Crawl4AI
Open-source web crawler for AI applications
Overview
Crawl4AI is an open-source, LLM-friendly web crawler that extracts clean content from websites.
The user shares a URL; the agent crawls the page and returns clean, structured content.
Installation
pip install "praisonai[tools]"Quick Start
- Simple Usage
python from praisonai_tools import Crawl4AITool # Initialize crawler = Crawl4AITool() # Crawl result = crawler.crawl("https://example.com") print(result) - With Configuration
Use the same tool with an agent — see Usage with Agent below, or pass env vars and options from the sections above.
Usage with Agent
from praisonaiagents import Agent
from praisonai_tools import Crawl4AITool
agent = Agent(
name="WebCrawler",
instructions="You crawl websites and extract content.",
tools=[Crawl4AITool()]
)
response = agent.chat("Crawl https://praison.ai/docs and summarize")
print(response)Available Methods
crawl(url)
Crawl a URL and extract content.
from praisonai_tools import Crawl4AITool
crawler = Crawl4AITool()
result = crawler.crawl("https://example.com")Common Errors
| Error | Cause | Solution |
|---|---|---|
crawl4ai not installed | Missing dependency | Run pip install crawl4ai |
Timeout | Page too slow | Increase timeout |
How It Works
Best Practices
Respect robots.txt
Crawl only pages you are allowed to access. Check the site’s crawl policy before large runs.
Limit crawl depth
Cap depth and page count so the agent processes focused content instead of an entire site.
Extract, don't dump
Ask for structured extraction rather than raw HTML so the agent works with fewer, cleaner tokens.