Skip to main content

​
Overview

Crawl4AI is an open-source, LLM-friendly web crawler that extracts clean content from websites. The user shares a URL; the agent crawls the page and returns clean, structured content.

​
Installation

pip install "praisonai[tools]"

​
Quick Start

1

Simple Usage

from praisonai_tools import Crawl4AITool

# Initialize
crawler = Crawl4AITool()

# Crawl
result = crawler.crawl("https://example.com")
print(result)
2

With Configuration

Use the same tool with an agent — see Usage with Agent below, or pass env vars and options from the sections above.

​
Usage with Agent

from praisonaiagents import Agent
from praisonai_tools import Crawl4AITool

agent = Agent(
    name="WebCrawler",
    instructions="You crawl websites and extract content.",
    tools=[Crawl4AITool()]
)

response = agent.chat("Crawl https://praison.ai/docs and summarize")
print(response)

​
Available Methods

​
crawl(url)

Crawl a URL and extract content. 
from praisonai_tools import Crawl4AITool

crawler = Crawl4AITool()
result = crawler.crawl("https://example.com")

​
Common Errors

ErrorCauseSolution
crawl4ai not installedMissing dependencyRun pip install crawl4ai
TimeoutPage too slowIncrease timeout

​
How It Works

​
Best Practices

Crawl only pages you are allowed to access. Check the site’s crawl policy before large runs.
Cap depth and page count so the agent processes focused content instead of an entire site.
Ask for structured extraction rather than raw HTML so the agent works with fewer, cleaner tokens.

Firecrawl

Web scraping API

Spider

Fast crawler

Jina

Reader API