Firecrawl

Web scraping and crawling with Firecrawl API

Firecrawl lets an agent scrape and crawl websites into clean, LLM-ready markdown.

Diagram

Overview

Firecrawl is a powerful web scraping API that converts websites into clean, LLM-ready markdown or structured data.

Installation

bash

pip install "praisonai[tools]"

Environment Variables

bash

export FIRECRAWL_API_KEY="${FIRECRAWL_API_KEY:?Set FIRECRAWL_API_KEY in your shell}"

Get your API key from Firecrawl.

How It Works

Diagram

Quick Start

Simple Usage

python

from praisonai_tools import FirecrawlTool

# Initialize
firecrawl = FirecrawlTool()

# Scrape a page
result = firecrawl.scrape("https://example.com")
print(result)

With Configuration
Use the same tool with an agent — see Usage with Agent below, or pass env vars and options from the sections above.

Usage with Agent

python

from praisonaiagents import Agent
from praisonai_tools import FirecrawlTool

agent = Agent(
    name="WebScraper",
    instructions="You are a web scraping assistant. Use Firecrawl to extract content.",
    tools=[FirecrawlTool()]
)

response = agent.chat("Scrape the content from https://praison.ai/docs")
print(response)

Available Methods

scrape(url)

Scrape a single URL and get markdown content.

python

from praisonai_tools import FirecrawlTool

firecrawl = FirecrawlTool()
result = firecrawl.scrape("https://example.com")

# Returns:
# {
#     "url": "https://example.com",
#     "markdown": "# Example Domain\n\nThis domain is...",
#     "metadata": {...}
# }

crawl(url, limit=10)

Crawl a website and get multiple pages.

python

results = firecrawl.crawl("https://docs.example.com", limit=5)

# Returns list of scraped pages

Configuration Options

python

firecrawl = FirecrawlTool(
    api_key="your_key",           # Optional: defaults to FIRECRAWL_API_KEY
    formats=["markdown", "html"], # Output formats
    only_main_content=True        # Extract only main content
)

Function-Based Usage

python

from praisonai_tools import firecrawl_scrape

# Quick scrape without instantiating class
result = firecrawl_scrape("https://example.com")

CLI Usage

bash

# Set API key
export FIRECRAWL_API_KEY=your_key

# Use with praisonai
praisonai --tools FirecrawlTool "Scrape the content from https://example.com"

Error Handling

python

from praisonai_tools import FirecrawlTool

firecrawl = FirecrawlTool()
result = firecrawl.scrape("https://example.com")

if "error" in result:
    print(f"Error: {result['error']}")
else:
    print(f"Content: {result['markdown'][:500]}")

Common Errors

Error	Cause	Solution
`FIRECRAWL_API_KEY not configured`	Missing API key	Set environment variable
`firecrawl not installed`	Missing dependency	Run `pip install firecrawl-py`
`Rate limited`	Too many requests	Upgrade plan or add delays

Best Practices

Let FIRECRAWL_API_KEY come from the environment

FirecrawlTool() defaults to the FIRECRAWL_API_KEY env var. Set it in your shell or .env rather than passing api_key= inline.

Limit crawl scope

crawl(url, limit=10) follows links up to the limit. Keep it small so the agent does not pull an entire site into context.

Handle rate limits

Firecrawl returns HTTP 429 when the plan quota is exceeded. Check for an error key in the result and back off or fall back to another scraper.

Related Tools

Crawl4AI

Open-source crawler

Tavily

Search-optimised web retrieval

Jina

Reader API