Spider Agent

Web scraping tools for AI agents.

python

from praisonaiagents import Agent
from praisonaiagents import scrape_page, extract_links, crawl, extract_text

agent = Agent(
    name="Spider",
    tools=[scrape_page, extract_links, crawl, extract_text],
)
agent.start("Scrape the homepage and list outbound links")

The user names a URL or crawl task; the agent uses spider tools to fetch and summarise page content.

Diagram

Spider Tools

Use Spider Tools to crawl and scrape web content with AI agents.

Install Dependencies
First, install the required packages:
bash
```
pip install praisonaiagents scrapy
```

Import Components

Import the necessary components:

python

from praisonaiagents import Agent, Task, AgentTeam
from praisonaiagents import scrape_page, extract_links, crawl, extract_text

Create Agent

Create a web scraping agent:

python

spider_agent = Agent(
    name="WebSpider",
    role="Web Scraping Specialist",
    goal="Extract and analyze web content efficiently.",
    backstory="Expert in web scraping and content extraction.",
    tools=[scrape_page, extract_links, crawl, extract_text],
    reflection=False
)

Define Task

Define the scraping task:

python

scraping_task = Task(
    description="Scrape product information from an e-commerce website.",
    expected_output="Structured product data with prices and descriptions.",
    agent=spider_agent,
    name="product_scraping"
)

Run Agent

Initialize and run the agent:

python

agents = AgentTeam(
    agents=[spider_agent],
    tasks=[scraping_task],
    process="sequential"
)
agents.start()

Understanding Spider Tools

What are Spider Tools?

Spider Tools provide web scraping capabilities for AI agents: – Page scraping and downloading – Content extraction and filtering – Link discovery and crawling – HTML parsing and cleaning – Data structuring and formatting

Key Components

Spider Agent

Create specialized scraping agents:

python

Agent(tools=[scrape_page, extract_content, crawl_links, parse_html, structure_data])

Spider Task

Define scraping tasks:

python

Task(description="scraping_query")

Process Types

Sequential or parallel processing:

python

process="sequential"

Scraping Options

Customize scraping parameters:

python

max_depth=2, delay=1

Examples

Basic Web Scraping Agent

python

from praisonaiagents import Agent, Task, AgentTeam
from praisonaiagents import scrape_page, extract_links, crawl, extract_text

# Create search agent
agent = Agent(
    name="AsyncWebCrawler",
    role="Web Scraping Specialist",
    goal="Extract and analyze web content efficiently.",
    backstory="Expert in web scraping and content extraction.",
    tools=[scrape_page, extract_links, crawl, extract_text],
    reflection=False
)

# Define task
task = Task(
    description="Scrape and analyze the content from 'https://example.com'",
    expected_output="Extracted content with links and text analysis",
    agent=agent,
    name="web_scraping"
)

# Run agent
agents = AgentTeam(
    agents=[agent],
    tasks=[task],
    process="sequential"
)
agents.start()

Advanced Scraping with Multiple Agents

python

# Create scraping agent
scraper_agent = Agent(
    name="Scraper",
    role="Content Scraper",
    goal="Extract web content systematically.",
    tools=[scrape_page, crawl_links, parse_html],
    reflection=False
)

# Create analysis agent
analysis_agent = Agent(
    name="Analyzer",
    role="Content Analyst",
    goal="Analyze and structure scraped content.",
    backstory="Expert in data analysis and organization.",
    tools=[extract_content, structure_data],
    reflection=False
)

# Define tasks
scraping_task = Task(
    description="Scrape product data from multiple pages.",
    agent=scraper_agent,
    name="product_scraping"
)

analysis_task = Task(
    description="Analyze and structure the scraped product data.",
    agent=analysis_agent,
    name="data_analysis"
)

# Run agents
agents = AgentTeam(
    agents=[scraper_agent, analysis_agent],
    tasks=[scraping_task, analysis_task],
    process="sequential"
)
agents.start()

Best Practices

Agent Configuration

Configure agents with clear scraping focus:

python

Agent(
    name="WebScraper",
    role="Content Extraction Specialist",
    goal="Extract web content ethically and efficiently",
    tools=[scrape_page, extract_content, crawl_links, parse_html, structure_data]
)

Task Definition

Define specific scraping objectives:

python

Task(
    description="Extract product prices and descriptions from e-commerce site",
    expected_output="Structured product database"
)

Built-in URL Safety

Spider tools (scrape_page, extract_links, crawl, extract_text) refuse to fetch dangerous URLs before any network request is made. You don’t need to wrap them in a custom validator.

Diagram

What gets refused

URL pattern	Example	Why
Non-`http`/`https` schemes	`file:///etc/passwd`, `gopher://x`	Only web protocols are allowed
Loopback	`http://127.0.0.1/`, `http://localhost/`	Blocks access to the local machine
Trailing-dot localhost	`http://localhost.:8765/`	Some HTTP clients strip the dot and hit loopback
Short-form IPv4	`http://127.1:8765/`, `http://127.1/`	`inet_aton` expands `127.1` → `127.0.0.1`
Octal IPv4	`http://0177.0.0.1:8765/`	`0177` = `127` in octal — `inet_aton` reads this as loopback
Octal that only the resolver reads as private	`http://010.0.0.1:8765/`	Some libcs read `010.0.0.1` as `10.0.0.1` (RFC1918) via `getaddrinfo`, though `inet_aton` reads it as public `8.0.0.1`. Both readings are consulted; either being internal blocks.
Hex integer host	`http://0x7f000001:8765/`	`0x7f000001` = `127.0.0.1` packed
Decimal integer host	`http://2130706433:8765/`	`2130706433` = `127.0.0.1` packed
Internal `.localhost` suffix	`http://api.localhost/`	All `*.localhost` is loopback by RFC
Private / reserved IPs	`http://10.0.0.5/`, `http://192.168.1.1/`	Blocks internal network access
Link-local	`http://169.254.169.254/`	Blocks cloud metadata services
Internal TLDs	`http://intranet.local/`, `http://svc.internal/`	Blocks corporate internal hosts
Backslash in URL	`http://127.0.0.1:6666\@1.1.1.1`	SSRF-smuggling: `urlparse` says `1.1.1.1`, `requests` actually hits `127.0.0.1`
ASCII control chars (`< 0x20` or `0x7f`)	`http://example.com\x00.evil.com`, `http://example.com\r\n.evil.com`	CRLF / NUL injection in the authority
Non-string input	`None`, `123`	Defensive — returns `False` instead of raising

Note

The backslash and control-character rejections (the last two rows above) were added in PraisonAI #1578 to close an SSRF bypass where urllib.parse.urlparse and the HTTP client (requests / httpx) disagreed on the destination host.

The alternative loopback encodings (trailing-dot localhost, short-form/octal/hex IPv4, etc.) are validated by the unified _host_is_blocked helper, which also powers URL safety for @url: mentions.

_host_is_blocked consults two independent readings of the host — socket.inet_aton (what libc-based HTTP clients dial) and socket.getaddrinfo (what a resolver returns) — and blocks if either yields an internal address. ipaddress.ip_address refuses ambiguous forms like 0177.0.0.1 (leading zeros) and 127.1 (short form) outright, so those would otherwise fall through to only the resolver — which some libcs read as the public 177.0.0.1, letting them dial loopback. Consulting both readings closes that gap in both directions (GHSA-5c6w-wwfq-7qqm, fixed in PR #4966).

The mentions handler (praisonaiagents.tools.mentions) uses this validator on every redirect hop, not just the first. Prior versions only validated the initial URL, so a redirect to 169.254.169.254 could exfiltrate cloud metadata. See Crawl4AI — URL Safety.

What it looks like to your agent

When the validator refuses a URL, the tool returns an error dict instead of fetching:

python

from praisonaiagents.tools import scrape_page

# Smuggled URL — looks like 1.1.1.1, would actually hit 127.0.0.1
scrape_page("http://127.0.0.1:6666\\@1.1.1.1")
# {'error': 'Invalid or potentially dangerous URL: http://127.0.0.1:6666\\@1.1.1.1'}

# Loopback
scrape_page("http://localhost/admin")
# {'error': 'Invalid or potentially dangerous URL: http://localhost/admin'}

# Cloud metadata endpoint
scrape_page("http://169.254.169.254/latest/meta-data/")
# {'error': 'Invalid or potentially dangerous URL: http://169.254.169.254/latest/meta-data/'}

# Octal-encoded loopback — 0177 = 127 in octal, dials 127.0.0.1
scrape_page("http://0177.0.0.1:8765/")
# {'error': 'Invalid or potentially dangerous URL: http://0177.0.0.1:8765/'}

# Normal public URL — works as expected
scrape_page("https://example.com/")
# {'url': 'https://example.com/', 'status_code': 200, 'content': '...', ...}

Common Patterns

Web Scraping Pipeline

python

# Scraping agent
scraper = Agent(
    name="Scraper",
    role="Web Scraper",
    tools=[scrape_page, crawl_links, parse_html]
)

# Processing agent
processor = Agent(
    name="Processor",
    role="Data Processor",
    tools=[extract_content, structure_data]
)

# Define tasks
scrape_task = Task(
    description="Scrape website content",
    agent=scraper
)

process_task = Task(
    description="Process scraped content",
    agent=processor
)

# Run workflow
agents = AgentTeam(
    agents=[scraper, processor],
    tasks=[scrape_task, process_task]
)

Custom Tools

Build your own agent tools

Tools Overview

Browse PraisonAI tool documentation

Spider Agent

Spider Tools

Understanding Spider Tools

Key Components

Examples

Basic Web Scraping Agent

Advanced Scraping with Multiple Agents

Best Practices

Built-in URL Safety

What gets refused

What it looks like to your agent

Common Patterns

Web Scraping Pipeline

Related