> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Spider Agent

> Web scraping tools for AI agents.

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent
from praisonaiagents import scrape_page, extract_links, crawl, extract_text

agent = Agent(
    name="Spider",
    tools=[scrape_page, extract_links, crawl, extract_text],
)
agent.start("Scrape the homepage and list outbound links")
```

The user names a URL or crawl task; the agent uses spider tools to fetch and summarise page content.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    U[Input] --> A[Agent]
    A --> T[Tool]
    T --> O[Output]

    classDef agent fill:#8B0000,color:#fff
    classDef tool fill:#189AB4,color:#fff

    class A agent
    class U,O tool
    class T tool
```

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
    participant User
    participant Agent
    participant Feature as Spider Agent

    User->>Agent: Request
    Agent->>Feature: Process request
    Feature-->>Agent: Result
    Agent-->>User: Response
```

<Note>
  **Prerequisites**

  * Python 3.10 or higher
  * PraisonAI Agents package installed
  * `scrapy` package installed
</Note>

## Spider Tools

Use Spider Tools to crawl and scrape web content with AI agents.

<Steps>
  <Step title="Install Dependencies">
    First, install the required packages:

    ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    pip install praisonaiagents scrapy
    ```
  </Step>

  <Step title="Import Components">
    Import the necessary components:

    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    from praisonaiagents import Agent, Task, AgentTeam
    from praisonaiagents import scrape_page, extract_links, crawl, extract_text
    ```
  </Step>

  <Step title="Create Agent">
    Create a web scraping agent:

    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    spider_agent = Agent(
        name="WebSpider",
        role="Web Scraping Specialist",
        goal="Extract and analyze web content efficiently.",
        backstory="Expert in web scraping and content extraction.",
        tools=[scrape_page, extract_links, crawl, extract_text],
        reflection=False
    )
    ```
  </Step>

  <Step title="Define Task">
    Define the scraping task:

    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    scraping_task = Task(
        description="Scrape product information from an e-commerce website.",
        expected_output="Structured product data with prices and descriptions.",
        agent=spider_agent,
        name="product_scraping"
    )
    ```
  </Step>

  <Step title="Run Agent">
    Initialize and run the agent:

    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    agents = AgentTeam(
        agents=[spider_agent],
        tasks=[scraping_task],
        process="sequential"
    )
    agents.start()
    ```
  </Step>
</Steps>

## Understanding Spider Tools

<Card title="What are Spider Tools?" icon="question">
  Spider Tools provide web scraping capabilities for AI agents:

  * Page scraping and downloading
  * Content extraction and filtering
  * Link discovery and crawling
  * HTML parsing and cleaning
  * Data structuring and formatting
</Card>

## Key Components

<CardGroup cols={2}>
  <Card title="Spider Agent" icon="user-robot">
    Create specialized scraping agents:

    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    Agent(tools=[scrape_page, extract_content, crawl_links, parse_html, structure_data])
    ```
  </Card>

  <Card title="Spider Task" icon="list-check">
    Define scraping tasks:

    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    Task(description="scraping_query")
    ```
  </Card>

  <Card title="Process Types" icon="arrows-split-up-and-left">
    Sequential or parallel processing:

    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    process="sequential"
    ```
  </Card>

  <Card title="Scraping Options" icon="sliders">
    Customize scraping parameters:

    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    max_depth=2, delay=1
    ```
  </Card>
</CardGroup>

## Examples

### Basic Web Scraping Agent

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents import Agent, Task, AgentTeam
from praisonaiagents import scrape_page, extract_links, crawl, extract_text

# Create search agent
agent = Agent(
    name="AsyncWebCrawler",
    role="Web Scraping Specialist",
    goal="Extract and analyze web content efficiently.",
    backstory="Expert in web scraping and content extraction.",
    tools=[scrape_page, extract_links, crawl, extract_text],
    reflection=False
)

# Define task
task = Task(
    description="Scrape and analyze the content from 'https://example.com'",
    expected_output="Extracted content with links and text analysis",
    agent=agent,
    name="web_scraping"
)

# Run agent
agents = AgentTeam(
    agents=[agent],
    tasks=[task],
    process="sequential"
)
agents.start()
```

### Advanced Scraping with Multiple Agents

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Create scraping agent
scraper_agent = Agent(
    name="Scraper",
    role="Content Scraper",
    goal="Extract web content systematically.",
    tools=[scrape_page, crawl_links, parse_html],
    reflection=False
)

# Create analysis agent
analysis_agent = Agent(
    name="Analyzer",
    role="Content Analyst",
    goal="Analyze and structure scraped content.",
    backstory="Expert in data analysis and organization.",
    tools=[extract_content, structure_data],
    reflection=False
)

# Define tasks
scraping_task = Task(
    description="Scrape product data from multiple pages.",
    agent=scraper_agent,
    name="product_scraping"
)

analysis_task = Task(
    description="Analyze and structure the scraped product data.",
    agent=analysis_agent,
    name="data_analysis"
)

# Run agents
agents = AgentTeam(
    agents=[scraper_agent, analysis_agent],
    tasks=[scraping_task, analysis_task],
    process="sequential"
)
agents.start()
```

## Best Practices

<AccordionGroup>
  <Accordion title="Agent Configuration">
    Configure agents with clear scraping focus:

    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    Agent(
        name="WebScraper",
        role="Content Extraction Specialist",
        goal="Extract web content ethically and efficiently",
        tools=[scrape_page, extract_content, crawl_links, parse_html, structure_data]
    )
    ```
  </Accordion>

  <Accordion title="Task Definition">
    Define specific scraping objectives:

    ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
    Task(
        description="Extract product prices and descriptions from e-commerce site",
        expected_output="Structured product database"
    )
    ```
  </Accordion>
</AccordionGroup>

## Built-in URL Safety

Spider tools (`scrape_page`, `extract_links`, `crawl`, `extract_text`) refuse to fetch dangerous URLs **before any network request is made**. You don't need to wrap them in a custom validator.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    URL([URL from agent]) --> V{Built-in<br/>_validate_url}
    V -->|Safe public URL| FETCH([HTTP request])
    V -->|Private / smuggled / control char| BLK([Refused — error returned])

    classDef input fill:#6366F1,stroke:#7C90A0,color:#fff
    classDef check fill:#F59E0B,stroke:#7C90A0,color:#fff
    classDef ok fill:#10B981,stroke:#7C90A0,color:#fff
    classDef bad fill:#8B0000,stroke:#7C90A0,color:#fff

    class URL input
    class V check
    class FETCH ok
    class BLK bad
```

### What gets refused

| URL pattern                              | Example                                                              | Why                                                                             |
| ---------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| Non-`http`/`https` schemes               | `file:///etc/passwd`, `gopher://x`                                   | Only web protocols are allowed                                                  |
| Loopback                                 | `http://127.0.0.1/`, `http://localhost/`                             | Blocks access to the local machine                                              |
| Trailing-dot localhost                   | `http://localhost.:8765/`                                            | Some HTTP clients strip the dot and hit loopback                                |
| Short-form IPv4                          | `http://127.1:8765/`, `http://127.1/`                                | `inet_aton` expands `127.1` → `127.0.0.1`                                       |
| Octal IPv4                               | `http://0177.0.0.1:8765/`                                            | `0177` = `127` in octal                                                         |
| Hex integer host                         | `http://0x7f000001:8765/`                                            | `0x7f000001` = `127.0.0.1` packed                                               |
| Decimal integer host                     | `http://2130706433:8765/`                                            | `2130706433` = `127.0.0.1` packed                                               |
| Internal `.localhost` suffix             | `http://api.localhost/`                                              | All `*.localhost` is loopback by RFC                                            |
| Private / reserved IPs                   | `http://10.0.0.5/`, `http://192.168.1.1/`                            | Blocks internal network access                                                  |
| Link-local                               | `http://169.254.169.254/`                                            | Blocks cloud metadata services                                                  |
| Internal TLDs                            | `http://intranet.local/`, `http://svc.internal/`                     | Blocks corporate internal hosts                                                 |
| Backslash in URL                         | `http://127.0.0.1:6666\@1.1.1.1`                                     | SSRF-smuggling: `urlparse` says `1.1.1.1`, `requests` actually hits `127.0.0.1` |
| ASCII control chars (`< 0x20` or `0x7f`) | `http://example.com\x00.evil.com`, `http://example.com\r\n.evil.com` | CRLF / NUL injection in the authority                                           |
| Non-string input                         | `None`, `123`                                                        | Defensive — returns `False` instead of raising                                  |

<Note>
  The backslash and control-character rejections (the last two rows above) were added in PraisonAI [#1578](https://github.com/MervinPraison/PraisonAI/pull/1578) to close an SSRF bypass where `urllib.parse.urlparse` and the HTTP client (`requests` / `httpx`) disagreed on the destination host.

  The alternative loopback encodings (trailing-dot localhost, short-form/octal/hex IPv4, etc.) are validated by the unified `_host_is_blocked` helper, which also powers URL safety for [`@url:` mentions](/docs/cli/mentions).
</Note>

### What it looks like to your agent

When the validator refuses a URL, the tool returns an error dict instead of fetching:

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
from praisonaiagents.tools import scrape_page

# Smuggled URL — looks like 1.1.1.1, would actually hit 127.0.0.1
scrape_page("http://127.0.0.1:6666\\@1.1.1.1")
# {'error': 'Invalid or potentially dangerous URL: http://127.0.0.1:6666\\@1.1.1.1'}

# Loopback
scrape_page("http://localhost/admin")
# {'error': 'Invalid or potentially dangerous URL: http://localhost/admin'}

# Cloud metadata endpoint
scrape_page("http://169.254.169.254/latest/meta-data/")
# {'error': 'Invalid or potentially dangerous URL: http://169.254.169.254/latest/meta-data/'}

# Normal public URL — works as expected
scrape_page("https://example.com/")
# {'url': 'https://example.com/', 'status_code': 200, 'content': '...', ...}
```

<Tip>
  This validation is **always on** for the bundled spider tools. It runs on every URL passed to `scrape_page`, `extract_links`, `crawl`, and `extract_text`. There is no flag to disable it, and it does not require `enable_security()`.
</Tip>

## Common Patterns

### Web Scraping Pipeline

```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
# Scraping agent
scraper = Agent(
    name="Scraper",
    role="Web Scraper",
    tools=[scrape_page, crawl_links, parse_html]
)

# Processing agent
processor = Agent(
    name="Processor",
    role="Data Processor",
    tools=[extract_content, structure_data]
)

# Define tasks
scrape_task = Task(
    description="Scrape website content",
    agent=scraper
)

process_task = Task(
    description="Process scraped content",
    agent=processor
)

# Run workflow
agents = AgentTeam(
    agents=[scraper, processor],
    tasks=[scrape_task, process_task]
)

## Related

<CardGroup cols={2}>
  <Card title="Custom Tools" icon="wrench" href="/docs/tools/custom">
    Build your own agent tools
  </Card>
  <Card title="Tools Overview" icon="toolbox" href="/docs/tools/tools">
    Browse PraisonAI tool documentation
  </Card>
</CardGroup>
```