MrScraper MCP Integration

Guide for retrieving web pages, searching the web, and checking account usage with PraisonAI agents using the hosted MrScraper MCP server

Add Web Scraping to AI Agent

Diagram

URL or Query

AI Agent

MrScraper MCP

Answer

Quick Start

  1. Install Dependencies

    Install PraisonAI with MCP support:

    bash
    pip install -U "praisonaiagents[mcp]>=1.7.12"

    praisonaiagents 1.7.12 and later load MrScraper’s tools over Streamable HTTP and install a compatible mcp 1.x.

  2. Set API Keys

    Create a MrScraper API key at app.mrscraper.com/api-tokens and set it with your model’s API key:

    bash
    export MRSCRAPER_API_KEY=your_mrscraper_api_key_here
    export OPENAI_API_KEY=your_openai_api_key_here
  3. Create a file

    Create a new file mrscraper_agent.py with the following code:

    python
    from praisonaiagents import Agent, MCP
    import os
    
    mrscraper_tools = MCP(
        "https://mcp.mrscraper.com/mcp",
        headers={"Authorization": f"Bearer {os.environ['MRSCRAPER_API_KEY']}"},
    )
    
    scraper_agent = Agent(
        instructions="""You research web pages with MrScraper.
        For a known URL, call scraperAPI and work from the page it returns;
        set markdown=true for readable text. Call webSearch when no URL
        is known, and status to check remaining tokens.""",
        model="gpt-4o-mini",
        tools=mrscraper_tools,
    )
    
    scraper_agent.start("Fetch https://www.scrapethissite.com/pages/simple/ and summarize the page.")
  4. Run the Agent

    Execute your script:

    bash
    python mrscraper_agent.py

Available Tools

ToolWhat it does
scraperAPIRetrieves a known page as HTML, Markdown, a screenshot, or a combination. Options include automatic browser fallback, real-device routing (super), geo targeting, proxies, and waiting for a selector.
webSearchSearches the web and returns ranked results (title, URL, description, source). Options include the results’ country (region) and the number of results (num_results, 10 to 100).
statusReports the account’s subscription, token usage, and remaining tokens.

Tool calls run on your MrScraper account and use its plan tokens. Only retrieve pages you are authorized to access.

CLI Configuration

Declare the server in praisonai.yaml at your project root to use it from praisonai run and praisonai code. ${MRSCRAPER_API_KEY} is read from the environment:

yaml
mcp:
  servers:
    mrscraper:
      type: remote
      url: https://mcp.mrscraper.com/mcp
      headers:
        Authorization: Bearer ${MRSCRAPER_API_KEY}

Check that the tools load:

bash
praisonai tools list