MrScraper MCP Integration
Guide for retrieving web pages, searching the web, and checking account usage with PraisonAI agents using the hosted MrScraper MCP server
Add Web Scraping to AI Agent
Quick Start
- Install Dependencies
Install PraisonAI with MCP support:
bash pip install -U "praisonaiagents[mcp]>=1.7.12"praisonaiagents1.7.12 and later load MrScraper’s tools over Streamable HTTP and install a compatiblemcp1.x. - Set API Keys
Create a MrScraper API key at app.mrscraper.com/api-tokens and set it with your model’s API key:
bash export MRSCRAPER_API_KEY=your_mrscraper_api_key_here export OPENAI_API_KEY=your_openai_api_key_here - Create a file
Create a new file
mrscraper_agent.pywith the following code:python from praisonaiagents import Agent, MCP import os mrscraper_tools = MCP( "https://mcp.mrscraper.com/mcp", headers={"Authorization": f"Bearer {os.environ['MRSCRAPER_API_KEY']}"}, ) scraper_agent = Agent( instructions="""You research web pages with MrScraper. For a known URL, call scraperAPI and work from the page it returns; set markdown=true for readable text. Call webSearch when no URL is known, and status to check remaining tokens.""", model="gpt-4o-mini", tools=mrscraper_tools, ) scraper_agent.start("Fetch https://www.scrapethissite.com/pages/simple/ and summarize the page.") - Run the Agent
Execute your script:
bash python mrscraper_agent.py
Available Tools
| Tool | What it does |
|---|---|
scraperAPI | Retrieves a known page as HTML, Markdown, a screenshot, or a combination. Options include automatic browser fallback, real-device routing (super), geo targeting, proxies, and waiting for a selector. |
webSearch | Searches the web and returns ranked results (title, URL, description, source). Options include the results’ country (region) and the number of results (num_results, 10 to 100). |
status | Reports the account’s subscription, token usage, and remaining tokens. |
Tool calls run on your MrScraper account and use its plan tokens. Only retrieve pages you are authorized to access.
CLI Configuration
Declare the server in praisonai.yaml at your project root to use it from praisonai run and praisonai code. ${MRSCRAPER_API_KEY} is read from the environment:
mcp:
servers:
mrscraper:
type: remote
url: https://mcp.mrscraper.com/mcp
headers:
Authorization: Bearer ${MRSCRAPER_API_KEY}Check that the tools load:
praisonai tools list