
mesharc
dev.mesharc/mesharc
Search the web, crawl and scrape websites to clean markdown, and track what changed between crawls.
Documentation
MeshArc web crawler
MeshArc turns any website into clean markdown your assistant can read, and keeps a record of what changed on it. It gives Claude, Cursor and any MCP client 23 tools to search the web, read pages, take a screenshot of a page, map and crawl sites, watch a site over time and ask what changed, hand a research question to an agent, and keep a search or a question on a schedule, all in plain English. Add it by URL, with nothing to install.
The free plan includes 1,000 credits with no card, and a plain fetch costs 1 credit.
https://mcp.mesharc.dev/mcp
Contents
- What can you do with it?
- What you need
- Quick start
- Example prompts
- Tools
- Errors and failure paths
- Pricing, free tier and limits
- Tool selection
- How it compares
- FAQ
- Source
- Links
- License
What can you do with it?
- Search the web. Ranked results, each with its title, URL and snippet, read from Google's results page (DuckDuckGo's and then Bing's when Google refuses or reads short). Keep it to some sites or leave some out, and read each result's page as markdown in the same call.
- Read any page as clean markdown. Pass one URL or up to 500. Navigation, cookie banners and boilerplate are stripped. PDF and Word links are read as text.
- See a page as a person does. A screenshot of one page, desktop or mobile, after browser steps if needed, handed to the assistant as an image.
- Get past walled sites. Each page climbs a fetch ladder: a plain request first, then a real browser, then stealth and residential exits only when the page needs them. You pay for the rung that worked. A page the site refuses comes back marked as blocked, and costs nothing.
- Map a site before you fetch it. Every URL a site declares in its sitemaps, with its sections, without fetching a single page.
- Crawl a whole site or one section. Use path globs such as
/blog/*, with a page limit and a depth. You get an index of every page plus readable excerpts, sized to fit a chat turn. - Watch a site over time. Make it a project, put it on a schedule, and ask "what changed?". You get pages added, modified and removed, and field-level changes. Removals are held back when a run reached too little of the site, so a blocked crawl never reports a site as gone.
- Run browser steps. Click "Load more" until it disappears, type into a search box, pick an option, then read the page.
- Extract structured fields. Give a JSON schema and get the same fields from every page.
- Ask a research question. An agent searches, maps sites and reads pages on its own, then answers with the pages it used, in the shape of a schema if you give one. You set its credit cap.
- Keep a search or a question on a schedule. It runs again hourly, daily or weekly and tells you what is new, what dropped and what changed, down to each record of a list answer.
What you need
- An MCP client that speaks remote MCP (Claude.ai, Claude Desktop, Claude Code, Cursor, VS Code, Windsurf…), or Python 3.10+ to run the server locally.
- A MeshArc account. You sign in when you first connect, and the free plan needs no card.
Quick start
Hosted (recommended). Nothing to install, and no key to copy: you sign in to mesharc.dev and approve once. The connection is read-only unless you tick write access.
| Field | Value |
|---|---|
| URL | https://mcp.mesharc.dev/mcp |
| Transport | HTTP, streamable |
| Auth | OAuth: sign in and approve in the browser |
Claude.ai: Settings → Connectors → Add custom connector → paste the URL and sign in.
Claude Code:
claude mcp add --transport http mesharc https://mcp.mesharc.dev/mcp
Cursor, Windsurf and other clients (mcp.json):
{
"mcpServers": {
"mesharc": { "url": "https://mcp.mesharc.dev/mcp" }
}
}
Local (stdio), with your own API key. Make a key under Settings → API keys on mesharc.dev.
pip install "mesharc[mcp]"
claude mcp add mesharc -e MESHARC_API_KEY=mesharc_... -- mesharc-mcp
Claude Desktop (claude_desktop_config.json) or Cursor:
{
"mcpServers": {
"mesharc": {
"command": "mesharc-mcp",
"env": { "MESHARC_API_KEY": "mesharc_..." }
}
}
}
Example prompts
These conversations were run against real sites; the results are the real ones.
- "Get Linear's pricing page and list every tier and price." The assistant used
scrape_urls: 1 credit and 4 seconds for the four tiers. - "How many pages does the FastAPI docs site have, and which are about security?" The assistant used
map_site: 151 URLs declared, 9 under /tutorial/security/, with no page fetched. - "Set up a project for hotspotseo.com that only scrapes the blog, weekly." The assistant used
get_project,map_siteandcreate_project. It found the blog under /blogs/, not /blog/, and the first run read only blog pages. - "Make it daily instead, and skip the tag and author pages." The assistant used
update_project; the include setting stayed as it was. - "Re-check the home page and page 2 now. What changed?" The assistant used
start_runwith the two URLs, thenget_changes. - "Read this arXiv paper (a PDF) and summarise the method." The assistant used
scrape_urls: 6,036 words of text for 2 credits. - "Which of these pages mention GDPR?" The assistant used
list_pageswith a search, on the stored crawl, at no credit cost.
Tools
Read the web
| Tool | What it does | Credits |
|---|---|---|
scrape_urls | Content of 1 to 500 known URLs; PDF and Word files are read as text. full=true reads one URL with every format: raw html, head and extracted fields, images, and browser steps run first. | per page |
screenshot_url | A picture of one page (JPEG, the visible window) as image content, with its title and cost: viewport desktop or mobile, browser actions first, a wait_ms. No picture comes back as no_screenshot with the browser's reason. | 5 (4 for the browser, 1 for the picture); a refused page is free |
map_site | Every URL a site declares in its sitemaps, with its sections, without fetching a page. | 1 per sitemap file |
crawl_site | Crawl a site or a section once, with page limit, depth and path globs. You get an index plus excerpts. | per page |
search_web | Ranked results for a query (title, URL, snippet) from Google's results page, or DuckDuckGo's and then Bing's when Google refuses or reads short. include_domains / exclude_domains keep it to sites; scrape=true also reads each result's page as markdown. | At most 4 per search (1 when DuckDuckGo or Bing answers alone); a refused page is free and an equal search within an hour comes from cache; scraped pages per page |
Watch a site over time
| Tool | What it does | Credits |
|---|---|---|
list_projects | The workspace's projects, with status and last run. | none |
get_project | One project's settings and state. With no id, every setting an assistant can set, with its meaning and default. | none |
create_project | A watched project from a URL, with any settings, or from a crawl_site crawl with nothing fetched again. | none |
update_project | Change the name, schedule or settings. Only the keys you give change. | none |
delete_project | Delete a project and its history once its name is repeated back. Needs an admin API key, so it works locally only. | none |
archive_project | Archive a project once its name is repeated back: it stops running, no longer counts toward the plan's projects, and is deleted after 30 days unless restored in the app. Works for a connected app, which cannot delete. | none |
start_run | Crawl a project now, or only the pages listed, compared with the last full run. You can wait for it to finish. | per page |
list_runs | A project's runs, newest first. | none |
list_pages | A run's pages: url, status, depth, words, when changed. With q, a search of them: words, a quoted phrase, or a CSS selector. With url, one stored page in full, with its versions across runs. | none |
get_changes | Pages added, modified and removed since the run before, with field changes. | none |
Research and monitors
| Tool | What it does | Credits |
|---|---|---|
run_agent | Hand a research question to an agent that searches, maps sites and reads pages on its own, then answers with the sources it used; a schema shapes the answer. It comes back as a job. | pages as read, plus the model's tokens at cost + 20%, within max_credits (2,000 by default) |
continue_agent | Carry on an agent run that stopped at its credit limit, with a new budget; the pages it read are not paid for again. | as run_agent |
create_monitor | Keep a web search or an agent question and run it again hourly, daily or weekly, each run compared with the last: new, dropped and moved results, or the answer's changed values. | each run as the search or agent run it is |
list_monitors | The workspace's monitors with schedule, spend and what the last run found; or one monitor with its runs. | none |
pause_monitor | Pause a monitor, or resume it. Nothing is deleted. | none |
get_monitor_rows | An agent monitor's records, one per item its answer lists, and what moved: added, changed (before and after), gone. | none |
Long jobs
| Tool | What it does | Credits |
|---|---|---|
get_job | Follow a crawl, run, batch, web search or agent run that came back as a job, and get its result. | none |
cancel_job | Stop a crawl, run, batch or agent run that is still going. Pages already read stay. | none |
Every tool has an output schema describing its answer. Read-only tools are marked read-only, and the four that overwrite, archive, stop or delete are marked destructive.
Errors and failure paths
- Blocked is not missing. A page the site refuses is reported as
blocked(orcaptcha,timeout,missingfor a 404 or 410), never as an empty page, and none of them is charged. - Read-only connections: any tool that fetches or changes something answers
code: read_only, with how to reconnect with write access. - Long work: a crawl, run, batch or web search that outlasts the connection's time limit comes back as a job.
get_jobpicks it up, and calling the original tool again returns the same job rather than starting a second one. - Out of credits:
402. A run that uses up the budget stops and keeps what it read. - A run already going:
start_runanswers409;list_runsshows it andcancel_jobstops it. - Plan limits:
create_projectanswersplan_limitwhen the plan's project count is full; values above a plan's caps are lowered to them. - Settings: an unknown config key is refused by name, never silently dropped.
- Size: a many-page answer is an index plus excerpts inside 60,000 characters; any one page comes back whole (each body up to 12,000 characters) on request.
Pricing, free tier and limits
- Free: 1,000 credits once, with no card. 2 projects, 500 pages per run, and 7-day retention.
- Paid plans: Starter $29/mo (100,000 credits), Growth $99/mo (500,000 credits, stealth tier), Scale $349/mo (2,500,000 credits), and Enterprise.
- A credit: a plain fetch costs 1, a fetch with a site's session 2, a real Chrome 4, a residential address 16, a solved widget 22.
- Free of charge: a refused page, a 404 or 410, a cached page, a results page the engines refused, change detection, and listing, searching and reading what is stored.
- The MCP server itself costs nothing extra. A tool call is an ordinary request through your account, with the same credits, limits and rate.
Tool selection
- You have the URLs →
scrape_urls. You need to find them on one site →map_site(cheap, declared URLs only) orcrawl_site(follows links). You need them from the whole web →search_web. - One page with clicks, structured fields or every format →
scrape_urlswithfull=true. - What a page looks like (its layout, a chart, a banner) →
screenshot_url. - You want to know what changes over time →
create_project, thenstart_runandget_changes. - You want to read what was already crawled, at no cost →
list_pages(withqto search it, withurlfor one page in full),get_changes. - A question that needs several pages or sites to answer →
run_agent. When you know what to fetch,search_webandscrape_urlsare quicker and cheaper. - You want to keep watching a search or a question (mentions, rankings, a competitor's prices) →
create_monitor. To watch a whole site's pages, use a project.
How it compares
| Firecrawl | MeshArc | |
|---|---|---|
| Blocked or error response billed | Yes | No |
| 404 billed | Yes | No |
| Cheaper repeat reads of protected sites | No | Yes, session reuse at 2 credits |
| Website change monitoring | Yes (Monitor) | Yes, with sitemap sections and held-back removals |
| Web search API | Yes | Yes (search_web: Google, then DuckDuckGo and Bing) |
| Autonomous agent / browser interaction | Yes | Yes (run_agent, and browser steps) |
| Open source | Yes | No |
Checked against public documentation and pricing pages.
FAQ
Does the assistant see my API key?
No. Hosted, you approve the app and MeshArc issues it a key of its own, which you can see and revoke under Settings → API keys. Run locally, the key stays in the server's environment on your machine.
Can it read pages behind a login?
No. It reads what a visitor can see. A login wall comes back as blocked, at no cost.
Does it respect robots.txt?
Yes, by default (respect_robots).
Can it change my projects?
Only if you allowed write access when you connected it. Deleting a project also needs an admin API key, which a connected app never holds.
Which clients work?
Any client that speaks remote MCP over streamable HTTP, or any client that runs a stdio server (mesharc-mcp).
Source
The server is mesharc/mcp.py in mesharc-org/mesharc-python, published on PyPI as mesharc, and the hosted one runs at https://mcp.mesharc.dev/mcp. This repository runs it: server.py starts it from that package, and the Dockerfile builds it.
docker build -t mesharc-mcp .
docker run -i --rm -e MESHARC_API_KEY=mesharc_... mesharc-mcp
Report issues with the server in this repository or in mesharc-python.
Links
- Website: https://mesharc.dev
- MCP page and setup per client: https://mesharc.dev/mcp
- Documentation: https://mesharc.dev/docs
- Python SDK and MCP server: https://github.com/mesharc-org/mesharc-python · https://pypi.org/project/mesharc/
- Node SDK: https://github.com/mesharc-org/mesharc-node
- Privacy: https://mesharc.dev/legal/privacy
- Support: hello@mesharc.dev
License
MIT. The SDK and the MCP server are MIT licensed too.
No installation packages available.