Repository avatar
Search & Data Extraction
v0.7.10
active

mesharc

dev.mesharc/mesharc

Search the web, crawl and scrape websites to clean markdown, and track what changed between crawls.

Documentation

MeshArc web crawler

MeshArc turns any website into clean markdown your assistant can read, and keeps a record of what changed on it. It gives Claude, Cursor and any MCP client 23 tools to search the web, read pages, take a screenshot of a page, map and crawl sites, watch a site over time and ask what changed, hand a research question to an agent, and keep a search or a question on a schedule, all in plain English. Add it by URL, with nothing to install.

The free plan includes 1,000 credits with no card, and a plain fetch costs 1 credit.

https://mcp.mesharc.dev/mcp

Glama score MCP remote, streamable HTTP tools 23 PyPI license MIT

Contents

What can you do with it?

  • Search the web. Ranked results, each with its title, URL and snippet, read from Google's results page (DuckDuckGo's and then Bing's when Google refuses or reads short). Keep it to some sites or leave some out, and read each result's page as markdown in the same call.
  • Read any page as clean markdown. Pass one URL or up to 500. Navigation, cookie banners and boilerplate are stripped. PDF and Word links are read as text.
  • See a page as a person does. A screenshot of one page, desktop or mobile, after browser steps if needed, handed to the assistant as an image.
  • Get past walled sites. Each page climbs a fetch ladder: a plain request first, then a real browser, then stealth and residential exits only when the page needs them. You pay for the rung that worked. A page the site refuses comes back marked as blocked, and costs nothing.
  • Map a site before you fetch it. Every URL a site declares in its sitemaps, with its sections, without fetching a single page.
  • Crawl a whole site or one section. Use path globs such as /blog/*, with a page limit and a depth. You get an index of every page plus readable excerpts, sized to fit a chat turn.
  • Watch a site over time. Make it a project, put it on a schedule, and ask "what changed?". You get pages added, modified and removed, and field-level changes. Removals are held back when a run reached too little of the site, so a blocked crawl never reports a site as gone.
  • Run browser steps. Click "Load more" until it disappears, type into a search box, pick an option, then read the page.
  • Extract structured fields. Give a JSON schema and get the same fields from every page.
  • Ask a research question. An agent searches, maps sites and reads pages on its own, then answers with the pages it used, in the shape of a schema if you give one. You set its credit cap.
  • Keep a search or a question on a schedule. It runs again hourly, daily or weekly and tells you what is new, what dropped and what changed, down to each record of a list answer.

What you need

  • An MCP client that speaks remote MCP (Claude.ai, Claude Desktop, Claude Code, Cursor, VS Code, Windsurf…), or Python 3.10+ to run the server locally.
  • A MeshArc account. You sign in when you first connect, and the free plan needs no card.

Quick start

Hosted (recommended). Nothing to install, and no key to copy: you sign in to mesharc.dev and approve once. The connection is read-only unless you tick write access.

FieldValue
URLhttps://mcp.mesharc.dev/mcp
TransportHTTP, streamable
AuthOAuth: sign in and approve in the browser

Claude.ai: Settings → Connectors → Add custom connector → paste the URL and sign in.

Claude Code:

claude mcp add --transport http mesharc https://mcp.mesharc.dev/mcp

Cursor, Windsurf and other clients (mcp.json):

{
  "mcpServers": {
    "mesharc": { "url": "https://mcp.mesharc.dev/mcp" }
  }
}

Local (stdio), with your own API key. Make a key under Settings → API keys on mesharc.dev.

pip install "mesharc[mcp]"
claude mcp add mesharc -e MESHARC_API_KEY=mesharc_... -- mesharc-mcp

Claude Desktop (claude_desktop_config.json) or Cursor:

{
  "mcpServers": {
    "mesharc": {
      "command": "mesharc-mcp",
      "env": { "MESHARC_API_KEY": "mesharc_..." }
    }
  }
}

Example prompts

These conversations were run against real sites; the results are the real ones.

  • "Get Linear's pricing page and list every tier and price." The assistant used scrape_urls: 1 credit and 4 seconds for the four tiers.
  • "How many pages does the FastAPI docs site have, and which are about security?" The assistant used map_site: 151 URLs declared, 9 under /tutorial/security/, with no page fetched.
  • "Set up a project for hotspotseo.com that only scrapes the blog, weekly." The assistant used get_project, map_site and create_project. It found the blog under /blogs/, not /blog/, and the first run read only blog pages.
  • "Make it daily instead, and skip the tag and author pages." The assistant used update_project; the include setting stayed as it was.
  • "Re-check the home page and page 2 now. What changed?" The assistant used start_run with the two URLs, then get_changes.
  • "Read this arXiv paper (a PDF) and summarise the method." The assistant used scrape_urls: 6,036 words of text for 2 credits.
  • "Which of these pages mention GDPR?" The assistant used list_pages with a search, on the stored crawl, at no credit cost.

Tools

Read the web

ToolWhat it doesCredits
scrape_urlsContent of 1 to 500 known URLs; PDF and Word files are read as text. full=true reads one URL with every format: raw html, head and extracted fields, images, and browser steps run first.per page
screenshot_urlA picture of one page (JPEG, the visible window) as image content, with its title and cost: viewport desktop or mobile, browser actions first, a wait_ms. No picture comes back as no_screenshot with the browser's reason.5 (4 for the browser, 1 for the picture); a refused page is free
map_siteEvery URL a site declares in its sitemaps, with its sections, without fetching a page.1 per sitemap file
crawl_siteCrawl a site or a section once, with page limit, depth and path globs. You get an index plus excerpts.per page
search_webRanked results for a query (title, URL, snippet) from Google's results page, or DuckDuckGo's and then Bing's when Google refuses or reads short. include_domains / exclude_domains keep it to sites; scrape=true also reads each result's page as markdown.At most 4 per search (1 when DuckDuckGo or Bing answers alone); a refused page is free and an equal search within an hour comes from cache; scraped pages per page

Watch a site over time

ToolWhat it doesCredits
list_projectsThe workspace's projects, with status and last run.none
get_projectOne project's settings and state. With no id, every setting an assistant can set, with its meaning and default.none
create_projectA watched project from a URL, with any settings, or from a crawl_site crawl with nothing fetched again.none
update_projectChange the name, schedule or settings. Only the keys you give change.none
delete_projectDelete a project and its history once its name is repeated back. Needs an admin API key, so it works locally only.none
archive_projectArchive a project once its name is repeated back: it stops running, no longer counts toward the plan's projects, and is deleted after 30 days unless restored in the app. Works for a connected app, which cannot delete.none
start_runCrawl a project now, or only the pages listed, compared with the last full run. You can wait for it to finish.per page
list_runsA project's runs, newest first.none
list_pagesA run's pages: url, status, depth, words, when changed. With q, a search of them: words, a quoted phrase, or a CSS selector. With url, one stored page in full, with its versions across runs.none
get_changesPages added, modified and removed since the run before, with field changes.none

Research and monitors

ToolWhat it doesCredits
run_agentHand a research question to an agent that searches, maps sites and reads pages on its own, then answers with the sources it used; a schema shapes the answer. It comes back as a job.pages as read, plus the model's tokens at cost + 20%, within max_credits (2,000 by default)
continue_agentCarry on an agent run that stopped at its credit limit, with a new budget; the pages it read are not paid for again.as run_agent
create_monitorKeep a web search or an agent question and run it again hourly, daily or weekly, each run compared with the last: new, dropped and moved results, or the answer's changed values.each run as the search or agent run it is
list_monitorsThe workspace's monitors with schedule, spend and what the last run found; or one monitor with its runs.none
pause_monitorPause a monitor, or resume it. Nothing is deleted.none
get_monitor_rowsAn agent monitor's records, one per item its answer lists, and what moved: added, changed (before and after), gone.none

Long jobs

ToolWhat it doesCredits
get_jobFollow a crawl, run, batch, web search or agent run that came back as a job, and get its result.none
cancel_jobStop a crawl, run, batch or agent run that is still going. Pages already read stay.none

Every tool has an output schema describing its answer. Read-only tools are marked read-only, and the four that overwrite, archive, stop or delete are marked destructive.

Errors and failure paths

  • Blocked is not missing. A page the site refuses is reported as blocked (or captcha, timeout, missing for a 404 or 410), never as an empty page, and none of them is charged.
  • Read-only connections: any tool that fetches or changes something answers code: read_only, with how to reconnect with write access.
  • Long work: a crawl, run, batch or web search that outlasts the connection's time limit comes back as a job. get_job picks it up, and calling the original tool again returns the same job rather than starting a second one.
  • Out of credits: 402. A run that uses up the budget stops and keeps what it read.
  • A run already going: start_run answers 409; list_runs shows it and cancel_job stops it.
  • Plan limits: create_project answers plan_limit when the plan's project count is full; values above a plan's caps are lowered to them.
  • Settings: an unknown config key is refused by name, never silently dropped.
  • Size: a many-page answer is an index plus excerpts inside 60,000 characters; any one page comes back whole (each body up to 12,000 characters) on request.

Pricing, free tier and limits

  • Free: 1,000 credits once, with no card. 2 projects, 500 pages per run, and 7-day retention.
  • Paid plans: Starter $29/mo (100,000 credits), Growth $99/mo (500,000 credits, stealth tier), Scale $349/mo (2,500,000 credits), and Enterprise.
  • A credit: a plain fetch costs 1, a fetch with a site's session 2, a real Chrome 4, a residential address 16, a solved widget 22.
  • Free of charge: a refused page, a 404 or 410, a cached page, a results page the engines refused, change detection, and listing, searching and reading what is stored.
  • The MCP server itself costs nothing extra. A tool call is an ordinary request through your account, with the same credits, limits and rate.

Tool selection

  • You have the URLs → scrape_urls. You need to find them on one site → map_site (cheap, declared URLs only) or crawl_site (follows links). You need them from the whole web → search_web.
  • One page with clicks, structured fields or every format → scrape_urls with full=true.
  • What a page looks like (its layout, a chart, a banner) → screenshot_url.
  • You want to know what changes over time → create_project, then start_run and get_changes.
  • You want to read what was already crawled, at no cost → list_pages (with q to search it, with url for one page in full), get_changes.
  • A question that needs several pages or sites to answer → run_agent. When you know what to fetch, search_web and scrape_urls are quicker and cheaper.
  • You want to keep watching a search or a question (mentions, rankings, a competitor's prices) → create_monitor. To watch a whole site's pages, use a project.

How it compares

FirecrawlMeshArc
Blocked or error response billedYesNo
404 billedYesNo
Cheaper repeat reads of protected sitesNoYes, session reuse at 2 credits
Website change monitoringYes (Monitor)Yes, with sitemap sections and held-back removals
Web search APIYesYes (search_web: Google, then DuckDuckGo and Bing)
Autonomous agent / browser interactionYesYes (run_agent, and browser steps)
Open sourceYesNo

Checked against public documentation and pricing pages.

FAQ

Does the assistant see my API key?

No. Hosted, you approve the app and MeshArc issues it a key of its own, which you can see and revoke under Settings → API keys. Run locally, the key stays in the server's environment on your machine.

Can it read pages behind a login?

No. It reads what a visitor can see. A login wall comes back as blocked, at no cost.

Does it respect robots.txt?

Yes, by default (respect_robots).

Can it change my projects?

Only if you allowed write access when you connected it. Deleting a project also needs an admin API key, which a connected app never holds.

Which clients work?

Any client that speaks remote MCP over streamable HTTP, or any client that runs a stdio server (mesharc-mcp).

Source

The server is mesharc/mcp.py in mesharc-org/mesharc-python, published on PyPI as mesharc, and the hosted one runs at https://mcp.mesharc.dev/mcp. This repository runs it: server.py starts it from that package, and the Dockerfile builds it.

docker build -t mesharc-mcp .
docker run -i --rm -e MESHARC_API_KEY=mesharc_... mesharc-mcp

Report issues with the server in this repository or in mesharc-python.

Links

License

MIT. The SDK and the MCP server are MIT licensed too.

No installation packages available.

Remote