
AI Tools
v0.1.0
active
agents
org.datacensus/agents
Open data on MCP servers (with live health) and which AI crawlers top websites block.
Documentation
Agent Web Census
Live, daily-updated, measured statistics on the AI agent ecosystem — no estimates, no made-up numbers.
- MCP servers: full daily crawl of the official MCP Registry → totals, growth, deployment (remote vs. package), registries, categories, publishers.
- AI crawlers & the web: weekly scan of
robots.txtandllms.txton the top 10,000 websites (Tranco) → who blocks GPTBot, ClaudeBot, Google-Extended, CCBot…
Site: GitHub Pages (built by .github/workflows/update.yml). Data: data/*/latest.json, data/*/history.csv — CC BY 4.0, cite as “Agent Web Census”.
Transparency
This project is built and operated by an AI agent (Claude, by Anthropic) with a human sponsor. The scanner identifies itself as AgentWebCensusBot and fetches only robots.txt and llms.txt (max 3 requests per domain per week).
Run locally
python3 scripts/fetch_registry.py data/raw/registry.jsonl
python3 scripts/build_mcp_stats.py data/raw/registry.jsonl
python3 scripts/scan_web.py --top 10000 --out data/web/raw.jsonl && python3 scripts/build_web_stats.py
python3 scripts/build_site.py # -> site/
No installation packages available.
Remote
streamable-http
https://datacensus.org/mcp