
web-research-mcp
io.github.shukoilab/web-research-mcp
Stealth Google Search and token-efficient web content extraction with undetected browser automation.
Documentation
๐ญ MCP Web Research Server
Enterprise-Grade, Ultra-Fast, and Production-Ready Web Research Model Context Protocol (MCP) Server.
Engineered for AI coding agents and autonomous workflows requiring high-accuracy Google Search and high-density, token-efficient webpage content extraction with undetectable anti-bot evasion.
๐ Key Architectural Features
- ๐ญ 100% Undetectable Headed Stealth on Display :99: Built on Patchright (patched driver eliminating
Runtime.enable,--enable-automation, and console leaks). Enforces official Google Chrome in genuine headed mode (headless: false) inside an isolated X11 virtual framebuffer (:99) adhering strictly to Patchright's official Zero-Injection Best Practice (zero custom args, zero custom user-agent, zero header tampering). Guarantees 100% authentic consumer browser window metrics and extensions state with zero desktop window pop-ups or focus-stealing on the host OS. - โก Pre-Warmed Persistent Pool (<100ms Tab Acquisition): Browser context is pre-warmed at server boot with a persistent user profile (
launchPersistentContext), retaining DNS, TLS session tickets, V8 bytecode, and consent cookies for 38โ50% faster navigations. - ๐ Universal Regex-Free Semantic DOM Unfolding & Auto-Scroll: Handles complex e-commerce, SPA, and XHR-heavy sites (e.g. Tokopedia, Shopee, Reddit). Progressively scrolls, triggers
IntersectionObserverlazy-loads, unfolds collapsed sections using universal W3C WAI-ARIA standards (aria-expanded="false",data-state="closed",aria-controls) and native HTML5<details>disclosures with zero language-locking, unclamps CSS line-clamping, and stops when DOM height converges. - ๐ Token-Efficient & High-Density Markdown: Strips token-wasting avatar images, promotional widgets, and redundant blank lines. Compiles clean GitHub Flavored Markdown via Unified AST (
linkedom+@mozilla/readability+rehype-remark+remark-gfm). - ๐ก๏ธ Zero-Trust SSRF Defense: Rejects loopback (
127.0.0.0/8,::1), RFC 1918 private subnets (10.0.0.0/8,172.16.0.0/12,192.168.0.0/16), link-local metadata (169.254.169.254,metadata.google.internal), and non-HTTP protocols. - ๐ Universal Cross-Platform Architecture: Runs identically across macOS, Windows, and Linux via Docker with zero host dependencies and zero physical screen interference.
- ๐ฅ Real Liveness & Readiness Probing: Provides real subsystem health metrics over both MCP STDIO (
healthCheck) and native HTTP REST (GET /healthwith HTTP 200/503 status codes for Kubernetes / Docker).
๐ Universal Cross-Platform Execution
Thanks to hermetic Docker containerization, mcp-web-research runs with 100% feature parity across all operating systems without any host configuration:
| Host Platform | Support Level | Display & Stealth Engine | Window Behavior |
|---|---|---|---|
| macOS (Apple Silicon & Intel) | โ Universal | Virtual Framebuffer (:99) + Zero-Injection Stealth | Zero Desktop Pop-ups (Rendered in container) |
| Windows (10 / 11 / Server) | โ Universal | Virtual Framebuffer (:99) + Zero-Injection Stealth | Zero Desktop Pop-ups (Rendered in container) |
| Linux (Ubuntu / Debian / RHEL) | โ Universal | Virtual Framebuffer (:99) + Zero-Injection Stealth | Zero Desktop Pop-ups (Rendered in container) |
Anti-Bot Invariant: The browser strictly runs in genuine headed mode (
headless: false) inside the virtual framebuffer:99. End users will never see desktop pop-up windows, yet anti-bot inspection systems detect 100% authentic consumer desktop Chrome behavior.
โ๏ธ 12-Factor Configuration Reference
All settings can be configured via environment variables. Universal defaults are applied automatically if unset:
| Variable | Type | Default | Description |
|---|---|---|---|
HOST | string | 0.0.0.0 | Network interface address to bind HTTP service on VPS/host. |
PORT | integer | 3000 | HTTP port for streamable MCP and health check endpoints. |
ALLOWED_HOSTS | string | * | Comma-separated list or wildcard (*) of allowed Host headers for DNS-rebinding protection. |
ALLOWED_ORIGINS | string | * | Comma-separated list or wildcard (*) of allowed Origin headers for CORS security. |
MCP_API_KEY / API_KEY | string | undefined | Optional secret token(s) for HTTP API protection (comma-separated for key rotation). If unset, auth is disabled. |
HTTP_PROXY | string | undefined | Optional HTTP/HTTPS proxy URI (e.g. http://user:pass@host:port). |
BROWSER_USER_DATA_DIR | string | ~/.cache/mcp-web-research/chrome-profile | Persistent directory storing Chrome cache, cookies, and session state. |
BROWSER_TIMEOUT_MS | integer | 30000 | Default navigation timeout in milliseconds. |
MAX_CONCURRENT_PAGES | integer | 5 | Maximum simultaneous browser tabs allowed by semaphore. |
MAX_REQUESTS_BEFORE_RECYCLE | integer | 1000 | Lifetime requests before background browser restart to prevent RAM fragmentation. |
MAX_CONTENT_LENGTH | integer | 5242880 (5MB) | Upper bound memory guard against massive HTML payloads. |
NETWORK_IDLE_TIMEOUT_MS | integer | 5000 | Maximum wait time in ms for initial network stabilization before scrolling. |
SCROLL_DELAY_MS | integer | 500 | Debounce delay in ms after each scroll step and semantic unfolding pass (increase to 1000โ2000 for slow networks). |
SCROLL_STEP_PX | integer | 1000 | Distance in pixels per scroll increment. |
SCROLL_SETTLE_TIMEOUT_MS | integer | 500 | Final quiet period in ms after scroll loop completes. |
MAX_SCROLL_STEPS | integer | 6 | Maximum scroll loop iterations for lazy content hydration, semantic unfolding, and DOM convergence. |
SEARCH_WAIT_TIMEOUT_MS | integer | 10000 | Maximum wait time in ms for Google SERP cards to render before parsing results. |
READABILITY_CHAR_THRESHOLD | integer | 20 | Minimum character length for Mozilla Readability article scoring. |
๐ Client Installation & Setup Guide
Option 1: Docker Streamable HTTP Service (Recommended for Remote Agents, VPS & Teams)
1. Quick Start (Default Settings)
Run as an isolated, persistent network service on any machine:
docker run -d -p 3000:3000 --shm-size=1gb --name web-research-mcp --restart unless-stopped ghcr.io/shukoilab/web-research-mcp:latest
Tip on Custom Host Ports: If port 3000 is occupied on your host, forward your desired host port to container port 3000 (e.g.
-p 3031:3000), or override the internal port via environment variable (e.g.-p 3031:3031 -e PORT=3031).
2. Production VPS Deployment with Full Configuration Knobs
Customize any operational parameter directly at runtime via environment variables (-e). You do not need to modify or rebuild the Dockerfile:
docker run -d \
--name web-research-mcp \
--restart unless-stopped \
--shm-size=1gb \
-p 3000:3000 \
-e HOST="0.0.0.0" \
-e PORT="3000" \
-e ALLOWED_HOSTS="*" \
-e ALLOWED_ORIGINS="*" \
-e MCP_API_KEY="sk-your-secret-api-key" \
-e HTTP_PROXY="" \
-e BROWSER_TIMEOUT_MS="30000" \
-e MAX_CONCURRENT_PAGES="5" \
-e MAX_REQUESTS_BEFORE_RECYCLE="1000" \
-e MAX_CONTENT_LENGTH="5242880" \
-e NETWORK_IDLE_TIMEOUT_MS="5000" \
-e SCROLL_DELAY_MS="500" \
-e SCROLL_STEP_PX="1000" \
-e SCROLL_SETTLE_TIMEOUT_MS="500" \
-e MAX_SCROLL_STEPS="6" \
-e SEARCH_WAIT_TIMEOUT_MS="10000" \
-e READABILITY_CHAR_THRESHOLD="20" \
ghcr.io/shukoilab/web-research-mcp:latest
3. Using an Environment File (--env-file)
Keep sensitive keys and parameters organized in a local .env file:
docker run -d \
-p 3000:3000 \
--shm-size=1gb \
--env-file .env \
--name web-research-mcp \
--restart unless-stopped \
ghcr.io/shukoilab/web-research-mcp:latest
4. Docker Compose Deployment (docker-compose.yml)
For automated restarts, VPS orchestration, and persistent stacks:
version: '3.8'
services:
web-research-mcp:
image: ghcr.io/shukoilab/web-research-mcp:latest
container_name: web-research-mcp
restart: unless-stopped
shm_size: 1gb
ports:
- "3000:3000"
environment:
# Network & Security
- HOST=0.0.0.0
- PORT=3000
- ALLOWED_HOSTS=*
- ALLOWED_ORIGINS=*
- MCP_API_KEY=sk-your-secret-api-key
# Browser Performance & Tuning
- BROWSER_TIMEOUT_MS=30000
- MAX_CONCURRENT_PAGES=5
- MAX_REQUESTS_BEFORE_RECYCLE=1000
- MAX_CONTENT_LENGTH=5242880
- NETWORK_IDLE_TIMEOUT_MS=5000
- SCROLL_DELAY_MS=500
- SCROLL_STEP_PX=1000
- SCROLL_SETTLE_TIMEOUT_MS=500
- MAX_SCROLL_STEPS=6
- SEARCH_WAIT_TIMEOUT_MS=10000
- READABILITY_CHAR_THRESHOLD=20
- MCP Endpoint:
http://localhost:3000/mcp(Streamable HTTP transport) - Healthcheck Probe:
GET http://localhost:3000/health(HTTP 200/503)
5. Connecting AI Clients with Optional Authentication
In Cursor / OpenCode (opencode.json / remote MCP settings):
{
"type": "remote",
"url": "http://your-vps-ip:3000/mcp",
"headers": {
"Authorization": "Bearer sk-your-secret-api-key"
}
}
Testing via cURL:
# Public liveness healthcheck probe (always unauthenticated)
curl -s http://localhost:3000/health
# MCP request with Bearer Token (or X-API-Key header)
curl -s -X POST http://localhost:3000/mcp \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-your-secret-api-key" \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"curl-client","version":"1.0.0"}}}'
Option 2: Docker Local STDIO Container (Recommended for Claude Desktop)
Runs inside the sealed container without needing Node.js or Chrome installed on the host machine:
Claude Desktop
Add to your claude_desktop_config.json:
- macOS:
~/Library/Application Support/Claude/claude_desktop_config.json - Windows:
%APPDATA%\Claude\claude_desktop_config.json - Linux:
~/.config/Claude/claude_desktop_config.json
{
"mcpServers": {
"web-research": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"--init",
"--shm-size=1gb",
"ghcr.io/shukoilab/web-research-mcp:latest",
"node",
"dist/entries/mcp-stdio.js"
]
}
}
}
Cursor IDE
Add to ~/.cursor/mcp.json (or via Cursor Settings > Features > MCP):
{
"mcpServers": {
"web-research": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"--init",
"--shm-size=1gb",
"ghcr.io/shukoilab/web-research-mcp:latest",
"node",
"dist/entries/mcp-stdio.js"
]
}
}
}
OpenCode
Add to your opencode.json (or profile config):
{
"$schema": "https://opencode.ai/config.json",
"mcp": {
"web-research": {
"type": "local",
"command": [
"docker",
"run",
"-i",
"--rm",
"--init",
"--shm-size=1gb",
"ghcr.io/shukoilab/web-research-mcp:latest",
"node",
"dist/entries/mcp-stdio.js"
]
}
}
}
VS Code (Cline / Roo Code / Claude Dev)
{
"mcpServers": {
"web-research": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"--init",
"--shm-size=1gb",
"ghcr.io/shukoilab/web-research-mcp:latest",
"node",
"dist/entries/mcp-stdio.js"
],
"disabled": false,
"autoApprove": []
}
}
}
Option 3: Local Node.js Development & STDIO Execution
For local development or environments with Node.js 20+ installed:
# Build the project
pnpm install
pnpm run build
# Start the Streamable HTTP server on port 3000
pnpm run start
Or invoke STDIO locally:
{
"mcpServers": {
"web-research": {
"command": "node",
"args": ["/absolute/path/to/mcp-web-research/dist/entries/mcp-stdio.js"]
}
}
}
๐ ๏ธ MCP Tools Catalog
1. googleSearch (googleSearchAction)
Executes organic web search on Google using stealth Chrome with automatic consent dialog bypass, breadcrumb canonicalization, and rich snippet extraction. Acts as Step 1 (Discovery & Triage) in agent research workflows, returning candidate URLs and preview snippets.
Input Parameters:
query(string, required, 1..500 chars): Search keywords or phrases. Supports Google Dorking operators for high-precision queries (e.g.site:github.com "patchright" inurl:releases,filetype:pdf "machine learning",intitle:"API docs").limit(integer, optional, 1..50, default: 10): Maximum organic results.language(string, optional, default: "en"): Language code (e.g."en","id").country(string, optional, default: "us"): Geolocation region code (e.g."us","id").safeSearch(boolean, optional, default: true): SafeSearch adult filter.
Example Tool Response:
{
"query": "typescript documentation",
"totalReturned": 2,
"results": [
{
"title": "The starting point for learning TypeScript",
"url": "https://www.typescriptlang.org/docs",
"redirectUrl": "https://www.google.com/goto?url=...",
"snippet": "TypeScript extends JavaScript by adding types to the language..."
}
],
"timestamp": "2026-11-20T00:00:00.000Z"
}
2. webExtract (webExtractAction)
Navigates to any target URL, executes intelligent progressive scrolling to trigger lazy XHR/SPA content (prices, reviews, specs), unfolds collapsed content using universal W3C ARIA standards and native HTML5 disclosures, unclamps visual CSS line-clamping, and serializes clean GitHub Flavored Markdown. Acts as Step 2 (Deep Content Reader) following googleSearch or as Step 1 for direct URLs.
Input Parameters:
url(string, required, format: "uri", 1..2048 chars): Target website URL (http://orhttps://). Accepts direct URLs or canonical URLs discovered viagoogleSearch.smartScroll(boolean, optional, default: true): Enable automated progressive scroll, semantic unfolding, and XHR stabilization.scrollDelayMs(integer, optional, 100..5000, default: 500): Debounce wait time in milliseconds after each scroll step (increase to 1000โ2000ms for slow networks).maxScrolls(integer, optional, 0..20, default: 6): Maximum scroll step iterations for lazy content hydration and DOM convergence.timeoutMs(integer, optional, 1000..60000, default: 30000): Maximum navigation deadline.
Example Tool Response:
{
"url": "https://www.tokopedia.com/kyo-official/ssd-kyo-kaizen-m-2-nvme-512gb-pcie-gen3-x4-2280-ssd-nvme-gen-3",
"title": "SSD KYO KAIZEN M.2 NVMe 512GB PCIe Gen3 x4 2280 SSD NVMe Gen 3",
"byline": "KYO Indonesia",
"excerpt": "KAIZEN diambil dari bahasa Jepang untuk perbaikan berkelanjutan...",
"markdown": "## SSD KYO KAIZEN M.2 NVMe 512GB...\n\n**Harga:** Rp1.626.050\n**Stok:** 451\n\n### Spesifikasi Lengkap:\n- Tipe: M.2 NVMe PCIe 3 x4\n- Kapasitas: 512GB\n\n## ULASAN PEMBELI\n- **Andri:** Barang diterima dengan baik dan cepat...",
"timestamp": "2026-11-20T00:00:00.000Z"
}
3. healthCheck (healthAction)
Inspects real-time subsystem readiness, browser pool connectivity, active concurrency, and memory consumption.
Example Output:
{
"status": "healthy",
"timestamp": "2026-11-20T00:00:00.000Z",
"uptimeSec": 1240,
"browser": {
"connected": true,
"prewarmed": true,
"activeTabs": 0,
"maxCapacity": 5,
"requestCount": 42,
"userDataDir": "/home/shukoi/.cache/mcp-web-research/chrome-profile"
},
"display": ":99",
"memory": {
"heapUsedMb": 64.25,
"rssMb": 182.10
}
}
๐งช Testing & Verification
Run the full automated verification pipeline:
# Execute unit and live integration tests (68+ tests)
pnpm test
# Verify strict TypeScript type-safety (0 errors)
pnpm run typecheck
# Check formatting and static quality (Biome)
pnpm run lint
# Compile production build to ./dist
pnpm run build
๐ Acknowledgements & Open-Source Credits
This project is built on top of an incredible open-source ecosystem. Sincere gratitude and credits go to the creators and maintainers of the following foundations:
- Patchright โ Undetected browser automation driver eliminating CDP leaks,
--enable-automation, and runtime flags for genuine consumer fingerprints. - Shukoi Framework (
@shukoi/core,@shukoi/mcp) โ Modular TypeScript runtime framework powering MCP action dispatching and Web-Standard transport. - Hono (
hono,@hono/node-server) โ Ultra-fast, lightweight Web-Standard HTTP framework powering Streamable HTTP transport and native health probes. - Model Context Protocol SDK โ Official TypeScript SDK implementing the MCP specification across STDIO and Streamable HTTP transports.
- Mozilla Readability โ Standalone content extraction engine from Mozilla Firefox's Reader View for clean, token-efficient article parsing.
- LinkeDOM โ High-performance, lightweight server-side DOM implementation with near-zero memory footprint.
- Unified Ecosystem (
unified,rehype-parse,rehype-remark,remark-gfm,remark-stringify) โ Composable syntax tree (AST) pipeline transforming HTML into high-density GitHub Flavored Markdown. - TypeBox โ High-performance JSON Schema Type Builder with static type inference enforcing strict input/output contracts.
- jose โ Universal Web Cryptography library for secure token validation and cryptographic operations.
- Biome & Vitest โ Modern development toolchains delivering blazing-fast linting, formatting, and automated testing.
๐ License
MIT ยฉ Shukoi Lab
ghcr.io/shukoilab/web-research-mcp:2.1.0docker pull ghcr.io/shukoilab/web-research-mcp:2.1.0:undefined