Repository avatar
Search & Data Extraction
v2.1.0
active

web-research-mcp

io.github.shukoilab/web-research-mcp

Stealth Google Search and token-efficient web content extraction with undetected browser automation.

Documentation

๐ŸŽญ MCP Web Research Server

Version: 2.1.0 Docker: Universal License: MIT TypeScript: 5.7+ Patchright: Undetected Shukoi: Enterprise

Enterprise-Grade, Ultra-Fast, and Production-Ready Web Research Model Context Protocol (MCP) Server.
Engineered for AI coding agents and autonomous workflows requiring high-accuracy Google Search and high-density, token-efficient webpage content extraction with undetectable anti-bot evasion.


๐ŸŒŸ Key Architectural Features

  • ๐ŸŽญ 100% Undetectable Headed Stealth on Display :99: Built on Patchright (patched driver eliminating Runtime.enable, --enable-automation, and console leaks). Enforces official Google Chrome in genuine headed mode (headless: false) inside an isolated X11 virtual framebuffer (:99) adhering strictly to Patchright's official Zero-Injection Best Practice (zero custom args, zero custom user-agent, zero header tampering). Guarantees 100% authentic consumer browser window metrics and extensions state with zero desktop window pop-ups or focus-stealing on the host OS.
  • โšก Pre-Warmed Persistent Pool (<100ms Tab Acquisition): Browser context is pre-warmed at server boot with a persistent user profile (launchPersistentContext), retaining DNS, TLS session tickets, V8 bytecode, and consent cookies for 38โ€“50% faster navigations.
  • ๐Ÿ”„ Universal Regex-Free Semantic DOM Unfolding & Auto-Scroll: Handles complex e-commerce, SPA, and XHR-heavy sites (e.g. Tokopedia, Shopee, Reddit). Progressively scrolls, triggers IntersectionObserver lazy-loads, unfolds collapsed sections using universal W3C WAI-ARIA standards (aria-expanded="false", data-state="closed", aria-controls) and native HTML5 <details> disclosures with zero language-locking, unclamps CSS line-clamping, and stops when DOM height converges.
  • ๐Ÿ“‰ Token-Efficient & High-Density Markdown: Strips token-wasting avatar images, promotional widgets, and redundant blank lines. Compiles clean GitHub Flavored Markdown via Unified AST (linkedom + @mozilla/readability + rehype-remark + remark-gfm).
  • ๐Ÿ›ก๏ธ Zero-Trust SSRF Defense: Rejects loopback (127.0.0.0/8, ::1), RFC 1918 private subnets (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), link-local metadata (169.254.169.254, metadata.google.internal), and non-HTTP protocols.
  • ๐ŸŒ Universal Cross-Platform Architecture: Runs identically across macOS, Windows, and Linux via Docker with zero host dependencies and zero physical screen interference.
  • ๐Ÿฅ Real Liveness & Readiness Probing: Provides real subsystem health metrics over both MCP STDIO (healthCheck) and native HTTP REST (GET /health with HTTP 200/503 status codes for Kubernetes / Docker).

๐ŸŒ Universal Cross-Platform Execution

Thanks to hermetic Docker containerization, mcp-web-research runs with 100% feature parity across all operating systems without any host configuration:

Host PlatformSupport LevelDisplay & Stealth EngineWindow Behavior
macOS (Apple Silicon & Intel)โœ… UniversalVirtual Framebuffer (:99) + Zero-Injection StealthZero Desktop Pop-ups (Rendered in container)
Windows (10 / 11 / Server)โœ… UniversalVirtual Framebuffer (:99) + Zero-Injection StealthZero Desktop Pop-ups (Rendered in container)
Linux (Ubuntu / Debian / RHEL)โœ… UniversalVirtual Framebuffer (:99) + Zero-Injection StealthZero Desktop Pop-ups (Rendered in container)

Anti-Bot Invariant: The browser strictly runs in genuine headed mode (headless: false) inside the virtual framebuffer :99. End users will never see desktop pop-up windows, yet anti-bot inspection systems detect 100% authentic consumer desktop Chrome behavior.


โš™๏ธ 12-Factor Configuration Reference

All settings can be configured via environment variables. Universal defaults are applied automatically if unset:

VariableTypeDefaultDescription
HOSTstring0.0.0.0Network interface address to bind HTTP service on VPS/host.
PORTinteger3000HTTP port for streamable MCP and health check endpoints.
ALLOWED_HOSTSstring*Comma-separated list or wildcard (*) of allowed Host headers for DNS-rebinding protection.
ALLOWED_ORIGINSstring*Comma-separated list or wildcard (*) of allowed Origin headers for CORS security.
MCP_API_KEY / API_KEYstringundefinedOptional secret token(s) for HTTP API protection (comma-separated for key rotation). If unset, auth is disabled.
HTTP_PROXYstringundefinedOptional HTTP/HTTPS proxy URI (e.g. http://user:pass@host:port).
BROWSER_USER_DATA_DIRstring~/.cache/mcp-web-research/chrome-profilePersistent directory storing Chrome cache, cookies, and session state.
BROWSER_TIMEOUT_MSinteger30000Default navigation timeout in milliseconds.
MAX_CONCURRENT_PAGESinteger5Maximum simultaneous browser tabs allowed by semaphore.
MAX_REQUESTS_BEFORE_RECYCLEinteger1000Lifetime requests before background browser restart to prevent RAM fragmentation.
MAX_CONTENT_LENGTHinteger5242880 (5MB)Upper bound memory guard against massive HTML payloads.
NETWORK_IDLE_TIMEOUT_MSinteger5000Maximum wait time in ms for initial network stabilization before scrolling.
SCROLL_DELAY_MSinteger500Debounce delay in ms after each scroll step and semantic unfolding pass (increase to 1000โ€“2000 for slow networks).
SCROLL_STEP_PXinteger1000Distance in pixels per scroll increment.
SCROLL_SETTLE_TIMEOUT_MSinteger500Final quiet period in ms after scroll loop completes.
MAX_SCROLL_STEPSinteger6Maximum scroll loop iterations for lazy content hydration, semantic unfolding, and DOM convergence.
SEARCH_WAIT_TIMEOUT_MSinteger10000Maximum wait time in ms for Google SERP cards to render before parsing results.
READABILITY_CHAR_THRESHOLDinteger20Minimum character length for Mozilla Readability article scoring.

๐Ÿš€ Client Installation & Setup Guide

Option 1: Docker Streamable HTTP Service (Recommended for Remote Agents, VPS & Teams)

1. Quick Start (Default Settings)

Run as an isolated, persistent network service on any machine:

docker run -d -p 3000:3000 --shm-size=1gb --name web-research-mcp --restart unless-stopped ghcr.io/shukoilab/web-research-mcp:latest

Tip on Custom Host Ports: If port 3000 is occupied on your host, forward your desired host port to container port 3000 (e.g. -p 3031:3000), or override the internal port via environment variable (e.g. -p 3031:3031 -e PORT=3031).

2. Production VPS Deployment with Full Configuration Knobs

Customize any operational parameter directly at runtime via environment variables (-e). You do not need to modify or rebuild the Dockerfile:

docker run -d \
  --name web-research-mcp \
  --restart unless-stopped \
  --shm-size=1gb \
  -p 3000:3000 \
  -e HOST="0.0.0.0" \
  -e PORT="3000" \
  -e ALLOWED_HOSTS="*" \
  -e ALLOWED_ORIGINS="*" \
  -e MCP_API_KEY="sk-your-secret-api-key" \
  -e HTTP_PROXY="" \
  -e BROWSER_TIMEOUT_MS="30000" \
  -e MAX_CONCURRENT_PAGES="5" \
  -e MAX_REQUESTS_BEFORE_RECYCLE="1000" \
  -e MAX_CONTENT_LENGTH="5242880" \
  -e NETWORK_IDLE_TIMEOUT_MS="5000" \
  -e SCROLL_DELAY_MS="500" \
  -e SCROLL_STEP_PX="1000" \
  -e SCROLL_SETTLE_TIMEOUT_MS="500" \
  -e MAX_SCROLL_STEPS="6" \
  -e SEARCH_WAIT_TIMEOUT_MS="10000" \
  -e READABILITY_CHAR_THRESHOLD="20" \
  ghcr.io/shukoilab/web-research-mcp:latest

3. Using an Environment File (--env-file)

Keep sensitive keys and parameters organized in a local .env file:

docker run -d \
  -p 3000:3000 \
  --shm-size=1gb \
  --env-file .env \
  --name web-research-mcp \
  --restart unless-stopped \
  ghcr.io/shukoilab/web-research-mcp:latest

4. Docker Compose Deployment (docker-compose.yml)

For automated restarts, VPS orchestration, and persistent stacks:

version: '3.8'

services:
  web-research-mcp:
    image: ghcr.io/shukoilab/web-research-mcp:latest
    container_name: web-research-mcp
    restart: unless-stopped
    shm_size: 1gb
    ports:
      - "3000:3000"
    environment:
      # Network & Security
      - HOST=0.0.0.0
      - PORT=3000
      - ALLOWED_HOSTS=*
      - ALLOWED_ORIGINS=*
      - MCP_API_KEY=sk-your-secret-api-key
      # Browser Performance & Tuning
      - BROWSER_TIMEOUT_MS=30000
      - MAX_CONCURRENT_PAGES=5
      - MAX_REQUESTS_BEFORE_RECYCLE=1000
      - MAX_CONTENT_LENGTH=5242880
      - NETWORK_IDLE_TIMEOUT_MS=5000
      - SCROLL_DELAY_MS=500
      - SCROLL_STEP_PX=1000
      - SCROLL_SETTLE_TIMEOUT_MS=500
      - MAX_SCROLL_STEPS=6
      - SEARCH_WAIT_TIMEOUT_MS=10000
      - READABILITY_CHAR_THRESHOLD=20
  • MCP Endpoint: http://localhost:3000/mcp (Streamable HTTP transport)
  • Healthcheck Probe: GET http://localhost:3000/health (HTTP 200/503)

5. Connecting AI Clients with Optional Authentication

In Cursor / OpenCode (opencode.json / remote MCP settings):

{
  "type": "remote",
  "url": "http://your-vps-ip:3000/mcp",
  "headers": {
    "Authorization": "Bearer sk-your-secret-api-key"
  }
}

Testing via cURL:

# Public liveness healthcheck probe (always unauthenticated)
curl -s http://localhost:3000/health

# MCP request with Bearer Token (or X-API-Key header)
curl -s -X POST http://localhost:3000/mcp \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-secret-api-key" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"curl-client","version":"1.0.0"}}}'

Option 2: Docker Local STDIO Container (Recommended for Claude Desktop)

Runs inside the sealed container without needing Node.js or Chrome installed on the host machine:

Claude Desktop

Add to your claude_desktop_config.json:

  • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
  • Windows: %APPDATA%\Claude\claude_desktop_config.json
  • Linux: ~/.config/Claude/claude_desktop_config.json
{
  "mcpServers": {
    "web-research": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "--init",
        "--shm-size=1gb",
        "ghcr.io/shukoilab/web-research-mcp:latest",
        "node",
        "dist/entries/mcp-stdio.js"
      ]
    }
  }
}

Cursor IDE

Add to ~/.cursor/mcp.json (or via Cursor Settings > Features > MCP):

{
  "mcpServers": {
    "web-research": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "--init",
        "--shm-size=1gb",
        "ghcr.io/shukoilab/web-research-mcp:latest",
        "node",
        "dist/entries/mcp-stdio.js"
      ]
    }
  }
}

OpenCode

Add to your opencode.json (or profile config):

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "web-research": {
      "type": "local",
      "command": [
        "docker",
        "run",
        "-i",
        "--rm",
        "--init",
        "--shm-size=1gb",
        "ghcr.io/shukoilab/web-research-mcp:latest",
        "node",
        "dist/entries/mcp-stdio.js"
      ]
    }
  }
}

VS Code (Cline / Roo Code / Claude Dev)

{
  "mcpServers": {
    "web-research": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "--init",
        "--shm-size=1gb",
        "ghcr.io/shukoilab/web-research-mcp:latest",
        "node",
        "dist/entries/mcp-stdio.js"
      ],
      "disabled": false,
      "autoApprove": []
    }
  }
}

Option 3: Local Node.js Development & STDIO Execution

For local development or environments with Node.js 20+ installed:

# Build the project
pnpm install
pnpm run build

# Start the Streamable HTTP server on port 3000
pnpm run start

Or invoke STDIO locally:

{
  "mcpServers": {
    "web-research": {
      "command": "node",
      "args": ["/absolute/path/to/mcp-web-research/dist/entries/mcp-stdio.js"]
    }
  }
}

๐Ÿ› ๏ธ MCP Tools Catalog

1. googleSearch (googleSearchAction)

Executes organic web search on Google using stealth Chrome with automatic consent dialog bypass, breadcrumb canonicalization, and rich snippet extraction. Acts as Step 1 (Discovery & Triage) in agent research workflows, returning candidate URLs and preview snippets.

Input Parameters:

  • query (string, required, 1..500 chars): Search keywords or phrases. Supports Google Dorking operators for high-precision queries (e.g. site:github.com "patchright" inurl:releases, filetype:pdf "machine learning", intitle:"API docs").
  • limit (integer, optional, 1..50, default: 10): Maximum organic results.
  • language (string, optional, default: "en"): Language code (e.g. "en", "id").
  • country (string, optional, default: "us"): Geolocation region code (e.g. "us", "id").
  • safeSearch (boolean, optional, default: true): SafeSearch adult filter.

Example Tool Response:

{
  "query": "typescript documentation",
  "totalReturned": 2,
  "results": [
    {
      "title": "The starting point for learning TypeScript",
      "url": "https://www.typescriptlang.org/docs",
      "redirectUrl": "https://www.google.com/goto?url=...",
      "snippet": "TypeScript extends JavaScript by adding types to the language..."
    }
  ],
  "timestamp": "2026-11-20T00:00:00.000Z"
}

2. webExtract (webExtractAction)

Navigates to any target URL, executes intelligent progressive scrolling to trigger lazy XHR/SPA content (prices, reviews, specs), unfolds collapsed content using universal W3C ARIA standards and native HTML5 disclosures, unclamps visual CSS line-clamping, and serializes clean GitHub Flavored Markdown. Acts as Step 2 (Deep Content Reader) following googleSearch or as Step 1 for direct URLs.

Input Parameters:

  • url (string, required, format: "uri", 1..2048 chars): Target website URL (http:// or https://). Accepts direct URLs or canonical URLs discovered via googleSearch.
  • smartScroll (boolean, optional, default: true): Enable automated progressive scroll, semantic unfolding, and XHR stabilization.
  • scrollDelayMs (integer, optional, 100..5000, default: 500): Debounce wait time in milliseconds after each scroll step (increase to 1000โ€“2000ms for slow networks).
  • maxScrolls (integer, optional, 0..20, default: 6): Maximum scroll step iterations for lazy content hydration and DOM convergence.
  • timeoutMs (integer, optional, 1000..60000, default: 30000): Maximum navigation deadline.

Example Tool Response:

{
  "url": "https://www.tokopedia.com/kyo-official/ssd-kyo-kaizen-m-2-nvme-512gb-pcie-gen3-x4-2280-ssd-nvme-gen-3",
  "title": "SSD KYO KAIZEN M.2 NVMe 512GB PCIe Gen3 x4 2280 SSD NVMe Gen 3",
  "byline": "KYO Indonesia",
  "excerpt": "KAIZEN diambil dari bahasa Jepang untuk perbaikan berkelanjutan...",
  "markdown": "## SSD KYO KAIZEN M.2 NVMe 512GB...\n\n**Harga:** Rp1.626.050\n**Stok:** 451\n\n### Spesifikasi Lengkap:\n- Tipe: M.2 NVMe PCIe 3 x4\n- Kapasitas: 512GB\n\n## ULASAN PEMBELI\n- **Andri:** Barang diterima dengan baik dan cepat...",
  "timestamp": "2026-11-20T00:00:00.000Z"
}

3. healthCheck (healthAction)

Inspects real-time subsystem readiness, browser pool connectivity, active concurrency, and memory consumption.

Example Output:

{
  "status": "healthy",
  "timestamp": "2026-11-20T00:00:00.000Z",
  "uptimeSec": 1240,
  "browser": {
    "connected": true,
    "prewarmed": true,
    "activeTabs": 0,
    "maxCapacity": 5,
    "requestCount": 42,
    "userDataDir": "/home/shukoi/.cache/mcp-web-research/chrome-profile"
  },
  "display": ":99",
  "memory": {
    "heapUsedMb": 64.25,
    "rssMb": 182.10
  }
}

๐Ÿงช Testing & Verification

Run the full automated verification pipeline:

# Execute unit and live integration tests (68+ tests)
pnpm test

# Verify strict TypeScript type-safety (0 errors)
pnpm run typecheck

# Check formatting and static quality (Biome)
pnpm run lint

# Compile production build to ./dist
pnpm run build

๐Ÿ™ Acknowledgements & Open-Source Credits

This project is built on top of an incredible open-source ecosystem. Sincere gratitude and credits go to the creators and maintainers of the following foundations:

  • Patchright โ€” Undetected browser automation driver eliminating CDP leaks, --enable-automation, and runtime flags for genuine consumer fingerprints.
  • Shukoi Framework (@shukoi/core, @shukoi/mcp) โ€” Modular TypeScript runtime framework powering MCP action dispatching and Web-Standard transport.
  • Hono (hono, @hono/node-server) โ€” Ultra-fast, lightweight Web-Standard HTTP framework powering Streamable HTTP transport and native health probes.
  • Model Context Protocol SDK โ€” Official TypeScript SDK implementing the MCP specification across STDIO and Streamable HTTP transports.
  • Mozilla Readability โ€” Standalone content extraction engine from Mozilla Firefox's Reader View for clean, token-efficient article parsing.
  • LinkeDOM โ€” High-performance, lightweight server-side DOM implementation with near-zero memory footprint.
  • Unified Ecosystem (unified, rehype-parse, rehype-remark, remark-gfm, remark-stringify) โ€” Composable syntax tree (AST) pipeline transforming HTML into high-density GitHub Flavored Markdown.
  • TypeBox โ€” High-performance JSON Schema Type Builder with static type inference enforcing strict input/output contracts.
  • jose โ€” Universal Web Cryptography library for secure token validation and cryptographic operations.
  • Biome & Vitest โ€” Modern development toolchains delivering blazing-fast linting, formatting, and automated testing.

๐Ÿ“„ License

MIT ยฉ Shukoi Lab

OCI
ghcr.io/shukoilab/web-research-mcp:2.1.0
Install Command
docker pull ghcr.io/shukoilab/web-research-mcp:2.1.0:undefined