
apexuse
io.github.Jackpkn/apexuse
Computer Use MCP server in Rust with Accessibility, Set-of-Mark vision, OCR, and workflow memory.
Documentation
ApexUse — Computer Use MCP Server for AI Agents (macOS, Windows & Linux)
ApexUse (apexuse-mcp) is an open-source, local-first Computer Use MCP server written in pure Rust and native OS bridges. It gives AI coding agents — including Claude Code, OpenAI Codex, Gemini CLI, and Antigravity — full desktop and browser control over macOS, Windows, and Linux using native Accessibility trees, Set-of-Mark ([m1]..[mN]) visual grounding, local OCR, a non-hijacking AgentCursor overlay, batched run_actions, and local workflow memory.
Zero external LLM keys. Zero external Rust crate dependencies. Zero cloud relay.
Why ApexUse vs. Alternative Computer-Use Tools?
| Capability | ApexUse (apexuse-mcp) | Screenshot-Only Computer Use | Browser-Only MCPs (Playwright / DevTools) |
|---|---|---|---|
| App Coverage | Full OS: native apps, Electron, terminals & any browser | Whole screen | Browser tabs only |
| UI Grounding | Hybrid: semantic Accessibility (e1..eN) + Set-of-Mark ([m1]..[mN]) + OCR | Pixel coordinates only | DOM only |
| Multi-Step Speed | run_actions (up to 12 local chained steps) + local wait_for polling | 1 LLM round-trip per click | Step-by-step tool calls |
| Scroll & Verification | Target-scoped scroll (ref/mark/x,y) + verify_change auto-retry | Blind scroll + manual re-screenshot | Page scroll only |
| Visual Pointer | Floating transparent AgentCursor overlay (never warps macOS hardware cursor) | Teleports real user cursor | None / page focus |
| Workflow Memory | Records & replays structural macros via workflow_memory | Re-plans every run | Scripts must be coded |
| Security Controls | Native password/secure-field blocking, protected-app list, STOP kill switch, audit log | Sends full raw screen pixels | Web origin scope |
Platform Support
| Platform | Backend Driver | Capabilities & Verification |
|---|---|---|
| macOS 14+ | MacosDriver + rust/native/macos_bridge.m | AXUIElement, ScreenCaptureKit, Apple Vision OCR, ApexUseCursorView overlay, per-PID CGEvent/NSEvent. Live verified on macOS. |
| Windows 10 / 11 | WindowsDriver (rust/src/driver/windows.rs) | Pure-Rust Win32 FFI (IUIAutomation COM, PrintWindow/BitBlt, WinRT Windows.Media.Ocr, SendInput, layered AgentCursor, som_png overlay). Cross-checked (x86_64-pc-windows-gnu) & built in Windows CI; live Windows host test pending. |
| Linux (X11 & Wayland) | LinuxDriver (rust/src/driver/linux.rs) | Runtime dlopen libX11 / libXtst + AT-SPI2 (gi.repository.Atspi), Wayland helpers (grim / spectacle / gnome-screenshot, wtype / ydotool), tesseract OCR, som_png overlay. Cross-checked (x86_64-unknown-linux-gnu) & built in Linux CI; live Linux host test pending. |
Quick Start
1. Install or build from source
Download prebuilt binaries for macOS (arm64), Linux (x86_64), or Windows (x86_64) from GitHub Releases, or install directly from crates.io:
cargo install apexuse-mcp
Or clone and build directly (requires only the Rust toolchain, plus Xcode Command Line Tools on macOS):
git clone https://github.com/Jackpkn/apexuse.git
cd apexuse
cargo build --release --manifest-path rust/Cargo.toml
Output binary: rust/target/release/apexuse-mcp (or apexuse-mcp.exe on Windows).
2. Check OS readiness & permissions
./rust/target/release/apexuse-mcp doctor
- macOS: Enable Accessibility and Screen Recording for the terminal or IDE launching
apexuse-mcpin System Settings → Privacy & Security. - Windows: Uses built-in Win32 / UI Automation / WinRT APIs with no extra dependencies.
- Linux: Runs on X11 via
libX11/libXtst(dlopen). For full semantic accessibility, OCR, and Wayland support install distro packages as needed (e.g.,python3-gi,gir1.2-atspi-2.0,tesseract-ocr,xdotool, orgrim/wtype/ydotoolon Wayland).
3. Connect to your AI coding agent
Generate a ready-to-paste snippet for your client:
./rust/target/release/apexuse-mcp install antigravity
# Supported: antigravity | gemini | claude | codex
Copy-paste snippets for Antigravity, Claude Code, OpenAI Codex, and Gemini CLI
- Claude Code:
claude mcp add apexuse --env APEXUSE_POLICY=/abs/path/to/apexuse/policy.autonomous.yaml -- /abs/path/to/apexuse/rust/target/release/apexuse-mcp - Antigravity / Gemini CLI (
~/.gemini/antigravity/mcp_config.jsonor~/.gemini/settings.json):{ "mcpServers": { "apexuse": { "command": "/abs/path/to/apexuse/rust/target/release/apexuse-mcp", "args": [], "env": { "APEXUSE_POLICY": "/abs/path/to/apexuse/policy.autonomous.yaml" } } } } - OpenAI Codex (
~/.codex/config.toml):[mcp_servers.apexuse] command = "/abs/path/to/apexuse/rust/target/release/apexuse-mcp" args = [] startup_timeout_sec = 60 tool_timeout_sec = 180 [mcp_servers.apexuse.env] APEXUSE_POLICY = "/abs/path/to/apexuse/policy.autonomous.yaml"
See agent-configs/README.md and the bundled skill at .agents/skills/apexuse/SKILL.md.
MCP Tools Reference (6 Tools)
| Tool | Actions & Key Parameters |
|---|---|
inspect_app | Read-only Accessibility snapshot for app (bundle ID, executable, app name, or "desktop"), optional window index and query filter. Returns e1..eN refs & workflow hints without taking a screenshot. |
visual_inspect | Captures an app window or full "desktop", fuses Accessibility + OCR (ocr="auto"|"fast"|"accurate"|"off", max_marks 1..=150), and returns an annotated PNG with [m1]..[mN] mark IDs. |
computer | inspect, click, double_click, right_click, hover (ref/mark/x,y), drag, type, press_key, scroll (optional ref/mark/x,y target), run_actions (up to 12 batched steps + inspect_after), wait_for (local AX/OCR poll up to 60s), confirm. Supports verify_change, max_retries (0..=3), settle_ms, retry_delay_ms, require_change. |
desktop | list_applications, launch_application, activate_application, open_url, visual_click, double_click, right_click, hover, drag. |
screenshot | Unannotated PNG capture of the largest window of a permitted running app after safety preflight. |
workflow_memory | list, lookup, record, replay (with optional step_args), clear stored in ~/.apexuse/memory/recipes.json. |
Policy, Safety & Environment Variables
| Variable | Default | Purpose |
|---|---|---|
APEXUSE_POLICY | ./policy.yaml (then ancestor search) | Policy YAML (policy.yaml or policy.autonomous.yaml). |
APEXUSE_STATE_DIR | ~/.apexuse | Directory for logs/audit.jsonl, memory/recipes.json, and STOP. |
APEXUSE_MOCK | 0 | Set to 1 (or --mock) to run the deterministic in-memory MockDriver. |
(Legacy HANDSMCP_* env vars remain supported as fallbacks.)
Built-in safety guardrails enforced regardless of policy:
- Protected app denial (
password,1password,bitwarden,keychain,wallet,bank, etc.) - Native password / secure text field redaction (
AXSecureTextField,CurrentIsPassword,AT-SPI PASSWORD_TEXT, bullet-mask checks) - Clipboard paste blocking (
Cmd+V/Ctrl+Vrejected; usecomputer(action="type")) - Emergency kill switch: create
~/.apexuse/STOPto halt all desktop mutations immediately
FAQ
Does ApexUse need an OpenAI, Anthropic, or Google API key?
No. apexuse-mcp is a pure local MCP stdio binary with zero built-in LLM calls. Your existing AI agent (Claude Code, Codex, Gemini CLI, Antigravity, etc.) drives it directly over MCP.
How does ApexUse avoid the slow screenshot loop of traditional Computer Use?
First, inspect_app reads compact semantic UI elements (e1..eN) without image encoding. Second, visual_inspect lets you set ocr="off" for instant AX-only visual marks. Third, computer(action="run_actions") batches up to 12 deterministic clicks, keystrokes, scrolls, and waits inside a single turn, and computer(action="wait_for") polls locally instead of burning model turns.
Does it hijack my physical mouse cursor while working?
On macOS, ApexUse renders an independent translucent AgentCursor overlay window and dispatches targeted events without warping your hardware cursor during normal visual feedback. On Windows, a layered click-through AgentCursor overlay tracks automated actions.
How is ApexUse different from Playwright or browser DevTools MCP servers? Browser MCPs only control a single automated browser tab via CDP. ApexUse controls the real desktop across native OS apps, Electron apps, terminals, dialogs, and user browsers (Chrome, Dia, Safari, Firefox, Edge).
Testing
# Rust unit & integration tests (38 tests)
cargo test --manifest-path rust/Cargo.toml
# End-to-end MCP stdio integration tests (requires python3 mcp + pytest)
.venv/bin/pytest -q
Layout
├── .agents/skills/apexuse/SKILL.md # Agent skill instructions for ApexUse MCP tools
├── .github/workflows/ci.yml # 3-OS CI matrix (macOS, Ubuntu Xvfb/AT-SPI, Windows)
├── .github/workflows/release.yml # Automated macOS/Linux/Windows release binaries
├── agent-configs/README.md # Client setup snippets (Antigravity, Gemini, Codex, Claude)
├── llms.txt # AI search & crawler index summary
├── policy.yaml # Default direct policy
├── policy.autonomous.yaml # Autonomous direct preset
├── pyproject.toml # Python MCP E2E test configuration
├── server.json # Official MCP Registry manifest
├── rust/
│ ├── Cargo.toml # apexuse-mcp crate (0 external dependencies)
│ ├── build.rs # Compiles Objective-C macOS bridge on macOS
│ ├── native/macos_bridge.{h,m} # AXUIElement, SCKit, Vision OCR, AgentCursor, events
│ └── src/
│ ├── driver/ # OsDriver trait + common.rs + macos.rs + windows.rs + linux.rs
│ ├── gateway.rs # Snapshots, policy enforcement, verify_change, run_actions
│ ├── json.rs # Zero-dependency JSON parser/serializer
│ ├── mcp.rs # JSON-RPC 2.0 MCP stdio server & tool schemas
│ ├── memory.rs # Structural workflow memory
│ ├── policy.rs # App/field security filters, rate limiter, confirmations
│ ├── som.rs # Set-of-Mark AX + OCR fusion
│ └── som_png.rs # Pure-Rust PNG codec, inflate, pixel digest & SoM renderer
└── tests/test_rust_mcp.py # End-to-end MCP stdio integration tests
Registry & License
- MIT License — see LICENSE
- Official MCP Registry: mcp-name: io.github.Jackpkn/apexuse
apexuse-mcpcargo install apexuse-mcp