
handy
io.github.miisodev/handy
Native mouse, keyboard, window, clipboard, and screen control for LLMs via stdio MCP.
Documentation
handy
Give your agent eyes and hands — see and drive the real desktop.
Real-time native mouse, keyboard, window, clipboard and coordinate-mapped screen control for LLMs and agents, served over the Model Context Protocol as a stdio server and CLI — with numbered Set-of-Mark targets, chained multi-step actions, an instant on/off permission toggle and a physical (0, 0) corner-slam failsafe.
Site · Quick start · The tool surface · How it works · Privacy
What is handy?
Agents get screenshots, but few get honest control of the machine they are looking at. Coordinates are guessed from pixels, one-shot clicks land where nothing happened, and the human watching has no switch to stop it all.
Handy is an MCP server (with a full CLI alongside) that gives a model real-time native control of a desktop — mouse, keyboard, clipboard and windows — paired with a live coordinate-mapped view of the screen: dual-contrast (x, y) rulers, active-window telemetry, pixel hex under the cursor, sub-2ms frame-to-frame diffs, and Set-of-Mark detection that numbers every interactive element on screen so the agent clicks mark 13 instead of guessing (631, 447). Up to 100 steps — focus a window, click, paste, wait for the UI to settle — run in one atomic chained call, with an after-shot that shows numbered trail markers where every action landed.
It installs cleanly on any PC with zero C++ addon compilation (node-gyp free), speaks plain stdio to any MCP client, and treats permission as a product surface: a ~/.handy/permissions.json toggle checked before every capture and every chain step, a --view-only mode, --ttl timers, a live loopback Control Panel, and a physical failsafe — slam your mouse into the top-left (0, 0) corner and any in-flight chain aborts instantly.
Who it's for
Developers and agent builders who want their assistant to actually operate the desktop — fill forms, drive native apps, verify what happened — rather than describe it. Any MCP-capable client works (Claude Desktop, Claude Code, Cursor, Windsurf, and everything else speaking MCP); the CLI makes the same surface usable from shell scripts and CI.
From experiment to open source
Handy began as the input layer for the author's own agents, then grew through daily real-machine use — real installs, real Wayland compositors, real failsafes — into a tool that behaves the same on Windows, macOS and Linux. What you're reading is the same code that runs the author's desktop every day, not a demo.
The core principle
An agent that can see and click must be interruptible.
Handy checks its permission store synchronously before every capture and before every individual step of a chain, and the failsafe reads the real cursor position each step, so a human can always stop the machine — by toggle, by corner-slam, or by timer. When the toggle is OFF, no MCP client can turn it back on by itself; handy_permission({ "action": "off" }) remains a one-way stop switch.
Design highlights
- Set-of-Mark vision (
marks: true) —handy_seedetects, classifies (control/text/icon/region), numbers and outlines the interactive elements on screen in reading order; every pointer step then accepts{ "mark": N }and the chain DSL understandsclick mark 5/click #5. Marks resolve to the element's own centre at execution time, so a chain survives layout shift and HiDPI scaling. Detection is DPI-independent (~18ms at 1280x800, ~22ms at 2560x1440, ~49ms at 4K) because the work buffer is area-capped rather than dimension-capped. - One atomic call for a whole interaction (
handy_act) — up to 100 sequential steps (focus_window,move,click,drag,scroll,paste,type,key,mouse_down/up,key_down/up,wait,wait_for_change,wait_for_settled,wait_for_color,screenshot) in a single call, withscreen/normalized(0..1000)/windowcoordinate spaces and numbered amber trail markers onscreenshotAfter. - Human-like pointer motion (
motion: "human") — Bézier arcs with Fitts's-law timing instead of teleporting straight lines, with the duration derived automatically from travel distance and target width when omitted. - Post-action verification (
verifyChange: true) — a baseline capture before the chain and a frame diff after, so the receipt says whether the click actually did anything. - Captures that survive real screens — dual-contrast rulers legible on
#000and#fffapps alike, area-average box-filter downscaling that keeps 1px text alive, and per-row PNG filter selection for ~26% smaller encodes. - Fail-soft enhancements — mark detection degrades to a plain screenshot (
marksError) instead of failinghandy_see, and stale marks fail loudly with the available indices instead of clicking somewhere arbitrary. - Strict chain-DSL validation — malformed pointer arguments throw with the offending segment quoted; out-of-range marks list what exists. Nothing degrades to "act wherever the cursor happens to be".
How it works
Handy is one TypeScript process with zero runtime dependencies beyond the MCP SDK and Zod. Each platform speaks its own native bridge — no wrappers, no VMs:
| Platform | Backend | How |
|---|---|---|
| Windows 10/11 | win32 | Persistent Win32 SendInput (KEYEVENTF_UNICODE, MOUSEEVENTF_VIRTUALDESK), EnumWindows/SetForegroundWindow, MTA-safe OpenClipboard, GDI CopyFromScreen, Per-Monitor V2 DPI awareness |
| macOS | darwin | Persistent CoreGraphics CGEvent + NSWorkspace + NSPasteboard, in-memory CGWindowListCreateImage PNG capture (with screencapture fallback), Retina 2x logical-point mapping |
| Linux — X11 | linux-x11 | Pure native X11 wire-protocol client (XTEST, GetImage, BIG-REQUESTS, RANDR, EWMH _NET_CLIENT_LIST/_NET_ACTIVE_WINDOW) — zero external packages |
| Linux — Wayland | linux-cli | Auto-selected on Wayland sessions: screen capture through the freedesktop screenshot portal (works out of the box on GNOME, whose Shell screenshot API rejects third-party callers) or grim on wlroots; input via ydotool (uinput daemon) or wtype/xdotool; clipboard via wl-clipboard/xclip/xsel |
| Headless / CI | virtual | Built-in 1280x800 multi-window interactive desktop workbench for safe agent sandboxing — no display required |
Everything above the bridge is shared: controller.ts enforces permissions and the failsafe between steps, marks.ts detects and numbers UI elements, overlay.ts renders rulers and reticles, png.ts handles decode/encode, and the MCP layer (mcp.ts) wraps it all as six tools.
A capture-and-act round trip looks like: the agent calls handy_see { marks: true } → the backend grabs a frame (portal on GNOME Wayland, native protocol on X11/Windows/macOS) → detection numbers the elements → the agent calls handy_act with a chain referencing mark indices → the controller re-checks permissions and the failsafe before each step → a receipt (and optionally a diff and after-shot) comes back.
Quick start
Requires Node.js 20+.
1. One-Command Auto-Setup (Claude Desktop, Cursor, Windsurf, Claude Code)
npx -y @miisodev/handy install
Preview changes first with --dry-run, target a single host (handy install cursor), or remove cleanly (handy uninstall). Backups (*.handy.bak) are created automatically before modifying existing config files.
2. Prepackaged Plugin (Claude Code / Agent Marketplaces)
Installs the stdio MCP server, the handy operational skill (skills/handy/SKILL.md), and the /handy:on, /handy:off, /handy:toggle, /handy:status, /handy:doctor, and /handy:windows commands together:
claude plugin marketplace add miisodev/handy
claude plugin install handy
3. Global CLI + Live Control Panel
npm install -g @miisodev/handy
handy doctor # verify input & capture readiness for your platform
handy ui # live Control Panel at http://127.0.0.1:7788
4. Manual Stdio MCP Configuration
{
"mcpServers": {
"handy": {
"command": "npx",
"args": ["-y", "@miisodev/handy", "mcp"]
}
}
}
Linux (Wayland) setup
Capture works out of the box on GNOME and any compositor whose portal serves Screenshot. Input needs a running ydotoold (uinput), which most distros ship:
sudo apt install ydotool # or your distro's package
sudo systemctl enable --now ydotoold
If your distro only ships a user-level unit (some recent Ubuntu releases do), it cannot open /dev/uinput; run the daemon as a root system service with a user-owned socket instead:
sudo systemctl edit --force --full ydotoold # then set:
# [Service]
# ExecStart=/usr/bin/ydotoold -p /run/user/1000/.ydotool_socket -P 0660 -o 1000:1000
# RuntimeDirectory=ydotoold
handy doctor tells you exactly which routes and tools it found — session type, capture routes, input tool.
The tool surface
All returns are structured JSON. Input mistakes come back as informational errors the model can self-correct from; every receipt names what was executed, where, and how long it took.
| Tool | Description |
|---|---|
handy_see | Capture the live screen, a single window, or a zoomed region with dual-contrast (x, y) rulers, cursor reticle, pixelUnderMouse, activeWindow, screenDiff, and highlightDiff. With marks: true it also detects, numbers, and returns every interactive UI element (elements). |
handy_act | Execute single or chained native mouse, keyboard, paste, window-focus, and smart visual-wait (wait_for_change, wait_for_settled, wait_for_color) controls with screen/normalized/window coordinates, mark element references, motion profiles, numbered trail markers, and optional verifyChange post-conditions. |
handy_windows | List visible desktop windows (title, app, bounds, focused) or focus a window by title, app, or id. |
handy_clipboard | Read or write plain-text system clipboard directly. |
handy_status | Inspect live permission toggle state (ON/OFF), mode (full/view_only), OS backend, displays, active window, and cursor coordinates. |
handy_permission | Query permission status or immediately turn Handy off. |
Example: See numbered elements, then act on them
// handy_see
{ "marks": true, "maxMarks": 30 }
// handy_act — marks 5 and 13 come from the capture above
{
"actions": [
{ "type": "click", "mark": 5 },
{ "type": "paste", "text": "Sam — ready for review 🚀" },
{ "type": "click", "mark": 13 },
{ "type": "wait_for_settled", "quietMs": 200 }
],
"verifyChange": true,
"screenshotAfter": true
}
handy_see returns each element's index, kind, bounds, centerX/centerY, and a score; marks stay valid in the server process until the next marked capture. The CLI re-detects on demand per invocation (detection is deterministic for an unchanged screen), and one chain can do both atomically:
handy see --marks --out marked.png
handy act 'click mark 5 -> paste "Sam" -> click mark 13 -> wait_for_settled' --screenshot-after after.png
handy act '[{"type":"screenshot","marks":true},{"type":"click","mark":5}]'
Standalone CLI
handy see --out screen.png # coordinate-mapped capture
handy see --window "Scratchpad" --highlight-diff --out shot.png
handy see --marks --max-marks 20 --json # numbered UI elements
handy windows && handy focus "Visual Studio Code"
handy clipboard get | handy clipboard set "Copied from terminal"
handy act 'focus "Scratchpad" -> click 800 420 -> paste "Hello 🌍" -> wait_for_settled'
handy key ctrl+s # chords work on every platform bridge
Deployment
Permission controls & human failsafe
Handy stores its permission toggle at ~/.handy/permissions.json (0600) and checks it synchronously before every handy_see capture, handy_windows/handy_clipboard call, and before every individual step of a handy_act chain:
handy off # immediately block all LLM screen viewing & input control
handy on # full screen vision & mouse/keyboard control
handy toggle # flip between ON and OFF
handy on --view-only # allow screen viewing, block mouse & keyboard
handy on --ttl 30m # auto-disable after 30 minutes
handy status # toggle state, backend, windows & displays
- Physical failsafe: slamming your mouse into the top-left
(0, 0)corner aborts any in-flight chain and releases held keys/buttons immediately — the controller reads the real cursor position each step, so it works even when the agent commands a different pointer path. - Prompt-injection guard: when turned OFF by the user, an MCP client cannot re-enable Handy on its own;
handy_permission({ "action": "off" })is always permitted as a one-way stop switch.
Backend selection & environment
The backend is picked automatically per platform; HANDY_BACKEND (auto, virtual, linux-x11, linux-cli/wayland, darwin, win32) overrides it. On a Wayland session the CLI backend is chosen even when DISPLAY is set (XWayland), because that is the only backend that can see the real desktop; on a pure X11 session the native X11 client takes over with full window listing and focus support. HANDY_ENABLED=false in the host environment is a second, independent kill switch.
Known platform limits, stated plainly: handy_windows/focus_window enumerate windows through the X11/Wayland-of-record APIs (EWMH, wmctrl) and see XWayland windows only on GNOME Wayland — native windows there are not publicly enumerable by any client-side API. Capture and input are unaffected.
Adapting to your environment
A handful of values reflect the author's machine rather than yours. None are secrets:
| What | Where | Make it yours |
|---|---|---|
| Linux input daemon | ydotool + ydotoold (see Linux (Wayland) setup) | Distros differ in how the daemon ships; handy doctor reports what it found |
| Capture tools (non-GNOME Wayland) | grim (wlroots) / maim / scrot | Optional — the portal route needs nothing installed |
| Display override | HANDY_BACKEND in the MCP host env | Only when auto-detection needs help |
| Permission store | ~/.handy/permissions.json | Created on first run; 0600 |
| Author · repo · funding | package.json, the badges above | Your own details if you fork; the funding links support the original author |
Architecture
src/
├── index.ts public API barrel
├── mcp.ts six MCP tools over stdio + resources + prompts
├── controller.ts permission + failsafe enforcement between steps, chains, receipts
├── cli.ts the `handy` command surface
├── installer.ts host auto-setup (`handy install`, backups, uninstall)
├── marks.ts Set-of-Mark detection, DPI-independent, fail-soft
├── overlay.ts dual-contrast rulers, reticles, trail markers
├── png.ts decode/encode, box-filter resize, per-row filters
├── diff.ts frame diffing with early exit
├── permissions.ts ~/.handy/permissions.json store, TTLs, view-only
├── keys.ts per-platform key maps (X11 keysyms, Darwin, Win32 VK, Linux EV_KEY)
├── ui.ts loopback Control Panel (SSE)
└── backends/
├── win32.ts SendInput, EnumWindows, GDI capture
├── darwin.ts CGEvent, NSWorkspace, CGWindowList
├── linux-x11.ts native X11 wire protocol client (XTEST, GetImage, RANDR, EWMH)
├── linux-cli.ts Wayland: portal capture + ydotool/wtype/xdotool/wl-clipboard
└── virtual.ts headless 1280x800 workbench for CI
Key properties: no native build steps (pure wire-protocol clients), permission checks synchronously before every effect, the failsafe reads the real cursor each step, mark references fail loudly when stale, and every route that can fail falls through with its error named.
Development
npm run typecheck # src + tests, strict
npm run build # lib/, bin/handy.js, public/handy.png
npm test # vitest across every backend bridge and the stdio MCP surface
npm run validate # version-site agreement, MCPB manifest, plugin commands, skill references
Real input is exercised against real desktop sessions — Windows SendInput, macOS CGEvent, Linux XTEST/uinput — with the virtual backend covering headless CI; a mock X11 server tests the binary protocol byte-for-byte.
Contributing
Handy is maintained by its author with contributions welcome: open an issue first for anything non-trivial, keep changes focused, match the existing code style, and update the docs a change touches — a tool change without its README/skill update is half a change. npm run typecheck, npm run build and npm test must pass clean; CI runs the same checks on every push and PR, on all three operating systems.
Security
Please do not report suspected vulnerabilities in a public issue — Handy is, by design, a tool that can click and type on someone's computer. Follow the private reporting process in SECURITY.md; never include screenshots or captured screen content in a report.
Privacy and support
Handy adds no analytics, no telemetry and no hosted account. Screen captures go only to the MCP client that asked for them; permission state lives in ~/.handy/; the Control Panel binds to loopback only. The full privacy policy says what is stored where. For help, open a GitHub issue or email miisodev@gmail.com.
Support the project
Handy is built and maintained by one person. If it's useful to you, sponsorship directly funds its continued development:
Starring the repo, reporting bugs, and spreading the word help too.
License
Credits
- Model Context Protocol — the open standard that lets one server drive every MCP-capable client.
- The GNOME, wlroots and freedesktop portal teams — Wayland's capture and input story works because of them.
@miisodev/handynpm install @miisodev/handy