Repository avatar
AI Tools
v1.0.2
active

tapir

io.github.anilcancakir/tapir

Drive a live Flutter app from an AI agent: snapshot the Semantics tree, tap, type and screenshot.

Documentation

Tapir logo

Tapir

Give your AI agent eyes and hands on a running Flutter app.
Snapshot the Semantics tree, tap, type, scroll, screenshot and profile the live app from one CLI and one MCP server, with no flutter_test harness and no test file between actions.

pub package CI codecov License: MIT pub points

Documentation · pub.dev · Issues

At a glance

FactValue
Packagetapir
Installflutter pub add tapir, then dart run tapir tapir:install
Importpackage:tapir/tapir.dart (TapirPlugin.install(), behind if (!kReleaseMode))
Entry pointsdart run tapir <command>, ./bin/dowel <command>, TapirPlugin.install(), TapirPlugin.enrichers, TapirDowelProvider (package:tapir/cli.dart), dart run tapir mcp:serve
PlatformsAny Flutter target that exposes a VM Service in a debug or profile build: Android, iOS, web, macOS, Windows, Linux; viewport and device emulation need Chrome with a CDP port; tapir_hot_reload_and_snap writes to the flutter run stdin FIFO, so it needs a POSIX host; on a Windows host mcp:install writes a dart run entry, never ./bin/dowel
RequiresDart 3.13, Flutter 3.47; a plugin of dowel (^1.0.0)
Dependenciesdowel, image, meta, wind_kit_diagnostics_contracts, yaml, the Flutter SDK
LicenseMIT

Install

flutter pub add tapir
dart run tapir tapir:install
dart run tapir start --device=macos   # dowel records the session every tapir:* command dials
dart run tapir tapir:snap

tapir:install patches lib/main.dart (the import, WidgetsFlutterBinding.ensureInitialized() when missing, and TapirPlugin.install() inside a !kReleaseMode guard), then scaffolds the fast ./bin/dowel wrapper and registers tapir as a dowel plugin. Re-running it changes nothing. The installation guide also adds dowel as a direct dependency (dart pub add dowel). Wired by hand, main.dart looks like this:

import 'package:flutter/foundation.dart';
import 'package:flutter/widgets.dart';
import 'package:tapir/tapir.dart';

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  if (!kReleaseMode) {
    TapirPlugin.install();
  }
  runApp(const MyApp()); // MyApp is the host app's root widget.
}

The guard is !kReleaseMode, not kDebugMode: debug and profile builds carry tapir (tapir:perf_run measures a profile build) and a release build tree-shakes it. Without the guard tapir ships in release, and anyone who reaches that app's VM Service can drive it. --dart-define=TAPIR_DISABLE=1 skips registration in a build that must not expose the extensions. The example/ app gives each verb a live target widget.

Snapshot and refs

dart run tapir tapir:snap --interactiveOnly
# - textbox "Email" [ref=e1]
#   typeable: true
# - button "Sign in" [ref=e2]
TokenMinted byLifetime
e<N>tapir:snapThe same Semantics node keeps its token across snapshots, and a token outlives its widget. A ref whose element left the tree is relocated by label (reresolved on the response) only when the snapshot recorded a binding for it, the live tree has the same number of nodes with that label and the match sits on the route the ref was minted on; otherwise it fails as defunct and you re-snap
q<N>tapir:find, tapir:observeRe-resolved against the live tree on every action; survives rebuilds

Narrow a read with --within=<eN>, --interactiveOnly and --grep=<regex>; they compose. --includeEnrichers adds Wind state under each W-widget that has a className: up to seven contract fields, of which bgColor and textColor appear only when they resolve (tapir:observe keeps only breakpoint and states unless you pass --includeEnrichers=full; the host calls Wind.installDebugResolver() from wind_kit, read through wind_kit_diagnostics_contracts) and Magic state (MagicTapirIntegration.install() from package:magic_devtools/tapir.dart, after Magic.init(); it exists only in a magic_devtools release built on tapir, and tapir:install warns and skips it otherwise; see Magic integration). A ref line ending in [offscreen] or [occluded] names why an action would fail before you try it.

Actions and the actionability gate

dart run tapir tapir:tap --ref=e2 --snapshot=diff
dart run tapir tapir:fill --ref=e1 --text="ada@example.com" --json

Gestures (tap, hover, drag, scroll, double_click, right_click, triple_click) and text verbs (type, clear, fill, press_key) dispatch real pointer and key events into the widget tree. tap, hover, drag, the click variants, type, fill and set_checkbox (on the control it taps) first pass the gate (clear, scroll, press_key and select_option skip it), in a fixed order, and a refusal names the reason an agent can branch on:

CheckRefusal substringRecovery
Defunct (preflight)defunct (...)Re-snap
Enablednot enabledRe-snap; the widget may enable later
Zero rectzero rectRe-snap or re-find
Off-viewport (scrolls into view first)off-viewporttapir:scroll into view, then retry
Stable across two framesnot stableWait for the animation; --no-checkStable
Receives events (hit test)obscured byDismiss the overlay (tapir:reset_overlays); --no-checkReceivesEvents

A green tick is dispatch, not effect. Add --json to print the whole response and read the effect block: effect.changed for tap and scroll, effect.verified and effect.value for type, fill and clear. When the hit test could not answer (routine on a web debug build) the response of a gated verb such as tap, type or fill carries a checks block with receivesEvents: indeterminate instead of a silent pass. A target tapir will not act on at all (a ref that is not a text field, say) answers the error type refused (code -32001), and a value it cannot use answers invalid_param. tapir:run "tap --ref=e5" "snap --grep Counter" batches verbs into one round trip.

Waiting and observing

dart run tapir tapir:observe --roles=button,textbox   # q<N> handles, re-resolved per action
dart run tapir tapir:wait_for --text="Welcome back" --timeoutMs=5000
VerbReturns
tapir:wait_forWaits for --text, --textGone or --expression
tapir:wait_for_network_idleWaits until in-flight HTTP requests stay at zero for --idleMs
tapir:observeA flat candidate list of interactive widgets with q<N> handles: observe once, act many times (no model call inside tapir)
tapir:console, tapir:exceptionsIn-app debugPrint output and FlutterErrors, overflow included; --since=<iso8601> scopes errors to one action
tapir:get_routesThe router's location and the top page (tapir:navigate and tapir:navigate_back change it)

When the app is not producing frames (a hidden window or a backgrounded tab) tapir forces frames outside the web and reports framesForced: true; where it cannot, every response carries a warnings block rather than a stale answer that looks fresh. See frame production.

Screenshots and web

dart run tapir tapir:screenshot --output=shot.jpg
dart run tapir start --device=chrome --cdp-port=9333
dart run tapir tapir:device_profile --preset=pixel-8
SurfaceBehaviour
tapir:screenshotJPEG (default) or PNG of the viewport, or of one widget with --ref (and a sub-rect of it with --rect)
Web screenshotWith --cdp-port, the CLI captures through CDP Page.captureScreenshot; the tapir_screenshot MCP tool stays in-isolate and can time out under CanvasKit, so prefer the CLI on web; a --cdp-port session cannot run tapir_evaluate (the web-server device has no evaluate)
tapir:device_profilePresets iphone-x, iphone-13, iphone-15-pro, pixel-5, pixel-8, ipad-pro-12.9, desktop-1440, desktop-1920 (viewport, DPR, touch, user agent); the emulation ends with the command, the window size stays. tapir:resize_viewport --width --height overrides the viewport until exit, or longer with --hold
tapir:hot_reload_and_snapHot reload, then snapshot, screenshot and recent exceptions in one round trip

Performance tracing

dart run tapir tapir:perf_begin --mode=attribution
dart run tapir tapir:tap --ref=e7
dart run tapir tapir:perf_end
VerbDoes
tapir:perf_begin, tapir:perf_endBracket one interaction; attribution ranks the widget builds that cost time next to Wind and Magic counters, timing reports frame timings you can quote. A session in which the engine rendered nothing is refused (refused: true, exit 1), never reported as zeros that read as fast
tapir:perf_insightThe rows behind one insight of the last report
tapir:perf_run, tapir:perf_compare, tapir:perf_campaignRepeat a scenario from a clean start and write medians, judge one run against another, run a campaign of scenarios from cold starts
tapir:perf_trace --out=trace.jsonThe last session as a Chrome Trace JSON for ui.perfetto.dev

MCP server and clients

The tapir_* tools ride on the dowel MCP server, so the client config key is dowel. dart run tapir mcp:install writes the entry into .mcp.json: ./bin/dowel mcp:serve when the fast wrapper exists on a POSIX host, and dart run tapir mcp:serve otherwise (always on Windows, where ./bin/dowel is not used). Standard config for any client that reads .mcp.json:

{ "mcpServers": { "dowel": { "command": "./bin/dowel", "args": ["mcp:serve"], "cwd": "." } } }
ClientInstall
Claude Codeclaude mcp add dowel -- ./bin/dowel mcp:serve, then /mcp reconnect dowel after a change
VS Codecode --add-mcp '{"name":"dowel","type":"stdio","command":"dart","args":["run","tapir","mcp:serve"]}'
Cursor.cursor/mcp.json (project) or ~/.cursor/mcp.json (user)
Windsurf~/.codeium/windsurf/mcp_config.json
Goose, WarpPaste the standard config into Goose (Advanced settings, Extensions, Add custom extension) or Warp (Settings, AI, Manage MCP Servers)
Claude Desktopclaude_desktop_config.json (macOS: ~/Library/Application Support/Claude/, Windows: %APPDATA%\Claude\)
Codexcodex mcp add dowel ./bin/dowel mcp:serve

Without tapir you hot reload, take a screenshot, drag the PNG into the chat, paste the widget source and ask why the button does nothing. With it the agent calls tapir_snap, then tapir_tap on e7: the gate checks the button, the tap lands and a fresh snap confirms it. A timing issue no longer needs a Future.delayed and a re-run to tell it from a bug, because the gate refuses with a reason that names the cause (not stable, obscured by, off-viewport).

For AI coding agents

DoBecause
Install tapir only behind if (!kReleaseMode)Anyone who reaches the VM Service of a build that carries tapir can drive the app
Start the app with dart run tapir start, not flutter runEvery tapir:* command dials the session dowel records
Snap again after a navigation or rebuild, or use a q<N> handle from tapir:finde<N> refs are frozen to the snapshot that minted them
Narrow a snapshot with --within, --interactiveOnly or --grepA full tree costs context, and a sidebar that repeats page labels misleads an exact-label lookup
Read effect (and checks when present) before trusting a green tickA tick says the event left tapir, not that the widget took it
Branch on the leading refusal substring, never on the trailing detailThe substrings and the gate order are a frozen contract; the detail is not
Read framesForced and warnings on a response that looks emptyA hidden window or tab stops frames, and the tree then reads as a broken screen
Check refused first on a tapir:perf_end reportA refused session carries no metrics
Point the MCP client at the dowel key and reconnect after mcp:installThe tools ride on the dowel server, and clients do not reload an entry on their own

Dependencies

dowel for the CLI, the MCP server and the VM Service connection; wind_kit_diagnostics_contracts to read Wind state without depending on Wind; image for screenshot encoding; yaml and meta. The rest is the Flutter SDK.

Links

License

MIT, see LICENSE. Built by Anılcan Çakır.