Repository avatar
Other Tools
v0.6.2
active

efaimo

io.github.efaimo-ai/efaimo

Read-only tools that grade an Agent Skill and measure its context-window token cost

Documentation

efaimo. The audit CLI for Agent Skills and MCP servers.

ci npm license node >= 22

efaimo audits everything your agent loads: MCP servers and Agent Skills.
One CLI lints them, weighs their context cost, diffs servers against the 2026-07-28 spec,
and A/B-tests whether a skill actually helps.

Everything you plug into an agent spends two budgets before any work happens: context-window tokens, and the trigger quality that decides whether the right tool fires at all. Good tools exist for single slices of that problem; efaimo's job is the whole audit in one command, for both halves of what an agent loads. Lint a skill, weigh a server's tool definitions, get a migration diff for the 2026-07-28 stateless MCP spec, and measure whether a skill actually improves task completion (SkillsBench found curated skills made agents worse on 16 of 84 tasks; linting alone cannot catch that).

Agent Skills

npx efaimo check --skill ./skills/

Point it at one skill or a whole folder. It validates each skill against the agentskills.io spec and grades it:

check skill  claude-api
grade C (73)   1 error  2 warnings  4 info

  x S101  description is 1068 chars (spec max 1024)
  ! S104  instructions are ~18.4k tokens (spec recommends staying under 5k)
          fix: move detail into references/ files loaded on demand (progressive disclosure)
  ! S104  SKILL.md is 570 lines (spec recommends under 500)
  i S104  metadata level is ~294 tokens; the spec targets ~100 and this loads at
          startup for every installed skill

(That is Anthropic's own claude-api skill.) efaimo checks frontmatter and trigger quality, collisions across an installed set, the context budget (metadata loaded every session, body loaded on trigger), reference integrity, and injection hygiene. Run it on a folder and every skill gets its own grade, not one aggregate.

We graded 36 public Agent Skills: every skill in anthropics/skills, anthropics/claude-cookbooks, and obra/superpowers, at pinned commits. 97% score an A, but even this curated set has real issues: Anthropic's own claude-api scores a C (73) with an over-limit description and ~18k-token instructions, and the median skill's instructions run ~1,632 tokens loaded on every trigger. Full report, the corpus manifest, and the two commands that reproduce it: the Skills Quality Index.

That index is a photograph. Measured twice re-runs the same corpus seven weeks later with the same ruleset: the cost that sits in context permanently moved -0.8%, the body +7.0%, and referenced files, which load out of sight and which no rule here counts, +61.4%. Grades moved one way only, 0 improved against 4 worse. The uncomfortable part points inward: the two columns this tool has thresholds for are the two that held.

Does the skill actually help? (experimental)

Linting tells you a skill is well-formed. It does not tell you the skill makes the agent better, and research shows some skills make it worse. efaimo test measures that directly: it runs a task with and without the skill, N trials each, and an LLM judge scores every attempt.

npx efaimo test scenario.yaml                 # dry run: validate + show the plan, no API calls
npx efaimo test scenario.yaml --live          # run it for real (Claude or GPT)
npx efaimo test scenario.yaml --live --model gpt-4o-mini

Works with Claude (ANTHROPIC_API_KEY) or GPT (OPENAI_API_KEY) models; the provider is picked from the model name. Put the key in your shell or a local .env file (copy .env.example); a real shell variable always wins.

Two real runs (claude-sonnet-5, 8 trials each) show why this matters: a generic csv-cleanup skill that a capable model does not need, and a skill that encodes a convention the model cannot guess.

efaimo test A/B results: csv-cleanup measures +0 points (no measurable effect) while contoso-crm-import measures +100 points (helps); both skills lint clean at grade A

The two runs as copyable text
test  csv-cleanup helps on a messy CSV
  with skill     8/8 pass  (100%)
  without skill  8/8 pass  (100%)
  delta          +0 points   no measurable effect
test  contoso-crm-import helps on an unknowable format
  with skill     8/8 pass  (100%)
  without skill  0/8 pass  (0%)
  delta          +100 points   helps

The first is pure context overhead, the second earns its tokens, and only the trial data tells them apart.

The verdict is a significance test, not a threshold on the gap. Every run reports a two-sided Fisher exact p and a 95% interval on the delta, and helps / hurts require p < 0.05; anything else is no measurable effect no matter how large the gap looks. That matters more than it sounds: at the old default of 5 trials per arm, 5/5 against 4/5 is +20 points and a p of exactly 1.0000, and the previous rule called it helps. The +100 run above is p = 0.0002; a +25 run at 8 trials per arm is p = 0.47 and now says so. The default is 20 trials per arm, because below about that a partial result cannot reach significance at all.

It is opt-in because a live run spends tokens on your key. See examples/scenario.example.yaml and examples/scenario.crm.yaml.

MCP servers

npx efaimo weigh "npx -y my-mcp-server"      # what does it cost my context window?
npx efaimo check --mcp "npx -y my-server"    # quality grade + 2026-07-28 migration diff

weigh reports the token cost of tool definitions in three real serializations, per tool, with an optional --anthropic exact Claude count.

efaimo weigh example output

check --mcp connects live (it speaks both the legacy handshake and the new stateless protocol, so a 2026-07-28 server audits fine) and reports two things, separately. First, a quality grade for what models actually experience: descriptions, schemas, annotations, tool count, token cost. Second, an ungraded migration diff for the 2026-07-28 stateless spec (which removes initialize and sessions, deprecates Sampling, Roots, and Logging, and requires server/discover, resultType, and cache fields): exactly what will break and how to fix it, each rule naming the SEP it came from. Readiness never drags the grade, because the two answer different questions: the grade is what a model experiences from this tool surface, readiness is whether a 2026-07-28 client can talk to this server. Combining them scored the reference server a C for a tool surface that is an A. Pass --strict-readiness when you want CI to fail on the diff. Full list: docs/RULES.md.

efaimo check example output: quality grade A (95) plus a one-item 2026-07-28 migration diff for the reference server

(That is the official reference server: solid quality, one thing left to migrate. It had four until it moved to the 2.x SDK, which answers server/discover and fills in the cache and resultType fields for you.)

The same output as copyable text
check mcp  npx -y @modelcontextprotocol/server-everything
grade A (95)   quality: 0 errors  1 warning  0 info

  ! E122  tool "echo": description misses 3/4 quality axes (length 40..600;
          says when to use it; mentions the result)

2026-07-28 readiness  1 item to migrate (a migration diff, not graded)
  ! E104  server declares the logging capability; MCP Logging is deprecated in
          2026-07-28 (SEP-2577) and logging/setLevel is removed

Can a search even single your tools out?

A host can mark tools defer_loading: true, which keeps their definitions out of the context window until a search returns them; Anthropic recommends turning that on once tool definitions pass ~10k tokens. Under deferral, a tool nothing surfaces costs no context and provides no capability, and neither weigh nor check looks at that.

npx efaimo find "npx -y my-mcp-server"

Two numbers come back, and they are different kinds of claim.

distinct is a property of your catalog: how many tools own a term no other tool has. A tool that owns none cannot be matched by any query that does not also match a competitor. That follows from the index, not from a model of how anyone searches, and it is the number --min-distinct gates on in CI.

distinct  22/24 (91.7%)   tools that own a word no other tool has
probe     24/24 (100%)    returned by a simulated search for their own description

  words  tool                     vocabulary no other tool has
      0  browser_close            none: every word also on browser_click, browser_drag, browser_drop
      0  browser_navigate         none: every word also on browser_tabs, browser_navigate_back, browser_network_requests
      1  browser_evaluate         evaluate
      1  browser_hover            hover
      2  browser_resize           window, resize

(That is @playwright/mcp 0.0.78, a well-maintained server, verbatim. Two of its twenty-four tools have no word of their own, and two more own only the word in their own name, which nobody describing a task would type.)

Four public servers measured this way, with the commands that reproduce each row, are in the findability report. The weakest was Notion's official server at 16 of 24, where API-get-user and API-get-users are different tools and nothing in their vocabulary separates them.

probe is a simulated BM25 search over the four fields tool search matches on, and it is labelled a simulation because it is one. It reads 100% on both the official reference server and playwright, so it is a floor test that catches an empty description or a literal duplicate, not a ranking. find says that on every run rather than letting a saturated number look like a good score.

There is no findability grade, on purpose: docs/METHODOLOGY.md has the method, its limits, and why.

From an agent

efaimo mcp runs efaimo as a small, read-only MCP server, so an agent can lint or weigh a skill mid-session, before it commits it to context:

npx efaimo mcp      # stdio server exposing efaimo_check_skill and efaimo_weigh_skill

It reads files only (it spawns no process and opens no socket), and the token-spending test is not exposed. Client config and recipes are in docs/INTEGRATIONS.md.

How it works

efaimo sits between your agent and everything it loads, measures the cost the way your host actually serializes it, and grades the quality, before any of it reaches your context window.

efaimo weighs, checks, and tests what your agent loads

How it compares

Focused tools already own single slices of this space, and some are very good. efaimo is the one tool that covers the whole audit surface. As of mid-2026:

efaimoskill-validatorskillgradeupskillmcp-spec-checkconformance (official)
Agent Skills lintingyesyesnononono
Skill token costyes (3-level split)yesnononono
Skill outcome testingwith/without A/B trials (experimental)static LLM scoringyes, and more thoroughly: Docker-isolated runs, script + LLM-rubric graders, weighted scores, CI pass-rate gatesyes (generate + eval, code agents)nono
MCP tool-definition costyes (3 serializations, --anthropic exact)nonononono
MCP quality rulesyesnonononono
2026-07-28 readinessmigration diff: what breaks and how to fixnononoyes/no verdictofficial test suite
CI budget gate + badgeyes (token budget)nopass-rate thresholdnonono

Read that table honestly: on skill outcome testing, skillgrade is the more serious tool and we would use it over our own efaimo test for anything depending on the result. Ours is one experimental subcommand in an audit CLI; theirs is a testing framework. efaimo's claim is the row count, not any single row.

SkillsBench, whose paper this README cites, is not in the table because it is a benchmark rather than a tool you point at your own work: a fixed suite of tasks with packaged environments and verifiers, extensible by contributing tasks upstream. Useful for the question "do skills help in general", which is why we cite it, and not for "does mine help", which is what efaimo test and skillgrade answer.

efaimo runs the official conformance suite for you (check --conformance on http targets) and complements security scanners such as Snyk agent-scan and the skills installer (npx skills) rather than replacing them. If your server is built on the TypeScript SDK, efaimo tells you what breaks and why, and @modelcontextprotocol/codemod rewrites the SDK v1 to v2 call sites for you. Diff, codemod, re-check.

"Doesn't tool search already solve this?"

Partly, and that is the point. Anthropic ships a tool search tool: mark tools defer_loading: true and the model discovers schemas on demand instead of carrying all of them, with discovered schemas appended so the prompt cache survives. Where it applies, it is a better fix than anything a linter can tell you, and efaimo's own rules point at it (E127 and E128 suggest deferred loading as a remedy).

It does not make measurement unnecessary, for three reasons. It is opt-in per tool, so somebody has to decide which tools are worth deferring, and that decision needs the per-tool numbers. It is a client-side feature, so it says nothing about what your server publishes to hosts that do not implement it. And a mitigation you cannot measure is a mitigation you are trusting on faith: weigh reports the definitions your server actually ships, which is the input every mitigation operates on.

And there is a fourth reason we did not have when this section was written. Deferring a tool does not remove the question, it replaces it: a deferred tool that no search returns costs nothing and does nothing, so the thing worth measuring becomes whether each tool can be picked out of the catalog at all. efaimo find measures that, and on four public servers it found two to eight tools apiece that own no word their neighbours lack. Deferral is what makes findability matter, not what makes it moot.

If deferred loading covers your whole setup, run weigh and find once and see what it costs you and what it hides; those answers are worth more than our opinion.

Install

Nothing to install. Use npx:

npx efaimo check --skill ./skills/

Or add it to a project with npm i -D efaimo, or the GitHub Action:

- uses: efaimo-ai/efaimo@v0
  with:
    command: check --skill ./skills --strict

The Action is listed on the GitHub Marketplace.

Gate a pull request on context-window growth, too:

npx efaimo weigh "npx -y my-server" --out base.json          # record a baseline
npx efaimo weigh "npx -y my-server" --diff base.json --allow-increase 10

--badge badge.svg writes an SVG plus a shields.io endpoint JSON for your README. These two are real efaimo output for the reference server:

context cost: 1759 tok (o200k) efaimo: A (95)

More recipes (pre-commit, GitLab, editor audit, programmatic use): docs/INTEGRATIONS.md.

Rules at a glance

familycovers
S101 to S108skills: frontmatter and trigger quality, trigger collisions, context budget, reference integrity, injection hygiene, a filename one capitalisation away from SKILL.md, and a reference file nothing points at
E101 to E118MCP 2026-07-28 readiness: deprecated primitives, statelessness, server/discover, resultType, cache fields, transport
E121 to E128, E130MCP quality: description quality, annotations, schema hygiene, tool-count and token-cost budgets
E141 to E146MCP findability: exclusive vocabulary, whether a search returns the tool, indistinguishable pairs, names that carry no domain word

E000 is not a finding about your server: it means one of efaimo's own rules threw and was skipped, so the report is incomplete. It is never scored.

Quality (E12x-E13x) and skill (S) findings set the letter grade; readiness findings (E101-E118) are reported as an ungraded migration diff, which --strict-readiness can turn into a non-zero exit without touching the grade. Findability findings (E14x) belong to efaimo find and never touch a grade either. Every finding carries a stable id you can suppress or link. See docs/RULES.md.

Roadmap

  • Harden efaimo test: a length-matched placebo arm (the control currently gets no system prompt at all, which confounds the skill's content with having any system prompt), multi-turn tool-use trials, and judge calibration, so the experimental harness earns unqualified trust. A separately chosen judge model and confidence intervals on the delta are done.
  • Readiness rules for the two 2026-07-28 changes that have none yet: the Mcp-Method / Mcp-Name request headers (SEP-2243), and Tasks moving out of core into the io.modelcontextprotocol/tasks extension (SEP-2663).
  • A public, continuously updated Agent Skills Quality Index over a broad corpus.

Stability

efaimo is 0.x. The commands are stable, but the rule set, grades, and exact output may change between minor versions until 1.0; pin a version in CI if you need reproducible thresholds. Every finding keeps a stable id.

Honest scope

efaimo is a linter and cost profiler, not a security scanner. Its injection checks are info-level heuristics that an attacker evades trivially; a clean report is not a security pass. For supply-chain safety use a dedicated scanner such as Snyk agent-scan. Token figures are estimates unless you opt into --anthropic; the method and its known bias are in docs/METHODOLOGY.md.

About

Built by efaimo ai, open tooling for the space between hosts and tools. efaimo is the flagship CLI; capabilities grow as subcommands under one name. Its site is efaimo.com. Apache-2.0. See CONTRIBUTING.md and docs/RULES.md to add a rule.

EFAIMO AI is a registered trademark of the publisher (KR 40-2585820, class 42). The lowercase efaimo ai used throughout is the same mark; the package name and the commands stay lowercase.

NPM
efaimo
Install Command
npm install efaimo
Runtime: npx