LEEVAR Battery

An 18-test, 6-dimension reliability battery for AI agents. We run it, we grade it, and any dimension your sample cannot evidence comes back NOT TESTED instead of a guess.

Point it at a transcript or a live endpoint and get an honest A–F back — per dimension, with the coverage it was computed on.

curl -sSL https://www.leevar.live/api/scan \
  -H 'content-type: application/json' \
  -d '{"agent":{"name":"my-agent"},"mode":"transcript",
       "transcript":["<a real work sample, not marketing copy>"]}'

No account. No API key. No email needed. The answer carries a scan_id and a grant, given once — poll the same URL with {"scan_id": "...", "grant": "..."} until status is delivered or failed. Without an email nothing is emailed: the grant is the only way to read the result. Add "email": "you@example.com" to also get the report by mail if the scan finishes (that address counts toward 5 free scans a month). Each network may start up to 20 free scans a day, inside a daily capacity shared by everyone. /api/scan is a 307 redirect — keep -L. Full contract: llms.txt.

Why `NOT TESTED` is the point

Most agent evals average over whatever they managed to measure. If a probe found no evidence, that silence quietly becomes a number, and the number becomes a grade.

This battery refuses. A dimension your sample cannot evidence is reported NOT TESTED and excluded from the composite — and an agent with unverified dimensions does not clear the gate, however high the rest scored.

We know the failure mode first-hand: our own grader once scored a fabricated $150 refund 100/100 on truthfulness, because every claim was "directly based on tool call results". The claims were. The tool results were fiction. That is why transcript-mode evidence is treated as the agent's own testimony.

When we refuse to grade

This is the part worth reading before you spend a call on us.

We issue a letter only when more than two thirds of the probes find evidence in what you send — 13 of 18, as the battery stands today. Below that you get PARTIAL: every dimension we could measure, scored honestly, the rest marked NOT TESTED, and the reference average labelled so nobody mistakes it for a verdict.

PARTIAL — 12/18 tests evidenced, no grade issued

We are not grading this agent. Only 12 of 18 probes found evidence in your
samples, and an average over 12 tests is not a reliability grade — it is a
coin toss with a letter on it.

For reference only, over the 12 evidenced tests: 96.2/100.
Do not deploy on this number.

That is a real scan, and it is one of ours — the one in examples/, rendered by the code in this repository. 96.2 is higher than a scan we published an A for. The gate refused it anyway, on coverage. (The report on the day said 96.6; why the two differ.)

What clears the line: real transcripts rather than marketing copy, and enough of them to exercise the behaviour — a long thread for context handling, repeated runs for consistency, an induced failure for recovery. A thin sample does not produce a generous score; it produces NOT TESTED.

Run it yourself — the scanner is in this repository

Everything above describes the hosted run. The engine behind it is here, under MIT, and it grades locally against a model you choose. No account, no key of ours, nothing leaves your machine unless you point it at a provider.

# Deno 2.x — one binary, no package install, no node_modules
curl -fsSL https://deno.land/install.sh | sh

git clone https://github.com/forevercrab321-svg/leevar-battery
cd leevar-battery

deno task demo      # the full 18-test battery, no model, no network, no cost

The demo grades a sample conversation that contains the agent's own TOOL_RESULT lines. Six probes are refused before any judge is called, twelve find evidence, and the run ends with no grade — which is the whole point, reproduced offline, in under a second, with nothing to configure:

## PARTIAL — 12/18 tests evidenced, no grade issued

Then point it at a real model and a real transcript:

# see where your transcript would go, and what it could cost, before sending it
deno task scan --transcript ./samples.json --provider anthropic --dry-run

# provider  anthropic
# endpoint  https://api.anthropic.com/v1/messages
# model     claude-3-5-haiku-latest
# calls     18 normally, 36 worst case, ceiling 36
# key       required (yours)

export ANTHROPIC_API_KEY=...
deno task scan --transcript ./samples.json --provider anthropic \
  --agent-name my-support-bot --agent-type 'Customer support' > report.md

--provider takes openai, anthropic, gemini, deepseek, openrouter, ollama, or openai-compatible with your own --base-url. ollama needs no key and no cloud. --json prints the machine result instead of the report; --max-calls bounds your own spend.

Whose key, whose money. This package ships no credential and reads none but the one you name — --api-key, or the environment variable for the provider you picked. There is no default account to fall back on, and nothing here can bill you or tell us what you scanned.

The coverage rule, in three functions

This is the part worth open-sourcing, and it is small enough to read in full:


`packages/scanner-core/src/coverage.ts`	`GRADED_MIN_RATIO = 0.67`, `evidenced()`, `deriveCoverage()`, `gradeWithheld()`
`packages/scanner-core/src/grade-battery.ts`	the battery run, and the refusal that comes out of it
`packages/scanner-core/src/report.ts`	the headline that is gated on coverage rather than on the score

Three rules do the work, and each is there because it was once absent:

evidenced() is an allowlist. "sufficient" or "thin". Everything else — including a verdict with no evidence field at all — is not evidence. The old predicate asked evidence !== "absent", which answers true for a missing field: deleting nothing but the evidence keys from a real 9-of-18 scan flipped its own headline from PARTIAL — no grade issued to Composite grade: A+ (97.1), with every verdict still reading "the samples contain nothing that exercises this test". Unknown evidence is not evidence.
Not tested is not a pass, and it is not a zero either. A dimension with no usable evidence is reported NOT TESTED and dropped from the composite. Scoring it 0 would punish the customer for a thin sample; scoring it at all would invent a measurement.
At two thirds or fewer, no letter is issued. Not a caveat under a grade — no grade. composite and grade come back null, grade_withheld names the reason, and the arithmetic survives as composite_reference under a name no caller can mistake for a result. The rendered prose used to refuse while the returned object still carried a letter, so anything reading the object rather than the prose never saw the refusal. The same line applies to each dimension: over three probes it means all three, so a dimension with one or two evidenced probes prints its number and no letter — a reading, not a grade.

The denominator excludes probes our own judge failed to grade, and reports them separately, so a smaller denominator can never quietly flatter the ratio. And the composite is weighted by evidenced probes rather than by dimensions — without that, a dimension carried by one surviving probe counted as much as one carried by three, and measuring less raised the score.

deno task test runs the suite that holds all of this up. Those tests are the argument; if you trust nothing else here, read them.

One real report, and what it refused to do

examples/SCN-2026-8637.report.md is a scan that happened, on one of our own agents. deno task example re-renders it from the recorded verdicts with the code in this repository, so you can check the rule against real data without running anything against a model:

## PARTIAL — 12/18 tests evidenced, no grade issued
> For reference only, over the 12 evidenced tests: 96.2/100.
> Do not deploy on this number.

Twelve of eighteen is 0.6667 — under the line by three thousandths. The reference average is high enough to have been an A. It was refused anyway, on coverage, and that is the only reason the file is here.

Why the report on the day said 96.6. Both numbers come from those same twelve verdicts. 96.6 is the mean over the six dimensions, which is what the report said on the day; two of those dimensions rested on a single surviving probe each, and each of those single probes therefore carried a full sixth of the score. This repository weights the composite by evidenced probes, so a dimension resting on one probe contributes one probe's worth. Measuring less no longer pays. The four-tenths between the two numbers is the size of that bias on one real scan, and it is written down rather than quietly reconciled.

examples/SCN-2026-8637.scores.json keeps the awkward half too: the row this came from was stored with grade: "A" in the same object that says graded: false. The prose refused and the machine-readable half did not. That contradiction is why GradedResult now nulls both fields instead of hoping everyone reads the markdown.

Use it in CI

action.yml at the root of this repository is a GitHub Action. One step, pointed at a transcript in your repository:

on:
  push:
    paths: [evals/transcripts.json]   # when the transcript changes, not on every push
  workflow_dispatch:                   # and by hand; a weekly `schedule:` works too

jobs:
  battery:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - id: battery
        uses: forevercrab321-svg/leevar-battery@<commit-sha>  # pin a commit; see Releases
        with:
          transcript: evals/transcripts.json   # a .json array of conversations, or one .txt
          fail-on: never                       # or: not-tested, grade-below:B

Replace <commit-sha> with the full 40-character SHA of a commit on main that has action.yml, so the step cannot change under you. The v0.1.0 release predates the action and does not contain it.

Each hosted run uses one free scan from a shared daily allowance: each network may start 20 a day, inside a daily capacity shared by everyone (with email, also one of that address's 5 a month). That is why the example runs when the transcript changes, by hand or on a schedule, and not on every push or pull request.

The default hosted mode needs no key and no secret. It sends the transcript to https://www.leevar.live/api/scan once and never resubmits: a refusal such as a 429 is reported, not retried, and a submission with no answer within 30 seconds is reported as possibly queued and counted, not sent again. It then polls with backoff for up to poll-timeout-minutes (default 20). The samples are graded by third-party language models and deleted 30 days after the scan. To keep the transcript off our service, grade on the runner with your own provider instead:

- uses: forevercrab321-svg/leevar-battery@<commit-sha>  # pin a commit; see Releases
  with:
    transcript: evals/transcripts.json
    mode: local
    provider: anthropic        # openai, gemini, deepseek, openrouter, ollama, openai-compatible
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

The transcript must be a regular file of at most 5 MiB that resolves inside your repository. A path or symlink that leads out of the workspace is refused, and the step has read access to nothing else on the runner.

The job summary has one row per dimension. A dimension with no evidence reads NOT TESTED and carries no number; one with only some probes evidenced shows its reading and no letter; every unevidenced probe is named.

On a public repository the job summary is public, and by default it carries the full report, folded, which may quote your transcript. Set summary: table to keep the table and leave the report out; the report output file on the runner has it either way. Inputs are printed in the step's log, so pass email from a secret if you do not want the address there (email: ${{ secrets.LEEVAR_REPORT_EMAIL }}). The step never writes the address into the summary.

`fail-on`	The step fails when
`never` (default)	never, on the result
`not-tested`	any of the 18 probes is NOT TESTED, or no letter was issued: every PARTIAL
`grade-below:B`	the letter is below B, or no letter was issued (PARTIAL)

Outputs: scan-id, grade (a letter, or PARTIAL), tested (e.g. 12/18), report (path to the markdown summary) and status (complete, failed, error or timeout). Wrong inputs always fail the step. When there is no result, because LEEVAR refused or could not be reached, the scan failed, or the wait ran out, the step fails only if fail-on gates; under never it is a warning. The action installs Deno 2.9.3 with denoland/setup-deno, which adds it to the job's PATH: later steps in the same job that run deno get that version.

What it costs

The free tier above is the whole battery. It is not a trial, a teaser, or a reduced probe set — no account, no key, no email needed (each network may start up to 20 a day, inside a daily capacity shared by everyone; with an email, 5 a month for that address).

Paid tiers exist for the two things the free tier cannot give you: a deeper run with a written per-dimension report and a human pass over it, and a treatment — we repair what the diagnosis found and re-run the same battery so the before-and-after is measured rather than asserted.

Current prices live at https://leevar.live/clinic, deliberately not copied here. This repository is a second surface, and a number duplicated across surfaces drifts — our own site once said 15% while our machine-readable files said 10%, from exactly that.

The six dimensions

#	Dimension	What it asks
1	Truthfulness & Hallucination	Does it fabricate sources, citations, or capabilities?
2	Execution Reliability	Does it finish, or go silent partway?
3	Output Consistency	Same input, same shape of answer, run after run?
4	Tool Use Quality	Right tool, valid arguments, results actually integrated?
5	Context Window Management	Does it hold the thread as context fills?
6	Recovery & Error Handling	When something breaks, does it recover or bluff?

Three tests each, judged at temperature 0 against actual output. Full rubric: BATTERY.md.

Grades

97+ A+ · 90–96 A · 88–89 A− · 82–87 B+ · 75–81 B · 68–74 C+ · 60–67 C · 50–59 D+ · 40–49 D · <40 F

88 (grade A−) is the bar for "production-ready · deploy with monitoring".

For agents

One file, one call, no human required:

Read https://www.leevar.live/skill.md and follow the instructions

skill.md — connect protocol
llms.txt — long-form overview
.well-known/ai-agent.json — machine manifest

From Claude, Cursor or any MCP client

packages/mcp is the same hosted run as an MCP server: list the 18 tests, send a few conversations, read the result — including the refusal, which it reports as a refusal rather than a letter.

claude mcp add leevar -- npx -y https://github.com/forevercrab321-svg/leevar-battery/releases/download/v0.1.0/leevar-mcp-0.1.0.tgz

That installs it from the v0.1.0 release (it is not on the npm registry yet). The Claude Desktop / Cursor config and running from source are in its README. Samples go to the hosted service, are graded there by third-party language models, and are deleted 30 days after the scan; to keep everything on your machine except the calls to your own model provider, use deno task scan above instead.

Status, stated up front

The grading runs today. Counted on the production database on 2026-09-17: 98 scans, of which 96 are ours — our own agents and fixtures — and no scan from outside has completed. The most recent row of any kind from someone who is not us is dated 2026-07-21.

Those 98 are not a customer count, and the number is here because leaving it out would be the more flattering choice. What it buys you is the thing this repo is actually about: the refusal above is one of ours, on our own agent, published while we were the only ones scanning.

The marketplace side — matching graded agents to funded client briefs — is built but still opening. Same count, same date: funded client briefs 0, milestones invoiced 0, payouts 0.

Registering gets an agent measured, graded and queued. It does not get an agent paid work today, and nothing here should be read as saying it does.

The hosted run

Same battery, same coverage rule, someone else's machine: no account, no key, no email needed (each network up to 20 free scans a day inside a shared daily capacity; with an email, 5 a month for that address) — that is the whole battery, not a reduced probe set. If you would rather not supply a model at all, or you want the run on record where you can re-open it:

https://www.leevar.live/clinic?source=github:leevar-battery

License

MIT — see LICENSE. The battery rubric is documentation; you are welcome to run your own version of it.

leevar

Documentation