Repository avatar
Other Tools
v0.1.5
active

mizan-ledger

io.github.qants-labs/mizan-ledger

Local stdio MCP server for a calibration ledger: declare confidence, commit a hash, score honesty.

Documentation

mizan-ledger

Declare how confident your agent is before a task, commit a salted hash of that declaration, then score honesty against the real outcome (your own tests, evals or checks). Local-first, hash-only, verifiable offline.

Status: v0.1.5, early alpha by QANTS. On PyPI (not on npm). MIT licensed (see LICENSE). No network access, no telemetry, no dependencies in the core library.

Why

Agents often report success with the same unearned confidence whether they are right or wrong. A calibration ledger asks one question: when the agent says "80% sure", is it right about 80% of the time? Brier score and ECE answer that. The commitment makes sure the declaration was fixed before the outcome was known.

Install

pip install mizan-ledger

Or from a local checkout:

pip install -e .            # Python >= 3.9, pytest only for development

30-second example

from mizan_ledger import Ledger

led = Ledger("ledger/", private_path="private/")          # public dir is safe to share; private dir is not
led.declare("fix-bug-421", 0.80, agent_id="my-agent", version="1.2.0")   # hash-only receipt, timestamped
# ... run the task, run the tests ...
led.resolve("fix-bug-421", outcome=True)                  # reveals the opening, scores it

print(led.summary("my-agent"))     # n, Brier + 95% interval, ECE, log loss, accuracy (Wilson), coverage
mizan-ledger verify ledger/        # offline check of hash chains, openings and scores

What gets recorded

FileVisibilityContents
receipts.jsonlpublicagent_id, agent_version, commitment (SHA-256), declared_at, hash-chain link
resolutions.jsonlpublic after resolutionoutcome, opening (task_id, p, salt), squared error, log loss, hash-chain link
private/openings.jsonlprivate (mode 600)the same openings, kept until the task is resolved

Commitment = SHA-256(canonical_json({agent_id, agent_version, p, salt, schema, task_id})) with p as a fixed 6-decimal string, a fresh 128-bit random salt per declaration, sorted keys and no whitespace. Shared test vectors (vectors/commitments.json) are checked by both the Python and the JavaScript implementation.

Guarantees and honest limits

  • A declaration cannot be replaced or re-resolved. Deleting or editing a record breaks the hash chain and verify fails.
  • Coverage is reported. Declared, resolved and pending are all counted, so you cannot quietly drop the bad ones. Anything unresolved shows as pending.
  • Timestamps are your local clock. They prove order only if the chain head is anchored somewhere independent (publish chain_head(), timestamp it, or write it to a transparency log). Without that, a ledger owner could rebuild the whole chain after the fact.
  • The library cannot tell whether the outcome you pass to resolve is honest. Use an outcome source you do not control through the agent (test results, an external check).
  • Small samples give wide intervals; ECE is biased upward for small n. The summary prints the interval and n; read them.
  • Scores are about calibration, not about whether the agent is good, safe or secure.

Scores (mizan_ledger.metrics)

Brier (mean squared error), log loss (p clipped to [0.01, 0.99], same as the Mizan API), ECE with 10 equal-width bins, reliability table, accuracy with a Wilson 95% interval, and a deterministic percentile-bootstrap 95% interval for Brier. The always-0.5 baseline Brier is 0.25.

Adapters

AdapterStatus
pytest plugin (mizan_ledger.pytest_plugin): @pytest.mark.confidence(0.9) declares before the test body, pass/fail resolves it. Run with pytest --mizan-ledger-dir .mizan --mizan-agent-id my-agent.Tested here (pytester).
Generic trace evaluator (mizan_ledger.adapters.trace_eval): plain-dict functions to declare from a trace's confidence field and resolve from its score.Unit-tested with synthetic dicts only. Not tested against Arize Phoenix, Braintrust or LangSmith; their evaluator signatures were not verified. Treat as a stub.
JavaScript shim (js/mizan-ledger.mjs): commitment, declare, verify opening. Node >= 18, node:crypto only.Tested against the shared vectors (node --test js/test.mjs). It does not implement the file ledger.

No CI workflow file is included in this repository.

Local MCP server (optional)

An optional stdio MCP server lets an agent host (Cursor, Claude Desktop, VS Code or any MCP client) use the ledger without leaving the machine. It needs Python >= 3.10; the core install is unchanged.

pip install "mizan-ledger[mcp]"
mizan-ledger mcp              # same server as the mizan-ledger-mcp script
{"mcpServers": {"mizan-ledger": {"command": "mizan-ledger-mcp",
  "env": {"MIZAN_LEDGER_DIR": ".mizan", "MIZAN_AGENT_ID": "my-agent"}}}}

Tools: commit_confidence (declare before the task), resolve_outcome (record the real outcome), get_calibration (read-only scores), verify_ledger (read-only offline check). Local files only: no network, no keys, no shell commands. The outcome you supply is not verified; prefer a source the agent does not control. MIZAN_PRIVATE_DIR (default <MIZAN_LEDGER_DIR>/private) holds the openings; keep it out of version control.

This is not the remote qants-diwan MCP server on the QANTS site (Mizan benchmark and directory); that is a different product.

Tests

python3 -m pytest -q          # Python suite
node --test js/test.mjs       # JS shim vs shared vectors

Seed agents

seeds/ holds three demonstration agents operated by QANTS (ids start with qants-). They use only free public data and local ledgers, never the production Mizan API, and they are always shown separately from external agents. Their raw ledgers are in seed_runs/ and can be checked with mizan-ledger verify seed_runs/<agent>.

Security notes

Keep private/ out of version control and backups you share. The salt is what stops guessing of low-entropy declarations.

License

MIT, see LICENSE. Copyright (c) 2026 QANTS.

PYPI
mizan-ledger
Install Command
pip install mizan-ledger
Runtime: uvx