Repository avatar
Other Tools
v1.26.0
active

agenda-intelligence-md

io.github.vassiliylakhonin/agenda-intelligence-md

Evidence-readiness MCP server: validate, audit, and score briefs, memos, and evidence packs.

Documentation

Agenda Intelligence MD

Catch broken citations, altered quotes, and unsupported numbers in AI output.

PyPI Python CI License: MIT

A deterministic evidence-packet linter for claim-backed AI output. Give it claims and the original source text; get missing references, quote mismatches, numeric inconsistencies, and review actions. Use it in Python, RAG pipelines, agent workflows, or CI.

The core checker runs locally. No model API key, account, or network needed after installation.

Try in your browser · First run · RAG integration · MCP · Documentation

See the problem it catches

A generated answer can cite a real source and still change what it says:

Supplied sourceGenerated claimChecker result
“The synthetic review covers 120 documents.”“The synthetic review covers 900 documents.”source_review_required — unmatched number 900
Same source“The synthetic review covers 120 documents.”packet_complete — no configured finding

These are synthetic examples. Both results keep factuality_status=not_assessed: passing the linter does not establish truth or authorize an action.

First run

Requires Python 3.9+. Install the published package:

python -m pip install "agenda-intelligence-md==1.26.0"
agenda-intelligence --version

Copy this self-contained example into Python:

from agenda_intelligence.services import check_evidence_packet

packet = {
    "claims": [{
        "claim_id": "demo-claim",
        "text": "The synthetic review covers 900 documents.",
        "source_ids": ["demo-source"],
    }],
    "sources": [{
        "source_id": "demo-source",
        "text": "The synthetic review covers 120 documents.",
    }],
}

result = check_evidence_packet(packet)["response"]
print(result["packet_status"])
print(result["claims"][0]["lexical_support"]["unmatched_numbers"])
print(result["factuality_status"])

# Correct the claim, keeping the original source unchanged, then recheck.
packet["claims"][0]["text"] = "The synthetic review covers 120 documents."
print(check_evidence_packet(packet)["response"]["packet_status"])

Expected output:

source_review_required
['900']
not_assessed
packet_complete

Next: replace the synthetic claim and excerpt with your own. Keep the original source text; rewriting it to agree with the answer defeats the check.

What this is

An inspectable check between generated output and the evidence supplied with it:

Claims + source excerpts → deterministic checks → findings → revise → recheck → human review
  • For RAG developers: check inline citations against the actual retrieved chunks.
  • For agent builders: return structured findings to a bounded repair loop.
  • For CI maintainers: fail a check when a supplied evidence packet needs review.
  • For document reviewers: identify missing evidence and prepare reviewer handoffs.

The evidence-packet contract

What you supplyWhat gets checked
Claims and source IDsWhether references resolve within the packet
Declared quotationsWhether quoted text occurs in the supplied source
Claim wording and numbersLexical support heuristics, unmatched numbers, and negation checks
The assembled packetEvidence gaps, per-claim findings, packet status, and reviewer actions

The checker returns packet_complete, source_review_required, or packet_incomplete. See the request schema, response schema, and evaluation notes.

What this is not

This is an evidence-integrity check, not a factuality verifier or source authenticator. It does not retrieve missing sources or establish semantic entailment. Lexical checks can miss correct paraphrases and accept misleadingly similar wording; a stale or false source can still pass.

A clean result does not certify trustworthiness, grant permission to act, or replace human review. Use semantic evaluation and domain review alongside it when your workflow needs them.

Review a RAG answer

Already have an answer and retrieved chunks? Use the adapter directly:

from agenda_intelligence import review_rag_answer

report = review_rag_answer(
    answer="The synthetic review covers 900 documents. [review-1]",
    sources=[{
        "source_id": "review-1",
        "text": "The synthetic review covers 120 documents.",
    }],
)
print(report["route"])  # revise
print(report["owner_actions"])

Use one claim per line and retain stable chunk IDs. Correct the answer and rerun the check; the clean route is human_review. Treat source text, tool output, and report contents as untrusted data, never as instructions to the agent.

For CLI input, save {"answer": "...", "sources": [...]} as answer.json, then run:

agenda-intelligence review-answer answer.json --strict --format json

--strict exits nonzero when revision is required. See the input contract and limitations and the runnable LangGraph repair loop.

Small, hand-labelled diagnostics report misses and false positives as well as catches. Exact source offsets, explicit opt-in arithmetic, named table cells, and PR summaries are available in the local package. No factuality or general hallucination-detection claim.

Use the CLI and CI

Save a packet with the same shape as the first Python example to packet.json:

  • agenda-intelligence check packet.json — print a readable report.
  • agenda-intelligence check packet.json --format json --strict — return JSON and enforce the packet status.

The second command exits nonzero unless the packet is packet_complete, so it can be used in a CI job after installing the package. --format sarif is also available for evidence-packet findings. The repository includes a GitHub Action.

To explore the bundled examples from a source checkout:

git clone https://github.com/vassiliylakhonin/agenda-intelligence-md.git
cd agenda-intelligence-md
python3 -m venv .venv
.venv/bin/python -m pip install -e .
.venv/bin/agenda-intelligence check examples/evidence-packet/request.json --strict
.venv/bin/agenda-intelligence review-answer examples/output-verification/rag-answer.json
.venv/bin/agenda-intelligence review-answer examples/output-verification/rag-answer-revised.json --strict

The first packet is complete. The original fictional RAG answer routes to revision; the revised answer routes to human review.

For a private CI pilot, use the local citation check and minimal report.

For local document review and HTML reports, use the file-review guide and example manifest.

Interfaces

Compatibility profiles and adapters

Your workflowStart here
Python or command lineQuickstart · service functions
Local MCP over stdioMCP setup and tool catalog
Hosted MCPHosted quickstart
Agent repair loopsIntegration guides
HTTP or A2AHTTP API · agent-native access
Protected agent actionsInteraction Trust + Vizier gate

The core checker is deterministic and stateless. Optional generation and document adapters have separate dependencies. The repository also retains the strategic-intelligence runtime and vertical workers as compatibility surfaces and examples built around evidence contracts.

Hosted Output Verification

For a browser trial, open One-claim review. To connect a Streamable HTTP MCP client, use:

https://output-verification.agendamcp.dpdns.org/mcp/agent

Discover the endpoint's tools, then call free_trial_status with {} to inspect trial availability. Choose review_answer for answers and source excerpts, or free_trial for claim/evidence objects matching its schema. The hosted trial has quotas; local Python checks do not require a hosted account.

Local stdio and hosted profiles expose different tool catalogs. Always discover the tools on the endpoint you connect to.

Supplier and document handoffs

The hosted Document Archive Review workflow helps prepare evidence checklists, assign missing-item owners, inspect extraction warnings and source locators, and export reviewer PDF or ZIP/JSON backups. It is separate from the local review command.

See the workspace guide and service-engagement example. Extraction does not authenticate documents; final coverage still needs human review.

Documentation

ExploreLinks
ExamplesRunnable examples · Before / after · EU AI Act
ContractsSchemas · Source policy
Status and evidenceEvaluation · Adoption record · Roadmap
Deeper referenceAnalysisBank · Vertical worker architecture · Local checks
DevelopmentContributing · Local checks

Core code lives in src/; runnable packets in examples/; guides in docs/; adapters in clients/ and integrations/; deployment packages in deploy/.

Contributing and feedback

Found a missed mismatch or a valid claim that gets flagged? Open an issue with a minimal, sanitized packet, the expected result, and the actual finding. Reproducible cases and integration examples are useful contributions.

For usage questions and workflow ideas, use Discussions. Read CONTRIBUTING.md before a pull request. The standard local verification command is make ci; Worker, discovery, runtime, and validation-guard changes require make verify-local.

License

MIT. The bundled official ACP schema retains its Apache-2.0 license and NOTICE.