
ai-host-optimizer
io.github.williamdoring-prog/ai-host-optimizer
Profile a host and get an evidence-graded checklist of AI-model optimizations.
Documentation
ai-host-optimizer
Profile a host, plan AI-model optimizations, and apply them — reversibly, with consent.
Every "speed up your local LLM" guide lists the same tweaks (Flash Attention, KV-cache quantization, keep-alive, NUMA). The lists are mostly right — they just never tell you which of them helps on your machine, or by how much.
This tool profiles the host, presents the checklist with an honest confidence tier on every number, resolves a conflict-free plan, and — the part that matters — measures the options you apply, with a noise floor, so a null result is reported as a null result instead of a fake win.
It is an MCP server and a CLI. Applying is dry-run by default, writes only into a managed config directory, and is reversible.
Status: 1.2. Profiling and planning are read-only. The applier writes only inside
~/.config/ai-host-optimizer(platform-appropriate), never touches system files, never runs a shell, and never escalates privilege.
Quickstart
npx -y ai-host-optimizer --cli
That prints a read-only profile and the checklist for your machine (trimmed):
Host profile (read-only)
role : workstation
platform : windows — x64
cpu : Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz (16 cores)
memory : 16 GiB total, 8 GiB free
gpu : nvidia NVIDIA GeForce RTX 3080 (10240 MiB VRAM)
cluster : 1 GPU(s)
tooling : ollama, python3, docker
Optimization checklist (read-only)
[ ] Enable Flash Attention (flash-attention)
improves decode tokens/s: qualitative (tier C)
risk low; fully-reversible; privilege false
stackable: throughput-stack
synergy: kv-cache-quant-q8
[ ] Quantize KV cache to q8_0 (kv-cache-quant-q8)
improves KV cache memory: 50% (tier B)
risk low; fully-reversible; privilege false
...
Then measure whether a change actually helped:
ai-host-optimizer --benchmark --save-baseline --samples 5 # before
ai-host-optimizer --apply --select flash-attention --yes # then restart the runtime
ai-host-optimizer --calibrate --select flash-attention # after → tier A, or "no change"
Why
Running local models well depends on host configuration (KV-cache size, Flash Attention, prefix caching, memory/NUMA/swap behaviour, provider prompt caching). The knowledge is scattered across guides. This tool profiles your host and tells you what is worth doing — honestly, labelling each estimate by confidence.
Install
# zero-install (downloads and runs)
npx -y ai-host-optimizer --cli
# or globally
npm install -g ai-host-optimizer
ai-host-optimizer --cli # or the short alias: aiho --cli
Usage
# report (read-only)
ai-host-optimizer --cli
ai-host-optimizer --cli --role container
# plan and apply
ai-host-optimizer --plan --select kv-cache-quant-q8,keep-alive-preload
ai-host-optimizer --apply --select kv-cache-quant-q8,keep-alive-preload # dry-run
ai-host-optimizer --apply --select kv-cache-quant-q8,keep-alive-preload --yes # apply
ai-host-optimizer --revert
# measure (tier A)
ai-host-optimizer --benchmark --save-baseline # before
ai-host-optimizer --benchmark # after (compares to baseline)
# calibrate: measure an option's real effect on THIS host and promote it to tier A
ai-host-optimizer --benchmark --save-baseline --samples 5
ai-host-optimizer --apply --select flash-attention --yes # then restart the runtime
ai-host-optimizer --calibrate --select flash-attention --samples 5
ai-host-optimizer --plan --select flash-attention # now shows [measured]
# generate declarative accessories from the same registry
ai-host-optimizer --generate ansible
ai-host-optimizer --generate dockerfile
ai-host-optimizer --generate checklist
# interfaces
ai-host-optimizer --mcp
ai-host-optimizer --browser # browser UI (not yet)
ai-host-optimizer --desktop # desktop UI (not yet)
Applying writes only into the managed directory and prints the source line to
activate it. Undo any time with --revert.
What is automated: options with an applyOp. The applier writes one
<runtime>.env file per target — ollama.env, llama.cpp.env,
huggingface.env, vllm.env — plus a single activate.sh / activate.ps1. On
Linux, the ollama-systemd-dropin option additionally writes a systemd drop-in
(so the Ollama service actually reads that environment) and prints the root
install commands rather than running them. Everything else is advisory:
--apply says so plainly rather than doing nothing silently.
Use as an MCP server
Add to your MCP client (e.g. Claude Desktop):
{
"mcpServers": {
"ai-host-optimizer": {
"command": "npx",
"args": ["-y", "ai-host-optimizer", "--mcp"]
}
}
}
Tools exposed:
| Tool | Purpose |
|---|---|
profile | Read-only host profile (OS, CPU, memory, GPU, detected tooling) |
checklist | Optimization options with estimated improvement + resource footprint |
plan | Conflict-free plan for selected ids, with a combined estimate (uses tier-A calibrations when present) |
benchmark | Measure local inference rate (decode tokens/s) + the noise floor — tier-A evidence |
validate-registry | Registry consistency check (development aid) |
Published to the official MCP Registry as
io.github.williamdoring-prog/ai-host-optimizer (see server.json).
Honesty contract
Every estimate carries a confidence tier:
- A — measured on your host (via
--benchmark/--calibrate). - B — labelled estimate with a cited method.
- C — qualitative (direction only, no number).
Tier B/C estimates are never presented as measurements. A combined estimate is only shown once its inputs are trustworthy (see the project plan).
Calibration is honest about noise. --benchmark takes several samples and
reports their spread; --calibrate compares before/after and only promotes an
option to tier A when the change exceeds the host's own noise floor (the
combined run-to-run range). A null result is reported as a null result and records
nothing — a small difference inside the noise is not claimed as a win. Use
--samples <n> to lower the noise floor when you need to detect a small effect.
What the benchmark measures
--benchmark measures decode tokens/second (and the prefill rate) by calling
the local runtime's HTTP API (/api/generate with stream: false), which returns
exact token counts and durations rather than a wall-clock guess. It takes several
samples, discards a warm-up model load, and reports the median plus the
run-to-run spread.
That spread is the whole point. Inference timing on a desktop is noisy — on a test
host the spread was ±7% run-to-run, so a single before/after comparison can show a
"15% improvement" that is pure noise. So --calibrate refuses to claim a change
smaller than the combined spread of the two measurements, and prints a null
result as a null result:
baseline: qwen2.5-coder:14b, 10.29 tok/s, 2638.9 ms, 5549 MiB free
current : qwen2.5-coder:14b, 10.16 tok/s, 2860.5 ms, 5541 MiB free
noise floor: ±13.9%
No significant change: decode tokens/s -1.3% is within the ±13.9% noise floor
of this host.
Nothing recorded — a null result is not evidence of an improvement.
A tool that will tell you "this didn't help" is more useful than one that always claims a win.
Safety
- Read-only by default. Profiling and planning never change anything.
- The applier writes only into the managed config directory
(
~/.config/ai-host-optimizer, or the platform equivalent) — never system files — runs no shell, and escalates no privilege. - Applying is dry-run first:
--applyshows the change;--yesperforms it. - Everything applied is reversible with
--revert(and--revertis idempotent). - Options are tagged with risk, reversibility and privilege needs.
- Mutually exclusive options (e.g. KV-cache q8_0 vs q4_0; zram vs disk swap) are one-of groups; stackable options are marked stackable.
- Role-aware: a
containercannot tune the host kernel; acluster-nodeis advisory-only.
Roadmap
- M0 — read-only profiler, registry, checklist, CLI + MCP. ✅
- M1 — reversible applier (user-space), plan/apply/revert, on-host benchmark. ✅
- M1b — measured tier-A calibration (median-of-samples benchmark, noise floor, per-option promotion). ✅
- M1c — multi-runtime applier: one
<runtime>.envper target (Ollama, llama.cpp, Hugging Face), 8 applyable options, idempotent revert. ✅ - M1d — more apply ops: a
systemd-dropinop (Ollama service env on Linux, install commands printed not run) and thevllmruntime. 32 options, 11 applyable. ✅ - M2 — accessory generation (Ansible/Dockerfile/checklist). ✅ (generation)
- M2b — distribution: official MCP Registry entry live
(
io.github.williamdoring-prog/ai-host-optimizer@1.2.0) + CI publish workflow; directory listings next. 🚧 - M4 — browser UI and Tauri desktop UI. (planned)
Documentation
docs/architecture.md— one core, many front-ends; detection coverage; roadmap.docs/registry-schema.md— the option schema and the honesty contract.docs/publishing.md— releasing to npm, the MCP Registry, and the directories.docs/launch.md— launch copy (Show HN / r/LocalLLaMA).SECURITY.md— security policy and the design commitments.
Contributing
Contributions are welcome — especially new optimization options and
corrections to unsupported claims. See CONTRIBUTING.md
for the dev setup and the registry contribution rules. Participation is governed
by CODE_OF_CONDUCT.md.
Licence
MIT. See LICENSE.
ai-host-optimizernpm install ai-host-optimizer