Repository avatar
Other Tools
v0.1.7
active

chopper

io.github.wiiiktor/chopper

Cuts the exact quotes that answer a question out of long text and reranks passages, in one pass.

Documentation

LLMs need a Chopper: watch the video
Option 1, universal chopper: a small separate model (mmBERT-base 0.31B or mmBERT-small 0.14B), text in and text out, works with any answering model. Links to the quick start. Option 2, integrated chopper: a 20% slice of MiniCPM3-4B itself, no second model, reads the model's own cache of the whole session. Links to how it works.

70-second video: youtu.be/KMvmdXCEOJU

Try it in your browser, nothing to install: English demo ยท Polish demo
They run the recommended base chopper (0.31B) on your CPU, with small (0.14B) as a faster option.

See it on real examples: 20 English questions, Chopper vs Opus 5.5 vs gold (PDF)

Where this is going. The standalone chopper is a first, practical step. The main idea is an integrated chopper: chopping built into the answering LLM itself, reading the model's own cache of the session instead of re-encoding the text in a second model, which should give the highest possible quality and speed. A proof of concept with MiniCPM3-4B is described here.

Chopper 0.3B close to Opus 5.5 in cutting out the best pieces

Chops are pieces cut out of the original text snippets retrieved from a database or a long conversation: exact quotes, never rewritten. The chopper (mmBERT, 0.31B) makes them; for comparison, Claude Opus 5.5 makes them too, also forced to quote directly.

What is new: a new method, invented in November 2026 in Poland (EU). The chopper is not an LLM summarising the snippets: a small encoder reads the question and all 20 snippets together in one pass and marks every word keep or drop. There is no text generation, so it is fast: 50 ms per question (20 snippets, ~2,400 tokens) on a consumer-grade GPU (RTX 3090), 36 ms with mmBERT-small.

One database retrieval per question, even at 4 hops. Every result below uses a single retrieval: each question gets its 20 snippets once, the chopper cuts them once, and Qwen3-14B answers once. There are no follow-up searches and no second rounds, so a 4-hop question has to be chained from one cut.

A reranker that also gives you the chops

Two pipelines. Top: 16 retrieved passages go into a reranker, which scores each passage and outputs them ranked; the LLM then reads whole passages, mostly text it does not need. Bottom: the same 16 passages go into the Chopper in one pass, which outputs the same ranking plus the chops highlighted inside the top passages; the LLM reads only the chops, exact quotes, about 2 to 5 percent of the text. Ranking quality nDCG@10 on 8 languages with 16 passages per query: Chopper 0.973, bge-reranker-v2-m3 0.984, Qwen3-Reranker-4B 0.986, at the same speed as bge (0.20 s), chops included.

One model replaces both the reranker and the compressor. Full comparison: below.

Qwen3-14B answering from full snippets or just short chops

400 MuSiQue questions, 100 each at 1, 2, 3 and 4 hops, each with its own 20 retrieved snippets. Qwen3-14B (4-bit, no fine-tuning, no thinking mode) answers from the given text only; gpt-5.6-luna judges. Length = prompt tokens. Each score counts every correct answer, including those from incomplete chops; the number in brackets counts only questions whose chops kept every hop and the answer.

Qwen3-14B correct answers at 1/2/3/4 hops (complete chops only in brackets) and length. All snippets retrieved from the database: 2,620 tokens, 84/28/22/12. Chops from the chopper: 228 tokens, 83 (82) / 48 (44) / 45 (35) / 29 (21). Chops from Opus 5.5: 144 tokens, 90 (90) / 64 (60) / 55 (40) / 43 (30). Gold chops: 217 tokens, 86/63/52/37.

Qwen3-14B: all snippets vs chops from the chopper vs chops from Opus 5.5

Read each diagram like website traffic: questions flow left to right, and at every hop the ones whose fact the chops missed drop out; the last stage is Qwen3-14B answering correctly. Top: all 20 retrieved snippets. Middle: the chopper's chops. Bottom: Opus 5.5's chops.

1-hop questions: Qwen3-14B correct 84% with all snippets, 83% with the chopper's chops, 90% with Opus's chops

2-hop questions: Qwen3-14B correct 28% with all snippets, 48% with the chopper's chops, 64% with Opus's chops

3-hop questions: Qwen3-14B correct 22% with all snippets, 45% with the chopper's chops, 55% with Opus's chops

4-hop questions: Qwen3-14B correct 12% with all snippets, 29% with the chopper's chops, 43% with Opus's chops

The last stage is the table's score; the lighter block under it counts answers Qwen got right from incomplete chops, and the number in brackets leaves them out. More: docs/DETAILS.md ยง7.

Also a reranker: ranking and chopping in one pass

The chopper scores every passage while it chops (mean p(KEEP) of the passage's 16 most likely tokens). On the 8-language rerank holdout (400 queries, 16 passages each, never trained on), ranked against dedicated rerankers:

ranking 16 passagesnDCG@10Hit@1GPU time per query (RTX 3090)
chopper, mmBERT-base (0.31B), one pass0.9730.9720.20 s, chops + ANSWER/SEARCH included
bge-reranker-v2-m3 (0.57B)0.9840.9850.20 s
Qwen3-Reranker-0.6B0.9770.9720.77 s
Qwen3-Reranker-4B0.9860.9903.3 s

The chopper was never trained to rank, yet it is within about one point of the best rerankers at the speed of the fastest, so a pipeline can drop its reranker: retrieve โ†’ chopper gives the ranking and the quotes. Caveats (this test is easy for every model; the labels favour Qwen3-Reranker-0.6B) in the details.

Against other context compressors

Same 400 questions, same 20 snippets, same Qwen3-14B prompt; each compressor cuts the snippets once:

Qwen3-14B answering fromtokens1-hop2-hop3-hop4-hop
all 20 snippets (no compression)2,62084282212
LLMLingua-2 (Microsoft), rate 0.137428624
Provence (Naver), its default threshold 0.126377272616
Provence, threshold 0.515271242111
chops from the chopper, mmBERT-small (0.14B)21772493223
chops from the chopper, mmBERT-base (0.31B)22883484529

LLMLingua-2 drops tokens without looking at the question, so the facts a chain needs rarely survive. Provence reads the question but prunes each snippet on its own. To be fair to them: the chopper was trained on MuSiQue's train split (this test uses its dev split) and on other multi-hop data, while both were trained for general single-passage use.


Chopper ๐Ÿ”ช keep only the part of RAG that answers the question

Standard RAG (retrieval-augmented generation) pastes whole passages into your model's context, and they stay there. About 95% of those tokens don't help. After a few lookups, a long session is mostly retrieval spam.

Chopper is a small model that reads the retrieved passages first and keeps only the fragments that matter. Your model then sees ~100 tokens instead of ~3,000 per lookup, and a session can carry dozens of lookups.

question โ”€โ–บ retrieve 16 passages โ”€โ–บ ๐Ÿ”ช rank + chop in one pass โ”€โ–บ model answers, or asks one follow-up search
  • ๐ŸŽฏ Clean context: only useful fragments enter the session; the raw passages never stay.
  • โšก Less compute: the main model reads 2โ€“5% of the retrieved text.
  • ๐Ÿ… Also a reranker: the same pass ranks the passages almost as well as bge-reranker-v2-m3 (nDCG@10 0.973 vs 0.984, 8 languages), at the same speed. No separate reranker needed.
  • ๐Ÿ”— Multi-hop works: the model writes a follow-up question when a fact is missing, and searches again.
  • ๐Ÿ  Built on one home GPU: every dataset and model here was made on a single RTX 3090 (some labels came from APIs).

Results at a glance

retrieved text the main model reads2โ€“5% (about 20โ€“40ร— less); the answer is still in the kept text for 85% of HotpotQA and 69% of TriviaQA questions
"nothing here": pools whose answer was deleted (8 languages, business documents)asks to search again 67% of the time, keeps 1.4% of the text
ranking 16 passages (8-language holdout, 400 queries)nDCG@10 0.973, vs bge-reranker-v2-m3 0.984 and Qwen3-Reranker-4B 0.986; 0.20 s like bge, 16ร— faster than Qwen-4B, chops included (details)
chopping time per lookup (RTX 3090)155 ms (separate mmBERT) ยท 277 ms (built into MiniCPM3-4B)
4-hop MuSiQue questions answered correctly by Qwen3-14B, one retrieval12% โ†’ 29% with the chopper's chops instead of all 20 snippets (11ร— fewer tokens); with Opus 5.5's chops: 43%
Polish questions (PoQuAD, PolQA) answered correctly by Qwen3-14B85.6% from the Polish chopper's chops (255 tokens) vs 85.2% from all 16 snippets (3,856 tokens)
licenseMIT: free for commercial use, weights and code. The closest alternatives, Provence and XProvence (Naver, sentence pruning + reranking), are CC BY-NC-ND 4.0: non-commercial only

All quality numbers in this repository (F1, recall, answer accuracy, nDCG) were measured on raw chops, without word or sentence snapping. Since chopper-rag 0.1.7 the package extends chops to whole sentences by default (97% of chops then end at a sentence end, vs 36% raw; about 4.1% of the text kept instead of 3.4%; details).

In training (Nov 2026), suggested by the OPI PIB AI Lab: the same Polish chopper recipe on two Polish encoders, pl-ModernBERT-base from OPI PIB (results expected about 16 Nov) and polish-roberta-base-8k (8k context, about 18 Nov), compared with mmBERT-base-pl on the same 856 Polish questions. Restarted 15 Nov with the training data fitted to the Polish tokenizers (why).

Long-conversation memory results (LoCoMo, LongMemEval) are in the details: there the 4B answering model is the limit.

Pick your chopper

๐ŸŸง universal๐ŸŸช integrated
what it isa separate small encoder (mmBERT-base, 0.31B, recommended; or mmBERT-small, 0.14B; both also in Polish versions)a 20% slice of MiniCPM3-4B's own weights + a small LoRA
works withany answering model: text in, text outMiniCPM3-4B
extra memory0.31B / 0.14B weights45M parameters; reads the model's own cache
weights๐Ÿค— wiiiktor/chopper_mmbert๐Ÿค— wiiiktor/chopper_minicpm3

Start with the universal one. The integrated one is the long-term bet: trained together with a model from the start, it should become almost free. If you pretrain models and want to build that, contact wiiiktor.

Quick start

No install: try the in-browser demo (po polsku). It runs base (0.31B) by default, like the code below, or small (0.14B).

In your own RAG pipeline (LangChain, LlamaIndex, Haystack or plain Python):

pip install chopper-rag                  # or "chopper-rag[langchain]", [llama-index], [haystack], [all]

As a tool for Claude (Claude Code, Claude Desktop or any MCP client): pip install "chopper-rag[mcp]", then claude mcp add chopper -- chopper-mcp. Claude can then call chop_passages / chop_files to read a few quotes instead of whole documents, or rank_passages to rerank; setup in integrations/chopper-rag.

from chopper_rag import Chopper
chopper = Chopper.from_pretrained(variant="base")          # or "small"; for Polish "base-pl", "small-pl"
result = chopper.run("When was the Eiffel Tower completed?", passages)   # passages: list of retrieved texts
print([c.text for c in result.chops], result.decision)    # exact quotes + ANSWER / SEARCH
print(result.ranking(top_k=5))                             # the same pass as a reranker: [(passage_index, relevance), ...]

Framework examples: integrations/chopper-rag. Listed in the LangChain integrations (ChopperCompressor). Or try the demo in this repo:

pip install -r requirements.txt

# your documents (a folder of .txt / .md, or a .jsonl with "title" and "text") + any Hugging Face causal LM
python operator-for-llm-with-mmbert.py --docs my_docs/ --model <hf model id> \
       --ask "Who founded the company that distributed the film UHF?"

# leave out --ask for an interactive session; it shows how many tokens plain RAG would have added

Weights download from Hugging Face on first use. Training data: ๐Ÿค— wiiiktor/chopper_demo.

Want the details?

Everything else is on one page, docs/DETAILS.md:

Credits

Built on mmBERT (JHU CLSP), MiniCPM3-4B (OpenBMB), Qwen3, Qwen3-Embedding and Qwen3-Reranker (Qwen team), bge-reranker (BAAI); labels and judgements also came from Claude Opus 5.5 (Anthropic) and gpt-5.6-luna / GPT-6 Luna (OpenAI). Compared with LLMLingua-2 (Microsoft) and Provence (Naver). Data: the MuSiQue, HotpotQA, 2WikiMultiHopQA, TriviaQA, QASPER, Multi-Session Chat, WildChat, FineWeb, Wikipedia, LoCoMo and LongMemEval datasets; for Polish, PoQuAD and PolQA (IPI PAN). Our memory evaluation follows Memory Tax (He et al., 2026) and compares with MemT-1.0 (MemTensor, 2026). Full table with licenses: docs/DETAILS.md.

Licenses

Commercial use is allowed. The mmBERT choppers and all code are MIT. For comparison, Naver's Provence and XProvence context pruners are licensed CC BY-NC-ND 4.0 (no commercial use, no derivatives).

whatlicense
mmBERT choppers, incl. Polish (mmbert-base, mmbert-small, mmbert-base-pl, mmbert-small-pl)MIT (as mmBERT)
code in this repo (LICENSE) and pip install chopper-ragMIT
MiniCPM3 chopperMiniCPM Model License (as MiniCPM3-4B)
training data wiiiktor/chopper_demoCC BY-SA 4.0 (share-alike from HotpotQA, Wikipedia and PolQA); Polish parts: PoQuAD CC BY 4.0, PolQA CC BY-SA 4.0

Some labels came from Claude Opus 5.5 and OpenAI Luna models, used to train a non-competing extraction tool; see docs/DETAILS.md.


Bored with reading LLMs' outputs? See also my other project, control_room: a VS Code panel for Claude Code that shows each answer as cards, charts and clickable options instead of a wall of text.

control_room: a Claude Code answer shown as stat cards, two charts and three clickable options

PYPI
chopper-rag
Install Command
pip install chopper-rag
Runtime: uvx