# Token-efficiency benchmark: protocol

Status: fixed before any model run. Committed to git ahead of the results; the commit time is the record.
This protocol is not lodged with an external registry.

## Question

When a model answers questions about a document, how many input tokens does each way of giving it the
document cost, and how much accuracy does each give up compared with the full text?

## Corpus

10 English Wikipedia articles, pinned to exact revisions (`corpus/manifest.json`), plain text, references
removed, cut at the first paragraph boundary after 6,000 words. 60,043 words in total.
Wikipedia text is CC BY-SA 4.0.

## Questions

100 questions (`questions.json`, SHA-256 in `corpus/SHA256SUMS`):

- 80 answerable: 8 per article, one from each of 8 paragraphs drawn at random (seed 20261108) from
  paragraphs of 45 words or more. Gold answers are short strings copied from the text; aliases allowed.
  `A&B` means both parts must appear.
- 20 unanswerable: 2 per article, plausible questions whose answer is not in the article (checked by
  searching the text for the key terms).

The questions and gold answers were written before any condition below was built or run.

## Model and settings

`qwen2.5:7b-instruct` via Ollama on one Apple M5 machine (17 GB). Temperature 0, seed 0, context window
16,384 tokens, at most 64 output tokens per answer. The same answer prompt for every condition:

> Answer the question using only the context. Reply with the answer in a few words. If the context
> does not contain the answer, reply exactly: not stated.

## Conditions

| ID | Name | Context given to the model per question |
| --- | --- | --- |
| A | Full text | The whole article |
| B2, B4 | Chunk retrieval (RAG) | The top 2 or top 4 chunks of about 200 words, ranked by BM25 against the question |
| C | Summary | A summary of at most 600 words, written once per article by the same model |
| D10, D25 | Croutons | The top 10 or top 25 croutons, ranked by BM25 against the question |
| Dall | All croutons | Every crouton for the article, no retrieval |

**Croutonization** here means rewriting the article once, by the same model, into atomic facts: one
sentence per fact, the subject named explicitly, numbers, names and dates kept exactly as written,
nothing added. The article is processed in sections of about 700 words.

**Prompt caching** is not a separate run. It changes the price of repeated input, not its size, so it
is reported as a cost calculation on condition A using providers' published cached-input rates.

**Not included:** learned prompt compressors such as LLMLingua-2 (they need a separate model download);
listed as a limitation and a next step.

## Measures

1. **Input tokens per question**: Ollama's `prompt_eval_count` (the model's own tokenizer).
2. **Accuracy on answerable questions**: the answer contains a gold string (case-insensitive, dashes and
   quotes normalized). "not stated" counts as wrong.
3. **Correct abstention on unanswerable questions**: the answer contains "not stated". Anything else
   counts as a made-up answer.
4. **Prefill time per question**: Ollama's `prompt_eval_duration`, as a proxy for compute.
5. **Preparation cost** for C and the D conditions: input plus output tokens spent building summaries
   and croutons, reported per article and amortized over its 10 questions, with the break-even number
   of questions against condition A.

## Analysis

- Point estimates with 95% intervals from 10,000 bootstrap resamples of questions.
- Paired bootstrap of the accuracy difference from condition A for every other condition.
- All results are published, including conditions that do worse than full text.

## Hypotheses (stated before running)

- H1: D25 uses at least 80% fewer input tokens per question than A.
- H2: D25 accuracy is within 5 points of A (the 95% interval of the difference stays above −5).
- H3: D25 makes up answers on unanswerable questions no more often than B4.

## Known limitations

- One model, 7B parameters, local; larger API models may behave differently.
- 100 questions, so intervals are wide.
- Questions were written by an AI model (Claude) reading the sampled paragraphs.
- Wikipedia prose is cleaner than most business documents.
- Single run per question at temperature 0.