How many tokens does an answer need?

← All research

Croutons is a product of ours. This result does not favor it. We publish it anyway, because a benchmark that only reports wins is marketing.

Setup

Everything below was written into the protocol before any condition was run.

• Corpus: 10 English Wikipedia articles pinned to exact revisions (James Webb Space Telescope, Lithium-ion battery, Panama Canal, Insulin, Great Barrier Reef, Rust, Tour de France, Marie Curie, Mount Everest, Penicillin), each cut at about 6,000 words. 60,043 words in total.
• Questions: 100. Of these, 80 are answerable: 8 per article, each from a paragraph drawn at random, with gold answers copied from the text. The other 20 are plausible questions whose answer is not in the article; they check whether the model makes things up.
• Model: qwen2.5:7b-instruct through Ollama on one laptop (Apple M5, 17 GB). Temperature 0, seed 0, 16,384-token context window, the same answer prompt in every condition.
• Scoring: an answer is right if it contains a gold string. On unanswerable questions, the model must reply "not stated"; anything else counts as made up.
• Statistics: 95% intervals from 10,000 bootstrap resamples of questions; accuracy differences from full text use a paired bootstrap.

The seven conditions

Croutons are the article rewritten once by the same model into atomic facts: one sentence per fact, the subject named every time, numbers, names and dates kept exactly. For example: "The Great Barrier Reef covers an area of approximately 344,400 square kilometres."

Results

700 answers, 1,393 seconds of model time. Tokens are counted by the model's own tokenizer.

With 80 answerable questions, one question is worth 1.25 points. Only the summary is clearly worse than full text. The differences among the other conditions are within a few questions of each other, and their intervals overlap.

Why the summaries failed

The model was allowed 600 words and wrote 164 to 331. The summaries kept the gold answer for only 17 of the 80 answerable questions. Of the summary condition's 80 answers, 11 were right, 64 were "not stated" and 5 were wrong. In most cases the model was not confused; the fact was simply no longer there.

Where the croutons lost points

The rewrite kept the gold answer for 78 of 80 questions. Two answers were lost in the rewrite itself (one insulin question and one Great Barrier Reef question), so every crouton condition missed them. The rest of the misses came from retrieval ranking, from the model, and from two borderline answers that strict scoring marks wrong. One D10 answer to an unanswerable question was made up ("Acropora coral").

Hypotheses

Stated in the protocol before any run, about the crouton condition we expected to do best.

H2 failing does not show that croutons are worse. It shows that 100 questions cannot rule out a loss of up to 10 points. Plain passage retrieval cleared the same bar without any rewrite.

Preparation cost

Summaries and croutons cost tokens to build before the first question is answered.

Compared with passage retrieval rather than full text, D25 saves about 370 tokens per question, so its 17,655-token build pays for itself only after about 48 questions on the same article. Build cost is counted in tokens of the same 7B model; with a larger model doing the rewrite, it would cost more.

Caching changes the arithmetic

Providers bill repeated input at a discount when a prompt starts with the same text as a recent one. The protocol treats this as a price calculation, not a separate run. At a 90% discount on cached input, asking a second question about the same full article costs the equivalent of about 860 input tokens, close to passage retrieval (963). At a 50% discount, it costs about 4,290. The discount applies only while the cache is alive (often minutes), only to an identical prefix, and only on providers and models that offer it; rates change, so check current pricing.

We did not use measured prefill time as a compute measure. The local server cached prompts across runs, so a "first" full-text call took 330 ms for 8,585 tokens. Calls with no cache reuse ran at about 270 tokens per second on this laptop. This is logged as a deviation.

Checking our fact-density calculator

Exploratory checks, not part of the protocol. They test the fact-density calculator in INGEST, which this benchmark motivated.

Is "characters ÷ 4" a good token estimate?

The calculator estimates tokens as characters divided by 4. We compared that estimate with this model's own tokenizer:

The estimate is fair for text taken as a whole, but it can be off by a fifth or more for a single page. For HTML it runs low, because markup splits into many short tokens. So the calculator understates what raw HTML costs, and the gap between HTML and text is wider than the character counts in the page sample below suggest. The fact-density percentage is a ratio of text to text, so the constant cancels out of it. Other models' tokenizers will give different numbers.

Does the "specific sentence" test keep the sentences that answer questions?

The calculator counts a sentence as specific when it carries a name, number, date or other concrete detail. For 76 of the 80 answerable questions (95%), the sentence holding the gold answer passed the test. The four it dropped explain a mechanism or define a term, for example why antistaphylococcal antibiotics have that name. The questions lean factual, so this recall is likely higher than it would be for "how" and "why" questions.

Across the 10 articles, 69% of the text was in specific sentences, ranging from 45% (Lithium-ion battery) to 92% (Mount Everest).

What a sample of real pages looks like

We ran the calculator on 12 pages, chosen for convenience to cover our site, SaaS homepages, pricing pages, reference pages, news and marketing guides. This is a description of 12 pages, not a statistic about the web.

Medians: 25.4% of the text in specific sentences, and visible text at 4.9% of the HTML. Pages were fetched on 2026-11-08 and will have changed since.

What this means for growth teams, content teams and agencies

1. Write passages that stand alone. Two to four passages of about 200 words were enough for 95–99% accuracy. If AI systems answer from a few retrieved passages rather than your whole page, each section has to answer its own question: name the subject, state the fact, and don't lean on "as mentioned above".
2. Put the specifics in the body. Summaries kept the answer to 17 of 80 questions. A TL;DR, meta description or "key takeaways" box helps people skim, but it will not carry your prices, numbers, dates and names. Put them in the sentences that make the claim.
3. Don't rewrite your site as fact lists to save tokens. The full fact rewrite was 8% smaller than the original and no more accurate. Clear, well-organized prose did as well as atomic facts once retrieval picked the right passage. Spend the effort on clarity and specifics, not on reformatting.
4. Most of a page is not the content. On the 12 pages we sampled, visible text was a median 4.9% of the HTML by characters, and markup costs more tokens per character than prose. Any system that reads raw markup pays for the rest. Keep important facts in server-rendered text, not only in images, tabs or scripts.
5. Test pages with questions. The method here is cheap: write the questions your buyers ask, give a model your page, and check the answers against the text. This whole benchmark ran on one laptop in 23 minutes of model time. It is a measurable deliverable an agency can run before and after a rewrite.
6. If you build AI tools for clients, try caching before building retrieval. With a 90% cache discount, repeat questions on the same document cost about as much as retrieval, without a search index to maintain. That holds only for repeat questions within the cache's lifetime.

Deviations from the protocol

1. An automatic retry on HTTP errors was added to the harness before preparation. It never fired.
2. A pilot of 140 answers ran before the full run to check the harness. Its answers are kept separately and excluded from every figure.
3. Prefill time was dropped as a compute measure because of cross-run prompt caching (see above).
4. The analysis script crashed on the pilot, which had no unanswerable questions. The fix changes no full-run figure.
5. The calculator checks and miss diagnostics were added after the run and are labelled exploratory.

Limitations

• One model with 7 billion parameters, run locally. Larger API models read long contexts better and may narrow the gap between full text and retrieval, or widen it.
• 100 questions, so intervals are wide: one question is 1.25 points.
• The questions were written by an AI model (Claude) reading the sampled paragraphs. They lean toward short facts, which favors retrieval and atomic facts over questions that need reasoning across a document.
• Wikipedia prose is cleaner and denser than most business pages.
• One run per question at temperature 0. Strict string scoring marks two borderline answers wrong.
• The summaries and croutons were written by the same 7B model. Human-written summaries, or a larger model's croutons, may do better.
• BM25 is simple keyword ranking. Embedding-based retrieval might change the passage and crouton results.
• The protocol was committed to git on 2026-11-08 at 01:57 (UTC+8), before any model run. The repository is private, so that timestamp cannot be independently verified. The protocol file is published below as it was committed.

Data

Everything needed to check the figures. Question and answer text derived from Wikipedia is CC BY-SA 4.0; article revisions are linked in the manifest.

• protocol.txt: the protocol as committed before the first run
• deviations.txt: every change from the protocol
• corpus-manifest.json: the 10 articles, pinned revision IDs and sampled paragraphs
• questions.json: the 100 questions and gold answers
• answers.json: all 700 answers, with token counts
• summary.json: per-condition statistics, intervals, hypotheses and caching calculation
• answer-recall.json: the specific-sentence check, question by question
• page-sample.json: calculator output for the 12 pages
• token-check.csv: the calculator's token estimate against the model's tokenizer
• bench-harness.py.txt and analysis.py.txt: the code that ran and scored the benchmark