← All research

−1.4
points of retrieval when passages stopped naming their own subject. Attaching the heading path or page title to each chunk won none of it back.
Joel MaldonadoNRLC.ai · 11 Nov 2026

Tom Johnson's chunking guide is careful to say its advice comes from vendor docs, not measurements. So we measured two of its questions on 797 benchmark questions: do headings attached to a chunk help it get found, and does it matter whether the passage names its own subject?

Short version: write the name into the sentence. The heading label didn't move retrieval.

Tom Johnson · I'd Rather Be Writing · Chunking your content for agent readability How documentation is split into chunks before an agent reads it through an MCP server or retrieval index, and how to write sections, headings, tables and procedures that still make sense when the agent sees only the chunk. idratherbewriting.com · summary in our words
797 questions50 articles, one pooled index9 chunking conditionsBM25 retrieval
The short answer Saying the subject's name in the passage itself mattered a little. Headings and page titles attached to a chunk did not help at all. We re-ran retrieval for the 800 answerable questions of our token-efficiency benchmark under the chunking settings that documentation teams are now being advised to plan for.
  • Chunk size made no measurable difference. Sections packed to about 200 words or about 375 words found the answer 97.7% to 99.5% of the time.
  • Heading paths and page titles added nothing. Prefixing every chunk with "Article > Section > Subsection" changed retrieval by 0.0 points, with a 95% interval of −0.4 to +0.4.
  • Removing the subject's name from the body cost 1.4 points (interval −2.6 to −0.1): 13 questions lost, 2 gained, out of 797.
  • Putting the name back only as a heading recovered none of it (+0.1 points, interval −0.5 to +0.8).
What this test is and is not This is a retrieval test with keyword search (BM25) on Wikipedia. It measures whether the passage holding the answer reaches the model, not what the model does with it. The conditions were chosen after reading the article above and run once, so treat it as exploratory.

The advice we tested

Tom Johnson's Chunking your content for agent readability (31 October 2026) describes how documentation is split into chunks before an agent sees it, through an MCP server or another retrieval index. The agent often receives only the chunks, not the page.

Using Google's Agent Search as the example, he notes that chunks follow the page layout, are capped at 500 tokens, and do not carry their ancestor headings by default. His advice follows from that:

  • keep each section shorter than a chunk
  • write headings that make sense on their own
  • use real heading tags
  • split long tables, procedures and tabbed content
  • don't leave information only in images

The piece is careful about its evidence. The chunker behaviour comes from vendor documentation, and he reports no before-and-after retrieval measurements. That gap is what this re-test starts to fill. We tested two of the questions it raises:

  1. Does it matter whether the chunk carries its headings?
  2. Does it matter whether the passage names its own subject?

Setup

The corpus and questions are the token-efficiency benchmark's, unchanged. Only the chunking varies.

Corpus. The same 50 English Wikipedia articles, pinned to the same revisions, cut after about 6,000 words each.

  • We rebuilt the text from each pinned revision, keeping section headings this time: 298,551 words, against 298,717 in the original corpus.
  • 797 of the 800 answerable questions have their gold answer in the rebuilt text. Those 797 are the test set.

Chunking conditions.

ConditionHow the text is cutChunks retrieved
200-word windowsThe benchmark's own method: paragraphs packed to about 200 words, ignoring sections4
Section chunks, about 375 wordsPacked within each h2 or h3 section; 375 words is roughly the 500-token cap2
Section chunks + page titleAs above, with the article title prefixed2
Section chunks + heading pathAs above, with "Article > Section > Subsection" prefixed2
De-namedEvery occurrence of the article title in the body replaced with "it" (3,963 replacements); run with no prefix, with the title and with the heading path2

The 200-word section chunks were also run with 4 retrieved, with and without headings.

Retrieval.

  • BM25 keyword ranking, the benchmark's own code.
  • All 50 articles pooled into one index of about 1,400 to 2,200 chunks, as an MCP server or site search would hold them.

Scoring.

  • A question counts as retrieved if a chunk from the correct article, among those returned, contains the gold answer.
  • Answers are matched as whole words after the benchmark's normalisation.

Statistics. Differences between conditions are paired by question. The 95% intervals come from 2,000 bootstrap resamples of whole articles, because questions from one article are related.

Results

Every condition found the answer at least 96.4% of the time, and only removing the subject's name moved the result by more than chance. Results are for 797 questions across the pooled 50-article index.

ConditionChunks returnedAnswer retrievedRight article in resultsWords returned (mean)
200-word windows498.2%99.7%630
Section chunks, ~375 words297.7%99.6%521
+ page title297.7%99.6%524
+ heading path297.7%99.6%533
De-named296.4%99.0%520
De-named + page title296.4%99.2%520
De-named + heading path296.5%99.4%528
De-named, 200-word sections497.7%99.4%583
De-named, 200-word sections + heading path498.0%99.6%605

Paired differences, in percentage points, with 95% intervals from resampling articles:

ComparisonDifference95% intervalQuestions that changed
Removing the subject's name (section chunks)−1.4−2.6 to −0.115
Adding the heading path to de-named chunks+0.1−0.5 to +0.87
Adding the page title to de-named chunks0.0−0.8 to +0.68
Adding the heading path to named chunks0.0−0.4 to +0.42
Adding the heading path, 200-word de-named chunks+0.30.0 to +0.62
200-word windows (4) vs section chunks (2)+0.5−0.6 to +1.412

Retrieving within each article alone, instead of from the pooled index, every condition scored 98.1% to 99.5%.

Why removing the name hurt

When a passage stopped naming its subject, another page that did name it often took its place. Of the 13 questions lost after de-naming, 9 had at least one returned chunk from a different article, and in 6 both returned chunks were from other articles.

  • "On what date did Fleming coin the term penicillin?" returned two chunks from Antibiotic instead of Penicillin.
  • "How many amino acids long is preproinsulin?" returned a chunk from DNA ahead of Insulin.
  • "What area in square kilometres does the Taj Trapezium Zone cover?" returned two chunks from Sahara.

A heading path did not win these back. The label names the subject once per chunk. A competing article that keeps naming the subject in its sentences scores higher on the same word.

This is the case that matters for businesses: a buyer's question about your product competes with other sites' pages that name your category, your competitors and sometimes your product, all in their body text.

The other 4 losses stayed within the right article, and they are partly an artefact of the test. For one-word titles such as Penicillin, Coffee or DNA, replacing the title also removed ordinary topic words, not just the subject's name. So the 1.4-point cost is probably an overestimate for those articles.

How this fits with contextual retrieval

Our null result for heading paths does not contradict Anthropic's Contextual Retrieval work. The two tests added different things to each chunk.

  • What Anthropic added: a model-written explanation of 50 to 100 tokens for each chunk, saying where it sits in its document. Prepended before both embedding and BM25 indexing, it cut the top-20 retrieval failure rate by 49%, from 5.7% to 2.9%.
  • What we added: a heading path of a few words, and only BM25 was tested.
  • How much room there was: on Wikipedia the passages already name their subjects, so failures were rare to begin with. Only 17 of our 1,371 section chunks open with a word like "it", "this" or "they".
Read together, the two results fit one explanation, which we have not yet tested directly: a bare label is too little context, while a sentence stating what the passage is about can be enough.

What this means for writers

Name the subject in the passage itself, and don't rely on headings or metadata to do it for you. Tom Johnson's advice to write specific headings still stands; this test suggests the sentences under the heading matter more.

  • Restate the subject in the first sentence of every section, by name: "Croutons' pilot takes one week", not "The pilot takes one week". In our test this was the one change that moved retrieval.
  • Don't count on heading paths or page titles in chunk metadata. Adding them changed nothing measurable, and many chunkers drop them anyway.
  • Chunk size within the 200 to 375 word range is not the lever. Keeping sections under the chunker's cap still matters for fit, but it did not change how often the answer was found.
  • Expect competition. Your passage is ranked against other sites' pages on the same topic. A page that names the product, the category and the specifics in its sentences gives keyword retrieval something to match.

We are adding the first rule to our answer and decision page methodology as a tested rule, and marking heading-path metadata as not shown to help.

Limitations

This is a narrow test, and its results apply only as far as its setup goes.

  • Retrieval only. No model answered the questions, so this measures whether the answer reached the model, not whether it was used.
  • Keyword ranking only. BM25 was the only ranker. Embedding retrieval, hybrid search and rerankers may treat headings differently, and Anthropic's results suggest they can.
  • An easy corpus. Wikipedia prose names its subjects, the 50 articles cover distinct topics, and the questions were written from the passages that answer them, so they share their wording. Business sites, where many pages cover the same product, are a harder case.
  • Synthetic de-naming. Replacing the title also removed ordinary words when the title is a common noun, which overstates the cost for those articles.
  • Reconstructed text. The corpus was rebuilt from the pinned revisions with headings kept; it matches the original to within 166 words in 298,717, and 3 of 800 gold answers are not in the rebuilt text.
  • One run, no protocol committed in advance. These conditions were chosen after reading Tom Johnson's article and run once. Treat this as exploratory, not as a test of hypotheses stated beforehand.

Next tests

  1. The same conditions with embedding and hybrid retrieval.
  2. Answer accuracy with the benchmark's model, for the conditions above.
  3. Model-written context sentences, the contextual retrieval approach, against heading paths.
  4. A business-site corpus, where pages compete on the same topic.

Data and code

The questions, gold answers and pinned revisions are those already published with the token-efficiency benchmark. The script that rebuilds the corpus, cuts the chunks, runs retrieval and computes the intervals will be added to this page, together with its per-question results.

Sources