# Deviations from PROTOCOL.md

Everything that differs from the protocol committed in f63c9f4, listed before the write-up.

1. **Retry on HTTP errors (harness, before prep).** `bench.py` retries an Ollama HTTP error up to
   three times. It never fired: `prep.log`, `results/pilot.log` and `results/full.log` contain no
   retries. Conditions, prompts and settings are unchanged.

2. **Pilot run (not in the protocol).** A pilot of 140 answers (2 answerable questions per article,
   all 7 conditions) ran before the full run to check the harness. Its answers are kept separately in
   `results/pilot.jsonl` and are not part of any reported figure. The full run re-answered every
   question from scratch.

3. **Prefill time is not reported as a compute measure.** Ollama's llama.cpp server keeps a prompt
   cache in memory, including across runs, so `prompt_eval_duration` measures cache hits rather than
   compute (for example, a "first" full-text call took 330 ms for 8,585 tokens). Calls with no cache
   reuse ran at about 270 tokens per second. Input tokens are used as the compute measure instead.
   During the pilot, one request also took 31 minutes while the machine was idle-throttled; the full
   run was made with idle sleep prevented.

4. **Analysis bug fix.** `analyze.py` crashed when a run had no unanswerable questions (the pilot).
   It now skips H3 in that case. No figure for the full run changes.

5. **Additional, exploratory checks (not in the protocol), reported separately and labelled as such:**
   - `tokencheck/`: the ingest calculator's estimate (tokens = characters / 4) compared with the
     benchmark model's tokenizer on the 10 articles and on 12 web pages (visible text and the first
     100,000 characters of HTML).
   - `tokencheck/answer_recall.json`: whether ingest's "specific sentence" test keeps the sentence that
     holds each answerable question's gold answer.
   - Diagnostics of misses: whether each gold answer survives in the summary and in the croutons.