{
    "schemaVersion": "1.0.0",
    "@context": "https://schema.org",
    "@type": "WebPage",
    "@id": "https://nrlc.ai/research/more-answers-per-chip/#webpage",
    "url": "https://nrlc.ai/research/more-answers-per-chip/",
    "name": "More Answers per Chip: Request Shaping for Energy-Efficient LLM Inference | NRLC.ai",
    "description": "A research agenda for cutting energy per AI answer outside the serving engine. Five objectives, pass criteria fixed in advance, results published either way.",
    "inLanguage": "en",
    "content": {
        "format": "text/markdown",
        "wordCount": 2133,
        "headings": [
            {
                "level": 1,
                "text": "More Answers per Chip"
            },
            {
                "level": 2,
                "text": "Abstract"
            },
            {
                "level": 2,
                "text": "1The gap"
            },
            {
                "level": 2,
                "text": "2What we measured"
            },
            {
                "level": 2,
                "text": "3Proposed system: Memory-Pressure-Aware Request Shaping"
            },
            {
                "level": 2,
                "text": "4Pre-registered objectives"
            },
            {
                "level": 2,
                "text": "5Why it matters"
            },
            {
                "level": 2,
                "text": "6Related work"
            },
            {
                "level": 2,
                "text": "7Open method"
            },
            {
                "level": 2,
                "text": "References"
            }
        ],
        "markdown": "---\ntitle: More Answers per Chip: Request Shaping for Energy-Efficient LLM Inference | NRLC.ai\ndescription: A research agenda for cutting energy per AI answer outside the serving engine. Five objectives, pass criteria fixed in advance, results published either way.\norganization: Neural Command LLC\ncanonical: https://nrlc.ai/research/more-answers-per-chip/\n---\n\n## Abstract\n\nEnergy per answer, not energy per token, is the figure AI data centres will increasingly be judged on. Measurements on a production-class inference stack show that accelerator energy follows busy time: a three-term model of idle, busy and per-token draw explains 99.8% of energy, while tokens alone explain 58%. Working memory, not compute, is the binding limit, and 4.7–15.7% of accelerator energy goes to work that did not need doing [1]. We propose Memory-Pressure-Aware Request Shaping (MPRS), a control layer in front of any serving engine that attributes energy to each request, forecasts KV-cache pressure at arrival, rewrites prompts so shared context can be reused, verifies answer reuse, and schedules work to minimise energy per answer under quality and latency limits. We set out five objectives with pass criteria fixed before any data is collected, ending in a 10% reduction in accelerator energy per answer on replayed production traffic at equal quality. Results will be published whether they pass or fail.\n\nKeywords: LLM inference, energy per answer, KV cache, request scheduling, prefix caching, data-centre efficiency cs.DC · cs.LG\n\n## 1The gap\n\nLarge language model (LLM) inference is becoming the main load in new AI data centres, yet the figure operators, utilities and regulators now care about, energy per useful answer, is controlled by nothing in the serving stack. Serving engines schedule work inside one engine to raise throughput [2, 3, 8]. API gateways route on price per token [9]. Neither sees the accelerator's memory state, and neither is accountable for energy per answer.\n\n## 2What we measured\n\n99.8%\n\nof accelerator energy explained by idle + busy + per-token draw. Tokens alone: 58%.\n\n[\n\n[1](#ref-1)]\n\n3.3×\n\nless energy per request at peak load than at low load, on the same hardware.\n\n[\n\n[1](#ref-1)]\n\n0% → 55%\n\nreuse of earlier computation when a shared document moves from mid-prompt to the front.\n\n[\n\n[1](#ref-1)]\n\nOur measurements on a production-class inference stack [1] show why this gap matters. A three-term model (idle draw, a fixed draw while the accelerator is busy, and a small per-token cost) explains 99.8% of measured accelerator energy; token counts alone explain 58%. Energy follows busy time, not tokens. The same hardware used 3.3 times less energy per request at peak load than at low load. The working memory of active requests (the KV cache) was above 90% full a third of the time and reached 100%, so requests were paused and recomputed while compute still had room [13]. Long-form summaries were 7% of requests and 37% of energy. Between 4.7% and 15.7% of accelerator energy went to work that did not need doing: identical retried calls, repeated questions, and shared documents placed mid-prompt, which gave 0% reuse of earlier computation against 55% when the same document came first.\n\nTokens only\n\n58%\n\nIdle + busy + per-token\n\n99.8%\n\n0%\n\n50%\n\n100%\n\nFigure 1.\n\nShare of measured accelerator energy explained by two models. Energy tracks busy time, not tokens\n\n[\n\n[1](#ref-1)]\n\n.\n\nshare of requests\n\nshare of energy\n\nMulti-step agents\n\n64%\n\n38%\n\nChat + document Q&A\n\n29%\n\n25%\n\nLong-form summaries\n\n7%\n\n37%\n\n0%\n\n35%\n\n70%\n\nFigure 2.\n\nShare of requests vs share of energy by workload class. Summaries are a small share of traffic and a large share of energy\n\n[\n\n[1](#ref-1)]\n\n.\n\n## 3Proposed system: Memory-Pressure-Aware Request Shaping\n\nMPRS is a closed-loop control layer that sits in front of any serving engine and reshapes the request stream before it reaches the accelerator, with energy per answer as its objective and answer quality as a hard constraint. It joins four mechanisms that have not been combined:\n\n- per-request energy attribution, fitted online from hardware counters and engine logs, that prices each request in busy-time joules rather than tokens;\n- a forecaster that predicts KV-cache pressure from features known at request arrival (workload class, prompt structure, expected output length) early enough to act;\n- quality-preserving prompt canonicalization, which moves shared context into a stable prefix the engine can reuse, plus verified reuse of repeated answers;\n- admission, batching and routing that move long-generation and agent fan-out work into the windows and model sizes where energy per answer is lowest, within latency targets.\nApps & agents\n\nrequests\n\nMPRS · request-shaping layer\n\n1  Energy attribution · J per request\n\n2  KV-pressure forecaster · at arrival\n\n3  Canonicalizer + verified reuse\n\n4  Controller · admit, defer, batch, route\n\nServing engine\n\nvLLM · SGLang · TensorRT-LLM\n\nAccelerators\n\nH100 · MI300X · GB200\n\ntelemetry: GPU counters · engine metrics · gateway logs\n\nFigure 3.\n\nMPRS sits between clients and any serving engine. It acts on requests before they reach the accelerator and learns from telemetry operators already collect.\n\nToday's prefix caches reuse only what clients happen to send in the same order; MPRS makes reuse a property of the request stream. Today's gateways optimise cost per token; our data show tokens explain barely half of energy. No current system ties a request-level action to a verified change in energy per answer.\n\nBecause MPRS runs outside the engine and needs only telemetry operators already collect (GPU counters, engine metrics, gateway logs), it can be adopted without retraining models or replacing infrastructure.\n\n## 4Pre-registered objectives\n\nEach objective has a pass criterion fixed before data collection, under the protocol of our published benchmarks: pre-registered hypotheses, paired bootstrap intervals, and negative results reported [4, 5, 6].\n\nO1. Portable energy attribution. Fit the idle + busy + per-token model on at least three accelerator families and two serving engines. Pass: at least 95% of measured energy explained on each. Challenge: busy draw shifts with batch mix and power capping. Approach: add batch-occupancy and clock terms only where held-out fit improves [11].\n\nO2. Early forecasting of memory pressure. Predict at arrival whether admitting a request pushes KV occupancy past 90% during its lifetime. Pass: AUROC of at least 0.85 on held-out traffic. Challenge: output length, the main driver of KV growth, is unknown at arrival. Approach: class-conditional length distributions, conformal bounds, and online correction as tokens stream.\n\nO3. Quality-preserving canonicalization. Move shared documents into a stable prefix. Pass: at least 50% prefix reuse on traffic with shared context, with no significant accuracy loss. Challenge: position changes what models use [7], and in our chunking study removing a passage's subject cost 1.4 points [5]. Approach: reorder only segments whose answer-relevant content does not depend on order, verified per workload class by paired evaluation.\n\nO4. Verified answer reuse. Serve repeated and near-repeated questions from cache only when an independent check confirms equivalence. Pass: at most 1% answer disagreement on audited hits. Challenge: semantic matching yields confident false hits. Approach: a two-stage gate (canonical form, then a small verifier model), with the verifier's own energy charged against the saving.\n\nO5. Closed-loop control. Combine O1–O4 into a controller that admits, defers, batches and routes requests to minimise energy per answer under p95 latency and quality limits. Pass: at least 10% lower accelerator energy per answer on replayed production traffic. Challenge: actions interact; deferral raises peak occupancy and can cause the preemption it was meant to avoid. Approach: model-predictive control on the O2 forecast, with a safe fall-back to pass-through.\n\nTable 2.\n\nObjectives and pass criteria, fixed before data collection.\n\nId\n\nObjective\n\nPass criterion\n\nO1\n\nPortable energy attribution\n\n≥ 95% of measured energy explained on each of ≥ 3 accelerator families, 2 engines\n\nO2\n\nEarly KV-pressure forecast\n\nAUROC ≥ 0.85 for crossing 90% occupancy, held-out traffic\n\nO3\n\nQuality-preserving canonicalization\n\n≥ 50% prefix reuse on shared-context traffic, no significant accuracy loss\n\nO4\n\nVerified answer reuse\n\n≤ 1% answer disagreement on audited cache hits\n\nO5\n\nClosed-loop control\n\n≥ 10% lower accelerator energy per answer at equal quality and p95 latency\n\nKey risk: the avoidable share we measured depends on how often traffic repeats [1]. If it repeats less, O4 adds little, and we test whether O2 and O3 alone recover energy. A negative result on any objective will be published.\n\n## 5Why it matters\n\nUS data centres used 176 TWh in 2023, 4.4% of national electricity, and are projected to use 325–580 TWh by 2028 [10]. Power, not chips, now limits where AI capacity is built, and regulators are moving toward output metrics. Malaysia, one of the fastest-growing AI data-centre markets, ties incentives to declared PUE, WUE and CUE, and requires large energy users to report and carry out energy saving measures under its 2024 energy efficiency act [15].\n\n0.170 Wh\n\nfacility energy per answer in the reference 20 MW hall. Published median for a Gemini text prompt: 0.24 Wh.\n\n[\n\n[1](#ref-1),\n\n[12](#ref-12)]\n\nFor a reference 20 MW inference hall, our benchmark implies 20,300–67,900 GJ of avoidable facility energy a year and 0.7–2.2 MW of accelerator capacity freed [1].\n\nTable 1.\n\nEnergy saving measures in EECA 2024 format for a reference 20 MW inference hall (70% load, PUE 1.4). Each counts only after replay shows answer quality holds\n\n[\n\n[1](#ref-1)]\n\n.\n\nId\n\nMeasure\n\nFacility GJ/yr\n\nt CO₂/yr\n\nWater m³/yr\n\nESM 1\n\nRemove identical retried calls\n\n9,086\n\n1,913\n\n3,966\n\nESM 2\n\nPlace shared documents first\n\n11,250\n\n2,369\n\n4,911\n\nConservative total (ESM 1 + 2)\n\n20,336\n\n4,282\n\n8,877\n\nESM 3\n\nCache repeated questions (upper bound)\n\n≤ 47,594\n\n≤ 10,021\n\n≤ 20,775\n\nBenchmark-mix total\n\n67,930\n\n14,303\n\n29,652\n\n## 6Related work\n\nServing engines (vLLM, SGLang, TensorRT-LLM) optimise inside one engine [2, 3, 8]; gateways (LiteLLM, Portkey, Helicone) optimise price per token [9]; facility tools measure buildings, not answers. Recent measurement work diagnoses where inference energy goes [11, 12] and how long contexts strain GPU memory [13]. MPRS uses engine caches and gateway logs as inputs and adds the missing layer: one that is accountable for energy per answer. Croutons, our inference-management product, already reads these logs and telemetry on the operator's own machine [14].\n\n## 7Open method\n\n−89%\n\ninput tokens with four retrieved passages, at 98.5% accuracy vs 95.1% for the full article.\n\n[\n\n[4](#ref-4)]\n\nEverything in this agenda is open. Method, data and code will be published under CC BY 4.0 as each objective is run, in the same way as our token-efficiency benchmark, which reported a result against our own product: our atomic-fact rewrite scored 5.4 points below full text, while passage retrieval cut input tokens 89% and raised accuracy to 98.5% [4].\n\n## References\n\n- [1]Croutons (Neural Command, LLC). Croutons Malaysia: energy per AI answer, in MITI and EECA terms. Benchmark and report, November 2026. https://croutons.ai/malaysia\n- [2]W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. arXiv:2309.06180. https://arxiv.org/abs/2309.06180\n- [3]L. Zheng et al. SGLang: Efficient Execution of Structured Language Model Programs. NeurIPS 2024. arXiv:2312.07104. https://arxiv.org/abs/2312.07104\n- [4]J. Maldonado. How Many Tokens Does an Answer Need? A Token-Efficiency Benchmark. NRLC.ai, November 2026. https://nrlc.ai/research/token-efficiency-benchmark/\n- [5]J. Maldonado. Do Headings Survive Chunking? A Re-test on 800 Questions. NRLC.ai, 11 November 2026. https://nrlc.ai/research/headings-chunking-retest/\n- [6]NRLC.ai. Does AI search optimization work? A randomized trial. Public pre-registered plan, 2026. https://nrlc.ai/en-us/insights/ai-search-optimization-randomized-trial/\n- [7]N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang. Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. arXiv:2307.03172. https://arxiv.org/abs/2307.03172\n- [8]A. Agrawal et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024. arXiv:2403.02310. https://arxiv.org/abs/2403.02310\n- [9]I. Ong et al. RouteLLM: Learning to Route LLMs with Preference Data. arXiv:2406.18665, 2024. https://arxiv.org/abs/2406.18665\n- [10]A. Shehabi et al. 2024 United States Data Center Energy Usage Report. Lawrence Berkeley National Laboratory, December 2024. https://www.energy.gov/articles/doe-releases-new-report-evaluating-increase-electricity-demand-data-centers\n- [11]J.-W. Chung, R. Wu, J. J. Ma, M. Chowdhury. Where Do the Joules Go? Diagnosing Inference Energy Consumption. arXiv:2601.22076, 2026. https://arxiv.org/abs/2601.22076\n- [12]C. Elsworth et al. Measuring the environmental impact of delivering AI at Google Scale. arXiv:2508.15734, 2025. https://arxiv.org/abs/2508.15734\n- [13]S. Luo, H. Shen. No Buffer, No Bottleneck: Efficient Zero-Copy KV Cache Offloading for Long-Context LLMs. OSDI 2026. https://www.usenix.org/system/files/osdi26-luo.pdf\n- [14]Croutons. Documentation: Crouton Units and the croutons CLI. 2026. https://croutons.ai/docs\n- [15]Croutons. Building data centres in Malaysia: power, water and approval rules (MITI guideline, DESAC, EECA 2024). 2026. https://croutons.ai/malaysia/building-data-centers\n© 2026 Neural Command, LLC · Text and figures CC BY 4.0 · Benchmark figures: Croutons Malaysia [1]\n\n\n\n---\n\nSource: https://nrlc.ai/research/more-answers-per-chip/\nPublisher: Neural Command LLC\nLicense: Editorial use with attribution\n"
    },
    "structuredData": [
        {
            "@context": "https://schema.org",
            "@type": "ScholarlyArticle",
            "@id": "https://nrlc.ai/research/more-answers-per-chip/",
            "url": "https://nrlc.ai/research/more-answers-per-chip/",
            "mainEntityOfPage": "https://nrlc.ai/research/more-answers-per-chip/",
            "headline": "More Answers per Chip: Memory-Pressure-Aware Request Shaping for Energy-Efficient LLM Inference",
            "name": "More Answers per Chip",
            "alternativeHeadline": "Memory-Pressure-Aware Request Shaping for energy-efficient LLM inference",
            "description": "A research agenda for cutting energy per AI answer outside the serving engine. Five objectives, pass criteria fixed in advance, results published either way.",
            "datePublished": "2026-11-11",
            "dateModified": "2026-11-11",
            "inLanguage": "en",
            "author": {
                "@type": "Person",
                "@id": "https://nrlc.ai/en-us/about/joel-maldonado/#person",
                "name": "Joel Maldonado",
                "affiliation": {
                    "@type": "Organization",
                    "@id": "https://nrlc.ai/#organization",
                    "name": "Neural Command, LLC",
                    "url": "https://nrlc.ai"
                }
            },
            "publisher": {
                "@type": "Organization",
                "@id": "https://nrlc.ai/#organization",
                "name": "Neural Command, LLC",
                "url": "https://nrlc.ai"
            },
            "license": "https://creativecommons.org/licenses/by/4.0/",
            "keywords": "LLM inference, energy per answer, KV cache, request scheduling, prefix caching, data-centre efficiency",
            "about": [
                {
                    "@type": "Thing",
                    "name": "LLM inference"
                },
                {
                    "@type": "Thing",
                    "name": "Energy efficiency"
                },
                {
                    "@type": "Thing",
                    "name": "KV cache"
                },
                {
                    "@type": "Thing",
                    "name": "Data centres"
                }
            ],
            "isBasedOn": "https://croutons.ai/malaysia",
            "associatedMedia": {
                "@type": "MediaObject",
                "encodingFormat": "application/pdf",
                "contentUrl": "https://nrlc.ai/research/more-answers-per-chip/more-answers-per-chip.pdf"
            },
            "citation": [
                "https://croutons.ai/malaysia",
                "https://arxiv.org/abs/2309.06180",
                "https://arxiv.org/abs/2312.07104",
                "https://nrlc.ai/research/token-efficiency-benchmark/",
                "https://nrlc.ai/research/headings-chunking-retest/",
                "https://nrlc.ai/en-us/insights/ai-search-optimization-randomized-trial/",
                "https://arxiv.org/abs/2307.03172",
                "https://arxiv.org/abs/2403.02310",
                "https://arxiv.org/abs/2406.18665",
                "https://www.energy.gov/articles/doe-releases-new-report-evaluating-increase-electricity-demand-data-centers",
                "https://arxiv.org/abs/2601.22076",
                "https://arxiv.org/abs/2508.15734",
                "https://www.usenix.org/system/files/osdi26-luo.pdf",
                "https://croutons.ai/docs",
                "https://croutons.ai/malaysia/building-data-centers"
            ]
        }
    ],
    "internalLinks": [
        {
            "url": "https://nrlc.ai/research/more-answers-per-chip/more-answers-per-chip.pdf",
            "label": "PDF"
        },
        {
            "url": "https://nrlc.ai/research/token-efficiency-benchmark/",
            "label": "https://nrlc.ai/research/token-efficiency-benchmark/"
        },
        {
            "url": "https://nrlc.ai/research/headings-chunking-retest/",
            "label": "https://nrlc.ai/research/headings-chunking-retest/"
        },
        {
            "url": "https://nrlc.ai/en-us/insights/ai-search-optimization-randomized-trial/",
            "label": "https://nrlc.ai/en-us/insights/ai-search-optimization-randomized-trial/"
        }
    ],
    "representations": {
        "html": "https://nrlc.ai/research/more-answers-per-chip/",
        "markdown": "https://nrlc.ai/research/more-answers-per-chip.md",
        "json": "https://nrlc.ai/research/more-answers-per-chip.json"
    },
    "publisher": {
        "@id": "https://nrlc.ai/#organization",
        "name": "Neural Command, LLC"
    }
}
