acceptodds
Under review as a conference paper at ICLR 2027

Watt Does a Benchmark Point Cost? Measuring the Energy Efficiency of LLM Capability under Production Serving

Abstract

Benchmark scores tell us what an LLM can do, but not how much energy is required to deliver that capability. We directly measure this trade-off for 86 open-weight model checkpoints, evaluated on seven benchmarks and six hardware platforms ranging from a consumer GPU to an eight-GPU server. We find large differences in energy efficiency even among models with similar capability. Most configurations lie away from the score–energy Pareto frontier: they use a median of 2.6–4.8 more energy than another configuration that scores at least as well, with differences reaching 27. Reasoning can improve capability, but enabling thinking increases energy by about 3.7 on average, and the benefit depends strongly on the model family and output budget. Generated-token count is also an imperfect proxy for energy, misordering 18–24% of model pairs. In contrast, FP8 reduces energy by roughly a quarter with little change in score, while changing hardware largely rescales energy without changing the relative ranking of models. These results show that capability alone gives an incomplete picture of efficiency. We argue that benchmark scores should be reported together with measured energy consumption under a specified serving regime and output budget.

Primary area:
datasets and benchmarks
Keywords:
large language models, energy efficiency, benchmarking, evaluation, inference serving, green AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8f9vOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the trade-off between the LLM benchmark capability and the directly measured inference energy under high-unutilization serving. This paper evaluates 86 open-weight checkpoints, i.e., 134 serving configurations, 7 benchmarks, and 6 hardware platforms. The analysis includes score-energy Pareto frontiers, controlled comparisons of reasoning vs. non-reasoning modes, output-budget sensitivity, reduced-precision weights, and hardware-dependent energy measurements.

The paper's main empirical findings are that many evaluated configurations are Pareto-dominated, reasoning often incurs a substantial energy premium, generated-token count is an imperfect proxy for physical energy, FPB reduces energy with little aggregate score change, and model efficiency rankings appear relatively stable across the hardware platforms evaluated in the 96-GB tier.

The work is carefully executed and potentially useful as a measure study. However, I am not convinced the current evidence is sufficiently representative or generalizable to support the breadth of the paper's claims about contemporary LLM serving. In more detail, the effective diversity of the model roster is narrower than the headline checkpoint count suggests, and the hardware conclusions are substantially broader than the evaluated hardware and serving regimes justify.

Critical strengths

Careful and unusually broad direct measurement campaign Good robustness and sensitivity analysis within the evaluated setup

Critical weaknesses

a) The model roster is broad in count (i.e., 86 checkpoints and 134 serving, etc.), but not sufficiently representative of the contemporary frontier serving landscape. In particular, many study rows are precision variants, reasoning-mode variants, or reasoning-effort variants of the same underlying model families. The study is also restricted to open-weight checkpoints that are compatible with the authors’ pinned serving stack. These points matter as the main Pareto conclusions are conditional on the candidate set. The authors also note that the roster is not a random sample of the model ecosystem. I therefore do not believe that the current study supports broad conclusions about "current LLMS" as strongly as the authors highlight.

b) The hardware conclusions substantially exceed their evaluation and serving space: they claim that changing hardware largely rescales absolute energy while preserving relative model efficiency. Nevertheless, the like-for-like hardware comparison is concentrated in the 96-GiB tier with four hardware platforms, while the 32- as well as 512-GiB are presented by RTX 5090 and 8×H100 node, respectively. In my opinion, this is not sufficient evidence to generalize hardware-invariant model rankings across the full model-size spectrum. In practice, small single-GPU workloads and large multi-GPU/MoE deployments can exhibit qualitatively different bottlenecks. Moreover, the serving regime is narrow. The authors use saturated continuous batching with platform-dependent admission limits. This presents a high-utilization, throughput-oriented operating point, not production serving in general. The evaluation scenarios (e.g., low-QPS serving, bursty traffic, latency-constrained serving, etc.) may produce different energy rankings. In fact, the C1 appendix shows that energy per generated token can increase severalfold and throughput can collapse relative to the saturated regime, demonstrating that utilization is not a minor detail. Finally, the compared systems differ in more than accelerator architecture, namely host CPU, memory technology, interconnect, tensor/expert parallelism, admission limit, and so on. The primary measurement boundary also differs across device classes (e.g., discrete-GPU board power, GH200 GPU+HBM, GB10 GPU rail). Appendix A and D show that the apparent GB10 advantage changes substantially when a wider package-level boundary is considered. This makes the authors' conclusion that hardware "largely rescales energy" weaker than the paper suggests.

Questions for authors

a) How sensitive are the reported Pareto-front memberships, dominance ratios, and reasoning-energy conclusions to the compositions of the model roster?

b) The hardware-invariance claim is supported by the 96-GiB tier. What evidence supports extrapolating this result to small single-GPU models and large multi-GPU/MoE models?

c) Do Pareto membership and relative model rankings remain stable across realistic utilization/concurrency levels, rather than only under saturated continuous batching?

d) How much do the cross-platform rankings change when all systems are compared under the broadest practically comparable measurement boundary?

6 comments

Thank you for the valuable review. We agree with its main point: several claims were broader than the evidence. Below we state the narrower claims the evidence supports, which the final version will adopt, and add sensitivity analyses on our existing data. Answers follow your questions; section and appendix numbers refer to the submitted PDF.

(a) Roster composition

We agree that the roster is not a sample of the ecosystem. Our claims cover 86 open-weight checkpoints served with one pinned stack, not "current LLMs" in general, and the final version will say so (abstract, Sec. 1 and Sec. 9). On duplication: the main frontier comparison uses one weight format per tier (BF16, BF16 and FP8; 82 configurations), so precision twins enter only the controlled comparison in Sec. 7. Reasoning modes are separate configurations by design, because a deployer chooses the mode and it changes energy 3.7× on average.

Sensitivity to the roster (new analyses of the Fig. 1 panels):

  • Leave one lineage out. The median energy gap of dominated configurations stays at 3.9–5.1× on the 32 GiB tier (4.8× with all lineages) and 3.2–5.0× on the 96 GiB tier (4.5×). On these two tiers no remaining front member leaves the front, and at most three configurations join it. On the 512 GiB tier the result depends on one lineage: without Qwen (8 of 27 configurations), 8 of the remaining 19 are dominated, by a median of 1.5× instead of 2.6×.
  • One configuration per base model, choosing one of its modes or effort levels at random (2,000 draws): 67–76% of configurations are dominated, by a median of 2.4–4.3× (medians over draws, per tier).
  • Reasoning energy. The 3.7× premium over 44 pairs is 3.5–3.9× when any one lineage is left out and 4.3× with lineages weighted equally; a bootstrap over lineages gives a 95% interval of 3.2–4.7×. The score gain depends on lineage, as the paper reports: its median is 3.4–8.4 points with one lineage left out (8.4 without Qwen).
  • FP8: 0.70–0.74× with any lineage left out, and 0.66–0.77× by lineage bootstrap.

The qualitative findings and the reasoning and precision effects are therefore robust to the roster, whereas front membership and the size of the dominance gaps are properties of the candidate set, as you note. The final version will say so where those numbers are reported (Sec. 5).

(b) Hardware claims

Nothing in our data supports extrapolating the hardware result beyond the 96 GiB tier: no configuration was served in two tiers. The final version will state the result only where it was measured (abstract, Sec. 7, Sec. 9): on four platforms serving the same 96 GiB-tier configurations, hardware rescales energy 0.5–2.2× while preserving their ranking (ρ = 0.99). Within that tier, the agreement does not rest on one kind of model or system. Spearman ρ with the RTX PRO 6000 is at least 0.90 within every subgroup we checked: dense and MoE models, thinking and non-thinking configurations, and each weight format. It holds on a two-GPU tensor-parallel system (2×RTX A6000) as well as on single GPUs and on two superchips whose memory bandwidth differs 15×. We agree that the platforms differ in more than the accelerator: this is a comparison between complete serving systems as configured (Table 6), and small single-GPU and large multi-GPU MoE deployments may behave differently. The final version will say both in Sec. 7 and Sec. 8.

(c) Utilisation and concurrency

No, we have not shown this, and we do not claim it. The paper's description of the regime also needs correcting: runs were not uniformly saturated. Each benchmark's requests are offered together, up to 1.5× vLLM's default sequence limit at a time, and run with continuous batching. Concurrency is therefore bounded by the sequence limit, the KV cache or the requests remaining (our general comment gives how often each applied), and small benchmarks (AIME has 30 requests per attempt) and the 8×H100 node, whose KV cache exceeds most request sets, run well below capacity. What we measure is a batch evaluation workload, not production serving in general. In the final version we will remove "saturated", change the title's "…under Production Serving" to "…under Batched Serving", and state in the limitations: "All main experiments offer each benchmark's requests to the server together, up to 1.5× its sequence limit at a time, so utilisation depends on the benchmark's size and the platform's memory; it is lowest on the 8×H100 node, whose KV cache exceeds most benchmarks' request sets. The energy values and rankings should not be assumed to hold at other loads, such as low request rates, bursty traffic or response-time targets."

The one same-configuration contrast we have with full benchmark traffic is mild. An earlier round on the RTX 5090 with the client limited to 100 concurrent requests (instead of 384) gives a suite-energy rank correlation of 0.82 with the final runs over 16 configurations, and the Pareto fronts share 6 of 7 members. We do not take this as evidence for other loads. App. C.1 shows how much utilisation can matter: energy per token is 6.6× higher at batch size 1.

(d) Broadest comparable boundary

The rankings do not change, but the absolute ratios do. On the two superchips we also record package (GB10) and module (GH200) energy, which add the CPU and the LPDDR5X memory. Against the RTX PRO 6000's board power, Spearman ρ is at least 0.98 at both GPU and package scope, and each platform's Pareto front is the same at both scopes, because the additional draw is nearly constant. Absolute energy moves substantially: the GB10 uses 0.61× the RTX PRO 6000's energy at GPU-rail scope and 1.03× at package scope, and the GH200 0.64× and 0.73×. Measured like a discrete GPU, with its memory but without its CPU, the GB10 would fall between these two values. A full system boundary would also need the discrete-GPU hosts' CPU and memory, which we did not meter. The final version will give both scopes in Sec. 7 and limit the hardware sentence in the abstract accordingly.

We hope the narrowed claims and the added analyses address your concerns, and we are happy to answer further questions.

The authors’ response considerably addresses my concerns about overgeneralization by narrowing the claims and providing useful roster sensitivity analyses. However, some points still require clarification. In particular, the energy-measurement procedure, including the calculation of energy per attempt after trimming the measurement interval, needs to be reconciled with the acknowledgment that runs were not uniformly saturated. The dependence of the 512 GiB dominance results on Qwen should also be stated explicitly. Finally, stable model rankings should be distinguished from comparisons of hardware efficiency, since a common full-system measurement boundary remains unavailable. Overall, the rebuttal is constructive and reduces the severity of my original concerns, but I would appreciate these clarifications before considering them fully resolved.

Reviewer_bmJNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper conducted a large-scale evaluation on 86 open-weight model checkpoints across 134 serving configurations, 7 benchmarks, 6 hardware platforms. They focus on the tradeoffs between score and energy consumption by constructing the score-energy Pareto frontier. They found most configurations are dominated; enabling thinking increases energy significantly while capability gains depends on models; generated token count is not a good proxy for energy consumption.

Critical strengths

This paper conducted a large-scale evaluation on 86 open-weight model checkpoints across 134 serving configurations, 7 benchmarks, 6 hardware platforms. The controlled comparisons of reasoning modes, output budgets, and precision provide useful deployment data.

Critical weaknesses
  • The contribution is primarily broader empirical coverage: the paper does not clearly establish which methodological advances or generalizable insights go beyond existing studies of energy–quality tradeoffs and workload-dependent serving efficiency.
  • The results come from one high-load setup, with different decoding settings for each model and parts of each run excluded from the energy measurements, so it is unclear whether the efficiency rankings would hold in other production settings.
Questions for authors
  1. What does the present study establish that these earlier characterization approaches could not? Please explain the main differences from the following two papers.
  • The Hidden Joules, which studies accuracy–energy tradeoffs across 1,200 vision models. ICML 2025.
  • Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective. arXiv 2508.16712.
  1. Section 4 suggests that you reran the benchmarks yourselves. Can you confirm that each score and its energy measurement come from the same run? Would the rankings change if all models used the same sampling settings instead of their recommended settings?

  2. Were all models actually running under sustained high load? How did you check this, especially for small benchmarks such as AIME and GPQA? Would the rankings hold if models received requests at the same rate or had to meet the same response-time limit?

  3. Does the energy measurement cover the work being scored? You remove the first 10% and last 15% of each run when measuring energy, but score the full benchmark. How do you account for this difference? Could removing the last part leave out costly long responses, and would measuring the full run change your conclusions?

  4. How much do the results depend on how benchmarks are combined? You average the benchmark scores and use a geometric mean for energy. Would the main rankings change if some tasks were given more weight, as they might be in a real application?

3 comments

Thank you for the precise questions. In short: every score and its energy come from the same attempt (Q2); not all runs were at sustained high load (Q3); every reported energy value covers the whole run, and no figure uses the cropped window described in Sec. 4 (Q4); and the main conclusions hold under other benchmark weightings (Q5). Section and appendix numbers refer to the submitted PDF.

Q1 / W1. What the study establishes beyond The Hidden Joules and qMeter

For the LLMs we measure, the energy needed to reach a score depends both on how many tokens a model chooses to generate and on how efficiently the system serves them, and the two vary independently. Between the 10th and 90th percentiles, generated tokens differ 6–14× and energy per token 3–10×, and the two are essentially uncorrelated (Sec. 5). Energy per token also depends on the workload: for the same configuration and GPU, it differs by a median of about 4× across our seven benchmarks, and MATH-500's per-token rate applied to AIME accounts for only about a third of AIME's measured energy. Capability–energy comparisons therefore need energy measured on each benchmark's own scored traffic. Per-token or per-query constants do not transfer between workloads, and token counts misorder energy for 18–24% of configuration pairs (score–token fronts keep 10 of the 25 models on the score–energy fronts). On that basis the paper establishes:

  1. how far configurations lie from the capability–energy front: a median of 2.6–4.8× the energy of an alternative that scores at least as well, and up to 27× (Fig. 1);
  2. the energy price of reasoning with the same weights and hardware, 3.7× over 44 pairs, and how its benefit depends on the lineage and the output budget (Fig. 3);
  3. that the output budget can reorder models, and that late tokens cost 2.0–4.9× as much as early ones (Fig. 4);
  4. the effect of FP8 and of hardware on a fixed configuration (Figs. 5–6).

The Hidden Joules (Yang & Armour, 2025) jointly measures accuracy and energy for 1,200 ImageNet models at saturating batch sizes, and it is the methodological precedent we build on (Sec. 3). In image classification each input costs one fixed forward pass, so energy per inference follows from the architecture and the batch size. LLMs differ in exactly the respect that drives findings 1–3: the computation per task is decided at inference time, through output length, reasoning mode and output budget. None of these has a counterpart in that study.

qMeter (arXiv 2508.16712) characterises quantised Llama-2-family models (7B–70B and CodeLlama-34B) on A100 and H100 with TensorRT-LLM, measuring GPU energy per token on short serving traces together with quality. Like ours, it measures energy rather than estimating it, but for one model family, one deployment choice, and models without reasoning modes, whose cost per query does not depend on a mode or a budget. Our precision result agrees with it in direction (FP8: 0.73× energy with no measurable change in score over 28 pairs). The final version will discuss both papers in Sec. 3.

Q2. Same run; sampling settings

Yes, each score and its energy come from the same attempt. The harness sends the benchmark's requests through our logging proxy while the GPUs are metered, and the responses of that attempt are scored. No score is taken from a leaderboard or from a separate run. For GPQA (2 attempts) and AIME (4), score and energy are averaged over the same attempts.

We did not rerun with common settings, because no common setting is neutral. Developers tune sampling for their models, and some reasoning models in our set (Qwen3, the DeepSeek-R1 distills) are documented to fall into endless repetition under greedy or low-temperature decoding. In a benchmark this generates up to the output cap, inflating energy and truncation for those models in particular. In practice the settings are similar: 79 of the 82 configurations in the main comparison sample at temperature 0.6–1.0 (the three Ministral-3 Instruct models use 0.05, as recommended for them), and 16 use our default of temperature 0.6 and top-p 0.95 because their developers give no recommendation. The differences we report are also large relative to sampling noise: repeated seeded attempts vary by a median of 6–8% in energy. App. A.1 lists each configuration's settings and their source.

Q3. Sustained load

Not all runs were under sustained high load; "saturated" overstates it, and the final version will describe the load as measured. We checked with the server's own metrics, sampled every 5 s during each attempt: sequences running, KV-cache use and requests waiting. The client offers each benchmark's requests together, up to 1.5× the sequence limit at a time (the whole request set of most benchmarks), so concurrency is bounded by the sequence limit, by the KV cache, or by the requests remaining. Our general comment gives how often each bound applied.

Load depends mostly on benchmark size. Large benchmarks such as LiveCodeBench and BFCL keep the 32 and 96 GB platforms at capacity most of the time. AIME, as you suspected, does not: an attempt has 30 requests, so a median of 9–15 sequences run at once, and it is at capacity at most 14% of the time except on the RTX 5090 (44%). The 8×H100 node, whose KV cache exceeds most benchmarks' request sets, is rarely at capacity; its numbers measure the energy of evaluating each benchmark on that node and should be compared within its tier only.

We did not measure equal request rates or response-time limits, and we do not claim the rankings hold there. The one other operating point we measured, one request at a time on the 8×H100 (App. C.1), raises energy per generated token 6.6× on average, so the operating point matters.

Q4. Energy window

Yes, the energy covers the work being scored. Each reported value integrates GPU power over the attempt's entire run, from harness start to the end of scoring, so ramp-up and the drain in which the longest responses finish are included. Our pipeline also computes a cropped window (first 10% and last 15% removed), which Sec. 4 describes, but no figure or number in the paper uses it; the final version will correct that sentence. Using the cropped window instead would change suite energy by a median of 3–11% per platform and leave the rankings essentially unchanged: Spearman ρ ≥ 0.99, the same Pareto fronts on four of the six platforms, and one additional front member on the other two.

Q5. Benchmark weighting

The main conclusions do not depend on equal weights; front membership is the most sensitive part. New analyses on the existing data:

  • Random weights over the seven components (2,000 draws per tier). In 95% of draws the energy ranking keeps ρ ≥ 0.95 with the equal-weight ranking on every tier, and the score ranking ρ ≥ 0.92 (≥ 0.80 on the 512 GiB tier, whose scores lie closer together). Between 64% and 85% of configurations remain dominated (5th–95th percentile), and off-front configurations still use a median of 1.9–7.9× the energy of an alternative that scores at least as well. The fronts change most: their median Jaccard similarity with the equal-weight fronts is 0.75–0.83.
  • One task at half weight, as for an application focused on it: energy ρ ≥ 0.93 and score ρ ≥ 0.83 against equal weights; 64–82% dominated.
  • Task groups: reasoning tasks alone (MATH-500, AIME, GPQA, LiveCodeBench) or the other three alone leave 71–86% of configurations dominated, and the reasoning-group fronts keep 6 of the suite fronts' 6–8 members.

These complement the checks already in App. A.4: summing energies instead of taking the geometric mean, or dropping any one component, keeps ρ ≥ 0.98.

Thank you again for your precise questions. As the discussion period ends on 18 November, we would be grateful to know whether our responses address your concerns, and we are happy to clarify any remaining point.

Reviewer_3zYpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate the accuracy-energy pareto-frontier across a range of models and Nvidida GPU platforms with evaluations on math, science, code and reasoning tasks. The authors find that highest performance models rely on large architectures and high token count, reasoning tokens is an inconsistent proxy for energy use when evaluated across models, and that reduced precision is effective in reducing energy requirements.

Critical strengths
  1. Compared to work which studies energy use or benchmark performance in isolation, this work examines energy and task performance across tasks, model size, and model families; with wide coverage on model sizes and both datacenter and workstation hardware
Critical weaknesses
  1. Methodology Clarity
    1. Batch Size and Concurrency.
      • What is the the "engine's default admission limit" (L209)? Is this the max concurrency in vLLM?
      • For smaller models, the default 256 concurrency does not saturate the device GPU memory which would underestimate the efficiency of serving.
      • When increasing the number of devices to 8x GPUs, is the batch size/allowed concurrency increased accordingly?
      • Likewise when reducing precision? This would explain why the energy savings are reduced on the 8xH100 platform (L416)
    2. Limited evaluations compared to prior work benchmarking system energy efficiency.
      • GB10 vs Server GPU are not fair comparisons as the GB10 energy evaluation does not include the LPDDR whereas the the other serving platforms include HBM energy cost
      • Compared to prior work such as ML Energy, AI Energy Score and Luccioni 24 which evaluate other modalities, evaluations are limited to text-only tasks,
  2. The findings are largely known from prior work. For example:
    • 5.2/5.3 It is common knowledge that (especially for the math/code/science reasoning tasks evaluated) large models with reasoning are utilized to achieve SoTA performance; and understood that the model size and associated test-time compute increase cost. Likewise, that the energy and FLOPS cost-per-token increases with size is unsurprising.
    • 6.1 Overthinking is known to increase cost of model inference with the potential to reduce performance (See [1] and referenced work)
    • 6.3 Traffic for the KV Cache is known to dominate execution for test-time reasoning models [2]

Ref

  1. Sui, Yang, et al. "Stop overthinking: A survey on efficient reasoning for large language models." arXiv preprint arXiv:2503.16419 (2025).
  2. Sadhukhan, Ranajoy, et al. "Kinetics: Rethinking test-time scaling law." Advances in Neural Information Processing Systems 38 (2026): 84346-84379.
Questions for authors

Questions.

  1. L160. How is grouping conducted? What is the headroom threshold?
  2. L181. What is the deterministic scoring? Is this greedy decoding?
  3. How were the benchmarks selected? Analysis across groups of tasks?
  4. The work does not measure CPU energy. What is the relative scale of power consumption on the GPU vs the CPU and host memory? CPU energy is measurable via RAPL or Intel VTune?

Nit

  1. L65 and L151. For clarity, "hybrid models" would be better referred to as "hybrid reasoning models" (with a clarification), as hybrid models is ambiguous and could also refer to hybrid model architectures (e.g. Qwen 3.5)
  2. L341. Define model lineage
  3. L430. Citation needed for pricing; TCO is not defined.
3 comments

Thank you for the detailed review. We answer in your order; section and appendix numbers refer to the PDF.

W1.1 Batch size and concurrency

What is the "engine's default admission limit" (L209)? Yes, it is vLLM's maximum concurrency: max_num_seqs, the largest number of sequences the scheduler runs at once. We kept vLLM's default on each platform: 256 on the RTX 5090, 2×RTX A6000 and GB10, and 1,024 on the GH200, RTX PRO 6000 and 8×H100 (App. A.2). The client keeps 1.5× that many requests outstanding; BFCL's harness sends at most 512 conversations at a time.

Does 256 under-fill small models? The limit was rarely what bounded concurrency: for the smallest models on the RTX 5090 (weights ≤ 5 GiB), it was reached in 3.5% of serving time, while the KV cache was full or requests were waiting in 48%. Consistent with this, an earlier round on the RTX 5090 in which the client sent at most 100 requests at a time gave per-benchmark energy within 7% of the final runs (31 configurations).

With 8 GPUs, is allowed concurrency increased? No. The 8×H100 uses vLLM's default of 1,024 sequences; it was not scaled with the number of GPUs, and a higher limit would not have changed these runs, because it was never reached. Each model runs as one tensor-parallel replica whose KV cache, about three times that of a 96 GB GPU, holds most benchmarks' entire request set, so the number of sequences in flight is set by the requests remaining (a median of 12 for AIME, which has 30 requests per attempt). The 8×H100 numbers therefore measure the energy of evaluating each benchmark on this node, which is more per token than a fully loaded node would use, and they should be compared within the 512 GiB tier only. The paper's two cross-tier comparisons, the Introduction's first example and Sec. 6.2's 1 vs 239 Wh per AIME point, mix this effect with model size; the final version will restate them on a single platform.

And when reducing precision (L416)? The limit does not change with precision. FP8 weights leave more memory for the KV cache, so more sequences run at once where the cache is the bound. On the paper's 28 pairs, FP8 raised the mean number of sequences in flight by ×1.33 on the RTX PRO 6000, ×1.10 on the RTX 5090 and ×1.04 on the 8×H100. This fits your explanation of the smaller 8×H100 saving, but it is not the whole explanation: on the GH200, the same 13 pairs as on the RTX PRO 6000 gain as much concurrency (×1.32) yet save less energy (median 0.86× against 0.69×).

W1.2 Scope of the evaluation

GB10 boundary. We agree that the GB10's GPU-rail reading is not comparable to discrete-GPU board power, which includes device memory. We also record package energy on the GB10 and module energy on the GH200 (App. A.3). At package scope the GB10 uses 1.03× the RTX PRO 6000's energy instead of 0.61×, and the GH200 0.73× instead of 0.64×, while the ranking of configurations is unchanged (Spearman ρ ≥ 0.98 with the RTX PRO 6000 at both scopes, and the same Pareto fronts). The GB10's GPU-rail energy in Sec. 7 should therefore not be read as an advantage of the platform; the final version will give both scopes there.

Text-only. This is a scope choice: every energy measurement is paired with a scored run, so the suite needs benchmarks with rule-based scoring. Scored multimodal benchmarks, with the encoders and preprocessing they add to serving, are left to future work.

W2 Findings already known

We agree that the directions of these effects are expected. What the paper adds is measured energy, which can differ from the proxies used to reason about them. Neither cited work measures energy: the survey measures efficiency as reasoning length, and Kinetics uses an analytical cost model. The final version will cite and discuss both.

  • 5.2/5.3. Size and test-time compute raise cost, but they do not identify the efficient models. Active parameters correlate only weakly with energy (ρ = 0.28–0.52), and generated tokens misorder energy for 18–24% of configuration pairs. In every tier, a non-thinking configuration comes within five points of the best score at a quarter to a sixth of its energy, and 8 of the 44 hybrid pairs lose score with thinking enabled.
  • 6.1. Overthinking is documented in tokens; we measure its energy at fixed weights and hardware. Raising the output budget beyond 32K adds 7–25 points on AIME 2026 and 6–21 on LiveCodeBench at 1.1–6.1× the energy, and it can reorder models: DeepSeek-V4-Flash ranks last of four at 32K and first at the largest cap.
  • 6.3. Kinetics leaves out the cost of reading model weights, assuming large batches amortise it. In our runs, late tokens are generated with 0.19–0.37× as many sequences in flight at nearly unchanged GPU power, which makes them 2.0–4.9× more energy-intensive: Sec. 6.3 measures the cost that Kinetics leaves out.

Questions

  1. Grouping and headroom (L160). Checkpoints are grouped by weight footprint, and the headroom threshold is a quarter of memory: a tier's platforms must hold each checkpoint's weights in at most three-quarters of their memory (the 512 GB tier runs on an 8×80 GB node), leaving at least a quarter for the KV cache and the serving engine's working memory. This is close to the split the vLLM authors report for serving, about 65% weights and 30% KV cache (Kwon et al., 2023). The margin matters for energy: the KV cache bounds the batch (Pope et al., 2023), and serving limited by memory capacity uses more energy per token (Chung et al., 2026). Once loaded, weights took at most 74% of device memory on every platform (vLLM's load-time report), and no assignment is borderline: the next checkpoint up in size would take 82% of the RTX 5090 or does not fit in 96 GB at all. The final version will state this rule in Sec. 4 and App. A.1.
  2. "Deterministic scoring" (L181). It is not greedy decoding. We meant that answers are scored by rules (exact match, executed unit tests, IFEval's programmatic checks and BFCL's checks of the function calls) rather than by an LLM judge. Decoding is sampled with each model's recommended settings, at temperature 0.6–1.0 for 79 of the 82 configurations in the main comparison, with fixed per-attempt seeds. "Rule-based scoring" is the accurate term, and the final version will use it.
  3. Benchmark selection; task groups. We chose one widely reported benchmark per capability area (instruction following, knowledge, science, mathematics, competition mathematics, coding and tool use), each scored by rules, with recent items where available (AIME 2026, LiveCodeBench) and fixed subsets where the full set was too costly to run on six platforms (MMLU-Pro, BFCL). The suite also spans very different serving workloads: the ratio of prompt to output tokens ranges from about 30 (BFCL) to 0.02 (AIME), and requests per attempt from 30 to about 4,200. For the same configuration and GPU, energy per generated token differs by a median of about 4× across benchmarks. For task groups (a new analysis), we split the suite into reasoning tasks (MATH-500, AIME, GPQA, LiveCodeBench) and the other three. In either group 71–86% of configurations are dominated, and the reasoning-group fronts keep 6 of the suite fronts' 6–8 members (the other group's fronts keep 4–5). One difference: on the 96 GiB tier, no non-thinking configuration comes within five points of the best reasoning-group score.
  4. CPU and host energy. We report CPU energy only where the CPU is part of the accelerator package. On the two superchips, the CPU and its LPDDR5X sit in the same package or module as the GPU, and on the GB10 that memory is also the GPU's, so package energy is a property of the platform (App. A.3). There, the CPU cores take 10% of the GB10's package energy and the rest of the package, including the memory, 31%; the Grace CPU, LPDDR5X and NVLink-C2C take 11% of the GH200's module energy. A discrete GPU, in contrast, can be installed in very different hosts, so the power of the host CPU and memory largely reflects the host build rather than the model or the accelerator, and adding it would make platform comparisons partly comparisons of hosts. We use accelerator energy as the common boundary and did not meter the discrete hosts (RAPL could measure their CPUs), so we cannot give the share there. Host energy still matters in deployment: Google reports total serving energy at about 1.7× active accelerator energy once host systems, idle capacity and data-centre overhead are included (Elsworth et al., 2025; Sec. 2). On the superchips the additional draw is nearly constant, so including it rescales energy without changing the ranking of configurations (see W1.2).

Nits. Agreed. The final version will define "hybrid reasoning models" at first use and "lineage" as a vendor's model line across generations (e.g., Qwen3, Qwen3.5 and Qwen3.6 form the Qwen lineage), write out "TCO" as total cost of ownership with a note that our estimate covers only accelerator purchase and electricity, and cite the price sources. All prices are for the GPUs alone. The GB10 is a complete computer, so USD 1,000 is deducted for its CPU, memory and storage. A GH200's price also covers the Grace CPU and its LPDDR5X, so we price its GPU side as the equivalent 96 GB Hopper PCIe card, the H100 NVL. Market prices are the mean of the top ten eBay listings for each card, taken in September 2026, about a week before the submission deadline, and launch prices are prices at release; for the GH200, both are the H100 NVL's.

Thank you again for your detailed review. One addition to our answer on host energy (Q4): adding a constant host draw of 100 W, 300 W or 1 kW to every run on the discrete-GPU platforms keeps each platform’s ranking of configurations at ρ ≥ 0.98 with the accelerator-only ranking and changes at most one Pareto-front member. As the discussion period ends on 18 November, we would be grateful to know whether our responses address your concerns, and we are happy to clarify any remaining point.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General: Summary of our responses

We thank the reviewers for their careful reviews. All three found the measurement campaign careful and broad. The shared concerns were the serving regime, what the study adds to prior work, and claims broader than the evidence. We answer them with the submitted paper and further analyses of the data we already have.

  1. Serving load, as measured. Each benchmark's requests are offered to vLLM together, up to 1.5× its default sequence limit at a time, and run with continuous batching. Server metrics sampled every 5 s show that the limit was reached in about 2% of serving time and the KV cache was full, or requests were waiting, in 45%; the rest of the time every remaining request was running. The share of time at capacity (KV cache at least 95% full, requests waiting, or the limit reached) is 71% on the RTX 5090, 34–59% on the 96 GB platforms and 11% on the 8×H100 node, whose KV cache exceeds most benchmarks' request sets. It depends mostly on benchmark size: LiveCodeBench, MMLU-Pro and BFCL are at capacity 51–92% of the time on the 32 and 96 GB platforms, and AIME, with 30 requests per attempt, at most 14% outside the RTX 5090. "Saturated" overstates this regime, and our claims are limited to it.
  2. Energy covers the whole run. Every reported energy value integrates GPU power over the attempt's entire run, including ramp-up and the final drain. Sec. 4 describes a cropped window that our pipeline also computes but that no figure uses; that description is inaccurate. Using the cropped window would change suite energy by a median of 3–11% per platform, with rank correlations of at least 0.99.
  3. Narrower claims. The evidence covers 86 open-weight checkpoints served with one pinned stack, not current LLMs in general. The hardware result holds within the 96 GiB tier, where it was measured, and energy should be compared within a tier. On the two superchips, package-scope energy changes the absolute ratios but not the rankings.
  4. Sensitivity analyses on existing data: model roster, benchmark weighting and task groups, FP8 concurrency, and host CPU and memory energy on the two superchips.
  5. Positioning: how this work differs from Kinetics, the overthinking survey of Sui et al., qMeter and The Hidden Joules.
  6. Clarifications: tier assignment, rule-based scoring, sampling settings, benchmark selection and price sources.
AuthorsOpenReview ↗

Summary of the discussion

As the discussion closes, we briefly summarise it for the AC and the reviewers.

Final version: The claims will be limited to what was measured: 86 open-weight checkpoints served with one pinned stack, under a batched evaluation workload rather than production serving in general (the title will say “Batched Serving”). The hardware conclusions will be limited to the 96 GiB tier, and the 512 GiB results will be stated as depending on the Qwen lineage. Sec. 4 will describe the energy measurement as performed, integrated over each complete attempt. Reviewer 8f9v considered the main concerns of their review substantially addressed.

What the measurements add: Energy is measured on each benchmark’s own scored traffic. Energy per generated token differs by a median of about 4× across benchmarks for the same configuration and GPU, and generated-token counts misorder 18–24% of configuration pairs, so token-based estimates do not substitute for measurement. About three-quarters of configurations use more energy than an alternative that scores at least as well, by a median of 2.6–4.8× per tier, and enabling thinking costs 3.7× on average for a benefit that depends on the model family and the output budget. These effects persist under the sensitivity analyses added during the discussion (roster, benchmark weighting, energy window and measurement boundary).

Why it matters. Electricity is becoming a constraint on AI deployment. The IEA projects data-centre electricity use to more than double by 2030, to about 945 TWh, and estimates that about 20% of planned data-centre projects risk delays from grid constraints (IEA, Energy and AI, 2025). In Northern Virginia, the largest data-centre market, Dominion Energy said in 2024 that large new data centres could wait up to seven years for a grid connection; xAI powered its Memphis data centre with a fleet of mobile gas turbines before its grid connection was complete; and local governments have begun to pause new data-centre construction, as Boise and San Mateo County did with temporary moratoria in November 2026. For a required capability, the choice of model, reasoning mode, output budget and weight format is a lever that users and providers control today, and our measurements show how large it is.

We thank the reviewers for their time and would welcome any final comments before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.