- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the trade-off between the LLM benchmark capability and the directly measured inference energy under high-unutilization serving. This paper evaluates 86 open-weight checkpoints, i.e., 134 serving configurations, 7 benchmarks, and 6 hardware platforms. The analysis includes score-energy Pareto frontiers, controlled comparisons of reasoning vs. non-reasoning modes, output-budget sensitivity, reduced-precision weights, and hardware-dependent energy measurements.
The paper's main empirical findings are that many evaluated configurations are Pareto-dominated, reasoning often incurs a substantial energy premium, generated-token count is an imperfect proxy for physical energy, FPB reduces energy with little aggregate score change, and model efficiency rankings appear relatively stable across the hardware platforms evaluated in the 96-GB tier.
The work is carefully executed and potentially useful as a measure study. However, I am not convinced the current evidence is sufficiently representative or generalizable to support the breadth of the paper's claims about contemporary LLM serving. In more detail, the effective diversity of the model roster is narrower than the headline checkpoint count suggests, and the hardware conclusions are substantially broader than the evaluated hardware and serving regimes justify.
Critical strengths
Careful and unusually broad direct measurement campaign Good robustness and sensitivity analysis within the evaluated setup
Critical weaknesses
a) The model roster is broad in count (i.e., 86 checkpoints and 134 serving, etc.), but not sufficiently representative of the contemporary frontier serving landscape. In particular, many study rows are precision variants, reasoning-mode variants, or reasoning-effort variants of the same underlying model families. The study is also restricted to open-weight checkpoints that are compatible with the authors’ pinned serving stack. These points matter as the main Pareto conclusions are conditional on the candidate set. The authors also note that the roster is not a random sample of the model ecosystem. I therefore do not believe that the current study supports broad conclusions about "current LLMS" as strongly as the authors highlight.
b) The hardware conclusions substantially exceed their evaluation and serving space: they claim that changing hardware largely rescales absolute energy while preserving relative model efficiency. Nevertheless, the like-for-like hardware comparison is concentrated in the 96-GiB tier with four hardware platforms, while the 32- as well as 512-GiB are presented by RTX 5090 and 8×H100 node, respectively. In my opinion, this is not sufficient evidence to generalize hardware-invariant model rankings across the full model-size spectrum. In practice, small single-GPU workloads and large multi-GPU/MoE deployments can exhibit qualitatively different bottlenecks. Moreover, the serving regime is narrow. The authors use saturated continuous batching with platform-dependent admission limits. This presents a high-utilization, throughput-oriented operating point, not production serving in general. The evaluation scenarios (e.g., low-QPS serving, bursty traffic, latency-constrained serving, etc.) may produce different energy rankings. In fact, the C1 appendix shows that energy per generated token can increase severalfold and throughput can collapse relative to the saturated regime, demonstrating that utilization is not a minor detail. Finally, the compared systems differ in more than accelerator architecture, namely host CPU, memory technology, interconnect, tensor/expert parallelism, admission limit, and so on. The primary measurement boundary also differs across device classes (e.g., discrete-GPU board power, GH200 GPU+HBM, GB10 GPU rail). Appendix A and D show that the apparent GB10 advantage changes substantially when a wider package-level boundary is considered. This makes the authors' conclusion that hardware "largely rescales energy" weaker than the paper suggests.
Questions for authors
a) How sensitive are the reported Pareto-front memberships, dominance ratios, and reasoning-energy conclusions to the compositions of the model roster?
b) The hardware-invariance claim is supported by the 96-GiB tier. What evidence supports extrapolating this result to small single-GPU models and large multi-GPU/MoE models?
c) Do Pareto membership and relative model rankings remain stable across realistic utilization/concurrency levels, rather than only under saturated continuous batching?
d) How much do the cross-platform rankings change when all systems are compared under the broadest practically comparable measurement boundary?
6 comments
(c) Utilisation and concurrency
No, we have not shown this, and we do not claim it. The paper's description of the regime also needs correcting: runs were not uniformly saturated. Each benchmark's requests are offered together, up to 1.5× vLLM's default sequence limit at a time, and run with continuous batching. Concurrency is therefore bounded by the sequence limit, the KV cache or the requests remaining (our general comment gives how often each applied), and small benchmarks (AIME has 30 requests per attempt) and the 8×H100 node, whose KV cache exceeds most request sets, run well below capacity. What we measure is a batch evaluation workload, not production serving in general. In the final version we will remove "saturated", change the title's "…under Production Serving" to "…under Batched Serving", and state in the limitations: "All main experiments offer each benchmark's requests to the server together, up to 1.5× its sequence limit at a time, so utilisation depends on the benchmark's size and the platform's memory; it is lowest on the 8×H100 node, whose KV cache exceeds most benchmarks' request sets. The energy values and rankings should not be assumed to hold at other loads, such as low request rates, bursty traffic or response-time targets."
The one same-configuration contrast we have with full benchmark traffic is mild. An earlier round on the RTX 5090 with the client limited to 100 concurrent requests (instead of 384) gives a suite-energy rank correlation of 0.82 with the final runs over 16 configurations, and the Pareto fronts share 6 of 7 members. We do not take this as evidence for other loads. App. C.1 shows how much utilisation can matter: energy per token is 6.6× higher at batch size 1.
(d) Broadest comparable boundary
The rankings do not change, but the absolute ratios do. On the two superchips we also record package (GB10) and module (GH200) energy, which add the CPU and the LPDDR5X memory. Against the RTX PRO 6000's board power, Spearman ρ is at least 0.98 at both GPU and package scope, and each platform's Pareto front is the same at both scopes, because the additional draw is nearly constant. Absolute energy moves substantially: the GB10 uses 0.61× the RTX PRO 6000's energy at GPU-rail scope and 1.03× at package scope, and the GH200 0.64× and 0.73×. Measured like a discrete GPU, with its memory but without its CPU, the GB10 would fall between these two values. A full system boundary would also need the discrete-GPU hosts' CPU and memory, which we did not meter. The final version will give both scopes in Sec. 7 and limit the hardware sentence in the abstract accordingly.
We hope the narrowed claims and the added analyses address your concerns, and we are happy to answer further questions.
The authors’ response considerably addresses my concerns about overgeneralization by narrowing the claims and providing useful roster sensitivity analyses. However, some points still require clarification. In particular, the energy-measurement procedure, including the calculation of energy per attempt after trimming the measurement interval, needs to be reconciled with the acknowledgment that runs were not uniformly saturated. The dependence of the 512 GiB dominance results on Qwen should also be stated explicitly. Finally, stable model rankings should be distinguished from comparisons of hardware efficiency, since a common full-system measurement boundary remains unavailable. Overall, the rebuttal is constructive and reduces the severity of my original concerns, but I would appreciate these clarifications before considering them fully resolved.
Thank you for the valuable review. We agree with its main point: several claims were broader than the evidence. Below we state the narrower claims the evidence supports, which the final version will adopt, and add sensitivity analyses on our existing data. Answers follow your questions; section and appendix numbers refer to the submitted PDF.
(a) Roster composition
We agree that the roster is not a sample of the ecosystem. Our claims cover 86 open-weight checkpoints served with one pinned stack, not "current LLMs" in general, and the final version will say so (abstract, Sec. 1 and Sec. 9). On duplication: the main frontier comparison uses one weight format per tier (BF16, BF16 and FP8; 82 configurations), so precision twins enter only the controlled comparison in Sec. 7. Reasoning modes are separate configurations by design, because a deployer chooses the mode and it changes energy 3.7× on average.
Sensitivity to the roster (new analyses of the Fig. 1 panels):
The qualitative findings and the reasoning and precision effects are therefore robust to the roster, whereas front membership and the size of the dominance gaps are properties of the candidate set, as you note. The final version will say so where those numbers are reported (Sec. 5).
(b) Hardware claims
Nothing in our data supports extrapolating the hardware result beyond the 96 GiB tier: no configuration was served in two tiers. The final version will state the result only where it was measured (abstract, Sec. 7, Sec. 9): on four platforms serving the same 96 GiB-tier configurations, hardware rescales energy 0.5–2.2× while preserving their ranking (ρ = 0.99). Within that tier, the agreement does not rest on one kind of model or system. Spearman ρ with the RTX PRO 6000 is at least 0.90 within every subgroup we checked: dense and MoE models, thinking and non-thinking configurations, and each weight format. It holds on a two-GPU tensor-parallel system (2×RTX A6000) as well as on single GPUs and on two superchips whose memory bandwidth differs 15×. We agree that the platforms differ in more than the accelerator: this is a comparison between complete serving systems as configured (Table 6), and small single-GPU and large multi-GPU MoE deployments may behave differently. The final version will say both in Sec. 7 and Sec. 8.