- Rating
- 2
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies energy-optimal test-time scaling for LLM reasoning. It argues that the common proxies for test-time compute allocation—FLOPs and latency—can diverge from energy-optimal allocation because GPU power draw is not constant: increasing parallel samples raises instantaneous power, whereas increasing reasoning length has a comparatively small effect on power. The paper proposes a joint sequential–parallel scaling model for task accuracy, combines it with measured latency, steady-state power, and energy across Qwen3 models and several GPUs, and characterizes how energy-optimal configurations tend to favor longer reasoning chains at lower parallelism, smaller dense models, and sparse mixture-of-experts models. The claimed contributions are the joint accuracy scaling model, an accuracy–latency–energy benchmark, and a characterization of energy-optimal test-time scaling across models, tasks, and hardware platforms.
Critical strengths
Strength 1: The problem is practically important and the central observation is consequential. Test-time scaling is increasingly used in reasoning deployments, yet FLOPs and latency are often treated as sufficient proxies for cost. The paper’s argument that transient power behavior can make energy-optimal allocation systematically different from latency-optimal allocation is directly relevant to serving and infrastructure decisions. This is the strongest reason to consider the paper: it identifies a real mismatch between commonly used efficiency proxies and actual energy use.
Strength 2: The joint sequential–parallel accuracy model provides a useful planning abstraction. Modeling pass@k coverage as a function of both reasoning length and number of samples allows iso-accuracy comparisons across test-time configurations, which is more informative than comparing methods at a single fixed budget. Even if the model is empirical, it gives a concrete way to reason about trade-offs among accuracy, latency, and energy.
Critical weaknesses
Weakness 1: The provided manuscript is materially corrupted, and key sections or figures are missing, which prevents proper evaluation of the method and experimental setup. Page 1 contains only line numbers, Page 3 and Page 5 are largely repeated indices, and the paper’s opening framing, part of Section 2, and at least some figures or surrounding text are not readable in the submitted file. This is not a cosmetic issue: the missing material likely contains the formal problem setup, measurement protocol details, and possibly Figure 2 or Figure 4, which are needed to assess how power and energy are measured and how the scaling model is fitted. For a paper whose central claims depend on empirical power measurements and configuration comparisons, this is a serious evaluability failure and supports rejection in the current form.
Weakness 2: The core claims rest on empirical power measurements and an empirically fitted accuracy model over a limited set of models, tasks, GPUs, and serving configurations, without sufficient evidence that the conclusions generalize or are causally attributable to the stated mechanisms. The experiments center on the Qwen3 family, vLLM serving, a maximum of 16 parallel samples, and a few reasoning/code tasks. The joint scaling model is a curve fit with , but fit quality does not establish that the model extrapolates to other regions, other model families, longer contexts, or multi-turn agent workloads. Power draw is measured with DCGM under a specific serving stack; the paper does not adequately separate the effects of model architecture, batch scheduling, KV-cache behavior, and implementation details from the claimed transient-power mechanism. The headline energy savings (up to 60.1% vs latency-optimal and 55.8% vs FLOPs-optimal) are therefore suggestive but not yet decisive. For a paper whose contribution is primarily empirical characterization, this limits the strength of the conclusions.
Questions for authors
-
The submitted manuscript has corrupted pages: Page 1 is only line numbers, and Pages 3 and 5 are largely repeated indices. Can you provide a complete, readable version? Without the missing setup and figure material, the measurement protocol and several central claims cannot be assessed.
-
How many repeated measurements were used for steady-state power, latency, and energy for each configuration? Can you report variance across runs and clarify whether the energy-optimal rankings are stable under run-to-run variation?
-
The paper limits parallel samples to 16 because of KV-cache throttling. How would the energy-optimal versus latency-optimal divergence change at higher concurrency, where the transient power regime may shrink or disappear?
-
The joint scaling model is an empirical fit. How well does it extrapolate beyond the measured grid of and , and how sensitive are the iso-accuracy contours and optimal configurations to model misspecification?
-
The power measurements depend on vLLM and DCGM. How much of the observed power scaling is attributable to the serving stack, batching policy, and KV-cache implementation rather than to the model or test-time configuration itself?
2 comments
Thank you again for your feedback! As the discussion period ends today, we would be grateful to know whether our responses address your concerns, and we are happy to clarify any remaining point.
We would like to thank the reviewer for their feedback. We appreciate that they found our work practical, consequential, and useful for understanding tradeoffs in test-time computing. Below we address their concerns:
Submission Formatting Issues. We have checked that the pdf for our submission renders properly and no issues with the file format are reported by any of our other reviewers. Please let us know if we can be of any assistance.
Measurement Methodology. As stated in Appendix A, batches of k requests are issued to the inference engine and repeated 8 times (or 4 times for the larger 14B and 32B dense models). Over the duration of the run and across all models, the median coefficient of variation in steady state power draw is 6.3%, and the P90 variation is 12.5%. Additionally, we find there is minimal drift in the power draw over the duration of the run, with only an increase of 0.5% in steady state power draw for the median run, and +1.4% for P90.
In regards to variation in latency per wave, we find minimal variation. The coefficient of variations in wave processing time is Qwen3-32B 0.9%, 14B 1.8%, 8B 3.4%, 4B 4.6%, 30B-A3B 5.6%, 1.7B 6.5%.
Evaluations at Higher Concurrency. In additional experiments on the H100 NVL, we consider a setting in which multiple parallel requests are processed concurrently in addition to pass@k parallel sampling. While increased concurrency reduces the difference between latency and energy-optimal scaling, we find that even when serving up to 8 batched requests with up to 128 concurrent sequences, energy-optimal configurations differ from latency-optimal configurations at 29% of iso-accuracy levels.
Note: In these experiments, we conduct our evaluations on H100 NVLs due to limited compute availability. However, as we note in Section 3.4, power proportional scaling is limited on this platform due to power capping. We expect that the difference between energy-optimal and latency-optimal serving would remain more pronounced on the uncapped platform.
Generalizing Performance of the Joint Scaling Model. Below we provide the held-out accuracy of the joint scaling model (pooled over models) and find that with the exception of AIME leave-budget-out fitting, the RMSE error of predicted pass@k is less than 0.05:
Limited Model Families. We would like to highlight that while our paper primarily focuses on Qwen3 for its main analysis. We find that the distinction between energy-optimal and latency-optimal scaling is pronounced for other model architectures as well. Concretely, on Qwen 3.5 which utilizes a different Gated Delta Net based architecture as opposed to Group Query Attention similarly observes distinct energy-optimal scaling.
Variability Under Different Measurement and Serving Configurations. We note that DCGM and vLLM are standard tools for inference energy measurement and serving, used across recent benchmarks and characterization studies (Chung et al., 2026; Patel et al., 2024; Jin et al., 2025); DCGM reads the same board-power sensor as NVML/nvidia-smi while additionally exposing the hardware counters we report in Appendix F.
To validate the effectiveness of our evaluations, we also consider SGLang as another commonly used inference serving engine. Across all Qwen3 model sizes and sequence lengths, we find that the difference in median energy consumption was within 1%, and the difference in median latency differs by no more than 5%.
Additionally, we compare the efficiency of joint scaling using the default bf16 precision with reduced precision fp8 KV Cache. We find that distinct energy-optimal scaling persists.