- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that energy per token is the wrong efficiency metric for on-device LLMs and proposes PACE, an evaluation and configuration-selection framework (not a new inference engine). The goal is to find the lowest-energy configuration (deployment, execution mode, token budget) that still produces enough correct answers before a deadline. A motivating measurement shows two runtimes (QNN and llm.npu) differing by only ~1% in energy per token, yet 31.2% vs. 63.0% correct within 10 s and 2.14x apart in energy per timely correct answer. From measurements across several small models, runtimes and two Snapdragon 8 Elite phones (GSM8K, MATH-500, HammerBench), the paper reports three observations: speed matters only when it moves correct answers across the deadline, more tokens matter only when they improve correctness, and final accuracy is a hard feasibility ceiling. PACE-SELECT then uses a calibration set to keep configurations meeting a (deadline, accuracy) target and picks the lowest mean request energy; in a replay over 20 splits it meets the target 19/20 times vs. 5/20 for minimum-J/token selection, with up to 38.6% less request energy than the fastest mode.
Critical strengths
- Clear motivation and presentation. The introduction and motivation are clear and easy to follow, and the point that cheap tokens do not necessarily produce useful requests is well illustrated.
- Careful measurement with actionable numbers. The measurement study is careful and reproducible in spirit (external power meter, repeated runs, paired comparisons), and the feasibility-first framing yields concrete numbers practitioners can act on, e.g., a lower-power mode meeting the same target with 38.6% less energy.
Critical weaknesses
- Limited novelty. The three observations come directly from measurements without providing non-trivial insight, and the selection rule and conclusions are fairly straightforward. Much of the contribution lies in the measurements; the method follows a typical systems design pattern (profile configurations into tables, then select) applied to a narrow problem.
- Oversimplified utility and no reliability guarantees. Treating every correct answer that arrives after the deadline as a failure is an oversimplified notion of task utility. The method depends on hyperparameters such as the size of the calibration set, but the paper gives neither experiments nor theory on how to choose them, which matters for an ML venue.
Questions for authors
- How sensitive is PACE-SELECT to the calibration-set size and other hyperparameters, and is there guidance or a guarantee for choosing them?
- How would the conclusions change under a softer utility than a hard deadline (e.g., utility that decays with latency)?
4 comments
W2 (Part 1) & Q1. Sensitivity to calibration size and finite-sample guarantees
"How sensitive is PACE-SELECT to the calibration-set size and other hyperparameters, and is there guidance or a guarantee for choosing them?"
Screening candidates with a one-sided Clopper–Pearson lower confidence bound guarantees that the probability of picking an infeasible configuration is at most in finite samples, with sample size scaling as .
- Finite-sample risk bound in App. F.3: For candidates calibrated on independent requests and risk level , screening via bounds family-wise infeasible selection by a union bound:
- Practical sample-size guide: A feasible candidate with margin passes once . For example, , , and needs , whereas a borderline candidate with safely results in no selection.
- Empirical results in Table 20 and Fig. 11b:
- 5%–40% calibration sweep across 23 configurations: Out of 1,200 decisions across 12 targets, 500 have no truly feasible candidate, so at most 700 can be met and abstaining safely triggers fallback rather than a silent miss. At a 20% share, setting cuts missed selections by 95%, from 107 down to 5 out of 1,200 splits, while tuning or smoothly trades risk for coverage on borderline targets.
- Small sample counts on OnePlus Ace 6 and Redmi K90: Unadjusted misses drop from 20/120 at 16 questions to 0/120 at 263 questions, while the confidence bound gives 0/120 misses at every sample size.
- Unseen requests on OnePlus Ace7 Pro: 32 calibration questions achieve 0 misses and 6/6 agreement with the best fixed test choice, compared with 4.35–5.09 misses out of 6 for unscreened baselines.
[Updated in Sec. 6, Eq. 4 & App. F.3, Table 20, Fig. 11b]
W2 (Part 2) & Q2. Smooth latency-decay utilities instead of a hard deadline
"How would the conclusions change under a softer utility than a hard deadline (e.g., utility that decays with latency)?"
The timely-accuracy curve completely determines mean utility for every smooth decreasing latency penalty, and minimum J/token remains suboptimal across all 8/8 exponential-decay settings.
- Exact relation in App. F.3: For any utility that decreases with latency toward , writing gives , so PACE-SELECT applies directly by screening .
- Exponential-decay results across 23 configurations in Fig. 11c: Under for , minimum J/token fails in all 8/8 task and settings, while increasing on MATH smoothly shifts the best choice along PACE's frontier from
llm.npuBalanced at 512 toNo capand then to 6-threadT-MACwithNo cap.
[Updated in Sec. 2 & App. F.3, Fig. 11c]
Note on Appendix A: Since you highlighted our measurement care, we also share a wording correction in Apps. A.2 and A.4 detailed under Reviewer kxJU Q1: three-repeat averaging applied to our stage and budget controls in Apps. B.1 and D.1 while full sweeps ran one pass, and energy integrates unplugged Android battery power every ~0.5 s following llm.npu (Xu et al., 2024), HeteroLLM (Chen et al., 2025), and MNN-AECS (Huang et al., 2025). Unplugged checks kept below 42 °C across 384 OnePlus Ace 6 runs in Table 3 & App. F.3, Table 19, Panel B and 150 OnePlus Ace7 Pro 3-repeat runs in App. F.1 reproduce our 38.6% saving at 38.2% [36.7%, 39.7%] with 5.3%–7.0% median variation.
We hope these finite-sample guarantees, smooth-utility results, and unseen-request studies merit reconsidering your rating!
Thanks for the rebuttal. It has addressed my concerns. I've updated my score accordingly.
Thank you for your support and feedback
Dear Reviewer ooo7,
Thank you very much for reading our response and raising your score! We truly appreciate your constructive feedback and the time you dedicated to reviewing our work. We will make sure to incorporate all the new results, discussions, and clarifications into the final version of the paper.
Best regards,
The Authors
Dear Reviewer ooo7,
Thank you for your thoughtful review and for noting our clear motivation, careful experiments, and practical takeaways. Below, we address your questions on insights beyond table lookup, calibration sample-size guarantees, and smooth latency utilities one by one.
W1. Non-trivial findings and low-cost calibration beyond brute-force table lookup
While PACE-SELECT keeps its selection rule simple on purpose, (1) our request-level measurements uncover five counter-intuitive hardware and quantization reversals where common rules fail, and (2) exact output consistency reduces accuracy profiling from O(R⋅M⋅B⋅P) to O(R), separating PACE from brute-force grid profiling.
Five reversals where common systems rules fail:
llm.npuat 0.0268 J/tok differ by 1.01% in J/token yet reach 31.2% vs. 63.0% timely accuracy at 10 s—a 2.14$\times$ gap in energy per timely correct answer in Fig. 1 & App. B.3.(8, 6)threads has slower mean TTFT (5.00 vs. 4.78 s) yet finishes 21.0% faster than(8, 8)because moving decode from 6 to 8 threads slows decoding in 35 of 36 paired runs in Sec. 3, Fig. 4b & App. B.1.No capsends 100% of extra tokens to answers that stay wrong, whereas on MATH wrong traces burn 65.3%–79.0% of recorded energy whileNo caprecovers 32–62 truncated answers in Table 1 & App. B.2, Fig. 8.llm.npuBurst at 1.56 s runs 20.2% faster than QNN Balanced at 1.95 s, yet is 8.4 pp less accurate and 47% higher energy due to falseQLinearLPBQtool calls.Why calibration is O(R) rather than brute-force O(R⋅M⋅B⋅P) lookup:
Brute-force profiling measures every model artifact R, mode M, budget B, and same-SoC phone P separately. We show that correctness Yic is 100% identical across M modes and P same-SoC phones—matching on all 15 shared Redmi K90 and OnePlus Ace 6 configurations and meeting 10, 20, and 30 s / 60% in 17/20, 20/20, and 20/20 cross-phone splits with zero re-labeling—while shorter-budget runs are exact prefixes of unclipped runs in App. D.1, Table 12. Thus one unclipped pass per artifact r fixes accuracy across all modes, budgets, and same-SoC phones, while target settings need only small timing/energy samples (k=16 in App. C.1, Table 11). Even on OnePlus Ace7 Pro with Snapdragon 8 Elite Gen 5, calibrating 9 configurations on just 32 GSM8K questions matches the best fixed test choice at all 6/6 targets and saves 33.9%
[31.4%, 36.3%]energy over Fastest [Sec. 6, Table 3; Apps. C.1, F.1–F.2].