acceptodds
Under review as a conference paper at ICLR 2027

PACE: Rethinking On-Device LLM Efficiency From Token Throughput to Timely Correctness

Abstract

On-device LLM systems are often optimized for token throughput or energy per token, but these metrics do not capture whether a request is answered correctly and on time. We present PACE, a task-aware framework for evaluating on-device LLM efficiency through timely accuracy (i.e., the fraction of requests answered correctly before a deadline) and total request energy. PACE studies how deployment, execution mode, and generation budget jointly affect task completion. Across three LLMs and three deployment paths on two real mobile phones, we find that similar token-level efficiency can hide large differences in task-level cost: two deployments with only a 1.01% difference in energy per token differ by 2.14× in energy per timely correct answer at a 10-second deadline. We further find that faster execution matters only when it helps correct responses meet the deadline, while larger generation budgets matter only when the extra tokens improve answer correctness. We then introduce PACE-SELECT, a simple calibration-based rule that chooses the lowest-energy configuration satisfying a target timely accuracy. Our results show that the most efficient configuration depends on the task requirement and deadline, and that on-device LLM efficiency should be evaluated by timely task completion rather than token efficiency alone.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
On-device Large Language Models, Task-level Efficiency, Energy-efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ooo7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that energy per token is the wrong efficiency metric for on-device LLMs and proposes PACE, an evaluation and configuration-selection framework (not a new inference engine). The goal is to find the lowest-energy configuration (deployment, execution mode, token budget) that still produces enough correct answers before a deadline. A motivating measurement shows two runtimes (QNN and llm.npu) differing by only ~1% in energy per token, yet 31.2% vs. 63.0% correct within 10 s and 2.14x apart in energy per timely correct answer. From measurements across several small models, runtimes and two Snapdragon 8 Elite phones (GSM8K, MATH-500, HammerBench), the paper reports three observations: speed matters only when it moves correct answers across the deadline, more tokens matter only when they improve correctness, and final accuracy is a hard feasibility ceiling. PACE-SELECT then uses a calibration set to keep configurations meeting a (deadline, accuracy) target and picks the lowest mean request energy; in a replay over 20 splits it meets the target 19/20 times vs. 5/20 for minimum-J/token selection, with up to 38.6% less request energy than the fastest mode.

Critical strengths
  1. Clear motivation and presentation. The introduction and motivation are clear and easy to follow, and the point that cheap tokens do not necessarily produce useful requests is well illustrated.
  2. Careful measurement with actionable numbers. The measurement study is careful and reproducible in spirit (external power meter, repeated runs, paired comparisons), and the feasibility-first framing yields concrete numbers practitioners can act on, e.g., a lower-power mode meeting the same target with 38.6% less energy.
Critical weaknesses
  1. Limited novelty. The three observations come directly from measurements without providing non-trivial insight, and the selection rule and conclusions are fairly straightforward. Much of the contribution lies in the measurements; the method follows a typical systems design pattern (profile configurations into tables, then select) applied to a narrow problem.
  2. Oversimplified utility and no reliability guarantees. Treating every correct answer that arrives after the deadline as a failure is an oversimplified notion of task utility. The method depends on hyperparameters such as the size of the calibration set, but the paper gives neither experiments nor theory on how to choose them, which matters for an ML venue.
Questions for authors
  1. How sensitive is PACE-SELECT to the calibration-set size and other hyperparameters, and is there guidance or a guarantee for choosing them?
  2. How would the conclusions change under a softer utility than a hard deadline (e.g., utility that decays with latency)?
4 comments

Dear Reviewer ooo7,

Thank you for your thoughtful review and for noting our clear motivation, careful experiments, and practical takeaways. Below, we address your questions on insights beyond table lookup, calibration sample-size guarantees, and smooth latency utilities one by one.

W1. Non-trivial findings and low-cost calibration beyond brute-force table lookup

"The three observations come directly from measurements without providing non-trivial insight, and the selection rule and conclusions are fairly straightforward. Much of the contribution lies in the measurements; the method follows a typical systems design pattern (profile configurations into tables, then select) applied to a narrow problem."

While PACE-SELECT keeps its selection rule simple on purpose, (1) our request-level measurements uncover five counter-intuitive hardware and quantization reversals where common rules fail, and (2) exact output consistency reduces accuracy profiling from O(R⋅M⋅B⋅P)O(R \cdot M \cdot B \cdot P) to O(R)O(R), separating PACE from brute-force grid profiling.

  1. Five reversals where common systems rules fail:

    Common HeuristicWhat Request-Level Measurement Reveals
    "Equal J/token means equal efficiency"QNN at 0.0266 J/tok and llm.npu at 0.0268 J/tok differ by 1.01% in J/token yet reach 31.2% vs. 63.0% timely accuracy at 10 s—a 2.14$\times$ gap in energy per timely correct answer in Fig. 1 & App. B.3.
    "Lowest power or J/token saves battery"QNN Low-power cuts power by 46.9% yet doubles runtime by +105.9%, raising request energy by +9.1% on GSM8K and +18.7% on HammerBench. On OnePlus Ace7 Pro with Snapdragon 8 Elite Gen 5 in App. F.1, Table 17, it has the lowest J/token across all 9 configs at 0.0669 yet misses 60% at both 20 s and 30 s.
    "Lower TTFT means faster completion"On CPU T-MAC, (8, 6) threads has slower mean TTFT (5.00 vs. 4.78 s) yet finishes 21.0% faster than (8, 8) because moving decode from 6 to 8 threads slows decoding in 35 of 36 paired runs in Sec. 3, Fig. 4b & App. B.1.
    "Larger token budget means higher accuracy"Raising GSM8K's budget from 512 to No cap sends 100% of extra tokens to answers that stay wrong, whereas on MATH wrong traces burn 65.3%–79.0% of recorded energy while No cap recovers 32–62 truncated answers in Table 1 & App. B.2, Fig. 8.
    "Fastest mode gives best deadline accuracy"Across 2,620 unplugged HammerBench tool-calling runs on OnePlus Ace 6 in App. F.2, Table 18, llm.npu Burst at 1.56 s runs 20.2% faster than QNN Balanced at 1.95 s, yet is 8.4 pp less accurate and 47% higher energy due to false QLinearLPBQ tool calls.
  2. Why calibration is O(R)O(R) rather than brute-force O(R⋅M⋅B⋅P)O(R \cdot M \cdot B \cdot P) lookup:
    Brute-force profiling measures every model artifact RR, mode MM, budget BB, and same-SoC phone PP separately. We show that correctness YicY_{ic} is 100% identical across MM modes and PP same-SoC phones—matching on all 15 shared Redmi K90 and OnePlus Ace 6 configurations and meeting 10, 20, and 30 s / 60% in 17/20, 20/20, and 20/20 cross-phone splits with zero re-labeling—while shorter-budget runs are exact prefixes of unclipped runs in App. D.1, Table 12. Thus one unclipped pass per artifact rr fixes accuracy across all modes, budgets, and same-SoC phones, while target settings need only small timing/energy samples (k=16k=16 in App. C.1, Table 11). Even on OnePlus Ace7 Pro with Snapdragon 8 Elite Gen 5, calibrating 9 configurations on just 32 GSM8K questions matches the best fixed test choice at all 6/6 targets and saves 33.9% [31.4%, 36.3%] energy over Fastest [Sec. 6, Table 3; Apps. C.1, F.1–F.2].

W2 (Part 1) & Q1. Sensitivity to calibration size and finite-sample guarantees

"How sensitive is PACE-SELECT to the calibration-set size and other hyperparameters, and is there guidance or a guarantee for choosing them?"

Screening candidates with a one-sided Clopper–Pearson lower confidence bound guarantees that the probability of picking an infeasible configuration is at most δ\delta in finite samples, with sample size scaling as ncal≍2ln⁡(m/δ)Δ2n_{\mathrm{cal}} \asymp \frac{2\ln(m/\delta)}{\Delta^2}.

  1. Finite-sample risk bound in App. F.3: For mm candidates calibrated on ncaln_{\mathrm{cal}} independent requests and risk level δ∈(0,1)\delta \in (0, 1), screening via C^LCB(D,q)={c:LCP(ncalS^c(D); ncal, δ/m)≥q}\widehat{\mathcal{C}}_{\mathrm{LCB}}(D, q) = \{c : L_{\mathrm{CP}}(n_{\mathrm{cal}}\widehat S_c(D);\, n_{\mathrm{cal}},\, \delta/m) \ge q\} bounds family-wise infeasible selection by a union bound: Pr⁡ ⁣(∃ c∈C^LCB(D,q):Sc(D)<q)  ≤  ∑c: Sc(D)<qPr⁡ ⁣(LCP(kc;ncal,δ/m)≥q)  ≤  m⋅δm  =  δ.\Pr\!\left(\exists\, c \in \widehat{\mathcal{C}}_{\mathrm{LCB}}(D, q) : S_c(D) < q\right) \;\le\; \sum_{c:\, S_c(D) < q} \Pr\!\left(L_{\mathrm{CP}}(k_c; n_{\mathrm{cal}}, \delta/m) \ge q\right) \;\le\; m \cdot \frac{\delta}{m} \;=\; \delta.
  2. Practical sample-size guide: A feasible candidate with margin Δ=Sc(D)−q>0\Delta = S_c(D) - q > 0 passes once ncal≍2ln⁡(m/δ)Δ2n_{\mathrm{cal}} \asymp \frac{2\ln(m/\delta)}{\Delta^2}. For example, Δ=0.10\Delta = 0.10, m=9m = 9, and δ=0.10\delta = 0.10 needs ncal≈225n_{\mathrm{cal}} \approx 225, whereas a borderline candidate with Δ=0.0027\Delta = 0.0027 safely results in no selection.
  3. Empirical results in Table 20 and Fig. 11b:
    • 5%–40% calibration sweep across 23 configurations: Out of 1,200 decisions across 12 targets, 500 have no truly feasible candidate, so at most 700 can be met and abstaining safely triggers fallback rather than a silent miss. At a 20% share, setting δ=0.10\delta = 0.10 cuts missed selections by 95%, from 107 down to 5 out of 1,200 splits, while tuning δ\delta or ncaln_{\mathrm{cal}} smoothly trades risk for coverage on borderline targets.
    • Small sample counts ncal∈{16,…,263}n_{\mathrm{cal}} \in \{16,\dots,263\} on OnePlus Ace 6 and Redmi K90: Unadjusted misses drop from 20/120 at 16 questions to 0/120 at 263 questions, while the confidence bound gives 0/120 misses at every sample size.
    • Unseen requests on OnePlus Ace7 Pro: 32 calibration questions achieve 0 misses and 6/6 agreement with the best fixed test choice, compared with 4.35–5.09 misses out of 6 for unscreened baselines.
      [Updated in Sec. 6, Eq. 4 & App. F.3, Table 20, Fig. 11b]

W2 (Part 2) & Q2. Smooth latency-decay utilities instead of a hard deadline

"How would the conclusions change under a softer utility than a hard deadline (e.g., utility that decays with latency)?"

The timely-accuracy curve {Sc(D)}D≥0\{S_c(D)\}_{D\ge 0} completely determines mean utility for every smooth decreasing latency penalty, and minimum J/token remains suboptimal across all 8/8 exponential-decay settings.

  • Exact relation in App. F.3: For any utility u:[0,∞)→[0,1]u:[0,\infty)\to[0,1] that decreases with latency toward u(∞)=0u(\infty) = 0, writing u(Tic)=∫0∞1{Tic≤D} (−du(D))u(T_{ic}) = \int_0^\infty \mathbf{1}\{T_{ic} \le D\}\,(-du(D)) gives Uc=1n∑iYicu(Tic)=∫0∞Sc(D) (−du(D))U_c = \frac{1}{n}\sum_i Y_{ic}u(T_{ic}) = \int_0^\infty S_c(D)\,(-du(D)), so PACE-SELECT applies directly by screening U^c≥qu\widehat U_c \ge q_u.
  • Exponential-decay results across 23 configurations in Fig. 11c: Under Uc(τ)=1n∑iYice−Tic/τU_c(\tau) = \frac{1}{n}\sum_i Y_{ic} e^{-T_{ic}/\tau} for τ∈{5,10,20,40} s\tau \in \{5, 10, 20, 40\}\text{ s}, minimum J/token fails in all 8/8 task and τ\tau settings, while increasing τ\tau on MATH smoothly shifts the best choice along PACE's frontier from llm.npu Balanced at 512 to No cap and then to 6-thread T-MAC with No cap.
    [Updated in Sec. 2 & App. F.3, Fig. 11c]

Note on Appendix A: Since you highlighted our measurement care, we also share a wording correction in Apps. A.2 and A.4 detailed under Reviewer kxJU Q1: three-repeat averaging applied to our stage and budget controls in Apps. B.1 and D.1 while full sweeps ran one pass, and energy integrates unplugged Android battery power Vbat×IbatV_{\mathrm{bat}}\times I_{\mathrm{bat}} every ~0.5 s following llm.npu (Xu et al., 2024), HeteroLLM (Chen et al., 2025), and MNN-AECS (Huang et al., 2025). Unplugged checks kept below 42 °C across 384 OnePlus Ace 6 runs in Table 3 & App. F.3, Table 19, Panel B and 150 OnePlus Ace7 Pro 3-repeat runs in App. F.1 reproduce our 38.6% saving at 38.2% [36.7%, 39.7%] with 5.3%–7.0% median variation.

We hope these finite-sample guarantees, smooth-utility results, and unseen-request studies merit reconsidering your rating!

Thanks for the rebuttal. It has addressed my concerns. I've updated my score accordingly.

AuthorsOpenReview ↗

Thank you for your support and feedback

Dear Reviewer ooo7,

Thank you very much for reading our response and raising your score! We truly appreciate your constructive feedback and the time you dedicated to reviewing our work. We will make sure to incorporate all the new results, discussions, and clarifications into the final version of the paper.

Best regards,

The Authors

Reviewer_kxJUOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a task-aware framework, evaluating on-device LLM inference using timely accuracy and total request energy. It studies deployment paths, execution modes, and output budgets, then selects the lowest-energy configuration meeting an empirical timely accuracy target. Measurements cover three LLMs and two mobile phones.

Critical strengths

Measurement studies with careful experimental controls. Byte-identical outputs across execution modes allow time and energy to be compared at fixed answer quality, while paired output-budget experiments distinguish answer recovery from additional generation that does not improve correctness. Overall, paper is well-written.

Critical weaknesses
  • The motivation and value of timely accuracy need a stronger case. The paper contrasts it with throughput and energy per token but does not establish what it offers beyond the standard accuracy-latency trade-off, of which it is a single point. The tested deadlines and accuracy targets are not tied to any concrete application requirement, so the basis for hard feasibility thresholds is unclear. In the Qwen vs. Llama example, the models reach 61.94% and 70.20% timely accuracy at 30 seconds with mean request energies of 4.35 J and 8.75 J. Selecting Llama at a 70% target follows the stated rule, but the comparison itself pits a 3B model against an 8B one at a fixed deadline, which measures model capacity rather than the runtime efficiency the paper claims to evaluate.
  • The energy comparisons may not reflect a system that stops generation at the deadline. All requests run to completion, including those that have already missed the deadline. Stopping these requests could reduce energy differently across configurations, potentially changing the ranking among feasible configurations and the reported savings. The paper acknowledges the retrospective design but does not assess its effect on these comparisons. A comparison with deadline-based stopping, accounting for energy consumed by all requests, would clarify whether the configuration rankings and reported savings still hold.
  • This paper states three LLMs, three deployment paths, and two phones, but the measured coverage is narrower: the full deployment grid exists only for Qwen2.5-3B (Table 4), BitNet-2B covers five QNN configurations on GSM8K with a different QNN version (Table 15), Llama 3.1-8B covers a single configuration (Table 13), and Redmi covers GSM8K only. The selection replay uses Qwen alone. Please scope the claims in the abstract and introduction to match.
  • The takeaways stay close to the definitions. Timely accuracy is bounded by final accuracy, speed helps only when correct answers cross the deadline, a larger budget helps only when it recovers answers, and the cheapest feasible configuration is the stated objective. These would hold without any measurement. The data show that the trade-offs vary across workloads and models, but the paper stops at recording them and does not explain why, or how to anticipate them without re-measuring the full grid for each new setting.
Questions for authors
  1. Please clarify the energy measurement setup. The paper states only that power was integrated at the USB input. With a battery in the circuit, USB input power does not equal device consumption: part of it may go to charging, and under burst mode the battery may supply peak current that the meter does not see. Please report whether the battery was bypassed or held at a fixed charge state, whether the screen was off, and the instrument and sampling rate used.
  2. Please explain why PACE-Select makes no selection in 11/20 splits on QNN/GSM8K at 20s/ 60% when the full-batch timely accuracy (60.27%) clears the target, and whether a confidence-adjusted screen was considered.
5 comments

Dear Reviewer kxJU,

Thank you for recognizing our byte-identical mode checks, paired budget design, and clear writing. Below, we address each point one by one.

W1. Value beyond a single accuracy–latency point, realistic deadlines, and 3B vs. 8B scope

"The paper... does not establish what it offers beyond the standard accuracy-latency trade-off, of which it is a single point. The tested deadlines and accuracy targets are not tied to any concrete application requirement... pitting a 3B model against an 8B one at a fixed deadline... measures model capacity rather than the runtime efficiency..."

The curve {Sc(D)}D≥0\{S_c(D)\}_{D\ge 0} determines every smooth decreasing latency utility rather than just one cutoff; we tested 2–20 s interactive deadlines on unseen HammerBench tool calls; and we explicitly separated 3B vs. 8B model capacity from single-model runtime efficiency.

  1. Why {Sc(D)}D≥0\{S_c(D)\}_{D\ge 0} goes beyond separate accuracy and latency means: Separate means (Yˉc,Tˉc)(\bar Y_c, \bar T_c) miss request-level coupling between correctness, length, and energy—e.g., incorrect MATH traces run longer and use 65.3%–79.0% of recorded energy in App. B.2, Fig. 8. For any utility u:[0,∞)→[0,1]u:[0,\infty)\to[0,1] decreasing toward u(∞)=0u(\infty)=0, writing u(Tic)=∫0∞1{Tic≤D} (−du(D))u(T_{ic})=\int_0^\infty \mathbf{1}\{T_{ic}\le D\}\,(-du(D)) gives Uc=1n∑i=1nYic u(Tic)=∫0∞Sc(D) (−du(D)).U_c = \frac{1}{n}\sum_{i=1}^n Y_{ic}\,u(T_{ic}) = \int_0^\infty S_c(D)\,(-du(D)). Thus {Sc(D)}D≥0\{S_c(D)\}_{D\ge 0} fixes mean utility for every smooth deadline penalty. Under u(t)=e−t/τu(t)=e^{-t/\tau} for τ∈{5,10,20,40} s\tau\in\{5, 10, 20, 40\}\text{ s} across 23 OnePlus Ace 6 configs, minimum J/token is suboptimal in all 8/8 task and τ\tau settings [Sec. 2; App. F.3, Fig. 11c].
  2. Interactive 2–5 s tool-calling deadlines on HammerBench: Grounding deadlines in 2–5 s tool calls vs. 10–60 s local reasoning, we ran 2,620 unplugged tests on OnePlus Ace 6 HammerBench using 130 calibration and 524 unseen test requests across 5 Qwen2.5-3B configs. PACE-SELECT picks QNN Balanced at 1.03 J and 1.95 s over llm.npu Burst at 1.52 J and 1.56 s, saving 32.1% energy while raising strict call and rejection accuracy by +6.3 pp at 2 s and +8.4 pp at 5–20 s [Secs. 1, 2, 6, Table 3; App. F.2, Table 18].
  3. 3B vs. 8B scope: Agreed—comparing Qwen2.5-3B at 61.94% and 4.35 J against Llama 3.1-8B at 70.20% and 8.75 J tests model capacity, not runtime efficiency. Revised Section 5 states explicitly in blue that rr spans model sizes here only to check whether 3B reaches the target before paying 8B's higher cost.

W2. Stopping late runs at deadline DD vs. running to completion

"Stopping these requests could reduce energy differently across configurations, potentially changing the ranking among feasible configurations and the reported savings."

Stopping late runs at D=10 sD = 10\text{ s} on the phone keeps the same configuration ranking and preserves a 38.4% energy saving.

We implemented hard process stopping at D=10 sD = 10\text{ s} on OnePlus Ace 6 for llm.npu at budget 512 across 16 GSM8K questions and 64 paired unplugged runs, recording battery energy until process teardown completed:

Execution Mode on OnePlus Ace 6 GSM8KFull Run EnergyStop-at-10 s EnergyTimely Correct (≤10 s\le 10\text{ s})
llm.npu Burst, Fastest9.89 J8.75 J, -11.5% with 2 late runs killed11 / 16
llm.npu Balanced, PACE-SELECT5.40 J5.39 J, -0.1% with 2 late runs killed11 / 16
Paired Saving, Balanced vs. Burst-45.4%-38.4% [35.1%, 41.2%]Same, 11 / 16

Even when late runs are killed at 10 s, Burst draws much more power during all timely runs and during the first 10 s of late runs. Applying proportional truncation Eic′=Eicmin⁡(1,D/Tic)E'_{ic}=E_{ic}\min(1, D/T_{ic}) across all 23 OnePlus Ace 6 configurations also leaves 100% of selections unchanged, keeping savings at 38.6% on GSM8K and 41.1% vs. 41.0% on MATH.
[Updated in Secs. 2, 6, Table 3 & App. F.3, Table 19, Panel A]


W3. Matching abstract and introduction claims to measured coverage

"This paper states three LLMs, three deployment paths, and two phones, but the measured coverage is narrower... Please scope the claims in the abstract and introduction to match."

Agreed—we revised the Abstract, Sections 1 and 2, and Appendix A.1, Table 5 (submitted Table 4) in blue to state exact coverage: Qwen2.5-3B across three paths on OnePlus Ace 6 and Redmi K90 with 23 and 15 configurations, targeted QNN comparisons on BitNet-2B and Llama 3.1-8B, plus unseen-request Qwen2.5-3B studies on OnePlus Ace7 Pro with Snapdragon 8 Elite Gen 5 across 9 configurations and OnePlus Ace 6 HammerBench across 5 configurations.

W4. Insights beyond definitions and avoiding full-grid re-measurement

"The takeaways stay close to the definitions... the paper stops at recording them and does not explain why, or how to anticipate them without re-measuring the full grid for each new setting."

Definitions enforce Sc(D)≤AcS_c(D)\le A_c, but cannot predict the hardware and quantization reversals we measure; meanwhile, output consistency across modes, budgets, and same-SoC phones reduces accuracy profiling from O(R⋅M⋅B⋅P)O(R \cdot M \cdot B \cdot P) to O(R)O(R), avoiding full-grid re-measurement.

  • Why definitions alone cannot predict these results: Definitions do not predict that halving power in QNN Low-power raises request energy by +9.1% on GSM8K and +18.7% on HammerBench as active-idle SoC and DRAM draw integrates over twice the runtime; that (8, 6) CPU threads beats (8, 8) by 21.0% despite slower mean TTFT because 8 decode threads slow decoding in 35/36 runs; or that llm.npu Burst runs 20.2% faster on HammerBench yet loses 8.4 pp accuracy from false QLinearLPBQ tool calls in App. F.2.
  • Avoiding full-grid profiling, O(R⋅M⋅B⋅P)→O(R)O(R \cdot M \cdot B \cdot P) \to O(R): Brute-force search across RR artifacts, MM modes, BB budgets, and PP same-SoC phones needs RMBPR M B P passes. Because correctness YicY_{ic} matches 100% across MM modes and PP phones—hitting 10, 20, and 30 s / 60% in 17/20, 20/20, and 20/20 Redmi K90 →\to OnePlus Ace 6 splits—and capped runs are exact prefixes in App. D.1, Table 12, one unclipped pass per artifact rr fixes accuracy, while target modes need only small cost samples (k=16k=16 in App. C.1, Table 11). On OnePlus Ace7 Pro with Snapdragon 8 Elite Gen 5, calibrating 9 configs on 32 GSM8K questions matches the best fixed test choice at all 6/6 targets and saves 33.9% [31.4%, 36.3%] energy.
    [Updated in Sec. 6, Table 3 & Apps. C.1, F.1–F.2, Tables 17–18, Fig. 11a]

Q1. Energy measurement setup, battery circuit, and repeats

"With a battery in the circuit, USB input power does not equal device consumption... Please report whether the battery was bypassed or held at a fixed charge state, whether the screen was off, and the instrument and sampling rate used."

Thank you for catching this—checking our logs against your question uncovered a wording mistake in submitted Apps. A.2 and A.4, for which we sincerely apologize and which we corrected in blue:

  1. Actual sensor and unplugged circuit: Rather than an external USB meter, our harness follows standard mobile LLM practice on sealed phones (llm.npu, Xu et al., 2024; HeteroLLM, Chen et al., 2025; MNN-AECS, Huang et al., 2025; ShadowNPU, Yin et al., 2025). It integrates battery power Vbat×IbatV_{\mathrm{bat}}\times I_{\mathrm{bat}} from Android power_supply and BatteryManager every ~0.5 s, with screen off, active-idle power kept, and USB charging unplugged verified in per-run logs.
  2. Sampling window and multi-repeat checks: Unplugged battery sensing avoids opening sealed phones and avoids USB-charging or PMIC buffering distortion, while 2–60 s LLM requests span many ~0.5 s updates (Huang et al., 2025). Three-repeat averaging applied to our stage and budget controls in Apps. B.1 and D.1, while full sweeps ran one pass. Re-running 64 GSM8K questions across 3 llm.npu modes and 2 unplugged repeats on OnePlus Ace 6 below 42 °C over 384 runs reproduces Balanced vs. Burst savings at 38.2% [36.7%, 39.7%] vs. 38.6%, and 150 OnePlus Ace7 Pro 3-repeat runs show 5.3%–7.0% median variation.
    [Corrected in Sec. 2 & Apps. A.2, A.4; new checks in Sec. 6, Table 3 & Apps. F.1, F.3, Table 19, Panel B]

Q2. Why QNN/GSM8K makes no selection in 11/20 splits at 20 s / 60% and confidence bounds

"Please explain why PACE-Select makes no selection in 11/20 splits on QNN/GSM8K at 20s/ 60% when the full-batch timely accuracy (60.27%) clears the target, and whether a confidence-adjusted screen was considered."

Full-batch QNN Balanced clears 60% by only 0.27 pp at 60.27%, while Burst reaches 58.23% and Low-power reaches 15.69%. With ncal=263n_{\mathrm{cal}} = 263 and SE ~3.02 pp, split variation puts S^c(20)\widehat S_c(20) below 60% in 11/20 splits, so PACE-SELECT makes no selection (6/3/11, now in Table 2). Replacing S^c(D)≥q\widehat S_c(D)\ge q with a one-sided Clopper–Pearson lower bound at level δ/m\delta/m over mm candidates keeps infeasible selection probability below δ\delta, requiring ncal≍2ln⁡(m/δ)/Δ2n_{\mathrm{cal}} \asymp 2\ln(m/\delta)/\Delta^2. At a 20% share and δ=0.10\delta = 0.10, this cuts missed selections by 95%, from 107 down to 5 of 1,200 splits, and down to 0/20 on QNN 20 s / 60% [Sec. 6, Table 2; App. F.3, Table 20, Fig. 11b].

We hope these hardware stopping experiments, unseen-request tests, and transparent corrections earn your support!

AuthorsOpenReview ↗

Reminder to reply

Dear Reviewer kxJU,

As the discussion period is nearing its end with less than one day remaining, we wanted to follow up and ensure that our previous response and revised manuscript have addressed all your concerns satisfactorily. If there are any additional points or feedback you would like us to clarify, please let us know. Your insights are invaluable to us, and we are eager to address any remaining questions to further improve our work.

Thank you again for your time and effort in reviewing our paper!

Best regards,

The Authors

Reviewer_8xGLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that on‑device LLM efficiency should be judged by whether a request is answered correctly before a deadline, not by tokens/s or J/token. They define timely accuracy S_c(D) (fraction correct and finished by D) and energy per timely correct answer, and measure Qwen2.5‑3B through three deployment paths (ExecuTorch/QNN, llm.npu, T‑MAC) on two Snapdragon 8 Elite phones over GSM8K and MATH‑500, varying execution mode and output budget, with BitNet‑2B and Llama‑3.1‑8B as side comparisons. Main observations: speed only helps when it moves correct answers across the deadline, bigger budgets only help when they recover truncated‑but‑correct answers (GSM8K saturates at 512, MATH doesn’t), and final accuracy caps what any deadline can give you. PACE‑SELECT is then a rule that, from calibration requests, keeps configs that meet (D, q) and picks the lowest‑energy one; replayed over 23 measured configs with 20% calibration splits, it hits GSM8K 10 s/60% in 19/20 splits vs 5/20 for min‑J/token selection and saves 38.6% energy vs the fastest mode on llm.npu.

Critical strengths
  • The measurement work is careful and honest: external power meter, three repeats, paired requests, byte‑identical output checks across modes (App. A.3), missing records documented rather than imputed, and a full appendix grid (Tables 7–9) that includes the cases where the method loses

  • A few of the concrete findings are interesting, non-intuitive findings: QNN low‑power nearly halves power but more than doubles time so request energy goes up 9.1% (Fig 3c); lowest TTFT ≠ fastest request (Fig 4b); and in GSM8K 512 -> No cap, 100% of added tokens go to requests that continue to be incorrect.

Critical weaknesses
  • lack of generalization. Calibration and evaluation come from the same 1,319‑request pool. Nothing is tested on unseen prompts, a different workload, or the second phone, so the selection rule’s real use case is untested.

  • lack of real-world tasks: Only math benchmarks are used for selection, with 10–60 s deadlines that don’t obviously correspond to any on‑device application.

Questions for authors

Why does Table 2 omit the QNN mode rows and the largest‑budget baseline where PACE‑SELECT is worse or equal?

Energy: is idle/baseline device power subtracted? Is thermal throttling controlled across 1,319‑request batches (especially burst modes)?

Cross‑deployment comparisons mix runtime with different quantization artifacts (W4A16 LPBQ vs G32 vs GPTQ INT4; Table 3). Can you separate how much of the 819 vs 915 gap is the quantizer vs the runtime?

How is timely accuracy different from goodput with a correctness term (DistServe/Sarathi define SLO attainment)? Worth positioning against that and ML.ENERGY/EnerInfer, which already report energy + accuracy.

3 comments

Dear Reviewer 8xGL,

Thank you for your constructive review and support, and for highlighting our byte-identical mode checks, explicit failure accounting, and empirical findings. Below, we address each point directly.

W1. Generalization to unseen requests, a new workload, and another phone

"Calibration and evaluation come from the same 1,319-request pool. Nothing is tested on unseen prompts, a different workload, or the second phone, so the selection rule's real use case is untested."

We tested PACE-SELECT with fixed calibration sets on unseen requests across a newer phone and SoC, a non-math tool-calling workload, and cross-phone transfer from Redmi K90 to OnePlus Ace 6.

Held-Out Study with Qwen2.5-3BCalibration →\to Unseen TestPACE-SELECT vs. Fastest on Test SetToken-Level / Low-Power Baseline
Newer Phone: OnePlus Ace7 Pro with Snapdragon 8 Elite Gen 5, 9 GSM8K configs, full run incl. load32 →\to 64 questions, 859 unplugged runs, 0 test failuresMatches best fixed choice at 6/6 targets; saves 33.9% energy [31.4%, 36.3%] at 30 s / 60% at 11.43 vs. 17.30 J, both 68.8%Lowest J/token, QNN Low-power at 0.0669, reaches only 56.3% at 30 s and misses 60%
New Workload: OnePlus Ace 6 HammerBench tool calling, 5 configs at budget 256130 →\to 524 requests, 2,620 unplugged runsPicks QNN Balanced at 1.03 J over Burst at 1.52 J: -32.1% energy, +6.3 pp at 2 s, and +8.4 pp at 5–20 sQNN Low-power drops to 5.3% at 2 s while using +18.7% more energy
Second Phone: Redmi K90 →\to OnePlus Ace 6 on Snapdragon 8 Elite, 15 shared GSM8K configs20% Redmi cal →\to 80% unseen OnePlus test100% per-question correctness match across phones; meets 10, 20, and 30 s / 60% in 17/20, 20/20, and 20/20 splitsNo accuracy re-labeling needed across same-SoC phones

Across all three settings, small fixed calibration sets generalize cleanly to unseen requests while avoiding the missed targets of per-token baselines.
[Updated in Sec. 6, Table 3 & Apps. C.1, F.1–F.2, Tables 17–18, Fig. 11a]


W2. Non-math workloads and realistic on-device deadlines

"Only math benchmarks are used for selection, with 10–60 s deadlines that don't obviously correspond to any on-device application."

We grounded our deadlines in two mobile assistant settings—2–5 s for interactive tool calls and 10–60 s for local multi-step reasoning before a timeout or cloud fallback—and tested 2–20 s deadlines on unseen HammerBench function-calling requests across 2,620 unplugged runs on OnePlus Ace 6.

Among the 524 unseen HammerBench test requests, half require an exact function and argument call and half require rejecting irrelevant tools. The fastest mode, llm.npu Burst at 1.56 s, finishes 20.2% faster than QNN Balanced at 1.95 s, yet scores 8.4 pp lower at 5 s with 31.9% vs. 40.3% accuracy and uses 47% more energy at 1.52 vs. 1.03 J. Both deployments match on valid tool calls at 66/262 vs. 65/262, but llm.npu's LPBQ quantization triggers false tool calls on irrelevant prompts, cutting correct rejections from 146/262 down to 101/262. PACE-SELECT catches this on the 130 calibration requests, saving 32.1% energy while improving timely accuracy at both 2 s from 30.3% to 36.6% and 5–20 s from 31.9% to 40.3%.
[Updated in Secs. 1, 2, 6, Table 3 & App. F.2, Table 18]

Q1. Adding QNN mode rows and the largest-budget baseline to Table 2

"Why does Table 2 omit the QNN mode rows and the largest-budget baseline where PACE-SELECT is worse or equal?"

That is a fair point—we added both rows directly to main-text Table 2 (Sec. 6) with full met / missed / no-selection counts, and added a confidence-bound screen to prevent over-selecting borderline candidates.

  • Why QNN mode makes no selection in 11/20 splits at 20 s / 60% (6/3/11 vs. Fastest 17/3/0): Full-batch QNN Balanced exceeds 60% by just 0.27 pp at 60.27%, while Burst sits at 58.23%. With ncal=263n_{\mathrm{cal}} = 263 and SE ~3.0 pp, split variation puts S^c(20)\widehat S_c(20) below 60% in 11/20 splits, so PACE-SELECT makes no selection while Fastest always picks a mode.
  • Why QNN budget misses 5 splits on MATH 60 s / 40% (14/5/1 vs. Largest-budget 20/0/0): With ncal=98n_{\mathrm{cal}} = 98 and SE ~4.9 pp, five splits overestimate budget 512 above 40% against its true 37.4%, while No cap meets all 20 splits at the same energy of 13.13 J.
  • Simple statistical fix: Replacing S^c(D)≥q\widehat S_c(D)\ge q with a one-sided Clopper–Pearson lower confidence bound at δ=0.10\delta = 0.10 keeps the chance of picking an infeasible candidate below δ\delta, cutting missed selections at a 20% share by 95%, from 107 down to 5 out of 1,200 splits, and down to 0/20 on QNN 20 s / 60%.
    [Updated in Sec. 6, Table 2 & App. F.3, Table 20, Fig. 11b]

Q2. Active-idle power, energy measurement details, and thermal control

"Energy: is idle/baseline device power subtracted? Is thermal throttling controlled across 1,319-request batches (especially burst modes)?"

Active-idle device power is kept intentionally with the screen off, and an unplugged two-repeat check kept below 42 °C across 384 runs reproduces our 38.6% headline saving at 38.2% with a 95% CI of [36.7%, 39.7%].

  • Why active-idle power is kept: Slower modes keep the SoC and DRAM active longer. Keeping active-idle power captures this cost and explains why QNN Low-power cuts power by 46.9% yet doubles runtime by +105.9% and raises request energy by 9.1% in Sec. 3, Fig. 3c & App. B.1.
  • Appendix A correction: Checking our logs alongside kxJU Q1 caught a wording mistake in submitted Apps. A.2 and A.4, for which we apologize; accuracy, latency, Sc(D)S_c(D), and byte-identical checks are unaffected. Three-repeat averaging applied to our stage and budget controls in Apps. B.1 and D.1 while full sweeps ran one pass. Rather than an external USB meter, our harness follows standard mobile LLM practice (llm.npu, Xu et al., 2024; HeteroLLM, Chen et al., 2025; MNN-AECS, Huang et al., 2025; ShadowNPU, Yin et al., 2025), integrating unplugged battery power Vbat×IbatV_{\mathrm{bat}}\times I_{\mathrm{bat}} from Android power_supply and BatteryManager every ~0.5 s over 2–60 s request windows.
  • Thermal and repeat check: Re-running 64 GSM8K questions across 3 llm.npu modes and 2 unplugged repeats on OnePlus Ace 6 below 42 °C gives 4.48 J for Balanced vs. 7.25 J for Burst at 84/128 timely answers, saving 38.2% [36.7%, 39.7%] vs. 38.6% in the full sweep. OnePlus Ace7 Pro 3-repeat median energy variation is 5.3%–7.0%.
    [Corrected in Sec. 2 & Apps. A.2, A.4; new checks in Sec. 6, Table 3 & Apps. F.1, F.3, Table 19, Panel B]

Q3. Quantizer vs. runtime in the 819 vs. 915 accuracy difference

"Can you separate how much of the 819 vs 915 gap is the quantizer vs the runtime?"

All of the gap comes from the quantized model export, and none from runtime mode scheduling or phone hardware. With the exported artifact fixed, outputs are byte-identical across all 6 mode and thread pairs on 1,319 GSM8K questions and match 100% between OnePlus Ace 6 and Redmi K90 across all 15 shared configs. Thus, 819 on QNN W4A16 vs. 915 on llm.npu LPBQ W4A16 G32 comes entirely from the exported artifact; a host FP32/CUDA check in App. F.2 traces question-level flips to QLinearLPBQ scale compression.
[Updated in Sec. 6 & App. F.2]


Q4. Comparison with server goodput and ML.ENERGY / EnerInfer

"How is timely accuracy different from goodput with a correctness term... Worth positioning against that and ML.ENERGY/EnerInfer..."

Expanded Section 7 (Related Work) compares both directly: server goodput systems like DistServe and Sarathi-Serve schedule concurrent requests for a fixed model whose accuracy does not change, whereas on-device choices c=(r,m,b)c=(r,m,b) jointly change quantization accuracy, truncation recovery, latency, and energy. Likewise, ML.ENERGY and EnerInfer tune server settings or mobile frequency scaling without screening a task-level (D,q)(D, q) accuracy target.

We hope these new experiments and clear revisions fully address your questions!

AuthorsOpenReview ↗

Reminder to reply

Dear Reviewer 8xGL,

As the discussion period is nearing its end with less than one day remaining, we wanted to follow up and ensure that our previous response and revised manuscript have addressed all your concerns satisfactorily. If there are any additional points or feedback you would like us to clarify, please let us know. Your insights are invaluable to us, and we are eager to address any remaining questions to further improve our work.

Thank you again for your time and effort in reviewing our paper!

Best regards,

The Authors

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and Reviewers 8xGL, kxJU, and ooo7 for their constructive feedback. We appreciate that all three reviewers found the paper clear, valued our experimental controls—including byte-identical mode outputs, paired budget comparisons, and full failure accounting—and recognized our central finding that token-level metrics (tokens/s, J/token) do not reflect task-level efficiency.

During the rebuttal period, we ran four unplugged phone studies totaling more than 3,600 runs, derived finite-sample risk bounds and smooth-utility guarantees, and checked quantization differences on host hardware. All changes are highlighted in blue in the revised PDF, including a concise 2-page Appendix F.

Summary of New Experiments and Revisions

  1. Generalization to a Newer Phone, Tool Calling, and Cross-Phone Transfer — Addresses 8xGL W1–W2; kxJU W1, W4; ooo7 W1

    • Newer smartphone and SoC: On a OnePlus Ace7 Pro phone with Snapdragon 8 Elite Gen 5, we measured full runs including model loading across 9 Qwen2.5-3B configurations, fixing calibration on 32 GSM8K questions and testing on 64 unseen questions over 859 unplugged runs. PACE-SELECT matches the best fixed choice at all 6/6 targets and cuts mean request energy by 33.9% at 30 s / 60% with a 95% CI of [31.4%, 36.3%], whereas minimum J/token misses 60% [Sec. 6, Table 3; App. F.1, Table 17, Fig. 11a].
    • Interactive 2–5 s tool calling: On OnePlus Ace 6 HammerBench function calling across 5 configurations, calibrating on 130 requests and testing on 524 unseen requests over 2,620 unplugged runs cuts energy by 32.1%, from 1.52 J down to 1.03 J, while raising timely accuracy over the fastest mode by +6.3 pp at 2 s and +8.4 pp at 5–20 s [Secs. 1, 6, Table 3; App. F.2, Table 18].
    • Cross-phone transfer and low-cost calibration: Per-question correctness matches 100% across all 15 shared Redmi K90 and OnePlus Ace 6 configurations on Snapdragon 8 Elite, meeting 10, 20, and 30 s / 60% in 17/20, 20/20, and 20/20 splits without re-measuring accuracy, so one unclipped run per model artifact fixes accuracy across modes, budgets, and same-SoC phones [Sec. 6; Apps. C.1, F.2].
  2. Stopping Late Runs at Deadline DD — Addresses kxJU W2

    • Stopping late llm.npu runs at D=10 sD = 10\text{ s} on OnePlus Ace 6 across 64 paired unplugged runs lowers Burst energy by 11.5%, from 9.89 to 8.75 J, yet Balanced still matches timely accuracy at 11/16 and saves 38.4% energy with a 95% CI of [35.1%, 41.2%]. Proportional truncation across all 23 configurations also leaves 100% of selections unchanged [Secs. 2, 6, Table 3; App. F.3, Table 19, Panel A].
  3. Complete Table 2, Finite-Sample Risk Bounds, and Smooth Deadline Utilities — Addresses 8xGL Q1; kxJU W1, Q2; ooo7 W2, Q1–Q2

    • We added the borderline QNN mode and max-budget rows to Table 2, showed that a one-sided Clopper–Pearson lower confidence bound with ncal≍2ln⁡(m/δ)/Δ2n_{\mathrm{cal}} \asymp 2\ln(m/\delta)/\Delta^2 cuts missed selections at a 20% share by 95% from 107 to 5 out of 1,200 splits, and proved that the curve {Sc(D)}D≥0\{S_c(D)\}_{D\ge 0} determines every smooth decreasing latency utility—with minimum J/token suboptimal in all 8/8 exponential-decay settings [Secs. 2, 6, Table 2; App. F.3, Table 20, Figs. 11b–c].
  4. Appendix A Documentation Correction and Thermal Repeatability — Addresses 8xGL Q2; kxJU Q1; ooo7

    • Checking our harness logs against kxJU Q1 caught a wording mistake in submitted Apps. A.2 and A.4, for which we apologize: three-repeat averaging applied to our stage and budget control experiments in Apps. B.1 and D.1, whereas full-batch sweeps ran one pass; and request energy integrates unplugged Android battery power Vbat×IbatV_{\mathrm{bat}}\times I_{\mathrm{bat}} every ~0.5 s with the screen off and active-idle power kept, following recent mobile LLM studies (llm.npu, Xu et al., 2024; HeteroLLM, Chen et al., 2025; MNN-AECS, Huang et al., 2025; ShadowNPU, Yin et al., 2025). Accuracy, latency, and Sc(D)S_c(D) are unaffected, and unplugged multi-repeat checks—384 OnePlus Ace 6 runs kept below 42 °C saving 38.2% [36.7%, 39.7%] vs. 38.6% in Table 3 & App. F.3, Table 19, Panel B, plus 150 OnePlus Ace7 Pro 3-repeat runs with 5.3%–7.0% median variation in App. F.1—confirm our energy results.
  5. Clearer Scope, Related Work, and Quantization Check — Addresses 8xGL Q3–Q4; kxJU W1, W3

    • We clarified exact phone and model coverage (OnePlus Ace 6, Redmi K90, and OnePlus Ace7 Pro) in the Abstract, Secs. 1, 2, 5, and App. A.1, Table 5, contrasted server goodput and runtime energy controllers in Sec. 7, and traced the 819 vs. 915 NPU accuracy gap to QLinearLPBQ scale compression in App. F.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.