- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Q2F, a CPU-NPU cooperative framework for efficient on-device language generation. It uses a low-bit LM-head on the NPU for candidate selection and refines the selected tokens on the CPU using existing embeddings. The method further introduces Fisher-based VGMP to preserve candidate quality. Experiments demonstrate reduced memory usage and improved inference efficiency with minimal accuracy degradation.
Critical strengths
The proposed method is well-motivated and addresses an important challenge in on-device LLM inference. The experiments are comprehensive, including real-device evaluations that demonstrate improvements in memory efficiency and inference throughput while largely preserving model accuracy.
Critical weaknesses
- The coarse-to-fine candidate selection and refinement framework builds upon existing retrieval-and-reranking approaches, while Fisher-based mixed-precision quantization is also a well-established technique. The main contribution seems to be their integration into CPU-NPU cooperative inference.
- VGMP strategy is not always necessary for accuracy. As shown in Table 2, simple group-wise 2-bit RTN combined with Q2F already achieves performance comparable to FP16, without Fisher-based precision allocation or additional fine-tuning. So how much does VGMP contribute beyond deployment compatibility?
- The comparison with FlashHead is mainly based on extrapolated CPU costs rather than direct end-to-end measurements. Since Q2F still performs full-vocabulary projection while FlashHead uses cluster-based retrieval, it is unclear whether Q2F offers better overall inference efficiency under comparable settings.
Questions for authors
- Could you explain whether the reported speedup is representative of diverse generation workloads, especially different context lengths and output lengths?
2 comments
Response to W2. 2-bit group-wise RTN
We agree that group-wise 2-bit RTN + Q2F is already strong in terms of accuracy. Our contribution addresses a different constraint: deployment on the target NPU. Since 2-bit group quantization is not supported by the backend (QAIRT 2.43.0.260116), the relevant comparison is among NPU-supported formats, where VGMP enables FP-level accuracy with per-channel mixed 2/4-bit quantization. We will clarify this distinction in the revision.
Response to W3. FlashHead comparison
We thank the reviewer for this point and agree that a CPU-cost extrapolation alone is insufficient. We therefore implemented FlashHead's default Llama3.2-1B configuration (8,016 centroids, 512 probes × 16 tokens = 8,192 candidates) on the same Snapdragon 8 Elite Gen 5 device used in the paper, in two execution scenarios, and directly measured its head-side costs.
| Config | Decoder graph (ms) | Head graph (ms) | CPU refinement (ms) | Measured/bound TPS |
|---|---|---|---|---|
| 16-bit head | 11.95 | 7.54 | – | 48.93 |
| 8-bit head | 11.93 | 3.94 | – | 59.91 |
| VGMP + FT + Q2F (K = 64) | 11.95 | 1.36 | < 0.2 | 71.63 |
| FlashHead, NPU-only | 11.94 | 7.99 | – | ≤ 50.18* |
| FlashHead, NPU + CPU refinement | 11.94 | - | 4.12 | ≤ 62.27* |
TPS in rows 1–3 is measured end-to-end (mean of five runs, Table 4). *Upper bound computed as 1000 / (decoder + head + refinement). It assigns zero cost to centroid scoring and cluster selection in the NPU+CPU mapping, and to sampling and runtime overhead in both FlashHead cases. In the measured rows, those costs add roughly 0.5–1 ms/token.
NPU-only. The centroid projection is a static-weight matmul and takes only 0.12 ms. Gathering the selected clusters (4.04 ms) and rescoring with the resulting input-dependent weights (3.54 ms) account for about 95% of the 7.99 ms head graph. This makes the FlashHead head slower than the dense 16-bit head (7.54 ms). Mobile NPUs favor static-weight kernels, and these dynamic-weight operations offset the savings from scoring fewer rows.
NPU + CPU refinement. If gathering and rescoring are instead moved to the CPU, as in Q2F, refining 8,192 candidates alone takes 4.12 ms/token. That is more than the entire 8-bit NPU head (3.94 ms). Q2F refines only 64 candidates, which takes under 0.2 ms.
Both bounds are optimistic: they omit every cost other than the decoder, the head graph and refinement. Even so, they remain well below Q2F's measured 71.63 tokens/s. Although FlashHead's reports a 1.75× end-to-end speedup on a laptop GPU against BF16 head, our conclusion concerns a different deployment setting. On CPU–NPU mobile backends, a static low-bit full-vocabulary head with a small CPU refinement budget is more efficient. We will add this table to Section 4.3 / Appendix A.9 and replace the extrapolation.
Response to Q1. Generation workloads and sequence lengths
We added a workload sweep on Gemma3-1B with K=64, varying prompt length from 25 to 4096 tokens and output length from 128 to 4096 tokens. The 4-bit decoder and INT8 input embeddings are fixed across head configurations. We follow the earlier device-profiling protocol, with two runs per configuration for this additional sweep rather than five.
We report decode throughput and end-to-end (E2E) throughput, defined as generated tokens divided by total request time, including prefill.
Decode throughput improves by 1.20–1.21× across these configurations. Including prefill, throughput improves by 1.19–1.21×, reducing total request latency by 16.0–17.1%. TTFT is slightly higher for Q2F, by 2.9–8.6 ms, but the decoding savings outweigh this increase in every tested configuration.
These results extend the efficiency evidence beyond the original Llama3.2-1B experiment and show consistent benefits across the tested prompt and output lengths. Our NPU deployment uses a static execution context of 8,192 tokens: varying the occupied context within this capacity does not change the compiled decode graph’s tensor dimensions. The LM-head projection and candidate-refinement dimensions also remain fixed for a given K. This execution regime helps explain the consistent decode speedups observed across the sweep. Prefill time nevertheless increases with prompt length, and its contribution to total request latency depends on output length; the E2E measurements account for both effects. These results characterize our fixed-context deployment, and we will document the execution configuration, full timings, and repetition protocol in the appendix.