acceptodds
Under review as a conference paper at ICLR 2027

CPU-NPU Cooperative LM-head Quantization for Efficient On-Device Language Generation

Abstract

On-device language generation is constrained by memory capacity and bandwidth. Although low-bit quantization shrinks Transformer decoders, large vocabularies leave the language-model head (LM-head) costly to store and execute on NPUs. However, existing methods either limit compression to prevent quality loss or rely on retrieval-based acceleration that imposes heavy CPU overhead. Efficient edge inference therefore requires reducing both LM-head storage and decoding cost without restricting the vocabulary. We introduce Q2F, which combines NPU-compatible LM-head compression with lightweight CPU refinement using existing embeddings. By exploiting the ability of low-bit predictions to retain promising candidates, Q2F supports greedy and stochastic decoding at 2.2 bits per LM-head weight. Across three models with FP16 decoders, the four-task average accuracy closely matches FP16 baselines, staying within 0.32 percentage points. On Snapdragon 8 Elite Gen 5, Q2F reduces LM-head weight storage by 73% over an INT8 baseline for Llama3.2-1B with a 4-bit decoder, improving end-to-end generation throughput by 19.6% with little CPU refinement overhead.

Primary area:
optimization
Keywords:
Quantization, Edge Language model, NPU, lm head, heterogeneous execution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_krvAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Q2F, a CPU-NPU cooperative framework for efficient on-device language generation. It uses a low-bit LM-head on the NPU for candidate selection and refines the selected tokens on the CPU using existing embeddings. The method further introduces Fisher-based VGMP to preserve candidate quality. Experiments demonstrate reduced memory usage and improved inference efficiency with minimal accuracy degradation.

Critical strengths

The proposed method is well-motivated and addresses an important challenge in on-device LLM inference. The experiments are comprehensive, including real-device evaluations that demonstrate improvements in memory efficiency and inference throughput while largely preserving model accuracy.

Critical weaknesses
  1. The coarse-to-fine candidate selection and refinement framework builds upon existing retrieval-and-reranking approaches, while Fisher-based mixed-precision quantization is also a well-established technique. The main contribution seems to be their integration into CPU-NPU cooperative inference.
  2. VGMP strategy is not always necessary for accuracy. As shown in Table 2, simple group-wise 2-bit RTN combined with Q2F already achieves performance comparable to FP16, without Fisher-based precision allocation or additional fine-tuning. So how much does VGMP contribute beyond deployment compatibility?
  3. The comparison with FlashHead is mainly based on extrapolated CPU costs rather than direct end-to-end measurements. Since Q2F still performs full-vocabulary projection while FlashHead uses cluster-based retrieval, it is unclear whether Q2F offers better overall inference efficiency under comparable settings.
Questions for authors
  1. Could you explain whether the reported speedup is representative of diverse generation workloads, especially different context lengths and output lengths?
2 comments

Response to Q1. Generation workloads and sequence lengths

We added a workload sweep on Gemma3-1B with K=64, varying prompt length from 25 to 4096 tokens and output length from 128 to 4096 tokens. The 4-bit decoder and INT8 input embeddings are fixed across head configurations. We follow the earlier device-profiling protocol, with two runs per configuration for this additional sweep rather than five.

We report decode throughput and end-to-end (E2E) throughput, defined as generated tokens divided by total request time, including prefill.

Prompt / output tokensDecode speedup over INT8E2E speedup over INT8Total latency: INT8 → Q2F
25 / 1281.20×1.19×1.87 → 1.56 s
25 / 10241.21×1.21×14.51 → 12.02 s
25 / 40961.20×1.20×58.03 → 48.20 s
1024 / 10241.21×1.21×14.71 → 12.20 s
4096 / 10241.20×1.19×15.15 → 12.72 s

Decode throughput improves by 1.20–1.21× across these configurations. Including prefill, throughput improves by 1.19–1.21×, reducing total request latency by 16.0–17.1%. TTFT is slightly higher for Q2F, by 2.9–8.6 ms, but the decoding savings outweigh this increase in every tested configuration.

These results extend the efficiency evidence beyond the original Llama3.2-1B experiment and show consistent benefits across the tested prompt and output lengths. Our NPU deployment uses a static execution context of 8,192 tokens: varying the occupied context within this capacity does not change the compiled decode graph’s tensor dimensions. The LM-head projection and candidate-refinement dimensions also remain fixed for a given K. This execution regime helps explain the consistent decode speedups observed across the sweep. Prefill time nevertheless increases with prompt length, and its contribution to total request latency depends on output length; the E2E measurements account for both effects. These results characterize our fixed-context deployment, and we will document the execution configuration, full timings, and repetition protocol in the appendix.

Response to W2. 2-bit group-wise RTN

We agree that group-wise 2-bit RTN + Q2F is already strong in terms of accuracy. Our contribution addresses a different constraint: deployment on the target NPU. Since 2-bit group quantization is not supported by the backend (QAIRT 2.43.0.260116), the relevant comparison is among NPU-supported formats, where VGMP enables FP-level accuracy with per-channel mixed 2/4-bit quantization. We will clarify this distinction in the revision.

Response to W3. FlashHead comparison

We thank the reviewer for this point and agree that a CPU-cost extrapolation alone is insufficient. We therefore implemented FlashHead's default Llama3.2-1B configuration (8,016 centroids, 512 probes × 16 tokens = 8,192 candidates) on the same Snapdragon 8 Elite Gen 5 device used in the paper, in two execution scenarios, and directly measured its head-side costs.

ConfigDecoder graph (ms)Head graph (ms)CPU refinement (ms)Measured/bound TPS
16-bit head11.957.54–48.93
8-bit head11.933.94–59.91
VGMP + FT + Q2F (K = 64)11.951.36< 0.271.63
FlashHead, NPU-only11.947.99–≤ 50.18*
FlashHead, NPU + CPU refinement11.94-4.12≤ 62.27*

TPS in rows 1–3 is measured end-to-end (mean of five runs, Table 4). *Upper bound computed as 1000 / (decoder + head + refinement). It assigns zero cost to centroid scoring and cluster selection in the NPU+CPU mapping, and to sampling and runtime overhead in both FlashHead cases. In the measured rows, those costs add roughly 0.5–1 ms/token.

NPU-only. The centroid projection is a static-weight matmul and takes only 0.12 ms. Gathering the selected clusters (4.04 ms) and rescoring with the resulting input-dependent weights (3.54 ms) account for about 95% of the 7.99 ms head graph. This makes the FlashHead head slower than the dense 16-bit head (7.54 ms). Mobile NPUs favor static-weight kernels, and these dynamic-weight operations offset the savings from scoring fewer rows.

NPU + CPU refinement. If gathering and rescoring are instead moved to the CPU, as in Q2F, refining 8,192 candidates alone takes 4.12 ms/token. That is more than the entire 8-bit NPU head (3.94 ms). Q2F refines only 64 candidates, which takes under 0.2 ms.

Both bounds are optimistic: they omit every cost other than the decoder, the head graph and refinement. Even so, they remain well below Q2F's measured 71.63 tokens/s. Although FlashHead's reports a 1.75× end-to-end speedup on a laptop GPU against BF16 head, our conclusion concerns a different deployment setting. On CPU–NPU mobile backends, a static low-bit full-vocabulary head with a small CPU refinement budget is more efficient. We will add this table to Section 4.3 / Appendix A.9 and replace the extrapolation.

Reviewer_1zo8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Q2F is a two stage decoding scheme for CPU-NPU devices: the NPU scores the full vocabulary with a very low bit LM-head (about 2.2 bits per weight), and the CPU then rescores only the top 64 to 128 candidates using rows of the existing tied embedding table. The coarse head uses vocabulary-grouped mixed-precision quantization (VGMP), where a Fisher based score promotes sensitive rows to 4 bits and an offline permutation keeps the two precision groups contiguous for dense NPU kernels; across Qwen3-4B, Gemma3-4B, and Llama3.2-3B the refined four task averages stay within 0.67 points of the FP16 baselines, and throughput rises 19.6% over an INT8-head baseline on a Snapdragon 8 Elite Gen 5.

Critical strengths
  • The motivating observation is crisp and well measured: a 2-bit head often misses the FP16 top-1 token, yet it keeps that token in its top 128 candidates at 95.85% to 99.90% of positions across four models (Figure 2, Table 5). Section 3.1 makes this precise with a greedy recovery condition and retained mass metrics, so the design rationale is testable, not anecdotal.
  • VGMP is built around real NPU limits, not just accuracy: per-channel scales with two contiguous precision blocks avoid the partial projections that group-wise formats need on this backend (Section 3.2). The Fisher score is checked against frequency, logit MSE, random, and original order selection, and it holds up under distribution shift, reaching 88.23% Recall@128 versus 78.17% for frequency on Gemma3-4B (Figure 3, Table 6).
  • The on-device evaluation is end to end on real hardware (Snapdragon 8 Elite Gen 5, QAIRT, and a custom Genie sampler), with throughput including candidate selection, refinement, and sampling, reported as mean and standard deviation over five runs of 2,500 tokens. Refinement is also not tied to one quantizer: it improves RTN, GPTQ, and fine-tuned heads (Table 13), and Table 14 cleanly separates what fine-tuning adds from what refinement adds.
Critical weaknesses
  • I could not find any seeds, repeats, or confidence intervals for the downstream accuracy numbers, even though decoding is stochastic and the claimed differences are only 0.03 to 0.67 points. The refined model sometimes even beats the FP16 baseline (Llama3.2-3B average 45.86 versus 45.19 in Table 2), and without variance estimates I cannot tell whether those wins are real or run to run noise of the same size as the claimed effects.
  • All hardware evidence comes from one 1B model on one chipset, against the most favorable baseline. The 73% storage reduction is for LM-head weights only, relative to an INT8 head; total NPU graph memory falls by 24.5% (Table 4), and against the paper's own 4-bit head row the gains shrink to about 9% memory and about 6% throughput, while the 3B and 4B models are simulated only. There are also no energy, power, or sustained load numbers, which matter for a method sold on deployment efficiency.
  • The comparisons to the closest alternatives are indirect. FlashHead is compared only through an extrapolated gather-and-GEMV cost of about 4.1 ms at its 8,192 candidate budget, not through an implementation (Appendix A.9); ARCHead only on packed storage ratio, 3.7 to 3.9x versus BF16 (Section 1); and the authors' own selective low-rank variant is profiled for speed and memory with no matched accuracy comparison (Appendix A.10). A direct accuracy and speed comparison against at least one of these would strengthen the paper.
Questions for authors
  1. Can you report downstream accuracy over multiple sampling seeds, or with greedy decoding, with confidence intervals for Tables 2 and 4? That would let me judge differences like the 0.32 point maximum drop, and the cases where Q2F beats the FP16 baseline, against noise.
  2. What are the throughput and total memory numbers if the headline comparison uses the 4-bit per-channel head from Table 4 as the baseline instead of the INT8 head? Please include the CPU side embedding footprint in the memory total.
  3. Can you add a matched accuracy comparison against the selective low-rank variant of Appendix A.10, or against ARCHead itself, at equal bits per weight? That is the closest alternative design.
2 comments

Response to Q1. Multi-seed downstream evaluation

We repeated the stochastic downstream evaluations with five sampling seeds. The complete per-task means and 95% confidence intervals for all 4 models are provided in our response to Reviewer v4nP, “Response to Q1. Statistical robustness and task-level regressions”. Task average scores are 55.76 ± 0.37 versus 55.84 ± 0.21 for Gemma3-4B, 74.05 ± 0.35 versus 74.08 ± 0.32 for Qwen3-4B, and 45.13 ± 0.80 versus 44.96 ± 0.54 for Llama3.2-3B, with VGMP + Q2F (K=128) and FP reference each. In each case the two confidence intervals overlap. This includes Llama3.2-3B, where the single-run result had Q2F above the FP16 baseline (45.86 vs. 45.19). Neither the small gains nor the small drops are distinguishable from seed-to-seed variability. These results support near-reference average quality while making decoding variability explicit. These additional evaluations use FP16 decoders and include the supplementary Llama3.2-1B results. We will incorporate the complete multi-seed results and evaluation details into the revised manuscript.

Response to Q2. The 4-bit baseline and total memory accounting

Using the 4-bit per-channel head as the baseline, the original Llama3.2-1B experiment gives:

ConfigurationNPU graph (MiB)Graph + INT8 embedding weights (MiB)Throughput (tokens/s)
4-bit head617.69868.1967.39 ± 1.21
VGMP + Q2F, K=64561.07811.5771.63 ± 0.68

The CPU embedding weight payload is 250.50 MiB in both configurations. Q2F therefore improves throughput by 6.3%, reduces NPU graph memory by 9.2%, and reduces combined graph-plus-embedding storage by 6.5%.

We also added complete Gemma3-4B and Gemma3-1B profiling, including INT8, 4-bit, 2-bit, and multiple Q2F candidate budgets. These results and the separate embedding footprints appear in our response to Reviewer v4nP, "Response to Q2. Larger-model deployment and the 4-bit baseline”. At K=64, throughput improves over the 4-bit head by 3.0% and 6.6%, respectively.

We have not measured energy or power. Our profiling runs include cooldowns and do not establish continuous thermal steady-state performance; we will state this scope explicitly.

Response to Q3. Comparisons with FlashHead, ARCHead, and selective low-rank correction

We thank the reviewer for suggesting these comparisons.

FlashHead. We replaced the extrapolation with an on-device implementation of FlashHead's default Llama3.2-1B configuration (8,192 candidates). We measured both an NPU-only mapping and a mapping with CPU refinement on the same device; details are in our response to Reviewer krvA, “Response to W3. Comparison with FlashHead”. Even when we count only the decoder, the head graph and refinement, the two mappings are bounded at ≤50.2 and ≤62.3 tokens/s, respectively, compared with 71.63 tokens/s measured for Q2F.

ARCHead and selective low-rank correction. We did not reproduce ARCHead. Instead, we report the matched comparison the reviewer also suggested, against our selective low-rank variant (VGLoRA). VGLoRA is an alternative to VGMP: it uses the same sensitivity-based vocabulary permutation, but instead of promoting the leading rows to 4 bits, it attaches a low-rank correction to them. The additional results clarify why we selected VGMP over selective low-rank correction: VGLoRA improves standalone coarse-head accuracy, but VGMP provides faster execution, while Q2F recovers the accuracy lost to aggressive quantization.

We evaluated VGLoRA, our selective low-rank variant from Appendix A.10, using the same FP16 decoder, INT8 input embeddings and permuted LM-head as the VGMP configurations below. All coarse heads are fine-tuned. Accuracy is mean ± 95% CI over five seeds:

Llama3.2-1B headIFEvalGSM8K
VGMP, M=12k51.94 ± 1.3622.55 ± 1.03
VGLoRA, M=10k, rank=6453.55 ± 2.8125.17 ± 1.29
VGLoRA, M=20k, rank=25654.36 ± 2.2225.22 ± 0.99
VGMP + Q2F, K=6459.64 ± 1.0630.40 ± 0.80

Separately, device profiling with a fixed 4-bit decoder gives the following results. Throughput is mean ± SD over five runs:

HeadNPU graph memory (MiB)Tokens/s
VGMP, M=12k560.6072.03 ± 0.69
VGLoRA, M=10k, rank=64555.2169.73 ± 0.89
VGLoRA, M=20k, rank=256560.35Not measured
VGMP + Q2F, K=64561.0771.63 ± 0.68

At nearly matched deployed memory, rank-256 VGLoRA achieves higher mean standalone accuracy than VGMP. However, the profiled rank-64 variant already reduces throughput by 3.2% relative to VGMP, despite its smaller memory footprint. Its additional low-rank projections and accumulation introduce execution costs that motivated our choice of the simpler, contiguous mixed-precision head.

Crucially, Q2F does not require the coarse head to produce accurate final predictions; it requires the head to retain useful candidates. Adding Q2F to VGMP raises IFEval from 51.94 to 59.64 and GSM8K from 22.55 to 30.40 in the accuracy evaluation, exceeding both tested standalone VGLoRA configurations. In device profiling, adding Q2F reduces VGMP throughput by only 0.56%, retaining 71.63 tokens/s.

These complementary results support our design choice: allocate the coarse-head budget to fast candidate generation and use inexpensive refinement to recover quality. We will add the results and this rationale to Appendix A.10.

Reviewer_v4nPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Q2F, a heterogeneous CPU–NPU decoding framework for aggressively quantized LM heads. An NPU performs low-bit full-vocabulary scoring, after which the CPU refines the top-K candidates using existing tied input embeddings. The proposed vocabulary-grouped mixed-precision quantization (VGMP) promotes Fisher-selected vocabulary rows to 4 bits, quantizes the remaining rows to 2 bits, and permutes rows into contiguous precision blocks for efficient NPU execution. Experiments across Llama, Qwen, and Gemma models report near-reference downstream accuracy at approximately 2.2 bits per head weight. On-device evaluation with Llama3.2-1B demonstrates substantial memory savings and a 19.6% throughput improvement over an INT8-head baseline.

Critical strengths
  1. Simple and hardware-aware formulation. The observation that inaccurate low-bit heads can still retain high top-K candidate recall effectively reframes LM-head quantization as candidate generation followed by selective recovery. Reusing existing embeddings avoids a separate high-precision head, making the approach particularly suitable for tied-weight edge models.
  2. Convincing combination of quality analysis and deployment evidence. Candidate-recall and retained-mass analyses complement downstream evaluations, ablations, and experiments with quantized decoders. The integrated Snapdragon evaluation is especially valuable because it demonstrates that the accuracy benefits translate into real memory and throughput improvements rather than merely favorable nominal bit-widths.
Critical weaknesses
  1. The near-reference quality claim lacks statistical robustness. Downstream results use stochastic decoding but report neither multiple seeds nor variability. Consequently, small average differences—and occasional improvements over FP16—are difficult to interpret. Task-level regressions can also be hidden by averaging; for example, Llama3.2-3B loses 3.57 GSM8K points despite an improved four-task average. This weakens the central claim that Q2F reliably preserves generation quality.
  2. The end-to-end efficiency evidence is narrowly scoped. On-device evaluation covers only Llama3.2-1B, one device, and one generation prompt, although CPU refinement cost depends on vocabulary size, hidden dimension, and K . Moreover, the headline 19.6% speedup uses the INT8-head baseline; relative to the practical 4-bit head in Table 4, the improvement is approximately 6.3%. Thus, the advantage over strong deployable alternatives and its generalization to larger models remain insufficiently established.
Questions for authors
  1. Can you report multi-seed mean and standard deviation for the stochastic downstream evaluations, or complementary greedy-decoding results, to establish whether the small quality gaps are significant?
  2. Can you provide on-device results for a larger-vocabulary or 4B-scale model and clarify the comparison against the strongest NPU-feasible baseline, particularly the 4-bit head?
  3. How does candidate-restricted sampling affect free-running generation relative to full-vocabulary top-p /top-k sampling, especially when substantial reference probability mass falls outside the candidate set?
3 comments

Response to Q1. Statistical robustness and task-level regressions

We thank the reviewer for requesting this evaluation. We repeated the stochastic downstream evaluations with five sampling seeds and report mean ± 95% confidence interval below (Table 2 and Table 10)

Gemma3-4B

LM-headMMLU-ProAIMEIFEvalGSM8KAvg.
FP16 reference41.42 ± 0.4118.78 ± 0.5882.95 ± 0.6980.21 ± 0.3355.84 ± 0.21
W2 GS6412.52 ± 0.500.15 ± 0.1265.73 ± 0.6123.41 ± 1.9225.45 ± 0.56
W2 GS64 + Q2F, K=6441.21 ± 0.6418.86 ± 0.7883.29 ± 0.8781.02 ± 0.9956.09 ± 0.41
VGMP39.12 ± 0.8114.26 ± 0.8677.17 ± 1.8878.21 ± 0.9252.19 ± 0.46
VGMP + Q2F, K=12841.49 ± 0.3519.01 ± 0.4781.97 ± 0.8780.56 ± 1.0055.76 ± 0.37

Qwen3-4B

LM-headMMLU-ProAIMEIFEvalGSM8KAvg.
FP16 reference66.63 ± 0.4549.02 ± 1.2188.78 ± 0.6591.90 ± 0.4874.08 ± 0.32
W2 GS6438.89 ± 0.3018.39 ± 0.9768.01 ± 0.4689.81 ± 0.4453.78 ± 0.20
W2 GS64 + Q2F, K=6466.55 ± 0.3049.09 ± 0.8088.68 ± 0.5792.25 ± 0.4074.14 ± 0.29
VGMP65.41 ± 0.1738.48 ± 0.7068.59 ± 1.2178.21 ± 1.6662.67 ± 0.42
VGMP + Q2F, K=12866.68 ± 0.3849.58 ± 1.6787.89 ± 0.6192.05 ± 0.5274.05 ± 0.35

Llama3.2-3B

LM-headMMLU-ProAIMEIFEvalGSM8KAvg.
FP16 reference35.19 ± 0.523.94 ± 0.7974.00 ± 0.7666.70 ± 1.2344.96 ± 0.54
W2 GS6410.00 ± 5.140.32 ± 0.1646.09 ± 2.446.28 ± 0.8715.67 ± 1.88
W2 GS64 + Q2F, K=6435.36 ± 0.484.12 ± 0.6373.72 ± 1.5266.13 ± 1.2844.83 ± 0.40
VGMP35.10 ± 0.314.42 ± 1.1050.29 ± 2.0630.31 ± 1.7030.03 ± 0.84
VGMP + Q2F, K=12835.71 ± 0.594.09 ± 0.9274.22 ± 0.8966.52 ± 0.8945.13 ± 0.80

Llama3.2-1B

LM-headMMLU-ProAIMEIFEvalGSM8KAvg.
FP16 reference15.19 ± 0.461.80 ± 0.7560.38 ± 1.6931.77 ± 1.2727.29 ± 0.74
RTN W815.29 ± 0.231.56 ± 0.8358.92 ± 0.5031.24 ± 1.1426.75 ± 0.42
RTN W415.69 ± 0.231.59 ± 0.4758.80 ± 1.5029.55 ± 1.9426.41 ± 0.31
RTN W20.00 ± 0.000.00 ± 0.0021.75 ± 3.390.65 ± 1.155.60 ± 1.09
VGMP17.92 ± 0.441.48 ± 0.5851.94 ± 1.3622.55 ± 1.0323.47 ± 0.45
VGMP + Q2F, K=3215.27 ± 0.411.89 ± 0.6459.81 ± 1.0330.63 ± 0.8526.90 ± 0.37
VGMP + Q2F, K=6415.47 ± 0.451.50 ± 0.5459.64 ± 1.0630.40 ± 0.8026.75 ± 0.31
VGMP + Q2F, K=12815.29 ± 0.431.80 ± 0.3359.74 ± 1.6530.49 ± 1.1426.83 ± 0.31

*VGMP include fine-tuning throughout the responses

Interpretation. VGMP + Q2F at K=128 differs from the FP16 reference by −0.08, −0.03, +0.17, and −0.46 percentage points in average accuracy for Gemma3-4B, Qwen3-4B, Llama3.2-3B, and Llama3.2-1B, respectively, while recovering most of the degradation from the unrefined heads.

The Llama3.2-3B GSM8K deficit highlighted by the reviewer is much smaller in the five-seed means: 66.70 ± 1.23 versus 66.52 ± 0.89. We agree that average recovery does not imply identical task-level performance; the full tables expose remaining differences, including IFEval on Gemma3-4B and Qwen3-4B.

The five-seed results show that Q2F consistently recovers most of the accuracy loss from aggressive LM-head quantization and achieves near-reference average performance across all 4 models. Reporting per-task means and confidence intervals makes the remaining differences and decoding variability explicit. We will replace the single-run results with these multi-seed evaluations and document the seeds and confidence-interval calculation in the revised manuscript.

Response to Q2. Larger vocab-size and 4B model deployment

We added on-device measurements for Gemma3-4B and Gemma3-1B on the same Snapdragon platform, extending evaluation to a 4B-scale model and a larger vocabulary. Within each model, 4-bit decoder and INT8 input embeddings are fixed across LM-head configurations. Throughput includes candidate selection, refinement, and sampling and is reported as mean ± SD over five runs.

Gemma3-4B: device profiling with a 4-bit decoder

LM-headNPU graph memory (MiB)Throughput (tokens/s)
8-bit RTN2222.3919.39 ± 0.04
4-bit RTN1900.8920.86 ± 0.03
2-bit RTN1740.2021.68 ± 0.07
VGMP + Q2F, K=641751.2921.49 ± 0.04
VGMP + Q2F, K=1281751.2921.45 ± 0.04
VGMP + Q2F, K=2561751.2921.34 ± 0.06

Gemma3-1B: device profiling with a 4-bit decoder

LM-headNPU graph memory (MiB)Throughput (tokens/s)
8-bit RTN646.3170.84 ± 0.08
4-bit RTN501.7579.99 ± 0.12
2-bit RTN429.4286.99 ± 0.13
VGMP + Q2F, K=64433.5585.27 ± 0.13
VGMP + Q2F, K=128433.5585.06 ± 0.19
VGMP + Q2F, K=256433.5584.66 ± 0.07

The separate INT8 embedding artifacts occupy 642.68 MiB for Gemma3-4B and 289.14 MiB for Gemma3-1B, unchanged across configurations. They are excluded from the NPU graph memory column.

Using K=64, consistent with the original Llama3.2-1B deployment comparison, Q2F improves throughput over the 4-bit head by 3.0% on Gemma3-4B and 6.6% on Gemma3-1B, while reducing NPU graph memory by 7.9% and 13.6%, respectively. Relative to the INT8 head, the corresponding throughput improvements are 10.8% and 20.4%. For the original Llama3.2-1B experiment, the gain is 6.3% over the 4-bit head, compared with 19.6% over INT8. We will state both baseline comparisons explicitly.

Accuracy reference for Gemma3-4B with an FP16 decoder

The following results provide a complementary reference for the quality impact of the LM-head configurations. The decoder is FP16 throughout; the reference model is fully FP16, while quantized-head configurations use INT8 input embeddings. Values are 4-task average accuracy, mean ± 95% CI over five sampling seeds.

LM-headAverage accuracy (%)
FP16 reference55.84 ± 0.21
4-bit RTN50.64 ± 0.40
2-bit RTN7.45 ± 0.24
VGMP + Q2F, K=6455.46 ± 0.63
VGMP + Q2F, K=12855.76 ± 0.37
VGMP + Q2F, K=25655.73 ± 0.37

At K=64, Q2F improves average accuracy over the 4-bit head by 4.82 percentage points, remaining 0.38 points below the FP16 reference. K=128 brings the mean within 0.08 points of the reference, with slightly lower measured device throughput.

These accuracy results characterize head compression with an FP16 decoder; they are not direct accuracy measurements of the 4-bit-decoder Gemma deployments. Appendices A.5–A.6 provide additional evidence of Q2F's effectiveness with quantized decoders on other evaluated models. We will include the full profiling, accuracy results in the revision.

Response to Q3. Candidate-restricted sampling

We thank the reviewer for this question. We conducted two additional analyses on Qwen3-4B with VGMP + Q2F on AIME: measuring how often substantial reference mass is excluded, and testing the downstream effect of a candidate-induced token substitution.

1. Large candidate-mass losses are rare.

We teacher-forced 933 FP-generated AIME responses, comprising 2.77M generated positions. At each position, we evaluated the reference distribution and Q2F candidates at the same hidden state.

KRetained Avg. MassMass (<0.9)Mass (<0.5)Ref. Miss Rate
3299.75%0.48%0.13%0.18%
6499.87%0.25%0.07%0.10%
12899.92%0.15%0.04%0.06%

Reference sampling uses T=0.7, top-k=20, and top-p=0.8. At K=128, less than 90% of the reference mass is retained at only 0.15% of positions, and a reference sample falls outside the candidates with probability 0.06%.

Inspection of low-retention cases identified underestimated logits for tokens assigned to the 2-bit portion of the head. Excluded tokens were concentrated among formatting and discourse markers such as “✅,” **, and “Similarly.” The remaining high-probability candidates often provided plausible alternatives in these cases. This explains how a large local mass loss can have a limited semantic effect, without assuming that every omitted token is interchangeable with a retained one.

2. A targeted intervention found no detectable accuracy degradation.

We identified 236 originally solved AIME problems whose FP trajectories contained at least one position where Q2F excluded FP's top-1 token. For each problem, we selected the lowest-retention position among those omissions at K=128.

At that position, we forced either (i) FP's full-vocabulary top-1 token or (ii) Q2F's highest-scoring available token. Both branches then continued with the same FP model, at T=1.0 and with the original token budget. We generated eight continuations per branch.

Forced tokenAIME accuracy
FP full-vocabulary top-189.90%
Q2F candidate-restricted top-190.60%
Paired difference+0.7 ± 1.6 percentage points, 95% CI

The branches had equal empirical accuracy on 169 of 236 problems; the Q2F-token branch performed better on 36 and worse on 31. Failure to finish within the token budget was also similar: 9.0% versus 8.4%, respectively.

This intervention tests a deliberately severe omission on each selected problem and finds no detectable degradation in downstream accuracy. It isolates one substitution followed by FP continuation; it does not test cumulative omissions during fully Q2F-generated trajectories.

3. Example: losing approximately half of the reference mass.

After the context “…102/68 = 3/2…”, retained FP mass at K=128 is only 0.49, because FP's highest-probability token is excluded:

RankFP full vocabularyQ2F, K=128
1✅ 0.511Yes 0.861
2Yes 0.429So 0.124
3So 0.054✔ 0.007
4✔ 0.003Thus 0.004

Here, excluding “✅” shifts probability toward alternatives already favored by FP, preserving the displayed shared alternatives' ordering. The substituted continuation begins, “Yes, 102/68 = 3/2. ### Final Step: Multiply by 10 …”, instead of FP trajectory "✅ So \cot\theta=3/2 ### Step 5: Multiply by 10 ...". Both branches solve the problem in all eight continuations. The example illustrates why losing substantial reference mass can still leave the final answer unchanged.

These results show that large omissions are uncommon on the evaluated FP trajectories, and even severe single-token substitutions do not necessarily harm task success while end-to-end Q2F retains near-reference downstream quality. We will add the detailed analyses and the example to the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.