- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes μ-ZERO, a zero-anchored, piecewise log-companded scalar quantizer for LLM KV caches. It allocates more quantization levels near zero, where the authors argue attention logits are sensitive to perturbations caused by cancellation among signed query–key products. The method retains the standard per-group affine metadata and selects the companding strength α during prefill. Experiments across several model families and tasks show that μ-ZERO improves on KIVI in a number of 2-bit evaluations, including GSM8K, HumanEval, and LongBench. The paper also reports lower memory use and higher decode throughput, though the source of the throughput advantage over KIVI is not isolated.
Critical strengths
- The paper gives a clear motivation for nonuniform KV-cache quantization and connects it to attention-logit error under cancellation.
- The method retains the usual per-group quantization metadata and does not require learned codebooks or offline calibration.
- The 2-bit results are promising across several models and tasks, including reasoning, code generation, and long-context evaluation.
- The evaluation covers multiple model scales, long-context settings, MQA and MLA architectures, group-size sensitivity, and kernel measurements.
Critical weaknesses
- The contribution of adaptive α selection is not sufficiently quantified. The paper says that the default 2-bit and 4-bit settings retain α=2, while larger values are more common for token-wise keys and very low bit widths. The main downstream results do not make clear how often α=4 or 16 is selected, or how much adaptation improves on fixed α=2.
- The speedup over KIVI is difficult to interpret. Table 15 uses different decode harnesses, so the end-to-end comparison does not isolate the source of the gain. In Table 16, μ-ZERO’s QK kernel is faster but its AV kernel is slower than KIVI’s. The paper does not explain what accounts for the QK latency difference.
- The AIME’25 gains have substantial uncertainty. The evaluation uses 30 problems with repeated responses and seeds. The paper acknowledges that the confidence intervals do not establish statistical superiority, but the limited number of distinct problems makes the relatively small gains difficult to assess.
- The benefit at 4 bits is less clear, and the tail tradeoff needs more analysis. On some LongBench results, μ-ZERO is close to BF16 and other 4-bit baselines. Stronger companding also gives up resolution in the tails to improve accuracy near zero, but the paper does not clearly show how this affects large-magnitude outliers or attention-sink tokens during long-context generation.
Questions for authors
- How often does the adaptive rule select α=4 or 16 in each main experiment? Could the authors report downstream results against a fixed α=2 baseline to show the independent contribution of adaptation?
- Do the KIVI and μ-ZERO kernels use comparable layouts, tiling, warp mappings, and compiler settings? What accounts for μ-ZERO’s lower QK latency despite its higher AV latency?
- Could the authors provide a paired uncertainty analysis by problem for AIME’25, or results on another reasoning benchmark, to clarify whether the observed gains are robust?
- On which tasks and models are the 4-bit gains consistent enough to matter in practice? Could the authors also report how companding affects large-magnitude outliers or attention-sink tokens in long-context generation?
5 comments
Response to Reviewer YzE7 (2/4)
R3.W2. Fair runtime comparison and QK/AV latency analysis
Weakness: The speedup over KIVI is difficult to interpret. Table 15 uses different decode harnesses, so the end-to-end comparison does not isolate the source of the gain. In Table 16, μ-ZERO’s QK kernel is faster but its AV kernel is slower than KIVI’s. The paper does not explain what accounts for the QK latency difference.
Question: Do the KIVI and μ-ZERO kernels use comparable layouts, tiling, warp mappings, and compiler settings? What accounts for μ-ZERO’s lower QK latency despite its higher AV latency?
We address this concern through a shared-harness comparison and an analysis of QK/AV implementation differences.
(1) Shared-harness runtime comparison. We repeated Llama-3.1-8B-Instruct measurements on the same dedicated RTX PRO 6000 Blackwell GPU using a shared Transformers driver, identical inputs, FP16 computation, 64-element groups, native 128-token windows, chunked dense prefill, and 160 warmup/512 measured decode updates. KIVI retains its official attention/cache core through a compatibility adapter.
Table R3c. Shared-harness throughput and decode memory. Throughput is the five-run median in aggregate tokens/s; speedups are relative to same-bit KIVI. Memory is decode-phase peak reserved allocation (GiB). Full results are reported in Appendix F.4.
| Context / batch | Bits | KIVI | μ-ZERO | Decode speedup | Decode memory: KIVI / μ-ZERO (GiB) |
|---|---|---|---|---|---|
| 8k / 16 | 2 | 322.94 | 466.09 | 1.44× | 18.52 / 18.46 |
| 8k / 16 | 4 | 322.74 | 409.25 | 1.27× | 21.49 / 21.00 |
| 32k / 16 | 2 | 189.40 | 233.58 | 1.23× | 29.08 / 26.66 |
| 32k / 16 | 4 | 143.39 | 194.85 | 1.36× | 41.23 / 38.52 |
| 4k / 120 | 2 | 1288.96 | 1416.05 | 1.10× | 30.90 / 32.24 |
| 4k / 120 | 4 | 1003.43 | 1213.83 | 1.21× | 41.68 / 40.01 |
As shown in Table R3c, μ-ZERO delivers 1.10–1.44×/1.21–1.36× KIVI's decode throughput at 2/4 bits. Including prefill, gains remain 1.05–1.24×/1.08–1.16×.
(2) Kernel comparability and QK latency. The implementation audit uses the same GPU and major compiler settings (NVCC 13.1, -O3, --use_fast_math, sm_120) for both methods; native layouts and thread mappings differ. KIVI materializes packed codes and metadata into its operator layout on each call and uses four warps with a 128-element reduction tile. μ-ZERO reads packed data directly, uses two warps over 64-token tiles, and reuses reconstructed Keys across four query heads. These differences explain the lower QK operator latency. The 16-entry lookup table (LUT) amortizes nonlinear reconstruction through indexed loads.
Table R3d. Attention-operator latency (μs). Batch 64, context 4096, 32 query heads, 8 KV heads, head dimension 128, 4-bit codes, and 64-element groups. Full results are reported in Appendix F.5.
| Path | QK | AV | QK+AV |
|---|---|---|---|
| KIVI affine | 775.0 | 451.0 | 1226.0 |
| μ-ZERO LUT · full AV | 634.2 | 520.7 | 1154.9 |
| μ-ZERO LUT · auto AV | 653.9 | 296.6 | 950.6 |
(3) AV scheduling and combined latency. The higher full-AV latency reflects a different reduction schedule. Unlike QK's independent token-position outputs, AV reduces over context; the full path traverses all 4096 tokens per block with repeated LUT staging and synchronization. Automatic AV uses two 2048-token segments with FP32 partial sums and a final reduction, doubling producing blocks while preserving the quantizer.
Relative to Table R3d's KIVI reference, auto AV has 22.5% lower reported QK+AV latency (950.6 versus 1226.0 μs). These operator-level results complement the shared-harness measurements in Table R3c. Together, they demonstrate that direct packed-cache access, reconstruction reuse, and automatic AV scheduling enable efficient execution of μ-ZERO's nonuniform quantizer.
Paper locations: Appendix F.4, Table 28 (shared-harness runtime); Appendix F.5, Table 29 (operator timings, layouts, and scheduling controls).
Response to Reviewer YzE7 (3/4)
R3.W3. Paired uncertainty analysis and additional reasoning evaluation
Weakness: The AIME’25 gains have substantial uncertainty. The evaluation uses 30 problems with repeated responses and seeds. The paper acknowledges that the confidence intervals do not establish statistical superiority, but the limited number of distinct problems makes the relatively small gains difficult to assess.
Question: Could the authors provide a paired uncertainty analysis by problem for AIME’25, or results on another reasoning benchmark, to clarify whether the observed gains are robust?
To assess robustness, we provide problem-paired AIME25 analysis and complementary evidence from a larger LiveCodeBench evaluation.
(1) Paired uncertainty analysis on AIME25. We evaluate Qwen3-8B in thinking mode on all 30 problems with 12 responses each, a 38,912-token output cap, and a 40,960-token context limit. Paired 95% CIs use 50,000 bootstrap resamples of the 30 problem-level accuracy scores.
Table R3e. Paired AIME25 accuracy gains on Qwen3-8B. Gains are μ-ZERO minus the baseline, in percentage points (pp). Paired 95% CIs apply to individual comparisons; adjusted CIs use Bonferroni correction over seven reported pass@1 contrasts. Full results are reported in Appendix E.3.
| Comparison | Gain (pp) [95% CI] | Adjusted 95% CI |
|---|---|---|
| μ-ZERO-2 − KIVI-2 | +6.67 [1.67, 12.50] | [0.00, 14.72] |
| μ-ZERO-3 − TurboQuant-3 | +6.94 [2.22, 11.94] | [0.56, 14.17] |
| μ-ZERO-4 − TurboQuant-4 | +4.72 [0.56, 9.17] | [−0.83, 11.11] |
| μ-ZERO-4 − KIVI-4 | +0.28 [−3.06, 3.89] | [−4.44, 5.00] |
Table R3e shows the largest observed AIME25 gains at 2–3 bits. At 3 bits, μ-ZERO improves pass@1 from 60.28% to 67.22% (+6.94 pp; paired 95% CI [2.22, 11.94]). At 4 bits, μ-ZERO and KIVI have close point estimates (70.28% versus 70.00%), with their paired interval including zero. We additionally assess sensitivity to multiple comparisons using Bonferroni correction over the seven reported pass@1 contrasts: four against quantized baselines and three against BF16. This correction widens the intervals to account for examining several comparisons together. The 3-bit gain retains a positive 95% CI [0.56, 14.17] under this more conservative check, strengthening the evidence for μ-ZERO's advantage over TurboQuant-3.
(2) Additional evaluation on LiveCodeBench. To broaden problem coverage beyond AIME25, we evaluate LiveCodeBench v6 incremental on 175 distinct programming problems using Qwen3-8B and DeepSeek-R1-Distill-Llama-8B (DR-Llama). Each method generates one completion per problem in thinking mode, with a 32,768-token output cap and a 40,960-token context limit. We report pass@1 and problem-paired confidence intervals.
Table R3f. LiveCodeBench pass@1 and paired gains. Scores are percentages; gains are μ-ZERO minus the baseline in pp, with 95% CIs for individual comparisons. TQ denotes TurboQuant. Full results are reported in Appendix E.3.
| Model | Bits | Baseline (%) | μ-ZERO (%) | Gain (pp) [95% CI] |
|---|---|---|---|---|
| Qwen3-8B | 2 | KIVI: 34.29 | 40.00 | +5.71 [0.57, 10.86] |
| Qwen3-8B | 3 | TQ: 44.57 | 49.71 | +5.14 [0.00, 10.29] |
| Qwen3-8B | 4 | KIVI: 46.86 | 52.00 | +5.14 [1.71, 9.14] |
| DR-Llama | 2 | KIVI: 25.14 | 29.71 | +4.57 [−0.57, 9.71] |
| DR-Llama | 3 | TQ: 32.57 | 33.14 | +0.57 [−4.57, 5.71] |
| DR-Llama | 4 | KIVI: 34.86 | 37.14 | +2.29 [−2.29, 6.86] |
Table R3f shows higher μ-ZERO point estimates across both models and all three bit widths in these comparisons. On Qwen3-8B, 4-bit μ-ZERO reaches 52.00% versus KIVI's 46.86%, a gain of 5.14 pp (95% CI [1.71, 9.14]). Accounting for 14 reported contrasts across both models gives a sensitivity interval of [0.00, 10.86] for this gain. This larger programming benchmark complements the mathematical evaluation through broader problem coverage and an additional model family.
Paper locations: Section 5.2; Appendix E.3, Tables 15–16 (AIME25 scores and paired uncertainty) and Tables 17–18 (LiveCodeBench scores and paired uncertainty).
Response to Reviewer YzE7 (1/4)
We thank the reviewer for the constructive feedback and have revised the manuscript accordingly. Revisions are highlighted in blue and annotated with margin labels (e.g., R3.W1 and R3.W2) linking each change to the corresponding concern. Each reply below identifies the relevant manuscript locations.
R3.W1. Independent contribution of adaptive α
To quantify adaptation's independent contribution, we added three matched Qwen3-8B ablations reporting α-selection frequencies and gains over fixed α=2. Our 2- and 4-bit configurations initialize α at 2; prefill statistics select α∈{2,4,16} per layer and cache type for reuse during decoding. We clarified this rule in Sections 4 and 5.1 and Appendix C.7.
The ablations use channel-wise Key grouping, token-wise Value grouping, 64-element groups, and a 128-token residual window. RULER-16k evaluates 2/4-bit caches on 13 tasks × 50 questions (650 total). The 2-bit AIME25 evaluation uses 30 problems × 12 responses in thinking mode, with a 38,912-token output cap and a 40,960-token context limit. Within each benchmark and bit width, only the α policy varies; weights, metadata precision, backend, prompts, and decoding settings are matched (Appendix G.3).
Table R3a. α-selection frequencies on Qwen3-8B. Each observation is the α chosen during prefill for one layer's Key (K) or Value (V) cache in one generation. N is the number of selections: 650 generations × 36 layers for RULER, or 30 problems × 12 responses × 36 layers for AIME25. Percentages are independently rounded. Full results are reported in Appendix G.3.
As shown in Table R3a, mild companding predominates, with stronger companding selected more often for keys than values. α=2 accounts for 86.22–100% of selections. On 4-bit RULER, α=4 accounts for 13.38% of Key selections and 3.00% of Value selections; α=16 accounts for at most 0.41% in any setting. This matches the conservative, cache-specific promotion rule in Appendix C.7: α=2 is retained unless prefill statistics cross a promotion threshold. To assess the downstream benefit of these selections, Table R3b compares adaptive α with fixed α_K=α_V=2 under the same settings.
Table R3b. Downstream effect of adaptive α on Qwen3-8B. Scores are RULER macro scores or AIME25 accuracy (%) on a 0–100 scale; gains are in percentage points (pp). Paired 95% confidence intervals (CIs) use 10,000 bootstrap resamples within RULER tasks or over AIME problems. Full results are reported in Appendix G.3.
The matched comparisons in Table R3b show that selective Key promotion provides a modest independent benefit on 2-bit RULER. The score rises from 78.90 to 80.31 (+1.41 pp; paired 95% CI [0.04, 2.82]). Since Value α remains 2 (Table R3a), the gain is attributable to adaptive Key selection under the matched protocol. The gains on 4-bit RULER and 2-bit AIME25 are +0.12 and +0.56 pp, respectively, with paired intervals including zero. Thus, more frequent promotion at 4 bits does not imply a larger downstream benefit.
The fixed-strength controls in Appendix G.3 further support selective promotion. Table 39 shows that setting Key α to 4 or 16 throughout the model while retaining Value α=2 reduces the 2-bit RULER score by 10.58 or 47.72 pp, respectively. Stronger companding improves near-zero resolution at the cost of coarser tail coverage (Appendix C.4); applying it uniformly can therefore be harmful. These controls support α=2 as a conservative starting point, with selective promotion yielding a modest gain on 2-bit RULER.
Paper locations: Section 4, Appendices C.4 and C.7 (tradeoff and promotion rule); Appendix G.3 (controlled ablations).