acceptodds
Under review as a conference paper at ICLR 2027

-Zero: Zero-Anchored Log Companding for KV Cache Quantization

Abstract

Long-context LLM inference is increasingly limited by KV cache memory traffic, making low-bit cache quantization essential for efficient deployment. Most calibration-free methods use per-group affine min–max quantization, uniformly allocating levels across the observed range. We show that this rule is poorly aligned with attention: after standard key/value grouping, online cache groups exhibit dense zero-centered mass and sparse, approximately log-tailed extremes, causing uniform grids to waste codes in low-density regions. Moreover, query-key logits are often residuals of cancelling signed products, so small perturbations near zero can induce large relative errors or sign flips despite low entrywise reconstruction error. These observations motivate protecting the zero neighborhood rather than only the range endpoints. We propose -Zero, a zero-anchored piecewise log-companded quantizer that applies a monotone -law transform to the positive and negative sides around an exact zero level. -Zero concentrates resolution near zero, preserves asymmetric side extents, and uses coarser logarithmic spacing in the tails. A lightweight prefill-time rule selects the compander strength per layer and cache type from scale-free statistics and reuses it during decoding. Across reasoning, code generation, and long-context understanding, -Zero achieves the best quantized results across all evaluated 2-4-bit settings against representative scalar, transform/sketch-domain, and mixed-precision baselines. Further evaluations support long-output reasoning and applicability to models up to 70B and selected MQA/MLA architectures. System-level evaluations show up to 2.38 higher throughput and 69% lower peak memory usage than BF16, while extending feasible batch size by at least 3.3 and context length by 4.7. Code available at: https://github.com/muZeroQuant/muZero.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
quantization, kv-cache, compression, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YzE7OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes μ-ZERO, a zero-anchored, piecewise log-companded scalar quantizer for LLM KV caches. It allocates more quantization levels near zero, where the authors argue attention logits are sensitive to perturbations caused by cancellation among signed query–key products. The method retains the standard per-group affine metadata and selects the companding strength α during prefill. Experiments across several model families and tasks show that μ-ZERO improves on KIVI in a number of 2-bit evaluations, including GSM8K, HumanEval, and LongBench. The paper also reports lower memory use and higher decode throughput, though the source of the throughput advantage over KIVI is not isolated.

Critical strengths
  1. The paper gives a clear motivation for nonuniform KV-cache quantization and connects it to attention-logit error under cancellation.
  2. The method retains the usual per-group quantization metadata and does not require learned codebooks or offline calibration.
  3. The 2-bit results are promising across several models and tasks, including reasoning, code generation, and long-context evaluation.
  4. The evaluation covers multiple model scales, long-context settings, MQA and MLA architectures, group-size sensitivity, and kernel measurements.
Critical weaknesses
  1. The contribution of adaptive α selection is not sufficiently quantified. The paper says that the default 2-bit and 4-bit settings retain α=2, while larger values are more common for token-wise keys and very low bit widths. The main downstream results do not make clear how often α=4 or 16 is selected, or how much adaptation improves on fixed α=2.
  2. The speedup over KIVI is difficult to interpret. Table 15 uses different decode harnesses, so the end-to-end comparison does not isolate the source of the gain. In Table 16, μ-ZERO’s QK kernel is faster but its AV kernel is slower than KIVI’s. The paper does not explain what accounts for the QK latency difference.
  3. The AIME’25 gains have substantial uncertainty. The evaluation uses 30 problems with repeated responses and seeds. The paper acknowledges that the confidence intervals do not establish statistical superiority, but the limited number of distinct problems makes the relatively small gains difficult to assess.
  4. The benefit at 4 bits is less clear, and the tail tradeoff needs more analysis. On some LongBench results, μ-ZERO is close to BF16 and other 4-bit baselines. Stronger companding also gives up resolution in the tails to improve accuracy near zero, but the paper does not clearly show how this affects large-magnitude outliers or attention-sink tokens during long-context generation.
Questions for authors
  1. How often does the adaptive rule select α=4 or 16 in each main experiment? Could the authors report downstream results against a fixed α=2 baseline to show the independent contribution of adaptation?
  2. Do the KIVI and μ-ZERO kernels use comparable layouts, tiling, warp mappings, and compiler settings? What accounts for μ-ZERO’s lower QK latency despite its higher AV latency?
  3. Could the authors provide a paired uncertainty analysis by problem for AIME’25, or results on another reasoning benchmark, to clarify whether the observed gains are robust?
  4. On which tasks and models are the 4-bit gains consistent enough to matter in practice? Could the authors also report how companding affects large-magnitude outliers or attention-sink tokens in long-context generation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer YzE7 (1/4)

We thank the reviewer for the constructive feedback and have revised the manuscript accordingly. Revisions are highlighted in blue and annotated with margin labels (e.g., R3.W1 and R3.W2) linking each change to the corresponding concern. Each reply below identifies the relevant manuscript locations.

R3.W1. Independent contribution of adaptive α

Weakness: The contribution of adaptive α selection is not sufficiently quantified. The paper says that the default 2-bit and 4-bit settings retain α=2, while larger values are more common for token-wise keys and very low bit widths. The main downstream results do not make clear how often α=4 or 16 is selected, or how much adaptation improves on fixed α=2.

Question: How often does the adaptive rule select α=4 or 16 in each main experiment? Could the authors report downstream results against a fixed α=2 baseline to show the independent contribution of adaptation?

To quantify adaptation's independent contribution, we added three matched Qwen3-8B ablations reporting α-selection frequencies and gains over fixed α=2. Our 2- and 4-bit configurations initialize α at 2; prefill statistics select α∈{2,4,16} per layer and cache type for reuse during decoding. We clarified this rule in Sections 4 and 5.1 and Appendix C.7.

The ablations use channel-wise Key grouping, token-wise Value grouping, 64-element groups, and a 128-token residual window. RULER-16k evaluates 2/4-bit caches on 13 tasks × 50 questions (650 total). The 2-bit AIME25 evaluation uses 30 problems × 12 responses in thinking mode, with a 38,912-token output cap and a 40,960-token context limit. Within each benchmark and bit width, only the α policy varies; weights, metadata precision, backend, prompts, and decoding settings are matched (Appendix G.3).

Table R3a. α-selection frequencies on Qwen3-8B. Each observation is the α chosen during prefill for one layer's Key (K) or Value (V) cache in one generation. N is the number of selections: 650 generations × 36 layers for RULER, or 30 problems × 12 responses × 36 layers for AIME25. Percentages are independently rounded. Full results are reported in Appendix G.3.

EvaluationBitsCacheα=2 (%)α=4 (%)α=16 (%)N
RULER-16k2K97.772.210.0223,400
RULER-16k2V100.000.000.0023,400
RULER-16k4K86.2213.380.4123,400
RULER-16k4V96.663.000.3423,400
AIME252K97.112.780.1112,960
AIME252V100.000.000.0012,960

As shown in Table R3a, mild companding predominates, with stronger companding selected more often for keys than values. α=2 accounts for 86.22–100% of selections. On 4-bit RULER, α=4 accounts for 13.38% of Key selections and 3.00% of Value selections; α=16 accounts for at most 0.41% in any setting. This matches the conservative, cache-specific promotion rule in Appendix C.7: α=2 is retained unless prefill statistics cross a promotion threshold. To assess the downstream benefit of these selections, Table R3b compares adaptive α with fixed α_K=α_V=2 under the same settings.

Table R3b. Downstream effect of adaptive α on Qwen3-8B. Scores are RULER macro scores or AIME25 accuracy (%) on a 0–100 scale; gains are in percentage points (pp). Paired 95% confidence intervals (CIs) use 10,000 bootstrap resamples within RULER tasks or over AIME problems. Full results are reported in Appendix G.3.

EvaluationBitsFixed α=2Adaptive αGain (pp) [95% CI]
RULER-16k278.9080.31+1.41 [0.04, 2.82]
RULER-16k493.9294.04+0.12 [−0.54, 0.79]
AIME25250.2850.83+0.56 [−1.67, 2.50]

The matched comparisons in Table R3b show that selective Key promotion provides a modest independent benefit on 2-bit RULER. The score rises from 78.90 to 80.31 (+1.41 pp; paired 95% CI [0.04, 2.82]). Since Value α remains 2 (Table R3a), the gain is attributable to adaptive Key selection under the matched protocol. The gains on 4-bit RULER and 2-bit AIME25 are +0.12 and +0.56 pp, respectively, with paired intervals including zero. Thus, more frequent promotion at 4 bits does not imply a larger downstream benefit.

The fixed-strength controls in Appendix G.3 further support selective promotion. Table 39 shows that setting Key α to 4 or 16 throughout the model while retaining Value α=2 reduces the 2-bit RULER score by 10.58 or 47.72 pp, respectively. Stronger companding improves near-zero resolution at the cost of coarser tail coverage (Appendix C.4); applying it uniformly can therefore be harmful. These controls support α=2 as a conservative starting point, with selective promotion yielding a modest gain on 2-bit RULER.

Paper locations: Section 4, Appendices C.4 and C.7 (tradeoff and promotion rule); Appendix G.3 (controlled ablations).

AuthorsOpenReview ↗

Response to Reviewer YzE7 (2/4)

R3.W2. Fair runtime comparison and QK/AV latency analysis

Weakness: The speedup over KIVI is difficult to interpret. Table 15 uses different decode harnesses, so the end-to-end comparison does not isolate the source of the gain. In Table 16, μ-ZERO’s QK kernel is faster but its AV kernel is slower than KIVI’s. The paper does not explain what accounts for the QK latency difference.

Question: Do the KIVI and μ-ZERO kernels use comparable layouts, tiling, warp mappings, and compiler settings? What accounts for μ-ZERO’s lower QK latency despite its higher AV latency?

We address this concern through a shared-harness comparison and an analysis of QK/AV implementation differences.

(1) Shared-harness runtime comparison. We repeated Llama-3.1-8B-Instruct measurements on the same dedicated RTX PRO 6000 Blackwell GPU using a shared Transformers driver, identical inputs, FP16 computation, 64-element groups, native 128-token windows, chunked dense prefill, and 160 warmup/512 measured decode updates. KIVI retains its official attention/cache core through a compatibility adapter.

Table R3c. Shared-harness throughput and decode memory. Throughput is the five-run median in aggregate tokens/s; speedups are relative to same-bit KIVI. Memory is decode-phase peak reserved allocation (GiB). Full results are reported in Appendix F.4.

Context / batchBitsKIVIμ-ZERODecode speedupDecode memory: KIVI / μ-ZERO (GiB)
8k / 162322.94466.091.44×18.52 / 18.46
8k / 164322.74409.251.27×21.49 / 21.00
32k / 162189.40233.581.23×29.08 / 26.66
32k / 164143.39194.851.36×41.23 / 38.52
4k / 12021288.961416.051.10×30.90 / 32.24
4k / 12041003.431213.831.21×41.68 / 40.01

As shown in Table R3c, μ-ZERO delivers 1.10–1.44×/1.21–1.36× KIVI's decode throughput at 2/4 bits. Including prefill, gains remain 1.05–1.24×/1.08–1.16×.

(2) Kernel comparability and QK latency. The implementation audit uses the same GPU and major compiler settings (NVCC 13.1, -O3, --use_fast_math, sm_120) for both methods; native layouts and thread mappings differ. KIVI materializes packed codes and metadata into its operator layout on each call and uses four warps with a 128-element reduction tile. μ-ZERO reads packed data directly, uses two warps over 64-token tiles, and reuses reconstructed Keys across four query heads. These differences explain the lower QK operator latency. The 16-entry lookup table (LUT) amortizes nonlinear reconstruction through indexed loads.

Table R3d. Attention-operator latency (μs). Batch 64, context 4096, 32 query heads, 8 KV heads, head dimension 128, 4-bit codes, and 64-element groups. Full results are reported in Appendix F.5.

PathQKAVQK+AV
KIVI affine775.0451.01226.0
μ-ZERO LUT · full AV634.2520.71154.9
μ-ZERO LUT · auto AV653.9296.6950.6

(3) AV scheduling and combined latency. The higher full-AV latency reflects a different reduction schedule. Unlike QK's independent token-position outputs, AV reduces over context; the full path traverses all 4096 tokens per block with repeated LUT staging and synchronization. Automatic AV uses two 2048-token segments with FP32 partial sums and a final reduction, doubling producing blocks while preserving the quantizer.

Relative to Table R3d's KIVI reference, auto AV has 22.5% lower reported QK+AV latency (950.6 versus 1226.0 μs). These operator-level results complement the shared-harness measurements in Table R3c. Together, they demonstrate that direct packed-cache access, reconstruction reuse, and automatic AV scheduling enable efficient execution of μ-ZERO's nonuniform quantizer.

Paper locations: Appendix F.4, Table 28 (shared-harness runtime); Appendix F.5, Table 29 (operator timings, layouts, and scheduling controls).

AuthorsOpenReview ↗

Response to Reviewer YzE7 (3/4)

R3.W3. Paired uncertainty analysis and additional reasoning evaluation

Weakness: The AIME’25 gains have substantial uncertainty. The evaluation uses 30 problems with repeated responses and seeds. The paper acknowledges that the confidence intervals do not establish statistical superiority, but the limited number of distinct problems makes the relatively small gains difficult to assess.

Question: Could the authors provide a paired uncertainty analysis by problem for AIME’25, or results on another reasoning benchmark, to clarify whether the observed gains are robust?

To assess robustness, we provide problem-paired AIME25 analysis and complementary evidence from a larger LiveCodeBench evaluation.

(1) Paired uncertainty analysis on AIME25. We evaluate Qwen3-8B in thinking mode on all 30 problems with 12 responses each, a 38,912-token output cap, and a 40,960-token context limit. Paired 95% CIs use 50,000 bootstrap resamples of the 30 problem-level accuracy scores.

Table R3e. Paired AIME25 accuracy gains on Qwen3-8B. Gains are μ-ZERO minus the baseline, in percentage points (pp). Paired 95% CIs apply to individual comparisons; adjusted CIs use Bonferroni correction over seven reported pass@1 contrasts. Full results are reported in Appendix E.3.

ComparisonGain (pp) [95% CI]Adjusted 95% CI
μ-ZERO-2 − KIVI-2+6.67 [1.67, 12.50][0.00, 14.72]
μ-ZERO-3 − TurboQuant-3+6.94 [2.22, 11.94][0.56, 14.17]
μ-ZERO-4 − TurboQuant-4+4.72 [0.56, 9.17][−0.83, 11.11]
μ-ZERO-4 − KIVI-4+0.28 [−3.06, 3.89][−4.44, 5.00]

Table R3e shows the largest observed AIME25 gains at 2–3 bits. At 3 bits, μ-ZERO improves pass@1 from 60.28% to 67.22% (+6.94 pp; paired 95% CI [2.22, 11.94]). At 4 bits, μ-ZERO and KIVI have close point estimates (70.28% versus 70.00%), with their paired interval including zero. We additionally assess sensitivity to multiple comparisons using Bonferroni correction over the seven reported pass@1 contrasts: four against quantized baselines and three against BF16. This correction widens the intervals to account for examining several comparisons together. The 3-bit gain retains a positive 95% CI [0.56, 14.17] under this more conservative check, strengthening the evidence for μ-ZERO's advantage over TurboQuant-3.

(2) Additional evaluation on LiveCodeBench. To broaden problem coverage beyond AIME25, we evaluate LiveCodeBench v6 incremental on 175 distinct programming problems using Qwen3-8B and DeepSeek-R1-Distill-Llama-8B (DR-Llama). Each method generates one completion per problem in thinking mode, with a 32,768-token output cap and a 40,960-token context limit. We report pass@1 and problem-paired confidence intervals.

Table R3f. LiveCodeBench pass@1 and paired gains. Scores are percentages; gains are μ-ZERO minus the baseline in pp, with 95% CIs for individual comparisons. TQ denotes TurboQuant. Full results are reported in Appendix E.3.

ModelBitsBaseline (%)μ-ZERO (%)Gain (pp) [95% CI]
Qwen3-8B2KIVI: 34.2940.00+5.71 [0.57, 10.86]
Qwen3-8B3TQ: 44.5749.71+5.14 [0.00, 10.29]
Qwen3-8B4KIVI: 46.8652.00+5.14 [1.71, 9.14]
DR-Llama2KIVI: 25.1429.71+4.57 [−0.57, 9.71]
DR-Llama3TQ: 32.5733.14+0.57 [−4.57, 5.71]
DR-Llama4KIVI: 34.8637.14+2.29 [−2.29, 6.86]

Table R3f shows higher μ-ZERO point estimates across both models and all three bit widths in these comparisons. On Qwen3-8B, 4-bit μ-ZERO reaches 52.00% versus KIVI's 46.86%, a gain of 5.14 pp (95% CI [1.71, 9.14]). Accounting for 14 reported contrasts across both models gives a sensitivity interval of [0.00, 10.86] for this gain. This larger programming benchmark complements the mathematical evaluation through broader problem coverage and an additional model family.

Paper locations: Section 5.2; Appendix E.3, Tables 15–16 (AIME25 scores and paired uncertainty) and Tables 17–18 (LiveCodeBench scores and paired uncertainty).

Reviewer_bxxeOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes µ-Zero, a zero-anchored log-companded quantizer for KV cache compression. By allocating more quantization levels near zero and adapting the companding strength across layers, the method improves low-bit quantization accuracy while preserving the standard scalar KV cache interface.

Critical strengths
  1. Strong empirical performance. The method consistently improves low-bit quantization accuracy over KIVI across multiple models and benchmarks, with particularly meaningful gains at 2 bits. Evaluations on long-context tasks and larger models further support its practical applicability.

  2. Simple and deployment-friendly design. The proposed quantizer preserves the existing scalar KV cache interface without additional per-group metadata. The LUT-based reconstruction enables efficient integration with fused attention kernels.

Critical weaknesses
  1. Insufficient discussion and comparison with LogQuant. The proposed method is closely related to existing logarithmic KV cache quantization approaches, particularly LogQuant. However, the paper does not sufficiently clarify the methodological differences or provide direct experimental comparisons. This makes it difficult to assess the novelty and unique benefits of µ-Zero.

  2. Limited system-level evaluation. The implementation is based on HuggingFace Transformers rather than a production-oriented LLM serving framework such as vLLM. Although the paper reports throughput improvements over KIVI, KIVI's relatively simple implementation does not constitute a strong serving-system baseline. Without integration into a modern serving framework and evaluation under realistic serving workloads, the practical efficiency benefits remain insufficiently demonstrated.

Questions for authors
  1. What are the key methodological differences between µ-Zero and LogQuant? Can the authors provide direct comparisons under matched bit widths, grouping strategies, and memory budgets to establish the advantages of the proposed approach?

  2. Can µ-Zero be integrated into a production-oriented serving framework such as vLLM? How does it affect throughput, latency, and memory usage under realistic serving workloads, particularly with continuous batching and varying context lengths?

  3. How does µ-Zero compare against optimized KV cache quantization implementations within the same serving framework, rather than relying primarily on KIVI as the system-level baseline?

3 comments
AuthorsOpenReview ↗

Response to Reviewer bxxe (1/2)

We thank the reviewer for the constructive feedback and have revised the manuscript accordingly. The added LogQuant comparisons and vLLM integration clarify μ-ZERO's contribution and demonstrate its serving benefits over native low-bit baselines. Revisions are highlighted in blue and annotated with margin labels R2.W1 (LogQuant) and R2.W2 (vLLM); each reply below identifies the relevant manuscript locations.

W1. Comparison with LogQuant

Insufficient discussion and comparison with LogQuant. The proposed method is closely related to existing logarithmic KV cache quantization approaches, particularly LogQuant. However, the paper does not sufficiently clarify the methodological differences or provide direct experimental comparisons. This makes it difficult to assess the novelty and unique benefits of µ-Zero.

We agree that a direct comparison with LogQuant should clarify both the methodological distinction and its empirical consequences. We therefore examine the two methods' designs, standalone accuracy and storage, and composition.

The key methodological distinction concerns where precision is allocated. μ-ZERO introduces a zero-anchored logarithmic scalar reconstruction grid, allocating fine resolution to the attention-sensitive region near zero and coarser resolution to the tails. LogQuant's logarithmic structure instead governs token positions: it retains recent and selected earlier KV vectors in full precision and applies affine quantization to the remainder. μ-ZERO changes reconstruction within each quantization group, preserving the standard scalar cache interface and deriving its zero anchor from the existing per-group minimum and scale without additional per-group metadata.

To evaluate the empirical consequences of this distinction, we added comparisons with LogQuant at 2 and 4 bits on reasoning, code generation, and long-context retrieval. At 2 bits, μ-ZERO exceeds LogQuant on RULER-16k by 9.10 percentage points (pp; paired 95% confidence interval (CI) [8.32, 9.88]) on Qwen3-8B and 31.86 pp ([31.03, 32.71]) on Llama-3.1-8B-Instruct. Its LiveCodeBench pass@1 point estimate is also 6.86 pp higher on DeepSeek-R1-Distill-Llama-8B (DR-Llama). As precision increases to 4 bits, both methods approach BF16 on RULER-16k, while μ-ZERO retains higher generation point estimates: GSM8K exact-match gains are 3.41/3.11 pp on Qwen/Llama, and LiveCodeBench pass@1 is 2.29 pp higher on DR-Llama. Sections 5.2–5.3 summarize these results; Appendices E.2.1, E.3, and E.6 provide the full comparisons and uncertainty analyses.

Appendix F.2 complements these accuracy comparisons with historical-cache accounting in bits per K/V element (BPE), including codes, metadata, protected values, and resident codec state. On RULER-16k, native vLLM layout calculations give μ-ZERO-2 approximately 2.534 BPE on both models, versus LogQuant-2's 2.559/2.565 on Qwen/Llama. The gains thus accompany similar calculated historical-cache storage.

Because μ-ZERO changes scalar reconstruction while LogQuant controls token retention, we also evaluate their composition. We combine μ-ZERO with LogQuant-2's full-precision token-selection policy and compare the resulting configuration with LogQuant-2 on GSM8K and RULER-16k.

Table R2a. Composition gains over LogQuant-2 (pp). Brackets give paired 95% CIs.

EvaluationQwen3-8BLlama-3.1-8B-Instruct
GSM8K+36.92 [32.98, 40.86]+3.94 [0.91, 6.97]
RULER-16k+7.17 [6.35, 7.98]+29.77 [28.89, 30.64]

All four paired intervals exclude zero, supporting μ-ZERO's use as a reconstruction primitive within LogQuant's token-retention policy and extending its utility beyond standalone quantization. Appendices E.2.1 and E.6 report the full composition results.

Paper locations: Appendix D.2 (method distinctions and configurations); Sections 5.2–5.3 and Appendices E.2.1, E.3, and E.6 (comparisons, composition, and uncertainty); Appendices F.1–F.2 (storage accounting).

AuthorsOpenReview ↗

Response to Reviewer bxxe (2/2)

W2. Native vLLM integration and serving evaluation

Limited system-level evaluation. The implementation is based on HuggingFace Transformers rather than a production-oriented LLM serving framework such as vLLM. Although the paper reports throughput improvements over KIVI, KIVI's relatively simple implementation does not constitute a strong serving-system baseline. Without integration into a modern serving framework and evaluation under realistic serving workloads, the practical efficiency benefits remain insufficiently demonstrated.

We added native vLLM integration and comparisons with optimized serving baselines.

(1) Native integration and controlled evaluation. We integrated μ-ZERO into vLLM 0.32.0 with packed paged KV storage, reconstruction within attention, continuous batching, and chunked prefill. We evaluate Qwen3-8B (BF16 weights) against BF16 and vLLM's TurboQuant (TQ)-3/4, INT4, and NVFP4 on one RTX PRO 6000 Blackwell, with a 12 GiB persistent KV/state budget including metadata and protected regions. Weights, graphs, and workspace are excluded. Each baseline retains its quantization format and full-precision cache regions.

Section 5.5 reports vLLM sweeps over 8k–32k inputs and 1–32 concurrent clients; the earlier Transformers/BF16 test is now in Appendix F.3. Appendix F.6.2 adds Poisson workloads with 32k inputs/256 outputs and ShareGPT/EOS requests (2,048-token output cap), each using three shared 128-request traces.

(2) Evidence from 2 to 4 bits. Table R2b compares all serving baselines, with LiveCodeBench (LCB) as a separate quality benchmark.

Table R2b. Quality and serving tradeoffs. Serving: 32k inputs, 256 outputs, 16 concurrent clients. BPE: persistent bits per K/V element, including metadata/protected regions. TTFT/TPOT: time to first token/per output token. LCB: vLLM pass@1 (%) on 175 problems. Bold marks column-best point estimates; † marks the observed Pareto frontier in LCB, capacity, and throughput: no alternative improves one point estimate without reducing another.

MethodCache tokens ↑BPE ↓Output tokens/s ↑p95 TTFT (s) ↓Peak running requestsp95 TPOT (ms) ↓LCB pass@1 ↑
BF1687,29616.00052.9770.37227.4450.29
μ-ZERO-2†519,4882.68963.9230.7315218.1740.00
TQ-3302,8484.61351.9142.949157.5344.57
μ-ZERO-3†371,0723.76562.0430.6311162.5449.71
INT4328,7044.25030.3969.9010311.4746.29
NVFP4†310,4644.50067.3034.849119.8346.29
TQ-4253,8885.50254.1453.257114.9649.14
μ-ZERO-4†288,5764.84161.0638.098117.4752.00

At 2 bits, μ-ZERO offers 5.95× BF16's cache capacity. At 16k/concurrency 32, its running peak is 31 versus BF16's 5, with 21.0% higher throughput and 42.2% lower p95 TTFT (Appendix F.6.1). In Table R2b, it retains 95.0% of NVFP4's throughput with 67.3% more capacity. Its LCB point estimate is lower and TPOT higher than μ-ZERO-3/4.

At 3 bits, μ-ZERO offers 22.5% more capacity than TQ-3 and 17.1–24.0% higher throughput across six scaling settings. Against NVFP4, μ-ZERO-3 provides 19.5% more capacity and higher LCB pass@1 (49.71% versus 46.29%), retaining 92.2% of NVFP4's throughput. Under Poisson arrivals, μ-ZERO-3 reduces p95 TTFT by 41.3% at 32k versus TQ-3; on ShareGPT, it increases request throughput by 4.5% and reduces p95 TPOT by 16.6% (Appendix F.6.2).

At 4 bits, μ-ZERO has the highest LCB point estimate (52.00%), 13.7% more capacity and 8.7–27.7% higher scaling throughput than TQ-4, and approximately twice INT4's 32k throughput. Under Poisson arrivals, μ-ZERO-4 reduces p95 TTFT by 30.1% at 32k versus TQ-4; on ShareGPT, it increases request throughput by 6.2% and reduces p95 TPOT by 10.6%. The μ-ZERO-3/4 scaling throughput advantage holds in all 18 matched comparisons per width (Appendices F.6.1–F.6.2).

(3) Broader applicability and tradeoffs. The integration broadens μ-ZERO's deployment scope to paged, continuously batched serving of long-context and variable-length requests. Table R2b places μ-ZERO on the observed Pareto frontier. NVFP4 achieves the highest throughput in our evaluated settings, using FlashInfer's optimized attention backend with XQA decoding on Blackwell (Section 5.5). TPOT reflects dynamic admission (15/11/8 running requests for μ-ZERO-2/3/4 versus BF16's 2), not matched decode cost. At controlled capacity, μ-ZERO-3/4 use 28.3%/15.3% less time per decode forward than TQ-3/4 (Appendix F.8). LCB rankings describe point estimates; Appendix E.3 reports paired CIs.

Paper locations: Section 5.5 (vLLM sweeps); Appendices F.3 (Transformers/BF16 test), F.6.1–F.6.2 (capacity, memory, serving), F.7 (load pilot), F.8 (controlled diagnostics), and E.3 (LCB quality/uncertainty).

Dear Reviewer,

Thank you again for your valuable review. Your suggestions motivated direct LogQuant comparisons and helped us uncover additional strengths of μ-ZERO, particularly through native vLLM integration. The resulting evaluations place μ-ZERO on the observed quality–capacity–throughput Pareto frontier against optimized serving baselines. This integration substantially strengthens the practical contribution of our work by providing an additional low-bit KV-cache quantization option for modern LLM serving.

As the discussion period draws to a close, we would greatly appreciate your thoughts on the revised manuscript and the additional results. Thank you again for your time and consideration.

Reviewer_ozpkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a KV-cache quantization method based on zero-anchored log companding. The problem is relevant and the implementation appears reasonably complete. However, I am not yet convinced that the empirical evidence supports the paper’s strongest claims. My main concerns are the missing closest baselines, the unclear role of the adaptive parameter, possible test-set tuning, and the fairness of the runtime comparison.

Critical strengths
  • The problem is important for long-context inference.
  • The method is described with enough implementation detail to be potentially reproducible.
  • The paper includes accuracy, ablation, and system-level experiments.
  • The overall presentation is fairly clean.
Critical weaknesses
  • The comparison misses several closely related non-uniform KV-quantization baselines, such as LogQuant, KVQuant, and WKVQuant/QAQ.
  • The claim that the method is best across all 2–4 bit settings appears too strong. In Table 2, several 3-bit results are below KIVI.
  • The contribution of adaptive α is unclear. The released implementation fixes α=2 for 4-bit quantization, while the main 2-bit path also appears close to a fixed setting.
  • The paper does not clearly state whether the threshold was selected without using test-set information. This raises a possible data-leakage concern.
  • The runtime comparison with KIVI uses a different harness, so the reported speed advantage is difficult to interpret. The baselines should use the same timing protocol and hardware conditions.
  • The analysis focuses mainly on the Key cache and does not provide a comparable explanation for the Value cache.
  • Appendix C.3 contains an apparent mathematical error: 1+α < log(1+α) has the wrong direction.
  • The paper claims arbitrary-bit support, but the public configuration accepts only {1, 2, 3, 4, 7, 8} bits. Several 4-bit gains are small, but no confidence intervals or repeated-run statistics are reported.
Questions for authors

NA

6 comments
AuthorsOpenReview ↗

Response to Reviewer ozpk (1/4)

We thank the reviewer for the constructive feedback and have revised the manuscript accordingly. Revisions are highlighted in blue and annotated with margin labels (e.g., R1.W1 and R1.W2) linking each change to the corresponding concern. Each reply below identifies the relevant manuscript locations.

W1. Closely related baselines

The comparison misses several closely related non-uniform KV-quantization baselines, such as LogQuant, KVQuant, and WKVQuant/QAQ.

We agree that these comparisons are needed to position μ-ZERO. We have added LogQuant-2/4 and KVQuant-3/4 on GSM8K, RULER-16k, and LiveCodeBench, and QAQ (τ=1,3) on GSM8K. KVQuant is evaluated on Llama-family checkpoints because the official release lacks Qwen3 integration. We did not identify an official public WKVQuant implementation and therefore do not report a full empirical comparison.

Methodological distinction. μ-ZERO defines a closed-form, zero-anchored scalar reconstruction grid, retaining packed codes and per-group minimum/scale metadata without offline codebook calibration. LogQuant keeps the K/V states of recent tokens and selected earlier tokens in full precision according to a logarithmic schedule, while applying affine quantization to the remaining cache. QAQ adapts token-wise K/V precision; KVQuant uses calibrated nonuniform codebooks and outlier protection. Table R1a compares these approaches on reasoning, long-context retrieval, and code generation.

Table R1a. Accuracy across tasks and models. Qwen/Llama denote Qwen3-8B/Llama-3.1-8B-Instruct; DR-Llama denotes DeepSeek-R1-Distill-Llama-8B. GSM8K reports original-script EM (%) on 1,319 problems; RULER-16k reports the macro score (0–100) on 6,500 examples; LiveCodeBench reports pass@1 (%) on 175 problems. Entries are point estimates.

BitsMethodGSM8K · QwenGSM8K · LlamaRULER-16k · QwenRULER-16k · LlamaLiveCodeBench · DR-Llama
16BF1688.6375.5192.5093.4941.71
2LogQuant43.8271.5776.7157.6722.86
2μ-ZERO85.7571.1185.8189.5429.71
3KVQuant—58.91—93.0932.00
3μ-ZERO88.8676.6592.4393.1633.14
4LogQuant84.8472.7892.4393.5134.86
4KVQuant—65.96—93.5234.86
4μ-ZERO88.2575.8992.5093.5337.14
1–4 (adaptive)QAQ† (τ=1)32.0732.60———
1–4 (adaptive)QAQ† (τ=3)23.2810.99———

† QAQ assigns 1–4-bit prompt codes and retains generated KV in BF16. Its results describe the two evaluated configurations, with coverage limited to GSM8K.

Main findings. μ-ZERO provides its largest gains under aggressive 2-bit compression, improving both long-context retrieval and generation quality. On RULER-16k, it exceeds LogQuant-2 by 9.10 percentage points (pp) on Qwen (paired 95% confidence interval (CI) [8.32, 9.88]) and 31.86 pp on Llama ([31.03, 32.71]). Qwen GSM8K improves by 41.93 pp, and DR-Llama LiveCodeBench by 6.86 pp. Llama GSM8K remains close to LogQuant-2 (71.11% versus 71.57%). At 3–4 bits, μ-ZERO combines higher reasoning and code-generation point estimates with retrieval scores close to BF16. At 3 bits, it exceeds KVQuant by 17.74 pp on Llama GSM8K and 1.14 pp on LiveCodeBench. At 4 bits, GSM8K gains are 3.41/3.11 pp over LogQuant on Qwen/Llama and 9.93 pp over KVQuant on Llama; LiveCodeBench improves by 2.29 pp over both baselines. Meanwhile, 4-bit RULER scores reach 92.50/93.53 on Qwen/Llama, compared with BF16's 92.50/93.49. Paired differences from BF16 are −0.002 pp (95% CI [−0.22, 0.22]) and +0.04 pp ([−0.08, 0.16]), respectively.

Composition with LogQuant. μ-ZERO can also serve as the scalar reconstruction primitive within a policy that allocates precision across tokens. We test this compatibility by combining μ-ZERO with LogQuant-2's cache policy. Table R1b reports gains over LogQuant-2 under both GSM8K scoring rules and on RULER-16k.

Table R1b. Composition gains over LogQuant-2 (pp). Brackets give paired 95% CIs.

MetricQwen3-8BLlama-3.1-8B-Instruct
GSM8K+36.92 [32.98, 40.86]+3.94 [0.91, 6.97]
RULER-16k · macro score+7.17 [6.35, 7.98]+29.77 [28.89, 30.64]

Paper locations: Sections 2 and 4 (method distinctions); Sections 5.2–5.3 and Appendices E.2.1, E.3, and E.6 (results, uncertainty, and composition); Appendices D.2–D.3 and F.1–F.2 (configurations and storage).

AuthorsOpenReview ↗

Response to Reviewer ozpk (2/4)

W2. Scope of the accuracy claim

The claim that the method is best across all 2–4 bit settings appears too strong. In Table 2, several 3-bit results are below KIVI.

We respectfully clarify that Table 3 (submitted Table 2) reports KIVI at 2/4 bits only, with no KIVI-3 column. The cited μ-ZERO-3 comparison therefore concerns higher-precision KIVI-4. Our claim concerns rankings within each nominal bit-width group; μ-ZERO leads the highlighted 3-bit group on all four models.

Added GSM8K, RULER-16k, AIME25, and LiveCodeBench evaluations likewise give μ-ZERO-3 the highest 3-bit point estimates among the evaluated baselines, including KVQuant-3 and vLLM-native TurboQuant-3. On AIME25, the gain over TurboQuant-3 is 6.94 pp (paired 95% CI [0.56, 14.17]). On Qwen3-8B LiveCodeBench, μ-ZERO-3 scores 49.71%, also above the evaluated 4-bit native presets: TurboQuant 49.14% and INT4/NVFP4 46.29%.

Paper locations: Sections 5.2–5.3, Tables 1–4 (quality comparisons); Appendices E.2.1, E.3, and E.6 (extended results and uncertainty).

W3. Independent contribution of adaptive α

The contribution of adaptive α is unclear. The released implementation fixes α=2 for 4-bit quantization, while the main 2-bit path also appears close to a fixed setting.

Our main results use layer-wise adaptive α, initialized at 2. Prefill statistics determine α∈{2,4,16} separately for each layer and cache type; the selected value is reused during decoding. To address this concern, we added matched Qwen3-8B ablations to quantify adaptation's independent contribution. RULER-16k uses 13 tasks × 50 questions; AIME25 uses 30 problems × 12 responses. Within each benchmark/bit-width setting, only the α policy varies; weights, cache configuration, backend, prompts, and decoding are matched. Table R1c compares adaptive selection with fixed α=2 for both K and V.

Table R1c. Independent contribution of adaptive α. Scores are RULER macro scores or AIME25 accuracy (%); gains are in percentage points (pp). Brackets give paired 95% CIs from 10,000 bootstrap resamples within RULER tasks or over AIME problems.

EvaluationBitsFixed α=2Adaptive αGain [95% CI]
RULER-16k278.9080.31+1.41 [0.04, 2.82]
RULER-16k493.9294.04+0.12 [−0.54, 0.79]
AIME25250.2850.83+0.56 [−1.67, 2.50]

The clearest independent benefit occurs on 2-bit RULER: adaptive α increases the score from 78.90 to 80.31 (+1.41 pp; paired 95% CI [0.04, 2.82]). The 4-bit RULER and 2-bit AIME25 differences remain statistically unresolved, with paired intervals spanning zero. To examine how often adaptation retains the initial value or selects α=4/16, Table R1d reports K/V selection frequencies from the same adaptive evaluations.

Table R1d. α-selection frequencies. K/V entries list percentages for α=2/4/16, respectively. N counts layer-wise decisions per cache type. Percentages are independently rounded.

EvaluationBitsK: α=2 / 4 / 16 (%)V: α=2 / 4 / 16 (%)N per cache type
RULER-16k297.77 / 2.21 / 0.02100.00 / 0.00 / 0.0023,400
RULER-16k486.22 / 13.38 / 0.4196.66 / 3.00 / 0.3423,400
AIME25297.11 / 2.78 / 0.11100.00 / 0.00 / 0.0012,960

In the added 4-bit RULER evaluation, α=4 is selected for 13.38% of Key and 3.00% of Value decisions, demonstrating promotion beyond the initial α=2. The additional fixed-α controls support α=2 as a conservative starting point: on 2-bit RULER, fixing Key α at 4/16 while retaining Value α=2 lowers scores by 10.58/47.72 pp relative to fixed (2,2). Appendix G.3 provides the complete K/V controls.

Paper locations: Appendix C.7 (promotion rule); Appendix G.3, Tables 39–40 (new evaluations).

W4. Threshold calibration and test independence

The paper does not clearly state whether the threshold was selected without using test-set information. This raises a possible data-leakage concern.

We thank the reviewer for pointing out this ambiguity. Promotion thresholds were selected through held-out layer-wise attention search on WikiText development data, disjoint from downstream evaluation examples, and frozen before benchmarking. Evaluation prompts, reference answers, and scores were excluded from threshold selection and tuning. At inference, α is selected using only the current request's prefill statistics under the frozen thresholds. We have added these clarifications to Appendix C.7 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer ozpk (3/4)

W5. Fairness of the runtime comparison

The runtime comparison with KIVI uses a different harness, so the reported speed advantage is difficult to interpret. The baselines should use the same timing protocol and hardware conditions.

We thank the reviewer for raising this concern. We repeated the Llama-3.1-8B-Instruct comparison using a shared Transformers harness on the same dedicated RTX PRO 6000 Blackwell GPU. Both methods share synthetic inputs, FP16 computation, a quantization group size of 64 elements, native 128-token window parameters, chunked dense prefill, and 160 warmup and 512 measured decode updates. KIVI retains its official fused attention/cache core via a compatibility adapter. Table R5 reports decode throughput and memory.

Table R5. Shared-harness comparison. Throughput is the five-run median in aggregate tokens/s; memory is peak reserved GPU memory during decoding (GiB). Speedups are relative to same-bit KIVI.

Context / batchBitsKIVIμ-ZERODecode speedupDecode memory: KIVI / μ-ZERO (GiB)
8k / 162322.94466.091.44×18.52 / 18.46
8k / 164322.74409.251.27×21.49 / 21.00
32k / 162189.40233.581.23×29.08 / 26.66
32k / 164143.39194.851.36×41.23 / 38.52
4k / 12021288.961416.051.10×30.90 / 32.24
4k / 12041003.431213.831.21×41.68 / 40.01

μ-ZERO delivers 1.10–1.44×/1.21–1.36× KIVI decode throughput at 2/4 bits across all tested settings. The advantage persists with prefill included (1.05–1.24×/1.08–1.16×). Peak reserved decode memory is 2.27–6.58% lower at 4 bits across all settings and 0.32–8.32% lower at 2 bits with batch 16.

Paper locations: Appendix F.4, Table 28.

W6. Value-cache explanation

The analysis focuses mainly on the Key cache and does not provide a comparable explanation for the Value cache.

We thank the reviewer for suggesting a fuller Value-cache explanation. Our main-text analysis emphasizes Keys because their quantization errors perturb query–key logits and can redirect attention across tokens, whereas Value errors perturb the vectors aggregated under fixed attention weights. This asymmetry can make Keys more sensitive to quantization and Values comparatively more robust through attention-weighted averaging and partial error cancellation. The original submission also analyzed Value distributions (Appendix A) and their attention-dependent sensitivity (Appendix C.6). We have now expanded the Value-specific explanation and validation in Appendix G.4.

For fixed attention probabilities AA, the Value-only output perturbation is exactly δoV=A(V^−V)\delta o_V=A(\widehat V-V). Its magnitude depends on attention weights and whether errors across tokens reinforce or cancel; attention-output error is therefore the relevant diagnostic. Appendix A shows that Value groups also concentrate near zero, motivating finer reconstruction there. We test this allocation directly through Value-only and masked-error replay. Table R6 reports relative attention-output MSE across prefill and decoding on 32 LongBench-derived prompts per model (4k/16k/32k contexts). Affine min–max and μ-ZERO at the default α=2 use the same reference queries, keys, and attention weights, with matched 64-element groups, metadata, and full-precision regions.

Table R6. Relative attention-output MSE under Value-only quantization. Reductions are relative to affine min–max; brackets give paired 95% CIs from 10,000 prompt-bootstrap resamples stratified by context length.

ModelValue bitsAffine min–max (V-only)μ-ZERO (V-only)MSE reduction (%) [95% CI]
Qwen3-8B20.1019040.03674363.94 [62.68, 65.21]
Qwen3-8B40.0040580.00258336.35 [35.27, 37.41]
Llama-3.1-8B-Instruct20.1883280.07407960.66 [59.55, 61.74]
Llama-3.1-8B-Instruct40.0072900.00437140.04 [39.53, 40.56]

Value-only output error falls 60.66–63.94% at 2 bits and 36.35–40.04% at 4 bits, with positive paired CIs in all four settings. In 2-bit masked comparisons with affine, center-only output MSE (smallest 50% of Value coordinates) falls 90.0–91.5%, and error on the highest-attention 1% of tokens falls 65.1–71.8%. Tail-only error (largest 6.25% of coordinates) rises 18.7–35.2%. Together, these results support finer near-zero precision for Values while quantifying its tail tradeoff.

As complementary downstream evidence, Appendix G.3's separate K/V controls show that increasing only Key or only Value α from 2 to 4 shifts Qwen3-8B's 2-bit RULER score by −10.58 and −2.07 pp, respectively. Values are more tolerant of this companding change in this setting.

Paper locations: Appendices A and C.6 (original Value analysis); G.3 (K/V controls); G.4, Tables 41–42 (Value-only diagnostics).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful assessment and constructive feedback. In response, we have substantially expanded the experimental evaluation with additional baseline comparisons, controlled ablations, and native vLLM serving studies, and clarified the method and experimental protocols. Changes in the revised manuscript are highlighted in blue, with margin notes linking them to the relevant reviewer comments. We address the reviewers' principal concerns in the following four points.

(1) Closely related baselines and methodological distinction. [R1.W1; R2.W1] We clarify μ-ZERO's contribution as a scalar reconstruction primitive, distinguishing it from methods based on token retention or calibrated codebooks. We added LogQuant, KVQuant, and supplementary QAQ comparisons in their supported settings. On RULER-16k, μ-ZERO-2 exceeds LogQuant-2 by 9.10 and 31.86 percentage points (pp) on Qwen3-8B and Llama-3.1-8B-Instruct, respectively. We also integrate μ-ZERO's scalar quantizer with LogQuant's token-retention design, yielding higher GSM8K and RULER-16k scores than LogQuant-2 (Sections 5.2–5.3; Appendices D.2, E.2.1, and E.6).

(2) Runtime fairness and practical serving performance. [R1.W5; R2.W2; R3.W2] We repeated KIVI comparisons using the same GPU, harness, and timing protocol, and added QK/AV analyses to clarify implementation differences. We also integrated μ-ZERO into vLLM and compared it with TurboQuant-3/4, INT4, and NVFP4 under a common 12 GiB persistent KV/state budget. Evaluations cover continuous batching with 8k–32k inputs and varying concurrency, plus Poisson arrivals and ShareGPT/EOS workloads. These evaluations demonstrate favorable quality–capacity–throughput tradeoffs for LLM serving. Relative to BF16, 3-bit μ-ZERO provides 4.25× the KV-cache capacity and up to 18.2% higher output throughput under the shared memory budget (Section 5.5; Appendices F.4–F.6 and F.8).

(3) Controlled ablations and attention fidelity. [R1.W3; R1.W4; R1.W6; R3.W1; R3.W4] Matched comparisons against fixed α=2 quantify adaptation's independent contribution; selection frequencies show that mild companding predominates. Selective promotion yields a modest gain on 2-bit RULER. Controlled analyses of the Value cache, tail reconstruction, and attention-sink candidates show that finer near-zero resolution can improve attention-output fidelity despite higher tail-coordinate error. We also document that promotion thresholds were selected on disjoint WikiText development data and frozen before benchmarking (Sections 4 and 5.4; Appendices C.7 and G.3–G.4).

(4) Broader workloads and quantified uncertainty. [R1.W2; R1.W8; R3.W3; R3.W4] We added AIME25 evaluation with extended generation budgets on Qwen3-8B (all 30 problems, 12 responses each) and LiveCodeBench evaluation on 175 problems across two model families. Paired 95% confidence intervals (CIs) and multiple-comparison sensitivity analyses quantify uncertainty in the reported gains. The 3-bit AIME25 gain over TurboQuant-3 retains a positive CI under Bonferroni correction. On Qwen3-8B, μ-ZERO-4 improves LiveCodeBench pass@1 over KIVI-4 by 5.14 pp (pointwise 95% CI [1.71, 9.14]). These results identify useful 4-bit gains while clarifying their dependence on the task and baseline (Sections 5.2–5.3; Appendices E.2.1 and E.3).

Our separate responses to each reviewer provide detailed results, experimental protocols, and references to the relevant manuscript sections.

We thank the reviewers again for the comments and suggestions that motivated these additional experiments and helped us better substantiate μ-ZERO's strengths and practical value. We would be grateful if the reviewers could revisit their assessment in light of the expanded evidence and revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.