$\mu$-Zero: Zero-Anchored Log Companding for KV Cache Quantization

Long-context LLM inference is increasingly limited by KV cache memory traffic, making low-bit cache quantization essential for efficient deployment. Most calibration-free methods use per-group affine min–max quantization, uniformly allocating levels across the observed range. We show that this rule is poorly aligned with attention: after standard key/value grouping, online cache groups exhibit dense zero-centered mass and sparse, approximately log-tailed extremes, causing uniform grids to waste codes in low-density regions. Moreover, query-key logits are often residuals of cancelling signed products, so small perturbations near zero can induce large relative errors or sign flips despite low entrywise reconstruction error. These observations motivate protecting the zero neighborhood rather than only the range endpoints. We propose $\mu$-Zero, a zero-anchored piecewise log-companded quantizer that applies a monotone $\mu$-law transform to the positive and negative sides around an exact zero level. $\mu$-Zero concentrates resolution near zero, preserves asymmetric side extents, and uses coarser logarithmic spacing in the tails. A lightweight prefill-time rule selects the compander strength per layer and cache type from scale-free statistics and reuses it during decoding. Across reasoning, code generation, and long-context understanding, $\mu$-Zero achieves the best quantized results across all evaluated 2-4-bit settings against representative scalar, transform/sketch-domain, and mixed-precision baselines. Further evaluations support long-output reasoning and applicability to models up to 70B and selected MQA/MLA architectures. System-level evaluations show up to 2.38 $\times$ higher throughput and 69% lower peak memory usage than BF16, while extending feasible batch size by at least 3.3$\times$ and context length by 4.7$\times$. Code available at: https://github.com/muZeroQuant/muZero.