acceptodds
Under review as a conference paper at ICLR 2027

D-Quant: Driftable Entropy Coding for KV Cache Quantization

Abstract

The KV cache has become a major bottleneck in deploying LLMs, as its memory footprint grows linearly with sequence length and batch size, imposing substantial pressure on both memory capacity and bandwidth. Among various KV cache compression techniques, quantization is particularly attractive due to its effectiveness and ease of deployment. However, most existing methods rely on fixed-width quantization, where a bit representation is inherently limited to quantization levels. As the bit width decreases, the number of available levels shrinks exponentially, leading to severe information loss and rapid performance degradation. We further observe that fixed-width quantization fails to exploit the highly non-uniform distribution of KV cache. After rotation and normalization, KV values approximately follow a normal distribution, with most values concentrated near the center and only a small fraction appearing in the tails. Nevertheless, fixed-width coding allocates the same number of bits to frequent and rare symbols. Entropy coding naturally exploits such non-uniformity by assigning shorter codewords to frequent symbols and longer ones to rare symbols, substantially reducing the average number of bits required for representation. However, its variable-length output is not suited to highly parallel attention kernels, where efficient dequantization and computation rely on regular memory layouts and fixed-stride accesses. To bridge this gap, we propose D-Quant, a flexible KV cache quantization framework that introduces a drift mechanism to convert entropy-coded representations of each token into fixed-size bitstreams, enabling regular memory access and parallel dequantization within attention kernels. D-Quant also provides greater flexibility than conventional quantization methods, allowing the bit budget and number of quantization levels to be configured more freely rather than being constrained by fixed bit widths and group sizes. Furthermore, since the rotated and normalized KV values closely follow a normal distribution, the probability model for entropy coding can be directly derived from the standard normal distribution, making D-Quant entirely calibration-free. Extensive experiments show that D-Quant preserves near BF16 performance at only bits per element while substantially outperforming existing baselines. D-Quant further achieves up to a reduction in KV cache memory footprint and a improvement in decoding throughput.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache quantization, entropy coding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_E5KcOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes D-Quant, which uses entropy coding to encode each vector in the KV cache in a more efficient way than fixed bit quantization by allowing more bits for rare ones and fewer bits for more common ones. Also, the authors notice that this will lead to variable-length bitstreams, making it difficult to integrate with highly parallel attention kernels. Therefore, they further propose a drift mechanism to convert their entropy-coded representations into fixed-size bitstreams, enabling regular memory access and parallel dequantization within attention kernels. The innovation in both algorithm and engineering lead to much better and almost lossless accuracy in low bit regime, while the save in KV cache and their well-written kernel successfully increase the aggregated throughput up to 3.5x.

Critical strengths
  1. This paper is algorithmically sound. Although the idea of applying entropy coding to compress KV cache is not novel, the author supposes that the bottleneck could be due to the fact that entropy coding produces variable-length representation, thus making it not suitable for modern attention kernel. Therefore, they propose a mechanism to overcome this challenge. The whole idea is natural and makes sense.

  2. The authors also implement efficient kernels to convert the theoretical advantage to actual measurable benefit, making this work technically solid.

Critical weaknesses
  1. The main technical weakness is the lack of a direct ablation on the effect of a fixed-size per-token storage. The paper argues that variable-length entropy coding causes irregular memory access and inefficient decoding, but does not quantify this cost. It remains unknown how slow the decoding will be when using variable-length entropy coding and how faster it can be when using the authors' proposed version. The lack of this clarification will weaken the claim and the necessity of the proposed method.

  2. The authors do not compare with other KV cache compression methods such as KIVI on decoding throughput. It is interesting to see how this method trade the quantization accuracy with throughput when compared with other baseline methods.

Questions for authors
  1. How much decoding overhead does variable-length entropy coding introduce compared with the proposed fixed-size per-token storage?

  2. How does D-Quant compare with existing KV cache compression methods such as KIVI in terms of decoding throughput?

Reviewer_FBiDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes D-Quant, a KV cache quantization framework for Large Language Models (LLMs) that replaces conventional fixed-width quantization with entropy coding. The authors observe that after Hadamard rotation and mean removal, KV cache values closely follow a standard normal distribution, allowing the probability model for entropy coding to be derived analytically without calibration data. To reconcile the variable-length nature of entropy-coded streams with the requirement for regular memory access in parallel attention kernels, D-Quant introduces a "drift" mechanism. This mechanism formulates quantization as a length-constrained rate-distortion optimization problem, solved via Lagrangian relaxation and bisection, which shifts a small fraction of symbols to cheaper codewords so that every token's stream fits exactly into a fixed-size byte container. Evaluated on Qwen3-8B and Llama-3.1-8B across RULER and LongBench-E benchmarks, D-Quant achieves near-BF16 performance at 2.26 bits per element, outperforming existing low-bit baselines (KIVI, QuaRot, TurboQuant, OScaR), particularly at long context lengths (up to 128K). The authors also report up to 7× memory reduction and 3.5× decoding throughput improvement due to increased batch sizes enabled by the compressed cache.

Critical strengths
  1. A principled and elegant bridge between information theory and systems-level constraints. The core contribution—using entropy coding to decouple the number of quantization levels from the bit-width, while using the "drift" mechanism to pack variable-length streams back into fixed-size containers—is conceptually strong. Standard fixed-width quantization wastes bits on frequent central values and starves rare tail values. By formulating this as a constrained rate-distortion optimization (Eq. 5 & 6) and solving it efficiently via warp-level bisection, the authors achieve a flexible representation that respects the strict memory layout requirements of GPU attention kernels. This is highly consequential because it solves a fundamental tension: entropy coding offers better compression but breaks parallel memory access; D-Quant successfully gets the best of both worlds.

  2. Calibration-free design leveraging analytic distributions. Many advanced KV cache quantization methods require calibration datasets or online profiling to estimate distributions, adding deployment friction. D-Quant’s observation that post-rotation/mean-removal KV caches approximate a standard normal distribution—and its subsequent use of Gaussian order statistics to directly construct the rANS probability table—is a significant practical advantage. This makes the method truly plug-and-play. The empirical validation in Appendix C (showing the analytic model incurs only ~0.004 excess bits/value over an empirical model) strongly supports the viability of this design choice.

Critical weaknesses
  1. Encoding overhead and latency are insufficiently analyzed. While the paper demonstrates excellent decoding throughput gains (achieved primarily by enabling larger batch sizes due to memory savings), it largely ignores the computational cost of the encoding step during the prefill phase. The drift mechanism requires a bisection search where distortion is recomputed on the fly for M candidate levels per warp. For long-context applications (e.g., 128K tokens), the prefill phase is already a severe bottleneck. If D-Quant significantly inflates prefill time-to-first-token (TTFT), the overall system utility is compromised. The paper lacks any wall-clock measurement of encoding latency compared to simple min-max INT2 quantization. This is consequential because a method targeting deployment efficiency must not introduce prohibitive overhead in the prefill stage.
Questions for authors
  1. Encoding Latency: Could you provide wall-clock measurements for the prefill (encoding) phase? Specifically, how much additional time does the drift/bisection solver add per token compared to standard affine INT2 quantization (e.g., KIVI or QuaRot)? Does this overhead become a bottleneck for Time-To-First-Token (TTFT) at 64K/128K context lengths?
  2. GQA/MQA Compatibility: How does D-Quant handle Grouped Query Attention (GQA) or Multi-Query Attention (MQA)? Since multiple query heads share a single KV head, does the fixed-container size change, and how does the shared structure affect the drift optimization?
Reviewer_Jwh6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

D-Quant compresses KV caches using entropy coding within fixed-size per-token containers. It combines rotation and key-mean removal with a length-constrained symbol assignment procedure, termed drift, and integrates rANS decoding into attention computation. Experiments on Qwen3-8B and Llama-3.1-8B report near-BF16 long-context performance at approximately 2.26 bits per value, alongside memory and throughput improvements over BF16.

Critical strengths

The long-context accuracy results are strong.

Appendix B provides a convincing controlled comparison using identical preprocessing, grouping, and storage budgets. D-Quant approximately halves output-distribution KL divergence, with improvements on all 27 documents per model and paired confidence intervals.

Critical weaknesses

The systems evaluation does not establish the trade-off against competing compressed-cache methods. Section 5.3 compares only against BF16, and the headline 3.5× throughput gain uses different batch sizes. Although Figure 3 also shows improvements at matched larger batches, the benefit relative to similarly compressed representations remains unclear.

The contribution relative to closely related work needs clearer positioning. TurboQuant already discusses entropy coding with analytically derived symbol probabilities, while KVComp already integrates entropy decoding with attention computation. D-Quant’s distinctive contribution therefore appears to be its fixed-size per-token containers and budget-constrained drift mechanism. A more explicit comparison with these approaches would help establish the significance of this design, particularly the trade-off between regular memory access and the additional distortion introduced by drift.

Questions for authors

How does the ideal code-length constraint guarantee that actual rANS output fits within the container? Please specify the state overhead, normalization parameters, and any overflow checks or fallback.

Which stages are included in the throughput measurements? Are existing results available against a representative quantized baseline under comparable storage and workload conditions?

Relative to KVComp’s fused decoding design, what concrete execution or memory-management benefits do fixed-size containers provide, and what additional cost does drift introduce?

1 comment

Thank you for the clarifications and additional experiments. The accuracy–throughput trade-off is clearer, but the practical advantage at comparable accuracy remains insufficiently established. I will maintain my rating of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.