- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes D-Quant, which uses entropy coding to encode each vector in the KV cache in a more efficient way than fixed bit quantization by allowing more bits for rare ones and fewer bits for more common ones. Also, the authors notice that this will lead to variable-length bitstreams, making it difficult to integrate with highly parallel attention kernels. Therefore, they further propose a drift mechanism to convert their entropy-coded representations into fixed-size bitstreams, enabling regular memory access and parallel dequantization within attention kernels. The innovation in both algorithm and engineering lead to much better and almost lossless accuracy in low bit regime, while the save in KV cache and their well-written kernel successfully increase the aggregated throughput up to 3.5x.
Critical strengths
-
This paper is algorithmically sound. Although the idea of applying entropy coding to compress KV cache is not novel, the author supposes that the bottleneck could be due to the fact that entropy coding produces variable-length representation, thus making it not suitable for modern attention kernel. Therefore, they propose a mechanism to overcome this challenge. The whole idea is natural and makes sense.
-
The authors also implement efficient kernels to convert the theoretical advantage to actual measurable benefit, making this work technically solid.
Critical weaknesses
-
The main technical weakness is the lack of a direct ablation on the effect of a fixed-size per-token storage. The paper argues that variable-length entropy coding causes irregular memory access and inefficient decoding, but does not quantify this cost. It remains unknown how slow the decoding will be when using variable-length entropy coding and how faster it can be when using the authors' proposed version. The lack of this clarification will weaken the claim and the necessity of the proposed method.
-
The authors do not compare with other KV cache compression methods such as KIVI on decoding throughput. It is interesting to see how this method trade the quantization accuracy with throughput when compared with other baseline methods.
Questions for authors
-
How much decoding overhead does variable-length entropy coding introduce compared with the proposed fixed-size per-token storage?
-
How does D-Quant compare with existing KV cache compression methods such as KIVI in terms of decoding throughput?
Thank you for the clarifications and additional experiments. The accuracy–throughput trade-off is clearer, but the practical advantage at comparable accuracy remains insufficiently established. I will maintain my rating of 2.