- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Dual-QK, which aims to simultaneously achieve low-bit KV cache quantization and dynamic channel pruning. The authors point out that while rotation-based quantization usually mitigates Key outliers by flattening their distribution, the shared rotation can also disperse Query energy, reducing the effectiveness of channel pruning based on Query magnitudes. To address this, Dual-QK uses paired non-orthogonal transforms that preserve the attention dot product: it applies partial whitening to the Key, and then applies PCA to the reversely compensated Query, making Key scales more balanced and Query energy more concentrated. During inference, the method dynamically selects the Key channels to read based on the channel energy of the current GQA Query, while retaining the full cache. The system also combines INT2 quantization, BF16 protection for the leading Key channel, diagonal scale calibration, and bucket-relative RoPE to improve quantization accuracy and long-context adaptability. The paper evaluates generation quality, long-context retrieval, and decoding throughput on four models. Under a uniform setting of 40% channel pruning, Dual-QK generally maintains accuracy better than the compared quantization baselines and achieves speedups by reducing Key cache reads. Its main contribution is a joint algorithm and system design targeting the compatibility of quantization and channel pruning.
Critical strengths
S1. The motivation of the paper is relatively novel and intuitive.
S2. The practical performance of the method is good.
Critical weaknesses
W1. Limited technical novelty: The method is essentially a combination of several existing techniques, which can be broken down as follows: 1) Using different transforms for Q and K (FPTQuant 2025); 2) Applying a fractional power transform to the Key eigenvalues to reduce variance, which is a well-established classical operation; 3) Applying PCA to the Query, which is also a standard classical operation; 4) Channel pruning, token protection, etc., for which the authors explicitly stated in the text that they referred to papers like OSCAR; 5) Bucket RoPE appears to be the only original component (no prior work found), but its contribution is not very fundamental. Therefore, the overall novelty of this work is questionable.
W2. Misleading experimental setup in the main table: The main comparison applies query-channel pruning uniformly to all SOTA baselines. However, methods like OSCAR are inherently not robust to this level of pruning, causing their scores to plummet. As shown in the ablation study (Table 4), when pruning is removed, OSCAR comprehensively outperforms this method on Qwen models. Comparing under a setting where baselines are forced to fail is a trick to make the proposed method look better.
W3. Insufficient experimental evaluation: The overall experiments are somewhat thin. Apart from the main table, the subsequent tables almost exclusively compare with OSCAR, missing several relevant baselines in the literature. For example, the authors could include comparisons with methods like Kitty and MixKVQ.
Questions for authors
Please explain the weaknesses above.