- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Block-GTQ, a RoPE-aware mixed-precision KV-cache quantization method that allocates key precision across two-dimensional RoPE frequency blocks. The method derives a blockwise attention-logit error bound, estimates block importance using calibration-based Q/K energy, and combines these scores with codec-dependent rate penalties to allocate integer bit widths under a fixed budget. It uses TQ-MSE quantization and implements a packed-cache attention kernel.
Experiments across ten model configurations show substantial reductions in attention-logit error compared with uniform TQ-MSE. The authors also report improvements on NIAH, LongBench, and AIME, together with packed-serving results demonstrating reduced KV-cache memory consumption.
Critical strengths
-
A structured approach to fine-grained key precision allocation. The paper identifies RoPE-aligned two-dimensional blocks as allocation units and formulates a lightweight, label-free importance score. This provides a mathematically motivated alternative to uniform key quantization and enables fine-grained allocation without requiring task labels or expensive gradient computation.
-
Strong empirical improvements over uniform quantization. The experiments cover ten model configurations, including different attention architectures. Block-GTQ reduces logit MAE across all 367 evaluated layers and achieves substantial improvements on NIAH, LongBench, and AIME under tight precision budgets. The inclusion of packed-cache serving experiments further strengthens the practical relevance of the work.
Critical weaknesses
-
Limited methodological novelty beyond existing mixed-precision allocation methods. The main distinction from existing KV-cache mixed-precision methods appears to be the use of RoPE-aligned 2D pairs as precision-allocation units, rather than a fundamentally new allocation framework. Prior methods such as ChanMix, Kitty, and MixKVQ already explore channel-level sensitivity-aware precision allocation, while AATC studies attention-aware rate-distortion allocation. Related RoPE-aware methods, including SpectrumQuant, EliteKV, and RAP, also exploit rotary or frequency structures for KV compression, albeit through different mechanisms. Furthermore, the individual technical components offer limited novelty: Q/K energy is a relatively conventional importance proxy, the rate penalty is adopted from prior TurboQuant analysis, and the greedy marginal-gain optimization follows classical separable resource-allocation principles. Consequently, the main novelty rests on the choice of RoPE-aligned 2D grouping. However, the importance score does not explicitly incorporate RoPE frequencies or relative positions, and the paper does not establish why coupling two RoPE channels should outperform existing, potentially more flexible channel-wise allocation strategies. The independent benefit of the proposed RoPE-specific design therefore remains insufficiently demonstrated.
-
Incomplete experimental evidence for superiority over existing methods and limited practical efficiency gains. The main controlled experiments primarily establish improvements over uniform TQ-MSE. Although comparisons with AATC, Kitty, and PM-KVQ are included on selected tasks, their physical memory budgets and evaluation configurations are not fully matched. Direct comparisons with closely related channel-aware methods such as ChanMix and MixKVQ are also missing. The packed-serving results further raise concerns about practical efficiency. At the same nominal K3V3 rate, Block-GTQ requires 158 bytes/token/KV head compared with 132 for uniform TQ-MSE. More importantly, its decoding latency exceeds FP16 FlashAttention-2 throughout the quality-validated 4K–128K range (9.50 vs. 8.99 ms/step at 128K). The reported 8% latency improvement at 1M is not accompanied by corresponding quality evaluation. While the improved accuracy over uniform TQ-MSE is valuable, the current results do not establish a competitive accuracy–memory–latency Pareto frontier against existing optimized KV quantization methods. Furthermore, packed serving is evaluated only on a single 3B model at batch size one.
Questions for authors
Q1. Methodological novelty and RoPE-specific effectiveness. Given that Q/K energy scoring, rate-distortion-based allocation, and greedy optimization have established precedents, what is the fundamental contribution beyond RoPE-aligned 2D grouping? The existing AATC comparison uses different quantizers and representation spaces, making it difficult to isolate the benefit of RoPE grouping. Could the authors provide controlled comparisons under the same codec and bit budget with (1) channel-wise Q/K energy allocation, (2) AATC-inspired attention-distortion sensitivity allocation, and (3) MixKVQ/ChanMix-inspired channel sensitivity allocation? In particular, what additional benefit comes from enforcing shared precision within each RoPE pair?
Q2. Fair comparisons with existing KV-cache quantization methods. The paper includes comparisons with AATC, Kitty, and PM-KVQ on selected reasoning tasks, but the physical memory budgets and evaluation configurations are not fully matched. Direct comparisons with closely related methods such as ChanMix and MixKVQ are also missing. Could the authors provide more comprehensive comparisons under matched physical KV-cache memory budgets or comparable accuracy, accounting for metadata, protected tokens, and auxiliary storage?
Q3. Practical efficiency and scalability. Block-GTQ incurs higher decoding latency than FP16 throughout the quality-validated 4K–128K range (e.g., 9.50 vs. 8.99 ms/step at 128K), while the reported 8% speedup at 1M lacks corresponding quality validation. Could the authors evaluate accuracy and latency jointly at longer supported contexts and larger batch sizes, and provide a breakdown of the overhead introduced by mixed-rate grouping, metadata, packing, and decoding?