acceptodds
Under review as a conference paper at ICLR 2027

Block-GTQ: RoPE-Block Rate Allocation for KV-Cache Quantization

Abstract

Larger models and longer contexts make the KV cache a memory and bandwidth bottleneck. We introduce Block-GTQ, a framework that uses the two-dimensional frequency blocks of rotary position embeddings (RoPE) to allocate cached-key precision. A blockwise logit-error bound motivates a label-free query–key energy score, which we combine with codec-dependent rate penalties. Under equal-cost budgets and diminishing marginal gains, largest-marginal-gain allocation exactly minimizes the separable integer proxy. Our TurboQuant-MSE (TQ-MSE) instantiation groups equal-rate blocks and supports attention over packed caches without materializing a decoded fp16 KV cache. Across ten model/path configurations, K-only diagnostics reduce layer-averaged logit mean absolute error by about – relative to uniform TQ-MSE, improving all layers at both 2- and 3-bit average logical key rates. At K2V2 (2 average logical bits per K/V coordinate) on Llama-3.1-8B-Instruct, Block-GTQ raises the six-task needle-in-a-haystack (NIAH) average from to and LongBench-EN from to ; K2V2 gains also hold on the tested 32B and 70B models. On DeepSeek-R1-Distill-Qwen-7B at K3V2, without fp16 sink or recent-token buffers, it achieves avg@8 on AIME 2024/2025 versus fp16’s , where uniform TQ-MSE scores zero on both. On Qwen2.5-3B-Instruct, packed K3V3 serving on one H800 reduces resident KV footprint by a factor of and retains perplexity close to fp16 at 128K, while uniform TQ-MSE degrades severely. At a 1M-token context, Block-GTQ reduces decode latency by 8.0% relative to FP16 FlashAttention-2.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV-cache quantization, efficient LLM inference, long-context language models, mixed-precision bit allocation, rotary position embeddings (RoPE)

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CjYaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Block-GTQ, a RoPE-aware mixed-precision KV-cache quantization method that allocates key precision across two-dimensional RoPE frequency blocks. The method derives a blockwise attention-logit error bound, estimates block importance using calibration-based Q/K energy, and combines these scores with codec-dependent rate penalties to allocate integer bit widths under a fixed budget. It uses TQ-MSE quantization and implements a packed-cache attention kernel.

Experiments across ten model configurations show substantial reductions in attention-logit error compared with uniform TQ-MSE. The authors also report improvements on NIAH, LongBench, and AIME, together with packed-serving results demonstrating reduced KV-cache memory consumption.

Critical strengths
  1. A structured approach to fine-grained key precision allocation. The paper identifies RoPE-aligned two-dimensional blocks as allocation units and formulates a lightweight, label-free importance score. This provides a mathematically motivated alternative to uniform key quantization and enables fine-grained allocation without requiring task labels or expensive gradient computation.

  2. Strong empirical improvements over uniform quantization. The experiments cover ten model configurations, including different attention architectures. Block-GTQ reduces logit MAE across all 367 evaluated layers and achieves substantial improvements on NIAH, LongBench, and AIME under tight precision budgets. The inclusion of packed-cache serving experiments further strengthens the practical relevance of the work.

Critical weaknesses
  1. Limited methodological novelty beyond existing mixed-precision allocation methods. The main distinction from existing KV-cache mixed-precision methods appears to be the use of RoPE-aligned 2D pairs as precision-allocation units, rather than a fundamentally new allocation framework. Prior methods such as ChanMix, Kitty, and MixKVQ already explore channel-level sensitivity-aware precision allocation, while AATC studies attention-aware rate-distortion allocation. Related RoPE-aware methods, including SpectrumQuant, EliteKV, and RAP, also exploit rotary or frequency structures for KV compression, albeit through different mechanisms. Furthermore, the individual technical components offer limited novelty: Q/K energy is a relatively conventional importance proxy, the 4−b4^{-b} rate penalty is adopted from prior TurboQuant analysis, and the greedy marginal-gain optimization follows classical separable resource-allocation principles. Consequently, the main novelty rests on the choice of RoPE-aligned 2D grouping. However, the importance score does not explicitly incorporate RoPE frequencies or relative positions, and the paper does not establish why coupling two RoPE channels should outperform existing, potentially more flexible channel-wise allocation strategies. The independent benefit of the proposed RoPE-specific design therefore remains insufficiently demonstrated.

  2. Incomplete experimental evidence for superiority over existing methods and limited practical efficiency gains. The main controlled experiments primarily establish improvements over uniform TQ-MSE. Although comparisons with AATC, Kitty, and PM-KVQ are included on selected tasks, their physical memory budgets and evaluation configurations are not fully matched. Direct comparisons with closely related channel-aware methods such as ChanMix and MixKVQ are also missing. The packed-serving results further raise concerns about practical efficiency. At the same nominal K3V3 rate, Block-GTQ requires 158 bytes/token/KV head compared with 132 for uniform TQ-MSE. More importantly, its decoding latency exceeds FP16 FlashAttention-2 throughout the quality-validated 4K–128K range (9.50 vs. 8.99 ms/step at 128K). The reported 8% latency improvement at 1M is not accompanied by corresponding quality evaluation. While the improved accuracy over uniform TQ-MSE is valuable, the current results do not establish a competitive accuracy–memory–latency Pareto frontier against existing optimized KV quantization methods. Furthermore, packed serving is evaluated only on a single 3B model at batch size one.

Questions for authors

Q1. Methodological novelty and RoPE-specific effectiveness. Given that Q/K energy scoring, rate-distortion-based allocation, and greedy optimization have established precedents, what is the fundamental contribution beyond RoPE-aligned 2D grouping? The existing AATC comparison uses different quantizers and representation spaces, making it difficult to isolate the benefit of RoPE grouping. Could the authors provide controlled comparisons under the same codec and bit budget with (1) channel-wise Q/K energy allocation, (2) AATC-inspired attention-distortion sensitivity allocation, and (3) MixKVQ/ChanMix-inspired channel sensitivity allocation? In particular, what additional benefit comes from enforcing shared precision within each RoPE pair?

Q2. Fair comparisons with existing KV-cache quantization methods. The paper includes comparisons with AATC, Kitty, and PM-KVQ on selected reasoning tasks, but the physical memory budgets and evaluation configurations are not fully matched. Direct comparisons with closely related methods such as ChanMix and MixKVQ are also missing. Could the authors provide more comprehensive comparisons under matched physical KV-cache memory budgets or comparable accuracy, accounting for metadata, protected tokens, and auxiliary storage?

Q3. Practical efficiency and scalability. Block-GTQ incurs higher decoding latency than FP16 throughout the quality-validated 4K–128K range (e.g., 9.50 vs. 8.99 ms/step at 128K), while the reported 8% speedup at 1M lacks corresponding quality validation. Could the authors evaluate accuracy and latency jointly at longer supported contexts and larger batch sizes, and provide a breakdown of the overhead introduced by mixed-rate grouping, metadata, packing, and decoding?

Reviewer_PkpFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper allocates key-cache bits per RoPE frequency pair instead of uniformly. Each pair gets a score from calibration data: the average of its mean squared query norm and its mean squared key norm. Bit widths are then chosen greedily under a fixed per-head budget. The penalty, taken from the TurboQuant analysis, shrinks 4x for each extra bit. Pairs with the same width are grouped and quantized with TurboQuant-MSE, and values stay uniform. There is also a packed CUDA kernel. Evaluation covers NIAH, LongBench, AIME and a deployment study on Qwen2.5-3B.

Critical strengths
  1. Simple and cheap. The schedule is static and needs no labels. The RoPE pair is a sensible unit because the rotation doesn't change a pair's norm.
  2. Large gains over uniform TQ-MSE at 2-3 bits across many architectures, including partial RoPE and MLA. The kernel is real rather than simulated quantization.
Critical weaknesses
  1. Novelty is limited. Lemma 1 says the logit error is at most the sum, over pairs, of the query norm times the key error norm. That is just the triangle inequality plus Cauchy-Schwarz. Theorem 2 is classical greedy marginal allocation (Fox, 1966). Mixed key precision from Q/K statistics has been done before (MixKVQ, Kitty, AATC), and the paper should say more clearly what's new.
  2. Memory isn't matched. The per-segment fp16 norms aren't counted in the bit rate. Table 33 shows 158 B vs 132 B per token per KV head. Uniform K4V4 also fits in 132 B, so K4V4 is the fair comparison.
  3. Deployment is oversold. Block-GTQ is slower than FP16 at every length up to 256K (1.26x FP16 latency at 4K, 1.06x at 128K). It only wins at 512K and 1M, beyond the 128K where quality was checked. Everything is batch size 1, with no FP8 KV baseline.
  4. Some numbers look off.
    • Qwen fp16 NIAH is exactly 72.22 at every length. I suspect the 30-token cap cuts off multi-answer outputs. • KIVI-ScaleOnly barely improves from 2 to 4 bits (KL goes from 1.97 to 1.79), which looks like a bug. • There are no error bars on AIME.
Questions for authors
  1. What is the effective key rate once norms are included?
  2. How does uniform K4V4 do at the same 132 B?
  3. Can you add the top-pairs and per-channel-scaled baselines at matched total bits?
  4. Does Qwen NIAH change with a longer generation cap?
Reviewer_8jzKOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BLOCK-GTQ, which allocates key-cache quantization rates across two-dimensional RoPE frequency blocks. Its label-free score uses block energy and codec penalties to form a separable allocation proxy, for which the paper derives a greedy optimum under stated equal-cost and diminishing-return assumptions. The method is evaluated across full-RoPE GQA, partial-RoPE, and MLA configurations, with long-context retrieval, LongBench, reasoning, and language-modeling tests. It reports consistent improvements over uniform TQ-MSE allocation in controlled comparisons.

Critical strengths

1.The motivation is clear and the paper organization is good

2.The method is tested across several attention configurations and model sizes, with comparisons at matched logical rates against uniform TQ-MSE allocation.

Critical weaknesses

1.Novelty of the allocation idea is limited. Mixed-precision and non-uniform KV cache compression is already a crowded area. MixKVQ selects mixed key-channel precision, SpectrumQuant uses hybrid bit-widths across DCT frequency bands, Kitty boosts selected key channels, AATC allocates precision across transformed channels and layers, and PM-KVQ progressively reduces layer-wise precision. Against this background, “assigning different bit-widths to different frequency blocks” is not a fundamentally new idea; Block-GTQ mainly contributes a specific allocation unit and procedure rather than a new compression paradigm.

2.The motivation-discovery path is similar to FreqKV and SpectrumQuant. These works start from the difficulty of uniformly quantizing RoPE-modulated key caches, analyze the structural or energy non-uniformity induced by RoPE, and then assign higher precision to high-contribution components and lower precision to low-contribution ones. The high-level idea-discovery trajectory is close.

Questions for authors

na

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.