- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GaugeRoPE, a training-free KV cache compression method for pretrained RoPE-based Transformers. The method aims to address the overhead of reconstructing historical keys in existing low-rank KV compression approaches before performing RoPE-aware attention. The paper first characterizes the structural conditions under which a latent representation remains exactly compatible with pretrained RoPE. Under the stated assumptions, the analysis shows that exact RoPE compatibility restricts linear mixing to coordinates sharing the same rotary frequency. Based on this observation, GaugeRoPE combines a RoPE-compatible carrier, which preserves the rotary operation in the retained subspace, with a position-independent residual key-value (RKV) correction that provides additional capacity for key and value information. Through offline calibration and projection weight folding, GaugeRoPE enables attention scoring and value aggregation directly in the compressed latent space, avoiding per-token reconstruction of historical native-dimensional KV states during decoding. Experiments on Mistral-7B show competitive long-context quality at 2x KV cache compression. At a context length of 32K and batch size 4, GaugeRoPE achieves a 1.59x decoding throughput improvement over the uncompressed baseline.
Critical strengths
- Well-motivated and practically relevant problem. The paper addresses not only the memory footprint of KV cache compression but also the execution overhead associated with compressed representations. Enabling direct attention computation in the latent space is a meaningful direction for improving decoding efficiency, particularly in long-context scenarios.
- Principled method design with a clear theoretical foundation. The characterization of RoPE-compatible latent spaces provides a theoretical basis for the proposed compression structure. The connection between same-frequency cross-head mixing and direct latent attention execution is clearly motivated. Furthermore, the combination of a RoPE-compatible carrier and an RKV correction offers an interesting trade-off between positional compatibility and representation capacity.
- Training-free adaptation. GaugeRoPE relies on offline calibration and projection weight folding without requiring gradient-based model retraining. This is a practical advantage when adapting existing pretrained models.
- Extensive analysis of the proposed components. The paper evaluates model quality on RULER and LongBench and includes several useful ablation studies and diagnostics, such as carrier rank analysis, carrier/RKV branch ablations, teacher-forced attention diagnostics, and rolling-window experiments. The execution experiments also demonstrate substantial improvements from direct latent execution over materialized execution. The results suggest promising throughput and memory benefits in long-context decoding scenarios.
Critical weaknesses
- Insufficient performance comparison with existing KV cache compression methods. While the paper provides quality comparisons with Palu and ReCalKV, corresponding decoding throughput comparisons under matched settings are missing. Although the direct execution results demonstrate clear benefits over materialized reconstruction, comparisons with existing compression methods using similarly optimized implementations would better establish GaugeRoPE's practical performance advantages.
- Limited analysis of the position-independent RKV approximation. The RKV branch approximates the position-dependent residual key contribution without applying the original RoPE operation. Although Appendix D provides useful carrier/RKV ablations, it does not isolate the accuracy loss caused by this approximation from that caused by low-rank compression. A controlled comparison using reconstructed residual keys with the original RoPE would help quantify this trade-off.
- Limited evaluation of practical inference performance. Table 14 shows a significant slowdown at batch size 1 and 8K context (25.36 vs. 55.96 tok/s), while the gap becomes small at 32K and GaugeRoPE outperforms Full at 64K. A detailed analysis of this context-dependent behavior, along with evaluations using larger batch sizes and optimized attention kernels, would help clarify the method's practical applicability.
Questions for authors
- Performance comparison Could the authors provide throughput and memory comparisons with Palu and ReCalKV under matched settings and comparable kernel implementations?
- Small-batch performance Table 14 shows that GaugeRoPE is slower than Full at batch size 1 for 8K and 32K contexts. What are the main sources of this performance degradation?
- RKV approximation Could the authors quantify the accuracy loss caused by omitting RoPE in the RKV branch, separately from the loss due to low-rank compression?
- Practical scalability How does GaugeRoPE perform with larger batch sizes and models? Can the carrier and RKV scoring paths be fused into an optimized attention kernel?
4 comments
Response to Reviewer 8HML (Part 2)
3. Separating residual positional error from compression error [W2, Q3]
Restoring residual RoPE at fixed compressed codes reduces attention KL by 65.68% and output error by 39.13%. We compare three paths on the same normalized Full attention states: 32 prompts from four tasks, the final 512 queries, and all layers and heads. All paths use the same historical mask and Full values.
A is Gauge's position-independent compressed residual; B restores native RoPE to the same decoded residual; C uses the exact residual/Full-key reference. A→B measures the positional approximation, while B→C identifies remaining compression error. FP32 operator algebra isolates these effects from latent-storage rounding.
Table R4. Attention fidelity with fixed Full states and values.
| Metric | A | B | C |
|---|---|---|---|
| Attention KL ↓ | 0.072916 | 0.025022 | 1.55e−9 |
| Top-8 overlap ↑ | 0.825609 | 0.912959 | 0.999999 |
| Output relative L2 ↓ | 0.251463 | 0.153077 | 9.39e−7 |
All three metrics improve in every layer mean and every prompt mean. The KL difference has a paired 95% CI of [−0.048606, −0.047178], resampling prompts within tasks. Output error here is scoring-induced drift, since values are identical.
Across four relative-distance bins from (0, 256) to [4K, 8K), the positional A−B scaled-logit MAE rises 0.3492→0.5710, while B−C remains near 0.1598–0.1631. We also test the two compressed paths in generation with the value codec and decoding protocol fixed.
Table R5. Closed-loop RULER attribution, 50 examples per task and arm.
| Task | A | B | B−A | Paired 95% CI |
|---|---|---|---|---|
| niah_single_2 | 100.00 | 100.00 | 0.00 | [0.00, 0.00] |
| cwe | 85.20 | 88.40 | +3.20 | [−1.20, +8.40] |
| qa_1 | 74.00 | 80.00 | +6.00 | [−6.00, +18.00] |
| Three-task macro | 86.40 | 89.47 | +3.07 | [−1.07, +7.27] |
The task-score trend is positive but not statistically significant. The distinction is now explicit in the paper: exact compatibility applies to the reconstructed carrier; omitting RoPE from the compressed residual has the measurable cost shown in Table R4.
Revised paper: §5.3 and Appendix D.3/Table 6; Appendix D.1 defines KL, scaling, aggregation, and bootstrap sampling.
4. Larger batches, larger models, and fusion [W3, Q4]
Gauge exceeds 2× Full throughput at B8/B16 and on Qwen2.5-14B-Instruct. These runs use the same optimized kernels with model-specific calibration and frozen pretrained weights.
Table R6. Full-model decode scalability. BF16; one warmup and three measured requests of 128 steps. Batch-total throughput and full-model peak allocated memory.
| Model / configuration | Full tok/s | Gauge tok/s | Speedup | Full GiB | Gauge GiB |
|---|---|---|---|---|---|
| Mistral-7B, B8 / 16K | 39.69 | 91.77 | 2.31× | 31.67 | 24.42 |
| Mistral-7B, B16 / 16K | 41.76 | 105.00 | 2.51× | 49.82 | 34.18 |
| Qwen2.5-14B-Instruct, B4 / 32K | 11.91 | 26.92 | 2.26× | 54.29 | 43.21 |
Doubling batch from 8 to 16 raises Gauge throughput by 14.4%. Peak memory is 22.9%/31.4% lower than Full for the Mistral configurations and 20.4% lower on Qwen14B.
Can carrier and RKV scoring be fused? Yes: both contribute to the same logit and share one softmax. Writing the transformed carrier and residual queries as and , historical scoring is
A joint QK tile loads both latent codes, accumulates the two dot products, and adds them before softmax. The resulting weights aggregate once; the value decoder acts on that aggregate. Thus the two branches do not require separate attention passes or historical K/V reconstruction. We have implemented this joint-QK operation in Triton.
For the reported decode results, we retain FP32 GEMMs for scoring and fuse exponential-weight computation with latent-value aggregation in split-PV. This preserves the validated scoring precision while tiling the 768-dimensional value code and history separately to control register pressure and expose parallelism. It stores one combined score matrix and avoids materializing attention weights. The 2.18× Full speedup at B4/32K and the scalability results above use this implementation. Fusing QK, softmax, and PV into a single kernel additionally requires balancing memory-traffic savings against register pressure and reduction precision.
Revised paper: §4.2 (shared scoring and latent aggregation), §5.4, and Appendix G/Table 9.
Dear Reviewer 8HML,
Thank you again for your time and insightful review. As the discussion closes on November 18 (AoE), we would appreciate your thoughts on whether the additional experiments and revisions address your concerns. We would be happy to clarify any remaining questions before the deadline.
Best regards,
The authors
- Thank you for the detailed rebuttal and the additional experiments. The new throughput comparisons, scalability evaluations, and controlled RKV ablations address several of my original concerns.
- However, the new results also raise a fundamental concern regarding the method's applicability to long-context inference. While GaugeRoPE achieves increasingly large speedups at longer context lengths, the positional approximation error caused by omitting RoPE in the RKV branch also increases with relative distance within the evaluated range. Furthermore, the newly reported retrieval results show a widening quality gap relative to the uncompressed baseline as context length increases.
- Given that GaugeRoPE achieves its largest measured speedups at longer context lengths, the absence of sufficient quality evaluation at 32/64K and controlled analysis of the RKV approximation at longer context lengths leaves the practical quality-efficiency trade-off insufficiently established.
- Therefore, despite acknowledging the technical contributions and the substantial improvements in the rebuttal, I maintain my original rating of weak rejection.
Response to Reviewer 8HML (Part 1)
Thank you for the suggestions on runtime evaluation. We have optimized the Gauge kernel and measured it against fused Palu/ReCalKV implementations, both before and after quantization.
1. Decode throughput and memory [W1, Q1]
At B4/32K, Gauge is 40.8% faster than Palu and 38.7% faster than ReCalKV. We developed a Triton split-PV kernel that fuses attention-weight computation with latent-value aggregation, together with incremental cache updates. The comparisons were rerun on our new H100 PCIe server in BF16, with Full using PyTorch's FlashAttention backend. All three low-rank configurations halve historical KV dimensions. Palu/ReCalKV use implementations based on Palu's official fused reconstruction kernels, adapted for GQA, batch, nonuniform ranks, and head permutations.
Table R1. Full-model decode throughput (tok/s), Mistral, B4. One warmup followed by six measured requests, each with 128 decode steps; entries are medians.
Gauge overtakes both compressed baselines from 16K onward and reaches 2.18×/2.48× Full throughput at 32K/64K.
Table R2. Full-model decode peak allocated memory (GiB). Includes weights, codec, cache, and temporary tensors; medians of request peaks.
At 32K, Gauge's persistent cache is 8.34 GiB, versus 8.05/8.03 GiB for Palu/ReCalKV; the recent-token policy is specified in Appendix C.1. Its smaller codec allocations contribute to 5.1%/13.7% lower model-level peak memory. Palu/ReCalKV's 64K failures occur during initialization.
We also measure attention latency, including projections, attention, output mapping, and cache updates. Gauge reduces latency relative to Palu/ReCalKV by 32.0%/31.3% at 32K and 36.8%/35.7% at 64K. Palu's fusion avoids writing reconstructed keys to global memory; Gauge additionally removes the reconstruction arithmetic from historical scoring.
The original same-state Direct/Materialized comparison isolates execution form within one representation. The revised Figure 4 instead compares full-model decoding and attention latency against external implementations. Timing includes forward, argmax, and cache updates, excluding prefill; it uses no CUDA graphs or torch.compile. Within-setting throughput ranges are below 1%.
Revised paper: §5.4/Figure 4 and Appendix G.
2. Quantized caches and small batches [W1, W3, Q1, Q2]
We also tested whether the speed advantage survives quantization. Gauge-50% (8/4 bits) uses the same representation and quantization rules for quality and runtime. It reaches 85.56 RULER at 5.85× compression, versus 82.92/80.92 at 5.20×/5.24× for 6-bit Palu/ReCalKV. At B4/32K, throughput is 26.32 vs. 17.84/18.26 tok/s, with 2.745 vs. 3.106/3.080 GiB of persistent cache. At 64K, the throughput advantage grows to 1.58×/1.55×.
The main tables now also include a 128-recent-token variant, which improves quality to 86.31 RULER and 38.63 LongBench at approximately 5.4× compression; the throughput results above evaluate the 5.85× configuration.
This implementation uses packed QK and fused RKV dequantization into a reusable workspace, followed by Torch softmax/PV. Palu/ReCalKV retain their official quantization rules and fused reconstruction kernels with a physical 6-bit cache adapter.
At small batch, query projections, kernel launches, and reductions are harder to amortize. Gauge reaches 1.29× Full throughput at B1/32K, versus 0.56× at B1/8K.
Table R3. B1 throughput (tok/s), unquantized caches.
Gauge stays near 23 tok/s as context increases, while Full falls from 41.37 to 17.94. Longer histories make the reduction in historical work more valuable.
Revised paper: §5.2/Tables 1/2; Appendices C.1/C.3 and Appendix G/Figure 5.