acceptodds
Under review as a conference paper at ICLR 2027

GaugeRoPE: RoPE-Compatible Executable Latent Attention

Abstract

Latent KV cache compression reduces memory footprint by mapping historical keys and values into compact representations. However, generic compressed key states require reconstruction to native dimension before applying RoPE and computing attention scores, adding sequence-scaling overhead to the decode path. We characterize when a latent KV representation can serve directly as the execution state of pretrained RoPE. For RoPE-based transformers, exact static linear execution of the rotary key path requires the latent-to-native map to preserve the pretrained RoPE action through an intertwining relation, restricting exact mixing to coordinates that share the same RoPE frequency. Guided by this characterization, we build GaugeRoPE, a training-free executable latent KV representation. It combines a RoPE-compatible carrier that preserves the rotary action with an approximate position-independent residual key–value (RKV) correction for complementary key and value information, both evaluated without per-token dense historical K/V reconstruction. Experiments on Mistral-7B show competitive quality at KV-state reduction and substantially stronger RULER and LongBench performance as the state budget tightens. Additional MHA results validate the construction beyond GQA. By enabling direct latent execution, GaugeRoPE achieves the decode throughput of the vanilla uncompressed baseline on 32K-context generation with batch size 4 while halving the KV cache.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, Rotary Position Embedding (RoPE), latent attention, efficient LLM inference, long-context inference, low-rank KV compression, executable latent representations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8HMLOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GaugeRoPE, a training-free KV cache compression method for pretrained RoPE-based Transformers. The method aims to address the overhead of reconstructing historical keys in existing low-rank KV compression approaches before performing RoPE-aware attention. The paper first characterizes the structural conditions under which a latent representation remains exactly compatible with pretrained RoPE. Under the stated assumptions, the analysis shows that exact RoPE compatibility restricts linear mixing to coordinates sharing the same rotary frequency. Based on this observation, GaugeRoPE combines a RoPE-compatible carrier, which preserves the rotary operation in the retained subspace, with a position-independent residual key-value (RKV) correction that provides additional capacity for key and value information. Through offline calibration and projection weight folding, GaugeRoPE enables attention scoring and value aggregation directly in the compressed latent space, avoiding per-token reconstruction of historical native-dimensional KV states during decoding. Experiments on Mistral-7B show competitive long-context quality at 2x KV cache compression. At a context length of 32K and batch size 4, GaugeRoPE achieves a 1.59x decoding throughput improvement over the uncompressed baseline.

Critical strengths
  1. Well-motivated and practically relevant problem. The paper addresses not only the memory footprint of KV cache compression but also the execution overhead associated with compressed representations. Enabling direct attention computation in the latent space is a meaningful direction for improving decoding efficiency, particularly in long-context scenarios.
  2. Principled method design with a clear theoretical foundation. The characterization of RoPE-compatible latent spaces provides a theoretical basis for the proposed compression structure. The connection between same-frequency cross-head mixing and direct latent attention execution is clearly motivated. Furthermore, the combination of a RoPE-compatible carrier and an RKV correction offers an interesting trade-off between positional compatibility and representation capacity.
  3. Training-free adaptation. GaugeRoPE relies on offline calibration and projection weight folding without requiring gradient-based model retraining. This is a practical advantage when adapting existing pretrained models.
  4. Extensive analysis of the proposed components. The paper evaluates model quality on RULER and LongBench and includes several useful ablation studies and diagnostics, such as carrier rank analysis, carrier/RKV branch ablations, teacher-forced attention diagnostics, and rolling-window experiments. The execution experiments also demonstrate substantial improvements from direct latent execution over materialized execution. The results suggest promising throughput and memory benefits in long-context decoding scenarios.
Critical weaknesses
  1. Insufficient performance comparison with existing KV cache compression methods. While the paper provides quality comparisons with Palu and ReCalKV, corresponding decoding throughput comparisons under matched settings are missing. Although the direct execution results demonstrate clear benefits over materialized reconstruction, comparisons with existing compression methods using similarly optimized implementations would better establish GaugeRoPE's practical performance advantages.
  2. Limited analysis of the position-independent RKV approximation. The RKV branch approximates the position-dependent residual key contribution without applying the original RoPE operation. Although Appendix D provides useful carrier/RKV ablations, it does not isolate the accuracy loss caused by this approximation from that caused by low-rank compression. A controlled comparison using reconstructed residual keys with the original RoPE would help quantify this trade-off.
  3. Limited evaluation of practical inference performance. Table 14 shows a significant slowdown at batch size 1 and 8K context (25.36 vs. 55.96 tok/s), while the gap becomes small at 32K and GaugeRoPE outperforms Full at 64K. A detailed analysis of this context-dependent behavior, along with evaluations using larger batch sizes and optimized attention kernels, would help clarify the method's practical applicability.
Questions for authors
  1. Performance comparison Could the authors provide throughput and memory comparisons with Palu and ReCalKV under matched settings and comparable kernel implementations?
  2. Small-batch performance Table 14 shows that GaugeRoPE is slower than Full at batch size 1 for 8K and 32K contexts. What are the main sources of this performance degradation?
  3. RKV approximation Could the authors quantify the accuracy loss caused by omitting RoPE in the RKV branch, separately from the loss due to low-rank compression?
  4. Practical scalability How does GaugeRoPE perform with larger batch sizes and models? Can the carrier and RKV scoring paths be fused into an optimized attention kernel?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 8HML (Part 1)

Thank you for the suggestions on runtime evaluation. We have optimized the Gauge kernel and measured it against fused Palu/ReCalKV implementations, both before and after quantization.

1. Decode throughput and memory [W1, Q1]

At B4/32K, Gauge is 40.8% faster than Palu and 38.7% faster than ReCalKV. We developed a Triton split-PV kernel that fuses attention-weight computation with latent-value aggregation, together with incremental cache updates. The comparisons were rerun on our new H100 PCIe server in BF16, with Full using PyTorch's FlashAttention backend. All three low-rank configurations halve historical KV dimensions. Palu/ReCalKV use implementations based on Palu's official fused reconstruction kernels, adapted for GQA, batch, nonuniform ranks, and head permutations.

Table R1. Full-model decode throughput (tok/s), Mistral, B4. One warmup followed by six measured requests, each with 128 decode steps; entries are medians.

ContextFull: FlashAttentionPaluReCalKVGauge
8K71.1496.5695.9286.60
16K39.9558.7959.2372.22
32K21.2632.9733.4846.43
64K10.78Init. OOMInit. OOM26.75

Gauge overtakes both compressed baselines from 16K onward and reaches 2.18×/2.48× Full throughput at 32K/64K.

Table R2. Full-model decode peak allocated memory (GiB). Includes weights, codec, cache, and temporary tensors; medians of request peaks.

ContextFullPaluReCalKVGauge
8K18.1018.9621.4917.38
16K22.6021.0323.5619.55
32K31.6025.1627.6823.88
64K49.61——32.53

At 32K, Gauge's persistent cache is 8.34 GiB, versus 8.05/8.03 GiB for Palu/ReCalKV; the recent-token policy is specified in Appendix C.1. Its smaller codec allocations contribute to 5.1%/13.7% lower model-level peak memory. Palu/ReCalKV's 64K failures occur during initialization.

We also measure attention latency, including projections, attention, output mapping, and cache updates. Gauge reduces latency relative to Palu/ReCalKV by 32.0%/31.3% at 32K and 36.8%/35.7% at 64K. Palu's fusion avoids writing reconstructed keys to global memory; Gauge additionally removes the reconstruction arithmetic from historical scoring.

The original same-state Direct/Materialized comparison isolates execution form within one representation. The revised Figure 4 instead compares full-model decoding and attention latency against external implementations. Timing includes forward, argmax, and cache updates, excluding prefill; it uses no CUDA graphs or torch.compile. Within-setting throughput ranges are below 1%.

Revised paper: §5.4/Figure 4 and Appendix G.

2. Quantized caches and small batches [W1, W3, Q1, Q2]

We also tested whether the speed advantage survives quantization. Gauge-50% (8/4 bits) uses the same representation and quantization rules for quality and runtime. It reaches 85.56 RULER at 5.85× compression, versus 82.92/80.92 at 5.20×/5.24× for 6-bit Palu/ReCalKV. At B4/32K, throughput is 26.32 vs. 17.84/18.26 tok/s, with 2.745 vs. 3.106/3.080 GiB of persistent cache. At 64K, the throughput advantage grows to 1.58×/1.55×.

The main tables now also include a 128-recent-token variant, which improves quality to 86.31 RULER and 38.63 LongBench at approximately 5.4× compression; the throughput results above evaluate the 5.85× configuration.

This implementation uses packed QK and fused RKV dequantization into a reusable workspace, followed by Torch softmax/PV. Palu/ReCalKV retain their official quantization rules and fused reconstruction kernels with a physical 6-bit cache adapter.

At small batch, query projections, kernel launches, and reductions are harder to amortize. Gauge reaches 1.29× Full throughput at B1/32K, versus 0.56× at B1/8K.

Table R3. B1 throughput (tok/s), unquantized caches.

ContextFullGaugeGauge / Full
8K41.3723.070.56×
32K17.9423.061.29×

Gauge stays near 23 tok/s as context increases, while Full falls from 41.37 to 17.94. Longer histories make the reduction in historical work more valuable.

Revised paper: §5.2/Tables 1/2; Appendices C.1/C.3 and Appendix G/Figure 5.

AuthorsOpenReview ↗

Response to Reviewer 8HML (Part 2)

3. Separating residual positional error from compression error [W2, Q3]

Restoring residual RoPE at fixed compressed codes reduces attention KL by 65.68% and output error by 39.13%. We compare three paths on the same normalized Full attention states: 32 prompts from four tasks, the final 512 queries, and all layers and heads. All paths use the same historical mask and Full values.

A is Gauge's position-independent compressed residual; B restores native RoPE to the same decoded residual; C uses the exact residual/Full-key reference. A→B measures the positional approximation, while B→C identifies remaining compression error. FP32 operator algebra isolates these effects from latent-storage rounding.

Table R4. Attention fidelity with fixed Full states and values.

MetricABC
Attention KL ↓0.0729160.0250221.55e−9
Top-8 overlap ↑0.8256090.9129590.999999
Output relative L2 ↓0.2514630.1530779.39e−7

All three metrics improve in every layer mean and every prompt mean. The KL difference has a paired 95% CI of [−0.048606, −0.047178], resampling prompts within tasks. Output error here is scoring-induced drift, since values are identical.

Across four relative-distance bins from (0, 256) to [4K, 8K), the positional A−B scaled-logit MAE rises 0.3492→0.5710, while B−C remains near 0.1598–0.1631. We also test the two compressed paths in generation with the value codec and decoding protocol fixed.

Table R5. Closed-loop RULER attribution, 50 examples per task and arm.

TaskABB−APaired 95% CI
niah_single_2100.00100.000.00[0.00, 0.00]
cwe85.2088.40+3.20[−1.20, +8.40]
qa_174.0080.00+6.00[−6.00, +18.00]
Three-task macro86.4089.47+3.07[−1.07, +7.27]

The task-score trend is positive but not statistically significant. The distinction is now explicit in the paper: exact compatibility applies to the reconstructed carrier; omitting RoPE from the compressed residual has the measurable cost shown in Table R4.

Revised paper: §5.3 and Appendix D.3/Table 6; Appendix D.1 defines KL, scaling, aggregation, and bootstrap sampling.

4. Larger batches, larger models, and fusion [W3, Q4]

Gauge exceeds 2× Full throughput at B8/B16 and on Qwen2.5-14B-Instruct. These runs use the same optimized kernels with model-specific calibration and frozen pretrained weights.

Table R6. Full-model decode scalability. BF16; one warmup and three measured requests of 128 steps. Batch-total throughput and full-model peak allocated memory.

Model / configurationFull tok/sGauge tok/sSpeedupFull GiBGauge GiB
Mistral-7B, B8 / 16K39.6991.772.31×31.6724.42
Mistral-7B, B16 / 16K41.76105.002.51×49.8234.18
Qwen2.5-14B-Instruct, B4 / 32K11.9126.922.26×54.2943.21

Doubling batch from 8 to 16 raises Gauge throughput by 14.4%. Peak memory is 22.9%/31.4% lower than Full for the Mistral configurations and 20.4% lower on Qwen14B.

Can carrier and RKV scoring be fused? Yes: both contribute to the same logit and share one softmax. Writing the transformed carrier and residual queries as qtcq^c_t and qtrq^r_t, historical scoring is

ℓt,s=(qtc)⊤zs+(qtr)⊤bsdh.\ell_{t,s}=\frac{(q^c_t)^\top z_s+(q^r_t)^\top b_s}{\sqrt{d_h}}.

A joint QK tile loads both latent codes, accumulates the two dot products, and adds them before softmax. The resulting weights aggregate bsb_s once; the value decoder acts on that aggregate. Thus the two branches do not require separate attention passes or historical K/V reconstruction. We have implemented this joint-QK operation in Triton.

For the reported decode results, we retain FP32 GEMMs for scoring and fuse exponential-weight computation with latent-value aggregation in split-PV. This preserves the validated scoring precision while tiling the 768-dimensional value code and history separately to control register pressure and expose parallelism. It stores one combined score matrix and avoids materializing attention weights. The 2.18× Full speedup at B4/32K and the scalability results above use this implementation. Fusing QK, softmax, and PV into a single kernel additionally requires balancing memory-traffic savings against register pressure and reduction precision.

Revised paper: §4.2 (shared scoring and latent aggregation), §5.4, and Appendix G/Table 9.

Dear Reviewer 8HML,

Thank you again for your time and insightful review. As the discussion closes on November 18 (AoE), we would appreciate your thoughts on whether the additional experiments and revisions address your concerns. We would be happy to clarify any remaining questions before the deadline.

Best regards,

The authors

  • Thank you for the detailed rebuttal and the additional experiments. The new throughput comparisons, scalability evaluations, and controlled RKV ablations address several of my original concerns.
  • However, the new results also raise a fundamental concern regarding the method's applicability to long-context inference. While GaugeRoPE achieves increasingly large speedups at longer context lengths, the positional approximation error caused by omitting RoPE in the RKV branch also increases with relative distance within the evaluated range. Furthermore, the newly reported retrieval results show a widening quality gap relative to the uncompressed baseline as context length increases.
  • Given that GaugeRoPE achieves its largest measured speedups at longer context lengths, the absence of sufficient quality evaluation at 32/64K and controlled analysis of the RKV approximation at longer context lengths leaves the practical quality-efficiency trade-off insufficiently established.
  • Therefore, despite acknowledging the technical contributions and the substantial improvements in the rebuttal, I maintain my original rating of weak rejection.
Reviewer_N74GOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how low-rank KV representations for pretrained RoPE Transformers can compress the cache while supporting direct attention computation. Under conditions including static linear maps and preservation of the original RoPE frequencies, the authors characterize the compatibility conditions for direct execution in latent space. They use this characterization to construct carriers that mix components with the same frequency and a shared residual key-value (RKV) representation. The retained rotary components can be executed exactly, whereas the remaining key scores use a position-independent approximation, so the overall attention computation remains approximate. The mappings are constructed through calibration without model training. Experiments cover Mistral-7B, CodeLlama-7B, and Gemma-7B, reporting long-context quality, representation ablations, and decoding throughput and GPU memory measurements on an H100.

Critical strengths
  1. The paper provides a clear structural analysis of the relationship between RoPE-compatible representations and direct execution, supported by several controlled experiments. It distinguishes whether a cache can be compressed from whether computation can operate directly on the compressed representation, and the proposed carriers have a clear theoretical basis. At the tighter B768 budget, GaugeRoPE achieves RULER/LongBench average scores of 62.33/28.84, compared with 47.04/23.67 for Palu and 26.49/25.31 for ReCalKV at the same budget. Execution comparisons using the same cached state also support the value of avoiding explicit reconstruction. These results indicate potential under particular budgets and execution conditions.
Critical weaknesses
  1. The quality losses are substantial, while the measured resource savings have yet to establish a compelling practical advantage. With a 2× reduction in KV state in the B1024 configuration, the average RULER score decreases from 90.84 for the uncompressed baseline (Full) to 85.92, and LongBench decreases from 43.15 to 36.46. The LongBench few-shot category drops from 46.37 to 32.94. At approximately 2.67× state reduction, the two average scores fall to 62.33 and 28.84. Although GaugeRoPE outperforms the paper's low-rank compression baselines at this tighter budget, the absolute quality gap to Full remains large, leaving unclear which applications can tolerate this loss.

    Appendix Table 14 shows that, relative to Full, throughput reaches approximately 1.59× at 32K with batch size 4, while total peak GPU memory decreases by approximately 26%. At 64K with batch size 4, the corresponding improvements are approximately 1.84× throughput and 35% lower peak memory. These are real gains, but they depend on the operating conditions: Gauge is substantially slower than Full at 8K with batch size 1 and slightly slower at 32K with batch size 1. A 2× reduction in KV state does not imply halving total memory, and the 6.09× speedup in Figure 3 compares Direct with Materialized execution of the same Gauge state. Overall, the results do not yet clearly identify an operating regime with both acceptable quality loss and sufficiently strong memory or latency benefits. This weakens the practical case for deploying the method on existing models.

  2. The comparisons do not yet establish the competitiveness of this trade-off against existing cache compression methods, and the quality and system evaluations need closer alignment. The main comparisons concern low-rank methods such as Palu and ReCalKV, without direct comparisons against token selection or per-head budget allocation methods such as SnapKV and HeadKV on the same model, tasks, and actual cache budget in bytes. Although they follow different compression approaches, these methods are relevant deployment alternatives for reducing the KV cache. Without these comparisons, it is difficult to determine whether GaugeRoPE offers an attractive quality–cost trade-off. The highest compression ratios or speedups reported in different papers cannot establish relative superiority directly.

    Furthermore, the main RULER quality evaluation is fixed at 8K, whereas the strongest measured system benefits occur at 32K–64K. A 32K input cap on LongBench does not substitute for quality evaluation with controlled context lengths. The native recent-token window in the appendix does improve quality, but changes the actual cache budget, and Palu/ReCalKV do not use the same native window in that comparison. Full's attention backend, the measured throughput of external compression baselines, and comparisons with more thoroughly optimized reconstruction paths also remain insufficiently specified. Consequently, the current evidence does not adequately distinguish the benefits of the representation from the effects of window policies, execution implementations, and evaluation configurations.

Questions for authors
  1. Direct comparisons with relevant cache methods: Could the authors add SnapKV and HeadKV, at least on the primary Mistral model, alongside the existing Palu/ReCalKV comparisons, and report quality–cost curves using the same tasks, context lengths, and actual cache budgets in bytes? In addition to comparisons at fixed cache budgets, it would be useful to show the memory and decoding benefits each method can achieve under a controlled limit on quality degradation. Please account for necessary index and auxiliary-state overhead, and report prefill time, decoding latency or throughput, and peak GPU memory separately. These results would directly affect my assessment of the method's practical competitiveness.

  2. Combination with token selection methods: Can the current carrier shared across heads and the RKV layout be combined with SnapKV-style token selection or HeadKV-style per-head budget allocation? If a direct combination is not feasible, do the obstacles arise from representation constraints, obtaining selection scores, different token-position sets across heads, or the execution implementation? If a combination is feasible, please discuss the conditions for retaining direct latent-space execution and, where practical, provide a small-scale evaluation of the resulting quality and resource benefits.

  3. Potential for quality recovery: With the compatible structure, cache budget, and recent-window configuration held fixed, could improved calibration, rank allocation, or modest-budget fine-tuning/distillation recover performance closer to Full while preserving the efficiency advantages of direct execution? Does the remaining error mainly arise from carrier truncation, position-independent residual scoring, or value approximation? If recovery training is used, please specify its data and budget, remeasure quality, decoding efficiency, and memory, and clarify whether it adds cached state or changes the execution path. Since the paper primarily targets training-free conversion, this is a supplementary question about whether the quality gap is recoverable. If no such results are available, please clarify the range of applications supported by the current training-free evidence.

  4. Alignment of long-context and window configurations: For the same B1024 configuration used in the system experiments, could the authors provide retrieval quality at 16K, 32K, and 64K, or results stratified by context length and distance to relevant evidence, and confirm whether the native recent-token window W=256 is used? In Appendix C.1, only Gauge uses this window, while baseline quality still comes from whole-history compression. Please report each method's actual cache size in bytes and isolate the benefits of the windowed variant through comparisons with the same window policy or matched actual budgets. Does the position-independent RKV approximation introduce larger errors at greater relative distances?

  5. Implementation and timing conditions for system comparisons: Which attention backend is used for Full? What are the timing boundaries, generation length, precision, warm-up procedure, and repeated-measurement protocol for Table 14, and how do these differ from the separate same-state experiment in Figure 3? Could the authors provide measurements under matched conditions for available optimized attention backends and relevant compression baselines, or comparisons with more fully fused reconstruction paths? Please distinguish speedups between execution forms of the same state from speedups over Full or external methods, and clarify the conclusions supported by the current implementation.

  6. Robustness to capacity and calibration choices: At fixed B1024, increasing the carrier from R1 to R2 raises the average RULER score from 6.36 to 85.92, indicating substantial sensitivity to carrier capacity. Is this behavior stable across calibration samples, calibration domains, and backbone models? Is there a criterion for selecting ranks and detecting impending degradation in autoregressive quality without relying on the evaluation set? Since Gauge and the baselines use different method-specific calibration procedures, controls with matched calibration data or corresponding sensitivity analyses would also help determine whether the gains arise from the representation or the calibration settings.

4 comments
AuthorsOpenReview ↗

Response to Reviewer N74G (Part 1)

Thank you for your comments. GaugeRoPE combines a theoretical analysis of direct execution under pretrained RoPE with experimental validation.

1. Aligned quality and cache cost [W1, W2, Q1, Q4]

Gauge-50% (8/4 bits) achieves 5.85× compression with 85.56 RULER, compared with Full's 90.84. The mixed-precision quality and runtime experiments use the same calibrated representation, scales, and quantization rules, compressing all historical tokens. Palu/ReCalKV's 6-bit configurations also compress all history, at 5.20×/5.24×.

Table R1. Quality and system cost for the same configurations. RULER: all 13 tasks at 8K, 50 examples each. LongBench: 16 tasks × 50, variable inputs capped at 32K. Runtime: B4/8K, 128 steps; cache includes metadata and allocated auxiliary state.

MethodRULERLongBench8K tok/s8K cache GiB
Full90.8442.5971.144.063
Palu-50% (6 bits)82.9237.8853.310.786
ReCalKV-50% (6 bits)80.9237.6154.940.779
Gauge-50% (8/4 bits)85.5635.7359.430.694

Gauge retains 94.2% of Full's RULER score with approximately one-sixth of the cache. This operating point targets cache-limited serving, combining higher RULER quality, a smaller cache, and faster decoding than the quantized low-rank baselines. On LongBench, Gauge leads Palu/ReCalKV in summarization, synthetic tasks, and code; they perform better on document QA and few-shot tasks.

A small native recent-token cache further improves this tradeoff. Retaining 128 tokens gives 86.31 RULER and 38.63 LongBench at 5.42×/5.41× compression, preserving 95.0%/90.7% of Full's scores. LongBench improves by 2.90 points (paired 95% CI [1.50, 4.34]). This variant is now included in the main tables; Appendix C.1 compares 128 and 256 tokens. Table R1's runtime measurements remain for the 5.85× configuration.

Palu/ReCalKV address the same feature-compression problem through reconstructive scoring, making them the direct comparison for our execution claim. Our added experiments focus on this comparison. SnapKV/HeadKV select historical positions, a complementary operation whose integration with Gauge is described in Part 2.

Table R2. Quantized full-model decoding at longer contexts. Each entry is tok/s / peak decode GiB; peak is the maximum across measured requests, including weights, codec, cache, and workspace.

ContextPalu (6 bits)ReCalKV (6 bits)Gauge (8/4 bits)
16K32.06 / 18.5832.94 / 21.1042.03 / 16.30
32K17.84 / 20.3018.26 / 22.7826.32 / 17.87
64K9.53 / 24.159.74 / 26.1415.05 / 21.05

At 32K, Gauge is 1.48×/1.44× faster than Palu/ReCalKV, using 2.745 GiB of cache versus 3.106/3.080 GiB. At 64K, the speedups reach 1.58×/1.55×. These measurements use physical packed caches and include quantization and dequantization costs.

Revised paper: §5.2/Tables 1/2, Appendix C.3, and Appendix G/Figure 5.

2. Long-context retrieval and execution [W1, W2, Q4, Q5]

Table R3. Mistral NIAH length scaling. Four tasks × 20 examples per length; Gauge retains 256 recent native tokens, matching its BF16 system configuration.

ContextFullGauge
8K92.5086.56
16K88.7582.19
32K90.6380.31

Gauge retains 88.6% of Full's retrieval score at 32K. On Qwen2.5-7B, the same four-task evaluation gives 78.75 vs. Full's 93.13. Table R3 evaluates the BF16 configuration; fixed-length 32K/64K RULER for mixed precision remains unmeasured.

Gauge's optimized split-PV avoids reconstruction arithmetic, reducing attention latency by 32.0%/31.3% at 32K and yielding 2.18× Full model throughput. At B4/32K, Gauge's full-model peak is 23.88 GiB, versus Full's 31.60 GiB. At B1, Gauge/Full throughput rises from 0.56× at 8K to 1.29× at 32K, as longer histories amortize projection and launch costs. The original Direct/Materialized comparison held Gauge states fixed to isolate execution form; the new results measure complete models against Full and external baselines.

The B4 full-model comparisons use H100 PCIe, one warmup, six measured requests, and 128 fixed decode steps on matched inputs. Timing covers model forward, argmax, and cache updates, excluding prefill; no CUDA graphs or torch.compile are used. Full and the low-rank methods use BF16 computation. Quantized Palu/ReCalKV preserve their official quantization rules and fused kernels with a physical 6-bit cache adapter.

Revised paper: §5.2/Tables 1/2, Appendices C.1/C.3, and Appendix G/Figure 5.

AuthorsOpenReview ↗

Response to Reviewer N74G (Part 2)

3. Combining with token selection [Q2]

Yes: common token selection directly selects carrier/RKV cache rows. With fixed projections and original position IDs, this equals masking the omitted positions in Gauge attention. Head-specific latent QK scores support attention-based selection without reconstructing historical K/V.

For per-head sets ShS_h, store each position once in U=⋃hShU=\bigcup_h S_h and apply head-specific indices or masks. Storage is blatent∣U∣b_{\mathrm{latent}}|U| plus metadata, so savings depend on overlap between heads, rather than independent head budgets. Sparse indexing and grouped layouts need engineering; token selection and score access preserve compatibility.

4. Approximation error and quality recovery [Q3, Q4]

Carrier capacity and residual correction both improve fidelity. At fixed cache size, R1→R2 reduces carrier-only KL 0.64589→0.40609; adding the residual to R2 reduces it further to 0.07292. To isolate residual positional error, we then keep compressed codes and Full values fixed:

Table R4. Residual-RoPE attribution on normalized Full states. 32 prompts, final 512 queries, all layers/heads, identical historical masks.

Residual scoreKL ↓Top-8 ↑Output rel-L2 ↓
A: position independent0.0729160.8256090.251463
B: same compressed residual, RoPE restored0.0250220.9129590.153077
C: exact residual / Full-key reference1.55e−90.9999999.39e−7

Restoring RoPE reduces KL by 65.68% and scoring-induced output error by 39.13%; B→C identifies remaining compression error. Across distance bins (0,256), [256,1K), [1K,4K), [4K,8K), positional A−B scaled-logit MAE grows 0.349→0.460→0.534→0.571, while B−C remains 0.160–0.163.

With the same value codec, closed-loop scores change 100→100 (single-needle), 85.20→88.40 (CWE), and 74→80 (QA), 50 examples each. The +3.07 macro change has 95% CI [−1.07,+7.27], so the task gain is not statistically established. Full values isolate scoring error. In the Real/Complex control, Real initially reconstructs values slightly better but diverges during prefill, so initial value error alone does not explain the collapse.

Recovery training. TransMLA (Meng et al., 2025, Table 1) recovers much of the conversion loss on LLaMA-2-7B with 0.5–6B tokens from a SmolLM-based mixture. Gauge could similarly optimize its calibrated projections and decoders through next-token training or distillation, constraining the carrier to same-frequency complex-linear maps. Keeping ranks, storage precision, and the recent-token policy fixed preserves cache size and direct latent execution without extra cached state. Our results remain training-free; evaluating recovery would require reporting the corpus and token/compute budget and remeasuring quality, decode throughput, and peak memory.

Error analysis in the revised paper: §5.3 and Appendices D.1–D.3, including Table 6.

5. Calibration and capacity robustness [Q6]

R2's advantage persists across calibration samples and corpora, with token budgets and fitting procedure fixed.

Table R5. Four-task RULER macro, 20 examples per task. Each fit uses 200,704 carrier and 100,352 RKV tokens. CIs resample prompt pairs within tasks.

Corpus / seedR1Real R2Complex R295% CI: Complex R2−R1
WikiText-103 / 530.000.0078.00[69.75,85.75]
WikiText-103 / 540.000.0084.75[77.62,91.25]
WikiText-103 / 550.000.0080.00[72.25,87.12]
WikiText-2 / 530.00—82.13[74.88,89.00]

R2 reduces carrier-only/hybrid KL by approximately 39%/31% across the three samples and 39.05%/31.43% on WikiText-2. WikiText-2 is also used by Palu/ReCalKV; all 256 prompt/layer pairs improve in KL, Top-8, and output error.

At fixed R2 rank, Complex gains 3.78–3.80 points in spectral capture and reduces hybrid KL by approximately 10% across samples. Its 56.5% reduction in L0 KL, together with divergence during prefill before generation, helps explain why modest average-KL differences can produce large closed-loop effects. On Qwen, increasing carrier and residual capacity raises 32K NIAH scores 25.31→58.75→78.75.

Spectral capture and local KL diagnose fixed-rank configurations; a reliable rank-selection or collapse-prediction rule from calibration statistics alone remains open.

Revised paper: §5.3; Appendix D.4/Table 7 and the cross-corpus paragraph; Appendix H/Table 10.

Dear Reviewer N74G,

Thank you again for your time and constructive feedback. As the discussion closes on November 18 (AoE), we would appreciate hearing whether our responses and the new evidence address your concerns. Please let us know if any points would benefit from further clarification before the deadline.

Best regards,

The authors

I’d like to express my sincere gratitude to the authors for their detailed response, particularly for including relevant experiments such as RULER and LongBench, along with the corresponding compression ratios and decoding speeds. I’d also like to thank the authors for their additional explanations regarding certain details. Although there are still some areas for improvement, overall I believe this version is a significant improvement over the original paper. Now, regarding the most critical issue: as shown by the supplementary experiments, while Gauge demonstrates improved efficiency and significant advantages in terms of video memory usage, it suffers from a severe loss in performance scores—a level that is unacceptable for many tasks. Furthermore, the comparison baselines should ideally be methods that are widely recognized within the field; relying solely on Palu and ReCalKV is insufficient.

In summary, I could raise the score to 2. But for the same reasons as reviewer 8HML, I still maintain my position of “weak rejection”. Finally, I would like to express my sincere gratitude for your dedication and hard work, and I hope you will continue to refine this method in your future research.

Reviewer_srauOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when low-rank KV caches in pre-trained RoPE models can be scored in latent space without reconstructing full key vectors. The authors show that a static key decoder commutes with RoPE if and only if it mixes coordinates of the same frequency across different attention heads. And this is also a necessary condition for achieving exact latent-space scoring. Based on this, GaugeRoPE uese a frequency-partitioned complex-PCA representation scored with latent RoPE and a joint PCA encoding for key residuals and values scored without RoPE. On Mistral-7B, GaugeRoPE matches Palu and ReCalKV at 2x compression, outperforms both at 2.67x compression, and delivers 1.59x decoding speedup in 32K context (4 bsz).

(I am not an expert in this field. If possible, please assign more experienced reviewers and take their review into account.)

Critical strengths

1.This paper is well-motivated and meaningful. For models using RoPE, reconstructing the keys during the decoding phase negates the performance gains achieved through low-rank KV compression.

2.The proposed method is simple but effective. At 2.67x compression, it outperforms Palu and ReCalKV, achieving 62.33 on RULER, compared to 47.04 for Palu and 26.49 for ReCalKV.

3.The appendix provides comprehensive ablation studies, more experimental results and the opensource code.

Critical weaknesses

1.The main concern is the novelty. The method may be somewhat similar to TransMLA[1], which already uses per-frequency PCA across kv heads and keeps RoPE only on leading components, and BKV-PCA then compresses the remaining keys with values. The authors use complex instead of real mixing and two modes per frequency. But the paper's own ablation shows the phase gauge improves little at R1, and Proposition 1 isn't used at R2.

2.Some conclusions may overcalim. At 2×, GaugeRoPE achieves the lowest on LongBench, and on few-shot it is 13.4 points below Full, against 1.7 for Palu. "Degrades more gracefully" rests on a limit budget where each method is below Full.

3.Only one GQA model is evaluated (Mistral-7B, D = 8) in baseline comparisons, and the result improvement from R1 to R2 (RULER 6.36 to 85.92) makes me worry that the right rank is model-specific. Llama-3.1-8B or a GQA model with fewer kv heads such as Qwen2.5-7B would help.

[1] F. Meng, P. Tang, Z. Yao, X. Sun, M. Zhang. "TransMLA: Migrating GQA Models to MLA with Full DeepSeek Compatibility and Speedup." NeurIPS 2025.

Questions for authors

1.The configuration of TransMLA used in Table 11 (RoPE dimension, latent rank, FreqFold)?

2.Can the latent state be combined with 8-bit KV quantization?

4 comments
AuthorsOpenReview ↗

Response to Reviewer srau (Part 1)

Thank you for raising the connection to TransMLA. We have made this relationship explicit and added an equal-rank comparison to test what complex coefficients contribute in practice.

W1. What complex coefficients add

TransMLA introduces same-frequency real mixing through RoRoPE and balanced joint KV compression. Under the stated native-frequency assumptions, GaugeRoPE characterizes the full compatible family: a static real-linear decoder must be frequency-block-diagonal, with each same-frequency block the realification of a complex-linear map. This necessary and sufficient condition connects representation structure to direct latent scoring:

\quad\longleftrightarrow\quad \begin{pmatrix}A_j&-B_j\\\B_j&A_j\end{pmatrix}.$$ RoRoPE's real carrier is the $B_j=0$ case. **Complex coefficients also capture relative phase relationships across heads, without adding cached dimensions at equal rank.** Computation remains real-valued. We compare Real and Complex R2 at native frequencies, without frequency folding, using the same calibration moments and 256 carrier + 768 RKV dimensions. Real retains the leading two eigenvectors of $\operatorname{Re}(G_j)$; Complex uses the Hermitian $G_j$. Both refit their residual/value code with the same balanced PCA procedure. **Table R1. Equal-rank, equal-cache comparison, Mistral.** Operator metrics use identical normalized Full states and Full values. | Metric | Real R2 | Complex R2 | | --------------------------- | ------: | ----------: | | Spectral capture | 79.27% | **83.06%** | | Carrier-only KL ↓ | 0.47232 | **0.40609** | | Hybrid KL ↓ | 0.08018 | **0.07292** | | Hybrid output relative L2 ↓ | 0.26783 | **0.25146** | | RULER, 13 tasks × 50 | 6.05 | **85.92** | | LongBench-6, six tasks × 30 | 0.13 | **42.98** | Complex improves capture in all 2048 layer-frequency blocks, reducing uncaptured energy by **18.3%**. Repeating calibration on three independent samples gives a consistent **3.78–3.80-point capture gain** and approximately **10% lower hybrid KL**. Complex scores **78.00/84.75/80.00** on the four-task evaluation; Real scores zero in every run. The quality jump is accompanied by a much larger early-layer improvement: **L0 KL falls by 56.5%**, versus approximately 10% averaged over layers. Two prompt traces diverge during prefill, before generated tokens differ. Dense/direct checks agree within 3.4e−6 relative L2 for scores and value aggregation, and cache encoding is identical. The large early-layer difference and divergence during prefill help explain why the generation gap is much larger than the average KL difference suggests. Carrier-only diagnostics isolate coefficient freedom; closed-loop scores also include its effect on the fitted residual code. Proposition 1 gives the rank-one phase-gauge implementation. R2 uses the general complex-linear construction of Theorem 1 through the real blocks above. We therefore added Table R1 to evaluate coefficient freedom at the R2 capacity used in our main experiments. **Revised paper:** §3, §6, §5.3/Figure 3, and Appendices D.2/D.4 (Tables 5/7). ### Q1. TransMLA configuration The **original submission's Table 11** used fold=collapse=8, a positional cache of 128 real scalars, and joint-BKV widths 896/640 at total budgets 1024/768. These are implemented cache widths; the reimplementation does not directly set the official qk_rope_head_dim and kv_lora_rank fields. The revised **Qwen length comparison** uses FreqFold=4, 128 positional + 384 joint-BKV scalars, and no uptraining, matching Gauge's 512-scalar budget. The Real/Complex control above keeps native frequencies and rank fixed to separate the coefficient effect. **Revised paper:** Appendix H, Table 10 and its configuration paragraph.
AuthorsOpenReview ↗

Response to Reviewer srau (Part 2)

W2. Quality at the evaluated operating points

We have removed “degrades more gracefully” and revised the discussion around the measured results below. The revised main tables include quantized configurations and use 50 examples per task: all 13 RULER tasks and 16 English LongBench tasks. The latter is a common 800-example evaluation for every method, replacing the original main-table LongBench protocol.

Table R2. Revised Mistral main-table results. Compression includes quantization metadata; “-50%” denotes halving cached dimensions. RULER uses 8K inputs; LongBench uses variable-length inputs capped at 32K.

MethodCompressionRULERLongBench
Full1.00×90.8442.59
Palu-50%2.00×83.3338.00
ReCalKV-50%2.00×81.5436.74
Gauge-50%2.00×85.9235.93
Palu-50% (6 bits)5.20×82.9237.88
ReCalKV-50% (6 bits)5.24×80.9237.61
Gauge-50% (8/4 bits)5.85×85.5635.73
+ 128 recent tokens5.42× / 5.41×86.3138.63

Gauge achieves higher RULER quality with a smaller cache than the quantized low-rank baselines. Its 8/4-bit configuration improves RULER by 2.64/4.64 points over Palu/ReCalKV while using 11.2%/10.4% less per-token storage. On LongBench, it leads these baselines in summarization, synthetic tasks, and code; Palu/ReCalKV perform better on document QA and few-shot tasks. Quantization changes Gauge's LongBench score by only 0.20 points with shared calibrated bases, reducing per-token storage by 65.8%. For the recent-token variant, compression is reported separately for RULER and LongBench and includes the native cache.

Retaining 128 recent tokens raises quality to 86.31 RULER and 38.63 LongBench while maintaining approximately 5.4× compression. The LongBench gain is 2.90 points (paired 95% CI [1.50, 4.34]). Increasing the window to 256 yields 86.41/38.92 at approximately 5.0× compression, showing that 128 tokens capture most of the improvement. Appendix C.1 reports both variants alongside the original BF16 ablation. The quantized runtime results evaluate the 5.85× configuration.

Revised paper: §5.2, Tables 1/2; Appendices C.1/C.3 and G/Figure 5.

W3. Why R1→R2 changes generation so sharply

R2 uses more of the fixed cache budget to preserve native positional interactions. On the same Full attention states, adding the second mode raises capture from 75.03% to 83.06% and improves both carrier-only and hybrid scoring.

Table R3. Fixed 1024-scalar cache, Mistral.

MetricR1R2
Carrier / RKV dimensions128 / 896256 / 768
Carrier-only KL ↓0.645890.40609
Hybrid KL ↓0.106360.07292
Hybrid output relative L2 ↓0.313460.25146
Complete RULER6.3685.92

Carrier-only KL falls by 37.13%; hybrid KL and output error fall by 31.44%/19.78%, despite less RKV capacity. KL, Top-8 overlap, and output error improve in all 32 layer means and all 32 prompt means. R1 produces widespread repetitive continuations, whereas R2 restores useful retrieval and generation. The Real/Complex traces in Part 1 further show how early operator differences accumulate before decoding begins.

This capacity effect repeats across three calibration samples: R1 scores zero and R2 scores 78.00–84.75 on the four-task test. Switching from WikiText-103 to WikiText-2 yields 0.00/82.13, with carrier-only/hybrid KL reductions of 39.05%/31.43%.

Revised paper: §5.3/Figure 3 and Appendices D.1/D.4.

AuthorsOpenReview ↗

Response to Reviewer srau (Part 3)

W3. Generalization to fewer KV heads

Qwen2.5-7B has four KV heads, versus Mistral's eight. Gauge reaches 78.96 on its complete RULER evaluation, retaining 88.10% of Full's score without uptraining. The configuration uses 256 carrier and 256 RKV scalars, with approximately 1.94× effective cache reduction.

Table R4. Complete Qwen RULER by category. 13 tasks × 50 examples; the overall macro weights tasks equally.

Category (tasks)FullGauge
NIAH (8)98.0091.44
VT (1)96.4066.80
CWE/FWE (2)81.4059.10
QA (2)61.0055.00
Overall89.6378.96

Table R5. Qwen NIAH length scaling. Four tasks × 20 examples at each length; Gauge and training-free TransMLA use the same total cache budget and recent-token policy.

Method8K16K32K
Full96.8895.9493.13
TransMLA76.2537.1911.88
Gauge90.0083.7578.75

At 32K, Gauge retains 84.56% of Full's retrieval score. Its compatible carrier also benefits standard MHA: on CodeLlama, increasing the carrier share at fixed cache size raises RULER 58.80→67.78→69.50. These results show that useful capacity depends on the backbone, while the compatible construction transfers across GQA and MHA.

Revised paper: §5.2/Table 3, Appendix E/Table 8, and Appendix H/Table 10.

Q2. Combining Gauge with quantization

Yes. We now combine an 8-bit carrier with 4-bit RKV, reaching 5.85× compression. It scores 85.56 RULER / 35.73 LongBench, versus 85.92/35.93 for Gauge-50%. The cache uses 700 B per token/layer, including dynamic scales and zero points, plus 32,768 B of shared carrier scales for the model. The carrier uses static symmetric quantization with paired real/imaginary scales; RKV uses dynamic asymmetric quantization in groups of 64.

The same representation, scales, and quantization rules are used in the optimized decode comparison. At B4/32K, Gauge delivers 26.32 tok/s with 2.745 GiB of persistent cache, versus 17.84/18.26 tok/s and 3.106/3.080 GiB for 6-bit Palu/ReCalKV. At 64K, its throughput advantage reaches 1.58×/1.55×.

Revised paper: §5.2/Tables 1/2; Appendix C.3 (quantization) and Appendix G/Figure 5 (runtime).

Dear Reviewer srau,

Thank you again for your time and thoughtful review. As the discussion closes on November 18 (AoE), we would appreciate hearing whether our rebuttal and revised manuscript address your concerns. We would be happy to clarify any remaining points before the deadline.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.