acceptodds
Under review as a conference paper at ICLR 2027

RACE: RoPE-Aligned Attention Compression Based On Query-Key Activation Energies

10 views

Abstract

Head-dimension compression can reduce the memory required by the key-value (KV) cache, but existing methods often reconstruct full-dimensional representations during inference, adding significant decoding FLOPs overhead. We formalize the design considerations for converting pretrained attention into a natively reduced-dimensional form that reduces both KV-cache storage and attention computation using a calibration set. Specifically, we consider reductions that (1) apply a shared projection to queries and keys so that they remain in a common reduced feature space, (2) derive this projection jointly from query and key activations to better preserve their interaction, and (3) are compatible with the pretrained RoPE transformation. To obtain a reconstruction-free realization of this reduction, we restrict it to coordinate selection and theoretically show that RoPE compatibility then holds if and only if complete rotary pairs are retained. Building on this characterization, we introduce RACE (RoPE-Aligned Attention Compression based on activation Energy), a post-training method that selects rotary pairs using their joint query-key activation energy. We theoretically show that this criterion minimizes a separable, calibration-dependent upper bound on post-RoPE query-key product approximation error. The resulting reductions are folded into the model weights and require neither runtime reconstruction nor specialized kernels. Across three model families and different compression levels, RACE outperforms structured pruning and remains competitive with reconstruction-based methods on language-modeling, zero-shot, and long-context tasks. At 50% KV-cache retention and 256K context, RACE reduces total inference FLOPs by 47.8%, yielding 1.79× higher throughput, 43.2% lower time-to-first-token (TTFT), and 28.1% lower allocated GPU memory in comparison to an uncompressed model. We also show that RACE is compatible with token eviction or compaction as well as quantization based KV-cache compression methods.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
KV-cache compression, Efficient LLM inference, Attention compression, Rotary positional embeddings

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_G5o5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed RACE - new method for KV$ compression, which reduce head dimension for query and keys. RACE ranks complete RoPE pairs using joint activation energy and retains the highest-energy pairs on some calibration set. This post-training method doesn't require run-time reconstruction. It also preserves long-context model capabilities. Authors demonstrated that RACE is competitive vs reconstruction-based methods. The new method is compatible with token eviction and with quantization based KV-cache compression.

Critical strengths

The key paper idea is simple, and it is completive comparing to similar to other simiulr KV$ compression methods. Good, practical post-training solution. Paper is well written.

Critical weaknesses

The primary paper idea - binding coordinates for droping into RoPE pairs is sound, but the paper has very limited novelty.

Questions for authors

Algorithms1: Will the coordinates selection change if one will compute E_m using pre-RoPE (NoPE) activations for each par m? In other words, how RoPE change E_m?

It would be interesting to extend section of effect of calibration data for RACE, using instead of FineWeb-Edu the calibration data for more narrow domain like math or coding.

Can you fix non-existing references please? For example, Fig. 8, Eq.7 ( on page 2, line 104-105, page 3, line 129,...)

2 comments
AuthorsOpenReview ↗

Response to Reviewer G5o5

We thank the reviewer for recognizing the practical benefits of RACE and for the constructive suggestions.

W1: Contributions We would like to clarify that the contributions of this work extends beyond complete RoPE pair binding. We formulate native attention compression as a constrained approximate matrix multiplication problem and provide a necessary and sufficient characterization of RoPE-compatible coordinate selection (Theorem 1). Within this admissible family, we derive a forward-only joint QK activation-energy criterion, RACE, with theoretical bounds connecting pair selection to QK approximation error, attention-output perturbation, and downstream loss perturbation (Appendix A.6). By folding the selected Q/K coordinates and calibrated value subspace into the model weights, RACE enables reconstruction-free native attention compression without specialized kernels. Thus, our contribution lies in the combination of a formal structural characterization, a principled and efficient selection criterion, and an effective native implementation, rather than simply grouping coordinates into RoPE pairs.

Q1: Effect of pre-RoPE vs. post-RoPE activations on pair selection. Computing the proposed energy criterion before or after RoPE produces exactly the same pair rankings and coordinate selection. This follows from the orthogonality of the RoPE rotation within each pair, as derived in Appendix A.7 Lines 1160–1173. Specifically, for a rotary pair (m), since (R_{m,t}^{\top}R_{m,t}=I), we have (|R_{m,t}q_{m,t}|2^2=|q{m,t}|_2^2), and similarly for keys. Consequently, the proposed criterion (E_m=|Q_m|_F^2+|K_m|_F^2) and pair selection is identical before and after RoPE.

Q2: Calibration using specialized math and coding datasets (additional results included in the revised manuscript). We specially thank the reviewer for this suggestion as this has given us interesting insights into the behaviour of RACE under highly specialized tasks like code generation. We have expanded our calibration-distribution study in Fig. 7 (Section 4.4) to include MMLU College Mathematics and LCC (code completion), alongside general-language and reasoning tasks. These additional results indicate that for highly specialized downstream tasks like LCC, same calibration data can meaningfully adapt the retained attention subspace while strongly mismatched calibration can be detrimental. For instance, task-matched LCC calibration improves performance by 13.3% relative to FineWeb-Edu calibration, while mismatched calibration can reduce performance by up to 17.8%. For more general downstream tasks, FineWeb-Edu provides a robust generic calibration source when the downstream distribution is unknown.

Q3. Missing references. The referenced figures and equations were included in the supplementary material, which was submitted as a separate PDF. Consequently, these references may have appeared unresolved when reading the main manuscript independently. We apologize for the confusion. We have consolidated the main paper and supplementary material into a single document in the revised manuscript to make the referenced content easier to locate.

Again, we thank the reviewer for the constructive feedback, especially regarding effect of calibration on highly specialized domains, which has helped us improve the paper significantly.

Dear reviewer, as we approach the end of the rebuttal period, please let us know if any concerns remain or if further clarification would be helpful. We would appreciate it if the reviewer could reconsider their score in light of our responses and additional experiments. Thank you!

Reviewer_DU7EOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RACE, a post-training KV-cache compression method that reduces attention head dimensions without runtime reconstruction. By formulating query-key compression as constrained approximate matrix multiplication, the authors establish that RoPE-compatible coordinate selection must preserve complete rotary pairs. RACE selects these pairs using joint query-key activation energy and compresses values through SVD, folding both reductions into model weights. Experiments across three model families demonstrate competitive accuracy against reconstruction-based approaches while reducing KV-cache memory, attention FLOPs, and inference latency.

Critical strengths
  • Practical reconstruction-free attention compression. RACE reduces both KV-cache storage and native attention computation without specialized kernels. Its compatibility with standard attention implementations makes the method particularly attractive for practical deployment.

  • Convincing empirical improvements over existing compression methods. RACE consistently outperforms RAP and achieves accuracy competitive with reconstruction-based methods across multiple models and compression levels. The reported 1.79× throughput improvement at 256K context further demonstrates meaningful systems benefits.

Critical weaknesses
  • Limited theoretical justification for the proposed selection criterion. The joint activation-energy criterion minimizes a loose separable upper bound rather than the actual query-key product approximation error. Moreover, this criterion is not invariant to reciprocal query/key rescaling that preserves attention logits, raising questions about how reliably it identifies important rotary pairs.

  • Insufficient validation of the quality-efficiency trade-off at aggressive compression. The largest reported efficiency gains occur at 50% KV-cache retention, where Figure 3 shows substantial degradation in downstream accuracy, particularly on LongBench. Additionally, the 256K-context systems measurements are not accompanied by corresponding long-context quality evaluations, making the practical trade-off difficult to assess.

Questions for authors
  • Can the authors quantify how tightly the proposed energy bound correlates with actual query-key product approximation error? How does RACE compare with a scale-invariant selection criterion that accounts for the interaction between query and key magnitudes?

  • Can the authors evaluate long-context retrieval quality at the 128K–256K lengths used in the efficiency benchmarks and provide throughput comparisons at matched downstream accuracy? This would help establish whether the reported systems improvements translate into practical deployment advantages.

3 comments
AuthorsOpenReview ↗

Response to Reviewer DU7E

We sincerely thank the reviewer for the constructive feedback and for recognizing the motivation, theoretical formulation, and systems benefits of RACE. We have revised the manuscript and added experiments specifically clarifying and addressing the concerns raised.

W1 & Q1. Theoretical Justification Error bound and connection to query-key product approximation error: We thank the reviewer for the observation and we agree that the proposed joint QK activation-energy criterion minimizes a separable upper bound rather than the exact query-key product approximation error. However, our theoretical justification extends beyond QK approximation error. As derived in Appendix A.6 (Lines 1175–1209), the proposed criterion also minimizes a derived upper bounds on attention-output perturbation and, under a local Lipschitz assumption, downstream loss perturbation. Specifically, we show that both perturbations are upper-bounded by quantities proportional to the total activation energy of the discarded RoPE pairs, which is minimized by retaining the highest-energy pairs. We further validate its effectiveness empirically in Figure 14, where joint QK selection achieves lower normalized QK approximation error (0.1051) than Q-only (0.1159) and K-only (0.1120) selection.

Invariance under reciprocal Q/K rescaling: We agree with the reviewer that the proposed joint energy selection criterion is not invariant to reciprocal query/key rescaling. Importantly, this criterion is not claimed to be the uniquely optimal selection criterion. Our framework distinguishes the structural requirements for native RoPE-compatible compression, established in Theorem 1, from the choice of scoring functions like the joint energy selection within this admissible family. We evaluate alternate scoring functions in Sec. 4.5 including an alternate attention-product criterion that more directly captures QK interactions (Fig. 8, Tab. 6). Despite being scale-invariant, it performs worse than joint energy selection (17.86 vs. 16.02 WikiText perplexity at 50% retention), while requiring 4.3× more calibration time (11.616s vs. 2.717s) and 1.7× more peak memory (10.592 GB vs. 6.142 GB). Thus, we position joint QK energy as a theoretically motivated, empirically effective, and computationally efficient choice within the broader family of valid RoPE-compatible selection criteria, rather than as an invariant or exact optimizer. We also acknowledge that other selection criteria within this admissible family may offer further improvements, potentially combining scale invariance with stronger approximation quality and computational efficiency, which presents a very interesting direction for future work.

W2 & Q2. Long-context performance beyond 64K (additional results included in the revised manuscript) To address this concern, we have added RULER evaluations at 64K, 128K, and 256K context lengths using Llama-3.2-3B-Instruct and RACE at 75% and 50% KV-cache retention as Fig.5 (detailed in Appendix A.8). These results supplement the systems benefits at long-contexts given in Fig.4. At 75% KV-cache retention, RACE preserves approximately 92% of the uncompressed model's RULER score on average across these context lengths. This demonstrates that moderate head-dimension compression preserves most of the measured long-context capability in addition to higher throughput and lower memory and TTFT. More aggressive compression incurs larger accuracy degradation similar to the language modeling, zero-shot, and LongBench results given in Fig.3. This emphasizes the quality-efficiency tradeoff, especially at higher compression rates. Note that the 256K context used in Fig. 5 exceeds the model’s native 128K context window and we evaluate this setting using simple context extension without additional training.

Practical deployment advantages at matched accuracy: We thank the reviewer for this suggestion. Fig. 1 characterizes the quality–efficiency trade-off across compression levels, comparing average zero-shot accuracy against relative inference decoding FLOPs for RACE and baseline methods. At comparable downstream accuracy, RACE achieves substantially lower FLOPs than reconstruction-based methods such as PALU and EigenAttention, while maintaining better accuracy than the reconstruction-free baselines such as RAP and ThinKeys at matched compression ratios. This advantage arises from executing attention natively in the reduced dimensions without runtime reconstruction. We complement this comparison with measured throughput, TTFT, and memory results in Fig. 4 and long-context quality evaluations in Fig. 5.

Again, we sincerely thank the reviewer for the very thoughtful comments and constructive feedback which has helped us improve the paper significantly.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. The additional theoretical analysis strengthens the justification for the proposed selection criterion, and the empirical comparison against the scale-invariant attention-product criterion provides useful evidence supporting the proposed design choice. The new RULER evaluations at 64K–256K substantially address my concerns about long-context quality. While aggressive compression still introduces noticeable degradation, the reported results at 75% retention demonstrate a promising quality-efficiency trade-off. Overall, I consider my primary concerns substantially mitigated.

Thank you for the thoughtful follow-up and for increasing the score. We appreciate that the additional clarifications, analyses, and experiments helped address your concerns.

Reviewer_p4WpOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies native attention compression along the head dimension to preserve low-dimensional representation during attention computation. The proposed method, RACE, structurally selects the rotary pairs on the shared coordinates of the query and key dimensions for RoPE compatibility, and it optimizes the selection by retaining rotary pairs with higher joint query-key energy from the calibration set. The method avoids full-dimensional reconstruction requirements of keys and obtains both prefilling and decoding FLOPs reduction with KV cache compressed. The evaluation shows this method remains competitive with reconstruction-based methods with optional post-finetuning. It can also be combined with other orthogonal compression methods such as token eviction and quantization.

Critical strengths
  • This paper addresses the full-dimensional reconstruction requirement, considering RoPE in previous works, enabling potential efficiency improvements in reduced-dimension attention. This paper proposes an insightful approach to address this problem, providing theoretical derivations to demonstrate the RoPE compatibility of coordinate selection and the rationality of the rotary-pair selection strategy on the query-key joint energy.

  • Estimation results show the efficiency improvement of the approach through TTFT/decoding speed improvements. This paper also experimentally validates the compatibility of other orthogonal methods, such as quantization and token compression.

Critical weaknesses
  • The evaluation of long-context inference is insufficient. The context length in evaluated tasks, including zero-shot and selected Longbench tasks, does not adequately cover the range in efficiency estimation, which hardly exceeds 64K. Therefore, the presented results are not sufficient to imply whether the model performance can be preserved in the context-length range that shows the significant efficiency improvement. This raises the concern of the limited impact of this approach in long-context inference scenarios. The performance evaluation beyond 64K context length would be helpful to clarify this concern.

  • The robustness of calibration data selection in the experiment needs to be further clarified. The authors discuss the dataset selection but do not explain the procedure to select the samples/tokens in the specified dataset and how the amount of it is determined. It is unclear whether the approach is sensitive to the selected set with different distributions and amounts. This paper lacks a sensitivity analysis on calibration data sampling.

  • The empirical advantage of Joint QK Energy needs to be clarified. The ablation for various selection criteria involved post-finetuning, if I understand correctly. Although the joint turns out to be the best choice in the end, the results do not isolate the contribution of the selection criteria itself.

Minor: There appears to be a typo in L455: “athese”.

Questions for authors
  • Can you provide the performance evaluation beyond 64K context-length?
  • Is the ablation results for selection criteria (Appendix, Table 6) involved fine-tuning? I am wondering whether joint QK Energy is superior before fine-tuning.
4 comments
AuthorsOpenReview ↗

Response to Reviewer p4Wp

We sincerely thank the reviewer for the constructive feedback and for recognizing the motivation, theoretical formulation, and systems benefits of RACE. We have revised the manuscript and added experiments specifically addressing the concerns raised.

W1 & Q1. Long-context performance beyond 64K (additional results included in the revised manuscript) To address this concern, we have added RULER evaluations at 64K, 128K, and 256K context lengths using Llama-3.2-3B-Instruct and RACE at 75% and 50% KV-cache retention as Fig.5 (detailed in Appendix A.8). These results supplement the systems benefits at long-contexts given in Fig.4. At 75% KV-cache retention, RACE preserves approximately 92% of the uncompressed model's RULER score on average across these context lengths. This demonstrates that moderate head-dimension compression preserves most of the measured long-context capability in addition to higher throughput and lower memory and TTFT. More aggressive compression incurs larger accuracy degradation similar to the language modeling, zero-shot, and LongBench results given in Fig.3. This emphasizes the quality-efficiency tradeoff at higher compression. Note that the 256K context used in Fig. 5 exceeds the model’s native 128K context window and we evaluate this setting using simple context extension without additional training.

W2. Robustness to calibration data sampling and size (additional results included in the revised manuscript's Appendix) (i) Calibration sampling: For the calibration tokens, we randomly shuffle the FineWeb-Edu training stream, tokenize examples sequentially from the shuffled stream with EOS separators, and truncate the resulting stream to the specified token budget. We have included the details in A.7 (L1260). (ii) Calibration size: We evaluate calibration budgets ranging from 2K to 512K tokens at both 75% and 50% KV-cache retention. We observe that increasing calibration size generally improves performance, with diminishing returns beyond 32K tokens for both 75% and 50% retention. Thus 32K offers a practical trade-off between calibration cost and model quality. This result is given in Fig. 11 of Appendix A.7 (iii) Sampling variability: We additionally evaluate five independently sampled 32K-token calibration sets with varying seeds. The resulting WikiText perplexity is 14.53 ± 0.37 at 75% retention and 18.05 ± 0.14 at 50% retention

W3 & Q2. Isolating the benefit of Joint QK Energy before recovery (additional results included in the revised manuscript's Appendix) We thank the reviewer for this observation. The selection-criteria results in Table 6 include the post-compression recovery, as do the other main results. To isolate the contribution of the selection criterion itself, we have additionally evaluated joint QK, Q-only, K-only, and magnitude based selection without any post-compression recovery (Figure 15 in Appendix A.7 L1535). The results show that Joint QK Energy remains the strongest of the evaluated selection criteria before recovery, similar to the post-recovery results. In particular, at 50% KV-cache retention, Joint QK Energy achieves a WikiText perplexity of 124.35, compared with 209.87 for Q-only, 144.76 for K-only, and 152.71 for magnitude-based selection. Although the absolute perplexities at this aggressive compression level are substantially higher without recovery, these results demonstrate that the advantage of joint selection is not solely attributable to subsequent optimization. It is also worth noting that at 87.5% and 75% KV retention, the trend remains the same while the absolute perplexities are lower and more meaningful making recovery optional for these moderate compression levels.

We thank the reviewer for pointing out the typographical error and have corrected it in the revised manuscript. Again, we sincerely thank the reviewer for the constructive feedback which has helped the paper substantially.

Dear reviewer, as we approach the end of the rebuttal period, please let us know if any concerns remain or if further clarification would be helpful. We would appreciate it if the reviewer could reconsider their score in light of our responses and additional experiments. Thank you!

Reviewer_p4WpOpenReview ↗

Response to Rebuttal

Thank you for the response. The additional experiments and discussion in the revised version have not only addressed all my concerns but also enriched the analysis of the proposed approach. I am happy to raise my score.

We appreciate that the additional results helped address your concerns and thank you for raising the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.