- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposed RACE - new method for KV$ compression, which reduce head dimension for query and keys. RACE ranks complete RoPE pairs using joint activation energy and retains the highest-energy pairs on some calibration set. This post-training method doesn't require run-time reconstruction. It also preserves long-context model capabilities. Authors demonstrated that RACE is competitive vs reconstruction-based methods. The new method is compatible with token eviction and with quantization based KV-cache compression.
Critical strengths
The key paper idea is simple, and it is completive comparing to similar to other simiulr KV$ compression methods. Good, practical post-training solution. Paper is well written.
Critical weaknesses
The primary paper idea - binding coordinates for droping into RoPE pairs is sound, but the paper has very limited novelty.
Questions for authors
Algorithms1: Will the coordinates selection change if one will compute E_m using pre-RoPE (NoPE) activations for each par m? In other words, how RoPE change E_m?
It would be interesting to extend section of effect of calibration data for RACE, using instead of FineWeb-Edu the calibration data for more narrow domain like math or coding.
Can you fix non-existing references please? For example, Fig. 8, Eq.7 ( on page 2, line 104-105, page 3, line 129,...)
2 comments
Dear reviewer, as we approach the end of the rebuttal period, please let us know if any concerns remain or if further clarification would be helpful. We would appreciate it if the reviewer could reconsider their score in light of our responses and additional experiments. Thank you!
Response to Reviewer G5o5
We thank the reviewer for recognizing the practical benefits of RACE and for the constructive suggestions.
W1: Contributions We would like to clarify that the contributions of this work extends beyond complete RoPE pair binding. We formulate native attention compression as a constrained approximate matrix multiplication problem and provide a necessary and sufficient characterization of RoPE-compatible coordinate selection (Theorem 1). Within this admissible family, we derive a forward-only joint QK activation-energy criterion, RACE, with theoretical bounds connecting pair selection to QK approximation error, attention-output perturbation, and downstream loss perturbation (Appendix A.6). By folding the selected Q/K coordinates and calibrated value subspace into the model weights, RACE enables reconstruction-free native attention compression without specialized kernels. Thus, our contribution lies in the combination of a formal structural characterization, a principled and efficient selection criterion, and an effective native implementation, rather than simply grouping coordinates into RoPE pairs.
Q1: Effect of pre-RoPE vs. post-RoPE activations on pair selection. Computing the proposed energy criterion before or after RoPE produces exactly the same pair rankings and coordinate selection. This follows from the orthogonality of the RoPE rotation within each pair, as derived in Appendix A.7 Lines 1160–1173. Specifically, for a rotary pair (m), since (R_{m,t}^{\top}R_{m,t}=I), we have (|R_{m,t}q_{m,t}|2^2=|q{m,t}|_2^2), and similarly for keys. Consequently, the proposed criterion (E_m=|Q_m|_F^2+|K_m|_F^2) and pair selection is identical before and after RoPE.
Q2: Calibration using specialized math and coding datasets (additional results included in the revised manuscript). We specially thank the reviewer for this suggestion as this has given us interesting insights into the behaviour of RACE under highly specialized tasks like code generation. We have expanded our calibration-distribution study in Fig. 7 (Section 4.4) to include MMLU College Mathematics and LCC (code completion), alongside general-language and reasoning tasks. These additional results indicate that for highly specialized downstream tasks like LCC, same calibration data can meaningfully adapt the retained attention subspace while strongly mismatched calibration can be detrimental. For instance, task-matched LCC calibration improves performance by 13.3% relative to FineWeb-Edu calibration, while mismatched calibration can reduce performance by up to 17.8%. For more general downstream tasks, FineWeb-Edu provides a robust generic calibration source when the downstream distribution is unknown.
Q3. Missing references. The referenced figures and equations were included in the supplementary material, which was submitted as a separate PDF. Consequently, these references may have appeared unresolved when reading the main manuscript independently. We apologize for the confusion. We have consolidated the main paper and supplementary material into a single document in the revised manuscript to make the referenced content easier to locate.
Again, we thank the reviewer for the constructive feedback, especially regarding effect of calibration on highly specialized domains, which has helped us improve the paper significantly.