- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes RoFold, a post-hoc KV-cache compression method that enables RoPE operations and attention scoring directly in a compressed latent space without reconstructing full-dimensional keys. It characterizes the subspaces that preserve RoPE compatibility and constructs compression bases by exploiting shared RoPE frequencies across attention heads. An attention-aware encoder is fitted to reduce approximation error, while a hybrid variant selectively retains reconstruction for some attention units. Experiments on Llama-3.1-8B compare perplexity and decoding throughput against PaLU and truncated SVD under a matched KV-cache budget.
Critical strengths
The paper provides a clear mathematical characterization of when RoPE can be executed exactly in a compressed subspace. The cross-head construction extends beyond selecting complete RoPE coordinate pairs, and the corresponding ablation (Table 4) demonstrates a meaningful perplexity improvement over this restricted alternative. The attention-aware encoder ablation (Table 3) also provides evidence that the fitted encoder contributes to reconstruction quality. The evaluation reports both perplexity and decoding throughput at several context lengths, making the quality–efficiency trade-off visible.
Critical weaknesses
The main concern is that the claimed decoding advantage is not convincingly established. In Figure 3b, RoFold-Pure is slower than the hybrid configuration at both evaluated context lengths, despite eliminating reconstruction entirely. It is also slower than PaLU at 8K, contradicting the statement in Section 5.4 that Pure achieves the highest decoding rate. Moreover, the uncompressed baseline remains faster than every compressed method, and the reported absolute throughput is unusually low for an A100. Without implementation details, optimized-kernel comparisons, or latency profiling, it is unclear whether the reported speedups reflect an algorithmic advantage or implementation-specific overhead.
The evaluation is also limited to one model, one compression budget, and perplexity measurements. RoFold consistently produces worse perplexity than PaLU at the same memory budget, while the fully reconstruction-free variant incurs a substantially larger quality penalty. The comparison with related methods is incomplete, particularly the absence of a quantitative comparison with RAP, which also targets reconstruction-free execution. Finally, the hybrid routing procedure and its reconstruction budget are insufficiently specified, making it difficult to understand the contribution of the reconstruction-free mechanism independently of the fallback strategy.
Questions for authors
-
Figure 3b shows that RoFold-Pure is slower than the hybrid at both evaluated context lengths, despite eliminating reconstruction. What explains this result, and how does it support the claimed benefit of reconstruction-free execution?
-
What implementation and kernels were used for RoFold and PaLU? Could the authors provide matched-kernel comparisons and a breakdown of decoding latency, including latent rotation, reconstruction, and attention computation?
-
How are the 23 fallback units selected in the hybrid configuration, and what are the reconstruction budget and cost function used in Eq. (8)? How sensitive are quality and throughput to this routing decision?
-
Was PaLU evaluated using its original rank allocation and whitening strategy, or was it constrained to RoFold's rank allocation? Could the authors clarify whether the baseline configuration is optimal under the matched memory budget?
-
Could the authors provide a quantitative comparison with RAP, as well as evaluations on additional model families, compression ratios, and downstream long-context tasks?
3 comments
Response to Reviewer uEby (2/2)
Q3. Routing: an explicit objective, cost and budget, solved on calibration data. The routing problem is Eq. (8),
where is the set of 64 units and the units that reconstruct; the others use the latent path. Appendix C.7 gives the full specification.
- Why route. Units differ in how much quality they lose under the RoPE-invariance constraint, so the reconstruction budget is spent where it helps most.
- Objective, cost, budget. is the mean per-token NLL on the 8K and 16K calibration corpora; is proportional to the unit's key rank ; is set in advance to about 30% of the total key-reconstruction work, and the deployed set uses 30.9%.
- Selection. A cost-aware heuristic: starting from 14 units from a single-unit screening shortlist, units are added by calibration-loss reduction per cost until the budget is reached (23 units), then refined by swapping reconstructed and latent units at a fixed count, keeping a round only when it lowers by more than a preset threshold (three of four rounds kept). Ranking uses calibration data only, never throughput.
- Validation. Each kept set passed a pass/fail check on a document-disjoint 16K validation span and an unused training-split span; the initial set was also compared with an earlier 14-unit set on the Table 2 8K/16K corpora, where the candidates ranked as on calibration data. The 32K and 64K spans were never used.
- Final configuration. Fixed before evaluation, shared by all tokens and lengths, with no extra cache.
Sensitivity (Figure 3b, at fixed memory):
| Fallback units | 0 (Pure) | 14 | 23 (deployed) |
|---|---|---|---|
| 16K perplexity | 14.34 | 9.48 | 8.40 |
| 16K decode (tok/s) | 2.30 | 2.70 | 2.60 |
Quality responds strongly to the budget and throughput only weakly: going from 14 to 23 units lowers 16K perplexity by 1.08 at a 3.9% throughput cost. This also separates the mechanism from the fallback: the 41 latent units carry the reconstruction-free path, and Table 4 shows that they are usable only because of the cross-head family.
Q4. PaLU was evaluated with its own rank allocation and whitening; RoFold adopts PaLU's allocation.
- Rank allocation. The shared allocation comes from PaLU's own Fisher-weighted rank search (head-group size 4) at the matched budget.
- Decomposition. The PaLU baseline uses PaLU's own whitening-based decomposition with its default calibration settings and no extra tuning (Appendix C.1).
- Verification. The key ranks match on 64/64 units, and the value factors are bitwise identical across all compressed arms.
The allocation is PaLU's own choice at this budget, and RoFold's advantage holds across four PaLU budgets (Table R2), so the comparison does not rest on a configuration chosen in RoFold's favor.
Q5. RAP, compression ratios, model families and downstream tasks.
- RAP. To our knowledge RAP has no public implementation, and its accuracy relies on LoRA recovery training. Table 4 isolates its core idea, whole RoPE pairs, under identical training-free settings: the cross-head family lowers perplexity by 30%.
- Compression ratios. PaLU at four ratios (Table R2).
- Downstream. Official 16K RULER, every task reported (Table R1; Appendix C.8).
- Model families. The theory applies to any RoPE model whose frequencies repeat across KV heads (standard MHA and GQA). The experiments use Llama-3.1-8B at the 40.1% budget.
Contribution and significance. The paper asks how much data-adaptive freedom an exactly RoPE-compatible low-rank key representation can retain, and gives a complete answer.
- A necessary and sufficient criterion. Theorem 1 turns "can this compressor skip reconstruction under RoPE?" into a checkable property of its subspace, for any fixed low-rank key representation.
- A strictly larger exact family. Because RoPE frequencies repeat across KV heads, the family contains data-adaptive cross-head modes; using it instead of whole pairs lowers perplexity by 30% with everything else fixed (Table 4).
- Training-free and exact as deployed. The deployed basis matches the canonical construction to 1.2×10⁻¹¹, with no training or recovery step.
- A controllable systems trade-off. At a fixed cache, the decode advantage over PaLU grows from 23.4% at 16K to 56.4% at 64K, and the routing budget sets the quality–throughput point.
- A direction for further work. Units differ widely in their sensitivity to the invariance constraint; per-unit routing already closes 92% of the 16K perplexity gap between Pure and PaLU, and finer per-unit choices are a natural next step.
We would be grateful if the reviewer would reconsider the score in light of these points.
Follow-up on Author Response
Dear Reviewer uEby,
Thank you again for your constructive review. We would like to briefly follow up on our two responses, as the discussion period is approaching its end.
We have addressed your main technical concerns with additional evidence: the implementation overhead behind RoFold-Pure's throughput, a semantics-preserving runtime refinement, the complete hybrid routing procedure, verification of PaLU's original rank allocation and whitening, and new results on 16K RULER and four PaLU compression budgets. The revised manuscript incorporates these clarifications and experiments.
We understand that some limitations remain, particularly regarding broader model coverage and comparisons with other RoPE-compatible methods. Nevertheless, we hope the new evidence helps establish the validity and significance of the proposed approach.
We would greatly appreciate your updated assessment. If any specific concerns remain unresolved, we would welcome the opportunity to clarify them before the discussion closes.
Thank you for your time and consideration.
Response to Reviewer uEby (1/2)
We thank the reviewer for the careful review, and for recognizing the mathematical characterization, the gain of the cross-head construction over whole RoPE pairs (Table 4) and the contribution of the attention-aware encoder (Table 3). We answer each question below.
Q1. Pure is slower than the Hybrid because of per-unit overhead in our implementation, not because of reconstruction-free attention.
The §5.4 sentence now matches Figure 3b. The paper's throughput tables keep the original implementation.
Q2. Implementation and kernels: one shared harness, no fused kernels.
(continued in the next comment)