acceptodds
Under review as a conference paper at ICLR 2027

RoFold: Reconstruction-Free Low-Rank KV Caching under RoPE

Abstract

Low-rank KV caching reduces the storage and bandwidth costs of long-context inference, but pre-RoPE methods reconstruct compressed historical keys to full width before rotation and scoring. We ask whether keys can instead remain latent throughout the read path without giving up a data-adaptive basis. We show that exact latent execution, RoPE invariance of the decoder subspace, and commutation of its orthogonal projector with RoPE are equivalent, and that the execution error of a general subspace decomposes orthogonally into subspace escape and latent-dynamics mismatch. Based on this characterization, we introduce RoFold, which combines (i) a data-adaptive construction of same-frequency cross-head modes and an attention-aware left-inverse encoder fitted by a convex ridge problem without increasing cache width or retained rank, and (ii) a reconstruction-free key path that never rebuilds full-width keys. This path is exactly equivalent, in exact arithmetic, to reconstruct-then-rotate for the same compressed representation, and a frozen unit-level hybrid applies it to 41 of 64 units. On Llama-3.1-8B, at a matched 40.1% KV-reduction budget, RoFold's autoregressive decode advantage over the reconstruct-then-rotate PaLU baseline grows from 9.1% at 8K to 56.4% at 64K, while the relative PPL increase remains below 10% across all evaluated lengths.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, Rotary Position Embeddings(RoPE), Long-Context LLM Inference, Low-Rank Compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_uEbyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RoFold, a post-hoc KV-cache compression method that enables RoPE operations and attention scoring directly in a compressed latent space without reconstructing full-dimensional keys. It characterizes the subspaces that preserve RoPE compatibility and constructs compression bases by exploiting shared RoPE frequencies across attention heads. An attention-aware encoder is fitted to reduce approximation error, while a hybrid variant selectively retains reconstruction for some attention units. Experiments on Llama-3.1-8B compare perplexity and decoding throughput against PaLU and truncated SVD under a matched KV-cache budget.

Critical strengths

The paper provides a clear mathematical characterization of when RoPE can be executed exactly in a compressed subspace. The cross-head construction extends beyond selecting complete RoPE coordinate pairs, and the corresponding ablation (Table 4) demonstrates a meaningful perplexity improvement over this restricted alternative. The attention-aware encoder ablation (Table 3) also provides evidence that the fitted encoder contributes to reconstruction quality. The evaluation reports both perplexity and decoding throughput at several context lengths, making the quality–efficiency trade-off visible.

Critical weaknesses

The main concern is that the claimed decoding advantage is not convincingly established. In Figure 3b, RoFold-Pure is slower than the hybrid configuration at both evaluated context lengths, despite eliminating reconstruction entirely. It is also slower than PaLU at 8K, contradicting the statement in Section 5.4 that Pure achieves the highest decoding rate. Moreover, the uncompressed baseline remains faster than every compressed method, and the reported absolute throughput is unusually low for an A100. Without implementation details, optimized-kernel comparisons, or latency profiling, it is unclear whether the reported speedups reflect an algorithmic advantage or implementation-specific overhead.

The evaluation is also limited to one model, one compression budget, and perplexity measurements. RoFold consistently produces worse perplexity than PaLU at the same memory budget, while the fully reconstruction-free variant incurs a substantially larger quality penalty. The comparison with related methods is incomplete, particularly the absence of a quantitative comparison with RAP, which also targets reconstruction-free execution. Finally, the hybrid routing procedure and its reconstruction budget are insufficiently specified, making it difficult to understand the contribution of the reconstruction-free mechanism independently of the fallback strategy.

Questions for authors
  1. Figure 3b shows that RoFold-Pure is slower than the hybrid at both evaluated context lengths, despite eliminating reconstruction. What explains this result, and how does it support the claimed benefit of reconstruction-free execution?

  2. What implementation and kernels were used for RoFold and PaLU? Could the authors provide matched-kernel comparisons and a breakdown of decoding latency, including latent rotation, reconstruction, and attention computation?

  3. How are the 23 fallback units selected in the hybrid configuration, and what are the reconstruction budget and cost function used in Eq. (8)? How sensitive are quality and throughput to this routing decision?

  4. Was PaLU evaluated using its original rank allocation and whitening strategy, or was it constrained to RoFold's rank allocation? Could the authors clarify whether the baseline configuration is optimal under the matched memory budget?

  5. Could the authors provide a quantitative comparison with RAP, as well as evaluations on additional model families, compression ratios, and downstream long-context tasks?

3 comments
AuthorsOpenReview ↗

Response to Reviewer uEby (1/2)

We thank the reviewer for the careful review, and for recognizing the mathematical characterization, the gain of the cross-head construction over whole RoPE pairs (Table 4) and the contribution of the attention-aware encoder (Table 3). We answer each question below.

Q1. Pure is slower than the Hybrid because of per-unit overhead in our implementation, not because of reconstruction-free attention.

  • Where it comes from. In our PyTorch reference implementation, each latent unit launches many small rotation operations per decode step and runs per-step finite-value checks that synchronize the host with the GPU. Pure routes all 64 units through this path; the latent arithmetic itself needs neither.
  • What we changed. A semantics-preserving refinement defers the checks to one device-side flush per decoder call and rewrites the rotation into fewer equivalent operations. Representation, ranks, routing and RoPE semantics are unchanged, and logits and 8K/16K perplexity are bitwise identical.
  • Result. At 16K, Pure's decode throughput rises by 1.22×, more than any other method (1.05–1.15×; Table R4 in the general response).
  • How this supports the claimed benefit. The benefit shows where this overhead does not dominate: with 41 latent units, the Hybrid is faster than PaLU at every length, by 23.4% at 16K and 56.4% at 64K, and the gap grows with context, as expected from removing a cost proportional to the number of cached keys.

The §5.4 sentence now matches Figure 3b. The paper's throughput tables keep the original implementation.

Q2. Implementation and kernels: one shared harness, no fused kernels.

  • One harness for every method. Full-KV, Truncated SVD, PaLU and RoFold all run in the same PyTorch reference harness: bf16, batch 1, one A100, the same decode loop and the same timing protocol.
  • No fused kernels. No method uses a custom fused kernel; Full-KV uses plain matmul and softmax. A matched-kernel comparison would need equally optimized kernels for every method, whereas PaLU's public release provides only a standalone attention-level kernel that is not wired into an end-to-end decode path.
  • What the numbers mean. The absolute rates are those of an unfused research harness, which is why they are low for an A100. Because every method shares it, the differences reflect what each method computes per step: rebuilding every cached key (PaLU, Truncated SVD) versus rotating and scoring in latent space (RoFold).
  • Where the time goes. RoFold's advantage over PaLU grows from 23.4% at 16K to 56.4% at 64K, the signature of removing a cost proportional to the number of cached keys. It is reconstruction, not cache size: across 30.1%–60.0% KV reduction PaLU stays at 2.05–2.10 tokens/s at 16K, whereas RoFold reaches 2.56 (Table R2).
  • Why Full-KV is faster. Full-KV is a memory reference, not a matched baseline: it stores 131,072 B/token, 1.67× the cache of every compressed arm. Low-rank KV compression (here Truncated SVD, PaLU and RoFold) is by design a trade-off among memory, quality and speed. A pre-RoPE low-rank cache, as used by every compressed method here, stores keys before rotation, so every cached key has to be processed again at each decode step; Full-KV stores rotated keys once and reads them directly. Truncated SVD and PaLU rebuild every cached key to full width before RoPE and scoring; RoFold's latent units instead rotate each latent with an r×r operator. In exchange, the cache is 40.1% smaller, which permits longer contexts or larger batches in the same memory. No compressed method in Table 2 is faster than Full-KV at batch 1, so the relevant comparison is at a fixed cache size, where RoFold is the fastest. From 16K to 64K, RoFold's decode rate also rises from 51% to 63% of Full-KV, the fastest closing among the compressed methods (PaLU stays near 40%).

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer uEby (2/2)

Q3. Routing: an explicit objective, cost and budget, solved on calibration data. The routing problem is Eq. (8),

min⁡S⊆NL(S)subject to∑u∈Scu≤Brec,\min_{\mathcal S\subseteq\mathcal N} \mathcal L(\mathcal S)\quad\text{subject to}\quad\sum_{u\in\mathcal S}c_u\le B_{\rm rec},

where N\mathcal N is the set of 64 units and S\mathcal S the units that reconstruct; the others use the latent path. Appendix C.7 gives the full specification.

  • Why route. Units differ in how much quality they lose under the RoPE-invariance constraint, so the reconstruction budget is spent where it helps most.
  • Objective, cost, budget. L\mathcal L is the mean per-token NLL on the 8K and 16K calibration corpora; cuc_u is proportional to the unit's key rank rur_u; BrecB_{\rm rec} is set in advance to about 30% of the total key-reconstruction work, and the deployed set uses 30.9%.
  • Selection. A cost-aware heuristic: starting from 14 units from a single-unit screening shortlist, units are added by calibration-loss reduction per cost until the budget is reached (23 units), then refined by swapping reconstructed and latent units at a fixed count, keeping a round only when it lowers L\mathcal L by more than a preset threshold (three of four rounds kept). Ranking uses calibration data only, never throughput.
  • Validation. Each kept set passed a pass/fail check on a document-disjoint 16K validation span and an unused training-split span; the initial set was also compared with an earlier 14-unit set on the Table 2 8K/16K corpora, where the candidates ranked as on calibration data. The 32K and 64K spans were never used.
  • Final configuration. Fixed before evaluation, shared by all tokens and lengths, with no extra cache.

Sensitivity (Figure 3b, at fixed memory):

Fallback units0 (Pure)1423 (deployed)
16K perplexity14.349.488.40
16K decode (tok/s)2.302.702.60

Quality responds strongly to the budget and throughput only weakly: going from 14 to 23 units lowers 16K perplexity by 1.08 at a 3.9% throughput cost. This also separates the mechanism from the fallback: the 41 latent units carry the reconstruction-free path, and Table 4 shows that they are usable only because of the cross-head family.

Q4. PaLU was evaluated with its own rank allocation and whitening; RoFold adopts PaLU's allocation.

  • Rank allocation. The shared allocation comes from PaLU's own Fisher-weighted rank search (head-group size 4) at the matched budget.
  • Decomposition. The PaLU baseline uses PaLU's own whitening-based decomposition with its default calibration settings and no extra tuning (Appendix C.1).
  • Verification. The key ranks match on 64/64 units, and the value factors are bitwise identical across all compressed arms.

The allocation is PaLU's own choice at this budget, and RoFold's advantage holds across four PaLU budgets (Table R2), so the comparison does not rest on a configuration chosen in RoFold's favor.

Q5. RAP, compression ratios, model families and downstream tasks.

  • RAP. To our knowledge RAP has no public implementation, and its accuracy relies on LoRA recovery training. Table 4 isolates its core idea, whole RoPE pairs, under identical training-free settings: the cross-head family lowers perplexity by 30%.
  • Compression ratios. PaLU at four ratios (Table R2).
  • Downstream. Official 16K RULER, every task reported (Table R1; Appendix C.8).
  • Model families. The theory applies to any RoPE model whose frequencies repeat across KV heads (standard MHA and GQA). The experiments use Llama-3.1-8B at the 40.1% budget.

Contribution and significance. The paper asks how much data-adaptive freedom an exactly RoPE-compatible low-rank key representation can retain, and gives a complete answer.

  • A necessary and sufficient criterion. Theorem 1 turns "can this compressor skip reconstruction under RoPE?" into a checkable property of its subspace, for any fixed low-rank key representation.
  • A strictly larger exact family. Because RoPE frequencies repeat across KV heads, the family contains data-adaptive cross-head modes; using it instead of whole pairs lowers perplexity by 30% with everything else fixed (Table 4).
  • Training-free and exact as deployed. The deployed basis matches the canonical construction to 1.2×10⁻¹¹, with no training or recovery step.
  • A controllable systems trade-off. At a fixed cache, the decode advantage over PaLU grows from 23.4% at 16K to 56.4% at 64K, and the routing budget sets the quality–throughput point.
  • A direction for further work. Units differ widely in their sensitivity to the invariance constraint; per-unit routing already closes 92% of the 16K perplexity gap between Pure and PaLU, and finer per-unit choices are a natural next step.

We would be grateful if the reviewer would reconsider the score in light of these points.

AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer uEby,

Thank you again for your constructive review. We would like to briefly follow up on our two responses, as the discussion period is approaching its end.

We have addressed your main technical concerns with additional evidence: the implementation overhead behind RoFold-Pure's throughput, a semantics-preserving runtime refinement, the complete hybrid routing procedure, verification of PaLU's original rank allocation and whitening, and new results on 16K RULER and four PaLU compression budgets. The revised manuscript incorporates these clarifications and experiments.

We understand that some limitations remain, particularly regarding broader model coverage and comparisons with other RoPE-compatible methods. Nevertheless, we hope the new evidence helps establish the validity and significance of the proposed approach.

We would greatly appreciate your updated assessment. If any specific concerns remain unresolved, we would welcome the opportunity to clarify them before the discussion closes.

Thank you for your time and consideration.

Reviewer_hkbBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RoFold, a low-rank KV cache method for RoPE-based models. It mixes coordinates with the same RoPE frequency across KV heads, allowing rotation and attention scoring without reconstructing full-width keys. An attention-aware encoder and a static mix of reconstruction-free and reconstruction-based units provide a quality–speed trade-off.

Critical strengths
  1. The paper addresses a practical limitation of low-rank KV caching: reducing cache size does not necessarily reduce decoding cost when historical keys must be reconstructed.
  2. Exploiting repeated RoPE frequencies across KV heads is a useful structural insight. It allows more flexible compatible subspaces than selecting original coordinate pairs.
Critical weaknesses
  1. The main contribution is not evaluated against the closest prior work. Methods such as RAP already support reconstruction-free execution, so the main addition here is the continuous cross-head representation. However, the quantitative comparisons focus on PaLU and truncated SVD, both of which require reconstruction. The coordinate-pair ablation is informative, but it does not replace a comparison with a complete reconstruction-free method. The practical advantage of the proposed representation remains unclear.

  2. The evidence for a better quality–speed trade-off is incomplete. Speedups over PaLU come with worse perplexity. Although the paper shows several RoFold operating points, it does not provide corresponding baseline curves across compression levels. It is therefore unclear whether the advantage would remain if the baselines also traded some quality for speed.

  3. Downstream task quality is not sufficiently evaluated. The quality evaluation relies mainly on perplexity. This leaves open whether the compressed model retains its ability to answer questions and reason using information in the cache, which is central to the method’s practical value.

Questions for authors
  1. How does RoFold compare with RAP or another reconstruction-free method under the same cache budget and comparable calibration or recovery costs? This comparison is needed to establish the value of continuous cross-head mixing.
  2. Does RoFold retain an advantage when baseline ranks are varied under the same cache-memory limit? Alternatively, how do speed and memory use compare at similar model quality?
  3. Could the authors add downstream evaluations, such as question answering and reasoning, alongside perplexity? KV reconstruction similarity does not by itself establish downstream task quality, and equivalence between execution paths does not show that compression preserves the original model’s capabilities.
2 comments
AuthorsOpenReview ↗

Response to Reviewer hkbB

We thank the reviewer for the constructive review, and for recognizing both the practical limitation the paper addresses and the cross-head structural insight.

W1/Q1. Comparison with RAP: we isolate its core idea under identical settings. A full-pipeline comparison with RAP is not possible at comparable cost: to our knowledge RAP has no public implementation, and its accuracy relies on LoRA recovery training, whereas RoFold is training-free. Reconstruction-free key compressors are also few (Table R0 in the general response). We therefore isolate RAP's core idea, keeping whole original RoPE pairs. In terms of Theorem 1 (exact latent execution holds if and only if RΔU=UR^ΔR_\Delta U=U\widehat R_\Delta for every offset Δ\Delta; full statement in the general response), whole-pair selection is the head-aligned subfamily of the RoPE-invariant subspaces, while RoFold optimizes over the whole family, including cross-head complex modes.

  • The controlled test. Table 4 swaps only the family; both arms are training-free and share the rank table, 40.1% budget, 41/23 routing, value path, calibration data and the encoder E = Uᵀ.
  • The result. Perplexity is 8.47 vs. 12.17 at 8K (−30%) and 8.90 vs. 12.89 at 16K (−31%). This is the measured value of continuous cross-head mixing, the representation the reviewer identifies as the main addition.

W2/Q2. RoFold keeps its advantage at every PaLU budget. Across four PaLU compression levels (30.1%–60.0% KV reduction, each with PaLU's own rank search and whitened decomposition; Table R2 in the general response, Appendix C.9), RoFold decodes faster than every one, and none beats it on cache, perplexity and throughput at once.

  • Why PaLU does not speed up. At 16K, PaLU stays at 2.05–2.10 tokens/s at every budget (RoFold: 2.56), because each step still rebuilds every cached key before rotation, the cost RoFold removes.
  • At similar quality. RoFold's 16K perplexity (8.40) lies between PaLU's matched (7.86) and 50.1% (9.47) points, and RoFold is faster than both; the 50.1% point saves memory but is 19% slower and has higher perplexity.
  • The advantage widens with context. At the matched budget, RoFold is 23.4% faster at 16K and 56.4% at 64K (Table 2), at a 6.8–9.3% perplexity gap; Truncated SVD is dominated on both axes at all four lengths.
  • Within RoFold, reconstructing 23 instead of 14 units at fixed memory lowers 16K perplexity from 9.48 to 8.40 for a 3.9% throughput cost (Figure 3b).

W3/Q3. Downstream QA and reasoning: added. The official 16K RULER includes two QA tasks and multi-hop variable tracking, alongside retrieval (Appendix C.8; Table R1 in the general response).

  • QA-1: the Hybrid scores 90 (PaLU 75, Full-KV 85; 20 prompts per task).
  • QA-2: it ties PaLU at 30.
  • Retrieval: single-needle retrieval is preserved (100 / 100 / 95, the same as PaLU).
  • Variable tracking: 84, against PaLU's 97.

Path equivalence guarantees that the latent path computes exactly what reconstruction computes; RULER then tests capability directly, with every task reported. The RoPE-invariance constraint costs different units very different amounts of quality. We quantify this per-unit cost (Appendix C.7) and reconstruct only the 23 of 64 units where it matters most: this raises the RULER average from 30.6 (Pure, no reconstruction) to 76.4 and restores single-needle retrieval to PaLU's level, while keeping the decode advantage over PaLU.

Contribution and significance. The paper asks how much data-adaptive freedom an exactly RoPE-compatible low-rank key representation can retain. Whole-pair selection keeps only part of it. Theorem 1 gives the complete family, which contains data-adaptive cross-head modes, and RoFold uses it without training (30% lower perplexity than whole pairs, Table 4), with a controllable quality–throughput trade-off at a fixed cache. The finding that units differ widely in their sensitivity to the invariance constraint leaves clear room for further work. We would be grateful if the reviewer would reconsider the score in light of these points.

Thank you for the response and the additional experiments. The clarifications on the baselines and the new downstream results largely address my main concerns, so I am raising my score.

Reviewer_xk4jOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RoFold, a low-rank KV-cache compression method that enables reconstruction-free key-side attention under RoPE.

Critical strengths
  • This paper makes sense at the theoretical parts. It avoids the reconstruction and also makes sure the RoPE is performed well at the latent space, not on the full rank key.
  • The use of repeated RoPE frequencies across KV heads is interesting.
  • There are some engineering optimizations, like hybrid execution, to help keep the performance good.
Critical weaknesses
  • The paper is theoretically making sense, but the experiment part is weak. There are no experiments showing the down stream task performance of this method (for example, RULER, LongBench...). Only PPL does not show the strength of a method.
  • The paper only compares PaLU as a main baseline. However, PaLU is a weight decomposition method. But RoFold performs at the activation space. I'd like to see the comparison between RoFold and methods that directly compress Key but without the optimization on the latent RoPE (for example, Eigen Attention, OjaKV...).
  • I understand that the latency of RoFold is higher than FullKV but lower than PaLU. But FullKV uses larger memory, so it will help if the author compares the batch throughput under the same memory budget between FullKV and RoFold; maybe under this setting we can see that RoFold will have a higher throughput.
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer xk4j

We thank the reviewer for recognizing the theoretical part and the use of repeated RoPE frequencies across KV heads, and for the concrete suggestions on the experiments, which the revision follows.

Downstream tasks: added (official 16K RULER). All methods use the same 220 prompts (20 per task) under the official protocol. All 11 tasks are reported with 95% intervals (Appendix C.8; Table R1 in the general response).

  • Retrieval holds where PaLU's does. The Hybrid scores 100 / 100 / 95 on the single-needle tasks and 100 on multi-key-1, identical to PaLU.
  • QA is not degraded relative to PaLU. QA-1 is 90 (PaLU 75, Full-KV 85); QA-2 ties PaLU at 30.
  • Aggregation costs more. The Hybrid is lower on tasks that combine many retrieved items (Table R1); the task average is 76.4 vs. 87.7.

The RoPE-invariance constraint costs different units very different amounts of quality. We quantify this per-unit cost (Appendix C.7) and reconstruct only the 23 of 64 units where it matters most: this raises the RULER average from 30.6 (Pure, no reconstruction) to 76.4 and lifts single-needle retrieval from 60 / 80 / 55 to PaLU's 100 / 100 / 95, while the Hybrid keeps its 23.4–56.4% decode advantage over PaLU at 16K–64K.

Activation-space key compressors: our setting, and what Theorem 1 implies. Our setting is specific: RoFold is training-free and executes RoPE in latent space, a combination that none of the closest methods has (Table R0 in the general response). Eigen Attention and OjaKV, like PaLU and Truncated SVD, are training-free in their base form but reconstruct-then-rotate, so they share one execution path. We therefore compare against the calibration-optimal static fit of this class, PaLU, under one setting: same model, rank table, cache size, value path and harness. PaLU, as evaluated here, is not a raw weight decomposition. It whitens with calibration activations. Within each head group, its truncated SVD minimizes ‖X(W − W′)‖_F, which (for invertible XᵀX) is exactly the Frobenius-optimal rank-r approximation of the calibration keys K = XW. That is the optimum activation PCA attains (Eckart–Young). Its optimality is for the unconstrained problem.

Theorem 1 says what that implies under RoPE. A key subspace supports exact latent RoPE if and only if it is RoPE-invariant, i.e. RΔU=UR^ΔR_\Delta U=U\widehat R_\Delta for every offset Δ\Delta (full statement in the general response).

  • The criterion applies to every fixed linear representation of pre-RoPE keys. An unconstrained fit, however good, cannot run RoPE exactly in latent space unless its subspace happens to be invariant. PaLU is such a fit, and so is activation PCA of pre-RoPE keys.
  • Eigen Attention and OjaKV. Our inspection of the publicly released Eigen Attention and OjaKV implementations confirms that their key-side execution reconstructs full-width keys before RoPE: both cache low-rank coefficients of pre-RoPE keys and rebuild the full-width key history at every decode step before rotating it. In contrast, RoFold directly performs RoPE-compatible computation in the compressed latent space, with the admissible subspaces characterized by Theorem 1.

RoFold's contribution is to constrain the fit to the invariant family, trading a small amount of unconstrained fit for an exactly executable latent path. Table 2 measures that trade at a matched budget.

Matched-memory batching. We agree with the reviewer's reasoning. At a fixed cache memory, RoFold's 59.9% cache holds 1.67× more cached tokens than Full-KV, which translates directly into a larger batch or context capacity. Batch-1 measurements already show the trend: from 16K to 64K, RoFold's decode rate rises from 51% to 63% of Full-KV (Table 2) as the cache becomes a larger share of each step's memory traffic. A fair batched comparison needs equally optimized kernels for every method. Full-KV has mature serving kernels, whereas PaLU's public release provides only a standalone attention-level kernel that is not wired into an end-to-end decode path. We therefore run all methods in the same unfused harness at batch 1.

Contribution and significance. The paper asks how much data-adaptive freedom an exactly RoPE-compatible low-rank key representation can retain. Prior reconstruction-free designs keep only part of it; RAP, for example, fixes whole RoPE pairs. Theorem 1 gives the complete answer: the exact family is strictly larger and, because RoPE frequencies repeat across KV heads, contains data-adaptive cross-head modes. RoFold uses this whole family without training (30% lower perplexity than whole-pair selection, Table 4) and gives a controllable quality–throughput trade-off at a fixed cache. We also find that units differ widely in their sensitivity to the invariance constraint, which leaves clear room for further work. We would be grateful if the reviewer would reconsider the score in light of these points.

AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer xk4j,

Thank you again for your constructive review and for recognizing the theoretical contributions of RoFold.

We would like to briefly follow up on our response and revised manuscript, particularly regarding your main concern about the lack of downstream evaluation.

We have now added official 16K RULER experiments across 11 tasks, reporting all results, including both the preserved retrieval performance and the degradation on more demanding multi-item aggregation tasks. We also clarified the distinction between RoFold and activation-space compressors such as Eigen Attention and OjaKV through their public implementations, and provided additional matched-memory efficiency evidence.

We acknowledge that batched serving experiments and further external baselines would strengthen the empirical evaluation. Nevertheless, we hope these additions help clarify both the practical capabilities and the limitations of the proposed method, while the theoretical characterization and controlled ablations establish its distinct methodological contribution.

We would greatly appreciate your updated assessment of the paper in light of these new results. If any specific concerns remain unresolved, we would welcome the opportunity to clarify them before the discussion period closes.

Thank you for your time and consideration.

Thank you for your clarification. It addresses some of my concerns. I now better understand why certain baselines are missing, and I appreciate your explanation of the differences between RoFold and Eigen Attention. (especially since I know PaLU is a relatively weak baseline, so it makes the results not convincing...)

However, if the main difference is the handling of RoPE, I would expect your method to outperform these approaches. Including experimental comparisons to support this claim would make the paper more convincing.

Therefore, I will maintain my original score.

Reviewer_Wt1zOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper addresses the reconstruction bottleneck in pre-RoPE low-rank KV cache compression, where compressed historical keys must be rebuilt to full width before applying rotary position embeddings (RoPE) at every decode step. The authors prove that exact latent-space RoPE execution is equivalent to RoPE invariance of the decoder subspace (UU), and propose RoFold, which constructs UU from same-frequency cross-head complex modes in grouped-query attention (GQA) and fits an attention-aware left-inverse encoder E=U⊤+C(I−UU⊤)E = U^\top + C(I - UU^\top) via convex ridge regression. To demonstrate the effectiveness of the proposed method, the authors evaluate a static hybrid configuration (applying the reconstruction-free path to 41 of 64 units) on Llama-3.1-8B at a 40.1% KV-reduction budget, comparing perplexity and batch-1 decode throughput against PaLU and Truncated SVD across 8K to 64K context lengths.

Critical strengths

(S1) Theoretical characterization of latent RoPE invariance and two-stage encoder-decoder design: The paper cleanly formalizes the equivalence between exact latent RoPE execution, decoder-subspace invariance (RΔU=UR^ΔR_\Delta U = U \widehat{R}_\Delta), and projector commutation (Theorem 1), while orthogonally decomposing the execution error of general subspaces (Equation 4 and Proposition 2). Exploiting repeated RoPE frequencies across GQA heads to extract data-adaptive modes (HpH_p in Equation 6) and pairing the fixed basis with a ridge-fitted left-inverse correction (EE in Equation 7) provides a well-motivated offline construction that requires no model retraining.

Critical weaknesses

(W1) Severe quality degradation of the pure reconstruction-free endpoint and reliance on hybrid fallback: Although the title and introduction emphasize reconstruction-free execution, applying the proposed path to all units (RoFold-Pure) causes 8K and 16K perplexity to degrade to above 14 in Figure 3b, performing much worse than both PaLU (7.49 and 7.86) and naive Truncated SVD (9.43 and 9.82). To achieve the results in Table 2, the deployed model must fall back to conventional full-width key reconstruction on 23 out of 64 units (35.9%, the 41/23 hybrid in Section 4.3 and Appendix C.7), and still incurs a 6.8% to 9.3% relative perplexity increase over PaLU across all context lengths.

(W2) Slower decoding than uncompressed KV and lack of quantitative comparison with recent RoPE-aware baselines: In Table 2, RoFold's decode throughput (3.294 tok/s at 8K down to 1.153 tok/s at 64K) remains 37% to 49% slower than the uncompressed Full-Width KV baseline (6.456 tok/s at 8K and 1.823 tok/s at 64K), showing that it only reduces the slowdown relative to PaLU rather than accelerating inference. Moreover, while Table 1 and Appendix A discuss directly related RoPE-aware compressors such as RAP (Xin et al., 2026), EliteKV (Zhou et al., 2025), and CommVQ (Li et al., 2025), the paper provides no quantitative comparisons against any of them (explicitly leaving a direct comparison with RAP's pipeline to future work in Appendix C.4.2 and Appendix C.8).

(W3) Narrow empirical scope limited to a single model, a single budget, and perplexity-only evaluation: All experiments are restricted to a single model (Llama-3.1-8B with Hkv=4H_{\text{kv}}=4) at a single 40.1% total KV-reduction budget (compressing keys from 65,536 to 21,120 B/token while leaving values largely uncompressed at 57,344 B/token in Table 6), leaving generalization across model families and GQA group sizes unverified. Furthermore, the paper reports only teacher-forced perplexity without evaluating any downstream long-context retrieval or reasoning benchmarks (such as NIAH, RULER, or LongBench), making it unclear how the 6.8% to 9.3% perplexity gap affects actual generation quality.

(W4) Discrepancy between the canonical whole-mode algorithm and the deployed individual-column implementation: While Section 4.1 (Algorithm 1) and Appendix B.3.4 (Definition 11) present whole-mode selection as the canonical method to guarantee subspace closure, Remark 10 and Appendix C.1 disclose that the bases actually used in the experiments were generated by an earlier individual-column selection recipe (Definition 8, which is not compatible by construction) and only verified numerically post hoc (Appendix C.5).

References

  • Junyan Li, Yang Zhang, Muhammad Yusuf Hassan, Talha Chafekar, Tianle Cai, Zhile Ren, et al. CommVQ: Commutative vector quantization for KV cache compression. In International Conference on Machine Learning (ICML), 2025.
  • Jihao Xin, Tian Lyu, David Keyes, Hatem Ltaief, and Marco Canini. RAP: KV-cache compression via RoPE-aligned pruning. arXiv preprint arXiv:2602.02599, 2026.
  • Yuhao Zhou, Sirui Song, Boyang Liu, Zhiheng Xi, Senjie Jin, Xiaoran Fan, Zhihao Zhang, Wei Li, and Xuanjing Huang. EliteKV: Scalable KV cache compression via RoPE frequency selection and joint low-rank projection. arXiv preprint arXiv:2503.01586, 2025.
Questions for authors

Please see the critical weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Wt1z (1/2)

We thank the reviewer for the detailed review and the specific questions on the empirical case. We answer each point below; Theorem 1 and Tables R0–R4 are in the general response.

W1. RoFold is designed as a trade-off at matched memory, not as a compressor that beats reconstruction on every axis. Pure is an analysis endpoint; the Hybrid is the proposed method.

We do not claim that RoFold outperforms reconstruction-based compressors in both quality and throughput. The claim is a new operating point at the same persistent cache: a faster decode for a bounded quality cost, with an explicit knob between the two.

  • Pure is a research endpoint, not a recommended setting. Routing all 64 units through the latent path shows that an entire 8B model can run its key side with no reconstruction at all. It also measures what the invariance constraint costs in the units where it is most restrictive; this is why Pure's perplexity is above even Truncated SVD's. The revision states that Pure is not the deployed configuration. The reconstruction-free path named in the title is the mechanism, and the Hybrid runs it on 41 of 64 units.
  • The Hybrid is the proposed configuration, and its finding is that a minority of units carries most of the quality loss. Reconstructing 23 of 64 units (30.9% of the key-reconstruction work), at the same cache size:
    • lowers 16K perplexity from 14.34 to 8.40, closing 92% of the gap between Pure and PaLU (7.86);
    • raises the RULER average from 30.6 to 76.4;
    • keeps the throughput advantage: the Hybrid is faster than PaLU at every evaluated length, by 23.4–56.4% at 16K–64K, and it dominates Truncated SVD in both perplexity and throughput at all four lengths.

Why some units need the fallback. The RoPE-invariance constraint costs different units very different amounts of quality. A minority of units carries most of Pure's loss, which is why reconstructing 23 of them closes 92% of the gap; the single-unit screening in Appendix C.7 measures this per-unit cost. Exactness and quality are therefore separate properties:

  • Exactness. Theorem 1 (full statement in the general response) guarantees that the latent path computes exactly what reconstruct-then-rotate computes, (U⊤q)⊤R^Δz=q⊤RΔUz(U^\top q)^\top\widehat R_\Delta z=q^\top R_\Delta Uz, for the same compressed representation.
  • Quality. This depends on how much key energy a rank-r RoPE-invariant subspace retains in each unit, and that differs from unit to unit.

The routing problem, Eq. (8), is min⁡SL(S)\min_{\mathcal S}\mathcal L(\mathcal S) subject to ∑u∈Scu≤Brec\sum_{u\in\mathcal S}c_u\le B_{\rm rec}, where S\mathcal S is the set of units to reconstruct. It assigns the expensive units to a reconstructed path at the same memory, under an explicit reconstruction-work budget.

Evidence that the loss sits in the representation, not the execution. We ran the same compressed representation (same basis, encoder and ranks) through both read paths on 10 pre-registered RULER examples (variable tracking and multi-key-2, under the official protocol):

  • greedy outputs were token-identical on 10/10 examples;
  • gold-answer log-probabilities agreed within 0.05 nat/token;
  • the last-query attention distributions matched (top-10 key overlap 1.00, total variation 0.005).

The layerwise output difference sits at our pre-registered 1×10⁻² median tolerance (1.0–1.1×10⁻²), uniformly across layers. Consistent with Theorem 1, on these examples the read path does not cause the gap; the loss is in the representation.

W2a. Full-KV is a memory reference, not a matched baseline. Full-KV stores 131,072 B/token, 1.67× the cache of every compressed arm. Low-rank KV compression (here Truncated SVD, PaLU and RoFold, i.e. every method in Table 2 other than Full-KV) is by design a trade-off among memory, quality and speed. Because RoPE depends on position, a low-rank cache must store keys before rotation, so every cached key has to be processed again at each decode step; Full-KV stores rotated keys once and reads them directly. Truncated SVD and PaLU, like RoFold's 23 reconstructed units, rebuild every cached key to full width before RoPE and scoring; RoFold's latent units instead rotate each latent with an r×r operator and score it against the projected query U⊤qU^\top q. In exchange, the cache is 40.1% smaller, which permits longer contexts or larger batches in the same memory. No compressed method in Table 2, PaLU included, is faster than Full-KV at batch 1. The relevant comparison is therefore at a fixed cache size, and at that size RoFold is the fastest method in Table 2. Table 2 also shows the trend that matters for long contexts:

  • from 16K to 64K, RoFold's decode rate rises from 51% to 63% of Full-KV;
  • PaLU stays near 40% (41% → 40%);
  • Truncated SVD moves from 39% to 44%.

RoFold closes the gap fastest as context grows, because it removes reconstruction.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer Wt1z (2/2)

W2b. RAP, EliteKV, CommVQ. These methods solve different problems. Theorem 1 places each of them precisely.

  • RAP keeps whole original RoPE pairs. In Theorem 1's terms, that is the head-aligned subfamily of invariant subspaces. Table 4 isolates exactly this representational difference at a fixed rank table, budget, routing, value path, calibration data and encoder (E = Uᵀ in both arms): the cross-head family lowers perplexity from 12.17 to 8.47 at 8K, and from 12.89 to 8.90 at 16K. RAP's full pipeline adds LoRA fine-tuning, whereas RoFold is training-free. A comparison with that full pipeline was not possible: to our knowledge, RAP has no public implementation at the time of writing.
  • EliteKV reallocates frequency capacity with uptraining; it is orthogonal to the choice of an exact latent subspace and could be combined with it. CommVQ is a quantization method rather than a low-rank one: it reduces the bits per cached entry, not the key dimension, so it belongs to a different class from the low-rank methods compared here and could be applied on top of them.

RoFold's contribution is the complete characterization of the invariant family, together with a training-free construction that uses all of it.

W3. Scope.

  • Downstream: we added official 16K RULER (general response; Appendix C.8). The Hybrid matches PaLU on all single-needle tasks, is not lower than PaLU on the two QA tasks, and is lower on multi-item aggregation. Every task is reported.
  • Generality of the theory: the theory needs only RoPE frequencies that repeat across KV heads. That holds for standard MHA and GQA RoPE models, so the characterization is not specific to Llama.
  • Budget composition: the 40.1% budget is matched in total and in composition. Every compressed arm stores 78,464 B/token with the same value factors (bitwise identical), so all differences come from the key path, which is the part RoFold changes.
  • One RoFold budget, four PaLU budgets: RoFold is evaluated at one budget. To check that the comparison is not an artifact of this choice, we ran PaLU at four budgets (Table R2 in the general response); RoFold is faster than all four, and none dominates it.
  • Group size: Theorem 1 does not depend on the group size. The group size (Hkv=4H_{\rm kv}=4 here) sets how many heads share each frequency, and hence how rich the cross-head family is. Other group sizes are not measured in this revision.
  • Empirical scope: the evaluation is on Llama-3.1-8B at the 40.1% budget, as stated in the paper.

W4. The deployed basis is the canonical construction, verified to machine precision. The deployed individual-column basis and the canonical whole-mode basis span the same subspace: the projector distance is 1.2×10⁻¹¹, and the representation-map residual of 1.3×10⁻¹¹ shows that they differ by an orthogonal change of basis. All 64 units pass every closure and locality check (Table 5).

We also corrected two table details; all PASS decisions are unchanged.

  • Rank cutoff. The global rank-cutoff entry is 7.7×10⁻³.
  • BF16 encoder check. The revision now states the actual rule: a unit passes if either its absolute residual or its normalized residual is ≤ 10⁻². The largest normalized residual is 1.7×10⁻³. The largest absolute residual is 9.9×10⁻², in one unit, which passes on the normalized criterion.

Summary of the contribution. The weaknesses raised concern the empirical scope. The paper's soundness rests on results that the quality–speed trade-off does not affect:

  • An exact criterion. Theorem 1 is a necessary and sufficient condition for running RoPE rotation and QK scoring exactly on a low-rank key cache: the subspace must be RoPE-invariant. Whether a compressor can skip reconstruction becomes a checkable property of its subspace. Whole-pair selection (RAP) is a special case; unconstrained fits (PaLU, Truncated SVD) in general fall outside the class.
  • A strictly larger exact family. Because RoPE frequencies repeat across KV heads, the invariant family contains a continuum of cross-head complex modes. Using it instead of whole pairs lowers perplexity by 30% with everything else fixed (Table 4).
  • A training-free construction, executed exactly. The deployed basis matches the canonical construction to a projector distance of 1.2×10⁻¹¹, and all 64 units pass every closure and locality check.
  • A measured systems effect at matched memory. RoFold's latent units remove a per-step cost that grows with context; its decode advantage over PaLU grows from 23.4% at 16K to 56.4% at 64K.

We hope these clarifications address the reviewer's concerns, in particular on soundness, and we would be grateful if the reviewer would reconsider the assessment in light of them.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2): Position, main results and revision summary

We thank the reviewers for recognizing the theoretical characterization of exact latent RoPE execution and the cross-head structure it exposes. The concerns are about the empirical case. Below we state our position and what the revision adds; Theorem 1 and all result tables are in the next comment.

What the paper is about. We ask how much data-adaptive freedom an exactly RoPE-compatible low-rank key representation can retain. RoFold characterizes the full family of exactly compatible subspaces (Theorem 1), quantifies how far any other subspace deviates from it, and builds a training-free basis over this family, giving a controllable quality–throughput trade-off at a fixed cache: at the same memory, it decodes 23–56% faster than PaLU at 16K–64K for a 7–9% perplexity gap. It is not proposed as a compressor that wins on every axis.

How RoFold differs from RAP. RAP prunes the key to a fixed subset of whole RoPE pairs and recovers accuracy with LoRA training. RoFold differs at every stage: (i) theory: it characterizes the whole exactly compatible family, of which whole pairs are one special case; (ii) basis: it chooses data-adaptive directions in that family, including cross-head modes, instead of selecting existing coordinates; (iii) encoder: an attention-aware encoder recovers quality without changing the subspace (Table 3: 11–12% higher perplexity without it); (iv) training: none. RAP has no public implementation to our knowledge, so Table 4 isolates the basis difference under identical settings: restricting RoFold to whole original pairs raises perplexity by 43.6% (8K) and 44.8% (16K).

Why the quantitative baselines are PaLU and Truncated SVD. The other RoPE-aware compressors change the model: RAP adds LoRA recovery and EliteKV adds uptraining (Table R0). Their results combine the key representation with training, so they cannot be matched to a training-free method at the same cache. We therefore compare the training-free methods under one setting: same model, rank table, cache size, value path and harness.

What the evidence establishes. Several reviewers ask for broader experiments. Our evaluation is built around the paper's question, and each claim has a direct, controlled test: exactness is certified on all 64 deployed units and checked end to end through both read paths (Table R3); the value of the larger exact family is isolated with everything else fixed (Table 4, 30% lower perplexity than whole pairs); the systems effect is measured at a matched cache over four context lengths and four PaLU budgets (Table 2, Table R2); and downstream capability is tested on all 11 RULER tasks (Table R1). The setting is deliberately strict, training-free and matched in cache, rank table and value path, so every difference is attributable to the key representation. No prior method combines exact reconstruction-free execution with a training-free, data-adaptive basis (Table R0).

Main results, all at the same 78,464 B/token cache.

  • RoFold-Hybrid 41/23 (41 of 64 units latent, 23 reconstructed) is better than Truncated SVD in both perplexity and throughput at all four lengths.
  • It decodes faster than PaLU at every length, by 23.4% (16K), 32.2% (32K) and 56.4% (64K), at a 6.8–9.3% perplexity gap. The advantage grows with context because its 41 latent units never rebuild their cached keys.
  • PaLU at four compression levels (30.1–60.0% KV reduction) is slower than RoFold at every level, and none is better than RoFold in cache, perplexity and throughput at once (Table R2).
  • The mechanism is exact as deployed: all 64 units pass every closure and locality check, and the deployed basis matches the canonical construction to 1.2×10−111.2\times10^{-11}.

What is new (blue in the revision; Tables R3–R4 in this response).

  • Downstream: 16K RULER, 11 tasks, identical prompts for all methods (Table R1; Appendix C.8). The Hybrid matches PaLU on single-needle retrieval and is not lower on QA; its gap is concentrated in tasks that aggregate many items.
  • Read-path check: the same compressed representation gives token-identical outputs through both read paths on 10/10 examples (Table R3).
  • Runtime: a semantics-preserving refinement speeds up Pure by 1.22× at 16K with bitwise-identical outputs (Table R4; uEby Q1).
  • Baselines and routing: PaLU at four compression levels (Appendix C.9), PaLU's setup (Appendix C.1) and the full routing specification (Appendix C.7).
AuthorsOpenReview ↗

General Response (2/2): Theorem 1 and result tables

This comment collects the reference material for the general response: the statement of Theorem 1, the setting of the closest methods (Table R0), and the new experiments of the revision (Tables R1–R4).

Theorem 1 (Appendix B.1.1). Let UU (U⊤U=IrU^\top U=I_r) be the decoder basis of a low-rank key cache, zs=Eksz_s=Ek_s the cached latent of key ksk_s, P=UU⊤P=UU^\top, and RΔR_\Delta the RoPE rotation at relative offset Δ\Delta. For each Δ\Delta, the following are equivalent:

∃R^Δ:RΔU=UR^Δ  ⟺  RΔrange(U)=range(U)  ⟺  RΔP=PRΔ  ⟺  q⊤RΔUz=(U⊤q)⊤R^Δz∀q,z,\exists\widehat R_\Delta: R_\Delta U=U\widehat R_\Delta \iff R_\Delta \mathrm{range}(U)=\mathrm{range}(U) \iff R_\Delta P=PR_\Delta \iff q^\top R_\Delta U z=(U^\top q)^\top \widehat R_\Delta z \quad\forall q,z,

and then R^Δ=U⊤RΔU\widehat R_\Delta=U^\top R_\Delta U is unique. The last condition is exact latent execution: each cached latent is rotated and scored by an r×rr\times r operator, without reconstructing UzsUz_s. Whole-pair selection (RAP) is an invariant special case; unconstrained fits such as PaLU and Truncated SVD are in general not invariant.

Table R0. Setting of the closest key-side methods (as in Table 1 of the paper).

MethodTraining-freeReconstruction-freeKey basis
PaLU✓×low-rank factors of the key projection
EliteKV× (uptraining)×selected RoPE frequencies with a joint low-rank projection
RAP× (LoRA recovery)✓a fixed subset of whole RoPE pairs
RoFold-Pure✓✓cross-head modes chosen from calibration data, over the full RoPE-invariant family
RoFold-Hybrid✓41 of 64 unitssame basis; 23 units reconstruct

Table R1. RULER at 16K (Appendix C.8). Official RULER protocol, 11 tasks, 20 prompts per task, identical prompts for every method; Llama-3.1-8B-Instruct; all compressed methods at the matched 40.1% budget. The 95% intervals are in C.8.

TaskFull-KVPaLURoFold-HybridRoFold-Pure
NIAH single 110010010060.0
NIAH single 210010010080.0
NIAH single 310095.095.055.0
NIAH multi-key 110010010060.0
NIAH multi-key 210010075.00.0
NIAH multi-value98.887.565.023.8
NIAH multi-query98.885.051.316.3
Variable tracking99.097.084.00.0
Frequent words88.395.050.016.7
QA 185.075.090.020.0
QA 255.030.030.05.0
Average93.287.776.430.6

Table R2. PaLU at four compression levels (Appendix C.9). Each PaLU point uses PaLU's own rank search and whitened decomposition at that budget. Throughput comes from one session on one GPU with all arms interleaved, three repeats each, so ratios within this table are directly comparable; the matched PaLU and RoFold values agree with Table 2 to within 2%.

MethodKV reductionB/tokenPPL 8K / 16Ktok/s 8K / 16K
PaLU60.0%52,48012.95 / 13.742.99 / 2.10
PaLU50.1%65,4089.09 / 9.472.99 / 2.07
PaLU (matched)40.1%78,4647.49 / 7.863.02 / 2.09
PaLU30.1%91,6486.90 / 7.232.93 / 2.05
RoFold-Hybrid40.1%78,4648.00 / 8.403.25 / 2.56

Table R3. One compressed representation, two read paths (Wt1z W1). The same basis, encoder and ranks are executed by the latent path and by reconstruct-then-rotate on 10 pre-registered RULER examples (variable tracking and multi-key 2, official protocol).

QuantityLatent vs. reconstruct-then-rotate
Greedy outputstoken-identical on 10/10
Gold-answer log-probability difference≤ 0.05 nat/token
Last-query attention: top-10 key overlap / total variation1.00 / 0.005
Layerwise output difference (median relative Frobenius)1.0–1.1×10⁻², uniform across layers

Table R4. Semantics-preserving runtime refinement at 16K (uEby Q1). Paired runs on the same harness; logits and 8K/16K perplexity are bitwise identical before and after. The paper's throughput tables keep the original runtime.

Full-KVPaLURoFold-HybridRoFold-Pure
Decode throughput, refined / original1.05×1.06×1.15×1.22×

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.