- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the view of Rotary associative memory for exploring and analyzing KV cache compression. The cache is decomposed into frequency bands (inspired by the motivation of RoPE), and discussions on the retrieval range, capacity at distance, and positional aliasing have been conducted. A perturbation certificate is also proposed.
Critical strengths
- This work proposes an analytical framework (RoAM) for KV cache compression and a corresponding set of certificates.
Critical weaknesses
- This paper analyzes only RoPE. I suggest considering the extrapolation schemes commonly used in real-world systems, such as YaRN, and conducting a more in-depth analysis of them. In addition, during mid-training in some industrial LLMs, there is often a long-context stage in which the RoPE base is progressively increased to accommodate longer sequences (e.g., RoPE base from 10k to 500k). Does the analytical framework still hold in such cases?
- Retrieval is mainly evaluated at 8192 tokens, which is shorter than practical settings (at least 128k or even up to 1M for recent LLMs). More discussions and experiments are required.
Questions for authors
- Plain retrieval alone may not be sufficient to fully reflect an LLM’s long-context capability. I suggest evaluating on more realistic long-context tasks besides retrieval.
- It is interesting to provide an additional analysis of partial RoPE and NoPE, as well as of this situation under hybrid models, since the current trend implies that RoPE may not be the only effective mechanism for long-context capability (e.g., NoPE and linear attention may work).
3 comments
Thank you for the timely response. It mostly addresses my concerns about the extrapolation schemes and the experiments with longer context lengths. Since my main concerns have been largely resolved, I am inclined to raise my score to 3.
Thank you for reading the response and the revision, and for the updated assessment.
Executive Summary
Thank you for the careful review. The revised PDF (changes in blue; pointers use its numbering) answers each point:
New Results
New Table 5 (seed 42; complete in new Table 31): PG-19 perplexity and mean RULER accuracy, full cache and with the slowest or fastest half of the bands zeroed.
W1: YaRN, Interpolation and Base Changes
New Corollary 4 (proof in Appendix D): interpolation divides every frequency by s, YaRN only the slow bands. (i) Both satisfy cov(Θ′;sL0)=cov(Θ;L0), the extended context inherits the coherent bands of the original, no others; (ii) YaRN leaves the fast bands, hence resolution and aliasing, unchanged, while interpolation moves the top-band onset off Δ=1; (iii) YaRN's attention factor is a change of β, not of the spectrum. Predictions were registered before any run: all 17 buffers match to 3×10−7, all 272 onset and alias integers reproduce, the coverage identity holds on all 11 variants, and the per-head fit holds (median r 0.82 to 0.94). Stock models that fail past their trained length recover under YaRN with no fine-tuning; interpolation restores coverage but spends resolution (Llama-3-8B in range: perplexity 8.0 to 118, retrieval 0.84 to 0.00); in every row the slow half carries retrieval. Weights trained under a schedule retrieve better than stock weights (0.23 against 0.10).
A base change from θ to θ′ multiplies band m by (θ′/θ)−2m/d: a per-band rescaling of the same form, so Theorem 5 gives the new certified range L∗ in closed form and Theorem 4 the alias integers. The fleet spans bases 104 (Mistral v0.1, Gemma) to 107 (Granite), Llama-3.1-8B carries its long-context schedule, and every spectrum prediction holds on each (Table 3). What a progressive schedule transfers across stages is a property of the weights, not the spectrum; the Mistral pair is that measurement.
W2: Length
F.12 repeats the capture-side measurements at 32,768 on two models (detrended r 0.924 and 0.888 against 0.895 and 0.885; Table 29), retrieval at 32,768 (macro 0.853 against 0.882, three seeds), and band ablation at 65,536 on Llama-3.1-8B (Table 30): macro 0.820, the slow-half arm 0.000 in all fifteen (subtask, seed) cells with perplexity multiplied by 161, the fast-half arm −0.008±0.012. F.13 runs to 131,072. The compression audit stays at 8,192, where ten models, four methods and three seeds fit; 131,072-token cells with a compressor in the loop exceed 48 GB (Section 9).
Q1: Beyond Plain Retrieval
The seven-task retrieval macro of Table 4 contains HotpotQA (multi-hop F1) and LongBench passage retrieval. New Table 36 (F.15) generates five LongBench subtasks (NarrativeQA, Qasper, MultiFieldQA, HotpotQA, passage retrieval) on Llama-3.1-8B-Instruct after a compressed 8,192-token prompt: zeroing the slow half takes all five to zero; KIVI-2 moves the four QA subtasks by at most 0.04 and passage retrieval by 0.42; H2O costs the QA subtasks 0.08 to 0.19.
Q2: Partial RoPE, NoPE, Hybrids
Section 2 now states the scope. Un-rotated dimensions are bands at θ=0: coherent at every distance, a floor to the kernel, no resolution, no aliasing; Theorem 2 permits any linear map on them. Under NoPE the band statements do not apply; Proposition 3 holds. In hybrids, RoPE layers are covered layer by layer; state-space and linear-attention layers keep a fixed-size state, not an addressed store, and lie outside the frame.
We thank the reviewer for the thorough reading and gladly discuss further.