- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper devise iS-KV, an online low-rank KV-cache compression method. Instead of evicting tokens, it keeps a compressed token representation for every position. A window of recent tokens stays exact, and older K and V states are folded in blocks into a bounded-rank factorization. The central design point is that when new blocks change the basis, the coordinates of all previously stored tokens are updated jointly via Brand’s block incremental SVD. The authors show that updating the basis alone causes the stored history to “drift” (Fig. 1c: 1.55/1.40 vs 0.023/0.051 for K/V). On MATH-500 at matched persistent-KV bytes, iS-KV beats other SotA compression methods such as R-KV and SnapKV in all six settings (e.g. Qwen3-8B at 5.64×: 89.2% vs 68.8% for R-KV). Further results cover AIME 2024/25, a RULER prefill test, an ablation and one latency/memory measurement.
Critical strengths
S1. Well-motivated problem and clear thesis. Eviction is irreversible and long reasoning traces revisit old tokens. Fig. 1a supports this with a probe, although it is a weak one (see Weaknesses). S2. Careful memory comparrison to other compressors. This is better practice than much of the literature. S3. Large gains over evictors on MATH-500 tasks. On Qwen3-8B the gap to R-KV (20+ points at similar compression) is far outside sampling noise. Paired bootstrap intervals are given for the R1-Distill comparisons. S4. Honest reporting. The paper discloses the FP16 overflow at long generation, the inferred SnapKV counts, the absence of a fused kernel, and the lack of any throughput claim. Appendix A is thorough, with settings, propositions and diagnostics. S5. Although it was hard for me to follow, the algorithmic core is clean.
Critical weaknesses
W1. Novelty is limited and the central “drift” finding is framed too strongly. The algorithm is Brand’s incremental SVD, which the paper acknowledges. Joint update of coefficients is intrinsic to that algorithm. The “drift” comparison in Fig. 1c is nice empirical demonstrarion on KV Cache, but it's simply is a linear-algebra identity of the algorithm. The comparison that bears on real methods is the frozen-basis ablation (Table 3), but it uses 80 questions, one model and one sample each. “Drift” is also used for two different quantities: Fig. 1c measures relative change against the history before each update, while Eq. 8 and Table 5 measure excess error against a fresh SVD. I recommend to use separate names or mention it clearly
W2. The generality is weakly established. Only two 8B dense models with GQA are evaluated on reasoning. Both use 8 KV heads of dimension 128, so d = 1024. Architectures whose KV representation is already compact such as MLA models -- DeepSeek V2/V3 are absent. Their cached latent is already low-rank by construction. I would expect much less headroom, and the paper’s low-rank-structure premise may not hold. If rank 64-128 gives a modest ratio there, the practical scope is narrower than the abstract suggests. This needs testing, or at least a discussion. Note that “DeepSeek-R1-Distill-Llama-8B” is not an MLA model! so it does not cover this case. The 4x to7x ratios are on top of GQA, and the authors can state clearly the gain relative to an MHA baseline. Larger models, other families and non-math tasks (code, agentic, long-form) are absent. The RULER experiment tests prefill only, with 10 examples per task and degradations of 2-14 points. Llama-3.1-8B at rank 96 falls from 85.6 to 71.7. A single global rank is used for all layers, though the paper itself reports that the 99%-energy rank varies from 122 to 338 across layers. Adaptive-rank methods (e.g. STAR-KV, cited) are not compared.
W3. Baselines and positioning relative to prior low-rank work. Related work and comparisons do not cover the closest competitors. The paper’s claim is that iS-KV works better than eviction because it keeps every token in low-rank form. The natural competitors are therefore other SVD/low-rank KV methods. The only ones compared against are eviction methods (R-KV, SnapKV), which the paper itself argues are weak for reasoning, so large margins over them are expected and say less about the method. §2 lists Palu, Eigen Attention, ShadowKV, StiefAttention, STAR-KV and KV-CoRE, but dismisses them as informing “basis construction and rank choice.” None is evaluated, even though several target the same design space (low-rank factorization to shrink the KV cache). The authors need to explain why they cannot be run in this setting, or run them. Static-basis methods such as Palu or ShadowKV are especially relevant because they are what the “frozen basis” ablation (Table 3) approximates. Other low-rank KV-compression methods, e.g. LoRC and ReCalKV, are not cited or discussed. Related work should cover this area more fully, and at least some of these methods should appear in the comparison.
The one online low-rank comparison OjaKV, is under-introduced and inconclusive. OjaKV is the closest prior work: online, context-aware, and low-rank. Yet it is absent from section 2. It first appears in section 4.1 (“Comparison with OjaKV”), and the paper never explains how it works or how it differs from iS-KV (e.g. how it updates its basis, whether old coordinates are kept consistent). A reader cannot tell whether the comparison tests the “joint update of basis and history” idea, which is the paper’s central claim. The comparison is also incomplete. The released configuration reaches only about 1.30× compression, so it is not matched in memory. The reduced-rank runs were stopped after 30, 28 and 60 generations, all of which looped to the length limit, and none was scored (Table 2). This could reflect a limitation of OjaKV, a poor rank configuration, or an integration problem. Without analysis, it supports at most the narrow statement that OjaKV did not work at these settings in the authors’ hands. Please introduce OjaKV in related work, explain the mechanism difference, and run it to completion (full 500 questions, or a subset with intervals) at its recommended settings and at matched memory.
W4. Performance claims are not established and the likely costs are under-measured. The only latency result is a single run per configuration at batch size 1 on an A40, with 1,024 decode steps after synthetic 8K/16K/32K prompts. iS-KV is 9–22% slower, and the authors state the numbers are “not statistically stable.” This slowing down per-se, is acceptable as a result, but it should be well established and explained. A. The motivating regime (10K–30K generated tokens) is untimed. B. Per-block update cost as well as reconstruction cost O(n) per step (with n being the sequence length), so overhead should grow with generated length. The reported overhead shrinking with longer input is not informative for long generation. C. Dense K/V are reconstructed at every decode step, so peak memory includes a temporary that the reported savings exclude. “Persistent-KV compression” is therefore not peak-memory compression, and the paper acknowledges this only in the appendix. I’d ask for peak-HBM numbers in the main text.
Back of the envelope evaluation that is missing in the main body or Appendix: Without compression, computing attention scores and the weighted value sum takes approximately Reconstruct compressed cached keys and values requires: . The ordinary attention computation still follows. This amount of work is comparable to the whole attention computation and hence, it must be discussed -- attended to in the paper (and there is also the addition of the block of new token every b tokens)
Questions for authors
Q1. How does iS-KV behave on an MLA model (e.g. a DeepSeek-V2/V3-family model), where the cache is already a low-rank latent? What compression at what accuracy cost is achievable there? Q2. What are wall-clock latency and peak-HBM at 16K–32K generated tokens, with batch sizes above 1? At what generation length does reconstruction dominate? Q3 How does iS-KV compare with KIVI/KVQuant at matched bytes, and does it compose with them? Q4. Can you compare against a frozen-basis or periodic-refit baseline on the full MATH-500, since those are the realistic alternatives to the stale-coordinate strawman? Q5. Is the OjaKV failure (all generations looping) attributable to the method, or to rank/configuration choices? Can you evaluate it with its recommended settings across more questions? Q6. Why do the values degrade more than the keys, and would a separate (higher) rank for values change the picture at equal total memory? Q7. The writing is clear and the appendix is complete. Small improvements:
- separate the two uses of “drift,”
- in figure 1. "the share that later receives strong attention." What is strong attention? this is explained only in the appendix...