acceptodds
Under review as a conference paper at ICLR 2027

iS-KV: Online Low-Rank KV Cache Compression via Block-Incremental SVD

Abstract

Long chain-of-thought reasoning substantially increases KV-cache memory during autoregressive decoding, as every generated token introduces new key and value states and causes the cache to grow linearly with decoding length. Existing KV-cache compression methods typically control this growth through token eviction, but irreversible deletion can remove historical states that later reasoning may need to revisit. SVD-based low-rank compression provides an alternative by retaining all positions with a more compact representation. However, extending it from a fixed prompt cache to online decoding is non-trivial. Through our investigation, we find that if the basis is updated for new tokens while old tokens keep their coordinates in the old basis, the stored history drifts substantially. Based on this observation, we propose iS-KV, an online low-rank KV-cache compression method for long-horizon reasoning. iS-KV keeps a recent window exact while incrementally folding older states into bounded-rank representations. As the low-rank basis evolves, it synchronizes historical coordinates with the updated basis to maintain representation consistency. On DeepSeek-R1-Distill-Llama-8B, iS-KV achieves 82.6% accuracy at 4.06× persistent-KV compression, close to the original model’s 83.6%. On Qwen3-8B, it achieves 89.2% accuracy at 5.64× compression. Under matched memory budgets, iS-KV consistently outperforms token-eviction baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
NLP: (Large) Language Models, KV Cache

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vwLVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper devise iS-KV, an online low-rank KV-cache compression method. Instead of evicting tokens, it keeps a compressed token representation for every position. A window of recent tokens stays exact, and older K and V states are folded in blocks into a bounded-rank factorization. The central design point is that when new blocks change the basis, the coordinates of all previously stored tokens are updated jointly via Brand’s block incremental SVD. The authors show that updating the basis alone causes the stored history to “drift” (Fig. 1c: 1.55/1.40 vs 0.023/0.051 for K/V). On MATH-500 at matched persistent-KV bytes, iS-KV beats other SotA compression methods such as R-KV and SnapKV in all six settings (e.g. Qwen3-8B at 5.64×: 89.2% vs 68.8% for R-KV). Further results cover AIME 2024/25, a RULER prefill test, an ablation and one latency/memory measurement.

Critical strengths

S1. Well-motivated problem and clear thesis. Eviction is irreversible and long reasoning traces revisit old tokens. Fig. 1a supports this with a probe, although it is a weak one (see Weaknesses). S2. Careful memory comparrison to other compressors. This is better practice than much of the literature. S3. Large gains over evictors on MATH-500 tasks. On Qwen3-8B the gap to R-KV (20+ points at similar compression) is far outside sampling noise. Paired bootstrap intervals are given for the R1-Distill comparisons. S4. Honest reporting. The paper discloses the FP16 overflow at long generation, the inferred SnapKV counts, the absence of a fused kernel, and the lack of any throughput claim. Appendix A is thorough, with settings, propositions and diagnostics. S5. Although it was hard for me to follow, the algorithmic core is clean.

Critical weaknesses

W1. Novelty is limited and the central “drift” finding is framed too strongly. The algorithm is Brand’s incremental SVD, which the paper acknowledges. Joint update of coefficients is intrinsic to that algorithm. The “drift” comparison in Fig. 1c is nice empirical demonstrarion on KV Cache, but it's simply is a linear-algebra identity of the algorithm. The comparison that bears on real methods is the frozen-basis ablation (Table 3), but it uses 80 questions, one model and one sample each. “Drift” is also used for two different quantities: Fig. 1c measures relative change against the history before each update, while Eq. 8 and Table 5 measure excess error against a fresh SVD. I recommend to use separate names or mention it clearly

W2. The generality is weakly established. Only two 8B dense models with GQA are evaluated on reasoning. Both use 8 KV heads of dimension 128, so d = 1024. Architectures whose KV representation is already compact such as MLA models -- DeepSeek V2/V3 are absent. Their cached latent is already low-rank by construction. I would expect much less headroom, and the paper’s low-rank-structure premise may not hold. If rank 64-128 gives a modest ratio there, the practical scope is narrower than the abstract suggests. This needs testing, or at least a discussion. Note that “DeepSeek-R1-Distill-Llama-8B” is not an MLA model! so it does not cover this case. The 4x to7x ratios are on top of GQA, and the authors can state clearly the gain relative to an MHA baseline. Larger models, other families and non-math tasks (code, agentic, long-form) are absent. The RULER experiment tests prefill only, with 10 examples per task and degradations of 2-14 points. Llama-3.1-8B at rank 96 falls from 85.6 to 71.7. A single global rank is used for all layers, though the paper itself reports that the 99%-energy rank varies from 122 to 338 across layers. Adaptive-rank methods (e.g. STAR-KV, cited) are not compared.

W3. Baselines and positioning relative to prior low-rank work. Related work and comparisons do not cover the closest competitors. The paper’s claim is that iS-KV works better than eviction because it keeps every token in low-rank form. The natural competitors are therefore other SVD/low-rank KV methods. The only ones compared against are eviction methods (R-KV, SnapKV), which the paper itself argues are weak for reasoning, so large margins over them are expected and say less about the method. §2 lists Palu, Eigen Attention, ShadowKV, StiefAttention, STAR-KV and KV-CoRE, but dismisses them as informing “basis construction and rank choice.” None is evaluated, even though several target the same design space (low-rank factorization to shrink the KV cache). The authors need to explain why they cannot be run in this setting, or run them. Static-basis methods such as Palu or ShadowKV are especially relevant because they are what the “frozen basis” ablation (Table 3) approximates. Other low-rank KV-compression methods, e.g. LoRC and ReCalKV, are not cited or discussed. Related work should cover this area more fully, and at least some of these methods should appear in the comparison.

The one online low-rank comparison OjaKV, is under-introduced and inconclusive. OjaKV is the closest prior work: online, context-aware, and low-rank. Yet it is absent from section 2. It first appears in section 4.1 (“Comparison with OjaKV”), and the paper never explains how it works or how it differs from iS-KV (e.g. how it updates its basis, whether old coordinates are kept consistent). A reader cannot tell whether the comparison tests the “joint update of basis and history” idea, which is the paper’s central claim. The comparison is also incomplete. The released configuration reaches only about 1.30× compression, so it is not matched in memory. The reduced-rank runs were stopped after 30, 28 and 60 generations, all of which looped to the length limit, and none was scored (Table 2). This could reflect a limitation of OjaKV, a poor rank configuration, or an integration problem. Without analysis, it supports at most the narrow statement that OjaKV did not work at these settings in the authors’ hands. Please introduce OjaKV in related work, explain the mechanism difference, and run it to completion (full 500 questions, or a subset with intervals) at its recommended settings and at matched memory.

W4. Performance claims are not established and the likely costs are under-measured. The only latency result is a single run per configuration at batch size 1 on an A40, with 1,024 decode steps after synthetic 8K/16K/32K prompts. iS-KV is 9–22% slower, and the authors state the numbers are “not statistically stable.” This slowing down per-se, is acceptable as a result, but it should be well established and explained. A. The motivating regime (10K–30K generated tokens) is untimed. B. Per-block update cost as well as reconstruction cost O(n) per step (with n being the sequence length), so overhead should grow with generated length. The reported overhead shrinking with longer input is not informative for long generation. C. Dense K/V are reconstructed at every decode step, so peak memory includes a temporary that the reported savings exclude. “Persistent-KV compression” is therefore not peak-memory compression, and the paper acknowledges this only in the appendix. I’d ask for peak-HBM numbers in the main text.

Back of the envelope evaluation that is missing in the main body or Appendix: Without compression, computing attention scores and the weighted value sum takes approximately Fattention=4NHdF_{\mathrm{attention}}=4NH d Reconstruct compressed cached keys and values requires: O(Nd(kK+kV))O\left(Nd(k_K+k_V)\right). The ordinary attention computation still follows. This amount of work is comparable to the whole attention computation and hence, it must be discussed -- attended to in the paper (and there is also the addition of the block of new token every b tokens)

Questions for authors

Q1. How does iS-KV behave on an MLA model (e.g. a DeepSeek-V2/V3-family model), where the cache is already a low-rank latent? What compression at what accuracy cost is achievable there? Q2. What are wall-clock latency and peak-HBM at 16K–32K generated tokens, with batch sizes above 1? At what generation length does reconstruction dominate? Q3 How does iS-KV compare with KIVI/KVQuant at matched bytes, and does it compose with them? Q4. Can you compare against a frozen-basis or periodic-refit baseline on the full MATH-500, since those are the realistic alternatives to the stale-coordinate strawman? Q5. Is the OjaKV failure (all generations looping) attributable to the method, or to rank/configuration choices? Can you evaluate it with its recommended settings across more questions? Q6. Why do the values degrade more than the keys, and would a separate (higher) rank for values change the picture at equal total memory? Q7. The writing is clear and the appendix is complete. Small improvements:

  • separate the two uses of “drift,”
  • in figure 1. "the share that later receives strong attention." What is strong attention? this is explained only in the appendix...
Reviewer_QK4JOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

iS-KV compresses historical KV states using block-incremental SVD while retaining recent and protected states at their original precision. It jointly updates the low-rank bases and historical coordinates as new tokens arrive. Evaluations on MATH-500, AIME, and 64K RULER examine accuracy and persistent-cache compression, supplemented by component ablations and limited latency measurements.

Critical strengths

On two 8B reasoning models, iS-KV achieves higher MATH-500 accuracy than token-eviction baselines at matched persistent-KV storage budgets. These results support low-rank history compression as a promising way to preserve reasoning capabilities without discarding token positions.

Critical weaknesses

The practical memory–compute trade-off is not established. Every decoding step reconstructs dense historical K/V and reapplies RoPE to historical keys, incurring substantial computation and temporary storage. However, reported compression ratios exclude reconstruction buffers and workspace, and reconstruction latency is not measured separately. Efficiency tests use single runs combining prefill with only 1,024 decoding steps, without 64K/128K efficiency measurements or peak-GPU-memory results. Consequently, the paper does not establish whether persistent-cache compression translates into useful memory savings at an acceptable decoding cost.

Questions for authors

At 64K and 128K context lengths, what are the measured peak GPU memory and per-token decoding latency, including dense K/V reconstruction, RoPE, and incremental updates? After accounting for these costs, does iS-KV offer a favorable memory–latency trade-off over full-cache attention and matched-budget baselines?

Reviewer_b6tkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The work compresses the KV cache that grows during long Cot generations. It maintains a 128 token sliding/recent window and the prompt exact and collapses everything older into rank r factors using an incremental SVD method every 64 tokens. This update rewrites the old coeffs along with the basis and they refer to this as "coordinate synchronization". At every decode step the full dense KV is reconstructed from the factors, RoPE is then applied and normal attention computed after that. Evaluation is on MATH-500 and AIME on R1 distill Llama 8B and Qwen3 8B, against RKV and SnapKV.

Negatives

Nothing gets faster The full dense KV is reconstructed at every step for every layer. The paper says itself that no throughput or peak HBM advantage is present. Generation is in fact up to 22% slower than the original model. The saving is storage only.

No quantization baseline test (honestly the most problematid of the negatives) The paper literally says "We do not evaluate additional low-bit quantization." Eg 4-bit KV (KIVI/KVQuant) arguably achieves the same ~4x, needs no SVD and has real kernels to support it (should be easy now with agents!). Beating eviction is a low bar in 2026 I am afraid.. Also missing: an offline basline which would be a direct test of whether actually the online update is even needed.

Critical strengths

Evidence that low rank beats eviction on long CoT..

Critical weaknesses
  1. No efficiency gain: the dense cache is rebuilt every step and it runs slower.
  2. The only comparison is against eviction without 4-bit KV at matched peak memory there's literally no evidence this is the right way to do things..
Questions for authors

At matched peak memory, how does your method compare with 4-bit KV (KIVI/KVQuant etc) and what is the decode latency at 16 or 32K context? If it clearly wins both, I'd reconsider.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.