- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes AttSVD, a training-free, per-prompt KV-cache compression method based on SVD. At prefill, it computes a truncated SVD of each layer/head's keys and values, using a weighted metric: The keys are weighted by the query covariance (preserving the attention logits ) while the values are weighted by the per-key attention mass (preserving the output ). Yielding the optimal rank- approximation of the quantities attention actually consumes, with no calibration corpus and no training. Two decode-time caching strategies are proposed (accumulating, for short generations; streaming, with a bounded sliding window, for long generations), plus a per-matrix dynamic rank rule based on spectral energy. Experiments on Phi-4-14B, Qwen3-8B, and Llama-3.1-8B-Instruct, across a custom agentic benchmark and the full LongBench suite, show AttSVD matching the dense cache within noise at ~50% memory reduction and outperforming eviction baselines (H2O, KeyDiff, KVzip) and other reconstruction baselines (raw SVD, KQ-SVD, OjaKV) at matched memory budgets.
Critical strengths
S1. The theoretical grounding are clean. The reduction of the weighted truncation problems in Eqs. 3–4 is elegantly presented, with accompaning proofs in Appendix B. I like the "standard SVD on weighted inputs," without not oversold as a new decomposition. S2. The paper is easy to read and provides plenty of figures to help the reader understand the results also intuitively, and not only follow the mathematical formalism. S3. Section 3.6 gives a clear interpretability of the solution. (also in Appendix D) S4. Table 5 positions AttSVD against the related-work landscape (weight-decomposition, calibration, cross-layer, and online families). This also helps the reader to sort things out. S5. Reasonably thorough empirical validation. Two complementary benchmarks (short-prompt/long-generation agentic eval and long-context LongBench), three backbones, and six baselines (3 eviction + 3 reconstruction) is a solid evaluation footprint for a training-free KV-cache paper. S6. Both Accumulating and Streaming solutions are natural strategies and easy to understand and evaluate.
Critical weaknesses
W1. The motivation is stated clearly at a high level: reduce the memory consumed by a growing KV cache without evicting tokens or substantially hurting answer quality. However, a concrete real-world deployment motivation is missing. the words latency or throughput are not even mentioned once in the paper.
W2. Using the dynamic mode (in Section 3.4) rather than a fixed rank produces a different rank for every sequence (and even every head, before aggregation). The paper never discusses how this interacts with batched attention kernels, either ranks must be padded to the batch max (eroding the claimed savings when one prompt in a batch needs high rank) or variable-shape kernels are needed (which is totally non-trivial engineering). This also makes AttSVD less practical especially when compared/compete with other, more batching friendly solutions.
W3. Althuogh ShadowKV is one of the most similar prior method (per-prompt, online, no calibration) it is excluded from the comparison Tables 2–4. ShadowKV has public code here https://github.com/ByteDance-Seed/ShadowKV.
W4. The Quality gains over SoTA eviction baselines are modest to nonexistent. The paper's own text in Section 4.1 reports the Qwen3-8B gaps between AttSVD and H2O/KVzip "fall within the reported 95% CI", that is, statistically tied. The paper is honest about this, but it means the practical case for AttSVD over simple, far cheaper eviction methods rests almost entirely on the reversibility/interpretability argument rather than a demonstrated quality win. That argument would be much stronger with a concrete experiment showing a scenario where an evicted token is needed later and eviction fails while AttSVD succeeds (e.g., a multi-turn or needle-in-the-haystack style task where the queried content shifts across turns). This is never constructed, despite being the paper's main motivating claim in the introduction. W5. Rank-budget coverage in the main quality tables stops at 50% cache. Table 1 reports memory at 75% and 87.5% reduction, but no corresponding LongBench/agentic quality numbers are given at those aggressive budgets. As a reader I want to understand where the quality breaks down. W6. What is the agentic benchmark? Is it released together with the paper? W7. Showing that quantization can be combined with AttSVD is highly beneficial to show that these method are indeed complementary (as noted very briefly in the Appendix A.)
Questions for authors
Q1. Without compression, computing attention scores and the weighted value sum takes approximately Reconstruct compressed cached keys and values: . The ordinary attention computation still follows. This amount of work is comparable to the whole attention computation. I would even claim that it is larger since . Thus it is crucial to show that the overall latency is not affected by this addition
What is the measured prefill latency overhead of AttSVD relative to dense attention and to the eviction baselines, at e.g. 32K context, across the tested models? Does the SVD cost amortize acceptably even in short-decode (accumulating) scenarios? Q2. How is computed under GQA when a key head is shared by multiple query heads? Please quantify any quality sensitivity to this choice. Q3. Can you provide a direct quantitative comparison against ShadowKV (keys-only, pre-RoPE, per-prompt) on at least one shared benchmark/model, even if values are left uncompressed for that baseline as in the original ShadowKV design? Q4. Show a scenario where irrevocable eviction demonstrably fails on a later-needed token while AttSVD's reconstruction-based approach succeeds. This would substantiate the paper's core irreversibility argument. Q5. In section 3.4, given a fixed measured cache budget, does optimizing the key/value rank split improve quality over the paper’s similar-rank setting, and what does that selection cost?
5 comments
Continuing responses: Query Shift, Aggressive Budgets, Benchmark, Quantization, GQA
W4/Q4: Irrevocable eviction. We built exactly this scenario: 64 needles in an 8K haystack, so a cache compressed for question A must also keep many other facts. Shift: the cache is compressed once with question A visible; we ask A (T1), then B on the same cache (T2). No-shift: compressed with B visible (oracle for B). Qwen3-8B, KV budget 0.5, n = 20, accuracy %:
| Method | Shift T1 (A) | Shift T2 (B) | No-shift (B) |
|---|---|---|---|
| Dense | 100 | 100 | 100 |
| AttSVD | 100 | 85 | 100 |
| H2O | 75 | 60 | 80 |
| SnapKV | 100 | 55 | 100 |
SnapKV answers A perfectly but loses B in almost half the samples, a direct instance of irrevocable eviction; AttSVD keeps a low-rank summary of every token. KVzip is query-agnostic by construction (at the cost of an extra reconstruction pass), so this test does not apply to it.
W5/Q5: Aggressive budgets. All 21 LongBench tasks, macro average; last column = tasks whose 95% CI of AttSVD − Dense is below 0:
| Model | Reduction | AttSVD − Dense | AttSVD − NaiveSVD | Below Dense |
|---|---|---|---|---|
| Qwen3-8B | 50% / 75% / 87.5% | −0.45 / −6.16 / −21.73 | +7.0 / +13.7 / +10.8 | 4 / 18 / 19 |
| Phi-4 | 50% / 75% / 87.5% | −2.32 / −9.90 / −15.15 | +2.7 / +1.1 / +2.2 | 7 / 19 / 20 |
Quality holds at 50% and breaks between 50% and 75%; 87.5% is outside the useful range of a uniform rank. It is task dependent: retrieval and few-shot classification are most sensitive (Qwen3-8B passage_retrieval_zh 98.0 -> 6.75 at 87.5%), while Qwen3-8B VCSUM and Phi-4 TriviaQA stay within ≈3 points of Dense even at 87.5%. Phi-4 passage_retrieval_en already drops at 50% (72.08 -> 37.92).
K/V split at identical cache bytes (Δ vs. symmetric; * = 95% CI excludes 0):
| Split K / V | Qwen3 GovR | Qwen3 Qasper | Qwen3 MFQA | Phi-4 GovR | Phi-4 Qasper | Phi-4 MFQA |
|---|---|---|---|---|---|---|
| 0.25 / 0.75 | −2.75* | −0.27 | −3.03* | −2.14* | −4.40* | −3.33* |
| 0.375 / 0.625 | +0.27 | +0.33 | −1.79* | +0.58* | −0.90 | +0.15 |
| 0.5 / 0.5 (symmetric) | ref. | ref. | ref. | ref. | ref. | ref. |
| 0.625 / 0.375 | −0.79* | −0.86 | −2.34* | −0.97* | −0.15 | +0.18 |
| 0.75 / 0.25 | −5.14* | −3.77* | −4.26* | −3.60* | −1.75 | −1.79* |
No split reliably beats symmetric. A fixed split adds no inference cost but needs held-out tuning; the energy rule suits accuracy-first use, fixed ranks a specific memory budget.
W6: Agentic benchmark. Private agentic tool-use (action-planning) evaluation; sandboxed harness with simulated tool outputs, identical across methods:
| Prompts | Categories | Actions | Tiers (easy/medium/hard) | Answer types | Scoring |
|---|---|---|---|---|---|
| 184 | 13 | ≈77 | 33 / 74 / 77 | 106 single action, 68 sequences of 2–5, 10 clarify | LLM judge 1–5 (grounding, reasoning, response, context); pass@4 (≥4) |
Generations are relatively long (median ≈800 tokens per sample, up to ≈10K), so the benchmark stresses the decode-time cache. With w = 64 and ℓ = 128, streaming compression triggers on 97–99% of samples. By "rarely", App. D.1 meant that streaming does little work per call, not that it seldom triggers: the median generation call is only ≈310–340 tokens, so each call compresses just a few 128-token chunks. We will clarify this in App. D.1. The benchmark is not yet released for legal reasons; all LongBench results are fully reproducible.
W7: Quantization. Phi-4*, 32K, AttSVD streaming r = 64; mean judge / pass@4:
| Cache | Size | General, non-thinking | General, thinking | Code, thinking | Code, non-thinking |
|---|---|---|---|---|---|
| Dense 16-bit | 6.25 GiB (100%) | 4.53 / 82.6% | 4.38 / 78.1% | 4.03 / 55.6% | 4.24 / 75.0% |
| Dense INT8 | 50% | 4.53 / 82.1% | 4.36 / 76.1% | 4.06 / 63.9% | 4.26 / 75.0% |
| AttSVD 16-bit | 50% | 4.51 / 81.0% | 4.38 / 76.6% | 4.18 / 69.4% | 4.10 / 66.7% |
| AttSVD + INT8 | 25% | 4.50 / 82.1% | 4.32 / 73.8% | 4.18 / 58.3% | 4.14 / 58.3% |
Effects are largely additive: 25% of Dense with mean judge within 0.06 on the 184 prompts; the cost shows in thinking-mode pass@4. The code subset has only 36 prompts, so it is indicative only.
*Public base Phi-4 checkpoint,microsoft/phi-4 (14B, Hugging Face), and a different LLM judge version(GPT5.2), so absolute scores are not comparable to the paper's; only within-table comparisons apply; the final version will use a single best checkpoint and judge throughout.
Q1: GQA. Both models use GQA, 4 query heads per KV head. Keys: C_Q = QᵀQ (post-RoPE) is pooled over a layer's query heads; one C_Q^{1/2} serves all its KV heads. Values: per-token attention mass is summed over query heads. Each KV head is factorized separately; only the metric is shared, adding no per-head state; all reported results use it. Pooling simplifies the per-group metric (it also counts other groups' queries); a per-group variant has equal storage.
Reviewer Reply
Thank you for the detailed rebuttal and additional experiments. The new results address several of my questions. I remain concerned that the paper does not yet present the central memory–runtime trade-off clearly.
W1/Q1. Runtime and deployment. All three reviewers raised latency or throughput. Your statement that “reconstruction-based KV-cache compression targets memory at preserved accuracy, not speed” is an important qualification for readers, not just reviewers. Please state it prominently in the paper and include the new runtime tables in the experimental results. On the reported H100 batch-1 test, AttSVD increases total time by approximately 23% for Qwen3-8B and 18% for Phi-4; prefill is substantially slower. A memory saving may justify that cost, but the trade-off should be explicit. The remaining deployment question is whether the saved GPU memory permits enough additional concurrent requests to improve throughput at a fixed GPU memory budget, accounting for peak memory during reconstruction.
W2. Batching with adaptive ranks. Thank you for explaining that each layer uses a shared rank, so standard dense batched matmuls apply. My concern is the behavior with multiple, heterogeneous requests: the maximum-rank rule lets one high-rank request raise the rank, storage, and reconstruction work for the entire batch. Since the new measurements use batch 1, they do not quantify this effect. Please report memory savings and latency or throughput for representative mixed-prompt batches, or clearly identify this as an unevaluated limitation.
W3/Q3. ShadowKV. Please include the direct comparison tables in the paper. The NarrativeQA F1 scores of AttSVD and ShadowKV GPU are nearly identical (14.93 versus 14.89), without uncertainty estimates, while their memory and runtime trade-offs differ. On GovReport, CPU-offloaded ShadowKV decodes at 39.3 tokens/s versus 30.9 for AttSVD and uses slightly less GPU cache memory, although its prefill is slower and it also uses CPU memory. That table omits quality scores, making the trade-off difficult to assess. Please report GovReport quality for the methods in that table. The LongBench comparison with NaiveSVD is useful evidence for attention weighting, but it is not a direct comparison with the full ShadowKV method.
W4/Q4. Eviction. The shifted-needle experiment addresses the demonstration I requested. Please include it in the paper and note its small sample size.
W7. Quantization. The combined experiment supports compatibility, but the near-constant mean judge score does not capture all quality changes. Relative to unquantized AttSVD, code pass@4 falls from 69.4% to 58.3% in thinking mode and from 66.7% to 58.3% in non-thinking mode. Given the 36-prompt code subset, please report uncertainty and describe the combination as promising rather than broadly lossless.
Rsponse to the Third Reviewer: Cost, Adaptive Ranks, ShadowKV, Query Shift, Aggressive Budgets, Benchmark, Quantization, GQA
We thank the reviewer for the constructive comments and address each point below.
W1/Q1: Deployment motivation and cost. Our motivation is memory in long-context serving: the KV cache grows with context and batch (≈4.5 GiB per 32K request, Qwen3-8B). Unlike methods that need calibration data or treat K and V alike, AttSVD drops the feature directions that matter least for attention, per prompt, with schedules for thinking/non-thinking modes and a fixed-rank or energy-rule budget. Reconstruction-based KV-cache compression targets memory at preserved accuracy, not speed; Almost all reconstruction-based methods introduce extra FLOPs; this is the computational cost paid for maintaining accuracy by retaining information in compressed form rather than discarding it. We report runtimes for transparency and do not claim a speedup; runtime depends on hardware and kernels, and speed techniques (speculative decoding, optimized kernels, kernel fusion, continuous batching) are usually combined with compression. Baselines: NaiveSVD (SVD on raw K, V; same storage and reconstruction as AttSVD) and ShadowKV.
(a) Quality and memory. Llama-3-8B-Instruct-Gradient-1048k, 200 NarrativeQA prompts (8K), rank 64, A100.
(b) Latency (s). H100, 4,096 in / 2,048 out, batch 1, rank 64, 3 repetitions. Prefill includes capture and SVD; decode includes reconstruction.
W2: Adaptive ranks and batching. Ranks adapt across layers and between K and V, not across heads: per layer, the energy rule takes the smallest rank keeping τ of the singular-value energy per (sequence, head), then the max, so all heads and sequences share one r_K, r_V. Factors are dense (b,h,s,r), (b,h,r,d): no padding or ragged kernels, standard batched matmuls. The max guarantees τ per head at some storage cost (mean/percentile also supported). One high-rank request raises the shared rank; we use batch 1 and do not quantify batching. Ranks on GovReport/Qasper/MultiFieldQA-en (head dim 128; ranges over examples/layers; Δ vs. r_K = r_V = 64, 3-task mean):
Weighted keys need far fewer components than weighted values; ranks vary widely across layers and inputs. The energy rule suits accuracy-first use; fixed ranks suit a specific memory budget.
W3/Q3: ShadowKV. AttSVD differs in three ways: post-RoPE keys (no re-rotation), attention-aware weighting (C_Q, attention mass), and compresses K and V, not K only. Accuracy vs. official ShadowKV code (commit e51904c; same model, prompts) is in W1 table (a): AttSVD at 0.56 GB beats ShadowKV at 0.73 GB (14.93 vs. 14.89 F1; 15.39 with window 64). ShadowKV's code supports only Llama-3/3.1-8B, GLM-4-9B, Yi-9B (Transformers 4.43, pre-Qwen3/Phi-4), so we used its primary model; all 200 prompts fill the 8K context.
Why NaiveSVD. ShadowKV's low-rank part is a per-prompt SVD of raw keys, i.e. our NaiveSVD (its chunk selection and value offloading are orthogonal), so NaiveSVD isolates attention-aware weighting at equal rank/storage, on models ShadowKV can't run. LongBench, 21 tasks (tasks with 95% CI of AttSVD − NaiveSVD > / < 0):
GovReport, 180 examples, ≤512 generated tokens, window 64: