acceptodds
Under review as a conference paper at ICLR 2027

ATTSVD: Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD

Abstract

The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free remedies evict low-importance tokens, an irreversible choice along the sequence axis. We instead keep every token and store it more cheaply along the "feature” axis. We therefore propose AttSVD, a new interpretable low-rank compression whose basis is derived from each prompt's own attention geometry: an online, per-prompt truncated SVD that keeps only the directions attention actually reads, cutting persistent per-head KV memory in proportion to the retained rank. We propose two decode-time caching strategies, accumulating and streaming, for short and long generation regimes. Furthermore, we propose two refinements that make compression adaptive. A per-matrix energy rule sizes the logit space and the attention mass independently. An attention-aware basis truncates only in the spaces attention actually reads, preserving both the attention logits and the attention output. The same factors also provide free, per-head interpretability insights into the effective rank and the geometry attention consumes. Across multiple models, on both an agentic benchmark and the full LongBench suite AttSVD stays on par with the dense cache while using up to 50% of the KV-cache memory.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
key-value (KV) cache compression, autoregressive transformers, SVD Decomposition, Attention Aware

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s1vyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes AttSVD, a training-free, per-prompt KV-cache compression method based on SVD. At prefill, it computes a truncated SVD of each layer/head's keys and values, using a weighted metric: The keys are weighted by the query covariance CQ=Q⊤QC_Q = Q^\top Q (preserving the attention logits QK⊤QK^\top) while the values are weighted by the per-key attention mass (preserving the output PVPV). Yielding the optimal rank-rr approximation of the quantities attention actually consumes, with no calibration corpus and no training. Two decode-time caching strategies are proposed (accumulating, for short generations; streaming, with a bounded sliding window, for long generations), plus a per-matrix dynamic rank rule based on spectral energy. Experiments on Phi-4-14B, Qwen3-8B, and Llama-3.1-8B-Instruct, across a custom agentic benchmark and the full LongBench suite, show AttSVD matching the dense cache within noise at ~50% memory reduction and outperforming eviction baselines (H2O, KeyDiff, KVzip) and other reconstruction baselines (raw SVD, KQ-SVD, OjaKV) at matched memory budgets.

Critical strengths

S1. The theoretical grounding are clean. The reduction of the weighted truncation problems in Eqs. 3–4 is elegantly presented, with accompaning proofs in Appendix B. I like the "standard SVD on weighted inputs," without not oversold as a new decomposition. S2. The paper is easy to read and provides plenty of figures to help the reader understand the results also intuitively, and not only follow the mathematical formalism. S3. Section 3.6 gives a clear interpretability of the solution. (also in Appendix D) S4. Table 5 positions AttSVD against the related-work landscape (weight-decomposition, calibration, cross-layer, and online families). This also helps the reader to sort things out. S5. Reasonably thorough empirical validation. Two complementary benchmarks (short-prompt/long-generation agentic eval and long-context LongBench), three backbones, and six baselines (3 eviction + 3 reconstruction) is a solid evaluation footprint for a training-free KV-cache paper. S6. Both Accumulating and Streaming solutions are natural strategies and easy to understand and evaluate.

Critical weaknesses

W1. The motivation is stated clearly at a high level: reduce the memory consumed by a growing KV cache without evicting tokens or substantially hurting answer quality. However, a concrete real-world deployment motivation is missing. the words latency or throughput are not even mentioned once in the paper.

W2. Using the dynamic mode (in Section 3.4) rather than a fixed rank produces a different rank for every sequence (and even every head, before aggregation). The paper never discusses how this interacts with batched attention kernels, either ranks must be padded to the batch max (eroding the claimed savings when one prompt in a batch needs high rank) or variable-shape kernels are needed (which is totally non-trivial engineering). This also makes AttSVD less practical especially when compared/compete with other, more batching friendly solutions.

W3. Althuogh ShadowKV is one of the most similar prior method (per-prompt, online, no calibration) it is excluded from the comparison Tables 2–4. ShadowKV has public code here https://github.com/ByteDance-Seed/ShadowKV.

W4. The Quality gains over SoTA eviction baselines are modest to nonexistent. The paper's own text in Section 4.1 reports the Qwen3-8B gaps between AttSVD and H2O/KVzip "fall within the reported 95% CI", that is, statistically tied. The paper is honest about this, but it means the practical case for AttSVD over simple, far cheaper eviction methods rests almost entirely on the reversibility/interpretability argument rather than a demonstrated quality win. That argument would be much stronger with a concrete experiment showing a scenario where an evicted token is needed later and eviction fails while AttSVD succeeds (e.g., a multi-turn or needle-in-the-haystack style task where the queried content shifts across turns). This is never constructed, despite being the paper's main motivating claim in the introduction. W5. Rank-budget coverage in the main quality tables stops at 50% cache. Table 1 reports memory at 75% and 87.5% reduction, but no corresponding LongBench/agentic quality numbers are given at those aggressive budgets. As a reader I want to understand where the quality breaks down. W6. What is the agentic benchmark? Is it released together with the paper? W7. Showing that quantization can be combined with AttSVD is highly beneficial to show that these method are indeed complementary (as noted very briefly in the Appendix A.)

Questions for authors

Q1. Without compression, computing attention scores and the weighted value sum takes approximately Fattention=4NHdF_{\mathrm{attention}}=4NH d Reconstruct compressed cached keys and values: O(Nd(rK+rV))O\left(Nd(r_K+r_V)\right). The ordinary attention computation still follows. This amount of work is comparable to the whole attention computation. I would even claim that it is larger since rK+rV=128≫4∗4r_K + r_V=128 \gg 4*4. Thus it is crucial to show that the overall latency is not affected by this addition

What is the measured prefill latency overhead of AttSVD relative to dense attention and to the eviction baselines, at e.g. 32K context, across the tested models? Does the SVD cost amortize acceptably even in short-decode (accumulating) scenarios? Q2. How is CQ=Q⊤QC_Q = Q^\top Q computed under GQA when a key head is shared by multiple query heads? Please quantify any quality sensitivity to this choice. Q3. Can you provide a direct quantitative comparison against ShadowKV (keys-only, pre-RoPE, per-prompt) on at least one shared benchmark/model, even if values are left uncompressed for that baseline as in the original ShadowKV design? Q4. Show a scenario where irrevocable eviction demonstrably fails on a later-needed token while AttSVD's reconstruction-based approach succeeds. This would substantiate the paper's core irreversibility argument. Q5. In section 3.4, given a fixed measured cache budget, does optimizing the key/value rank split improve quality over the paper’s similar-rank setting, and what does that selection cost?

5 comments
AuthorsOpenReview ↗

Rsponse to the Third Reviewer: Cost, Adaptive Ranks, ShadowKV, Query Shift, Aggressive Budgets, Benchmark, Quantization, GQA

We thank the reviewer for the constructive comments and address each point below.

W1/Q1: Deployment motivation and cost. Our motivation is memory in long-context serving: the KV cache grows with context and batch (≈4.5 GiB per 32K request, Qwen3-8B). Unlike methods that need calibration data or treat K and V alike, AttSVD drops the feature directions that matter least for attention, per prompt, with schedules for thinking/non-thinking modes and a fixed-rank or energy-rule budget. Reconstruction-based KV-cache compression targets memory at preserved accuracy, not speed; Almost all reconstruction-based methods introduce extra FLOPs; this is the computational cost paid for maintaining accuracy by retaining information in compressed form rather than discarding it. We report runtimes for transparency and do not claim a speedup; runtime depends on hardware and kernels, and speed techniques (speculative decoding, optimized kernels, kernel fusion, continuous batching) are usually combined with compression. Baselines: NaiveSVD (SVD on raw K, V; same storage and reconstruction as AttSVD) and ShadowKV.

(a) Quality and memory. Llama-3-8B-Instruct-Gradient-1048k, 200 NarrativeQA prompts (8K), rank 64, A100.

MethodF1 (×100)KV cache (GB)Prefill (s)Decode (tok/s)
Dense14.691.080.7739.1
NaiveSVD (window 64)12.160.56––
AttSVD (window 16)14.930.564.0131.2
AttSVD (window 512)14.710.603.9531.2
AttSVD (window 1,024)14.420.633.8831.5
ShadowKV (GPU)14.890.73 stored (1.58 on GPU)3.9029.8
ShadowKV (CPU offload)14.340.73 stored (0.19 GPU + 0.54 CPU)7.1038.7

(b) Latency (s). H100, 4,096 in / 2,048 out, batch 1, rank 64, 3 repetitions. Prefill includes capture and SVD; decode includes reconstruction.

ModelMethodTotalPrefillDecode
Qwen3-8BDense38.180.1538.02
Qwen3-8BNaiveSVD46.422.2344.19
Qwen3-8BAttSVD46.762.4044.35
Phi-4Dense37.840.2737.57
Phi-4NaiveSVD43.533.1340.40
Phi-4AttSVD44.783.3341.45

W2: Adaptive ranks and batching. Ranks adapt across layers and between K and V, not across heads: per layer, the energy rule takes the smallest rank keeping τ of the singular-value energy per (sequence, head), then the max, so all heads and sequences share one r_K, r_V. Factors are dense (b,h,s,r), (b,h,r,d): no padding or ragged kernels, standard batched matmuls. The max guarantees τ per head at some storage cost (mean/percentile also supported). One high-rank request raises the shared rank; we use batch 1 and do not quantify batching. Ranks on GovReport/Qasper/MultiFieldQA-en (head dim 128; ranges over examples/layers; Δ vs. r_K = r_V = 64, 3-task mean):

Modelτr_K mean [range]r_V mean [range]Δ
Qwen3-8B0.9540.8 [5, 73]85.8 [48, 106]−1.99
Qwen3-8B0.9986.7 [27, 109]116.0 [87, 123]+0.95
Qwen3-8B0.999118.5 [87, 124]126.9 [115, 128]+1.11
Phi-40.9518.7 [8, 37]91.7 [42, 107]−14.18
Phi-40.9963.7 [35, 91]118.4 [86, 123]+1.06
Phi-40.999112.2 [67, 122]127.3 [118, 128]+1.66

Weighted keys need far fewer components than weighted values; ranks vary widely across layers and inputs. The energy rule suits accuracy-first use; fixed ranks suit a specific memory budget.

W3/Q3: ShadowKV. AttSVD differs in three ways: post-RoPE keys (no re-rotation), attention-aware weighting (C_Q, attention mass), and compresses K and V, not K only. Accuracy vs. official ShadowKV code (commit e51904c; same model, prompts) is in W1 table (a): AttSVD at 0.56 GB beats ShadowKV at 0.73 GB (14.93 vs. 14.89 F1; 15.39 with window 64). ShadowKV's code supports only Llama-3/3.1-8B, GLM-4-9B, Yi-9B (Transformers 4.43, pre-Qwen3/Phi-4), so we used its primary model; all 200 prompts fill the 8K context.

Why NaiveSVD. ShadowKV's low-rank part is a per-prompt SVD of raw keys, i.e. our NaiveSVD (its chunk selection and value offloading are orthogonal), so NaiveSVD isolates attention-aware weighting at equal rank/storage, on models ShadowKV can't run. LongBench, 21 tasks (tasks with 95% CI of AttSVD − NaiveSVD > / < 0):

ModelDenseNaiveSVD (0.5 / 0.25 / 0.125)AttSVD (0.5 / 0.25 / 0.125)Tasks > / <
Qwen3-8B49.6442.18 / 29.75 / 17.1349.19 / 43.48 / 27.9115/1, 15/0, 18/0
Phi-444.4839.44 / 33.50 / 27.1042.16 / 34.58 / 29.338/1, 10/4, 9/4

GovReport, 180 examples, ≤512 generated tokens, window 64:

MethodPrefill (s)Decode (tok/s)KV cache (GB)
Dense0.6935.51.14
NaiveSVD3.6131.20.62
AttSVD3.8330.90.62
ShadowKV (GPU)3.8128.70.79 (1.60 on GPU)
ShadowKV (CPU offload)6.9639.30.58 GPU + 0.64 CPU
AuthorsOpenReview ↗

Continuing responses: Query Shift, Aggressive Budgets, Benchmark, Quantization, GQA

W4/Q4: Irrevocable eviction. We built exactly this scenario: 64 needles in an 8K haystack, so a cache compressed for question A must also keep many other facts. Shift: the cache is compressed once with question A visible; we ask A (T1), then B on the same cache (T2). No-shift: compressed with B visible (oracle for B). Qwen3-8B, KV budget 0.5, n = 20, accuracy %:

MethodShift T1 (A)Shift T2 (B)No-shift (B)
Dense100100100
AttSVD10085100
H2O756080
SnapKV10055100

SnapKV answers A perfectly but loses B in almost half the samples, a direct instance of irrevocable eviction; AttSVD keeps a low-rank summary of every token. KVzip is query-agnostic by construction (at the cost of an extra reconstruction pass), so this test does not apply to it.

W5/Q5: Aggressive budgets. All 21 LongBench tasks, macro average; last column = tasks whose 95% CI of AttSVD − Dense is below 0:

ModelReductionAttSVD − DenseAttSVD − NaiveSVDBelow Dense
Qwen3-8B50% / 75% / 87.5%−0.45 / −6.16 / −21.73+7.0 / +13.7 / +10.84 / 18 / 19
Phi-450% / 75% / 87.5%−2.32 / −9.90 / −15.15+2.7 / +1.1 / +2.27 / 19 / 20

Quality holds at 50% and breaks between 50% and 75%; 87.5% is outside the useful range of a uniform rank. It is task dependent: retrieval and few-shot classification are most sensitive (Qwen3-8B passage_retrieval_zh 98.0 -> 6.75 at 87.5%), while Qwen3-8B VCSUM and Phi-4 TriviaQA stay within ≈3 points of Dense even at 87.5%. Phi-4 passage_retrieval_en already drops at 50% (72.08 -> 37.92).

K/V split at identical cache bytes (Δ vs. symmetric; * = 95% CI excludes 0):

Split K / VQwen3 GovRQwen3 QasperQwen3 MFQAPhi-4 GovRPhi-4 QasperPhi-4 MFQA
0.25 / 0.75−2.75*−0.27−3.03*−2.14*−4.40*−3.33*
0.375 / 0.625+0.27+0.33−1.79*+0.58*−0.90+0.15
0.5 / 0.5 (symmetric)ref.ref.ref.ref.ref.ref.
0.625 / 0.375−0.79*−0.86−2.34*−0.97*−0.15+0.18
0.75 / 0.25−5.14*−3.77*−4.26*−3.60*−1.75−1.79*

No split reliably beats symmetric. A fixed split adds no inference cost but needs held-out tuning; the energy rule suits accuracy-first use, fixed ranks a specific memory budget.

W6: Agentic benchmark. Private agentic tool-use (action-planning) evaluation; sandboxed harness with simulated tool outputs, identical across methods:

PromptsCategoriesActionsTiers (easy/medium/hard)Answer typesScoring
18413≈7733 / 74 / 77106 single action, 68 sequences of 2–5, 10 clarifyLLM judge 1–5 (grounding, reasoning, response, context); pass@4 (≥4)

Generations are relatively long (median ≈800 tokens per sample, up to ≈10K), so the benchmark stresses the decode-time cache. With w = 64 and ℓ = 128, streaming compression triggers on 97–99% of samples. By "rarely", App. D.1 meant that streaming does little work per call, not that it seldom triggers: the median generation call is only ≈310–340 tokens, so each call compresses just a few 128-token chunks. We will clarify this in App. D.1. The benchmark is not yet released for legal reasons; all LongBench results are fully reproducible.

W7: Quantization. Phi-4*, 32K, AttSVD streaming r = 64; mean judge / pass@4:

CacheSizeGeneral, non-thinkingGeneral, thinkingCode, thinkingCode, non-thinking
Dense 16-bit6.25 GiB (100%)4.53 / 82.6%4.38 / 78.1%4.03 / 55.6%4.24 / 75.0%
Dense INT850%4.53 / 82.1%4.36 / 76.1%4.06 / 63.9%4.26 / 75.0%
AttSVD 16-bit50%4.51 / 81.0%4.38 / 76.6%4.18 / 69.4%4.10 / 66.7%
AttSVD + INT825%4.50 / 82.1%4.32 / 73.8%4.18 / 58.3%4.14 / 58.3%

Effects are largely additive: 25% of Dense with mean judge within 0.06 on the 184 prompts; the cost shows in thinking-mode pass@4. The code subset has only 36 prompts, so it is indicative only.

*Public base Phi-4 checkpoint,microsoft/phi-4 (14B, Hugging Face), and a different LLM judge version(GPT5.2), so absolute scores are not comparable to the paper's; only within-table comparisons apply; the final version will use a single best checkpoint and judge throughout.

Q1: GQA. Both models use GQA, 4 query heads per KV head. Keys: C_Q = QᵀQ (post-RoPE) is pooled over a layer's query heads; one C_Q^{1/2} serves all its KV heads. Values: per-token attention mass is summed over query heads. Each KV head is factorized separately; only the metric is shared, adding no per-head state; all reported results use it. Pooling simplifies the per-group metric (it also counts other groups' queries); a per-group variant has equal storage.

Reviewer_s1vyOpenReview ↗

Reviewer Reply

Thank you for the detailed rebuttal and additional experiments. The new results address several of my questions. I remain concerned that the paper does not yet present the central memory–runtime trade-off clearly.

W1/Q1. Runtime and deployment. All three reviewers raised latency or throughput. Your statement that “reconstruction-based KV-cache compression targets memory at preserved accuracy, not speed” is an important qualification for readers, not just reviewers. Please state it prominently in the paper and include the new runtime tables in the experimental results. On the reported H100 batch-1 test, AttSVD increases total time by approximately 23% for Qwen3-8B and 18% for Phi-4; prefill is substantially slower. A memory saving may justify that cost, but the trade-off should be explicit. The remaining deployment question is whether the saved GPU memory permits enough additional concurrent requests to improve throughput at a fixed GPU memory budget, accounting for peak memory during reconstruction.

W2. Batching with adaptive ranks. Thank you for explaining that each layer uses a shared rank, so standard dense batched matmuls apply. My concern is the behavior with multiple, heterogeneous requests: the maximum-rank rule lets one high-rank request raise the rank, storage, and reconstruction work for the entire batch. Since the new measurements use batch 1, they do not quantify this effect. Please report memory savings and latency or throughput for representative mixed-prompt batches, or clearly identify this as an unevaluated limitation.

W3/Q3. ShadowKV. Please include the direct comparison tables in the paper. The NarrativeQA F1 scores of AttSVD and ShadowKV GPU are nearly identical (14.93 versus 14.89), without uncertainty estimates, while their memory and runtime trade-offs differ. On GovReport, CPU-offloaded ShadowKV decodes at 39.3 tokens/s versus 30.9 for AttSVD and uses slightly less GPU cache memory, although its prefill is slower and it also uses CPU memory. That table omits quality scores, making the trade-off difficult to assess. Please report GovReport quality for the methods in that table. The LongBench comparison with NaiveSVD is useful evidence for attention weighting, but it is not a direct comparison with the full ShadowKV method.

W4/Q4. Eviction. The shifted-needle experiment addresses the demonstration I requested. Please include it in the paper and note its small sample size.

W7. Quantization. The combined experiment supports compatibility, but the near-constant mean judge score does not capture all quality changes. Relative to unquantized AttSVD, code pass@4 falls from 69.4% to 58.3% in thinking mode and from 66.7% to 58.3% in non-thinking mode. Given the 36-prompt code subset, please report uncertainty and describe the combination as promising rather than broadly lossless.

Reviewer_pZcXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AttSVD, a low-rank compression method for KV caches that requires no additional training or calibration data, aimed at reducing the memory overhead of KV caches during long-context inference. Unlike traditional methods that perform singular value decomposition directly on the Key and Value matrices, AttSVD constructs a weighting matrix using the attention information from the current prompt, thereby prioritizing the preservation of low-rank subspaces that are more critical for attention computation.

Specifically, the method weights the Key using the Query covariance matrix to minimize attention logit errors caused by compression; simultaneously, it weights the Value using the attention mass received by each token to improve the approximation quality of the attention output. Furthermore, the authors propose an adaptive rank selection mechanism based on singular value energy, allowing the Key and Value matrices to use different compression ranks, and design two caching strategies to accommodate generation tasks of varying lengths.

Experiments report that, with approximately 50% of the KV-cache storage budget, AttSVD achieves performance close to that of uncompressed models on the evaluated LongBench subtasks and agentic benchmarks, outperforming multiple token eviction and low-rank compression baselines.

Critical strengths

This paper addresses the limitation of traditional SVD, which selects retention directions based solely on the singular value energy of a matrix, by proposing a low-rank decomposition method guided by the attention information from the current prompt. By separately optimizing the query-weighted reconstruction objective for the Key and the attention-mass-weighted objective for the Value, this method can more effectively preserve information related to attention computation while maintaining a fixed compression rank.

Critical weaknesses
  1. The absence of inference efficiency measurements makes it difficult to assess the practical benefits of the proposed compression method. According to Section 3.3 and Algorithm 1, AttSVD requires reconstructing low-rank factors into full Key and Value matrices at each decoding step before performing standard attention computations. This process introduces additional matrix multiplication overhead, while the SVD decomposition and attention-aware weighting during the prefill phase also increase computational costs. However, the paper primarily reports on the storage size of the persistent KV-cache and lacks key metrics such as end-to-end inference latency, decoding throughput, and reconstruction overhead.

  2. This paper presents the streaming cache as one of its major contributions, aiming to support long sequence generation by compressing historical generated tokens in blocks. However, Appendix D.1 explicitly states that, among the 9 LongBench tasks evaluated, streaming compression was not actually triggered in eight of them due to the short length of the generated sequences; consequently, the “accumulating” and “streaming” approaches employed the same caching process for these tasks. Only GovReport triggered streaming compression, and the additional sliding-window ablation was conducted only under the 80% cache setting for that task. These experiments are insufficient to fully validate the effectiveness of the streaming strategy in long-generation scenarios.

Questions for authors
  1. Could the authors provide end-to-end inference latency, decoding throughput, and prefill overhead measurements to clarify whether the additional SVD and KV reconstruction costs offset the memory savings?
  2. Given that streaming compression is not triggered in eight of the nine LongBench tasks, could the authors evaluate longer generations (e.g., 4K–16K tokens) and investigate how the use of standard truncated SVD for historical tokens affects generation quality?
1 comment
AuthorsOpenReview ↗

Response to the Second Reviewer: Inference Efficiency (W1/Q1) and Long-Generation Streaming (W2/Q2)

We thank the reviewer for the constructive comments and address each point.

W1/Q1: Inference efficiency. Reconstruction-based KV-cache compression targets memory at preserved accuracy, not speed; it adds FLOPs, the cost of keeping information compressed instead of discarding it. We report runtimes for transparency and do not claim a speedup; runtime depends on hardware and kernels, and speed techniques (speculative decoding, optimized kernels, kernel fusion, continuous batching) are usually combined with compression.

(a) Quality and memory. Llama-3-8B-Instruct-Gradient-1048k, 200 NarrativeQA prompts (8K), rank 64, A100.

MethodF1 (×100)KV cache (GB)Prefill (s)Decode (tok/s)
Dense14.691.080.7739.1
NaiveSVD (window 64)12.160.56––
AttSVD (window 16)14.930.564.0131.2
AttSVD (window 512)14.710.603.9531.2
AttSVD (window 1,024)14.420.633.8831.5
ShadowKV (GPU)14.890.73 stored (1.58 on GPU)3.9029.8
ShadowKV (CPU offload)14.340.73 stored (0.19 GPU + 0.54 CPU)7.1038.7

ShadowKV: official code, defaults (sparse budget 2,048, key rank 160). CPU offload relocates values to host memory; AttSVD compresses keys and values on the GPU.

(b) Latency (s). H100, 4,096 in / 2,048 out, batch 1, rank 64, 3 repetitions. Prefill includes capture and SVD; decode includes reconstruction.

ModelMethodTotalPrefillDecode
Qwen3-8BDense38.180.1538.02
Qwen3-8BNaiveSVD46.422.2344.19
Qwen3-8BAttSVD46.762.4044.35
Phi-4Dense37.840.2737.57
Phi-4NaiveSVD43.533.1340.40
Phi-4AttSVD44.783.3341.45

(c) Persistent KV cache vs. peak GPU memory. We reduce the stored cache; peak also includes weights, activations and buffers. Qwen3-8B, H100, 4K prompt + 16K generated:

MethodPeak GPU (GiB)
Dense18.1
AttSVD, rank 6417.9
AttSVD, rank 1617.7

W2/Q2: Long generation and SVD of historical tokens. Streaming triggers when generated tokens overflow the exact window w; older chunks of c tokens get truncated SVD (no queries are captured for them).

(a) Paper results (App. D).

SettingAccumulatingStreamingDense
LongBench, n = 1,750, Phi-4 / Qwen3-8B, w = 64 (Table 6)ref.Δ ∈ [−0.29, +0.24] at every budget–
GovReport ROUGE-L, Llama-3.1-8B, c = 0.2, w = 256 to 16 (Table 7)33.7933.74–33.8834.66

(b) New: 12K–16K generated tokens. Teacher-forced PG-19 perplexity (lower is better; Δ vs. Dense), up to 20 books, 4,096-token prompt, rank 64; streaming triggers in every run (up to 127 compressed chunks per layer).

Methodw / cPhi-4 (12K)Qwen3-8B (16K)
Dense–8.6113.49
Accumulating–8.77 (+0.16)14.27 (+0.78)
Streaming512 / 2568.83 (+0.22)15.05 (+1.56)
Streaming64 / 1288.80 (+0.19)14.93 (+1.44)
Streaming32 / 1288.80 (+0.19)14.76 (+1.27)
Streaming16 / 1288.80 (+0.19)14.29 (+0.80)

Streaming adds at most 0.06 (Phi-4) and 0.78 (Qwen3-8B) perplexity over accumulating, and only 0.02 with a 16-token window: little degradation even over 12K–16K tokens. It is worth mentioning that by "rarely", App. D.1 meant that streaming does little work, not that it never triggers: generations are short (median ≈310 tokens per call), so although it fires on 97–99% of samples, it compresses only a few 128-token chunks per call. To prevent confusion, we will clarify this in App. D.1. The benchmark is not yet released for legal reasons; all LongBench results are fully reproducible.

Reviewer_UkjrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AttSVD, a training-free KV-cache compression method based on prompt-adaptive, attention-weighted SVD. The method uses query covariance to weight keys and attention mass to weight values, aiming to preserve attention-relevant information under low-rank approximation. It also introduces accumulating and streaming strategies for managing generated tokens, along with an energy-based rule for selecting compression ranks. Experiments on three language models using LongBench and an agentic benchmark show that AttSVD generally maintains performance close to the dense KV cache at approximately 50% memory usage.

Critical strengths

The attention-aware formulation is mathematically well motivated and relatively straightforward to implement. By incorporating query covariance and attention mass into the SVD, the method achieves better preservation of attention-relevant information than standard low-rank approximation. The improvements over unweighted SVD are consistent across the evaluated models and benchmarks.

The paper also evaluates several existing KV-cache compression methods, including eviction-based and low-rank approaches. The reported results suggest that AttSVD can maintain competitive generation quality at reduced persistent KV-cache memory usage without requiring additional training or calibration data.

Critical weaknesses
  1. Limited methodological novelty. The method primarily combines standard truncated SVD with attention-based weighting and adaptive rank selection. Prior methods such as ShadowKV and KQ-SVD have already explored prompt-specific low-rank compression and attention-aware approximations. While the proposed weighting formulation is reasonable, the overall contribution appears incremental.

  2. Missing practical efficiency evaluation. This is my main concern. The paper reports reductions in persistent KV-cache storage but provides no measurements of decoding latency, throughput, or peak GPU memory. Reconstructing K and V at every decoding step introduces additional computation and temporary memory overhead, while prefill requires SVD and attention-statistics computation. Without end-to-end measurements, it is unclear whether the proposed method offers meaningful practical benefits.

  3. Insufficient experimental validation. The streaming strategy is not adequately evaluated, as it behaves identically to accumulating on 8 of 9 LongBench tasks. Improvements over competitive baselines are also relatively small, with limited statistical evidence. Furthermore, the paper lacks component-wise ablations of key/value weighting and matched-memory comparisons for adaptive rank selection. These limitations make it difficult to assess the contributions of the individual components.

Questions for authors
  1. Can the authors report end-to-end decoding latency, throughput, and peak GPU memory, including the overhead of prefill SVD and KV reconstruction?

  2. Can the authors provide experiments with substantially longer generations to validate the streaming strategy, and evaluate more aggressive compression ratios against competitive low-rank baselines such as ShadowKV and Palu?

  3. Can the authors provide component-wise ablations of key/value weighting and adaptive rank selection, along with statistical evidence supporting improvements over competitive baselines?

2 comments
AuthorsOpenReview ↗

Response to the First Reviewer (1/2): Inference Efficiency and Streaming

We appreciate the reviewer's thoughtful comments and address each point below.

W2/Q1: Inference efficiency. Reconstruction-based KV-cache compression targets memory at preserved accuracy, not speed; almost all reconstruction methods add FLOPs to rebuild the cache, the computational cost paid for maintaining accuracy by retaining information in compressed form rather than discarding it. Latency optimization is therefore outside the scope of this work. Our implementation adds latency over dense caching; we report prefill, decode and total request time separately to make this trade-off transparent and do not claim an inference speedup. Runtime efficiency depends on hardware, kernels and workload, so timings do not generalize across processors. Inference-efficiency techniques (speculative decoding, sliding-window attention, optimized kernels, kernel fusion, continuous batching) are a distinct line of work, usually combined with KV-cache compression to compensate for the introduced operations, improving execution and serving efficiency alongside the memory savings.

Baselines: NaiveSVD (SVD on raw K, V), the simplest low-rank method (no calibration data or RoPE re-application), with the same factor storage and decode reconstruction as AttSVD at matched ranks; and ShadowKV, as requested.

(a) Quality and memory. Llama-3-8B-Instruct-Gradient-1048k, 200 NarrativeQA prompts (8K), rank 64 for K and V, A100.

MethodF1 (×100)KV cache (GB)Prefill (s)Decode (tok/s)
Dense14.691.080.7739.1
NaiveSVD (window 64)12.160.56––
AttSVD (window 16)14.930.564.0131.2
AttSVD (window 512)14.710.603.9531.2
AttSVD (window 1,024)14.420.633.8831.5
ShadowKV (GPU)14.890.73 stored (1.58 on GPU)3.9029.8
ShadowKV (CPU offload)14.340.73 stored (0.19 GPU + 0.54 CPU)7.1038.7

ShadowKV: official code, defaults (sparse budget 2,048, key rank 160, chunk size 8); its larger stored state favors it in quality. Both runtimes store the same state: the GPU one keeps it all on the GPU, where gather buffers push it above Dense; CPU offload moves values to host memory, cutting GPU memory and speeding decode but slowing prefill, occupying a CPU core and lowering F1. Offloading relocates values; AttSVD compresses keys and values on the GPU.

(b) Latency (s). H100, 4,096 in / 2,048 out, batch 1, rank 64, 16-token window, accumulating decode, BF16, SDPA, 3 repetitions. Prefill includes query capture (prefill only), statistics and SVD; decode includes reconstruction; totals exclude loading and serving.

ModelMethodTotalPrefillDecode
Qwen3-8BDense38.180.1538.02
Qwen3-8BNaiveSVD46.422.2344.19
Qwen3-8BAttSVD46.762.4044.35
Phi-4Dense37.840.2737.57
Phi-4NaiveSVD43.533.1340.40
Phi-4AttSVD44.783.3341.45

(c) Persistent KV cache vs. peak GPU memory. We reduce the stored cache (keys and values kept for every token, growing with context and batch); peak also includes weights, activations and temporary buffers, and our statistics/factorization buffers raise the prefill peak. Lower ranks shrink stored factors, which matters most in long generation. Qwen3-8B, H100, 4K prompt + 16K generated:

MethodPeak GPU (GiB)
Dense18.1
AttSVD, rank 6417.9
AttSVD, rank 1617.7

W3/Q2–Q3: Streaming, rank selection, and ablations.

Streaming vs. accumulating (App. D.1, Table 6). On the paper's 9-task LongBench set (n = 1,750; Phi-4-14B, Qwen3-8B; w = 64), streaming stays within 0.3 points of decode-exact accumulating at every budget (Δ ∈ [−0.29, +0.24]). App. D.1's "rarely" (agentic benchmark) meant that streaming does little work, not that it never triggers: per-call generations are short (median ≈310 tokens), so although it fires on 97–99% of samples, it compresses only a few 128-token chunks per call. We will clarify this in App. D.1. The benchmark is not yet released for legal reasons; all LongBench results are fully reproducible.

Exact decode-window ablation (App. D.2, Table 7). On GovReport (Llama-3.1-8B-Instruct, c = 0.2), ROUGE-L stays flat as the exact window shrinks: streaming scores 33.74-33.88 for windows from 256 down to 16 tokens, versus 33.79 for accumulating and 34.66 for Dense.

Streaming stress tests. Forced 4K–16K-token generations (4,096-token prompt; window 512, chunk 256) trigger hundreds to thousands of compressed decode chunks, confirming that streaming fires at runtime (they do not measure quality).

AuthorsOpenReview ↗

Response to the First Reviewer (2/2): Long Generations, Rank Selection, Ablations

W3/Q2–Q3 (continued): Streaming, rank selection, and ablations.

Long generations (12K–16K tokens). To measure quality, we compute perplexity on the true PG-19 continuation (up to 20 books; 4,096-token prompt; 12,288 continuation tokens for Phi-4, 16,384 for Qwen3-8B), AttSVD rank 64 for K and V (head dim 128). Accumulating keeps every generated token exact; streaming keeps only the last w tokens exact and compresses older ones in chunks of c, so it triggers in every streaming run (up to 127 compressed chunks per layer).

Methodw / cPhi-4 (12K)Qwen3-8B (16K)
Dense–8.6113.49
Accumulating–8.77 (+0.16)14.27 (+0.78)
Streaming512 / 2568.83 (+0.22)15.05 (+1.56)
Streaming64 / 1288.80 (+0.19)14.93 (+1.44)
Streaming32 / 1288.80 (+0.19)14.76 (+1.27)
Streaming16 / 1288.80 (+0.19)14.29 (+0.80)

Perplexity over the full continuation: streaming adds at most 0.06 perplexity over accumulating on Phi-4 and at most 0.78 on Qwen3-8B, and is within 0.02 of accumulating on Qwen3-8B with a 16-token window. Compressing generated tokens therefore adds little degradation, even over 12K–16K tokens.

Energy-based vs. fixed rank (Sec. 4.3, Table 4). The per-matrix energy rule reaches near-dense LongBench averages: 36.5 and 36.4 at τ = 0.997 and 0.995 versus 36.7 for Dense on Phi-4-14B, and 52.0 and 52.2 versus 51.8 on Qwen3-8B.

Fixed symmetric rank (r_K = r_V = 64) and four K/V splits of the same total rank (identical bytes) vs. energy thresholds 0.95, 0.99, 0.999 on GovReport, Qasper, MultiFieldQA-en (MFQA), both models. The last column gives storage vs. the fixed symmetric setting. At comparable memory (τ = 0.95), the energy rule does not outperform the fixed ranks and drops sharply on Phi-4; its gains appear at τ = 0.99 and 0.999, which store 1.4-1.9× more. At τ = 0.999, all six scores are within 0.53 points of dense. **The energy rule therefore trades memory for quality rather than outperforming fixed ranks at equal memory. In practice, the energy rule suits accuracy-first use, and fixed ranks suit a fixed memory budget.

Setting* (r_K / r_V)Qwen3 GovRQwen3 QasperQwen3 MFQAPhi-4 GovRPhi-4 QasperPhi-4 MFQABytes vs. symmetric
Dense33.6648.3353.6733.1340.7347.75–
Fixed, symmetric (64 / 64)31.1746.3954.8030.3339.6146.581.00×
Fixed (32 / 96)28.4246.1251.7628.1935.2043.261.00×
Fixed (48 / 80)31.4446.7253.0130.9138.7146.731.00×
Fixed (80 / 48)30.3845.5452.4629.3739.4546.761.00×
Fixed (96 / 32)26.0342.6250.5426.7437.8644.791.00×
Energy τ = 0.9528.4545.4052.5621.9521.7030.330.85–1.01×
Energy τ = 0.9933.9447.4353.8532.3440.0647.311.41–1.60×
Energy τ = 0.99933.5448.1154.0332.9540.2648.281.86–1.92×

*All rebuttal LongBench results use a separate evaluation setup from the paper's Table 2. Because of time and privacy constraints, Phi-4 results use the public base Phi-4 checkpoint, microsoft/phi-4 (14B, Hugging Face), a different model from the Phi-4 in the paper. The setup also differs in prompt caps (32,768 tokens for Qwen3-8B; within Phi-4's 16,384-token limit), tokenization and truncation, and official LongBench answer postprocessing and scoring, and it covers every test example. Absolute scores, including Dense, therefore differ from Tables 2 and 4 and are not directly comparable to them. Comparisons are valid within each table, where all methods share the same checkpoint and setup. In the final version, we will rerun all results with the newest and best checkpoints under a single setup, so Tables 2, 4 and 6 and the rebuttal results are directly comparable.

Attention-weighting ablation. On all 21 LongBench tasks at rank fractions 0.5, 0.25 and 0.125, we compare plain, key-weighted, value-weighted and both-weighted SVD; all compress K and V and differ only in which factorization is attention-weighted. Weighting both is best or tied-best in all six settings, with gains over plain SVD of +7.0 to +13.7 points on Qwen3-8B and +1.1 to +2.7 on Phi-4.

ModelRank frac.DensePlain SVDKey-w.Value-w.Both (AttSVD)
Qwen3-8B0.549.6442.1847.7343.5349.19
Qwen3-8B0.2549.6429.7538.9430.5943.48
Qwen3-8B0.12549.6417.1323.5216.2527.91
Phi-40.544.4839.4440.7040.5542.16
Phi-40.2544.4833.5032.8734.5834.58
Phi-40.12544.4827.1027.1528.7529.33

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.