- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes PASVD, a training-free method for KV-Cache compression. It applies SVD on un-RoPE-ed cross-head completed key pages, it chooses a rank for each completed page, and stores the compressed form only. During decode, per-query token, only selected key pages are reconstructed for final attention. Values are not compressed using SVD, but rather using INT8 quantization. The reported quality is close to FullKV on several benchmarks, and decode-only throughput on Qwen3-8B at 32K context is 2.37× higher at batch size 1.
Critical strengths
The combination of page-specific ranks and selection before reconstruction is well motivated: reconstructing every historical key would erase much of the potential speed benefit. The paper evaluates quality across long-context tasks, MMLU, and perplexity, and reports the storage layout and implementation choices needed to make variable ranks practical.
Critical weaknesses
W1. The contribution of each compression choice to throughput is unclear. The FullKV comparison changes several components simultaneously: key SVD, INT8 storage for key bases and values, candidate selection, selective reconstruction, and fused kernels. Consequently, the reported speedup cannot be attributed specifically to the proposed key factorization. The paper also does not explain why keys receive SVD compression while values receive only quantization, despite showing spectral decay for both. (Figure 1, Left part) A useful ablation would report throughput, memory, and quality for at least: FullKV; INT8 K and V; SVD K with FP16 V; SVD K with INT8 V; and SVD K and V. The last comparison would test the proposed K/V asymmetry directly.
W2. The batch-3 speedup is difficult to interpret. For Qwen3-8B at 32K, an FP16 FullKV cache is approximately 4.5 GiB per sequence, or 13.5 GiB at batch size 3. Together with roughly 16 GB of model weights, this puts the experiment close to the RTX 5090’s 32 GB capacity before other allocations. FullKV throughput falls from 24.51 tokens/s at batch 2 to 11.68 at batch 3, while PASVD rises to 88.11 tokens/s. The resulting 7.54× ratio may substantially reflect FullKV operating under memory pressure. Fragmentation is one possible cause, but the paper does not measure it. Fitting more sequences on a constrained GPU is a genuine benefit. However, this result alone does not establish a 7.54× compute-efficiency advantage in a less constrained deployment. Please report peak allocated and reserved memory, memory-bandwidth use, and throughput over more batch sizes or on a GPU where both methods have comfortable capacity. A comparison using similarly optimized attention implementations would also help, since FullKV uses the Hugging Face SDPA dynamic cache.
Another concern regarding batch size: the throughput evaluation stops at batch size 3. At such small batches, decode performance may be dominated by memory traffic, and the GPU may not be fully utilized. To support the method’s deployment claims, the authors should evaluate larger concurrent batches which characterize real-deployments, for example, 16–64 sequences, on hardware with sufficient memory—and report throughput, per-token latency, and tail latency. PASVD’s variable page ranks also raise an implementation question. Different pages can require different amounts of work, potentially causing load imbalance between warps or thread blocks. Packed factors may further affect memory-access efficiency, although variable rank does not necessarily prevent coalesced reads within a warp. GPU profiling and scaling results are needed to determine whether these costs grow at higher concurrency. Without that evidence, the paper establishes a useful KV-cache compression result, but leaves its benefit for heavily loaded serving systems unresolved.
W3. Pre-RoPE selection may miss positionally important keys. PASVD selects candidates using scores computed before RoPE, then applies RoPE only to selected keys for final attention. These rankings can differ. The appendix diagnostic finds that the pre-RoPE top-64 captures 52.76% of selectable historical post-RoPE attention mass, compared with 60.74% for an equal-budget post-RoPE oracle. This diagnostic uses short 2,048-token segments and excludes sink, recent, and active-page tokens. It does not show how often selection errors affect answers at the evaluated 16K and 32K lengths. I would like to see attention-mass recall and task quality as a function of selection budget, context length, and query head, especially for tasks requiring information spread across the context.
Questions for authors
Q1. Can you separate the effects of key SVD, value INT8 quantization, candidate selection, and fused kernels on throughput, memory use, and quality? Q2. Why are keys compressed with SVD while values are quantized? How does this design compare with INT8 K and V, and with SVD applied to both K and V, at matched memory budgets? Q3. What causes FullKV throughput to fall sharply at batch size 3? Please report peak memory allocation, memory-bandwidth use, and GPU utilization for both methods. Q4. How does PASVD perform at larger concurrent batches, such as 16–64 sequences on hardware with sufficient memory? Do variable page ranks cause load imbalance or reduce memory-access efficiency? Please include throughput and tail latency. Q5. At 16K and 32K context lengths, how much post-RoPE attention mass does pre-RoPE selection retain, and how does this change with the selection budget?
2 comments
Q3: Why does FullKV slow down at batch=3? Were memory, bandwidth, and GPU utilization measured?
Response: We thank the reviewer for the question. We measured memory, bandwidth, and GPU utilization. The main cause of the slowdown is that memory usage approaches the device limit, triggering frequent allocation retries and synchronizations, rather than bandwidth saturation. The experimental configuration is consistent with Table 7 of the paper: Qwen3-8B FP16, 32K context, a single RTX 5090; FullKV uses the HF SDPA dynamic cache.
| Method | Batch | Throughput (tokens/s) | Peak allocated memory (GiB) | DRAM read/write bandwidth (%) | GPU utilization (%) | Allocation retries |
|---|---|---|---|---|---|---|
| FullKV | 1 | 20.96 | 20.33 | 51.4 / 28.8 | 96.9 | 0 |
| FullKV | 2 | 24.51 | 25.37 | 50.3 / 33.5 | 98.0 | 0 |
| FullKV | 3 | 11.68 | 30.41 | 22.5 / 21.3 | 58.8 | 6,897 |
| PASVD | 1 | 49.58 | 18.80 | 47.1 / 0.98 | 84.7 | 0 |
| PASVD | 2 | 73.59 | 22.17 | 35.8 / 0.95 | 77.0 | 0 |
| PASVD | 3 | 88.11 | 25.55 | 29.7 / 0.96 | 71.5 | 0 |
The FullKV dynamic cache extends K/V with torch.cat at every step and every layer, producing a large number of temporary tensors. At batch=3, its reserved memory reaches 30.75 GiB, close to the available limit of 31.36 GiB; 6,897 allocation retries occur within 256 decoding steps, GPU utilization drops from 98.0% to 58.8%, and bandwidth decreases accordingly. PASVD uses 4.86 GiB less memory and incurs no retries.
Q4: Do larger concurrency and variable page ranks affect throughput, tail latency, and load balance?
Response: We thank the reviewer for the suggestion. We conducted additional decoding experiments with 1–16 concurrent sequences on a single NVIDIA A800 80GB GPU: Qwen3-8B, WikiText, 32K context; after warm-up, each configuration is measured three times and the median is reported. Throughput is the aggregate output speed of the entire batch, excluding prefill; P95/P99 are per-step decoding latencies. Concurrency of 32 could not be measured because it exceeds the GPU memory.
| Concurrency | Throughput (tokens/s) | P95 (ms) | P99 (ms) |
|---|---|---|---|
| 1 | 39.62 | 22.36 | 137.02 |
| 2 | 54.24 | 31.53 | 213.62 |
| 3 | 61.79 | 40.86 | 282.93 |
| 8 | 73.91 | 87.24 | 796.19 |
| 16 | 81.85 | 161.62 | 1079.57 |
Throughput and tail latency. As concurrency increases from 1 to 16, throughput improves by about 2.1×, but does not scale linearly. Tail latency increases with concurrency, mainly because requests of the same length seal pages and rebuild the CUDA Graph at the same step. Load balance. At a concurrency of 16, the coefficient of variation of the attention kernel time across requests is only 0.95%; variable page ranks do not cause noticeable load imbalance. Memory access efficiency. Due to permission restrictions on the server, we were unable to measure DRAM bandwidth and L2 hit rate, and thus cannot yet quantify the impact of variable ranks on memory access efficiency.
Q5: At 16K/32K, how much attention weight does pre-RoPE selection retain? How does it compare with post-RoPE selection?
Response: On Qwen3-8B, we use 15 paired segments from WikiText-103 to compare where SVD and candidate scoring are performed. Both settings use cross-head paged SVD, page size , scoring rank 32, an INT8 basis matrix, and a global budget .
| Length | Budget | Global | post-RoPE SVD, post-RoPE scoring (%) | pre-RoPE SVD, pre-RoPE scoring (%) |
|---|---|---|---|---|
| 16K | 2340 | 89.634 | 95.084 | |
| 16K | 2048 | 88.932 | 94.640 | |
| 16K | 1820 | 88.340 | 94.229 | |
| 32K | 4681 | 88.145 | 95.660 | |
| 32K | 4096 | 87.215 | 95.225 | |
| 32K | 3640 | 86.413 | 94.821 |
Under the default budget , the coverage of pre-RoPE SVD + pre-RoPE scoring is 94.64%/95.23% (16K/32K), which is 5.71/8.01 percentage points higher than that of post-RoPE SVD + post-RoPE scoring; it is higher under all three budgets, and the advantage grows with context length.
We sincerely thank Reviewer for the constructive comments, and apologize for the delayed response, as we conducted several additional experiments to address the concerns. All new results will be incorporated into the revised version.
Q1: What are the individual contributions of each component to throughput, memory, and quality?
Response: We thank the reviewer for the suggestion. We add a component comparison on Qwen3-8B FP16 with RULER 32K, using the first 100 samples of each of MK1/MK2/MV/MQ/FWE. The table below reports the decoding throughput of this component-ablation implementation on an RTX 5090; the paging configuration is S=96, and a global budget of k=4096 is used when selection is enabled. The K-SVD rows use an INT8 basis matrix and FP16 latent variables; the cache column reports the actual KV storage, excluding the model and temporary workspace. The throughput is measured after adding batched computation to the method in our paper.
For the separate quality/storage comparison of pure SVD and basis-matrix quantization, please refer to Q1 in our response to Reviewer bbef: with S=128, r=64, no selection, and unquantized factors in FP16, the scores of K-SVD / + quantized K basis / + quantized V are 97.733 / 97.533 / 97.533%, with cumulative cache savings of 21.79 / 34.22 / 58.73%; this group has not yet been benchmarked for speed separately. The full-history INT8 row differs in quantization granularity and RoPE position; it serves only as an overall scheme-level comparison and is not equivalent to KVQuant.
Q2: Why use SVD for K and INT8 for V? Is this better than the other two designs under the same memory budget?
Response: The low-rank representation of K can be reused for compressed-domain scoring and candidate reconstruction, while V INT8 avoids additional value decomposition and low-rank reconstruction. We measure these designs on 8 RULER tasks at 32K; the throughput is measured after adding batched computation to the method in our paper. The results are as follows:
K-SVD / V-INT8 (S=128) has the smallest KV cache, 4.2% smaller than K/V both INT8; its throughput is essentially on par (66.69 vs. 66.73 tokens/s), and its average score is only 0.02 percentage points lower. K/V both SVD achieves the highest score, but its throughput drops by about 44%. Therefore, K-SVD / V-INT8 achieves a good trade-off among storage, speed, and accuracy.