acceptodds
Under review as a conference paper at ICLR 2027

PASVD: Adaptive Cross-Head Key Selection from Low-Rank KV Caches

Abstract

Large language models (LLMs) are widely used in interactive assistants, long-document understanding, and complex reasoning. As context windows grow, however, the key–value (KV) cache becomes a major memory and data-movement bottleneck. Many existing low-rank methods construct projection spaces offline from model weights or calibration data, leaving fixed projections unable to adapt to the evolving token distribution of each sequence. Online alternatives adapt during generation, but their approximate updates may compromise representation quality and introduce additional overhead. We introduce PASVD, a training- and calibration-free method that jointly factorizes pre-RoPE keys across KV heads within each sealed page and adaptively selects a page-specific rank from its singular-value spectrum. During decoding, PASVD computes historical query–key scores directly in the compressed space and reconstructs only the selected keys for RoPE and final attention computation. This selection-before-reconstruction design avoids reconstructing the full historical key cache while enabling content-adaptive compression and sparse access.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Adaptive Rank Selection, KV Cache Compression, Low-Rank Approximation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5DG4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PASVD, a training-free method for KV-Cache compression. It applies SVD on un-RoPE-ed cross-head completed key pages, it chooses a rank for each completed page, and stores the compressed form only. During decode, per-query token, only selected key pages are reconstructed for final attention. Values are not compressed using SVD, but rather using INT8 quantization. The reported quality is close to FullKV on several benchmarks, and decode-only throughput on Qwen3-8B at 32K context is 2.37× higher at batch size 1.

Critical strengths

The combination of page-specific ranks and selection before reconstruction is well motivated: reconstructing every historical key would erase much of the potential speed benefit. The paper evaluates quality across long-context tasks, MMLU, and perplexity, and reports the storage layout and implementation choices needed to make variable ranks practical.

Critical weaknesses

W1. The contribution of each compression choice to throughput is unclear. The FullKV comparison changes several components simultaneously: key SVD, INT8 storage for key bases and values, candidate selection, selective reconstruction, and fused kernels. Consequently, the reported speedup cannot be attributed specifically to the proposed key factorization. The paper also does not explain why keys receive SVD compression while values receive only quantization, despite showing spectral decay for both. (Figure 1, Left part) A useful ablation would report throughput, memory, and quality for at least: FullKV; INT8 K and V; SVD K with FP16 V; SVD K with INT8 V; and SVD K and V. The last comparison would test the proposed K/V asymmetry directly.

W2. The batch-3 speedup is difficult to interpret. For Qwen3-8B at 32K, an FP16 FullKV cache is approximately 4.5 GiB per sequence, or 13.5 GiB at batch size 3. Together with roughly 16 GB of model weights, this puts the experiment close to the RTX 5090’s 32 GB capacity before other allocations. FullKV throughput falls from 24.51 tokens/s at batch 2 to 11.68 at batch 3, while PASVD rises to 88.11 tokens/s. The resulting 7.54× ratio may substantially reflect FullKV operating under memory pressure. Fragmentation is one possible cause, but the paper does not measure it. Fitting more sequences on a constrained GPU is a genuine benefit. However, this result alone does not establish a 7.54× compute-efficiency advantage in a less constrained deployment. Please report peak allocated and reserved memory, memory-bandwidth use, and throughput over more batch sizes or on a GPU where both methods have comfortable capacity. A comparison using similarly optimized attention implementations would also help, since FullKV uses the Hugging Face SDPA dynamic cache.

Another concern regarding batch size: the throughput evaluation stops at batch size 3. At such small batches, decode performance may be dominated by memory traffic, and the GPU may not be fully utilized. To support the method’s deployment claims, the authors should evaluate larger concurrent batches which characterize real-deployments, for example, 16–64 sequences, on hardware with sufficient memory—and report throughput, per-token latency, and tail latency. PASVD’s variable page ranks also raise an implementation question. Different pages can require different amounts of work, potentially causing load imbalance between warps or thread blocks. Packed factors may further affect memory-access efficiency, although variable rank does not necessarily prevent coalesced reads within a warp. GPU profiling and scaling results are needed to determine whether these costs grow at higher concurrency. Without that evidence, the paper establishes a useful KV-cache compression result, but leaves its benefit for heavily loaded serving systems unresolved.

W3. Pre-RoPE selection may miss positionally important keys. PASVD selects candidates using scores computed before RoPE, then applies RoPE only to selected keys for final attention. These rankings can differ. The appendix diagnostic finds that the pre-RoPE top-64 captures 52.76% of selectable historical post-RoPE attention mass, compared with 60.74% for an equal-budget post-RoPE oracle. This diagnostic uses short 2,048-token segments and excludes sink, recent, and active-page tokens. It does not show how often selection errors affect answers at the evaluated 16K and 32K lengths. I would like to see attention-mass recall and task quality as a function of selection budget, context length, and query head, especially for tasks requiring information spread across the context.

Questions for authors

Q1. Can you separate the effects of key SVD, value INT8 quantization, candidate selection, and fused kernels on throughput, memory use, and quality? Q2. Why are keys compressed with SVD while values are quantized? How does this design compare with INT8 K and V, and with SVD applied to both K and V, at matched memory budgets? Q3. What causes FullKV throughput to fall sharply at batch size 3? Please report peak memory allocation, memory-bandwidth use, and GPU utilization for both methods. Q4. How does PASVD perform at larger concurrent batches, such as 16–64 sequences on hardware with sufficient memory? Do variable page ranks cause load imbalance or reduce memory-access efficiency? Please include throughput and tail latency. Q5. At 16K and 32K context lengths, how much post-RoPE attention mass does pre-RoPE selection retain, and how does this change with the selection budget?

2 comments

We sincerely thank Reviewer for the constructive comments, and apologize for the delayed response, as we conducted several additional experiments to address the concerns. All new results will be incorporated into the revised version.

Q1: What are the individual contributions of each component to throughput, memory, and quality?

Response: We thank the reviewer for the suggestion. We add a component comparison on Qwen3-8B FP16 with RULER 32K, using the first 100 samples of each of MK1/MK2/MV/MQ/FWE. The table below reports the decoding throughput of this component-ablation implementation on an RTX 5090; the paging configuration is S=96S=96, and a global budget of k=4096k=4096 is used when selection is enabled. The K-SVD rows use an INT8 basis matrix and FP16 latent variables; the cache column reports the actual KV storage, excluding the model and temporary workspace. The throughput is measured after adding batched computation to the method in our paper.

ConfigurationFive-task avg. (%)Decoding throughput (tokens/s)KV cache (GiB)
FullKV (FP16)97.45020.8844.403
Full-history K/V INT8 (independent baseline)97.45026.4392.285
K-SVD + V FP16, selection + fusion97.23366.7583.628
K-SVD + V INT8, selection, unfused decoding attention97.18312.1382.522
K-SVD + V INT8, selection + fused decoding attention (PASVD)97.23366.6852.522
  • Fused decoding attention is the main source of speedup: with the same K-SVD + V INT8 representation, throughput increases from 12.138 to 66.685 tokens/s (5.49×).
  • V INT8 reduces memory: compared with V FP16, the KV cache is reduced by 30.49% (3.628 → 2.522 GiB), while the score and throughput remain essentially unchanged.
  • Overall effect: compared with FullKV, PASVD improves throughput by 3.19×, reduces the KV cache by 42.7%, and lowers the average score by only 0.217 percentage points; compared with full-history K/V INT8, it improves throughput by 2.52×.

For the separate quality/storage comparison of pure SVD and basis-matrix quantization, please refer to Q1 in our response to Reviewer bbef: with S=128S=128, r=64r=64, no selection, and unquantized factors in FP16, the scores of K-SVD / + quantized K basis / + quantized V are 97.733 / 97.533 / 97.533%, with cumulative cache savings of 21.79 / 34.22 / 58.73%; this group has not yet been benchmarked for speed separately. The full-history INT8 row differs in quantization granularity and RoPE position; it serves only as an overall scheme-level comparison and is not equivalent to KVQuant.

Q2: Why use SVD for K and INT8 for V? Is this better than the other two designs under the same memory budget?

Response: The low-rank representation of K can be reused for compressed-domain scoring and candidate reconstruction, while V INT8 avoids additional value decomposition and low-rank reconstruction. We measure these designs on 8 RULER tasks at 32K; the throughput is measured after adding batched computation to the method in our paper. The results are as follows:

DesignAvg. (%)Decoding throughput (tokens/s)KV cache (GiB)
K-SVD / V-INT8 (S=96S=96)97.18366.0872.313
K-SVD / V-INT8 (S=128S=128)97.21366.6912.190
K/V both INT897.23366.732.285
K/V both SVD97.76737.6222.316

K-SVD / V-INT8 (S=128S=128) has the smallest KV cache, 4.2% smaller than K/V both INT8; its throughput is essentially on par (66.69 vs. 66.73 tokens/s), and its average score is only 0.02 percentage points lower. K/V both SVD achieves the highest score, but its throughput drops by about 44%. Therefore, K-SVD / V-INT8 achieves a good trade-off among storage, speed, and accuracy.

Q3: Why does FullKV slow down at batch=3? Were memory, bandwidth, and GPU utilization measured?

Response: We thank the reviewer for the question. We measured memory, bandwidth, and GPU utilization. The main cause of the slowdown is that memory usage approaches the device limit, triggering frequent allocation retries and synchronizations, rather than bandwidth saturation. The experimental configuration is consistent with Table 7 of the paper: Qwen3-8B FP16, 32K context, a single RTX 5090; FullKV uses the HF SDPA dynamic cache.

MethodBatchThroughput (tokens/s)Peak allocated memory (GiB)DRAM read/write bandwidth (%)GPU utilization (%)Allocation retries
FullKV120.9620.3351.4 / 28.896.90
FullKV224.5125.3750.3 / 33.598.00
FullKV311.6830.4122.5 / 21.358.86,897
PASVD149.5818.8047.1 / 0.9884.70
PASVD273.5922.1735.8 / 0.9577.00
PASVD388.1125.5529.7 / 0.9671.50

The FullKV dynamic cache extends K/V with torch.cat at every step and every layer, producing a large number of temporary tensors. At batch=3, its reserved memory reaches 30.75 GiB, close to the available limit of 31.36 GiB; 6,897 allocation retries occur within 256 decoding steps, GPU utilization drops from 98.0% to 58.8%, and bandwidth decreases accordingly. PASVD uses 4.86 GiB less memory and incurs no retries.

Q4: Do larger concurrency and variable page ranks affect throughput, tail latency, and load balance?

Response: We thank the reviewer for the suggestion. We conducted additional decoding experiments with 1–16 concurrent sequences on a single NVIDIA A800 80GB GPU: Qwen3-8B, WikiText, 32K context; after warm-up, each configuration is measured three times and the median is reported. Throughput is the aggregate output speed of the entire batch, excluding prefill; P95/P99 are per-step decoding latencies. Concurrency of 32 could not be measured because it exceeds the GPU memory.

ConcurrencyThroughput (tokens/s)P95 (ms)P99 (ms)
139.6222.36137.02
254.2431.53213.62
361.7940.86282.93
873.9187.24796.19
1681.85161.621079.57

Throughput and tail latency. As concurrency increases from 1 to 16, throughput improves by about 2.1×, but does not scale linearly. Tail latency increases with concurrency, mainly because requests of the same length seal pages and rebuild the CUDA Graph at the same step. Load balance. At a concurrency of 16, the coefficient of variation of the attention kernel time across requests is only 0.95%; variable page ranks do not cause noticeable load imbalance. Memory access efficiency. Due to permission restrictions on the server, we were unable to measure DRAM bandwidth and L2 hit rate, and thus cannot yet quantify the impact of variable ranks on memory access efficiency.

Q5: At 16K/32K, how much attention weight does pre-RoPE selection retain? How does it compare with post-RoPE selection?

Response: On Qwen3-8B, we use 15 paired segments from WikiText-103 to compare where SVD and candidate scoring are performed. Both settings use cross-head paged SVD, page size S=96S=96, scoring rank 32, an INT8 basis matrix, and a global budget k=⌊N/d⌋k=\lfloor N/d\rfloor.

LengthBudgetGlobal kkpost-RoPE SVD, post-RoPE scoring (%)pre-RoPE SVD, pre-RoPE scoring (%)
16KN/7N/7234089.63495.084
16KN/8N/8204888.93294.640
16KN/9N/9182088.34094.229
32KN/7N/7468188.14595.660
32KN/8N/8409687.21595.225
32KN/9N/9364086.41394.821

Under the default budget N/8N/8, the coverage of pre-RoPE SVD + pre-RoPE scoring is 94.64%/95.23% (16K/32K), which is 5.71/8.01 percentage points higher than that of post-RoPE SVD + post-RoPE scoring; it is higher under all three budgets, and the advantage grows with context length.

Reviewer_bbefOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PASVD, a training- and calibration-free framework for low-rank KV-cache compression in long-context LLM inference. The method jointly factorizes pre-RoPE keys across all KV heads within each fixed-size token page via a single truncated SVD, selects a per-page rank from the residual spectral energy under an error threshold instead of a fixed global rank, stores the resulting heterogeneous-rank factors in a compact layout, and during decoding scores historical keys directly in the compressed space and reconstructs only the top- k selected keys before applying RoPE at their original positions.

Critical strengths
  1. The method design is well motivated.

  2. The experimental results are promising.

Critical weaknesses
  1. The core contribution is incremental, largely a combination of existing techniques. Low-rank KV compression, page-wise SVD, energy-threshold rank selection, and score-then-reconstruct are all individually known. The paper, however, lacks per-component ablations (e.g., cross-head vs. per-head SVD, pre-RoPE vs. post-RoPE at equal rank) to show that each design is individually necessary, making it hard to distinguish essential components.

  2. The central efficiency design is thinly validated, and the adaptive-rank benefit is limited. Pre-RoPE scoring as a selection proxy is supported by only a single operating point (about 84% of oracle attention mass), with no analysis of how proxy error varies with page size, rank, or top-k. Moreover, the K-side rank saving from adaptive selection over a fixed rank is modest, while most of the reported KV reduction comes from V-side INT8 quantization—and no pure-INT8 baseline is provided to isolate the net benefit of low-rank K compression.

Questions for authors

Can you provide per-component ablations to establish that each design choice is individually necessary? Specifically, please report: (a) cross-head joint SVD vs. per-head SVD at equal rank and page size, in both quality and SVD-call overhead; (b) a direct pre-RoPE vs. post-RoPE comparison at equal rank (beyond the spectral-energy diagnostic in Figure 1); and (c) page-adaptive rank vs. a fixed rank under the same average rank budget. These would clarify which design is essential and which is marginal, and would substantiate the claim that the combination itself constitutes a contribution beyond prior work.

1 comment

We sincerely thank Reviewer for the constructive comments, and apologize for the delayed response, as we conducted several additional experiments to address the concerns. All new results will be incorporated into the revised version.

Q1: Could you provide a component ablation to show the contribution of each design?

Response: We thank the reviewer for the suggestion. On Qwen3-8B with RULER 32K, S=128S=128, and an average rank of 64, we use the first 100 samples of each of 8 tasks and incrementally add three representation components.

ConfigurationAverage score (%)KV cache savings
FullKV FP1697.4500.00%
K-SVD (FP16 factors)97.73321.79%
K-SVD + INT8 K basis97.53334.22%
K-SVD + INT8 K basis + INT8 V97.45058.73%

Each of the three steps further reduces the cache, and the final configuration maintains the task quality of FullKV. The savings ratio denotes the cumulative KV cache savings relative to FullKV.

Q2: Under the same rank and page size, how do cross-head joint SVD and per-head SVD compare in terms of performance and invocation overhead?

Response: We compare them with fixed S=96S=96, an average rank of 80:

Experimental setup: Qwen3-8B, 32K context, batch size 1, page size 96, decomposition rank 80.

MethodDecoding throughput (tokens/s)Prefill + compression (s/request)KV cache (GiB)Per-page SVD (ms)
Cross-head joint SVD66.1344.6372.6691.222
Per-head independent SVD62.0405.0326.4251.132

Compared with the per-key parallel reconstruction in Table 7 of the paper, both implementations in this experiment organize the selected keys within the same page into matrix blocks and reconstruct them in batches with Tensor Cores. Compared with per-head independent SVD with batched submission, cross-head joint SVD improves decoding throughput by 6.60% and reduces KV cache usage by 58.45%.

Q3: Under the same rank, how does decomposition before RoPE differ from decomposition after RoPE?

Response: We fix S=96S=96, the factor precision, and V, and use full-history attention. We compare task scores on the first 100 samples of each of MK3/CWE/FWE/QA1, and measure the L2 error of the attention output relative to FullKV:

LengthRankpre-RoPE avg. score (%)post-RoPE avg. score (%)pre-RoPE output error (%)post-RoPE output error (%)Relative error reduction of pre-RoPE
16K6487.9087.902.5843.79231.87%
16K8087.9088.001.0721.60233.07%
32K6485.4785.932.1663.47737.71%
32K8085.4285.870.9001.40535.96%

The table shows that pre-RoPE decomposition reduces the sampled output error by 31.87%–37.71% in relative terms.

Q4: Under the same average rank budget, is page-adaptive rank better than fixed rank?

Response: Using the same tasks and samples as in Q3, we fix S=96S=96. For the adaptive scheme, the candidate ranks range from 64 to 96 with a step of 8, and the energy threshold is 0.997; the fixed rank is set to the average adaptive page rank for each input and each layer. The total prefill rank budgets of the two settings differ by less than 0.26%. We compare task scores on the first 100 samples of each of MK3/CWE/FWE/QA1.

LengthFixed / adaptive avg. rankFixed-rank avg. score (%)Adaptive-rank avg. score (%)Relative output error reduction
16K69.344 / 69.38887.9088.4012.88%
32K68.993 / 69.01784.8785.4712.46%

The adaptive scheme improves the average score by 0.50/0.60 percentage points and reduces the local output error by 12.88%/12.46% in relative terms.

Reviewer_fFCKOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed PASVD a page-adaptive low-rank key compression at the pre-RoPE stage. Eventhough the paper didn’t explicitly mentioned this, but the work combines most of the related works topics such as quantization, token selection (top-k at the page level) and low-rank decomposition.

Critical strengths

The paper addresses a critical problem on long context KV management. It compares against the recent low-rank KV decomposition methods and shows results closer to FullKV.

Critical weaknesses

Major Weakness:

  1. Since the rank is chosen from {80 or 96} and the S=96, It seems with \epsilon=0.997, it tends to retain most of the ranks as can be seen in Wikitext (~90) with the range of ~(81-94) imply it is skewed towards the right.
  2. Based on Line 215, it would take a lot of storage to keep the compressed elements than the raw elements per page I.e., r_p(D+S) > SD when r_p=[~90,..,96]. Similar analysis can be done for the scoring to show that scoring is better in raw element than the compressed elements when the r_p =~90.
  3. I am not sure I got the motivation on the pre-RoPE decomposition clearly. What is the impact of the RoPE in the low-rank KV decomposition methods?

Minor Weakness

  1. Impact of quantization error, paged vs full SVD error , top-K selection error and selection in the quantized space is unclear.
  2. No comparison with other quantization baselines such as KVQuant.
  3. Impact of token selection is not studies properly, what if we select all the 96 tokens per page? Since the baselines considered doesn’t do top-k, this seems like an unfair comparison.
  4. What is the impact of the GQA on the results? Since the baselines might not be using GQA, will this affect the comparison?
  5. How is your work different from ShadowKV from ICML 2025?
  6. Zero-shot MMLU might not be a good choice for this experiment due to the short token size per query and response and 16 sink tokens and 64 recent tokens and active page requirements.
Questions for authors
  1. Are there any hyper-parameter or ablations done on some of the key hyper parameters? how is page size S=96 chosen? Does it have any impact on the performance e.g., S=1 results in quantization method, S=k will be without token selection where as S=t (number of tokens so far) imply fullKV SVD?
  2. Top-k=64 in line 293 seems arbitrary, does it mean 64 out of 96 tokens were dequantized and uncompressed per page along with the 16 sink tokens and 64 recent tokens and active page?
  3. What is the impact of the quantization in the results?
  4. Minor corrections: Use \citep in line 32.
  5. Are all the results repeated 3 times as in Table 7?
2 comments

We sincerely thank Reviewer for the constructive comments, and apologize for the delayed response, as we conducted several additional experiments to address the concerns. All new results will be incorporated into the revised version.

Q1: With page size S=96S=96, a default rank candidate set of {80,96}\{80,96\}, the selected ranks may be high, so the savings brought by adaptivity are limited.

Response: We thank the reviewer. Our default configuration, with candidate ranks {80,96}\{80,96\} and a spectral energy retention threshold of 0.9970.997, is a conservative setting that prioritizes model quality. Table 6 shows that, over 166 WikiText-2 segments, the overall average rank is 90.24, and the per-segment average rank ranges from 81.3 to 94.6; compared with a fixed rank of 96, the actual K-cache storage is reduced by 5.85% on average. We therefore agree that the default configuration is biased toward higher ranks and that the additional savings from adaptivity are relatively limited. However, PASVD is not restricted to this candidate set. In Table 5, after expanding the candidate set to {64,72,80,88,96}\{64,72,80,88,96\}, the average rank drops to 67.68, reducing the K rank allocation by 29.50% relative to a fixed rank of 96, while the average score over four RULER-16K tasks decreases from 89.08 to 88.38, a drop of only 0.70%. Appendix Table A1 further shows that on LongBench, the average rank is 75.58, the K rank allocation is reduced by 21.27%, and the average score decreases from 34.57 to 34.34, a drop of 0.23%.

Q2: When the page size is S=96S=96 and the rank rpr_p approaches 96, would storing the low-rank factors consume more memory than storing the original keys?

Response: We thank the reviewer. When the rank approaches the page length, unquantized SVD factors may indeed take more storage than the original keys. Taking Qwen3-8B (D=1024D=1024) as an example, with S=rp=96S=r_p=96, the two FP16 factors occupy 2rp(D+S)=2150402r_p(D+S)=215040 bytes per page, which is 9.38% more than the 2SD=1966082SD=196608 bytes of the original FP16 K. After quantizing the basis matrix to INT8 following Eq. (10) in Section 4.4, and storing the latent variables and quantization scales in FP16, each page occupies rpD+2Srp+2rp=116928r_pD+2Sr_p+2r_p=116928 bytes, which is 40.53% less than the original K. In this case, the storage savings come from quantization.

When the page size is S=128S=128 and the average rank is 64, even with both SVD factors stored in FP16, each page requires on average only 2×64×(1024+128)=1474562\times64\times(1024+128)=147456 bytes, compared with 262144262144 bytes for the original FP16 K; low-rank K alone saves 43.75% of K storage. In the eight-task RULER 32K ablation, with S=128S=128 and an average rank of 64, K-SVD alone reduces the KV cache from 4.500 GiB to 3.519 GiB; after further quantizing the K basis matrix and V to INT8, the cache is reduced to 1.857 GiB, and the peak allocated memory decreases from 19.425 GiB to 17.763 GiB. Before and after quantization, the eight-task average score changes from 97.733% to 97.533%, a decrease of 0.2%.

Q3: The impact of RoPE on low-rank decomposition

Response: We thank the reviewer for the question. Let KK denote the pre-RoPE key matrix and RtR_t the RoPE rotation matrix at position tt. RoPE is an orthogonal transformation; therefore, if all tokens used the same rotation matrix RR, the transformed matrix would be KR⊤KR^\top, whose singular values are identical to those of KK, and its low-rank compressibility would remain unchanged. In practice, however, RoPE applies a different rotation matrix at each position, i.e., the tt-th row becomes ktRt⊤k_tR_t^\top. This is no longer a single orthogonal transformation applied to the whole matrix, so the rank and the singular value distribution of the matrix may change. For example, let two pre-RoPE key vectors both be (1,0)(1,0); the matrix they form has rank 1. If the two positions are rotated by 00 and θ\theta, respectively, the transformed vectors are (1,0)(1,0) and (cos⁡θ,sin⁡θ)(\cos\theta,\sin\theta); when sin⁡θ≠0\sin\theta\ne 0, the transformed matrix has rank 2. This example shows that position-dependent rotations may disperse originally shared low-dimensional directions, increasing the low-rank approximation error under the same rank budget. Moreover, Wertheimer et al. (2026), "Frayed RoPE and Long Inputs: A Geometric Perspective" prove in Lemma 1, Lemma 2, and Theorem 1 of Section 3.1.2 that, for rank-one inputs, under their magnitude-variation conditions and in the long-sequence limit, RoPE preserves the total squared energy of the singular values but reduces the energy disparity among the singular values. Meanwhile, Figure 1(a) of our paper shows that the pre-RoPE key matrix captures most of its spectral energy with only a small number of feature dimensions, indicating that it is well suited for low-rank compression. Therefore, we decompose K before RoPE (pre-RoPE).

Q4: Top-k=64 in line 293 seems arbitrary. Does it mean that 64 out of every 96 tokens in a page are dequantized and decompressed, in addition to 16 sink tokens, 64 recent tokens, and the active page?

Response: We apologize for the error in line 293 of the paper: describing the candidate budget as a fixed Top-k=64 is incorrect. For evaluation, the default global budget is k=⌊T/8⌋k=\lfloor T/8\rfloor, where TT is the context length in tokens; for 32K, k=4096k=4096. Each layer selects at most kk tokens in total from all sealed pages, rather than 64 tokens from each page. The final attention is computed over the union of these candidates, the 16 sink tokens, the 64 recent tokens, and the current active page, with overlapping positions computed only once.

Q5: How do the page size and the selection budget affect task quality, decoding throughput, and KV cache usage?

Response: Thank you for the question. We evaluate eight RULER 32K tasks on Qwen3-8B, using the first 100 samples of each task. The table below ablates the divisor dd in the budget formula k=⌊T/d⌋k=\lfloor T/d\rfloor: d=7,8,9d=7,8,9 correspond to global k=3640,4096,4681k=3640,4096,4681, respectively, where d=8d=8 is the default setting, and the average adaptive rank is 89.

Page size SSGlobal kkAvg. (%)Decoding throughput (tokens/s)KV cache (GiB)
64364097.0366.572.58
96364097.0367.312.55
128364097.2367.772.37
64409697.2366.352.58
96409697.2366.982.56
128409697.4367.582.37
64468197.3065.982.58
96468197.2566.812.56
128468197.4567.162.37

As shown above, S=128S=128 yields the smallest KV cache and the highest decoding throughput. With S=128S=128 fixed, increasing kk from 4096 to 4681 improves the average score by 0.02%. We will update the paper with the task results of the best configuration.

Q6: The impact of GQA on the results and on the fairness of comparisons.

Response: We thank the reviewer for the question. Qwen3-8B and Llama-3.1-8B-Instruct natively adopt GQA, and our locally reproduced baselines retain the same architecture; therefore, there is no inconsistency in the GQA configuration. In addition, Table 4 also reports results on Llama-2-7B, which does not use GQA, showing that PASVD is applicable to both MHA and GQA. On both types of architectures, PASVD reduces the KV cache while maintaining performance close to FullKV.

Q7: Differences from ShadowKV.

Response: ShadowKV builds a low-rank representation over the entire history and offloads V to the CPU, fetching it back when needed. In contrast, PASVD partitions the history into small pages, compresses each page independently, and determines how many ranks to retain according to the information distribution of each page.

Q8: Is each value in Table 7 repeated three times?

Response: We thank the reviewer for the question. In the speed tests, the result of the first run is excluded; we then run the test three more times and report the average.

Q9: Use \citep in line 32.

Response: We thank the reviewer for pointing this out. We will use \citep in line 32 to correct the citation format.

Q10: How does quantization affect task quality and cache usage?

Response: In the RULER 32K evaluation on Qwen3-8B, we use the first 100 samples of each of 8 tasks, fix the page size to 128 and the rank to 64, and use full-history attention in all settings. With K-SVD alone, the average score over the 8 tasks is 97.733%, saving 21.79% of the KV cache compared with FullKV; after quantizing the K basis matrix to INT8, the average score decreases by 0.2% to 97.533%, with cumulative savings of 34.22%; further quantizing V to INT8 leaves the average score unchanged and increases the cumulative savings to 58.73%. The average score of FullKV is 97.450%.

Q11: Comparison between full-history INT8 and PASVD

Speed test with fixed inputs. Qwen3-8B, 32K context, batch size 1. We report the mean of 3 formal measurements. PASVD uses a page size of 128, an average rank of 64, and global k=4681k=4681, with the FP16 key reconstruction cache enabled during speed testing.

MethodEight-task avg. (%)Decoding throughput (tokens/s)
Full-history K/V INT897.31766.736
PASVD97.45069.508

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.