acceptodds
Under review as a conference paper at ICLR 2027

ProjCache: Output-Projected KV Cache Eviction with OBS-Inspired Compensation

Abstract

Evicting key-value (KV) pairs reduces the memory cost of large language model inference but perturbs attention outputs. We introduce ProjCache, which unifies KV cache eviction and compensation under a shared layer-wise objective: the reconstruction error measured after the attention output projection. We parameterize both operations using token masks applied to attention weights and extend the saliency analysis of Optimal Brain Damage (OBD) to these mask coordinates. By summing all terms in the Taylor expansion of the reconstruction error, we derive an exact single-pair eviction score. This score accounts for attention renormalization and retains cross terms between query heads sharing a KV head in grouped-query attention. For compensation, we adapt Optimal Brain Surgeon (OBS) to jointly optimize the masks of retained tokens using a quadratic approximation of the same reconstruction objective. Experiments on LongBench and RULER with LLaMA-3.1-8B and Qwen3-8B show that replacing heuristic attention scores with ProjCache scores improves performance in 106 of the 108 evaluated settings, with gains of up to 3.4 points on LongBench and 25.6 points on RULER. ProjCache achieves performance comparable to state-of-the-art methods. For compensation, ProjCache lowers the reconstruction error by 17% to 44% in all 24 LongBench and RULER-4K settings, whereas prior compensation methods increase it in every setting.

Primary area:
generative models
Keywords:
KV Cache Eviction, KV Cache Compression, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VnfoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ProjCache unifies KV cache eviction and compensation under a single layer-wise objective—the reconstruction error measured after the attention output projection—parameterizing both as token masks on attention weights. For eviction, an exact single-pair score is derived from the full Taylor expansion of the objective (capturing attention renormalization and GQA cross-head terms), and for compensation, Optimal Brain Surgeon is adapted to jointly optimize the masks of retained tokens.

Critical strengths

The theoretical derivations in this paper are thorough and well-grounded.

Critical weaknesses
  1. The method clearly beats earlier baselines such as H2O and SnapKV. Against stronger, more recent methods like CriticalKV and OBCache, however, the improvements are marginal.

  2. The algorithm is more complex than existing methods, which likely adds computational overhead. The paper doesn't measure this. I'd suggest reporting latency, throughput, and memory usage against the baselines.

  3. Quantization is now widely used for KV caches. Showing that the method works alongside it would make the work more practical.

Questions for authors

See weaknesses.

8 comments
AuthorsOpenReview ↗

Author response to Reviewer Vnfo

Thank you for your comments.

W1. Comparisons with stronger baselines.

We have compared ProjCache with recent methods, including CriticalKV and OBCache (both ICML 2026). In manuscript Table 1, ProjCache achieves the best score in 15 of 36 settings on LLaMA-3.1-8B, compared with 12 for CriticalKV, 8 for OBCache-VK, and 3 for DropKV. We also added comparisons with ReST-KV (An et al., 2026). Results are shown below in Table 1. Here, ProjCache denotes ProjCache-AllFull, and ReST-KV denotes its score without smoothing. On Qwen3-8B RULER-4K, ProjCache outperforms ReST-KV in five of six settings (Table 1).

Table 1. RULER-4K accuracy (%) on Qwen3-8B at a 10% KV budget, without compensation. Scores are averaged over 13 tasks.

MethodQ-AwareQ-Agnostic
H2O
+ DropKV42.417.8
+ OBCache-VK35.417.4
+ CriticalKV36.417.6
+ ReST-KV45.918.6
+ ProjCache-SelfFull48.818.3
+ ProjCache49.319.3
SnapKV
+ DropKV43.418.1
+ OBCache-VK38.816.9
+ CriticalKV51.220.3
+ ReST-KV50.520.1
+ ProjCache-SelfFull47.720.2
+ ProjCache48.721.3
AdaKV
+ DropKV60.325.9
+ OBCache-VK57.123.8
+ CriticalKV52.923.9
+ ReST-KV57.024.9
+ ProjCache-SelfFull62.827.2
+ ProjCache63.527.9

We added experiments on a larger model, Qwen2.5-32B-Instruct. ProjCache averages 41.09 across all 16 LongBench tasks, versus 39.41 for CriticalKV, and scores higher on 13 tasks. Both use H2O in the Q-Aware setting with a 10% KV budget (Table 2).

Table 2. LongBench scores on Qwen2.5-32B-Instruct (Q-Aware) at a 10% KV budget with H2O, without compensation.

Single-Doc QA: NQA, Qasper, MF-en; Multi-Doc QA: HQA, 2WQA, MSQ; Summarization: GovR, QMSum, MNews; Few-shot: TREC, TQA, SAMSum; Synthetic: PCount, PRe; Code: LCC, RB-P.

MethodNQAQasperMF-enHQA2WQAMSQGovRQMSumMNewsTRECTQASAMSumPCountPReLCCRB-PAvg.
H2O27.6628.7138.5344.7737.9923.0025.2822.9318.7549.0084.7144.458.3191.0244.1636.2039.09
+ CriticalKV29.3628.5539.3348.7738.0823.1225.6223.0919.0048.5084.9943.997.6192.0243.2735.2139.41
+ ProjCache29.0131.7540.2451.0543.4725.5026.8423.0419.6953.5086.1844.718.9891.1843.6138.6941.09

We also added experiments on Qwen3.5-9B, which uses output gates. ProjCache-SelfFull and ProjCache average 88.79 and 88.65 across all 13 RULER-48K tasks, versus 88.01 for CriticalKV and 87.31 for H2O. These experiments use H2O in the Q-Aware setting with a 5% KV budget (Table 3). We compress KV only in the eight softmax-attention layers and leave the 24 Gated DeltaNet states unchanged.

Table 3. RULER-48K accuracy (%) on Qwen3.5-9B (Q-Aware) at a 5% KV budget with H2O, without compensation.

MethodCWEFWEMK-1MK-2MK-3MQMVS-1S-2S-3QA-1QA-2VTAvg.
H2O43.6884.80100.0076.0082.40100.0097.80100.00100.0096.0089.6064.80100.0087.31
+ CriticalKV48.7283.47100.0076.0086.40100.0099.20100.00100.0096.0089.6064.80100.0088.01
+ ProjCache-SelfFull53.6084.27100.0089.6087.20100.0095.60100.00100.0089.6089.6064.80100.0088.79
+ ProjCache52.0883.73100.0088.0087.20100.0095.80100.00100.0090.4089.6065.60100.0088.65
AuthorsOpenReview ↗

Response continued

We suggest that the large gap between CriticalKV and ProjCache in some Q-Agnostic settings can be explained by the query-independent projected value-norm term in CriticalKV's score implementation. We follow the notation in the CriticalKV paper. Its first stage reserves tokens using attention scores, and its second stage computes

\qquad \mathcal V={\boldsymbol{V}}{\boldsymbol{W}}_O,$$ $$\mathcal A = ({\boldsymbol{A}}+\epsilon)\odot\\|\mathcal V\\|_{1,\mathrm{row}} = {\boldsymbol{A}}\odot\\|\mathcal V\\|_{1,\mathrm{row}} +\epsilon\\|\mathcal V\\|_{1,\mathrm{row}}.$$ CriticalKV heuristically adds the query-independent term $\epsilon\\|\mathcal V\\|_{1,\mathrm{row}}$ in its algorithm and implementation. This term is absent from its derivation. Our code followed the default $\epsilon=10^{-4}$. We investigate its effect by rerunning CriticalKV w/ AdaKV with $\epsilon\in\\{10^{-4},10^{-7},0\\}$ on Qwen3-8B, using Q-Agnostic RULER-32K and a 20% KV budget (Table 4). We further test two variants: CriticalKV using only $\\|\mathcal V\\|_{1,\mathrm{row}}$ for second-stage ranking, and ProjCache adding $10^{-4}\\|\mathcal V\\|_{1,\mathrm{row}}$ to its original score. The two variants score 68.33 and 75.12, respectively. We find that this term has a large effect on performance. ProjCache outperforms CriticalKV both without it (41.0 vs. 37.5) and with it (75.12 vs. 68.0), achieving the highest score in Table 4. **Table 4.** Epsilon and value-norm ablations on Qwen3-8B (Q-Agnostic), using RULER-32K at a 20% KV budget. Scores are averaged over 13 tasks. | Method | $\epsilon$ | Average score | | --- | :---: | :---: | | CriticalKV w/ AdaKV | $10^{-4}$ | 68.00 | | CriticalKV w/ AdaKV | $10^{-7}$ | 37.60 | | CriticalKV w/ AdaKV | $0$ | 37.50 | | CriticalKV w/ AdaKV (norm-only) | – | 68.33 | | ProjCache w/ AdaKV | – | 41.00 | | ProjCache w/ AdaKV + value norm | $10^{-4}$ | 75.12 | At layers 0, 12, 24, and 35, respectively, 97.73%, 98.55%, 98.61%, and 98.98% of attention scores are below $10^{-4}$ (Table 5). When ${\boldsymbol{A}}\ll\epsilon$, the second-stage score is approximately $\epsilon\\|\mathcal V\\|_{1,\mathrm{row}}$, so selection depends mainly on the projected value norm. This could help explain the large effect of epsilon. **Table 5.** Attention scores below $10^{-4}$ on Qwen3-8B (Q-Agnostic), using AdaKV on RULER-32K at a 20% KV budget. Results are averaged over all RULER-32K samples; the observation window is excluded. | Layer | Attention scores below $10^{-4}$ | | --- | :---: | | 0 | 97.73% | | 12 | 98.55% | | 24 | 98.61% | | 35 | 98.98% | ## W2. Latency, throughput, and memory. Without compensation, ProjCache has similar decoding latency and memory use to OBCache-VK and CriticalKV (Appendix B.1, Figures 4–5). For example, at 96K and batch size one, ProjCache and CriticalKV have decoding latencies of 14.388 and 14.337 ms/token and peak decoding memory of 16.397 and 16.272 GiB, respectively. TTFT is slightly higher in this setting, at 11.534 versus 11.081 s. The appendix covers 24K–96K inputs and batch sizes one and four on LLaMA-3.1-8B, using one H100 in BF16. Appendix C.3 reports the additional TTFT and memory costs of compensation, including Hessian construction and the solve. Appendix B.1 (manuscript Table 4) also shows that ProjCache has scoring complexity comparable to CriticalKV. With the required Gram blocks precomputed from ${\boldsymbol{W}}_O$, ProjCache computes scores in the KV-group dimension, while both methods scale linearly with cache length. We also measure the TTFT of compensation (Table 6). TTFT increases from 2.247 to 5.268 seconds at batch size 1 and from 9.123 to 17.826 seconds at batch size 4. Optimized GPU kernels could reduce this overhead. We leave this optimization for future work. **Table 6.** TTFT and memory usage with and without compensation on LLaMA-3.1-8B with synthetic 32K prompts and a 10% KV budget, measured in BF16 on one H100. | Method | TTFT (s) | Memory (GiB) | | --- | :---: | :---: | | *Batch size 1* | | | | ProjCache | $2.247\pm.004$ | 15.584 | | ProjCache + OBS | $5.268\pm.001$ | 15.793 | | *Batch size 4* | | | | ProjCache | $9.123\pm.008$ | 16.994 | | ProjCache + OBS | $17.826\pm.025$ | 17.831 |
AuthorsOpenReview ↗

Response continued

W3. Compatibility with KV-cache quantization.

Following ReST-KV (An et al., 2026), we apply grouped 4-bit quantization with KVQuant-style outlier handling (Hooper et al., 2024). ProjCache with packed 4-bit KV scores 44.76 on LongBench, versus 45.14 for the unquantized manuscript reference (Table 7). This evaluation covers all 16 tasks (3,750 examples) on LongBench, using LLaMA-3.1-8B with H2O in the Q-Aware setting with a 10% KV budget and a window size of 64.

Table 7. LongBench scores of ProjCache with unquantized and packed 4-bit KV on LLaMA-3.1-8B (Q-Aware) at a 10% KV budget with H2O, without compensation. Scores are averaged over 16 tasks.

Single-Doc QA: NQA, Qasper, MF-en; Multi-Doc QA: HQA, 2WQA, MSQ; Summarization: GovR, QMSum, MNews; Few-shot: TREC, TQA, SAMSum; Synthetic: PCount, PRe; Code: LCC, RB-P.

MethodNQAQasperMF-enHQA2WQAMSQGovRQMSumMNewsTRECTQASAMSumPCountPReLCCRB-PAvg.
ProjCache29.9234.0952.5056.0048.3231.1827.0724.3421.1048.5089.9943.5912.62100.0050.4452.6145.14
+ INT4 KV30.2735.2952.6755.1148.5828.3426.8923.9920.8848.5090.0442.5111.22100.0050.4651.3644.76

We leave the joint treatment of quantization and eviction within our objective for future work.

References

  • Yongqi An, Chang Lu, Kuan Zhu, Tao Yu, Chaoyang Zhao, Hong Wu, Ming Tang, and Jinqiao Wang. ReST-KV: Robust KV cache eviction with layer-wise output reconstruction and spatial-temporal smoothing. In International Conference on Learning Representations (ICLR), 2026. URL OpenReview.

  • Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024. URL NeurIPS proceedings.

Reviewer_Tp7tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies KV-cache eviction based on attention-output reconstruction error. It derives an exact output-projected error score for deleting a single KV entry with attention renormalization, accounting for interactions among query heads sharing a KV head. It also introduces OBS-inspired compensation that reweights retained entries. Experiments evaluate two 8B models on LongBench, RULER, and PG19, with model-level latency and memory measurements for scoring variants.

Critical strengths
  1. A well-defined scoring objective and clear derivations. The method explicitly accounts for attention renormalization, output projection, and within-group cross-head interactions, providing a reasonable basis for cache-importance scoring. The paper also clearly distinguishes exact single-entry deletion error from its approximate use for multi-entry eviction.

  2. Relevant baselines and practical implementation evidence. Experiments compare against DropKV, OBCache, and CriticalKV across multiple cache-selection frameworks. The paper reports model-level latency and memory measurements and transparently acknowledges the inconsistent task-level benefits of compensation.

Critical weaknesses
  1. Limited methodological novelty and insufficient validation of the core additions. DropKV and CriticalKV have already studied exact deletion error and output projection, respectively. In Table 2, full scoring and its diagonal approximation perform similarly in most settings, while ablations do not isolate cross-head terms from within-block off-diagonal terms. Advantages over strong baselines also depend on the configuration.

  2. Limited task-level benefits from compensation. Despite reducing reconstruction error, compensation changes task-average scores by only −0.3 to +0.7 points in Table 3. The error is measured separately for each KV group; improvements in full-layer output error and on subsequent queries not used for fitting remain unverified. Its benefits for future generation quality are therefore unclear.

  3. Limited coverage of model scales and tasks. Experiments cover only two 8B models, and dynamic eviction is evaluated solely through teacher-forced perplexity on PG19. Although code completion and multi-hop QA are included, long-chain mathematical reasoning and execution-based code generation are not evaluated. Effectiveness across scales and during sustained generation remains unclear.

  4. Unclear deployment costs of compensation. The system evaluation does not explicitly report the additional latency and memory costs of the full compensation pipeline. The damping parameter and fallback for negative weights also receive little analysis, leaving it unclear whether compensation provides sufficient benefits to justify its costs.

Questions for authors

Please address the four concerns outlined under Critical Weaknesses.

7 comments
AuthorsOpenReview ↗

Author response to Reviewer Tp7t

Thank you for your comments.

W1. Contribution and scoring comparisons.

Our token-mask formulation connects Taylor-based eviction scoring with OBS-inspired compensation under a shared output-projected reconstruction objective. Summing all Taylor orders yields the exact single-pair eviction score, accounting for attention renormalization and GQA cross-head interactions. Our OBS-inspired compensation jointly adjusts retained token masks within each KV head using a regularized quadratic approximation of the same objective. The eviction set and cached keys and values stay fixed.

Compared with DropKV, our unprojected score recovers its exact deletion score for the same observation queries, providing a Taylor interpretation. We extend this analysis to reconstruction error after WO{\boldsymbol{W}}_O, including interactions between query heads sharing a KV head under GQA.

Compared with CriticalKV, our score measures the exact projected squared-L2L_2 error of deleting one KV pair. CriticalKV derives its score from an upper bound on each query head's projected L1L_1 error.

We compare three scoring variants. Define the Gram matrix and its query-head blocks as

{\boldsymbol{M}}_{h,h'}={\boldsymbol{W}}_{O,h}{\boldsymbol{W}}_{O,h'}^{\top}.$$ For query heads $h,h'$ sharing a KV head: - ProjCache-AllFull (shown as ProjCache in the tables) uses all blocks ${\boldsymbol{M}}_{h,h'}$. - ProjCache-SelfFull uses only blocks ${\boldsymbol{M}}_{h,h}$. - ProjCache-SelfDiag uses only the diagonal entries of ${\boldsymbol{M}}_{h,h}$. We added ProjCache-SelfFull results on LongBench and Qwen3 RULER-4K to isolate the effect of cross-head interactions. Compared with ProjCache-SelfFull, ProjCache scores higher in all four H2O settings on LongBench, while SnapKV and AdaKV show both gains and drops (Table 1). On Qwen3 RULER-4K, ProjCache outperforms ProjCache-SelfFull in all six settings (Table 2). **Table 1.** LongBench scores on LLaMA-3.1-8B and Qwen3-8B at a 10% KV budget, without compensation. Scores are averaged over 16 tasks. | Method | LLaMA-3.1-8B Q-Aware | LLaMA-3.1-8B Q-Agnostic | Qwen3-8B Q-Aware | Qwen3-8B Q-Agnostic | | --- | :---: | :---: | :---: | :---: | | H2O | | | | | | + DropKV | 45.1 | 36.3 | 44.8 | 37.4 | | + OBCache-VK | 44.9 | 36.0 | 44.5 | 37.1 | | + CriticalKV | 44.6 | 34.9 | 44.3 | 36.2 | | + ReST-KV | 44.7 | 35.2 | 44.3 | 36.5 | | + ProjCache-SelfDiag | **45.2** | 36.5 | 44.8 | 37.2 | | + ProjCache-SelfFull | 45.0 | 36.7 | 44.6 | 37.5 | | + ProjCache | 45.1 | **37.0** | **44.9** | **37.9** | | SnapKV | | | | | | + DropKV | 45.1 | 35.6 | 44.6 | 36.0 | | + OBCache-VK | 45.0 | 35.7 | 44.9 | 36.4 | | + CriticalKV | **45.4** | **36.8** | **45.2** | **38.0** | | + ReST-KV | 44.9 | 35.0 | 44.2 | 36.0 | | + ProjCache-SelfDiag | 45.1 | 35.8 | 44.6 | 36.7 | | + ProjCache-SelfFull | 45.0 | 36.1 | 44.8 | 36.9 | | + ProjCache | 45.1 | 35.9 | 44.8 | 36.7 | | AdaKV | | | | | | + DropKV | 46.7 | 39.6 | 45.9 | 39.3 | | + OBCache-VK | 46.8 | 38.9 | **46.6** | **39.8** | | + CriticalKV | 46.2 | 38.1 | 45.6 | 39.7 | | + ReST-KV | 45.5 | 36.6 | 45.0 | 36.8 | | + ProjCache-SelfDiag | **46.9** | **40.1** | 45.9 | 38.9 | | + ProjCache-SelfFull | **46.9** | 39.8 | 45.9 | 39.2 | | + ProjCache | **46.9** | 39.7 | 46.1 | 39.4 | **Table 2.** RULER-4K accuracy (%) on Qwen3-8B at a 10% KV budget, without compensation. Scores are averaged over 13 tasks. | Method | Q-Aware | Q-Agnostic | | --- | :---: | :---: | | H2O | | | | + DropKV | 42.4 | 17.8 | | + OBCache-VK | 35.4 | 17.4 | | + CriticalKV | 36.4 | 17.6 | | + ReST-KV | 45.9 | 18.6 | | + ProjCache-SelfFull | 48.8 | 18.3 | | + ProjCache | **49.3** | **19.3** | | SnapKV | | | | + DropKV | 43.4 | 18.1 | | + OBCache-VK | 38.8 | 16.9 | | + CriticalKV | **51.2** | 20.3 | | + ReST-KV | 50.5 | 20.1 | | + ProjCache-SelfFull | 47.7 | 20.2 | | + ProjCache | 48.7 | **21.3** | | AdaKV | | | | + DropKV | 60.3 | 25.9 | | + OBCache-VK | 57.1 | 23.8 | | + CriticalKV | 52.9 | 23.9 | | + ReST-KV | 57.0 | 24.9 | | + ProjCache-SelfFull | 62.8 | 27.2 | | + ProjCache | **63.5** | **27.9** |
AuthorsOpenReview ↗

Response continued

Compared with strong baselines, ProjCache achieves the best score in 15 of 36 settings on LLaMA-3.1-8B, versus 12 for CriticalKV, 8 for OBCache-VK, and 3 for DropKV (manuscript Table 1).

We suggest that the large gap between CriticalKV and ProjCache in some Q-Agnostic settings can be explained by the query-independent projected value-norm term in CriticalKV's score implementation. We follow the notation in the CriticalKV paper. Its first stage reserves tokens using attention scores, and its second stage computes

\qquad \mathcal V={\boldsymbol{V}}{\boldsymbol{W}}_O,$$ $$\mathcal A = ({\boldsymbol{A}}+\epsilon)\odot\\|\mathcal V\\|_{1,\mathrm{row}} = {\boldsymbol{A}}\odot\\|\mathcal V\\|_{1,\mathrm{row}} +\epsilon\\|\mathcal V\\|_{1,\mathrm{row}}.$$ CriticalKV heuristically adds the query-independent term $\epsilon\\|\mathcal V\\|_{1,\mathrm{row}}$ in its algorithm and implementation. This term is absent from its derivation. Our code followed the default $\epsilon=10^{-4}$. We investigate its effect by rerunning CriticalKV w/ AdaKV with $\epsilon\in\\{10^{-4},10^{-7},0\\}$ on Qwen3-8B, using Q-Agnostic RULER-32K and a 20% KV budget (Table 3). We further test two variants: CriticalKV using only $\\|\mathcal V\\|_{1,\mathrm{row}}$ for second-stage ranking, and ProjCache adding $10^{-4}\\|\mathcal V\\|_{1,\mathrm{row}}$ to its original score. The two variants score 68.33 and 75.12, respectively. We find that this term has a large effect on performance. ProjCache outperforms CriticalKV both without it (41.0 vs. 37.5) and with it (75.12 vs. 68.0), achieving the highest score in Table 3. **Table 3.** Epsilon and value-norm ablations on Qwen3-8B (Q-Agnostic), using RULER-32K at a 20% KV budget. Scores are averaged over 13 tasks. | Method | $\epsilon$ | Average score | | --- | :---: | :---: | | CriticalKV w/ AdaKV | $10^{-4}$ | 68.00 | | CriticalKV w/ AdaKV | $10^{-7}$ | 37.60 | | CriticalKV w/ AdaKV | $0$ | 37.50 | | CriticalKV w/ AdaKV (norm-only) | – | 68.33 | | ProjCache w/ AdaKV | – | 41.00 | | ProjCache w/ AdaKV + value norm | $10^{-4}$ | 75.12 | At layers 0, 12, 24, and 35, respectively, 97.73%, 98.55%, 98.61%, and 98.98% of attention scores are below $10^{-4}$ (Table 4). When ${\boldsymbol{A}}\ll\epsilon$, the second-stage score is approximately $\epsilon\\|\mathcal V\\|_{1,\mathrm{row}}$, so selection depends mainly on the projected value norm. This could help explain the large effect of epsilon. **Table 4.** Attention scores below $10^{-4}$ on Qwen3-8B (Q-Agnostic), using AdaKV on RULER-32K at a 20% KV budget. Results are averaged over all RULER-32K samples; the observation window is excluded. | Layer | Attention scores below $10^{-4}$ | | --- | :---: | | 0 | 97.73% | | 12 | 98.55% | | 24 | 98.61% | | 35 | 98.98% |
AuthorsOpenReview ↗

Response continued

W2/W4. Compensation quality and cost.

In the manuscript experiments, we set the damping parameter to λ=50\lambda=50. Compensation is skipped for 34.35% of KV heads over the full LongBench evaluation. Details can be found in Appendix C.3 (Table 25). We skip a head when the solve fails or returns negative or invalid weights.

We sample 50 examples from five LongBench tasks (10 per task) for each of the MSE and damping evaluations below.

Compensation reduces mean relative output MSE by 12.55% on answer queries excluded from fitting (Table 5). Specifically, we first prefill the context and question and use the observation queries for KV selection and compensation. Compensation keeps the eviction set fixed and adjusts only retained token masks. Second, we feed the same reference answer tokens using three caches: Full KV, ProjCache without compensation, and ProjCache with compensation. These answer tokens are not used for selection or compensation. Third, we measure the MSE of attention outputs after WO{\boldsymbol{W}}_O at answer positions in each layer, using Full KV as the reference. We report the relative MSE change, 100×(MSEcomp/MSEprune−1)100\times(\mathrm{MSE}_{\mathrm{comp}}/\mathrm{MSE}_{\mathrm{prune}}-1), averaged over examples and layers.

Table 5. Answer-token output MSE on Qwen3-8B (Q-Aware), with ProjCache + SnapKV, a 20% KV budget, and λ=50\lambda=50. Results are based on 10 examples per task.

TaskMean Δ\DeltaMSE (%)
Qasper−12.96-12.96
MF-en−15.45-15.45
HotpotQA−9.05-9.05
2WQA−21.30-21.30
MuSiQue−4.01-4.01
All 50 examples−12.55-12.55

We investigate the effect of damping values λ∈5,50,500\lambda\in\\{5,50,500\\} on per-KV-head MSE using Qwen3-8B, ProjCache + SnapKV, and a 20% KV budget. With Full-KV layer inputs and eviction sets fixed, we compute compensation mask weights using an observation window of 64 queries at the end of a 4K prefix. We report output MSE on both the observation window and the next 64 queries, using the same mask weights. The next 64 queries are excluded from mask computation. λ=50\lambda=50 gives the largest MSE reductions: 22.94% on the observation window and 13.27% on the next 64 queries (Table 6). Larger damping reduces skips but weakens compensation at λ=500\lambda=500. Heads with negative weights remain uncompensated.

Table 6. Damping sensitivity of OBS compensation on Qwen3-8B.

λ\lambdaObservation (%)Next 64 (%)Skipped (%)
5−20.48-20.48−9.06-9.0657.52
50−22.94-22.94−13.27-13.2718.74
500−5.03-5.03−3.26-3.264.35

We also measure the TTFT of compensation (Table 7). TTFT increases from 2.247 to 5.268 seconds at batch size 1 and from 9.123 to 17.826 seconds at batch size 4. Optimized GPU kernels could reduce this overhead. We leave this optimization for future work.

Table 7. TTFT and memory usage with and without compensation on LLaMA-3.1-8B with synthetic 32K prompts and a 10% KV budget, measured in BF16 on one H100.

MethodTTFT (s)Memory (GiB)
Batch size 1
ProjCache2.247±.0042.247\pm.00415.584
ProjCache + OBS5.268±.0015.268\pm.00115.793
Batch size 4
ProjCache9.123±.0089.123\pm.00816.994
ProjCache + OBS17.826±.02517.826\pm.02517.831
Reviewer_aThpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProjCache scores each KV entry by how much the layer’s attention output changes, measured after the output projection, when that entry is removed and the remaining attention weights are renormalized. The paper writes this error as the sum of all Taylor terms in token-mask coordinates, which gives an exact closed form for removing one entry. Under GQA, the score adds up the output changes of the query heads that share a KV head before taking the norm, so it includes cross-head terms. Cheaper variants keep only the per-head blocks of W_O W_Oᵀ (SELF-FULL), or only their diagonals (SELF-DIAG). A separate OBS-inspired step keeps the eviction set fixed and solves a regularized quadratic problem to rescale the masks of retained tokens. Cached keys and values are not changed. The score replaces the attention scores in H2O, SnapKV, and AdaKV. It is evaluated on LLaMA-3.1-8B and Qwen3-8B with LongBench and RULER (4K/32K), plus PG19 streaming decoding and runtime and memory measurements. Replacing attention scores improves 106 of 108 settings. Compensation lowers reconstruction MSE by 17%–44% in all 24 tested settings, but task averages change only by −0.3 to +0.7.

Critical strengths
  1. The paper derives the exact error of removing one KV entry after the output projection, with renormalization and an explicit distinction from setting the entry to zero. Summing within a GQA group lets the score capture interactions between query heads. The Taylor view also relates the score to OBCache and DropKV.
  2. The score is cheap to use. SELF-DIAG adds almost no cost over attention scores, and it plugs into H2O, SnapKV, and AdaKV without other changes. Across two models and three benchmarks, this replacement improves 106 of 108 settings.
  3. The evaluation is careful and honest. Runtime and memory are reported with standard deviations up to 96K tokens. The paper also states plainly that compensation lowers reconstruction error without improving task scores.
  4. The paper is clearly written. Notation is consistent, derivations are complete, and assumptions such as the single-deletion limitation are stated.
Critical weaknesses
  1. ReST-KV [1] (Eqs. 6–7) already gives the exact single-pair deletion error after W_O, with renormalization, computed per head. For one query and one head per KV head, ProjCache Eq. 10 is its square. The paper neither compares with [1], so the new part reduces to the GQA cross-head terms. Moreover, SELF-DIAG stays within 0.5 points of ALL-FULL in 32 of 36 LongBench settings, and there are no SELF-FULL accuracy results. The Qwen3 RULER tables lack DropKV and OBCache.

  2. The paper doesn’t discuss a large gap on query-agnostic RULER-32K. CriticalKV beats ProjCache in 7 of 9 settings in Table 16, for example 68.0 vs 41.0 (AdaKV, 20%). A similar gap appears for LLaMA in Table 1 (48.8 vs 31.6).

  3. The compensation results are weak. ΔMSE is measured on the same queries used to fit the masks. Task gains are small and inconsistent. Damping sensitivity, skip rate, and runtime are missing.

  4. Model and architecture coverage is narrow.

  • Only two 8B dense GQA models are tested, with accuracy up to 32K tokens.
  • Recent models such as Qwen3.5 and Qwen3.8-Flash-Next keep a KV cache in only one layer in four (the rest are Gated DeltaNet). They use larger GQA groups (24 query heads over 4 or 2 KV heads), output gates before W_O (Qwen3.5), or top-k sparse attention (QSA).
  • Each of these features breaks an assumption the score relies on: a fixed Gram matrix, dense renormalization, or a cache in every layer.
Questions for authors
  1. How does Eq. 10 relate to Eq. 7 of ReST-KV? Under the same observation queries, budget, and policy, how do ALL-FULL, SELF-FULL, and a per-head projected score (such as ReST-KV without smoothing) compare, including on Qwen3 RULER?

  2. How would ProjCache adapt to hybrid and sparse-attention models?

  • With an output gate, is the gate folded into the score per query, and what does that cost?
  • Do the gains hold when only the softmax-attention layers keep a cache, for example on Qwen3.5-9B or 27B beyond 32K?
  • Under QSA-style token selection, should the score be computed over the selected subset, and how should the indexer’s cache be kept consistent?
  • Do larger GQA groups make the cross-head terms matter more?
4 comments
AuthorsOpenReview ↗

Author response to Reviewer aThp

W1/Q1. ReST-KV and scoring variants.

Thank you for pointing out ReST-KV. We now cite it and explain the relationship in the manuscript, with the additions highlighted in red. For one observation query and one query head per KV head, our Eq. 10 is the square of ReST-KV's unsmoothed per-head score and gives the same ranking.

We list three scores below. Here, ProjCache denotes ProjCache-AllFull, and ReST-KV denotes its score without smoothing:

S_{g,p}^{\mathrm{ProjCache}} &=\sum_i\left\\| \sum_{h\in{\mathbb{Q}}_g}\frac{a_{i,p}^{h}}{1-a_{i,p}^{h}} ({\boldsymbol{o}}_i^h-{\boldsymbol{v}}_{g,p}){\boldsymbol{W}}_{O,h} \right\\|_2^2,\\\\ S_{g,p}^{\mathrm{ProjCache\text{-}SelfFull}} &=\sum_i\sum_{h\in{\mathbb{Q}}_g}\left\\| \frac{a_{i,p}^{h}}{1-a_{i,p}^{h}} ({\boldsymbol{o}}_i^h-{\boldsymbol{v}}_{g,p}){\boldsymbol{W}}_{O,h} \right\\|_2^2,\\\\ S_{g,p}^{\mathrm{ReST\text{-}KV}} &=\sum_i\sum_{h\in{\mathbb{Q}}_g}\left\\| \frac{a_{i,p}^{h}}{1-a_{i,p}^{h}} ({\boldsymbol{o}}_i^h-{\boldsymbol{v}}_{g,p}){\boldsymbol{W}}_{O,h} \right\\|_2. \end{aligned}$$ ProjCache sums projected perturbations across heads before taking the squared norm. ProjCache-SelfFull instead retains only the diagonal head blocks ${\boldsymbol{W}}_{O,h}{\boldsymbol{W}}_{O,h}^{\top}$ of ${\boldsymbol{W}}_O{\boldsymbol{W}}_O^{\top}$: each query–head term is the square of ReST-KV's per-head score. ProjCache-SelfDiag also removes within-head off-diagonal terms. We added ProjCache-SelfFull and ReST-KV results on LongBench and Qwen3 RULER-4K, plus DropKV and OBCache-VK results on Qwen3 RULER-4K. We find that squaring the norm improves scores. ProjCache-SelfFull outperforms ReST-KV in all 12 LongBench settings (Table 1). **Table 1.** LongBench scores on LLaMA-3.1-8B and Qwen3-8B at a 10% KV budget, without compensation. Scores are averaged over 16 tasks. | Method | LLaMA-3.1-8B Q-Aware | LLaMA-3.1-8B Q-Agnostic | Qwen3-8B Q-Aware | Qwen3-8B Q-Agnostic | | --- | :---: | :---: | :---: | :---: | | H2O | | | | | | + DropKV | 45.1 | 36.3 | 44.8 | 37.4 | | + OBCache-VK | 44.9 | 36.0 | 44.5 | 37.1 | | + CriticalKV | 44.6 | 34.9 | 44.3 | 36.2 | | + ReST-KV | 44.7 | 35.2 | 44.3 | 36.5 | | + ProjCache-SelfDiag | **45.2** | 36.5 | 44.8 | 37.2 | | + ProjCache-SelfFull | 45.0 | 36.7 | 44.6 | 37.5 | | + ProjCache | 45.1 | **37.0** | **44.9** | **37.9** | | SnapKV | | | | | | + DropKV | 45.1 | 35.6 | 44.6 | 36.0 | | + OBCache-VK | 45.0 | 35.7 | 44.9 | 36.4 | | + CriticalKV | **45.4** | **36.8** | **45.2** | **38.0** | | + ReST-KV | 44.9 | 35.0 | 44.2 | 36.0 | | + ProjCache-SelfDiag | 45.1 | 35.8 | 44.6 | 36.7 | | + ProjCache-SelfFull | 45.0 | 36.1 | 44.8 | 36.9 | | + ProjCache | 45.1 | 35.9 | 44.8 | 36.7 | | AdaKV | | | | | | + DropKV | 46.7 | 39.6 | 45.9 | 39.3 | | + OBCache-VK | 46.8 | 38.9 | **46.6** | **39.8** | | + CriticalKV | 46.2 | 38.1 | 45.6 | 39.7 | | + ReST-KV | 45.5 | 36.6 | 45.0 | 36.8 | | + ProjCache-SelfDiag | **46.9** | **40.1** | 45.9 | 38.9 | | + ProjCache-SelfFull | **46.9** | 39.8 | 45.9 | 39.2 | | + ProjCache | **46.9** | 39.7 | 46.1 | 39.4 | On Qwen3 RULER-4K, ProjCache-SelfFull outperforms ReST-KV in four of six settings. Including cross-head interactions also helps: ProjCache outperforms ProjCache-SelfFull in all six settings (Table 2). **Table 2.** RULER-4K accuracy (%) on Qwen3-8B at a 10% KV budget, without compensation. Scores are averaged over 13 tasks. | Method | Q-Aware | Q-Agnostic | | --- | :---: | :---: | | H2O | | | | + DropKV | 42.4 | 17.8 | | + OBCache-VK | 35.4 | 17.4 | | + CriticalKV | 36.4 | 17.6 | | + ReST-KV | 45.9 | 18.6 | | + ProjCache-SelfFull | 48.8 | 18.3 | | + ProjCache | **49.3** | **19.3** | | SnapKV | | | | + DropKV | 43.4 | 18.1 | | + OBCache-VK | 38.8 | 16.9 | | + CriticalKV | **51.2** | 20.3 | | + ReST-KV | 50.5 | 20.1 | | + ProjCache-SelfFull | 47.7 | 20.2 | | + ProjCache | 48.7 | **21.3** | | AdaKV | | | | + DropKV | 60.3 | 25.9 | | + OBCache-VK | 57.1 | 23.8 | | + CriticalKV | 52.9 | 23.9 | | + ReST-KV | 57.0 | 24.9 | | + ProjCache-SelfFull | 62.8 | 27.2 | | + ProjCache | **63.5** | **27.9** |
AuthorsOpenReview ↗

Response continued

W2. CriticalKV on Q-Agnostic RULER-32K.

We suggest that the large gap can be explained by the query-independent projected value-norm term in CriticalKV's score implementation. We follow the notation in the CriticalKV paper. Its first stage reserves tokens using attention scores, and its second stage computes

\qquad \mathcal V={\boldsymbol{V}}{\boldsymbol{W}}_O,$$ $$\mathcal A = ({\boldsymbol{A}}+\epsilon)\odot\\|\mathcal V\\|_{1,\mathrm{row}} = {\boldsymbol{A}}\odot\\|\mathcal V\\|_{1,\mathrm{row}} +\epsilon\\|\mathcal V\\|_{1,\mathrm{row}}.$$ CriticalKV heuristically adds the query-independent term $\epsilon\\|\mathcal V\\|_{1,\mathrm{row}}$ in its algorithm and implementation. This term is absent from its derivation. Our code followed the default $\epsilon=10^{-4}$. We investigate its effect by rerunning CriticalKV w/ AdaKV with $\epsilon\in\\{10^{-4},10^{-7},0\\}$ on Qwen3-8B, using Q-Agnostic RULER-32K and a 20% KV budget (Table 3). We further test two variants: CriticalKV using only $\\|\mathcal V\\|_{1,\mathrm{row}}$ for second-stage ranking, and ProjCache adding $10^{-4}\\|\mathcal V\\|_{1,\mathrm{row}}$ to its original score. The two variants score 68.33 and 75.12, respectively. We find that this term has a large effect on performance. ProjCache outperforms CriticalKV both without it (41.0 vs. 37.5) and with it (75.12 vs. 68.0), achieving the highest score in Table 3. **Table 3.** Epsilon and value-norm ablations on Qwen3-8B (Q-Agnostic), using RULER-32K at a 20% KV budget. Scores are averaged over 13 tasks. | Method | $\epsilon$ | Average score | | --- | :---: | :---: | | CriticalKV w/ AdaKV | $10^{-4}$ | 68.00 | | CriticalKV w/ AdaKV | $10^{-7}$ | 37.60 | | CriticalKV w/ AdaKV | $0$ | 37.50 | | CriticalKV w/ AdaKV (norm-only) | – | 68.33 | | ProjCache w/ AdaKV | – | 41.00 | | ProjCache w/ AdaKV + value norm | $10^{-4}$ | 75.12 | At layers 0, 12, 24, and 35, respectively, 97.73%, 98.55%, 98.61%, and 98.98% of attention scores are below $10^{-4}$ (Table 4). When ${\boldsymbol{A}}\ll\epsilon$, the second-stage score is approximately $\epsilon\\|\mathcal V\\|_{1,\mathrm{row}}$, so selection depends mainly on the projected value norm. This could help explain the large effect of epsilon. **Table 4.** Attention scores below $10^{-4}$ on Qwen3-8B (Q-Agnostic), using AdaKV on RULER-32K at a 20% KV budget. Results are averaged over all RULER-32K samples; the observation window is excluded. | Layer | Attention scores below $10^{-4}$ | | --- | :---: | | 0 | 97.73% | | 12 | 98.55% | | 24 | 98.61% | | 35 | 98.98% |
AuthorsOpenReview ↗

Response continued

W3. Compensation quality and cost.

In the manuscript experiments, we set the damping parameter to λ=50\lambda=50. Compensation is skipped for 34.35% of KV heads over the full LongBench evaluation. Details can be found in Appendix C.3 (Table 25). We skip a head when the solve fails or returns negative or invalid weights.

We sample 50 examples from five LongBench tasks (10 per task) for each of the MSE and damping evaluations below.

Compensation reduces mean relative output MSE by 12.55% on answer queries excluded from fitting (Table 5). Specifically, we first prefill the context and question and use the observation queries for KV selection and compensation. Compensation keeps the eviction set fixed and adjusts only retained token masks. Second, we feed the same reference answer tokens using three caches: Full KV, ProjCache without compensation, and ProjCache with compensation. These answer tokens are not used for selection or compensation. Third, we measure the MSE of attention outputs after WO{\boldsymbol{W}}_O at answer positions in each layer, using Full KV as the reference. We report the relative MSE change, 100×(MSEcomp/MSEprune−1)100\times(\mathrm{MSE}_{\mathrm{comp}}/\mathrm{MSE}_{\mathrm{prune}}-1), averaged over examples and layers.

Table 5. Answer-token output MSE on Qwen3-8B (Q-Aware), with ProjCache + SnapKV, a 20% KV budget, and λ=50\lambda=50. Results are based on 10 examples per task.

TaskMean Δ\DeltaMSE (%)
Qasper−12.96-12.96
MF-en−15.45-15.45
HotpotQA−9.05-9.05
2WQA−21.30-21.30
MuSiQue−4.01-4.01
All 50 examples−12.55-12.55

We investigate the effect of damping values λ∈5,50,500\lambda\in\\{5,50,500\\} on per-KV-head MSE using Qwen3-8B, ProjCache + SnapKV, and a 20% KV budget. With Full-KV layer inputs and eviction sets fixed, we compute compensation mask weights using an observation window of 64 queries at the end of a 4K prefix. We report output MSE on both the observation window and the next 64 queries, using the same mask weights. The next 64 queries are excluded from mask computation. λ=50\lambda=50 gives the largest MSE reductions: 22.94% on the observation window and 13.27% on the next 64 queries (Table 6). Larger damping reduces skips but weakens compensation at λ=500\lambda=500. Heads with negative weights remain uncompensated.

Table 6. Damping sensitivity of OBS compensation on Qwen3-8B.

λ\lambdaObservation (%)Next 64 (%)Skipped (%)
5−20.48-20.48−9.06-9.0657.52
50−22.94-22.94−13.27-13.2718.74
500−5.03-5.03−3.26-3.264.35

We also measure the TTFT of compensation (Table 7). TTFT increases from 2.247 to 5.268 seconds at batch size 1 and from 9.123 to 17.826 seconds at batch size 4. Optimized GPU kernels could reduce this overhead. We leave this optimization for future work.

Table 7. TTFT and memory usage with and without compensation on LLaMA-3.1-8B with synthetic 32K prompts and a 10% KV budget, measured in BF16 on one H100.

MethodTTFT (s)Memory (GiB)
Batch size 1
ProjCache2.247±.0042.247\pm.00415.584
ProjCache + OBS5.268±.0015.268\pm.00115.793
Batch size 4
ProjCache9.123±.0089.123\pm.00816.994
ProjCache + OBS17.826±.02517.826\pm.02517.831
AuthorsOpenReview ↗

Response continued

W4/Q2. Model scale and hybrid or sparse attention.

ProjCache also improves scores on a 32B model and on a hybrid model with inputs longer than 32K tokens. On Qwen2.5-32B-Instruct, it averages 41.09 versus 39.41 for CriticalKV and scores higher on 13 of 16 LongBench tasks, using H2O and a 10% KV budget (Table 8).

Table 8. LongBench scores on Qwen2.5-32B-Instruct (Q-Aware) at a 10% KV budget with H2O, without compensation.

Single-Doc QA: NQA, Qasper, MF-en; Multi-Doc QA: HQA, 2WQA, MSQ; Summarization: GovR, QMSum, MNews; Few-shot: TREC, TQA, SAMSum; Synthetic: PCount, PRe; Code: LCC, RB-P.

MethodNQAQasperMF-enHQA2WQAMSQGovRQMSumMNewsTRECTQASAMSumPCountPReLCCRB-PAvg.
H2O27.6628.7138.5344.7737.9923.0025.2822.9318.7549.0084.7144.458.3191.0244.1636.2039.09
+ CriticalKV29.3628.5539.3348.7738.0823.1225.6223.0919.0048.5084.9943.997.6192.0243.2735.2139.41
+ ProjCache29.0131.7540.2451.0543.4725.5026.8423.0419.6953.5086.1844.718.9891.1843.6138.6941.09

Hybrid and sparse attention. On Qwen3.5-9B, ProjCache-SelfFull and ProjCache average 88.79 and 88.65 across 13 RULER-48K tasks, versus 88.01 for CriticalKV and 87.31 for H2O (Table 9). We compress KV only in the eight softmax-attention layers and leave the 24 Gated DeltaNet states unchanged.

Table 9. RULER-48K accuracy (%) on Qwen3.5-9B (Q-Aware) at a 5% KV budget with H2O, without compensation.

MethodCWEFWEMK-1MK-2MK-3MQMVS-1S-2S-3QA-1QA-2VTAvg.
H2O43.6884.80100.0076.0082.40100.0097.80100.00100.0096.0089.6064.80100.0087.31
+ CriticalKV48.7283.47100.0076.0086.40100.0099.20100.00100.0096.0089.6064.80100.0088.01
+ ProjCache-SelfFull53.6084.27100.0089.6087.20100.0095.60100.00100.0089.6089.6064.80100.0088.79
+ ProjCache52.0883.73100.0088.0087.20100.0095.80100.00100.0090.4089.6065.60100.0088.65

QSA retains historical KV, but each query attends only to a selected KV subset. We could score and evict individual KV pairs, evict a chosen fraction of entire KV blocks, or remove a fixed fraction of KV pairs within each block. With block selection fixed, all three can be represented by different mask constraints in the same reconstruction objective. The indexer would then score the compressed cache. We leave this adaptation and its evaluation for future work.

Larger GQA groups. We expect cross-head interactions to matter more when more query heads share each KV head. However, isolating the effect of GQA group size requires keeping other model characteristics fixed, which is difficult when comparing different models.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their comments. We summarize the main clarifications and additional experiments below.

1. Contribution and relation to prior work.

Our token-mask formulation connects Taylor-based eviction scoring with OBS-inspired compensation under a shared output-projected reconstruction objective. Summing all Taylor orders yields the exact single-pair eviction score, accounting for attention renormalization and GQA cross-head interactions. Our OBS-inspired compensation jointly adjusts retained token masks within each KV head using a regularized quadratic approximation of the same objective. The eviction set and cached keys and values stay fixed.

Compared with DropKV, our unprojected score recovers its exact deletion score for the same observation queries, providing a Taylor interpretation. We extend this analysis to reconstruction error after WO{\boldsymbol{W}}_O, including interactions between query heads sharing a KV head under GQA.

Compared with CriticalKV, our score measures the exact projected squared-L2L_2 error of deleting one KV pair. CriticalKV derives its score from an upper bound on each query head's projected L1L_1 error.

Compared with ReST-KV, our score includes interactions between query heads sharing a KV head under GQA.

We added ProjCache-SelfFull and unsmoothed ReST-KV comparisons on LongBench and Qwen3 RULER-4K to test cross-head terms and norm squaring separately.

2. Comparisons with stronger baselines.

Compared with strong baselines, ProjCache achieves the most wins on LLaMA-3.1-8B (manuscript Table 1). Our ablations show that CriticalKV's heuristically added query-independent projected value-norm term (default ϵ=10−4\epsilon=10^{-4}) strongly affects Q-Agnostic RULER-32K performance. This term is absent from its derivation. ProjCache outperforms CriticalKV both without this value-norm term (41.0 vs. 37.5) and with the term (75.12 vs. 68.0).

3. Compensation quality and cost.

We measure output MSE after OBS compensation on answer queries excluded from KV selection and mask fitting, and find a mean relative reduction of 12.55%. We also analyze damping sensitivity and skip rates, and measure the added TTFT and memory costs.

4. Model coverage and quantization.

We added evaluations on larger and hybrid models and tested packed INT4 storage for retained prefix KV, all using H2O in the Q-Aware setting. On Qwen2.5-32B-Instruct at a 10% KV budget, ProjCache scores 41.09 on LongBench versus 39.41 for CriticalKV, with higher scores on 13 of 16 tasks. On Qwen3.5-9B at a 5% KV budget, ProjCache-SelfFull and ProjCache score 88.79 and 88.65 on RULER-48K, versus 88.01 for CriticalKV. On LLaMA-3.1-8B at a 10% KV budget, ProjCache with packed INT4 prefix KV scores 44.76 on LongBench, compared with 45.14 for the unquantized result reported in the manuscript.

Further details and point-by-point responses are provided in our replies to each reviewer below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.