- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ProjCache unifies KV cache eviction and compensation under a single layer-wise objective—the reconstruction error measured after the attention output projection—parameterizing both as token masks on attention weights. For eviction, an exact single-pair score is derived from the full Taylor expansion of the objective (capturing attention renormalization and GQA cross-head terms), and for compensation, Optimal Brain Surgeon is adapted to jointly optimize the masks of retained tokens.
Critical strengths
The theoretical derivations in this paper are thorough and well-grounded.
Critical weaknesses
-
The method clearly beats earlier baselines such as H2O and SnapKV. Against stronger, more recent methods like CriticalKV and OBCache, however, the improvements are marginal.
-
The algorithm is more complex than existing methods, which likely adds computational overhead. The paper doesn't measure this. I'd suggest reporting latency, throughput, and memory usage against the baselines.
-
Quantization is now widely used for KV caches. Showing that the method works alongside it would make the work more practical.
Questions for authors
See weaknesses.
8 comments
Response continued
We suggest that the large gap between CriticalKV and ProjCache in some Q-Agnostic settings can be explained by the query-independent projected value-norm term in CriticalKV's score implementation. We follow the notation in the CriticalKV paper. Its first stage reserves tokens using attention scores, and its second stage computes
\qquad \mathcal V={\boldsymbol{V}}{\boldsymbol{W}}_O,$$ $$\mathcal A = ({\boldsymbol{A}}+\epsilon)\odot\\|\mathcal V\\|_{1,\mathrm{row}} = {\boldsymbol{A}}\odot\\|\mathcal V\\|_{1,\mathrm{row}} +\epsilon\\|\mathcal V\\|_{1,\mathrm{row}}.$$ CriticalKV heuristically adds the query-independent term $\epsilon\\|\mathcal V\\|_{1,\mathrm{row}}$ in its algorithm and implementation. This term is absent from its derivation. Our code followed the default $\epsilon=10^{-4}$. We investigate its effect by rerunning CriticalKV w/ AdaKV with $\epsilon\in\\{10^{-4},10^{-7},0\\}$ on Qwen3-8B, using Q-Agnostic RULER-32K and a 20% KV budget (Table 4). We further test two variants: CriticalKV using only $\\|\mathcal V\\|_{1,\mathrm{row}}$ for second-stage ranking, and ProjCache adding $10^{-4}\\|\mathcal V\\|_{1,\mathrm{row}}$ to its original score. The two variants score 68.33 and 75.12, respectively. We find that this term has a large effect on performance. ProjCache outperforms CriticalKV both without it (41.0 vs. 37.5) and with it (75.12 vs. 68.0), achieving the highest score in Table 4. **Table 4.** Epsilon and value-norm ablations on Qwen3-8B (Q-Agnostic), using RULER-32K at a 20% KV budget. Scores are averaged over 13 tasks. | Method | $\epsilon$ | Average score | | --- | :---: | :---: | | CriticalKV w/ AdaKV | $10^{-4}$ | 68.00 | | CriticalKV w/ AdaKV | $10^{-7}$ | 37.60 | | CriticalKV w/ AdaKV | $0$ | 37.50 | | CriticalKV w/ AdaKV (norm-only) | – | 68.33 | | ProjCache w/ AdaKV | – | 41.00 | | ProjCache w/ AdaKV + value norm | $10^{-4}$ | 75.12 | At layers 0, 12, 24, and 35, respectively, 97.73%, 98.55%, 98.61%, and 98.98% of attention scores are below $10^{-4}$ (Table 5). When ${\boldsymbol{A}}\ll\epsilon$, the second-stage score is approximately $\epsilon\\|\mathcal V\\|_{1,\mathrm{row}}$, so selection depends mainly on the projected value norm. This could help explain the large effect of epsilon. **Table 5.** Attention scores below $10^{-4}$ on Qwen3-8B (Q-Agnostic), using AdaKV on RULER-32K at a 20% KV budget. Results are averaged over all RULER-32K samples; the observation window is excluded. | Layer | Attention scores below $10^{-4}$ | | --- | :---: | | 0 | 97.73% | | 12 | 98.55% | | 24 | 98.61% | | 35 | 98.98% | ## W2. Latency, throughput, and memory. Without compensation, ProjCache has similar decoding latency and memory use to OBCache-VK and CriticalKV (Appendix B.1, Figures 4–5). For example, at 96K and batch size one, ProjCache and CriticalKV have decoding latencies of 14.388 and 14.337 ms/token and peak decoding memory of 16.397 and 16.272 GiB, respectively. TTFT is slightly higher in this setting, at 11.534 versus 11.081 s. The appendix covers 24K–96K inputs and batch sizes one and four on LLaMA-3.1-8B, using one H100 in BF16. Appendix C.3 reports the additional TTFT and memory costs of compensation, including Hessian construction and the solve. Appendix B.1 (manuscript Table 4) also shows that ProjCache has scoring complexity comparable to CriticalKV. With the required Gram blocks precomputed from ${\boldsymbol{W}}_O$, ProjCache computes scores in the KV-group dimension, while both methods scale linearly with cache length. We also measure the TTFT of compensation (Table 6). TTFT increases from 2.247 to 5.268 seconds at batch size 1 and from 9.123 to 17.826 seconds at batch size 4. Optimized GPU kernels could reduce this overhead. We leave this optimization for future work. **Table 6.** TTFT and memory usage with and without compensation on LLaMA-3.1-8B with synthetic 32K prompts and a 10% KV budget, measured in BF16 on one H100. | Method | TTFT (s) | Memory (GiB) | | --- | :---: | :---: | | *Batch size 1* | | | | ProjCache | $2.247\pm.004$ | 15.584 | | ProjCache + OBS | $5.268\pm.001$ | 15.793 | | *Batch size 4* | | | | ProjCache | $9.123\pm.008$ | 16.994 | | ProjCache + OBS | $17.826\pm.025$ | 17.831 |Response continued
W3. Compatibility with KV-cache quantization.
Following ReST-KV (An et al., 2026), we apply grouped 4-bit quantization with KVQuant-style outlier handling (Hooper et al., 2024). ProjCache with packed 4-bit KV scores 44.76 on LongBench, versus 45.14 for the unquantized manuscript reference (Table 7). This evaluation covers all 16 tasks (3,750 examples) on LongBench, using LLaMA-3.1-8B with H2O in the Q-Aware setting with a 10% KV budget and a window size of 64.
Table 7. LongBench scores of ProjCache with unquantized and packed 4-bit KV on LLaMA-3.1-8B (Q-Aware) at a 10% KV budget with H2O, without compensation. Scores are averaged over 16 tasks.
Single-Doc QA: NQA, Qasper, MF-en; Multi-Doc QA: HQA, 2WQA, MSQ; Summarization: GovR, QMSum, MNews; Few-shot: TREC, TQA, SAMSum; Synthetic: PCount, PRe; Code: LCC, RB-P.
| Method | NQA | Qasper | MF-en | HQA | 2WQA | MSQ | GovR | QMSum | MNews | TREC | TQA | SAMSum | PCount | PRe | LCC | RB-P | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ProjCache | 29.92 | 34.09 | 52.50 | 56.00 | 48.32 | 31.18 | 27.07 | 24.34 | 21.10 | 48.50 | 89.99 | 43.59 | 12.62 | 100.00 | 50.44 | 52.61 | 45.14 |
| + INT4 KV | 30.27 | 35.29 | 52.67 | 55.11 | 48.58 | 28.34 | 26.89 | 23.99 | 20.88 | 48.50 | 90.04 | 42.51 | 11.22 | 100.00 | 50.46 | 51.36 | 44.76 |
We leave the joint treatment of quantization and eviction within our objective for future work.
References
-
Yongqi An, Chang Lu, Kuan Zhu, Tao Yu, Chaoyang Zhao, Hong Wu, Ming Tang, and Jinqiao Wang. ReST-KV: Robust KV cache eviction with layer-wise output reconstruction and spatial-temporal smoothing. In International Conference on Learning Representations (ICLR), 2026. URL OpenReview.
-
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024. URL NeurIPS proceedings.
Author response to Reviewer Vnfo
Thank you for your comments.
W1. Comparisons with stronger baselines.
We have compared ProjCache with recent methods, including CriticalKV and OBCache (both ICML 2026). In manuscript Table 1, ProjCache achieves the best score in 15 of 36 settings on LLaMA-3.1-8B, compared with 12 for CriticalKV, 8 for OBCache-VK, and 3 for DropKV. We also added comparisons with ReST-KV (An et al., 2026). Results are shown below in Table 1. Here, ProjCache denotes ProjCache-AllFull, and ReST-KV denotes its score without smoothing. On Qwen3-8B RULER-4K, ProjCache outperforms ReST-KV in five of six settings (Table 1).
Table 1. RULER-4K accuracy (%) on Qwen3-8B at a 10% KV budget, without compensation. Scores are averaged over 13 tasks.
We added experiments on a larger model, Qwen2.5-32B-Instruct. ProjCache averages 41.09 across all 16 LongBench tasks, versus 39.41 for CriticalKV, and scores higher on 13 tasks. Both use H2O in the Q-Aware setting with a 10% KV budget (Table 2).
Table 2. LongBench scores on Qwen2.5-32B-Instruct (Q-Aware) at a 10% KV budget with H2O, without compensation.
Single-Doc QA: NQA, Qasper, MF-en; Multi-Doc QA: HQA, 2WQA, MSQ; Summarization: GovR, QMSum, MNews; Few-shot: TREC, TQA, SAMSum; Synthetic: PCount, PRe; Code: LCC, RB-P.
We also added experiments on Qwen3.5-9B, which uses output gates. ProjCache-SelfFull and ProjCache average 88.79 and 88.65 across all 13 RULER-48K tasks, versus 88.01 for CriticalKV and 87.31 for H2O. These experiments use H2O in the Q-Aware setting with a 5% KV budget (Table 3). We compress KV only in the eight softmax-attention layers and leave the 24 Gated DeltaNet states unchanged.
Table 3. RULER-48K accuracy (%) on Qwen3.5-9B (Q-Aware) at a 5% KV budget with H2O, without compensation.