- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RestoreKV complements query-agnostic KV eviction with a small learned restore cache. After context prefill, eight restore tokens read the full cache through a LoRA-adapted pass. Their generated KV states replace an equal-sized portion of the retained cache. Training uses self-distillation from the full-cache model, while subsequent queries and decoding use the original frozen backbone.
Critical strengths
The method is intuitive and reports strong accuracy improvements under aggressive compression. Combining retained original states with a learned complement is a reasonable way to compensate for information lost through eviction. The improvements across multiple backbones and eviction methods, under matched KV budgets, support its practical potential.
Critical weaknesses
-
Training and deployment introduce additional infrastructure complexity. Although no per-context optimization is required, a separate restore generator must be trained for each backbone–evictor configuration. Deployment also requires managing these checkpoints, selectively activating LoRA during cache construction, and integrating the generated states into the compressed cache. This adds integration and maintenance requirements beyond a training-free eviction pipeline.
-
Sensitivity to training data and broader task generalization remain insufficiently characterized. The learned restoration mechanism may depend on the context and query distributions used for distillation. While transfer across several long-context benchmarks is encouraging, more systematic studies of training-data composition and broader general-purpose tasks are needed to establish its robustness.
Questions for authors
N/A
2 comments
Comment 2/2 for Reviewer 5eSj
Weakness 2: Sensitivity to training data and broader task generalization remain insufficiently characterized. The learned restoration mechanism may depend on the context and query distributions used for distillation. While transfer across several long-context benchmarks is encouraging, more systematic studies of training-data composition and broader general-purpose tasks are needed to establish its robustness.
Response: Thank you for this important comment. We varied the training-data composition, expanded evaluation to eight domains, and tested code-only training. Both the original and augmented training mixtures improve KVzip on all eight domains, while code-only training also retains gains on the four original benchmarks.
Training-data composition. Our original data comprise LongAlpaca, PG-19, and Tulu-3 Flan. Queries for LongAlpaca and PG-19 are teacher-generated from generic prompts (e.g., summarization, explanation), rather than tailored to the evaluation benchmarks. We augment this mixture with legal, finance, biomedical, and web data, generating queries in the same way. None of these four added datasets is used as an evaluation benchmark.
| Group | Domain | Dataset | Samples | Avg. length |
|---|---|---|---|---|
| Orig. train | Long documents | LongAlpaca | 2.5k | 7.1k |
| Books | PG-19 | 2.2k | 3.1k | |
| General | Tulu-3 Flan | 1.5k | 0.9k | |
| New train | Legal | The Case-law [1] | 0.5k | 9.4k |
| Finance | EDGAR-CORPUS [2] | 0.5k | 10.0k | |
| Web | FineWeb-Edu [3] | 0.5k | 6.8k | |
| Biomedical | PubMed Central OA [4] | 0.5k | 10.4k |
Lengths are in tokens. Both runs use 5,000 updates with the same batch size: the number of processed examples is fixed, while the data pools differ (approximately 6.2K vs. 8.2K examples).
Broader evaluation. Using Llama-3.1-8B-Instruct with KVzip at r=0.05, we evaluate on eight domains: the original medical and academic benchmarks, plus Wikipedia, legal, finance, novels, meetings, and code. We compare generators trained on the original mixture (orig.) and the augmented mixture (orig.+new).
| Wikipedia | Legal | Finance | Novel | Medical | Academic | Meeting | Code | |
|---|---|---|---|---|---|---|---|---|
| Benchmark | HELMET NQ [5] | CUAD [6] | SECQUE [7] | ∞Bench En.MC [8] | LongHealth | QASPER | QMSum [9] | LCC [10] |
| Full KV | 50.5 | 37.6 | 48.4 | 57.2 | 81.3 | 45.7 | 23.2 | 34.1 |
| KVzip | 38.4 | 27.6 | 24.9 | 49.8 | 67.8 | 26.0 | 20.4 | 22.9 |
| RestoreKV (orig.) | 39.2 | 31.2 | 32.1 | 53.3 | 76.5 | 35.5 | 22.7 | 27.7 |
| RestoreKV (orig.+new) | 40.8 | 32.0 | 31.4 | 53.3 | 77.0 | 36.5 | 22.3 | 27.1 |
Bold compares the two RestoreKV variants.
Both variants outperform KVzip on all eight domains. Augmentation raises the Wikipedia score by 1.6 points; changes on the other seven domains are within 1.0 point, with both increases and decreases. Under this fixed update budget, adding domain coverage has a modest effect, while the original mixture already provides gains across the evaluated domains.
Code-only training. Following Reviewer cgWs's suggestion, we separately trained a Llama-3.1-8B-Instruct generator only on code data (LCC, RepoBench-P, RepoQA) and evaluated it on the four original general-domain benchmarks.
| r=0.05 | RULER-4K | QASPER | QuALITY | LongHealth |
|---|---|---|---|---|
| KVzip | 63.1 | 26.0 | 58.6 | 67.8 |
| RestoreKV (code only) | 73.5 | 33.0 | 63.0 | 76.3 |
| RestoreKV (orig.) | 81.1 | 35.5 | 63.2 | 76.5 |
Code-only training improves KVzip by 10.4/7.0/4.4/8.5 points, respectively. The original mixture remains stronger at this budget, particularly on RULER (81.1 vs. 73.5), showing that the training distribution still matters. Nevertheless, restoration remains useful under this code-to-general shift. Results across budgets and the reverse transfer direction are in our response to Reviewer cgWs (W1).
Overall, the original mixture is effective across these eight domains, and useful gains persist with code-only training. These results support robustness to the tested changes in training composition and domain, rather than independence from the training distribution. We have added the results and analysis to Appendix A.4.
References:
[1] Naudet and Dolan. The Case-law. Hugging Face, 2024.
[2] Loukas et al. EDGAR-CORPUS. ECONLP 2021.
[3] Penedo et al. The FineWeb Datasets. NeurIPS D&B 2024.
[4] National Library of Medicine. PMC Open Access Subset. 2003–.
[5] Yen et al. HELMET. ICLR 2025.
[6] Hendrycks et al. CUAD. NeurIPS D&B 2021.
[7] BenYoash et al. SECQUE. GEM² Workshop at ACL 2025.
[8] Zhang et al. ∞Bench. ACL 2024.
[9] Zhong et al. QMSum. NAACL 2021.
[10] Guo et al. LongCoder. ICML 2023.
Comment 1/2 for Reviewer 5eSj
We thank the reviewer for the careful reading and thoughtful comments.
Weakness 1: Training and deployment introduce additional infrastructure complexity. Although no per-context optimization is required, a separate restore generator must be trained for each backbone–evictor configuration. Deployment also requires managing these checkpoints, selectively activating LoRA during cache construction, and integrating the generated states into the compressed cache. This adds integration and maintenance requirements beyond a training-free eviction pipeline.
Response: Thank you for raising this practical concern. We agree that RestoreKV adds training and checkpoint management beyond training-free eviction. We quantify generator reuse and training costs and provide an integration example.
(1) Checkpoint management. We normally train a generator per backbone–evictor configuration. We test reuse of the same generator trained with KVzip [3], without retraining, on six other evictors (Qwen3-4B, RULER-4K). Each group compares the base evictor, + RestoreKV with the shared KVzip-trained generator, and + RestoreKV with an evictor-specific generator.
All KVzip-trained rows reuse the same generator. Other RestoreKV rows use evictor-specific generators. Bold marks the highest score within each evictor group and budget.
The transferred generator improves the base evictor in all 18 settings. For KVgrad, EchoPress, KVzip+, and ContrastKV, it is close to or better than evictor-specific training; the largest deficit on KVgrad is 0.9 points. Thus, these results show that one generator can be reused across multiple evictors within a backbone.
For SnapKV and H2O, transfer still improves the baselines, but evictor-specific training yields larger gains. Reuse therefore depends on the inference evictor; we do not claim a universal generator. We clarified this limitation in Sec. 5 and expanded Appendix A.3 (A.4 in the original submission).
(2) Training cost. The Qwen3-4B generator has 16.5M trainable parameters (0.4% of the backbone). For Llama-3.1-8B-Instruct with KVzip, Table 7 reports approximately 1.5 hours on one RTX PRO 6000 and 22.9 GB peak memory, only 0.1 GB above KVzip inference at 15K tokens. Long-context prefill and importance scoring run without gradients. These measurements apply to this configuration, not every backbone–evictor pair.
(3) Integration and deployment. Our new supplementary
rebuttal_restorekv_kvpressexample subclassesKVzipPressin KVPress.These results reduce, but do not eliminate, the additional training and maintenance burden.
References:
[1] Kwak et al. KVgrad. ICML Workshop (AdaptFM) 2026.
[2] Lin et al. EchoPress. arXiv:2610.00412.
[3] Kim et al. KVzip. NeurIPS 2025.
[4] Jegou and Jeblick. KVzap. arXiv:2601.07891.
[5] Chen et al. ContrastKV. ACL 2026.
[6] Li et al. SnapKV. NeurIPS 2024.
[7] Zhang et al. H2O. NeurIPS 2023.