acceptodds
Under review as a conference paper at ICLR 2027

RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction

16 views

Abstract

Query-agnostic KV cache eviction compresses a context once and reuses the resulting cache for arbitrary future queries, but performance can collapse under tight budgets. Existing methods primarily improve which original KV pairs are retained. We introduce RestoreKV, which complements this selection-based formulation with learned restoration under the same total KV budget. The restore generator is trained to complement retained original KV states through predictions from their combined cache, rather than to replace the entire context. After context prefill, a few restore tokens attend to the full KV cache in a single LoRA-adapted pass, generating a compact, context-conditioned restore cache. The base importance scorer and eviction rule remain unchanged, and the adapters are disabled for all subsequent queries and decoding. RestoreKV is trained through parameter-efficient self-distillation from the frozen full-cache model, optimizing only 0.4% of the parameters and requiring no task-specific tuning. Across four backbones and four long-context benchmarks, RestoreKV substantially reduces compression-induced degradation. On Qwen3-4B, it improves 59 of 60 paired, budget-matched settings across five base eviction methods; at a 5% budget, it raises KVzip from 38.2 to 73.2 on RULER-4K. On Qwen3-8B, applying RestoreKV to KVzip+ reaches 86.4 RULER accuracy at 16× compression on the KVPress Benchmark, while adding at most 0.5% one-time cache construction overhead in a 32K-context evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
KV Cache Eviction, Inference Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5eSjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RestoreKV complements query-agnostic KV eviction with a small learned restore cache. After context prefill, eight restore tokens read the full cache through a LoRA-adapted pass. Their generated KV states replace an equal-sized portion of the retained cache. Training uses self-distillation from the full-cache model, while subsequent queries and decoding use the original frozen backbone.

Critical strengths

The method is intuitive and reports strong accuracy improvements under aggressive compression. Combining retained original states with a learned complement is a reasonable way to compensate for information lost through eviction. The improvements across multiple backbones and eviction methods, under matched KV budgets, support its practical potential.

Critical weaknesses
  1. Training and deployment introduce additional infrastructure complexity. Although no per-context optimization is required, a separate restore generator must be trained for each backbone–evictor configuration. Deployment also requires managing these checkpoints, selectively activating LoRA during cache construction, and integrating the generated states into the compressed cache. This adds integration and maintenance requirements beyond a training-free eviction pipeline.

  2. Sensitivity to training data and broader task generalization remain insufficiently characterized. The learned restoration mechanism may depend on the context and query distributions used for distillation. While transfer across several long-context benchmarks is encouraging, more systematic studies of training-data composition and broader general-purpose tasks are needed to establish its robustness.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Comment 1/2 for Reviewer 5eSj

We thank the reviewer for the careful reading and thoughtful comments.

Weakness 1: Training and deployment introduce additional infrastructure complexity. Although no per-context optimization is required, a separate restore generator must be trained for each backbone–evictor configuration. Deployment also requires managing these checkpoints, selectively activating LoRA during cache construction, and integrating the generated states into the compressed cache. This adds integration and maintenance requirements beyond a training-free eviction pipeline.

Response: Thank you for raising this practical concern. We agree that RestoreKV adds training and checkpoint management beyond training-free eviction. We quantify generator reuse and training costs and provide an integration example.

(1) Checkpoint management. We normally train a generator per backbone–evictor configuration. We test reuse of the same generator trained with KVzip [3], without retraining, on six other evictors (Qwen3-4B, RULER-4K). Each group compares the base evictor, + RestoreKV with the shared KVzip-trained generator, and + RestoreKV with an evictor-specific generator.

Inference evictorMethodr=0.2r=0.1r=0.05
KVgrad [1] (ICMLw 2026)Baseline92.385.567.1
+ RestoreKV (KVzip-trained)93.588.672.9
+ RestoreKV (KVgrad-trained)93.289.073.8
EchoPress [2] (arXiv 2026.10)Baseline91.479.845.3
+ RestoreKV (KVzip-trained)93.188.175.1
+ RestoreKV (EchoPress-trained)92.786.971.2
KVzip+ [4] (arXiv 2026.01)Baseline91.384.851.6
+ RestoreKV (KVzip-trained)93.690.473.2
+ RestoreKV (KVzip+-trained)93.389.770.7
ContrastKV [5] (ACL 2026)Baseline91.680.738.0
+ RestoreKV (KVzip-trained)92.585.642.8
+ RestoreKV (ContrastKV-trained)92.384.340.2
SnapKV [6] (NeurIPS 2024)Baseline29.719.212.8
+ RestoreKV (KVzip-trained)34.821.713.5
+ RestoreKV (SnapKV-trained)38.625.914.2
H2O [7] (NeurIPS 2023)Baseline8.03.53.2
+ RestoreKV (KVzip-trained)12.18.24.9
+ RestoreKV (H2O-trained)17.311.65.7

All KVzip-trained rows reuse the same generator. Other RestoreKV rows use evictor-specific generators. Bold marks the highest score within each evictor group and budget.

The transferred generator improves the base evictor in all 18 settings. For KVgrad, EchoPress, KVzip+, and ContrastKV, it is close to or better than evictor-specific training; the largest deficit on KVgrad is 0.9 points. Thus, these results show that one generator can be reused across multiple evictors within a backbone.

For SnapKV and H2O, transfer still improves the baselines, but evictor-specific training yields larger gains. Reuse therefore depends on the inference evictor; we do not claim a universal generator. We clarified this limitation in Sec. 5 and expanded Appendix A.3 (A.4 in the original submission).

(2) Training cost. The Qwen3-4B generator has 16.5M trainable parameters (0.4% of the backbone). For Llama-3.1-8B-Instruct with KVzip, Table 7 reports approximately 1.5 hours on one RTX PRO 6000 and 22.9 GB peak memory, only 0.1 GB above KVzip inference at 15K tokens. Long-context prefill and importance scoring run without gradients. These measurements apply to this configuration, not every backbone–evictor pair.

(3) Integration and deployment. Our new supplementary rebuttal_restorekv_kvpress example subclasses KVzipPress in KVPress.

  • Implementation: about 130 lines in one file, 8 lines of changes to the library core, and three overridden hooks, integrated with the existing compression and generation pipeline.
  • Checkpoints: one bundle per trained generator, containing a PEFT adapter and restore embeddings, loaded with PEFT and safetensors.
  • LoRA: adapter weights remain resident and are enabled only for the restore pass; prefill, query processing, and decoding use the original model.
  • Attention: restore states are ordinary KV pairs, requiring no custom attention kernel or bias.

These results reduce, but do not eliminate, the additional training and maintenance burden.

References:

[1] Kwak et al. KVgrad. ICML Workshop (AdaptFM) 2026.

[2] Lin et al. EchoPress. arXiv:2610.00412.

[3] Kim et al. KVzip. NeurIPS 2025.

[4] Jegou and Jeblick. KVzap. arXiv:2601.07891.

[5] Chen et al. ContrastKV. ACL 2026.

[6] Li et al. SnapKV. NeurIPS 2024.

[7] Zhang et al. H2O. NeurIPS 2023.

AuthorsOpenReview ↗

Comment 2/2 for Reviewer 5eSj

Weakness 2: Sensitivity to training data and broader task generalization remain insufficiently characterized. The learned restoration mechanism may depend on the context and query distributions used for distillation. While transfer across several long-context benchmarks is encouraging, more systematic studies of training-data composition and broader general-purpose tasks are needed to establish its robustness.

Response: Thank you for this important comment. We varied the training-data composition, expanded evaluation to eight domains, and tested code-only training. Both the original and augmented training mixtures improve KVzip on all eight domains, while code-only training also retains gains on the four original benchmarks.

Training-data composition. Our original data comprise LongAlpaca, PG-19, and Tulu-3 Flan. Queries for LongAlpaca and PG-19 are teacher-generated from generic prompts (e.g., summarization, explanation), rather than tailored to the evaluation benchmarks. We augment this mixture with legal, finance, biomedical, and web data, generating queries in the same way. None of these four added datasets is used as an evaluation benchmark.

GroupDomainDatasetSamplesAvg. length
Orig. trainLong documentsLongAlpaca2.5k7.1k
BooksPG-192.2k3.1k
GeneralTulu-3 Flan1.5k0.9k
New trainLegalThe Case-law [1]0.5k9.4k
FinanceEDGAR-CORPUS [2]0.5k10.0k
WebFineWeb-Edu [3]0.5k6.8k
BiomedicalPubMed Central OA [4]0.5k10.4k

Lengths are in tokens. Both runs use 5,000 updates with the same batch size: the number of processed examples is fixed, while the data pools differ (approximately 6.2K vs. 8.2K examples).

Broader evaluation. Using Llama-3.1-8B-Instruct with KVzip at r=0.05, we evaluate on eight domains: the original medical and academic benchmarks, plus Wikipedia, legal, finance, novels, meetings, and code. We compare generators trained on the original mixture (orig.) and the augmented mixture (orig.+new).

WikipediaLegalFinanceNovelMedicalAcademicMeetingCode
BenchmarkHELMET NQ [5]CUAD [6]SECQUE [7]∞Bench En.MC [8]LongHealthQASPERQMSum [9]LCC [10]
Full KV50.537.648.457.281.345.723.234.1
KVzip38.427.624.949.867.826.020.422.9
RestoreKV (orig.)39.231.232.153.376.535.522.727.7
RestoreKV (orig.+new)40.832.031.453.377.036.522.327.1

Bold compares the two RestoreKV variants.

Both variants outperform KVzip on all eight domains. Augmentation raises the Wikipedia score by 1.6 points; changes on the other seven domains are within 1.0 point, with both increases and decreases. Under this fixed update budget, adding domain coverage has a modest effect, while the original mixture already provides gains across the evaluated domains.

Code-only training. Following Reviewer cgWs's suggestion, we separately trained a Llama-3.1-8B-Instruct generator only on code data (LCC, RepoBench-P, RepoQA) and evaluated it on the four original general-domain benchmarks.

r=0.05RULER-4KQASPERQuALITYLongHealth
KVzip63.126.058.667.8
RestoreKV (code only)73.533.063.076.3
RestoreKV (orig.)81.135.563.276.5

Code-only training improves KVzip by 10.4/7.0/4.4/8.5 points, respectively. The original mixture remains stronger at this budget, particularly on RULER (81.1 vs. 73.5), showing that the training distribution still matters. Nevertheless, restoration remains useful under this code-to-general shift. Results across budgets and the reverse transfer direction are in our response to Reviewer cgWs (W1).

Overall, the original mixture is effective across these eight domains, and useful gains persist with code-only training. These results support robustness to the tested changes in training composition and domain, rather than independence from the training distribution. We have added the results and analysis to Appendix A.4.

References:

[1] Naudet and Dolan. The Case-law. Hugging Face, 2024.

[2] Loukas et al. EDGAR-CORPUS. ECONLP 2021.

[3] Penedo et al. The FineWeb Datasets. NeurIPS D&B 2024.

[4] National Library of Medicine. PMC Open Access Subset. 2003–.

[5] Yen et al. HELMET. ICLR 2025.

[6] Hendrycks et al. CUAD. NeurIPS D&B 2021.

[7] BenYoash et al. SECQUE. GEM² Workshop at ACL 2025.

[8] Zhang et al. ∞Bench. ACL 2024.

[9] Zhong et al. QMSum. NAACL 2021.

[10] Guo et al. LongCoder. ICML 2023.

Reviewer_cgWsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

For query-agnostic KV-cache eviction, the paper introduces a learnable cache that complements the KV states retained by existing selection-based eviction methods. The restore generator is trained to compensate for useful evicted information by jointly using the retained original KV states and the learnable cache, with parameter-efficient self-distillation from a frozen full-cache model. Since the underlying importance scorer and eviction rule remain unchanged, the proposed method can be integrated with a variety of KV-cache eviction methods. The paper demonstrates strong performance across different eviction strategies, backbone models, compression ratios, and benchmarks.

Critical strengths
  1. The paper addresses an timely and important issue by introducing a post-hoc method that complements existing KV-cache eviction methods with a learnable restoration mechanism. The core idea of generating a restoration cache using a LoRA-tuned model is simple yet effective. Moreover, the plug-in nature of RestoreKV (can be combined with various KV-cache eviction methods) is a significant advantage, with reasonable additional overhead in both training and inference.

  2. RestoreKV shows consistently strong performance across a wide range of benchmarks and experimental settings, achieving substantial gains over existing methods.

  3. The experimental section is well organized and comprehensive. The paper includes useful ablation studies and analyses covering LoRA configurations, the number of restoration tokens, performance at long context lengths (up to 128K tokens), and computational efficiency. The generalization experiments reported in Tables G and H are also well conducted and provide further evidence of the method's effectiveness.

  4. The paper is generally well written and easy to follow.

Critical weaknesses
  1. Generalization under Domain Shift. Given the results and practicality, the generalization ability of RestoreKV is the main point, particularly the robustness of the trained restoration generator under distribution shifts. Although Tables G and H provide some evidence of generalization, it would be interesting to evaluate whether the method remains effective under more substantial domain shifts, such as training the generator on code data and evaluating it on general-language tasks, and vice versa.

  2. In Table 1, the performance gains seem to vary substantially across benchmarks and settings. For example, the improvement at a compression ratio of (r=0.05) on RULER seems notably different from those on other benchmarks. It would be great if some explanations are added to this.

  3. The KV-Distill method explained in [Lines 139–141] seems closely related to RestoreKV. It would be helpful to clarify the key differences between the two approaches, and to include an empirical comparison if feasible.

  4. Several figures seems empty or missing plots

Questions for authors

Wrote above

3 comments
AuthorsOpenReview ↗

Comment 1/3 for Reviewer cgWs

We thank the reviewer for the careful reading and thoughtful comments.

Weakness 1: Generalization under Domain Shift. Given the results and practicality, the generalization ability of RestoreKV is the main point, particularly the robustness of the trained restoration generator under distribution shifts. Although Tables G and H provide some evidence of generalization, it would be interesting to evaluate whether the method remains effective under more substantial domain shifts, such as training the generator on code data and evaluating it on general-language tasks, and vice versa.

Response: Thank you for the helpful suggestion. We evaluated both transfer directions using Llama-3.1-8B-Instruct with KVzip. The two directions use separately trained restore generators: the general-trained generator is evaluated on code, whereas the code-trained generator is evaluated only on the four general-domain benchmarks. RestoreKV improves KVzip in 8 of 9 benchmark–budget settings for General → Code and all 12 settings for Code → General.

GroupDomainDatasetSource / evaluation task
Orig. train (general)Long documentsLongAlpacaLong-context instruction following
BooksPG-19Language modeling
GeneralTulu-3 FlanInstruction following
Orig. testSyntheticRULERLong-context retrieval, tracing, aggregation, and QA
ScienceQASPERQA over a paper
StoriesQuALITYReading comprehension
ClinicalLongHealthQA over patient records
CodeCodeLCC (LongBench) [1]Next-line code completion
CodeRepoBench-P (LongBench) [1]Repository-level code completion
CodeRepoQA (SCBench) [2]Function retrieval from a repository

The task column describes the source datasets or evaluation tasks; RestoreKV is trained through full-cache self-distillation.

(1) General → Code. We apply RestoreKV trained on our original three general-domain datasets to the three code benchmarks.

Test setMethodr=1.0r=0.2r=0.1r=0.05
LCCKVzip34.137.331.622.9
+ Ours–37.131.727.7
RepoBench-PKVzip45.646.645.746.1
+ Ours–48.147.446.3
RepoQAKVzip51.020.014.02.0
+ Ours–33.021.04.0

RestoreKV improves KVzip in 8 of 9 settings. The only decrease is 0.2 points on LCC at r=0.2 (37.3→37.1). The largest gains are +4.8 on LCC at r=0.05 and +13.0 on RepoQA at r=0.2.

(2) Code → General. We train a separate generator only on LCC, RepoBench-P, and RepoQA, then evaluate it on our four original general-domain benchmarks. The original general-trained generator is included as a reference.

Test setMethodr=1.0r=0.2r=0.1r=0.05
RULER-4KKVzip95.595.290.363.1
+ Ours (code-trained)–95.492.573.5
+ Ours (general-trained)–95.893.481.1
QASPERKVzip45.745.138.926.0
+ Ours (code-trained)–45.639.833.0
+ Ours (general-trained)–47.045.235.5
QuALITYKVzip76.069.967.258.6
+ Ours (code-trained)–70.668.163.0
+ Ours (general-trained)–70.468.463.2
LongHealthKVzip81.378.574.567.8
+ Ours (code-trained)–80.078.876.3
+ Ours (general-trained)–80.879.876.5

r=1.0 denotes the uncompressed full-cache reference.

The code-trained generator improves KVzip in all 12 settings. At r=0.05, the gains are +10.4/+7.0/+4.4/+8.5 points on RULER, QASPER, QuALITY, and LongHealth. Gains are generally smaller than with the original general-domain training mixture, indicating that the training distribution still matters. Nevertheless, useful restoration transfers to these benchmarks even when the generator is trained only on code.

We also added legal, finance, biomedical, and web data to the training mixture and evaluated on eight domains. Both the original and augmented mixtures outperform KVzip in all eight domains at r=0.05. Full results are in Appendix A.4 and our response to Reviewer 5eSj (W2).

Overall, these experiments support transfer across the tested code/general-domain shifts: although the training distribution affects absolute performance, the restoration benefit persists. We have added these results to Appendix A.4 of the revision.

References:

[1] Bai et al. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. ACL 2024.

[2] Li et al. SCBench: A KV Cache-Centric Analysis of Long-Context Methods. ICLR 2025.

AuthorsOpenReview ↗

Comment 2/3 for Reviewer cgWs

Weakness 2: In Table 1, the performance gains seem to vary substantially across benchmarks and settings. For example, the improvement at a compression ratio of (r=0.05) on RULER seems notably different from those on other benchmarks. It would be great if some explanations are added to this.

Response: Thank you for pointing this out. The particularly large gain on RULER at r=0.05 is associated with two observations in our results.

(1) Eviction causes a much larger drop on RULER. In Table 1, going from r=0.1 to 0.05, KVzip drops by 41.9 points on RULER but by 9.0–15.9 points on the other benchmarks. This larger baseline degradation leaves more room for recovery on RULER.

Qwen3-4BRULER-4KQASPERQuALITYLongHealth
KVzip (r=0.1)80.133.958.570.0
KVzip (r=0.05)38.218.046.361.0
KVzip + ours (r=0.05)73.228.752.467.5

(2) The RULER gain is concentrated in retrieval tasks. At r=0.05, the average improvement on the eight NIAH retrieval tasks is 50.1 points. QA and aggregation tasks show smaller gains (+12.8 and +14.2), while the tracing task (vt) is near ceiling at 99.0 for both methods.

RULER-4K (Qwen3-4B)Retrieval (NIAH, 8 tasks)QA (qa_1, qa_2)Aggregation (cwe, fwe)Tracing (vt)Avg. (13 tasks)
KVzip (r=0.05)34.227.734.599.038.2
KVzip + ours (r=0.05)84.340.548.799.073.2

Retrieval (NIAH): S1–S3 (single), MK1–MK3 (multikey), MV (multivalue), MQ (multiquery); QA: QA1–QA2; Aggregation: CWE, FWE; Tracing: VT.

Because NIAH accounts for 8 of the 13 equally weighted tasks, its 50.1-point gain contributes approximately 30.8 points of the 35.0-point overall improvement (about 88%). Recovery is especially large where KVzip nearly fails: niah_multikey_3 improves from 0.4 to 67.4, and niah_multiquery from 11.9 to 92.8. Thus, the unusually large RULER average reflects both severe baseline degradation and the benchmark's substantial weight on retrieval tasks, rather than uniformly larger gains across every task type.

In addition, at r=0.05, Table 6 (Table 5 in the original submission) shows that generating restore states from the full cache achieves 73.2, compared with 64.4 when the generator accesses only the surviving KV. This supports the benefit of full-context access under aggressive compression.

We have added the full per-task RULER results and this analysis to Appendix A.6 of the revision.

AuthorsOpenReview ↗

Comment 3/3 for Reviewer cgWs

Weakness 3: The KV-Distill method explained in [Lines 139–141] seems closely related to RestoreKV. It would be helpful to clarify the key differences between the two approaches, and to include an empirical comparison if feasible.

Response: Thank you for the comment. We clarify the conceptual differences from KV-Distill and provide an empirical comparison.

(1) Conceptual differences. RestoreKV and KV-Distill share LoRA and distillation from the full-cache model, but the role of the learned KV differs. KV-Distill constructs replacement KV through selected-token re-encoding, while Still [2] synthesizes compact latent states. In contrast, RestoreKV preserves the base evictor and retained original KV, learning only eight complementary states within the same final budget. These states are trained through predictions from the combined retained-and-restore cache; adapters are disabled for query processing and decoding.

An additional control supports the role of this training objective: on RULER-4K with Qwen3-4B at r=0.05, joint training achieves 73.2, versus 54.3 for standalone eight-state memory training followed by combination with the same retained KV at inference (Sec. 4.4, Table 6).

(2) Accuracy and cache-construction time. Since KV-Distill and Still have not released code or checkpoints, and reproducing them is costly (KV-Distill reports that most models converged within 3 days on eight A100 GPUs [1]), we compare with the results reported in Still [2], including its reproduction of KV-Distill. Our evaluations use Qwen3-4B-Instruct-2507, 32K contexts, the same seven RULER tasks, and final cache budgets K that include our eight restore states. The published RULER scores are from the zero-shot transfer rows of Table 12 in [2]. Training data differ across methods; this is a comparison with reported results rather than a common-codebase reproduction.

RULER-32K, 7 tasksK=2048 (r=0.063)K=1024 (r=0.031)K=512 (r=0.016)
KV-Distill [1]24.122.721.3
Still [2]37.234.029.6
KVzip + ours57.639.725.2
Fast KVzip [3] + ours61.146.926.8
QuALITY-concat, 32KK=2048 (r=0.063)K=1024 (r=0.031)K=512 (r=0.016)
KV-Distill [1]NRNR40.0
Still [2]42.5NRNR
KVzip + ours53.246.142.5
Fast KVzip [3] + ours51.539.237.8

For QuALITY-concat, Table 9 of [2] reports best-over-K scores of 40.0 for KV-Distill at K=512 and 42.5 for Still at K=2048. NR denotes an unreported per-budget score.

Both RestoreKV combinations exceed KV-Distill's reported RULER scores at every budget and Still's at K=2048 and 1024, while Still is stronger at K=512. On QuALITY-concat, KVzip + RestoreKV exceeds KV-Distill's reported best-over-K score at all three budgets.

Method (K=2048)Construction timeRULER-32K
KV-Distill [1]2.13 s (est.)24.1
Still [2]2.14 s (est.)37.2
KVzip7.23 s49.5
KVzip + RestoreKV7.28 s57.6
Fast KVzip [3]1.81 s53.9
Fast KVzip [3] + RestoreKV1.85 s61.1

KV-Distill/Still accuracies are reported in [2], and their times are architecture-level profiling estimates from our random-weight reimplementations, not measurements of trained systems. Other entries are measured in our setup. All times include prefill at K=2048 on one RTX PRO 6000.

In our directly measured comparison, adding RestoreKV to Fast KVzip improves RULER accuracy from 53.9 to 61.1 (+7.2 points) while increasing construction time from 1.81 to 1.85 s (+0.04 s; 2.2%). Thus, restoration also benefits a fast learned evictor. The profiling estimates do not establish a directly measured speed advantage over KV-Distill or Still.

We have clarified the relation to KV-Distill in Sec. 2.2, added these comparisons to Sec. 4.2 and Appendix A.7, and added the standalone control to Sec. 4.4. Further details are provided in our response to Reviewer QRC6 (W1 and Q1).

References:

[1] Chari et al. KV-Distill: Nearly Lossless Learnable Context Compression for LLMs. arXiv:2503.10337.

[2] O'Neill et al. Still: Amortized KV Cache Compaction in a Single Forward Pass. arXiv:2606.07878.

[3] Kim et al. Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction. NeurIPS 2026.


Weakness 4: Several figures seems empty or missing plots

Response: Thank you for pointing this out. We have re-exported Figs. 1, 5, and 6 and checked that all panels are visible in the revised PDF.

Reviewer_QRC6OpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RestoreKV trains a small adapter to generate eight extra KV states before eviction, replacing an equal number of retained states. These states improve compressed cache accuracy while leaving the model and decoding unchanged.

Critical strengths
  • Eight generated states substantially improve accuracy without increasing the final cache budget or requiring adapters during decoding.

  • Table 5 shows that much of the degradation can be repaired using information already present in the pruned cache, accuracy rises without accessing discarded states. This suggests that aggressive eviction damages how the model uses surviving information, beyond simply removing needed facts.

Critical weaknesses
  • The paper’s main distinction is learning a small complement instead of replacing the entire cache but it does not compare against the closest amortized learned compressors; KV-Distill and Still. Its speed comparison uses Attention Matching, which fits each context separately, so it does not establish the advantage over methods that already avoid that fitting cost.

[Chari, V., Qin, G., and Van Durme, B. (2025). KV-Distill: Nearly Lossless Learnable Context Compression for LLMs]

[O’Neill et al. (2026), Still: Amortized KV Cache Compaction in a Single Forward Pass]

Questions for authors
  • Does training the restore states jointly with retained KV outperform training a standalone eight-state memory and then combining it with the same retained cache?
4 comments
AuthorsOpenReview ↗

Comment 1/2 for Reviewer QRC6

We thank the reviewer for the careful reading and thoughtful comments.

Weakness 1: The paper's main distinction is learning a small complement instead of replacing the entire cache but it does not compare against the closest amortized learned compressors; KV-Distill [1] and Still [2]. Its speed comparison uses Attention Matching, which fits each context separately, so it does not establish the advantage over methods that already avoid that fitting cost.

Response: Thank you for highlighting these related methods. We add a conceptual comparison, accuracy results on the same seven RULER tasks as Still, and cache-construction times.

(1) Conceptual comparison.

KV-Distill [1]Still [2]KVzip [3]RestoreKV
Mechanism*SynthesisSynthesisEvictionEviction + synthesis
Original KV valuesReplacedReplacedPreservedPreserved
Role of learned KVCompact cacheCompact cacheNone8-state complement

Here, synthesis broadly denotes learned construction of replacement KV representations: KV-Distill uses selected-token re-encoding, while Still generates compact latent states.

RestoreKV shares LoRA and distillation components with KV-Distill. Its distinction is to preserve the base evictor and retained original KV unchanged, while learning eight complementary states through predictions from the combined retained-and-restore cache. These states count within the same budget, and adapters are disabled for query processing and decoding.

(2) Accuracy comparison.

Since KV-Distill and Still have not released code or checkpoints, reproducing them requires substantial compute: [1] reports most models converged within 3 days on eight A100 GPUs, while [2] uses eight H200 GPUs for Qwen3-4B. We therefore compare with their published results in [2], rather than retrain them, on RULER and QuALITY-concat. Our evaluations use Qwen3-4B-Instruct-2507, 32K contexts, and final budgets K including our restore states. RULER uses the seven tasks in Still: niah_single_2, niah_multikey_1, vt, cwe, fwe, qa_1, qa_2. Published RULER scores come from the zero-shot transfer rows of Table 12 in [2], including its KV-Distill reproduction. Our entries are new evaluations; training data differ.

RULER-32K, 7 tasksK=2048 (r=0.063)K=1024 (r=0.031)K=512 (r=0.016)
KV-Distill [1] (arXiv 2025.03)24.122.721.3
Still [2] (arXiv 2026.06)37.234.029.6
KVzip [3] (NeurIPS 2025)49.520.818.6
KVzip + RestoreKV57.639.725.2
Fast KVzip [4] (NeurIPS 2026)53.923.017.5
Fast KVzip + RestoreKV61.146.926.8

RestoreKV improves KVzip by 8.1/18.9/6.6 points and Fast KVzip by 7.2/23.9/9.3 points. Both exceed Still's reported scores at K=2048 and 1024, while Still performs better at K=512.

QuALITY-concat, 32KK=2048 (r=0.063)K=1024 (r=0.031)K=512 (r=0.016)
KV-Distill [1] (arXiv 2025.03)NRNR40.0
Still [2] (arXiv 2026.06)42.5NRNR
KVzip [3] (NeurIPS 2025)47.043.341.8
KVzip + RestoreKV53.246.142.5
Fast KVzip [4] (NeurIPS 2026)50.838.636.9
Fast KVzip + RestoreKV51.539.237.8

K is the final budget; NR denotes unreported per-budget scores. Table 9 of [2] reports best-over-K scores: 40.0 for KV-Distill and 42.5 for Still.

KVzip + RestoreKV exceeds both reported best-over-K scores at K=2048 and 1024, and matches Still's best at K=512.

(3) Cache-construction time.

To address efficiency beyond per-context fitting, we profile KV-Distill and Still and evaluate RestoreKV with Fast KVzip, a learned evictor. The following results use 32K contexts and K=2048 on one RTX PRO 6000, including prefill.

MethodConstruction timeRULER-32K Acc.
KV-Distill*2.13 s24.1
Still*2.14 s37.2
KVzip7.23 s49.5
KVzip + RestoreKV7.28 s57.6
Fast KVzip1.81 s53.9
Fast KVzip + RestoreKV1.85 s61.1

KV-Distill/Still times are architectural profiling estimates from random-weight reimplementations, not measurements of trained systems; accuracies are from [2]. Other rows report measured time and accuracy.

Adding RestoreKV to Fast KVzip improves accuracy from 53.9 to 61.1 for only 0.04 s additional construction time (2.2%). This is faster and more accurate than KVzip + RestoreKV (1.85 vs. 7.28 s; 61.1 vs. 57.6). The estimated times do not establish a measured speed advantage over KV-Distill or Still. Thus, RestoreKV also benefits a fast learned evictor.

These results show that a small learned complement improves evictors at low additional cost. We updated Sec. 2.2, Sec. 4.2, and Appendix A.7.

References:

[1] Chari et al. KV-Distill. arXiv:2503.10337.

[2] O'Neill et al. Still. arXiv:2606.07878.

[3] Kim et al. KVzip. NeurIPS 2025.

[4] Kim et al. Fast KVzip. NeurIPS 2026.

AuthorsOpenReview ↗

Comment 2/2 for Reviewer QRC6

Q1: Does training the restore states jointly with retained KV outperform training a standalone eight-state memory and then combining it with the same retained cache?

Response: Thank you for this insightful question. Yes. We implemented the standalone eight-state memory control you suggested. At r=0.05, joint training achieves 73.2, versus 54.3 for standalone training followed by combination with the same retained KV at inference (+18.9 points). Both generators access the full cache before eviction and use the same eight-state capacity and final inference budget. The key distinction is whether the generated states are trained through predictions from the combined retained-and-restore cache.

We motivate RestoreKV through two related aspects:

  • (i) Semantic restoration: Accessing the full cache before eviction allows the generator to incorporate information from entries that will be discarded.
  • (ii) Behavioral restoration: Training through predictions from the combined cache encourages the generated states to complement the retained KV and compensate for eviction-induced changes in model behavior.

We evaluate these design choices with three variants:

Standalone training: The generator accesses the full cache, but the student's predictions during training use only the eight generated states as context memory. At inference, these states are combined with the same retained KV used by joint training.

From evicted cache: The generator accesses only the remaining cache after eviction and is trained through predictions from the combined retained-and-restore cache.

Joint training (default): The generator accesses the full cache before eviction and is trained through predictions from the combined cache.

All three variants use the same backbone, base evictor, number of restore states, and final KV budget. "Joint" does not mean updating the retained KV: the backbone and base evictor remain frozen, the retained KV representations are unchanged, and only the restore-token embeddings and LoRA parameters are trained.

RULER-4K (Qwen3-4B)Generator accesses full cacheTrained with retained KVr=0.2r=0.1r=0.05
KVzip (baseline)––91.480.138.2
Standalone training✓✗93.385.154.3
From evicted cache✗✓93.787.664.4
Joint training (default)✓✓93.588.873.2

First, joint vs. standalone training directly answers your question. With the same retained cache at inference, joint training improves accuracy by 0.2/3.7/18.9 points at r=0.2/0.1/0.05. The largest benefit occurs under the most aggressive compression, supporting the value of training the states for use with the retained cache rather than attaching an independently trained memory afterward.

Second, the from-evicted-cache variant tests the benefit of full-cache access. As you noted in your review, substantial recovery is possible using surviving KV alone: at r=0.05, this variant reaches 64.4, compared with 38.2 for KVzip. Allowing the generator to access the full cache further improves accuracy to 73.2 (+8.8 points). This benefit is budget-dependent: at r=0.2, the from-evicted-cache variant is slightly higher (93.7 vs. 93.5).

These controls support two design choices: training through predictions from the combined cache and providing full-context access before eviction. We use semantic and behavioral restoration to motivate these choices, not to claim a strict decomposition of their effects.

Crucially, the 73.2 vs. 54.3 comparison supports our distinction in W1: learning a complement specifically for use with retained original KV, rather than learning an independent compact memory and combining it with that cache only afterward.

We have clarified these motivations in the Introduction and added the standalone control and analysis to Sec. 4.4 and Table 6 (Table 5 in the original submission). We thank the reviewer for this insightful question, which helped us clarify our interpretation of RestoreKV’s gains and better articulate the role of joint training.

Thank you for running the requested experiments. These results materially strengthen the paper’s distinguishing contribution. I will raise my recommendation.

AuthorsOpenReview ↗

Thank You to Reviewer QRC6

Thank you for your updated assessment. Your insightful comments helped improve the paper, and we appreciate your time and engagement.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Summary of Revisions

We thank all reviewers for their constructive feedback. Below we summarize the major updates in the revised manuscript.

Key Updates

  • Generalization and reusability (cgWs, 5eSj): RestoreKV improves KVzip in 12/12 code-to-general and 8/9 general-to-code settings, and in all domains of an expanded eight-domain evaluation; adding new-domain training data yields comparable results. A single KVzip-trained generator also improves all six other evictors without retraining (18/18 settings), performing close to or better than evictor-specific training on four reconstruction-based evictors. (Sec. 5, Appendix A.3, A.4)

  • Comparison with amortized learned compressors (QRC6, cgWs): Clarified that, unlike KV-Distill and Still, which learn a compact replacement cache, RestoreKV preserves the base evictor and retained original KV and learns only eight complementary states within the same budget. Compared with their published results on the same seven RULER-32K tasks, RestoreKV achieves higher accuracy at K=1024 and 2048. Adding RestoreKV to Fast KVzip increases construction time by only 0.04 s. (Sec. 2.2, Sec. 4.2, Appendix A.7)

  • Joint vs. standalone training (QRC6): Training restore states through the combined retained-and-restore cache outperforms a standalone eight-state memory (73.2 vs. 54.3 at r=0.05). This result strengthens our core contribution of learning restore states specifically to complement the retained KV. (Introduction, Sec. 4.4)

  • Explanation of RULER gains (cgWs): The large gain on RULER at r=0.05 reflects severe baseline degradation and recovery on retrieval tasks, which account for about 88% of the 35-point improvement. (Appendix A.6)

  • Training cost (5eSj): Training a generator takes about 1.5 hours on a single GPU for Llama-3.1-8B-Instruct, with peak memory comparable to KVzip inference. (Table 7)

  • Integration cost (5eSj): We provide an example implementation in KVPress, NVIDIA's open-source KV cache compression library. It requires about 130 lines in a single file plus eight lines of changes to the library core. Since restore states are ordinary KV pairs, no custom attention kernel is needed, and LoRA is disabled after cache construction. (Supplementary)

Main Paper Revisions

  • Introduction: Clarified motivation via semantic and behavioral restoration.
  • Sec. 2.2 / 4.2: Added relation to and comparisons with amortized learned compressors (KV-Distill and Still).
  • Sec. 4.4, Table 6: Added standalone-training and from-evicted-cache controls.
  • Sec. 5: Clarified the limitation on generator reuse across evictors.
  • Figures: Re-exported Figs. 1, 5, and 6.

Appendix Updates

  • A.3: Generator reuse across six evictors (improves all 18 settings).
  • A.4: Domain-shift transfer and training-data composition study.
  • A.6: Per-task RULER analysis of the large gain at r=0.05.
  • A.7: Full comparison details with KV-Distill and Still.
  • Supplementary: Uploaded KVPress integration example code.

We would appreciate your updated assessment in light of these additions and would be happy to clarify any remaining concerns. Thank you again for your time and thoughtful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.