acceptodds
Withdrawn from ICLR 2027

Unified Damage-Based KV Cache Compression for Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language Models (VLMs) cache the keys and values of context tokens for autoregressive decoding, but the KV cache grows linearly with context length and can be dominated by thousands of visual tokens. Existing eviction methods typically rank tokens by accumulated attention, measuring the sum of attention weights a token receives across queries rather than what is lost when it is removed, while separate heuristic sets per-layer budgets. We present DEKV (Damage-Equalized KV Eviction), a training-free framework that uses a single damage metric to determine both token importance and per-layer budgets. The metric is the exact attention-output perturbation caused by evicting one token, obtained in closed form from a softmax leave-one-out identity. DEKV sums these per-token damages as an additive surrogate for joint eviction, which we bound analytically and validate empirically. Normalizing by each layer's output scale makes damages comparable across layers, and a single global threshold, equivalent to water-filling over the per-layer damage curves, allocates the memory budget without an external allocation policy. DEKV scores only text-query attention rows and keeps the cache at exactly the budget after prefill and after every decoding step. Across six VLMs from the LLaVA, Qwen3-VL, and Phi-3.5 families on four VQA benchmarks, DEKV outperforms the strongest evaluated baseline, TGV-KV, in 22 of 24 settings at a 5% KV budget, retaining 86.6% of full-cache accuracy on average versus 80.0%, with gains of up to 27.5 points on ChartQA and 13.0 points on DocVQA for Phi-3.5-vision, while reducing KV cache memory by 20×.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, Vision-Language Models, Damage-Aware Eviction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eyEmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that KV cache eviction should account for output distortion caused by both changes in attention weights and the removal of informative value vectors. The proposed metric simply combines these two effects. The method is evaluated primarily on image benchmarks across diverse model families.

Critical strengths

1 This paper is easy to read

2 This method is evaluated in diverse model family

Critical weaknesses

1 The proposed damage metric has limited novelty. Changes in attention scores induced by KV cache eviction have been extensively studied in prior work [1], as has the resulting approximation error from discarding value vectors. The paper should clearly distinguish this metric from existing formulations and explain what new insight it provides. The motivation should explicitly explain why the proposed method is expected to be more effective under tight KV cache budgets, which is distinct from general discussion of top-p and top-k sampling [2].

[1] Cross-Self KV Cache Pruning for Efficient Vision-Language Inference [2] Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruning

2 The proposed method appears to combine the two existing approaches discussed above and reformulate them as a single metric. I do not see a substantive methodological innovation beyond this reformulation. A straightforward combination of the existing approaches could plausibly achieve similar results, and the paper should include this baseline to demonstrate the benefit of the proposed metric.

3 The evaluation is limited to image benchmarks, which do not adequately assess KV cache compression in long-context, memory-bound settings. These experiments provide insufficient evidence of practical benefits for long-video or multi-image workloads, where cache growth and memory bandwidth become major bottlenecks. The paper should evaluate these settings and report peak memory usage, latency, and throughput alongside accuracy.

Questions for authors

NA

Reviewer_d44gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DEKV, a training-free KV cache eviction framework for vision-language models that unifies token importance estimation and per-layer budget allocation using a single damage metric. The metric measures the exact attention-output perturbation caused by evicting an individual token, derived from a softmax leave-one-out identity and aggregated over text-query attention rows. The damage scores are normalized across layers to enable global water-filling-based selection, which dynamically determines per-layer KV cache budgets. Experiments across multiple VLM families and benchmarks demonstrate that DEKV generally achieves better accuracy retention than existing baselines, particularly under aggressive KV cache compression.

Critical strengths
  • The experimental evaluation is relatively comprehensive. The paper evaluates DEKV across multiple VLM families and model scales, covering a variety of benchmarks for visual question answering and image captioning. These experiments provide evidence of the method's effectiveness and generalizability across different model architectures and tasks.
  • The paper introduces a damage-based importance metric for KV cache tokens that considers both attention weights and the difference between value vectors and attention outputs. This metric provides a more informative criterion for distinguishing redundant tokens from important ones than attention-only scoring. Furthermore, the paper employs the same damage metric for both token selection and cross-layer budget allocation, resulting in a unified compression framework.
Critical weaknesses
  • Missing comparison with LAVa. LAVa has already proposed a unified framework for KV cache eviction and dynamic budget allocation based on information loss. A comparison with LAVa would help better establish the effectiveness and novelty of DEKV.
  • Eq. (3) computes the exact attention-output perturbation caused by evicting a single token. However, the actual damage caused by jointly evicting multiple tokens is generally not equal to the sum of their individual damages. Therefore, the additive damage surrogate used in Eq. (2) requires further theoretical justification. Although the abstract claims that this surrogate is analytically bounded and empirically validated, I could not find an explicit theoretical bound or its derivation in the main text or appendix. The authors should provide the claimed theoretical analysis.
  • The readability of Table 4 could be improved by adding horizontal lines to separate different evaluation metrics. Additionally, the best-performing results should be highlighted in bold to facilitate comparisons across methods.
  • It is suggested to add a comparison of latency and throughput with other KV Cache compression methods in Table 4 to more comprehensively evaluate the actual inference efficiency and additional computational overhead of DEKV.
  • Eq. (4) introduces the query weight wiw_i, but its computation is only briefly described in the main text, with the explicit definition provided in Appendix Table 11. The authors are encouraged to include the formula for computing wiw_i near Eq. (4) to improve the completeness and clarity of the methodology.
  • The paper lacks sensitivity analysis on hyperparameters qq、ρ\rho、γ\gamma.
Questions for authors

The abstract claims that the additive surrogate for joint token eviction is analytically bounded and empirically validated. However, I could not find an explicit theoretical bound or its derivation in the main text or appendix.

Reviewer_hUrtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes DEKV, a training-free KV cache compression framework for VLMs that unifies token importance scoring and layer-wise budget allocation under a single attention-output perturbation metric. Specifically, DEKV estimates the damage caused by evicting individual KV entries using a closed-form softmax leave-one-out formulation. It then applies global damage-based selection to dynamically allocate memory across VLM layers. Experiments across six VLMs and multiple VQA benchmarks demonstrate improved accuracy retention over existing eviction methods, particularly under very-aggressive retention rates as low as 5%.

Critical strengths

S1: The paper presents a meaningful token-importance criterion with supporting ablation. The damage score explicitly accounts for value content and softmax renormalization, capturing information that attention weights alone omit. Its single-token interpretation is mathematically clear. The ablation also supports its usefulness: at 10% retention, LLaVA-1.5 TextCaps score falls significantly when damage is replaced simply by attention.

S2: Substantial gains across several architectures at extreme compression. DEKV beats existing baselines in most model-benchmark pairs, supporting the paper’s claims of quality preservation under extreme compression rates.

S3: Efficiency gains over vanilla VLMs. DEKV significantly reduces KV cache memory consumption, achieving up to 20× compression at a 5% retention budget.At 4K context length, peak GPU memory decreases from 49.79 GB (vanilla) to 19.28 GB, while at 8K context length, DEKV enables inference where the vanilla implementation runs out of memory, demonstrating the practical benefits of DEKV.

Critical weaknesses

W1: Peak memory during prefill. DEKV evicts vision tokens for decoding. However, the method requires additional compute and storage of attention during the decode stage. Based on past experience working on this area, I personally ran into OOM problems with ElasticCache on A100-80GB GPUs with long requests or large images, even with batch size 1. Although DEVK undoubtedly saves on VRAM during decoding stage, the authors should provide profiling details on the prefill stage. Furthermore, how are the original full KV tensors managed while waiting for global allocation? Do the authors retain them on GPU, temporarily offload them, or perform an online selection?

W2: Additional baselines. The authors position DEKV as direct competitor to post-prefill eviction methods, the authors should still include a few in-prefill baselines to help the community better understand where DEKV is positioned. For the rebuttal, I request the authors to include one of the following: FastV [1], HiRED [2] or similar, in terms of both performance and efficiency.

W3: Limited modality-specific novelty and extension to textonly LLMs. DEKV is not inherently vision-specific, as its damage metric is derived entirely from standard softmax attention without relying on multimodal properties. Prior methods such as CAOTE [3] and ReST-KV[4] have explored closely related attention-output perturbation criteria, while LAVa [5] and LaProx [6] have investigated unified token ranking and adaptive or global KV budget allocation. The primary vision-specific components of DEKV appear to be restricting damage estimation to text-query attention rows and imposing a minimum vision-token retention floor. This raises the question of whether DEKV introduces a fundamentally multimodal formulation or primarily adapts and combines existing output-aware KV eviction techniques for VLMs. The authors should more clearly distinguish their methodological contributions from these prior approaches and ideally evaluate DEKV on text-only LLMs to establish its generality. Furthermore, controlled comparisons against prior output-aware scoring and global allocation methods would strengthen the paper’s positioning.

W4: Retention of positional embeddings. Since DEKV evicts a different number of tokens per layer, a token may end up at different positions across different layers (e.g. position i in layer l and position j in layer l+1, where i ≠ j). How do the authors deal with this in implementation? The paper also reports results on Qwen3-VL, which uses interleaved MRoPE for spatial temporal modelling. Do the authors have a special way to treat the positional embedding for this?

W5: Efficiency metrics. Although DEKV achieves better results, Table 4’s latency metrics are less than desirable. Specifically, it is slower than Full KV and TGV-KV on nearly all latency-related measures. Although the savings on peak memories are significant, the latencies tell an adverse story.

W6: Inconsistent retention rates across tables. Table 2 reports 20% (and 5%) retention rates, while Table 4 reports 10% (and 5%), which makes the comparison unfair. The method could have better performance at 20% but worse efficiency measures. The authors should report both performance and efficiency measures at the same ratios.

W7: Substantial failure case overshadowed. Table 2 reports DEKV’s VizWiz accuracy to be 19.2 at 5% retention rate, compare the 54.4% for vanilla LLaVA. This is a significant drop not observed in other benchmarks and retention rates. The authors should elaborate on this failure case for the benefit of the community.

W8: Mixed evidence for the necessity of DWF. Table 5 shows that DEKV without DWF allocation sometimes performs better. At 5% retention, TextCaps is 0.567 without DWF vs 0.550 with full DEKV. On ChartQA, DWF improves the 5% result only from 15.7 to 15.9, while no-DWF performs better at 50% and 20%. These results support a task-dependent contribution, but not the statement that both components are “strictly necessary” for extreme-compression gains (line 163). The allocation ablation also covers only one model and two tasks, without uncertainty estimates. The paper should include additional ablations on other model families and sizes.

W9: Minor: the figures are illustrative, but the text is too small to be read comfortably. Table 4 is hard to read without lines to indicate the groupings for each metric. Table 3 should also include the retention rate in the caption.

References: [1] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024. [2] Arif, Kazi Hasan Ibn, et al. "Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 2. 2025. [3] Goel, Raghavv, et al. "Caote: Kv cache selection for llms via attention output error-based token eviction." arXiv preprint arXiv:2504.14051 (2025). [4] An, Yongqi, et al. "ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing." arXiv preprint arXiv:2605.08840 (2026). [5] Yiqun Shen, Song Yuan, Zhengze Zhang, Xiaoliang Wang, Daxin Jiang, and Cam-Tu Nguyen. LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 13672–13692. [6] Mai, Tho, and Joo-Young Kim. "Reformulating KV Cache Eviction Problem for Long-Context LLM Inference." arXiv preprint arXiv:2605.07234 (2026).

Questions for authors

For questions, please see the questions in W1 to W9 above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.