W1: Peak memory during prefill. DEKV evicts vision tokens for decoding.
However, the method requires additional compute and storage of attention during
the decode stage. Based on past experience working on this area, I personally
ran into OOM problems with ElasticCache on A100-80GB GPUs with long requests or
large images, even with batch size 1. Although DEVK undoubtedly saves on VRAM
during decoding stage, the authors should provide profiling details on the
prefill stage. Furthermore, how are the original full KV tensors managed while
waiting for global allocation? Do the authors retain them on GPU, temporarily
offload them, or perform an online selection?
W2: Additional baselines. The authors position DEKV as direct competitor to
post-prefill eviction methods, the authors should still include a few
in-prefill baselines to help the community better understand where DEKV is
positioned. For the rebuttal, I request the authors to include one of the
following: FastV [1], HiRED [2] or similar, in terms of both performance and
efficiency.
W3: Limited modality-specific novelty and extension to textonly LLMs. DEKV is
not inherently vision-specific, as its damage metric is derived entirely from
standard softmax attention without relying on multimodal properties. Prior
methods such as CAOTE [3] and ReST-KV[4] have explored closely related
attention-output perturbation criteria, while LAVa [5] and LaProx [6] have
investigated unified token ranking and adaptive or global KV budget allocation.
The primary vision-specific components of DEKV appear to be restricting damage
estimation to text-query attention rows and imposing a minimum vision-token
retention floor. This raises the question of whether DEKV introduces a
fundamentally multimodal formulation or primarily adapts and combines existing
output-aware KV eviction techniques for VLMs. The authors should more clearly
distinguish their methodological contributions from these prior approaches and
ideally evaluate DEKV on text-only LLMs to establish its generality.
Furthermore, controlled comparisons against prior output-aware scoring and
global allocation methods would strengthen the paper’s positioning.
W4: Retention of positional embeddings. Since DEKV evicts a different number of
tokens per layer, a token may end up at different positions across different
layers (e.g. position i in layer l and position j in layer l+1, where i ≠ j).
How do the authors deal with this in implementation? The paper also reports
results on Qwen3-VL, which uses interleaved MRoPE for spatial temporal
modelling. Do the authors have a special way to treat the positional embedding
for this?
W5: Efficiency metrics. Although DEKV achieves better results, Table 4’s
latency metrics are less than desirable. Specifically, it is slower than Full
KV and TGV-KV on nearly all latency-related measures. Although the savings on
peak memories are significant, the latencies tell an adverse story.
W6: Inconsistent retention rates across tables. Table 2 reports 20% (and 5%)
retention rates, while Table 4 reports 10% (and 5%), which makes the comparison
unfair. The method could have better performance at 20% but worse efficiency
measures. The authors should report both performance and efficiency measures at
the same ratios.
W7: Substantial failure case overshadowed. Table 2 reports DEKV’s VizWiz
accuracy to be 19.2 at 5% retention rate, compare the 54.4% for vanilla LLaVA.
This is a significant drop not observed in other benchmarks and retention
rates. The authors should elaborate on this failure case for the benefit of the
community.
W8: Mixed evidence for the necessity of DWF. Table 5 shows that DEKV without
DWF allocation sometimes performs better. At 5% retention, TextCaps is 0.567
without DWF vs 0.550 with full DEKV. On ChartQA, DWF improves the 5% result
only from 15.7 to 15.9, while no-DWF performs better at 50% and 20%. These
results support a task-dependent contribution, but not the statement that both
components are “strictly necessary” for extreme-compression gains (line 163).
The allocation ablation also covers only one model and two tasks, without
uncertainty estimates. The paper should include additional ablations on other
model families and sizes.
W9: Minor: the figures are illustrative, but the text is too small to be read
comfortably. Table 4 is hard to read without lines to indicate the groupings
for each metric. Table 3 should also include the retention rate in the caption.
References:
[1] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.
[2] Arif, Kazi Hasan Ibn, et al. "Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 2. 2025.
[3] Goel, Raghavv, et al. "Caote: Kv cache selection for llms via attention output error-based token eviction." arXiv preprint arXiv:2504.14051 (2025).
[4] An, Yongqi, et al. "ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing." arXiv preprint arXiv:2605.08840 (2026).
[5] Yiqun Shen, Song Yuan, Zhengze Zhang, Xiaoliang Wang, Daxin Jiang, and Cam-Tu Nguyen. LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 13672–13692.
[6] Mai, Tho, and Joo-Young Kim. "Reformulating KV Cache Eviction Problem for Long-Context LLM Inference." arXiv preprint arXiv:2605.07234 (2026).