- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
Average accuracy and other aggregate metrics used to judge a compression’s success or failure can hide significant per-sample damage (and repair). At matched accuracy, these two proportions of the dataset are close to equal and cancel each other out, making them hard to see. This paper builds on that observation by studying post-prefill KV-cache eviction in VLMs across several datasets, models, and methods. It pairs each compressed run with the uncompressed or low-compressed baseline and uses the compressed model’s output confidence to measure, and paired runs to predict, and diagnose compression-induced failures.
Critical strengths
- Experimental breadth is substantial, including many models, datasets, and KV compression methods.
- A new angle; compressed to compressed prediction changes in sample correctness is new, Figure 1 illustrate that diffrent compressions break diffrent samples.
Critical weaknesses
-
The paper confounds two different problems. Starts with negative flips under a matched aggregate metric, i.e., failures that cancel against positive flips so that accuracy looks preserved. It then evolves to lost performance under aggressive compression, where the model loses significant aggregate performance. CIF then becomes mostly about broken prediction comparisons from compression A vs compression B, rather than invisible failures that matter.
-
The paper becomes about falling back to the non or low-compressed model on low-confidence samples to restore broken predictions. This requires a second inference (without compression) on the flagged samples.
-
Figure 1 illustrates negative prediction churns between two non-degraded (and then degraded) compressed models, and not flips (which are changes between a compressed model and its original), yet it is used to describe the latter; paragraph 1 of the introduction starts by introducing the flips problem, paragraph 2 starts linked to it with "However" and lists Figure 1, which is then directly defined by compressed-to-compressed prediction churns, making the core problem addressed by the paper hard to understand in order to assess significance.
Questions for authors
Please see the weaknesses, if any misinterpretation is present state which and why.
I also have the following questions:
-
Could you please state in a few sentences the exact problem that you to adress (flips, churns, failure under degradation or how diffrent compressions (methods and aggresivity) break diffrent predictions), and how the problem is significant.
-
Title may be misleading; the paper only treats post-prefill KV-cache eviction, but uses the term "Compression-induced Failures" while adding KV could make the metric and title less appealing, it remains necessary to not include other compression methods such as Pruning (model), sparsification, quantization, tensorial decomposition, other token reduction methods etc., do you have elements that show generalizability to these compression methods so that the current title is supported?
-
The new term "reverse flips" is very ambiguous; it's closer to being understood as a flip that was reversed (a flip that was reverted as in corrected). Why wasn't the already established "positive flip" used instead? How do they differ?
5 comments
Post-rebuttal Update
I thank the authors for their answers, which have answered most of my concerns and questions.
The remaining important ones are the following:
- The proposed fallback is too costly, as it requires a second inference without compression on the flagged samples. However, there are use-cases where it would be justified.
- As accuracy degradation increases, the CIF count becomes redundant with accuracy, as positive flips become negligible. I see CIFs as an informative metric for comparing the negative flips of two KV-eviction methods of similar accuracy, but not beyond that (not in the accuracy degraded regime)
Overall, I now find the work to be more informative and novel when focusing only on the first regime and motivation, as it is the first to my knowledge to compare individual sample failures across compressed models (KV-eviction in this case, which I suspect would generalize across other compression methods) in VLMs. However, extending the work to aggressive compressions, where accuracy is severely degraded, has added a lot of noise to the paper: the motivation of the work, coupled with the informativeness of the CIFs (which I still encourage renaming the metric), does not carry beyond that point.
I thus increase the informativeness and overall rating.
Response to Reviewer 6jG9
Thank you for reading our response and the revised paper, and for raising your scores. We also thank you for stating the two remaining points so precisely.
We agree that at a high eviction ratio the CIF count is close to the accuracy drop. The positive flips are few there, and by Proposition 1 the drop equals the CIF rate minus the positive-flip rate. What the paired runs add at this ratio is the identity of the failing requests. The comparison you describe, between methods of similar accuracy, is the first result of the paper (Figure 1a, Section 5.2). We use as the stress setting, and the final version will state this role of the paired runs in the introduction and in Section 5.4.
Fallback requires a second inference, and we agree that this is a real cost. Its use case is a workload in which the affected requests are a minority of the traffic (Table E in our response to Reviewer fjNy, Table 17 in the revised PDF). The paper uses fallback to test whether the per-request signal is actionable.
On the name of the metric CIF, we follow your suggestion. The final version will call it an eviction-induced failure (EIF), in line with the revised title.
Response to Reviewer 6jG9
Thank you for the careful reading. Your reading of Figure 1 is correct. It showed us that the introduction mixed two questions, and we rewrote the introduction and the figure accordingly. The title now names KV-cache eviction, and your phrase "methods and aggressivity" led us to add the aggressivity axis as Figure 1b.
[Q1] The exact problem and its significance
We study which requests a KV-cache eviction breaks, and how that set changes with the method and the eviction ratio. This corresponds to the last option in your list: how different compressions (methods and aggressivity) break different predictions. Every prediction is compared against the same reference run.
A compression-induced failure (CIF) is a negative flip of the eviction: the reference run answers correctly and the compressed run answers wrongly. The drop in accuracy equals the CIF rate minus the positive-flip rate, so accuracy fixes this net count and leaves open which requests fail (Table J).
Table J: What the benchmark score shows and what the paired comparison adds.
The problem is significant for three reasons. First, methods with nearly equal accuracy break different requests, so hard inputs collected under one method miss many failures of another. Second, failures nest along the eviction ratio (aggressivity), so the ρ=0.9 stress setting covers most failures at milder ratios. Third, each failure can be acted on. Eviction is applied per request and the weights stay the same, so the same request can be rerun with the full cache. Such a rerun repairs a CIF and leaves an error of the uncompressed model as it is, so the flip is the event to detect. On ChartQA (Qwen3-VL-4B) at 10% rerun coverage, a trigger trained on CIF labels recovers 53.7 CIFs, and one trained on generic errors recovers 23.5.
[W1] Two problems: cancellation and aggressive eviction
We agree that the introduction presented them as one. The revised introduction states one problem, which requests fail, and Proposition 1 stays as a paired accounting identity. This problem is common to both regimes. We use ρ=0.9 because it covers most failures at milder ratios and yields enough CIFs to study prediction and repair.
[W3] Figure 1: churn between compressed runs
You are right that the y-axis is churn between two compressed runs, restricted to reference-correct samples. The revised caption states it as the fraction of samples that are a CIF under exactly one method, with both CIF sets defined against the same reference. The introduction defines a CIF before it refers to the figure, and panel (b) adds the aggressivity axis.
[Misinterpretation, as you requested] One reference for every CIF
A CIF is always defined against the reference run, and prediction, fallback, and restoration all use these labels. Figure 1a and Section 5.2 compare two methods through their CIF sets, both defined against the same reference. The revision states this explicitly.
[W2] The second inference of fallback
Yes, fallback requires a second inference. It reruns the flagged requests with the same weights and the full cache, and every fallback curve reports this rerun coverage as its cost. We use fallback to test whether the per-request signal is actionable.
[Q2] Title and generality beyond KV-cache eviction
We agree, and the title now names KV-cache eviction: "Measuring, Predicting, and Diagnosing Failures under KV-Cache Eviction in Vision-Language Models". Weight pruning, sparsification, quantization, and decomposition are outside the study. For one other token-reduction placement, we ran FastV with pruning inside prefill (general response, item 3). The same paired measurement applies, and failure sets also overlap and nest there. The title and the claims stay with KV-cache eviction.
[Q3] The term "reverse flips"
They are the same event as positive flips (wrong under the reference, right after compression). The revision uses "positive flips" throughout.