acceptodds
Under review as a conference paper at ICLR 2027

Measuring, Predicting, and Diagnosing Compression-Induced Failures in Vision-Language Models

Abstract

Visual-token compression reduces the inference cost of vision-language models (VLMs). It is usually evaluated by average accuracy, which does not show which correct answers become incorrect. We characterize these compression-induced failures (CIFs) through paired reference and compressed runs, defining a CIF as a sample answered correctly by the reference but incorrectly after compression. We focus on post-prefill KV-cache eviction across ten benchmarks, three model families, four controlled retention policies, and adaptations of six published strategies. Methods with nearly equal accuracy often break different requests. On DocVQA, two adapted attention-based eviction methods differ by 0.19 points, yet 17.5% of the dataset fails under exactly one of them. The failures concentrate on benchmarks whose answers rely on small visual details such as text and layout. The paired runs also make these failures predictable and repairable. Output-confidence features rank CIFs at AUROC up to 0.89 and support selective fallback in paired simulation. Restoring pruned tokens repairs most failures, and at tight budgets score-guided selection repairs more than random or spatial selection. These results support reporting paired failure counts alongside average accuracy when evaluating compressed VLMs.

Primary area:
datasets and benchmarks
Keywords:
KV-cache compression, compression-induced failures, vision-language models, reliability auditing, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6jG9OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
1
Soundness
3
Read review
Paper summary

Average accuracy and other aggregate metrics used to judge a compression’s success or failure can hide significant per-sample damage (and repair). At matched accuracy, these two proportions of the dataset are close to equal and cancel each other out, making them hard to see. This paper builds on that observation by studying post-prefill KV-cache eviction in VLMs across several datasets, models, and methods. It pairs each compressed run with the uncompressed or low-compressed baseline and uses the compressed model’s output confidence to measure, and paired runs to predict, and diagnose compression-induced failures.

Critical strengths
  • Experimental breadth is substantial, including many models, datasets, and KV compression methods.
  • A new angle; compressed to compressed prediction changes in sample correctness is new, Figure 1 illustrate that diffrent compressions break diffrent samples.
Critical weaknesses
  • The paper confounds two different problems. Starts with negative flips under a matched aggregate metric, i.e., failures that cancel against positive flips so that accuracy looks preserved. It then evolves to lost performance under aggressive compression, where the model loses significant aggregate performance. CIF then becomes mostly about broken prediction comparisons from compression A vs compression B, rather than invisible failures that matter.

  • The paper becomes about falling back to the non or low-compressed model on low-confidence samples to restore broken predictions. This requires a second inference (without compression) on the flagged samples.

  • Figure 1 illustrates negative prediction churns between two non-degraded (and then degraded) compressed models, and not flips (which are changes between a compressed model and its original), yet it is used to describe the latter; paragraph 1 of the introduction starts by introducing the flips problem, paragraph 2 starts linked to it with "However" and lists Figure 1, which is then directly defined by compressed-to-compressed prediction churns, making the core problem addressed by the paper hard to understand in order to assess significance.

Questions for authors

Please see the weaknesses, if any misinterpretation is present state which and why.

I also have the following questions:

  • Could you please state in a few sentences the exact problem that you to adress (flips, churns, failure under degradation or how diffrent compressions (methods and aggresivity) break diffrent predictions), and how the problem is significant.

  • Title may be misleading; the paper only treats post-prefill KV-cache eviction, but uses the term "Compression-induced Failures" while adding KV could make the metric and title less appealing, it remains necessary to not include other compression methods such as Pruning (model), sparsification, quantization, tensorial decomposition, other token reduction methods etc., do you have elements that show generalizability to these compression methods so that the current title is supported?

  • The new term "reverse flips" is very ambiguous; it's closer to being understood as a flip that was reversed (a flip that was reverted as in corrected). Why wasn't the already established "positive flip" used instead? How do they differ?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6jG9

Thank you for the careful reading. Your reading of Figure 1 is correct. It showed us that the introduction mixed two questions, and we rewrote the introduction and the figure accordingly. The title now names KV-cache eviction, and your phrase "methods and aggressivity" led us to add the aggressivity axis as Figure 1b.

[Q1] The exact problem and its significance

We study which requests a KV-cache eviction breaks, and how that set changes with the method and the eviction ratio. This corresponds to the last option in your list: how different compressions (methods and aggressivity) break different predictions. Every prediction is compared against the same reference run.

A compression-induced failure (CIF) is a negative flip of the eviction: the reference run answers correctly and the compressed run answers wrongly. The drop in accuracy equals the CIF rate minus the positive-flip rate, so accuracy fixes this net count and leaves open which requests fail (Table J).

Table J: What the benchmark score shows and what the paired comparison adds.

Figure 1What the benchmark score showsWhat the paired comparison shows
(a) Two attention-based methods on DocVQA (Qwen3-VL-8B)They differ by 0.79 points18.3% of the dataset fails under exactly one of them
(a) The 26 method pairs within 2 pointsThey are nearly equalIn 18 pairs, more than 5% of the dataset fails under exactly one method
(b) CIFs at ρ=0.5\rho = 0.5 to 0.80.8The net drop at each ratio63-99% of them are also CIFs at ρ=0.9\rho = 0.9, against 2-63% of all correctly answered requests

The problem is significant for three reasons. First, methods with nearly equal accuracy break different requests, so hard inputs collected under one method miss many failures of another. Second, failures nest along the eviction ratio (aggressivity), so the ρ=0.9\rho = 0.9 stress setting covers most failures at milder ratios. Third, each failure can be acted on. Eviction is applied per request and the weights stay the same, so the same request can be rerun with the full cache. Such a rerun repairs a CIF and leaves an error of the uncompressed model as it is, so the flip is the event to detect. On ChartQA (Qwen3-VL-4B) at 10% rerun coverage, a trigger trained on CIF labels recovers 53.7 CIFs, and one trained on generic errors recovers 23.5.

[W1] Two problems: cancellation and aggressive eviction

We agree that the introduction presented them as one. The revised introduction states one problem, which requests fail, and Proposition 1 stays as a paired accounting identity. This problem is common to both regimes. We use ρ=0.9\rho = 0.9 because it covers most failures at milder ratios and yields enough CIFs to study prediction and repair.

[W3] Figure 1: churn between compressed runs

You are right that the y-axis is churn between two compressed runs, restricted to reference-correct samples. The revised caption states it as the fraction of samples that are a CIF under exactly one method, with both CIF sets defined against the same reference. The introduction defines a CIF before it refers to the figure, and panel (b) adds the aggressivity axis.

[Misinterpretation, as you requested] One reference for every CIF

A CIF is always defined against the reference run, and prediction, fallback, and restoration all use these labels. Figure 1a and Section 5.2 compare two methods through their CIF sets, both defined against the same reference. The revision states this explicitly.

[W2] The second inference of fallback

Yes, fallback requires a second inference. It reruns the flagged requests with the same weights and the full cache, and every fallback curve reports this rerun coverage as its cost. We use fallback to test whether the per-request signal is actionable.

[Q2] Title and generality beyond KV-cache eviction

We agree, and the title now names KV-cache eviction: "Measuring, Predicting, and Diagnosing Failures under KV-Cache Eviction in Vision-Language Models". Weight pruning, sparsification, quantization, and decomposition are outside the study. For one other token-reduction placement, we ran FastV with pruning inside prefill (general response, item 3). The same paired measurement applies, and failure sets also overlap and nest there. The title and the claims stay with KV-cache eviction.

[Q3] The term "reverse flips"

They are the same event as positive flips (wrong under the reference, right after compression). The revision uses "positive flips" throughout.

Reviewer_6jG9OpenReview ↗

Post-rebuttal Update

I thank the authors for their answers, which have answered most of my concerns and questions.

The remaining important ones are the following:

  • The proposed fallback is too costly, as it requires a second inference without compression on the flagged samples. However, there are use-cases where it would be justified.
  • As accuracy degradation increases, the CIF count becomes redundant with accuracy, as positive flips become negligible. I see CIFs as an informative metric for comparing the negative flips of two KV-eviction methods of similar accuracy, but not beyond that (not in the accuracy degraded regime)

Overall, I now find the work to be more informative and novel when focusing only on the first regime and motivation, as it is the first to my knowledge to compare individual sample failures across compressed models (KV-eviction in this case, which I suspect would generalize across other compression methods) in VLMs. However, extending the work to aggressive compressions, where accuracy is severely degraded, has added a lot of noise to the paper: the motivation of the work, coupled with the informativeness of the CIFs (which I still encourage renaming the metric), does not carry beyond that point.

I thus increase the informativeness and overall rating.

AuthorsOpenReview ↗

Response to Reviewer 6jG9

Thank you for reading our response and the revised paper, and for raising your scores. We also thank you for stating the two remaining points so precisely.

We agree that at a high eviction ratio the CIF count is close to the accuracy drop. The positive flips are few there, and by Proposition 1 the drop equals the CIF rate minus the positive-flip rate. What the paired runs add at this ratio is the identity of the failing requests. The comparison you describe, between methods of similar accuracy, is the first result of the paper (Figure 1a, Section 5.2). We use ρ=0.9\rho = 0.9 as the stress setting, and the final version will state this role of the paired runs in the introduction and in Section 5.4.

Fallback requires a second inference, and we agree that this is a real cost. Its use case is a workload in which the affected requests are a minority of the traffic (Table E in our response to Reviewer fjNy, Table 17 in the revised PDF). The paper uses fallback to test whether the per-request signal is actionable.

On the name of the metric CIF, we follow your suggestion. The final version will call it an eviction-induced failure (EIF), in line with the revised title.

Reviewer_jAKwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies compression-induced failures (CIFs) in vision-language models, defined as requests answered correctly by a reference model but incorrectly after compression. The authors introduce a paired evaluation protocol that measures these failures, reverse flips, and differences between the failure sets of compression methods. Experiments across ten benchmarks and three model families show that methods with nearly identical average accuracy can fail on substantially different requests, and that benchmark sensitivity depends on whether the scored answer is generated after compression takes effect. The paper further shows that output-confidence features can predict CIFs and guide selective fallback in paired simulations. Token-restoration experiments suggest that many failures are recoverable and that the identity of restored tokens matters at limited budgets. Overall, the results motivate reporting sample-level reliability alongside average accuracy when evaluating VLM compression.

Critical strengths

1.The paper is well motivated and makes the limitations of average accuracy concrete. In particular, the DocVQA example, where two methods differ by only 0.19 percentage points in accuracy but induce exclusive failures on 17.5% of the dataset, provides compelling evidence that aggregate performance can obscure meaningful differences in request-level reliability. Accounting for reverse flips and failure-set size differences further strengthens this analysis.

2.The experimental investigation is systematic and includes useful controls. Comparing retention policies under the same model, prompts, decoding settings, and token budgets helps isolate the effect of token selection. The answer-placement and noise-image controls are particularly informative, as they explain why some benchmarks can remain insensitive to substantial post-prefill cache eviction despite requiring visual information.

3.The paper presents a coherent progression from measuring failures to predicting and diagnosing them. The confidence-based predictor is simple and shows useful transfer across datasets, compression backends, and model families. Meanwhile, the equal-budget restoration experiments provide intervention-based evidence that recovery depends on which tokens are restored, rather than solely on increasing cache capacity.

Critical weaknesses

1.All evaluated methods operate within a post-prefill eviction setting, which limits the generality of the empirical conclusions. In this setting, the first generated token has already been computed using the full visual context. The FastV-style adaptation also changes important aspects of the original method, including its pruning placement. Although these limitations are acknowledged, the observed benchmark insensitivity and differences between compression methods cannot directly establish corresponding behavior under their original implementations. The official SnapKV comparison is helpful but covers only one method on a limited set of conditions.

2.The correctness criterion changes across some important experiments, making the results harder to compare. For the same Qwen3-VL-8B DocVQA run, the ANLS-threshold criterion produces 171 CIFs, whereas exact match produces 596 CIFs; the latter is used in parts of the prediction, fallback, and restoration analyses. This difference is substantial enough to change both the composition of the failure set and the interpretation of recovery rates. Although the switch is disclosed, it remains unclear whether the main conclusions hold consistently under both criteria.

3.The practical benefit of selective fallback is not yet established at a matched computational budget. Fallback is evaluated through paired-output simulation, and its cost is primarily represented by rerun coverage. However, the instrumented compressed pass is slower than the reference pass in the reported profiles, and peak memory is unchanged. Consequently, the results demonstrate potential accuracy recovery but do not establish a favorable efficiency–reliability trade-off relative to simply using less aggressive compression.

Questions for authors

1.For the adapted compression methods, could the authors provide a concise comparison of their original and adapted behavior, particularly regarding text-token protection, layer-wise budget allocation, and decode-time eviction? Beyond the official SnapKV comparison, is there evidence that the main failure-set turnover findings remain stable when more of the original implementation choices are preserved?

2.What are the exact binarization rules and thresholds used to define correctness on each benchmark? For Qwen3-VL-8B DocVQA, could the authors report the confidence-prediction, fallback, and restoration results under both ANLS-threshold and exact-match correctness? Understanding whether the reported gains persist under a consistent criterion would substantially strengthen the interpretation of these results.

3.At a matched end-to-end inference budget, how does confidence-triggered fallback compare with uniformly using a less aggressive compression ratio? A comparison that accounts for the compressed pass, risk estimation, and reference reruns would help establish whether selective fallback provides a practical advantage beyond the recovery demonstrated by paired simulation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer jAKw (Part 1 of 2)

Thank you for the concrete questions. Your caution about the original implementations led us to run FastV at its released placement. Failure sets also overlap and nest there, and the CIF rates and the benchmark insensitivity change, which confirms your caution (Tables F to H). Part 2 of this response covers the correctness criteria and the matched-budget comparison. The details are in Appendices A.2, A.3, C.2, D.3, D.6, and D.7 of the revised PDF.

[W1, Q1] Original and adapted behavior of each method

On the three choices you name, StreamingLLM, ElasticCache, and H2O follow the released code, and SnapKV and PrefixKV each differ on one (Table F). The FastV-style variant differs on all of them, so we also ran FastV with its released settings.

Table F: Behavior of each adapted method, and its difference from the released code.

MethodText-token protectionLayer and head budgetsDecode-time evictionDifference from the released code
StreamingLLMNoneSame in every layer, shared by headsEvery stepNone
ElasticCacheNoneSame in every layer, shared by headsEvery stepNone
SnapKVObservation window keptSame in every layer, per KV headNoneObservation window capped at half the budget (released: 64 tokens)
H2ONoneOne token set for all layers and headsEvery stepNone (the H2O paper selects per head)
PrefixKVNoneA separate budget for each layerEvery stepLayer budgets profiled per sample (released: offline profiles, none for Qwen3-VL)
FastV-style variantText evicted lastAll layers prunedEvery stepAll three choices, and placement

(All six evict after prefill. For H2O and StreamingLLM, the released code is the re-implementation published with ElasticCache.)

[W1, Q1 (continued)] Stability of the turnover findings under the original choices

We ran FastV at its released placement on Qwen3-VL-4B: pruning inside prefill, last-token ranking, first two layers intact, all text kept, no decode-time eviction. Its failures still recur across ratios (Table G) and still coincide with those of the post-prefill methods, and the CIF rates change (Table H).

Table G: Failures of FastV at its released placement recur at a higher ratio.

CIFs at ρ=0.9\rho = 0.9 among the CIFs at ρ=0.5\rho = 0.5 (%)CIFs at ρ=0.9\rho = 0.9 among all correctly answered requests (%)
ChartQA91.280.8
OCRBench98.390.1
TextVQA82.550.0
DocVQA91.374.9
POPE88.420.4
MMStar76.749.8

(Full evaluation sets. Correctly answered refers to the uncompressed model.)

Requests that fail at a mild ratio keep failing at a strong one. Of the CIFs at ρ=0.5\rho = 0.5, 77-98% are again CIFs at ρ=0.9\rho = 0.9, a higher share than among all correctly answered requests on every benchmark.

The released FastV also tends to break the requests that the post-prefill methods break. On TextVQA at ρ=0.9\rho = 0.9, it breaks 50.0% of all correctly answered requests (Table G) and 65-76% of the requests that a post-prefill method breaks. We made this comparison wherever a post-prefill run has CIFs at the same ratio: ten methods at ρ=0.9\rho = 0.9 on four benchmarks, and three retention policies at ρ=0.5\rho = 0.5 and 0.70.7 on three of them. The second share is the higher one in 57 of these 58 comparisons (Appendix C.2). The exception is on OCRBench at ρ=0.9\rho = 0.9, where FastV breaks 90% of the correctly answered requests.

Table H: CIF rate (%) of FastV.

Released, ρ=0.5\rho = 0.5Released, ρ=0.7\rho = 0.7Released, ρ=0.9\rho = 0.9Post-prefill adaptation, ρ=0.9\rho = 0.9
ChartQA12.835.967.816.4
OCRBench24.142.170.154.6
TextVQA3.39.142.626.3
DocVQA15.835.170.734.3
POPE1.44.218.60
MMStar8.616.229.70

Pruning inside prefill changes the first generated token, so POPE and MMStar register CIFs, and at ρ=0.9\rho = 0.9 the rate exceeds that of our adaptation on every benchmark. Your caution is therefore justified here. The answer-placement account of Section 5.3 predicts this, and the insensitivity of POPE and MMStar is specific to post-prefill eviction.

The released FastV is 10-68 points below the post-prefill methods in accuracy, so it lies outside the near-equal-accuracy window of Figure 1a. The official SnapKV code on LLaVA-NeXT errs on mostly the same requests as our adaptation. Over all errors of the compressed runs, their turnover is 0.16-0.18, while different methods on the same backbone reach 0.51.

Our H2O adaptation now re-indexes its accumulated scores after every eviction. With this correction, 18 of the 26 pairs within 2 points have more than 5% exclusive failures (19 of 27 in the submission). The DocVQA example you quote becomes the same method pair on Qwen3-VL-8B, 0.79 points apart with 18.3% of the dataset failing under exactly one method.

AuthorsOpenReview ↗

Response to Reviewer jAKw (Part 2 of 2)

[W2, Q2] Binarization rules and results under both criteria

The default rule is the same for every benchmark: a sample is correct if its standard per-sample metric is positive. It is applied to the reference and the compressed outputs alike. The one exception, stated in Appendix A.2, is that the Qwen3-VL-8B DocVQA rows of the prediction, transition, fallback, and restoration analyses use exact match, and Table A below gives both criteria.

Table I: Correctness rule per benchmark.

BenchmarkA sample is correct ifAlternative criterion tested
DocVQAANLS >0> 0, i.e., normalized edit similarity ≥0.5\geq 0.5 to a reference answerExact match
TextVQA, VizWizVQA accuracy >0> 0, i.e., at least one matching annotatorThree matching annotators
ChartQARelaxed accuracy (5% tolerance for numbers, exact match otherwise)-
OCRBenchThe binary per-sample score is 1-
GQAExact match-
POPE, HallusionBenchThe yes/no answer matches-
MMStarThe option matches-
AMBERThe per-sample non-hallucination score is positive-

The gains persist under either criterion. We repeated confidence prediction, fallback, and restoration on the same Qwen3-VL-8B DocVQA outputs under each.

Table A: The same DocVQA outputs under each criterion (repeated from our response to Reviewer fjNy).

Qwen3-VL-8B DocVQAANLS-thresholdExact match
CIFs171596
Positive flips2754
Confidence AUROC.866.880
CIFs recovered by fallback at 20% coverage76%74%
CIFs repaired at the 5% restoration budget, routing-score selection76.0%58.1%
Same budget, high-attention selection65.5%47.0%
Same budget, spatial selection56.1%38.3%
Same budget, random selection51.5%26.2%

As you note, the criterion changes the size of the failure set: exact match counts about 3.5 times as many CIFs. The conclusions are the same under both. Confidence ranks the CIFs about equally well, fallback recovers about three quarters of them, and the four restoration policies keep the same order, with routing-score selection first. The reanalysis corrected the 8B DocVQA restoration row, which had scored repairs with ANLS inside the exact-match pool (561 of 596). The consistent count is 346 of 596, and the order of the policies is unchanged.

[W3, Q3] Fallback against milder eviction at a matched budget

We agree with your reading of the cost profile. On our research stack, eviction reduces the memory of a request and adds time. On ChartQA (Qwen3-VL-4B), the uncompressed pass takes 252 ms per request, and the compressed pass takes 325 ms at ρ=0.7\rho = 0.7 and 359 ms at ρ=0.9\rho = 0.9. The detector takes 0.25 ms, and a rerun is one uncompressed pass. The decode-time KV cache of a request is 68 MiB uncompressed, 20 MiB at ρ=0.7\rho = 0.7, and 6.8 MiB at ρ=0.9\rho = 0.9. We therefore match budgets by KV entries, the resource that eviction reduces.

The smaller cache turns into capacity and speed under concurrency. At batch size 1 the weights dominate peak memory and a decode step. At a context length of 4K tokens, one 84 GB GPU fits 1,216 concurrent requests at ρ=0.9\rho = 0.9, against 130 with the full cache. The decode throughput at batch 128 rises from 1,499 to 6,070 tokens/s (Table D in our response to Reviewer fjNy). Fallback is evaluated as a paired simulation.

Table E: Error (%) when OCRBench requests are mixed into a POPE + MMStar workload (repeated from our response to Reviewer fjNy).

OCRBench share of the trafficUncompressedρ=0.9\rho = 0.9, no fallbackρ=0.9\rho = 0.9 + 20% fallbackUniform ρ=0.7\rho = 0.7
10%19.423.019.520.6
20%19.726.619.922.0
30%19.730.220.923.2
50%20.938.326.826.7
100% (OCRBench alone)22.257.043.733.7

(Qwen3-VL-4B, routing retention with the confidence trigger. The last two columns use the same decode-time KV budget of 0.3. POPE and MMStar have no CIFs at ρ=0.9\rho = 0.9.)

At this matched budget, the answer to your question depends on how concentrated the failures are. Fallback improves on ρ=0.9\rho = 0.9 in every mix, from 23.0% to 19.5% error at a 10% share and from 57.0% to 43.7% on OCRBench alone. Against uniform ρ=0.7\rho = 0.7, it gives lower error when OCRBench is 10-30% of the traffic. The two nearly tie at a 50% share, and on a single affected task uniform ρ=0.7\rho = 0.7 gives lower error (33.7% against 43.7% on OCRBench). Fallback also adds 0.2 prefill passes. Selective fallback therefore improves on a milder ratio when the affected requests are a minority of the traffic. The revision uses fallback to show that the per-request signal is actionable.

Thank you for the detailed rebuttal. The authors' responses have adequately addressed my main concerns, so I will maintain my rating accordingly.

AuthorsOpenReview ↗

Response to Reviewer jAKw

Thank you for reading our response and for confirming your assessment. Your caution about the original implementations led to Table 4 and to the experiment with FastV at its released placement (Table 8), and both are in the revised paper.

Reviewer_fjNyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies compression-induced failures (CIFs) in vision-language models: requests answered correctly by a reference configuration but incorrectly after compression. Paired outputs expose failures and reverse flips that average accuracy can conceal. The study focuses on post-prefill KV-cache eviction and compares failure sets across retention policies and compression adaptations. It predicts CIF risk and simulates selective fallback using paired outputs. Equal-budget token restoration tests whether discarded information can repair failures.

Critical strengths
  1. A useful sample-level evaluation perspective.

Showing that nearly equally accurate compression methods can break different requests identifies a relevant blind spot in average-accuracy reporting. The paired-transition accounting and answer-placement analysis help characterize the scope of this effect.

  1. Measurement is connected to prediction and intervention.

Confidence-based risk ranking and equal-budget restoration extend the analysis beyond counting failures. Random, spatial, and attention-based controls help distinguish additional token capacity from token identity. These results support the paper as an empirical measurement and diagnostic study.

Critical weaknesses
  1. Robustness to the correctness criterion needs clearer evidence.

For the same Qwen3-VL-8B DocVQA run, exact match yields 596 CIFs while the ANLS-threshold criterion yields 171. The paper discloses this distinction. However, the label definition changes the prediction target and restoration pool. A consistent reanalysis on the same paired outputs would clarify whether failure-set differences, risk ranking, and restoration conclusions remain stable under reasonable correctness criteria. The concern is robustness of the conclusions, rather than the existence of a single universal correctness rule.

  1. The incremental decision value of CIF supervision is not directly established.

Appendix D.2 already includes generic-error targets and CIF-specificity analyses, which provide useful evidence beyond generic error detection. The missing comparison is fallback decision quality after training on CIF labels versus generic compressed-output errors. Features, splits, and rerun coverage should be matched. Fallback is simulated and restoration is conditioned on known CIF sets. The instrumented stack is also slower than the reference, as disclosed, so these experiments do not establish net serving gains. These limitations constrain operational claims, but do not by themselves invalidate the measurement and diagnostic findings.

Overall assessment: I lean toward weak rejection. The measurement and diagnostic perspective is valuable, but the criterion sensitivity and incremental decision value remain important unresolved questions. I regard these concerns as potentially addressable rather than evidence of an irreparable flaw.

Questions for authors
  1. Please apply exact match and ANLS-threshold correctness consistently to the same DocVQA paired outputs. How do prediction, fallback, and restoration findings change?

  2. With identical features, splits, and rerun budgets, does CIF-label training improve fallback outcomes over generic-error-label training? Please report the recovery and reverse-flip trade-off directly.

  3. For deployment claims, please separate compression, detector, and rerun costs. Under what conditions is net benefit possible? High-concurrency serving measurements would strengthen this claim, but should be distinguished from validation of the paper’s measurement contribution.

5 comments
AuthorsOpenReview ↗

Response to Reviewer fjNy (Part 1 of 2)

Thank you for the precise questions. We ran each analysis you requested on the same paired outputs. Each correctness criterion leads to the same conclusions (Table A), and CIF labels give better fallback decisions than generic-error labels in most settings (Table B). Part 2 of this response separates the costs per request and under concurrency (Tables C to E). The details are in Appendices D.3, D.4, D.6, and D.7 of the revised PDF.

[W1, Q1] Consistent reanalysis on DocVQA under each criterion

We scored the same Qwen3-VL-8B DocVQA outputs under the ANLS-threshold criterion (normalized edit similarity ≥0.5\geq 0.5 to a reference answer) and under exact match. We then reran prediction, fallback, and restoration under each.

Table A: The same DocVQA outputs under each criterion.

Qwen3-VL-8B DocVQAANLS-thresholdExact match
CIFs171596
Positive flips2754
Confidence AUROC.866.880
CIFs recovered by fallback at 20% coverage76%74%
CIFs repaired at the 5% restoration budget, routing-score selection76.0%58.1%
Same budget, high-attention selection65.5%47.0%
Same budget, spatial selection56.1%38.3%
Same budget, random selection51.5%26.2%

(Full validation set of 5,349 questions.)

The criterion changes how many failures are counted and leaves the three findings unchanged. Exact match counts about 3.5 times as many CIFs. Confidence ranks them about equally well, and fallback at 20% coverage recovers about three quarters of them in either case. In restoration, the four selection policies keep the same order. Routing-score selection is first, and its lead over high-attention selection is significant in a paired sign test (p=.001p = .001 under the ANLS-threshold criterion and p<.001p < .001 under exact match). The identity of the restored tokens therefore matters under either criterion.

This reanalysis corrected one row. The submitted 8B DocVQA restoration row had scored repairs with ANLS inside the exact-match pool (561 of 596), and scored consistently it is 346 of 596. The order of the policies is unchanged. The 5% range in the text becomes 38-64%, and routing still repairs the most in every setting at 5%.

Other runs show the same stability. On TextVQA and VizWiz, requiring three matching annotators instead of one keeps confidence AUROC at .848-.884 and fallback recovery at 70-82%. On DocVQA, the method pairs that differ most under one criterion tend to differ most under the other (Spearman correlation of the exclusive-failure shares .71 on 4B and .73 on 8B).

[W2, Q2] CIF labels against generic-error labels for fallback

CIF labels repair more CIFs in four of the five settings below, at the cost of a few more positive flips. Features, folds, seeds, and rerun coverage are identical, and only the training label changes.

Table B: Fallback at 10% rerun coverage, with the trigger trained on CIF labels or on generic-error labels.

Model, datasetTraining labelCIFs repairedPositive flips lostError after fallback (%)
4B, ChartQACIF53.78.520.4
Generic error23.56.021.5
8B, ChartQACIF54.83.818.1
Generic error36.01.518.7
4B, VizWizCIF134.810.824.5
Generic error118.27.524.8
4B, AMBERCIF32.86.313.7
Generic error32.36.513.8
4B, TextVQACIF114.217.215.6
Generic error120.315.315.4

(Qwen3-VL-4B and 8B, confidence features, mean of six seeds. CIFs repaired and positive flips lost count the rerun requests. The error is simulated on the paired outputs.)

The label matters because a rerun can repair only a CIF. An error that the uncompressed model also makes stays wrong after the rerun, so a trigger trained on generic errors spends part of its reruns on such requests. The gain is accordingly largest on ChartQA, where 32% of the compressed-output errors are CIFs and the CIF label repairs about twice as many of them (53.7 against 23.5 on 4B). On AMBER, 88% of the errors are CIFs, and the two labels perform alike.

On the trade-off you asked about, the CIF label also reruns a few more positive flips (8.5 against 6.0 on 4B ChartQA), and each of these turns a correct answer into a wrong one. The net effect still favors the CIF label: the error after fallback is lower or equal in four of the five settings, and in all 11 model-dataset settings with confidence and routing features combined.

AuthorsOpenReview ↗

Response to Reviewer fjNy (Part 2 of 2)

[Q3] Separate costs and conditions for a net benefit

Tables C to E separate these costs. Eviction saves memory per request, and it saves decode time under concurrency.

Table C: Cost per request.

DatasetSettingPrefill (ms)Eviction after prefill (ms)Time per decode step (ms)Decode steps per requestTotal (ms)KV cache per request at decode (MiB)
ChartQAUncompressed95-443.525268
ChartQAρ=0.7\rho = 0.711550453.532520
ChartQAρ=0.9\rho = 0.912255523.43596.8
OCRBenchUncompressed104-4816.187160
OCRBenchρ=0.7\rho = 0.7128565316.21,06618
OCRBenchρ=0.9\rho = 0.9132585617.01,1716.0

(The 2,500 ChartQA and 1,000 OCRBench requests at batch size 1 on a single GPU. The means exclude the first request. The time per decode step includes decode-time eviction and loop overhead.)

At batch size 1, the gain from eviction is the smaller cache. Peak allocated GPU memory is 9.0 GiB in every setting against a cache of 68 MiB per request, so a decode step is dominated by the pass over the weights. The decode forward takes 44-47 ms with the full cache and 42-51 ms with 10-30% of it.

The extra time in Table C is the eviction itself in our instrumented stack. It adds 20-28 ms in prefill, 50-58 ms once after prefill, and 3-5 ms per decode step, which accumulates over the 16-17 steps of an OCRBench answer. The detector takes 0.25 ms per request on a CPU, and a rerun is one uncompressed pass for the flagged fraction.

Table D: Batched decode on one GPU at a context length of 4K tokens, by retained KV cache.

SettingKV cache per requestConcurrent requests that fitTokens/s at batch 1Tokens/s at batch 32Tokens/s at batch 128Tokens/s at the largest batch
Uncompressed580 MiB130571,1871,4991,432
ρ=0.7\rho = 0.7177 MiB424571,8113,8723,882
ρ=0.9\rho = 0.962 MiB1,216521,8426,0707,456

(One 84 GB GPU, Qwen3-VL-4B, stock Hugging Face decode loop, 32 greedy steps, median of three runs. Each request keeps the retained fraction of its prompt cache, so the table applies to any eviction method at that ratio. The eviction operation is timed separately below.)

The smaller cache pays off once many requests are in decode together, because the cache then fills the GPU and is read at every step. At ρ=0.9\rho = 0.9 the GPU fits 1,216 requests, against 130 with the full cache, and the decode throughput at batch 128 rises from 1,499 to 6,070 tokens/s. At batch size 1 the throughput is about the same at every cache size, as in Table C. The eviction operation is cheap at this scale. Written as batched tensor operations, a SnapKV-style selection and the compaction of the cache take 180 ms for these 128 requests at ρ=0.9\rho = 0.9, and the faster decode recovers this within three steps. Eviction follows prefill, so the time saving applies to the decode phase. Table D covers eviction alone.

Table E: Error (%) when OCRBench requests are mixed into a POPE + MMStar workload.

OCRBench share of the trafficUncompressedρ=0.9\rho = 0.9, no fallbackρ=0.9\rho = 0.9 + 20% fallbackUniform ρ=0.7\rho = 0.7
10%19.423.019.520.6
20%19.726.619.922.0
30%19.730.220.923.2
50%20.938.326.826.7
100% (OCRBench alone)22.257.043.733.7

(Qwen3-VL-4B, routing retention with the confidence trigger. The last two columns use the same decode-time KV budget of 0.3. POPE and MMStar have no CIFs at ρ=0.9\rho = 0.9.)

Fallback improves on ρ=0.9\rho = 0.9 in every mix, from 23.0% to 19.5% error at a 10% share and from 57.0% to 43.7% on OCRBench alone. Whether it also improves on uniform ρ=0.7\rho = 0.7 at the same KV budget depends on how concentrated the failures are. When OCRBench is 10-30% of the traffic, fallback gives lower error and stays within 1.2 points of the uncompressed model. The two nearly tie at a 50% share, and on a single affected task uniform ρ=0.7\rho = 0.7 gives lower error (33.7% against 43.7% on OCRBench). Fallback also adds 0.2 prefill passes. A net benefit is therefore possible when the affected requests are a minority of the traffic.

We agree that fallback is a paired simulation and that restoration is a diagnostic on known CIF sets. The revision states both and reports fallback as recovery at a given coverage, as a test of whether the signal is actionable.

AuthorsOpenReview ↗

Follow-up to Reviewer fjNy

Dear Reviewer fjNy,

Thank you again for your review. You named two questions as unresolved: (1) the sensitivity to the correctness criterion and (2) the incremental decision value of CIF labels.

We ran both analyses as you specified them:

  1. Correctness criterion. On the same Qwen3-VL-8B DocVQA outputs, the ANLS-threshold and exact-match criteria give confidence AUROC of .866 and .880. Fallback at 20% coverage recovers 76% and 74% of the CIFs, and the four restoration policies keep the same order (Table A).
  2. CIF labels. With identical features, folds, seeds, and rerun coverage, a trigger trained on CIF labels gives lower or equal error after fallback than one trained on generic-error labels in four of five settings (Table B).

Tables C to E separate the costs of eviction, detection, and reruns, and Table D reports batched decode under concurrency.

We would be grateful to know whether these results resolve your two questions. We are glad to provide any further analysis before the discussion period closes.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers and ACs

We thank the three reviewers and the Area Chairs for their specific questions. The reviews call the sample-level evaluation perspective useful (fjNy), the paper well motivated (jAKw), and the comparison between compressed runs a new angle (6jG9). Their questions led to the four changes below. The checks they asked for also corrected one adapted baseline and one restoration row, and the conclusions are unchanged. The new benchmark results use the full evaluation sets, and the revised PDF marks changed text in blue.

(1) The problem studied and the scope of the title [6jG9]

The paper studies which requests a KV-cache eviction breaks. The title now names KV-cache eviction, and the introduction defines a compression-induced failure (CIF) before it refers to Figure 1. New Figure 1b shows that failures nest along the eviction ratio: of the CIFs at ρ=0.5\rho = 0.5 to 0.80.8, 63-99% recur at ρ=0.9\rho = 0.9.

(2) One correctness criterion and the value of CIF labels [fjNy, jAKw]

Each correctness criterion leads to the same conclusions. On the same Qwen3-VL-8B DocVQA outputs, the ANLS-threshold and exact-match criteria give confidence AUROC of .866 and .880, and routing-score restoration repairs the most CIFs under both. At the same rerun coverage, a fallback trigger trained on CIF labels gives a lower or equal error than one trained on generic-error labels in four of five settings (Tables A and B in our response to fjNy).

(3) Findings under released implementation choices [jAKw, 6jG9]

We ran FastV at its released placement, with pruning inside prefill (Tables G and H in our response to jAKw). Failure sets overlap and nest there as well. The CIF rates differ, and POPE and MMStar register CIFs, as Section 5.3 implies. Our H2O adaptation now re-indexes its accumulated scores after every eviction. In Figure 1a, 18 of the 26 method pairs within 2 points have more than 5% exclusive failures (19 of 27 in the submission).

(4) Costs of eviction and fallback at a matched budget [All reviewers]

We separate the costs of eviction, detection, and reruns (Tables C to E in our response to fjNy). Eviction at ρ=0.9\rho = 0.9 reduces the decode-time KV cache of a request to one tenth. On our research stack it adds time at batch size 1, and at batch 128 it raises the decode throughput from 1,499 to 6,070 tokens/s (4K-token contexts). At the same KV budget, fallback gives higher error than uniformly milder eviction on a single affected task and lower error when most traffic is unaffected. The paper now uses fallback as a test of whether the per-request signal is actionable.

AuthorsOpenReview ↗

Summary of Changes in the Revised PDF

All changes are marked in blue. The table lists where each change is and which reviewer raised it.

LocationChangeRaised by
Title and abstractThe title names KV-cache eviction. The abstract defines a CIF as a negative flip under eviction. Its DocVQA example is now the same method pair on Qwen3-VL-8B (0.79 points, 18.3%).6jG9, jAKw
Introduction and Figure 1The introduction states one problem and defines a CIF before Figure 1. New panel (b) shows that failures nest along the eviction ratio. "Positive flips" replaces "reverse flips".6jG9
Section 5.2, Appendices A.3 and B.2Table 4 lists the behavior of each adapted method against its released code. The H2O adaptation re-indexes its accumulated scores after every eviction (18 of the 26 pairs in Figure 1a).jAKw
Section 5.3, Appendix C.2FastV at its released placement, with pruning inside prefill (Table 8).jAKw, 6jG9
Section 5.4, Appendices D.3 and D.4Results under each correctness criterion (Tables 12 and 13). CIF labels against generic-error labels (Table 14).fjNy, jAKw
Section 5.4, Appendices D.6 and D.7Fallback as a test of whether the per-request signal is actionable. Fallback against uniformly milder eviction (Table 17). Cost per request and batched decode (Tables 19 and 20).fjNy, jAKw, 6jG9
Section 5.5, Appendix EThe 8B DocVQA restoration row is scored consistently (346 of 596), and the range at the 5% budget becomes 38-64%.fjNy
Appendix A.2The correctness rule of each benchmark.jAKw
AuthorsOpenReview ↗

Summary of the Discussion by Authors

We thank the reviewers and the Area Chairs for the discussion. This comment records where each point stands and what the final version will change.

  • Reviewer jAKw wrote that our responses "have adequately addressed my main concerns".

  • Reviewer fjNy wrote that the reanalysis and the matched comparison "have addressed my concerns regarding Q1 and Q2". On Q3, the reviewer considers the deployment benefit "only partially established". We agree. Our reply to that comment measures the components of a request together at a fixed batch size (Tables K and L) and states that our benchmark requests lie outside the regime with a net time saving.

  • Reviewer 6jG9 wrote that our answers "have answered most of my concerns and questions" and finds the work "more informative and novel when focusing only on the first regime and motivation". The reviewer sees CIFs as "an informative metric for comparing the negative flips of two KV-eviction methods of similar accuracy, but not beyond that". Two points remain. Fallback is "too costly", although "there are use-cases where it would be justified". The extension to aggressive compression "has added a lot of noise to the paper", and the CIF count "becomes redundant with accuracy" as accuracy degrades. Our reply agrees that the cost is real and that the count is close to the accuracy drop at a high eviction ratio. In the latest comment, the reviewer writes "Although I remain uncomfortable increasing my score further, I do see real value and potential in this work", and suggests that we "focus solely on the no-degradation and low-degradation regimes and introduce a method that selects the best compression using EIFs as the criterion".

The final version will make three changes. Following the suggestion of Reviewer 6jG9 to rename the metric, it will be called an eviction-induced failure (EIF). The introduction and Section 5.4 will state that ρ=0.9\rho = 0.9 is the stress setting and that what the paired runs add at this ratio is the identity of the failing requests. The limitations of the timing measurements will replace the sentence of Section 6 on the serving benefit, and Tables K and L will be reported in Appendix D.7.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.