acceptodds
Under review as a conference paper at ICLR 2027

FIRA: Fixed-Support Intervention and Readout Audit for Visual Token Pruning

Abstract

A stable visual-token ranking can coexist with failures shared by the compared selectors. Fixed-Support Intervention and Readout Audit (FIRA) holds selected positions fixed while recording actual outputs and candidate-restricted decisions. On 800 A-OKVQA images, changing the readout reduces historical LLaVA delete–zero ranking interactions by 19.625 points in NF4 and 18.625 in FP16. Qwen2.5- VL with MMTok/DivPrune supports instead exhibits shared failures that largely cancel in selector comparisons. Native OCRBench-v2 text-image choices provide 536 questions and seven complete arms. On 429 confirmation questions, full-input accuracy is 68.30%, the shared excluded-correct shift is 19.35 [15.81,23.10] points and the differential shift is 3.73 [-1.18,8.51], with 97.5% paired-image intervals. The task meets its frozen difficulty criterion. Another 7,504 decisions confirm positive shared shifts at 25/75% retention. Saved traces from 194 receipts reveal different multi-token candidate decisions in 59/970 Qwen and 116/970 Monkey OCR cells. These results motivate reporting absolute answer outcomes alongside relative rankings under explicit answer contracts.

Primary area:
interpretability and explainable AI
Keywords:
Visual Token Importance, Vision-Language Models, Ablation Evaluation, Interpretability, OCR

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LJ9pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how visual-token pruning comparisons depend on the intervention and the rule used to score answers. It fixes each selector's retained positions and compares actual outputs with choices restricted to an allowed answer set. An excluded-correct event occurs when the restricted choice is correct but the actual answer falls outside the allowed set. J compares the selectors’ zero-minus-delete changes in these events, while L averages those changes. LLaVA shows substantial readout sensitivity, while Qwen often shows shared failures despite small changes in the selector comparison. A separate analysis of receipt traces compares free generation, token-by-token selection within a candidate trie, and whole-string probability ranking.

The experiments are carefully done, and I find the reporting approach useful. I lean toward rejection because the clearest single-token results come from cases where zeroing causes the model to produce mostly invalid answers. Outside this setting, I am less convinced that the choice of readout meaningfully changes how we compare selectors. I could move toward acceptance if the receipt analysis shows a reliable, practically meaningful change in the selector comparison, or the chance comparison clarifies what the exclusion measures add beyond accuracy and valid-output rates.

Critical strengths
  1. The authors explain their experimental controls clearly. They keep the same token positions across interventions, include invalid outputs in the scores, and preserve the paired comparisons when estimating uncertainty. They also explain when masking and deletion should give the same result and report differences caused by numerical precision. I appreciate that they disclose both the TextVQA task being easier than planned and the implementation fix in the native-choice study. These checks and disclosures give me more confidence in the reported comparisons.

  2. The breakdown helps explain why the scores change. For LLaVA, Table 2 shows how the comparison between selectors depends on how answers are chosen. The receipt analysis also distinguishes choosing an allowed answer one token at a time from choosing the most likely complete answer. The authors check these reconstructed decisions against saved traces and make clear that a correct restricted choice does not fix the answer the model originally generated. This makes it easier to interpret what the readout differences actually show.

Critical weaknesses
  1. I am not yet convinced how much the audit adds in practice for single-token answers. On the OCRBench-v2 confirmation subset, zeroing gives 1.40% accuracy for actual outputs and 20.75% when choices are restricted to allowed answers, giving the reported 19.35-point shared shift (Table 29). Section 5.6 shows that the restricted accuracy is compatible with the 21.80% expected from uniform guessing. The gap is measured correctly, but both ways of choosing answers still perform poorly. What does reporting this gap tell us about the selectors beyond their accuracy and how often they produce valid answers? I would also make the chance comparison more prominent when presenting the headline result.

  2. The evidence is less clear when we look beyond zeroing. The historical LLaVA panel has no excluded-correct events under mean replacement (Table 5; Appendix L.2), but the newer Qwen single-token experiments do not include this operator. The older Qwen TextVQA experiments do, but they intervene at a different point in the model and score complete answers (Appendix AF). The receipt study helps extend the analysis beyond single-token answers, although both planned interaction tests have intervals that include zero (Table 31). One result worth looking at more closely is MonkeyOCR under deletion. High-norm selection scores about 9.80 points above DivPrune with exact-match scoring, but 0.51 points below it when answers are chosen through the trie (Table 32). Without uncertainty estimates for these gaps and their difference, it is hard to tell how much weight to put on this change.

Questions for authors

Questions 1 and 2 could change my rating. Question 3 concerns a limited additional experiment that would help assess how broadly the findings apply.

  1. How do the observed exclusion effects compare with what would be expected from uniform guessing among the allowed answers when the actual output is invalid? Would this comparison change how you interpret J and L?

  2. How reliable is the roughly 10.31-point change in the high-norm versus DivPrune gap between exact match and trie decisions under deletion (Table 32)? A paired bootstrap interval for this change, resampling receipt groups, would help assess whether the readout meaningfully changes the MonkeyOCR comparison.

  3. Does the absence of excluded-correct events under mean replacement in LLaVA also hold in the newer Qwen single-token setting?

Reviewer_GLamOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies if the current evaluation methods for visual token pruning reflect model performance. The paper prsents FIRA, a framework that compares different ways of removing ore replacing visual tokens while keeping the selected tokens fixed. examine how different answer-scoring methods affect the evaluation results. Through experiments on several VLMs and datasets, including A-OKVQA, TextVQA, OCRBench-v2, and receipt-reading tasks, the authors show that two pruning methods can appear to perform similarly even when both suffer substantial performance losses. The paper highlights the importance of examining actual generated answers and shared failures, rather than relying only on relative comparisons between pruning methods.

Critical strengths

The paper addresses an important practical problem in evaluating visual token pruning methods. I particularly liked the finding that two methods can appear equally effective while both have significant failures that are hidden in their relative comparison. This is a useful observation that could help researchers avoid misleading conclusions.

Critical weaknesses

My main concern is the novelty and significance of the contribution. It is already known that different token removal methods and answer-scoring rules can affect evaluation results. While FIRA combines these factors in a systematic way, I am not fully convinced that it provides a sufficiently new insight beyond demonstrating their combined effects. The paper also includes many experiments with different models and evaluation settings, making the central contribution somewhat difficult to follow. More importantly, although the authors show that current evaluations can be misleading, they do not clearly demonstrate how FIRA would change practical decisions about which pruning method to use or how to improve pruning performance. I would like to see a concrete example where FIRA leads to a different and better-supported conclusion about a pruning method. I also see three presentation problems:

  1. The motivation is not explained intuitively. The authors introduce technical definitions and statistical comparisons before clearly demonstrating the practical problem with a simple example.
  2. There are too many experimental directions. The paper moves between pruning operators, scoring rules, model families, datasets, and different statistical measures. Although these experiments are relevant, their relationships are not always easy to follow.
  3. The central contribution gets buried in technical details. It is difficult to distinguish which findings are genuinely new, which confirm known evaluation problems, and which are additional validation.
Questions for authors

I understand that FIRA can reveal shared failures that are hidden in relative comparisons. However, I am less clear about how these findings would influence the development or selection of pruning methods. Could the authors provide a concrete example where FIRA leads to a different conclusion about which pruning method is better or more reliable?

Reviewer_o2ftOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FIRA, a framework for evaluating visual token pruning in VLMs. The main idea is to keep the tokens selected by each method fixed while varying how the remaining tokens are handled (zeroing, mean replacement, masking, or deletion) and how the model's answers are evaluated (actual outputs versus candidate-restricted decisions). The authors use paired measures to distinguish differences between selectors from failures shared by both selectors, which may be hidden when only relative rankings are considered. Across A-OKVQA, TextVQA-derived four-choice tasks, native OCRBench-v2 choices, and receipt extraction, the authors show that comparing selectors via ranking interactions can hide substantial shared failures and argue for reporting absolute outcomes and explicit answer contracts alongside relative rankings.

Critical strengths

The paper addresses an important issue in evaluating visual-token pruning by showing that relative comparisons between selectors can hide substantial failures shared by both methods. The experiments cover multiple models, selectors, and datasets. The additional evaluation on native OCRBench-v2 and across different token-retention budgets provides further support for the findings.

Critical weaknesses

The main findings are largely driven by zero replacement, which causes a substantial drop in performance. It remains unclear whether similar shared failures would occur under less disruptive interventions. Most experiments intervene at layer 2, where visual and textual information have already interacted. Moreover, MMTok and DivPrune are evaluated under a common audit setup rather than their native pruning settings. It is unclear whether the findings would hold at earlier layers or under the methods' original implementations.

Questions for authors

Most of the analysis focuses on single-token multiple-choice decisions. Do the authors expect the shared-failure findings to hold for open-ended generation tasks beyond the receipt experiments? In the Qwen experiments, a large part of the deletion–zero accuracy gap comes from candidate-decision loss rather than excluded-correct outputs. Could the authors provide more insight into what causes this difference?

2 comments
AuthorsOpenReview ↗

Response to reviewer o2ft (Part 2/2)

(Part 2/2 — Continued Response to Reviewer o2ft)

3. Why Candidate-Decision Loss Is Large in Qwen Deletion-vs.-Zero Comparisons (Addressing Q2)

On the 429 OCRBench-v2 confirmation questions at Site 2, the 66.55 pp66.55\text{ pp} actual accuracy drop from deletion (67.9567.95\\%) to zeroing (1.401.40\\%) partitions into 47.20 pp47.20\text{ pp} of candidate-decision loss (QQ drops from 67.9567.95\\% to 20.7520.75\\%) plus 19.35 pp19.35\text{ pp} of excluded-correct loss (RR) (and on A-OKVQA, 86.88 pp=61.63 pp+25.25 pp86.88\text{ pp} = 61.63\text{ pp} + 25.25\text{ pp}). Our probability-mass and chance-decomposition diagnostics explain why both components occur together:

  1. Collapse Out of the Option Space (R=19.35 ppR = 19.35\text{ pp}): Layer-2 zeroing reduces the total probability mass assigned to all valid option letters from 99.6999.69\\% (under deletion) down to 1.861.86\\% (under zeroing), causing valid-output eligibility EE to drop to 2.562.56\\% (MMTok) and 5.595.59\\% (DivPrune).
  2. Collapse Within the Option Space (QQ Drops to Chance): When the model is forced to pick among the allowed option letters despite putting 98.1498.14\\% of its probability mass on non-option tokens, its conditional choice within the option set collapses to a nearly image-blind letter bias (the modal candidate letter C is chosen on 48.9548.95\\% / 47.0947.09\\% of questions vs. 25.4125.41\\% / 25.8725.87\\% under deletion). As a result, restricted candidate accuracy Q=20.75Q = 20.75\\% falls right to the uniform-guessing expectation (21.8021.80\\%), decomposing R=19.35 ppR = 19.35\text{ pp} into 20.98 pp20.98\text{ pp} of uniform-chance credit and −1.63 pp-1.63\text{ pp} of above-chance excess. Thus, zeroing simultaneously destroys within-set visual discrimination (producing the 47.20 pp47.20\text{ pp} candidate loss) and valid-token emission (producing the 19.35 pp19.35\text{ pp} exclusion shift).

4. Scope for Open-Ended Generation Tasks Beyond Receipts (Addressing Q1)

Whenever an open-ended or semi-structured VLM task is evaluated by comparing unconstrained generation against a finite candidate set, rubric, or constrained vocabulary (such as entity extraction, key-value OCR, classification-by-generation, or multiple-choice reasoning), the exact prefix-trie identity A=EQA = E Q applies and shared/differential exclusion shifts can arise under physical deletion—as demonstrated by the 19.0719.07\\%\text{--}29.38\\% exclusion rates on WildReceipt. For unconstrained free-form description tasks scored by continuous similarity without a discrete candidate set, single-step exclusion does not apply directly; instead, FIRA's broader principle—reporting absolute task outcomes and valid-format compliance alongside relative selector differences—remains essential. We have made this boundary explicit in Section 6.

We thank you again for your supportive review and hope these new native experiments and analyses further reinforce your confidence in the paper.

Thank you for submitting the author response. I acknowledged it and most of my concerns are resolved, and I would like to keep the current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.