acceptodds
Under review as a conference paper at ICLR 2027

VIGOR: Visual Intervention-Grounded Evaluation for Observation and Reasoning

Abstract

Visual reasoning benchmarks are read as evidence that a vision-language model (VLM) answers from the image, yet final-answer accuracy cannot show whether the image was used. Intervening on the image and reading the change in accuracy is the usual remedy, but an accuracy change can reflect a shift in the model's response prior as well as the removal of task-relevant content. We ask when such a delta supports a claim that the intended evidence was used. We introduce VIGOR, an audit that evaluates each item under five views and summarizes them as a validity profile, and VIGOR-Stress300, an evidence-annotated subset of MMStar, BLINK-Twice, and MMPerspective. Across open VLMs from three families, masking the annotated evidence lowers accuracy on MMStar and raises it on BLINK-Twice; read on its own, the gain would be taken as improved use of the image. Two requirements make the deltas interpretable. Under score legibility, a panel of option structure, label marginal, and answers without evidence shows that the BLINK-Twice gain is a response shift toward what the models answer for an occluded image, that a text-only polarity rule outscores every masked view, and that original-view predictions show no positive chance-adjusted agreement with the labels. Under matched intervention, MMStar's degradation survives label adjustment, six removal styles, location- and magnification-matched controls, and boxes drawn by independent auditors. On MMPerspective the tested interventions leave a small, model-dependent effect unresolved. We recommend that intervention-based audits report the legibility panel and the matched controls with every delta.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, visual reasoning, benchmark validity, evaluation methodology, image interventions, language priors

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9JovOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VIGOR, an audit of VLMs’ use of visual evidence across five views: original images, evidence masking, off-evidence masking, evidence cropping, and question-only inputs. It combines label distributions, κ statistics, and intervention controls. Across 300 items from three benchmarks, MMStar shows relatively robust region dependence, BLINK-Twice’s masking gains are associated with response shifts, and the MMPerspective results remain inconclusive.

Critical strengths
  • The BLINK-Twice analysis offers practical value: an image-free question-polarity rule achieves 80% accuracy on the full benchmark, clearly exposing the risks of evaluating visual reasoning solely through final-answer accuracy.

  • The experiments include extensive controls covering removal styles, area-matched subsets, crop geometry, and independent re-annotation. Explicitly acknowledging the inconclusive MMPerspective results makes the analysis more credible than reporting intervention deltas alone.

Critical weaknesses
  • The central motivation involves a logical leap: improved accuracy after masking does not naturally imply “more faithful image use,” and the paper does not demonstrate that existing audits actually adopt this interpretation. Majority-class, blank-image, and text-only baselines already expose much of the BLINK-Twice issue; the full framework’s incremental value over these simpler diagnostics needs direct validation.

  • The framework lacks calibration against positive and negative controls with known mechanisms. κ only adjusts for label marginals, while region specificity establishes dependence on a region without distinguishing task-relevant evidence from shortcuts within it. Although the authors acknowledge that their requirements do not certify grounding, they have not shown that the framework reliably distinguishes different failure mechanisms.

  • The interventions remain incompletely matched: off-evidence masking uses multiple peripheral rectangles, whereas evidence masking uses a single large box. Even with equal areas, shape and spatial distribution are not matched. The feasibility rate of the low-overlap random-crop condition is also unreported. The three-axis controls therefore do not fully rule out intervention-induced confounds.

  • Evaluation settings may affect the central conclusions. Qwen inputs are capped at 384×384, and its processor neutralizes the zoom control. The appendix also reports LLaVA accuracy differences of up to nine percentage points across library versions. Without replication at native resolution and under official evaluation settings, model limitations are difficult to separate from failures introduced by input processing.

  • The re-annotation experiments contain an important reporting gap: in Table A15, the two auditors mark 27 and 25 MMPerspective items, respectively, as not localizable with a single box, yet Table A16 still reports intervention results on 49 items. It is unclear whether these cases use approximate boxes, fallback boxes, or another procedure, limiting the verifiability of the annotation-robustness conclusions.

Questions for authors

Please refer to Weaknesses.

Reviewer_zwr9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduces VIGOR, an evaluation framework that assesses visual grounding beyond final-answer accuracy by examining model behavior across five evaluation views and summarizing the results as validity profiles. Experiments on three benchmarks (MMStar, BLINK-Twice, and MMPerspective) reveal different patterns: evidence masking reduces accuracy on MMStar, increases accuracy on BLINK-Twice due to response shifts, and produces inconclusive effects on MMPerspective. Based on these findings, the authors argue that intervention-based evaluations should incorporate score-legibility analysis and matched intervention controls to distinguish genuine evidence dependence from response biases and intervention artifacts.

Critical strengths
  1. The authors conduct systematic control experiments across masking locations, removal styles, crop geometries, and independent evidence annotations. The consistent findings across multiple VLM families, particularly on MMStar and BLINK-Twice, provide evidence for the robustness of the observed intervention effects.
Critical weaknesses
  1. The methodological novelty of VIGOR remains unclear, as the framework mainly combines existing evaluation metrics and intervention controls. Moreover, the three observations (§4.2) are each tied to a specific benchmark, making it difficult to identify a unified insight or generalizable diagnostic principle. It is unclear what broader understanding these observations provide beyond explaining benchmark-specific behaviors. Additional experiments evaluating whether VIGOR can consistently distinguish these diagnostic patterns on unseen benchmarks, along with an ablation demonstrating its added value over existing evaluation methods, would strengthen the claimed contribution.
  2. A central motivation of this paper is that accuracy changes cannot establish whether models faithfully use the intended visual evidence. However, VIGOR primarily measures region dependence through accuracy-based interventions and statistical controls, without independently validating whether its diagnoses reflect faithful evidence use. Furthermore, the reliance on a single evidence box is particularly problematic for MMPerspective, where most samples require global-layout information (§3.3). Additional evaluation on controlled tasks with known evidence-use mechanisms, including models relying on spurious visual shortcuts, as well as experiments involving spatial relationships and multiple evidence regions, would help clarify the reliability and scope of VIGOR's diagnostic capabilities.
Questions for authors

Q1. Dataset Selection Reason. Why did the authors restrict the evaluation to 100 samples each from MMStar and MMPerspective rather than using the full benchmarks? Could the authors clarify the sampling criteria and how representative these subsets are of the original benchmarks?

Q2. Consistency of Diagnostic Interpretation. While §5 provides qualitative interpretations of the three observed patterns, what explicit criteria determine how VIGOR assigns these diagnoses to a new benchmark? In particular, how can the authors ensure consistent diagnostic interpretations rather than relying on benchmark-specific, post-hoc explanations?

Reviewer_mhPAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

VIGOR is an evaluation-methodology paper arguing that a positive intervention delta (masking evidence lowers accuracy) is not sufficient evidence of "evidence use": the delta can be a response shift (e.g., toward the majority label or the occluded-image answer). It proposes two requirements - a score-legibility panel (option structure, gold marginal, kappa, balanced accuracy, no-evidence answers) and matched interventions (location/style/geometry-matched off-evidence controls with realizability checks) - and instantiates them on VIGOR-Stress300 (300 items from MMStar, BLINK-Twice, MMPerspective; 4 VLMs x 21 views). Main findings: MMStar shows the genuine evidence-use signature (masking hurts via harmful flips, ES_e 0.24-0.29, crop restores); BLINK-Twice's large masked gain (up to +0.47) is re-read as a response shift toward occluded-image answers (original kappa = -0.14, masked kappa = +0.26, constant-No predicts 0.71; masked predictions agree with blank-image predictions 78-91%); MMPerspective's small effects are mostly non-significant after Holm correction.

Critical strengths
  1. Sharp, important methodological thesis: intervention deltas conflate evidence use with response shift; the paper demonstrates this concretely with a striking reversal on BLINK-Twice, a widely used benchmark.2. The matched-intervention discipline is rigorous: same-area off-evidence controls, matched crop magnification, and - unusually honestly - reporting that realizability fails on 68/46/82 of 100 items, exposing why naive location controls are invalid.
Critical weaknesses
  1. The dataset (300 items, 100 per source) is a stress profile, not a full-benchmark replacement; the paper itself states it does not estimate full-benchmark accuracy - the practical impact therefore depends on whether authors of MMStar/BLINK adopt the protocol, which is unverified.2. Only four open VLMs, one prompt/decoding setting, and binary/4-option MCQ formats are tested; generalization to open-ended VLQA (MM-Vet style) is not shown in the read portion.
Questions for authors
  1. For BLINK-Twice, is the "response shift" interpretation testable by further controls (e.g., asking the model to justify answers, or removing the question entirely)? What mechanistic account (attention on the masked region? no-evidence default?) does the paper propose? 2. How were the 100 items per source sampled (stratification, item-type balance), and what are the selection criteria to keep the stress profile representative? 3. Does the legibility panel change conclusions on any existing published benchmark claims beyond BLINK-Twice?

Comments on the paper

1 comment
Area_Chair_pidrOpenReview ↗

Please review the authors’ responses and provide feedback

Dear Reviewers,

The authors have submitted their responses. Please review them carefully, assess whether they address your concerns, and leave a follow-up comment for the authors.

Thank you for your time and thoughtful feedback.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.