- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces VIGOR, an audit of VLMs’ use of visual evidence across five views: original images, evidence masking, off-evidence masking, evidence cropping, and question-only inputs. It combines label distributions, κ statistics, and intervention controls. Across 300 items from three benchmarks, MMStar shows relatively robust region dependence, BLINK-Twice’s masking gains are associated with response shifts, and the MMPerspective results remain inconclusive.
Critical strengths
-
The BLINK-Twice analysis offers practical value: an image-free question-polarity rule achieves 80% accuracy on the full benchmark, clearly exposing the risks of evaluating visual reasoning solely through final-answer accuracy.
-
The experiments include extensive controls covering removal styles, area-matched subsets, crop geometry, and independent re-annotation. Explicitly acknowledging the inconclusive MMPerspective results makes the analysis more credible than reporting intervention deltas alone.
Critical weaknesses
-
The central motivation involves a logical leap: improved accuracy after masking does not naturally imply “more faithful image use,” and the paper does not demonstrate that existing audits actually adopt this interpretation. Majority-class, blank-image, and text-only baselines already expose much of the BLINK-Twice issue; the full framework’s incremental value over these simpler diagnostics needs direct validation.
-
The framework lacks calibration against positive and negative controls with known mechanisms. κ only adjusts for label marginals, while region specificity establishes dependence on a region without distinguishing task-relevant evidence from shortcuts within it. Although the authors acknowledge that their requirements do not certify grounding, they have not shown that the framework reliably distinguishes different failure mechanisms.
-
The interventions remain incompletely matched: off-evidence masking uses multiple peripheral rectangles, whereas evidence masking uses a single large box. Even with equal areas, shape and spatial distribution are not matched. The feasibility rate of the low-overlap random-crop condition is also unreported. The three-axis controls therefore do not fully rule out intervention-induced confounds.
-
Evaluation settings may affect the central conclusions. Qwen inputs are capped at 384×384, and its processor neutralizes the zoom control. The appendix also reports LLaVA accuracy differences of up to nine percentage points across library versions. Without replication at native resolution and under official evaluation settings, model limitations are difficult to separate from failures introduced by input processing.
-
The re-annotation experiments contain an important reporting gap: in Table A15, the two auditors mark 27 and 25 MMPerspective items, respectively, as not localizable with a single box, yet Table A16 still reports intervention results on 49 items. It is unclear whether these cases use approximate boxes, fallback boxes, or another procedure, limiting the verifiability of the annotation-robustness conclusions.
Questions for authors
Please refer to Weaknesses.
Please review the authors’ responses and provide feedback
Dear Reviewers,
The authors have submitted their responses. Please review them carefully, assess whether they address your concerns, and leave a follow-up comment for the authors.
Thank you for your time and thoughtful feedback.
AC