- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies whether VLM reasoning steps are more sensitive to the image regions they cite than to distinct regions cited by other steps. It changes each cited image region and measures how that affects the likelihood of the model producing the same reasoning step. When each eligible region pair is compared in both directions, the paper’s null model predicts that a step’s own region will win no more than half the comparisons on average. Qwen3-VL-4B and GLM-4.6V-Flash show own-region advantages, while InternVL3.5-4B's results are weaker and cohort-variable. A held-out study links the sensitivity margin to relative content changes under own-region versus sibling-region blur when regenerating the current step with preceding steps fixed.
I lean toward acceptance because the evaluation is useful and the main findings have good experimental support. A more direct localization comparison and better guidance on scoring-sensitive comparisons would strengthen its practical value. Clarifying the behavioral timeline would also help me assess the study's prospective design.
Critical strengths
-
The own-versus-sibling comparison is simple and meaningful. Using other regions from the same chain as controls is more informative than just showing that a perturbation affects the text. It needs no region annotations, and the authors are clear about its limits.
-
The evidence spans several cohorts, and for Qwen the advantage survives preregistered changes to the scoring context and intervention operator. The held-out generation test is a useful addition. Planned and exploratory analyses are clearly separated, and the documented protocol freezes and re-implementation checks strengthen the reproducibility record (Appendix J).
Critical weaknesses
-
The empirical relationship to localization remains underexplored. Appendix E uses best-object IoU, which need not establish overlap with the referent actually described by a step. The planned robustness-cohort localization breakdown was also not implemented (Table 12). On a subset of steps with an unambiguous object referent, comparing sensitivity with localization against that referent would help explain what the assay adds in practice, given Section 6's recommendation against reporting localization alone.
-
Individual results depend on how the score is calculated. Scoring only the content tokens changes whether the cited region wins in 13–22% of comparisons in the Qwen robustness cohort, even though the overall advantage remains (Table 6). This makes individual results harder to interpret. The authors rightly avoid treating one loss as evidence that a step is unfaithful. Looking at how the generated text changes in cases where the two scoring methods disagree would help clarify how to use these scores in practice.
Questions for authors
-
Appendix I (p. 24) mentions "earlier behavioral attempts" separately from the technical set and describes one pre-generation runtime abort. Were these attempts all part of the run that stopped before generating any output, or were there other runs? If additional runs produced outputs, when did this occur relative to the protocol freeze, and did those outputs inform the endpoint or analysis plan?
-
On the held-out behavioral cohort, how often do full-token and content-only scoring disagree on the win/non-win outcome? If there are enough cases, does either margin predict the relative content-change contrast within that subset? Reporting the subset size and uncertainty would help; Table 10's overall correlations do not resolve this question.
-
Could you include the fourth GQA cohort's fixed-text ordering estimate in Table 1, labeled descriptive: 0.557, 95% CI [0.498, 0.616] (Section 4.3; Appendix I)? It was not a prespecified replication, and its interval is compatible with the earlier estimates. Showing the four cohorts together would make the overall evidence easier to assess.