- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how vision-language models (VLMs) effectively utilize visual evidence acquired through zoom-in tools. The authors identify a limitation of existing evaluations: end-to-end answer accuracy cannot distinguish evidence acquisition failures from evidence utilization failures. To address this, they first propose a self-verified benchmarking protocol and construct the EvidenceZoom benchmark to systematically investigate the gap between acquiring useful visual evidence and utilizing it effectively.
Based on attention analyses, the authors further propose Head-Gated Attention Reallocation, a training-free method that adaptively shifts attention from generated reasoning toward zoomed-in visual evidence. Experiments across six datasets and five VLM backbones demonstrate consistent accuracy improvements.
Critical strengths
Strength 1: EvidenceZoom Benchmark
The proposed benchmarking protocol provides a useful framework for evaluating reasoning over self-verified visual evidence. It enables a more focused analysis of evidence utilization failures that are difficult to distinguish using answer accuracy alone.
Strength 2: Well-motivated and Effective Method
The proposed Head-Gated Attention Reallocation is well motivated by the attention diagnosis of reasoning–zoom imbalance and head-dependent focus. Its effectiveness is supported by consistent accuracy improvements across six datasets and five VLM backbones, with modest decoding overhead.
Critical weaknesses
Weakness 1: Reliability of Self-verification
The reliability of the self-verification protocol remains insufficiently established on challenging benchmarks. Although the authors validate self-verification using V*Bench, a constructed test set, and human evaluation, these analyses exclude UltraVR, which also provides localized evidence annotations (Table 5) and exhibits substantially lower answer accuracy.
Moreover, the protocol accepts partially useful evidence that may be insufficient to answer the question, leaving potential ambiguity between evidence acquisition and utilization failures. This limits the reliability of EvidenceZoom in isolating evidence utilization failures.
Weakness 2: Limited Comparison with Closely Related Attention Interventions
The external training-free baselines, ResDec and FLB, operate on output logits, whereas the proposed method directly reallocates attention toward zoomed-in evidence. Although Section 2 discusses prior visual-attention calibration and reallocation methods, these are not included in the experimental comparison.
This leaves the incremental benefit of the proposed zoom-specific intervention over existing attention-based approaches insufficiently established.
Questions for authors
Question 1: Self-verification on UltraVR
UltraVR provides localized evidence annotations (Table 5). Why are these annotations not used to evaluate self-verification? Could the authors report evidence overlap and false acceptance rates on UltraVR, particularly for self-verified trajectories that still produce incorrect answers?
Question 2: Disentangling Evidence Acquisition and Utilization
The paper attributes accuracy improvements to more effective utilization of zoomed-in evidence. However, the reported accuracy gains do not clearly distinguish two possible sources: (1) acquiring additional relevant regions through subsequent zoom-in calls, and (2) better utilizing already acquired visual evidence through Head-Gated Attention Reallocation.
Could the authors disentangle these effects, for example, by evaluating accuracy with fixed zoom evidence (without additional tool calls) and reporting how many corrected predictions involve newly acquired regions versus improved utilization of existing evidence?
Question 3: Comparison with Attention-based Methods
Could the authors compare their method with representative visual-attention calibration or reallocation methods discussed in Section 2, under the same post-zoom continuation setting? Such comparisons would help clarify the incremental benefit of the proposed intervention over existing attention-based approaches.