acceptodds
Withdrawn from ICLR 2027

Lost in the Zoom: Diagnosing and Rebalancing Evidence Use in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) with zoom-in tools improve visual question answering, yet acquiring useful visual evidence does not ensure correct reasoning over it. To understand this, we propose a new protocol to analyze tool-use trajectories and reasoning. The protocol separates trace curation, tool execution, evidence verification, and answer evaluation, then reuses trajectories for continuation comparisons. Applied to VLMs with zoom-in tools across six datasets, the protocol yields the *EvidenceZoom* benchmark, comprising 35.4K trajectories covering 3.9K questions. Attention diagnosis over trajectories reveals that reasoning consistently dominates attention allocation and that zoomed-in images are overlooked, with attention head patterns showing substantial variation in their focus within the zoomed-in region. Based on these findings, we propose *Head-Gated Attention Reallocation*, a training-free method that corrects the zoom–reasoning attention imbalance, scaling each head's correction by its relative within-zoom focus. Across five VLMs from the GLM, Qwen, and Gemma series, our method improves the six-dataset mean accuracy over the best training-free baseline and even surpasses the training-based method on Qwen3-VL-8B. To summarize, our work highlights the gap between acquiring visual evidence and using it effectively, and suggests attention reallocation as a promising direction for improving visual reasoning with zoom-in tools.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision language models, Reasoning, Tool use, Benchmarking, Diagnosis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5m1HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how vision-language models (VLMs) effectively utilize visual evidence acquired through zoom-in tools. The authors identify a limitation of existing evaluations: end-to-end answer accuracy cannot distinguish evidence acquisition failures from evidence utilization failures. To address this, they first propose a self-verified benchmarking protocol and construct the EvidenceZoom benchmark to systematically investigate the gap between acquiring useful visual evidence and utilizing it effectively.

Based on attention analyses, the authors further propose Head-Gated Attention Reallocation, a training-free method that adaptively shifts attention from generated reasoning toward zoomed-in visual evidence. Experiments across six datasets and five VLM backbones demonstrate consistent accuracy improvements.

Critical strengths

Strength 1: EvidenceZoom Benchmark

The proposed benchmarking protocol provides a useful framework for evaluating reasoning over self-verified visual evidence. It enables a more focused analysis of evidence utilization failures that are difficult to distinguish using answer accuracy alone.

Strength 2: Well-motivated and Effective Method

The proposed Head-Gated Attention Reallocation is well motivated by the attention diagnosis of reasoning–zoom imbalance and head-dependent focus. Its effectiveness is supported by consistent accuracy improvements across six datasets and five VLM backbones, with modest decoding overhead.

Critical weaknesses

Weakness 1: Reliability of Self-verification

The reliability of the self-verification protocol remains insufficiently established on challenging benchmarks. Although the authors validate self-verification using V*Bench, a constructed test set, and human evaluation, these analyses exclude UltraVR, which also provides localized evidence annotations (Table 5) and exhibits substantially lower answer accuracy.

Moreover, the protocol accepts partially useful evidence that may be insufficient to answer the question, leaving potential ambiguity between evidence acquisition and utilization failures. This limits the reliability of EvidenceZoom in isolating evidence utilization failures.

Weakness 2: Limited Comparison with Closely Related Attention Interventions

The external training-free baselines, ResDec and FLB, operate on output logits, whereas the proposed method directly reallocates attention toward zoomed-in evidence. Although Section 2 discusses prior visual-attention calibration and reallocation methods, these are not included in the experimental comparison.

This leaves the incremental benefit of the proposed zoom-specific intervention over existing attention-based approaches insufficiently established.

Questions for authors

Question 1: Self-verification on UltraVR

UltraVR provides localized evidence annotations (Table 5). Why are these annotations not used to evaluate self-verification? Could the authors report evidence overlap and false acceptance rates on UltraVR, particularly for self-verified trajectories that still produce incorrect answers?

Question 2: Disentangling Evidence Acquisition and Utilization

The paper attributes accuracy improvements to more effective utilization of zoomed-in evidence. However, the reported accuracy gains do not clearly distinguish two possible sources: (1) acquiring additional relevant regions through subsequent zoom-in calls, and (2) better utilizing already acquired visual evidence through Head-Gated Attention Reallocation.

Could the authors disentangle these effects, for example, by evaluating accuracy with fixed zoom evidence (without additional tool calls) and reporting how many corrected predictions involve newly acquired regions versus improved utilization of existing evidence?

Question 3: Comparison with Attention-based Methods

Could the authors compare their method with representative visual-attention calibration or reallocation methods discussed in Section 2, under the same post-zoom continuation setting? Such comparisons would help clarify the incremental benefit of the proposed intervention over existing attention-based approaches.

Reviewer_f9EGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks why VLMs with a zoom-in tool still answer incorrectly after acquiring useful visual evidence. It builds EvidenceZoom, 35.4K self-verified zoom-in trajectories from six VLMs on six datasets, and a last-layer attention diagnosis (Eqs. 3-4) showing that reasoning tokens receive more attention than zoom tokens, that incorrect trajectories attend less to the zoom (Fig. 1d), and that heads differ stably in within-zoom focus (Fig. 3). It then proposes Head-Gated Attention Reallocation, a training-free last-layer logit shift toward zoom tokens, capped by Delta_cap and gated per head by relative focus (Eqs. 5-7). On five VLMs it improves six-dataset mean accuracy over ResDec and FLB by 0.65-1.85 pp (Table 2), beats an ungated variant (Table 3) and the training-based ZwZ-8B on Qwen3-VL-8B (Fig. 5), with about 3.5% per-token overhead.

Critical strengths
  • The acquisition-vs-utilization decomposition is a clear, relevant framing, and resuming decoding from a fixed post-zoom context gives a controlled, paired comparison across interventions that end-to-end accuracy cannot.
  • The diagnosis is methodologically careful: log-mean-exp avoids token-count confounds, within-question correct/incorrect pairing (Fig. 1d) controls for difficulty, and the swap experiment (Fig. 4) rules out a simple recency explanation.
  • The intervention is simple, uses a single Delta_cap shared across all models, needs no extra forward pass, and improves over Base in 29/30 model-dataset pairs with McNemar tests and bootstrap SDs; the Uniform ablation (Tables 3/8) isolates the head-gate contribution.
  • Self-verification reliability is examined from several angles (IoG contrasts in Table 7, 81.7% cross-attribute agreement, Fleiss kappa 0.737), and the timing protocol (A.5, D.3) is unusually rigorous.
Critical weaknesses
  • The benchmark does not cleanly isolate reasoning failures. The verification prompt (A.3) accepts "partial" evidence that "does not need to establish the final answer", so retained trajectories can still fail for acquisition reasons. Verifier quality is also uneven: Qwen3-VL-8B's positive/negative IoG contrast is only 0.182 on VBench (Table 7), and the human study covers only HR-Bench/VBench, not CV-Bench-2D, MMStar, or UltraVR, which make up ~70% of trajectories.
  • Because each model's retained set depends on its own verifier and sampling (Table 1: 439 CV-2D trajectories for Qwen3-VL-8B vs. 4,320 for GLM), accuracies are not comparable across models and new models cannot be scored on a fixed set. It is unclear how EvidenceZoom functions as a reusable benchmark rather than per-model trajectory collections.
  • Baselines are weak for an attention-intervention paper. ResDec and FLB are logit-level hallucination methods; the natural competitors are the attention-based visual reweighting methods the paper cites (attention-sink redistribution, cross-modal attention calibration, layer-wise reallocation) plus a plain constant boost to zoom-token logits. Without these, the claim that the diagnosis-motivated design drives the gains is not established.
  • The ZwZ-8B comparison (Fig. 5) is underspecified: how a different model is evaluated on trajectories generated by Qwen3-VL-8B, and whether "w/ zoom" uses the same retained contexts, is not explained.
  • Gains are modest and often within noise. Several per-dataset results fall below a baseline (GLM VStar: 87.99 vs. FLB 88.25; Qwen3.8 UltraVR 62.23 +/- 1.03 vs. ResDec 60.78 +/- 0.89, not significant), and significance is mainly reported for the mean. The equally weighted mean is swayed by UltraVR, where Qwen3-VL-8B's 5.28% Base accuracy suggests output-parsing problems rather than evidence use.
  • The diagnosis rests on 20 questions per setting (Sec. 3.2.3), single-round trajectories, and two models in Fig. 1(b)/(d), with no confidence intervals on the +0.226/-0.173 gap differences. The method is applied only at the last layer and only from the first zoom, with no sensitivity analysis for Delta_cap, layer choice, or reallocating from pre- vs. post-zoom reasoning separately.
  • Novelty is incremental: inference-time redistribution of attention toward visual tokens is well established, and the main additions are the tool-use setting and an entropy-based per-head gate.
Questions for authors

see weakness

Reviewer_gwyuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies failures after VLMs acquire enlarged image regions using a zoom tool. EvidenceZoom contains 35,412 trajectories covering 3,923 questions across six datasets and six collection models. The filter asks the same model, without its original reasoning or answer, whether a crop provides useful evidence; both correct and incorrect accepted trajectories are retained. Subsequent experiments resume generation from acquired prefixes, allowing matched comparisons of post-zoom continuations.

The analysis uses last-layer log-mean-exp attention scores for prompt, reasoning, and crop groups, and normalized entropy to measure each head's concentration within the crop. The proposed Head-Gated Attention Reallocation increases crop logits and decreases reasoning logits according to their score imbalance, scaled by each head's relative concentration. It requires no training or extra model pass. Across five intervention models, the method improves average accuracy over the training-free baselines, with supporting crop-retention, human-agreement, ablation, and timing studies.

Critical strengths

1. A controlled setting for studying post-zoom reasoning.

Reusing the same acquired prefixes across interventions holds the preceding tool interaction fixed, enabling a focused comparison of how models use the resulting evidence. Retaining both correct and incorrect self-accepted trajectories avoids restricting the benchmark to successful continuations. The collection across six datasets and six models provides a substantial basis for studying these failures.

2. Consistent improvements with a lightweight intervention.

The proposed method improves over the base model in 29 of 30 model–dataset settings and exceeds the strongest training-free baseline in the six-dataset average for each intervention model. It requires neither additional training nor an extra model pass. These results support its usefulness as a relatively inexpensive intervention for the evaluated continuation task.

3. Supporting validation and implementation analysis.

The crop-retention tests and human evaluation provide evidence that the filtering procedure identifies relevant visual information. The ablations examine components of the attention adjustment, while the timing study controls continuation tokens and distinguishes prefill from cached decoding. These analyses help assess both the benchmark construction and the implementation cost.

Critical weaknesses

1. Limited methodological novelty and comparisons with related interventions.

Prior work has addressed visual evidence use through attention redistribution, layer-wise reallocation, and attention calibration; OPERA also addresses over-reliance on generated tokens through decoding interventions. The main distinctions here are the post-zoom setting and entropy-based head gating. However, the main comparisons use ResDec and FLB, rather than the more closely related attention-based methods discussed in the paper. Evaluating Visual Attention Redistribution, Reallocating Attention Across Layers, and Cross-Modal Attention Calibration under the same zoom-based protocol would help establish the contribution beyond adapting attention intervention to this setting. [1–4]

2. Insufficient validation of the head-gating hypothesis.

The gate assumes that heads with more concentrated attention within the crop are better suited to using visual evidence. Concentration, however, does not establish task relevance: a head may focus on an irrelevant detail, while distributed attention may be necessary for counting or spatial relations. The uniform-reallocation ablation does not directly validate this assumption and also changes correction strength. Relating head attention to independently annotated evidence regions, together with strength-matched controls, would provide a stronger justification for the gating mechanism.

3. Self-verification does not establish evidence sufficiency.

The filter relies on the VLM's own judgment of crop usefulness and explicitly permits partial evidence. An incorrect retained continuation may therefore reflect incomplete acquisition rather than failure to use sufficient evidence. Human agreement and target-retention analyses support relevance more directly than answer sufficiency. Model-specific verification may also introduce different acceptance criteria across collection models. An independently verified, answer-sufficient subset with a common annotation standard would better support the claimed separation of evidence acquisition and utilization.

4. Limited evidence of end-to-end tool-use improvements.

The intervention experiments resume generation from saved prefixes after the first zoom, restricted to trajectories that pass self-verification. This controls the continuation comparison but does not establish an improvement over the full question set, where reasoning, crop selection, and subsequent tool calls interact. An end-to-end evaluation without conditioning on self-verification would better establish the method's practical effectiveness and total tool-use cost.

References

[1] Qidong Huang et al. OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation. CVPR 2024. https://arxiv.org/abs/2311.17911

[2] Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See What You Are Told: Visual Attention Sink in Large Multimodal Models. ICLR 2025. https://arxiv.org/abs/2503.03321

[3] Haolang Lu et al. Reallocating Attention Across Layers to Reduce Multimodal Hallucination. CVPR 2026. https://openaccess.thecvf.com/content/CVPR2026/papers/Lu_Reallocating_Attention_Across_Layers_to_Reduce_Multimodal_Hallucination_CVPR_2026_paper.pdf

[4] Jiaming Li et al. Cross-Modal Attention Calibration for LVLM Hallucination Mitigation. CVPR 2026. https://openaccess.thecvf.com/content/CVPR2026/papers/Li_Cross-Modal_Attention_Calibration_for_LVLM_Hallucination_Mitigation_CVPR_2026_paper.pdf

Questions for authors
  1. How does the method compare with closely related visual-attention interventions under the same post-zoom evaluation protocol?

  2. Does attention concentration predict attention to task-relevant evidence, and does the proposed gate outperform strength-matched alternatives?

  3. Do the findings hold on an independently verified, answer-sufficient subset using a common annotation standard across collection models?

  4. Does the intervention improve end-to-end accuracy on the original questions without conditioning on self-verification, and at what total tool-use cost?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.