acceptodds
Withdrawn from ICLR 2027

AMR: Enhancing Visual Evidence Selection via Attention-Guided Multimodal Re-Reading

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) have advanced rapidly across vision-language tasks, yet their performance on visually grounded reasoning remains far from perfect. We investigate whether repeating the same image–question pair can address a limitation of causal, left-to-right processing, whereby visual evidence may be encountered before the model fully understands which information is relevant to the question. Repeating the complete multimodal input consistently improves inference, and an analysis of the repeated pass reveals a systematic redistribution of text-to-vision attention between its first and second reads. We refer to this read-to-read redistribution as *attention shift*. Rather than uniformly replaying the first allocation, the second read selectively increases attention to a subset of visual tokens, indicating that repetition can expose complementary visual evidence. Building on this observation, we propose *AMR* (Attention-Guided Multimodal Re-Reading), a training-free, plug-and-play intervention that leverages the model's own attention shift to identify and selectively reinforce visual evidence that becomes more relevant during the second read. Across six MLLMs and four multimodal reasoning benchmarks, AMR improves all 24 model–benchmark combinations over the zero-shot baseline, with an average gain of 2.61 percentage points. These results suggest that multimodal re-reading provides more than additional computation: it yields a state-dependent signal that can guide visual evidence selection at inference time, without parameter updates or task-specific training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal large language models, multimodal reasoning, visual attention, attention shift

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PYZjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes AMR (Attention-guided Multimodal Re-reading). AMR is a training-free, plug-and-play intervention method that leverages the model’s own attention shift to identify and selectively reinforce visual evidence. It provides the model with the same image–question pair repeated consecutively. The authors notice a read-to-read redistribution of visual attention and formalize it as an attention shift. Then, to identify visual evidence, AMR tracks how visual attention changes between the two reads of the same multimodal input, identifies candidate evidence whose relevance increases, and selectively reinforces that evidence during decoding. They report multiple experiments across six MLLMs and four multimodal reasoning benchmarks, obtaining consistent gains across model scales, architectures, and task types.

Critical strengths
  • AMR is a simple, training-free intervention that requires no parameter updates and is evaluated across three MLLM families and multiple model sizes.
  • AMR shows consistent improvements across six different MLLMs and four multimodal reasoning benchmarks. This breadth of evaluation supports the generality of the approach across architectures, scales, and task types.
Critical weaknesses
  • AMR claims that the attention shift can be exploited to identify visual evidence, but no direct evaluation of this process is provided beside the downstream answer accuracy on the benchmarks. A direct assessment of visual evidence quality would definitely be more useful to validate the ability of AMR to identify meaningful visual evidence. TreeBench provides traceable visual evidence annotations through bounding boxes, which could be used to directly assess whether the visual tokens selected by AMR align with the ground-truth evidence regions.
  • Several key components remain unexplored. Table 2 compares text-only, image-only, and complete multimodal repetition, while the supplementary analysis studies the intervention strength α. However, the paper does not evaluate the sensitivity to the attention layer, the ratio-based shift metric, the median-based selection threshold, or the logarithmic bias transformation. Although these choices are motivated in the method section, comparisons with simple alternatives would help establish their contribution. The number of repeated reads and the ordering of the image and question blocks also remain unexplored.
  • The comparison between the multimodal input repetition and the full AMR intervention is reported only for Qwen2.5-VL-7B. Since the reported ablation already shows that repetition alone accounts for a substantial fraction of the gains, evaluating Original vs. Repeat vs. AMR across all backbones for at least one model size would be important to establish the actual contribution of the proposed attention-guided mechanism.
Questions for authors
  • Could the authors directly evaluate the quality of AMR’s visual evidence selection? TreeBench provides bounding-box annotations for the relevant visual evidence that should be used when answering the question. For example, it would be informative to measure whether the selected visual tokens or the redistributed attention mass increasingly overlap with the annotated evidence regions from the first to the second read, and after applying the AMR intervention.
  • How does AMR interact with the visual attention sink phenomenon identified in prior work [1]? Kang et al. show that some visual tokens can systematically attract high attention despite containing no useful information to answer the question. Since AMR interprets read-to-read increases in visual attention as a proxy for increasing relevance, it is unclear whether some selected tokens may correspond to visual attention sinks rather than genuinely useful evidence. Figure 3 clearly shows some high attention visual tokens at the top left corner of the image that doesn’t contain any useful information for the question. Could the authors analyze the overlap between AMR-selected tokens and sink tokens, and evaluate whether explicitly filtering out sink tokens affects performance?

[1] Kang, Seil, et al. “See what you are told: Visual attention sink in large multimodal models.” International Conference on Learning Representations, 2025.

Reviewer_thDwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors observe that feeding the same image–question pair to an MLLM twice reliably improves reasoning. Closer analysis attributes this to a systematic redistribution of text-to-vision attention between the two reads — an "attention shift": the second pass does not uniformly replay the first allocation but selectively up-weights a subset of visual tokens, suggesting that re-reading surfaces complementary evidence. Building on this, AMR is proposed as a training-free, plug-and-play inference-time intervention: per-token attention ratios across the two reads are computed, unstable signals are filtered with a median threshold, and a log-ratio additive bias is applied to the key logits of the second visual block during decoding to reinforce the rising evidence. Across six MLLMs and four benchmarks, all 24 model–benchmark pairs improve, by +2.61 points on average.

Critical strengths
  1. An interesting phenomenon with solid mechanistic analysis. The attention-shift characterization explains why re-reading works at a mechanistic level; the method built on it is simple, architecture-agnostic, and training-free, and 24/24 positive outcomes attest to its robustness.
  2. Well-designed ablations. The RepeatText / RepeatImage / REPEAT / Full comparisons cleanly separate the contribution of mere repeated computation from attention-guided reinforcement, showing the gains are not just a byproduct of extra compute.
Critical weaknesses
  1. Questionable cost-effectiveness. The improvements are modest (~2.6 points) while the input roughly doubles and prefill cost nearly doubles with it; there is no comparison against test-time alternatives of equal budget, such as self-consistency voting or longer CoT, leaving the cost–benefit story incomplete.
  2. Attention as a relevance signal is theoretically contested. The pipeline embeds several heuristics (log-ratio, median threshold, strength α), and their sensitivity and failure modes — for instance, when the measured shift is merely noise — receive limited analysis.
  3. There are no test results on the most advanced open-source models, like Qwen3-VL.
Questions for authors
  1. Under a matched FLOPs budget, how does AMR fare against self-consistency or repeat-plus-majority-vote?
  2. How sensitive are the results to α and the median threshold τ, and must they be tuned per model?
  3. Is there a quantitative correlation between the magnitude of the attention shift and the answers actually being corrected?
Reviewer_imJKOpenReview ↗
Rating
1
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Attention-Guided Multimodal Re-Reading (AMR), a training-free method for improving visual reasoning in multimodal large language models. The method repeats the same image-question pair, measures attention shifts between the first and second reads, and selectively enhances visual tokens receiving increased attention. Experiments across six MLLMs and four benchmarks report an average improvement of 2.61 percentage points over the zero-shot baseline.

Critical strengths

Simple and training-free framework. AMR requires no additional training or parameter updates and can be applied to different MLLM architectures.

Consistent empirical improvements. The method achieves positive gains across all 24 model-benchmark combinations, demonstrating reasonable effectiveness across different models and tasks.

Critical weaknesses

Limited novelty compared with existing visual attention enhancement methods. Previous studies, such as MLLMs Know Where to Look, have already explored identifying and emphasizing important visual regions to improve multimodal reasoning. AMR follows a highly similar direction by using attention signals to select and reinforce visual evidence. The paper does not sufficiently clarify its fundamental methodological differences from these closely related approaches.

Incremental methodological contribution. The proposed method mainly combines input repetition with attention-based token reweighting, both of which have been explored in prior research. Computing attention ratios and adding biases to selected visual tokens appears to be a relatively straightforward extension rather than a substantial methodological innovation.

Limited improvements beyond simple repetition. A considerable portion of the reported gains comes from repeating the multimodal input rather than the proposed attention-guided intervention. For example, on MMBench, simple repetition achieves 79.30% accuracy, while the full AMR method reaches only 79.38%. This raises questions about the necessity and effectiveness of the additional attention mechanism.

Insufficient justification for attention-based evidence selection. The method assumes that visual tokens receiving increased attention during the second read are more relevant to the correct answer. However, attention shifts do not necessarily indicate reliable or causally important visual evidence. The limited qualitative analysis does not adequately validate this central assumption.

Insufficient comparisons and computational analysis. The paper lacks comprehensive comparisons with closely related visual attention steering and evidence enhancement methods. Moreover, the additional computational overhead introduced by repeated multimodal inputs and attention extraction is not sufficiently evaluated. Comparisons under matched inference budgets are necessary to establish the practical advantages of AMR.

Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.