acceptodds
Under review as a conference paper at ICLR 2027

Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoning

Abstract

We introduce Self-Saliency, a method for training Vision-Language Models (VLMs) to increase the alignment between their visual attention and the image regions mentioned in their reasoning. Self-Saliency uses a grounding model to localize the objects mentioned in each reasoning step and treats the resulting areas as supervision for the model's visual attention. Previous work on steering visual attention determines target image regions based solely on the image and question. In contrast, we show that conditioning the target regions on the model's generated reasoning improves downstream performance. For proper evaluation, we build a unified, broad suite of 25 visual reasoning benchmarks, where we reproduce the results of previous methods. We find that Self-Saliency significantly outperforms both prior attention-steering methods and baselines that ground image-level text, achieving both a better average rank, and better mean score. Post-training analysis shows that the model primarily adapts its reasoning text to existing attention patterns, producing shorter steps that refer to larger regions. Nevertheless, when controlling for generated text, attention to grounded regions increases significantly across the relevant layer. Finally, we identify a consistent geometric bias in VLM visual attention toward the image border. However, our ablations show that Self-Saliency's gains cannot be explained by simply aligning attention with the center of the image, highlighting the importance of aligning visual attention with the regions mentioned in the model's reasoning. We will publish our code and model weights upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual reasoning, Reinforcement learning, attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bc9nOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SELF-SALIENCY, a post-training method that aligns a vision-language model’s visual attention with image regions mentioned in its own generated reasoning. The method identifies observation steps, grounds them using an external grounding model, and incorporates an attention-alignment reward into GRPO training. Experiments on Qwen3-VL-8B across 25 benchmarks show the highest average score and best average rank among the evaluated methods. The paper also includes comparisons with prior attention-steering approaches, ablations of the grounding targets, and analyses of changes in reasoning text and attention patterns.

Critical strengths

++ An interesting approach to reasoning-conditioned attention supervision. Deriving supervision from the model’s own observation steps allows the target regions to follow its evolving reasoning. This offers a conceptually appealing alternative to fixed, question-level supervision and provides a useful direction for improving the connection between generated reasoning and visual evidence.

++ Extensive and transparent empirical evaluation. The paper evaluates 25 benchmarks under a common evaluation protocol, includes relevant baselines and ablations, and openly reports both improvements and regressions. This breadth and transparency are substantial strengths: they provide a more informative picture of the method’s behavior and make its overall benefits and limitations easier to assess.

Critical weaknesses

-- The discussion does not adequately reconcile the overall gains with benchmark-specific regressions. The comprehensive evaluation reveals results that qualify the paper’s broader narrative about improved visual reasoning. For example, HallusionBench performance decreases from 57.1 for No-Sal to 55.7 for SELF-SALIENCY, despite the method’s motivation of better aligning reasoning with visual evidence. DailyClue and several mathematical benchmarks also decline relative to No-Sal. The current discussion focuses primarily on improvements and provides little analysis of these trade-offs. Discussing the declining benchmarks, while distinguishing effects of general post-training from those associated with the saliency objective, would make the conclusions more complete and clarify the method’s scope of applicability.

-- Concrete failure-case analysis is missing. The qualitative examples illustrate successful corrections, but the paper does not examine cases where SELF-SALIENCY fails or turns a previously correct prediction into an incorrect one. Representative examples comparing the reasoning, grounded regions, and attention maps of SELF-SALIENCY and No-Sal would help identify whether failures involve inaccurate grounding, missed visual evidence, or reasoning errors. Such analysis should be included at least in the supplementary material to give readers a clearer understanding of the method’s practical limitations.

Questions for authors

See weakness

1 comment

We thank the reviewer for highlighting the paper’s strengths and for the insightful comments.

Benchmark-specific analysis

We agree that the benchmark-specific regressions deserve further analysis. We therefore examined failures on HallusionBench and DailyClue and identified a recurring pattern. In many cases, the model correctly interprets the visual evidence and reaches the correct conclusion in its reasoning. However, its final answer contradicts that conclusion or is missing. For example, the model may produce: So the answer should be yes. <Answer>No.</Answer>

These answer flips became more frequent after training. On HallusionBench, their frequency increased from 1.8% for the coldstart model (the checkpoint used to initialize training) to 10.3% for Self-Saliency. On DailyClue, it increased from 9.5% to 18.2%.

This finding highlights a distinction between errors in visual reasoning and errors in producing the final answer. We will add this analysis to the revised manuscript.

Concrete failure-case analysis

Following the reviewer’s suggestion, we have added a figure to the appendix showing several representative failure cases. These examples complement the successful cases presented in Figure 3 and illustrate practical limitations of our method.

Reviewer_u7ynOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SELF-SALIECNY, a RL method that improves VLM by aligning their visual attention with the image regions referenced in their own reasoning steps. SELF-SALIENCY uses a grounding model to identify objects mentioned during reasoning and rewards the VLM for attending to those regions

Critical strengths

S1. The paper suggests the RL training framework using the attention steering supervision.

Critical weaknesses

W1. The benchmark regression after the training

  • Even the model is trained with such amount of curated data, the trained model shows certain point of regression from the non-trained baseline.

W2. Lack of ablation for the saliency reward itself and the reward weight parameter.

W3. The statement of "SELF-SALIENCY is the first method to derive grounded supervision directly from the reasoning chain generated by the policy being trained" is not clear

  • Several works with "thinking with images" [1,2] uses the cropped image as their implicit grounding visual information at the reasoning stage. And [3] uses the cropped image as the explicit reward.

W4. Insufficient baseline models for comparison

  • Wide range of baseline models solving the similar tasks with similar approach (e.g., [4] aux vision encoder used at training, [1,2,3] crop based reasoning) are not being compared at the experiment [1,2,3,4]
[1] Zheng, Ziwei, et al. "Deepeyes: Incentivizing" thinking with images" via reinforcement learning." International Conference on Learning Representations. Vol. 2026. 2026.
[2] Lai, Xin, et al. "Mini-o3: Scaling up reasoning patterns and interaction turns for visual search." International Conference on Learning Representations. Vol. 2026. 2026.
[3] Lee, Jewon, et al. "Ergo: Efficient high-resolution visual understanding for vision-language models." International Conference on Learning Representations. Vol. 2026. 2026.
[4] Wang, Qixun, et al. "Monet: Reasoning in latent visual space beyond images and language (2025)." URL https://arxiv. org/abs/2511.21395.
Questions for authors

Q1. How does the training pipe work when the train data include high-resolution image (e.g., 1920x1080, 3840x2160)

  • Does grounding-dino model support dynamic resolution? or simply reduce the resolution to the supported fixed resolution. Simply reducing the resolution would hurt the training objective (as GroundingDino would miss the target) when the query image has small obj and text query asks for it.

Q2. What would be the model performance of αdirect=0,αsal=0\alpha_{direct} = 0, \alpha_{sal}=0 setting?

  • The performance gain from this setting could support the impact of saliency score at the training stage and isolate its contribution as core reward component.
2 comments

We thank the reviewer for the detailed and insightful comments. Regarding the requested saliency-reward ablation and the setting alpha_direct=alpha_sal=0:

  1. We report an ablation without the saliency reward in the No-Sal column of the main results table (Table 3).
  2. To isolate the saliency reward’s contribution, we set alpha_sal=0 and retain the other rewards. Setting both alpha_sal and alpha_direct to zero, as suggested by the reviewer, removes two reward components and therefore does not isolate the saliency reward. If the reviewer intended a different ablation, we would appreciate further clarification.

Regarding high-resolution images, Grounding-DINO supports variable image dimensions. Its standard preprocessing preserves the aspect ratio and targets a short side of 800 pixels, with a maximum long side of 1,333 pixels. It therefore accepts high-resolution source images, although preprocessing can reduce their resolution. As reported in Appendix A.2, our GRPO training resizes images to a maximum long side of 512 pixels. We agree that downsampling can obscure small objects and weaken the grounding supervision. We do not assume perfect localization. The saliency reward serves as an auxiliary signal alongside the answer-correctness rewards. Despite this limitation, Self-Saliency improves over No-Sal on MME-RealWorld (66.7 → 68.7), HRBench4K (77.4 → 79.5), and HRBench8K (72.0 → 74.5). These results show benefits on high-resolution benchmarks. They do not establish robustness to every small-object case. Grounding-DINO is used only during training, so its preprocessing does not constrain inference resolution.

Regarding the reviewer proposal for additional baselines. We thank the reviewer for these suggestions. Saliency-R1 and EASE are closer comparisons to our method because they directly supervise internal visual attention. DeepEyes learns to generate region coordinates and inspect the corresponding image crops during reasoning. Mini-o3 performs multi-turn visual search through repeated cropping and inspection. ERGO rewards crop selection based on whether the crop supports a correct answer, while also controlling its size. Monet uses auxiliary images during training to teach the model to generate latent visual representations as intermediate reasoning steps. These methods address related visual reasoning problems. However, they focus on acquiring visual evidence or changing the reasoning representation, making them less direct comparisons for our attention-supervision objective.

Thanks for answers from the authors. But as much of concerns (e.g., W1/W3) from the paper have not been resolved yet, I would like to keep my score.

Reviewer_WKVEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SELF-SALIENCY, a method that uses a grounding model to localize regions relevant to reasoning steps and uses them to supervise visual attention during RL training. The method is evaluated across 25 visual reasoning benchmarks and outperforms prior work and various baselines.

Critical strengths
  • Supervising visual attention with a grounding model to enhance reasoning is intuitive and makes sense.

  • Empirical results show that SELF-SALIENCY outperforms prior work and various baselines.

Critical weaknesses
  • The improvement over prior work, such as Saliency-R1 [1], is incremental.

  • The experiments are conducted only on Qwen3-VL-8B, so the method's generalization capability remains unclear.

Reference:

[1] Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward

Questions for authors

Does the proposed method generalize to different model sizes?

1 comment

We thank the reviewer for the insightful comments.

Improvement is incremental

We respectfully disagree that the improvement over Saliency-R1 is incremental, both in the method and in the results.

Our main methodological novelty is the source of grounding supervision. Saliency-R1 uses fixed, human-annotated bounding boxes. Our method derives boxes during training from the model’s own reasoning steps. This design tests the hypothesis that multiple grounding paths can lead to the correct answer. Human annotations identify specific image regions, but other regions may also support a correct answer. A model may already mention these regions in its reasoning without directing its attention toward them. Our method rewards alignment between these references and visual attention. A competing hypothesis is that a single predefined grounding path is sufficient. Our results support the value of grounding the model’s own reasoning, as we discuss below. This distinction is the central novelty of our method.

We agree that the overall mean accuracy gain is modest. However, our evaluation covers 25 benchmarks, a broader suite than those used in the recent studies we compare against. Across this suite, our method improves over Saliency-R1 on most benchmarks and achieves a higher mean score. We consider these gains meaningful evidence of the value of our approach.

Only used a single model

Our focus on the Qwen family follows a common practice in visual reasoning research. Both widely cited work [1,2,3], and closely related methods such as Saliency-R1 [4] and EASE [5], evaluate exclusively on Qwen models. Nevertheless, we acknowledge that our experiments do not establish generalization across model families or sizes. We are extending our evaluation to an additional model, but complete results are not yet available.

[1] Huang, Wenxuan, et al. "Vision-r1: Incentivizing reasoning capability in multimodal large language models." International Conference on Learning Representations. Vol. 2026. 2026. [2] Zhang, Jingyi, et al. "R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025. [3] Yang, Yi, et al. "R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization, 2025." URL https://arxiv. org/abs/2503.10615 54. [4] Gong, Shizhan, et al. "Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward." arXiv preprint arXiv:2604.04500 (2026). [5] Hu, Ruina, et al. "Attend to evidence: Evidence-anchored spatial attention supervision for multimodal RLVR." Findings of the Association for Computational Linguistics: EMNLP 2026. 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.