acceptodds
Under review as a conference paper at ICLR 2027

Learning to Look: A Foundation for Visual Understanding

Abstract

Vision-language models (VLMs) suffer from visual hallucination, such as saying an object that is not present in an image. We ask a more fundamental question: whether a VLM is even aware of where it is looking before understanding the semantic meaning in the image. We formulate this question as a task: identifying which parts of its visual input were available to the model. We construct the task in a self-supervised way without manually labeling the available region to the model. Our experiments show that existing VLMs fail to localize the visible region. We further show that fine-tuning VLMs on the proposed task improves a model's awareness of visual input.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-language model, Visual understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_igMXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the LOOK task for improving visual grounding. The constructed LOOK training data is employed to fine-tune pre-trained VLMs through self-supervision to localize the crop region in images using global positional information. Furthermore, the VisABench is designed to evaluate responses when visual content is withheld. Experimental results yielded on Qwen3-VL show improved crop localization and fewer false object-presence under the benchmark's no-image condition compared with instruction-only fine-tuning, while largely preserving general benchmark performance.

Critical strengths
  1. Annotation-free task construction. LOOK gets localization supervision directly from image crops, providing a simple auxiliary training task that requires no additional semantic annotations.
  2. Controlled evaluation of missing visual input. VisABench preserves sequence length and token positions when withholding visual content, while real-image controls help distinguish reduced unsupported assertions from a general tendency to answer No.
Critical weaknesses
  1. Positional decoding is presented as a multimodal capability. The target bounding box is determined by the supplied global positional metadata and remains unchanged when the crop's semantic content changes. LOOK therefore admits a solution that ignores image content entirely. However, the paper provides neither a position-only baseline nor a fixed-position content-replacement control, leaving the contribution of the visual content unestablished.
  2. The central interpretation conflates coordinate readout with visual grounding. Learning to recover supplied spatial information does not demonstrate that a model relies on image evidence when making semantic claims. A model could solve LOOK perfectly while continuing to answer visual questions from language priors. The narrow improvements under artificial no-image conditions do not establish this missing connection. Consequently, the paper overstates the significance of positional decoding as a foundation for multimodal understanding.
Questions for authors

Could you compare LOOK with a position-only baseline and a control that replaces crop content while preserving positional information, under matched training budgets, and evaluate their transfer to semantic grounding on real images to determine whether the benefits extend beyond decoding the supplied positional metadata?

2 comments
AuthorsOpenReview ↗

Response to Reviewer igMX

We thank the reviewer for the precise critique and for recognizing our annotation-free task construction and the design of VisABench. The baseline suggested by the reviewer turned out to be the most informative experiment in our paper.

W1 (Position-only baseline).

the paper provides neither a position-only baseline nor a fixed-position content-replacement control

Thank you for this suggestion. We trained the position-only baseline (PosOnly) with the same training budget as our model (see the General Response for details). It replaces the crop content with zeros while keeping all positions, which also serves as the fixed-position content replacement. In addition, we test our model with the crop content removed at test time.

LOOK IoU (content given)LOOK IoU (content removed)No-image Yes (%) ↓p(Yes) ↓
PosOnly0.2140.99113.10.310
Ours 10:100.7420.5100.80.101

We agree that LOOK can be solved without the image content, and PosOnly indeed learns this solution. However, our model does not solve LOOK in this way. When the content is removed, the IoU of our model drops by 0.23, while PosOnly fails when the real content is given (0.214). Moreover, the content-free solution does not help on VisABench. The no-image Yes rate of PosOnly is 16 times higher than ours. Therefore, the improvement on VisABench comes from learning LOOK with the image content, not from reading out the positions.

W2 (Coordinate readout vs. visual grounding).

A model could solve LOOK perfectly while continuing to answer visual questions from language priors.

We agree that LOOK alone does not guarantee the complete elimination of visual hallucinations. As illustrated in Figure 1, we propose a three-level visual hierarchy, in which hallucinations can arise at any level. Existing research primarily focuses on the latter two levels, Perceive and Understand, while largely overlooking the more fundamental Look level.

And we also agree that solutions to high LOOK IoU scores does not mean it reduces visual hallucination. Our new ablation, PosOnly, achieves nearly perfect performance on the LOOK task (IoU 0.991), it still produces false "Yes" responses on 13.1% of no-image inputs. This is also why we design the training of LOOK by combining both the visual content input and the global positional embeddings. This also motivates us to design a non-training task benchmark VisABench to further evaluate the model's awareness of visual input.

We also agree that our claim should be stated more precisely. LOOK targets visual-input awareness, i.e., whether the model's responses depend on the visual input it actually receives, rather than semantic grounding. On real-image inputs, LOOK maintains comparable Yes rates (94.1% vs. 93.3% for Base), wrong-image Yes rates (15.1% vs. 15.6%), and general benchmark performance (Table 4). Crucially, on no-image inputs, LOOK reduces false "Yes" responses from 31.2% (SFT) to 0.8%, revealing an improvement that conventional hallucination benchmarks such as POPE fail to capture.

Q1 (Matched-budget comparison and transfer).

Could you compare LOOK with a position-only baseline and a control that replaces crop content while preserving positional information, under matched training budgets...

Please see the comparison above and also in the general reponses. Both experiments are reported above with the same training budget. We hope these results address your concern that our contribution reduces to decoding the positional information.

AuthorsOpenReview ↗

Gentle Reminder: Rebuttal Discussion

Dear Reviewer igMX,

We would like to kindly follow up on our rebuttal, as we have not yet received your feedback. Since the discussion period is approaching its end, we would greatly appreciate it if you could review our responses and share your thoughts.

Best,

All authors

Reviewer_SH4COpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors claim that visual hallucination can be organized into a three-level hierarchy: LOOK, PERCEIVE, and UNDERSTAND. This paper identifies the main cause of hallucination as LOOK, the ability of a vision-language model to localize the visual content of a given image. To address it, the LOOK task is proposed, and models are trained to predict the coordinates of the visible region. To evaluate the effect of training, VisABench is proposed. VisABench is a benchmark dataset that asks whether a given expression is visible in a correct image, a blank image (no image), or a mismatched image. As a result, training on the LOOK task shows improvement on VisABench, but it is still limited in out-of-domain hallucination benchmarks.

Critical strengths
  • The paper proposes a new view of visual hallucination: rather than treating it as a semantic-level error, it decomposes it into three levels (LOOK, PERCEIVE, and UNDERSTAND) and argues that a hallucination can originate at any of them.
Critical weaknesses
  • There is no analysis showing that LOOK is the most massive problem among the three levels. The paper emphasizes that knowing at which level hallucination arises is important, yet it does not explain why it focuses only on LOOK.
    • MIRAGE appears to be cited as the justification, but there is a clear difference: in MIRAGE, no image is given at all, whereas in LOOK, an image is given but the model does not know where it can look. Citing MIRAGE alone is therefore insufficient, and additional analysis specific to the LOOK task is needed.
  • Prior work (VCD, region attention enhancement, VA-neuron) suggests that when an image is given, the model is looking at the image (in terms of visual attention), but the response comes out differently. This paper appears to make an opposing claim (that even when an image is given, the model fails to properly look at the region containing the visual content). However, the paper's claim may stem from the response type (bbox coordinates) or the input image format (normally the full image and its corresponding positional encoding are given, but here, a cropped image with absolute positional encoding), and it is likely that the model is actually looking at the image region well when its attention is observed. In addition, the evidence supporting the claim is not sufficiently visible in the paper, so it is hard to regard this as a contribution.
  • Regarding the LOOK task design) If the goal is to measure whether the model is actually looking at the image well, it seems more natural to erase semantic information, for example by painting the background outside the cropped region black, and feed the entire image as input. What is the reason for choosing a structure like the LOOK task? In particular, it is a setting that cannot be analyzed for VLMs that do not maintain absolute positions of visual tokens (e.g., query-based compression) or for closed models, so the analysis may be limited.
    • Since positional encoding is fed in a way completely different from the original VLM input format, the model is likely to find it harder to output coordinates compared to similar situations, and it is questionable whether a model that was not trained that way can output coordinates well in the first place.
  • For VisABench, (i)&(ii) is the evaluation setting of the MIRAGE paper, and (i)&(iii) is a format used in many hallucination datasets such as POPE. It is unclear whether it was created because of a problem that existing benchmarks do not satisfy or merely for analysis, and it is hard to regard it as a contribution.
  • Performance gains are only slightly visible in the in-domain setting, and in the out-of-domain setting there is almost no gain, or performance even drops, so it is hard to regard this as a method that solves hallucination. Also, there is no comparison at all with existing hallucination mitigation methods.
  • Minor
    • I don't understand what Table 1 wants to show. Is it that (iii) performs worst and therefore was chosen as the training task? If it performs poorly, why should it be chosen?
    • The example for UNDERSTAND is ambiguous (Figure 1): a question about an action appearing in the image is closer to Perceive, and Understand should be a question that requires more inference, e.g., an inference about why it seems to be doing that action. Because of this issue, the criterion for dividing the three hierarchy levels also seems ambiguous.
    • The definitions of symbols are missing in some equations.
Questions for authors

Please refer to the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer SH4C (1/2)

We thank the reviewer for the detailed comments and for recognizing our new view of visual hallucination. We address the concerns below.

W1 (Why focus on LOOK, and the difference from MIRAGE).

There is no analysis showing that LOOK is the most massive problem among the three levels. Difference from MIRAGE.

We do not claim that Look is the largest source of hallucination. Our claim is that Look is the most basic level, and existing benchmarks do not test it. Before a model can say what it sees, it first needs to know which visual input it has. Our results also show that this failure is large in practice. For example, Gemma-4-12B claims an object is present on 42.9% of the no-image items, and instruction-tuned Qwen3-VL does so on 31.2%, while its POPE score stays the same (88.5).

MIRAGE generally shows this severe hallucination problem in frontier models bur does not address it. MIRAGE removes the whole image for testing, while LOOK keeps a partial input for training the model to be aware of its visual input. And the no-image condition in VisABench is the same case as MIRAGE where we test the model by setting the visual input empty.

W2 (Prior work shows that models attend to the image).

Prior work (VCD, region attention enhancement, VA-neuron) suggests that when an image is given, the model is looking at the image. The paper's claim may stem from the response type (bbox coordinates) or the input image format

First, we do not claim that VLMs fully fail to attend to image regions. Second, even though the attention is computed from visual tokens, it does not mean the visual understanding is obtained from the images. This has been shown in many papers[1,2].

Our claim is narrower: pretrained VLMs cannot report which visual input they received. Given the full image, the pretrained model boxes the domainent object in the image instead of the whole view (Fig. 5). In the permutation test (Fig. 7), the predictions of Base and SFT match a random other window as well as their own window. A model can attend to visual tokens and still not know which part of the image these tokens cover, so our finding is consistent with the attention studies.

We agree that a pretrained model has not seen crops with absolute positions, which is why we train on LOOK. The new baselines show that what the model learns is not an effect of the input format. LocPos sees the same format with local positions and does not learn the task (IoU 0.371). PosOnly learns to read out the positions (IoU 0.991) but becomes less aware of missing input (13.1% vs. 0.8%).

[1] Wang, Zichuan, et al. "Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2026.

[2] Zhe Cheng, et al. "Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating." Proceedings of the 43rd International Conference on Machine Learning (ICML). 2026.

AuthorsOpenReview ↗

Response to Reviewer SH4C (2/2)

W3 (Why not black out the area outside the crop).

it seems more natural to erase semantic information, for example by painting the background outside the cropped region black

Thanks. We have tested this in Table 1 with black, grey, zero and mean masking. With masking, the model can uses a semantic shortcut to achieve high IoU scores such as always predicting the bbox of the domainent objects in the images as shown in Fig. 5. The higher IoU of masking (0.75–0.77, Table 1) comes from these shortcuts, as the model simply boxes the main object (Fig. 5 ii–iii). Cropping is the only design where the hidden content is truly absent while the image geometry is kept. For closed models and models with query-based compression, LOOK training cannot be applied (Sec. 7), but VisABench can still be used on any open-weight VLM.

W4 (Contribution of VisABench).

It is unclear whether it was created because of a problem that existing benchmarks do not satisfy

The individual conditions are similar to MIRAGE and POPE. What is new is that we evaluate all three conditions on the same 10,752 items, with the same sequence length and positions, and score them from logits. This allows us to separate "does not use the image" from "tends to say No". VisABench finds a failure that POPE misses, since SFT and our model have the same POPE score but differ by 39 times on the no-image Yes rate. We will explain this motivation more clearly in Sec. 4.

W5 (Out-of-domain gains and comparisons).

Performance gains are only slightly visible in the in-domain setting.

We want to remind the reviewer that we trained the model with LOOK tasks, but tested it on the VisABench, which is ready an out-of-distribution (OOD) testing. VisABench is designed to test if the model has awareness of visual input and no existing pretraining or posttraining including existing anti-hallucination methods targets for this ablity, since existing anti-hallucination methods are still limited to the perception and understanding levels. The results in Table 3 show that our model is the only one that can transfer the training to the OOD VisABench testing.

Moreover, our method keeps general performance within 2.5 points of SFT (Table 4) for general benchmarks and can be combined with semantic-level methods (please see our response to Reviewer YnBv, W3) to further improve the performance.

Minor.

  • Table 1. compares different probe designs. We deliberately choose the crop task since it shows the models have a gap on this task and the model thus can learn something from this task. The higher IoU under masking may come from semantic shortcuts. We will revise the caption to clarify it.
  • Fig. 1. Thank you for this suggestion. We agree that the Understand example is too close to Perceive, and we will replace it with a question that requires inference, e.g., "Why is the sheep's head lowered?" → "It is likely grazing."
  • Notation. We will define all symbols in Eqs. (1)–(4).
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer SH4C,

As the discussion period is coming to an end, we would appreciate it if you could take some time to review our rebuttal and provide your feedback. Your response would be very helpful for the discussion.

Thank you!

Best,

All authors

Reviewer_YnBvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether VLMs are aware of the visual input available to them before interpreting its content. It proposes a three-level hierarchy of visual understanding (Look, Perceive, Understand) and introduces the LOOK task. In LOOK, only an image crop is encoded, but the crop keeps the positional embeddings it would have in the full image, and the model must predict the crop's bounding box in the full image. LOOK data are generated automatically from arbitrary images. The paper also introduces VisABench, which tests whether a model claims an object is present when the visual tokens are replaced with null placeholders.

Critical strengths
  1. Novel and interesting perspective. The paper studies hallucination from the angle of whether a model knows which visual input it actually receives. The LOOK task is self-supervised: the crop itself provides the label, so it requires no manual annotation and can be built from any image dataset.
  2. Clear presentation and thorough implementation details. The paper documents logit-based scoring, positional restoration and implementation checks in detail. It also shows that general capabilities stay within 2.5 points of the SFT baseline.
Critical weaknesses
  1. Unclear link between LOOK and recognizing missing visual input. LOOK task trains the model to locate a crop within the full image using positional encodings, and no training sample is missing its visual input. It is therefore unclear why this skill should help the model recognize that no image is given. The ablation also suggests that global positional awareness, the paper's core mechanism, contributes only a small part of the gain. LocPos uses the same crop inputs and bbox answers as Ours, and the only difference is that it uses crop-local positions instead of global ones. Yet LocPos already lowers the no-image Yes rate from 31.2% (SFT) to 5.7%, and adding global positions lowers it only from 5.7% to 0.8%. This suggests that most of the reduction may come from adding crop inputs and bbox answers to training, not from positional awareness. In addition, LOOK IoU rises across mixing ratios (0.609 → 0.665 → 0.742), but the no-image Yes rate does not drop with it (2.3% → 4.9% → 0.8%). Could the authors clarify why the improvement should be attributed to positional awareness?
  2. Limited improvement over the pretrained model. The large VisABench gain is measured mainly against SFT, whose no-image Yes rate is itself raised from 2.5% to 31.2% by the replay data. Relative to the pretrained model (2.5%), LOOK models range from 0.8% to 4.9%. On mean p(Yes), all LOOK models (0.101–0.241) are worse than the pretrained model (0.041). Could the authors explain why replay SFT causes this increase, and whether "LOOK fine-tuning consistently improves performance" still holds relative to the pretrained model?
  3. Limited evidence on realistic hallucination. On the wrong-image condition, LOOK models match the pretrained model (15.1–16.0% vs. 15.6%). On POPE, Ours (10:10) equals SFT (88.5) and is only 0.5 above the pretrained model. There is also no comparison with existing hallucination-mitigation methods, and all experiments use a single model. Could the authors provide such comparisons and discuss how LOOK applies to other architectures?
Questions for authors

See weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer YnBv

We thank the reviewer for appreciating our perspective, the self-supervised task construction and the clear presentation. We address the concerns below.

W1 (Link between LOOK and missing visual input).

LocPos already lowers the no-image Yes rate from 31.2% (SFT) to 5.7% ... most of the reduction may come from adding crop inputs and bbox answers to training, not from positional awareness.

  • It is true that training with crops and bounding-box answers already helps, as LocPos reaches 5.7%. However, the new PosOnly baseline is trained on exactly the same crops and bounding-box answers, and it only reaches 13.1%. From LocPos to our model, the no-image Yes rate further drops from 5.7% to 0.8%, and the mean p(Yes) drops from 0.272 to 0.101. So neither the crop content nor the global position alone is enough, and the gain comes from combining the two.
Crop contentGlobal positionNo-image Yes (%) ↓p(Yes) ↓
SFT––31.20.409
PosOnly✗✓13.10.310
LocPos✓✗5.70.272
Ours 10:10✓✓0.80.101
  • For the mixing ratios, all three models learn LOOK (IoU ≥ 0.61) and all bring the no-image Yes rate back to the pretrained level, substantially improving over the SFT baseline (31.2%). Specifically, the rates are 2.3%, 4.9%, and 0.8% for the three mixing ratios, compared with 2.5% for the pretrained model. We agree that the trend is not monotonic, but these are the actual experimental results. We hypothesize that fine-tuning may benefit from more careful data-mixture design, although investigating this is beyond the scope of this work.

W2 (Improvement over the pretrained model).

Could the authors explain why replay SFT causes this increase, and whether "LOOK fine-tuning consistently improves performance" still holds relative to the pretrained model?

We thank the reviewer for pointing this out. We were also surprised to observe that SFT increases the No-image Yes rate. We believe this highlights the importance of carefully designing the data mixture for SFT, as naive fine-tuning may improve the model's confidence on certain tasks while compromising its generalization ability. Our method can be integrated with large-scale pretraining and fine-tuning to further enhance the model's visual awareness and grounding capabilities.

W3 (Realistic hallucination, comparisons and architectures).

There is also no comparison with existing hallucination-mitigation methods, and all experiments use a single model.

  • Existing visual hallucination benchmarks primarily evaluate the Perceive level in our proposed visual hierarchy, i.e., whether the model's claims about objects are consistent with the image. In contrast, LOOK targets the Look level, i.e., whether the model relies on visual input at all. These two aspects are fundamentally different: SFT and our model achieve the same POPE score (88.5, Table 4), yet our model has 39x lower no-image Yes rates. This distinction also motivates the introduction of VisABench. Existing hallucination-mitigation methods primarily address semantic claims during decoding or alignment and are therefore complementary to LOOK. We will clarify this relationship in the related work section.

  • VisABench is applicable to any open-weight VLM, and we have already reported results for Qwen2.5-VL, LLaVA-1.5, and Gemma-4 in Table 3. LOOK training, however, requires absolute positional information for image patches (Appendix B), which is available in the widely used Qwen3-VL family. We acknowledge and discuss this limitation in Section 7.

AuthorsOpenReview ↗

Gentle Reminder: Rebuttal Discussion

Dear Reviewer YnBv,

We hope you have had a chance to look at our rebuttal. We would greatly appreciate your feedback on whether our responses have addressed your concerns, especially as the discussion period is nearing its end.

Thank you for your time!

Best,

All authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to AC and all reviewers

We thank all reviewers for their careful and critical reading of our paper. We are glad that the reviewers recognized the novelty of our perspective (YnBv: "novel and interesting perspective"), the new insight into hallucination (SH4C: "a new view of visual hallucination"), and our self-supervised data construction (igMX: "annotation-free task construction"). This encouraged us to address every concern with new experiments.

A question shared by Reviewers YnBv and igMX is whether the improvement comes from global positional information, or simply from training on crops with bounding-box answers. Reviewer igMX further points out that LOOK could be solved from the positions alone, without looking at the image content. To answer this, we trained the position-only baseline (PosOnly) suggested by Reviewer igMX. PosOnly uses the same settings as our model. The only difference is that we set the patch embeddings of every LOOK crop to zero, so that each crop token keeps its global position but carries no image content.

The results on VisABench (all 10,752 items) and on LOOK (the 1,000 views of Table 2) are shown below. For LOOK, we also evaluate each model after removing the crop content at test time.

BaseSFTLocPos (content, local pos.)PosOnly (global pos., no content)Ours 10:10 (content + global pos.)
No-image Yes (%) ↓2.531.25.713.10.8
Real-image Yes (%) ↑93.394.992.695.294.1
Wrong-image Yes (%) ↓15.617.015.018.415.1
Gap ↑90.763.786.982.093.2
LOOK IoU (content given)0.181–0.3710.2140.742
LOOK IoU (content removed at test time)0.113––0.9910.510

From these results, we observe the following:

  • Positions alone solve LOOK perfectly. We agree with Reviewer igMX that LOOK can be solved without the image content: PosOnly predicts the visible window almost perfectly from the positions (IoU 0.991).
  • However, this does not make the model more aware of its visual input. PosOnly still answers "Yes" on 13.1% of the no-image items, which is 16 times higher than our model, and its gap (82.0) is even lower than that of the pretrained model (90.7). It also means that the box answer format does not help much for the VisaBench performance.
  • Content alone is also not enough. LocPos, which has the content but only local positions, stays at 5.7%. Only the model that learns to relate the crop content to its true position reaches 0.8%.
  • Our model uses both the image content and positional information. Its LOOK prediction depends on what is inside the crop, as the IoU drops from 0.742 to 0.510 when the content is removed.
  • Our method is designed to force the model to be aware of visual evidence and hence ground the following visual understanding and reasoning better.
AuthorsOpenReview ↗

Please take a look at our rebuttal, thank you!

Dear AC and reviewers,

Thank you for your time and consideration. We would greatly appreciate it if you could take another look at our responses and let us know whether they address your concerns. We are happy to provide further clarification if needed.

Best,

All authors

Dear reviewers, if you haven't already, could you please respond to the author rebuttal?

Thank you,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.