- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the LOOK task for improving visual grounding. The constructed LOOK training data is employed to fine-tune pre-trained VLMs through self-supervision to localize the crop region in images using global positional information. Furthermore, the VisABench is designed to evaluate responses when visual content is withheld. Experimental results yielded on Qwen3-VL show improved crop localization and fewer false object-presence under the benchmark's no-image condition compared with instruction-only fine-tuning, while largely preserving general benchmark performance.
Critical strengths
- Annotation-free task construction. LOOK gets localization supervision directly from image crops, providing a simple auxiliary training task that requires no additional semantic annotations.
- Controlled evaluation of missing visual input. VisABench preserves sequence length and token positions when withholding visual content, while real-image controls help distinguish reduced unsupported assertions from a general tendency to answer No.
Critical weaknesses
- Positional decoding is presented as a multimodal capability. The target bounding box is determined by the supplied global positional metadata and remains unchanged when the crop's semantic content changes. LOOK therefore admits a solution that ignores image content entirely. However, the paper provides neither a position-only baseline nor a fixed-position content-replacement control, leaving the contribution of the visual content unestablished.
- The central interpretation conflates coordinate readout with visual grounding. Learning to recover supplied spatial information does not demonstrate that a model relies on image evidence when making semantic claims. A model could solve LOOK perfectly while continuing to answer visual questions from language priors. The narrow improvements under artificial no-image conditions do not establish this missing connection. Consequently, the paper overstates the significance of positional decoding as a foundation for multimodal understanding.
Questions for authors
Could you compare LOOK with a position-only baseline and a control that replaces crop content while preserving positional information, under matched training budgets, and evaluate their transfer to semantic grounding on real images to determine whether the benefits extend beyond decoding the supplied positional metadata?
2 comments
Gentle Reminder: Rebuttal Discussion
Dear Reviewer igMX,
We would like to kindly follow up on our rebuttal, as we have not yet received your feedback. Since the discussion period is approaching its end, we would greatly appreciate it if you could review our responses and share your thoughts.
Best,
All authors
Response to Reviewer igMX
We thank the reviewer for the precise critique and for recognizing our annotation-free task construction and the design of VisABench. The baseline suggested by the reviewer turned out to be the most informative experiment in our paper.
W1 (Position-only baseline).
Thank you for this suggestion. We trained the position-only baseline (PosOnly) with the same training budget as our model (see the General Response for details). It replaces the crop content with zeros while keeping all positions, which also serves as the fixed-position content replacement. In addition, we test our model with the crop content removed at test time.
We agree that LOOK can be solved without the image content, and PosOnly indeed learns this solution. However, our model does not solve LOOK in this way. When the content is removed, the IoU of our model drops by 0.23, while PosOnly fails when the real content is given (0.214). Moreover, the content-free solution does not help on VisABench. The no-image Yes rate of PosOnly is 16 times higher than ours. Therefore, the improvement on VisABench comes from learning LOOK with the image content, not from reading out the positions.
W2 (Coordinate readout vs. visual grounding).
We agree that LOOK alone does not guarantee the complete elimination of visual hallucinations. As illustrated in Figure 1, we propose a three-level visual hierarchy, in which hallucinations can arise at any level. Existing research primarily focuses on the latter two levels, Perceive and Understand, while largely overlooking the more fundamental Look level.
And we also agree that solutions to high LOOK IoU scores does not mean it reduces visual hallucination. Our new ablation, PosOnly, achieves nearly perfect performance on the LOOK task (IoU 0.991), it still produces false "Yes" responses on 13.1% of no-image inputs. This is also why we design the training of LOOK by combining both the visual content input and the global positional embeddings. This also motivates us to design a non-training task benchmark VisABench to further evaluate the model's awareness of visual input.
We also agree that our claim should be stated more precisely. LOOK targets visual-input awareness, i.e., whether the model's responses depend on the visual input it actually receives, rather than semantic grounding. On real-image inputs, LOOK maintains comparable Yes rates (94.1% vs. 93.3% for Base), wrong-image Yes rates (15.1% vs. 15.6%), and general benchmark performance (Table 4). Crucially, on no-image inputs, LOOK reduces false "Yes" responses from 31.2% (SFT) to 0.8%, revealing an improvement that conventional hallucination benchmarks such as POPE fail to capture.
Q1 (Matched-budget comparison and transfer).
Please see the comparison above and also in the general reponses. Both experiments are reported above with the same training budget. We hope these results address your concern that our contribution reduces to decoding the positional information.