- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GazeVLM, a visual reasoning framework that uses self-generated gaze actions to guide attention over visual tokens. A spatial attention suppression bias is applied during SFT and GRPO training but removed at inference, allowing the model to operate without external cropping tools. The method demonstrates consistent improvements across several multimodal benchmarks, particularly HRBench.
Critical strengths
- Interesting methodological contribution. Using training-time attention modulation to learn gaze-1conditioned visual reasoning without additional inference-time visual tokens is a promising direction.
- Meaningful experimental improvements. The method achieves consistent gains across multiple benchmarks and model backbones.
- Useful ablation studies. Comparisons with the passive grounding baseline and analyses of attention alignment provide evidence supporting the proposed training mechanism.
Critical weaknesses
- Insufficient causal evidence for internal attention control. The attention–region agreement does not establish that generated gaze coordinates causally control attention. Moreover, removing LOOK blocks also removes observation content, making it difficult to isolate the effect of gaze actions.
- Limited semantic grounding of gaze actions. The GRPO reward primarily encourages geometrically valid LOOK actions without verifying whether the selected regions contain task-relevant visual evidence.
Questions for authors
- Causal attention control: Could the authors perturb or shuffle LOOK coordinates while preserving the reasoning context to verify whether gaze actions causally influence attention and predictions?
- Semantic grounding: How often do the generated LOOK regions contain task-relevant evidence? Could the authors evaluate the effect of irrelevant or randomly selected bounding boxes?
3 comments
Dear Reviewer,
As the rebuttal is approaching to an end, would you mind letting the authors know if their rebuttal addressed your concerns? Thanks!
AC
Thank you for your clear response. My concerns are fully addressed, and please make sure to incorporate these clarifications into the revised manuscript. The topic you are studying is very interesting and promising, I encourage you to explore this direction further. Accordingly, I will raise my rating from 3 to 4. Best of luck with your paper and future research! :)
We thank the reviewer for their review of our paper and we provide our responses to the questions below.
Q1. Causal attention control
We ran the causal intervention experiment suggested by the reviewer: we replace the coordinates in the
<LOOK at="..." bbox=[...]>tag generated by GazeVLM with randomly perturbed coordinates (of a randomly positioned box of the same size) as soon as the tag is generated, and then continue decoding normally. We perform this intervention with GazeVLM (Qwen3-VL-4B) on 200 HRBench samples (4k+8k), using greedy decoding.We report the effect of perturbing the coordinates of the bounding box of the
LOOKblock by measuring the location of the model's attention weights. In particular, we quantify the IoU between the model's top-k most-attended image patches (where k is the number of patches within the box) and the "perturbed box" (i.e. the replaced box with perturbed coordinates), and with the "unperturbed box" (i.e. the box originally generated by the model):Surprisingly, we observe that the model pays far less attention to the perturbed box than it would to the generated unperturbed box (IoU=0.012 vs IoU=0.312). This suggests that, after training, the model's visual attention is not directly controlled by the
LOOKblock coordinates and that the model has learned to be robust to perturbations of the bounding box.We therefore hypothesize that the
LOOKblock helps the model to focus on the gaze bias primarily during training, but it progressively learns to ignore it as training proceeds. This hypothesis would predict that as training progresses we should see the model becoming more robust to perturbations of the bounding box. To test this, we ran an experiment where we perturb the bounding box at different stages of GRPO training, starting at a checkpoint at step 0 (right after SFT) and quantify the model's behavior on a subset of HRBench (4k+8k). We show the change in accuracy together with the standard error over the samplesConsistent with the main observation, we see that accuracy progressively improves with GRPO training when measured on a normally generated unperturbed box. In addition, as predicted by our hypothesis that
LOOKblocks are more important at the beginning of training but then are essentially ignored at the end, we see that perturbing the box at inference time causes a drop in performance that is maximal at around halfway through learning, to then become minimal by the end.Q2. Semantic grounding
Although our reward does not check the content of the boxes directly, training with the bias ties it to the answer: during GRPO, a box that misses relevant evidence will tend to lead to a wrong answer and therefore to low reward. HRBench has no ground-truth region annotations, so we measured grounding on V* [1], which provides target-object boxes for every question (191 questions). We count a target as found when its center lies inside a
LOOKbox, and compare against a random box of the same size:LOOK boxes are tight (median 0.5% of the image area), so these hits are not explained by large boxes. Grounding is linked to correctness: correct answers find the target in 88% of cases vs 74% for wrong answers, and accuracy is 85.5% when all targets are found vs 69.2% when one is missed.
[1] V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs, CVPR 2024