acceptodds
Under review as a conference paper at ICLR 2027

GazeVLM: Active Vision via Internal Attention Control for Multimodal Reasoning

Abstract

Human visual reasoning is governed by active vision, a process where meta-cognitive control drives top-down goal-directed attention, dynamically routing foveal focus toward task-relevant details while maintaining peripheral awareness of the global scene. In contrast, modern Vision-Language Models (VLMs) process visual information passively, relying on the static accumulation of massive token contexts that dilute the visual evidence as the reasoning chain grows. Here we propose GazeVLM, a multimodal model that learns to exert this meta-cognitive oversight over its attention resources by generating gaze actions in its reasoning chain, in the form of <LOOK> tags specifying the coordinates of the region to inspect. During training, each <LOOK> block triggers a continuous suppression bias on the attention logits that dampens features outside the regions gazed at so far, and that lifts at the end of the block, restoring the global view. The model is trained with this suppression bias as a teaching signal, first by Supervised Fine-Tuning on curated gaze-reasoning traces, then through Group Relative Policy Optimization (GRPO) with rewards for correct answers and valid grounding. At deployment, the external suppression is removed and the model, running its unmodified forward pass, steers its own attention to the regions indicated by the <LOOK> blocks, retaining most of the effect of the bias and emulating top-down control of spatial attention. This model thus transitions between global spatial awareness and localized focal reasoning without relying on external agentic tools like cropping, and without adding visual tokens from re-encoded patches. Applied to 4B-parameter backbones, GazeVLM demonstrates strong high-resolution multimodal reasoning on HRBench-4k and HRBench-8k, surpassing its base models by about 4 points, and outperforming agentic multimodal pipelines built around thinking with images by 5 to 12 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Language Models, Multimodal Reasoning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Feg2OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GazeVLM, a visual reasoning framework that uses self-generated gaze actions to guide attention over visual tokens. A spatial attention suppression bias is applied during SFT and GRPO training but removed at inference, allowing the model to operate without external cropping tools. The method demonstrates consistent improvements across several multimodal benchmarks, particularly HRBench.

Critical strengths
  1. Interesting methodological contribution. Using training-time attention modulation to learn gaze-1conditioned visual reasoning without additional inference-time visual tokens is a promising direction.
  2. Meaningful experimental improvements. The method achieves consistent gains across multiple benchmarks and model backbones.
  3. Useful ablation studies. Comparisons with the passive grounding baseline and analyses of attention alignment provide evidence supporting the proposed training mechanism.
Critical weaknesses
  1. Insufficient causal evidence for internal attention control. The attention–region agreement does not establish that generated gaze coordinates causally control attention. Moreover, removing LOOK blocks also removes observation content, making it difficult to isolate the effect of gaze actions.
  2. Limited semantic grounding of gaze actions. The GRPO reward primarily encourages geometrically valid LOOK actions without verifying whether the selected regions contain task-relevant visual evidence.
Questions for authors
  1. Causal attention control: Could the authors perturb or shuffle LOOK coordinates while preserving the reasoning context to verify whether gaze actions causally influence attention and predictions?
  2. Semantic grounding: How often do the generated LOOK regions contain task-relevant evidence? Could the authors evaluate the effect of irrelevant or randomly selected bounding boxes?
3 comments

We thank the reviewer for their review of our paper and we provide our responses to the questions below.

Q1. Causal attention control

We ran the causal intervention experiment suggested by the reviewer: we replace the coordinates in the <LOOK at="..." bbox=[...]> tag generated by GazeVLM with randomly perturbed coordinates (of a randomly positioned box of the same size) as soon as the tag is generated, and then continue decoding normally. We perform this intervention with GazeVLM (Qwen3-VL-4B) on 200 HRBench samples (4k+8k), using greedy decoding.

We report the effect of perturbing the coordinates of the bounding box of the LOOK block by measuring the location of the model's attention weights. In particular, we quantify the IoU between the model's top-k most-attended image patches (where k is the number of patches within the box) and the "perturbed box" (i.e. the replaced box with perturbed coordinates), and with the "unperturbed box" (i.e. the box originally generated by the model):

ConditionIoU with unperturbed boxIoU with perturbed box
Perturbed box (gaze bias off)0.3120.012

Surprisingly, we observe that the model pays far less attention to the perturbed box than it would to the generated unperturbed box (IoU=0.012 vs IoU=0.312). This suggests that, after training, the model's visual attention is not directly controlled by the LOOK block coordinates and that the model has learned to be robust to perturbations of the bounding box.

We therefore hypothesize that the LOOK block helps the model to focus on the gaze bias primarily during training, but it progressively learns to ignore it as training proceeds. This hypothesis would predict that as training progresses we should see the model becoming more robust to perturbations of the bounding box. To test this, we ran an experiment where we perturb the bounding box at different stages of GRPO training, starting at a checkpoint at step 0 (right after SFT) and quantify the model's behavior on a subset of HRBench (4k+8k). We show the change in accuracy together with the standard error over the samples

CheckpointUnperturbed boxPerturbed boxChange
GRPO step 075.5 ± 3.073.0 ± 3.1−2.5 ± 2.5
GRPO step 10079.0 ± 2.975.0 ± 3.1−4.0 ± 2.2
GRPO step 30078.0 ± 2.977.5 ± 3.0−0.5 ± 1.7

Consistent with the main observation, we see that accuracy progressively improves with GRPO training when measured on a normally generated unperturbed box. In addition, as predicted by our hypothesis that LOOK blocks are more important at the beginning of training but then are essentially ignored at the end, we see that perturbing the box at inference time causes a drop in performance that is maximal at around halfway through learning, to then become minimal by the end.

Q2. Semantic grounding

Although our reward does not check the content of the boxes directly, training with the bias ties it to the answer: during GRPO, a box that misses relevant evidence will tend to lead to a wrong answer and therefore to low reward. HRBench has no ground-truth region annotations, so we measured grounding on V* [1], which provides target-object boxes for every question (191 questions). We count a target as found when its center lies inside a LOOK box, and compare against a random box of the same size:

GazeVLM LOOK boxRandom box, same size
Targets found (all)85.3%4.0%
Direct attributes (1 target)90.4%1.3%
Relative position (2 targets, both found)59.3%1.1%
First LOOK hits a target90.6%4.9%

LOOK boxes are tight (median 0.5% of the image area), so these hits are not explained by large boxes. Grounding is linked to correctness: correct answers find the target in 88% of cases vs 74% for wrong answers, and accuracy is 85.5% when all targets are found vs 69.2% when one is missed.

  • For the effect of irrelevant or random boxes, please see our answer to Q1.

[1] V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs, CVPR 2024

Dear Reviewer,

As the rebuttal is approaching to an end, would you mind letting the authors know if their rebuttal addressed your concerns? Thanks!

AC

Thank you for your clear response. My concerns are fully addressed, and please make sure to incorporate these clarifications into the revised manuscript. The topic you are studying is very interesting and promising, I encourage you to explore this direction further. Accordingly, I will raise my rating from 3 to 4. Best of luck with your paper and future research! :)

Reviewer_48AgOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GazeVLM teaches a VLM to control its own attention over the visual tokens it has already encoded, instead of cropping and re-encoding image regions. The model emits <LOOK bbox=…> blocks in its reasoning. During training only, each block adds a Gaussian suppression bias to text-to-image attention logits outside the boxes gazed at so far, and the bias is lifted at </LOOK>. Training is SFT on about 11k gaze traces distilled from Qwen3-VL-235B, followed by GRPO with correctness, grounding, and efficiency rewards. At deployment the bias is removed and the model runs its unmodified forward pass. On Qwen3-VL-4B, GazeVLM improves HRBench-4k/8k by about 4 points and general benchmarks by 1.5–2.6. It beats a matched “passive grounding” baseline that uses the same data and recipe without the bias, and emits about 5× fewer tokens than DeepEyes.

Critical strengths

Simple, cheap mechanism: The method needs no architecture change, adds no visual tokens, and has no inference-time overhead. That makes it a practical alternative to crop-and-re-encode agents for high-resolution reasoning.

Matched control: The passive-grounding baseline shares the teacher traces, rewards, and SFT+GRPO recipe, and the bias variant beats it on every benchmark (e.g., +4.4/+3.6 on HRBench-4k/8k, +5.2 on MathVista). This attributes the gain to the bias rather than to data distillation.

Mechanistic evidence and evaluation care: The attention–box IoU rises from 0.13 to 0.36, removing <LOOK> reduces accuracy, and LLM-judge scores are checked against string matching (99.8% agreement). Together these partly support the internalization claim.

Critical weaknesses

Unclear why the gain appears: The bias brings no gain after SFT and helps only during GRPO. Applying it at inference hurts (MathVista 73.3 → 69.4), and GRPO rollouts are sampled under the biased policy while deployment uses the unbiased one. The paper does not explain this train–test mismatch, so the gap could reflect different RL dynamics rather than learned “active vision.”

Single-seed results with small margins: Gains over the base model on general benchmarks are 1.5–2.6 points and come from single runs. The <LOOK>-removal ablation uses only 100 samples per split (a 6-sample difference), and that intervention is out-of-distribution for a model always trained with <LOOK>.

Backbone-confounded agentic comparison: The “5–12 points over zoom-in agents” and 5× token-efficiency claims compare a Qwen3-VL-4B model against Qwen2.5-VL-7B agents. The Qwen3-VL-4B base already beats those agents (79.5 vs 75.1 on HRBench-4k). A crop-and-re-encode baseline on the same backbone and data is missing.

Weak internalization evidence: An IoU of 0.36 is moderate. The key mechanism figures (Figs. 2, 3, 5) are single examples, and Fig. 2’s oracle-box curve rests on one question.

Underspecified Qwen3.5 result: It is unclear whether GazeVLM-Qwen3.5 starts from the Instruct or Think model, and there is no passive-baseline control for it. The +8.6 MathVista gain is therefore hard to interpret.

Questions for authors

Can you report mean ± std over several GRPO seeds for GazeVLM and the passive baseline, to show the RL-stage gap is reliable?

Why does applying the bias at inference reduce accuracy if the model has internalized it? Have you tried GRPO with unbiased rollouts and a bias-based auxiliary loss, to separate the regularization effect from the rollout distribution?

Can you compare against a crop-and-re-encode zoom agent trained on Qwen3-VL-4B with the same data and rewards, reporting both accuracy and total (visual + text) tokens?

Which Qwen3.5 checkpoint initializes GazeVLM-Qwen3.5, and how does the passive baseline perform on it?

Some related works [1] also strengthen vision activation by designing perception-oriented rewards. It's encouraged to compare with these peer works.

[1] More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models, ICLR 2026

4 comments

We want to thank the reviewer for their feedback and high-quality review. Below, we respond to the concerns and questions.

Q1. Results on multiple GRPO seeds

We managed to run three repetitions of GRPO training and evaluated the trained models on the benchmarks. We summarize our results in the table below by displaying uncertainty intervals as standard deviation around the mean, which we will also include in the revised paper. These results show that GazeVLM consistently improves over the base Qwen3-VL-4B model, with particularly clear gains on the high-resolution benchmarks (+2.1 and +3.4 points on HRBench-4k and HRBench-8k, respectively).

Chart & Math ReasoningGeneral Reasoning & Vision criticalHigh-Resolution
MethodChartQAMathVistaMMBenchMMStarCV-BenchHRBench-4kHRBench-8k
Qwen3-VL-4B83.271.787.767.383.479.573.6
GazeVLM (mean ± std)85.3 ± 0.572.0 ± 1.189.0 ± 0.268.1 ± 1.085.2 ± 0.381.6 ± 1.677.0 ± 1.2
Δ (vs Qwen3-VL-4B)+2.1+0.3+1.3+0.8+1.8+2.1+3.4

Q2. Why does applying the bias at inference reduce accuracy?

  • We thank the reviewer for raising this important question. We want to emphasize that the drop in performance when using the bias at inference is not substantial, although it is large enough to show that not using the bias at inference is better. This observation, combined with the benefit of not adding the overhead of enforcing the bias at inference, led us to our setup, which was the intended behaviour.

  • We hypothesize that, when the bias is used at inference, the model is more sensitive to the correctness of the bounding boxes. If it loses some information because it chooses an incorrect bounding box, it is more difficult for the model to recover. During training, this is acceptable because of the supervision that we provide with our expert traces as well as the correct answer, which helps to steer the model appropriately.

  • Regarding GRPO with unbiased rollouts and a bias-based loss, we did not run such an experiment. However, we ran an experiment with unbiased rollouts, reported in Table 1 as GazeVLM (w/o gaze bias). This represents the traditional grounding method. For our model, GazeVLM, we use gaze-biased rollouts during the GRPO training.

Q3. Comparison against crop-and-re-encode zoom

We thank the reviewer for this suggestion. We implemented a crop-and-re-encode baseline using the same Qwen3-VL-4B backbone, training data, and reward functions as GazeVLM. Following each <LOOK> action, the corresponding image region is cropped, re-encoded, and provided to the model as additional visual tokens. We use a different KL-divergence configuration for this baseline because the original configuration led to unstable GRPO training. Below, you can see the table of the results of our GazeVLM vs. Zoom baseline on the HRBench dataset.

Method4K Tokens / Trace ↓4K Calls / Trace ↓4K Accuracy ↑8K Tokens / Trace ↓8K Calls / Trace ↓8K Accuracy ↑
Qwen3-VL-4B-Instruct113.50.079.5125.20.073.6
DeepEyes (ZOOM)13251.8175.112681.7172.6
Zoom baseline (<LOOK> + crop)760.51.1583.11479.41.1579.1
GazeVLM (<LOOK>)265.91.1983.4292.91.1978.0
Gain vs. DeepEyes4.98×1.52×+8.3%4.33×1.44×+5.4%
Gain vs. Zoom baseline2.86×0.97×+0.3%5.05×0.97×−1.1%

The results show that crop-and-re-encode (Zoom baseline) achieves comparable accuracy to GazeVLM, while GazeVLM requires substantially fewer tokens (2x to 5x fewer output tokens). This highlights the main advantage of our approach: competitive visual reasoning without the additional visual token processing required by explicit zooming.

Q4. Checkpoint for GazeVLM-Qwen3.5

  • We initialized GazeVLM-Qwen3.5 with Qwen3.5-4B in Instruct mode (enable_thinking=False). This was necessary, as we wanted to teach the model to follow our format and not its native thinking pipeline.

  • As for the performance of the passive baseline, we report the accuracy in the table below and we will update the corresponding table in the paper accordingly.

GroupMethodChartQAMathVistaMMBenchMMStarCV-BenchHRBench-4kHRBench-8k
Vanilla open-source VLMsQwen2.5-VL-7B83.367.983.857.973.768.559.5
Qwen3-VL-4B83.271.787.767.383.479.573.6
Qwen3.5-4B-Think83.168.888.065.484.583.078.4
Ours (Qwen3.5-4B)GazeVLM-Qwen3.5 (w/o gaze bias)86.976.089.469.385.982.380.0
GazeVLM-Qwen3.5 (Ours)85.877.490.071.286.286.982.1
Δ (vs Qwen3.5-4B-Think)+2.7+8.6+2.0+5.8+1.7+3.9+3.7

We observe a similar performance trend to the one we observed with GazeVLM on Qwen3-VL: the gaze-trained model performs better than the passive baseline, except on ChartQA in this case. We will also add the passive baseline to the revised paper.

Q5. Related works

We thank the reviewer for pointing us to this relevant work. We will include VAPO in the related work and discuss its connection to GazeVLM. Indeed, VAPO-Thinker achieves 67.1 and 75.6 on MathVista for its 3B and 7B models, respectively, compared to 77.4 for GazeVLM-Qwen3.5-4B. Similarly, on MMStar, VAPO-Thinker-7B achieves 63.0, compared to 71.2 for GazeVLM-Qwen3.5-4B.

Dear Reviewer,

As the rebuttal is approaching to an end, would you mind letting the authors know if their rebuttal addressed your concerns? Thanks!

AC

Thanks for the rebuttal. The rebuttal addresses most of my concerns. Therefore I would like to keep my rating. Thanks.

Reviewer_HFq7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce GazeVLM, a novel mechanism inspired by human visual attention (from global to local view), in order to improve MLLM abilities in focusing and describing objects and entities in VQA settings. By focusing on 4B MLLM models (specifically Qwen3 and Qwen3.5), GazeVLM is a framework that preserves the original architecture of models while introducing a modification on the forward pass: during the reasoning phase, if the model has to 'look' for a specific object in an image, it produces <LOOK> tags with coordinates of the corresponding object bounding box and then applies an attention suppression bias that modulate the decoder's attention logits, by weighting down visual tokens outside the bounding box through Gaussian spatial decay. Once the </LOOK> is produced, the model attention is restored to its global view. The suppression bias is done only during the training phase, so that in inference it automatically adapts to the strategy and focuses to the important regions, all of this while operating directly on pre-encoded visual tokens without external cropping, image re-encoding or changing the initial model architecture. GazeVLM is then validated through different VQA benchmarks and compared against existing SOTA models. Also different ablations are presented to investigate the contribution of the new mechanism and how it impacts performances, particularly on high-resolution tasks.

Critical strengths

Existing models in literature try to improve MLLM abilities in reasoning in VQA settings with the introduction of external tools, like re-encoding of the image patches (Deep Eyes), zoom-in and cropping (Ground-R1 and Pixel Reasoner). GazeVLM, instead, does it without any external tools and without changing the original architecture and mostly maintain the same training process of the original model. This makes the model able to reason and focus on specific region automatically during inference without auxiliary tools to help it reasoning. Thanks to this, we can see how is efficient in terms of computational analysis, by producing 5x fewer tokens in respect to DeepEyes, but still acquiring more than 5% in accuracy in the high-resolution benchmark of HRBench-4k/8k.

Critical weaknesses

The main concerns are related to the experimental results and limited discussion of some of the reported results. The abstract states that GazeVLM is applied to 4B-parameter backbones, but this refers only to two Qwen models (Qwen3VL and Qwen 3.5) of which only Qwen3VL has results explained and described throughout all the paper.

Moreover, Table 1 reports only one version of Qwen3VL, presumably the Instruct version. However, the framework relies on reasoning processes, and given the promising results obtained with GazeVLM-Qwen 3.5, an evaluation on Qwen3VL-Thinking could be interesting to analyze. A minor issue is related to bold results not always being the best ones, see o1 in MathVista (73.9) and Qwen3.5-4B-Think in HRBench8k (78.4) beating GazeVLM scores (73.3 and 78.4).

Another main concern follows Qwen3.5 analyses. Following the previous concerns on missing evaluations, the GazeVLM-Qwen3.5 w/o gaze bias is missing in Table 1. Similiarly, neither in Table 3 and 4, the 3.5 model and the 'without gaze bias' are included. The paper claims that GazeVLM internalizes the suppression bias and it contributes to improving model performance, but this is only proved for one model and so not applicable to other backbones. Including these other results can provide stronger evidence in supporting the claim.

In Table 1, Qwen3.5 Instruct and Thinking are presented as Vanilla Open-Source VLMs, but there is no official checkpoint that separates Instruct and Thinking versions of this model and there is no discussion of how this configuration was applied in the experiments. Moreover, authors should clairify whether the GazeVLM-Qwen3.5 was used with Thinking version as the paper does not provide a detailed discussion of its performance.

Finally, one of the main important results is GazeVLM-Qwen3.5 on HRBench4k/8k, with a value on 86.9 and 82.1, but it is unclear if the official protocol was followed to produce these results. For reference here (https://github.com/DreamMr/HR-Bench) these numbers exceed those of much larger models on the official benchmark leaderboard, also exceeding one with more parameters, and they are not adequately addressed throughout the paper.

Questions for authors
  1. Could the authors provide details about the training and inference configurations used for GazeVLM Qwen 3.5, and also clarify the configuration of both Instruct and Thinking of Qwen3.5? Could they also clarify how these two configuration were evaluated in Table 1?

  2. Some experiments are missing. Could the authors provide:

2.1 Results of Table 3 and 4 also evaluating GazeVLM without bias?

2.2 Results of GazeVLM-Qwen 3.5 in Table 1 without gaze bias?

2.3 Results of GazeVLM-Qwen3.5 applied to study of Table 3 and 4?

This can be helpful in claiming and supporting the idea of how internalized bias is helping to improve the model.

  1. Could the authors clarify if the evaluation on HRBench follows the official protocol? Given the strong results, it would be important to assess if these results can be comparable to those reported by the official leaderboard.
4 comments

We want to thank the reviewer for taking the time to assess our work, and for the thoughtful review.

Weakness: Analysis of Qwen3-VL-Think

We thank the reviewer for suggesting this comparison. We evaluated the original Qwen3-VL-4B-Thinking model using greedy decoding with a maximum of 2,048 generated tokens. GazeVLM outperforms this baseline across all seven benchmarks, with particularly notable improvements on HRBench-4k (+8.6 points) and HRBench-8k (+7.0 points).

GroupMethodChartQAMathVistaMMBenchMMStarCV-BenchHRBench-4kHRBench-8k
Vanilla open-source VLMsQwen2.5-VL-7B83.367.983.857.973.768.559.5
Qwen3-VL-4B83.271.787.767.383.479.573.6
Qwen3-VL-4B-Think83.867.388.665.785.374.871.0
RL-trained VLMsVision-R1-7B-73.5-----
Pixel-Reasoner-7B-----74.066.9
DeepEyes-70.1---75.172.6
Ground-R1-7B-----75.071.1
Ours (Qwen3-VL-4B)GazeVLM (w/o gaze bias)85.568.188.767.184.579.074.4
GazeVLM (Ours)85.873.389.269.085.483.478.0
Δ (vs Qwen3-VL-4B-Think)+2.0+6.0+0.6+3.3+0.1+8.6+7.0

Weakness: Bolded results that are not best

Thanks for remarking that. In the revision, we made sure to only denote best results in bold.

Q1. Details on the training and inference configurations

GazeVLM Qwen3.5

  • For the training and inference configurations used for GazeVLM Qwen3.5, we use the same configuration as GazeVLM Qwen3-VL with the main difference being the base model change. Specifically, for training, we use LoRA (r=64, α\alpha=256), lr=1e-5 and a maximum sequence length of 2,048.

  • We disable the native thinking mode of Qwen3.5 throughout training and inference (enable_thinking=False in the chat template). Setting the argument to False corresponds to Instruct, and True to Thinking. Both modes are evaluated using greedy decoding. During training, it was also necessary to set the argument to False to make the model reason using our format (<THINK>, <LOOK at=... bbox=[...]>, <ANSWER>).

  • For inference, we used our trained GazeVLM-Qwen3.5 with the gaze bias turned off, and have greedy decoding with up to 2,048 tokens, following our evaluation protocol.

Configuration for Qwen3.5 Instruct and Thinking

  • The Instruct and Thinking modes of Qwen3.5 are enabled by changing the enable_thinking argument of the model during inference. They both use the same checkpoint.
  • We use greedy decoding for inference and max_new_tokens=2,048.

Q2. Additional Results

  1. In Tables 3 and 4, GazeVLM does not use the bias at inference. When using the bias at inference, we explicitly state that in the paper (as in Table 3, last line). If the reviewer was instead referring to GazeVLM trained without the gaze bias, we have extended Table 3 with this model below.
ModelIoU
Qwen3-VL-4B (base)0.130 ± 0.009
GazeVLM w/o gaze bias0.379 ± 0.007
  1. Regarding GazeVLM-Qwen3.5 w/o gaze bias, here is the updated table with this result. We only kept the main numbers and removed other models for simplicity. We will have the full updated table in the paper.

We observe a similar performance trend to the one we observed with GazeVLM on Qwen3-VL: as the gaze-trained model performs better than the passive baseline, except on ChartQA in this case.

GroupMethodChartQAMathVistaMMBenchMMStarCV-BenchHRBench-4kHRBench-8k
Vanilla open-source VLMsQwen2.5-VL-7B83.367.983.857.973.768.559.5
Qwen3-VL-4B83.271.787.767.383.479.573.6
Qwen3.5-4B-Think83.168.888.065.484.583.078.4
RL-trained VLMsVision-R1-7B-73.5-----
Pixel-Reasoner-7B-----74.066.9
DeepEyes-70.1---75.172.6
Ground-R1-7B-----75.071.1
Ours (Qwen3.5-4B)GazeVLM-Qwen3.5 (w/o gaze bias)86.976.089.469.385.982.380.0
GazeVLM-Qwen3.5 (Ours)85.877.490.071.286.286.982.1
Δ (vs Qwen3.5-4B-Think)+2.7+8.6+2.0+5.8+1.7+3.9+3.7
  1. Regarding GazeVLM-Qwen3.5 in Table 3 and 4, here are the results that we obtain. We observe similar trends to the ones we reported for Qwen3-VL.

Table 3

ModelIoU
Qwen3.5-4B (base)0.210 ± 0.005
GazeVLM-Qwen3.50.256 ± 0.005
GazeVLM-Qwen3.5 + bias0.340 ± 0.005

Table 4

ModelHRBench-4kHRBench-8k
GazeVLM-Qwen3.592.081.0
GazeVLM-Qwen3.5 (w/o <LOOK>)77.070.0

Q3. Evaluation of HRBench

We evaluated our models on the full HRBench benchmark (800 samples per resolution). As in the official protocol, we use an LLM judge (gpt-oss-120b in our case) to judge the answers. We additionally evaluated the predictions using strict answer matching, reported in Table 6 in the Appendix, which agrees with the judge on 99.8% of the samples. We will expand the experimental details in the revised manuscript.

We hope these clarifications address the reviewer’s questions. We thank the reviewer for their constructive feedback.

Dear Reviewer,

As the rebuttal is approaching to an end, would you mind letting the authors know if their rebuttal addressed your concerns? Thanks!

AC

I thank the authors for their detailed response and for conducting all the additional experiments to address the concerns raised in the review.

The new results on Qwen3-VL-Thinking, the additional Qwen3.5 ablations, and the clarification of the HRBench evaluation protocol address most of my original concerns and strengthen the experimental evaluation. I encourage the authors to include and explicitly discuss the new Qwen3.5 results in the main paper, as these results are important for how the proposed mechanism behaves consistently across the evaluated backbones.
Although there are some aspects that need some clarifications. For the Qwen3-VL version, the passive baseline trained without gaze bias achieves a slightly higher IoU than GazeVLM (0.379 vs. 0.360). This can suggest that the improved attention alignment cannot be attributed specifically to training with the gaze bias, as we see alignment increasing only when the bias is applied again at inference (0.440). This is also consistent with the experiment the authors provided in the other comments: where the <LOOK> coordinates are replaced with a random box, the model’s attention doesn’t follow the new box (0.012) but remains on the originally predicted region (0.312). This indicates that the model already knows where to look, independently of the coordinates it emits. For Qwen3.5, the corresponding passive baseline is still missing from this analysis. If we assume that both backbones follow the same trend, as we noticed in the other experiments, I would expect a similar behavior, so the main question about the impact of the bias on attention alignment is still open.
Another connection is to the results in Table 4. Removing <LOOK> blocks eliminates both the gaze and the observation text, and despite seeing the drop in accuracy, it doesn't mean it fully isolates the contribution of the gaze action. This is because we see the passive baseline still attending to the relevant region as well as GazeVLM, so the drop could mean something different than the bias itself, like producing explicit observations. Reporting the same ablation of <LOOK> tags removed for the passive baseline could help clarify this point.

Overall, I appreciate the additional experiments and clarifications, and I revise my rating from 2 to 3, while encouraging a more careful discussion of the mechanism and its limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.