- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how human gaze can be used to train VLMs to produce grounded reasoning, where each reasoning step is tied to an image coordinate. The authors collect about 7.2k eye-tracking trials on chart QA, counting, and visual search. They then propose GGR: a large teacher model receives the image, question, and human fixations, and writes a grounded reasoning trace using only a selected subset of the fixations. These traces are used to fine-tune Qwen3-VL-8B, followed by standard RL without gaze. At test time the model sees only the image and question. The main finding is that selective use of gaze works better than supervising every fixation. GGR also stays more aligned with human gaze than grounded baselines that do not use gaze, both before and after RL.
Critical strengths
-
The human-collected dataset is of high quality. The eye-tracking data was collected with a research-grade tracker under an IRB protocol, and covers three fairly hard task types: chart understanding, counting, and high-resolution visual search. Gaze recorded during goal-directed problem solving, together with think-aloud audio, is rare. This rich gaze data can be used to fine-tune models on downstream tasks, and is likely to be useful beyond this paper. The authors commit to releasing it.
-
The paper reaches an interesting conclusion about how to use gaze: selective use beats point-by-point imitation. The GGR vs. Fix. All comparison is clean, since both use the same teacher and pipeline and differ only in whether the teacher may skip fixations. GGR is better on every benchmark after SFT, while Fix. All is actually more aligned with human gaze. This dissociation, where closer imitation of human behavior does not give better reasoning, is useful for anyone thinking about learning from human behavioral data.
-
The evaluation looks at the model from several angles, not just accuracy. The paper separately measures task accuracy, spatial alignment with held-out human fixations, and the reasoning behaviors expressed in the chain of thought. The alignment analysis goes beyond a single score: it reports coverage and precision and controls for the number of output points. The comparison with ViGoRL also uses the same teacher model, which makes the human-guided vs. model-guided comparison fairer than it would otherwise be.
Critical weaknesses
-
The claim that human gaze makes the model reason better is not established, and the method does little to help the VLM actually exploit the gaze signal. In GGR, the decisions about which points to use and in what order are made by the 122B teacher. Gaze only narrows the candidate pool, and the prompt in Listing A.1 explicitly allows citing points "in any semantic order". Fixation order, durations, and the think-aloud transcripts are all discarded, and gaze plays no role in RL. This does not quite match the paper's framing that gaze transfers human "exploration structure". The SFT gain could largely come from distilling a much larger teacher with heavy rejection sampling rather than from the human fixations. The experiment that would separate the two is missing: the same pipeline with no fixations, or with non-human candidates. A "Base CoT (grounded)" condition appears in Figures 1 and 3 but not in Table 1. The Shuffled control does not fill this gap either: §5.1 says it was run on the Fix. condition, where it scores essentially the same as Fix. All, so comparing it with GGR in §5.2 does not support "task-specific fixation guidance matters".
-
The baselines are limited, and the main table is essentially an internal ablation. Apart from three off-the-shelf models, every row in Table 1 is a variant the authors trained from the same base model. The main related work, ViGoRL, is reimplemented only for its MCTS data-construction step. There is no ViGoRL SFT → RL, even though the original method includes RL, and no comparison with its multi-turn variant that zooms into predicted coordinates. The RL recipe and most of the RL data come from Vero, yet the Vero paper's own model trained from the same base, Vero-Qwen3I-8B, is not reported. More broadly, there are few external baselines on the same benchmarks. Visual Probe comes from the Mini-o3 paper, and the RL data includes Pixel Reasoner subsets, yet none of the "thinking with images" methods that crop and zoom, such as Mini-o3, Pixel Reasoner, or DeepEyes, are evaluated. Prior gaze-supervision methods such as ChartGaze are also missing. This comparison matters because participants could zoom during data collection, and §6 itself notes that emitting a coordinate does not give the model a new view. Without these baselines, the abstract's claim of being "competitive with leading grounded-reasoning baselines" is hard to assess.
-
The evidence for transfer is weaker than presented. The "higher alignment does not mean better performance" effect is observed, but the alignment metric fits the KDE bandwidth σ and background weight ε separately for each condition (Appendix A.6), which makes cross-condition comparisons less direct. The held-out trials also come from the same collection as the training gaze, so gaze-trained models scoring higher is partly expected. In the behavioral analysis of §5.4, the effects are small, on the order of a tenth of an occurrence per 100 reasoning words or less. The counts come from an LLM coder with no validation against human annotation. On top of that, the shared system prompt in Listing A.2 explicitly instructs the model to show planning, exploration, evaluation, and re-examination. That confounds any claim that gaze supervision "induces" these behaviors.
-
After RL, the performance benefit of gaze supervision is marginal, and most of the final gain comes from the borrowed RL stage. GGR-RL beats Point-Grounded RL by only 0.6 points. The reported error bars come from repeated evaluation sampling only, not from different training seeds, so this gap is not shown to be robust. Meanwhile RL alone, apparently starting from the base model, already improves accuracy by about 11 points. The RL data is much larger than the gaze SFT data and mostly taken from Vero together with its recipe, the reward contains no gaze term, and there is no ablation of the RL data. So by the end of training, gaze contributes very little measurable accuracy, and the remaining advantage is mainly in spatial alignment. The suggestion in §6 that gaze-guided initialization shapes how RL optimizes would need learning curves or sample-efficiency results to support it.
Questions for authors
-
Can you report a no-gaze control through the same pipeline, such as the "Base CoT (grounded)" condition shown in Figures 1 and 3, and a GGR run with shuffled or saliency-predicted fixations, to show how much of the gain comes from human gaze rather than the teacher?
-
How many participants took part, where do the stimulus images and questions come from, and do they overlap with the evaluation benchmarks?
-
How are Point-Grounded RL and Ungrounded RL defined, and does the gap between GGR-RL and Point-Grounded RL hold across multiple training seeds?
Please see the weaknesses above. If my concerns can be resolved, I would consider raising my score.
4 comments
Response to Reviewer sMmQ (2/2)
In the behavioral analysis of §5.4, the effects are small, on the order of a tenth of an occurrence per 100 reasoning words or less. The counts come from an LLM coder with no validation against human annotation.
To clarify the magnitude of the behavioral differences, we additionally report the behavior presence in Figure 5 in the updated draft. This complements the per-100-word measure by examining behaviors across traces. After RL, GGR-RL increases exploration by +3.9 percentage points and task decomposition by +18.7 percentage points in behavior presence relative to the base model, compared with +1.0 and −4.1 percentage points for No-Gaze RL (Figure 5a). Both differences are significant (Holm-adjusted p < .001).
The shared system prompt in Listing A.2 explicitly instructs the model to show planning, exploration, evaluation, and re-examination. That confounds any claim that gaze supervision "induces" these behaviors.
For all conditions, we train and evaluate with the same system prompt (Listing A.4 in the updated draft), and thus any differences observed between conditions are the result of other factors. We have updated the phrasing of "induce" to instead state "exhibits" in the paper.
How many participants took part, where do the stimulus images and questions come from, and do they overlap with the evaluation benchmarks?
We have expanded on the dataset details in Appendix A.1. We collected 7,109 gaze trials, with 6,411 used for SFT and 698 held out. Participants provided informed consent under an IRB-approved protocol, and gaze was recorded at 1,000 Hz. In total, we had 58 participants. Participants were compensated $20 per hour and we required them to be English speakers that had normal or corrected-to-normal vision.
There is no overlap between individual training and testing samples. At the source-dataset level, Visual Probe and ChartQA are used to curate training images and questions. The remaining four benchmarks have no training datasets and all conditions use the same training sets. We check and remove stimuli that share exact images or image–question pairs with the evaluation sets.
How are Point-Grounded RL and Ungrounded RL defined?
We have updated Section 4.2 in the updated draft with clear definitions. Ungrounded-RL and Grounded-RL use the same RL setup as GGR-RL, but run RL directly on the initial model either without requiring grounding (Ungrounded-RL) or with a grounding constraint requiring at least one coordinate to receive full format reward (Grounded-RL). We have now also included the No-Gaze condition in Tables 1 and 2. We have updated the captions of Tables 1 and 2 in Section 4 to state that error bars reflect evaluation sampling rather than independent training runs.
Thank you to the authors for the detailed response and the substantial experimental additions in the revised draft. Most of my technical and evaluation questions have been addressed.
However, the newly added No-Gaze control confirms my main concern: under the same teacher distillation and RL pipeline, human gaze yields only +1.1% before RL and zero overall accuracy gain after RL (68.8% vs. 68.8%). Reframing the central claim as "using gaze as a spatial prior that maintains final accuracy while improving gaze alignment and grounded-claim support" substantially weakens the significance of the work compared to the original premise that human gaze improves reasoning performance.
Under this concern, I do not find the remaining contribution strong enough to meet the ICLR bar, so I will maintain my score.
Thank you for the response and we really appreciate the continued engagement. The original abstract described GGR as "achieving greater human alignment while maintaining performance competitive with leading grounded-reasoning baselines." The revised draft makes these claims even more precise.
Beyond benchmark accuracy, obtaining predictive models of where people look while solving a task is a longstanding goal in computer science and cognitive science (E.g., Torralba et al., 2006). GGR's grounding locations better predict held-out human fixations than those of No-Gaze (1.15 vs. 0.99 bits/fixation, Table 2).
Additionally, beyond accuracy, reducing visual hallucinations has been another longstanding goal. Across 1,500 matched questions, the sampled claim was supported by the image and its point landed on or near the referenced target in 74.9% of GGR outputs versus 70.5% for No-Gaze (Figure 3).
We also contribute scientific findings and insights into gaze guidance transfer and how RL affects gaze alignment across conditions, with multiple metrics to understand how gaze and model training relate in grounded reasoners.
Finally, we publicly release our collected dataset of 7,109 active viewing trials of humans solving hard visual tasks while tracking their eye gaze.
We hope this clarifies the additional contributions beyond benchmark accuracy that would be useful for the ICLR community.
Response to Reviewer sMmQ (1/2)
We thank the reviewer for their detailed comments and thoughtful feedback. Below we address specific questions.
We ran the GGR pipeline with shuffled fixations across examples provided to the annotation model (Appendix Table 10 in the updated draft). Shuffling reduced the overall accuracy (65.3% vs. 63.4%) and gaze alignment (0.84 vs. 0.68 bits/fix) compared to the unshuffled condition. These results demonstrate the usefulness of task-specific gaze for both task performance and gaze spatial alignment.
We have additionally run the same GGR pipeline but without gaze conditioning (No-Gaze condition, Tables 1 and 2 in the updated draft). All annotation and training settings (system prompts, teacher model annotation, etc.) remain the same with the difference being the use of gaze or not. After RL, gaze-guided SFT -> RL maintains the same task performance, but with higher gaze alignment (1.15 ± 0.07 vs. 0.99 ± 0.07 bits/fixation under the condition-fitted metric; Table 1), a higher rate of supported claims with a grounding point on or near the target (74.9% vs. 70.5%; Holm-adjusted p < 0.01; Section 4.2; Figure 3; Appendix Table 8), and more up-front visual exploration before arriving at an answer (Section 4.4; Figure 5). Pre-RL, the gaze-guided model also improves overall task accuracy by 1.1 percentage points (65.3 ± 0.2% vs. 64.2 ± 0.2%; Table 1), while both conditions achieve 68.8% after RL.
Our central claim is that gaze-guided supervision achieves higher gaze spatial alignment and better visual grounding while maintaining strong task performance. We find that after RL, gaze-guided SFT initialization leads to more reliable grounding, higher alignment to gaze, and changes in reasoning behavior at the same accuracy compared to initialization without gaze. We have made our claims more precise in the updated draft in the abstract, introduction, and results.
We have included the ViGoRL SFT → RL results in the updated manuscript. After RL, we find that GGR outperforms ViGoRL-MCTS → RL in overall task accuracy (68.8 ± 0.2% vs. 63.8 ± 0.2%) and gaze alignment (1.15 ± 0.07 vs. 0.71 ± 0.04 bits/fixation; Table 2). GGR is especially better on counting (CountQA 41.8% vs. 30.0% after SFT; 50.7% vs. 35.7% after RL), where the MCTS training is not expressive enough to iterate over high object counts, and this persists after RL.
In Table 2 of the updated draft, we provide additional baseline comparisons to the teacher model Qwen3.5-122B-A10B, Vero-Qwen3I-8B, and DeepEyes-7B with zooming. We were not able to find a public checkpoint for ChartGaze, but have discussed it in Section 2 and Section 5. GGR-SFT and GGR-RL obtain 65.3 ± 0.2% and 68.8 ± 0.2% overall accuracy, respectively, outperforming Vero-Qwen3I-8B (64.2%) and DeepEyes-7B (50.2%). Additionally, our 8B GGR-RL model performs within 2.0 percentage points of the larger teacher Qwen3.5-122B-A10B (70.8%).
We agree that condition-specific KDE fitting makes cross-condition comparisons less direct. To address this concern, we recomputed alignment using a single shared parameter pair, σ = 70 and ε = 0.4, fitted exclusively on the 336 calibration trials with equal weight across 13 conditions. We then froze these parameters and evaluated every condition on the 362 scoring trials.
Under this shared metric, GGR outperformed No-Gaze by 0.1533 bits/fixation after SFT (95% CI [0.1179, 0.1891]) and 0.1222 after RL (95% CI [0.0844, 0.1598]) (p < 0.001). The advantages also hold when the parameters were fitted using only No-Gaze SFT calibration data (+0.1177 after SFT and +0.0802 after RL; both adjusted p < 0.001). The advantage is also present in every fifth of the human scanpath, including the first (Appendices A.6 and A.7; Figure 10).
We also agree that these results establish generalization to held-out trials within a similar gaze collection, rather than transfer to new participants or completely different question categories. We have revised the paper in Section 4 to clarify this generalization.