acceptodds
Under review as a conference paper at ICLR 2027

Learning Grounded Reasoning from Human Gaze

Abstract

Human visual reasoning is driven by systematic exploration and extraction of task-relevant visual evidence. Human gaze provides a window into this process, making gaze a useful source of guidance for visually-grounded reasoning in vision-language models (VLMs). However, human gaze also includes noise and redundancies that may transfer poorly to VLMs. In this work, we investigate how human gaze data can be effectively leveraged to improve visual reasoning in VLMs, while maintaining alignment with human behavior. We collect 7,202 human gaze trials from a diverse and challenging set of visual reasoning tasks, and train grounded reasoning models with different forms of gaze guidance. Across evaluations in chart understanding, visual search, and counting, we find that selective use of human gaze consistently outperforms supervision that incorporates the full gaze sequence, achieving greater human alignment while maintaining performance competitive with leading grounded-reasoning baselines. These gains persist through reinforcement learning, while gaze-guided models remain more spatially aligned with human behavior. Behavioral analyses further show that gaze supervision induces persistent differences in exploration and verification. Our results suggest that the value of human gaze lies in transferring its exploration structure without requiring complete imitation of human behavior.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, grounded visual reasoning, human gaze, gaze-guided supervision, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sMmQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how human gaze can be used to train VLMs to produce grounded reasoning, where each reasoning step is tied to an image coordinate. The authors collect about 7.2k eye-tracking trials on chart QA, counting, and visual search. They then propose GGR: a large teacher model receives the image, question, and human fixations, and writes a grounded reasoning trace using only a selected subset of the fixations. These traces are used to fine-tune Qwen3-VL-8B, followed by standard RL without gaze. At test time the model sees only the image and question. The main finding is that selective use of gaze works better than supervising every fixation. GGR also stays more aligned with human gaze than grounded baselines that do not use gaze, both before and after RL.

Critical strengths
  1. The human-collected dataset is of high quality. The eye-tracking data was collected with a research-grade tracker under an IRB protocol, and covers three fairly hard task types: chart understanding, counting, and high-resolution visual search. Gaze recorded during goal-directed problem solving, together with think-aloud audio, is rare. This rich gaze data can be used to fine-tune models on downstream tasks, and is likely to be useful beyond this paper. The authors commit to releasing it.

  2. The paper reaches an interesting conclusion about how to use gaze: selective use beats point-by-point imitation. The GGR vs. Fix. All comparison is clean, since both use the same teacher and pipeline and differ only in whether the teacher may skip fixations. GGR is better on every benchmark after SFT, while Fix. All is actually more aligned with human gaze. This dissociation, where closer imitation of human behavior does not give better reasoning, is useful for anyone thinking about learning from human behavioral data.

  3. The evaluation looks at the model from several angles, not just accuracy. The paper separately measures task accuracy, spatial alignment with held-out human fixations, and the reasoning behaviors expressed in the chain of thought. The alignment analysis goes beyond a single score: it reports coverage and precision and controls for the number of output points. The comparison with ViGoRL also uses the same teacher model, which makes the human-guided vs. model-guided comparison fairer than it would otherwise be.

Critical weaknesses
  1. The claim that human gaze makes the model reason better is not established, and the method does little to help the VLM actually exploit the gaze signal. In GGR, the decisions about which points to use and in what order are made by the 122B teacher. Gaze only narrows the candidate pool, and the prompt in Listing A.1 explicitly allows citing points "in any semantic order". Fixation order, durations, and the think-aloud transcripts are all discarded, and gaze plays no role in RL. This does not quite match the paper's framing that gaze transfers human "exploration structure". The SFT gain could largely come from distilling a much larger teacher with heavy rejection sampling rather than from the human fixations. The experiment that would separate the two is missing: the same pipeline with no fixations, or with non-human candidates. A "Base CoT (grounded)" condition appears in Figures 1 and 3 but not in Table 1. The Shuffled control does not fill this gap either: §5.1 says it was run on the Fix. condition, where it scores essentially the same as Fix. All, so comparing it with GGR in §5.2 does not support "task-specific fixation guidance matters".

  2. The baselines are limited, and the main table is essentially an internal ablation. Apart from three off-the-shelf models, every row in Table 1 is a variant the authors trained from the same base model. The main related work, ViGoRL, is reimplemented only for its MCTS data-construction step. There is no ViGoRL SFT → RL, even though the original method includes RL, and no comparison with its multi-turn variant that zooms into predicted coordinates. The RL recipe and most of the RL data come from Vero, yet the Vero paper's own model trained from the same base, Vero-Qwen3I-8B, is not reported. More broadly, there are few external baselines on the same benchmarks. Visual Probe comes from the Mini-o3 paper, and the RL data includes Pixel Reasoner subsets, yet none of the "thinking with images" methods that crop and zoom, such as Mini-o3, Pixel Reasoner, or DeepEyes, are evaluated. Prior gaze-supervision methods such as ChartGaze are also missing. This comparison matters because participants could zoom during data collection, and §6 itself notes that emitting a coordinate does not give the model a new view. Without these baselines, the abstract's claim of being "competitive with leading grounded-reasoning baselines" is hard to assess.

  3. The evidence for transfer is weaker than presented. The "higher alignment does not mean better performance" effect is observed, but the alignment metric fits the KDE bandwidth σ and background weight ε separately for each condition (Appendix A.6), which makes cross-condition comparisons less direct. The held-out trials also come from the same collection as the training gaze, so gaze-trained models scoring higher is partly expected. In the behavioral analysis of §5.4, the effects are small, on the order of a tenth of an occurrence per 100 reasoning words or less. The counts come from an LLM coder with no validation against human annotation. On top of that, the shared system prompt in Listing A.2 explicitly instructs the model to show planning, exploration, evaluation, and re-examination. That confounds any claim that gaze supervision "induces" these behaviors.

  4. After RL, the performance benefit of gaze supervision is marginal, and most of the final gain comes from the borrowed RL stage. GGR-RL beats Point-Grounded RL by only 0.6 points. The reported error bars come from repeated evaluation sampling only, not from different training seeds, so this gap is not shown to be robust. Meanwhile RL alone, apparently starting from the base model, already improves accuracy by about 11 points. The RL data is much larger than the gaze SFT data and mostly taken from Vero together with its recipe, the reward contains no gaze term, and there is no ablation of the RL data. So by the end of training, gaze contributes very little measurable accuracy, and the remaining advantage is mainly in spatial alignment. The suggestion in §6 that gaze-guided initialization shapes how RL optimizes would need learning curves or sample-efficiency results to support it.

Questions for authors
  1. Can you report a no-gaze control through the same pipeline, such as the "Base CoT (grounded)" condition shown in Figures 1 and 3, and a GGR run with shuffled or saliency-predicted fixations, to show how much of the gain comes from human gaze rather than the teacher?

  2. How many participants took part, where do the stimulus images and questions come from, and do they overlap with the evaluation benchmarks?

  3. How are Point-Grounded RL and Ungrounded RL defined, and does the gap between GGR-RL and Point-Grounded RL hold across multiple training seeds?

Please see the weaknesses above. If my concerns can be resolved, I would consider raising my score.

4 comments
AuthorsOpenReview ↗

Response to Reviewer sMmQ (1/2)

We thank the reviewer for their detailed comments and thoughtful feedback. Below we address specific questions.

How much of the gain comes from human gaze rather than the teacher? Can you report a no-gaze control through the same pipeline, and a GGR run with shuffled or saliency-predicted fixations? After RL, the performance benefit of gaze supervision is marginal, and most of the final gain comes from the borrowed RL stage.

We ran the GGR pipeline with shuffled fixations across examples provided to the annotation model (Appendix Table 10 in the updated draft). Shuffling reduced the overall accuracy (65.3% vs. 63.4%) and gaze alignment (0.84 vs. 0.68 bits/fix) compared to the unshuffled condition. These results demonstrate the usefulness of task-specific gaze for both task performance and gaze spatial alignment.

We have additionally run the same GGR pipeline but without gaze conditioning (No-Gaze condition, Tables 1 and 2 in the updated draft). All annotation and training settings (system prompts, teacher model annotation, etc.) remain the same with the difference being the use of gaze or not. After RL, gaze-guided SFT -> RL maintains the same task performance, but with higher gaze alignment (1.15 ± 0.07 vs. 0.99 ± 0.07 bits/fixation under the condition-fitted metric; Table 1), a higher rate of supported claims with a grounding point on or near the target (74.9% vs. 70.5%; Holm-adjusted p < 0.01; Section 4.2; Figure 3; Appendix Table 8), and more up-front visual exploration before arriving at an answer (Section 4.4; Figure 5). Pre-RL, the gaze-guided model also improves overall task accuracy by 1.1 percentage points (65.3 ± 0.2% vs. 64.2 ± 0.2%; Table 1), while both conditions achieve 68.8% after RL.

Our central claim is that gaze-guided supervision achieves higher gaze spatial alignment and better visual grounding while maintaining strong task performance. We find that after RL, gaze-guided SFT initialization leads to more reliable grounding, higher alignment to gaze, and changes in reasoning behavior at the same accuracy compared to initialization without gaze. We have made our claims more precise in the updated draft in the abstract, introduction, and results.

The baselines are limited, and the main table is essentially an internal ablation. There is no ViGoRL SFT → RL, Vero, or thinking with images baselines.

We have included the ViGoRL SFT → RL results in the updated manuscript. After RL, we find that GGR outperforms ViGoRL-MCTS → RL in overall task accuracy (68.8 ± 0.2% vs. 63.8 ± 0.2%) and gaze alignment (1.15 ± 0.07 vs. 0.71 ± 0.04 bits/fixation; Table 2). GGR is especially better on counting (CountQA 41.8% vs. 30.0% after SFT; 50.7% vs. 35.7% after RL), where the MCTS training is not expressive enough to iterate over high object counts, and this persists after RL.

In Table 2 of the updated draft, we provide additional baseline comparisons to the teacher model Qwen3.5-122B-A10B, Vero-Qwen3I-8B, and DeepEyes-7B with zooming. We were not able to find a public checkpoint for ChartGaze, but have discussed it in Section 2 and Section 5. GGR-SFT and GGR-RL obtain 65.3 ± 0.2% and 68.8 ± 0.2% overall accuracy, respectively, outperforming Vero-Qwen3I-8B (64.2%) and DeepEyes-7B (50.2%). Additionally, our 8B GGR-RL model performs within 2.0 percentage points of the larger teacher Qwen3.5-122B-A10B (70.8%).

The alignment metric fits the KDE bandwidth σ and background weight ε separately for each condition (Appendix A.6), which makes cross-condition comparisons less direct. The held-out trials also come from the same collection as the training gaze, so gaze-trained models scoring higher is partly expected.

We agree that condition-specific KDE fitting makes cross-condition comparisons less direct. To address this concern, we recomputed alignment using a single shared parameter pair, σ = 70 and ε = 0.4, fitted exclusively on the 336 calibration trials with equal weight across 13 conditions. We then froze these parameters and evaluated every condition on the 362 scoring trials.

Under this shared metric, GGR outperformed No-Gaze by 0.1533 bits/fixation after SFT (95% CI [0.1179, 0.1891]) and 0.1222 after RL (95% CI [0.0844, 0.1598]) (p < 0.001). The advantages also hold when the parameters were fitted using only No-Gaze SFT calibration data (+0.1177 after SFT and +0.0802 after RL; both adjusted p < 0.001). The advantage is also present in every fifth of the human scanpath, including the first (Appendices A.6 and A.7; Figure 10).

We also agree that these results establish generalization to held-out trials within a similar gaze collection, rather than transfer to new participants or completely different question categories. We have revised the paper in Section 4 to clarify this generalization.

AuthorsOpenReview ↗

Response to Reviewer sMmQ (2/2)

In the behavioral analysis of §5.4, the effects are small, on the order of a tenth of an occurrence per 100 reasoning words or less. The counts come from an LLM coder with no validation against human annotation.

To clarify the magnitude of the behavioral differences, we additionally report the behavior presence in Figure 5 in the updated draft. This complements the per-100-word measure by examining behaviors across traces. After RL, GGR-RL increases exploration by +3.9 percentage points and task decomposition by +18.7 percentage points in behavior presence relative to the base model, compared with +1.0 and −4.1 percentage points for No-Gaze RL (Figure 5a). Both differences are significant (Holm-adjusted p < .001).

The shared system prompt in Listing A.2 explicitly instructs the model to show planning, exploration, evaluation, and re-examination. That confounds any claim that gaze supervision "induces" these behaviors.

For all conditions, we train and evaluate with the same system prompt (Listing A.4 in the updated draft), and thus any differences observed between conditions are the result of other factors. We have updated the phrasing of "induce" to instead state "exhibits" in the paper.

How many participants took part, where do the stimulus images and questions come from, and do they overlap with the evaluation benchmarks?

We have expanded on the dataset details in Appendix A.1. We collected 7,109 gaze trials, with 6,411 used for SFT and 698 held out. Participants provided informed consent under an IRB-approved protocol, and gaze was recorded at 1,000 Hz. In total, we had 58 participants. Participants were compensated $20 per hour and we required them to be English speakers that had normal or corrected-to-normal vision.

There is no overlap between individual training and testing samples. At the source-dataset level, Visual Probe and ChartQA are used to curate training images and questions. The remaining four benchmarks have no training datasets and all conditions use the same training sets. We check and remove stimuli that share exact images or image–question pairs with the evaluation sets.

How are Point-Grounded RL and Ungrounded RL defined?

We have updated Section 4.2 in the updated draft with clear definitions. Ungrounded-RL and Grounded-RL use the same RL setup as GGR-RL, but run RL directly on the initial model either without requiring grounding (Ungrounded-RL) or with a grounding constraint requiring at least one coordinate to receive full format reward (Grounded-RL). We have now also included the No-Gaze condition in Tables 1 and 2. We have updated the captions of Tables 1 and 2 in Section 4 to state that error bars reflect evaluation sampling rather than independent training runs.

Thank you to the authors for the detailed response and the substantial experimental additions in the revised draft. Most of my technical and evaluation questions have been addressed.

However, the newly added No-Gaze control confirms my main concern: under the same teacher distillation and RL pipeline, human gaze yields only +1.1% before RL and zero overall accuracy gain after RL (68.8% vs. 68.8%). Reframing the central claim as "using gaze as a spatial prior that maintains final accuracy while improving gaze alignment and grounded-claim support" substantially weakens the significance of the work compared to the original premise that human gaze improves reasoning performance.

Under this concern, I do not find the remaining contribution strong enough to meet the ICLR bar, so I will maintain my score.

Thank you for the response and we really appreciate the continued engagement. The original abstract described GGR as "achieving greater human alignment while maintaining performance competitive with leading grounded-reasoning baselines." The revised draft makes these claims even more precise.

Beyond benchmark accuracy, obtaining predictive models of where people look while solving a task is a longstanding goal in computer science and cognitive science (E.g., Torralba et al., 2006). GGR's grounding locations better predict held-out human fixations than those of No-Gaze (1.15 vs. 0.99 bits/fixation, Table 2).

Additionally, beyond accuracy, reducing visual hallucinations has been another longstanding goal. Across 1,500 matched questions, the sampled claim was supported by the image and its point landed on or near the referenced target in 74.9% of GGR outputs versus 70.5% for No-Gaze (Figure 3).

We also contribute scientific findings and insights into gaze guidance transfer and how RL affects gaze alignment across conditions, with multiple metrics to understand how gaze and model training relate in grounded reasoners.

Finally, we publicly release our collected dataset of 7,109 active viewing trials of humans solving hard visual tasks while tracking their eye gaze.

We hope this clarifies the additional contributions beyond benchmark accuracy that would be useful for the ICLR community.

Reviewer_eYWcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates how human gaze can be used to improve grounded visual reasoning in VLMs. The authors introduce GGR (Gaze-Guided Reasoner), which uses human fixations as candidate evidence locations and a strong VLM teacher to selectively construct grounded reasoning traces. These traces are used to fine-tune Qwen3-VL-8B, followed by reinforcement learning without additional gaze supervision. The authors collect 7,202 human eye-tracking trials across chart understanding, counting, and visual search tasks. Experiments show that selective gaze supervision outperforms full-fixation imitation during SFT, achieving higher task accuracy while maintaining strong spatial alignment with human gaze. The alignment advantages largely persist after RL, suggesting that gaze-guided supervision can influence subsequent grounded reasoning behavior.

Critical strengths

1. A meaningful and well-motivated research problem.

The paper explores an interesting intersection between human visual cognition and VLM reasoning. Rather than treating gaze simply as an attention map, it investigates how human visual exploration can provide structured guidance for grounded reasoning. I find this perspective valuable, particularly given the increasing interest in improving the faithfulness and interpretability of multimodal reasoning.

2. A valuable human eye-tracking dataset.

Collecting 7,202 gaze trials across chart understanding, counting, and visual search is a substantial effort. The inclusion of challenging, reasoning-oriented tasks makes the dataset particularly interesting, as it captures goal-directed visual exploration rather than passive viewing. The planned release of the dataset could also benefit future research on human-VLM alignment.

3. An interesting finding regarding selective gaze supervision.

One of the most compelling observations is that imitating every human fixation does not necessarily improve visual reasoning. The comparison between GGR and Fix. All provides useful empirical evidence that selective supervision can be more effective than directly transferring complete human gaze traces. This is an insightful finding that acknowledges the differences between human perception and VLM processing.

4. Comprehensive evaluation beyond task accuracy.

I appreciate that the authors evaluate not only task performance but also spatial gaze alignment, precision, coverage, and expressed reasoning behaviors. The analysis across SFT and RL stages is particularly interesting, as it suggests that gaze-guided initialization can produce persistent differences in model behavior even without explicit gaze rewards during RL.

Critical weaknesses

1. The final task improvement is marginal, and its contribution cannot be clearly attributed to human gaze.

GGR-RL achieves 67.3% overall accuracy, only 0.6 percentage points above Point-Grounded RL (66.7%), which starts directly from the same base model. Most of this difference comes from ChartQA-Pro (+4.1 points). When averaged within each task domain, GGR-RL actually performs slightly worse on counting and visual search.

Furthermore, two other gaze-supervised variants, Fix. + Think-Aloud (63.3%) and Fix. All (65.3%), perform worse after RL than Point-Grounded RL without gaze supervision (66.7%). This suggests that gaze supervision alone does not reliably improve task performance; the benefit appears specific to the selective annotation pipeline.

Crucially, GGR's training traces are generated and filtered by a 122B teacher model, with human gaze providing only candidate coordinates. The paper does not isolate whether the improvement comes from human gaze or from the teacher's selection, reasoning generation, and rejection sampling.

A direct baseline using the same teacher, annotation procedure, and training configuration, but allowing the teacher to select grounding points without human gaze, is missing. ViGoRL-MCTS does not fully address this concern because it uses a different search procedure, answer-guided trajectory selection, and correction examples.

2. The shuffled-fixation baseline does not clearly support the claimed benefit of task-specific gaze guidance.

Section 5.1 describes Shuffled as "Our Fix. Full condition using shuffled fixation lists across examples," but Fix. Full is not otherwise defined.

If Fix. Full refers to Fix. All, the appropriate controlled comparison would be Fix. All (55.9%) versus Shuffled (55.8%). These results show virtually no difference in task accuracy, despite a substantial difference in gaze alignment (1.04 versus 0.01 bits/fix).

However, Section 5.2 compares GGR-SFT (62.8%) with Shuffled (55.8%) and attributes the difference to task-specific gaze guidance. If Shuffled follows the full-fixation pipeline, this comparison also changes the selective annotation strategy and therefore cannot isolate the effect of correctly matched gaze.

The authors should clarify the construction of Shuffled and provide a shuffled-gaze baseline using the same selective GGR annotation pipeline.

3. Teacher-generated supervision lacks independent quality validation.

The same Qwen3.5-122B-A10B model is used for fixation descriptions, task-relevance annotation, reasoning generation, rejection sampling, and behavioral evaluation.

The paper does not report an independent human audit of the generated reasoning traces, rejection acceptance rates, or the distribution of retained training samples across conditions. These omissions make it difficult to assess the quality and comparability of the supervision datasets.

The behavioral analysis in Section 5.4 also relies on the same teacher model to identify exploration and evaluation behaviors in student-generated reasoning. This creates a potential evaluator-dependence issue, particularly because the measurements concern expressed reasoning patterns rather than independently verified visual actions, as acknowledged in Table 5.

4. Temporal gaze information is largely discarded despite its importance in the motivation.

The introduction motivates gaze through sequential visual exploration, such as moving from a chart legend to a curve, then to a relevant point and its corresponding axis.

However, GGR allows the teacher to reorder selected fixations arbitrarily. The training procedure does not explicitly use fixation duration, temporal transitions, or recorded zoom and marker interactions as reasoning supervision. The gaze-alignment metric also ignores fixation order.

5. Spatial gaze alignment does not establish faithful visual grounding.

The reported gaze-alignment and precision metrics measure whether model-generated grounding coordinates are spatially close to human fixations. For example, the reported precision of approximately 86–88% does not establish that the model actually relies on these regions when producing its answers.

The model may learn to generate plausible human-aligned coordinates without using the corresponding visual evidence.

A controlled intervention experiment, such as masking or replacing cited regions and comparing the resulting answer changes against uncited or random regions, would provide stronger evidence of faithful grounding.

6. Human data collection and potential benchmark overlap are insufficiently documented.

Although the paper reports 7,202 eye-tracking trials, it does not clearly specify the number of participants, trials per participant, participants per image, participant answer accuracy, or the number of trials excluded during quality control.

More importantly, the sources of the stimulus images and their relationship to the six evaluation benchmarks are not sufficiently documented.

Since the pipeline generates reasoning traces and filters them using answer correctness, any overlap between training stimuli and benchmark test examples could introduce evaluation contamination. The overlap between the Visual Probe data used during RL and the Visual Probe evaluation benchmark also requires clarification.

The paper should explicitly document dataset sources, split construction, and deduplication procedures.

7. Stronger model comparisons and clearer evaluation protocols are needed.

The reported zero-shot baselines include Qwen3-VL-8B-Instruct, Molmo2-O-7B, and GPT-5-nano. However, the 122B teacher's own performance is not reported, nor are results from stronger contemporary VLMs. These comparisons would help contextualize the practical significance of the proposed method.

Additionally, the paper selects the best checkpoints for both SFT and RL using validation accuracy, but does not clearly specify the validation datasets or their separation from the final benchmarks.

Finally, ViGoRL-MCTS is evaluated after SFT but not after the same RL procedure, leaving an important gap in the comparison between gaze-guided and model-generated reasoning supervision.

Questions for authors

1. How exactly is the Shuffled baseline constructed?

What does "Fix. Full" refer to in Section 5.1? If Shuffled uses the Fix. All pipeline, why does Section 5.2 compare it against GGR-SFT rather than Fix. All? Please provide a shuffled-fixation control using the same selective GGR annotation pipeline.

2. Can the authors isolate the contribution of human gaze from teacher-generated reasoning?

Two particularly important ablations are:

  • Teacher-only grounding: Use the same three-stage annotation pipeline, sampling, and rejection procedure, but allow the teacher to select grounding points without human fixations.
  • Supervision decomposition: Separately evaluate coordinate-only and reasoning-text-only supervision to determine which component contributes to task performance and spatial alignment.

A comparison with saliency-based or model-predicted fixation candidates would further clarify whether human eye-tracking data is necessary.

3. Can the authors provide more complete information about the human dataset and potential data overlap?

Please report the number of participants, trials per participant, number of observers per image, participant answer accuracy, quality-control exclusions, and sources of stimulus images.

Please also clarify whether any SFT or RL training images overlap with the six evaluation benchmarks, particularly the Visual Probe subsets used during RL.

4. How reliable are the teacher-generated annotations and behavioral evaluations?

Please report the acceptance rate of generated reasoning traces and provide an independent annotation-quality assessment. Has the behavioral analysis been validated against human judgments or an independent evaluation model?

5. Can the authors clarify the evaluation protocol and provide stronger reference baselines?

Please report the zero-shot performance of Qwen3.5-122B-A10B and at least one stronger contemporary VLM on the six benchmarks.

Which validation datasets are used for SFT and RL checkpoint selection? Are they strictly separated from the final evaluation sets?

Additionally, can the authors evaluate ViGoRL-MCTS SFT followed by the same GSPO RL procedure used for GGR?

4 comments
AuthorsOpenReview ↗

Response to Reviewer eYWc (1/2)

We thank the reviewer for their detailed comments and thoughtful feedback. Below we address specific questions.

Can the authors isolate the contribution of human gaze from teacher-generated reasoning? Can you run teacher-only grounding and supervision decomposition ablations?

We have included results from the same GGR pipeline without gaze conditioning in Table 1 of the updated draft. All annotation and training settings (system prompts, teacher model annotation, etc.) remain the same with the difference being the use of gaze or not. After RL, gaze-guided SFT -> RL maintains the same task performance (68.8% overall), but with higher gaze alignment (1.15 ± 0.07 vs. 0.99 ± 0.07 bits/fixation; Table 1), a higher rate of supported claims with a grounding point on or near the target (74.9% vs. 70.5%; p < 0.01; Figure 3; Appendix Table 8), and more up-front visual exploration before arriving at an answer (Section 4.4; Figure 5). Pre-RL, the gaze-guided model also improves overall task accuracy by 1.1 percentage points (65.3 ± 0.2% vs. 64.2 ± 0.2%; Table 1). Our central claim is that gaze-guided supervision achieves higher gaze spatial alignment and better visual grounding while maintaining strong task performance, and we have made this more precise throughout the draft.

We have additionally run the requested ablations on GGR-SFT using coordinate-only and reasoning-text-only supervision in Appendix Table 10 of the updated draft. Coordinate-only supervision retains a portion of the GGR-SFT gaze alignment (0.7784 versus 0.84 bits/fix), but with lower overall accuracy (65.3% to 58.57%). Text-only supervision also lowers accuracy (61.87%), while gaze alignment drops to nearly 0 due to its lack of grounded reasoning (0.0018 bits/fix). Thus, joint supervision of the coordinates and text reasoning therefore achieves higher overall accuracy and gaze alignment than either component alone.

Spatial gaze alignment does not establish faithful visual grounding.

To address your concern, we examine the visual grounding quality of grounded reasoning between GGR and GGR without gaze (No-Gaze in Tables 1 and 2 in the updated draft). We use GPT-6.1-Sol-xhigh given privileged zoomed-in views to assess grounding locations and their associated reasoning steps (N = 1,500). We validate the model ratings with two human raters (N = 100 samples; 84–90% agreement with the model, 90–92% agreement between the two raters; Appendix Table 7).

Compared to our method without gaze guidance (No-Gaze), GGR obtains a higher rate of supported claims that are correctly grounded after both SFT and RL, with gains of 4.5 and 4.4 percentage points, respectively (Figure 3; Appendix Figure 12; p = 0.0025). GGR-RL additionally achieves better grounding accuracy regardless of whether the claim is supported (Appendix Table 8 Point on/near target; 84.1% with GGR-RL compared with 79.8% for No-Gaze-RL; p = 0.0016). Our results demonstrate that gaze guidance improves the rate of supported reasoning claims and accuracy of the visual grounding.

Please provide a shuffled-fixation control using the same selective GGR annotation pipeline.

We ran the GGR pipeline with shuffled fixations across examples provided to the annotation model (Appendix Table 10 in the updated draft). Shuffling reduced the overall accuracy (65.3% vs. 63.4%) and gaze alignment (0.84 vs. 0.68 bits/fix) compared to the unshuffled condition. These results demonstrate the usefulness of task-specific gaze for both task performance and gaze spatial alignment.

Temporal gaze information is largely discarded despite its importance in the motivation.

To test how the alignment gain relates to the temporal structure of human gaze, we added a time-resolved alignment analysis and two fixation order-aware metrics (Appendix A.7; Figure 10 in the updated draft). We split each held-out scanpath into fifths in temporal order and compute our alignment metric on each fifth, with parameters shared across conditions. GGR is more aligned than No-Gaze in every fifth of the scanpath, from the first (+0.14 bits/fixation after SFT, 95% CI [0.09, 0.18]; +0.08 after RL, 95% CI [0.04, 0.13]) to the last (+0.18 and +0.13; all p < 0.001).

GGR traces are also more similar to human scanpaths under the ScanMatch-style metric (0.078 vs. 0.071 after SFT; 0.119 vs. 0.100 after RL; both p < 0.001). Fixation order itself is only weakly consistent in human gaze (split-half order score of 0.19 on a scale from −1 to 1), so we expect order-specific differences to be small.

We agree, however, that GGR does not explicitly supervise fixation order. To clarify this, we have revised the introduction and methods to state that GGR uses gaze to guide where the model looks.

AuthorsOpenReview ↗

Response to Reviewer eYWc (2/2)

The 122B teacher's own performance is not reported, nor are results from stronger contemporary VLMs. ViGoRL-MCTS is evaluated after SFT but not after the same RL procedure.

In Table 2 of the updated draft, we provide additional baseline comparisons to the teacher model Qwen3.5-122B-A10B, Vero-Qwen3I-8B, and DeepEyes-7B with zooming. GGR-SFT and GGR-RL obtain 65.3 ± 0.2% and 68.8 ± 0.2% overall accuracy, respectively, outperforming Vero-Qwen3I-8B (64.2%) and DeepEyes-7B (50.2%). Additionally, our 8B GGR-RL model performs within 2.0 percentage points of the larger teacher Qwen3.5-122B-A10B (70.8%).

We have additionally included the ViGoRL SFT → RL results in the updated manuscript. After RL, we find that GGR outperforms ViGoRL-MCTS → RL in overall task accuracy (68.8 ± 0.2% vs. 63.8 ± 0.2%) and gaze alignment (1.15 ± 0.07 vs. 0.71 ± 0.04 bits/fixation; Table 2). GGR is especially better on counting (CountQA 41.8% vs. 30.0% after SFT; 50.7% vs. 35.7% after RL), where the MCTS training is not expressive enough to iterate over high object counts, and this persists after RL.

How reliable are the teacher-generated annotations?

We validate the model ratings of the rejection sampling with two human raters for randomly selected samples (N = 50 samples). The participants had 100% agreement with the model ratings for whether the answer matched the reference (49/49), 96% for absence of numeric coordinates, 94% for internal contradictions, and 90% agreement for severe repetition.

Can the authors provide more complete information about the human dataset and potential data overlap?

We have expanded on the dataset details in Appendix A.1. We collected 7,109 gaze trials, with 6,411 used for SFT and 698 held out. Participants provided informed consent under an IRB-approved protocol, and gaze was recorded at 1,000 Hz. In total, we had 58 participants, compensated $20 per hour, and required English speakers that had normal or corrected-to-normal vision.

There is no overlap between individual training and testing samples. At the source-dataset level, Visual Probe and ChartQA training sets are used to curate a proportion of training images and questions. The remaining four benchmarks have no training datasets and all conditions use the same training sets. We check and remove stimuli that share exact images or image–question pairs with the evaluation sets. We use the 698 human trials as our validation set and only select based on accuracy. We have updated Sections 3.2 and 4 to clarify this.

Reviewer_eYWcOpenReview ↗

Response

Thank you for the detailed response and the additional experiments. Many of my concerns have been addressed. However, the reply to Weakness 1 (Question 2) raises a new concern about the contribution of human gaze.

1. Human gaze does not appear to contribute to final accuracy or to most of the alignment.

With the same pipeline, the No-Gaze variant reaches the same accuracy after RL (68.8%) and already obtains most of the gaze alignment (0.99 vs. 1.15 bits/fixation). This suggests that the teacher's own point selection accounts for most of the benefit.

2. Fixation order is not supervised.

As acknowledged in the response to Weakness 4, GGR does not supervise fixation order, which is arguably the aspect of human gaze most relevant to step-by-step reasoning.

Question:

Given the above, could the authors clarify why human gaze is still needed for this task? Specifically:

  • When accuracy is the same, what is the practical benefit of the higher gaze alignment and the 4.4-point gain in grounded claims?
  • Is there evidence that these gains matter downstream, for example with a weaker teacher, on domains where the teacher's prior is poor, or for interpretability as judged by humans?

Human gaze does not appear to contribute to final accuracy or to most of the alignment.

Thank you for the continued discussion. We agree that our results show that human gaze guidance maintains, but does not improve upon, the accuracy of the No-Gaze-RL condition. The contribution of gaze in this work is higher gaze alignment and more accurate grounding of visual claims. We also agree that the strong No-Gaze alignment result is an important one to highlight, and have updated Sections 4.1 and 4.2 to state these results more clearly. The practical benefit comes from reasoning explanations that are more accurately grounded, making it easier to verify the explanations in the image.

More broadly, our paper contributes to a better understanding of how human gaze transfers to VLM reasoning, and how various training conditions change gaze alignment and accuracy. For example, we find that selective use of gaze is more effective than supervising every fixation, and that RL increases gaze alignment through better coverage (Section 4.3).

We appreciate the suggestions regarding using a weaker teacher and interpretability as judged by human evaluators, and we view these as exciting future directions.

GGR does not supervise fixation order, which is arguably the aspect of human gaze most relevant to step-by-step reasoning.

We agree that fixation order is a potentially useful aspect of gaze, and that GGR does not supervise it. Our study and tasks focus on the contribution of spatial guidance. On the tasks studied here, human participants viewing the same image and answering the same question showed low agreement in the order they visited shared regions (mean γ = 0.086 across 212 tasks, 95% CI [0.033, 0.140]). Thus, fixation order is weakly consistent even across humans.

This does not suggest that ordering is unimportant for a given reasoning trace, but indicates there may be multiple useful orderings for tasks explored in this work. For example, on both visual search and chart problems, different participants often visit the relevant regions (searching around salient parts of a scene, visiting legend and axis labels, etc) in different orders. We therefore use spatial alignment as the primary metric and have updated the draft text accompanying Figure 10 to clarify this.

Reviewer_sY27OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors explore gaze-guided visually-grounded reasoning in vision-language models (VLMs). To mitigate noise inherent in human gaze, the authors propose Gaze-Guided Reasoning (GGR), which generates grounded reasoning traces from human fixations using a strong annotation model. To train their VLM, the authors also collect a multi-domain gaze fixation trace dataset consisting of more than 7K eye-tracking trials. Through a multi-faceted evaluation method, the authors show that their VLM trained with SFT and RL using GGR outperforms baselines while retaining alignment with human gaze.

Critical strengths
  1. The experimental evaluation and subsequent analysis is quite rigorous and reveals interesting trends of model reasoning behavior.
Critical weaknesses
  1. The GGR-SFT method closely resembles Mondal et al. [1] which introduced a similar technique for generating transcripts to guide gaze decoding. The authors should cite, discuss and compare.

  2. The authors claim to achieve greater gaze alignment, but mention in the experimental setup section (L. 322) that the metric capturing the said alignment "does not capture fixation order, duration, or the model’s internal attention." Reasoning usually follows an order and discarding that information even though the data contains order information doesn't make sense to me. Several metrics like ScanMatch [2] are prevalent in human gaze literature to compare gaze traces.

[1] Mondal, Sounak, Dimitris Samaras, Gregory Zelinsky, and Minh Hoai. "Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention." In European Conference on Computer Vision, pp. 411-430. Cham: Springer Nature Switzerland, 2026.

[2] Cristino, Filipe, Sebastiaan Mathôt, Jan Theeuwes, and Iain D. Gilchrist. "ScanMatch: A novel method for comparing fixation sequences." Behavior research methods 42, no. 3 (2010): 692-700.

Questions for authors
  1. Can the authors report some fixation order-aware similarity metrics for model-generated traces and real human traces?

  2. From the representative qualitative example in Figure 5, I could see that the model is expected to generate actual unnormalized image coordinates for fixations. However, images can be of different dimensions and are usually resized to a constant shape by the model preprocessing module. Did the authors consider normalizing the coordinates to a uniform range like 0-1?

5 comments
AuthorsOpenReview ↗

Response to Reviewer sY27

We thank the reviewer for their detailed comments and thoughtful feedback. Below we address specific questions.

The GGR-SFT method closely resembles Mondal et al. [1] which introduced a similar technique for generating transcripts to guide gaze decoding. The authors should cite, discuss and compare.

Thank you for highlighting this closely related work. Gazette and GGR both construct language supervision from human gaze, but use it for different objectives. Gazette learns to infer an observer’s task or goal from an image and scanpath, using synthetic attention-allocation transcripts as auxiliary supervision, while GGR uses gaze to construct grounded reasoning supervision for answering a given question, with the trained student subsequently operating without human gaze supervision. We have included this discussion of Gazette in the updated draft in Section 2.

We have also run a comparison between GGR and Gazette that examines how well these methods preserve human attention in the resulting reasoning traces (Appendix Section A.7 in the updated draft). Using the same teacher model, we compared a selective-backfill variant with a Gazette-inspired narrative-to-reasoning adaptation on trials held out from our initial training. On these trials, GGR backfills achieved higher ScanMatch similarity (0.172 versus 0.103; paired difference +0.069, 95% CI [0.053, 0.085]) and lower human-to-trace spatial coverage distance (0.147 versus 0.207, lower is better). These results indicate greater preservation of the supplied gaze’s spatial coverage and ordering in the generated supervision.

The authors claim to achieve greater gaze alignment, but mention in the experimental setup section (L. 322) that the metric capturing the said alignment "does not capture fixation order, duration, or the model’s internal attention." Can the authors report some fixation order-aware similarity metrics for model-generated traces and real human traces?

We now report fixation order-aware metrics between model traces and human scanpaths (Appendices A.6 and A.7; Figure 10 in the updated draft). We compute (i) a ScanMatch-style similarity, which aligns each model's sequence of grounded regions to the human fixation sequence with Needleman–Wunsch alignment, together with a shuffled-order baseline that separates order from spatial overlap, and (ii) our alignment metric computed separately within each fifth of the human scanpath.

Under the ScanMatch-style metric, GGR traces are more similar to human scanpaths than No-Gaze traces (0.078 vs. 0.071 after SFT; 0.119 vs. 0.100 after RL; both p < 0.001), and GGR is more aligned in every fifth of the scanpath, including the first (+0.14 bits/fixation after SFT, 95% CI [0.09, 0.18]; +0.08 after RL, 95% CI [0.04, 0.13]; all fifths p < 0.001).

From the representative qualitative example in Figure 5, I could see that the model is expected to generate actual unnormalized image coordinates for fixations. However, images can be of different dimensions and are usually resized to a constant shape by the model preprocessing module. Did the authors consider normalizing the coordinates to a uniform range like 0-1?

The coordinates are indeed normalized to a uniform range of 0-1000, while the preprocessing module adapts the number of image tokens to the image size. We choose this to be consistent with the grounding format of the pretrained models we are finetuning.

Reviewer_sY27OpenReview ↗

Thanks and additional query

Thank you for the clarification and the updated manuscript. I have one query: as per Fig. 10, No-Gaze RL appears to outperform GGR-RL on order agreement. Can you please comment on that?

Thank you for the continued discussion and for raising this point. You are right that the No-Gaze RL has a slightly higher mean order agreement. However, the paired difference for the order agreement metric was statistically nonsignificant (95% CI [−0.013, 0.054], p = 0.23).

For context, fixation order via the ordering agreement is not strongly consistent even across humans, and requires very fine-grained pairwise agreement of visit order across regions. Human participants viewing the same image and answering the same question showed low agreement in the order they visited shared regions (mean γ = 0.086 across 212 tasks, 95% CI [0.033, 0.140]).** Additionally, a trace built from half of a person's own fixations reaches an order score of only 0.19 against the other half (on a scale from −1 to 1).

We use spatial alignment as the primary metric to capture whether the model reasons about similar visual regions that humans visit. This is also a reflection of the tasks for which we collected human gaze data. For example, on both visual search and chart problems, different participants often visit the relevant regions (searching around salient parts of a scene, visiting legend and axis labels, etc) in different orders. GGR has higher ScanMatch similarity, and higher spatial alignment in each fifth of the human scanpath, while the difference in order agreement is not statistically significant. We have updated the paper text accompanying Figure 10 to distinguish these findings and claims.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Response and Summary of Revisions

We would like to thank the reviewers for their thoughtful feedback. We appreciate the recognition of the value of our gaze dataset and the rigor and usefulness of our experimental evaluation. Our work contributes a dataset of 7,109 human gaze trials during active viewing and trains GGR, which uses human gaze as a spatial prior for constructing grounded reasoning supervision. Our central finding is that gaze-guided supervision improves human-gaze alignment and visual grounding while maintaining strong task performance. The additional experiments in the updated draft strengthen our findings in the following ways:

  1. Contribution of human gaze. We compare GGR with the same methodology without gaze guidance, and find that GGR achieves higher gaze alignment (1.15 vs. 0.99 bits/fix) and a higher rate of correctly grounded claims, while maintaining accuracy (eYWc, sMmQ, Table 1 and 2). We additionally added experiments showing that shuffling fixations during GGR annotation decreases both performance and gaze alignment (eYWc, sMmQ), and that both the coordinates and text reasoning are important for GGR performance (eYWc, Appendix Table 10).

  2. Better visual grounding. GGR-RL significantly improves the rate of supported reasoning claims and accuracy of the visual grounding (eYWc, sMmQ, 74.9% vs. 70.5%, p < 0.01, Figure 3).

  3. Robustness of gaze alignment. The improvement in gaze alignment for GGR holds with a shared set of the fitted alignment parameters (sMmQ, Figure 10a) and metrics that take into account fixation ordering (eYWc, sY27, ScanMatch-style similarity, p < 0.001, Appendix A.7).

  4. Stronger baseline comparisons. GGR-RL outperforms RL run on ViGoRL-MCTS (68.8% vs. 63.8%), Vero-Qwen3I-8B (64.2%), and DeepEyes-7B (50.2%) (eYWc, sMmQ, Table 2). Our 8B model also performs within 2.0 percentage points of the 122B teacher model (eYWc, Table 2). We added a comparison to the Gazette supervision construction showing that GGR better preserves gaze coverage and order (sY27, Appendix A.7).

We have additionally added clarifications about the dataset and the training/evaluation setup (eYWc, sMmQ, Section 3.2, Appendix A.1). We thank the reviewers for helping us strengthen the work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.