acceptodds
Under review as a conference paper at ICLR 2027

Learning Where to Look: Reinforcement Learning from Radiologist Gaze for Visually Grounded Chest X-ray VQA

Abstract

Recent medical vision-language models have shown strong capability in chest X-ray (CXR) interpretation. Supervised fine-tuning (SFT), which optimizes the answer tokens, has enabled this capability through instruction tuning, and reinforcement learning (RL) has further improved it by updating the policy with rewards on answer correctness or reasoning quality. However, these training signals supervise only the generated text rather than how the model reaches the answer from the image. Neither objective constrains where the model looks, allowing a model to answer correctly while attending to clinically irrelevant regions. We propose CheXGaze, a CXR-specialized VLM for visual question answering that learns where to look by using radiologists' eye gaze as an RL reward rather than a target to imitate. Following radiologists' global-to-focal reading process, CheXGaze generates an <organ> bounding box, <gaze> points, and an <answer>. Experiments on MIMIC-CXR-VQA and the CXR subsets of SLAKE and VQA-RAD demonstrate its effectiveness, and attention analysis shows closer alignment with radiologists' gaze, indicating that CheXGaze learns where to look.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Vision-Language Model, Chest X-ray, Medical Visual Question Answering, Eye Gaze, Visual Grounding, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_JLjeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CheXGaze, a chest X-ray VQA model that incorporates radiologists’ eye gaze into supervised fine-tuning and reinforcement learning. Starting from Qwen2.5-VL-7B, the model generates anatomical bounding boxes, two gaze coordinates, and an answer. Gaze annotations are obtained by aligning questions with timestamped radiologist dictation and associated fixations. The authors construct duration-weighted heatmaps, subtract a population-level spatial prior, and reward predicted coordinates according to the gaze mass they capture.

Training alternates full-dataset SFT with GRPO on 2,494 gaze-annotated QA pairs, representing 0.86% of the training corpus. On MIMIC-CXR-VQA, CheXGaze improves accuracy from 65.46% to 66.18% and F1 from 64.45% to 65.24% over the fine-tuned backbone. Further adaptation to SLAKE-CXR and VQA-RAD-CXR shows transfer benefits, with the main reported results incorporating predicted organ/gaze traces. Tag and training-schedule ablations, fixation-region occlusion, and attention analysis provide additional evidence about the method’s behavior.

Critical strengths
  1. A thoughtful approach to using sparse, noisy expert gaze. The method combines question–dictation alignment, duration-weighted fixation maps, and a coverage-based reward that allows flexibility in predicted coordinates. The explicit treatment of the population spatial prior is particularly valuable: the reported reward obtainable by a constant policy decreases from 32% to 17% after correction. This addresses a concrete shortcut arising from the consistent anatomy of CXRs and strengthens the methodological contribution.

  2. Multiple complementary forms of empirical evidence. The paper includes comparisons using the same backbone and training data, three-run means and standard deviations, component and training-schedule ablations, and adaptation to two additional datasets. Importantly, the grounding analysis includes an actual image-occlusion intervention rather than relying exclusively on attention visualizations. Together, these provide meaningful initial evidence that gaze supervision changes model behavior and can improve downstream performance.

Critical weaknesses
  1. The evidence does not yet establish that generated gaze coordinates faithfully mediate the answer. The reward supervises coordinate tokens, but the architecture does not require the answer to use image content at those coordinates. Table 6 provides useful evidence that the model is sensitive to radiologist-fixated regions; however, it lacks equal-area random, anatomy-matched, or population-prior occlusion controls. Consequently, greater sensitivity to these regions cannot yet be distinguished from changes in sensitivity to anatomical regions or local image corruption more generally. The attention analysis also omits the layers, heads, and generated tokens used, leaving unclear whether alignment improves specifically during answer generation. Moreover, Table 6 shows a substantial F1 improvement even with a blank image—40.95 versus 30.56—so the answer gains cannot be attributed exclusively to improved visual grounding. These limitations directly affect the central interpretation that the model has learned where to look.

  2. The specific contribution of gaze-based RL is not sufficiently isolated. The approach combines structured gaze supervision during SFT, organ localization, answer and format rewards, gaze rewards, and an alternating training schedule. The Gaze-VLM comparison and Tables 3–4 are useful, but they do not cleanly isolate the gaze reward while preserving the output representation, other rewards, and training budget. This matters because the gains on the main benchmark are modest: in Table 3, the full model improves over the no-tag variant by 0.51 accuracy points and 0.14 F1 points, while gaze-only training does not improve F1. A matched experiment retaining the same output schema and optimization schedule while removing or shuffling only the gaze reward would more directly support the claimed advantage over imitation or auxiliary supervision. The current evidence supports the combined framework more strongly than a general conclusion that gaze-based RL is superior to supervised gaze alignment.

Questions for authors
  1. Under the same output schema, data exposure, and training budget, how does the full method compare with setting the gaze-reward weight to zero, using shuffled or population-prior gaze targets, and continuing structured SFT on the gaze subset? For Table 3, please clarify which training losses, rewards, and schedules change when a tag is removed.

  2. Do the differential performance drops in Table 6 persist relative to equal-area random and anatomy-matched occlusions? Which layers, heads, and output tokens are used for the attention analysis, and do the alignment gains remain when attention is measured only during answer generation?

  3. Has the question–dictation–fixation alignment been manually audited, and how sensitive are the results to gaze–speech delays? Because the aligned gaze supplies both the training reward and the grounding evaluation reference, quantifying its question relevance would materially strengthen the assessment.

5 comments

Dear Reviewer JLje, thank you for your constructive suggestions, which helped us strengthen our work. In the revision, we

  1. clarify the Table 3 ablation setting and isolate the gaze-based RL reward,
  2. test whether the generated gaze points mediate the answer,
  3. add equal-area random, population-prior, and anatomy-matched occlusion controls,
  4. specify the attention extraction protocol, and 5) assess the sensitivity of the reward in the question-dictation-fixation alignment errors.

Response to Q1 & W2 - Isolating the contribution of gaze-based RL

Table 3 clarification. Removing a tag changes both training and inference. The removed tag is excluded from the SFT target, the generated output, and its reward in GRPO. SFT uses token-level cross-entropy on the corresponding target, and all variants use the same two-cycle SFT–GRPO schedule with identical epochs and hyperparameters.

  • Table 3 setting.
OrganGazeSFT outputGRPO rewards
✗✗<answer>answer + format
✓✗<organ><answer>+ organ
✗✓<gaze><answer>+ gaze
✓✓<organ><gaze><answer>+ organ + gaze

Isolating the gaze reward. To isolate the contribution of the gaze reward, we evaluate CheXGaze on MIMIC-CXR-VQA with (1) the gaze reward weight set to zero and (2) each GRPO stage replaced by SFT on the same gaze subset for the same number of epochs.

  • Isolating the gaze reward.
Acc.F1
CheXGaze66.1064.82
Gaze reward = 064.7463.83
SFT instead of GRPO64.7563.03

Both removing the gaze reward and replacing GRPO with SFT reduce accuracy and F1. These results indicate that the gaze reward contributes beyond structured SFT.

  • Gaze target controls
MIMIC-CXR-VQA (n=13,793)SLAKE-CXRVQA-RAD-CXR
Overall ACCOverall F1Verify ACCChoose F1Query F1Closed ACCOpen RecallOverallClosed ACCOpen RecallOverall
Shuffled Targets65.0463.0378.8565.7062.2783.6382.7683.0482.1861.7375.37
Population-prior Targets65.7264.6678.7264.7664.6383.0483.1983.1482.1851.7072.02
CheXGaze66.1064.8278.7566.3664.3884.7985.0384.9585.3465.7378.80

MIMIC-CXR-VQA: single run. SLAKE-CXR and VQA-RAD-CXR: average over three runs.

Furthermore, to assess whether the gaze reward benefits from sample-specific radiologist fixations, we replace each gaze target with either a target from another training pair (shuffled) or the population prior. Both controls reduce overall performance across all three benchmarks. The population-prior target yields similar performance on MIMIC-CXR-VQA but larger drops on SLAKE-CXR and VQA-RAD-CXR. We add the Table 3 clarification and the gaze-reward ablation to Sec. 4.1 and Sec. 4.2 in the revised version.

Response to Q2 & W1 - How gaze supervision affects answer generation and visual grounding.

Gaze intervention. To assess whether the generated gaze coordinates mediate the answer, we replace the predicted gaze coordinates with the radiologist fixations and random points during inference. This experiment is conducted on the MIMIC-CXR-VQA gaze-annotated valid subset.

Acc.ΔAcc. [95% CI]F1ΔF1 [95% CI]
CheXGaze predicted gaze66.67—65.89—
Radiologist fixations66.53−0.14 [−0.55, +0.28]65.59−0.30 [−1.15, +0.43]
Uniform random points66.25−0.41 [−1.11, +0.14]65.71−0.18 [−1.01, +0.53]

Neither replacement substantially changes performance, suggesting limited dependence of the generated gaze coordinates on the answer. CheXGaze generates gaze coordinates as text tokens without using them to re-access image features. In contrast, shuffling the gaze targets during training lowers performance (Q1 & W2), providing evidence that radiologist gaze supervision improves visual grounding.

Controlled occlusion analysis. We add the equal-area random, population-prior, and anatomy-matched occlusion controls. All controls occlude 5% (115 of the 2,304 grid cells) of the image, the same area as the ROI. The random control shifts the ROI mask to a random location, and the population-prior control occludes the 115 highest-prior cells using the same mask across all images. The anatomy-matched control randomly samples 115 cells inside the organ box referenced by the question. This control applies to 527 of 723 questions with a matching organ box containing at least 115 non-ROI cells. To account for differences in overall occlusion sensitivity, we report Δ = control − ROI.

5% occlusionw/o GazeCheXGaze
All questions (n = 723)Acc.F1Acc.F1
Full63.6263.6666.6765.89
ROI59.8958.6858.6455.25
Random61.0059.7262.1059.40
Population prior59.0658.3460.0357.52
Δ (Random − ROI)1.111.043.464.15
Δ (Prior − ROI)-0.83-0.341.392.27
Organ box that holds mask (n = 527)
Full62.4361.6366.2264.58
ROI60.9158.6458.2552.99
Anatomy-matched57.1254.7659.0154.88
Δ (Anatomy − ROI)-3.79-3.880.761.89

Note. The w/o Gaze checkpoint used for Table 6 in the submission (Table 9 in the revised version) was not retained, so all w/o Gaze conditions were re-evaluated on a remaining run from Table A5 (Table A6 in the revised version) to keep them consistent. The CheXGaze results are unchanged, and the w/o Gaze results have been updated accordingly.

CheXGaze exhibits a larger performance drop under ROI occlusion than any control. In contrast, w/o Gaze exhibits smaller Δ values under random occlusion and negative Δ values under the population-prior and anatomy-matched controls. Overall, our occlusion test suggests that CheXGaze is more sensitive to gaze-relevant regions.

Attention analysis. In our attention analysis, we perform a teacher-forced forward pass in greedy-decoded sequences from the <answer> tokens to the image tokens, excluding the preceding <organ> and <gaze> tokens. Attention is renormalized within each layer and averaged across all heads and 28 layers, without head selection. Sec. 4.3 reports the ROI ratio and cosine similarity with the ROI (top 5% of the fixation heatmap). Layer-wise analysis shows positive gains in 26 of 28 layers, including all layers from 7 to 27, while the earliest layers show little or no gain. We add the occlusion controls to Appendix H and the attention protocol to Sec. 4.3 in the revised version.

Response to Q3 - Reliability of the question–dictation–fixation alignment

The aligned gaze spans have not been manually audited. Thus, we assess sensitivity to gaze–speech misalignment by perturbing span boundaries and reconstructing gaze targets using the original pipeline. Random perturbations independently shift each boundary (three seeds), whereas fixed shifts move both boundaries together (negative values indicate earlier times). As a control, we sample an equal number of fixations from random timestamps within the same reading session. For each alignment, we report the median correlation between reconstructed and default gaze targets (Corr.), along with the gaze rewards of CheXGaze and the population-prior baseline on gaze-annotated validation pairs, expressed as percentages of the attainable reward.

AlignmentCorr.CheXGaze (%)Population prior (%)CheXGaze − Prior
Default1.0020.316.73.6
Random ±1 s0.9420.116.73.4
Random ±2 s0.8719.816.73.1
Shift −2 s0.6522.017.74.3
Shift −1 s0.8422.217.64.6
Shift +1 s0.8317.215.81.4
Shift +2 s0.0615.114.90.2
Random time0.3715.416.3−0.9

Corr.: median correlation with the default targets. CheXGaze and Population prior: gaze reward of each placement on held-out QA pairs

Random perturbations up to ±2 s largely preserve target correlations and reward margins. The margin disappears under random-time sampling within the same reading session, suggesting that temporally aligned fixations provide question-specific information beyond general viewing patterns. Later shifts reduce the margin, whereas shifts of 1–2 s earlier increase it, consistent with radiologists fixating regions before describing them. Expert-validated alignment may improve supervision quality and remains a direction for future work. We add this analysis to Appendix C in the revised version.

Reviewer_krBeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CheXGaze, a chest X-ray VQA model trained using radiologist gaze as an RL reward. It sequentially generates organ boxes, gaze points, and an answer, trained via alternating SFT on the full dataset with GRPO on a small gaze-annotated subset. The authors report improvements on three VQA benchmarks, report various ablations, and analyze alignment with radiologist fixations.

Critical strengths
  1. I found the paper easy to read and the motivation interesting. Using radiologist gaze as a reward is a promising direction, exploiting new supervisory signals beyond radiologist report / text signals.

  2. I find the results somewhat encouraging, particularly the improvements on open-ended questions in the transfer datasets, given that gaze annotations cover only 0.86% of the main training set. While the improvements are moderate, it's suprising that this small amount of gaze data already helps performance a bit.

  3. I appreciate the comparison with the same fine-tuned backbone and the additional occlusion, attention, and corrupted-trace analyses.

Critical weaknesses
  1. My main uncertainty concerns what it means for the model to “look” at its predicted locations. From Sec. 3.1 and Appendix B, this appears to be sequential generation of coordinate tokens while retaining access to the full image. The additional analyses are useful, but I am still unsure how faithfully the generated organ/gaze traces reflect the evidence used for the answer. This feels a bit similarly limited to CoT not really representing the true "reasoning" but just some verbalisation that may not causally determine the final answer.

  2. It's a bit concerning to reward question-irrelevant fixations. The paper acknowledges gaze–speech asynchrony, and some illustrated fixations appear unrelated to the queried region. Population-prior subtraction addresses a particular shortcut, but I do not see evidence that the remaining reward consistently favors clinically relevant evidence. E.g., how do you prevent that noise in gaze (e.g. signals in lower left in Fig1 instead of lower right) do not impair the method? (e.g. how to ensure that you are not learning to "focus" on regions that sometimes are irrelevant?

  3. I find the ablation description insufficiently precise. Table 3 removes organ/gaze tags, but it is unclear whether this changes training, inference, or both. I would like to understand which reward components contribute the gains and whether the training schedule and tuning are comparable across variants. (e.g. w/o gaze, different hyperparameters may be optimal).

  4. The discussion of prior RL approaches seems overly text-based and misses previous works on radiology VLMs / Agents trained with RL. E.g., RadVLM-GRPO optimizes visual grounding, while RadAgent rewards aspects of the tool-use process for CT interpretation. These seem relevant to positioning the contribution, although neither establishes the same use of radiologist gaze.

Questions for authors
  1. Does answer generation retain unrestricted access to the image, or do the predicted regions constrain subsequent visual processing? How exactly are the traces corrupted in Table 5?

  2. For Table 3, what changes in the targets, prompts, and rewards during training and inference? Could the authors clarify how the comparisons isolate the organ and gaze rewards?

  3. How was potential train/test image overlap handled for VQA-RAD-CXR? (see e.g. problems with VQA-Rad dataset as reported in Med-Flamingo paper)

  4. What explains the very low MIMIC-CXR-VQA F1 scores for LLaVA-Med and LLaVA-Rad (1.06 and 3.80)? I would appreciate clarification on whether output formatting or answer extraction contributes to these results.

Small details:

  1. Why exactly 2 gaze points? What about 1,3 or more?
  2. I would cite some more medical VLM lit: e.g. Med-Gemma 1.5, Med-Flamingo, Med-Palm M..
3 comments

Dear Reviewer krBe, thank you for your detailed and constructive feedback. In response, we

  1. clarify the role of the generated traces with gaze interventions,
  2. address noise in the gaze data,
  3. clarify Table 3 and isolate the organ and gaze rewards,
  4. discuss related RL work, and 5) clarify the VQA-RAD overlap and the low F1 of LLaVA-based models.

Response to W1 & Q1 - Clarification of our work

CheXGaze generates <organ>, <gaze>, and <answer> sequentially, while retaining unrestricted access to the original image. Predicted coordinates do not constrain subsequent visual processing through cropping or re-encoding. As the reviewer points out, generated traces may share the limitations of CoT, as they do not necessarily reflect the visual evidence causally used for the final answer generation. CheXGaze instead uses radiologists' fixations as RL rewards to encourage the model to identify task-relevant local regions. Thus, our main claim is about learning where to look through gaze-based rewards, rather than the causal role of generated coordinates in answer generation.

Table 5 (Table 6 in the revised version) provides supporting evidence that correctly matched gaze traces carry useful information when transferred to datasets without gaze annotations (e.g., VQA-RAD-CXR open: 65.73 vs. 57.95). To clarify, the corrupted trace preserves the organ names and tag structure of each predicted trace but replaces its coordinates with those from another randomly selected training sample’s predicted trace.

To further examine the role of gaze rewards in learning and the dependence on generated coordinates, we conduct two additional interventions:

  • Training-time (MIMIC-CXR-VQA test): Shuffling gaze targets during GRPO, while keeping SFT and the organ reward unchanged, reduces accuracy (66.10 → 65.04) and F1 (64.82 → 63.03).

  • Inference-time (723 gaze-annotated validation pairs): Replacing generated gaze points with radiologist fixations or random points produces no significant change in accuracy (66.67 → 66.53 / 66.25) or F1 (65.89 → 65.59 / 65.71).

Together with ROI occlusion (Table 6; Table 9 in the revised version) and attention map analysis (Sec. 4.3), these findings support our claim that CheXGaze learns where to look through gaze-based rewards and relies more heavily on visual evidence from radiologist-fixated regions.

Response to W2 - Noise in human gaze data. Rewarding question-irrelevant fixations.

We acknowledge that gaze–speech asynchrony can introduce question-irrelevant fixations. Our pipeline mitigates this through (1) question-aligned dictation spans, (2) dwell-time weighting, and (3) population-prior subtraction.

The shuffled-target control (W1) shows that the original gaze–question correspondence provides a useful training signal despite potential noise. However, it does not establish the relevance of individual fixations. Expert validation could further improve target quality, but our approach is designed to leverage existing eye-tracking data without per-sample expert annotation. We add this limitation in the revised version.

Response to W3 & Q2 - Clarification of Table 3 & Ablation to isolate the organ and gaze reward

In Table 3, removing a component excludes its tag from the prompt, SFT target, and inference output, and removes its corresponding GRPO reward.

OrganGazeSFT outputGRPO rewards
✗✗<answer>answer + format
✓✗<organ><answer>+ organ
✗✓<gaze><answer>+ gaze
✓✓<organ><gaze><answer>+ organ + gaze

Thus, Table 3 evaluates each component jointly with its supervision. To isolate the GRPO rewards, we additionally retain identical SFT targets and the full output schema while disabling the organ or gaze reward.

Organ RewardGaze RewardOverall Acc.Overall F1Verify Acc.Choose F1Query F1
0.00.065.4262.8479.6463.5862.63
1.00.064.7463.8377.9866.2563.14
0.01.065.4963.3079.2963.4763.25
1.01.066.1064.8278.7566.3664.38

Removing either reward lowers overall accuracy and F1, indicating that both rewards are necessary for the full model's performance. The gains are concentrated in choose and query F1. Since all variants share the same SFT, these differences isolate the effects of GRPO reward components.

All variants use the same two-cycle SFT–GRPO schedule, epochs, and hyperparameters. The learning rate was selected for the answer-only SFT baseline and reused without variant-specific tuning. We add these clarifications and the reward ablation to Sec. 4.1 and Sec. 4.2 in the revised version.

Response to W4 - Related RL work on radiology VLMs and agents.

Thank you for pointing out these works. RadVLM-GRPO applies GRPO to CXR report generation and visual grounding using an IoU-based grounding reward. RadAgent trains a tool-using agent for chest CT interpretation with rewards for report quality and tool use. While both incorporate rewards beyond answer correctness, CheXGaze differs by using radiologists' eye gaze as an RL reward to learn where to look, rather than relying solely on anatomical annotations or tool-use feedback. We will add both works in the Related Works section (Sec 2.2).

Response to Q3 - Image overlap in VQA-RAD.

As the reviewer notes, the official VQA-RAD split shares images between train and test. We kept this split so that our results are directly comparable to prior models evaluated on it, such as LLaVA-Med. The overlap applies equally to all fine-tuned models in Table 2 and to all variants in Table 5 (Table 6 in the revised version), since they are trained and evaluated on the same split. The zero-shot comparison requires more care, as models trained on the VQA-RAD training split (Table A4; Table A5 in the revised version) may benefit from the overlap. We will state this in the Experimental Setup section (Sec. 4.1).

Response to Q4 - Low F1 of LLaVA-Med and LLaVA-Rad.

The low F1 in LLaVA-based models is largely due to output-format mismatch. F1 is computed by comparing predicted and ground-truth label sets drawn from 110 EHRXQA labels. However, LLaVA-Med and LLaVA-Rad frequently generate free-form sentences or reports rather than the expected labels. Their outputs are generally well-formed (0% empty, less than 1% truncated), but often fail label matching.

As a diagnostic, we additionally apply label extraction to their outputs. F1 increases from 1.06 to 14.85 for LLaVA-Med and from 3.80 to 19.74 for LLaVA-Rad, still substantially below CheXGaze (65.24). While this confirms that output formatting substantially affects their low F1, the remaining gap cannot be attributed to formatting alone. We add this analysis to Sec. 4.2 in the revised version.

Thanks for the responses.

I feel the W1, W2 points remain problematic:

Replacing generated gaze points with radiologist fixations or random points produces no significant change in accuracy (66.67 → 66.53 / 66.25) or F1 (65.89 → 65.59 / 65.71).

This suggests to me that the generated gaze is not really used / relevant (as I would suspect given that the gaze is just an add-on in the context - not a bottleneck through which the reasoning has to go through. So any type of inspectable / interpretable model output that is not directly useful / steering the down-stream prediction may be potentially misleading (like a bounding box that looks at wrong organ) for the practitioner.

Reviewer_fhhLOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CheXGaze, a chest X-ray (CXR)-specialized vision-language model (VLM) for visual question answering (VQA) that learns where to look by using radiologists' eye gaze as a reinforcement learning (RL) reward rather than a supervised target to imitate. The authors argue that both supervised fine-tuning (SFT) and existing RL reward designs (MedVLM-R1, Med-R1, CheXOne, CX-Mind) supervise only the generated text and never constrain where the model attends, so a model can answer correctly while attending to clinically irrelevant regions. The method has three components. (1) A structured output format where the model generates an <organ> bounding box, two <gaze> points, then an <answer>, motivated by the radiologist's global-to-focal reading process. (2) Gaze target construction from the REFLACX and EGD-CXR eye-tracking datasets, temporally aligned to MIMIC-CXR-VQA questions via keyword matching plus an LLM-based transcript-span matcher (Qwen3.6-27B), yielding only 2,494 gaze-annotated QA pairs (0.86% of the training set). A key technical detail is subtracting the population-prior projection of the fixation heatmap to prevent the model from exploiting the strong center/anatomy bias in CXRs (a constant policy that ignores the image would otherwise capture 32% of the attainable gaze reward; this correction drops it to 17%). (3) An iterative SFT–GRPO pipeline that alternates full-corpus SFT with gaze-guided GRPO to keep the policy from drifting toward the tiny gaze subset. Validation is on MIMIC-CXR-VQA plus the CXR subsets of SLAKE and VQA-RAD, using Qwen2.5-VL-7B as the backbone. CheXGaze outperforms general, medical, and CXR-specialized VLMs and, more importantly, outperforms the same backbone fine-tuned on the same data without gaze — while Gaze-VLM (attention KL-regularization baseline) yields no improvement. Ablations isolate the contribution of the organ and gaze tags, a step-wise study validates the alternating schedule, a transfer study on non-gaze datasets (direct vs. predicted vs. corrupted traces) shows the grounding generalizes, and occlusion plus attention-map analysis indicate the model genuinely relies on radiologist-fixated regions.

Critical strengths

Gaze as an RL reward, not a supervised target, with an explicit anti-shortcut mechanism. This is the most consequential strength. Treating gaze as an imitation target reduces expert guidance to copying one reader's variable viewing pattern (well-documented in the literature and illustrated in Appendix A). By instead rewarding the fraction of gaze mass captured and subtracting the population-prior projection, the authors address the central obstacle to using CXR gaze as a signal — the strong anatomy/center bias that makes a trivial prior highly predictive. The constant-policy analysis (32% → 17%) is a clean, convincing demonstration that the shortcut is closed. This is a genuine methodological contribution, not just another fine-tuning recipe. Controlled experimental design that isolates the causal claim. Rather than only beating large pre-trained models, the paper fine-tunes the same Qwen2.5-VL-7B backbone with and without gaze (and reproduces Gaze-VLM with the official implementation), so the improvement can be attributed to gaze-as-reward. The transfer study on gaze-free datasets with predicted-vs-corrupted traces is a particularly good design: corrupted traces degrade performance while predicted traces improve it, which supports the claim that the gain comes from learned grounding rather than the reasoning format itself.

Critical weaknesses

Modest, statistically unvalidated gains. The headline comparison (CheXGaze vs. the same backbone fine-tuned without gaze) shows improvements on the order of 0.5–1 points on MIMIC-CXR-VQA and SLAKE, and the largest single gain (VQA-RAD open recall 65.73 vs 57.03) sits on a very small test split (174 pairs) with a reported std of ±2.47 — i.e., within noise. No significance testing or confidence intervals are given on the main results, so it is not established that gaze-as-reward reliably beats gaze-free SFT+GRPO. Given that the entire contribution rests on this comparison, this is the most consequential weakness for my assessment. The "the model looks where radiologists look" evidence is indirect and the gaze supervision is noisy/scarce. The occlusion and attention-map analyses show correlation (higher attention ROI ratio, larger occlusion drop) but do not prove the model's improved accuracy is caused by gazing at the right region — the two could share a confound (e.g., the model simply learned better localization, which both raises accuracy and correlates with fixations). Compounding this, gaze covers only 0.86% of training data, is obtained through an LLM-assisted transcript matcher with acknowledged gaze–speech asynchrony, and the mechanistic analysis is run on the validation set (n=723) because the test split has essentially no gaze coverage.

Questions for authors

(Clarification / significance.) For the core comparison — CheXGaze vs. the same backbone fine-tuned without gaze (and vs. Gaze-VLM) — what is the statistical significance of the differences? With the reported per-run stds (Appendix A5/A6), are the MIMIC-CXR-VQA and SLAKE gains stable across seeds, and would you be willing to report a paired test over the held-out questions? (Clarification / causality.) The occlusion and attention analyses show CheXGaze attends more to fixated regions and drops more when they are occluded. Is there a direct causal test — e.g., does forcing the model to emit gaze points from a wrong region (while keeping the answer format identical) hurt accuracy more than forcing correct regions? This would more cleanly separate "the model learned to look correctly" from "the model simply became a better localizer." (Clarification.) How sensitive are results to the transcript-span matcher and the population-prior subtraction coefficient? An ablation reporting how the gaze-target quality (and thus the reward) degrades when the LLM matcher or the prior projection is removed would help gauge how much of the gain is robust to annotation noise.

2 comments

Dear Reviewer fhhL, thank you for your thoughtful and detailed review. In response, we added (1) paired-bootstrap tests for the main comparisons, (2) causal tests of gaze supervision, and (3) a sensitivity analysis of the gaze-target construction.

Response to W1 & Q1 - Statistical significance and stability of the gains.

To test the statistical significance of the gains of CheXGaze over the baselines, we run a paired bootstrap over test questions (B = 10,000, 95% percentile CI) averaged over the three runs of each model (Tables A5–A6; Tables A6–A7 in the revised version). Bold marks CIs that exclude zero. For stability across seeds, we count the run pairs (out of 9) in which CheXGaze leads.

  • Statistical significance on MIMIC-CXR-VQA.
Acc.F1VerifyChooseQuery
CheXGaze66.18 ± 0.3465.24 ± 0.5378.87 ± 0.1265.92 ± 0.9265.05 ± 0.64
vs Qwen2.5-VL-7B FT: Δ [95% CI]+0.72 [+0.33, +1.12]+0.79 [+0.27, +1.31]+1.40 [+0.86, +1.96]+0.07 [−1.21, +1.32]+1.00 [+0.44, +1.58]
vs Gaze-VLM: Δ [95% CI]+0.75 [+0.35, +1.13]+1.05 [+0.53, +1.59]+1.40 [+0.85, +1.96]+0.89 [−0.38, +2.17]+1.10 [+0.52, +1.65]
  • Statistical significance on SLAKE and VQA-RAD.
SLAKEVQA-RAD
ClosedOpenClosedOpen
CheXGaze84.79 ± 1.0185.03 ± 0.2985.34 ± 1.5065.73 ± 2.47
vs Qwen2.5-VL-7B FT: Δ [95% CI]+0.58 [−5.32, +6.45]+2.76 [−0.18, +5.85]+0.00 [−6.67, +6.55]+8.70 [+0.83, +16.80]
vs Gaze-VLM: Δ [95% CI]+7.59 [+0.00, +15.27]+11.15 [+7.42, +15.04]+4.02 [−0.62, +8.95]+10.27 [+4.91, +16.26]

[Statistical Significance] On MIMIC-CXR-VQA, the gains over both baselines are significant on overall accuracy and F1, as well as on verify and query questions, but not on choose questions. On the CXR subsets, CheXGaze is on par with the fine-tuned backbone on closed-ended questions. Its open-ended gain is significant on VQA-RAD but not on SLAKE, where the CI narrowly includes zero. The VQA-RAD open split contains only 58 questions, so its CI is wide. Against Gaze-VLM, the open-ended gains are significant on both benchmarks.

[Stability across seeds] On MIMIC-CXR-VQA, CheXGaze outperforms the fine-tuned backbone in 9 out of 9 run pairs on accuracy and 8/9 on F1, and Gaze-VLM in all runs on both metrics. On SLAKE, CheXGaze outperform the baseline in all run pairs on open questions, whereas the closed-question difference lies within run-to-run variation (6/9). It outperforms Gaze-VLM in 9/9 run pairs on both question types. We add these results to Appendix G and refer to them in Sec. 4.2 in the revised version.

Response to W2&Q2 - Causal effect of gaze supervision.

We thank the reviewer for raising this important distinction. We claim that gaze-guided training helps CheXGaze learn where to look, rather than that the generated gaze coordinates explicitly control answer generation at inference. We investigate these effects with training- and inference-time interventions.

[Training-time intervention] We keep SFT, the organ reward, and all other settings unchanged, and replace each GRPO gaze target with the fixations of another training pair. This preserves the distribution of radiologist gaze and the organ-level signal, and breaks only the gaze–case correspondence. The result shows that shuffling the gaze target significantly reduces accuracy and F1.

  • Radiologists vs. shuffled gaze targets during training (MIMIC-CXR-VQA test)
Acc.F1
Matched (own pair's gaze)66.1064.82
Shuffled (another pair's gaze)65.0463.03
Δ (matched − shuffled)+1.06 [+0.44, +1.68]+1.79 [+0.96, +2.62]

[Inference-time intervention] We then replace the predicted gaze points with radiologist fixations or uniform random points on the 723 gaze-annotated validation pairs, keeping the output format identical.

  • Inference-time replacement of gaze points (MIMIC-CXR-VQA validation gaze subset).
Acc.ΔAcc. [95% CI]F1ΔF1 [95% CI]
CheXGaze predicted gaze66.67—65.89—
Radiologist fixations66.53−0.14 [−0.55, +0.28]65.59−0.30 [−1.15, +0.43]
Uniform random points66.25−0.41 [−1.11, +0.14]65.71−0.18 [−1.01, +0.53]

Neither replacement significantly changes accuracy or F1, suggesting limited dependence on the generated gaze coordinates. This is consistent with CheXGaze's architecture, which generates gaze points as text tokens without modifying the visual input (e.g., cropping or re-encoding). However, limited dependence on gaze coordinates does not necessarily imply limited reliance on task-relevant visual evidence. Consistent with this distinction, the ROI occlusion results (Table 6; Table 9 in the revised version) show greater sensitivity to gaze-relevant regions.

These findings suggest that gaze supervision improves VQA performance and increases sensitivity to gaze-relevant visual evidence, despite limited observed dependence on generated gaze coordinates at inference.

Response to W2 & Q3 - Sensitivity to the span matcher and the prior projection

The subtraction coefficient βq in Eq. (2) is not a tuned hyperparameter, and it is determined in closed form for each question's gaze map by least-squares projection. We therefore assess sensitivity by removing each step, where removing the projection corresponds to βq = 0. We evaluate the target quality by whether the resulting gaze reward can be exploited by shortcuts. The span matcher prevents question-independent policies from obtaining a high reward, and the prior projection does the same for image-independent policies. For targets built with and without each step, we compare the reward of CheXGaze with that of two shortcut policies. Q-blind oracle places points on the image's whole-session gaze map regardless of the question. Population prior places points at the population fixation peaks regardless of the image and question. Rewards are normalized so that the optimal placement for each target scores 1. Margin = CheXGaze − Population prior on held-out targets, where a positive margin indicates that the reward favors image-specific gaze over the shortcut. CheXGaze is the same trained model in all rows.

  • Ablation of gaze-target construction.
Span MatcherPrior ProjectionQ-blind oracleCheXGazePopulation PriorMargin
--1.000.3670.468−0.101
✔-0.630.2930.323−0.030
-✔1.000.1940.182+0.012
✔✔0.560.2030.167+0.036

Each step prevents different shortcuts. Without the span matcher, all questions on an image share one target, and a question-blind placement attains the maximum reward. The matcher reduces this to 0.56, and CheXGaze’s points score higher on their own question’s target than on the targets of other questions on the same image (+0.034, 95% CI [+0.023, +0.044]). The image- and question-blind population prior receives a higher reward than CheXGaze without the prior projection, but a lower reward with it (0.167 vs. 0.203 with both steps). This indicates that both steps are necessary for the reward to enable question- and image-specific gaze.

Reviewer_T8qBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CheXGaze, a chest X-ray VQA framework that incorporates radiologists' eye gaze into reinforcement learning to improve visual grounding. The method constructs question-aligned gaze targets, corrects population-level spatial bias, and alternates SFT with gaze-guided GRPO to jointly optimize anatomical localization, gaze prediction, and answer correctness. Experiments across three CXR-VQA benchmarks demonstrate improved answer performance, while ablation studies provide supporting evidence for the proposed approach.

Critical strengths
  1. Meaningful motivation and practical method design. Incorporating radiologists' gaze into RL is an interesting direction for improving visual grounding in medical VLMs. The population-prior correction and alternating SFT–GRPO strategy are well motivated, particularly given the limited availability of gaze annotations.

  2. Solid experimental coverage and encouraging transfer results. The paper includes comparisons against multiple VLMs, component ablations, training-stage evaluations, and visual grounding analyses. The improvements on datasets without gaze annotations, particularly open-ended VQA-RAD-CXR, are promising and suggest potential transfer benefits.

Critical weaknesses
  1. The specific contribution of the gaze reward remains insufficiently isolated. Although Tables 3 and 4 provide meaningful ablations of output components and training strategies, they do not directly isolate the gaze reward within GRPO. Since the model already receives gaze-token supervision during SFT, and GRPO jointly optimizes answer, organ, format, and gaze rewards, it remains difficult to determine how much of the improvement is specifically attributable to gaze-guided RL. A controlled experiment that removes only the gaze reward while preserving the training schedule and output structure would clarify this contribution.

  2. The evidence for clinically meaningful visual grounding remains incomplete. The model explicitly predicts organ bounding boxes and gaze points, but their localization quality is not directly evaluated against held-out anatomical or clinical evidence annotations. The attention and occlusion analyses provide useful supporting evidence, but do not fully establish the correctness of the predicted spatial evidence. Furthermore, the substantially higher F1 obtained by CheXGaze on fully blank images in Table 6 suggests that improvements unrelated to visual evidence may also contribute to the results.

Questions for authors

As stated in "Critical Weaknesses"

2 comments

Dear Reviewer T8qB, thank you for your thoughtful feedback. We address both concerns through (1) additional ablation to isolate the gaze reward contribution and (2) direct assessment of visual grounding.

Response to W1 - Isolation of the gaze reward contribution

To isolate the contribution of gaze-guided RL, we conduct an additional ablation study by setting the gaze reward weight to zero during GRPO. The training schedule, output structure, and all other settings remain identical.

  • Experimental results on gaze reward w_gaze=0
MIMIC-CXR-VQASLAKE-CXRVQA-RAD-CXR
Overall ACCOverall F1Verify ACCChoose F1Query F1Closed ACCOpen RecallOverallClosed ACCOpen RecallOverall
w_gaze=064.7463.8377.9866.2563.1484.5083.6583.9281.9056.5173.43
CheXGaze66.1064.8278.7566.3664.3884.7985.0384.9585.3465.7378.80

MIMIC-CXR-VQA: single run. SLAKE-CXR and VQA-RAD-CXR: average over three runs.

Removing the gaze reward leads to performance degradation across all datasets. The larger performance drop on VQA-RAD-CXR further suggests the generalization ability of our gaze-guided to datasets without gaze annotations. Along with the structured-SFT ablations in Table 3, these results indicate that both structured supervision during SFT and gaze-guided RL contribute to performance improvements. We add these results to Sec. 4.2 in the revised version.

Response to W2 - Direct Assessment on Visual Grounding

  • Predicted <organ> boxes vs. Chest ImaGenome (test split).
MethodMean IoUIoU ≥ 0.5 (%)
Image-blind baseline0.4751.0
CheXGaze0.5664.7
  • Predicted <gaze> points vs. radiologist fixations (gaze-annotated valid split, 723 QAs).
Hit @ top-5% (%) ↑Hit @ top-10% (%) ↑≤ 200 px (%) ↑≤ 300 px (%) ↑> 500 px (%) ↓Median dist. (px) ↓Mean dist. (px) ↓
Uniform random9.719.36.315.660.3583632
Population prior placement31.150.936.460.710.7268295
CheXGaze36.154.144.466.714.4218296

We agree that our attention and occlusion analysis alone do not fully establish the spatial accuracy. We therefore directly evaluate the predicted <organ> boxes and <gaze> coordinates against anatomical annotations and radiologist fixations, respectively.

  • [Organ Localization] We compare predicted organ boxes with Chest ImaGenome annotations, using the median bounding box for each organ as an image-blind baseline. CheXGaze achieves a higher mean IoU (0.56 vs. 0.47) and IoU ≥ 0.5 rate (64.7% vs. 51.0%), indicating that CheXGaze predictions capture image-specific anatomical localization beyond canonical organ locations.

  • [Gaze Localization] Due to limited overlap between gaze annotations and the test split, we use the gaze-annotated validation split. We compare against random, which samples points uniformly over the entire image, and population prior, both using the same number of points as CheXGaze (K = 2). Hit@k% is the fraction of samples with at least one predicted point inside the radiologist’s top-k% dwell region, and distance is the pixel distance from each predicted point to the nearest radiologist fixation.

CheXGaze achieves higher hit rates than both baselines (Hit@5%: 36.1% vs. 31.1% for the prior) and a lower median distance (218 vs. 268 px). However, its mean distance is comparable to the prior (296 vs. 295 px), reflecting the population prior placed near the center of the chest, keeping distances moderate even when missing the target regions. As a result, the prior produces fewer large errors (>500 px: 10.7% vs. 14.4%) but achieves lower target-region hit rates. Some of CheXGaze's large errors involve predictions in the contralateral lung, suggesting laterality as an area for further improvement. Nevertheless, CheXGaze's lower median distance and higher target-region hit rates provide evidence of its ability to predict question- and image-specific gaze locations. We add these localization results to Sec. 4.3 in the revised version.

On the higher F1 with blank images (Table 6; Table 9 in the revised version). We thank the reviewer for this observation. CheXGaze's higher blank-image F1 may appear to indicate weaker reliance on visual input. However, this gap is due to the difference in answer distributions in blank-image. In query questions, the gaze-free model frequently generates negative answers ("none" for 49.6% of questions vs. 15.3% in the ground truth), whereas CheXGaze maintains an answer distribution closer to the ground truth (21.8% of “none”). Since F1 measures the overlap between predicted and ground-truth answer sets, this difference results in a much larger gap in F1 (40.95 vs. 32.55) than in accuracy (38.17 vs. 37.07). Nevertheless, this does not preclude our main claim that our model relies more on relevant regions. Blank-image evaluation measures performance without any visual input, whereas ROI occlusion in Table 6 (Table 9 in the revised version) assesses sensitivity to specific image regions. CheXGaze shows a larger performance drop under ROI occlusion than the gaze-free model, indicating greater sensitivity to gaze-relevant regions. Together with the direct localization results above, these findings support CheXGaze's ability to localize radiologist gaze and utilize gaze-relevant image regions for prediction.

Comments on the paper

1 comment

Dear Reviewers

We sincerely thank you for your valuable feedback and thoughtful comments on our work. We have carefully addressed your concerns and provided detailed responses in our rebuttal.

We would greatly appreciate it if you could take a moment to review our responses and let us know whether they adequately address your concerns. We would be happy to provide further clarification if needed.

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.