- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CheXGaze, a chest X-ray VQA model that incorporates radiologists’ eye gaze into supervised fine-tuning and reinforcement learning. Starting from Qwen2.5-VL-7B, the model generates anatomical bounding boxes, two gaze coordinates, and an answer. Gaze annotations are obtained by aligning questions with timestamped radiologist dictation and associated fixations. The authors construct duration-weighted heatmaps, subtract a population-level spatial prior, and reward predicted coordinates according to the gaze mass they capture.
Training alternates full-dataset SFT with GRPO on 2,494 gaze-annotated QA pairs, representing 0.86% of the training corpus. On MIMIC-CXR-VQA, CheXGaze improves accuracy from 65.46% to 66.18% and F1 from 64.45% to 65.24% over the fine-tuned backbone. Further adaptation to SLAKE-CXR and VQA-RAD-CXR shows transfer benefits, with the main reported results incorporating predicted organ/gaze traces. Tag and training-schedule ablations, fixation-region occlusion, and attention analysis provide additional evidence about the method’s behavior.
Critical strengths
-
A thoughtful approach to using sparse, noisy expert gaze. The method combines question–dictation alignment, duration-weighted fixation maps, and a coverage-based reward that allows flexibility in predicted coordinates. The explicit treatment of the population spatial prior is particularly valuable: the reported reward obtainable by a constant policy decreases from 32% to 17% after correction. This addresses a concrete shortcut arising from the consistent anatomy of CXRs and strengthens the methodological contribution.
-
Multiple complementary forms of empirical evidence. The paper includes comparisons using the same backbone and training data, three-run means and standard deviations, component and training-schedule ablations, and adaptation to two additional datasets. Importantly, the grounding analysis includes an actual image-occlusion intervention rather than relying exclusively on attention visualizations. Together, these provide meaningful initial evidence that gaze supervision changes model behavior and can improve downstream performance.
Critical weaknesses
-
The evidence does not yet establish that generated gaze coordinates faithfully mediate the answer. The reward supervises coordinate tokens, but the architecture does not require the answer to use image content at those coordinates. Table 6 provides useful evidence that the model is sensitive to radiologist-fixated regions; however, it lacks equal-area random, anatomy-matched, or population-prior occlusion controls. Consequently, greater sensitivity to these regions cannot yet be distinguished from changes in sensitivity to anatomical regions or local image corruption more generally. The attention analysis also omits the layers, heads, and generated tokens used, leaving unclear whether alignment improves specifically during answer generation. Moreover, Table 6 shows a substantial F1 improvement even with a blank image—40.95 versus 30.56—so the answer gains cannot be attributed exclusively to improved visual grounding. These limitations directly affect the central interpretation that the model has learned where to look.
-
The specific contribution of gaze-based RL is not sufficiently isolated. The approach combines structured gaze supervision during SFT, organ localization, answer and format rewards, gaze rewards, and an alternating training schedule. The Gaze-VLM comparison and Tables 3–4 are useful, but they do not cleanly isolate the gaze reward while preserving the output representation, other rewards, and training budget. This matters because the gains on the main benchmark are modest: in Table 3, the full model improves over the no-tag variant by 0.51 accuracy points and 0.14 F1 points, while gaze-only training does not improve F1. A matched experiment retaining the same output schema and optimization schedule while removing or shuffling only the gaze reward would more directly support the claimed advantage over imitation or auxiliary supervision. The current evidence supports the combined framework more strongly than a general conclusion that gaze-based RL is superior to supervised gaze alignment.
Questions for authors
-
Under the same output schema, data exposure, and training budget, how does the full method compare with setting the gaze-reward weight to zero, using shuffled or population-prior gaze targets, and continuing structured SFT on the gaze subset? For Table 3, please clarify which training losses, rewards, and schedules change when a tag is removed.
-
Do the differential performance drops in Table 6 persist relative to equal-area random and anatomy-matched occlusions? Which layers, heads, and output tokens are used for the attention analysis, and do the alignment gains remain when attention is measured only during answer generation?
-
Has the question–dictation–fixation alignment been manually audited, and how sensitive are the results to gaze–speech delays? Because the aligned gaze supplies both the training reward and the grounding evaluation reference, quantifying its question relevance would materially strengthen the assessment.
5 comments
Response to Q2 & W1 - How gaze supervision affects answer generation and visual grounding.
Gaze intervention. To assess whether the generated gaze coordinates mediate the answer, we replace the predicted gaze coordinates with the radiologist fixations and random points during inference. This experiment is conducted on the MIMIC-CXR-VQA gaze-annotated valid subset.
| Acc. | ΔAcc. [95% CI] | F1 | ΔF1 [95% CI] | |
|---|---|---|---|---|
| CheXGaze predicted gaze | 66.67 | — | 65.89 | — |
| Radiologist fixations | 66.53 | −0.14 [−0.55, +0.28] | 65.59 | −0.30 [−1.15, +0.43] |
| Uniform random points | 66.25 | −0.41 [−1.11, +0.14] | 65.71 | −0.18 [−1.01, +0.53] |
Neither replacement substantially changes performance, suggesting limited dependence of the generated gaze coordinates on the answer. CheXGaze generates gaze coordinates as text tokens without using them to re-access image features. In contrast, shuffling the gaze targets during training lowers performance (Q1 & W2), providing evidence that radiologist gaze supervision improves visual grounding.
Controlled occlusion analysis. We add the equal-area random, population-prior, and anatomy-matched occlusion controls. All controls occlude 5% (115 of the 2,304 grid cells) of the image, the same area as the ROI. The random control shifts the ROI mask to a random location, and the population-prior control occludes the 115 highest-prior cells using the same mask across all images. The anatomy-matched control randomly samples 115 cells inside the organ box referenced by the question. This control applies to 527 of 723 questions with a matching organ box containing at least 115 non-ROI cells. To account for differences in overall occlusion sensitivity, we report Δ = control − ROI.
| 5% occlusion | w/o Gaze | CheXGaze | ||
|---|---|---|---|---|
| All questions (n = 723) | Acc. | F1 | Acc. | F1 |
| Full | 63.62 | 63.66 | 66.67 | 65.89 |
| ROI | 59.89 | 58.68 | 58.64 | 55.25 |
| Random | 61.00 | 59.72 | 62.10 | 59.40 |
| Population prior | 59.06 | 58.34 | 60.03 | 57.52 |
| Δ (Random − ROI) | 1.11 | 1.04 | 3.46 | 4.15 |
| Δ (Prior − ROI) | -0.83 | -0.34 | 1.39 | 2.27 |
| Organ box that holds mask (n = 527) | ||||
| Full | 62.43 | 61.63 | 66.22 | 64.58 |
| ROI | 60.91 | 58.64 | 58.25 | 52.99 |
| Anatomy-matched | 57.12 | 54.76 | 59.01 | 54.88 |
| Δ (Anatomy − ROI) | -3.79 | -3.88 | 0.76 | 1.89 |
Note. The w/o Gaze checkpoint used for Table 6 in the submission (Table 9 in the revised version) was not retained, so all w/o Gaze conditions were re-evaluated on a remaining run from Table A5 (Table A6 in the revised version) to keep them consistent. The CheXGaze results are unchanged, and the w/o Gaze results have been updated accordingly.
CheXGaze exhibits a larger performance drop under ROI occlusion than any control. In contrast, w/o Gaze exhibits smaller Δ values under random occlusion and negative Δ values under the population-prior and anatomy-matched controls. Overall, our occlusion test suggests that CheXGaze is more sensitive to gaze-relevant regions.
Attention analysis. In our attention analysis, we perform a teacher-forced forward pass in greedy-decoded sequences from the <answer> tokens to the image tokens, excluding the preceding <organ> and <gaze> tokens. Attention is renormalized within each layer and averaged across all heads and 28 layers, without head selection. Sec. 4.3 reports the ROI ratio and cosine similarity with the ROI (top 5% of the fixation heatmap). Layer-wise analysis shows positive gains in 26 of 28 layers, including all layers from 7 to 27, while the earliest layers show little or no gain. We add the occlusion controls to Appendix H and the attention protocol to Sec. 4.3 in the revised version.
Response to Q3 - Reliability of the question–dictation–fixation alignment
The aligned gaze spans have not been manually audited. Thus, we assess sensitivity to gaze–speech misalignment by perturbing span boundaries and reconstructing gaze targets using the original pipeline. Random perturbations independently shift each boundary (three seeds), whereas fixed shifts move both boundaries together (negative values indicate earlier times). As a control, we sample an equal number of fixations from random timestamps within the same reading session. For each alignment, we report the median correlation between reconstructed and default gaze targets (Corr.), along with the gaze rewards of CheXGaze and the population-prior baseline on gaze-annotated validation pairs, expressed as percentages of the attainable reward.
| Alignment | Corr. | CheXGaze (%) | Population prior (%) | CheXGaze − Prior |
|---|---|---|---|---|
| Default | 1.00 | 20.3 | 16.7 | 3.6 |
| Random ±1 s | 0.94 | 20.1 | 16.7 | 3.4 |
| Random ±2 s | 0.87 | 19.8 | 16.7 | 3.1 |
| Shift −2 s | 0.65 | 22.0 | 17.7 | 4.3 |
| Shift −1 s | 0.84 | 22.2 | 17.6 | 4.6 |
| Shift +1 s | 0.83 | 17.2 | 15.8 | 1.4 |
| Shift +2 s | 0.06 | 15.1 | 14.9 | 0.2 |
| Random time | 0.37 | 15.4 | 16.3 | −0.9 |
Corr.: median correlation with the default targets. CheXGaze and Population prior: gaze reward of each placement on held-out QA pairs
Random perturbations up to ±2 s largely preserve target correlations and reward margins. The margin disappears under random-time sampling within the same reading session, suggesting that temporally aligned fixations provide question-specific information beyond general viewing patterns. Later shifts reduce the margin, whereas shifts of 1–2 s earlier increase it, consistent with radiologists fixating regions before describing them. Expert-validated alignment may improve supervision quality and remains a direction for future work. We add this analysis to Appendix C in the revised version.
Dear Reviewer JLje, thank you for your constructive suggestions, which helped us strengthen our work. In the revision, we
Response to Q1 & W2 - Isolating the contribution of gaze-based RL
Table 3 clarification. Removing a tag changes both training and inference. The removed tag is excluded from the SFT target, the generated output, and its reward in GRPO. SFT uses token-level cross-entropy on the corresponding target, and all variants use the same two-cycle SFT–GRPO schedule with identical epochs and hyperparameters.
<answer><organ><answer><gaze><answer><organ><gaze><answer>Isolating the gaze reward. To isolate the contribution of the gaze reward, we evaluate CheXGaze on MIMIC-CXR-VQA with (1) the gaze reward weight set to zero and (2) each GRPO stage replaced by SFT on the same gaze subset for the same number of epochs.
Both removing the gaze reward and replacing GRPO with SFT reduce accuracy and F1. These results indicate that the gaze reward contributes beyond structured SFT.
MIMIC-CXR-VQA: single run. SLAKE-CXR and VQA-RAD-CXR: average over three runs.
Furthermore, to assess whether the gaze reward benefits from sample-specific radiologist fixations, we replace each gaze target with either a target from another training pair (shuffled) or the population prior. Both controls reduce overall performance across all three benchmarks. The population-prior target yields similar performance on MIMIC-CXR-VQA but larger drops on SLAKE-CXR and VQA-RAD-CXR. We add the Table 3 clarification and the gaze-reward ablation to Sec. 4.1 and Sec. 4.2 in the revised version.