- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies on-policy distillation (OPD) for medical vision-language models to generate chest X-ray radiology reports. The paper first identifies two empirical failure modes of vanilla OPD: (1) the student incorrectly allocates visual attention toward template tokens rather than clinically relevant tokens, and (2) a notable fraction of student rollouts can outperform the teacher. To address these two issues, the authors propose VITAL, a two-module framework. The VAP module uses image perturbation to compute token-level visual dependency and reweights policy gradients to prioritize clinical tokens. The TAD module selectively filters student rollouts for distillation, retaining only samples where the teacher achieves superior performance measured by ROUGE-L. The method is validated on four chest X-ray datasets including both in-domain and out-of-distribution benchmarks for adult and neonatal radiology. Experimental results show VITAL improves OOD clinical F1 over standard OPD baselines. Ablation studies verify that both VAP and TAD contribute to performance gains.
Critical strengths
1.The paper identifies two non-trivial, empirically validated failure modes for on-policy distillation (OPD) in medical VLM report generation. The authors quantitatively show student models suffer from visual misallocation deficit (visual sensitivity leaks into template tokens instead of clinical anatomy/severity tokens) and task deficit (over 20% student rollouts outperform the teacher). These two counterintuitive observations directly motivate the design of the two modules in VITAL, and the motivating figure provides solid empirical evidence. This insight is valuable for the medical multimodal distillation community, which often assumes the student only lacks visual capacity and the teacher is universally better. 2.The proposed two-module framework achieves strong out-of-distribution performance gains on chest X-ray datasets. VITAL combines token-level visual reweighting (VAP) and rollout-level selective distillation (TAD). Evaluations across 4 chest X-ray datasets, including both in-domain and challenging OOD splits for adult and neonatal radiology, demonstrate consistent OOD F1 improvements compared with strong OPD baselines. Ablation studies confirm both modules contribute to performance, which supports the core claim that the two designed components are complementary. The code is anonymously released for reproducibility.
Critical weaknesses
1.The technical novelty of VAP is limited, and the boundary with Med-OPD is not sufficiently established. Med-OPD already uses image perturbation to estimate token-wise visual evidence. VAP repurposes this perturbation signal as a scaling factor for policy gradient weights. The paper does not include the critical ablation of Med-OPD + TAD. Without this comparison, we cannot confirm whether the performance gain comes from the novel VAP design or merely combining Med-OPD’s visual module with TAD selective filtering. This blurs the core technical contribution of this work. 2. The TAD module’s reliance on ROUGE-L as the proxy metric introduces unvalidated systematic risk for clinical diagnosis. TAD uses ROUGE-L for rollout filtering. ROUGE-L measures text overlap and does not perfectly align with clinical diagnostic correctness. The reported correlation between ROUGE-L and clinical F1 is only provided for NeoCXR. The paper lacks correlation statistics for MIMIC-CXR and CheXpert Plus, and there is no control experiment replacing ROUGE-L with clinical F1 to test robustness. For safety-critical medical tasks, using an unvalidated proxy metric for sample selection is a major concern and weakens the reliability of the proposed TAD module.
Questions for authors
1.The paper states that over 20% of student rollouts outperform the teacher in Figure 1(b), but the metric used to compute this statistic is not specified. Please clarify whether this percentage is calculated using ROUGE-L or clinical F1. Please also update both the main text and figure caption to explicitly state the metric used. 2. The in-domain gains on MIMIC-CXR and NeoCXR are tiny and may fall within random training variance. Could you report results across multiple random seeds with mean ± std and statistical significance tests for all primary benchmarks? Especially for CheXpert Plus, which has only 202 test samples. Please answer: Are the reported ID and OOD improvements statistically significant? 3. Could you add two baselines: (1) simple top-k rollout filtering baseline; (2) VITAL with vanilla SFT teacher (without GSPO). Please answer: How much performance difference exists between TAD and simple top-k filtering? Does VITAL maintain consistent gains when using a non-GSPO SFT teacher? 4. The authors swept and , but temperature parameter is missing from hyperparameter ablation. Please add ablation for . Please answer: How sensitive is final performance to different choices of ? 5. The paper only evaluates on chest X-ray. The current claim "for medical diagnosis" is overbroad. Please revise title and abstract to restrict scope to chest X-ray report generation. Also report training compute overhead (GPU hours, runtime) introduced by VAP’s extra perturbed-image forward passes. Please answer: What is the additional training cost of VITAL compared with Native OPD? 6. Could you add more qualitative failure cases in the appendix? Please discuss two risks: (1) VAP only models token-level visual dependency and cannot spatially localize pathology; (2) TAD may learn incorrect teacher outputs when the teacher produces medical hallucinations. Please answer: In what typical failure scenarios does VITAL still fail, and how severe is the teacher hallucination risk for TAD? 7. The two core motivating challenges in the introduction are conceptually reasonable, but their supporting empirical validation is insufficient. For Challenge 1, please quantitatively compare the visual dependency weights on template tokens versus clinically relevant tokens between the Med-OPD baseline and the full VITAL model, to verify that VAP effectively suppresses visual weights assigned to template tokens. For Challenge 2, please disentangle whether performance gains originate from the proposed teacher-relative filtering mechanism, or merely from discarding low-quality rollout samples. Please answer: Can you provide quantitative evidence showing VAP reduces visual attention on template tokens and strengthens visual dependency on clinical tokens? Can you confirm that TAD’s gain comes from the proposed teacher-relative rule rather than simple low-quality sample filtering?
8 comments
4. Sensitivity analysis for the temperature parameter ()
The results of our sensitivity analysis for the temperature parameter () on MIMIC-CXR are shown below. Performance varies slightly across the tested temperatures, with the best result at .
\hline \textbf{$\tau$} & \textbf{R-L } (\uparrow) & \textbf{R-1 } (\uparrow) & \textbf{MET } (\uparrow) & \textbf{RaTE } (\uparrow) & \textbf{P } (\uparrow) & \textbf{R } (\uparrow) & \textbf{F1 } (\uparrow) & \textbf{Avg. } (\uparrow) \\\\ \hline 0.1 & 27.85 & 29.24 & 20.45 & 51.05 & 42.59 & 40.05 & 41.28 & 36.07 \\\\ 0.5 & \textbf{27.94} & \textbf{29.33} & \textbf{20.54} & \textbf{51.14} & \textbf{42.68} & \textbf{40.14} & \textbf{41.37} & \textbf{36.16} \\\\ 1.0 & 27.89 & 29.28 & 20.49 & 51.09 & 42.63 & 40.09 & 41.32 & 36.11 \\\\ \hline \end{array}$$ **5. Scope and Computational Cost** 1)Clarification of Scope We agree with your assessment. We will change the title from "VITAL: Visual and Task Advantage Learning for On-Policy Distillation in Medical Diagnosis" to "VITAL: Visual and Task Advantage Learning for On-Policy Distillation in Medical Report Generation",and synchronously update abstract. 2)Computational Cost We quantified the computational resources required for training VITAL and Native OPD, see table below. Both runs used four NVIDIA A800 GPUs. According to the training logs, VITAL took 504 seconds (8.4 minutes) longer than Native OPD, a 14.6% increase in end-to-end wall-clock time and an additional 0.56 GPU-hours. The additional cost comes primarily from VAP. In Native OPD, the teacher uses the original image to compute log probabilities for the tokens generated by the student. VAP adds one teacher forward pass on the same GPU using a Gaussian-perturbed image. It scores the same tokens and computes the visual weights from the differences in log probabilities between the two passes. We have added this table to session Additional Analysis of VITAL in the appendix. $$\begin{array}{l|ccc} \hline\hline \textbf{Training cost} & \textbf{Native OPD} & \textbf{VITAL} & \textbf{Additional cost} \\\\ \hline \text{Training Time} & \text{3,463 s} & \text{3,967 s} & \text{+504 s (+14.6\\%)} \\\\ \text{GPU-hours} & \text{3.85} & \text{4.41} & \text{+0.56 (+14.6\\%)} \\\\ \hline\hline \end{array}$$ **6.Failure Cases** Thank you for this suggestion. VAP models visual dependence at the token level. Because VITAL optimizes report generation at the text level, it cannot spatially localize lesions. VAP models dependence on visual information at the token level. Because VITAL optimizes report generation at the text level, it cannot spatially localize lesions. TAD uses the teacher’s score advantage over the student to decide when to distill. However, because the rejection decision is made at the rollout level, the student may imitate the teacher’s entire output, including incorrect tokens. In future work, we will explore finer-grained rejection distillation to selectively learn from the teacher’s correct tokens. We have discussed both limitations in the Limitations section and provide qualitative examples of TAD failure cases in the appendix. In future work, we will explore region- and pixel-level lesion localization and finer-grained distillation that selectively learns from correct teacher tokens.7. Supplementary Evidence Addressing Two Core Challenges
1)Quantitative Comparison with MedOPD
As shown below, VAP shifts sensitivity from non-clinical to clinical tokens: Other drops from 32.01% to 23.64%, while Anatomy and Severity rise from 7.60% to 15.34% and 2.90% to 9.81%, respectively. In contrast, MedOPD increases overall sensitivity but yields lower clinical attention (Anatomy: 11.61%, Severity: 6.52%) and maintains high redundant sensitivity (Other: 24.62%, Visual Sensitivity Rate: 38.13%). VITAL achieves better calibration—driving stronger clinical attention while minimizing redundant visual dependencies (Visual Sensitivity Rate: 34.42%).
\hline\hline \textbf{Metric} & \textbf{Direction} & \textbf{2B Student} & \textbf{MedOPD} & \textbf{VITAL (Ours)} & \textbf{8B teacher} \\\\ \hline \text{other \\%} & \downarrow & 32.01\\% & 24.62\\% & \textbf{23.64\\%} & 21.78\\% \\\\ \text{anatomy \\%} & \uparrow & 7.60\\% & 11.61\\% & \textbf{15.34\\%} & 12.05\\% \\\\ \text{severity \\%} & \uparrow & 2.90\\% & 6.52\\% & \textbf{9.81\\%} & 6.94\\% \\\\ \text{Visual Sensitivity Rate} & \downarrow & 44.73\\% & 38.13\\% & \textbf{34.42\\%} & 35.65\\% \\\\ \hline\hline \end{array}$$ 2)Source of TAD's Advantage We compared our method with the Top-K distillation approach, which distills only the top-2 rollouts ranked by student task performance. The comparison shows that the improvement of our method stems from bridging the teacher-student gap rather than merely filtering out low-quality samples. $$\begin{array}{l|cc|cc} \hline\hline \textbf{Method} & \textbf{MIMIC R-L } (\uparrow) & \textbf{MIMIC F1 } (\uparrow) & \textbf{CheXpert R-L } (\uparrow) & \textbf{CheXpert F1 } (\uparrow) \\\\ \hline \text{VAP + Top-K (k=2)} & \text{27.31} & \text{40.01} & \text{24.69} & \text{38.85} \\\\ \textbf{VAP + TAD (Ours)} & \textbf{27.94 } (+0.63) & \textbf{41.37 } (+1.36) & \textbf{25.03 } (+0.34) & \textbf{40.47 } (+1.62) \\\\ \hline\hline \end{array}$$
Thank you for your thorough and expert feedback. We have done our best to address your concerns below.
1. Task Quality Proxy (ROUGE-L) . We use ROUGE-L, which is consistent with the definition of task advantage in Equation (9). Thank you for pointing this out. We have changed the sub-captions of Figure 1 to "(a) Visual deficit (Visual Sensitivity by Token Category)" and "(b) Task Deficit (ROUGE-L-Based Rollout Win Rates)". In addition, we have highlighted the phrase "task quality (proxied by ROUGE-L scores)," in red on Line 075 in the main text.
2. Mean and Standard Deviation.
As shown in the table below, VITAL achieves consistent results across runs. The standard deviation of the average score is 0.01 on MIMIC-CXR and rounds to 0.00 on NeoCXR-EV. We acknowledge that VITAL’s gains on the in-domain MIMIC-CXR benchmark are modest. However, our method generalizes substantially better than the state-of-the-art method on out-of-domain test sets, improving F1 by 3.66 percentage points on CheXpert Plus and 2.79 percentage points on NeoCXR-EV. We have added this table to session Additional Analysis of VITAL in the appendix.
\hline\hline \textbf{Metric} & \textbf{MIMIC-CXR} & \textbf{CheXpert Plus} & \textbf{NeoCXR} & \textbf{NeoCXR-EV} \\\\ \hline \text{R-L } (\uparrow) & 27.94 \pm 0.02 & 25.03 \pm 0.02 & 46.32 \pm 0.00 & 32.91 \pm 0.08 \\\\ \text{R-1 } (\uparrow) & 29.33 \pm 0.02 & 25.88 \pm 0.01 & 46.80 \pm 0.00 & 33.84 \pm 0.08 \\\\ \text{MET } (\uparrow) & 20.54 \pm 0.00 & 17.02 \pm 0.22 & 45.76 \pm 0.00 & 32.43 \pm 0.17 \\\\ \text{RaTE } (\uparrow) & 51.14 \pm 0.15 & 43.26 \pm 0.24 & 59.91 \pm 0.01 & 50.33 \pm 0.01 \\\\ \text{P } (\uparrow) & 42.68 \pm 0.20 & 38.27 \pm 1.08 & 55.19 \pm 0.03 & 55.14 \pm 0.00 \\\\ \text{R } (\uparrow) & 40.14 \pm 0.03 & 42.93 \pm 0.06 & 55.19 \pm 0.00 & 45.68 \pm 0.00 \\\\ \text{F1 } (\uparrow) & 41.37 \pm 0.11 & 40.47 \pm 0.52 & 55.19 \pm 0.00 & 49.97 \pm 0.00 \\\\ \hline \textbf{Avg. } (\uparrow) & \textbf{36.16} \pm 0.01 & \textbf{33.27} \pm 0.31 & \textbf{52.05} \pm 0.00 & \textbf{42.90} \pm 0.02 \\\\ \hline\hline \end{array}$$ **3. Two Baseline Settings** **1). Simple top-$k$ filtering.** We compared our method with a Top-K baseline that selects the top-2 rollouts based on student task performance, see table below. Although Top-K filters out poor samples, it ignores the hard samples the student truly needs guidance on. Moreover, it imposes constraints on samples where the student already outperforms the teacher, which can degrade the student's capabilities. As a result, this approach yields inferior performance compared to our TAD. We have added this ablation study to session Additional Analysis of VITAL in the appendix. $$\begin{array}{l|cc|cc} \hline\hline \textbf{Method} & \textbf{MIMIC R-L } (\uparrow) & \textbf{MIMIC F1 } (\uparrow) & \textbf{CheXpert R-L } (\uparrow) & \textbf{CheXpert F1 } (\uparrow) \\\\ \hline \text{VAP + Top-K (k=2)} & \text{27.31} & \text{40.01} & \text{24.69} & \text{38.85} \\\\ \textbf{VAP + TAD (Ours)} & \textbf{27.94 } \textcolor{teal}{(+0.63)} & \textbf{41.37 } \textcolor{teal}{(+1.36)} & \textbf{25.03 } \textcolor{teal}{(+0.34)} & \textbf{40.47 } \textcolor{teal}{(+1.62)} \\\\ \hline\hline \end{array}$$ **2). VITAL using only a standard SFT teacher (without GSPO).** Distillation from an SFT teacher proves suboptimal due to the limited performance gap between the teacher and student. As a result, neither VITAL nor native OPD achieves meaningful performance gains in this setting. $$\begin{array}{llccccccccc} \hline \textbf{Dataset} & \textbf{Model} & \textbf{R-L} & \textbf{R-1} & \textbf{MET} & \textbf{RaTE} & \textbf{P} & \textbf{R} & \textbf{F1} & \textbf{Avg} & \textbf{$\Delta$Avg} \\\\ \hline \text{MIMIC} & \text{8B SFT teacher} & 26.58 & 27.95 & 19.85 & 49.96 & 32.14 & 20.35 & 22.79 & 28.52 & \text{---} \\\\ \text{MIMIC} & \text{2B student} & 17.73 & 19.25 & 20.17 & 49.08 & 35.27 & 24.87 & 29.17 & 27.93 & \text{---} \\\\ \text{MIMIC} & \text{Native OPD} & 22.74 & 24.63 & 16.27 & 45.59 & 33.19 & 23.72 & 27.67 & 27.69 & -0.25 \\\\ \text{MIMIC} & \text{VITAL} & 21.02 & 22.51 & 14.90 & 44.12 & 27.20 & 17.28 & 21.14 & 24.03 & -3.91 \\\\ \hline \text{CXP} & \text{8B SFT teacher} & 21.78 & 22.82 & 16.56 & 41.42 & 31.11 & 28.04 & 28.18 & 27.13 & \text{---} \\\\ \text{CXP} & \text{2B student} & 18.92 & 19.98 & 17.28 & 42.76 & 36.45 & 38.71 & 37.55 & 30.23 & \text{---} \\\\ \text{CXP} & \text{Native OPD} & 17.42 & 18.74 & 8.59 & 35.83 & 31.73 & 24.57 & 27.69 & 23.51 & -6.73 \\\\ \text{CXP} & \text{VITAL} & 17.32 & 18.51 & 8.44 & 35.44 & 31.93 & 26.50 & 28.96 & 23.87 & -6.36 \\\\ \hline \end{array}$$