- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper highlights that the standard cross-entropy loss used to train models for medical report generation is blind to the clinical relevance of tokens. It punishes (or reinforces) stylistic tokens just as much as diagnostically relevant ones, despite their crucially different importance. Diagnostic expressions, extracted from the training set in an offline step, are used to filter for relevant tokens. Instead of just applying standard cross-entropy loss, relevant token probabilities belonging to supported targets are pooled using "noisy-OR" (should be present at least once), while token probabilities belonging to unsupported targets are pooled in a log-sum-exp aggregation (should never be present). These aggregated losses are added to the uniformly-pooled loss over all tokens. The method is validated across six methods and several metrics compared to plain cross-entropy on 3D radiology report generation.
Critical strengths
Tackles a relevant problem with a not runtime-increasing method
Critical weaknesses
- Eq. 5 aggregates the probabilities in such a way that a single high probability is enough to get a score close to . For a multi-token finding (eg let's assume "<chole><lith><iasis>" being a 3-token word) this will almost always reach a very high score, since <iasis> is very likely given <chole><lith>, despite the relevant token probability <chole> being perhaps very small. Thus the loss can be almost saturated even when the relevant finding is unlikely to be generated. This questions the motivation and alignment with the goal of improving diagnostic faithfulness severely.
- No comparison to baselines besides plain cross-entropy. Improvements over CE are a necessary validation, but there exists a large body of related work (eg. LLM-RG4, SAT-RRG) against which the method should be compared. Also a simple token reweighting (increase weight of relevant tokens) using the same diagnostic groups should be considered as baseline, to justify asymmetric construction. Without this the necessity and effectiveness of the specific asymmetric formulation is unclear.
- No information on several hyperparameters such as learning rate, batch size, and number of optimization steps. No information on whether, and if so, how, these are tuned. Without it reproducibility and understanding of experimental setup are very limited.
- No ablation on the benefit of log-sum-exp over a simpler max-aggregation for negative group (Table 3) or for different temperatures . Required to understand the dynamics of the loss construction.
- Main results (Table 1) are based on single training run per setup, the 9-run seed analysis (Appendix F) was performed on the one of the strongest methods (highest Merlin RadGraph-F1 improvement). 2-3 runs per method would have provided more information about the method's robustness across runs.
Questions for authors
- There are large, unaddressed differences between reported scores in Table 1 and Table 3: e.g. fVLM Radgraph F1 for CE in Table 1 is 46.5, but in Table 3 55.55, stronger than DLoss in Table 1 (53.09). Similar discrepancies for other metrics. How do the authors explain those?
- The diagnostic loss coefficients are surprisingly small (Table 5). What is the contribution of Diagnostic Loss to the total loss?
- Did the authors perform any evaluation on the extractor's quality?
- Equation 9 introduces a sample weight. It is never specified how it is determined or set in the experiments, could the authors clarify that?
- How were checkpoints selected? Based on validation loss, after fixed number of steps or differently?
8 comments
Response to Reviewer FNfa (2/4)
W2, W4: No comparison to baselines besides plain cross-entropy. Improvements over CE are a necessary validation, but there exists a large body of related work (eg. LLM-RG4, SAT-RRG) against which the method should be compared. Also a simple token reweighting (increase weight of relevant tokens) using the same diagnostic groups should be considered as baseline, to justify asymmetric construction. Without this the necessity and effectiveness of the specific asymmetric formulation is unclear. No ablation on the benefit of log-sum-exp over a simpler max-aggregation for negative group (Table 3) or for different temperatures . Required to understand the dynamics of the loss construction.
We added matched fVLM comparisons using seed 101 and evaluated them on the full Merlin test set. Diagnostic-token-weighted CE up-weights reference tokens that express supported targets. Unlikelihood penalizes the probability mass of the same unsupported targets used by DLoss at each report position. We also adapted LLM-RG4’s sentence-weighting strategy to abdominal CT by selecting sentences using the shared diagnostic vocabulary in place of the original CheXbert-based selection. Direct comparisons with the full LLM-RG4 and SAT-RRG methods on abdominal CT remain open.
For the pooling ablation, we fixed the supported-target noisy-OR term, the diagnostic target sets, and both loss coefficients. We varied only the aggregation of unsupported-target probability across report positions. At a unique maximum, hard max sends the gradient to one position. Log-sum-exp distributes it across positions, with larger concentrating more of the gradient on positions with high unsupported-target probability.
| Objective | GREEN | F1-RG |
|---|---|---|
| DLoss, (default) | +16.68 | +14.10 |
| Token-weighted CE, | +12.00 | +1.40 |
| Unlikelihood, | +4.70 | +2.45 |
| Adapted sentence-weighted CE, | +2.45 | −6.03 |
| DLoss, | +7.32 | +4.93 |
| DLoss, | +0.52 | −1.64 |
| DLoss, | +8.43 | +2.48 |
| DLoss, | +4.25 | +1.98 |
| DLoss, hard max | +2.35 | −0.18 |
All entries are relative changes from the same CE baseline. The token-weighted CE and unlikelihood rows show the settings with the highest GREEN in their respective grids. Default DLoss scored highest on all six reported metrics across the tested objectives. Performance varied non-monotonically with , and performed best among the tested settings. In the separate seed-13 experiment in Table 3, replacing supported-target noisy-OR with max or mean also lowered all three reported metrics.
Section 4.3 summarizes these results, and Appendix G.1–G.3 provides the full grids and objective definitions. The new comparisons cover one backbone, one seed, and one test cohort.
W3: No information on several hyperparameters such as learning rate, batch size, and number of optimization steps. No information on whether, and if so, how, these are tuned. Without it reproducibility and understanding of experimental setup are very limited.
For the main experiments in Table 1, we use AdamW with zero weight decay, a cosine learning-rate schedule with warmup, and no gradient accumulation. We train the LoRA parameters and the visual-to-text projector or adapter with fixed visual features. Appendix E.1 describes the LoRA configurations. All backbones use the pooled Merlin–CancerVerse training set of approximately 391,000 organ-level findings, with one training example per finding. GPU memory and availability determine the number of GPUs and the per-GPU batch size.
| Backbone | Decoder | Peak learning rate | Batch size (per GPU × GPUs) | Approx. optimizer steps per epoch |
|---|---|---|---|---|
| Merlin | RadLLaMA-7B | 1e-5 | 16 × 2 | 12.2k |
| fVLM | Qwen2.5-0.5B | 2e-4 | 32 × 2 | 6.1k |
| BTB3D | Llama-3.1-8B | 2e-5 | 8 × 2 | 24.5k |
| HLIP | Qwen2.5-0.5B | 2e-4 | 64 × 1 | 6.1k |
| RadFM | MedLLaMA-13B | 2e-5 | 64 × 2 | 3.1k |
| SPECTRE | Qwen2.5-0.5B | 2e-4 | 32 × 2 | 6.1k |
Checkpoint selection and tuning. As described in Appendix E.1, we evaluate token-level cross-entropy on the validation set after each epoch and retain the checkpoint with the lowest value. Training stops after three consecutive epochs without improvement. Within each backbone, CE and CE + DLoss use the same optimizer, learning rate and schedule, batch size, GPU configuration, and checkpoint-selection rule. We selected the two backbone-specific DLoss coefficients through a small grid search and fixed for the main DLoss runs. Appendix E.1 lists the coefficients, and Appendix G.3 reports the temperature sweep.
Response to Reviewer FNfa (3/4)
W5: Main results (Table 1) are based on single training run per setup, the 9-run seed analysis (Appendix F) was performed on the one of the strongest methods (highest Merlin RadGraph-F1 improvement). 2-3 runs per method would have provided more information about the method's robustness across runs.
Table 1 reports one matched CE and CE + DLoss training pair per backbone. We extended the repeated-run analysis beyond BTB3D by evaluating three matched fVLM pairs (seeds 101, 42, and 13) on the full Merlin test set. Within each pair, both objectives use the same seed and training setup.
| Backbone | Matched pairs | GREEN gain (95% CI) | F1-RG gain (95% CI) |
|---|---|---|---|
| BTB3D | 9 | +4.55 [1.97, 7.32] | +7.95 [2.03, 14.51] |
| fVLM | 3 | +4.85 [0.34, 8.99] | +3.66 [0.89, 6.55] |
Gains are mean within-pair differences in score points. The intervals bootstrap matched training pairs and test studies. GREEN and F1-RG each improve in eight of nine BTB3D pairs, and all nine reported metrics improve in all three fVLM pairs. Hardware varied across fVLM pairs, and the intervals summarize variability across the three observed pairs.
We also evaluated the same nine BTB3D pairs on the external cohort. GREEN improves in seven of nine pairs, with a mean gain of +1.53 points (95% CI [0.29, 2.54]). The intervals for the other five reported metrics include zero. Appendix F.2 and F.4 report the fVLM and external BTB3D results, respectively.
For Merlin and RadFM, Appendix F.5 reports paired bootstrap intervals that quantify test-study sampling uncertainty conditional on the fixed Table 1 checkpoints. Repeated-training evidence covers two of the six backbones, BTB3D and fVLM.
Q1: There are large, unaddressed differences between reported scores in Table 1 and Table 3: e.g. fVLM Radgraph F1 for CE in Table 1 is 46.5, but in Table 3 55.55, stronger than DLoss in Table 1 (53.09). Similar discrepancies for other metrics. How do the authors explain those?
Tables 1 and 3 report separate fVLM training-run pairs, which the original captions did not distinguish sufficiently clearly. Table 1 uses seed 101; Table 3 uses seed 13 and its own CE checkpoint. Within each pair, CE and CE + DLoss use the same seed and training setup. Both experiments evaluate the full Merlin test set.
We have added a three-seed fVLM analysis to Appendix F, including both originally reported pairs:
| Seed | CE F1-RG | CE + DLoss F1-RG | Paired gain |
|---|---|---|---|
| 101 — Table 1 | 46.53 | 53.09 | +6.56 points |
| 42 | 46.10 | 49.66 | +3.56 points |
| 13 — Table 3 | 55.55 | 56.41 | +0.86 points |
Thus, the absolute baseline scores and improvement sizes vary substantially between runs, while the matched DLoss-minus-CE differences remain positive. The ablations also used A100 hardware, whereas the main experiments used GH200 hardware; we therefore do not attribute all between-run variation to random seed alone.
Across the three matched pairs, mean improvements are +4.85 GREEN points and +3.66 F1-RG points. All nine evaluated metrics improve in every pair. Appendix F now reports the absolute scores, within-seed differences, bootstrap intervals, and the limitations of inference from three seed pairs.
Table 1 reports absolute-score differences, while Table 3 reports relative percentage changes. The +1.54% F1-RG gain in Table 3 corresponds to +0.86 points, from 55.55 to 56.41. Appendix F.2 identifies the matched baselines and reports the per-run scores.
Q2: The diagnostic loss coefficients are surprisingly small (Table 5). What is the contribution of Diagnostic Loss to the total loss?
Appendix E.1 reports the magnitude of the weighted diagnostic term, including and , relative to cross-entropy. Averaged over training up to the selected checkpoints, its absolute magnitude ranges from 0.2% to 4.8% of the CE loss across backbones. We also measured the gradient norms of the two terms on training batches at the selected checkpoints, using the same parameters for both terms.
| Backbone | Weighted DLoss magnitude relative to CE | Weighted DLoss gradient norm relative to CE |
|---|---|---|
| fVLM | 0.2% | 2.8% |
| BTB3D | 0.6% | 1.6% |
| RadFM | 4.8% | 18.5% |
Both columns use cross-entropy as the denominator. The loss ratios summarize training up to the selected checkpoint, and the gradient ratios describe measurements at that checkpoint.
Response to Reviewer FNfa (1/4)
Thank you for the constructive review, which has made our paper stronger.
The reviewer identifies a genuine limitation under teacher forcing: a continuation token can receive high probability after a reference-supplied prefix, allowing Eq. (5) to become large even when the realization’s first-token probability is low. We have clarified that q+ is an auxiliary token-level surrogate, not the probability of generating a complete diagnostic phrase. CE continues to supervise every reference token, including phrase starts.
We investigated this behavior using the RadFM CE and DLoss checkpoints from Table 1, evaluated under teacher forcing without weight updates. We randomly sampled 4,000 training and 4,000 validation organ-level findings and evaluated every supported target, yielding 8,135 and 8,266 supported-target instances.
An instance meets our continuation-only criterion when the reference contains multi-token lexical realizations of the target and the tokens after their first token alone yield a noisy-OR score of at least 0.9. We use this as an operational high-score threshold, with −log(0.9)≈0.105.
For the RadFM DLoss checkpoint, continuation tokens alone yielded a score of at least 0.9 while the first-token probability was below 0.1 in 3.0% of sampled training targets and 5.4% of sampled validation targets. The corresponding rates for CE were 2.9% and 5.3%. All percentages are computed over all sampled supported-target instances in the corresponding split (8,135 training and 8,266 validation). The joint event was uncommon across the sampled targets overall. Its frequency within the subset with matched multi-token realizations remains unknown.
Separately, the fraction of supported-target instances with the full Eq. (5) score below 0.1 decreases from 16.5% to 1.2% on training and from 25.6% to 2.1% on validation. This shows a change in the teacher-forced supervision-score distribution, rather than independently establishing improved complete-phrase generation probability. Generated-report performance is evaluated separately in Table 1.
We have added this diagnostic and its scope limitations in Appendix H and clarified the surrogate’s interpretation in Section 3.2 and Limitations.