acceptodds
Under review as a conference paper at ICLR 2027

Diagnostic Loss: Asymmetric Supervision for Radiology Report Generation

Abstract

Automated radiology report generation requires models to do more than produce fluent clinical narratives: they must explicitly state supported diagnoses while avoiding unsupported ones. However, existing medical vision-language models (MedVLMs) are typically trained with token-level maximum likelihood objectives, which prioritize agreement with reference text but do not directly constrain the probability assigned to diagnostically critical terms. As a result, models can generate plausible reports that omit key diagnoses, hedge on clinically evident findings, or introduce unsupported differential diagnoses. We identify this as an objective misalignment between conventional language modeling and the asymmetric diagnostic requirements of radiology reporting. To address this problem, we introduce Diagnostic Loss (DL), a differentiable, architecture-agnostic objective that operates directly on decoder token probabilities. DL separately aggregates probability mass over supported and unsupported diagnostic vocabularies using asymmetric temporal pooling: it encourages supported diagnoses to receive sufficient probability at at least one position while suppressing spurious probability spikes for unsupported diagnoses throughout the generated report. The loss can be applied to existing 2D and 3D MedVLMs without modifying their vision backbones, language decoders, or inference procedures, and introduces no additional inference-time computation. We evaluate DL across multiple 2D and 3D MedVLM architectures on both in-distribution and out-of-distribution report-generation benchmarks. Comprehensive experiments show that DL improves diagnostic faithfulness while maintaining competitive performance on conventional language-generation metrics, demonstrating the value of aligning training objectives with the clinical semantics of report generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
radiology report generation, llm, transformer, medical imaging

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FNfaOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper highlights that the standard cross-entropy loss used to train models for medical report generation is blind to the clinical relevance of tokens. It punishes (or reinforces) stylistic tokens just as much as diagnostically relevant ones, despite their crucially different importance. Diagnostic expressions, extracted from the training set in an offline step, are used to filter for relevant tokens. Instead of just applying standard cross-entropy loss, relevant token probabilities belonging to supported targets are pooled using "noisy-OR" (should be present at least once), while token probabilities belonging to unsupported targets are pooled in a log-sum-exp aggregation (should never be present). These aggregated losses are added to the uniformly-pooled loss over all tokens. The method is validated across six methods and several metrics compared to plain cross-entropy on 3D radiology report generation.

Critical strengths

Tackles a relevant problem with a not runtime-increasing method

Critical weaknesses
  • Eq. 5 aggregates the probabilities in such a way that a single high probability is enough to get a score close to q+=1.0q^+=1.0. For a multi-token finding (eg let's assume "<chole><lith><iasis>" being a 3-token word) this will almost always reach a very high score, since <iasis> is very likely given <chole><lith>, despite the relevant token probability <chole> being perhaps very small. Thus the loss can be almost saturated even when the relevant finding is unlikely to be generated. This questions the motivation and alignment with the goal of improving diagnostic faithfulness severely.
  • No comparison to baselines besides plain cross-entropy. Improvements over CE are a necessary validation, but there exists a large body of related work (eg. LLM-RG4, SAT-RRG) against which the method should be compared. Also a simple token reweighting (increase weight of relevant tokens) using the same diagnostic groups should be considered as baseline, to justify asymmetric construction. Without this the necessity and effectiveness of the specific asymmetric formulation is unclear.
  • No information on several hyperparameters such as learning rate, batch size, and number of optimization steps. No information on whether, and if so, how, these are tuned. Without it reproducibility and understanding of experimental setup are very limited.
  • No ablation on the benefit of log-sum-exp over a simpler max-aggregation for negative group (Table 3) or for different temperatures τ\tau. Required to understand the dynamics of the loss construction.
  • Main results (Table 1) are based on single training run per setup, the 9-run seed analysis (Appendix F) was performed on the one of the strongest methods (highest Merlin RadGraph-F1 improvement). 2-3 runs per method would have provided more information about the method's robustness across runs.
Questions for authors
  • There are large, unaddressed differences between reported scores in Table 1 and Table 3: e.g. fVLM Radgraph F1 for CE in Table 1 is 46.5, but in Table 3 55.55, stronger than DLoss in Table 1 (53.09). Similar discrepancies for other metrics. How do the authors explain those?
  • The diagnostic loss coefficients are surprisingly small (Table 5). What is the contribution of Diagnostic Loss to the total loss?
  • Did the authors perform any evaluation on the extractor's quality?
  • Equation 9 introduces a sample weight. It is never specified how it is determined or set in the experiments, could the authors clarify that?
  • How were checkpoints selected? Based on validation loss, after fixed number of steps or differently?
8 comments
AuthorsOpenReview ↗

Response to Reviewer FNfa (1/4)

Thank you for the constructive review, which has made our paper stronger.

W1: Eq. 5 aggregates the probabilities in such a way that a single high probability is enough to get a score close to q+=1.0q^+ = 1.0 . For a multi-token finding (eg let's assume "<chole><lith><iasis>" being a 3-token word) this will almost always reach a very high score, since <iasis> is very likely given <chole><lith>, despite the relevant token probability <chole> being perhaps very small. Thus the loss can be almost saturated even when the relevant finding is unlikely to be generated. This questions the motivation and alignment with the goal of improving diagnostic faithfulness severely.

The reviewer identifies a genuine limitation under teacher forcing: a continuation token can receive high probability after a reference-supplied prefix, allowing Eq. (5) to become large even when the realization’s first-token probability is low. We have clarified that q+q^+ is an auxiliary token-level surrogate, not the probability of generating a complete diagnostic phrase. CE continues to supervise every reference token, including phrase starts.

We investigated this behavior using the RadFM CE and DLoss checkpoints from Table 1, evaluated under teacher forcing without weight updates. We randomly sampled 4,000 training and 4,000 validation organ-level findings and evaluated every supported target, yielding 8,135 and 8,266 supported-target instances.

An instance meets our continuation-only criterion when the reference contains multi-token lexical realizations of the target and the tokens after their first token alone yield a noisy-OR score of at least 0.9. We use this as an operational high-score threshold, with −log⁡(0.9)≈0.105-\log(0.9)\approx0.105.

CheckpointSplitContinuation-only score ≥ 0.9Continuation-only score ≥ 0.9 and first-token probability < 0.5Continuation-only score ≥ 0.9 and first-token probability < 0.1
CE + DLossTraining7.8%5.1%3.0%
CE + DLossValidation11.7%8.1%5.4%
CETraining7.9%5.0%2.9%
CEValidation11.6%8.0%5.3%

For the RadFM DLoss checkpoint, continuation tokens alone yielded a score of at least 0.9 while the first-token probability was below 0.1 in 3.0% of sampled training targets and 5.4% of sampled validation targets. The corresponding rates for CE were 2.9% and 5.3%. All percentages are computed over all sampled supported-target instances in the corresponding split (8,135 training and 8,266 validation). The joint event was uncommon across the sampled targets overall. Its frequency within the subset with matched multi-token realizations remains unknown.

Separately, the fraction of supported-target instances with the full Eq. (5) score below 0.1 decreases from 16.5% to 1.2% on training and from 25.6% to 2.1% on validation. This shows a change in the teacher-forced supervision-score distribution, rather than independently establishing improved complete-phrase generation probability. Generated-report performance is evaluated separately in Table 1.

We have added this diagnostic and its scope limitations in Appendix H and clarified the surrogate’s interpretation in Section 3.2 and Limitations.

AuthorsOpenReview ↗

Response to Reviewer FNfa (2/4)

W2, W4: No comparison to baselines besides plain cross-entropy. Improvements over CE are a necessary validation, but there exists a large body of related work (eg. LLM-RG4, SAT-RRG) against which the method should be compared. Also a simple token reweighting (increase weight of relevant tokens) using the same diagnostic groups should be considered as baseline, to justify asymmetric construction. Without this the necessity and effectiveness of the specific asymmetric formulation is unclear. No ablation on the benefit of log-sum-exp over a simpler max-aggregation for negative group (Table 3) or for different temperatures . Required to understand the dynamics of the loss construction.

We added matched fVLM comparisons using seed 101 and evaluated them on the full Merlin test set. Diagnostic-token-weighted CE up-weights reference tokens that express supported targets. Unlikelihood penalizes the probability mass of the same unsupported targets used by DLoss at each report position. We also adapted LLM-RG4’s sentence-weighting strategy to abdominal CT by selecting sentences using the shared diagnostic vocabulary in place of the original CheXbert-based selection. Direct comparisons with the full LLM-RG4 and SAT-RRG methods on abdominal CT remain open.

For the pooling ablation, we fixed the supported-target noisy-OR term, the diagnostic target sets, and both loss coefficients. We varied only the aggregation of unsupported-target probability across report positions. At a unique maximum, hard max sends the gradient to one position. Log-sum-exp distributes it across positions, with larger τ\tau concentrating more of the gradient on positions with high unsupported-target probability.

ObjectiveGREENF1-RG
DLoss, τ=8\tau=8 (default)+16.68+14.10
Token-weighted CE, w=4w=4+12.00+1.40
Unlikelihood, α=0.6\alpha=0.6+4.70+2.45
Adapted sentence-weighted CE, w=1.75w=1.75+2.45−6.03
DLoss, τ=2\tau=2+7.32+4.93
DLoss, τ=4\tau=4+0.52−1.64
DLoss, τ=16\tau=16+8.43+2.48
DLoss, τ=32\tau=32+4.25+1.98
DLoss, hard max+2.35−0.18

All entries are relative changes from the same CE baseline. The token-weighted CE and unlikelihood rows show the settings with the highest GREEN in their respective grids. Default DLoss scored highest on all six reported metrics across the tested objectives. Performance varied non-monotonically with τ\tau, and τ=8\tau=8 performed best among the tested settings. In the separate seed-13 experiment in Table 3, replacing supported-target noisy-OR with max or mean also lowered all three reported metrics.

Section 4.3 summarizes these results, and Appendix G.1–G.3 provides the full grids and objective definitions. The new comparisons cover one backbone, one seed, and one test cohort.

W3: No information on several hyperparameters such as learning rate, batch size, and number of optimization steps. No information on whether, and if so, how, these are tuned. Without it reproducibility and understanding of experimental setup are very limited.

For the main experiments in Table 1, we use AdamW with zero weight decay, a cosine learning-rate schedule with warmup, and no gradient accumulation. We train the LoRA parameters and the visual-to-text projector or adapter with fixed visual features. Appendix E.1 describes the LoRA configurations. All backbones use the pooled Merlin–CancerVerse training set of approximately 391,000 organ-level findings, with one training example per finding. GPU memory and availability determine the number of GPUs and the per-GPU batch size.

BackboneDecoderPeak learning rateBatch size (per GPU × GPUs)Approx. optimizer steps per epoch
MerlinRadLLaMA-7B1e-516 × 212.2k
fVLMQwen2.5-0.5B2e-432 × 26.1k
BTB3DLlama-3.1-8B2e-58 × 224.5k
HLIPQwen2.5-0.5B2e-464 × 16.1k
RadFMMedLLaMA-13B2e-564 × 23.1k
SPECTREQwen2.5-0.5B2e-432 × 26.1k

Checkpoint selection and tuning. As described in Appendix E.1, we evaluate token-level cross-entropy on the validation set after each epoch and retain the checkpoint with the lowest value. Training stops after three consecutive epochs without improvement. Within each backbone, CE and CE + DLoss use the same optimizer, learning rate and schedule, batch size, GPU configuration, and checkpoint-selection rule. We selected the two backbone-specific DLoss coefficients through a small grid search and fixed τ=8\tau=8 for the main DLoss runs. Appendix E.1 lists the coefficients, and Appendix G.3 reports the temperature sweep.

AuthorsOpenReview ↗

Response to Reviewer FNfa (3/4)

W5: Main results (Table 1) are based on single training run per setup, the 9-run seed analysis (Appendix F) was performed on the one of the strongest methods (highest Merlin RadGraph-F1 improvement). 2-3 runs per method would have provided more information about the method's robustness across runs.

Table 1 reports one matched CE and CE + DLoss training pair per backbone. We extended the repeated-run analysis beyond BTB3D by evaluating three matched fVLM pairs (seeds 101, 42, and 13) on the full Merlin test set. Within each pair, both objectives use the same seed and training setup.

BackboneMatched pairsGREEN gain (95% CI)F1-RG gain (95% CI)
BTB3D9+4.55 [1.97, 7.32]+7.95 [2.03, 14.51]
fVLM3+4.85 [0.34, 8.99]+3.66 [0.89, 6.55]

Gains are mean within-pair differences in score points. The intervals bootstrap matched training pairs and test studies. GREEN and F1-RG each improve in eight of nine BTB3D pairs, and all nine reported metrics improve in all three fVLM pairs. Hardware varied across fVLM pairs, and the intervals summarize variability across the three observed pairs.

We also evaluated the same nine BTB3D pairs on the external cohort. GREEN improves in seven of nine pairs, with a mean gain of +1.53 points (95% CI [0.29, 2.54]). The intervals for the other five reported metrics include zero. Appendix F.2 and F.4 report the fVLM and external BTB3D results, respectively.

For Merlin and RadFM, Appendix F.5 reports paired bootstrap intervals that quantify test-study sampling uncertainty conditional on the fixed Table 1 checkpoints. Repeated-training evidence covers two of the six backbones, BTB3D and fVLM.

Q1: There are large, unaddressed differences between reported scores in Table 1 and Table 3: e.g. fVLM Radgraph F1 for CE in Table 1 is 46.5, but in Table 3 55.55, stronger than DLoss in Table 1 (53.09). Similar discrepancies for other metrics. How do the authors explain those?

Tables 1 and 3 report separate fVLM training-run pairs, which the original captions did not distinguish sufficiently clearly. Table 1 uses seed 101; Table 3 uses seed 13 and its own CE checkpoint. Within each pair, CE and CE + DLoss use the same seed and training setup. Both experiments evaluate the full Merlin test set.

We have added a three-seed fVLM analysis to Appendix F, including both originally reported pairs:

SeedCE F1-RGCE + DLoss F1-RGPaired gain
101 — Table 146.5353.09+6.56 points
4246.1049.66+3.56 points
13 — Table 355.5556.41+0.86 points

Thus, the absolute baseline scores and improvement sizes vary substantially between runs, while the matched DLoss-minus-CE differences remain positive. The ablations also used A100 hardware, whereas the main experiments used GH200 hardware; we therefore do not attribute all between-run variation to random seed alone.

Across the three matched pairs, mean improvements are +4.85 GREEN points and +3.66 F1-RG points. All nine evaluated metrics improve in every pair. Appendix F now reports the absolute scores, within-seed differences, bootstrap intervals, and the limitations of inference from three seed pairs.

Table 1 reports absolute-score differences, while Table 3 reports relative percentage changes. The +1.54% F1-RG gain in Table 3 corresponds to +0.86 points, from 55.55 to 56.41. Appendix F.2 identifies the matched baselines and reports the per-run scores.

Q2: The diagnostic loss coefficients are surprisingly small (Table 5). What is the contribution of Diagnostic Loss to the total loss?

Appendix E.1 reports the magnitude of the weighted diagnostic term, including λ+\lambda_+ and λ−\lambda_-, relative to cross-entropy. Averaged over training up to the selected checkpoints, its absolute magnitude ranges from 0.2% to 4.8% of the CE loss across backbones. We also measured the gradient norms of the two terms on training batches at the selected checkpoints, using the same parameters for both terms.

BackboneWeighted DLoss magnitude relative to CEWeighted DLoss gradient norm relative to CE
fVLM0.2%2.8%
BTB3D0.6%1.6%
RadFM4.8%18.5%

Both columns use cross-entropy as the denominator. The loss ratios summarize training up to the selected checkpoint, and the gradient ratios describe measurements at that checkpoint.

Reviewer_MuwpOpenReview ↗
Rating
2(−1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Diagnostic Loss (DLoss), an auxiliary training objective designed to improve diagnostic faithfulness in radiology report generation. In addition to conventional cross-entropy training, DLoss introduces asymmetric supervision over diagnostic concepts. It uses noisy-OR pooling to encourage the inclusion of diagnoses supported by the reference report and temperature-controlled log-sum-exp pooling to suppress high probabilities assigned to unsupported diagnostic alternatives. The objective can be incorporated into existing medical vision-language models without modifying their architectures or inference procedures. The paper evaluates DLoss across multiple 2D and 3D models, reporting improvements in diagnostic evaluation metrics on both in-distribution and external test cohorts.

Critical strengths
  1. Clinically meaningful problem formulation and targeted objective design.

The paper addresses an important limitation of conventional report-generation objectives: linguistically fluent reports may still omit relevant diagnoses or introduce unsupported findings. The asymmetric treatment of supported and unsupported diagnostic concepts provides a well-motivated approach to reducing these errors. This is a meaningful contribution because diagnostic faithfulness is more consequential in clinical applications than surface-level linguistic similarity alone.

  1. Practical applicability and encouraging cross-model results.

DLoss can be integrated into existing decoder training without changing the underlying architectures or introducing additional inference-time computation. The evaluation across multiple medical vision-language models and an external cohort provides encouraging evidence of its applicability. The reported training overhead is relatively modest, and the nine-seed analysis for BTB3D on Merlin further supports the stability of the improvements for that setting.

Critical weaknesses
  1. Uncertainty in the constructed diagnostic supervision targets.

The effectiveness of DLoss depends heavily on the accuracy of the supported and unsupported diagnostic concept sets, which are constructed through report parsing, semantic tagging, and synonym consolidation. However, a finding that is visible in an image may be omitted from the reference report. Consequently, treating unmentioned concepts as unsupported could introduce incorrect supervision. The current evidence does not sufficiently establish the reliability of these constructed targets. A representative manual audit and sensitivity analysis under target-label noise are important for assessing whether the proposed objective consistently promotes clinically valid predictions. This limitation also restricts the strength of conclusions about clinical reliability.

  1. Insufficient evidence isolating the benefits of asymmetric pooling.

Although the paper includes component and pooling ablations, the comparison with simpler diagnostic-aware training objectives appears incomplete. In particular, matched baselines using diagnostic-token-weighted cross-entropy or negative-token unlikelihood training would help determine whether the reported gains arise specifically from asymmetric temporal pooling rather than additional diagnostic supervision. Furthermore, most model and cohort comparisons appear to rely on single training runs, while the repeated-seed analysis is limited to one model and dataset. These issues weaken the evidence for the method's distinctive contribution and the consistency of its improvements.

Questions for authors

Q1. Reliability of diagnostic supervision: How frequently do the automated concept extraction and consolidation procedures produce incorrect positive or negative targets? Can the authors provide a manual audit of representative cases, particularly findings that are visible in the images but omitted from the reference reports? How sensitive is DLoss to such target-label noise?

Q2. Comparison with simpler objectives: How does DLoss compare with diagnostic-token-weighted cross-entropy and negative-token unlikelihood training under matched model architectures, training budgets, and supervision vocabularies? This comparison would help clarify the specific contribution of the proposed asymmetric pooling operators.

Q3. Independent clinical validity: Can the authors provide a blinded radiologist assessment or a clinically grounded error analysis distinguishing false positives, false negatives, anatomical localization errors, and severity errors? Since GREEN and RadGraph-F1 rely on automated reference-based evaluation, additional evidence would strengthen the claimed improvements in diagnostic faithfulness.

Q4. Robustness of the improvements: Can the authors provide repeated-run results or paired uncertainty estimates for at least one additional backbone and the external cohort? This is particularly relevant where the reported improvements over the corresponding baselines are relatively small.

11 comments
AuthorsOpenReview ↗

Response to Reviewer Muwp (1/4)

W1, Q1, Q3: The effectiveness of DLoss depends heavily on the accuracy of the supported and unsupported diagnostic concept sets [...] However, a finding that is visible in an image may be omitted from the reference report. Consequently, treating unmentioned concepts as unsupported could introduce incorrect supervision. [...] Can the authors provide a manual audit of representative cases, particularly findings that are visible in the images but omitted from the reference reports? How sensitive is DLoss to such target-label noise? [...] Can the authors provide a blinded radiologist assessment or a clinically grounded error analysis distinguishing false positives, false negatives, anatomical localization errors, and severity errors?

Target construction and report omissions (W1). DLoss does not treat every unmentioned diagnosis as unsupported. For an organ labeled abnormal, negative supervision is restricted to normal wording (e.g., “normal,” “unremarkable”); other unmentioned diagnoses are not explicitly assigned as negative targets. For an organ labeled normal, a fixed organ-specific list of abnormal expressions is pooled into one unsupported target. Organs omitted from the report are treated as normal during preprocessing. This last step is an assumption and can produce incorrect negative supervision when an omitted organ contains pathology. An explicit normal statement is stronger report evidence than omission, but is also fallible. Although CE trains the same reference wording, DLoss adds a distinct penalty and can reinforce an incorrect normality assumption. We clarified this rule and its assumptions in Section 3.1 (“Diagnostic target construction” paragraph), with the detailed selection procedure in Appendix C (“Diagnostic supervision construction”), and expanded the limitations in Section 5.

Manual audit (W1, Q1). One author manually compared the automated organ labels and diagnostic targets with clinician-prepared reference reports for 100 organ-level sections from the external test set in Table 1. The sample covered all 11 organ groups and included 50 sections labeled normal and 50 labeled abnormal. Normal/abnormal labels agreed with the reports in 97/100 sections. Every assigned diagnostic target agreed with the report in 95/100 sections, and no report finding was missing from the targets in 90/100 sections. These are section-level agreement and completeness measures, not per-target positive and negative error rates. The audit evaluates extraction and consolidation fidelity to the reports; it does not measure findings visible in the images but omitted from those reports, or establish the noise rate in routine training reports. We added the protocol, results table, and scope to Appendix C.1 (“Manual audit of report-derived diagnostic targets”). We also clarified the provenance of the external reference reports in Appendix A.2 (“Test set,” In-house paragraph).

Sensitivity to target noise (W1, Q1). All reported experiments use targets extracted from routine clinical reports. If extraction misses a diagnosis but still labels the organ abnormal, that diagnosis is not explicitly assigned as a negative target; CE continues to supervise its wording if it appears in the reference. This reduces one source of incorrect negative supervision. On the external set with clinician-prepared research references, DLoss improves GREEN by 4.28 points on average across six backbones, compared with 4.05 points on Merlin, and all 36 external model–metric comparisons improve. These results support generalization under the existing supervision. They do not isolate sensitivity to target-label noise or rule out learning report-derived errors. We do not report a controlled target-corruption experiment here, so this sensitivity remains unquantified. We added this distinction and the remaining failure modes to Appendix C.2 (“Target noise and independent clinical validity”) and the limitations paragraph in Section 5.

Independent clinical validity (Q3). The target audit is a report-based supervision audit, not a blinded radiologist assessment of generated reports against CT images. GREEN and RadGraph-F1 remain automated, reference-based measures, and the qualitative examples do not provide independently adjudicated rates of false positives, false negatives, anatomical localization errors, or severity errors. We therefore limit our conclusion to improved reference-based performance. A blinded image-based assessment separating these error types remains necessary to substantiate independent clinical validity; no such assessment is reported here. We revised Section 4.1 (“Evaluation metrics” paragraph), Section 4.2 (“Qualitative results” paragraph), and Section 5 (conclusion and limitations), and added the discussion in Appendix C.2.

AuthorsOpenReview ↗

Response to Reviewer Muwp (2/4)

W2: Insufficient evidence isolating the benefits of asymmetric pooling. Although the paper includes component and pooling ablations, the comparison with simpler diagnostic-aware training objectives appears incomplete. In particular, matched baselines using diagnostic-token-weighted cross-entropy or negative-token unlikelihood training would help determine whether the reported gains arise specifically from asymmetric temporal pooling rather than additional diagnostic supervision. Furthermore, most model and cohort comparisons appear to rely on single training runs, while the repeated-seed analysis is limited to one model and dataset. These issues weaken the evidence for the method's distinctive contribution and the consistency of its improvements.

We address the attribution to asymmetric pooling separately from the consistency of the improvements.

Benefits of asymmetric pooling. The fVLM ablations retain CE and both diagnostic target sets while replacing supported-target noisy-OR with max or mean pooling, or unsupported-target log-sum-exp with mean pooling. Each replacement lowers BLEU-4, GREEN, and F1-RG relative to full DLoss. Removing either diagnostic component also lowers these metrics. This provides evidence for the chosen pooling operators within the matched seed-13 experiment.

Please see the next comment for our joint response to W2 and Q4.

AuthorsOpenReview ↗

Response to Reviewer Muwp (3/4)

W2, Q4: Insufficient evidence isolating the benefits of asymmetric pooling. [...] comparison with simpler diagnostic-aware training objectives appears incomplete [...] matched baselines using diagnostic-token-weighted cross-entropy or negative-token unlikelihood training would help determine whether the reported gains arise specifically from asymmetric temporal pooling [...] most model and cohort comparisons appear to rely on single training runs [...] Can the authors provide repeated-run results or paired uncertainty estimates for at least one additional backbone and the external cohort?

Additional-backbone robustness: fVLM (W2, Q4). We report three matched CE–DLoss pairs on the full Merlin test set. Within each pair, CE and DLoss share the seed and training setup. All six metrics below improve in all three pairs, and both reported 95% intervals lie above zero. With only three pairs, the intervals describe the observed runs; moreover, hardware differs across runs, so this analysis does not isolate seed-only variability. The protocol, paired effects, and absolute scores are in Appendix F.2 (“fVLM run comparisons and three-seed analysis”).

All tables below use displayed point units: native score ×100, except CIDEr ×10. In all bootstrap analyses below, METEOR is the mean row-level score, whereas Table 1 uses corpus METEOR. The METEOR effects and intervals therefore concern the row-mean estimator, not the corpus METEOR effects in Table 1.

MetricMean effectSeed bootstrapSeed–study bootstrap
BLEU-4+4.02[+0.36, +7.04][+0.39, +7.03]
METEOR (row mean)+3.29[+0.82, +6.18][+0.85, +6.17]
ROUGE-L+3.90[+0.78, +6.70][+0.81, +6.69]
CIDEr+3.27[+0.36, +6.56][+0.39, +6.54]
GREEN+4.85[+0.31, +9.00][+0.34, +8.99]
F1-RG+3.66[+0.86, +6.56][+0.89, +6.55]

External-cohort robustness: BTB3D (W2, Q4). We evaluated the same nine matched BTB3D pairs on the external cohort. Seed-bootstrap intervals resample matched CE–DLoss training pairs; seed–study intervals additionally resample test studies while preserving pairing and organ-row clustering. The existing Merlin analysis gives a mean GREEN gain of 4.6 points (95% seed–study interval [2.0, 7.3]). The external results are:

MetricMean effectSeed bootstrapSeed–study bootstrap
BLEU-4+2.22[−0.73, +5.16][−0.68, +5.15]
METEOR (row mean)+2.11[−0.32, +4.24][−0.36, +4.25]
ROUGE-L+1.88[−0.38, +3.92][−0.40, +3.97]
CIDEr+1.33[−0.90, +3.23][−0.92, +3.26]
GREEN+1.53[+0.31, +2.54][+0.29, +2.54]
F1-RG+2.11[−0.05, +4.11][−0.04, +4.13]

All six external mean effects are positive. GREEN improves in seven of nine seed pairs, and both GREEN intervals lie above zero. The other five metrics have intervals that include zero, so we do not claim consistent improvement across every metric. We added the external results and their interpretation in Appendix F.4 (“BTB3D repeated-run results on the external cohort”), and summarized them in Section 4.2 (“Quantitative results” paragraph).

Uncertainty for small gains: Merlin and RadFM (W2, Q4). For the fixed checkpoint pairs underlying Table 1, we report paired test-study bootstrap intervals on the Merlin (M) and external (E) cohorts. These intervals quantify test-study sampling uncertainty conditional on those checkpoints; they do not measure training-seed variability.

MetricMerlin MMerlin ERadFM MRadFM E
BLEU[−0.06, +0.08][+0.07, +0.12][−0.001, +0.12][+0.11, +0.31]
ROUGE-L[+0.002, +0.09][+0.20, +0.33][+0.11, +0.32][+0.33, +0.63]
METEOR (row mean)[−0.02, +0.03][+0.05, +0.10][+0.02, +0.10][+0.12, +0.24]
CIDEr[+0.14, +0.22][+0.45, +0.55][+0.04, +0.18][+0.14, +0.37]
GREEN[+0.71, +0.94][+1.05, +1.44][+0.40, +0.58][+0.71, +1.00]
F1-RG[−0.07, +0.04][+0.28, +0.42][+0.13, +0.34][+0.27, +0.57]

On the external cohort, all six intervals lie above zero for both backbones. Some internal-cohort intervals include zero. We added these results and the distinction from repeated training in Appendix F.5 (“Study-bootstrap uncertainty for Merlin and RadFM”), with corresponding clarifications in Sections 4.2 and 5.

All GREEN intervals in the analyses reported here lie above zero. This supports a positive effect in these specific settings, while the limited seed coverage, three-pair fVLM analysis, and external intervals spanning zero for other BTB3D metrics limit broader robustness claims. Repeated-run evidence addresses consistency of the full objective; it does not replace the additional objective baselines requested in W2.

Reviewer_HbGoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Diagnostic Loss (DLoss), an auxiliary training objective for improving the diagnostic accuracy of radiology report generation. The method divides diagnostic concepts in the reference report into Supported and Unsupported sets, and uses Noisy-OR Pooling and Log-Sum-Exp Pooling, respectively, to aggregate the decoder’s predicted probabilities at different token positions, thereby encouraging the model to retain correct diagnoses and reduce incorrect diagnoses.

The authors conduct experiments on multiple 3D Medical Vision-Language Models. The results show that DLoss generally improves GREEN, RadGraph-F1, and traditional text generation metrics on both internal and external CT datasets, without increasing inference overhead.

Critical strengths

This paper proposes a training objective with clear clinical motivation targeting diagnostic omissions and incorrect diagnoses in radiology report generation.

DLoss is simple and efficient. It requires no modification to existing model architectures and adds no computational overhead at inference time, offering good generality and practical value.

The experiments cover six MedVLMs, internal and external test sets, and include component ablations, computational cost analysis, and random-seed robustness experiments. The overall experimental evaluation is relatively thorough, and the results are fairly consistent.

Critical weaknesses

The description and analysis of Diagnostic Target Construction are insufficient

The effectiveness of DLoss depends heavily on the construction of the Supported and Unsupported Diagnostic Target Sets. Although the authors present the overall construction pipeline in Section 3.1 and Appendix C, several key questions require further clarification:

1.1 Definition of diagnostic concepts and synonyms. Is there a specific lookup table, or are they generated dynamically according to other methods?

1.2 Selection strategy for Unsupported Targets. The authors do not sufficiently explain the specific source of Unsupported Targets. Are they selected from a predefined organ-specific diagnostic vocabulary, or constructed through other reports or specific rules? In addition, could this process introduce class imbalance or incorrect negative supervision? For example, for some organs, the unsupported targets may be substantially larger/longer than those for other organs.

1.3 Organ-level performance and Diagnostic Target Distribution. Different organs may have different numbers of diagnostic categories, frequencies of occurrence, and vocabulary sizes, but corresponding statistics and performance analyses are currently lacking. In particular, Eq. (4) directly sums token probabilities in the diagnostic vocabulary without normalization by vocabulary size. Could a larger vocabulary set (D+) make the loss easier to satisfy and thus affect the strength of supervision? And is this difference related to the performance improvements observed for different organs?

Questions for authors

Same as weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer HbGo (1/2)

Thank you for the constructive review, which has made our paper stronger.

W1: Definition of diagnostic concepts and synonyms. Is there a specific lookup table, or are they generated dynamically according to other methods?

Thank you for highlighting this ambiguity. The diagnostic concepts and their synonyms correspond to a fixed, organ-specific lookup table, constructed once, offline, from the training reports. Neither the targets nor their lexical realizations are generated or updated during report-generation training.

We construct the table in three steps:

  1. Semantic tagging: An LLM assigns structured labels to each organ-level finding, including normality status, broad diagnostic category, and specific diagnoses or findings. Manually defined seed targets and organ-specific examples guide this step; additional targets may be proposed during this offline stage.
  2. Target consolidation: An LLM merges labels with the same clinical meaning into a single diagnostic target, defining the controlled target space G\mathcal{G} in Section 3.1.
  3. Lexical collection: For each target, we collect recurrent diagnostic words and short phrases from the training reports assigned to it, for example, gallstones and cholelithiasis.

During training, each organ-level example uses its stored semantic labels and this fixed table to obtain its diagnostic target sets. Each decoder’s existing tokenizer maps the associated lexical realizations to vocabulary indices.

We clarify the fixed nature of the table in Section 3.1 and describe its offline construction in Appendix C. “Diagnostic supervision construction.”

W2: Selection strategy for Unsupported Targets. The authors do not sufficiently explain the specific source of Unsupported Targets. Are they selected from a predefined organ-specific diagnostic vocabulary, or constructed through other reports or specific rules? In addition, could this process introduce class imbalance or incorrect negative supervision? For example, for some organs, the unsupported targets may be substantially larger/longer than those for other organs.

Unsupported targets come from the fixed, organ-specific lookup table described in our response to W1. They are selected by a predefined rule, rather than sampled from other reports or generated during training:

  1. Abnormal organ: Negative supervision is restricted to normal wording, such as normal and unremarkable. Other diagnoses not mentioned for that organ are not assigned as negative targets, because additional abnormalities may coexist without being documented.
  2. Normal organ: Negative supervision uses a fixed list of abnormal expressions for that organ. These expressions are pooled into one unsupported target with a single vocabulary-index set, rather than introducing a separate negative loss term for each abnormal diagnosis.

Organs not mentioned in the report are treated as normal during preprocessing. We now explicitly identify this as an annotation assumption: it can introduce incorrect negative supervision when an omitted organ contains unreported pathology. Explicitly negated concepts are excluded from target construction.

Regarding imbalance, pooling the abnormal expressions into one target means that a larger organ-specific diagnostic inventory does not produce more target-level negative loss terms. However, this does not guarantee equal supervision strength: vocabulary size and tokenizer fragmentation can affect the aggregated probability mass, and the loss does not explicitly rebalance organ or normal/abnormal frequencies. We have clarified these distinctions rather than claiming that the selection rule eliminates imbalance or incorrect negatives.

We have revised Section 3.1 and the “Diagnostic supervision construction” Appendix C to specify the experimental selection rule and pooling procedure. We also added the annotation limitation and clarified that the schematics illustrate a stricter target assignment than the one used in the reported experiments.

AuthorsOpenReview ↗

Response to Reviewer HbGo (2/2)

W3: Organ-level performance and Diagnostic Target Distribution. Different organs may have different numbers of diagnostic categories, frequencies of occurrence, and vocabulary sizes, but corresponding statistics and performance analyses are currently lacking. In particular, Eq. (4) directly sums token probabilities in the diagnostic vocabulary without normalization by vocabulary size. Could a larger vocabulary set (D+) make the loss easier to satisfy and thus affect the strength of supervision? And is this difference related to the performance improvements observed for different organs?

A larger diagnostic vocabulary can make the supported-target loss easier to satisfy: for fixed decoder probabilities, adding vocabulary IDs can increase the mass in Eq. (4). We acknowledge this dependence.

Eq. (4) intentionally measures total probability assigned to the target’s token set. Distinct vocabulary IDs are mutually exclusive outcomes of one softmax, so their sum is the probability that the next token belongs to that set and lies in [0,1][0,1]. This measures token-level lexical evidence, rather than the probability of generating a complete diagnostic phrase. Dividing by vocabulary size would instead compute average probability per vocabulary item, changing the quantity being supervised.

We constrain lexical-list variation by capping each diagnostic concept at 24 phrases before unsupported-target pooling. Across the 11 organs, supported concepts in abnormal findings have organ-level mean sizes of 17–23 phrases and 17–28 unique vocabulary IDs under BTB3D’s Llama-3.1 tokenizer. This limits variation but does not make supervision invariant to vocabulary size. Separately, Eq. (8) averages over targets, preventing the loss from scaling directly with their number.

We have added an organ-level analysis using the nine matched BTB3D seed pairs described in Appendix F. Each organ’s GREEN gain is averaged over those nine pairs. Across the 11 organ-level observations, Spearman correlations with mean GREEN gain are ρ=−0.18\rho=-0.18 for phrase counts and ρ=−0.26\rho=-0.26 for unique token-ID counts. Mean GREEN improves in 10 of the 11 organs. These weak negative associations are descriptive and do not rule out vocabulary-size effects.

The revised manuscript includes these distinctions in Sections 3.1–3.2, the lexical-list construction details in Appendix C, and the summary statistics and size–performance analysis in Appendix F.

Please let us know if you have any follow-up questions. We are happy to clarify. If you think our response addresses your concerns, please consider raising your score.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

As fewer than two days remain before the Author–Reviewer discussion period closes, we would greatly appreciate it if you could take a look at our rebuttal to confirm whether we have addressed all of your concerns. In such a case, we would be grateful if you could consider raising your score.

Thank you again for your time and thoughtful contributions to the process.

Best regards,

Authors

Comments on the paper

1 comment
Area_Chair_2thFOpenReview ↗

Author-Reviewer Discussion

Dear Reviewers:

Thank you for your thoughtful reviews. The authors have responded to your reviews. Please check their response and reach out to them if you have any further questions. Please act as soon as possible, as the discussion period with the authors closes on Nov 18.

Best regards,

  • AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.