acceptodds
Under review as a conference paper at ICLR 2027

Beyond Average Risk: Diagnosis-Priority Robust Learning for Structured Radiology Report Generation

Abstract

Structured generation models are commonly optimized for average performance, although low frequency semantic events may carry disproportionate importance in high stakes domains. We study this problem in structured radiology report generation, where strong aggregate text and semantic metrics can coexist with degraded reliability on rare diagnostic findings. We formulate this failure as a semantic tail robustness problem and introduce diagnosis priority robust learning, which explicitly increases optimization pressure on diagnostically important long tail semantics and underperforming groups. Our framework uses reference side semantic representations to characterize abnormality and diagnostic rarity while separating average generation quality from worst group and rare-finding reliability. Across single-source and cross-source radiology report generation settings, we find that models with similar aggregate quality can exhibit substantially different semantic-tail behavior, and that robust training shifts the operating point between average fidelity and diagnosis priority reliability. Cross source experiments further indicate that this robustness problem becomes especially relevant under heterogeneous data distributions. We evaluate the approach using standard generation metrics together with semantic, worst group, rare recall, and false negative measures. These results motivate structured generation objectives that explicitly account for rare diagnostic semantics rather than relying solely on average sequence level performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
structured radiology report generation, robust learning, long-tail learning, semantic robustness, structured prediction, clinical natural language generation, medical artificial intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cgqbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces DFR (diagnosis-first robust training) for radiology report generation, focusing on rare and abnormal diagnosis (minority groups). This is done by combining diagnosis-priority sample weighting and group-aware reweighting. DFR uses semantic labels from reports and divide them into four groups. The framework is evaluated using T5-based models on CheXpert benchmark. Results show that DFR improves the performance on worst group, but degrades a bit on the performance of G11 and rare groups.

Critical strengths
  1. The paper studies an important problem in rediology report generation that strong average generation metrics might obscure poor performance on rare or abnormal conditions. This is also true for other medical domain and modalities.

  2. The paper evaluates several dimensions of robustness, including group-wise F1's, rather than only relying on general text generation metrics. The component level comparison, multi-seeds, and cross-source experiments provide empirical observations about the trade-offs between different robustness objectives.

Critical weaknesses
  1. Both the diagnosis-priority sample weighting and group-aware reweighting are well-established methods in long-tail learning and group-wise robustness experiments. The methodology novelty is limit in this case. More importantly, the experiments are not sufficient to justify the robustness of DFR. There is no comparisons towards other established baselines, such as Group DRO and JTT (even cited in the paper). The main table is basically comparing ablations (diagnosis-priority sample weighting alone; or group-aware reweighting alone) instead of baselines. Also, the experiments only use T5-Base and Flan-T5 backbone. It is hard to tell whether the improvements can be adopted to more recent models. Although the authors evaluate Path 1 and Path 2, and state this to be the cross-source validation, the paper does not include a direct cross-dataset generalization test.

  2. My other concern is that the results do not support the claimed benefits of the DFR. Even if the worst-group F1 increases with the help of DFR, the performance on Group 11 and rare recall remain unchanged or even worse. Even though the authors state this trade-off in their paper, it is still unclear why the worst-group is prioritized over Group 11 and rare. Since the paper is mainly about the minority findings, Group 11 should also be prioritized. In addition, the submitted pdf file still contains the default template title, "FORMATTING INSTRUCTIONS FOR ICLR 2027 CONFERENCE SUBMISSIONS," rather than the actual paper title. And the manuscript does not include the "AI Use Statement" section required by ICLR 2027. The authors should correct these issues and ensure compliance with the conference's submission requirements.

Questions for authors
  1. Could the authors justify the methodological contirbution of DFR beyond combining existing reweighting methods?
  2. Could the authors compare to other robustness learning methods such as Group DRO, JTT, or class-balanced reweighting?
  3. Since Group 11 is under the primary diagnosis-priority group, could the authors further justify the effectiveness of full DFR for diagnosis-priority reliability? How should the trade-offs in Table. 1 be interpreted?
  4. Could the authors extend the evaluation to more recent or stronger language models beyond T5-Base and Flan-T5?
  5. Could the authors provide evaluation across additional datasets to further justify the effectiveness of DFR as current two datasets are primarily focusing on chest radiology report? Additionally, could the author provide the direct cross-dataset generalization experiments, such as training on CheXpert-Plus and evaluating on the MIMIC-CXR-derived dataset, or vice verse?
Reviewer_QxccOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates semantic-tail robustness in structured radiology report generation, asking whether favorable aggregate generation scores imply reliable recovery of uncommon or abnormal findings. The authors propose Diagnosis-First Robust Training (DFR), combining diagnosis-priority example weights and group-aware reweighting. Reference-side semantic labels define abnormality and rarity, producing four nominal groups. The abnormal-and-rare slice (Group 11) is designated as the primary diagnosis-priority subset. Worst-group F1 is calculated over Groups 00, 10 and 11 because Group 01 is degenerate. Experiments cover a CheXpert-Plus-derived primary benchmark, MIMIC-CXR-derived second-source structured reports, and pooled training. Reported metrics include BLEU, ROUGE-L, BERTScore, semantic F1, worst-group F1, Group 11 F1, rare recall and rare false-negative rate. On the primary benchmark, Full DFR improves worst-group F1 from 0.781 to 0.820, but Group 11 F1 remains 0.837 and rare recall decreases from 0.735 to 0.723. In the pooled setting, worst-group F1 improves from 0.836 to 0.867 and rare recall from 0.733 to 0.757, whereas Group 11 F1 decreases from 0.879 to 0.867. The paper identifies a meaningful evaluation problem and honestly reports trade-offs. However, it does not establish a consistently beneficial change in the designated diagnosis-priority outcomes, and additional methodological, baseline and validation details are needed.

Critical strengths

S1. An important reliability question not adequately captured by aggregate report-generation scores. The paper draws attention to a clinically relevant limitation of BLEU, ROUGE-L, BERTScore and aggregate semantic scores: minority diagnostic findings may behave differently from the average case. Separating worst-group F1, Group 11 F1 and rare recall exposes error patterns that would otherwise be hidden in corpus-level scores. This distinction is relevant even when the proposed training objective is not uniformly superior. S2. Transparent analysis of competing objectives and an initial cross-source evaluation. The authors compare the baseline with diagnosis-priority-only, group-only and combined weighting, providing a starting point for identifying component effects. They additionally report multi-seed primary results and examine a second-source and a pooled setting. Importantly, the paper acknowledges that Full DFR improves worst-group F1 without dominating Group 11 F1 or rare recall. This operating-point interpretation is more credible than a claim of universal improvement.

Critical weaknesses

W1. Improvements in worst-group F1 do not establish improved reliability for the paper's designated abnormal-and-rare priority group. Evidence: Section 3.2, Tables 1–4, and Sections 4.2–4.5. The paper defines Group 11 (abnormal and rare) as the primary diagnosis-priority group, but its main positive result in the primary setting concerns worst-group F1. Table 1 reports an increase from 0.781 to 0.820, while Group 11 F1 remains unchanged at 0.837, rare recall decreases from 0.735 to 0.723, and rare false-negative rate increases from 0.265 to 0.277. The group-only variant performs better on the designated diagnostic outcomes (Group 11 F1 0.840; rare recall 0.738). These results are not mathematically inconsistent: the lowest-scoring group need not be Group 11. The authors also acknowledge the trade-off. Nonetheless, the paper does not identify which group is the worst performer under each configuration, or explain why an operating point with lower rare recall is clinically preferable in a diagnosis-first setting. The improvement could be driven by another subgroup. The cross-source tables reinforce the concern. In the pooled setting, Full DFR increases worst-group F1 from 0.836 to 0.867 and rare recall from 0.733 to 0.757, but Group 11 F1 decreases from 0.879 to 0.867. In the MIMIC-derived second-source setting, both worst-group F1 (0.925 to 0.921) and Group 11 F1 (0.925 to 0.921) decline. The test subsets are also small: 127 Group 11 cases in the primary setting, 98 in the second-source setting, and 91 in the pooled setting (Table 6). Three-seed mean ± SD on the primary benchmark is informative but does not replace paired uncertainty analysis at the case or patient level; the pooled results likewise require confidence intervals or repeated-seed reporting. Please report per-group F1 for Groups 00, 10 and 11, the identity of the worst group for every method, paired confidence intervals, and specific missed/recovered rare diagnoses. Without these analyses, the evidence supports a change in robustness operating point, not consistent improvement of rare-abnormal diagnostic reliability.

W2. DFR is insufficiently specified and lacks controlled comparisons needed to establish its distinct contribution; the semantic-slicing protocol requires independent validation. Evidence: Sections 3.2–3.5, Tables 5–6, and Appendices A and C. The instance-level weight is written as w_i = 1 + alpha phi(a_i,r_i), but phi is described rather than fully defined. Likewise, the group weights q_g are said to emphasize underperforming groups, without a complete specification of how the weights are calculated, normalized or updated. Providing alpha and eta in Appendix C does not by itself specify the optimization algorithm. The main experimental comparisons involve the baseline and internal ablations, rather than established group-robust alternatives such as Group DRO or conventional class-balanced weighting, although related approaches are discussed in Section 2.3. Thus the experiments do not establish whether the combination outperforms existing robust optimization under a matched training budget. The group labels and semantic scores also rely on the same automatic reference-side semantic extraction framework. Although reference-side assignment avoids defining groups from the model predictions, errors in the extractor may affect both grouping and evaluation. No independent expert-validated estimate of accuracy on the rare abnormal subset is reported. Finally, the stated training-support-based threshold procedure needs clarification. At tau = 2000, Table 5 reports Group 11 support of 127, matching the Path1 test Group 11 count in Table 6 rather than its train count of 188. This discrepancy does not establish test leakage, but the data partition behind the threshold-support table should be documented. Frequency-based rarity is not equivalent to clinical severity, which the authors appropriately acknowledge. A precise training algorithm, group-weight schedule, comparisons with robust-learning baselines, sensitivity to alternative thresholds, and an independent semantic-label audit would materially strengthen the paper.

Questions for authors

Q1. Which group determines worst-group F1 under each method and dataset setting? Please report F1 for Groups 00, 10 and 11 separately and identify the group achieving the minimum in each condition. Why is Full DFR preferable to group-only for diagnosis-priority reliability when the latter achieves higher Group 11 F1 and rare recall on the primary benchmark? Q2. What are the exact DFR weighting and optimization rules? Please give the explicit function phi(a,r), specify whether group weights q_g are static or dynamically updated, and detail normalization, update frequency and hyperparameter selection. How does DFR compare with Group DRO and conventional class-balanced weighting under the same training budget? Q3. How robust is the result to semantic-extractor errors and the definition of rarity? Can the authors provide independent expert validation of semantic labels, especially rare abnormal findings? Please clarify why the selected-threshold support count is 127 in Table 5 while Table 6 reports 188 training Group 11 instances for Path1, and show sensitivity to reasonable thresholds and grouping alternatives. Q4. Are cross-source improvements statistically reliable and distinguishable from the effect of dataset pooling? Please provide repeated-run results and paired confidence intervals for the pooled and second-source conditions, as well as group-level error breakdowns. Could controlled source mixtures clarify whether performance changes arise from DFR weighting or from data composition? Q5. What clinical reliability claim is justified by report-to-Impression restructuring? Section 3.1 specifies free-text radiology reports as inputs and structured Impressions as targets, rather than independent diagnosis from images. Could the authors clarify the deployment scenario and distinguish faithful preservation of existing diagnoses from independent abnormality detection? Expert evaluation of consequential omissions, unsupported findings and uncertainty changes would help contextualize these metrics.

Reviewer_tHUUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether standard aggregate metrics reliably capture model performance on rare and abnormal findings in structured radiology report generation. Framing the task as transforming narrative reports into structured impressions, the authors categorize samples by abnormality and rarity via semantic labels. They introduce Diagnosis-First Robust Training (DFR), blending diagnosis-prioritized loss weighting with group-aware reweighting. Benchmarked on T5-Base across CheXpert Plus and MIMIC-CXR-derived data, DFR yields marked improvements in worst-group F1; however, its efficacy on long-tail findings is mixed. Consequently, the study characterizes these outcomes as fundamental trade-offs between overall quality, worst-group robustness, and rare-finding sensitivity.

Critical strengths
  • The evaluation concern is well motivated. Aggregate text and semantic metrics can conceal failures on minority subsets. Separately reporting worst-group performance, abnormal-and-rare performance, and rare-finding recall provides a more informative evaluation.
  • The experiments include useful initial controls. Component ablations, multiple seeds on the primary setting, and evaluations across two data sources provide preliminary evidence about how reweighting changes model behavior.
Critical weaknesses
  • The task definition and practical motivation remain insufficiently established. It is unclear whether the input contains Findings only, the original Impression, or the complete report.
  • The comparisons do not establish relevance to the current state of report structuring. The experiments mainly compare weighting variants of T5-Base, with an additional T5-Flan baseline. They omit both stronger lightweight models and contemporary small language models. A particularly relevant predecessor, Structuring Radiology Reports: Challenging LLMs with Lightweight Models (Moll et al., EMNLP 2025), already compares lightweight encoder–decoder models with 1B–70B language models on this task. Small models remain potentially useful, but the present paper establishes neither a quality–efficiency advantage nor whether its identified failure modes persist in stronger systems.
  • The methodological contribution is unclear and underspecified. The priority weight depends only on abnormality and rarity, which also define group membership. It is therefore constant within each group and can be absorbed into the group weight in the stated objective. Distinct optimization dynamics may still arise, but the paper does not sufficiently specify the priority function, group-weight update, normalization, or DFR-A/B configurations to establish this distinction or enable reproduction.
  • The improvements do not consistently support the diagnosis-priority motivation. On the primary setting, full DFR improves worst-group F1 from 0.781 to 0.820, while rare recall decreases from 0.735 to 0.723. The pooled setting improves worst-group F1 and rare recall, but G11 F1 decreases. These results demonstrate trade-offs, but do not establish why the selected operating point is preferable for preserving rare abnormalities.
Questions for authors
  1. What exactly are the inputs and targets in each setting?
  2. How does DFR compare with existing report-structuring systems?
  3. What makes the two weighting components functionally distinct?
  4. How are the metrics computed? Please specify micro/macro/sample/weighted aggregation, treatment of negated and uncertain findings, rare-recall denominators, and empty-label cases.
  5. How was the rarity threshold selected?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.