- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces DFR (diagnosis-first robust training) for radiology report generation, focusing on rare and abnormal diagnosis (minority groups). This is done by combining diagnosis-priority sample weighting and group-aware reweighting. DFR uses semantic labels from reports and divide them into four groups. The framework is evaluated using T5-based models on CheXpert benchmark. Results show that DFR improves the performance on worst group, but degrades a bit on the performance of G11 and rare groups.
Critical strengths
-
The paper studies an important problem in rediology report generation that strong average generation metrics might obscure poor performance on rare or abnormal conditions. This is also true for other medical domain and modalities.
-
The paper evaluates several dimensions of robustness, including group-wise F1's, rather than only relying on general text generation metrics. The component level comparison, multi-seeds, and cross-source experiments provide empirical observations about the trade-offs between different robustness objectives.
Critical weaknesses
-
Both the diagnosis-priority sample weighting and group-aware reweighting are well-established methods in long-tail learning and group-wise robustness experiments. The methodology novelty is limit in this case. More importantly, the experiments are not sufficient to justify the robustness of DFR. There is no comparisons towards other established baselines, such as Group DRO and JTT (even cited in the paper). The main table is basically comparing ablations (diagnosis-priority sample weighting alone; or group-aware reweighting alone) instead of baselines. Also, the experiments only use T5-Base and Flan-T5 backbone. It is hard to tell whether the improvements can be adopted to more recent models. Although the authors evaluate Path 1 and Path 2, and state this to be the cross-source validation, the paper does not include a direct cross-dataset generalization test.
-
My other concern is that the results do not support the claimed benefits of the DFR. Even if the worst-group F1 increases with the help of DFR, the performance on Group 11 and rare recall remain unchanged or even worse. Even though the authors state this trade-off in their paper, it is still unclear why the worst-group is prioritized over Group 11 and rare. Since the paper is mainly about the minority findings, Group 11 should also be prioritized. In addition, the submitted pdf file still contains the default template title, "FORMATTING INSTRUCTIONS FOR ICLR 2027 CONFERENCE SUBMISSIONS," rather than the actual paper title. And the manuscript does not include the "AI Use Statement" section required by ICLR 2027. The authors should correct these issues and ensure compliance with the conference's submission requirements.
Questions for authors
- Could the authors justify the methodological contirbution of DFR beyond combining existing reweighting methods?
- Could the authors compare to other robustness learning methods such as Group DRO, JTT, or class-balanced reweighting?
- Since Group 11 is under the primary diagnosis-priority group, could the authors further justify the effectiveness of full DFR for diagnosis-priority reliability? How should the trade-offs in Table. 1 be interpreted?
- Could the authors extend the evaluation to more recent or stronger language models beyond T5-Base and Flan-T5?
- Could the authors provide evaluation across additional datasets to further justify the effectiveness of DFR as current two datasets are primarily focusing on chest radiology report? Additionally, could the author provide the direct cross-dataset generalization experiments, such as training on CheXpert-Plus and evaluating on the MIMIC-CXR-derived dataset, or vice verse?