- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ED-SR, a self-correcting framework for generating chest X-ray reports. The authors utilize a single visual language model to handle both report generation and script generation, and train the model through three stages: (1) supervised fine-tuning for report generation, (2) followed by training the model's editing capabilities using a more powerful model, (3) optimizing the model's initial report generation ability through report-level reinforcement learning, counterfactual editing rewards, and preference loss distillation. Furthermore, the authors propose a Gradient Conflict Resolution mechanism to address potential gradient conflicts between report-level strategy optimization and preference loss.
Critical strengths
- This study addresses an important and practical problem: the difficulty of accurately assigning credit for specific errors in medical report generation. It proposes a more granular form of supervision for reinforcement learning in this context.
- The method framework is comprehensive: incorporating counterfactual editing, preference distillation, and GCR to form a clear training process. The authors also conducted ablation studies on key modules.
Critical weaknesses
-
The reward function used in this paper consists of BLEU-2, ROUGE-L, CheXbert F1, and RadGraph F1. However, the paper does not clearly explain how the weights for each metric were determined. Furthermore, the authors do not specify the design rationale for using BLEU-2 instead of BLEU-4 or BLEU-1. If the NLG metric is removed and the reward function is constructed solely based on clinical metrics, how would this affect the distribution of the reward, the positive/negative judgments, and the final generated output? Finally, the paper only states that CheXbert F1 is calculated at the sample level using 14 classes of observation labels, but it does not explicitly detail how uncertain (-1) and blank (NaN) labels are handled. These implementation details could directly impact the difference in rewards and, consequently, influence the optimization direction of the editing actions.
-
Currently, the primary sources of reward signals are CheXbert and RadGraph, along with NLG metrics. However, while CheXbert and RadGraph provide a degree of medical reliability, there is still a gap between them and actual clinical accuracy, which raises concerns about potential error accumulation.
-
In Experiment A.7, the modifications generated directly by the editor resulted in a decrease in several metrics. However, using the Oracle-filtered scheme, as outlined in the reference report, yielded better results. The modifications proposed by the editor should not be considered reliable clinical corrections, as the final outcome depends on how effectively the edits are identified and selected. Therefore, this editor is less effective when the reasoning process lacks GT (presumably, a specific guideline or method).
-
The ablation study demonstrates that incorporating edit-level losses on top of report-level reinforcement learning yields additional benefits. However, the current experiments do not fully distinguish whether the performance improvement stems from a meaningful semantic correspondence between edit actions and rewards, or from the additional auxiliary training objective and its optimization process. If the same edit scripts and reward distributions are maintained, but the correspondence between edit actions and rewards is randomized, would the model still achieve similar improvements? Furthermore, this paper utilizes edit scripts generated by the model to construct preferences. Since the edit scripts are only used during the training phase to construct edit-level supervision and preferences, the inference process does not utilize the edit scripts. Therefore, it is necessary to further validate whether the edit-level supervision, constructed using the model's own generated edit scripts, provides independent benefits compared to a simpler reference report-guided preference learning scheme.
Questions for authors
(1) How was the reward function's weights determined? Was hyperparameter search or sensitivity analysis performed? Why was BLEU-2 chosen over BLEU-1, BLEU-4, or not using NLG metrics? (2) What is the specific method used by CheXbert F1? How are "uncertain" and "blank" labels handled? Could different approaches to handling these labels affect the positive/negative judgment of edit rewards or the final performance? (3) Can you provide independent evidence to demonstrate that rewards for editors actually increase the likelihood of correcting clinical errors, rather than simply improving composite automated evaluation metrics? (4) Can further experiments, such as disrupting the relationship between editing and rewards, be used to confirm whether the semantic information contained in editing-level rewards is a key factor in overall performance improvement? (5) What are the primary advantages of the self-editing mechanism compared to the approach of constructing revision preferences directly from reference reports? Where does the author see the mechanism's most significant contribution?