- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper combines MCTS-based tool search, counterfactual refinement of search-control rules, and curriculum-based on-policy self-distillation for radiology report generation. The trained model generates reports directly from images without using external tools at inference time.
Critical strengths
-
The method uses tool-based evidence during training without requiring external tools at inference time. This keeps the inference process relatively simple.
-
The gains show up on clinically oriented metrics: on IU X-Ray, RadGraph F1 rises from 0.408 to 0.450 and GREEN from 0.586 to 0.631 (Table 2). The paper also compares against related distillation methods (OPSD, PW-OPSD, and DASH) in Table 10.
Critical weaknesses
-
Section 3.3 (p. 5) assigns a Keep label when a one-step continuation does not improve F1. However, tool sequences such as grounding, segmentation, and verification may require several calls before improving the result. It remains unclear how often this leads to incorrect Keep labels or whether the validation gate can detect such cases.
-
I’m concerned that evidence selection does not catch attribute errors. Search and selection mostly use CheXpert-14 F1, plus BLEU for hard negatives (Appendix B.3). Neither looks at laterality, location, or severity. A piece of evidence could have the right disease label but the wrong side or severity, and still be used as positive teacher context.
Questions for authors
- How often does a node labeled Keep after one step turn out to improve with a longer continuation? Does the validation gate catch these cases?
- When two evidence candidates have the same disease labels but different attributes, how do you tell them apart? And what does the teacher do when the reference report and the tool evidence conflict?
7 comments
Author Response (Part 2/3): Critical Weakness 2 and Question 1
Response to Critical Weakness 2: Thank you for the constructive suggestion. We acknowledge that CheXpert-14 F1 and BLEU do not explicitly verify fine-grained clinical attributes such as Laterality, Location, and Severity. We would like to clarify that both MIMIC-CXR and IU X-Ray use natural-language radiology reports as ground truth (GT), and some reports do not explicitly describe the fine-grained clinical attributes of the corresponding findings, making it difficult to construct complete attribute-level supervision. Although RadGraph-F1 can capture certain fine-grained attribute differences through clinical entities and their relations, when the GT report does not explicitly specify these attributes, even clinically correct candidate evidence may be penalized due to incomplete matching with the structured information in the reference report, potentially introducing measurement bias arising from incomplete GT information.
To further investigate the impact of different MCTS state-value metrics, we have conducted additional comparative experiments using GREEN, RadGraph-F1, and CheXpert-F1. As shown in Table R15, CheXpert-F1 achieves the best overall evidence-quality score and search efficiency, with a Norm. Mean of 0.997 and an average of 5.27 tool calls. Furthermore, Table R16 shows that CA-OPSD trained on evidence collected using CheXpert-F1 achieves the highest overall score (0.990) for downstream report generation. Although different evaluation metrics exhibit advantages on individual measures, these results support the choice of CheXpert-F1 as the MCTS state-value metric in our current framework.
Table R15. Comparison of MCTS state-value metrics in terms of evidence quality and search efficiency on 100 cases from MIMIC-CXR and IU X-Ray. Norm. Mean denotes the average of the four normalized evidence-quality metrics, excluding Tool Calls. Norm.Mean denotes the arithmetic mean of all normalized metric scores, with each metric normalized by its highest score across methods.
| State-value Metrics | CheXpert-F1↑ | Rad-F1↑ | ↑ | ↑ | Norm.Mean↑ | Tool Calls↓ |
|---|---|---|---|---|---|---|
| GREEN | 0.707 | 0.604 | 0.713 | 0.839 | 0.986 | 5.38 |
| RadGraph-F1 | 0.702 | 0.614 | 0.721 | 0.840 | 0.991 | 5.56 |
| CheXpert-F1(Ours) | 0.723 | 0.608 | 0.719 | 0.846 | 0.997 | 5.27 |
Table R16. Effect of different MCTS state-value metrics on downstream radiology report generation after CA-OPSD training on IU X-Ray. Norm. Mean denotes the average of the six normalized report-generation metrics. Norm.Mean denotes the arithmetic mean of all normalized metric scores, with each metric normalized by its highest score across methods.
| State-value Metrics | NLG Metrics BL-4 | NLG Metrics MTR | NLG Metrics RG-L | CE Metrics GREEN | CE Metrics RadGraph-F1 | CE Metrics CheXpert-F1 | Norm.Mean |
|---|---|---|---|---|---|---|---|
| GREEN | 0.265 | 0.232 | 0.421 | 0.635 | 0.436 | 0.505 | 0.977 |
| RadGraph-F1 | 0.259 | 0.241 | 0.421 | 0.633 | 0.457 | 0.509 | 0.988 |
| CheXpert-F1(Ours) | 0.267 | 0.239 | 0.428 | 0.631 | 0.450 | 0.521 | 0.995 |
In future work, we will further explore automatic extraction and consistency verification of fine-grained clinical attributes, incorporating attribute-level constraints for Laterality, Location, and Severity into evidence selection and teacher-context construction to reduce the risk of disease-label agreement accompanied by incorrect clinical attributes. The corresponding experiments and analyses have been added to Tables 21 and 22 in Appendix C. 1.6 of the revised manuscript, with the changes highlighted for clarity.
Response to Question 1: As shown in Table R13 under Critical Weakness 1, within the evaluated counterfactual expansion depths ranging from 1 to 5, 9.5% of nodes labeled Keep after a one-step continuation yield performance gains under longer continuations. However, the Skill Validation Gate experiments demonstrate that the impact of such misleading feedback on subsequent skill refinement can be effectively mitigated. As shown in Table R14, our evaluation over 200 batches shows that 95.5% of candidate skill updates derived from full counterfactual contexts pass the Validation Gate, whereas only 2.5% of those derived exclusively from misleading Keep feedback are accepted, meaning that 97.5% of such updates are rejected. These results demonstrate that the Validation Gate effectively filters unreliable skill updates induced by misleading feedback, thereby reducing the impact of overlooked delayed benefits on subsequent skill refinement.
Author Response (Part 3/3): Question 2
Response to Question 2: Thank you for the constructive suggestion. When evidence candidates share the same disease labels but differ in fine-grained clinical attributes, the Decision Agent selects evidence by considering both the original X-ray image and the actual tool outputs during multi-tool evidence orchestration, rather than relying solely on disease labels. To further evaluate the clinical consistency of evidence selection, we have conducted additional experiments. As shown in Table R17, the MCTS-selected Positive Evidence achieves the highest RadGraph-F1 among candidate evidence in 94% of cases, indicating a high degree of agreement with the reference reports. Furthermore, Table R18 shows that, compared with MCTS, CSC-SkillEvo reduces the conflict rates of selected Positive Evidence in Laterality, Location, and Severity from 5.23%, 6.58%, and 4.77% to 4.32%, 5.27%, and 3.46%, respectively, with the mean conflict rate decreasing from 5.53% to 4.35%. These results further demonstrate improved fine-grained clinical attribute consistency in the selected evidence.
Table R17. Clinical agreement of MCTS-selected positive evidence with reference reports and its ranking among candidate evidence on IU X-Ray. Top-1 Consistency Rate denotes the percentage of cases in which the selected evidence achieves the highest RadGraph-F1 among candidate evidence.
| Evidence Selection | CheXpert-F1↑ | RadGraph-F1↑ | Top-1 Consistency Rate(%)↑ |
|---|---|---|---|
| MCTS-Selected Positive Evidence | 0.723 | 0.608 | 94 |
Table R18. Comparison of clinical attribute conflict rates in selected positive evidence (MCTS vs. CSC-SkillEvo) and generated reports (Base vs. Ours) on IU X-Ray. Mean Conflict denotes the average conflict rate across Laterality, Location, and Severity.
| Evaluation Targets | Methods | Laterality Conflict(%)↓ | Location Conflict(%)↓ | Severity Conflict(%)↓ | Mean Conflict(%)↓ |
|---|---|---|---|---|---|
| Evidence | MCTS | 5.23 | 6.58 | 4.77 | 5.53 |
| Evidence | CSC-SkillEvo | 4.32 | 5.27 | 3.46 | 4.35 |
| Generated Report | Base | 7.24 | 8.17 | 6.53 | 7.31 |
| Generated Report | Ours | 3.05 | 4.07 | 2.20 | 3.11 |
Regarding potential conflicts between reference reports and tool evidence, CA-OPSD adaptively adjusts Token-Level Weights based on teacher prediction differences under different evidence conditions during the first two curriculum stages, guiding the student to progressively distinguish and internalize clinical evidence. In the third curriculum stage, external Positive Evidence is removed, and the GT-only Teacher is directly used as the distillation target, thereby reducing the student's dependence on potentially inconsistent external evidence. As shown in Table R19, the conflict rates of generated reports in Laterality, Location, and Severity consistently decrease across the three curriculum stages, with the mean conflict rate dropping from 7.31% to 3.11%. These results support the effectiveness of CA-OPSD in improving clinical attribute consistency. We have added the corresponding tables and analyses to Tables 23, 24, and 25 in Appendix C.1.7 of the revised manuscript, with the changes highlighted for clarity.
Table R19. Clinical attribute conflict rates and cumulative conflict resolution across the three CA-OPSD curriculum stages on IU X-Ray. Mean Conflict denotes the average conflict rate across Laterality, Location, and Severity. Cumulative Conflict Resolution Rate measures the proportion of initially identified attribute conflicts resolved at each curriculum stage.
| Evaluation Target | Training Stages | Laterality Conflict (%)↓ | Location Conflict (%)↓ | Severity Conflict (%)↓ | Mean Conflict (%)↓ | Cumulative Conflict Resolution Rate (%)↑ |
|---|---|---|---|---|---|---|
| Generated Report | Base | 7.24 | 8.17 | 6.53 | 7.31 | 87.3 |
| Generated Report | After | 5.75 | 6.74 | 4.79 | 5.76 | 90.8 |
| Generated Report | After | 4.43 | 5.41 | 3.52 | 4.45 | 93.6 |
| Generated Report | After | 3.05 | 4.07 | 2.20 | 3.11 | 94.5 |
Author Response (Part 1/3): Critical Weakness 1
Response to Critical Weakness 1: Thank you for the constructive suggestion. We first clarify that the Keep/Revise feedback generated by one-step counterfactual supervision does not directly determine whether to stop or continue the MCTS search. Instead, it is used to evaluate previous search termination decisions, identify potential deficiencies, and guide the refinement of the Search-Termination Skill. The updated skill, once validated by the Validation Gate, is subsequently used to guide termination decisions in the next round of MCTS search. To investigate potential delayed benefits from multi-step tool calls, we have extended the Counterfactual Expansion Depth from 1 to 5 and measured the Misleading Keep Rate. As shown in Table R13, the Misleading Keep Rate progressively decreases with increasing expansion depth, while Evi-F1 improves only marginally and the cost of counterfactual data collection increases substantially. These results indicate that our one-step counterfactual supervision achieves a reasonable trade-off between supervision depth and computational cost.
Furthermore, we have evaluated the filtering capability of the Skill Validation Gate over 200 batches. As shown in Table R14, the acceptance rate of candidate skill updates derived from full counterfactual contexts is 95.5%, whereas only 2.5% of candidate updates derived exclusively from misleading Keep feedback are accepted. These results demonstrate that the Validation Gate effectively filters unreliable skill updates induced by misleading feedback, mitigating the limitations of one-step counterfactual supervision. We have added the corresponding experiments and analyses to Appendix C.1.5 (Stopping Supervision Analysis) of the revised manuscript, with the changes highlighted.
Table R13. Effect of counterfactual expansion depth on misleading Keep feedback, evidence quality, and collection cost in CSC-SkillEvo. Misleading Keep Rate denotes the percentage of Keep labels contradicted by improvements found through longer counterfactual expansions. Collection Time and Extra Nodes denote the average per-case time and number of additional nodes required for counterfactual context collection, respectively.
Table R14. Effectiveness of the skill validation gate in filtering updates derived from misleading Keep feedback in CSC-SkillEvo. Gate Acceptance Rate is evaluated over 200 batches at different expansion depths, comparing candidate skill updates derived from full counterfactual contexts with those derived exclusively from misleading Keep feedback.