acceptodds
Under review as a conference paper at ICLR 2027

From Evidence Orchestration to Reasoning Internalization for Radiology Report Generation

12 views

Abstract

Radiology report generation (RRG) methods often produce diagnostic findings without explicit clinical evidence, which may lead to factual hallucinations and lesion omissions. Although recent MLLM-based medical agents have begun to acquire clinical evidence through external tools, integrating conflicting clinical observations returned by different tools and mitigating error accumulation caused by early tool decisions remain challenging. Thus, we propose an evidence-informed RRG framework. Specifically, deliberative multi-tool evidence orchestration is first introduced, which employs Monte Carlo Tree Search (MCTS) to explore alternative tool-use trajectories and integrate outputs from different tools into structured clinical evidence. During MCTS decision-making, Counterfactual Search-Control Skill Evolution (CSC-SkillEvo) is further incorporated, which constructs counterfactual supervision signals from tool-use trajectories to refine tool-prioritization and search-termination decisions, thereby enabling holistic clinical evidence orchestration. Finally, we devise Curriculum-Annealed On-Policy Self-Distillation (CA-OPSD), which progressively restricts the clinical evidence accessible to the teacher to construct curriculum-based distillation signals, thereby enabling the RRG model to gradually internalize evidence-supported reasoning capabilities. Experiments on public datasets demonstrate that our framework consistently improves the clinical consistency of generated reports, thereby enhancing the reliability of RRG.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Radiology Report Generation, Multimodal Large Language Models, Tool-use, Clinical Evidence Orchestration, On-Policy Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yvoUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper combines MCTS-based tool search, counterfactual refinement of search-control rules, and curriculum-based on-policy self-distillation for radiology report generation. The trained model generates reports directly from images without using external tools at inference time.

Critical strengths
  1. The method uses tool-based evidence during training without requiring external tools at inference time. This keeps the inference process relatively simple.

  2. The gains show up on clinically oriented metrics: on IU X-Ray, RadGraph F1 rises from 0.408 to 0.450 and GREEN from 0.586 to 0.631 (Table 2). The paper also compares against related distillation methods (OPSD, PW-OPSD, and DASH) in Table 10.

Critical weaknesses
  1. Section 3.3 (p. 5) assigns a Keep label when a one-step continuation does not improve F1. However, tool sequences such as grounding, segmentation, and verification may require several calls before improving the result. It remains unclear how often this leads to incorrect Keep labels or whether the validation gate can detect such cases.

  2. I’m concerned that evidence selection does not catch attribute errors. Search and selection mostly use CheXpert-14 F1, plus BLEU for hard negatives (Appendix B.3). Neither looks at laterality, location, or severity. A piece of evidence could have the right disease label but the wrong side or severity, and still be used as positive teacher context.

Questions for authors
  1. How often does a node labeled Keep after one step turn out to improve with a longer continuation? Does the validation gate catch these cases?
  2. When two evidence candidates have the same disease labels but different attributes, how do you tell them apart? And what does the teacher do when the reference report and the tool evidence conflict?
7 comments
AuthorsOpenReview ↗

Author Response (Part 1/3): Critical Weakness 1

Response to Critical Weakness 1: Thank you for the constructive suggestion. We first clarify that the Keep/Revise feedback generated by one-step counterfactual supervision does not directly determine whether to stop or continue the MCTS search. Instead, it is used to evaluate previous search termination decisions, identify potential deficiencies, and guide the refinement of the Search-Termination Skill. The updated skill, once validated by the Validation Gate, is subsequently used to guide termination decisions in the next round of MCTS search. To investigate potential delayed benefits from multi-step tool calls, we have extended the Counterfactual Expansion Depth from 1 to 5 and measured the Misleading Keep Rate. As shown in Table R13, the Misleading Keep Rate progressively decreases with increasing expansion depth, while Evi-F1 improves only marginally and the cost of counterfactual data collection increases substantially. These results indicate that our one-step counterfactual supervision achieves a reasonable trade-off between supervision depth and computational cost.

Furthermore, we have evaluated the filtering capability of the Skill Validation Gate over 200 batches. As shown in Table R14, the acceptance rate of candidate skill updates derived from full counterfactual contexts is 95.5%, whereas only 2.5% of candidate updates derived exclusively from misleading Keep feedback are accepted. These results demonstrate that the Validation Gate effectively filters unreliable skill updates induced by misleading feedback, mitigating the limitations of one-step counterfactual supervision. We have added the corresponding experiments and analyses to Appendix C.1.5 (Stopping Supervision Analysis) of the revised manuscript, with the changes highlighted.

Table R13. Effect of counterfactual expansion depth on misleading Keep feedback, evidence quality, and collection cost in CSC-SkillEvo. Misleading Keep Rate denotes the percentage of Keep labels contradicted by improvements found through longer counterfactual expansions. Collection Time and Extra Nodes denote the average per-case time and number of additional nodes required for counterfactual context collection, respectively.

Expansion DepthsMisleading Keep Rate(%) ↓Evi-F1 ↑Collection Time(s) ↓Extra Nodes ↓
1 (Ours)9.50.72313.061.57
25.30.72827.243.27
32.90.73042.645.43
41.30.73159.587.64
500.73380.329.72

Table R14. Effectiveness of the skill validation gate in filtering updates derived from misleading Keep feedback in CSC-SkillEvo. Gate Acceptance Rate is evaluated over 200 batches at different expansion depths, comparing candidate skill updates derived from full counterfactual contexts with those derived exclusively from misleading Keep feedback.

Expansion DepthsGate Acceptance Rate (%) FullGate Acceptance Rate (%) Misleading Only
1(Ours)95.52.5
296.01.5
397.51.0
497.51.0
598.0-
AuthorsOpenReview ↗

Author Response (Part 2/3): Critical Weakness 2 and Question 1

Response to Critical Weakness 2: Thank you for the constructive suggestion. We acknowledge that CheXpert-14 F1 and BLEU do not explicitly verify fine-grained clinical attributes such as Laterality, Location, and Severity. We would like to clarify that both MIMIC-CXR and IU X-Ray use natural-language radiology reports as ground truth (GT), and some reports do not explicitly describe the fine-grained clinical attributes of the corresponding findings, making it difficult to construct complete attribute-level supervision. Although RadGraph-F1 can capture certain fine-grained attribute differences through clinical entities and their relations, when the GT report does not explicitly specify these attributes, even clinically correct candidate evidence may be penalized due to incomplete matching with the structured information in the reference report, potentially introducing measurement bias arising from incomplete GT information.

To further investigate the impact of different MCTS state-value metrics, we have conducted additional comparative experiments using GREEN, RadGraph-F1, and CheXpert-F1. As shown in Table R15, CheXpert-F1 achieves the best overall evidence-quality score and search efficiency, with a Norm. Mean of 0.997 and an average of 5.27 tool calls. Furthermore, Table R16 shows that CA-OPSD trained on evidence collected using CheXpert-F1 achieves the highest overall score (0.990) for downstream report generation. Although different evaluation metrics exhibit advantages on individual measures, these results support the choice of CheXpert-F1 as the MCTS state-value metric in our current framework.

Table R15. Comparison of MCTS state-value metrics in terms of evidence quality and search efficiency on 100 cases from MIMIC-CXR and IU X-Ray. Norm. Mean denotes the average of the four normalized evidence-quality metrics, excluding Tool Calls. Norm.Mean denotes the arithmetic mean of all normalized metric scores, with each metric normalized by its highest score across methods.

State-value MetricsCheXpert-F1↑Rad-F1↑CQLLM\mathrm{CQ}_{\mathrm{LLM}}↑CQRad\mathrm{CQ}_{\mathrm{Rad}}↑Norm.Mean↑Tool Calls↓
GREEN0.7070.6040.7130.8390.9865.38
RadGraph-F10.7020.6140.7210.8400.9915.56
CheXpert-F1(Ours)0.7230.6080.7190.8460.9975.27

Table R16. Effect of different MCTS state-value metrics on downstream radiology report generation after CA-OPSD training on IU X-Ray. Norm. Mean denotes the average of the six normalized report-generation metrics. Norm.Mean denotes the arithmetic mean of all normalized metric scores, with each metric normalized by its highest score across methods.

State-value MetricsNLG Metrics BL-4NLG Metrics MTRNLG Metrics RG-LCE Metrics GREENCE Metrics RadGraph-F1CE Metrics CheXpert-F1Norm.Mean
GREEN0.2650.2320.4210.6350.4360.5050.977
RadGraph-F10.2590.2410.4210.6330.4570.5090.988
CheXpert-F1(Ours)0.2670.2390.4280.6310.4500.5210.995

In future work, we will further explore automatic extraction and consistency verification of fine-grained clinical attributes, incorporating attribute-level constraints for Laterality, Location, and Severity into evidence selection and teacher-context construction to reduce the risk of disease-label agreement accompanied by incorrect clinical attributes. The corresponding experiments and analyses have been added to Tables 21 and 22 in Appendix C. 1.6 of the revised manuscript, with the changes highlighted for clarity.

Response to Question 1: As shown in Table R13 under Critical Weakness 1, within the evaluated counterfactual expansion depths ranging from 1 to 5, 9.5% of nodes labeled Keep after a one-step continuation yield performance gains under longer continuations. However, the Skill Validation Gate experiments demonstrate that the impact of such misleading feedback on subsequent skill refinement can be effectively mitigated. As shown in Table R14, our evaluation over 200 batches shows that 95.5% of candidate skill updates derived from full counterfactual contexts pass the Validation Gate, whereas only 2.5% of those derived exclusively from misleading Keep feedback are accepted, meaning that 97.5% of such updates are rejected. These results demonstrate that the Validation Gate effectively filters unreliable skill updates induced by misleading feedback, thereby reducing the impact of overlooked delayed benefits on subsequent skill refinement.

AuthorsOpenReview ↗

Author Response (Part 3/3): Question 2

Response to Question 2: Thank you for the constructive suggestion. When evidence candidates share the same disease labels but differ in fine-grained clinical attributes, the Decision Agent selects evidence by considering both the original X-ray image and the actual tool outputs during multi-tool evidence orchestration, rather than relying solely on disease labels. To further evaluate the clinical consistency of evidence selection, we have conducted additional experiments. As shown in Table R17, the MCTS-selected Positive Evidence achieves the highest RadGraph-F1 among candidate evidence in 94% of cases, indicating a high degree of agreement with the reference reports. Furthermore, Table R18 shows that, compared with MCTS, CSC-SkillEvo reduces the conflict rates of selected Positive Evidence in Laterality, Location, and Severity from 5.23%, 6.58%, and 4.77% to 4.32%, 5.27%, and 3.46%, respectively, with the mean conflict rate decreasing from 5.53% to 4.35%. These results further demonstrate improved fine-grained clinical attribute consistency in the selected evidence.

Table R17. Clinical agreement of MCTS-selected positive evidence with reference reports and its ranking among candidate evidence on IU X-Ray. Top-1 Consistency Rate denotes the percentage of cases in which the selected evidence achieves the highest RadGraph-F1 among candidate evidence.

Evidence SelectionCheXpert-F1↑RadGraph-F1↑Top-1 Consistency Rate(%)↑
MCTS-Selected Positive Evidence0.7230.60894

Table R18. Comparison of clinical attribute conflict rates in selected positive evidence (MCTS vs. CSC-SkillEvo) and generated reports (Base vs. Ours) on IU X-Ray. Mean Conflict denotes the average conflict rate across Laterality, Location, and Severity.

Evaluation TargetsMethodsLaterality Conflict(%)↓Location Conflict(%)↓Severity Conflict(%)↓Mean Conflict(%)↓
EvidenceMCTS5.236.584.775.53
EvidenceCSC-SkillEvo4.325.273.464.35
Generated ReportBase7.248.176.537.31
Generated ReportOurs3.054.072.203.11

Regarding potential conflicts between reference reports and tool evidence, CA-OPSD adaptively adjusts Token-Level Weights based on teacher prediction differences under different evidence conditions during the first two curriculum stages, guiding the student to progressively distinguish and internalize clinical evidence. In the third curriculum stage, external Positive Evidence is removed, and the GT-only Teacher is directly used as the distillation target, thereby reducing the student's dependence on potentially inconsistent external evidence. As shown in Table R19, the conflict rates of generated reports in Laterality, Location, and Severity consistently decrease across the three curriculum stages, with the mean conflict rate dropping from 7.31% to 3.11%. These results support the effectiveness of CA-OPSD in improving clinical attribute consistency. We have added the corresponding tables and analyses to Tables 23, 24, and 25 in Appendix C.1.7 of the revised manuscript, with the changes highlighted for clarity.

Table R19. Clinical attribute conflict rates and cumulative conflict resolution across the three CA-OPSD curriculum stages on IU X-Ray. Mean Conflict denotes the average conflict rate across Laterality, Location, and Severity. Cumulative Conflict Resolution Rate measures the proportion of initially identified attribute conflicts resolved at each curriculum stage.

Evaluation TargetTraining StagesLaterality Conflict (%)↓Location Conflict (%)↓Severity Conflict (%)↓Mean Conflict (%)↓Cumulative Conflict Resolution Rate (%)↑
Generated ReportBase7.248.176.537.3187.3
Generated ReportAfter Ced\mathcal{C}_{\mathrm{ed}}5.756.744.795.7690.8
Generated ReportAfter Ced+Ceg\mathcal{C}_{\mathrm{ed}}+\mathcal{C}_{\mathrm{eg}}4.435.413.524.4593.6
Generated ReportAfter Ced+Ceg+Crc\mathcal{C}_{\mathrm{ed}}+\mathcal{C}_{\mathrm{eg}}+\mathcal{C}_{\mathrm{rc}}3.054.072.203.1194.5
Reviewer_95RpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a two stage pipeline for chest X-ray report generation. Stage one runs MCTS over a library of seven radiology tools such as a classifier, retrieval, grounding, segmentation and VQA to collect clinical evidence for an image, with the search value defined by CheXbert F1 against the ground truth report. Between search rounds, a GPT-5.5 skill optimizer looks at counterfactual pairs from the tree, where a lower priority branch scored higher or where continuing the search would have helped, and rewrites two natural language skills for tool prioritization and search termination, gated by a validation check. Stage two is a three stage on-policy self-distillation where the teacher is the same model conditioned on the ground truth report plus positive evidence, hard negative evidence, or nothing, with token weights from teacher entropy and inter teacher JS divergence. At test time only the student runs with no tools. Results are reported on MIMIC-CXR and IU X-Ray with BLIP-3 as the backbone, plus ablations, evidence quality analysis with five radiologists, and comparisons to other OPSD variants.

Critical strengths
  1. The evidence quality evaluation in Appendix C.1 is better than usual. Five radiologists rate the evidence, and the comparison covers ReAct, Plan-and-Solve, beam, DFSDT, ToT and plain MCTS under the same tool library with tool call counts reported.
  2. The skill validation gate is a sensible safeguard against the LLM optimizer drifting, and the recovery from early planning error experiment in Table 6 is a good way to probe what the evolved skills actually buy.
  3. Table 8 is the right experiment to run. Comparing evidence prompting at inference, evidence augmented SFT, GRPO and CA-OPSD under matched conditions is what makes the internalization claim testable at all.
  4. The pipeline is fully described down to prompts and hyperparameters.
Critical weaknesses
  1. The main table does not show what the method contributes. On MIMIC-CXR the paper reports BL-4 of 0.224 against 0.175 for the best prior method. But the backbone is BLIP-3 with Phi-3 and the ablation rows that exist suggest the plain SFT model is already near 0.19 to 0.20, which would by itself beat every baseline in Table 1. There is no Base row for MIMIC-CXR anywhere. On top of that the paper says it performs data cleaning on MIMIC-CXR, so the test set is not the one the baselines were evaluated on. The IU X-Ray jump from 0.208 to 0.267 BL-4 has the same problem. Right now the reader cannot tell how much is the method and how much is a stronger backbone on a different split.
  2. GPT-5.5 is doing a lot of the work and also doing the grading. It is the skill optimizer, it is the CQ LLM judge, and the skills it writes are natural language rules. The paper frames this as self-evolution but the counterfactual labels are a simple Revise or Keep rule and the rewriting is a frontier model with high reasoning effort. Whether a weaker optimizer would get the same result is never tested.
  3. The distillation stage is tiny. 3,000 MIMIC-CXR cases, 30 optimization steps per stage, 8 rollouts each. That is on the order of a few thousand gradient steps worth of tokens total. Gains of this size from this little training with no seeds or variance reported are hard to believe at face value. No experiment uses more than one run.
  4. Several numbers do not agree with each other. Table 1 reports MIMIC-CXR recall of 0.554 for Ours while Table 3 reports 0.543 for the same configuration. Table 2 and Table 8 share the Evidence Augmented SFT row but this is the only place the two tables line up.
  5. The tool library models are never named. Figure 2 lists functions, not which classifier, which VQA model, which grounding model. This matters because the evidence quality is bounded by those tools.
  6. No cost is reported for the search stage. 30 simulations, depth 8, several tool calls per node, per training image.
Questions for authors
  1. What are the MIMIC-CXR and IU X-Ray numbers for BLIP-3 after SFT alone, with no evidence, no MCTS, no distillation, on the exact test split used in Table 1? Please add this row to Table 1.
  2. What does data cleaning on MIMIC-CXR remove, and how many test reports remain? Are the baseline numbers in Table 1 copied from papers that used the full official test set?
  3. Table 1 and Table 3 give different recall for the same model on MIMIC-CXR. Which is correct?
  4. How many seeds were run for Table 2, Table 3 and Table 8, and what is the variance? With 30 steps per curriculum stage I would expect it to be large.
  5. Does CSC-SkillEvo still beat the human written skill in Table 5 if the optimizer is a smaller open model instead of GPT-5.5?
  6. Which specific models are the seven tools?
10 comments
AuthorsOpenReview ↗

Author Response (Part 1/6): Critical Weakness 1

Response to Critical Weakness 1: Thank you for the constructive suggestion. The original main table followed common RRG practice by summarizing results from recent methods. As some methods lack available source code, we reimplemented representative open-source SOTA methods using the same BLIP-3 backbone, data splits, and preprocessing procedures as ours to improve the comparability and rigor of our evaluations. All methods were evaluated on identical test samples. We also included the SFT-only Base model in SOTA comparisons and conducted a complete component ablation on MIMIC-CXR. These results appear below; Tables 2, 3, 4, and 10 have been updated in the revised manuscript, with changes highlighted.

Table R1. Comparison with state-of-the-art methods on MIMIC-CXR. Results marked with * are reproduced using BLIP-3 as the backbone under the same data splits and preprocessing procedures as Ours. Results are reported as mean ± standard deviation over 3 random seeds.

MethodsBL-4MTRRG-LPRF1Rad-F1GREENCRIMSONICARE-AVG
AM-MRG*0.179±0.0050.185±0.0040.335±0.0060.541±0.100.423±0.0070.471±0.0080.297±0.0040.369±0.0050.164±0.0040.577±0.008
MLRG*0.196±0.0050.201±0.0030.347±0.0070.586±0.0070.544±0.0050.564±0.0060.310±0.0030.356±0.0020.179±0.0040.588±0.009
R2-LLM*0.172±0.0040.178±0.0040.332±0.0050.531±0.0060.403±0.0080.458±0.0070.298±0.0040.364±0.0050.157±0.0040.534±0.008
multiP-r2gen*0.184±0.0030.186±0.0040.343±0.0040.567±0.0050.424±0.0060.485±0.0050.307±0.0050.380±0.0060.176±0.0030.557±0.007
BLIP-3(Only SFT)0.165±0.0030.172±0.0020.326±0.0050.506±0.0060.385±0.0050.437±0.0060.284±0.0020.354±0.0040.144±0.0030.503±0.007
Ours0.223±0.0040.223±0.0030.357±0.0040.602±0.0050.554±0.0070.577±0.0060.328±0.0040.403±0.0030.192±0.0050.655±0.006

Table R2. Ablation study of different components on MIMIC-CXR. “Base” denotes the SFT RRG model. Results are reported as mean ± standard deviation over 3 random seeds.

BaseMCTSCSC-SkillEvo Stp\mathbb{S}_{tp}CSC-SkillEvo Sst\mathbb{S}_{st}CA-OPSDBL-4MTRRG-LPRF1Rad-F1GREENCRIMSONICARE-AVG
✓0.165±0.0030.172±0.0020.326±0.0050.506±0.0060.385±0.0050.437±0.0060.284±0.0020.354±0.0040.144±0.0030.503±0.007
✓✓0.177±0.0040.179±0.0030.332±0.0040.522±0.00100.398±0.0080.452±0.0080.289±0.0030.368±0.0030.156±0.0040.548±0.006
✓✓✓0.187±0.0050.190±0.0020.342±0.0040.547±0.0070.417±0.0050.473±0.0060.302±0.0050.375±0.0050.167±0.0040.582±0.008
✓✓✓0.183±0.0030.184±0.0030.337±0.0060.540±0.0090.402±0.0070.461±0.0080.297±0.0050.372±0.0040.164±0.0050.579±0.007
✓✓✓✓0.205±0.0050.201±0.0040.342±0.0050.573±0.0050.487±0.0060.527±0.0050.310±0.0030.380±0.0040.177±0.0050.613±0.009
✓✓✓✓✓0.223±0.0040.223±0.0030.357±0.0040.602±0.0050.554±0.0070.577±0.0060.328±0.0040.403±0.0030.192±0.0050.655±0.006

Table R3. Comparison with state-of-the-art methods on IU X-Ray. Results marked with * are reproduced using BLIP-3 as the backbone under the same data splits and preprocessing procedures as Ours. Results are reported as mean ± standard deviation over 3 random seeds.

MethodsBL-4MTRRG-LRad-F1GREENCRIMSONICARE-AVG
AM-MRG*0.213±0.0070.216±0.0050.384±0.0070.416±0.0060.599±0.0050.584±0.0060.671±0.10
MLRG*0.217±0.0040.220±0.0030.374±0.0050.422±0.0040.586±0.0030.549±0.0050.661±0.006
R2-LLM*0.218±0.0040.213±0.0040.375±0.0060.414±0.0050.594±0.0040.564±0.0070.653±0.008
multiP-r2gen*0.223±0.0060.221±0.0050.403±0.0050.422±0.0060.615±0.0040.597±0.0060.683±0.007
BLIP-3(Only SFT)0.194±0.0040.200±0.0030.352±0.0050.406±0.0030.583±0.0030.501±0.0060.617±0.007
Ours0.266±0.0050.239±0.0040.426±0.0050.452±0.0040.633±0.0030.655±0.0040.760±0.006

Table R4. Ablation study of different components on IU X-Ray. “Base” denotes the SFT RRG model. Results are reported as mean ± standard deviation over 3 random seeds.

BaseMCTSCSC-SkillEvo Stp\mathbb{S}_{tp}CSC-SkillEvo Sst\mathbb{S}_{st}CA-OPSDBL-4MTRRG-LRad-F1GREENCRIMSONICARE-AVG
✓0.194±0.0040.200±0.0030.352±0.0050.406±0.0030.583±0.0030.501±0.0060.617±0.007
✓✓0.216±0.0030.208±0.0030.377±0.0080.418±0.0030.603±0.0040.523±0.0050.664±0.009
✓✓✓0.238±0.0040.220±0.0020.395±0.0090.428±0.0040.612±0.0040.558±0.0070.682±0.011
✓✓✓0.237±0.0050.216±0.0040.393±0.0110.426±0.0030.608±0.0050.554±0.0060.680±0.008
✓✓✓✓0.246±0.0050.224±0.0040.416±0.0050.438±0.0050.617±0.0040.602±0.0050.705±0.007
✓✓✓✓✓0.266±0.0050.239±0.0040.426±0.0050.452±0.0040.633±0.0030.655±0.0040.760±0.006
AuthorsOpenReview ↗

Author Response (Part 2/6): Critical Weakness 2

Response to Critical Weakness 2: Thank you for the constructive suggestion. To distinguish the contributions of CSC-SkillEvo from GPT-5.5's reasoning capability, we have conducted additional ablation experiments with GPT-5.5 fixed as the skill optimizer, demonstrating the effectiveness of Counterfactual Refinement and Skill Validation beyond direct skill rewriting. Furthermore, we have evaluated different open-source and closed-source models as skill optimizers. The results show that CSC-SkillEvo consistently outperforms Direct Refinement across different optimizers, despite variations in their capabilities. Meanwhile, to avoid using GPT-5.5 for both skill optimization and evaluation, we have employed Claude Opus 5 as an independent LLM judge for CQ evaluation, together with Evi-F1 and radiologist-assessed CQ, to further validate the reliability of our results. These results are shown in Tables R8 and R9 below, and we have added the corresponding experiments and analyses to Tables 11 and 12 in Appendix C.1.2 of the revised manuscript, with the revisions highlighted for clarity.

Table R8.Ablation study of skill validation and counterfactual refinement in CSC-SkillEvo with GPT-5.5 as the skill optimizer. Claude Opus 5 serves as an independent evaluator for CQLLM\mathrm{CQ}_{\mathrm{LLM}}.

MethodsEvi-F1↑CQLLM\mathrm{CQ}_{\mathrm{LLM}}↑CQRad\mathrm{CQ}_{\mathrm{Rad}}↑Tool Calls↓
Direct Refinement0.6820.6630.8116.35
+Validation Gate0.6930.6720.8235.98
+Counterfactual Refinement0.7130.6940.8355.87
CSC-SkillEvo(Ours)0.7230.7080.8465.27

Table R9. Generalization of CSC-SkillEvo across open-source and closed-source VLMs as skill optimizers.

Skill OptimizerDirect Refinement Evi-F1↑Direct Refinement CQLLM\mathrm{CQ}_{\mathrm{LLM}}↑Direct Refinement CQRad\mathrm{CQ}_{\mathrm{Rad}}↑Direct Refinement Tool Calls↓CSC-SkillEvo Evi-F1↑CSC-SkillEvo CQLLM\mathrm{CQ}_{\mathrm{LLM}}↑CSC-SkillEvo CQRad\mathrm{CQ}_{\mathrm{Rad}}↑CSC-SkillEvo Tool Calls↓
Open-source
Qwen-3.5-27B0.6320.6130.7716.230.6910.6540.8095.73
Qwen-3.8-27B0.6430.6250.7906.500.6960.6680.8275.66
Closed-source
Claude Opus 4.70.6760.6560.8046.420.7190.6990.8375.30
GPT-5.5$\textbf{(Ours)}$0.6820.6630.8116.350.7230.7080.8465.27
AuthorsOpenReview ↗

Author Response (Part 3/6): Critical Weaknesses 3-4

Response to Critical Weakness 3: Thank you for the constructive suggestion. We first clarify that CA-OPSD is a post-SFT training stage, where the model has already acquired initial report generation capabilities. Importantly, the overall performance gains arise from the combined contributions of MCTS, CSC-SkillEvo, and CA-OPSD, rather than from the distillation stage alone. To address concerns about distillation effectiveness and result stability, we have conducted additional ablation experiments on the three core curriculum stages of CA-OPSD, reporting mean ± standard deviation (Mean ± Std) over three different random seeds to assess their incremental contributions. Furthermore, we have conducted three-seed evaluations of different Token Weight Strategies and Evidence Internalization Strategies during distillation. These results are shown in Tables R10, R6, and R7 below, and we have added the corresponding experiments and analyses to Tables 5, 14, and 15 in Appendix C.1.3 of the revised manuscript, with the revisions highlighted for clarity.

Table R10. Ablation study of progressive curriculum distillation in CA-OPSD on IU X-Ray. Results are reported as mean ± standard deviation over 3 random seeds.

MethodsBL-4MTRRG-LGREENRad-F1
Base0.194±0.0040.200±0.0030.352±0.0050.583±0.0030.406±0.003
+Ced\mathcal{C}_{\mathrm{ed}}0.224±0.0040.212±0.0020.378±0.0080.612±0.0040.414±0.003
+Ced+Ceg\mathcal{C}_{\mathrm{ed}}+\mathcal{C}_{\mathrm{eg}}0.243±0.0040.221±0.0020.408±0.0050.622±0.0020.424±0.004
+Ced+Ceg+Crc\mathcal{C}_{\mathrm{ed}}+\mathcal{C}_{\mathrm{eg}}+\mathcal{C}_{\mathrm{rc}} (Ours)0.266±0.0050.239±0.0040.426±0.0050.633±0.0030.452±0.004

Table R6. Ablation of token weight strategies for CA-OPSD on MIMIC-CXR. Results are reported as mean ± standard deviation over 3 random seeds.

Token Weight wτ\boldsymbol{w}_{\boldsymbol{τ}}BL-4MTRRG-LPRF1
wτ=1w_{τ} = 10.198±0.0030.193±0.0060.336±0.0050.565±0.0060.517±0.0100.541 ± 0.008
wτ=κτw_{τ} = κ_{τ}0.206±0.0040.204±0.0060.343±0.0030.586±0.0050.524±0.0090.553 ± 0.003
wτ=φτw_{τ} = φ_{τ}0.208±0.0050.211±0.0040.353±0.0060.589±0.0080.534±0.0050.560 ± 0.006
Ours0.223±0.0040.223±0.0030.357±0.0040.602±0.0050.554±0.0070.577±0.006

Table R7. Comparison of different clinical evidence reasoning internalization methods on IU X-Ray.“*” indicates our re-implementations. Results are reported as mean ± standard deviation over 3 random seeds.

Evidence Internalization MethodsTraining ConditionsInference ConditionsBL-4MTRRG-LGREENRad-F1
BaseImage onlyImage only0.194±0.0040.200±0.0030.352±0.0050.583±0.0030.406±0.003
Evidence PromptingImage onlyImage+Ex+E_{x}^{+}0.219±0.0080.221±0.0030.377±0.0070.616±0.0030.425±0.006
Evidence-Augmented SFTImage+Ex+E_{x}^{+}Image only0.246±0.0050.224±0.0040.416±0.0050.617±0.0040.438±0.005
GRPOImage+Ex+E_{x}^{+}Image only0.227±0.0070.230±0.0050.391±0.0080.615±0.0080.434±0.008
CA-OPSD(Ours)Image+Ex+E_{x}^{+} + Ex−E_{x}^{−}Image only0.266±0.0050.239±0.0040.426±0.0050.633±0.0030.452±0.004

Response to Critical Weakness 4: Thank you for pointing out these inconsistencies. Upon verification, we have identified transcription errors as the source of these inconsistencies. The correct Recall on MIMIC-CXR is 0.554, and we have corrected the corresponding value in Table 3. Furthermore, the results for Base, Evidence-Augmented SFT, and CA-OPSD should be consistent between Tables 2 and 8. However, some metrics in the Base row of Table 2 contained transcription errors. We have corrected these inconsistencies, thoroughly verified the experimental results, and incorporated the corrections into the corresponding tables in the revised manuscript (Tables 4, 5, and 15, corresponding to the original Tables 2, 3, and 8, respectively), with the changes highlighted for clarity.

Reviewer_9Fq6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a two-stage framework to improve the clinical reliability of RRG. It first uses MCTS to explore different tool-use trajectories and applies counterfactual skill evolution to improve tool selection and search termination. The acquired clinical evidence is then transferred into the RRG model using curriculum-annealed on-policy self-distillation. Experiments on MIMIC-CXR and IU X-Ray show improvements over existing methods.

Critical strengths
  • The proposed method is interesting, particularly the use of counterfactual feedback to improve tool selection and search termination during evidence acquisition.

  • The idea of using the acquired clinical evidence to improve the RRG model is interesting. This allows the model to generate reports without requiring external tools during inference.

  • The paper presents extensive experiments on two datasets, with comparisons against multiple baselines and ablation studies. It also includes an evaluation of the acquired clinical evidence by five radiologists.

Critical weaknesses
  1. Limited clinical evaluation metrics: On MIMIC-CXR, clinical accuracy is evaluated only using CheXpert-based precision, recall, and F1. GREEN and RadGraph F1 are reported only on the smaller IU X-Ray dataset. Recent metrics such as GREEN, CRIMSON, and ICARE have shown better agreement with clinician evaluations. I think it would be useful to report these metrics on both datasets as well.

  2. Limited clinical validation: The paper includes an evaluation of the acquired clinical evidence by five radiologists on 100 cases, but the final generated reports are not evaluated. Since the paper aims to improve clinical reliability, I believe a direct evaluation of the generated reports by radiologists is important.

  3. Lack of multi-seed experiments: The paper does not report variability across different random seeds. I think reporting results from multiple runs, with mean and standard deviation, would help assess whether the improvements are consistent.

Questions for authors

I would be open to raising my rating if the authors can adequately address the concerns mentioned above.

11 comments
AuthorsOpenReview ↗

Author Response (Part 1/2): Critical Weakness 1

Response to Critical Weakness 1: Thank you for the constructive suggestion. To strengthen our clinical evaluation metrics, we have selected and reproduced representative SOTA methods with publicly available source code and have added GREEN, CRIMSON, and ICARE-AVG to the SOTA comparisons and component ablation studies on MIMIC-CXR and IU X-Ray. We have also conducted repeated experiments with three different random seeds for the corresponding evaluations. These results are shown in the tables below, and we have accordingly updated Tables 2, 3, 4, and 10 in the revised manuscript, with the corresponding revisions highlighted for clarity.

Table R1. Comparison with state-of-the-art methods on MIMIC-CXR. Results marked with * are reproduced using BLIP-3 as the backbone under the same data splits and preprocessing procedures as Ours. Results are reported as mean ± standard deviation over 3 random seeds.

MethodsBL-4MTRRG-LPRF1Rad-F1GREENCRIMSONICARE-AVG
AM-MRG*0.179±0.0050.185±0.0040.335±0.0060.541±0.100.423±0.0070.471±0.0080.297±0.0040.369±0.0050.164±0.0040.577±0.008
MLRG*0.196±0.0050.201±0.0030.347±0.0070.586±0.0070.544±0.0050.564±0.0060.310±0.0030.356±0.0020.179±0.0040.588±0.009
R2-LLM*0.172±0.0040.178±0.0040.332±0.0050.531±0.0060.403±0.0080.458±0.0070.298±0.0040.364±0.0050.157±0.0040.534±0.008
multiP-r2gen*0.184±0.0030.186±0.0040.343±0.0040.567±0.0050.424±0.0060.485±0.0050.307±0.0050.380±0.0060.176±0.0030.557±0.007
Ours0.223±0.0040.223±0.0030.357±0.0040.602±0.0050.554±0.0070.577±0.0060.328±0.0040.403±0.0030.192±0.0050.655±0.006

Table R2. Ablation study of different components on MIMIC-CXR. “Base” denotes the SFT RRG model. Results are reported as mean ± standard deviation over 3 random seeds.

BaseMCTSCSC-SkillEvo Stp\mathbb{S}_{tp}CSC-SkillEvo Sst\mathbb{S}_{st}CA-OPSDBL-4MTRRG-LPRF1Rad-F1GREENCRIMSONICARE-AVG
✓0.165±0.0030.172±0.0020.326±0.0050.506±0.0060.385±0.0050.437±0.0060.284±0.0020.354±0.0040.144±0.0030.503±0.007
✓✓0.177±0.0040.179±0.0030.332±0.0040.522±0.00100.398±0.0080.452±0.0080.289±0.0030.368±0.0030.156±0.0040.548±0.006
✓✓✓0.187±0.0050.190±0.0020.342±0.0040.547±0.0070.417±0.0050.473±0.0060.302±0.0050.375±0.0050.167±0.0040.582±0.008
✓✓✓0.183±0.0030.184±0.0030.337±0.0060.540±0.0090.402±0.0070.461±0.0080.297±0.0050.372±0.0040.164±0.0050.579±0.007
✓✓✓✓0.205±0.0050.201±0.0040.342±0.0050.573±0.0050.487±0.0060.527±0.0050.310±0.0030.380±0.0040.177±0.0050.613±0.009
✓✓✓✓✓0.223±0.0040.223±0.0030.357±0.0040.602±0.0050.554±0.0070.577±0.0060.328±0.0040.403±0.0030.192±0.0050.655±0.006

Table R3. Comparison with state-of-the-art methods on IU X-Ray. Results marked with * are reproduced using BLIP-3 as the backbone under the same data splits and preprocessing procedures as Ours. Results are reported as mean ± standard deviation over 3 random seeds.

MethodsBL-4MTRRG-LRad-F1GREENCRIMSONICARE-AVG
AM-MRG*0.213±0.0070.216±0.0050.384±0.0070.416±0.0060.599±0.0050.584±0.0060.671±0.10
MLRG*0.217±0.0040.220±0.0030.374±0.0050.422±0.0040.586±0.0030.549±0.0050.661±0.006
R2-LLM*0.218±0.0040.213±0.0040.375±0.0060.414±0.0050.594±0.0040.564±0.0070.653±0.008
multiP-r2gen*0.223±0.0060.221±0.0050.403±0.0050.422±0.0060.615±0.0040.597±0.0060.683±0.007
BLIP-3(Only SFT)0.194±0.0040.200±0.0030.352±0.0050.406±0.0030.583±0.0030.501±0.0060.617±0.007
Ours0.266±0.0050.239±0.0040.426±0.0050.452±0.0040.633±0.0030.655±0.0040.760±0.006

Table R4. Ablation study of different components on IU X-Ray. “Base” denotes the SFT RRG model. Results are reported as mean ± standard deviation over 3 random seeds.

BaseMCTSCSC-SkillEvo Stp\mathbb{S}_{tp}CSC-SkillEvo Sst\mathbb{S}_{st}CA-OPSDBL-4MTRRG-LRad-F1GREENCRIMSONICARE-AVG
✓0.194±0.0040.200±0.0030.352±0.0050.406±0.0030.583±0.0030.501±0.0060.617±0.007
✓✓0.216±0.0030.208±0.0030.377±0.0080.418±0.0030.603±0.0040.523±0.0050.664±0.009
✓✓✓0.238±0.0040.220±0.0020.395±0.0090.428±0.0040.612±0.0040.558±0.0070.682±0.011
✓✓✓0.237±0.0050.216±0.0040.393±0.0110.426±0.0030.608±0.0050.554±0.0060.680±0.008
✓✓✓✓0.246±0.0050.224±0.0040.416±0.0050.438±0.0050.617±0.0040.602±0.0050.705±0.007
✓✓✓✓✓0.266±0.0050.239±0.0040.426±0.0050.452±0.0040.633±0.0030.655±0.0040.760±0.006
AuthorsOpenReview ↗

Author Response (Part 2/2): Critical Weaknesses 2-3

Response to Critical Weakness 2: Thank you for the constructive suggestion. To further evaluate the clinical reliability of the final generated reports, we have conducted an additional radiologist evaluation in the component ablation study. Specifically, based on CheXbert annotations, we have selected 50 cases from each of the MIMIC-CXR and IU X-Ray test sets through stratified random sampling, totaling 100 cases and covering normal and abnormal cases as well as common and rare findings. Five radiologists have independently assessed the generated reports in terms of Report Faithfulness and Clinical Accuracy. Report Faithfulness has been assessed using a four-level scale: Fully Faithful=1, Minor Inconsistency=0.6, Moderate Inconsistency=0.3, and Major Inconsistency =0. Clinical Accuracy has been assessed using a five-level scale: Clinically Acceptable=1, Nearly Acceptable=0.75, Moderately Acceptable=0.5, Unacceptable=0.25, and Unusable=0. The results are shown in the table below, and we have added this radiologist evaluation to Appendix C.1.4 (Radiologist Evaluation of Generated Reports) of the revised manuscript, with the corresponding revisions highlighted for clarity.

Table R5. Ablation study of model components based on radiologist assessments of report faithfulness and clinical accuracy on 100 cases sampled from the MIMIC-CXR and IU X-Ray.

BaseMCTSCSC-SkillEvo Stp\mathbb{S}_{tp}CSC-SkillEvo Sst\mathbb{S}_{st}CA-OPSDFactual ConsistencyClinical Accuracy
✓0.5340.379
✓✓0.5570.394
✓✓✓0.6230.447
✓✓✓0.6200.442
✓✓✓✓0.6850.487
✓✓✓✓✓0.7720.557

Response to Critical Weakness 3: Thank you for the constructive suggestion. As detailed in our response to Critical Weakness 1, we have conducted additional experiments with three different random seeds for the SOTA comparisons and component ablation studies on both MIMIC-CXR and IU X-Ray, with the corresponding results presented in our previous response. Furthermore, we have also conducted three-seed evaluations of different Token Weight Strategies and Evidence Internalization Strategies, reporting the results as mean ± standard deviation (Mean ± Std), as shown in the tables below. We have accordingly updated Tables 5 and 15 in the revised manuscript, with the corresponding revisions highlighted for clarity.

Table R6. Ablation of token weight strategies for CA-OPSD on MIMIC-CXR. Results are reported as mean ± standard deviation over 3 random seeds.

Token Weight wτ\boldsymbol{w}_{\boldsymbol{τ}}BL-4MTRRG-LPRF1
wτ=1w_{τ} = 10.198±0.0030.193±0.0060.336±0.0050.565±0.0060.517±0.0100.541 ± 0.008
wτ=κτw_{τ} = κ_{τ}0.206±0.0040.204±0.0060.343±0.0030.586±0.0050.524±0.0090.553 ± 0.003
wτ=φτw_{τ} = φ_{τ}0.208±0.0050.211±0.0040.353±0.0060.589±0.0080.534±0.0050.560 ± 0.006
Ours0.223±0.0040.223±0.0030.357±0.0040.602±0.0050.554±0.0070.577±0.006

Table R7. Comparison of different clinical evidence reasoning internalization methods on IU X-Ray. “*” indicates our re-implementations.

Evidence Internalization MethodsTraining ConditionsInference ConditionsBL-4MTRRG-LGREENRad-F1
BaseImage onlyImage only0.194±0.0040.200±0.0030.352±0.0050.583±0.0030.406±0.003
Evidence PromptingImage onlyImage+ Ex+E_{x}^{+}0.219±0.0080.221±0.0030.377±0.0070.616±0.0030.425±0.006
Evidence-Augmented SFTImage+ Ex+E_{x}^{+}Image only0.246±0.0050.224±0.0040.416±0.0050.617±0.0040.438±0.005
GRPOImage+ Ex+E_{x}^{+}Image only0.227±0.0070.230±0.0050.391±0.0080.615±0.0080.434±0.008
CA-OPSD(Ours)Image+Ex+E_{x}^{+} + Ex−E_{x}^{−}Image only0.266±0.0050.239±0.0040.426±0.0050.633±0.0030.452±0.004

Dear Reviewer 9Fq6,

Thank you for your constructive feedback. We have carefully considered your comments and provided detailed responses in our rebuttal. As the discussion period draws to a close, we would greatly appreciate any further feedback, especially if you have any remaining concerns.

Best regards,
The Authors

Comments on the paper

1 comment

Dear Reviewers,

The authors have responded to your original comments. Could you please take a look and indicate whether the response resolves the concerns, and whether your score changes? Thank you very much.

Best,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.