- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces an interactive clinical environment for history taking, testing, specialist consultation, and literature search. The authors apply reinforcement learning after supervised fine-tuning and adjust diagnostic rewards based on evidence coverage. Evaluation includes 2,500 cases, external benchmarks, ablations, and physician review. The trained 8B model shows gains in diagnostic accuracy and evidence acquisition.
Critical strengths
1.The environment is a useful research contribution. It separates test ordering, image access, and report interpretation. This makes it possible to examine what evidence the agent obtains and which actions it takes. The curated cases and image annotations also provide a foundation for future work. 2.The gains on the same 8B backbone are clear. Diagnostic accuracy rises from 24.7% to 37.1%, while strict accuracy improves from 9.0% to 15.3%. These results support the effectiveness of the full training pipeline.
Critical weaknesses
1.The method builds on GiGPO and an existing interaction curriculum. Yet two central choices—the coverage-adjusted reward and grouping by clinical state—lack direct ablations. These components are described in lines 178–186 and 198–205, but Table 7 does not isolate their effects. The results support the full pipeline, but they do not establish how much these specific additions contribute. 2.The evidence for a more trustworthy diagnostic process remains limited. Physician review covers only trajectories accepted by the automated evaluator, with no corresponding review of the SFT model (lines 1738–1746). This supports the quality of some successful trajectories, but it does not establish that RL improves evidence use. There is also an inconsistency in the example. The paper states that post-treatment findings are withheld (lines 827–828), yet the trajectory receives and uses imaging changes observed four months after steroid treatment (lines 2603–2605 and 3165–3167). The authors should explain this discrepancy and establish whether it affects the main evaluation. Its scope is currently unclear.
Questions for authors
1.The paper states that post-treatment findings are withheld, but the example uses imaging results obtained four months after steroid treatment. What explains this discrepancy? Does the same issue occur in the training or test data used for the reported experiments? 2.How do the results change when the coverage adjustment is removed or the clinical state grouping is replaced, with initialization, interaction budgets, and training compute held fixed? Do any existing experiments isolate the contribution of these two components? 3.Is there a blinded physician comparison of the RL and SFT models on the same cases? Such a comparison would help establish whether RL improves evidence use. Reviewing successful trajectories alone does not answer this question. 4.Lines 321–323 refer to validating GPT-5-mini but using GPT-5.1-mini for evaluation. Which judge produced the main results? Does the validation in the appendix use the same model and scoring configuration? 5. In Table 6, some standard deviations are close to 49, while others are around 1. Are these statistics computed at the same level?