acceptodds
Withdrawn from ICLR 2027

TrustMed-RL: Long-Horizon Reinforcement Learning for Evidence-Grounded Clinical Diagnosis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical language models can produce correct diagnoses despite incomplete investigations and unsupported reasoning. To support long-horizon, evidence-grounded diagnosis, we introduce TrustMed-RL. Built from PubMed rare-disease cases and over 24,000 manually annotated image panels, it integrates interviews, examinations, testing, specialist consultation, and literature search through state-dependent actions. Our 8B vision–language policy, trained with clinically adapted GiGPO and coverage-adjusted diagnostic rewards, achieves 37.1% diagnostic accuracy on 2,500 evaluation cases, outperforming all evaluated open-weight baselines and improving over supervised fine-tuning by 12.4%. When success additionally requires acquiring at least 50% of supporting test evidence, TrustMed-RL achieves 32.5%, exceeding GPT-4o by 6.8%. Furthermore, it surpasses all evaluated baselines on MTMedDialog and multiple larger 27–32B models on AgentClinic. In physician review of 200 diagnostically accepted test-set trajectories, 83.0% receive evidential-grounding scores of 4–5 out of 5. Physicians' assessments suggest that these diagnostic trajectories are trustworthy and aligned with human diagnostic reasoning.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
long horizon agent, clinical agent, trustworthiness, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_Rx36OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces an interactive clinical environment for history taking, testing, specialist consultation, and literature search. The authors apply reinforcement learning after supervised fine-tuning and adjust diagnostic rewards based on evidence coverage. Evaluation includes 2,500 cases, external benchmarks, ablations, and physician review. The trained 8B model shows gains in diagnostic accuracy and evidence acquisition.

Critical strengths

1.The environment is a useful research contribution. It separates test ordering, image access, and report interpretation. This makes it possible to examine what evidence the agent obtains and which actions it takes. The curated cases and image annotations also provide a foundation for future work. 2.The gains on the same 8B backbone are clear. Diagnostic accuracy rises from 24.7% to 37.1%, while strict accuracy improves from 9.0% to 15.3%. These results support the effectiveness of the full training pipeline.

Critical weaknesses

1.The method builds on GiGPO and an existing interaction curriculum. Yet two central choices—the coverage-adjusted reward and grouping by clinical state—lack direct ablations. These components are described in lines 178–186 and 198–205, but Table 7 does not isolate their effects. The results support the full pipeline, but they do not establish how much these specific additions contribute. 2.The evidence for a more trustworthy diagnostic process remains limited. Physician review covers only trajectories accepted by the automated evaluator, with no corresponding review of the SFT model (lines 1738–1746). This supports the quality of some successful trajectories, but it does not establish that RL improves evidence use. There is also an inconsistency in the example. The paper states that post-treatment findings are withheld (lines 827–828), yet the trajectory receives and uses imaging changes observed four months after steroid treatment (lines 2603–2605 and 3165–3167). The authors should explain this discrepancy and establish whether it affects the main evaluation. Its scope is currently unclear.

Questions for authors

1.The paper states that post-treatment findings are withheld, but the example uses imaging results obtained four months after steroid treatment. What explains this discrepancy? Does the same issue occur in the training or test data used for the reported experiments? 2.How do the results change when the coverage adjustment is removed or the clinical state grouping is replaced, with initialization, interaction budgets, and training compute held fixed? Do any existing experiments isolate the contribution of these two components? 3.Is there a blinded physician comparison of the RL and SFT models on the same cases? Such a comparison would help establish whether RL improves evidence use. Reviewing successful trajectories alone does not answer this question. 4.Lines 321–323 refer to validating GPT-5-mini but using GPT-5.1-mini for evaluation. Which judge produced the main results? Does the validation in the appendix use the same model and scoring configuration? 5. In Table 6, some standard deviations are close to 49, while others are around 1. Are these statistics computed at the same level?

Reviewer_vpazOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Based on rare disease cases from PubMed, this paper constructs an OSCE-style multimodal interactive diagnostic environment, featuring over 24,000 panel-level manually annotated images, atomized facts, and probes for image integrity. Building upon the GiGPO framework, the proposed method incorporates ''evidence-coverage-adjusted diagnostic rewards'', selective reinforcement of correct diagnostic trajectories, and a progressive interactive budget curriculum. The 8B-parameter model achieves an accuracy of 37.1% across 2,500 evaluation cases, and outperforms several 27–32B models on three OOD interactive benchmarks.

Critical strengths
  • S1. The environment is designed to be traceable and auditable at every step of the diagnostic process. Panel-level manual annotation, separation of patient self-report and examination findings, a PMCID-disjoint interference image library, and ontology-based diagnostic matching enable the recording and scoring of each intermediate action and its evidence source. This is a valuable infrastructure contribution.

  • S2. The reward design directly constrains evidence acquisition. Diagnostic scores are weighted by the coverage of acquired evidence, and only trajectories with correct diagnoses are reinforced. This prevents the model from receiving high rewards for guessing correctly when evidence is insufficient.

  • S3. Ablation and generalization experiments are thorough. Table 7 presents the ablation results for eight variants and reports the 95% CI after Bonferroni correction. Table 8 analyzes the behavior in the later stages of training. The three OOD benchmarks in Table 6 support a certain degree of cross-distribution generalization.

Critical weaknesses
  • W1. The physician evaluation claim regarding ''trustworthiness'' suffers from selection bias. The 200 physician review trajectories were selected only from cases automatically judged as ''correct or substantially correct'', excluding approximately 63% of cases not judged as correct, the most crucial sample for verifying trustworthiness. Without evaluation of failed trajectories, the ''trustworthy'' claim in the title lacks valid evidence.

  • W2. The training reward and evaluation judge belong to the same OpenAI model family, and the evaluation judge has not been validated. Training rewards use GPT-4.1-mini, and evaluation uses GPT-5.1-mini. However, Table 10 shows validation of GPT-5-mini, not GPT-5.1-mini. Existing ontology consistency checks only have N=100 participants, and physician-involved validation has only N=139 participants, insufficient to rule out the possibility of the strategy catering to the preferences of family-specific judges.

  • W3. Absolute performance lacks a reference, and the reward hyperparameters also lack sensitivity analysis. The 37.1% Acc and 13.5% Strict@50 rates lacked controls with randomization, always-give-up, or human doctors, making their practical implications difficult to determine. The coefficients, hyperparameters, and the nine end-game reward rules in Table 3 were all manually set, but the paper did not report a sensitivity analysis.

Questions for authors
  • Q1. Is it possible to conduct physician reviews on the same number of trajectories with incorrect diagnoses and compare the grounding scores of the correct and incorrect groups?
  • Q2. Table 1 and Table 4 contain almost identical content, but appear in the main text and appendix respectively, which is confusing.
Reviewer_7jwcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a long-horizon reinforcement learning framework for evidence-grounded clinical diagnosis named TrustMed-RL. Specifically, TrustMed-RL consists of three core components: the state-dependent clinical interaction, the evidence-conditioned policy optimization, and the outcome-adaptive interaction curriculum. The state-dependent clinical interaction framework supports multistage diagnostic encounters and traceable evidence acquisition. The evidence-conditioned policy optimization promotes accurate diagnoses under coverage-adjusted rewards via clinically adapted GiGPO. Finally, the outcome-adaptive interaction curriculum framework adjusts case sampling under progressively expanded interaction budgets. The experimental evaluation in this paper assesses the effectiveness of TrustMed-RL across four benchmark datasets. Results demonstrate that TrustMed-RL achieves higher diagnostic accuracy while improving evidence acquisition, outperforming several larger open-weight medical model baselines.

Critical strengths
  • The idea of constructing an interactive environment that releases clinical evidence in response to actions, and then using rewards tied to evidence coverage to train a doctor agent to complete the entire diagnostic pathway is considered novel.
  • The paper includes the construction of an environment, which contributes to the research community.
Critical weaknesses
  • In the authors’ design of the evidence-conditioned diagnosis reward in Sec. 5, the main diagnostic reward directly incentivizes a correct final diagnosis and the acquisition of more reports. However, the evidence-grounded diagnosis emphasized in the paper also involves correctly interpreting the findings in the reports, using these findings to support or rule out candidate diseases, and revising conclusions when contradictory evidence emerges; yet these aspects are not reflected in the paper. If two trajectories have the same final diagnosis, report coverage, length, and action validity, the current reward cannot distinguish between them.
  • Meanwhile, defining evidence sufficiency as report coverage may encourage redundant evidence acquisition. The denominator of coverage is the total number of available image-bearing evidence records in a case, and the reward favors acquiring more existing records rather than directly favoring the acquisition of a set of evidence sufficient to support the diagnosis. Furthermore, budgets and discounting cannot eliminate this issue. The current reward permits, and even encourages evidence acquisition that provides no additional diagnostic value.
  • In the design of the abstention reward, some cases are labeled as OOK through offline probing. However, in regular diagnostic tasks, an incorrect diagnosis can receive a higher reward than abstention. One possible scenario is a regular answer task with no image corruption, where some workup has already been performed, termination is allowed, and no formatting or other penalties are triggered. In this case, the reward for an incorrect diagnosis is 0, whereas the reward for abstention is −0.05. Thus, even if the evidence in the current trajectory is insufficient and the model is highly uncertain, the reward within this task category favors guessing a diagnosis over abstaining. This contradicts the paper’s goal of learning reliable diagnostic commitment and appropriate deferral to human clinicians.
  • The authors aim to improve the entire diagnostic investigation process through long-horizon RL, but on cases where the model most needs incremental exploration, improvements in the investigation become visible only after the final diagnosis crosses the diagnostic threshold. In other words, regardless of how much the evidence acquisition process improves, as long as the final diagnosis does not pass the threshold, the reward signal for progress in evidence acquisition on difficult cases is entirely zeroed out.
Questions for authors
  • Why can clinical anchors provide more useful step-level credit for clinical decisions? Do their benefits come from more appropriate local comparisons, or merely from regrouping and rescaling terminal rewards?
  • A POMDP allows hidden states to influence rewards, which is mathematically valid. However, if the evidence already acquired is sufficient to support the diagnosis, why should the model still be penalized for not triggering a prespecified anomaly that has not yet formed part of the basis for its diagnosis?
Reviewer_AxhUOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TrustMed-RL, an interactive, evidence-based clinical diagnostic framework for long-horizon reasoning. The authors leverage large language models (LLMs) to transform clinical cases from the RareArena dataset into interactive environments, enabling clinical actions such as patient questioning, ordering diagnostic tests, and requesting specialist consultations. Building on this environment, the author trained a Doctor Agent through SFT and clinically adapted GiGPO, incorporating evidence-conditioned rewards, step-level credit assignment, adaptive sampling, and curriculum learning. The method achieves 37.1% diagnostic accuracy and 32.5% Acc@50 on 2,500 test cases, outperforming the evaluated open-weight baselines. Furthermore, the appendix provides additional supporting results, including out-of-distribution (OOD) test performance, ablation studies, an assessment of evidence completeness, and evaluations conducted by physicians.

Critical strengths
  1. Well-motivated problem formulation and rigorous methodology This paper identifies a key limitation in existing clinical agents: a correct diagnostic outcome does not necessarily imply that sufficient evidence has been gathered. To address this, the authors propose a framework that integrates an interactive clinical environment, an evidence-based reward mechanism, and long-horizon reinforcement learning (RL), enabling the joint optimization of diagnostic accuracy and the evidence-gathering process.

  2. Comprehensive empirical evaluation This paper rigorously evaluate the agentic framework against diverse baselines, physician reviews, and out-of-distribution tests, examining both diagnostic accuracy and evidence completeness

Critical weaknesses
  1. The gap between evidence coverage and evidence grounding The authors primarily reward evidence acquisition via report coverage without demonstrating that these examinations are clinically necessary or that the final diagnosis genuinely relies on the gathered evidence.

  2. Overly Permissive Diagnostic Accuracy The accuracy metric accepts "CORE Match," tolerating errors in critical attributes such as disease subtype, etiology, or severity. GiGPO achieves an Accuracy of 37.1%, whereas its Strict Accuracy drops to only 15.3%(Table 1), indicating that the primary conclusions are notably sensitive to how diagnostic correctness is defined.

Questions for authors
  1. Under clinical setting, test reports are typically already accessible to the physician. Why is a “Specialist Routing” procedure still needed to obtain these reports? What additional diagnostic capability or decision-making process is this mechanism intended to emulate? What is the original intention behind this mechanism?

  2. The environment explicitly returns unavailable when requested examinations are absent from the record. In a medical context, "unavailable" is also informative (potential information leakage). This may lead the model to develop shortcuts, where it requests more "unavailable" responses to rule out certain diseases, especially under rare diseases setting. Could the authors report the frequency of such responses and analyze the relationship between unavailable examination requests and diagnostic accuracy? Please also clarify whether unavailable requests are included in the reported "Invalid" in Table 2.

  3. Does the reward system encourage excessive clinical exams? GiGPO increases workup turns while productive actions increase only modestly compared with SFT (10.2->17.8 and 7.7->8.5). Could the author provide detailed analysis regarding these additional steps?

  4. The paper claims iterative self-improvement under section 4, but the training process appears to consist mainly of standard RL policy updates and outcome-adaptive task sampling. Could the authors clarify whether these two components constitute the claimed iterative self-improvement mechanism? Whether an additional self-improvement mechanism (e.g. failure trajectory refinement) is implemented?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.