acceptodds
Withdrawn from ICLR 2027

MedAgentWorld: Privileged On-Policy Self-Distillation for World-Model-Augmented Clinical Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical large language models perform well on static case questions but are rarely evaluated on the sequential decisions of acute care. We present MedAgentWorld (MedWA), a world-model-augmented reinforcement learning framework for multi-turn clinical planning. We curate MedWA-118K and MedWA-Bench as time-ordered patient trajectories and build MedWA-Gym, where an agent acts on currently available information and receives replayed observations. Because historical records lack responses to plausible actions that were not performed, MedWA-WM predicts missing next observations. To make the world model robust to histories containing its own predictions, we introduce privileged on-policy self-distillation (OPSD): a student rolls out on its generated histories, while a teacher uses training-only access to the target observation, subsequent examination results, and final diagnosis to provide distillation targets. At inference, the student conditions only on the visible patient history and selected action. We train MedWA-Agent with supervised fine-tuning and reinforcement learning on replayed and world-model-generated branches. On MedWA-Bench, the agent reaches 44.92% diagnosis accuracy, compared with 42.73% without the world model, and MedWA-WM improves held-out next-observation prediction over prompted language-model baselines. These results support world-model-assisted learning for interactive clinical planning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent,World Models, OPSD, Reinforcement Learning.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3iV1OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MedAgentWorld, a hybrid replay-and-world-model environment for training clinical language agents on sequential emergency-care trajectories. The framework consists of MedWA-Gym for interactive rollouts, MedWA-WM for generating missing action feedback using privileged on-policy self-distillation (OPSD), and an SFT+GRPO agent trained over replayed and generated observations. The paper constructs MedWA-118K and MedWA-Bench from MC-MED, reports next-observation prediction results for the world model, evaluates transfer to MIMIC-IV-ED, and shows an agent-level improvement from 42.73% to 44.92% diagnosis accuracy when the world model is included.

Critical strengths
  1. The paper targets a meaningful limitation of static clinical QA: real clinical planning involves sequential evidence gathering under partial observability. The proposed routing principle, returning recorded feedback whenever it exists and invoking the learned model only for valid requests without recorded feedback, is a sensible attempt to preserve grounding in observed data.
  2. The paper gives a reasonably explicit formulation of the world-model prediction objective and the generalized JSD loss.
  3. The world-model results are substantial within the reported laboratory-prediction setting.
Critical weaknesses
  1. Figures 1, 2, and 3 show obvious signs of AI drawing. The text is filled with numerous low-quality AI-written elements, and I did not see any "AI use statement" in the paper. This is clearly a low-quality paper produced with AI assistance.
Questions for authors
  1. See "Weaknesses".
Reviewer_1DLiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MedAgentWorld (MedWA) for training LLM agents on multi-turn emergency-department workup. MC-MED encounters are converted into time-ordered trajectories (MedWA-118K for training and a 220-case MedWA-Bench). When the agent requests a bundle, MedWA-Gym replays the recorded result. For valid requests that lack a recorded result, MedWA-WM fills the gap; it is a Qwen3-4B model that generates the next observation as text. MedWA-WM is trained with SFT and then “privileged OPSD”: a frozen teacher that also sees the target observation, later exam results, and the final diagnosis supervises the student through token-level JSD on student-sampled prefixes. A Qwen3-4B agent is then trained with SFT and GRPO in the hybrid environment. Validation covers lab-value prediction on held-out MC-MED panels and 120 converted MIMIC-IV-ED cases against prompted LLMs, plus bundle F1 and diagnosis accuracy on MedWA-Bench. There, adding the world model raises accuracy from 42.73% to 44.92%.

Critical strengths
  1. A useful problem formulation and testbed. The paper recasts ED care as time-ordered request/feedback episodes in which recorded results always take priority over generated ones. This is a sensible way to move medical LLM evaluation from static case QA toward evidence gathering and deciding when to commit to a diagnosis. If the conversion pipeline and environment are released (within MC-MED’s access terms), others could use them to compare clinical agents. This is the main reason I see value in the submission.
  2. Candid scoping. The paper says plainly that: 1) generated observations have no clinician validation and no calibrated confidence; 2) cumulative error is unbounded; 3) the reward imitates historical practice rather than optimal care. This makes the claims easy to audit, although these admissions also narrow what the results can establish.
Critical weaknesses
  • Limited methodological novelty, and an overclaimed "world model."
    • "Privileged OPSD" is OPSD from Zhao et al. (2026), Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models, applied to lab values. That work already conditions the teacher on ground-truth answers, and its released code offers a fixed teacher. The JSDβ\mathrm{JSD}_\beta loss on student samples is GKD (Agarwal et al., ICLR 2024, On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes; uncited). The only new element is the content of ct+c_t^+. Since even the Base teacher sees the target oto_t, gains are small: MAE goes 9.18 (SFT) → 8.94 → 8.58/8.66 with no compute-matched SFT control, and the default (Future+Dx) loses to Future Hint on MAE, MAPE, S@10, and S@15.
    • Fig. 2 writes latent dynamics st+1=fϕ(st,at)s_{t+1}=f_\phi(s_t,a_t), but the model is a text generator pθ(ot∣ht,at)p_\theta(o_t \mid h_t,a_t) used for one-step fill-in, with no latent state, transition model, multi-step imagination, or planning.
    • The motivating counterfactual problem seems not to be what is implemented. Per Sec. 4.1 and App. B.1, requests must match the case's recorded requestable bundles, so the WM only fills recorded orders lacking a structured result (90 requests over 220 cases): missing-result imputation, not simulation of unobserved branches. Overall, an engineering integration of existing components.
  • The evaluation does not support the central claims.
    • The WM is scored only on recorded numeric lab panels, not on the likely non-random missing results it fills; imaging outputs (26/90 completions) are never evaluated, though Case 1's generated "reduced LVEF" points directly at the label. Uniform relative thresholds are clinically miscalibrated: under the headline S@25, sodium 105 vs. a true 140 counts as success. Clinical correctness (abnormal-flag agreement, analyte-specific tolerances, blinded clinician review) is the relevant test; even Fig. 3's illustrative teacher pushes B12 toward "normal" while citing pernicious anemia, which lowers B12.
    • Baselines are zero-shot LLMs against an in-distribution fine-tuned model. Missing: carry-forward, retrieval, and tabular WM baselines, fine-tuned peers, and the cited EHRWorld and MedDreamer; for the agent, no alternative training method on the same environment, e.g., LA-CDM (Bani-Harouni et al., ICLR 2026, Language Agents for Hypothesis-Driven Clinical Decision Making with Reinforcement Learning). Everything uses one Qwen3-4B backbone.
    • The 2.19-point agent gain is ≈5 of 220 cases, with no seeds, CI, or paired test; 44.92% is not a multiple of 1/220.
    • Data quality is unaudited: MedWA-118K restructures MC-MED without clinician review of bundle mapping or labels; some labels are symptom codes (e.g., hypoxia).
Questions for authors
  • [Clarification] Action space. Can the agent request any schema-valid bundle, or only bundles in the case's recorded requestable set (App. B.1)? How many WM calls involve tests that were never ordered? Is the requestable list shown to the agent? If so, it leaks the gold bundles (inflating F1) and diagnostic cues.
  • [Clarification] Test-time protocol and significance. Which agents receive WM completions at test time, including "SFT+GRPO w/o WM" and the prompted baselines? Please separate train-time from test-time WM effects (train with WM and evaluate without, and vice versa), report seeds, CIs, and a paired test, and explain how 44.92% arises from 220 cases.
  • [Clarification] Leakage. Are the cases used to train MedWA-WM with Future+Dx hints disjoint from the agent's RL cases and from MedWA-Bench? If not, diagnosis information can leak into generated observations.
  • [Clarification] Benchmark construction. How large is the diagnostic candidate set, what is chance accuracy, and how were the candidates, labels, and "quality score" defined? Was any subset audited by clinicians?
  • [Clarification] OPSD design. Why is Future+Dx the default when Future Hint wins four of seven metrics? How do compute-matched SFT and standard GKD with a stronger external teacher compare?
  • [Suggestion for revision] Clinical correctness and baselines. Evaluate generated observations by abnormal-flag agreement, analyte-specific tolerances, and blinded clinician rating (including imaging text), including on masked recorded panels whose missingness mimics the gaps the WM fills. Add simple WM baselines (carry-forward, disease-conditional mean, kNN retrieval) and one alternative agent-training method, such as LA-CDM, on MedWA-Gym.
Reviewer_fN4VOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MedAgentWorld (MedWA), a framework for training medical agents to make multi-turn clinical decisions under incomplete information. The authors construct MedWA-118K and MedWA-Bench from emergency EHRs and introduce MedWA-Gym, an interactive environment that replays recorded clinical observations. To address missing feedback for unobserved actions, they develop MedWA-WM, a world model trained with Privileged On-Policy Self-Distillation (OPSD). A fixed teacher leverages recorded observations, subsequent examination results, and final diagnoses to guide student training. MedWA-Agent is then optimized using SFT and GRPO with both recorded and world-model-generated feedback. Experiments demonstrate that MedWA-WM outperforms prompted LLM baselines in next-observation prediction, including on the out-of-distribution MIMIC-IV-ED dataset. On MedWA-Bench, MedWA-Agent achieves 44.92% diagnosis accuracy, compared with 42.73% without world-model feedback, alongside improvements in clinical action-matching F1 scores.

Critical strengths
  1. The research problem is practically meaningful and closely aligned with real-world clinical decision-making. This paper addresses multi-turn clinical decision-making, a research direction with significant practical relevance. Unlike traditional static medical question-answering tasks, real-world clinical practice requires physicians to progressively gather evidence, adjust their decisions, and reach diagnoses under incomplete information. Therefore, extending medical agents from static prediction to dynamic, sequential clinical decision-making is a meaningful research direction. Furthermore, the authors construct MedWA-Gym based on real EHR data, combining historical trajectory replay with world-model-generated feedback to provide a valuable interactive environment for studying this problem.

  2. The paper proposes a well-motivated world-model optimization approach supported by multi-level experimental validation. The authors introduce Privileged On-Policy Self-Distillation (OPSD) into clinical world-model training, leveraging subsequent examination results and final diagnoses to guide the student model on its self-generated response prefixes. This design is technically well-motivated. Experimental results demonstrate improvements in next-observation prediction, with consistent advantages on the out-of-distribution MIMIC-IV-ED dataset.

Critical weaknesses
  1. Limited methodological novelty and insufficient comparisons to justify the necessity of the proposed distillation strategy. The proposed Privileged OPSD largely follows the core mechanisms of existing On-Policy Self-Distillation, including a fixed teacher, student-generated prefixes, and token-level JSD optimization. The main modifications involve adapting this approach to clinical world modeling and incorporating subsequent examination results and final diagnoses as additional privileged information. However, the improvements over Base OPSD are relatively modest. Furthermore, the experiments only compare against standard SFT and different privileged information configurations, without direct comparisons to alternative approaches such as Off-Policy Distillation or teacher-generated SFT using stronger teacher models. These missing comparisons make it difficult to assess whether the improvements specifically arise from the proposed on-policy distillation strategy or could be achieved through simpler alternatives.

  2. Insufficient validation of world-model-generated feedback and its contribution to downstream clinical decision-making. A central component of this work is using the world model to generate missing clinical observations from EHR trajectories. However, while the authors evaluate laboratory value prediction, the quality of generated imaging feedback remains insufficiently validated. Additional evaluations on cases with available ground-truth imaging reports, using metrics such as BERTScore, clinical finding accuracy, or expert assessment, would help establish the reliability and clinical plausibility of the generated observations. Furthermore, incorporating the world model improves diagnosis accuracy only from 42.73% to 44.92% (+2.19 percentage points) on a relatively small benchmark of 220 cases. The paper also lacks multi-seed experiments or statistical significance analysis. Consequently, the current evidence is insufficient to establish the robustness and clinical significance of the downstream improvements brought by the world model.

Questions for authors

Q1. The original On-Policy Self-Distillation framework already employs a fixed teacher, student-generated prefixes, and token-level distribution matching. The main difference in this work appears to be the incorporation of subsequent examination results and final diagnoses as additional privileged information. Could the authors further clarify the substantive algorithmic novelty of the proposed method compared to the original OPSD? Additionally, could the authors provide comparisons with alternative approaches, such as Off-Policy Distillation or teacher-generated SFT, ideally under controlled settings with the same teacher and privileged information, to demonstrate the necessity of the on-policy mechanism?

Q2. The proposed world model is used to generate missing laboratory and imaging feedback, but the current quantitative evaluations primarily focus on laboratory value prediction. Could the authors provide an independent evaluation of the generated imaging feedback? For example, on test cases with available ground-truth imaging reports, metrics such as BERTScore or clinical expert assessments could be used to evaluate the accuracy and clinical plausibility of the generated feedback.

Q3. Incorporating the world model improves the agent's diagnosis accuracy from 42.73% to 44.92%, but the benchmark contains only 220 cases. Could the authors provide multi-seed experiments, confidence intervals, or paired statistical significance tests to assess the robustness of this improvement? Furthermore, given that the world model demonstrates out-of-distribution generalization, could the authors extend the downstream agent evaluation to additional clinical datasets or a larger test set to further validate the generalizability of the reported improvements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.