acceptodds
Withdrawn from ICLR 2027

Agentifying Patient Dynamics within LLMs through Interacting with Clinical World Model: A Case Study of Sepsis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sepsis management in the ICU requires sequential treatment decisions under rapidly evolving patient physiology. Although large language models (LLMs) encode broad clinical knowledge and can reason over guidelines, they are not inherently grounded in action-conditioned patient dynamics. We introduce SepsisAgent, a world model-augmented LLM agent for sepsis treatment recommendation. SepsisAgent uses a learned Clinical World Model to simulate patient responses under candidate fluid–vasopressor interventions, and follows a propose–simulate–refine workflow before committing to a prescription. We first show that world-model access alone yields inconsistent LLM decision performance, motivating agent-specific training. We then train SepsisAgent through a three-stage curriculum: patient-dynamics supervised fine-tuning, propose–simulate–refine behavior cloning, and world-model-based agentic reinforcement learning. On MIMIC-IV sepsis trajectories, SepsisAgent outperforms all traditional RL and LLM-based baselines in off-policy value while achieving the best safety profile under guideline adherence and unsafe-action metrics. Further analysis shows that repeated interaction with the Clinical World Model enables the agent to learn regularities in patient evolution, which remain useful even when simulator access is removed.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
LLM Agent, AI for health, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6JLROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose SepsisAgent, a LLM agent trained to query a world model before giving a treatment recommendation. They train this agent with a 3-step process to first add awareness of patient dynamics and guidelines, then achieve a propose-simulate-refine workflow, and finally use RL to inform its treatment recommendation. The authors demonstrate empirically that providing the LLM with a world model and clinical priors alone is insufficient for improving performance on treatment recommendation. They then show that their 3-step training process helps the model perform significantly better.

Critical strengths
  1. The experiments are set up well. The authors define a reward function that combines terminal outcome, intermediate physiology, and guideline consistency. They then measure off-policy evaluations of this reward, adherence to sepsis guidelines, and unsafe actions. They first demonstrate with 3 different LLMs that adding a world model and clinical priors is insufficient to improve performance and then demonstrate the efficacy of their 3-step training process. They also compare to results from both LLM-only and RL-only models. They also evaluate the effect of adding each stage of their training process.
  2. The paper is well-written. The motivation for interacting with a world model and the description of the 3-stage agent training process are clear. The prompts are provided in the appendix for reproducibility. The example reasoning trace provided in the appendix also helps with seeing how the model works.
Critical weaknesses
  1. I wonder if the clinical world model described in Section 3.1 is overly simplistic as next-step and final outcome prediction may be subpar compared to RL. If the LLM were given an RL model without training on how to use it, how would it perform?
  2. The authors switch from using o3, Gemini-3-Flash, and GPT4.1-mini to Qwen3-4B Instruct when comparing the world model only to their 3-step training process. This comparison does not seem fair as Qwen3-4B-Instruct performs worse than the other LLMs as shown in Table 3. The LLM used should be held constant for vanilla LLM, world-model augmented LLM, and Sepsis Agent. Why was the 3-step SepsisAgent process not tried with the better-performing LLMs? How does Qwen3-4B-Instruct perform with just the world model (and clinical prior)?
Questions for authors

In addition to the questions in the weaknesses section above:

  1. When you used RL to simulate trajectories, how many trajectories did you roll out? How did the model handle uncertainty / variance across trajectories?

  2. The example trace is great to see! The final response is not good though. If the patient received very high IV fluid over the past 4 hours, their condition is not considered stabilized sufficiently for discharge. What proportion of recommendations ended in an unsafe discharge while on fluids or vasopressors?

Reviewer_k2LdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SepsisAgent, a Qwen3-4B-Instruct agent for dosing fluids and vasopressors in sepsis. The agent follows a "propose–simulate–refine" loop: it proposes candidate actions, queries a learned Clinical World Model (a 2-layer GRU with a Gaussian next-state head and a 90-day mortality outcome head), and then prescribes a treatment. Training has three stages: Stage I: SFT on about 100k rationales generated by GPT-4.1, which analyse the state, predict in-hospital mortality (IHM) and vasopressor requirement (VR), and arrive at the clinician's action. Stage II: behaviour cloning on about 1k multi-turn traces of world-model interaction, synthesised by GPT-4.1, that end at the clinician's action. Stage III: GRPO in the world model, with a reward that combines terminal survival (±15), changes in SOFA and lactate, and a −10 penalty for guideline violations. Key experiments are conducted on 725 sampled test episodes.

Critical strengths
  1. Grounding an LLM's treatment reasoning in an action-conditioned patient model is a natural and interesting direction, and the finding that plain access to the world model does not reliably help general-purpose LLMs (Table 2) is a useful observation.
  2. Clinicians, three sepsis RL methods, six frontier and open LLMs, and world-model-augmented LLMs are compared on the same episodes using multiple OPE estimators.
  3. World-model architecture variants, cross-period (MIMIC-III) and cross-institution (eICU) evaluation, bootstrap CIs for the comparison with clinicians, and a clinician-validated failure analysis go beyond the minimum.
Critical weaknesses
  1. The claim of "outperforming all baselines" is contradicted by the paper's own tables, and the OPE evidence is weak. The abstract and the list of contributions state that SepsisAgent "outperforms all traditional RL and LLM-based baselines in off-policy value". However: MIMIC-IV: WD3QNE has a higher WIS, and the WPDIS values are almost identical; MIMIC-III: WD3QNE has a higher WPDIS; eICU: WD3QNE has a higher WIS. No tests are performed on the strongest baselines. Key OPE details are missing: How are the action probabilities π_e(a|s) defined for LLM policies that output a single action? How is the clinician behaviour policy π_b estimated? What are the effective sample sizes? With deterministic policies and 25 actions over about 12 steps, very few trajectories receive non-zero weight in WIS. Which reward is used for OPE? Does it include the −10 guideline penalty that SepsisAgent was trained on? The DR estimator uses a Q-function from the WD3QNE implementation, i.e., the value model of one of the baselines, which may bias the comparison in unknown ways.

  2. The claim of the "best safety profile" is not supported: the safety metrics ignore fluid overload, and the paper's own example demonstrates this harm. The unsafe-action rules cover only extreme underdosing and vasopressor overdosing. Excessive IV fluid, one of the main harms in sepsis care and one that the paper itself highlights, is never measured. The trace in the appendix shows exactly this harm. The patient starts with a MAP of 69.8 mmHg, a lactate of 2.6 and a urine output of 600 mL/4h, and is not in shock. SepsisAgent nonetheless prescribes Very High IV fluid at hour 0. At hour 48, with a MAP of 86.5 and a lactate of 1.2, it states that "ongoing aggressive fluid resuscitation is appropriate" and continues Very High fluids. This is clinically inappropriate under current guidelines. A proper safety assessment requires fluid-balance metrics (cumulative fluid, fluid given to patients without hypoperfusion) and clinician review of whole trajectories.

W3. Key baselines are missing, so the value of the LLM is unclear. A world model + classical planner (no LLM), e.g., MPC or random shooting over the 25 actions with the same world model and reward. If such a planner matches SepsisAgent, the LLM adds little. A small non-LLM policy trained with RL in the same world model (PPO/GRPO, or model-based RL such as MedDreamer, on which the world model is based). This would separate the contribution of "LLM + RL" from that of "RL in the simulator". Qwen3-4B trained with GRPO but without tool calls at inference, which would separate the benefit of RL training from that of test-time simulation. The LLM baselines are zero-shot, whereas SepsisAgent is trained on about 14k in-distribution ICU stays, so the comparison mainly shows that in-domain training helps.

W4. The novelty is limited. LLMs that plan with a world model (e.g., RAP, Hao et al., EMNLP 2023; WebDreamer, Gu et al., TMLR 2025), agentic tool-use RL with GRPO (e.g., Search-R1, Jin et al., 2025) and model-based RL for sepsis all already exist. The contribution is to apply this combination to sepsis dosing. That is reasonable, but without stronger evidence, its significance is limited.

W5. The discussion of limitations is inadequate, and several details are missing. The "Limitations" section is a two-sentence disclaimer. It does not discuss OPE validity, confounding, world-model fidelity, fluid safety, the discretisation of actions or the treatments that are left out (antibiotics, source control, steroids and ventilation). Unclear details: The units of the state MAE; The OPE reward and discount factor; How LLM policy probabilities are obtained; How the world model was used on eICU; Whether the code and generated data will be released (there is no reproducibility statement). Minor issues include the typo "Guidlines", the informal "sota", and a "Figure K" reference.

Questions for authors

Please see above.

Reviewer_QHwiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SepsisAgent combines an LLM and a GRU-based clinical world model for sequential fluid and vasopressor recommendations through a propose–simulate–refine workflow. Training uses supervised fine-tuning, behavior cloning, and world-model-based reinforcement learning with GRPO. Retrospective ICU evaluations report off-policy value, guideline adherence, safety metrics, and outcome prediction without simulator access. The claimed contribution is improved treatment decisions and internalized patient dynamics through world-model interaction.

Critical strengths

1 A meaningful problem and an intuitive workflow Sepsis treatment is an important sequential decision problem. The intuitive propose–simulate–refine workflow lets the LLM compare predicted treatment responses. 2 Useful comparisons across training stages and baselines RL and LLM baselines, staged ablations, external-dataset results, and expert review provide useful comparisons. They help distinguish training-stage effects from simply providing simulator access.

Critical weaknesses

1 The main policy-value claims rely on an incorrect DR estimator Sections 5.2–5.3 use DR results in Tables 3–4 to support SepsisAgent's treatment-value advantage and the RL stage's benefit. The estimator's mathematical error undermines this evidence and must be resolved before those results support the conclusions. Earlier importance weights are applied repeatedly. The evaluation specification defines each per-decision ratio as the product of all ratios up to that step, then inserts this cumulative quantity into a recursive DR estimator (Appendix G, pp. 22–23). The standard recursion uses the one-step ratio instead (Jiang and Li, 2016, Eq. 10). With two one-step ratios of 2 and 3, only the second reward equal to 1, no discounting, and zero value corrections, the correct contribution is 6; the stated recursion gives 12. If implemented as written, DR results in the main, ablation, and external-evaluation tables must be recomputed. If the implementation uses one-step ratios, provide the code and reconcile it with the reported method. The paired bootstrap in Appendix J addresses sampling uncertainty, not this estimator error. Behavior-policy fitting, LLM target-action probabilities, effective sample sizes, and weight concentration also need disclosure. Reliable retrospective evaluation also requires states that precede decisions. The main setup uses four-hour decision steps, but the preprocessing description combines within-window measurements and treatment aggregates without explicit observation cutoffs (Appendix C). Specify the decision timeline and verify strictly pre-decision inputs. The current description leaves temporal leakage unresolved; it does not establish that leakage occurred.

2 Counterfactual validity and clinical safety remain unestablished The main method uses simulated responses to choose alternative treatments and train long-horizon policies, yet Table 1 validates prediction only on observed trajectories. This does not establish reliable responses to alternative actions, especially outside historical support. Explicit causal assumptions, action-conditioned multi-step validation, and uncertainty diagnostics are needed because policies may exploit simulator errors. Physiological validity also fails in the showcased interaction: the Hour 4 update reports AST as −10.7 U/L (Appendix K, p. 29). This is invalid enzyme activity in the stated units. Explain its origin, quantify such violations, and test whether enforcing physiological bounds changes the policy or its reported gains. Section 5.2's safety interpretation also depends on the validity of its clinical supervision. The teacher prompt makes current MAP below 65 mmHg a necessary condition for septic shock, alongside vasopressor use and elevated lactate (Appendix F.1.1, Figure 4). Sepsis-3 instead concerns vasopressors required to maintain MAP at or above 65 mmHg, with lactate above 2 mmol/L despite adequate resuscitation (Singer et al., 2016). Pressure correction therefore does not exclude shock. Correct the prompt and assess its effect on supervision and safety results. The claimed avoidance of fluid overload needs direct support. At Hour 48, the showcased agent describes MAP 86.5 mmHg and lactate 1.2 mmol/L, yet recommends the highest fluid category using an early-resuscitation rationale (Appendix K, p. 30). Explain the actual dose, fluid balance, and fluid responsiveness supporting this recommendation in relation to Figure 7. The overdose metric tests high vasopressor use at high MAP, so it does not assess fluid overload. Finally, Section 5.4 interprets Table 4 as evidence of internalized patient dynamics, but Stage I directly supervises the same mortality and vasopressor-requirement prediction tasks. Their improvement supports predictive learning without simulator access, while leaving action-conditioned understanding unresolved. Controls matched for data, teacher reasoning, guideline conditioning, and training budget, together with controlled action-response tests and seed uncertainty, are needed to attribute the gains to world-model interaction.

Questions for authors
  1. Clarification: Does the DR implementation use cumulative or one-step ratios inside the recursion? Please provide the code and reconcile or recompute the affected results.
  2. Clarification: How are behavior and target-action probabilities obtained, and what weight diagnostics support OPE? Which state measurements are available before each decision?
  3. Validation: What evidence supports multi-step counterfactual reliability and physiological validity? How do the shock-prompt correction and the Hour 48 fluid recommendation affect the safety assessment?
  4. Attribution: Under matched training controls, what evidence separates action-conditioned dynamics understanding from improvement on the explicitly supervised prediction tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.