acceptodds
Under review as a conference paper at ICLR 2027

EHRTemp: Learning Dynamic Clinical Decision Making from Longitudinal Electronic Health Records

Abstract

Clinical diagnosis requires physicians to acquire evidence and revise differential diagnoses as the patient’s condition evolves over time. Existing sequential diagnostic agents often interact with a fixed patient snapshot, leaving the effects of clinical progression on observations and evidence acquisition underexplored. We formalize dynamic clinical decision making (DCDM) and introduce EHRTemp, an interactive environment for studying this setting through longitudinal electronic health record replay. Within EHRTemp, we train EHRTemp-AGENT to update its diagnostic belief through differential diagnosis and select between laboratory test requests and final diagnosis. We initialize the agent with differential-diagnosis supervision and optimize its clinical decisions through reinforcement learning with By-Turn Process-Advantage Estimation (BT-PAE), which aligns process supervision with the clinical stage of each decision. On EHRSHOT, our 8B agent achieves the highest Sample F1, Micro F1, and diagnostic precision among the evaluated models, and remains competitive on an independent MIMIC-IV cohort.

Primary area:
foundation or frontier models, including LLMs
Keywords:
dynamic clinical decision making, longitudinal electronic health records, LLM agents, sequential diagnosis, multi-turn reinforcement learning, advantage estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x8ppOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work formulates clinical diagnosis as a dynamic clinical decision making process and then learn this dynamic process from longitudinal electronic health record (EHR). First, the partially observed Markov decision process is adopted to model the dynamic clinical decision making. Then, a reinforcement learning strategy is developed to fine-tune a large language model using longitudinal EHR. Experiments are conducted on two public datasets. One is used to evaluate in-distribution performance, while the other is for out-of-distribution validation. The proposed method is compared with both general-purpose LLMs and medical LLMs.

Critical strengths
  • In practice, clinical diagnosis is based on longitudinal electronic health records. This work formulates the problem as dynamic clinical decision making, adopts partially observable Markov decision process to model the dynamic clinical decision making, and develops a reinforcement learning approach to fine-tune LLMs. The technical contributions are novel and interesting.
  • The proposed method is compared with commercial and open-source models on two public datasets. One is used to evaluate in-distribution performance, while the other aims to validate out-of-distribution performance. Experimental results show its effectiveness for dynamic clinical decision making.
Critical weaknesses
  • Inconsistency between modelling and problem. The authors used partially observable Markov decision process to model the dynamic clinical decision making, but it is unclear which variables remain partially observed and what could be missed during a patient journey.

  • Limited applicability to real-world scenarios. Data missing is common in EHR. It is unclear how would the proposed method address this issue when the expected/requested laboratory evidence is missing. This paper has not established any solutions to solve this real-world challenge.

  • Uncertainty has not been considered yet. This work uses clinicians’ actions to construct supervision. However, a clinical diagnostic action exists uncertainty due to inter-observer variability. The task objective in Eq. (8) does not consider the uncertainty of reference diagnosis set.

  • The aim of this study is unclear. Is this study aimed to replace clinicians for decision making or assist decision making? If former, how to guarantee its reliability for prospective data? There is no evaluation on prospective cohorts, and the temporal generalizability of the proposed method is not validated. If latter, there is no experiment to show the proposed method can benefit clinicians under its assistance.

  • AI-generated content. Fig 1 and fig 2 are largely AI-generated. “Laboratory-agreement metrics measure agreement with recorded testing and timing, not clinical optimality. Ablations and behavioral analyses examine DDx-guided initialization, process-supervision design, evidence acquisition, and diagnostic-belief revision.” This has weak connection with the context and leads to much confusion.

Questions for authors
  • The motivation on adapting POMDP is unclear. Particularly, what variables are missing in EHR? How would the proposed method solve uncertainly missing data in clinical environment.

  • The applicability of this study has been convincingly demonstrated. Clinical environment is complex and uncertain. Inter-observer variability exists across clinicians. This kind of uncertainty has not been considered yet. How would the proposed method address this uncertainty.

  • The aim of this study is unclear. Is this study aimed to replace clinicians for decision making or assist decision making? Please justify the aim of this study.

5 comments
AuthorsOpenReview ↗

Response to Reviewer (1/4):

Response to Weakness 1: Partial observability and our POMDP formulation

First, we would like to distinguish three concepts that can otherwise be conflated. The first is the inherent partial observability of clinical diagnosis: a patient’s underlying clinical state cannot be observed directly, which motivates our POMDP formulation. The second is the partial visibility of evidence during interaction: the evidence an agent can access depends on its requests and the current clinical time; our dynamic setting further captures how evidence emerges and changes over clinical time. The third is missingness in the EHR data itself, where clinical events that occurred, or their results, were not fully recorded. The first two persist even without EHR missingness. We address the third in our response to Weakness 2. Below, we explain the first two concepts and their implementation in Section 3.1.

(1) Partial observability inherent in clinical diagnosis.

In practice, clinicians cannot directly observe a patient’s underlying clinical state. They assess it through two channels: routine updates received as the clinical course unfolds, such as newly recorded symptoms and vital signs; and actively acquired evidence, most notably through laboratory requests. These findings provide only partial evidence about the underlying clinical state, which remains incompletely observed even as individual findings become available. Clinicians use this evidence to update their diagnostic beliefs and decide what to do next. This process closely aligns with the POMDP framework of evidence acquisition, belief updating, and decision-making under partial observability.

Even if an EHR contained no missing entries and all recorded information were disclosed to the agent, the records would still constitute evidence about the patient’s underlying clinical state, rather than direct and complete observation of that state.

(2) What the dynamic setting adds.

In sequential diagnostic settings based on a fixed patient snapshot, interaction primarily determines which existing information is revealed. Our dynamic setting introduces clinical time, thereby changing how evidence is revealed: evidence that emerges at a later clinical time cannot be accessed in advance, and the patient’s underlying condition may evolve, so measurements of the same test may differ across clinical times. The agent must therefore not only acquire information through requests, but also update its diagnostic beliefs in response to new evidence and changes arising as clinical time advances.

(3) How these distinctions appear in Section 3.1.

In our formulation, the state represents the patient’s underlying clinical condition, while observations represent the available evidence, including symptoms, vital signs, and laboratory results. Observations comprise automatically released information and actively requested evidence. Laboratory results are disclosed subject to both the agent’s requests and clinical-time constraints, while newly recorded non-laboratory findings are automatically released as the trajectory advances.

At decision turn tt, the agent acts only on the interaction context ct−1c_{t-1} and the current observation oto_t, and never directly on sts_t. Three sources of partial observability can therefore be identified:

(i) The latent state. The patient’s underlying condition sts_t, including the diagnosis the agent must eventually commit to, is never observed directly.

(ii) Unrequested laboratory evidence. The laboratory evidence available at turn tt is limited to

Et={v(ℓ,τi)∣∃k<t: ℓ∈ak, τi≤τk}.E_t = \{ v(\ell, \tau_i) \mid \exists k < t:\ \ell \in a_k,\ \tau_i \le \tau_k \}.

Thus, a recorded measurement of any test ℓ∉⋃k<tak\ell \notin \bigcup_{k<t} a_k remains unobserved even though it exists in the record. The agent therefore observes only the laboratory evidence selected through its own requests and eligible under the clinical-time constraints.

(iii) Time-gated evidence. At response turn tt, laboratory measurements recorded after the preceding request time τt−1\tau_{t-1} are not returned, even for requested tests. Non-laboratory findings recorded after the current clinical time τt\tau_t are not yet released.

Component (i) corresponds to the partial observability inherent in clinical diagnosis and persists even if the full record is disclosed. Components (ii) and (iii) correspond to the partial visibility of evidence during interaction, reflecting the constraints imposed by active requests and clinical time, respectively.

AuthorsOpenReview ↗

Response to Reviewer (2/4):

Response to Weakness 2: Handling missing laboratory evidence

Thank you for raising the issue of missing laboratory evidence. When a requested laboratory result is absent from the EHR, our environment does not impute it. Imputed values would introduce evidence that is absent from the patient record, which EHRTemp is designed to avoid, and for tests that were never performed they would be counterfactual values that cannot be verified against the patient’s course. In our environment, a request returns the values of matching recorded laboratory events that are eligible at the request time; when no such event exists, the environment returns no value for that test. During RL, a request that matches an eligible, previously uncredited recorded laboratory event receives positive process feedback, and other requests receive negative process feedback under our record-based reward. We discuss three forms of missing laboratory evidence.

First, a laboratory event may be recorded without a result value. EHRSHOT is derived from OMOP-CDM records. Under the OMOP convention for the MEASUREMENT table, a measurement record without a result is interpreted as a test that was conducted but whose result was not captured [1]. We follow this convention. Because the process reward is defined on recorded laboratory events rather than on returned values (Section 3.2.2), a request matching such an eligible, previously uncredited event receives positive process feedback, while the environment returns no result value. The absence of a result value alone therefore does not cause a request to be penalized, and the agent proceeds with the evidence that is available.

Second, a test may have been performed in practice but not captured in the accessible EHR. We recognize this as an important challenge in EHR-based research. Such a test is indistinguishable in the record from one that was never performed, so the environment returns no value and the request receives negative process feedback. The existence and timing of recorded clinical events nevertheless carry clinically relevant information: Agniel et al. (2018) [2] showed that whether and when laboratory tests were ordered predicted patient outcomes. Such records provide a basis for studying our central question of how agents acquire evidence and revise diagnostic judgments as clinical time advances, and we leave explicit modeling of the distinction between unperformed and performed-but-unrecorded tests to future work.

Third, a clinically informative test may never have been performed. Its result would be counterfactual to the recorded trajectory, and our record-based replay environment cannot provide it without generating unobserved laboratory values. A clinically useful request can consequently receive negative process feedback when it is unsupported by the historical record, and the same applies to the second case. We therefore interpret the process reward as supervision for agreement with recorded testing, and we do not claim that the current experiments establish the clinical optimality of the learned policy.

References

[1] Observational Health Data Sciences and Informatics (OHDSI). OMOP Common Data Model v5.3: MEASUREMENT table. https://ohdsi.github.io/CommonDataModel/cdm53.html#measurement

[2] Agniel, D., Kohane, I. S., and Weber, G. M. Biases in electronic health record data due to processes within the healthcare system: retrospective observational study. BMJ, 361, k1479, 2018. https://doi.org/10.1136/bmj.k1479

AuthorsOpenReview ↗

Response to Reviewer (3/4):

Response to Weakness 3: Uncertainty in reference diagnoses due to inter-observer variability

Thank you for highlighting the uncertainty arising from inter-observer variability. Different clinicians may reach different diagnostic judgments for the same case. Our cohorts provide one recorded diagnosis set per encounter, without independent assessments of the same case by multiple clinicians. This data limitation also applies to other studies relying on routine EHR diagnoses without additional multi-rater annotations.

Diagnostic objective and inter-observer variability. Recorded diagnosis sets can be viewed as samples of clinicians’ judgments. Let p(y∣x)p(y \mid x) denote the distribution of diagnosis sets produced by the target clinician population for case xx under comparable information conditions. For a prediction y^\hat y fixed independently of the sampled reference, its expected diagnostic utility is

Uˉdiag(y^∣x)=EY∼p(⋅∣x)[Udiag(y^,Y)].\bar{U}_{\mathrm{diag}}(\hat{y} \mid x) = \mathbb{E}_{Y \sim p(\cdot \mid x)}[U_{\mathrm{diag}}(\hat{y},Y)].

If the recorded reference YobsY_{\mathrm{obs}} is a representative draw from this distribution, then

EYobs∼p(⋅∣x)[Udiag(y^,Yobs)]=Uˉdiag(y^∣x).\mathbb{E}_{Y_{\mathrm{obs}} \sim p(\cdot \mid x)}[U_{\mathrm{diag}}(\hat{y},Y_{\mathrm{obs}})] = \bar{U}_{\mathrm{diag}}(\hat{y} \mid x).

Under this assumption, the diagnostic utility computed against the recorded reference in Eq. (8) is a single-sample unbiased estimate of expected utility over clinicians’ judgments. This provides an interpretation of the recorded-reference objective in the presence of inter-observer variability.

This interpretation depends on the representativeness of the records. The composition of the treating clinicians, the information available when judgments are made, and the translation of clinical judgments into structured diagnosis codes can cause the recorded distribution to differ from that of the target clinician population. Cross-institutional evaluation is relevant because these factors may vary across institutions: it examines whether the model remains competitive against references produced under a different set of clinical and documentation practices. On the independent MIMIC-IV cohort, our method achieves the second-highest Sample F1 (0.315), outperforming all evaluated medical-specialist baselines and ranking below only DeepSeek-V4-Flash (0.362). This result provides empirical support for the method’s effectiveness against diagnostic references from a different data source.

Directly quantifying inter-observer variability would require multiple clinicians to independently assess the same cases under comparable information conditions. This is difficult with the available data: we use the released, de-identified EHR records and cannot access the source institutions’ complete original charts to obtain additional information for such assessments. Our current method therefore uses fixed recorded references without explicitly estimating a distribution of clinicians’ judgments for each encounter.

Label space and evaluation design. Inter-observer variability can manifest as differences in diagnostic specificity or in which diagnoses clinicians include in a reference set. We use 82 three-character ICD-10-CM categories, selected in consultation with a physician, so disagreements confined to finer subcategories map to the same label. Set-level F1 gives partial credit when predicted and recorded diagnosis sets overlap, while Recall@5 measures coverage of reference diagnoses among the top five candidates. These choices accommodate some manifestations of diagnostic disagreement through coarser labels, partial credit, and evaluation of multiple diagnostic candidates. Incorporating reference distributions derived from independent multi-clinician assessments into training and evaluation is a direction for future work.

Response to Weakness 4: Study aim and clinical applicability

Thank you for asking us to clarify the study’s intended scope. Our aim is methodological: to develop and evaluate diagnostic agents under evolving clinical evidence. Comparisons with other LLMs and diagnostic agents demonstrate the promise of our approach for this dynamic decision-making problem.

Substantial work remains before clinical deployment. Our retrospective experiments assess agent performance, and the MIMIC-IV evaluation examines cross-source transfer. Establishing reliability on future patient cohorts and benefits to clinicians will require prospective validation and clinician–AI interaction studies.

Reviewer_4PoDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies dynamic diagnosis from longitudinal EHRs, where clinical time advances between decisions and newly available evidence changes over the trajectory. The authors build EHRTEMP, a replay environment in which agents request laboratory tests or terminate with a diagnosis; requested tests reveal only measurements already present in the recorded EHR up to the request time, while new non-laboratory observations are released as time advances. EHRTEMP-AGENT initializes Qwen3-8B with supervised training on model-generated differential-diagnosis (DDx) labels and recorded clinical actions, then applies GRPO with diagnostic outcome reward and a laboratory process reward. The proposed BT-PAE normalizes process rewards within the same encounter and decision turn to reduce confounding from turn-dependent reward distributions. On EHRSHOT, the final model reaches 0.415 Sample F1 versus 0.379 for DDx-guided SFT and 0.406 for outcome-only GRPO; on MIMIC-IV it reaches 0.315, above its SFT checkpoint (0.299) but below DeepSeek-V4-Flash (0.362).

Critical strengths
  1. The paper introduces a concrete temporal replay environment for dynamic diagnosis from longitudinal EHRs. EHRTEMP makes clinical time part of the interaction: future laboratory measurements are unavailable, repeated measurements can change over time, and each non-terminal action advances the agent to the next recorded decision point. This is a meaningful departure from sequential diagnostic benchmarks that reveal information from a fixed patient snapshot.

  2. The paper develops an end-to-end training pipeline for this setting rather than only proposing an evaluation environment. The authors construct intermediate DDx supervision, fine-tune Qwen3-8B on DDx-plus-action trajectories, and then perform interactive RL in EHRTEMP using both terminal diagnostic reward and laboratory process reward. This yields a complete system for training and evaluating policies under the proposed temporal interaction protocol.

  3. BT-PAE addresses a specific optimization issue created by the temporal environment. Process-reward distributions vary strongly across decision turns, so the proposed estimator compares rollouts from the same encounter at the same turn rather than pooling rewards across clinical stages. The paper also introduces a pooled within-turn variance estimate to stabilize normalization when later turns contain fewer rollouts.

  4. The empirical study covers both component-level ablations and cross-source evaluation. The paper compares zero-/few-shot Qwen3-8B, supervised variants, multiple RL estimators, general-purpose LLMs, and medical models on EHRSHOT, and evaluates the same trained checkpoints on an independent MIMIC-IV cohort. This gives useful evidence about where gains arise and how well they transfer across data sources.

Critical weaknesses
  1. The diagnostic benefit on encounters requiring multiple laboratory decision turns is not yet clearly established. Only 565 of the 2,098 ID test encounters belong to the multi-turn subset. The dedicated analysis mainly reports laboratory behavior and relative DDx evolution, rather than final diagnostic performance for the competing training variants on this same fixed subset. On the full test set, BT-PAE improves Sample F1 by 0.009 over across-turn estimation, but the three-seed versus single-seed comparison does not establish statistically reliable superiority, as the authors acknowledge.

  2. The process reward measures agreement with recorded ordering behavior rather than the diagnostic utility of a test. A request receives positive credit when it matches an eligible, previously uncredited historical order; a clinically useful but historically unperformed test cannot return a result and receives negative process reward. This is a limitation of the explicitly observational replay setting, rather than a contradiction in its design. Nevertheless, improvements in process reward or Lab–Day F1 should not by themselves be interpreted as evidence of improved clinical test selection. The terminal diagnostic reward provides a separate outcome-based signal.

  3. The DDx ablation supports a change in diagnostic behavior, but its interpretation should remain limited. Adding DDx supervision improves Sample F1, precision, and completion, while reducing Micro F1 and Recall@5. All RL variants then start from the DDx-guided SFT checkpoint. The existing comparisons therefore measure RL gains conditional on this initialization, but do not establish that DDx-guided initialization is necessary for the final RL performance.

Questions for authors
  1. The DDx annotator is trained on 1,806 EHRSHOT encounters from 279 patients. Are these patients disjoint from the downstream agent validation and test patients? Please clarify how annotator training and SFT filtering relate to the downstream patient-level split.

  2. On the fixed 565 multi-turn ID encounters, can the authors report final diagnostic Sample F1 for DDx-SFT, outcome-only GRPO, across-turn estimation, and BT-PAE using the existing checkpoints? Paired patient-level bootstrap intervals for the differences would help assess whether the observed diagnostic gains persist on this subset. These intervals would quantify test-cohort uncertainty, rather than variability across training seeds.

2 comments
AuthorsOpenReview ↗

Response to Reviewer(1/2): Multi-Turn & Multi-Seed Diagnostic Performance and Statistical Analysis

Response to Weakness 1 & Question 2:

Thank you very much for highlighting the need to evaluate final diagnostic performance on the fixed multi-turn subset and assess the statistical reliability of the observed gains. Based on your suggestion, we have added the following experiments and analyses:

(1) Three-seed evaluation and statistical analysis on the full ID test set.

We extended outcome-only GRPO and across-turn estimation to three training seeds each (41, 42, and 43), matching BT-PAE, and conducted two-sided Welch’s t-tests on the run-level scores.

  • Compared with across-turn estimation, BT-PAE achieves significantly higher Sample F1, Micro F1, and precision (p = 0.016, 0.028, and 0.007, respectively).
  • Compared with outcome-only GRPO, it achieves higher mean values on these three diagnostic metrics and significantly higher Lab–Day F1 (p < 0.001).

Full results are reported in the newly added Appendix H.4, Multi-Seed Results and Statistical Analysis.

(2) Final diagnostic performance on the fixed multi-turn subset.

We evaluated DDx-guided SFT and the three RL variants on the same 565 multi-turn ID encounters. The following table reports results using the original checkpoints, with seed 42 for the RL variants.

Method (s42)Sample F1Micro F1PrecisionRecall@5Lab–Day F1
DDx-guided SFT0.41160.44110.48730.41340.4812
Outcome-only GRPO0.42950.45780.48390.45680.4988
Across-turn estimation0.42070.45050.48070.44340.4994
BT-PAE0.43080.46020.50050.43600.5158

We additionally report the mean and sample standard deviation across three training seeds for the RL variants. BT-PAE achieves the highest mean Sample F1, Micro F1, precision, and Lab–Day F1 among these variants.

MethodSample F1Micro F1PrecisionRecall@5Lab–Day F1
Outcome-only GRPO0.4338 ± 0.00400.4653 ± 0.00770.4966 ± 0.01100.4475 ± 0.00830.4957 ± 0.0041
Across-turn estimation0.4254 ± 0.00660.4576 ± 0.00650.4886 ± 0.00950.4392 ± 0.00420.5032 ± 0.0043
BT-PAE0.4368 ± 0.00920.4685 ± 0.01070.5068 ± 0.00880.4391 ± 0.00810.5158 ± 0.0006

(3) Paired patient-level bootstrap analysis on the multi-turn subset.

Following your suggestion, we performed paired patient-level bootstrap analysis with 10,000 resamples on the fixed multi-turn subset. We first averaged encounter-level scores or sufficient counts across the three seeds, then resampled patients using the same sampled patient IDs for both methods. These intervals quantify test-cohort uncertainty for the evaluated checkpoints.

  • BT-PAE vs. across-turn estimation: Mean Sample F1 improves by 0.0114 (95% CI: [0.0007, 0.0216]). The intervals for Micro F1 and precision also exclude zero. Lab–Day F1 improves by 0.0126 (95% CI: [0.0060, 0.0198]).
  • BT-PAE vs. outcome-only GRPO: The mean Sample F1 difference is +0.0030, with a 95% CI that includes zero ([−0.0079, 0.0143]). Lab–Day F1 improves by 0.0200 (95% CI: [0.0128, 0.0274]).

The diagnostic advantage over across-turn estimation is supported by both analyses, with the multi-turn analysis additionally showing a significant improvement in Lab–Day F1.

Together, these findings support the value of incorporating temporal context into process-advantage estimation for dynamic sequential diagnosis. By comparing rollouts from the same encounter at the same decision turn, BT-PAE accounts for turn-dependent differences in process rewards and improves both final diagnostic performance and agreement with recorded laboratory selection and timing relative to across-turn estimation on multi-turn encounters.

Compared with outcome-only GRPO, BT-PAE achieves higher mean Sample F1, Micro F1, and precision, but these diagnostic differences do not reach statistical significance in either the full-test-set Welch’s t-tests or the multi-turn patient-level bootstrap analysis. Its Lab–Day F1 improvement is significant in both analyses, supporting the benefit of process supervision for learning laboratory selection and timing patterns recorded in EHR trajectories.

We have added these analyses to Appendix H.4, including the paragraph Paired Patient-Level Bootstrap on Multi-Turn Encounters, and updated Section 4.2 to reflect the statistical findings. All corresponding changes are highlighted in blue in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer (2/2): Laboratory Metric Interpretation, DDx Supervision, and Patient-Level Separation

Response to Weakness 2:

We agree that Lab–Day F1 measures agreement with recorded clinical trajectories, and appreciate your recognition of the limitations of observational replay. We use it as a behavioral metric to understand how effectively an algorithm learns the patterns of laboratory selection and timing present in longitudinal EHR trajectories.

Examining this metric together with diagnostic performance provides further insight into what different algorithms learn. For example, BT-PAE and across-turn estimation use the same process reward, yet BT-PAE achieves significantly higher Sample F1, Micro F1, precision, and Lab–Day F1 on the fixed multi-turn subset in the paired patient-level bootstrap analysis. This joint improvement suggests that by-turn advantage estimation helps RL better learn the evidence-acquisition patterns embedded in dynamic clinical trajectories while also improving final diagnosis.

This is the role of Lab–Day F1 in our analysis: it helps characterize the learning behavior of the algorithm, with its interpretation grounded in recorded clinical practice, as specified in Section 4.1.

Response to Weakness 3:

Thank you for highlighting this distinction. Without a controlled comparison that removes DDx supervision during SFT while keeping the subsequent RL configuration unchanged, we cannot attribute gains in final RL performance to DDx-guided initialization.

We have revised the DDx-Guided Initialization paragraph in Section 4.2 to limit its conclusions to the effects observed at the SFT stage: improved Sample F1, precision, laboratory agreement, and completion, alongside lower Micro F1 and Recall@5. The revised text is highlighted in blue.

Response to Question 1:

Thank you for asking us to clarify the patient-level separation. Yes, the 279 patients used to train the DDx annotator are disjoint from all downstream agent training, validation, and test patients.

We first selected these 279 patients from cases with a single laboratory decision point and reserved their data for annotator training. All data from these patients were excluded from the downstream patient pool before trajectory construction and the patient-level data split. SFT filtering, which uses annotator outputs, is applied afterwards to the training partition only, while validation and test encounters are selected independently of annotator outputs (Appendix A.5).

We have added this clarification at the end of the Patient cases paragraph in Appendix C.2.

Reviewer_ryUJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EHRTemp, a longitudinal EHR replay environment for dynamic clinical decision making, and EHRTemp-Agent, an 8B agent trained with differential-diagnosis-guided supervision and By-Turn Process-Advantage Estimation. Unlike static sequential diagnosis settings, EHRTemp gates evidence by clinical time and allows repeated tests to return different recorded values as the trajectory advances. Experiments on EHRSHOT and an independent MIMIC-IV cohort show strong in-distribution performance and competitive cross-source transfer.

Critical strengths
  • The paper studies an important limitation of existing sequential diagnostic agents by modeling changes in both the agent's information and the patient's recorded clinical trajectory.
  • EHRTemp provides a temporally grounded interaction environment, while the explicit differential diagnosis and turn-specific process supervision offer an intuitive design for belief revision and evidence acquisition.
  • The evaluation includes patient-disjoint splits, an independent EHR cohort, diverse external models, same-base-model ablations, behavioral analyses, detailed training configurations, and supplementary code.
Critical weaknesses
  • The laboratory process reward and Lab-Day F1 measure agreement with recorded clinician orders rather than clinical utility. Since agent actions do not alter the recorded trajectory and counterfactual test outcomes are unavailable, the experiments do not show that the learned evidence-acquisition policy is clinically optimal.
  • The central benefit of BT-PAE is not statistically established. Its diagnostic improvement over outcome-only and across-turn alternatives is modest, while only BT-PAE is evaluated over three training seeds and the competing variants use one seed each.
  • The headline comparisons with external models do not control for task-specific training or interaction compatibility. EHRTemp-Agent receives supervised and reinforcement learning in the proposed environment, whereas the external systems are used without matched training and some perform poorly under the required interface.
Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2): Multi-Seed and Multi-Turn Evaluation with Statistical Analysis

Response to Weakness 2: Multi-seed evaluation of BT-PAE

Following your suggestion, we extended outcome-only GRPO and across-turn estimation to three training seeds each (41, 42, and 43), matching BT-PAE, and conducted two-sided Welch’s t-tests on the run-level scores.

BT-PAE achieves the highest mean Sample F1, Micro F1, precision, and Lab–Day F1 among the three variants (Table 1).

  • Compared with across-turn estimation, it significantly improves Sample F1, Micro F1, and precision (p = 0.016, 0.028, and 0.007, respectively; Table 2).
  • Compared with outcome-only GRPO, it significantly improves Lab–Day F1 (p < 0.001), while its higher mean Sample F1, Micro F1, and precision do not reach statistical significance (Table 3).

Table 1. Performance across three training seeds on the full ID test set (mean ± sample standard deviation).

MethodSample F1Micro F1PrecisionRecall@5Lab–Day F1
Outcome-only GRPO0.4090 ± 0.00290.4404 ± 0.00270.4377 ± 0.01110.4437 ± 0.01010.5950 ± 0.0022
Across-turn estimation0.4053 ± 0.00110.4362 ± 0.00080.4354 ± 0.00080.4397 ± 0.00440.6152 ± 0.0032
BT-PAE0.4145 ± 0.00260.4469 ± 0.00350.4523 ± 0.00300.4370 ± 0.00280.6170 ± 0.0014

Table 2. BT-PAE vs. across-turn estimation: two-sided Welch’s t-tests.

MetricMean differencetdfp
Sample F1+0.00915.642.630.016
Micro F1+0.01085.152.230.028
Precision+0.01699.542.290.007
Recall@5−0.0027−0.893.410.431
Lab–Day F1+0.00180.912.750.435

Table 3. BT-PAE vs. outcome-only GRPO: two-sided Welch’s t-tests.

MetricMean differencetdfp
Sample F1+0.00542.423.960.073
Micro F1+0.00652.533.750.069
Precision+0.01462.202.280.143
Recall@5−0.0067−1.112.310.370
Lab–Day F1+0.022014.733.43<0.001

Mean differences are BT-PAE minus the corresponding baseline. Bold p-values indicate p < 0.05.

We additionally evaluated the fixed subset of 565 multi-turn ID encounters using paired patient-level bootstrap analysis of results aggregated across the three seeds.

  • BT-PAE vs. across-turn estimation: Sample F1 improves by 0.0114 (95% CI: [0.0007, 0.0216]), with intervals entirely above zero also observed for Micro F1 and precision. Lab–Day F1 improves by 0.0126 (95% CI: [0.0060, 0.0198]).
  • BT-PAE vs. outcome-only GRPO: The diagnostic intervals include zero (Sample F1 difference: +0.0030; 95% CI: [−0.0079, 0.0143]), whereas Lab–Day F1 improves by 0.0200 (95% CI: [0.0128, 0.0274]).

Together, these findings support the value of incorporating temporal context into process-advantage estimation. BT-PAE compares rollouts from the same encounter at the same decision turn to account for turn-dependent differences in process rewards. Its diagnostic advantage over across-turn estimation is supported by both the full-test-set multi-seed analysis and the multi-turn subset analysis, with the latter additionally demonstrating improved agreement with recorded laboratory selection and timing.

Compared with outcome-only GRPO, BT-PAE achieves a higher three-seed mean Sample F1, although the difference does not reach statistical significance. Its significant improvement in Lab–Day F1 supports the benefit of process supervision for learning laboratory selection and timing patterns recorded in EHR trajectories.

We have added these results and statistical analyses to Appendix H.4 and revised Section 4.2 to reflect these findings. Changes to the main text are highlighted in blue in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer (2/2): Clinical Interpretation and Baseline Comparability

Response to Weakness 1:

Thank you for highlighting the distinction between agreement with recorded testing and clinical utility. As noted in Section 4.1, Lab–Day F1 measures agreement with recorded testing and does not establish clinical optimality.

As you note, establishing clinical optimality would require evaluating alternative testing strategies, including tests whose outcomes are absent from the records. Reliably estimating these patient- and time-specific outcomes is challenging and requires additional assumptions and validation. Similar to Hager et al. (2024) [1], we ground interactions in actual patient records, preserving fidelity to observed evidence while accepting this counterfactual limitation.

Within this setting, our experiments demonstrate improvements in diagnostic performance under temporal constraints compared with the evaluated baselines on the ID EHRSHOT benchmark, with same-base-model ablations assessing the contribution of our training design (see the additional multi-seed results in our response to Weakness 2). Behavioral analyses further show repeated testing and improving DDx rankings over successive turns, while the independent MIMIC-IV evaluation assesses cross-source transfer. Together, these results support progress in dynamic diagnosis, without equating agreement with historical orders with clinical optimality.

Response to Weakness 3:

We agree that EHRTemp-Agent receives task-specific SFT and RL, so comparisons with external models reflect differences in task adaptation as well as model capabilities. To reduce additional disparities arising from the label space, laboratory terminology, and inference interfaces, we implemented the following measures.

  1. Shared diagnostic label space. We provide all baselines with the same list of 82 candidate diagnosis categories in their task prompts.

  2. Flexible laboratory-name matching. As described in Appendix B.2, free-text laboratory requests are mapped to supported tests using fuzzy matching and SapBERT-based semantic similarity. This reduces sensitivity to differences in how models express synonymous laboratory-test names.

  3. Inference settings. For medical-specialist baselines, we used the sampling parameters specified in their released inference configurations, including temperature and top-p where applicable. We also followed model-specific inference requirements, including HuatuoGPT-3’s prescribed thinking mode.

  4. Targeted completion-rate probes. Before the main evaluation, we conducted targeted probes for EHR-R1 and Baichuan-M2, using an 80% completion threshold to guide configuration choices. EHR-R1 completed only 2/40 encounters under our task format, compared with 5/5 in a separate probe using its EHR-Bench Markdown format. We therefore adapted the clinical input to that format and removed the explicit DDx output requirement for EHR-R1. For Baichuan-M2, enabling reasoning increased completion from 65% to 80% on 20 encounters per setting, so we retained our task format with reasoning enabled. We have refined the descriptions of these probes and configuration choices in Appendix G.2, Baseline Invocation, for greater precision; the revisions are highlighted in blue.

  5. Additional interface evaluation for DoctorAgent-RL. Your concern about task comparability prompted us to re-examine the baseline configurations, revealing an interface compatibility issue for DoctorAgent-RL. Given its completion rate of only 45.9% on the full ID test set under our task format, we conducted an additional probe following the procedure described in Point 4. Using its original response structure yielded 95% completion in this probe. We then reran the full evaluation with the adapted protocol, keeping the clinical information and EHRTemp task rules unchanged. The updated results appear in Tables 1 and 5, with corresponding revisions to the comparisons, OOD evaluation, and behavioral analyses. Appendix G.2 documents this additional probe and the revised invocation protocol, while the new DoctorAgent-RL native mode paragraph in Appendix B.4 details the prompt adaptation. These revisions are highlighted in blue. The updated results do not change the conclusions reported in the paper.

For HuatuoGPT-3 and Baichuan-M2, the official usage documentation provides natural-language interaction examples without prescribing a dedicated clinical prompt or task-specific response schema. We therefore use our system prompt to specify the shared diagnostic task, while retaining their documented inference settings. The inference prompts for each model, including the adaptations to the native formats of EHR-R1 and DoctorAgent-RL, are documented in Appendix B.4.

References

[1] Hager, P., et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine, 30, 2613–2622, 2024. https://doi.org/10.1038/s41591-024-03097-1

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.