- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work formulates clinical diagnosis as a dynamic clinical decision making process and then learn this dynamic process from longitudinal electronic health record (EHR). First, the partially observed Markov decision process is adopted to model the dynamic clinical decision making. Then, a reinforcement learning strategy is developed to fine-tune a large language model using longitudinal EHR. Experiments are conducted on two public datasets. One is used to evaluate in-distribution performance, while the other is for out-of-distribution validation. The proposed method is compared with both general-purpose LLMs and medical LLMs.
Critical strengths
- In practice, clinical diagnosis is based on longitudinal electronic health records. This work formulates the problem as dynamic clinical decision making, adopts partially observable Markov decision process to model the dynamic clinical decision making, and develops a reinforcement learning approach to fine-tune LLMs. The technical contributions are novel and interesting.
- The proposed method is compared with commercial and open-source models on two public datasets. One is used to evaluate in-distribution performance, while the other aims to validate out-of-distribution performance. Experimental results show its effectiveness for dynamic clinical decision making.
Critical weaknesses
-
Inconsistency between modelling and problem. The authors used partially observable Markov decision process to model the dynamic clinical decision making, but it is unclear which variables remain partially observed and what could be missed during a patient journey.
-
Limited applicability to real-world scenarios. Data missing is common in EHR. It is unclear how would the proposed method address this issue when the expected/requested laboratory evidence is missing. This paper has not established any solutions to solve this real-world challenge.
-
Uncertainty has not been considered yet. This work uses clinicians’ actions to construct supervision. However, a clinical diagnostic action exists uncertainty due to inter-observer variability. The task objective in Eq. (8) does not consider the uncertainty of reference diagnosis set.
-
The aim of this study is unclear. Is this study aimed to replace clinicians for decision making or assist decision making? If former, how to guarantee its reliability for prospective data? There is no evaluation on prospective cohorts, and the temporal generalizability of the proposed method is not validated. If latter, there is no experiment to show the proposed method can benefit clinicians under its assistance.
-
AI-generated content. Fig 1 and fig 2 are largely AI-generated. “Laboratory-agreement metrics measure agreement with recorded testing and timing, not clinical optimality. Ablations and behavioral analyses examine DDx-guided initialization, process-supervision design, evidence acquisition, and diagnostic-belief revision.” This has weak connection with the context and leads to much confusion.
Questions for authors
-
The motivation on adapting POMDP is unclear. Particularly, what variables are missing in EHR? How would the proposed method solve uncertainly missing data in clinical environment.
-
The applicability of this study has been convincingly demonstrated. Clinical environment is complex and uncertain. Inter-observer variability exists across clinicians. This kind of uncertainty has not been considered yet. How would the proposed method address this uncertainty.
-
The aim of this study is unclear. Is this study aimed to replace clinicians for decision making or assist decision making? Please justify the aim of this study.
5 comments
Response to Reviewer (2/4):
Response to Weakness 2: Handling missing laboratory evidence
Thank you for raising the issue of missing laboratory evidence. When a requested laboratory result is absent from the EHR, our environment does not impute it. Imputed values would introduce evidence that is absent from the patient record, which EHRTemp is designed to avoid, and for tests that were never performed they would be counterfactual values that cannot be verified against the patient’s course. In our environment, a request returns the values of matching recorded laboratory events that are eligible at the request time; when no such event exists, the environment returns no value for that test. During RL, a request that matches an eligible, previously uncredited recorded laboratory event receives positive process feedback, and other requests receive negative process feedback under our record-based reward. We discuss three forms of missing laboratory evidence.
First, a laboratory event may be recorded without a result value. EHRSHOT is derived from OMOP-CDM records. Under the OMOP convention for the MEASUREMENT table, a measurement record without a result is interpreted as a test that was conducted but whose result was not captured [1]. We follow this convention. Because the process reward is defined on recorded laboratory events rather than on returned values (Section 3.2.2), a request matching such an eligible, previously uncredited event receives positive process feedback, while the environment returns no result value. The absence of a result value alone therefore does not cause a request to be penalized, and the agent proceeds with the evidence that is available.
Second, a test may have been performed in practice but not captured in the accessible EHR. We recognize this as an important challenge in EHR-based research. Such a test is indistinguishable in the record from one that was never performed, so the environment returns no value and the request receives negative process feedback. The existence and timing of recorded clinical events nevertheless carry clinically relevant information: Agniel et al. (2018) [2] showed that whether and when laboratory tests were ordered predicted patient outcomes. Such records provide a basis for studying our central question of how agents acquire evidence and revise diagnostic judgments as clinical time advances, and we leave explicit modeling of the distinction between unperformed and performed-but-unrecorded tests to future work.
Third, a clinically informative test may never have been performed. Its result would be counterfactual to the recorded trajectory, and our record-based replay environment cannot provide it without generating unobserved laboratory values. A clinically useful request can consequently receive negative process feedback when it is unsupported by the historical record, and the same applies to the second case. We therefore interpret the process reward as supervision for agreement with recorded testing, and we do not claim that the current experiments establish the clinical optimality of the learned policy.
References
[1] Observational Health Data Sciences and Informatics (OHDSI). OMOP Common Data Model v5.3: MEASUREMENT table. https://ohdsi.github.io/CommonDataModel/cdm53.html#measurement
[2] Agniel, D., Kohane, I. S., and Weber, G. M. Biases in electronic health record data due to processes within the healthcare system: retrospective observational study. BMJ, 361, k1479, 2018. https://doi.org/10.1136/bmj.k1479
Response to Reviewer (3/4):
Response to Weakness 3: Uncertainty in reference diagnoses due to inter-observer variability
Thank you for highlighting the uncertainty arising from inter-observer variability. Different clinicians may reach different diagnostic judgments for the same case. Our cohorts provide one recorded diagnosis set per encounter, without independent assessments of the same case by multiple clinicians. This data limitation also applies to other studies relying on routine EHR diagnoses without additional multi-rater annotations.
Diagnostic objective and inter-observer variability. Recorded diagnosis sets can be viewed as samples of clinicians’ judgments. Let denote the distribution of diagnosis sets produced by the target clinician population for case under comparable information conditions. For a prediction fixed independently of the sampled reference, its expected diagnostic utility is
If the recorded reference is a representative draw from this distribution, then
Under this assumption, the diagnostic utility computed against the recorded reference in Eq. (8) is a single-sample unbiased estimate of expected utility over clinicians’ judgments. This provides an interpretation of the recorded-reference objective in the presence of inter-observer variability.
This interpretation depends on the representativeness of the records. The composition of the treating clinicians, the information available when judgments are made, and the translation of clinical judgments into structured diagnosis codes can cause the recorded distribution to differ from that of the target clinician population. Cross-institutional evaluation is relevant because these factors may vary across institutions: it examines whether the model remains competitive against references produced under a different set of clinical and documentation practices. On the independent MIMIC-IV cohort, our method achieves the second-highest Sample F1 (0.315), outperforming all evaluated medical-specialist baselines and ranking below only DeepSeek-V4-Flash (0.362). This result provides empirical support for the method’s effectiveness against diagnostic references from a different data source.
Directly quantifying inter-observer variability would require multiple clinicians to independently assess the same cases under comparable information conditions. This is difficult with the available data: we use the released, de-identified EHR records and cannot access the source institutions’ complete original charts to obtain additional information for such assessments. Our current method therefore uses fixed recorded references without explicitly estimating a distribution of clinicians’ judgments for each encounter.
Label space and evaluation design. Inter-observer variability can manifest as differences in diagnostic specificity or in which diagnoses clinicians include in a reference set. We use 82 three-character ICD-10-CM categories, selected in consultation with a physician, so disagreements confined to finer subcategories map to the same label. Set-level F1 gives partial credit when predicted and recorded diagnosis sets overlap, while Recall@5 measures coverage of reference diagnoses among the top five candidates. These choices accommodate some manifestations of diagnostic disagreement through coarser labels, partial credit, and evaluation of multiple diagnostic candidates. Incorporating reference distributions derived from independent multi-clinician assessments into training and evaluation is a direction for future work.
Response to Weakness 4: Study aim and clinical applicability
Thank you for asking us to clarify the study’s intended scope. Our aim is methodological: to develop and evaluate diagnostic agents under evolving clinical evidence. Comparisons with other LLMs and diagnostic agents demonstrate the promise of our approach for this dynamic decision-making problem.
Substantial work remains before clinical deployment. Our retrospective experiments assess agent performance, and the MIMIC-IV evaluation examines cross-source transfer. Establishing reliability on future patient cohorts and benefits to clinicians will require prospective validation and clinician–AI interaction studies.
Response to Reviewer (1/4):
Response to Weakness 1: Partial observability and our POMDP formulation
First, we would like to distinguish three concepts that can otherwise be conflated. The first is the inherent partial observability of clinical diagnosis: a patient’s underlying clinical state cannot be observed directly, which motivates our POMDP formulation. The second is the partial visibility of evidence during interaction: the evidence an agent can access depends on its requests and the current clinical time; our dynamic setting further captures how evidence emerges and changes over clinical time. The third is missingness in the EHR data itself, where clinical events that occurred, or their results, were not fully recorded. The first two persist even without EHR missingness. We address the third in our response to Weakness 2. Below, we explain the first two concepts and their implementation in Section 3.1.
(1) Partial observability inherent in clinical diagnosis.
In practice, clinicians cannot directly observe a patient’s underlying clinical state. They assess it through two channels: routine updates received as the clinical course unfolds, such as newly recorded symptoms and vital signs; and actively acquired evidence, most notably through laboratory requests. These findings provide only partial evidence about the underlying clinical state, which remains incompletely observed even as individual findings become available. Clinicians use this evidence to update their diagnostic beliefs and decide what to do next. This process closely aligns with the POMDP framework of evidence acquisition, belief updating, and decision-making under partial observability.
Even if an EHR contained no missing entries and all recorded information were disclosed to the agent, the records would still constitute evidence about the patient’s underlying clinical state, rather than direct and complete observation of that state.
(2) What the dynamic setting adds.
In sequential diagnostic settings based on a fixed patient snapshot, interaction primarily determines which existing information is revealed. Our dynamic setting introduces clinical time, thereby changing how evidence is revealed: evidence that emerges at a later clinical time cannot be accessed in advance, and the patient’s underlying condition may evolve, so measurements of the same test may differ across clinical times. The agent must therefore not only acquire information through requests, but also update its diagnostic beliefs in response to new evidence and changes arising as clinical time advances.
(3) How these distinctions appear in Section 3.1.
In our formulation, the state represents the patient’s underlying clinical condition, while observations represent the available evidence, including symptoms, vital signs, and laboratory results. Observations comprise automatically released information and actively requested evidence. Laboratory results are disclosed subject to both the agent’s requests and clinical-time constraints, while newly recorded non-laboratory findings are automatically released as the trajectory advances.
At decision turn t, the agent acts only on the interaction context ct−1 and the current observation ot, and never directly on st. Three sources of partial observability can therefore be identified:
(i) The latent state. The patient’s underlying condition st, including the diagnosis the agent must eventually commit to, is never observed directly.
(ii) Unrequested laboratory evidence. The laboratory evidence available at turn t is limited to
Et={v(ℓ,τi)∣∃k<t: ℓ∈ak, τi≤τk}.
Thus, a recorded measurement of any test ℓ∈/⋃k<tak remains unobserved even though it exists in the record. The agent therefore observes only the laboratory evidence selected through its own requests and eligible under the clinical-time constraints.
(iii) Time-gated evidence. At response turn t, laboratory measurements recorded after the preceding request time τt−1 are not returned, even for requested tests. Non-laboratory findings recorded after the current clinical time τt are not yet released.
Component (i) corresponds to the partial observability inherent in clinical diagnosis and persists even if the full record is disclosed. Components (ii) and (iii) correspond to the partial visibility of evidence during interaction, reflecting the constraints imposed by active requests and clinical time, respectively.