acceptodds
Under review as a conference paper at ICLR 2027

EHRGym: A Diagnostic Benchmark for Offline Reinforcement Learning

Abstract

A low offline-RL score on clinical-style logs can reflect little room to improve, missing information, or failed learning. EHRGym separates these possibilities in one synthetic ventilator task with hand-specified responses and rewards. Its reference ladder runs from the exactly computed best constant setting to a hidden-state oracle. Planners that know the simulator probe what past settings, timing, and measurements can support. A clock-only plan follows a fixed schedule of settings indexed by time and uses no measurements. On varied logs, this plan reaches about a quarter of the way from floor to ceiling. DQN/FQI and MedDreamer see the full record yet stay near the floor. With hidden-state access, DQN/FQI reaches more than halfway, exposing difficulty learning from the partial record. Under repetitive logging, CQL and Decision Transformer often copy starting settings and score below their stochastic behavior policy. Researchers can use information-matched references to interpret low scores and inspect chosen actions for logging habits.

Primary area:
datasets and benchmarks
Keywords:
offline reinforcement learning, reinforcement learning in healthcare, electronic health records, partial observability, benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NVkaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

--After rebuttal--- Many thanks for the authors' rebuttal, which addresses most of my major concerns, and I therefore increase my score.

The paper focuses on why offline RL methods score poorly on clinical-style logs, considering three possible explanations: there is little room for improvement, the record lacks the necessary information, or learning fails. To separate these explanations, the authors build a single synthetic ventilator task with hand-authored hidden dynamics. Observations are generated from aggregate MIMIC-IV measurement rates and a rank-4 PPCA background.

Critical strengths

S1. The question is sound. Decomposing a low score into "no room to improve", "information missing" and "learning failed" is a useful way to think about clinical offline RL, and the paper formulates this well.

S2. Well-designed experiment.

S3. Code, a public API, planners, tests and fresh-environment receipts are provided.

Critical weaknesses

W1. A single hand-authored toy task is too narrow for a "benchmark", and the "EHR" component is weak. Only one profile ("Respiratory") is evaluated, although the name "EHRGym" and the phrase "each EHRGym profile…" suggest a suite. All conclusions, such as "learning from partially observed, observation-driven logs remains the open problem", are drawn from a single small POMDP (2,592 states, 12 actions, horizon 11).The components that determine the results are hand-authored rather than learned from data: the response probabilities (0.65/0.10/0.45/0.20/0.25), the stabilisation and critical-event rates (0.12/0.10), the timing rules, the initial-state distribution and the behaviour policy (App. B, I). MIMIC-IV supplies only measurement acquisition rates and a noise background. The link to real EHR data is therefore largely cosmetic, and the findings have limited clinical relevance. The authors acknowledge this, but it still reduces the paper's significance.

W2. The data-scaling experiment does not test data scaling. The training data grow from 16,384 to 262,144 episodes while the number of updates stays fixed (6,000 updates × batch size 512), so the larger datasets are seen for only a few epochs. The finding of "no detectable gain" from more data is therefore uninformative, and the authors concede that "further gains remain possible".

W3. The practical recommendations do not transfer to the real-world setting that motivates the paper. The abstract and Sec. 1 suggest that "researchers can use information-matched references to interpret low scores". However, these references require a known simulator. On real ICU logs (the motivating use case), exact floors, oracles, clock schedules and known-model planners cannot be computed. The paper should state clearly what a practitioner who has only real logs can take from EHRGym (e.g., action audits and reporting of behaviour support), and what remains possible only with a simulator.

W4. Closely related work is missing: The widely used small synthetic sepsis simulator with a hidden state and exact values ("Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models"), which is the most direct precedent for exact references in clinical-style synthetic tasks. Causal confusion in imitation learning ("Causal Confusion in Imitation Learning"). Offline hyperparameter and checkpoint selection. Partially observable RL benchmarks (e.g., "POPGym: Benchmarking Partially Observable Reinforcement Learning").

W5. The presentation should be improved. The main text is extremely dense, relies heavily on private jargon ("reference ladder", "information-matched", "lifted initial state", "shared-seed run" vs. "separate run", "evidence roles", "core-state projection") and hedges almost every sentence ("consistent with, not identified as"). As a result, the main message is hard to extract. Terms are used before they are defined: NS and M1/T1 appear in Fig. 1 before Eq. 2 and Sec. 3.2, and QMDP and POMCP are explained only later. Much essential content is relegated to more than 25 pages of appendices. The abstract is hard for a general reader to follow. A plain-language summary of the task, the three diagnostic questions and the three findings, together with one clear table, would help considerably.

Questions for authors

Refer to Critical Weaknesses. Thanks.

2 comments

We thank you for finding the question sound and the experiment well designed, and for recognizing the released code, planners and tests.

W1. Added tasks and sensitivity checks extend coverage and support the EHR connection. We agree that one synthetic task did not support the earlier breadth. We added acute kidney injury (AKI) and sepsis, and now use "tasks" rather than "profiles". On logs with diverse actions, the best learner means are 21.3 and 2.0 normalized score (NS) points, versus 43.7 and 23.9 NS for the open-loop schedules, fixed time-only plans found by search with the simulator (Table 3). A separate sweep runs AKI and sepsis in all eight measurement, timing and logging conditions, and the open-loop schedule is above every learner mean in all 16 conditions (Appendix P).

The findings persist when we vary responses and rewards, and the EHR connection lies in record properties. Responses, rewards and timing rules remain hand-specified, but learner means still trail the open-loop schedule across 23 response models and two retrained rewards (Appendix L). The record properties include delayed measurements, decisions kept in force for several hours and persistent actions. The findings also hold with observations calibrated to MIMIC-IV persistence and severity differences. Excess action repetition also appears on real MIMIC-IV Respiratory records: behavior cloning and conservative Q-learning repeat previous actions at 97.1% and 92.2%, versus 81.6% for clinicians (Appendix O).

W2. We withdrew the no-gain claim. Fixed updates hid gains from more data. We increased training updates from 6,000 to 96,000 to hold expected epochs fixed. With 16-fold more data, the fitted-Q learner (DQN/FQI) improves from -2.7 to 17.7 NS, though it still trails the open-loop schedule (Appendix G).

W3. We clarified which checks transfer to real logs. Researchers can fit a schedule from logged returns (a log-fitted schedule), audit copying of previous actions and behavior-policy support, and compare choosing the most likely action with sampling from the learned distribution, while value comparisons still require off-policy evaluation (OPE). Exact floors, ceilings, hidden-state controls and known-model planners require a simulator (Section 6).

W4. We added the requested related work. The revision covers Oberst and Sontag on hidden state and counterfactual evaluation with a Gumbel-Max sepsis simulator, de Haan et al. on causal confusion and previous-action removal, POPGym on online partially observable benchmarks, and Tang and Wiens, Paine et al., and Kumar et al. on offline policy selection (Section 2).

W5. We simplified the presentation. We rewrote the abstract and Introduction, replaced "reference ladder", "information-matched", "lifted initial state", "evidence roles" and "core-state projection" with plain descriptions, and renamed the clock schedule the open-loop schedule. NS is defined before its first use in the Introduction, and QMDP and POMCP are defined in Section 4. Table 3 provides one summary table for all three tasks, and the appendix is reorganized by topic.

We appreciate your detailed suggestions on task coverage and presentation. They led to broader experiments and a clearer account of what researchers can check on real logs. Have we addressed your main concerns? We would be glad to clarify any remaining points.

Thank you for reading our rebuttal and for raising your score. We've reflected these changes in the revised PDF. If any concerns remain, please let us know, and we will do our best to address them during the remaining discussion period.

Reviewer_6PDpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces EHRGym, a diagnostic benchmark for offline reinforcement learning in healthcare. The benchmark uses EHR-informed observations and a synthetic respiratory treatment task to evaluate RL algorithms under different measurement, decision-timing, and behavior-policy conditions. Experiments with several offline RL algorithms show substantial performance gaps relative to reference policies and repeated-action copying under persistent behavior-policy logging.

Critical strengths

The paper addresses a meaningful challenge in offline RL: understanding why an algorithm performs poorly when alternative treatment outcomes cannot be directly observed. The authors conduct controlled experiments and ablation studies to examine how measurement availability, decision timing, and behavior-policy persistence affect learning.

Critical weaknesses
  1. Limited clinical task coverage and insufficient generalizability. My main concern is the limited scope of the benchmark. Despite being presented as EHRGym, the evaluation focuses on a single respiratory treatment task involving PEEP and FiO₂ adjustments during mechanical ventilation. This is only one small part of ICU treatment, which itself represents a small portion of the much broader EHR domain. EHR data cover diverse diseases, clinical settings, treatment decisions, and longitudinal patient trajectories.

Although the authors evaluate eight experimental conditions, these are variations of the same synthetic task rather than distinct clinical treatment scenarios. I find it difficult to assess the generalizability of the proposed benchmark based on such a narrow evaluation.

  1. Insufficient validation of the simulator against real-world clinical data.

Another major concern is the lack of comprehensive validation of the simulator. While the authors use MIMIC-IV aggregates to inform the observation process, the patient-state transitions, treatment responses, outcomes, and reward functions are manually specified. I would expect more thorough comparisons between simulator-generated trajectories and real-world EHR records, especially since all reported results depend on the simulator.

Matching a single behavioral statistic is not sufficient to establish the realism of the simulated environment. Without stronger validation, it is difficult to determine whether the reported algorithm failures reflect challenges relevant to real clinical decision-making or are driven by the assumptions of this particular simulator. This substantially limits my confidence in the broader conclusions and the value of EHRGym as a clinical RL benchmark.

Questions for authors
  1. Have the authors compared the distributions of simulator-generated patient trajectories with real MIMIC-IV trajectories, including physiological measurements, treatment patterns, and clinical outcomes? If so, could these validation results be provided? If not, what evidence supports the clinical representativeness of the simulator beyond the aggregate measurement inputs and action-repetition comparison?

  2. How sensitive are the main findings and algorithm rankings to the manually specified treatment-response dynamics and reward functions? Have the authors evaluated whether the conclusions remain consistent under alternative, clinically plausible simulator configurations?

  3. Given that the reference planners know the true simulator dynamics while the offline RL algorithms only have access to logged data, how can the observed performance gaps distinguish insufficient information from limitations in learning the underlying dynamics?

4 comments

We thank you for recognizing the importance of diagnosing poor performance when unchosen treatment outcomes are unknown, and the controlled study of measurement, timing and logging.

Your central concern is whether the reported failures reflect real clinical challenges or the assumptions of this simulator. EHRGym is built for offline RL method developers. The diagnostic strength you credit requires known dynamics, because real logs cannot distinguish little room to improve, missing information and failed learning when outcomes under unchosen treatments are never observed. We therefore target realism in the record properties that make learning hard, namely delayed measurements, decisions kept in force for several hours and persistent logging, and we test whether the findings survive changes to observations, responses and rewards.

W1. We expanded task coverage beyond ventilation. We agree that one ventilation task was insufficient coverage. We added acute kidney injury (AKI) and sepsis, which differ in treatments and dynamics but share the EHR record properties that EHRGym studies. On varied logs, in which the behavior policy chooses diverse actions, the best learner means are 21.3 and 2.0 normalized score (NS) points, versus 43.7 and 23.9 NS for the open-loop schedules, fixed time-only plans found by search with the simulator (Table 3).

W2 and Q1. Responses remain hand-specified, and we added trajectory comparisons with real records. Yes, we compared simulated trajectories with 13,883 MIMIC-IV training episodes, including physiological measurements and action-change rates (Appendix N). Calibrated step-to-step persistence (lag-1 autocorrelation) is within 0.05 of real records for SpO2, FiO2, heart rate, mean blood pressure and respiratory rate, and simulated measurement medians fall within real severity-group ranges for all six Respiratory measurements. Treatment patterns differ: the simulated varied behavior policy increases FiO2 in 23.1% of four-hour steps, versus 5.5% in real records. The findings hold with calibrated observations and across 23 response models, supporting the diagnostic comparisons beyond the default configuration. The main learner failure exposed by the simulator, excess repetition, also appears on real MIMIC-IV Respiratory records, where behavior cloning (BC) and conservative Q-learning (CQL) repeat previous actions at 97.1% and 92.2% versus 81.6% for clinicians, and removing previous-action input reduces this repetition (Appendix O). We do not compare outcome rates because simulator events do not map unambiguously to recorded outcomes.

Why we do not fit responses to records. Clinical trial evidence guides the response structure, but the response magnitudes and rewards are hand-set. Fitting responses to records does not resolve this concern because sicker patients receive more support and still worsen (confounding by indication), so fitted associations cannot be treated as clinical treatment effects. For a setting matched to hidden severity, the fitted model gives almost no benefit: its improvement probability is 0.016, versus 0.65 by default.

Q2. The varied-log gap persists across responses and rewards. Learner means on varied logs still trail the open-loop schedule across 23 response models and two retrained rewards that emphasize critical events or treatment costs (Appendix L). The value cost of copying, which repeats the starting or previous action, depends on reward: higher treatment costs remove CQL's copying penalty.

Q3. We withdrew the partial-observability attribution. Planners show achievable value, not whether finite logs suffice to learn it. Given the exact belief over hidden states, a deep Q-network learner (DQN/FQI) scores 5.1 NS, versus 31.5 NS for a known-model planner that uses the same inputs (Figure 3). Missing information alone therefore cannot explain the gap. Model and value estimation from finite logs remain possible explanations.

Thank you again for asking for stronger comparisons with real trajectories and broader task coverage. Your questions led to the calibration checks and added tasks that most strengthened this revision. Have we sufficiently addressed your main concerns? Please let us know if further questions remain.

Since our first reply, we have added an analysis that addresses your Q1 and W2 on treatment patterns. We built a behavior policy that matches clinician action-change rates, and the learner gap persists on its logs.

Q1 and W2 (treatment patterns) Our earlier behavior policy did not match clinician treatment patterns, so we built one that matches their action-change rates. For the new policy, FiO2 decrease, hold, and increase rates per four-hour step were 7.1%, 90.1%, and 2.8%. For MIMIC-IV clinicians, they were 7.5%, 89.8%, and 2.7%. The new policy repeated the previous action in 81.8% of steps, versus 81.4% for clinicians.

Every learner mean still fell below the open-loop schedule, a fixed sequence of actions based only on elapsed time. The schedule scored 25.8 on the normalized score (NS) scale, where 0 is the best constant action and 100 is a policy that sees the hidden state. Across eight log replicates, each an independently generated set of logs with its own learner fits, the highest learner mean was 4.9 NS [-0.6, 10.4].

Correction to our first reply: the FiO2 comparison of 23.1% versus 5.5% counted action classes for the simulator and raw ventilator settings for clinicians. With action classes for both, the earlier policy raised FiO2 in 15.9% of steps versus 2.7% for clinicians.

Q2 (response settings) The learner gap also holds when the model follows the effect directions reported in ventilation trials. In that setting, the schedule scored 18.5 NS and the best learner scored 3.6 NS [-6.2, 13.4]. The gap was at least 14.9 NS in each of the 23 response configurations (Appendix L).

W1 (task coverage) The schedule also leads on acute kidney injury (AKI) and sepsis, the two other tasks in the revised paper. It scored above every learner mean in all 16 observation and timing conditions across those tasks (Appendix P.7).

The new analysis uses the main Respiratory setting and is in the new Appendix S of the updated PDF (Tables 26 and 27). The policy matches action-change rates but is not a model of clinician decisions.

Does this address your concern about how simulated treatment patterns compare with real records?

I thank the authors for their detailed responses and additional experimental efforts. However, some of my orignal concerns will be challenged to solve during the short rebuttal period. I really appreciate the authors' motivations for this work, but the work should be further strenghed in future.

  1. Expanding task from 1 to 3 has represented a substantial change to the original submission. I am concerned that such extensive revisions during the rebuttal period do not allow sufficient time for a thorough validation.
  2. My fundamental concern regarding simulator fidelity remains unresolved. Clinical outcomes and treatment-response dynamics remain unvalidated against real outcomes, and these represent two important components for a simulator. If the simulator hasn't convincingly and comprehensively validated its fiedelity, any observations, conclusions or algorithms build on it will not be trusted.

Thank you for the follow-up. We respond to your two remaining concerns in turn.

Expanding task from 1 to 3 has represented a substantial change to the original submission. I am concerned that such extensive revisions during the rebuttal period do not allow sufficient time for a thorough validation.

Each conclusion holds on Respiratory, the submitted task with the most checks against MIMIC-IV records. AKI and sepsis show that the same failures recur under different hand-set treatments and dynamics. In each, the best algorithm scores well below a fixed schedule that ignores patient measurements (Table 3). Both ran the same eight measurement, timing, and logging conditions as Respiratory, with three independent logs per condition. The paper will state which comparisons with records each task has.

Clinical outcomes and treatment-response dynamics remain unvalidated against real outcomes.

We agree, and the abstract and Introduction will say so. EHRGym is a synthetic test bed with known answers for diagnosing offline RL algorithms. We make no claim about patient benefit. Our claims about EHR data will cover only measurement rates, action-change rates, and action repetition, the properties we compared with MIMIC-IV.

We set treatment responses by hand on purpose, and the main comparison holds when we change them. Fitting responses to records mixes a treatment's effect with the illness that prompted it. Our own fit to MIMIC-IV Respiratory training records makes treatment look almost useless (improvement probability 0.016 against 0.65 in the default model, Table 14), and neither number is a validated effect size. In each of 23 hand-set configurations, and in one more guided by PEEP and oxygen-target trials, every algorithm's mean score stays below the fixed schedule (Appendix L).

The repetition failure we found in EHRGym also appears on real records. In EHRGym, BC and CQL repeat the previous action almost always because they rely on that input (Appendix S). Trained on MIMIC-IV records, they repeat it 97.1% and 92.2% of the time, versus 81.6% for clinicians, and removing the input reduces repetition there too (Appendix O).

Reviewer_VmMROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EHRGym, a benchmark for diagnosing why offline RL methods score poorly on clinical-style logs. It is built on one synthetic ventilator POMDP with 12 PEEP/FiO2 actions, at most 11 four-hour ticks, and hand-specified dynamics and rewards. MIMIC-IV aggregates set the measurement acquisition rates and the observation background. Learners are compared against a reference ladder of baselines with increasing information: an exact best-constant floor, an open-loop clock schedule, known-model planners that see either the action history alone or the full record, and an exact hidden-state oracle ceiling. All six learners score at or below the floor and below the clock schedule. The authors conclude that learning from partially observed logs is the open problem. Under persistent logging, CQL, IQL and Decision Transformer repeat the starting action and score below their behavior policy.

Critical strengths

S1. The floor and ceiling are computed exactly, and the oracle plans over the full horizon, rather than being myopic or estimated from samples.

S2. The paper includes a useful sanity baseline: a fixed schedule that, for each time step, uses the setting with the best average outcome in the logs (13.2 NS). It ignores patient measurements, yet scores above five of the six learners trained on the same logs (MedDreamer’s interval is too wide to compare). This makes clear that the learners underperform even a naive use of their data.

S3. Reporting selected-action frequencies and behavior-policy support alongside returns (Fig. 4b, App. K) is a useful diagnostic practice.

Critical weaknesses

W1. Closely related ventilation RL work is missing. PEEP/FiO2 control on MIMIC and eICU is a well-studied task: VentAI (Peine et al., 2021), DeepVent (Kondrup et al., 2023), Liu et al. (JMIR 2024). Most directly, Chang and Kuo (AAAI 2026) released a ventilator RL benchmark built from MIMIC-IV and eICU transitions, with fidelity checks against real data and seven offline RL baselines. None of these is cited. I would like the paper to explain what it offers over a data-grounded ventilator environment.

W2. The role of the MIMIC-IV observations is unclear. The background is fit to pooled MIMIC patients without reference to the simulator’s hidden states, and the hidden state enters the emitted values only through a “synthetic state shift,” an authored offset added to the emitted measurements (App. B). Varying how measurements are acquired (M0 vs M1) also has little effect on any learner (Table 21: at most about 4 NS, with four of six intervals spanning zero). If the emitted values carry little information, any sufficiently rich background should give the same results. The MIMIC component would then contribute realism of appearance only, and the name “EHRGym” would overstate the benchmark’s EHR content; a ventilator-specific title would be more accurate. If the emitted values do matter, their dependence on the hidden state should be calibrated so that, for example, the oxygenation values of a severe simulated patient resemble those of severe MIMIC patients rather than pooled patients plus a small offset.

W3. I am unsure what C2 adds beyond confirming a known effect. Under S1, the behavior policy repeats its previous action with persistence 0.75, and the previous-action input is initialized to action 0 in every episode. A greedy policy anchored to such logs would be expected to keep selecting action 0, as in the copycat problem (Wen et al., 2020), and this is what the paper observes. Sampled deployment shows no detectable deficit against the behavior policy in the tested controls, so the loss appears specific to deterministic deployment. Separately, both behavior policies score below the best constant action (−15.2 and −20.8 NS), so improvement over the behavior policy is a low bar, and learners that constrain toward logged actions are tied to a poor policy by construction. Results with a competent, protocol-based behavior policy would show whether C1 and C2 hold when the logs are generated from good care.

W4. I am not yet convinced that the learners’ failure is due to partial observability, as the paper concludes from C1. Three issues leave the attribution open. First, the belief control points elsewhere: DQN/FQI given the exact belief, the best available summary of the hidden state, reaches only 5 NS [−11, 21], while QMDP given the same belief reaches 38 NS (Tables 3, 4). Learners given the full hidden state also reach only 42 to 57 NS (Table 3). Both results suggest the difficulty lies in estimating action values from sparse logs of a poor behavior policy rather than in inferring the hidden state. Second, the planners show what the logged record can support given the true simulator equations, not what can be learned from 16,384 episodes of this behavior policy (as the authors note in Sec. 7), so the learner-planner gap may reflect logs insufficient to estimate the patient model rather than a failure of the learners. Third, the learners may be undertuned: DQN/FQI is unstable even with hidden-state access (Fig. 3), results use one optimization seed, and modest tuning moves CQL by up to 14 NS (Table 28). A baseline that estimates the model from the logs and then plans, together with multiple seeds, would help separate these explanations.

W5. C3 and the practical recommendations seem preliminary. The two full-record POMCP runs differ by about 6 NS, planner convergence is not established (App. D.3, E.2), and the measurement effect is significant only for QMDP, so the ordering of the middle rungs carries uncertainty comparable to the gaps being interpreted. The paper also recommends reporting information-matched references, but most of the ladder cannot be computed on real logs, where the learner’s score would itself be an off-policy estimate; the benchmark is well placed to test how far such estimates fall from the exact values (see Q4).

Questions for authors
  1. What does EHRGym diagnose that a data-grounded ventilator environment (Chang and Kuo, 2026) cannot?
  2. Do C1 to C3 hold across clinically sourced parameter ranges, and with a behavior policy that beats the clock schedule?
  3. What does a baseline that fits the model from logs and then plans score?
  4. How far are standard OPE estimates (FQE, WIS, DR) from the exact values across the eight conditions?
  5. How large is the authored severity shift relative to the background variance, is the background temporally correlated across ticks, and do the measurement values carry any information about the three patient traits?
3 comments
AuthorsOpenReview ↗

Response to Reviewer VmMR (1/2)

We thank you for recognizing the exact floor and oracle values, the log-fitted schedule, and the action-frequency and behavior-policy support reporting.

For the original claims C1 to C3 that your review discusses:

  • C1 (learner gap) now covers only varied logs, in which the behavior policy chooses diverse actions.
  • C2 (copying) is kept, and its value cost depends on reward and deployment.
  • C3 (information use) keeps the exact history-only result and drops the POMCP ordering.

W1 and Q1. Known dynamics enable exact diagnostic comparisons. EHRGym's dynamics are known, so we can compute exact policy values, evaluate planners that see only the learner's record, and check off-policy evaluation (OPE) estimates against exact values. We added VentAI, DeepVent, Liu et al., and Chang and Kuo, whose environment is fitted to recorded transitions and cannot provide these exact values (Appendix R).

W2 and Q5. Calibrated observations preserve the findings. In the default model, measured values carry little severity information, helping explain the small learner gains when changing the measurement condition. The oxygen-saturation shift is 0.127 background standard deviations per severity level versus 0.47 to 1.17 SD in real-record severity contrasts. Within-episode correlation is 0.11 versus 0.58 in MIMIC-IV. We therefore added observation models calibrated to MIMIC-IV. Under calibration, measurements raise the full-record QMDP planner's value by +11.3 NS over the history-only optimum, versus +8.5 NS by default. Learner means on varied logs still trail the open-loop schedule, and copying persists on logs that mostly repeat the previous action (Appendix M). The open-loop schedule is your clock schedule, the best time-only schedule found by search with the simulator. We keep the EHRGym name because the tasks share EHR record properties, and the added acute kidney injury (AKI) and sepsis tasks extend coverage beyond ventilation.

W3 and Q2. The protocol policy scores below the open-loop schedule, and a high-value behavior policy changes the comparison. The PEEP/FiO2-table protocol scores below the open-loop schedule because stepwise escalation undertreats hidden severity, so it is not a competent comparator here. We therefore also tested a high-value behavior policy, which infers hidden state from the record and chooses actions with known-model values. On its logs, a learner kept close to logged actions has a mean above the open-loop schedule in Respiratory and sepsis, while unconstrained learners fall far behind the behavior policy.

Beyond identifying repetition, we show that the value cost of copying depends on reward and deployment: sampling actions improves Respiratory returns, and higher treatment costs remove the copying penalty of conservative Q-learning (CQL).

Q2, parameter ranges. The ranges are hand-set, and the varied-log comparison holds across them. The parameter ranges were not clinically sourced. Across 23 hand-set response models and two retrained rewards, learner means on varied logs still trail the open-loop schedule.

AuthorsOpenReview ↗

Response to Reviewer VmMR (2/2)

W4 and Q3. Missing information alone does not explain the learning gap. We therefore withdrew the partial-observability attribution. Given the exact belief over hidden states, the fitted-Q learner (DQN/FQI) has a normalized score (NS) of 5.1, versus 31.5 NS (a sampled estimate) for a known-model planner that uses the same inputs. Planning with a response model fitted from the logs scores below the best constant action. With new optimization seeds, none of the 54 learner fits scores above the open-loop schedule. Selecting settings with simulator returns, which is optimistic, gives CQL a gain of 13.7 NS on repetitive logs, the gain you note. Yet tuned CQL on varied logs still scores -2.9 NS (Appendix J). Model and value estimation from finite logs remain possible explanations.

W5. We keep the history-only result and no longer claim a POMCP ordering. POMCP convergence remains unestablished. The bounds on the exact optimal value of a policy that sees only past actions and timing, 31.1 to 31.3 NS, are above the open-loop schedule at 25.8 NS. Timing and past actions therefore have value regardless of POMCP convergence (Appendix K). The sampled 31.5 NS estimate has interval [22.5, 40.5], containing the exact bounds. Only QMDP gains significantly from access to measurements, so we do not generalize that gain to POMCP. These references require a simulator, and Section 6 lists real-log checks.

Q4. OPE errors are large and change which policy is selected. Mean absolute errors are at least 5.6 NS for every estimator in every condition. Across the eight Respiratory conditions, errors against exact values are 12.1 to 24.1 NS for fitted Q-evaluation (FQE). They are 13.4 to 38.9 NS for weighted importance sampling (WIS) and 5.6 to 15.4 NS for doubly robust (DR) estimates. Exact values rank the open-loop schedule above every learner in all eight conditions. FQE selects the schedule in none of eight, DR in at most one of eight, and WIS in five of eight. When an estimator fails to rank the schedule first, its selected policy loses 23.6 to 39.0 NS relative to that schedule (Figure 3c).

Thank you again for pressing us on the learning attribution and the copycat comparison. These questions led to stronger controls and more precise claims. Have we addressed your main concerns? We welcome any remaining questions.

Since our first reply, we have added an analysis for your W3 and Q2: logs from a behavior policy built without simulator values. On these logs, the learner gap (C1) holds and learners still repeat the previous action far more than the behavior policy (C2). The repetition carries no value cost there: the copying learners score above the behavior policy. The policy matches how often MIMIC-IV clinicians change ventilator settings.

W3 and Q2 (behavior policy). The new policy is clinician-matched, not high-value: it scores -15.7 on the normalized score (NS) scale. That is below the best of the 12 constant actions, which defines 0 NS, and above 8 of them. Our first reply covers the protocol policy you asked about and a high-value policy built with simulator values. Together, the two replies cover a high-value policy and a clinician-matched one.

An offline learner trains only on logs. The open-loop schedule is a fixed time-only action sequence. On the NS scale, 0 is the best constant action and 100 is a policy that sees the hidden state. Brackets give 95% intervals across independently generated log sets with separate learner fits.

Learner gap (C1). Every learner mean is below the open-loop schedule across eight log replicates.

PolicyNS
Open-loop schedule25.8 (exact)
Highest learner mean (MedDreamer, model-based)4.9 [-0.6, 10.4]
DQN/FQI learner given the hidden state37.8 [34.0, 41.5]
Behavior policy-15.7 [-17.1, -14.3]

Given the hidden state as an input, the deep Q-network / fitted Q-iteration (DQN/FQI) learner reaches 37.8 NS on these logs. So the actions covered by these logs do not limit the learners.

Copying (C2). Conservative Q-learning (CQL) repeats the previous action at about 99% of decisions versus 68.0% for the behavior policy. CQL still scores 10.0 NS [6.9, 13.1] above the behavior policy. As you note, copying costs depend on the behavior policy.

The policy holds an action between decisions, so one decision can cover several four-hour steps. Counted per four-hour step, it keeps the same action in 81.8% of steps, and clinicians in 81.4%.

Removing the previous-action input (C2). Learners receive the previous action as an input feature. Retraining without it lowers repetition similarly on simulated and MIMIC-IV records.

LearnerRecordsWith the inputWithout the input
Behavior cloningMatched logs99.8%79.9%
Behavior cloningMIMIC-IV97.1%78.6%
CQLMatched logs98.9%75.6%
CQLMIMIC-IV92.2%73.4%

Simulated rows use the first three log replicates.

See the new Appendix S of the updated PDF (Tables 26 and 27). The policy matches only action-change rates. It is not a model of clinician decisions. Appendix S also reports off-policy evaluation on these logs (Table 28) and a controlled analysis of the learner's shortfall (Table 29).

Please let us know if any part of W3 or Q2 remains open.

Comments on the paper

1 comment

Purpose

We thank the reviewers and area chair for comments that improved the manuscript. EHRGym helps offline reinforcement learning developers diagnose poor performance before or alongside tests on real electronic health record logs. Known, hand-set dynamics let us compute achievable planner values and audit off-policy evaluation (OPE) against exact values.

What we withdrew

  • Withdrawn: the claim that partial observability explains low learner scores. Missing information alone does not explain them.
  • Withdrawn: the claim that more data does not help. Holding expected epochs fixed reveals gains from more data.
  • Restricted: the claim that learners trail the open-loop schedule (which reads only time) now covers varied logs (the behavior policy chooses diverse actions) only. The open-loop schedule is the best schedule found by search with the simulator. High-value logs come from a behavior policy that infers hidden state from the record and chooses actions with known-model values. On high-value logs, a behavior-constrained learner (one that stays close to logged actions) has a mean above the schedule in Respiratory and sepsis.

What we added

  • Acute kidney injury (AKI) and sepsis, extending coverage across three tasks.
  • An OPE audit against exact simulator values.
  • A real MIMIC-IV check: learners repeat actions more than clinicians, and removing previous-action input reduces repetition.

On varied logs, learners trail the open-loop schedule in every task. On repetitive logs, whose behavior policy mostly repeats its previous action, learners mostly copy the starting action. Learners use only the logged record, while the full-record planner, a known-model planner that also sees measurements, shows achievable value.

QuantityRespiratoryAKISepsis
Open-loop schedule, NS25.843.723.9
Full-record planner, NS35.352.434.0
Best learner mean on varied logs, NS3.121.32.0
DQN/FQI gain when given the hidden state, NS+58.1+38.0+46.6
Repetitive logs: starting-action share, CQL / DT99.5% / 99.9%98.8% / 97.7%96.5% / 98.4%
Repetitive logs: CQL minus behavior policy, NS-8.5-6.3-54.1

NS (normalized score): best constant action = 0 and hidden-state oracle = 100 within each task. DQN/FQI: deep Q-network or fitted Q-iteration. CQL: conservative Q-learning. DT: Decision Transformer. Starting-action share: share of decisions that repeat the starting action (Table 3).

Copying lowers value relative to the behavior policy in our main comparisons under the default reward.

Revised manuscript

We uploaded a revised manuscript, rewrote the abstract and Introduction, added a three-panel evidence figure (Figure 3), and reorganized the appendix by topic, with several result tables now shown as figures. Each per-reviewer reply names the section, figure or appendix where the change can be checked. We would be glad to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.