- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
--After rebuttal--- Many thanks for the authors' rebuttal, which addresses most of my major concerns, and I therefore increase my score.
The paper focuses on why offline RL methods score poorly on clinical-style logs, considering three possible explanations: there is little room for improvement, the record lacks the necessary information, or learning fails. To separate these explanations, the authors build a single synthetic ventilator task with hand-authored hidden dynamics. Observations are generated from aggregate MIMIC-IV measurement rates and a rank-4 PPCA background.
Critical strengths
S1. The question is sound. Decomposing a low score into "no room to improve", "information missing" and "learning failed" is a useful way to think about clinical offline RL, and the paper formulates this well.
S2. Well-designed experiment.
S3. Code, a public API, planners, tests and fresh-environment receipts are provided.
Critical weaknesses
W1. A single hand-authored toy task is too narrow for a "benchmark", and the "EHR" component is weak. Only one profile ("Respiratory") is evaluated, although the name "EHRGym" and the phrase "each EHRGym profile…" suggest a suite. All conclusions, such as "learning from partially observed, observation-driven logs remains the open problem", are drawn from a single small POMDP (2,592 states, 12 actions, horizon 11).The components that determine the results are hand-authored rather than learned from data: the response probabilities (0.65/0.10/0.45/0.20/0.25), the stabilisation and critical-event rates (0.12/0.10), the timing rules, the initial-state distribution and the behaviour policy (App. B, I). MIMIC-IV supplies only measurement acquisition rates and a noise background. The link to real EHR data is therefore largely cosmetic, and the findings have limited clinical relevance. The authors acknowledge this, but it still reduces the paper's significance.
W2. The data-scaling experiment does not test data scaling. The training data grow from 16,384 to 262,144 episodes while the number of updates stays fixed (6,000 updates × batch size 512), so the larger datasets are seen for only a few epochs. The finding of "no detectable gain" from more data is therefore uninformative, and the authors concede that "further gains remain possible".
W3. The practical recommendations do not transfer to the real-world setting that motivates the paper. The abstract and Sec. 1 suggest that "researchers can use information-matched references to interpret low scores". However, these references require a known simulator. On real ICU logs (the motivating use case), exact floors, oracles, clock schedules and known-model planners cannot be computed. The paper should state clearly what a practitioner who has only real logs can take from EHRGym (e.g., action audits and reporting of behaviour support), and what remains possible only with a simulator.
W4. Closely related work is missing: The widely used small synthetic sepsis simulator with a hidden state and exact values ("Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models"), which is the most direct precedent for exact references in clinical-style synthetic tasks. Causal confusion in imitation learning ("Causal Confusion in Imitation Learning"). Offline hyperparameter and checkpoint selection. Partially observable RL benchmarks (e.g., "POPGym: Benchmarking Partially Observable Reinforcement Learning").
W5. The presentation should be improved. The main text is extremely dense, relies heavily on private jargon ("reference ladder", "information-matched", "lifted initial state", "shared-seed run" vs. "separate run", "evidence roles", "core-state projection") and hedges almost every sentence ("consistent with, not identified as"). As a result, the main message is hard to extract. Terms are used before they are defined: NS and M1/T1 appear in Fig. 1 before Eq. 2 and Sec. 3.2, and QMDP and POMCP are explained only later. Much essential content is relegated to more than 25 pages of appendices. The abstract is hard for a general reader to follow. A plain-language summary of the task, the three diagnostic questions and the three findings, together with one clear table, would help considerably.
Questions for authors
Refer to Critical Weaknesses. Thanks.
2 comments
Thank you for reading our rebuttal and for raising your score. We've reflected these changes in the revised PDF. If any concerns remain, please let us know, and we will do our best to address them during the remaining discussion period.
We thank you for finding the question sound and the experiment well designed, and for recognizing the released code, planners and tests.
W1. Added tasks and sensitivity checks extend coverage and support the EHR connection. We agree that one synthetic task did not support the earlier breadth. We added acute kidney injury (AKI) and sepsis, and now use "tasks" rather than "profiles". On logs with diverse actions, the best learner means are 21.3 and 2.0 normalized score (NS) points, versus 43.7 and 23.9 NS for the open-loop schedules, fixed time-only plans found by search with the simulator (Table 3). A separate sweep runs AKI and sepsis in all eight measurement, timing and logging conditions, and the open-loop schedule is above every learner mean in all 16 conditions (Appendix P).
The findings persist when we vary responses and rewards, and the EHR connection lies in record properties. Responses, rewards and timing rules remain hand-specified, but learner means still trail the open-loop schedule across 23 response models and two retrained rewards (Appendix L). The record properties include delayed measurements, decisions kept in force for several hours and persistent actions. The findings also hold with observations calibrated to MIMIC-IV persistence and severity differences. Excess action repetition also appears on real MIMIC-IV Respiratory records: behavior cloning and conservative Q-learning repeat previous actions at 97.1% and 92.2%, versus 81.6% for clinicians (Appendix O).
W2. We withdrew the no-gain claim. Fixed updates hid gains from more data. We increased training updates from 6,000 to 96,000 to hold expected epochs fixed. With 16-fold more data, the fitted-Q learner (DQN/FQI) improves from -2.7 to 17.7 NS, though it still trails the open-loop schedule (Appendix G).
W3. We clarified which checks transfer to real logs. Researchers can fit a schedule from logged returns (a log-fitted schedule), audit copying of previous actions and behavior-policy support, and compare choosing the most likely action with sampling from the learned distribution, while value comparisons still require off-policy evaluation (OPE). Exact floors, ceilings, hidden-state controls and known-model planners require a simulator (Section 6).
W4. We added the requested related work. The revision covers Oberst and Sontag on hidden state and counterfactual evaluation with a Gumbel-Max sepsis simulator, de Haan et al. on causal confusion and previous-action removal, POPGym on online partially observable benchmarks, and Tang and Wiens, Paine et al., and Kumar et al. on offline policy selection (Section 2).
W5. We simplified the presentation. We rewrote the abstract and Introduction, replaced "reference ladder", "information-matched", "lifted initial state", "evidence roles" and "core-state projection" with plain descriptions, and renamed the clock schedule the open-loop schedule. NS is defined before its first use in the Introduction, and QMDP and POMCP are defined in Section 4. Table 3 provides one summary table for all three tasks, and the appendix is reorganized by topic.
We appreciate your detailed suggestions on task coverage and presentation. They led to broader experiments and a clearer account of what researchers can check on real logs. Have we addressed your main concerns? We would be glad to clarify any remaining points.