acceptodds
Under review as a conference paper at ICLR 2027

MedGym: A Unified Benchmark for Dynamic Medical Treatment Reinforcement Learning

Abstract

Medical treatment recommendation poses several challenges to reinforcement learning (RL): patient physiology evolves in continuous time, measurements and interventions are performed at irregular intervals, and treatment effects vary substantially across individuals. Existing RL formulations and simulated environments, however, are based on discrete-time MDPs with fixed decision intervals. Thus, it remains difficult to evaluate whether RL methods can handle time-interval-dependent disease progression, personalized treatment response, and safety between consecutive measurement points. To address this gap, we introduce MedGym, a benchmark environment for dynamic treatment recommendation. MedGym models longitudinal patient evolution in a continuous-time framework and constructs a configurable medical RL benchmark from clinical data by using Physics-Informed Neural Networks. The resulting benchmark enables direct comparison between discrete-time and continuous-time methods under irregular treatment timing and patient-specific dynamics. Furthermore, MedGym supports evaluation from clinically important perspectives, such as personalization and trajectory-level safety. By providing a standardized and configurable benchmark for continuous-time dynamic treatment, MedGym enables more realistic and informative evaluation of medical RL methods.

Primary area:
datasets and benchmarks
Keywords:
Medical Treatment, Reinforcement Learning Benchmark, Continuous-Time Systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VzsdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MedGym, a unified benchmark for reinforcement learning in dynamic medical treatment. It combines physiology-inspired dynamics, learned neural residuals, and PINN-based continuous-time simulators to support realistic and informative evaluation across medical RL methods.

Critical strengths

The benchmark addresses two clinically relevant dimensions: intervention timing (continuous) and patient personalization. It supports comparisons between fixed and adaptive intervention schedules, as well as population-, cluster-, and individual-level policies within a common evaluation framework.

Critical weaknesses
  1. Insufficient evidence of added benchmark value. Tables 2–4 mainly demonstrate performance differences among configurations within MedGym. More experimental comparisons with existing benchmarks are needed to substantiate the advantages claimed in Table 1.

  2. Unbalanced presentation. Considerable space describes the platform broadly and introduces standard RL algorithms. This space would be better used for concrete environment training details.

  3. Unclear presentation and interpretation of figures. The figures contain many comparisons without clearly identifying the central findings. The authors should simplify their presentation and explain precisely why each result supports the corresponding claim.

Questions for authors
  1. In Table 3, DQN-Clu achieves a slightly lower mean SOFA than DQN-Ind. What explains this difference, and what is the boldface intended to emphasize?
  2. Beyond showing performance differences among individual-, cluster-, and population-level policies, what additional conclusions can be drawn from these comparisons?
  3. Since the simplified ODE is an approximate physiological prior, how is the impact of its misspecification assessed beyond trajectory prediction errors?
  4. Could the authors provide direct experimental evidence of MedGym’s advantages in continuous-time interaction over existing benchmarks or matched discrete-time baselines?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Vzsd (1/2): W1, Q4 and Q2

We thank the reviewer for the comments. Changes are marked in blue in the revised PDF.

W1 & Q4. Comparison with an existing benchmark, and evidence for continuous-time interaction. Compared with ICU-Sepsis, the closest benchmark in Table 1 (same database), MedGym can additionally evaluate safety between decisions, policies that choose when to act, and policies for individual patients. For continuous-time interaction, the matched discrete-time baseline is fixed timing inside MedGym; against it, adaptive timing lowers trajectory-wide SOFA when the number of interventions is limited.

What ICU-Sepsis cannot test (new Fig. 11a, one test patient of our cohort). Its state is defined only at the 4 h grid points (blue points; the blue segments show that nothing is modeled in between), a decision is taken every 4 h (black triangles), and one model is used for all patients. It therefore cannot test safety between decisions, policies that choose their own timing, or policies for individual patients.

What MedGym can test (Fig. 11b, trajectories simulated by this patient's model and decisions chosen by the policy).

  1. Safety between decisions (orange vs. green curve). Both curves pass through the same points at all 4 h grid points from 0 to 48 h, and all of these points lie below the grey safety threshold. The green dashed curve stays below the threshold throughout, whereas the orange curve crosses it twice, between 0 and 4 h and between 12 and 16 h (shaded areas). In Fig. 11a both reduce to the same blue points, so the violations are invisible. In our results, the safety metrics change when they are computed along the trajectory instead of at decision times (App. D.3).
  2. Timing (black triangles). In Fig. 11a a decision is taken at every grid point, twelve in 48 h. In Fig. 11b the policy takes seven decisions, placed where they are needed instead of on the 4 h grid. Fixed timing inside MedGym is the matched discrete-time baseline: the same patients, simulator, algorithm and number of interventions, with only the choice of intervals removed. Against it, with a limited number of interventions (K = 10), adaptive timing lowers trajectory-wide SOFA by 0.35 for individual and population policies (Holm p = 0.018 for Ind) at comparable safety (new App. D.8, Table 17).
  3. Personalization (one model per patient). Each patient has its own curves, whereas Fig. 11a is the same for every patient. MedGym can therefore compare individual, cluster and population policies on the same patients (Table 2); the individual advantage remains when the policy is not trained on the evaluation simulator (App. D.5).

The setting of ICU-Sepsis (population model, fixed interval, evaluation at decision times) is thus one configuration of MedGym, and Tables 2–4 show how the conclusions change when one leaves it along each of these three dimensions. The abstract now states what is compared: "direct comparison between fixed-interval discrete-time and time-adaptive continuous-time decision making in a common continuous-time environment". In the camera-ready version, we will add a quantitative comparison with ICU-Sepsis for the personalization part. Revised: abstract; Sec. 1; new App. D.8, D.11 (extended, see follow-up), Fig. 11 (updated), Fig. 12, Table 19; App. D.3.

Q2. Further conclusions. Two conclusions that a population-level, fixed-step benchmark cannot produce:

  1. Safety on average does not mean safety for each patient. The mean SafeTime% of cluster and population policies is 89–92%, but it varies strongly between patients (SD 21–29 points, against 13–17 for individual policies, whose mean is 93–97%; Table 2). Consistently, Lagrangian policies are the safest when trained for the individual patient, but show no consistent safety advantage at the cluster and population levels (Table 10, App. D.3). This suggests that safety should be constrained for each patient rather than on average.
  2. Adaptive timing pays off when the number of interventions is limited, so the budget K matters. With a limited number of interventions (K = 10), adaptive timing lowers trajectory-wide SOFA by 0.35 for individual and population policies at comparable safety; with K = 20 the benefit is smaller for individual policies (new App. D.6, D.8; Table 17). The number of allowed interventions is thus a choice that changes the conclusion, whereas a fixed-step benchmark fixes it in advance.
AuthorsOpenReview ↗

Response to Reviewer Vzsd (2/2): W2, W3, Q1 and Q3

W2. Space. Done: we cut the RL algorithm descriptions in Sec. 3 to a few sentences (details in App. C) and used the space for how the environment is built. Sec. 3 now states how each patient's PINN is trained, how cohorts and clusters are formed, how the cluster- and population-level models are trained without the test patients, and how fidelity is checked; the paragraphs on online and offline learning now point to the training hyperparameters (Table 7), and App. B.1 additionally describes how the cohorts are selected. We kept the platform description, since it defines the three levels and the two timing settings that all experiments use. Revised: Sec. 3; App. B.1.

W3. Figures. Done: the caption of each result figure and table in the main text now starts with its main finding. For example, the caption of Table 2 now begins "Individual policies reach the lowest SOFA scores and the highest SafeTime% under both timing settings, and adaptive timing lowers the trajectory-wide SOFA score on average at all three levels". The panel labels of Fig. 4 are also corrected. Revised: captions of Fig. 3–6 and Tables 2–4.

Q1. DQN-Clu vs. DQN-Ind (Table 3). The reason is out-of-distribution (OOD) actions: DQN leaves the support of the offline data, so it cannot exploit the patient's own data. With seven cohorts, DQN-Clu is still not worse than DQN-Ind. On 70 test patients (3 seeds; revised Table 3) the two are equal: 8.60 vs. 8.61 (submitted Table 3, one cohort: 7.84 vs. 7.88). DQN has no conservative term and is the farthest from the data support (Fig. 10). In these OOD regions its value estimates are unreliable, so the patient's own data bring no more than data from ten other patients. GCQL, which limits OOD actions and stays closest to the data, gains clearly from the patient's own data (6.87 for Ind against 7.31 for Pop; Table 3). Boldface marks the best mean among offline policies in each column, as now stated in the caption.

Q3. ODE misspecification beyond prediction error. The ODE prior and the neural residual are complementary: the residual corrects the misspecified ODE, and the prior makes treatment responses more plausible and the model less dependent on the seed. Ablation on 10 patients (2 seeds), with criteria independent of the fitted ODE:

12 h state / SOFA error ↓Implausible treatment response ↓
Full PINN0.24 / 0.3817%
ODE only0.42 / 0.660% (by construction)
NN only0.29 / 0.4724%

The ODE alone is misspecified (error 1.7–1.8× that of the full PINN), and the residual corrects it; without the prior, implausible responses rise from 17% to 24% and the fit varies more across seeds. The policy-level conclusion is also robust to the model fit: when the individual policy is trained on an independent refit of the patient's model, it is still better than the cluster and population policies (App. D.5). The ablation covers the observed range; we state that results beyond it are extrapolations conditional on the physiological prior. Revised: App. A; new App. D.5, D.9.

We added a comparison with ICU-Sepsis (new Fig. 11) together with a matched-baseline experiment (K = 10), ran an ablation of the ODE prior, and revised Sec. 3 and the captions as you suggested. If these changes address your concerns about the added value and the presentation, we would really appreciate it if you would kindly consider updating your rating. We are glad to clarify any remaining point.

AuthorsOpenReview ↗

Follow-up to Reviewer Vzsd: W1 & Q4

Thank you again for your careful review. We follow up because we have completed a new experiment that addresses W1 & Q4 quantitatively. In our earlier response, we only updated the references to the appendix, figures and tables in the Revised line; its content is unchanged.

Follow-up on W1 & Q4: a quantitative comparison with a discrete-time, fixed-step baseline (new experiment).

Our first response compared MedGym with ICU-Sepsis qualitatively. We have now added a quantitative experiment (extended App. D.11; new Table 19) and replaced Fig. 11 and added Fig. 12 to give more comprehensive results.

The three policies compared. ICU-Sepsis has 716 discrete states and 25 discrete actions, so its policies cannot be run in MedGym. We therefore trained three policies that differ in one factor at a time:

  • Discrete-time policy: trained on a discrete 4 h transition model without an interval input, fitted to the data of the same 100 training patients arranged on a 4 h grid, as in ICU-Sepsis; a decision every 4 h.
  • Continuous-time policy with fixed timing: trained on the MedGym population simulator; a decision every 4 h. Compared with the first policy, only the model of time changes.
  • Continuous-time policy with adaptive timing: trained on the same MedGym simulator; the policy chooses its intervals. Compared with the second policy, only the timing changes.

States, actions, reward, cost, algorithm (TRPOLag) and seeds are identical. All three are evaluated on the individual simulators of the 70 test patients, with 3 training seeds and 50 evaluation episodes per patient and seed.

Discrete 4 h model, fixed 4 hMedGym, fixed 4 hMedGym, adaptive
SOFA(all) ↓8.427.897.60
SOFA(96h) ↓8.107.407.24
SafeTime% ↑89.891.892.3
Episodes with hidden violations ↓1.81%1.58%1.08%
Decisions242412

The experiment supports our first response with numbers: continuous-time modeling gives better policies, violations between decisions are invisible on a 4 h grid, and adaptive timing lowers trajectory-wide SOFA with half the decisions.

  1. Model of time With a decision every 4 h in both cases, training on the continuous-time simulator lowers SOFA(all) by 0.53 (Holm p = 0.005) and SOFA(96h) by 0.70 (p < 0.001). The discrete model also overestimates its own policy: a mean SOFA of 6.97 inside the model against 8.42 on the patient simulators (p < 10⁻⁸).
  2. Safety between decisions (updated Fig. 11). Between 40 and 44 h and between 52 and 56 h, the trajectory (orange) exceeds the threshold while the grid values at both ends (green) are below it. Such hidden violations occur in 1.6–1.8% of the episodes with fixed 4 h decisions; they are rare (less than 1% of the unsafe time) and can only be measured in continuous time.
  3. Timing (new Fig. 12). On the same simulator, the adaptive policy lowers SOFA(all) by 0.29 (Holm p = 0.010) with about 12 decisions instead of 24.
  4. Personalization. ICU-Sepsis is a single MDP estimated from the pooled data of about 17,000 patients, so it has no patient-specific model on which an individual policy could be trained or evaluated. The three policies above are therefore all at the population level; the evidence for personalization is the comparison inside MedGym (Table 2; App. D.5).

Figures. Fig. 11 now shows an episode from this experiment (continuous-time policy with fixed timing), selected to illustrate a hidden violation, so the description in our first response refers to the previous version. Fig. 12 shows the three policies on one test patient.

Revised: App. D.11 (extended); Fig. 11 (replaced); new Table 19, Fig. 12.

With this quantitative comparison, we believe the revision is considerably stronger. We would be grateful if you could consider raising your score, and we are happy to answer any further questions.

Reviewer_TtT3OpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces a continuous-time RL benchmark for medical treatment decisions named MedGym. It addresses an important problem present in traditional RL models: assuming fixed discrete time steps, which do not reflect real-world clinical care where treatments generally occur at irregular intervals. The authors formulate the clinical decision process as a continuous-time Markov Decision Process. They build patient simulators using Physics-Informed Neural Networks (PINNs) that combine simplified organ ODEs with data-driven neural residuals across three levels: population, cluster, and individual. The benchmark is validated on two clinical datasets (MIMIC-III Sepsis and Synthetic Acute Hypotension) across online, offline, and safety-constrained RL algorithms.

Critical strengths

I think this work genuinely provides a leap towards more realistic benchmarks for medical treatment decision-making. Grounding the latter in a continuous-time framework with adaptive action timing addresses a fundamental gap in medical RL, moving beyond unrealistic fixed-interval assumptions.

Critical weaknesses

Experimental rigour may be weaker than it appears. All online and offline policy-learning algorithms across the paper are trained using a single random seed and the offline RL evaluation is restricted to a small number of test patients. I think using a kk-fold cross-validation style protocol across cluster patients and multi-seed training runs is necessary to establish reliable performance bounds.

Minor Comment on the Presentation
  • Labels (c) and (d) are inverted in the caption for Figure 4. Plot (c) displays the SOFA score trajectory and plot (d) displays the lactate trajectory, whereas the caption text states that (c) represents lactate and (d) represents SOFA. I think the caption is wrong.
  • There is a repeated sentence in lines 277–278.
Questions for authors
  • In the clinical management of sepsis, the SOFA score is fundamentally a measure of patient safety. Why is SOFA treated exclusively as a treatment effectiveness metric, whilst lactate elevation is treated as the sole safety constraint? I understand that this choice is not intrinsic to MedGym, as noted in line 222.
  • In Section 4.1, the authors state that "... The degradation in control is not a fidelity artefact" because all policies are evaluated inside the same Individual simulator. However, isn't the Population policy's poor control performance a direct consequence of being trained on a lower-fidelity Population simulator?
  • Again in Section 4.1, lines 312–314, the authors state that "This consistent ordering indicates that patient heterogeneity should be considered in both environment construction and policy learning." I do not quite see how this conclusion follows from the preceding statements; could the authors clarify?
3 comments
AuthorsOpenReview ↗

Response to Reviewer TtT3 (1/2): W1 and W2

We thank the reviewer for the positive assessment and the concrete suggestions. Changes are marked in blue in the revised PDF.

W1. Seeds, offline test patients and k-fold. We added all three.

(i) Seeds: the ordering Ind < Clu < Pop holds on average over three seeds, and the individual advantage holds in every seed and every cohort. We retrained all TRPOLag policies with 3 seeds and ran paired tests over the 70 test patients (new App. D.6). Ind improves SOFA(all) over Clu by 1.35 / 1.13 (F / A) and over Pop by 1.49 / 1.20 (all Holm p < 10⁻⁸). Clu is better than Pop on average, but by much less (SOFA(all) −0.14 / −0.07, n.s.; SOFA(96h) −0.13 / −0.23, the latter p = 0.017), comparable to seed-to-seed variation.

(ii) Offline: the individual-data advantage holds on 70 test patients and 3 seeds (revised Table 3). GCQL reaches 6.87 (Ind), 7.79 (Clu) and 7.31 (Pop): Ind beats Pop in 6/7 cohorts, while Clu does not improve on Pop, as the original text noted (l.426–431). DQN-Clu matches DQN-Ind (8.60 vs. 8.61), and both beat Pop (9.31). A control with ten random patients gives the same result as Clu, so offline the cluster result reflects how data are pooled rather than the grouping.

(iii) k-fold: the individual advantage holds in every fold. We rotated the test patient within one cluster (4 folds; models containing the test patient were retrained without it; 3 seeds). Ind is better than Clu and Pop in 4/4 folds under both timing modes, and the full ordering Ind < Clu < Pop holds in 3/4 folds. The mean differences in SOFA(all), F / A, are −1.15 / −0.74 (Ind − Clu) and −0.47 / −0.29 (Clu − Pop), in the same direction as the 70-patient results. We will extend this rotation to more clusters in the camera-ready version. The refit control (Q2), a random-grouping control and a randomly sampled cohort further test robustness to the patient partition (App. D.5–D.7).

Revised: Sec. 4, Tables 2–3; App. D.4 (new Table 13); new App. D.6.

W2 (minor). Both fixed: the Fig. 4 caption now reads (c) SOFA, (d) lactate; the duplicate at l.277–278 is removed.

AuthorsOpenReview ↗

Response to Reviewer TtT3 (2/2): Q1, Q2 and Q3

Q1. Roles of SOFA and lactate. SOFA is the standard measure of organ dysfunction in sepsis and what treatment aims to lower, so we use it as the objective; lactate gives an early safety signal that is independent of SOFA, so we use it as a constraint. Both are choices for the sepsis task: in MedGym, the reward and the cost can be set for each disease.

Why SOFA as effectiveness. Sepsis is defined by organ dysfunction, which SOFA measures, and a higher SOFA means a higher risk of death. It summarizes multi-organ dysfunction rather than a single variable, and it can be computed at every decision time, so it is more suitable for evaluating sequential treatment than a terminal outcome (App. B.2). Earlier RL work on sepsis also uses SOFA changes in the reward (Raghu et al., 2017). We agree that it also carries safety information; since the reward already drives SOFA down, we did not constrain it in addition, and a user can add SOFA terms to the cost.

Why lactate as the safety constraint. Lactate is not part of our SOFA surrogate and signals hypoperfusion earlier than the organ scores, so a threshold on it (4 mmol/L) gives a separate, early safety signal.

Reward and cost can be set per disease. For example, the acute hypotension task uses different terms.

Q2. Is Pop worse only because of a lower-fidelity training simulator? Not only. The population policy is less effective for a given patient because patients respond differently to the same treatment, while a policy learned from many patients' data, without the test patient, therefore improves the average outcome rather than that of each patient. This is the situation of any new patient, not an artefact of the simulator. Training on the evaluation simulator itself (a specific individual patient PINN) adds a further advantage for the individual policy, which a refit control quantifies: training on a refit of the same patient's PINN (80% of the data, new seed) costs 0.99 / 0.83 SOFA points, out of an Ind→Pop gap of 1.49 / 1.20 (70 test patients, 3 seeds). The refit policy is still better than Pop by 0.50 / 0.37 (p = 0.001 / 0.016). The sentence at l.311–312 now reads "not only a fidelity artefact". Revised: Sec. 4.1; new App. D.5.

Q3. Why heterogeneity should be considered. It follows from two results: patients respond differently to the same treatment, which is visible in the raw data (new App. B.5), and the individual advantage remains when the policy is not trained on the evaluation simulator (Q2). A benchmark with only a population simulator would hide this gap. Closing it for new patients means learning policies from mixed data that depend on patient features, which MedGym is designed to evaluate. Revised: Sec. 4.1; new App. B.5, D.5.

We added the three analyses you asked for (more seeds, 70 offline test patients and a within-cluster rotation), fixed the two presentation issues, and narrowed the claims at l.311–314, which are now supported by a refit control and the raw data. We would be grateful to know whether these changes resolve your concerns, and we are glad to add anything further.

AuthorsOpenReview ↗

Follow-up to Reviewer TtT3: W1 (i) and (iii)

Thank you again for your constructive review and your positive assessment. We follow up because we have completed the remaining experiments for W1 (i) and (iii). Our earlier responses are unchanged.

Follow-up on W1 (i): three seeds for the other online algorithms (extended Table 15).

In our first response, only TRPOLag was repeated with three seeds. We have now retrained SAC, PPO, TRPO and PPOLag with three seeds as well and evaluated them on the same 70 test patients.

SOFA(all) ↓, F / AIndCluPop
SAC5.93 / 5.387.11 / 6.977.35 / 7.19
PPO5.72 / 5.458.09 / 8.118.41 / 8.04
TRPO6.47 / 6.277.94 / 7.658.39 / 7.98
PPOLag5.74 / 5.567.85 / 7.708.11 / 7.66
TRPOLag6.27 / 6.227.62 / 7.357.76 / 7.42

The individual advantage holds for every algorithm. Ind improves SOFA(all) over Clu by 1.13–2.66 and over Pop by 1.20–2.69 under both timing settings (all Holm p < 10⁻⁷).

Follow-up on W1 (iii): the rotation now covers every patient of the cluster (11 folds).

In our first response, the test patient was rotated over 4 patients of one cluster. We have extended this to a leave-one-out over all 11 patients of the cluster (cluster and population models retrained without the test patient; 3 seeds).

SOFA(all)Ind < CluInd < PopClu < Popmean Ind − Clumean Clu − Pop
Fixed11/1111/118/11−1.50−0.59
Adaptive10/1111/117/11−1.14−0.03

The individual advantage holds in every fold under fixed timing and in 10 and 11 of the 11 folds under adaptive timing. The 4-fold result therefore extends to every member of the cluster.

Revised: Table 15, App. D.6; App. A.

We hope these additions further support your positive assessment, and we would be grateful if you could maintain your score. We are happy to answer any further questions.

Reviewer_r5hFOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MedGym, a benchmark for dynamic treatment recommendation in which patient evolution is modeled in continuous time. Each patient gets an individual hybrid model, which the paper calls a PINN: a physiology-inspired ODE plus a neural residual. The model maps the current state, the treatment, and the elapsed time δt to the next state, and adds Gaussian noise whose variance scales with δt. Patient models are grouped by capacity-constrained medoid clustering, using how well each model predicts the others' transitions as the distance. Cluster-level and population-level models are then trained on the pooled data. The evaluation protocol compares three dimensions:

  • (i) fixed versus adaptive intervention timing, using the TACOS bounded-interaction formulation (K = 20, T = 96 h);
  • (ii) population-, cluster-, and individual-level policies, all evaluated on the held-out patient's own simulator;
  • (iii) online learning (SAC, PPO, TRPO and their Lagrangian variants) versus offline learning (DQN, CQL, GCQL).

Treatment effectiveness is measured with SOFA-based metrics and safety with lactate-based metrics. The main instantiation uses MIMIC-III sepsis data (770 patient models, 70 held-out test patients). A second instantiation uses the Synthetic Acute Hypotension Dataset (10 test patients). The authors conclude that intervention timing and the degree of personalization materially affect policy performance, and that effectiveness and safety need not improve together.

Critical strengths
  1. Well-motivated problem. Existing medical RL benchmarks mostly use discrete-time MDPs with fixed decision intervals and evaluate population-level policies. The paper states and formalizes the two missing elements precisely: transitions that depend on the elapsed interval, and patient-specific dynamics (l.129–143). Both are relevant to the field.
  2. Broad coverage and candid limitations.
  • The experiments cover five online and three offline algorithms, two clinical settings, an alternative clustering scheme (App. D.2), and several trajectory-level safety metrics (App. D.3).
  • Appendix A openly acknowledges that the individual setting is an oracle and that observational data cannot validate counterfactual treatment responses.
Critical weaknesses
  1. The personalization conclusion is confounded with a mismatch between training and evaluation simulators. The individual policy is trained on the same patient-specific simulator it is later evaluated on, while the cluster and population policies are trained on different PINNs and then transferred to that simulator (l.270–274). The ordering Ind < Clu < Pop in Tables 2 and 8 is therefore what one would expect from model mismatch alone, even without any genuine patient heterogeneity, and the argument that the degradation "is not a fidelity artefact, since all three policies are evaluated on the same patient-specific simulator" (l.311–312) overlooks that the training dynamics differ as well. The offline experiments have the same structure, since the individual dataset is generated by an oracle TRPOLag policy trained on the evaluation simulator itself (l.1366–1379). It is also unclear how much of the difference between patient models reflects physiology rather than fitting variance: each model has about 5×10⁴ parameters and is fitted to a single short trajectory (l.943–945), and temporal hold-out is shown for only one patient (Fig. 7). The supplementary code further indicates that the cohort was selected as tight, mutually separated groups in PINN-distance space, a step not described in the paper, which would make the cluster structure partly a product of cohort construction. A control that retrains the simulator for the same patient, trains a policy on the retrained copy, and evaluates it on the original would show how much of the Ind→Pop gap comes from model mismatch alone; a random-grouping control would do the same for the cluster advantage.
  2. The evidence for the timing claim is weak, and the experiment does not test the gap the paper motivates. In Table 2, adaptive timing reduces SOFA(all) by only 0.21, 0.07, and 0.51 for Ind, Clu, and Pop, against across-patient standard deviations of about 1.6–2.0. Each policy is trained with a single seed (l.415), and no paired test is reported. For the individual policy, adaptive timing actually worsens SOFA(96h) (5.63 → 5.87) and SafeTime%, and Table 4 is similarly mixed, so the claim that timing "materially affect[s] treatment performance" (l.84–86) is overstated. More fundamentally, Section 2 motivates the benchmark by the failure of discrete-time transitions to depend on δt (l.129–136). Yet the fixed-interval baseline already runs on the continuous-time simulator with a constant δt, so the experiment measures whether letting the policy choose its own timing helps, not whether continuous-time modeling helps. No genuinely continuous-time RL method is evaluated beyond the TACOS reformulation of discrete-time algorithms, so the abstract's claim of "direct comparison between discrete-time and continuous-time methods" is not supported. The safety side of this comparison is also not well defined: the training cost is evaluated only at decision states (l.1240–1241), even though the paper itself identifies unsafe evolution between decision times as the key risk (l.822–823), so an adaptive policy can step over unsafe stretches by choosing long intervals. The supplementary code also suggests that the reward actually optimized is not the interval integral of Eq. 33 and that γ ≠ 1, in which case the equivalence in Eq. 37 would not hold for the reported experiments.
Questions for authors
  1. To separate model mismatch from genuine heterogeneity, could you retrain the simulator for the same patient with a different seed or data subsample, train a policy on that retrained copy, and evaluate it on the original simulator? How large is this degradation compared with the Ind→Pop gap? Does the cluster advantage over the population policy remain under random grouping?
  2. How was the 770-patient cohort selected from the available patients, and were patients chosen so that the dynamics-based groups are tight and well separated? Do the main conclusions hold on a randomly sampled cohort?
  3. Can you report temporal hold-out errors, including SOFA error, for all patients, together with the distribution of observed trajectory lengths, given that policies are evaluated over 96 h?
4 comments
AuthorsOpenReview ↗

Response to Reviewer r5hF (1/3): W1 and Q1

We thank the reviewer for the careful reading, including of the code. Changes are marked in blue in the revised PDF.

W1 & Q1. Is the Ind < Clu < Pop gap only simulator mismatch?

Not only. The gap comes from whether the policy can adapt to each patient's own reaction to treatment, and the essential reason is whose data the policy is trained on. Patients respond differently to the same treatment, which is visible in the raw data, and only the individual models capture this. An individual policy is learned for the patient it treats; cluster and population policies are learned from several patients' data, without the test patient, so they improve the average outcome rather than that of each patient. This is the situation of any new patient, not an artefact of the simulator. Training on the evaluation simulator itself adds a further advantage for the individual policy, which the refit control quantifies (point 3). For future work, this means learning policies from mixed data that depend on patient features, and extracting such feature variables is important for this.

  1. Patients differ in the raw data (770 patients, no simulator; new App. B.5). Between-patient differences explain 77% of the variance of SOFA and 69% of log-lactate; the patient-specific response of urine output to fluids varies about 5× the population slope (p < 10⁻²⁰); at similar severity, patient identity explains 72% of the variance in vasopressor dose. These differences are consistent with known sepsis subphenotypes (Seymour et al., 2019).

  2. The individual models capture the patient, not fitting noise. A refit of the same patient is much closer to the original model than models of other patients (median distance 0.16 vs. 0.77).

  3. The individual advantage remains without training on the evaluation simulator (the refit control you suggested; 70 test patients, 3 seeds). A policy trained on a refit of the patient's PINN (80% of the data, new seed), Ind′, is evaluated on the original model.

SOFA(all) ↓, F / AIndInd′CluPop
TRPOLag6.27 / 6.227.26 / 7.057.62 / 7.357.76 / 7.42

Training on the refit costs 0.99 / 0.83 of the Ind→Pop gap of 1.49 / 1.20, as you expected. Ind′ is still better than Pop by 0.50 / 0.37 (p = 0.001 / 0.016) and than Clu by 0.36 / 0.30 (p = 0.001 / 0.055). Offline, the Ind dataset is indeed an oracle, which we now call an upper bound; still, replacing a patient's own data by the same amount of data from ten random patients worsens GCQL by 0.91 (7/7 cohorts). Accordingly, we revised the sentence at l.311–312 ("not a fidelity artefact"); it now states that training on the evaluation simulator explains part of the gap and refers to this control (Sec. 4.1).

Cluster level and random grouping (Q1). The cluster level is a practical intermediate level: one policy is trained once and shared by a group of patients. Cluster policies are better than population policies on average (SOFA(all) −0.14 / −0.07; final SOFA −0.13 / −0.23, the latter p = 0.017). Since this level depends on the grouping, the paper already tests a second grouping (App. D.2), and we added the random grouping you suggested: in cohort 1, random groups perform like the dynamics-based clusters (Clu 7.97 / 7.80, Clu-rand 7.99 / 7.76, Pop 8.03 / 7.69; Ind 6.87 / 6.94). No grouping reaches the individual level; MedGym provides the three levels as references to measure how much of this gap future methods recover.

Revised: Sec. 4.1; App. C.2; new App. B.5, D.5–D.7; Table 13.

AuthorsOpenReview ↗

Response to Reviewer r5hF (2/3): W2

W2(1). Timing, continuous time and safety between decisions. When the fixed interval is short enough, fixed timing already treats the patient well and differs little from adaptive timing. When the number of interventions is limited (K = 10), adaptive timing lowers trajectory-wide SOFA clearly at comparable safety.

We therefore tested a limited number of interventions, K = 10, i.e. a fixed interval of 9.6 h (cohort 1, 10 test patients, 3 seeds; new Table 17):

F / AIndCluPop
SOFA(all) ↓6.84 / 6.497.77 / 7.707.99 / 7.64
SafeTime% ↑99.0 / 95.287.4 / 88.882.1 / 83.3

Adaptive timing lowers SOFA(all) by 0.35 for Ind (Holm p = 0.018) and for Pop, and by 0.06 for Clu; the differences in SafeTime% (−3.8, +1.5 and +1.2 points) are not significant. The reason is that, with few interventions, a fixed schedule has to spread them evenly over the 96 h, whereas an adaptive policy can place them where the patient's state changes quickly and hold a treatment while the patient is stable. Paired tests with three training seeds for K = 20 are given in App. D.6.

Both timing modes run on the same continuous-time simulator, so the experiment compares two ways of making decisions, as you point out. The abstract now reads "direct comparison between fixed-interval discrete-time and time-adaptive continuous-time decision making in a common continuous-time environment".

Safety is now measured between decisions. You are right that violations between decisions were not counted; this also affected our evaluation, which held the end-of-interval lactate over each interval. All safety metrics are now computed along the simulator's sub-step trajectory (Tables 2, 3, 8–12), and the SafeTime% values above are computed in this way. Policies are still trained with the cost at decision times; the trajectory-level metrics now show how much unsafe time this leaves between decisions (App. D.3), and training with a trajectory-level cost is left for future work.

Revised: abstract, Sec. 1, Sec. 4.1, Conclusion; Tables 2, 3, 8–12, 17; App. A, D.3, D.6, D.8.

W2(2). Reward and γ. You are right; we now describe the reward and γ as implemented. The runs use a right-endpoint approximation with four shaping terms and γ = 0.997; Eq. 37 is now stated as the limit case. Shaping (26–39% of the absolute reward) is unlikely to drive the timing choice: the time term is the same constant for every policy, the SOFA-improvement term is potential-based, the short-interval penalty is at most 0.16 per episode, and adaptive policies use only 7–14 of 20 decisions. Evaluation does not use the training reward. Revised: App. C.1, Table 7.

AuthorsOpenReview ↗

Response to Reviewer r5hF (3/3): Q2 and Q3

Q2. Cohort selection. Yes: each cohort is the union of tight, well-separated groups in PINN-distance space, as the supplement document describes; the paper omitted this step and now states it (App. B.1). On a randomly sampled cohort, the ordering Ind < Clu < Pop is the same. Within each cohort, clusters are formed as in the paper (Eq. 19–21). The seven cohorts contain 770 of the 831 patients with an individual model, so the selection excludes few patients. It favors the cluster and population levels; even so, Ind is better than both in 7/7 cohorts, Clu and Pop are close (full ordering in 5/7 and 3/7 cohorts, F / A), and random groups perform like the clusters (W1). For 110 patients drawn at random from all 831 (clustered with Eq. 19–21, one random test patient per cluster, all models retrained), the ordering is the same: SOFA(all), F / A, is 6.25 / 6.40 (Ind), 7.33 / 7.14 (Clu) and 7.87 / 7.49 (Pop) (10 test patients, 3 seeds). Revised: App. A, B.1; new App. D.6.

Q3. Hold-out errors. Now reported for all 770 patients: the fit holds over short horizons, and 96 h lies beyond every observed trajectory, which we now state. With a 70/30 temporal split, the mean absolute error (one-step / open-loop) is 0.35 / 0.70 mmol/L for lactate, 0.67 / 1.47 for GCS and 0.32 / 0.45 for the SOFA surrogate (new App. D.10). Observed trajectories last 24–82 h (median 52 h), so results at 96 h are extrapolations conditional on the model. Two points limit the impact on our conclusions: (i) the individual advantage is already present within the observed window: within the first 24 h, which every patient's data cover, Ind is better than Pop by 0.83 / 0.54 and than Clu by 0.77 / 0.68, about half of the full-horizon gap; (ii) the gap also remains when the individual policy is trained on a refit instead of the evaluation simulator (W1, point 3), so it does not come only from exploiting the simulator's errors. Revised: App. A; new App. D.6 (Table 16), D.10.

We ran the controls you proposed (refit, random grouping, a random cohort, hold-out for all patients), added a K = 10 experiment with paired tests, and revised the three points you found in the code (safety between decisions, reward and γ, cohort selection). If these changes address your concerns about soundness, we would really appreciate it if you would kindly consider updating your rating. We are glad to run further checks within the discussion period.

AuthorsOpenReview ↗

Follow-up to Reviewer r5hF: W1 & Q1 and W2(1)

Thank you again for your detailed review. We follow up because we have completed additional experiments that further address W1 & Q1 and W2(1). Our earlier responses are unchanged.

Follow-up on W1 & Q1 and W2(1): the conclusions hold for the other online algorithms (extended Table 15).

We have now retrained SAC, PPO, TRPO and PPOLag with three seeds, in addition to TRPOLag (70 test patients).

SOFA(all) ↓, F / AIndCluPop
SAC5.93 / 5.387.11 / 6.977.35 / 7.19
PPO5.72 / 5.458.09 / 8.118.41 / 8.04
TRPO6.47 / 6.277.94 / 7.658.39 / 7.98
PPOLag5.74 / 5.567.85 / 7.708.11 / 7.66
TRPOLag6.27 / 6.227.62 / 7.357.76 / 7.42

With three training seeds, the individual policy remains the best for every online algorithm and under both timing settings, and adaptive timing lowers the trajectory-wide SOFA. Ind improves SOFA(all) over Clu and Pop by 1.13–2.69 (all Holm p < 10⁻⁷). Adaptive timing lowers SOFA(all) in 14 of the 15 combinations of algorithm and level, by 0.05–0.55.

Revised: Table 15 (App. D.6); App. A.

With these additional results, we believe the revision is considerably stronger. We would be grateful if you could consider raising your score, and we are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.