- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MedGym, a unified benchmark for reinforcement learning in dynamic medical treatment. It combines physiology-inspired dynamics, learned neural residuals, and PINN-based continuous-time simulators to support realistic and informative evaluation across medical RL methods.
Critical strengths
The benchmark addresses two clinically relevant dimensions: intervention timing (continuous) and patient personalization. It supports comparisons between fixed and adaptive intervention schedules, as well as population-, cluster-, and individual-level policies within a common evaluation framework.
Critical weaknesses
-
Insufficient evidence of added benchmark value. Tables 2–4 mainly demonstrate performance differences among configurations within MedGym. More experimental comparisons with existing benchmarks are needed to substantiate the advantages claimed in Table 1.
-
Unbalanced presentation. Considerable space describes the platform broadly and introduces standard RL algorithms. This space would be better used for concrete environment training details.
-
Unclear presentation and interpretation of figures. The figures contain many comparisons without clearly identifying the central findings. The authors should simplify their presentation and explain precisely why each result supports the corresponding claim.
Questions for authors
- In Table 3, DQN-Clu achieves a slightly lower mean SOFA than DQN-Ind. What explains this difference, and what is the boldface intended to emphasize?
- Beyond showing performance differences among individual-, cluster-, and population-level policies, what additional conclusions can be drawn from these comparisons?
- Since the simplified ODE is an approximate physiological prior, how is the impact of its misspecification assessed beyond trajectory prediction errors?
- Could the authors provide direct experimental evidence of MedGym’s advantages in continuous-time interaction over existing benchmarks or matched discrete-time baselines?
3 comments
Response to Reviewer Vzsd (2/2): W2, W3, Q1 and Q3
W2. Space. Done: we cut the RL algorithm descriptions in Sec. 3 to a few sentences (details in App. C) and used the space for how the environment is built. Sec. 3 now states how each patient's PINN is trained, how cohorts and clusters are formed, how the cluster- and population-level models are trained without the test patients, and how fidelity is checked; the paragraphs on online and offline learning now point to the training hyperparameters (Table 7), and App. B.1 additionally describes how the cohorts are selected. We kept the platform description, since it defines the three levels and the two timing settings that all experiments use. Revised: Sec. 3; App. B.1.
W3. Figures. Done: the caption of each result figure and table in the main text now starts with its main finding. For example, the caption of Table 2 now begins "Individual policies reach the lowest SOFA scores and the highest SafeTime% under both timing settings, and adaptive timing lowers the trajectory-wide SOFA score on average at all three levels". The panel labels of Fig. 4 are also corrected. Revised: captions of Fig. 3–6 and Tables 2–4.
Q1. DQN-Clu vs. DQN-Ind (Table 3). The reason is out-of-distribution (OOD) actions: DQN leaves the support of the offline data, so it cannot exploit the patient's own data. With seven cohorts, DQN-Clu is still not worse than DQN-Ind. On 70 test patients (3 seeds; revised Table 3) the two are equal: 8.60 vs. 8.61 (submitted Table 3, one cohort: 7.84 vs. 7.88). DQN has no conservative term and is the farthest from the data support (Fig. 10). In these OOD regions its value estimates are unreliable, so the patient's own data bring no more than data from ten other patients. GCQL, which limits OOD actions and stays closest to the data, gains clearly from the patient's own data (6.87 for Ind against 7.31 for Pop; Table 3). Boldface marks the best mean among offline policies in each column, as now stated in the caption.
Q3. ODE misspecification beyond prediction error. The ODE prior and the neural residual are complementary: the residual corrects the misspecified ODE, and the prior makes treatment responses more plausible and the model less dependent on the seed. Ablation on 10 patients (2 seeds), with criteria independent of the fitted ODE:
| 12 h state / SOFA error ↓ | Implausible treatment response ↓ | |
|---|---|---|
| Full PINN | 0.24 / 0.38 | 17% |
| ODE only | 0.42 / 0.66 | 0% (by construction) |
| NN only | 0.29 / 0.47 | 24% |
The ODE alone is misspecified (error 1.7–1.8× that of the full PINN), and the residual corrects it; without the prior, implausible responses rise from 17% to 24% and the fit varies more across seeds. The policy-level conclusion is also robust to the model fit: when the individual policy is trained on an independent refit of the patient's model, it is still better than the cluster and population policies (App. D.5). The ablation covers the observed range; we state that results beyond it are extrapolations conditional on the physiological prior. Revised: App. A; new App. D.5, D.9.
We added a comparison with ICU-Sepsis (new Fig. 11) together with a matched-baseline experiment (K = 10), ran an ablation of the ODE prior, and revised Sec. 3 and the captions as you suggested. If these changes address your concerns about the added value and the presentation, we would really appreciate it if you would kindly consider updating your rating. We are glad to clarify any remaining point.
Follow-up to Reviewer Vzsd: W1 & Q4
Thank you again for your careful review. We follow up because we have completed a new experiment that addresses W1 & Q4 quantitatively. In our earlier response, we only updated the references to the appendix, figures and tables in the Revised line; its content is unchanged.
Follow-up on W1 & Q4: a quantitative comparison with a discrete-time, fixed-step baseline (new experiment).
Our first response compared MedGym with ICU-Sepsis qualitatively. We have now added a quantitative experiment (extended App. D.11; new Table 19) and replaced Fig. 11 and added Fig. 12 to give more comprehensive results.
The three policies compared. ICU-Sepsis has 716 discrete states and 25 discrete actions, so its policies cannot be run in MedGym. We therefore trained three policies that differ in one factor at a time:
- Discrete-time policy: trained on a discrete 4 h transition model without an interval input, fitted to the data of the same 100 training patients arranged on a 4 h grid, as in ICU-Sepsis; a decision every 4 h.
- Continuous-time policy with fixed timing: trained on the MedGym population simulator; a decision every 4 h. Compared with the first policy, only the model of time changes.
- Continuous-time policy with adaptive timing: trained on the same MedGym simulator; the policy chooses its intervals. Compared with the second policy, only the timing changes.
States, actions, reward, cost, algorithm (TRPOLag) and seeds are identical. All three are evaluated on the individual simulators of the 70 test patients, with 3 training seeds and 50 evaluation episodes per patient and seed.
| Discrete 4 h model, fixed 4 h | MedGym, fixed 4 h | MedGym, adaptive | |
|---|---|---|---|
| SOFA(all) ↓ | 8.42 | 7.89 | 7.60 |
| SOFA(96h) ↓ | 8.10 | 7.40 | 7.24 |
| SafeTime% ↑ | 89.8 | 91.8 | 92.3 |
| Episodes with hidden violations ↓ | 1.81% | 1.58% | 1.08% |
| Decisions | 24 | 24 | 12 |
The experiment supports our first response with numbers: continuous-time modeling gives better policies, violations between decisions are invisible on a 4 h grid, and adaptive timing lowers trajectory-wide SOFA with half the decisions.
- Model of time With a decision every 4 h in both cases, training on the continuous-time simulator lowers SOFA(all) by 0.53 (Holm p = 0.005) and SOFA(96h) by 0.70 (p < 0.001). The discrete model also overestimates its own policy: a mean SOFA of 6.97 inside the model against 8.42 on the patient simulators (p < 10⁻⁸).
- Safety between decisions (updated Fig. 11). Between 40 and 44 h and between 52 and 56 h, the trajectory (orange) exceeds the threshold while the grid values at both ends (green) are below it. Such hidden violations occur in 1.6–1.8% of the episodes with fixed 4 h decisions; they are rare (less than 1% of the unsafe time) and can only be measured in continuous time.
- Timing (new Fig. 12). On the same simulator, the adaptive policy lowers SOFA(all) by 0.29 (Holm p = 0.010) with about 12 decisions instead of 24.
- Personalization. ICU-Sepsis is a single MDP estimated from the pooled data of about 17,000 patients, so it has no patient-specific model on which an individual policy could be trained or evaluated. The three policies above are therefore all at the population level; the evidence for personalization is the comparison inside MedGym (Table 2; App. D.5).
Figures. Fig. 11 now shows an episode from this experiment (continuous-time policy with fixed timing), selected to illustrate a hidden violation, so the description in our first response refers to the previous version. Fig. 12 shows the three policies on one test patient.
Revised: App. D.11 (extended); Fig. 11 (replaced); new Table 19, Fig. 12.
With this quantitative comparison, we believe the revision is considerably stronger. We would be grateful if you could consider raising your score, and we are happy to answer any further questions.
Response to Reviewer Vzsd (1/2): W1, Q4 and Q2
We thank the reviewer for the comments. Changes are marked in blue in the revised PDF.
W1 & Q4. Comparison with an existing benchmark, and evidence for continuous-time interaction. Compared with ICU-Sepsis, the closest benchmark in Table 1 (same database), MedGym can additionally evaluate safety between decisions, policies that choose when to act, and policies for individual patients. For continuous-time interaction, the matched discrete-time baseline is fixed timing inside MedGym; against it, adaptive timing lowers trajectory-wide SOFA when the number of interventions is limited.
What ICU-Sepsis cannot test (new Fig. 11a, one test patient of our cohort). Its state is defined only at the 4 h grid points (blue points; the blue segments show that nothing is modeled in between), a decision is taken every 4 h (black triangles), and one model is used for all patients. It therefore cannot test safety between decisions, policies that choose their own timing, or policies for individual patients.
What MedGym can test (Fig. 11b, trajectories simulated by this patient's model and decisions chosen by the policy).
The setting of ICU-Sepsis (population model, fixed interval, evaluation at decision times) is thus one configuration of MedGym, and Tables 2–4 show how the conclusions change when one leaves it along each of these three dimensions. The abstract now states what is compared: "direct comparison between fixed-interval discrete-time and time-adaptive continuous-time decision making in a common continuous-time environment". In the camera-ready version, we will add a quantitative comparison with ICU-Sepsis for the personalization part. Revised: abstract; Sec. 1; new App. D.8, D.11 (extended, see follow-up), Fig. 11 (updated), Fig. 12, Table 19; App. D.3.
Q2. Further conclusions. Two conclusions that a population-level, fixed-step benchmark cannot produce: