acceptodds
Under review as a conference paper at ICLR 2027

REAL-EHR: Benchmarking Open-Loop Clinical Event Forecasting

Abstract

Open-loop EHR forecasting requires models to predict beyond the observed patient record without receiving future observations, yet most models are evaluated with teacher-forced next-event prediction. Teacher-forced performance may not reflect open-loop forecasting, and open-loop evaluation itself depends on how generated futures are scored. We introduce REAL-EHR, a benchmark that compares matched patients and targets while varying information access, scoring criteria, and rollout budget. We adapt 17 model architectures to a common forecasting interface. At a 32-rollout budget on MIMIC-IV, teacher-forced and estimated open-loop NLL rankings show little agreement (), with 62 of 136 pairwise model orders reversed. At the same budget, all 17 learned models have higher estimated marginal NLL than a frequency baseline, while 16 improve candidate coverage and window occurrence. These NLL comparisons are budget-sensitive: increasing the rollout budget to 512 moves nine of ten audited models below the frequency baseline by point estimate, while weak agreement with teacher forcing persists within this subset (). Model implementations and evaluation code are available at https://anonymous.4open.science/r/REAL-EHR-E237.

Primary area:
datasets and benchmarks
Keywords:
electronic health records, long-horizon forecasting, open-loop forecasting, evaluation benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JyFdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes to evaluate EHR forecasting models by measuring the NLL for trajectories

Critical strengths

Long horizon prediction performance is important and useful.

Critical weaknesses

This paper is incredibly difficult to follow and makes no sense due to the authors skipping steps when explaining their work. A well written paper should introduce the problem, explain why the proposed solution is correct, walk through why existing alternatives didn't work, and then demonstrate that their solution works.

I don't even know what problem this paper is trying to solve? Why would you evaluate long-term trajectory prediction using NLL? What is wrong with the existing solution of defining a suite of prediction tasks and measuring AUROC? For an example of standard prior work, see Figure 3 in the ETHOS paper for how this is currently done: https://arxiv.org/pdf/2407.21124.

"yet most models are evaluated with teacher-forced next-event prediction" is flat out wrong.

That's not getting into how I'm not even sure the proposed setup is sound. Where are your proofs that your evaluation metrics are proper scoring rules? Why are you ignoring the time aspect of the problem.

The whole setup seems to solve a problem that is already solved (AUROC for candidate tasks), but in a worse way.

Questions for authors

What is wrong with the traditional healthcare ML evaluation setup of predicting a range of targets within a range of time horizons and measuring AUROC?

How do you know that your metrics are proper scoring rules?

6 comments
AuthorsOpenReview ↗

Response to Reviewer JyFd (Part 1/5)

Dear Reviewer JyFd,

Thank you for your careful review and for recognizing the importance of long-horizon prediction. We rewrote the Introduction around a lactate test, evaluated both predictions side by side, and corrected our description of how earlier work evaluates clinical outcomes.

Q1. “What is wrong with the traditional healthcare ML evaluation setup of predicting a range of targets within a range of time horizons and measuring AUROC?”

AUROC is appropriate for asking which patients will receive a lactate test within 48 hours. Suppose two models give each patient exactly the same risk. Their AUROCs will be identical, even if one model regularly generates the test three events early. If we also want to know whether the model anticipated a lactate measurement at the step where it was recorded, we can look at the probability it assigned to lactate at that step.

We evaluated both questions on lactate and troponin (Sec. 5.4, Tables 2B, 12 and 14). The nine models below started at ICU hour 24 with 64 observed events. RSSM had the lowest average negative log-likelihood (NLL) for predicting each test at its recorded position. Transformer had the highest 48-hour AUROC for lactate, and Foresight for troponin.

PredictorLactate position NLLLactate 48-h AUROC [95%95\% CI]Troponin position NLLTroponin 48-h AUROC [95%95\% CI]
LaCT5.775.770.715  [0.610,0.797]0.715\;[0.610,0.797]6.276.270.770  [0.636,0.882]0.770\;[0.636,0.882]
RSSM4.684.680.646  [0.555,0.737]0.646\;[0.555,0.737]6.216.210.700  [0.557,0.835]0.700\;[0.557,0.835]
Transformer5.805.800.794  [0.709,0.869]0.794\;[0.709,0.869]7.027.020.771  [0.618,0.901]0.771\;[0.618,0.901]
TTT-Linear6.096.090.759  [0.670,0.837]0.759\;[0.670,0.837]6.536.530.766  [0.639,0.895]0.766\;[0.639,0.895]
Delphi6.246.240.638  [0.531,0.729]0.638\;[0.531,0.729]7.797.790.590  [0.413,0.744]0.590\;[0.413,0.744]
Foresight6.446.440.788  [0.709,0.866]0.788\;[0.709,0.866]6.876.870.869  [0.759,0.957]0.869\;[0.759,0.957]
MIRA6.406.400.769  [0.679,0.852]0.769\;[0.679,0.852]6.886.880.835  [0.706,0.942]0.835\;[0.706,0.942]
Delphi+Value6.296.290.650  [0.556,0.745]0.650\;[0.556,0.745]8.118.110.602  [0.436,0.748]0.602\;[0.436,0.748]
Mamba-26.166.160.761  [0.683,0.839]0.761\;[0.683,0.839]6.436.430.846  [0.723,0.941]0.846\;[0.723,0.941]
Last observed valueN/A0.7570.757N/A0.7930.793
Prior test countN/A0.6240.624N/A0.6790.679

We scored the first recorded test position on 40 patients per test who received it, using 1,024 draws each. The model saw only the history. We used the recorded position after generation, and extra tests elsewhere did not affect this score. For AUROC we used 256 draws and patients with and without the test. Of 300 sampled patients, 122 had assessable lactate status (67 positive) and 111 had assessable troponin status (15 positive). Brackets give 95%95\% patient intervals with the seed-41 checkpoints and simulated paths held fixed.

“Why would you evaluate long-term trajectory prediction using NLL?”

We use NLL to score the probability of the measurement recorded at one future step. For example, after seeing 16 events, the model generates many possible 63-event continuations. For each one it predicts a distribution for event 64. We average the probabilities assigned to the measurement that was actually recorded, then take the negative logarithm (Sec. 3.2, Eq. 1). Teacher forcing gives a valid likelihood for the whole observed sequence by the chain rule. At step 64, though, it has seen the intervening 63 events.

“‘yet most models are evaluated with teacher-forced next-event prediction’ is flat out wrong.”

We agree, and we removed that claim. ETHOS already used simulated timelines to predict clinical outcomes. FoMoH evaluates how well models distinguish patients with and without an outcome, how well calibrated their risks are, and how performance varies across groups. Sec. 1 and Sec. 2 now credit these approaches and explain why a reader interested in the generated record might also want a score for a particular step. ETHOS · FoMoH

AuthorsOpenReview ↗

Response to Reviewer JyFd (Part 2/5)

Q2. “How do you know that your metrics are proper scoring rules?”

Two of the three scores are strictly proper when the forecast probabilities are exact. For a true event distribution rr at a fixed position and a forecast qq, excess expected log loss equals the Kullback–Leibler divergence from rr to qq. It is zero only when q=rq=r. For a binary event with true probability rr and forecast pp, expected Brier loss is r(1−r)+(p−r)2r(1-r)+(p-r)^2, uniquely minimized at p=rp=r. Scoring only patients who received a test, as in the position analysis above, cannot establish calibration in all patients. We clarified this distinction in Sec. 3.2 and added the Brier derivation to Appendix F. Gneiting and Raftery (2007)

EventPass@K is not a proper scoring rule (Appendix F). It measures the chance that at least one of KK candidates includes the recorded event. When the true probabilities are (0.8, 0.1, 0.1), shifting probability toward the two rare events increases five-candidate coverage from 0.882 to 0.926, even as expected NLL gets worse.

Forecast for three eventsExpected NLL ↓\downarrowExpected EventPass@5 ↑\uparrow
True probabilities (0.8,0.1,0.1)(0.8,0.1,0.1)0.6390.6390.8820.882
Forecast (0.6,0.2,0.2)(0.6,0.2,0.2)0.7310.7310.9260.926

What changes when we estimate these probabilities from simulated paths? Taking a negative logarithm introduces upward bias in the expected NLL because the negative logarithm is convex. Two models can have different biases even with the same number of draws. For Brier loss, let ss be the fraction of NN independent paths containing the event and aa the observed label. The expected plug-in loss is (p−a)2+p(1−p)/N(p-a)^2+p(1-p)/N. For N>1N > 1, subtracting s(1−s)/(N−1)s(1-s)/(N-1) from the plug-in loss gives an unbiased estimate of the exact Brier loss. We report the uncorrected score used in the experiments and give the correction in Appendix F. Ferro (2014)

New exact check of the NLL estimator (Table 7). On 512 patients the exact event-64 NLL was 3.199 for Markov and 3.200 for Frequency. We repeated the sampled Markov calculation in 32 independent pools with six nested budgets per pool. Errors and across-pool SDs are in nats.

Draws per patientMean sampled NLLMean signed errorSD across pools
32324.1954.195+0.996+0.9960.0700.070
64643.8323.832+0.633+0.6330.0700.070
1281283.5583.558+0.359+0.3590.0680.068
2562563.3733.373+0.174+0.1740.0440.044
5125123.2763.276+0.077+0.0770.0230.023
1,0241{,}0243.2303.230+0.031+0.0310.0160.016

Increasing the budget reduced mean error by 97%97\%. Even at 1,024 draws, the error exceeded the exact gap of about 0.001 nats. Every point estimate at every budget favored Frequency. The simultaneous Monte Carlo intervals included both zero and the exact difference, with an infinite upper endpoint.

How large is the simulation uncertainty for the neural models? (Table 9) We held the 512 patients and one checkpoint per model (seed 41) fixed. Δ\Delta is the estimated NLL minus the exact loss for Frequency. The brackets give simultaneous Monte Carlo intervals for this difference. Six of the ten models were evaluated at both budgets.

Model512 draws: NLLΔ\Delta [MC interval]1,024 draws: NLLΔ\Delta [MC interval]
Transformer3.1143.114−0.086  [−1.690,+∞]-0.086\;[-1.690,+\infty]3.0443.044−0.156  [−1.480,+∞]-0.156\;[-1.480,+\infty]
RSSM3.2103.2100.010  [−1.638,+∞]0.010\;[-1.638,+\infty]3.1863.186−0.014  [−1.415,+∞]-0.014\;[-1.415,+\infty]
Mamba-23.0703.070−0.130  [−1.681,+∞]-0.130\;[-1.681,+\infty]N/AN/A
Delphi3.1323.132−0.068  [−1.676,+∞]-0.068\;[-1.676,+\infty]3.0893.089−0.111  [−1.459,+∞]-0.111\;[-1.459,+\infty]
Delphi+Value3.0953.095−0.104  [−1.679,+∞]-0.104\;[-1.679,+\infty]3.0423.042−0.157  [−1.465,+∞]-0.157\;[-1.465,+\infty]
Foresight3.0463.046−0.154  [−1.683,+∞]-0.154\;[-1.683,+\infty]N/AN/A
TTT-Linear3.0973.097−0.102  [−1.678,+∞]-0.102\;[-1.678,+\infty]N/AN/A
LaCT3.1743.174−0.026  [−1.661,+∞]-0.026\;[-1.661,+\infty]3.0853.085−0.115  [−1.446,+∞]-0.115\;[-1.446,+\infty]
MIRA3.1093.109−0.091  [−1.679,+∞]-0.091\;[-1.679,+\infty]N/AN/A
Latent ODE4.8804.8801.681  [−1.377,+∞]1.681\;[-1.377,+\infty]4.8764.8761.676  [−1.082,+∞]1.676\;[-1.082,+\infty]

In the original three-seed study, nine of ten models went from above Frequency’s loss at 32 draws to below it at 512 (Table 6). Its intervals measured variation across patients with the paths fixed. Here we ask what can change if we draw new paths for the same patients. All six intervals at 1,024 draws include zero.

AuthorsOpenReview ↗

Response to Reviewer JyFd (Part 3/5)

Clinical outcomes. Does a forecast that scores well on recorded events help predict an outcome within a clinical time window?

We tested whether simulated records help predict which patients will leave the ICU alive within 48 hours. We evaluated patients still alive in the index ICU stay at hour 24. Of 4,632 previously examined MIMIC-IV test patients, 4,620 had an assessable outcome and 2,932 left that ICU stay alive within 48 hours. We compared classifiers using observed history, features from 128 futures generated by a frozen 10M Transformer, or both. All three used the same 36,659 training patients and 4,598 validation patients.

Clinical endpoint results (new Sec. 5.4, Table 16). Brackets are 95%95\% patient-bootstrap intervals conditional on the fitted models and trajectories.

Classifier featuresAUROC [95%95\% CI]Average precision [95%95\% CI]Brier [95%95\% CI]
Observed history0.707  [0.691,0.723]0.707\;[0.691,0.723]0.767  [0.750,0.786]0.767\;[0.750,0.786]0.198  [0.193,0.204]0.198\;[0.193,0.204]
Rollouts0.689  [0.672,0.705]0.689\;[0.672,0.705]0.753  [0.734,0.770]0.753\;[0.734,0.770]0.199  [0.194,0.205]0.199\;[0.194,0.205]
History and rollouts0.699  [0.682,0.714]0.699\;[0.682,0.714]0.767  [0.750,0.785]0.767\;[0.750,0.785]0.200  [0.195,0.204]0.200\;[0.195,0.204]
Rollouts minus history−0.018  [−0.030,−0.006]-0.018\;[-0.030,-0.006]−0.014  [−0.031,4×10−5]-0.014\;[-0.031,4\times 10^{-5}]+0.001  [−0.001,0.004]+0.001\;[-0.001,0.004]
Combined minus history−0.009  [−0.018,0.001]-0.009\;[-0.018,0.001]−1.3×10−4  [−0.015,0.013]-1.3\times 10^{-4}\;[-0.015,0.013]+0.001  [−0.001,0.004]+0.001\;[-0.001,0.004]

History alone gave an AUROC of 0.707, compared with 0.699 after adding rollout features (paired change −0.009-0.009, 95%95\% CI [−0.018,0.001][-0.018,0.001]). We found no clear gain for this outcome. A patient who transferred alive from the ICU to a hospital ward counted as a live ICU exit.

Does predicting a creatinine measurement mean the model also predicted a rise in creatinine? We checked this in 40 patients with a recorded rise. Each model started from the 64 events observed at ICU hour 24 and generated 1,024 futures. At the recorded position, we first scored the probability of a creatinine measurement. We then counted that probability only on paths whose predicted value also met the rise criterion (Tables 12 and 13).

ModelIdentity NLLIdentity ++ rise NLLZero before floor, patientsIdentity Pass@5Identity ++ rise Pass@5
Foresight4.904.908.818.815/405/400.0430.0430.0090.009
MIRA4.874.878.788.787/407/400.0430.0430.0090.009
Delphi+Value5.125.1210.4610.4619/4019/400.0360.0360.0070.007

For Delphi+Value, adding the threshold increased loss from 5.12 to 10.46. In 19 of 40 patients none of its paths contributed any probability to the rise. We used a floor of one in a million to take the logarithm in those cases. A rise here means an increase of ≥0.3\geq 0.3 mg/dL above the lowest creatinine observed before ICU hour 24 or a value ≥1.5\geq 1.5 times that minimum.

What happens when we ask the conventional 48-hour question? Table 15 reports AUROC for the three models across five threshold-defined outcomes. At the same landmark and seed, each model generated 256 paths per patient. Positives met the rule within 48 hours. A negative patient needed a measurement in that window. Hypotension used MAP <60<60 mmHg, hypoxemia used SpO2<88%\mathrm{SpO}_2 < 88\%, tachypnea used respiratory rate >35>35/min, and tachycardia used heart rate >130>130/min.

Outcome (positive patients)Foresight AUROCMIRA AUROCDelphi+Value AUROCLast value AUROCPrior count AUROC
Creatinine rise (3737)0.540  [0.422,0.644]0.540\;[0.422,0.644]0.497  [0.385,0.607]0.497\;[0.385,0.607]0.335  [0.242,0.442]0.335\;[0.242,0.442]0.7420.7420.5720.572
Hypotension (2929)0.480  [0.361,0.604]0.480\;[0.361,0.604]0.519  [0.410,0.630]0.519\;[0.410,0.630]0.589  [0.461,0.711]0.589\;[0.461,0.711]0.6110.6110.5300.530
Hypoxemia (2020)0.537  [0.421,0.630]0.537\;[0.421,0.630]0.493  [0.437,0.527]0.493\;[0.437,0.527]0.517  [0.421,0.603]0.517\;[0.421,0.603]0.5040.5040.5390.539
Tachypnea (2828)0.556  [0.446,0.665]0.556\;[0.446,0.665]0.459  [0.354,0.576]0.459\;[0.354,0.576]0.471  [0.355,0.583]0.471\;[0.355,0.583]0.6780.6780.5630.563
Tachycardia (1515)0.576  [0.412,0.719]0.576\;[0.412,0.719]0.500  [0.345,0.646]0.500\;[0.345,0.646]0.492  [0.352,0.622]0.492\;[0.352,0.622]0.6780.6780.5670.567

The last observed value had higher AUROC than the rollout in 13 of 15 comparisons. Brackets give 95%95\% intervals for the rollout AUROCs. We estimated risk using paths that reached 48 hours or already contained the event, so early termination can affect the estimate.

Reviewer_sTqeOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a new benchmark for open-loop generative EHR rollouts. It evaluates 17 different models and scores them against three metrics: marginal event probability, candidate coverage, and occurrence within a future window.

Critical strengths
  1. This paper comprehensively evaluates 17 ehr foundation models, covering different tokenizations, backbones and loss functions.
  2. The benchmark compared performance on both teaching-forcing and open-loop tasks, showing how rankings could be different based on evaluation tasks.
Critical weaknesses
  1. The novelty of the paper is limited. It mainly proposes three new evaluation metrics and doesn't release any new tasks or datasets.

  2. The paper uses toy-scale setups. All models are around 10M parameters with at most 128 events as context. The evaluation cohort only includes 512 patients. However, most recent EHR foundation models use at least 50-100M parameters, 1k-2k context length and several thousand patients for evaluation cohorts [1][2]. The paper's conclusions may not generalize to a larger scale. For example, section 6.1 mentions that “Training on longer histories helps when the test input is also long.” However, if we scale up the context length beyond 1k, increasing the context length may decrease the downstream performance. (see figure 9 in the context clue paper [1])

  3. This benchmark has no clinical outcome tasks like mortality prediction or disease diagnosis. The performance of NLLs or coverage can’t reflect models’ performance in real-life clinical use cases.

  4. The benchmark only evaluates recorded event identity and ignores other event properties of EHR trajectories, like time interval and numerical measurement.

[1] Wornow, Michael, et al. "Context clues: evaluating long context models for clinical prediction tasks on EHR data." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Jeanselme, Vincent, et al. "FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records." arXiv preprint arXiv:2505.16941 (2025).

Questions for authors
  1. There is no in-depth analysis of the performance difference across different models. For instance, do any tokenizers, backbones, or loss functions favor downstream performance?
7 comments
AuthorsOpenReview ↗

Response to Reviewer sTqe (Part 1/6)

Dear Reviewer sTqe,

Thank you for reading the paper so carefully. We appreciate your recognition of the breadth of the 17-model comparison and your observation that model rankings can change with the evaluation task. We now test models up to roughly 100M parameters on two cohorts, evaluate a clinical outcome in 4,620 patients, and report timing and lab errors in hours and mg/dL.

W1. “It mainly proposes three new evaluation metrics and doesn't release any new tasks or datasets.”

REAL-EHR uses existing ICU data to study a forecasting task. Given only the record available now, how much probability does a model assign to the measurement that will be recorded 64 events later? It has to generate the intervening events itself. We use established scoring rules for event and occurrence probabilities and report candidate coverage separately. We provide rollout and scoring code and plan to release the trained checkpoints for all 17 adaptations. Anonymous code

At 32 draws per patient, RSSM had lower estimated loss than Foresight (3.565 vs. 4.193 nats). At 512 draws, Foresight had lower loss (3.073 vs. 3.173). We held the fitted models, patients and targets fixed. Nine of the ten models also went from above the Frequency baseline’s loss to below it. We added the Markov comparison in Table 7 to see whether sampling could actually favor the wrong model.

Event-64 negative log-likelihood (NLL) for the ten models evaluated at both budgets (Table 6). Lower is better. We give means ±\pm SD over three training seeds. The paired 95%95\% intervals in the final column resample patients while holding the simulated paths fixed.

ModelNLL at 32 drawsNLL at 512 drawsDifference from Frequency at 512 [95%95\% patient interval]
Transformer4.252±0.0484.252\pm 0.0483.125±0.0123.125\pm 0.012−0.075  [−0.157,+0.018]-0.075\;[-0.157,+0.018]
LaCT3.940±0.0913.940\pm 0.0913.181±0.0173.181\pm 0.017−0.019  [−0.080,+0.045]-0.019\;[-0.080,+0.045]
Foresight4.193±0.0494.193\pm 0.0493.073±0.0243.073\pm 0.024−0.127  [−0.193,−0.050]-0.127\;[-0.193,-0.050]
TTT-Linear4.132±0.0314.132\pm 0.0313.092±0.0053.092\pm 0.005−0.108  [−0.179,−0.026]-0.108\;[-0.179,-0.026]
RSSM3.565±0.0163.565\pm 0.0163.173±0.0323.173\pm 0.032−0.027  [−0.080,+0.028]-0.027\;[-0.080,+0.028]
MIRA4.185±0.0494.185\pm 0.0493.111±0.0093.111\pm 0.009−0.089  [−0.162,−0.002]-0.089\;[-0.162,-0.002]
Mamba-24.166±0.0104.166\pm 0.0103.103±0.0373.103\pm 0.037−0.097  [−0.166,−0.016]-0.097\;[-0.166,-0.016]
Delphi3.904±0.0513.904\pm 0.0513.109±0.0383.109\pm 0.038−0.091  [−0.158,−0.011]-0.091\;[-0.158,-0.011]
Delphi+Value3.938±0.0163.938\pm 0.0163.124±0.0313.124\pm 0.031−0.076  [−0.144,+0.002]-0.076\;[-0.144,+0.002]
Latent ODE5.260±0.5705.260\pm 0.5705.203±0.5225.203\pm 0.522+2.003  [+1.856,+2.150]+2.003\;[+1.856,+2.150]

How close was the estimated Markov loss to its exact value? (New Table 7) We scored the 64th future event after 16 observed events. Error is sampled minus exact loss, in nats. We repeated the simulation 32 times and evaluated all six budgets in each run.

Draws per patientMean sampled NLLMean signed errorSD across pools
32324.1954.195+0.996+0.9960.0700.070
64643.8323.832+0.633+0.6330.0700.070
1281283.5583.558+0.359+0.3590.0680.068
2562563.3733.373+0.174+0.1740.0440.044
5125123.2763.276+0.077+0.0770.0230.023
1,0241{,}0243.2303.230+0.031+0.0310.0160.016

Increasing the number of draws reduced mean error by 97%97\%. The exact loss favored Markov (3.199 vs. 3.200 nats for Frequency), yet every run favored Frequency even at 1,024 draws. The Monte Carlo intervals included zero.

AuthorsOpenReview ↗

Response to Reviewer sTqe (Part 2/6)

W2. “The paper uses toy-scale setups.” Do the results change when models are an order of magnitude larger?

At roughly 100M parameters, Hi-BEHRT made better teacher-forced predictions on both cohorts. Its open-loop forecasts improved on eICU and worsened on MIMIC-IV. We added these comparisons in Sec. 5.6 and Appendix D.4. Each row compares two sizes on the same patients. Under teacher forcing (TF), each model sees the recorded intervening events. In open loop (OL), it generates those events itself. We estimated loss at event 64 from 32 draws per patient. A negative change favors the larger model.

MIMIC-IV adult ICU, 512 patients (new Table 20).

ModelRegime10M NLL [95%95\% CI]100M NLL [95%95\% CI]Paired change [95%95\% CI]
CEHR-BERTTF0.394  [0.311,0.484]0.394\;[0.311,0.484]0.383  [0.300,0.475]0.383\;[0.300,0.475]−0.011  [−0.031,0.011]-0.011\;[-0.031,0.011]
CEHR-BERTOL3.831  [3.570,4.093]3.831\;[3.570,4.093]3.883  [3.608,4.163]3.883\;[3.608,4.163]0.052  [−0.092,0.196]0.052\;[-0.092,0.196]
CEHR-GPTTF0.383  [0.303,0.469]0.383\;[0.303,0.469]0.385  [0.301,0.475]0.385\;[0.301,0.475]0.003  [−0.016,0.021]0.003\;[-0.016,0.021]
CEHR-GPTOL3.810  [3.551,4.068]3.810\;[3.551,4.068]3.982  [3.703,4.280]3.982\;[3.703,4.280]0.172  [3×10−4,0.334]0.172\;[3\times 10^{-4},0.334]
ETHOSTF0.390  [0.305,0.480]0.390\;[0.305,0.480]0.387  [0.302,0.482]0.387\;[0.302,0.482]−0.003  [−0.023,0.018]-0.003\;[-0.023,0.018]
ETHOSOL3.848  [3.577,4.118]3.848\;[3.577,4.118]3.879  [3.614,4.144]3.879\;[3.614,4.144]0.031  [−0.103,0.161]0.031\;[-0.103,0.161]
Hi-BEHRTTF0.417  [0.330,0.512]0.417\;[0.330,0.512]0.379  [0.295,0.471]0.379\;[0.295,0.471]−0.038  [−0.074,−0.004]-0.038\;[-0.074,-0.004]
Hi-BEHRTOL3.706  [3.464,3.940]3.706\;[3.464,3.940]3.952  [3.677,4.232]3.952\;[3.677,4.232]0.247  [0.127,0.378]0.247\;[0.127,0.378]
LaCTTF0.398  [0.313,0.493]0.398\;[0.313,0.493]0.396  [0.313,0.487]0.396\;[0.313,0.487]−0.001  [−0.029,0.025]-0.001\;[-0.029,0.025]
LaCTOL3.890  [3.643,4.146]3.890\;[3.643,4.146]4.006  [3.738,4.275]4.006\;[3.738,4.275]0.115  [−0.067,0.304]0.115\;[-0.067,0.304]
Med-BERTTF0.401  [0.316,0.493]0.401\;[0.316,0.493]0.391  [0.309,0.482]0.391\;[0.309,0.482]−0.010  [−0.026,0.004]-0.010\;[-0.026,0.004]
Med-BERTOL4.233  [3.923,4.561]4.233\;[3.923,4.561]4.206  [3.880,4.524]4.206\;[3.880,4.524]−0.027  [−0.221,0.159]-0.027\;[-0.221,0.159]
TTT-LinearTF0.396  [0.313,0.486]0.396\;[0.313,0.486]0.395  [0.310,0.484]0.395\;[0.310,0.484]−0.001  [−0.025,0.023]-0.001\;[-0.025,0.023]
TTT-LinearOL4.108  [3.813,4.412]4.108\;[3.813,4.412]4.254  [3.937,4.577]4.254\;[3.937,4.577]0.146  [−0.066,0.357]0.146\;[-0.066,0.357]
TransformEHRTF0.401  [0.319,0.490]0.401\;[0.319,0.490]0.375  [0.292,0.465]0.375\;[0.292,0.465]−0.026  [−0.047,−0.006]-0.026\;[-0.047,-0.006]
TransformEHROL3.761  [3.513,4.018]3.761\;[3.513,4.018]3.938  [3.667,4.216]3.938\;[3.667,4.216]0.177  [0.037,0.312]0.177\;[0.037,0.312]

eICU, 395 patients per comparison except LaCT (389), from 512 source patients (new Table 21). TransformEHR had a teacher-forced result only.

ModelRegime10M NLL [95%95\% CI]100M NLL [95%95\% CI]Paired change [95%95\% CI]
BEHRTTF1.828  [1.672,1.997]1.828\;[1.672,1.997]1.812  [1.647,1.971]1.812\;[1.647,1.971]−0.017  [−0.100,0.068]-0.017\;[-0.100,0.068]
BEHRTOL4.804  [4.624,4.980]4.804\;[4.624,4.980]5.336  [5.108,5.566]5.336\;[5.108,5.566]0.532  [0.373,0.703]0.532\;[0.373,0.703]
CEHR-GPTTF1.705  [1.551,1.867]1.705\;[1.551,1.867]1.686  [1.533,1.836]1.686\;[1.533,1.836]−0.019  [−0.099,0.056]-0.019\;[-0.099,0.056]
CEHR-GPTOL5.169  [4.960,5.376]5.169\;[4.960,5.376]5.436  [5.249,5.625]5.436\;[5.249,5.625]0.267  [0.106,0.421]0.267\;[0.106,0.421]
ETHOSTF1.713  [1.559,1.871]1.713\;[1.559,1.871]1.696  [1.547,1.846]1.696\;[1.547,1.846]−0.016  [−0.104,0.065]-0.016\;[-0.104,0.065]
ETHOSOL5.820  [5.616,6.027]5.820\;[5.616,6.027]5.213  [5.037,5.386]5.213\;[5.037,5.386]−0.607  [−0.756,−0.460]-0.607\;[-0.756,-0.460]
Hi-BEHRTTF1.835  [1.670,2.009]1.835\;[1.670,2.009]1.686  [1.536,1.835]1.686\;[1.536,1.835]−0.148  [−0.247,−0.054]-0.148\;[-0.247,-0.054]
Hi-BEHRTOL5.865  [5.607,6.129]5.865\;[5.607,6.129]4.937  [4.758,5.121]4.937\;[4.758,5.121]−0.928  [−1.118,−0.744]-0.928\;[-1.118,-0.744]
LaCTTF1.660  [1.513,1.816]1.660\;[1.513,1.816]1.785  [1.602,1.974]1.785\;[1.602,1.974]0.125  [0.020,0.231]0.125\;[0.020,0.231]
LaCTOL4.740  [4.593,4.904]4.740\;[4.593,4.904]4.820  [4.658,4.978]4.820\;[4.658,4.978]0.080  [−0.055,0.220]0.080\;[-0.055,0.220]
Med-BERTTF1.747  [1.587,1.919]1.747\;[1.587,1.919]1.822  [1.646,1.996]1.822\;[1.646,1.996]0.075  [−0.030,0.173]0.075\;[-0.030,0.173]
Med-BERTOL4.765  [4.619,4.917]4.765\;[4.619,4.917]4.885  [4.713,5.060]4.885\;[4.713,5.060]0.120  [−0.024,0.272]0.120\;[-0.024,0.272]
TTT-LinearTF1.662  [1.503,1.823]1.662\;[1.503,1.823]1.846  [1.663,2.042]1.846\;[1.663,2.042]0.184  [0.075,0.300]0.184\;[0.075,0.300]
TTT-LinearOL4.830  [4.682,4.981]4.830\;[4.682,4.981]4.896  [4.734,5.071]4.896\;[4.734,5.071]0.066  [−0.074,0.207]0.066\;[-0.074,0.207]
TransformEHRTF1.825  [1.661,1.995]1.825\;[1.661,1.995]1.730  [1.579,1.883]1.730\;[1.579,1.883]−0.095  [−0.174,−0.025]-0.095\;[-0.174,-0.025]
TransformerTF1.596  [1.446,1.751]1.596\;[1.446,1.751]1.844  [1.655,2.040]1.844\;[1.655,2.040]0.248  [0.121,0.380]0.248\;[0.121,0.380]
TransformerOL4.797  [4.660,4.940]4.797\;[4.660,4.940]4.991  [4.793,5.180]4.991\;[4.793,5.180]0.194  [0.057,0.333]0.194\;[0.057,0.333]

For Hi-BEHRT, scaling up increased open-loop loss by 0.247 nats on MIMIC-IV and reduced it by 0.928 on eICU. The 95%95\% intervals resample patients while holding each seed-41 checkpoint and its own simulated paths fixed.

AuthorsOpenReview ↗

Response to Reviewer sTqe (Part 3/6)

W2 (continued). Can more history hurt, and how far do the 512-patient results generalize?

Yes. A longer test history hurt the Transformer on eICU (Sec. 6.1). Training on a mixture of history lengths improved the Transformer when both models saw 512 events at test time. But once that model was trained, giving it 512 events at test time produced a worse forecast than giving it 128.

Open-loop NLL on the same 304 eICU event-64 targets (Fig. 5, Table 22). The two training schedules shared targets and 50,000 updates. Mixed training used 128-, 256- and 512-event histories and processed 2.33 times as many input events. Results use three seeds and 512 draws per patient. Brackets are paired 95%95\% patient intervals for adding test history.

Model128-only training, 512 at testMixed training, 512 at testMixed minus 128-only trainingMixed training, 128 at test512 minus 128 at test, mixed fit
Transformer5.1275.1274.8294.829−0.298-0.2984.2474.247+0.582  [0.470,0.695]+0.582\;[0.470,0.695]
Mamba-24.4584.4584.3494.349−0.108-0.1084.3464.346+0.003  [−0.018,0.025]+0.003\;[-0.018,0.025]

The mixed-trained Transformer's loss rose by 0.582 nats, while Mamba-2's change was close to zero.

Does more history help predict a particular clinical event? (Table 11) For patients who later had a measured creatinine rise, we asked how much probability the model assigned to a creatinine measurement at the recorded step. Each cell gives NLL (lower is better) and EventPass@5, the chance that at least one of five candidates predicts that measurement (higher is better). We used 1,024 draws per patient and averaged results over three training seeds.

Patient groupModel12-event history36-event history64-event history
All positivesLaCT4.94/0.0464.94/0.0464.88/0.0484.88/0.0484.85/0.0504.85/0.050
All positivesTTT-Linear4.90/0.0424.90/0.0424.86/0.0454.86/0.0454.85/0.0464.85/0.046
All positivesRSSM5.02/0.0355.02/0.0354.98/0.0374.98/0.0374.94/0.0394.94/0.039
All positivesTransformer5.15/0.0345.15/0.0345.15/0.0365.15/0.0365.32/0.0335.32/0.033
All positivesLatent ODE5.75/0.0275.75/0.0275.75/0.0275.75/0.0275.76/0.0275.76/0.027
No prior labelLaCT5.01/0.0445.01/0.0444.96/0.0464.96/0.0464.94/0.0474.94/0.047
No prior labelTTT-Linear4.94/0.0414.94/0.0414.93/0.0444.93/0.0444.92/0.0444.92/0.044
No prior labelRSSM5.03/0.0355.03/0.0355.00/0.0365.00/0.0364.98/0.0384.98/0.038
No prior labelTransformer5.20/0.0335.20/0.0335.23/0.0345.23/0.0345.38/0.0325.38/0.032
No prior labelLatent ODE5.79/0.0275.79/0.0275.79/0.0265.79/0.0265.79/0.0265.79/0.026

The two samples included 512 patients with a measured rise and 348 selected for having no prior creatinine-rise label. They may overlap. LaCT improved modestly with more history in both samples, while Transformer’s loss increased with 64 events. Latent ODE’s results use only paths that reached the target. For 142 patients in the first sample, none did. Part 6 examines the harder question of predicting the rise in value as well as the measurement itself.

How much can we infer from 512 patients? We sampled the primary 512 from 4,747 eligible MIMIC-IV test patients. We could also check sampling error directly on three larger sets of targets because the Markov model has an exact solution (Fig. 9).

RepresentationEligible targetsSampled minus exact at 32Sampled minus exact at 512
MIMIC-IV, historical hospital-wide1,3851{,}385+0.404+0.404+0.026+0.026
eICU, train-fitted1,5531{,}553+0.741+0.741+0.073+0.073
MIMIC-III1,7421{,}742+1.122+1.122+0.135+0.135

In all three settings, increasing the budget from 32 to 512 brought the estimate closer to the exact loss. The 4,620-patient experiment below tests the clinical usefulness of one fixed Transformer.

A separate 10M–100M Transformer comparison (Sec. 5.6). For validation-selected checkpoints on another 512 MIMIC-IV patients (seed 41, 32 draws), the larger Transformer had slightly lower loss and slightly higher candidate coverage. Both paired intervals included zero.

Outcome at event 64100M minus 10M [95%95\% patient CI]
Marginal NLL, nats−0.104  [−0.306,0.101]-0.104\;[-0.306,0.101]
EventPass@5, fraction+0.004  [−0.012,0.020]+0.004\;[-0.012,0.020]
Reviewer_w9ntOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper deals with the issue of teacher forcing in the field of clinical event forecasting. The authors insist that the current methods are mainly evaluated using teacher forcing, which differs from real-world usage. To ensure a fair and more realistic evaluation scenario, the authors suggest open-loop forecasting instead of teacher forcing and compare it with conventional teacher forcing. Based on the evaluation results under this condition, the authors find that the metric and rollout are significant for evaluation.

Critical strengths
  • I believe that there is a need for such an open-loop evaluation method for EHR forecasting.
Critical weaknesses
  • I believe the novelty of this study is very limited. As the authors themselves also acknowledge, the difference between teacher forcing and open generation has been discussed in NLP community. While the authors mainly proposed a benchmark to compare teacher forcing and open-loop and demonstrated the difference between them, I think such a difference would already be expected by other researchers.
Questions for authors

See Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer w9nt (Part 1/2)

Dear Reviewer w9nt,

Thank you for your thoughtful review and for recognizing the need for open-loop evaluation in EHR forecasting. We agree that a difference between teacher forcing and open-loop forecasting is expected. In the revised Abstract and Introduction, we show how the choice of clinical question and even the number of simulated records per patient can change which model appears to perform best when both models forecast in open loop.

“I think such a difference would already be expected by other researchers.”

Two models changed places when we changed only the number of draws per patient. At 32 draws, RSSM had lower estimated loss than Foresight for the 64th future event (3.565 vs. 4.193 nats). At 512 draws, Foresight had lower estimated loss (3.073 vs. 3.173). We used the same checkpoints, the same 512 patients, and the same targets following 16 observed events at both budgets.

How often did the estimated ranking change? (Table 6) We report negative log-likelihood (NLL), where lower is better. The first two columns give means ±\pm SD across three training seeds. The last compares each model with Frequency, a baseline that predicts from training-event frequencies and has an exact NLL of 3.200 nats. The paired intervals resample patients while holding the simulated paths fixed.

ModelNLL at 32 drawsNLL at 512 drawsDifference from Frequency at 512 [95%95\% patient interval]
Transformer4.252±0.0484.252\pm 0.0483.125±0.0123.125\pm 0.012−0.075  [−0.157,+0.018]-0.075\;[-0.157,+0.018]
LaCT3.940±0.0913.940\pm 0.0913.181±0.0173.181\pm 0.017−0.019  [−0.080,+0.045]-0.019\;[-0.080,+0.045]
Foresight4.193±0.0494.193\pm 0.0493.073±0.0243.073\pm 0.024−0.127  [−0.193,−0.050]-0.127\;[-0.193,-0.050]
TTT-Linear4.132±0.0314.132\pm 0.0313.092±0.0053.092\pm 0.005−0.108  [−0.179,−0.026]-0.108\;[-0.179,-0.026]
RSSM3.565±0.0163.565\pm 0.0163.173±0.0323.173\pm 0.032−0.027  [−0.080,+0.028]-0.027\;[-0.080,+0.028]
MIRA4.185±0.0494.185\pm 0.0493.111±0.0093.111\pm 0.009−0.089  [−0.162,−0.002]-0.089\;[-0.162,-0.002]
Mamba-24.166±0.0104.166\pm 0.0103.103±0.0373.103\pm 0.037−0.097  [−0.166,−0.016]-0.097\;[-0.166,-0.016]
Delphi3.904±0.0513.904\pm 0.0513.109±0.0383.109\pm 0.038−0.091  [−0.158,−0.011]-0.091\;[-0.158,-0.011]
Delphi+Value3.938±0.0163.938\pm 0.0163.124±0.0313.124\pm 0.031−0.076  [−0.144,+0.002]-0.076\;[-0.144,+0.002]
Latent ODE5.260±0.5705.260\pm 0.5705.203±0.5225.203\pm 0.522+2.003  [+1.856,+2.150]+2.003\;[+1.856,+2.150]

Nine of ten models went from above Frequency’s loss at 32 draws to below it at 512.

Does sampling recover the correct ranking when we know the exact loss? (Sec. 5.3, new Table 7) Markov’s exact NLL was 3.199 nats on the same 512 patients, slightly lower than Frequency’s 3.200. We repeated the simulation 32 times, evaluating all six budgets in each run. Error is sampled minus exact loss. The SD measures variation across simulation runs for the same patients and model.

Draws per patientMean sampled NLLMean signed errorSD across pools
32324.1954.195+0.996+0.9960.0700.070
64643.8323.832+0.633+0.6330.0700.070
1281283.5583.558+0.359+0.3590.0680.068
2562563.3733.373+0.174+0.1740.0440.044
5125123.2763.276+0.077+0.0770.0230.023
1,0241{,}0243.2303.230+0.031+0.0310.0160.016

Increasing the budget cut mean error by 97%97\%. Yet even at 1,024 draws the remaining error exceeded the exact model difference of about 0.001 nats. All 32 runs favored Frequency at every budget. The Monte Carlo intervals included both zero and the exact difference and had no finite upper bound.

ETHOS used 20 simulated futures per discharge to estimate 30-day readmission risk. In the Markov comparison, even 1,024 were insufficient because the exact difference was so small. Renc et al. (2024)

AuthorsOpenReview ↗

Response to Reviewer w9nt (Part 2/2)

Novelty (continued). What do we learn by scoring the generated record alongside a clinical prediction task?

A model can predict who will receive a test even if it places that test at the wrong step in the record. We added both evaluations in Sec. 5.4. Starting from the last 64 events at ICU hour 24, we asked how much probability each model assigned to the test at its first recorded position, and how well it distinguished patients who received the test within 48 hours from those who did not.

Nine models with both scores and two clinical references (new Table 2B and Tables 12 and 14). NLL is in nats. Brackets are 95%95\% patient-bootstrap intervals for AUROC, conditional on the fitted model and its simulated paths.

PredictorLactate position NLLLactate 48-h AUROC [95%95\% CI]Troponin position NLLTroponin 48-h AUROC [95%95\% CI]
LaCT5.775.770.715  [0.610,0.797]0.715\;[0.610,0.797]6.276.270.770  [0.636,0.882]0.770\;[0.636,0.882]
RSSM4.684.680.646  [0.555,0.737]0.646\;[0.555,0.737]6.216.210.700  [0.557,0.835]0.700\;[0.557,0.835]
Transformer5.805.800.794  [0.709,0.869]0.794\;[0.709,0.869]7.027.020.771  [0.618,0.901]0.771\;[0.618,0.901]
TTT-Linear6.096.090.759  [0.670,0.837]0.759\;[0.670,0.837]6.536.530.766  [0.639,0.895]0.766\;[0.639,0.895]
Delphi6.246.240.638  [0.531,0.729]0.638\;[0.531,0.729]7.797.790.590  [0.413,0.744]0.590\;[0.413,0.744]
Foresight6.446.440.788  [0.709,0.866]0.788\;[0.709,0.866]6.876.870.869  [0.759,0.957]0.869\;[0.759,0.957]
MIRA6.406.400.769  [0.679,0.852]0.769\;[0.679,0.852]6.886.880.835  [0.706,0.942]0.835\;[0.706,0.942]
Delphi+Value6.296.290.650  [0.556,0.745]0.650\;[0.556,0.745]8.118.110.602  [0.436,0.748]0.602\;[0.436,0.748]
Mamba-26.166.160.761  [0.683,0.839]0.761\;[0.683,0.839]6.436.430.846  [0.723,0.941]0.846\;[0.723,0.941]
Last observed valueN/A0.7570.757N/A0.7930.793
Prior test countN/A0.6240.624N/A0.6790.679

Among the nine models, RSSM had the lowest average position NLL for both tests. Transformer had the highest 48-hour AUROC for lactate (0.794), and Foresight had the highest for troponin (0.869). A model that predicts a test three events early can still rank the right patients highly for the 48-hour outcome. Conversely, a model can assign high probability at the recorded step even if it also generates extra tests elsewhere.

We scored position on 40 positive patients per test, using 1,024 draws each (seed 41). Window AUROC used 256 draws and both positives and negatives. Of 300 sampled patients, 122 had assessable lactate status (67 positive) and 111 had assessable troponin status (15 positive). Risks use paths that produced the event or reached 48 hours. Across the nine models, the NLL–AUROC rank correlations were +0.28+0.28 for lactate and −0.28-0.28 for troponin (permutation p=0.46p=0.46 for each).

Did 1,024 draws suffice to distinguish a neural model from Frequency? (Sec. 5.3, Table 9) For six models we could compare both budgets using the same seed-41 checkpoints and the same 512 patients. The intervals show how much uncertainty in the loss difference comes from drawing a finite number of futures.

ModelSeed-41 NLL at 512Seed-41 NLL at 1,024Difference from Frequency at 1,024Monte Carlo interval for difference
Transformer3.1143.1143.0443.044−0.156-0.156[−1.480,+∞][-1.480,+\infty]
RSSM3.2103.2103.1863.186−0.014-0.014[−1.415,+∞][-1.415,+\infty]
Delphi3.1323.1323.0893.089−0.111-0.111[−1.459,+∞][-1.459,+\infty]
Delphi+Value3.0953.0953.0423.042−0.157-0.157[−1.465,+∞][-1.465,+\infty]
LaCT3.1743.1743.0853.085−0.115-0.115[−1.446,+∞][-1.446,+\infty]
Latent ODE4.8804.8804.8764.876+1.676+1.676[−1.082,+∞][-1.082,+\infty]

All six intervals at 1,024 draws include zero. The original teacher-forcing comparison (62 of 136 model pairs reversed at 32 draws) motivated the study.

Thank you for asking what the benchmark adds beyond the familiar teacher-forcing gap. We added the exact Markov comparison in Sec. 5.3 (Table 7). It shows how sampling alone can reverse the apparent ranking of two fitted models even when the patients and forecast targets stay the same.

Best regards,

The Authors

Thank you for your response. After carefully considering the response, I decided to keep my current ratings.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response

We thank the reviewers for their thoughtful feedback. Reviewer w9nt emphasized the need to evaluate EHR models in open loop, where models use their own predictions to forecast subsequent events. Reviewer JyFd highlighted the importance of measuring how well models predict clinical events over longer horizons. Reviewer sTqe noted the breadth of our comparison of 17 models with different tokenizations, architectures, and training objectives. They also noted that switching from teacher forcing to open-loop forecasting can change how the models rank. We summarize how we addressed the reviewers comments below:

1. What is new beyond the expected teacher-forcing gap? (w9nt) We added a comparison whose true ordering we can calculate. On 512 patients, Markov had slightly lower exact loss than the Frequency baseline (3.199 vs. 3.200 nats). Every one of 32 independent simulations favored Frequency, even at 1,024 draws per patient. The uncertainty intervals included zero (Sec. 5.3, Table 7).

2. Do the results extend to larger models and clinical outcomes? (sTqe) Scaling Hi-BEHRT from 10M to 100M parameters improved teacher-forced loss on both cohorts. Its estimated open-loop loss worsened on MIMIC-IV and improved on eICU (Sec. 5.6). On 4,620 patients, adding Transformer rollout features to observed history gave no clear gain in predicting live ICU exit within 48 hours (AUROC 0.699 vs. 0.707 for history alone, Sec. 5.4).

3. What does event-level scoring add to clinical AUROC? (JyFd) RSSM assigned more probability to lactate at its recorded position than Transformer (NLL 4.68 vs. 5.80). Transformer better distinguished patients who received the test within 48 hours (AUROC 0.794 vs. 0.646, Table 2B). We show why exact log and Brier losses reward accurate probabilities and why candidate coverage can reward an inaccurate forecast (Sec. 3.2, Appendix F). At JyFd’s and sTqe’s suggestion, we also evaluated elapsed time. LaCT and Frequency had similar event-identity loss, yet their mean timing errors were 470.39 and 2.13 hours (Sec. 5.5, Table 17).

Thank you for considering the revised paper.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.