- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes to evaluate EHR forecasting models by measuring the NLL for trajectories
Critical strengths
Long horizon prediction performance is important and useful.
Critical weaknesses
This paper is incredibly difficult to follow and makes no sense due to the authors skipping steps when explaining their work. A well written paper should introduce the problem, explain why the proposed solution is correct, walk through why existing alternatives didn't work, and then demonstrate that their solution works.
I don't even know what problem this paper is trying to solve? Why would you evaluate long-term trajectory prediction using NLL? What is wrong with the existing solution of defining a suite of prediction tasks and measuring AUROC? For an example of standard prior work, see Figure 3 in the ETHOS paper for how this is currently done: https://arxiv.org/pdf/2407.21124.
"yet most models are evaluated with teacher-forced next-event prediction" is flat out wrong.
That's not getting into how I'm not even sure the proposed setup is sound. Where are your proofs that your evaluation metrics are proper scoring rules? Why are you ignoring the time aspect of the problem.
The whole setup seems to solve a problem that is already solved (AUROC for candidate tasks), but in a worse way.
Questions for authors
What is wrong with the traditional healthcare ML evaluation setup of predicting a range of targets within a range of time horizons and measuring AUROC?
How do you know that your metrics are proper scoring rules?
6 comments
Response to Reviewer JyFd (Part 2/5)
Q2. “How do you know that your metrics are proper scoring rules?”
Two of the three scores are strictly proper when the forecast probabilities are exact. For a true event distribution at a fixed position and a forecast , excess expected log loss equals the Kullback–Leibler divergence from to . It is zero only when . For a binary event with true probability and forecast , expected Brier loss is , uniquely minimized at . Scoring only patients who received a test, as in the position analysis above, cannot establish calibration in all patients. We clarified this distinction in Sec. 3.2 and added the Brier derivation to Appendix F. Gneiting and Raftery (2007)
EventPass@K is not a proper scoring rule (Appendix F). It measures the chance that at least one of candidates includes the recorded event. When the true probabilities are (0.8, 0.1, 0.1), shifting probability toward the two rare events increases five-candidate coverage from 0.882 to 0.926, even as expected NLL gets worse.
| Forecast for three events | Expected NLL | Expected EventPass@5 |
|---|---|---|
| True probabilities | ||
| Forecast |
What changes when we estimate these probabilities from simulated paths? Taking a negative logarithm introduces upward bias in the expected NLL because the negative logarithm is convex. Two models can have different biases even with the same number of draws. For Brier loss, let be the fraction of independent paths containing the event and the observed label. The expected plug-in loss is . For , subtracting from the plug-in loss gives an unbiased estimate of the exact Brier loss. We report the uncorrected score used in the experiments and give the correction in Appendix F. Ferro (2014)
New exact check of the NLL estimator (Table 7). On 512 patients the exact event-64 NLL was 3.199 for Markov and 3.200 for Frequency. We repeated the sampled Markov calculation in 32 independent pools with six nested budgets per pool. Errors and across-pool SDs are in nats.
| Draws per patient | Mean sampled NLL | Mean signed error | SD across pools |
|---|---|---|---|
Increasing the budget reduced mean error by . Even at 1,024 draws, the error exceeded the exact gap of about 0.001 nats. Every point estimate at every budget favored Frequency. The simultaneous Monte Carlo intervals included both zero and the exact difference, with an infinite upper endpoint.
How large is the simulation uncertainty for the neural models? (Table 9) We held the 512 patients and one checkpoint per model (seed 41) fixed. is the estimated NLL minus the exact loss for Frequency. The brackets give simultaneous Monte Carlo intervals for this difference. Six of the ten models were evaluated at both budgets.
| Model | 512 draws: NLL | [MC interval] | 1,024 draws: NLL | [MC interval] |
|---|---|---|---|---|
| Transformer | ||||
| RSSM | ||||
| Mamba-2 | N/A | N/A | ||
| Delphi | ||||
| Delphi+Value | ||||
| Foresight | N/A | N/A | ||
| TTT-Linear | N/A | N/A | ||
| LaCT | ||||
| MIRA | N/A | N/A | ||
| Latent ODE |
In the original three-seed study, nine of ten models went from above Frequency’s loss at 32 draws to below it at 512 (Table 6). Its intervals measured variation across patients with the paths fixed. Here we ask what can change if we draw new paths for the same patients. All six intervals at 1,024 draws include zero.
Response to Reviewer JyFd (Part 3/5)
Clinical outcomes. Does a forecast that scores well on recorded events help predict an outcome within a clinical time window?
We tested whether simulated records help predict which patients will leave the ICU alive within 48 hours. We evaluated patients still alive in the index ICU stay at hour 24. Of 4,632 previously examined MIMIC-IV test patients, 4,620 had an assessable outcome and 2,932 left that ICU stay alive within 48 hours. We compared classifiers using observed history, features from 128 futures generated by a frozen 10M Transformer, or both. All three used the same 36,659 training patients and 4,598 validation patients.
Clinical endpoint results (new Sec. 5.4, Table 16). Brackets are patient-bootstrap intervals conditional on the fitted models and trajectories.
| Classifier features | AUROC [ CI] | Average precision [ CI] | Brier [ CI] |
|---|---|---|---|
| Observed history | |||
| Rollouts | |||
| History and rollouts | |||
| Rollouts minus history | |||
| Combined minus history |
History alone gave an AUROC of 0.707, compared with 0.699 after adding rollout features (paired change , CI ). We found no clear gain for this outcome. A patient who transferred alive from the ICU to a hospital ward counted as a live ICU exit.
Does predicting a creatinine measurement mean the model also predicted a rise in creatinine? We checked this in 40 patients with a recorded rise. Each model started from the 64 events observed at ICU hour 24 and generated 1,024 futures. At the recorded position, we first scored the probability of a creatinine measurement. We then counted that probability only on paths whose predicted value also met the rise criterion (Tables 12 and 13).
| Model | Identity NLL | Identity rise NLL | Zero before floor, patients | Identity Pass@5 | Identity rise Pass@5 |
|---|---|---|---|---|---|
| Foresight | |||||
| MIRA | |||||
| Delphi+Value |
For Delphi+Value, adding the threshold increased loss from 5.12 to 10.46. In 19 of 40 patients none of its paths contributed any probability to the rise. We used a floor of one in a million to take the logarithm in those cases. A rise here means an increase of mg/dL above the lowest creatinine observed before ICU hour 24 or a value times that minimum.
What happens when we ask the conventional 48-hour question? Table 15 reports AUROC for the three models across five threshold-defined outcomes. At the same landmark and seed, each model generated 256 paths per patient. Positives met the rule within 48 hours. A negative patient needed a measurement in that window. Hypotension used MAP mmHg, hypoxemia used , tachypnea used respiratory rate /min, and tachycardia used heart rate /min.
| Outcome (positive patients) | Foresight AUROC | MIRA AUROC | Delphi+Value AUROC | Last value AUROC | Prior count AUROC |
|---|---|---|---|---|---|
| Creatinine rise () | |||||
| Hypotension () | |||||
| Hypoxemia () | |||||
| Tachypnea () | |||||
| Tachycardia () |
The last observed value had higher AUROC than the rollout in 13 of 15 comparisons. Brackets give intervals for the rollout AUROCs. We estimated risk using paths that reached 48 hours or already contained the event, so early termination can affect the estimate.
Response to Reviewer JyFd (Part 1/5)
Dear Reviewer JyFd,
Thank you for your careful review and for recognizing the importance of long-horizon prediction. We rewrote the Introduction around a lactate test, evaluated both predictions side by side, and corrected our description of how earlier work evaluates clinical outcomes.
AUROC is appropriate for asking which patients will receive a lactate test within 48 hours. Suppose two models give each patient exactly the same risk. Their AUROCs will be identical, even if one model regularly generates the test three events early. If we also want to know whether the model anticipated a lactate measurement at the step where it was recorded, we can look at the probability it assigned to lactate at that step.
We evaluated both questions on lactate and troponin (Sec. 5.4, Tables 2B, 12 and 14). The nine models below started at ICU hour 24 with 64 observed events. RSSM had the lowest average negative log-likelihood (NLL) for predicting each test at its recorded position. Transformer had the highest 48-hour AUROC for lactate, and Foresight for troponin.
We scored the first recorded test position on 40 patients per test who received it, using 1,024 draws each. The model saw only the history. We used the recorded position after generation, and extra tests elsewhere did not affect this score. For AUROC we used 256 draws and patients with and without the test. Of 300 sampled patients, 122 had assessable lactate status (67 positive) and 111 had assessable troponin status (15 positive). Brackets give 95% patient intervals with the seed-41 checkpoints and simulated paths held fixed.
We use NLL to score the probability of the measurement recorded at one future step. For example, after seeing 16 events, the model generates many possible 63-event continuations. For each one it predicts a distribution for event 64. We average the probabilities assigned to the measurement that was actually recorded, then take the negative logarithm (Sec. 3.2, Eq. 1). Teacher forcing gives a valid likelihood for the whole observed sequence by the chain rule. At step 64, though, it has seen the intervening 63 events.
We agree, and we removed that claim. ETHOS already used simulated timelines to predict clinical outcomes. FoMoH evaluates how well models distinguish patients with and without an outcome, how well calibrated their risks are, and how performance varies across groups. Sec. 1 and Sec. 2 now credit these approaches and explain why a reader interested in the generated record might also want a score for a particular step. ETHOS · FoMoH