acceptodds
Under review as a conference paper at ICLR 2027

From Forecast Error to Triage Decisions: Patient-Centered Evaluation of ICU Vital-Sign Forecasting

Abstract

In intensive care, a vital-sign forecast would be acted on through an early-warning score, a step function that bins each parameter into a few ordinal levels and sums the points. We therefore evaluate ICU vital-sign forecasts through that score. Each forecast is mapped through a four-parameter NEWS2-style risk index (respiratory rate, SpO, systolic blood pressure and heart rate), converted into triage levels and scored against two separate ground truths: the index on the observed vitals and events recorded in the 24 h after the forecast origin (termination, vasopressor start, dialysis start, invasive ventilation or any of these). On 347,648 windows from the MIMIC-IV dataset, the 9.3% masked-MSE lead of the best in-domain-trained model over the best zero-shot foundation models shrinks to 1% in weighted . With the same 9.6% of windows flagged in both states, so that every extra event recovered is one fewer false alarm, triage on a zero-shot TimesFM forecast recovers 18% more deterioration-bearing windows than triage on the currently observed vitals at  h (sensitivity 12.4% to 14.7%; 95% CI on the gain 13.9 to 22.3, resampled over patients). Ranked by the raw subscore rather than by level, the gain is 9.0%; ranked by level, it decays to 7.2% at 24 h. Squared error does not select the forecast model family that recovers the most events: at a forced 10% budget, GRU, the model with the lowest MSE, recovers fewer deteriorations than every zero-shot model (Chronos-2 minus GRU relative-gain points, 95% CI 3.0 to 9.4, paired on the same patients) and 27 of 36 paired zero-shot-minus-trained contrasts exclude zero. In addition, we report the harmonised benchmark behind the forecasts (MIMIC-IV-CLIF and eICU-CRD, six vitals, h, to h, 16 models) and release the risk and triage protocols as secondary evaluations for vital-sign forecasting benchmarks.

Primary area:
datasets and benchmarks
Keywords:
time-series forecasting, vital signs, intensive care, early-warning score, NEWS2, clinical evaluation, triage, time-series foundation models, zero-shot forecasting, scoring rules, MIMIC-IV, eICU

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6jK9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper discussed about the evaluation of ICU vital-sign forecasts. And emphasize they should be evaluated through standard composite risk score, rather than by MSE alone. Forecasts are mapped through a four-parameter NEWS2-style index, and scored against the index on observed vitals and against events within 24 hours (death, vasopressor start, dialysis start, ventilation) on MIMIC-IV. The authors report that the MSE gap between trained and zero-shot models largely disappears under the index, and that triage on a TimesFM forecast recovers 18% more deteriorations than triage on current vitals.

Evaluating forecasts at the decision level is a sensible idea, indeed. However, I am afraid the clinical premise is not well justified, and the headline gains rely heavily on favourable settings.

Critical strengths
  • Evaluating vital sign forecasting models , is a well-motivated and practical question. Although, I don’t think it is that novel.

Gu, Xiao, et al. "Are time series foundation models ready for vital sign forecasting in healthcare?." ML4H@ NeurIPS. 2024.

Critical weaknesses

I am mainly concerned about the clinical premise of this work, and whether the headline gains hold under fairer settings.

Clarity and Presentation

  • The paper is quite dense, and many terms come without context, e.g. "read-out", "rate-calibrated", "complete-case" and "quantisation ceiling". It took me a while to follow.
  • There is no related work section, which makes it hard to position this work against existing early-warning and ICU forecasting studies.

Methodology

  • NEWS2 is designed for spot checks on general wards rather than the ICU. Here only four of its seven parameters are kept (no supplemental oxygen, consciousness or temperature), and the cut-offs are applied unchanged (L139-143). In the ICU, most patients are on oxygen, sedated, or on vasopressors titrated to blood pressure, so I am not sure this index reflects how deterioration is actually picked up there.
  • The index is applied to the mean over the forecast horizon rather than to a spot value). Averaging would therefore hide exactly what an early-warning score is meant to catch, right?
  • Why forecast vitals and then score them, rather than predict deterioration directly? A direct classifier on the look-back window, at the same alarm budget, would be expected as a baseline.

Experiment and Results

  • Threshold-free AUROC is basically identical for all six forecasting models (0.573 to 0.575, Table 8), and around 0.57 is quite low for triage use anyway. So I am not convinced by the zero-shot versus trained story.
  • L250. Inclusion requires observed vitals within the forecast horizon, which conditions on the future, and treatment starts can fall inside the horizon
  • Risk and triage results are on MIMIC-IV only, and eICU appears only in the forecasting leaderboard. Generalisability therefore remains unclear.
Questions for authors

I would expect some clarification on:

  1. Why a four-parameter NEWS2 for ICU patients, and how do the conclusions change with the full NEWS2 or an ICU-specific score? Even threshold hold based baseline apply for parameters that would be better

  2. How does a direct classifier predicting the 24-hour events from the look-back window perform at the same alarm budget?

Reviewer_DrbxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper examines whether ICU vital-sign forecasting accuracy translates into useful risk assessment. It maps forecasts to a four-variable NEWS2-style risk index and evaluates both agreement with scores calculated from observed future vitals and detection of recorded deterioration outcomes within 24 hours. Forecast-based triage recovers more deterioration-bearing windows than scoring currently observed vitals at a matched alert budget, while squared-error rankings do not consistently identify the models that recover the most events. Forecasting is evaluated on MIMIC-IV and eICU, while the risk and outcome analyses are restricted to MIMIC-IV.

Critical strengths

The paper evaluates forecasts against recorded deterioration outcomes rather than relying solely on forecasting error or agreement with a derived risk score. Matched alert-budget comparisons and patient-clustered uncertainty analyses make the results more interpretable by controlling alarm volume and accounting for repeated observations within patients. Analyses across forecast horizons, ranking rules, and model families provide useful evidence that selecting a forecasting model by squared error alone can overlook differences relevant to downstream risk assessment.

Critical weaknesses

W1. The added contribution beyond closely related prior work is insufficiently distinguished. Gu et al., Are Time Series Foundation Models Ready for Vital Sign Forecasting in Healthcare? (ML4H 2024, PMLR 259), already evaluate forecasts through a NEWS2-based score using the same four physiological variables and window-averaged values (Section 3.1.2, Equations 2–3). They also identify discrepancies between numerical forecasting accuracy and clinically motivated evaluation, including competitive score-change prediction from zero-shot models despite weaker numerical accuracy (Section 4.1). This work is not cited. The present paper’s clearest extension is evaluation against recorded outcomes at matched alert budgets, which is meaningfully different from predicting score changes. However, Contributions 1–2 do not adequately distinguish this extension from the earlier framework and findings. W2. The central risk-index and outcome-based triage findings are evaluated only on MIMIC-IV, despite including eICU-CRD in the forecasting benchmark. This leaves unclear whether the principal findings generalize beyond a single dataset. W3. Presentation obscures the principal contribution and supporting evidence. The abstract and introduction compress numerous metrics, comparisons, and protocol details without sufficiently explaining their significance. Contribution 2 largely restates that rankings depend on the scoring rule, while Contribution 3 introduces “three estimands” without explicitly identifying the three quantities or clearly connecting them to the subsequent analyses. Changes in model subsets, evaluation populations, scoring schemes, and alarm budgets also make the analyses difficult to track. Although Section 5 explicitly distinguishes risk-score agreement from recorded-outcome discrimination, the changing evaluation configurations make it difficult to connect each headline claim to its supporting comparison.

Questions for authors

Q1. How does the proposed evaluation advance beyond Gu et al.’s forecast-based NEWS2 evaluation? Please distinguish the procedures and qualitative findings already established there from the additional findings enabled by recorded outcomes and matched alert budgets. Q2. Section 9 acknowledges that the zero-shot models’ advantage cannot be separated from differences in training objectives. Could the authors test whether changing how the in-domain models are trained, or correcting systematic over- or underprediction using validation data, reduces their disadvantage in detecting deterioration? Appendix A.4 explains differences between forecasting error and risk-score agreement, but how far does that explanation account for the outcome-detection results in Section 8?

Q3. In Figure 7(b), forecast-based risk scores achieve higher AUROC for dialysis and intubation than scores calculated from the observed six-hour mean vitals. Could the authors explain why using predicted rather than measured future averages produces better outcome detection through this risk score? What does this result indicate about the relationship between forecasting accuracy and outcome detection?

Q4. How does forecasting vital signs and converting them into a risk score compare with directly predicting whether deterioration will occur within the next 24 hours? Could the authors compare these approaches using the same patient history, training and test patients, outcome definitions, and number of alerts? This would help clarify the practical value of the intermediate forecasting step.

Reviewer_UfxmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper asks whether ICU vital-sign forecasts should be judged by what a clinician would do with them, and not only by squared error. The authors map forecasts through a four-parameter NEWS2-style risk index and then into triage levels. They score the result in two ways: agreement with the index computed on the observed vitals, and discrimination of recorded events in the next 24 hours (death or hospice, vasopressor start, dialysis start, invasive ventilation, and their union). On 347,648 MIMIC-IV windows, they claim three things. First, the 9.3% MSE gap between the best trained model and the best zero-shot model shrinks to about 1% in weighted κ. Second, at a matched alarm budget of 9.6% of windows, triage on a 6-hour TimesFM forecast recovers 18% more deteriorations than triage on the current vitals. Third, MSE does not identify the model family that recovers the most events, because zero-shot models beat trained ones in 27 of 36 paired contrasts.

Critical strengths
  1. Separating “agreement with the index” from “discrimination of recorded events” is a useful way to frame the evaluation. The paper also shows that the observed-state comparator is a ceiling in one stage and an ordinary comparator in the other.
  2. The authors are open about their limitations and include several helpful controls: smoothing baselines, level versus subscore ranking, an Expired-only termination check, and a length-of-stay control. Intervals are clustered by patient and model contrasts are paired.
Critical weaknesses
  1. The headline gains rest on favorable setups. The 18% gain uses TimesFM’s own T3 share as the budget. The comparator is cut down at random to match. The gain falls to 9.0% under subscore ranking. It falls to 6.2% at a forced 10% budget. The comparator is weak. The “compression” claim compares MSE with a bounded metric (κ). The real compression is about 2×.
  2. The family-level claim is not isolated. Zero-shot forecasts are more dispersed, not less. This does not fit the stated mechanism. Trained models under-call risk more. Tie-breaking may hurt them at a fixed budget. T3 occupancy is not reported. No recalibrated trained model is tested. Section 9 admits this. The 36 contrasts are not independent. No multiplicity correction is applied. The AR baseline and the adapted TimesFM are not scored for triage. The source of the gain is unclear.
  3. The clinical premise is not established. No evidence shows forecasts would be used through an early-warning score. Half the windows are already on organ support. Baseline AUROC is near chance. The unit is the window, not the patient. There are no alarms per patient-day or lead times. “Patient-centered” is not supported. All results come from MIMIC-IV. Pre-training contamination is not checked. There is no related-work section.
Questions for authors
  1. What are the headline gain and interval at a forced 10% budget on the full 542,987-window set, and under population bins?
  2. What are the T3 occupancy and tie-break share for each model? Do the family conclusions hold under continuous-score ranking and after affine recalibration of the trained forecasts?
  3. How do the AR baseline, a 48-hour mean or trend extrapolation, and the adapted TimesFM perform under triage?
  4. What are the event-level results (alarms per patient-day, lead time), and what are the results for patients not on organ support at the origin?
  5. On which split were the rate-calibrated thresholds fit?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.