- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper discussed about the evaluation of ICU vital-sign forecasts. And emphasize they should be evaluated through standard composite risk score, rather than by MSE alone. Forecasts are mapped through a four-parameter NEWS2-style index, and scored against the index on observed vitals and against events within 24 hours (death, vasopressor start, dialysis start, ventilation) on MIMIC-IV. The authors report that the MSE gap between trained and zero-shot models largely disappears under the index, and that triage on a TimesFM forecast recovers 18% more deteriorations than triage on current vitals.
Evaluating forecasts at the decision level is a sensible idea, indeed. However, I am afraid the clinical premise is not well justified, and the headline gains rely heavily on favourable settings.
Critical strengths
- Evaluating vital sign forecasting models , is a well-motivated and practical question. Although, I don’t think it is that novel.
Gu, Xiao, et al. "Are time series foundation models ready for vital sign forecasting in healthcare?." ML4H@ NeurIPS. 2024.
Critical weaknesses
I am mainly concerned about the clinical premise of this work, and whether the headline gains hold under fairer settings.
Clarity and Presentation
- The paper is quite dense, and many terms come without context, e.g. "read-out", "rate-calibrated", "complete-case" and "quantisation ceiling". It took me a while to follow.
- There is no related work section, which makes it hard to position this work against existing early-warning and ICU forecasting studies.
Methodology
- NEWS2 is designed for spot checks on general wards rather than the ICU. Here only four of its seven parameters are kept (no supplemental oxygen, consciousness or temperature), and the cut-offs are applied unchanged (L139-143). In the ICU, most patients are on oxygen, sedated, or on vasopressors titrated to blood pressure, so I am not sure this index reflects how deterioration is actually picked up there.
- The index is applied to the mean over the forecast horizon rather than to a spot value). Averaging would therefore hide exactly what an early-warning score is meant to catch, right?
- Why forecast vitals and then score them, rather than predict deterioration directly? A direct classifier on the look-back window, at the same alarm budget, would be expected as a baseline.
Experiment and Results
- Threshold-free AUROC is basically identical for all six forecasting models (0.573 to 0.575, Table 8), and around 0.57 is quite low for triage use anyway. So I am not convinced by the zero-shot versus trained story.
- L250. Inclusion requires observed vitals within the forecast horizon, which conditions on the future, and treatment starts can fall inside the horizon
- Risk and triage results are on MIMIC-IV only, and eICU appears only in the forecasting leaderboard. Generalisability therefore remains unclear.
Questions for authors
I would expect some clarification on:
-
Why a four-parameter NEWS2 for ICU patients, and how do the conclusions change with the full NEWS2 or an ICU-specific score? Even threshold hold based baseline apply for parameters that would be better
-
How does a direct classifier predicting the 24-hour events from the look-back window perform at the same alarm budget?