- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper conducted an empirical study of model robustness on clinical tabular data using three ICU databases: MIMIC-III, MIMIC-IV, and eICU. It evaluates traditional machine learning models, neural networks, and tabular foundation models on two prediction tasks: in-hospital mortality and ICU length of stay. The study considers a broad range of perturbations and conducted extensive experiments.
Critical strengths
The paper considers a broad range of perturbations and compares traditional machine learning methods with recent tabular foundation models on large-scale ICU datasets.
Critical weaknesses
The authors claim to provide a systematic robustness evaluation of clinical prediction models and offer insights into their suitability for clinical deployment. Although the paper presents extensive experiments, with many additional results in the appendix, I have several concerns about the study design.
First, the study only considers mortality and length of stay prediction, while many other widely studied ICU tasks, such as sepsis and respiratory failure prediction, are not included.
Second, several experimental settings remain unclear, including patient overlap, the use of clinical baselines, missing-value handling across models, calibration, and context selection for foundation models.
Third, some perturbations seem quite far from real clinical settings. For example, the authors randomly combine three perturbations without explaining why these combinations were chosen. The synthetic missingness mechanisms also may not reflect how clinical data are collected in practice.
For an empirical study, I would expect more careful justification of these choices. While the paper reports many experiments, it is still unclear whether they are sufficient to support the broad claims about robustness and clinical deployment.
Questions for authors
- Patient-overlap. In the paper the authors restrict MIMIC-IV to admissions from 2014 onward avoids admission-level overlap with MIMIC-III. But the same patients may have been admitted in different years.
MIMIC-III and MIMIC-IV mainly reflects temporal rather than cross-institutional generalization.
-
Clinical baseline For mortality prediction, SAPS II could be included.
-
Fairness of model comparison. a. Authors described that different models use different missing-value handling strategies. Does it influence robustness? b. Whether Platt scaling was fitted using labeled samples from the external datasets or only from the development dataset?
-
Foundation model setting How the context samples were selected during in-context inference?
-
Prediction tasks. Why did the paper only consider mortality and length of stay prediction? Other common ICU prediction tasks, such as sepsis or respiratory failure prediction, could be included to better assess robustness.
-
TYPES OF PERTURBATIONS
There are several concerns about the perturbation design. For example: The authors randomly combine three perturbations and evaluate 50 combinations. Why were three perturbations and 50 combinations chosen? Missingness in ICU data often depends on clinical decisions and patient conditions. The synthetic MAR and MNAR mechanisms may not reflect real clinical missingness patterns.
We thank the reviewer for highlighting limitations within design and interpretation. We clarified that we constitute controlled robustness rather than a clinical deployment test.
Please find below our response.
1. We agree that temporal filtering cannot exclude patients who appear in both MIMIC-III and MIMIC-IV, since patient ID cannot be linked across releases. Our aim was to avoid overlapping admissions by restricting the MIMIC-IV subset, only when paired with MIMIC-III for subgroup analyses, to admissions whose earliest possible calendar year was 2013 or later.
We strengthened this distinction in the revised manuscript.
Moreover, We corrected an error in interpreting MIMIC-IV's patient-level reference-year interval; the eligible paired-analysis subset consequently increased from 32,062 to 38,737 patients. This correction does not affect the main robustness experiments, which did not use that filter. The affected subgroup results still reflect the original subset. Given the computational cost of the affected analyses we will rerun these specific analyses and update the associated results, figures, and interpretations in the camera-ready manuscript, should the paper be accepted.
2. We added SAPS II for in-hospital mortality and updated the results and figures. As a fixed clinical score, its predictions are unaffected by perturbations applied during training.
3. Our objective is to assess the robustness of methods under their standard modeling pipeline, including native missing-value handling when available, rather than imposing a common imputation procedure across all models. This choice reflects how these methods are implemented in practice and avoids disadvantaging models designed to handle missing values directly. Previous work has also highlighted the effectiveness and computational efficiency of native missing-value handling strategies, particularly for tree-based models [1].
We have revised the Discussion to explicitly acknowledge this limitation and clarify that our findings characterize the robustness of complete modeling pipelines.
For external validation, Platt scaling was tuned only on the development data. The calibrated models were evaluated on external cohorts without recalibration or access to external outcome labels.
This has been clarified in the revised section 2.3 (line 189-197).
4. All foundation models were provided with the complete training fold. The model versions used in our experiments do not perform internal subsampling of the training samples provided for in-context inference.
This has been clarified in the section 2.3 (line 180-182).
5. We selected in-hospital mortality as a consistently recorded binary endpoint with an established SAPS II comparator, and length of stay to assess regression robustness. Adding sepsis or respiratory failure would require additional, harmonized outcome definitions and prediction horizons across datasets. Sepsis labels can vary across clinical definitions [2]; for respiratory failure, consistent identification of conditions such as ARDS may require information, including imaging, that is not comparably available across these datasets [3, 4].
6. Each of the 50 sampled composites sequentially applies three distinct perturbation families, with randomly selected subtypes, noise levels, and order. We chose three to examine joint degradation while retaining interpretability and 50 for computational tractability, not statistical optimality.
We agree that synthetic mechanisms can not be assumed to fully reproduce ICU measurement processes, which depend on patient condition, clinical decisions, and institutional practice. There is no single missingness percentage that can represent all ICU variables, patients, hospitals, and clinical settings. However, one method that performs well across different noise levels is also more likely to perform at task-specific perturbation level. Therefore, broad severity ranges are designed for controlled sensitivity analyses, not as estimates of clinically typical missingness, input noise, or label noise levels. As also stated in answer to reviewer 1, we believe a broad range of perturbations allows the ML community to design models that are broadly robust, whereas researchers applying ML in ICU can report their own observed perturbation mechanisms and levels informing their methodological choice. Related simulation work also varies missingness or noise conditions [5–8].
We clarified this distinction in Methods and added clinically informed perturbations as a future direction in the Limitations.