acceptodds
Under review as a conference paper at ICLR 2027

Stress-testing prediction models: A systematic robustness evaluation on large-scale ICU data

Abstract

Limited robustness to real-world perturbations is a key source of generalization failure in healthcare, yet it remains underexplored in systematic evaluations. We introduce a robustness evaluation that assesses multiple model classes under systematized clinically motivated perturbations ranging from missingness, measurement noise, label errors, to distribution shifts. Using three large-scale intensive care databases (MIMIC-III, MIMIC-IV, and eICU), we evaluate linear models, tree-based ensembles, neural networks, and tabular foundation models on both in-hospital mortality classification and ICU length-of-stay regression. Across most perturbation settings, transformer-based models exhibit smaller performance degradation than tree-based methods. However, robustness varies substantially across perturbation types and model implementations, and tuned gradient-boosting methods remain competitive in several settings. We further characterize the computational cost of robustness, finding that the robustness gains of tabular foundation models are accompanied by substantially higher prediction time than optimized gradient-boosting models. These results highlight that robustness is multidimensional and model-dependent, and that model selection for clinical deployment should consider not only predictive performance under nominal conditions, but also sensitivity to realistic data perturbations and computational constraints. Our benchmark provides a systematic framework for evaluating these trade-offs in clinical tabular prediction.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Robustness, tabular machine learning, healthcare, tabular foundation models, distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iZH3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper conducted an empirical study of model robustness on clinical tabular data using three ICU databases: MIMIC-III, MIMIC-IV, and eICU. It evaluates traditional machine learning models, neural networks, and tabular foundation models on two prediction tasks: in-hospital mortality and ICU length of stay. The study considers a broad range of perturbations and conducted extensive experiments.

Critical strengths

The paper considers a broad range of perturbations and compares traditional machine learning methods with recent tabular foundation models on large-scale ICU datasets.

Critical weaknesses

The authors claim to provide a systematic robustness evaluation of clinical prediction models and offer insights into their suitability for clinical deployment. Although the paper presents extensive experiments, with many additional results in the appendix, I have several concerns about the study design.

First, the study only considers mortality and length of stay prediction, while many other widely studied ICU tasks, such as sepsis and respiratory failure prediction, are not included.

Second, several experimental settings remain unclear, including patient overlap, the use of clinical baselines, missing-value handling across models, calibration, and context selection for foundation models.

Third, some perturbations seem quite far from real clinical settings. For example, the authors randomly combine three perturbations without explaining why these combinations were chosen. The synthetic missingness mechanisms also may not reflect how clinical data are collected in practice.

For an empirical study, I would expect more careful justification of these choices. While the paper reports many experiments, it is still unclear whether they are sufficient to support the broad claims about robustness and clinical deployment.

Questions for authors
  1. Patient-overlap. In the paper the authors restrict MIMIC-IV to admissions from 2014 onward avoids admission-level overlap with MIMIC-III. But the same patients may have been admitted in different years.

MIMIC-III and MIMIC-IV mainly reflects temporal rather than cross-institutional generalization.

  1. Clinical baseline For mortality prediction, SAPS II could be included.

  2. Fairness of model comparison. a. Authors described that different models use different missing-value handling strategies. Does it influence robustness? b. Whether Platt scaling was fitted using labeled samples from the external datasets or only from the development dataset?

  3. Foundation model setting How the context samples were selected during in-context inference?

  4. Prediction tasks. Why did the paper only consider mortality and length of stay prediction? Other common ICU prediction tasks, such as sepsis or respiratory failure prediction, could be included to better assess robustness.

  5. TYPES OF PERTURBATIONS

There are several concerns about the perturbation design. For example: The authors randomly combine three perturbations and evaluate 50 combinations. Why were three perturbations and 50 combinations chosen? Missingness in ICU data often depends on clinical decisions and patient conditions. The synthetic MAR and MNAR mechanisms may not reflect real clinical missingness patterns.

1 comment

We thank the reviewer for highlighting limitations within design and interpretation. We clarified that we constitute controlled robustness rather than a clinical deployment test.

Please find below our response.

1. We agree that temporal filtering cannot exclude patients who appear in both MIMIC-III and MIMIC-IV, since patient ID cannot be linked across releases. Our aim was to avoid overlapping admissions by restricting the MIMIC-IV subset, only when paired with MIMIC-III for subgroup analyses, to admissions whose earliest possible calendar year was 2013 or later.

We strengthened this distinction in the revised manuscript.

Moreover, We corrected an error in interpreting MIMIC-IV's patient-level reference-year interval; the eligible paired-analysis subset consequently increased from 32,062 to 38,737 patients. This correction does not affect the main robustness experiments, which did not use that filter. The affected subgroup results still reflect the original subset. Given the computational cost of the affected analyses we will rerun these specific analyses and update the associated results, figures, and interpretations in the camera-ready manuscript, should the paper be accepted.

2. We added SAPS II for in-hospital mortality and updated the results and figures. As a fixed clinical score, its predictions are unaffected by perturbations applied during training.

3. Our objective is to assess the robustness of methods under their standard modeling pipeline, including native missing-value handling when available, rather than imposing a common imputation procedure across all models. This choice reflects how these methods are implemented in practice and avoids disadvantaging models designed to handle missing values directly. Previous work has also highlighted the effectiveness and computational efficiency of native missing-value handling strategies, particularly for tree-based models [1].

We have revised the Discussion to explicitly acknowledge this limitation and clarify that our findings characterize the robustness of complete modeling pipelines.

For external validation, Platt scaling was tuned only on the development data. The calibrated models were evaluated on external cohorts without recalibration or access to external outcome labels.

This has been clarified in the revised section 2.3 (line 189-197).

4. All foundation models were provided with the complete training fold. The model versions used in our experiments do not perform internal subsampling of the training samples provided for in-context inference.

This has been clarified in the section 2.3 (line 180-182).

5. We selected in-hospital mortality as a consistently recorded binary endpoint with an established SAPS II comparator, and length of stay to assess regression robustness. Adding sepsis or respiratory failure would require additional, harmonized outcome definitions and prediction horizons across datasets. Sepsis labels can vary across clinical definitions [2]; for respiratory failure, consistent identification of conditions such as ARDS may require information, including imaging, that is not comparably available across these datasets [3, 4].

6. Each of the 50 sampled composites sequentially applies three distinct perturbation families, with randomly selected subtypes, noise levels, and order. We chose three to examine joint degradation while retaining interpretability and 50 for computational tractability, not statistical optimality.

We agree that synthetic mechanisms can not be assumed to fully reproduce ICU measurement processes, which depend on patient condition, clinical decisions, and institutional practice. There is no single missingness percentage that can represent all ICU variables, patients, hospitals, and clinical settings. However, one method that performs well across different noise levels is also more likely to perform at task-specific perturbation level. Therefore, broad severity ranges are designed for controlled sensitivity analyses, not as estimates of clinically typical missingness, input noise, or label noise levels. As also stated in answer to reviewer 1, we believe a broad range of perturbations allows the ML community to design models that are broadly robust, whereas researchers applying ML in ICU can report their own observed perturbation mechanisms and levels informing their methodological choice. Related simulation work also varies missingness or noise conditions [5–8].

We clarified this distinction in Methods and added clinically informed perturbations as a future direction in the Limitations.

  1. Perez-Lebel et al., GigaScience, 2022
  2. Rhee et al., Critical Care, 2016
  3. Levy et al., Critical Care Medicine, 2025
  4. Marshall et al., Intensive Care Medicine Experimental, 2026
  5. N'kam Suguem et al., Bioinformatics, 2026
  6. Ehrig et al., JMIR Medical Informatics, 2025
  7. Luijken et al., Statistics in Medicine, 2019
  8. Yang et al., BMC Medical Informatics and Decision Making, 2024
Reviewer_mwW1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This study presents a comprehensive robustness benchmark for clinical tabular prediction across three large-scale ICU cohorts (MIMIC-III, MIMIC-IV, and eICU) on mortality classification and length-of-stay regression tasks. The authors systematically stress-test linear models, gradient-boosted decision trees (GBDTs), neural networks, and tabular foundation models against clinically realistic univariate and composite data perturbations, while also evaluating the impact of hyperparameter optimization. Although tabular foundation models like TabPFNv3 generally demonstrate superior resilience to data corruption and distribution shifts compared to tree-based ensembles, their advantage is dataset-dependent and comes at the cost of orders-of-magnitude slower inference times (e.g., 5.12s vs. 0.02s for LightGBM). Ultimately, the findings demonstrate that clinical model selection cannot rely solely on clean-data leaderboards, but must balance multi-dimensional robustness against computational and operational deployment constraints.

Critical strengths
  1. Perturbation mechanisms for robustness evaluation is well motivated and represents realistic data problems
  2. cross dataset validation and leakage control seems well executed
  3. Meaningful performance measurements of tabular foundation models in various settings and their performance constrasts well versus tree-based methods.
Critical weaknesses
  1. Formatting is all over the place. Citation brackets are not consistently used. Bolding and emphasis are illogical.
  2. There is a side-ways figure in the manuscript. There shouldn't be a side-ways figure in conference proceedings.
  3. AI polished paragraphs are much easier to read and in clear contrast to paragraphs that wasn't polished. It is unclear how much human effort was behind the AI written paragraphs given how poorly written the non-AI paragraphs appear.
  4. Most legend and labels in figures are entirely illegible.
Questions for authors

Please fix various formatting and writting issues (ideally with a good human editor) for an otherwise interesting paper.

1 comment

We thank the reviewer for identifying problems with the manuscript's presentation. We have revised the text and figures to improve clarity, consistency, and legibility.

Formatting and citations. We standardized citation formatting, terminology, and emphasis throughout the manuscript, particularly in the Introduction and Methods. We corrected inconsistently placed citations and revised unclear passages to improve readability and coherence.

Figures We corrected the orientation of Figure 1 and improved the legibility of axis labels and legends across figures in both the main text and appendix. The revised figures are included in the updated manuscript.

Tables. In response to Reviewer 1, we added tables reporting each model's performance degradation relative to its clean baseline across perturbation scenarios. To improve readability, the detailed results tables in the appendix are presented in landscape orientation, while the main-text table retains portrait orientation to accommodate the 10-page limit.

Scientific writing. We acknowledge the concerns regarding uneven writing quality and have thoroughly revised the manuscript for clarity and consistency, incorporating feedback from all reviewers. In particular, the Introduction better highlights our contribution, the Model Development section clarifies the experimental protocol, the Results include the updated analysis with results on relative degradation, and the Discussion provides a revised interpretation of model robustness and its limitations.

AI-use statement. During the revision, we also used generative AI tools to assist with the interpretation and reporting of study results. We have updated the AI-use statement to transparently disclose this assistance.

Reviewer_xZmcOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper evaluates linear models, tree ensembles, RealMLP, and three tabular foundation models on MIMIC-III, MIMIC-IV, and eICU. It covers in-hospital mortality and ICU length of stay under label noise, input noise, missingness, class imbalance, reduced training size, feature shuffling, subgroup and dataset shift, and 50 random composite perturbations. It claims that foundation models degrade less than tree-based methods in most settings, that hyperparameter optimization (HPO) has inconsistent effects, and that robustness gains come with longer inference times.

Critical strengths
  • The perturbation design is broad and structured, and models see identical perturbations.
  • The evaluation uses nested cross-validation, external validation in all directions, and steps to reduce MIMIC-III/IV overlap. It includes an explicit HPO sensitivity analysis.
  • The limitations and reproducibility statements are candid.
Critical weaknesses

From my perspective, I found that the paper conducts good experiments on testing different machine learning models. However, I found it a bit difficult to capture what the novelty / contribution of the paper is to the ICLR community. It does not seem to offer significantly new insights other than running and stress-testing various models.

Other issues are:

  • The metrics measure performance under perturbation, not degradation.
  • The main finding depends on the development dataset.
  • External shift is confounded with differences in feature coding.
  • Absolute effect sizes are largely missing.
  • The paper offers little explanation of why models differ.
Questions for authors
  1. Can you report each model’s degradation relative to its own clean performance?
  2. Do foundation models receive the full training fold as context? If not, how is it subsampled, especially for eICU?
  3. Are HPO and Platt scaling run on clean or perturbed data, and is the development calibrator reused for external evaluation?
1 comment

We thank the reviewer for the constructive comments. We agree that the original manuscript did not sufficiently highlight our contribution to the ML community. While existing tabular benchmarks primarily compare predictive performance across datasets, systematic robustness comparisons between established tabular methods and recent tabular foundation models remain limited, particularly across diverse perturbation mechanisms and severity levels. As highlighted in [1], stress-testing AI models in healthcare across diverse perturbations can reveal important vulnerabilities that conventional benchmark evaluations may overlook, underscoring the importance of identifying such blind spots in healthcare applications. Our study addresses this gap through a unified evaluation of model robustness across multiple perturbations, development datasets, and prediction tasks. We believe a broad range of perturbations allows the ML community to design models that are broadly robust, whereas researchers applying ML in ICU can report their own observed perturbation mechanisms and levels informing their methodological choice. Importantly, our choice of perturbations was informed by a systematic review of ML robustness in healthcare ensuring that the evaluation framework addresses robustness challenges identified in clinical applications in ICU.

We have strengthened this contribution in the revised Introduction.

1. Relative degradation. We extended the results to measure each model's degradation relative to its own clean baseline. More specifically, for each perturbation scenario, we provide the performance on clean data, the performance at the highest level of noise, the absolute decrease and the relative degradation and the degradation relative to the clean performance. The results are computed both for AUC and MAE and presented in tables.

By considering the relative degradation, the interpretation of our study has changed. In particular, model rankings (e.g., under continuous-feature noise, TabPFNv3 experienced a higher relative AUC decrease, compared with to LightGBM, despite having a higher clean-data AUC) based on absolute predictive performance under perturbations do not always align with those based on relative performance degradation. We emphasize that relative degradation should be interpreted alongside absolute performance.

We revised the abstract, results, discussion, and conclusion section accordingly.

2. In-context inference. We thank the reviewer for requesting clarification of the in-context inference protocol.

All three tabular foundation models were provided with the complete training fold, including for eICU. The model versions used in our experiments do not perform additional internal subsampling of the training observations provided for in-context inference.

We have clarified this protocol in Section 2.3 of the revised manuscript (line 180-182).

3. HPO and Calibration protocol. The HPO procedure was conducted for each experimental scenario and perturbation severity level and within the inner cross-validation folds of each outer training fold. Therefore, depending on the scenario, HPO search receives the training data either in its perturbed or clean state: Training-set perturbations: HPO used the corresponding perturbed training data. Validation-set-only perturbations: HPO used clean training data. Training- and evaluation-set perturbations: HPO used the corresponding perturbed training data. Calibration. Similarly, calibration was applied after perturbation: training-only perturbations used unperturbed validation data for calibration, while validation-side perturbations used perturbed validation data. The predictive model remained fixed during calibration, and calibration samples were excluded from performance evaluation. For external validation, models were trained and tuned on 90% of the source development data, with the remaining 10% reserved for Platt scaling. The calibrator was evaluated on external cohorts without recalibration or access to external outcome labels.

We revised the model development section to better clarify the HPO and calibration protocol (line 175-197).

[1] Gu Y, el al. The illusion of readiness: Stress testing large frontier models on multimodal medical benchmarks, 2025. https://arxiv.org/abs/2509.18234

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.