- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how clinical dataset construction affects predictive performance and model comparisons. The proposed self-contained dataset design system (SDDS) organizes construction choices into source, population, features, missingness, and longitudinal structure, while specifying controls for model training and evaluation. Experiments cover NHANES, EHRSHOT, and two private cohorts, with up to eight model families. Object-at-a-time studies are complemented by joint design-of-experiments analyses on the public datasets. The results describe design-dependent performance changes, missingness-stage and temporal sensitivities, and reversals in pairwise model orderings.
Critical strengths
-
Relevant question and substantial empirical effort. The study examines clinically meaningful construction choices across multiple cohorts, tasks, and model families. Considering both individual objects and joint factor variation provides a broader assessment than a conventional fixed-dataset comparison.
-
Clear organization and transparent qualifications.
Critical weaknesses
-
The contribution beyond prior data-centric evaluation work needs clearer differentiation.
The general premise has substantial precedent outside clinical prediction. Tschalzev et al. (2024), cited in the manuscript, propose a data-centric tabular evaluation framework and demonstrate that preprocessing and feature engineering change model rankings. More recently, Tian et al. (2026) describe a system-level LLM evaluation framework incorporating factorial analysis of main and interaction effects.
These studies do not duplicate the present clinical experiments, but they motivate a clearer explanation of SDDS’s additional contribution. Beyond the clinically motivated factors and experimental coverage, the additional empirical understanding appears modest. For example, the paper documents different sample-size versus feature priorities in NHANES and EHRSHOT but provides limited explanation of what accounts for this difference.
-
The practical interpretation of design priorities remains limited.
Figure 2 compares absolute performance changes across objects with different intervention ranges, candidate configurations, and baselines. The authors appropriately qualify the ordering as conditional on these choices. Nevertheless, its relevance to data-collection priorities would be stronger if the ordering were shown to persist under alternative reasonable intervention sets or configuration weights.
Figure 3 is explicitly descriptive, and Appendix B.4 identifies factors that cannot be separately estimated. This is not an incorrect analysis, but it limits attribution to individual construction choices. More focused comparisons would strengthen the connection between the broad summaries and practical recommendations.
-
Ranking reversals are not sufficiently characterized in terms of magnitude and reproducibility.
Table 2 counts a reversal whenever the observed model gap changes sign, without distinguishing near-ties from substantial changes in relative performance. The authors acknowledge the absence of uncertainty estimates. Because model selection is a central motivation, reporting reversal magnitudes and stability across training and perturbation realizations is important. Also, the substantial reversal in Table 3 may warrant a more detailed examination.
Ref:
- Tschalzev et al. (2024). A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data. NeurIPS.
- Tian et al. (2026). Beyond Benchmarks: Toward Causally Faithful Evaluation of Large Language Models. ICML.
Questions for authors
-
How sensitive are the reported design priorities to the intervention ranges and configuration weights? Which analyses retain a common test population, and which change its eligibility or composition?
-
What fraction of reversals remains after excluding near-ties or requiring meaningful model gaps in both configurations? How was the celiac example selected, and does it persist across training seeds and masking realizations?
-
Section 5.4 includes a paired missingness analysis at rate 0.4, but combines several different application stages within each comparison group. Could all-split and evaluation-only masking be compared directly under otherwise identical conditions? Please clarify the associated training, preprocessing, and validation procedures.