acceptodds
Under review as a conference paper at ICLR 2027

Beyond Model Leaderboards: Separating Dataset Design from Model Capability in Clinical Prediction

Abstract

Clinical prediction benchmarks typically fix a dataset and compare models, leaving the consequences of dataset construction underexamined. We introduce a self-contained dataset design system (SDDS) that separates design objects—source, population, features, missingness, and longitudinal structure—from confounding objects governing the model, task, preprocessing, and execution conditions. Individual-object studies and joint experiments over representative factors distinguish object impacts, within-object factor variation, joint factor effects, and changes in model comparisons. Across NHANES, EHRSHOT, and two private health-examination cohorts, covering up to eight model families, features and population generally produce the largest within-source impacts, while missingness has a smaller aggregate impact. Under joint variation, however, features lead sample size in NHANES, whereas sample size leads in EHRSHOT. Small aggregate impacts can also conceal consequential design choices: across the evaluated EHRSHOT missingness configurations, evaluation-only masking produces a task-balanced median AUROC loss of 0.070, compared with 0.002 for all-split masking. In a celiac prediction example, applying 5% missingness within the recent 30-day window reverses the ordering of histogram gradient boosting and a linear model. Dataset design therefore shapes model selection and priorities for data collection; neither follows from sample size or a single benchmark score alone.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Clinical Prediction, Dataset Design

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9LGBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how clinical dataset construction affects predictive performance and model comparisons. The proposed self-contained dataset design system (SDDS) organizes construction choices into source, population, features, missingness, and longitudinal structure, while specifying controls for model training and evaluation. Experiments cover NHANES, EHRSHOT, and two private cohorts, with up to eight model families. Object-at-a-time studies are complemented by joint design-of-experiments analyses on the public datasets. The results describe design-dependent performance changes, missingness-stage and temporal sensitivities, and reversals in pairwise model orderings.

Critical strengths
  1. Relevant question and substantial empirical effort. The study examines clinically meaningful construction choices across multiple cohorts, tasks, and model families. Considering both individual objects and joint factor variation provides a broader assessment than a conventional fixed-dataset comparison.

  2. Clear organization and transparent qualifications.

Critical weaknesses
  1. The contribution beyond prior data-centric evaluation work needs clearer differentiation.

    The general premise has substantial precedent outside clinical prediction. Tschalzev et al. (2024), cited in the manuscript, propose a data-centric tabular evaluation framework and demonstrate that preprocessing and feature engineering change model rankings. More recently, Tian et al. (2026) describe a system-level LLM evaluation framework incorporating factorial analysis of main and interaction effects.

    These studies do not duplicate the present clinical experiments, but they motivate a clearer explanation of SDDS’s additional contribution. Beyond the clinically motivated factors and experimental coverage, the additional empirical understanding appears modest. For example, the paper documents different sample-size versus feature priorities in NHANES and EHRSHOT but provides limited explanation of what accounts for this difference.

  2. The practical interpretation of design priorities remains limited.

    Figure 2 compares absolute performance changes across objects with different intervention ranges, candidate configurations, and baselines. The authors appropriately qualify the ordering as conditional on these choices. Nevertheless, its relevance to data-collection priorities would be stronger if the ordering were shown to persist under alternative reasonable intervention sets or configuration weights.

    Figure 3 is explicitly descriptive, and Appendix B.4 identifies factors that cannot be separately estimated. This is not an incorrect analysis, but it limits attribution to individual construction choices. More focused comparisons would strengthen the connection between the broad summaries and practical recommendations.

  3. Ranking reversals are not sufficiently characterized in terms of magnitude and reproducibility.

    Table 2 counts a reversal whenever the observed model gap changes sign, without distinguishing near-ties from substantial changes in relative performance. The authors acknowledge the absence of uncertainty estimates. Because model selection is a central motivation, reporting reversal magnitudes and stability across training and perturbation realizations is important. Also, the substantial reversal in Table 3 may warrant a more detailed examination.

Ref:

  • Tschalzev et al. (2024). A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data. NeurIPS.
  • Tian et al. (2026). Beyond Benchmarks: Toward Causally Faithful Evaluation of Large Language Models. ICML.
Questions for authors
  1. How sensitive are the reported design priorities to the intervention ranges and configuration weights? Which analyses retain a common test population, and which change its eligibility or composition?

  2. What fraction of reversals remains after excluding near-ties or requiring meaningful model gaps in both configurations? How was the celiac example selected, and does it persist across training seeds and masking realizations?

  3. Section 5.4 includes a paired missingness analysis at rate 0.4, but combines several different application stages within each comparison group. Could all-split and evaluation-only masking be compared directly under otherwise identical conditions? Please clarify the associated training, preprocessing, and validation procedures.

Reviewer_AD2xOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the self-contained dataset design system (SDDS), a framework that treats the construction of clinical prediction datasets as an experimental variable in model evaluation. Rather than fixing an existing dataset and comparing only models, the paper analyzes how model performance changes with dataset construction. The work is motivated by the observation that clinical prediction benchmarks typically keep the dataset fixed and conduct research solely through model comparisons, leaving the influence of dataset configurations such as age, sex, and feature availability underexamined. The paper systematically separates design objects (source, population, features, missingness, and longitudinal history) from confounding objects (model, task, preprocessing, and execution conditions), with the source constraining the feasible settings of the remaining design objects. Under these constraints, the paper evaluates NHANES, EHRSHOT, PHE1, and PHE2 using object-at-a-time (OAT) studies, which vary one design object at a time, and joint design-of-experiments (DOE) studies, which vary representative factors from multiple objects simultaneously. Through these analyses, the authors quantify the impact of dataset construction on model performance in clinical prediction.

Critical strengths

The main strength of the paper lies in its large-scale, systematic experiments and the way the results are organized. Using two public and two private datasets, the study evaluates 57 tasks (29 from public datasets) with up to eight model families, employing OAT and DOE studies with tens to hundreds of configurations per design object. This is valuable as a large-scale empirical measurement in the clinical domain that quantifies several common intuitions about dataset construction. Furthermore, the paper provides a detailed and well-organized breakdown of the design factors and transparently discloses its metric definitions and analytical limitations.

Critical weaknesses
  • The central argument of the paper, that model performance and behavior depend on dataset construction, may be seen as somewhat expected given that machine learning models fundamentally learn from the data distribution. To strengthen this claim, it would be important to go beyond showing that dataset construction changes results and to clearly demonstrate how large these effects are relative to model selection, why they occur, and what practitioners should do as a result. These aspects are not yet clearly addressed in the current version. In addition, the clinical characteristics and interpretation of the findings are not discussed in sufficient depth, and Evaluatology, which serves as the conceptual foundation of the analysis, is not described in enough detail.

  • Several aspects of the methodology could be further developed. Although the source is defined as a design object, it appears to function mainly as a criterion for separating experiments rather than as a factor that is actually varied. In addition, the section describing an object causal model does not include a formal causal analysis. The paper states that the closure of each intervention is recorded, but the actual dependent updates are not reported, and the admissibility criteria and the dependencies that determine which updates are allowed or blocked are not justified. Furthermore, the core OAT and DOE analyses are built on selecting data subsets and applying artificial masking, and the rationale for the chosen candidate levels is not provided.

  • Several aspects of the experimental design remain unclear. Changing the patient population can alter the test data itself, yet it is not possible to verify whether each comparison is made on the same test population. In addition, the number of repeated runs and how they were performed are not reported. The stage at which missingness is applied in the DOE also differs between datasets (training and validation for NHANES versus all splits for EHRSHOT), and no justification is provided for this choice. Most importantly, the absolute model performance under the baseline configurations, its reliability, and its variation across seeds or resampling are not reported. Without this information, it is difficult to rule out that the observed changes, including the ranking reversals, arise from noise rather than from dataset construction.

  • Several key metrics are missing from the reported results. Per-class performance changes for the multiclass tasks are not reported for individual models, and most results are summarized as medians across models or as aggregate metrics such as AUROC, which makes it difficult to observe model-specific trends in most analyses. In addition, the number and proportion of positive cases are not reported for each task. As a result, for tasks with few positive cases, it is not possible to determine whether changes in AUROC reflect the effect of dataset design or the instability of the metric itself.

  • The interpretation of the results largely remains at the level of describing the tables and figures. In most cases, the significance of the findings and the reasons behind them are not discussed, and the interpretations are not clearly stated. Many sections conclude with qualifications such as the results not being representative or requiring further validation, rather than with clear conclusions. These shortcomings weaken the overall completeness of the paper.

Questions for authors
  1. Beyond showing that dataset construction changes the results, can you demonstrate how large these effects are (relative to model selection), why they occur, and what should be done as a result? Specifically:
  • A direct comparison of effect sizes with model selection (decomposing the variation attributable to dataset design versus model choice)
  • A baseline that distinguishes these effects from chance (e.g., the reversal rate obtained when only the random seed or resampling is changed)
  • An explanation of the underlying mechanisms (e.g., distribution shift)
  • General principles that extend beyond the specific datasets
  • Practical recommendations (e.g., reporting standards or external validation)
  • The realism of the interventions (e.g., their connection to actual differences between hospitals)
  1. Can you provide a justification for how and why each dataset modification was chosen, grounded in clinical considerations or the data distribution?

  2. Can you report the performance and robustness of each model, and examine whether the trends observed in the existing analysis hold? If they differ, can you explain why, and strengthen the weaknesses of the existing analysis accordingly?

Reviewer_GCKcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that clinical prediction benchmarks fix a dataset and compare models, while ignoring how the dataset was built. It proposes a framework (SDDS) that splits dataset construction into five “design objects”: source, population, features, missingness and longitudinal history. It then perturbs each object across NHANES, EHRSHOT and two private health-exam cohorts, with tabular models from linear to TabPFN. It reports that feature and population choices usually move performance most, that missingness and history matter in specific configurations, and that 13–30% of model-pair orderings flip under some design change.

Critical strengths

At least the writting is better than the average AI slop this conference has been receiving.

Critical weaknesses

This paper presents a broad and comprehensive framework of clinical data and methods evaluation, but struggles to draw insightful findings or impactful conclusions. The framework's broadness impedes proper deep dives in to understanding underlying mechanisms, instead locks the authors into making observational claims. For example:

  1. Key findings such as "Features and Population generally dominate aggregate impacts" are rather unsurprising and common knowledge in my opinoin; of course dropping entire test panels from the inference problem would have significant impact in both the fitted model and its aggregate predictions.
  2. Another example when discussing Missingness: "Most missingness interventions produce changes near zero, consistent with its small aggregate impact in Figure 2. However, larger changes occur for particular units, mechanisms, and application stages. Thus, a small aggregate object impact does not imply that all missingness choices are inconsequential; Section 5.4 examines these conditional effects directly." which basically says "missingness doesn't have much effect except in few cases where it has big effect" (a self-proving statement), and further on reading 5.4, a reader's only takeaway would be "if you drop data with sufficient bias, your remaining data will be biased, and thus the inference results will show bias".
  3. Another example is the Celiac example the authors emphasized in abstract and discussions: when a model's performance improves by more than 0.1 AUROC (0.618 to 0.723) from removing samples, that usually warrant a detailed discussion of what potential mechanisms could be cause such odd behavior, and repeat experiments with different random draws would have been conducted to confirm if it is due to random lottery ticket-type issues. Instead the authors triumphantly declare that this observation "illustrates a broader pattern in the evaluated design spaces" and counts it as a win.

A separate issue is that the performance of baselines (the non perturbed ones) seems unreasonable. E.g. when population changes induce a R2 change of 1.72, that would require one of the comparison sets to have a significantly negative R2 - i.e. negative correlation at predicting the target, worse than even predicting the population mean.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.