- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates two components of cross-validation-based pipelines: configuration selection and model averaging. To separate their effects, the authors design a controlled experiment in which one component is varied while the other is held fixed. The first component is configuration selection, which uses either out-of-fold validation predictions from all eight folds (CV) or predictions from a single held-out fold (holdout). The second component is output construction, which uses either a single fold-trained model bank or an average of models trained across eight folds.
The paper further investigates how evaluation workflows affect benchmark rankings compared with deployment decisions, showing that changes in the workflow can substantially alter comparisons between model families while having a smaller effect on which family is selected for deployment. Additional experiments examine the relationship between fold-model disagreement and averaging gains, the effect of using a small labeled confirmation panel to re-rank models, and the tradeoff between predictive performance and computational cost when models are refitted on the full training data.
Critical strengths
- Controlled experimental design: The paper presents an approach for separating the effects of validation information and output model averaging. The 2×2 experimental framework in Section 2 allows the authors to analyze the contribution of each component independently, rather than attributing the entire performance gain to cross-validation. The results show that the benefits of additional validation information and fold-model averaging are not additive. In particular, the improvement from using full out-of-fold selection is substantially smaller when predictions are already averaged across folds.
- Useful findings about evaluation workflow sensitivity: The results demonstrate that benchmark comparisons can change substantially depending on how models are selected and how their predictions are combined. This is relevant for researchers comparing machine-learning algorithms and interpreting reported benchmark results.
- Evaluation across multiple prediction tasks and metrics: The authors investigate binary classification, multiclass classification, and regression across 51 TabArena tasks. The use of different metrics and multiple held-back splits provides useful evidence about the consistency of the observed effects within the studied benchmark.
Critical weaknesses
- Limited generalizability across model-selection methods:The experiments rely exclusively on Greedy Ensemble Selection (GES). It would be interesting to see if similar findings hold under other selection methods, such as selecting the best single configuration (Caruana et al., 2004) or learning ensemble weights through Super Learner (van der Laan et al., 2007). Testing these alternatives using the existing saved predictions would strengthen the paper's conclusions.
- Limited validation on independent datasets: The evaluation uses 765 held-back outer splits from the same 51 TabArena tasks considered during development. Although the analysis is appropriately conducted at the task level, the additional splits assess stability across resampling rather than generalization to previously unseen datasets. Evaluation on an independent collection of tasks would strengthen the evidence that the findings extend beyond the studied benchmark.
- Unclear practical significance of the measured improvements: Although the paper reports statistically supported differences between selection and averaging policies, many of the absolute improvements in predictive performance are modest. A more explicit discussion of practical effect sizes, particularly relative to the additional computational cost of cross-validation, would help interpret the results.
- Small multiclass sample size and unresolved effects. Both multiclass effects have positive task-level directions and Holm-adjusted p-values of 0.023, but their simultaneous confidence intervals contain zero. The authors appropriately classify them as unresolved; nevertheless, the limited number of multiclass tasks constrains the strength of the conclusions.
- Limited comparability of the refitting experiment: Section 8 investigates full-data refitting using a separate collection of tasks, model configurations, cross-validation folds, and evaluation metric (half-Brier loss). Although this experiment is useful for studying practical deployment and computational costs, the authors do not sufficiently explain how its results support the broader conclusions established using ROC-AUC, log loss, and NRMSE in the TabArena replay.
Questions for authors
- Generalization to other selection algorithms: How sensitive are the results to the choice of GES? In particular, does the relative contribution of validation information and averaging remain similar when selecting a single best-performing configuration instead of constructing a greedy ensemble?
- Practical computational tradeoff: Given the relatively small improvement after full-data refitting and the substantially higher computational cost, under which task or dataset characteristics would the authors recommend full CV over a simpler holdout-and-refit procedure?
- Comparison with existing findings: Although the authors identify their methodological contributions relative to TabArena, could they clarify which empirical conclusions are new, beyond the controlled factorization and statistical confirmation provided here?
- Independent evaluation: Would the main findings remain consistent on previously unseen datasets, rather than additional train/test partitions of the datasets used during development?