acceptodds
Under review as a conference paper at ICLR 2027

A Cross-Validation Gain Does Not Identify the Value of Its Selection Information

Abstract

Cross-validation changes two things at once. It gives the selector more validation information, and it usually changes the deployed predictor, because the fold-trained models are averaged. A gain from the complete pipeline cannot say which change produced it. We separate the two with an output-matched design that varies one component at a time, and add a third component that often travels with them: a small labelled panel that confirms the final choice. The evidence is a replay of four TabArena model families on 51 tasks, planned and sealed before any of the 765 held-back outer splits was opened. The value of selection information depends on what it is added to. For the validation-selected predictor, full out-of-fold selection raises ROC-AUC by 0.91 points when a single fold model is deployed but by only 0.28 points once fold outputs are averaged, while averaging alone is worth 0.98 points; both components lower normalized RMSE on all 13 regression tasks. An ordinary average of separately selected holdouts recovers about three quarters of the AUC gain of cross-validation. The same change of workflow matters far more for a benchmark than for deployment. RealMLP leads CatBoost on most splits of 16 tasks under a single holdout and of 32 tasks under cross-validation with averaging, yet validation picks RealMLP in 86 to 88 percent of splits either way. A confirmation panel of 32 labels that re-ranks all 800 saved models lowers AUC by 1.24 points on every binary task, and a shortlist removes most of that damage. A controlled study with actual refitting shows that the selection advantage survives full-data refitting and puts a cost on each policy.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
cross-validation, model selection, ensemble learning, tabular learning, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VV7DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates two components of cross-validation-based pipelines: configuration selection and model averaging. To separate their effects, the authors design a controlled experiment in which one component is varied while the other is held fixed. The first component is configuration selection, which uses either out-of-fold validation predictions from all eight folds (CV) or predictions from a single held-out fold (holdout). The second component is output construction, which uses either a single fold-trained model bank or an average of models trained across eight folds.

The paper further investigates how evaluation workflows affect benchmark rankings compared with deployment decisions, showing that changes in the workflow can substantially alter comparisons between model families while having a smaller effect on which family is selected for deployment. Additional experiments examine the relationship between fold-model disagreement and averaging gains, the effect of using a small labeled confirmation panel to re-rank models, and the tradeoff between predictive performance and computational cost when models are refitted on the full training data.

Critical strengths
  • Controlled experimental design: The paper presents an approach for separating the effects of validation information and output model averaging. The 2×2 experimental framework in Section 2 allows the authors to analyze the contribution of each component independently, rather than attributing the entire performance gain to cross-validation. The results show that the benefits of additional validation information and fold-model averaging are not additive. In particular, the improvement from using full out-of-fold selection is substantially smaller when predictions are already averaged across folds.
  • Useful findings about evaluation workflow sensitivity: The results demonstrate that benchmark comparisons can change substantially depending on how models are selected and how their predictions are combined. This is relevant for researchers comparing machine-learning algorithms and interpreting reported benchmark results.
  • Evaluation across multiple prediction tasks and metrics: The authors investigate binary classification, multiclass classification, and regression across 51 TabArena tasks. The use of different metrics and multiple held-back splits provides useful evidence about the consistency of the observed effects within the studied benchmark.
Critical weaknesses
  • Limited generalizability across model-selection methods:The experiments rely exclusively on Greedy Ensemble Selection (GES). It would be interesting to see if similar findings hold under other selection methods, such as selecting the best single configuration (Caruana et al., 2004) or learning ensemble weights through Super Learner (van der Laan et al., 2007). Testing these alternatives using the existing saved predictions would strengthen the paper's conclusions.
  • Limited validation on independent datasets: The evaluation uses 765 held-back outer splits from the same 51 TabArena tasks considered during development. Although the analysis is appropriately conducted at the task level, the additional splits assess stability across resampling rather than generalization to previously unseen datasets. Evaluation on an independent collection of tasks would strengthen the evidence that the findings extend beyond the studied benchmark.
  • Unclear practical significance of the measured improvements: Although the paper reports statistically supported differences between selection and averaging policies, many of the absolute improvements in predictive performance are modest. A more explicit discussion of practical effect sizes, particularly relative to the additional computational cost of cross-validation, would help interpret the results.
  • Small multiclass sample size and unresolved effects. Both multiclass effects have positive task-level directions and Holm-adjusted p-values of 0.023, but their simultaneous confidence intervals contain zero. The authors appropriately classify them as unresolved; nevertheless, the limited number of multiclass tasks constrains the strength of the conclusions.
  • Limited comparability of the refitting experiment: Section 8 investigates full-data refitting using a separate collection of tasks, model configurations, cross-validation folds, and evaluation metric (half-Brier loss). Although this experiment is useful for studying practical deployment and computational costs, the authors do not sufficiently explain how its results support the broader conclusions established using ROC-AUC, log loss, and NRMSE in the TabArena replay.
Questions for authors
  • Generalization to other selection algorithms: How sensitive are the results to the choice of GES? In particular, does the relative contribution of validation information and averaging remain similar when selecting a single best-performing configuration instead of constructing a greedy ensemble?
  • Practical computational tradeoff: Given the relatively small improvement after full-data refitting and the substantially higher computational cost, under which task or dataset characteristics would the authors recommend full CV over a simpler holdout-and-refit procedure?
  • Comparison with existing findings: Although the authors identify their methodological contributions relative to TabArena, could they clarify which empirical conclusions are new, beyond the controlled factorization and statistical confirmation provided here?
  • Independent evaluation: Would the main findings remain consistent on previously unseen datasets, rather than additional train/test partitions of the datasets used during development?
Reviewer_AYGoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how much of the performance improvement associated with a cross-validation pipeline comes from additional validation information, and how much comes from averaging the predictions of fold-trained models. It introduces a two-by-two design that crosses the source of selection information—one holdout fold or the full out-of-fold prediction matrix—with the output construction—a single fold’s model bank or an average across fold-trained models. Greedy ensemble selection is used throughout.

The main study replays saved predictions from four model families on 51 TabArena tasks, using 765 outer splits that were held back during analysis development. On binary classification tasks, fold averaging under holdout selection improves ROC-AUC by 0.98 percentage points, while adding full out-of-fold selection information to averaged outputs provides a further 0.28 percentage points. An ordinary average of separately selected holdout predictors recovers approximately 75% of the overall AUC improvement from the complete CV pipeline.

The paper also examines changes in model-family comparisons and deployment choices, the effect of small additional labeled panels used to revise the final selection, and actual full-data refitting and computational costs in a separate study on 18 classification tasks. Its main contribution is an empirical framework for attributing gains to individual components of evaluation and deployment workflows.

Critical strengths
  1. The factorial design and equal-fit comparator provide a useful decomposition of pipeline gains
  2. The study handles statistical units carefully and evaluates meaningful deployment alternatives
Critical weaknesses
  1. The interpretation of partial substitution is stronger than the evidence for the interaction itself. The paper emphasizes that selection information contributes 0.91 AUC percentage points with single-fold outputs but only 0.28 percentage points with averaged outputs, describing the components as partial substitutes and highlighting an approximately threefold difference.However, the estimated AUC interaction is −0.63 percentage points, with a simultaneous 95% confidence interval of [−1.33, +0.07] and a Holm-adjusted p-value of 0.06. The win-share interaction interval also contains zero. Under the paper’s own prespecified criterion, these interactions are therefore unresolved.
  2. The claim that the workflow affects benchmarking much more than deployment is not established using directly comparable measures. The paper supports this claim by contrasting two observations: the number of tasks on which RealMLP beats CatBoost on a majority of splits increases from 16 to 32, whereas the selected model family changes in only 92 of 765 splits.
Questions for authors

1.Interpretation of the interaction. Given that the simultaneous confidence intervals for the AUC and win-share interactions include zero, should “partial substitution” and the approximately threefold difference be understood as descriptive findings or confirmatory conclusions? Please reconcile the wording across the abstract, results, and conclusion, and clarify the metrics and settings to which the interpretation applies. 2. A common basis for comparing benchmark and deployment sensitivity. Can the authors report benchmark-winner changes and deployment-family changes using the same aggregation procedure, preferably calculating rates within each task and then averaging tasks equally? How much does the selected predictor’s quality change in cases where the selected family remains unchanged? These results would materially affect my assessment of the claim that deployment is less sensitive. 3. Availability of reproducibility materials. Are the frozen plans, code, input hashes, and unit-level results available in the anonymous supplementary material? The provided PDF does not identify an accessible location. Please also clarify where the 82 scikit-learn configurations, exact task identifiers, group definitions, and sampling and splitting procedures for the local refit study are documented. 4. Confirmation-panel selection rules. How are the Q32/Q128 panels stratified, and how are ties between candidate scores resolved? With few labeled examples and hundreds of candidates, discrete AUC scores can make tie-breaking consequential. Please clarify whether these rules were fixed in advance and where their implementation is documented.

Reviewer_4h1UOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

As far as I can tell the authors study different ways of doing cross validation. The authors consider a tabular data benchmark and show that averaging different folds is mildly beneficial.

Critical strengths

My most generous reading is that the paper shows that averaging information from different folds in multi-fold cross validation is mildly beneficial.

I'm an expert on this topic and I really like this topic. But I can't quite say what this paper is actually about.

Critical weaknesses

The findings are rudimentary at best, essentially Table 1, a few numbers on one benchmark.

The numbers are all pretty similar. The sentence "On binary tasks, averaging the fold-trained outputs of a fixed one-fold selection raised the AUC of the chosen predictor by 0.98 [0.33, 1.63] points" refers to percent, as in, the difference is 1% or 0.01.

Averaging is anyway the default in cross validation. So, I'm not sure what's being demonstrated here.

This paper has all the marks of a Claude written paper. Lots of AI slop throughout like:

"add a third component that often travels with them."

"One pair of model families shows what is at stake"

Questions for authors

I recommend precisely formulating the question in a human-understandable way.

For this to be above the bar, it would need a far more comprehensive evaluation and a clear claim that's being demonstrated.

As it is now, I don't think that any minor questions that I could ask would make a difference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.