acceptodds
Under review as a conference paper at ICLR 2027

Same Risks, Different Error Bars: Disagreement-Aware Uncertainty for Cross-Validated Model Selection

Abstract

A practitioner who chooses between two fixed classifiers by cross-validation reports the selected model's cross-validated score, usually with a fold-based or binomial error bar. Fold complements overlap heavily, so the selection is re-run on almost the same data in every fold, and when the two models are close the folds flip between them. We show that this adds a reuse term to the mean squared error of the score that the usual error bars omit, and that the term is set by how the two models make mistakes jointly, not by their risks alone: pairs with identical risks can differ in it fifteen-fold. A pathwise identity (the cross-validated score equals the winner's empirical risk plus 1/n times the margins of the folds whose selection flips) yields an exact finite-sample evaluator of this error for any law of the paired 2x2 loss table and any fold count dividing n. Applied to the observed table it gives a reuse-aware root mean squared error (RMSE), which apart from a floor on empty cells is the ideal paired-loss bootstrap of the select-and-score pipeline for the two fixed models, computed without resampling; its coverage is measured, not guaranteed. On a synthetic grid fixed before any run the usual intervals cover as little as 75.0% at nominal 95%, while the reuse-aware interval's worst cell is 0.895 against 0.857 for a constant widening matched to its width. On 252 cells built from open-weight language-model checkpoints its worst cell is 0.907, which an exactly calibrated interval would fall to or below with probability 0.07 (expected worst cell 0.917), against 0.877 for the binomial and Agresti-Coull intervals (probability below 1e-5). On our declared grid of 1212 cells of real fitted classifiers, however, a textbook Agresti-Coull interval does as well or better, mostly at n = 200 where the estimated table is noisiest; at n = 1000 the reuse-aware interval's worst cell is the higher, 0.898 against 0.878. In practice, with two fixed candidates, report the winner's empirical risk, which the identity recovers from a cross-validated score and its fold contrasts; a cross-validated score that is reported should carry the reuse-aware RMSE, not a fold-based or binomial error bar.

Primary area:
learning theory
Keywords:
cross-validation, model selection, uncertainty quantification, confidence intervals, coverage, bootstrap, model comparison, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UDjJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper looks at the uncertainty in the cross-validation score when performing model selection between two fixed classifiers. It points out that the sets used to select a classifier across CV folds can overlap heavily, and proposes a reuse-aware RMSE to quantify the uncertainty of the cross-validation score. As mentioned in the paper, this reuse-aware RMSE is "an uncertainty measurement for a number the practitioner already reports", and is not a model selection criterion; winners are still selected based on empirical risk on the full evaluation set.

Overall, I lean towards rejection. The paper considers an interesting question on the reliability of performance estimates reported after cross-validation for model selection. My concerns relate primarily to the significance and practical usefulness of method. The proposed uncertainty estimate may provide useful additional information, but its practical benefits are insufficiently demonstrated. The presentation and organisation of the paper unfortunately also make the main recommendations difficult to follow.

Critical strengths
  • The paper provides an exact evaluator for the reuse-aware RMSE quantity that they propose, that is fast even for large sample sizes.
  • The paper provides an interpretable explanation of the difference between the CV score and the full-sample winner’s empirical risk.
  • The authors also acknowledge that these intervals have no coverage guarantee and report settings in which established alternatives perform better. This transparency is commendable.
Critical weaknesses
  • The additional practical value of the proposed reporting procedure is insufficiently established. The proposed RMSE leaves model selection unchanged, and its contribution therefore lies in performance assessment. The paper does not sufficiently demonstrate how this additional information would inform practical decisions or improve on existing reporting approaches.

  • The manuscript is difficult to follow and is very dense at times, with the connections between the main points needing substantial improvement in my opinion. Please consider restructuring and rewriting the manuscript to make the setting, contribution, and practical takeaway more explicit.

Questions for authors
  1. What practical benefit does the proposed uncertainty estimate provide? Under what conditions should a practitioner prefer the CV score with the reuse-aware RMSE over the winner’s empirical risk?
  2. Can the proposed RMSE evaluator be extended to more than two candidate classifiers?
  3. Please clarify the applicability to multiclass classification. If I understand correctly this is possible within your framework but some clarity would be appreciated.
Reviewer_LRnGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes cross-validation of a selection between two fixed classifiers under paired binary losses. Its central identity expresses the cross-validated score as the full-sample winner's empirical risk plus a nonnegative correction from folds whose selections differ. The paper uses this identity to decompose mean squared error and compute it exactly for a specified paired-loss law, then proposes a plug-in RMSE and studies the resulting intervals. Validation combines algebraic derivations with synthetic and frozen-classifier experiments, alongside comparisons with binomial intervals.

Critical strengths
  1. The pathwise identity and MSE decomposition explain why equal marginal risks can produce different uncertainty after model selection. Sections S1-S2 provide derivations that account for ties and lead to an exact finite-sum evaluator for a supplied paired-loss law. Proposition 2 also identifies a precise condition for the sign of the reuse term. This is a useful analytical contribution that separates the role of joint errors from marginal accuracy.

  2. The empirical analysis reports negative results and distinguishes exact evaluation, plug-in estimation, and interval coverage. The width-matched control and conventional binomial comparisons, including the better performance of Agresti-Coull on the declared real grid, make the practical limitations visible and help readers assess the contribution fairly.

Critical weaknesses
  1. The practical contribution is not convincingly established. The exact evaluator is proved for a supplied loss law, while the proposed procedure substitutes an estimated table. Section 6 acknowledges that near a tie the relative error of the reuse correction need not vanish, and the oracle-versus-plug-in comparison shows substantial remaining undercoverage. On the declared real grid, Agresti-Coull performs better on every coverage summary while being only slightly wider. Although other experiments show gains, the evidence does not establish a sufficiently clear regime of reliable practical benefit to justify the broad recommendation. The issue is the link between the exact analysis and a useful uncertainty procedure, rather than the absence of a universal coverage theorem.

  2. The theoretical presentation is too compressed in places. Section S2 introduces g1g_1, g2g_2, and eDGe_{DG} before explaining that they are moments involving Γ\Gamma, a different quantity from the main-text correction GG. Section 3 also uses p^\hat p for both the empirical table and the table after flooring and renormalization. The standing assumptions in Section 3 and the additional conditions in individual propositions could be linked more explicitly. A brief account of which conditions support the pathwise identity, MSE decomposition, and nonnegativity result would clarify their scope. Distinguishing the two tables and explaining the auxiliary moments before presenting the formulas would also make the connection to the plug-in procedure easier to follow.

Questions for authors
  1. In which regimes is the plug-in RMSE sufficiently reliable for the proposed normal-approximation interval? Please relate the recommendation to the near-tie estimation limitation and the reported undercoverage. A concise comparison with the winner-centered simultaneous Clopper-Pearson option already evaluated in Section S3 would help clarify the practical tradeoff, while preserving the different interval centers and common target. A universal coverage theorem is not needed to clarify the intended scope.

  2. How were the grid-wide noise floor and probability 0.07 calculated? Were coverage counts modeled as independent binomials with success probability 0.95, how were differing replicate counts handled, and were evaluation samples or random draws shared across configurations? Please give the calculation and account for dependence if applicable.

  3. The empty-cell floor binds in roughly 15-17% of replicates at the worst cells. Is there evidence about sensitivity to the choice 1/(2n)1/(2n), separate from ordinary table-estimation error? If this has not been assessed, please say so and explain the resulting limitation on recommendations in the low-disagreement regime. This question concerns the plug-in interval, not the exact identities.

  4. Is the advice that nesting is unnecessary for a handful of candidates but useful for large searches intended only for the reported studies? Proposition 5 concerns selection from a fixed out-of-fold loss table, as Section S3 correctly notes, and does not establish a general rule for refitted learners. Clarifying the intended scope would determine how broadly the practical conclusion is supported.

Reviewer_wofdOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the uncertainty of the cross-validated score when a practitioner uses k-fold cross-validation on a labeled evaluation set to choose between two fixed classifiers (such as two released language model checkpoints) and ships the full-sample winner.

Critical strengths

Nice theoretical insight and exact resampling-free computation. Isolating the exact relationship between the cross-validated score and the full-sample winner's empirical risk via the flipped-fold margins makes the role of joint model disagreement clear. Turning this decomposition into an exact finite-sample evaluator that computes the ideal paired-loss bootstrap error without Monte Carlo simulation is a clean and practical methodological contribution.

Critical weaknesses

Narrow practical applicability of cross-validating two already-fixed models. Cross-validation is primarily needed when models are refitted on each fold or tuned over a larger hyperparameter grid. When comparing just two frozen models on an evaluation set, no parameters are fitted inside the folds, and as the paper's own experiments show, cross-validating the two-way selection never improves mean squared error over simply taking the winner's empirical risk on the full sample.

Questions for authors
  1. Given your finding that cross-validating the choice between two fixed models never improves mean squared error over simply reporting the full-sample winner's empirical risk on any real dataset tested, in what concrete practical scenario should a practitioner run cross-validation with the reuse-aware RMSE rather than just reporting the winner's plain empirical risk with an Agresti-Coull or union-bound interval?

  2. Can the exact finite-sample evaluator be extended to comparisons of more than two fixed models, such as selecting the best checkpoint from a leaderboard of five or ten language models, where fold-level flipping across overlapping complements is even more common?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.