- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper looks at the uncertainty in the cross-validation score when performing model selection between two fixed classifiers. It points out that the sets used to select a classifier across CV folds can overlap heavily, and proposes a reuse-aware RMSE to quantify the uncertainty of the cross-validation score. As mentioned in the paper, this reuse-aware RMSE is "an uncertainty measurement for a number the practitioner already reports", and is not a model selection criterion; winners are still selected based on empirical risk on the full evaluation set.
Overall, I lean towards rejection. The paper considers an interesting question on the reliability of performance estimates reported after cross-validation for model selection. My concerns relate primarily to the significance and practical usefulness of method. The proposed uncertainty estimate may provide useful additional information, but its practical benefits are insufficiently demonstrated. The presentation and organisation of the paper unfortunately also make the main recommendations difficult to follow.
Critical strengths
- The paper provides an exact evaluator for the reuse-aware RMSE quantity that they propose, that is fast even for large sample sizes.
- The paper provides an interpretable explanation of the difference between the CV score and the full-sample winner’s empirical risk.
- The authors also acknowledge that these intervals have no coverage guarantee and report settings in which established alternatives perform better. This transparency is commendable.
Critical weaknesses
-
The additional practical value of the proposed reporting procedure is insufficiently established. The proposed RMSE leaves model selection unchanged, and its contribution therefore lies in performance assessment. The paper does not sufficiently demonstrate how this additional information would inform practical decisions or improve on existing reporting approaches.
-
The manuscript is difficult to follow and is very dense at times, with the connections between the main points needing substantial improvement in my opinion. Please consider restructuring and rewriting the manuscript to make the setting, contribution, and practical takeaway more explicit.
Questions for authors
- What practical benefit does the proposed uncertainty estimate provide? Under what conditions should a practitioner prefer the CV score with the reuse-aware RMSE over the winner’s empirical risk?
- Can the proposed RMSE evaluator be extended to more than two candidate classifiers?
- Please clarify the applicability to multiclass classification. If I understand correctly this is possible within your framework but some clarity would be appreciated.