- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates the topic of performance score inflation from selecting hyperparameters on the same test set used for reporting. Authors introduce a paired protocol comparing validation- and test-selected (optimal) hyperparameter configurations from the same pool obtained via a random search. Their evaluation is larger scale and covers 5 benchmarks, 8 architectures each and 30 repetitions each. They observe test-set selection causes detectable but generally small inflation of reported performance scores relative to run to run variance. They further observe validation and test selection often result in choosing the same configuration and rankings averaged across repetitions are often preserved. The study only covers fixed holdout random search with 30 configurations. It measures selection-based meta-overfitting, not adaptive overfitting, overtuning, or degradation on unseen data (compared to other related existing literature). Data presented in the paper supports modest score inflation in this narrow setting but not the general claim the authors make that test-set tuning is generally benign.
Critical strengths
Strengths of the paper are the larger scale empirical evaluation and the paired experimental design. These allow for isolating the effect of validation- vs. test-based selection across their large corpus of trained models. Moreover, the paper directly analyzes how leaderboard stability would be affected, which is relevant for the field of benchmarking and reporting even though the analysis only covers the setting where all models use the same selection protocol.
Critical weaknesses
In my opinion the two core weaknesses are:
- The paper does not adequately engage with and relate to the closest terminology and related literature. Most of what the paper discusses and presents as novel has been discussed and investigated in depth in existing literature. The measured gap in the paper largely corresponds to selection-based meta-overfitting, i.e., the known optimism from selecting and reporting the best configuration on the same holdout set (see Cawley and Talbot, 2010; Nagler et al., 2024; Schneider et al., 2025; Schröder et al., 2025). As a consequence, the empirical contribution and novelty are much more narrower than claim. Moreover, the term "adaptive overfitting" is misleading for a non-adaptive random search.
- The conclusions the authors draw are not supported by their experimental design. The study considers a fixed-holdout random search with 30 configurations but excludes resampling methods such as CV, the concept of reshuffling (Nagler et al., 2024), Bayesian optimization as an alternative to random search, checkpoint selection or heterogeneous tuning protocols. Moreover, the significance analysis presented in the paper will as I understand it almost always detect an effect because the defined gap is non-negative by construction, while leaderboard stability is evaluated only after averaging across repetitions and when all models use the same selection rule. The experiments overall only support modest overoptimism in a quite narrow setting but not the conclusion that test-set tuning is generally benign.
Literature:
Cawley and Talbot, 2010: https://www.jmlr.org/papers/volume11/cawley10a/cawley10a.pdf
Nagler et al., 2024: https://proceedings.neurips.cc/paper_files/paper/2024/file/47811ee68103bfcde7ca2223fccefb3a-Paper-Conference.pdf
Schneider et al., 2025: https://proceedings.mlr.press/v293/schneider25a.html
Schröder et al., 2025: https://link.springer.com/chapter/10.1007/978-3-031-91398-3_14
Questions for authors
How does the proposed () differ from selection-based meta-overfitting or selection-induced optimism as studied by Cawley and Talbot (2010), Schneider et al. (2025) and Schröder et al. (2025)? Please clarify the claimed novelty relative to this literature.
The leaderboard analysis assumes that all models use the same selection protocol and computes rankings from means over 30 resampled splits. How often do rankings change within individual runs or when a test-tuned model is compared against validation-tuned baselines?
Since () (as I understand it) by construction, the reported significance test will detect an effect whenever the two selection procedures differ sufficiently often. What information does this test provide beyond the frequency of nonzero gaps?
How we differ or follow existing literature, and that we model the current practice of test-tuning.
Thank you for your time, engagement, and critical questions.
Thank you for the citations! These papers show that comparing val-tuned models vs test-tuned models is a problem. There is quite some work on this, and we already cited work with similar findings e.g. (Schroder et al.,2025; Schneider et al., 2025; Makarova et al., 2021). Where our paper differs, is that we compare test-tuning models vs other test-tuned models. None of these papers do this, and we are the first to consider this setting. We evaluate the emerging practice of published papers that explicitly tune on the test-set. We added the citations, and clarified our novelty.
Apologies. This was a mistake, we removed the term from the paper.
We use a fixed-holdout, because that is the setting where published papers tune on the test set (see our paragraph with "Test set tuning is common practice" on page 2). We agree that more advanced settings are interesting, but we start with the setting for which we have evidence that it is already done in practice. We added this to page 4 "problem setup". We agree that this is interesting to investigate as follow-up research, which we added to the paper, see limitations on page 10.
Exactly. This is how existing work evaluates leaderboard stability, we follow Bouthillier et al. (2021) and we see no reason to deviate from accepted practice. We do go beyond their setting, and also investigate how bad test-tuning inflation is compared to other sources of variance, such as random data-splitting, where the gap is not non-negative by construction. We better clarified our intentions in the paper.
This is true, and not our intention. We changed the paper in multiple places to clarify that our conclusions are not general, but only the test tuning that is done by existing papers. This, in our opinion, is the setting that is interesting, because this is what is currently happening, and has not been investigated yet.
Good point, we should have done this. We added a thorough validation of our experimental setup, see "Exp 1: validating if the experimental setup captures overfitting" on page 5. See table 1 for these results.
This test validates our experimental setup, ie: if our setting sufficiently models the situation where test-tuning is better than val-tuning. We need to to show that our setting is suitable to model overfitting. It shows if our setting confirms with existing literature on comparing val-tuning to test-tuning. We clarified this in the paper, see the new caption of Fig 1.
The updated paper has improved by addressing all points of the reviewer. As such, we do not see on what grounds the paper should be rejected.