acceptodds
Under review as a conference paper at ICLR 2027

On Hyperparameter Tuning on the Test Set

Abstract

"Don't tune hyperparameters on the test set" is often stated in machine learning textbooks. Violating it is considered a cardinal sin that produces misleadingly optimistic results, corrupts benchmark integrity, and thus can even be interpreted as scientific fraud. Yet evidence suggests that test set hyperparameter tuning does occur in practice, making it all the more important to understand its actual consequences. So how bad is it, really? In this work we question this dogma and put it to an empirical test. We systematically study the magnitude of the performance inflation caused by tuning the hyperparameters on the test set for MNIST-1D, CIFAR-10, and three tasks from the GLUE benchmark. Our experiments show that while the effect is real and significant, it is frequently small relative to other sources of noise. In many cases, we find that tuning on the test set recovers exactly the same model as when tuning on the validation set. Most importantly, we find that the rankings of models remain essentially preserved after tuning on the test set and therefore that consistent test-set tuning does not invalidate benchmarks or model selection. Our results call for a more nuanced view of tuning hyperparameters on the test set, stimulating researchers to openly report test tuning.

Primary area:
learning theory
Keywords:
adaptive overfitting, hyperparameter optimization, test set reuse, benchmarking, random search

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PHSLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the topic of performance score inflation from selecting hyperparameters on the same test set used for reporting. Authors introduce a paired protocol comparing validation- and test-selected (optimal) hyperparameter configurations from the same pool obtained via a random search. Their evaluation is larger scale and covers 5 benchmarks, 8 architectures each and 30 repetitions each. They observe test-set selection causes detectable but generally small inflation of reported performance scores relative to run to run variance. They further observe validation and test selection often result in choosing the same configuration and rankings averaged across repetitions are often preserved. The study only covers fixed holdout random search with 30 configurations. It measures selection-based meta-overfitting, not adaptive overfitting, overtuning, or degradation on unseen data (compared to other related existing literature). Data presented in the paper supports modest score inflation in this narrow setting but not the general claim the authors make that test-set tuning is generally benign.

Critical strengths

Strengths of the paper are the larger scale empirical evaluation and the paired experimental design. These allow for isolating the effect of validation- vs. test-based selection across their large corpus of trained models. Moreover, the paper directly analyzes how leaderboard stability would be affected, which is relevant for the field of benchmarking and reporting even though the analysis only covers the setting where all models use the same selection protocol.

Critical weaknesses

In my opinion the two core weaknesses are:

  1. The paper does not adequately engage with and relate to the closest terminology and related literature. Most of what the paper discusses and presents as novel has been discussed and investigated in depth in existing literature. The measured gap in the paper largely corresponds to selection-based meta-overfitting, i.e., the known optimism from selecting and reporting the best configuration on the same holdout set (see Cawley and Talbot, 2010; Nagler et al., 2024; Schneider et al., 2025; Schröder et al., 2025). As a consequence, the empirical contribution and novelty are much more narrower than claim. Moreover, the term "adaptive overfitting" is misleading for a non-adaptive random search.
  2. The conclusions the authors draw are not supported by their experimental design. The study considers a fixed-holdout random search with 30 configurations but excludes resampling methods such as CV, the concept of reshuffling (Nagler et al., 2024), Bayesian optimization as an alternative to random search, checkpoint selection or heterogeneous tuning protocols. Moreover, the significance analysis presented in the paper will as I understand it almost always detect an effect because the defined gap is non-negative by construction, while leaderboard stability is evaluated only after averaging across repetitions and when all models use the same selection rule. The experiments overall only support modest overoptimism in a quite narrow setting but not the conclusion that test-set tuning is generally benign.

Literature:

Cawley and Talbot, 2010: https://www.jmlr.org/papers/volume11/cawley10a/cawley10a.pdf

Nagler et al., 2024: https://proceedings.neurips.cc/paper_files/paper/2024/file/47811ee68103bfcde7ca2223fccefb3a-Paper-Conference.pdf

Schneider et al., 2025: https://proceedings.mlr.press/v293/schneider25a.html

Schröder et al., 2025: https://link.springer.com/chapter/10.1007/978-3-031-91398-3_14

Questions for authors

How does the proposed (ΔAO\Delta_{\mathrm{AO}}) differ from selection-based meta-overfitting or selection-induced optimism as studied by Cawley and Talbot (2010), Schneider et al. (2025) and Schröder et al. (2025)? Please clarify the claimed novelty relative to this literature.

The leaderboard analysis assumes that all models use the same selection protocol and computes rankings from means over 30 resampled splits. How often do rankings change within individual runs or when a test-tuned model is compared against validation-tuned baselines?

Since (ΔAO≥0\Delta_{\mathrm{AO}}\geq0) (as I understand it) by construction, the reported significance test will detect an effect whenever the two selection procedures differ sufficiently often. What information does this test provide beyond the frequency of nonzero gaps?

1 comment
AuthorsOpenReview ↗

How we differ or follow existing literature, and that we model the current practice of test-tuning.

Thank you for your time, engagement, and critical questions.

The paper does not adequately engage with and relate to the closest terminology and related literature. ... As a consequence, the empirical contribution and novelty are much more narrower than claim.

Thank you for the citations! These papers show that comparing val-tuned models vs test-tuned models is a problem. There is quite some work on this, and we already cited work with similar findings e.g. (Schroder et al.,2025; Schneider et al., 2025; Makarova et al., 2021). Where our paper differs, is that we compare test-tuning models vs other test-tuned models. None of these papers do this, and we are the first to consider this setting. We evaluate the emerging practice of published papers that explicitly tune on the test-set. We added the citations, and clarified our novelty.

Moreover, the term "adaptive overfitting" is misleading for a non-adaptive random search.

Apologies. This was a mistake, we removed the term from the paper.

The conclusions the authors draw are not supported by their experimental design. The study considers a fixed-holdout random search with 30 configurations but excludes resampling methods such as CV, the concept of reshuffling (Nagler et al., 2024), Bayesian optimization as an alternative to random search, checkpoint selection or heterogeneous tuning protocols.

We use a fixed-holdout, because that is the setting where published papers tune on the test set (see our paragraph with "Test set tuning is common practice" on page 2). We agree that more advanced settings are interesting, but we start with the setting for which we have evidence that it is already done in practice. We added this to page 4 "problem setup". We agree that this is interesting to investigate as follow-up research, which we added to the paper, see limitations on page 10.

Moreover, the significance analysis presented in the paper will as I understand it almost always detect an effect because the defined gap is non-negative by construction, while leaderboard stability is evaluated only after averaging across repetitions and when all models use the same selection rule.

Exactly. This is how existing work evaluates leaderboard stability, we follow Bouthillier et al. (2021) and we see no reason to deviate from accepted practice. We do go beyond their setting, and also investigate how bad test-tuning inflation is compared to other sources of variance, such as random data-splitting, where the gap is not non-negative by construction. We better clarified our intentions in the paper.

The experiments overall only support modest overoptimism in a quite narrow setting but not the conclusion that test-set tuning is generally benign.

This is true, and not our intention. We changed the paper in multiple places to clarify that our conclusions are not general, but only the test tuning that is done by existing papers. This, in our opinion, is the setting that is interesting, because this is what is currently happening, and has not been investigated yet.

The leaderboard analysis assumes that all models use the same selection protocol and computes rankings from means over 30 resampled splits. How often do rankings change within individual runs or when a test-tuned model is compared against validation-tuned baselines?

Good point, we should have done this. We added a thorough validation of our experimental setup, see "Exp 1: validating if the experimental setup captures overfitting" on page 5. See table 1 for these results.

Since ... What information does this test provide beyond the frequency of nonzero gaps?

This test validates our experimental setup, ie: if our setting sufficiently models the situation where test-tuning is better than val-tuning. We need to to show that our setting is suitable to model overfitting. It shows if our setting confirms with existing literature on comparing val-tuning to test-tuning. We clarified this in the paper, see the new caption of Fig 1.

Rating: 1: Clear rejection

The updated paper has improved by addressing all points of the reviewer. As such, we do not see on what grounds the paper should be rejected.

Reviewer_ekr8OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper empirically investigates the effects of tuning hyperparameters directly on the test set (i.e. not using a separate validation set). The authors measure the resulting performance change and its impact on model rankings through experiments across multiple classification tasks and model architectures. They find that test-set tuning introduces statistically significant but generally modest performance inflation, while model rankings remain largely unchanged. Based on these findings, the authors argue that test-set tuning may be less harmful to benchmark reliability than commonly assumed.

Critical strengths
  • The paper investigates a relevant methodological question concerning the reliability of machine learning benchmarks. The underlying research question is truly interesting.
  • The experiments are extensive in computational scale, covering multiple model architectures and classification tasks.
  • The paired experimental design provides a controlled comparison between validation-set and test-set hyperparameter tuning.
  • The paper is generally clearly written, with well-organized experiments and informative visualizations.
Critical weaknesses
  • The paper does not provide sufficiently useful conclusions that would benefit the community. It essentially argues that test-set contamination is not so bad, but why should this change anything? Test-set tuning still introduces bias, there is still no justification for it, and hyperparameters should still be tuned on a separate validation set. It is unclear what practical implications the paper offers. While positive implications of this paper are unclear, the negative implications are substantial - this paper could encourage authors to commit val-test contamination, even in cases where such contamination could widely distort how favorable the claims of these authors may look.

  • I think this paper risks sending the wrong message to the community. The authors advocate for the normalization of test-set tuning, arguing that researchers should openly report it because its effects are relatively small. This cannot be accepted as motivation - authors should be candid about flaws in their experimental frameworks because avoiding this would push back science and would be unethical, not because the implications are allegedly less destructive. Presenting improper evaluation practices as relatively harmless sends the wrong message to the community. Showing that test-set contamination sometimes has limited consequences does not justify accepting it. I don't think we (as a community) should publish such papers without unequivocal, truly thorough empirical and theoretical results that endorse them (which I don't believe this paper demonstrates).

  • The hyperparameter search is very limited - I don't think it's enough to draw the conclusions drawn by the authors. The authors mostly test sensitivity to optimization hyperparameters. What about batch size, architectural hyperparameters, data augmentation, and other factors? Furthermore, they only consider random search, while real-world hyperparameter tuning is often iterative and adaptive.

  • I don't think that the experiments are convincing enough. If the authors want to challenge a notion so deeply grounded in machine learning, both theoretically and in how widely adopted it is, they need substantially more convincing evidence. One synthetic classification dataset (MNIST-1D), CIFAR-10, and three small GLUE classification tasks are not nearly enough. What if these datasets happen to support the authors' conclusions while others do not? This is especially concerning given the reliance on MNIST-1D, which is artificial and may not represent more complex datasets well.

  • The paper provides little theoretical reasoning for why the observed behavior occurs, or under what conditions it should hold. Given the established theoretical concerns about test-set contamination, the authors should explain more convincingly why their empirical observations arise and how they relate to those concerns.

  • The authors repeatedly argue that test-set contamination is relatively harmless because the resulting performance inflation is smaller than the run-to-run standard deviation. Why should this make it acceptable? A systematic bias can still be meaningful, especially when improvements between competing methods are often smaller than that bias.

  • The model-ranking experiments are also not convincing. The authors conclude that test-set tuning does not meaningfully change model selection, but only compare eight architectures per dataset, averaged over 30 runs, using identical tuning procedures and budgets. Real-world benchmarks involve different researchers, tuning procedures, and budgets, often with very small differences between methods. There is insufficient evidence that their conclusions about ranking preservation generalize to such settings.

  • The authors claim they are the first to explore deliberate tuning of HPs over the test set - this is novelty is overstated. There is overlapping work the authors should position against. See Varma and Simon (2006), Cawley and Talbot (2010), Koleiev et al. (2026)

Questions for authors
  1. The positive implications of this paper are unclear, while the negative implications of normalizing potentially misleading or dishonest experimental frameworks (i.e. such that tune HPs over the test set) are substantial, especially whilst the community is already drowning in potentially misleading/dishonest reports in papers within the LLM and generative AI era. For me to consider a score change, I'd ask the authors to respond to this concern.
  2. I also specifically ask the authors to address my concerns related to the experimental setup the paper presents. To change such a fundamental approach in how we perform HP tuning in ML, that has been considered basic for decades, one expects overwhelmingly thorough and extensive experimentation, that is probably impractical for a 9-pager at a conference. Conferences frequently publish results with limited experimental conviction compared to journals, but in this case of challenging something so fundamental, I think limited evidence and POC-style results are not enough to justify acceptance. I am curious how the authors respond to this concern.
  3. While I consider the weaknesses related to my questions 1,2 most important to consider a score change, I would like to see other weaknesses I raised addressed in the rebuttal.
2 comments
AuthorsOpenReview ↗

On experimental soundness and the role of academic discourse.

Thank you! We appreciate the caution, your time, and the engagement.

Technical concerns

The hyperparameter search is very limited - I don't think it's enough to draw the conclusions.

I don't think that the experiments are convincing enough. What if these datasets happen to support the authors' conclusions

The model-ranking experiments are also not convincing ... only compare eight architectures per dataset

These points rightfully question if our setup is sufficient We added a thorough validation of our experimental setup, see "Exp 1: validating if the experimental setup captures overfitting" on page 5. Our experimental setups captures overfitting.

The authors claim they are the first .. There is overlapping work

Thanks for the refs. Those are not unique in showing bias for HP test tuning, and we already cite work with similar findings eg (Schroder et al.,2025; Schneider et al., 2025; Makarova et al., 2021). What none of these works show, however, is an empirical analysis of explicit tuning all HPs and all models on the test-set, for which we are the first. We added the references and clarified the distinction.

... only consider random search, while real-world hyperparameter tuning is often iterative and adaptive.

Real-world benchmarks involve different researchers, tuning procedures, and budgets,

We follow the existing real-world benchmarks that do test-tuning (see "Test set tuning is common practice" on page 2) . We agree that more advanced settings are interesting, but we start with the setting for which we have evidence that it is already done in practice.See page 4 "problem setup". We agree that this is interesting to investigate as follow-up research, which we added to the paper, see limitations on page 10.

The paper provides little theoretical reasoning for why the observed behavior occurs, or under what conditions it should hold.

Agreed. We added a theoretical motivation on page 3.

Impact and ethical concerns.

It is unclear what practical implications the paper offers.

The positive implications of this paper are unclear

The paper does not provide sufficiently useful conclusions that would benefit the community.

We agree that authors should not do test tuning, but they do, and this is now the reality, and thus warrants investigation. Benefits include:

  • We show that existing literature with test-tuning results (see "Test set tuning is common practice" on page 2) should not be dismissed simply because it's test tuned, but these works do have value, at least when compared to other test-tuned results.
  • Because val-tuning will be worse, a fair comparison to published test-tuned work is not possible. If authors openly mention they test-tune, (instead of simply hiding it, as they do now) then our paper shows that comparing test-tuned models to each other is reasonable, offering a way out for new papers to compare to published test-tuned papers.

We added these benefits in the paper, see lines 064-068.

The authors repeatedly argue that test-set contamination is relatively harmless

We apologize for this unclarity. Contamination by comparing val-tuning to test-tuning is certainly not harmless. We updated the papers at several places to make this clearer.

one expects overwhelmingly thorough and extensive experimentation, that is probably impractical for a 9-pager at a conference.

I think this paper risks sending the wrong message to the community.

I don't think we (as a community) should publish such papers without unequivocal, truly thorough empirical and theoretical results that endorse them (which I don't believe this paper demonstrates).

.. harmful insights related to premature acceptance of potentially wrong experimental paradigms, that could lead to publication of wrong or misleading results in the future. These concerns are not sufficiently acknowledged or addressed by the authors

We addressed all technical concerns above. Yet, we feel the reviewer blocks the paper from publication because of its impact on the community. Is it a good reason to block a publication because its results are inconvenient? Should the community not decide what to do with these results openly? A single conference paper is just the first step, and the academic discussion should be had openly, by building on, or challenging the results through new empirical evidence in future conference papers.

We do agree that our paper could easily be misunderstood, and we rewrote the paper to make it clearer that we think comparing val-tuning to test-tuning is wrong, but that comparing test-tuned models among themselves is reasonable. See "Misinterpretation risks" on page 10.

1: Clear rejection

We addressed all technical concerns, the impact question remains: should the community not be allowed to see our results?

I saw the authors' replies and am updating my score to 2.

Reviewer_tEtVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission investigates the empirical impact of hyperparameter optimization conducted directly on the test set across five classification benchmarks: one-dimensional MNIST, CIFAR-10, and three tasks from the GLUE benchmark (RTE, MRPC, and CoLA). Evaluating eight distinct model architectures per benchmark, the authors establish a paired bootstrap evaluation framework across thirty independent data splits, with thirty randomly sampled hyperparameter configurations per trial, training a total of 36,000 models. The study measures the adaptive overfitting gap (defined as the test accuracy difference between models selected via test performance versus validation performance), the frequency of identical configuration choices across splits, and the preservation of model rankings. The central findings are threefold: first, hyperparameter selection on the test set leads to statistically detectable overfitting after testing as few as three configurations on language tasks and thirteen on CIFAR-10, yet the absolute magnitude of this gap remains small and consistently below the natural run-to-run standard deviation of validation-tuned models. Second, validation and test tuning select the exact same hyperparameter configuration in 13 to 48% of trials. Third, architectural rankings remain largely preserved, with the top-ranked model unchanged across all benchmarks. On this basis, the authors propose normalizing the practice of test-set hyperparameter tuning and encourage researchers to report it openly rather than treating it as scientific misconduct.

Critical strengths
  1. Rigorous Paired Protocol and Large Empirical Scale: the paired-run design cleanly isolates the hyperparameter selection rule from data-split variability and initialization noise across 36,000 trained models. This extensive computational effort provides a controlled and highly reproducible empirical baseline.
  2. Timely Investigation of an Overlooked Practice: the paper directly examines a widespread but rarely documented habit in deep learning, where validation splits are routinely reused for checkpoint selection. Showing that small-budget independent searches frequently recover identical configurations across both validation and test sets provides valuable clarity on benchmark stability.
Critical weaknesses
  1. Mismatch with Adaptive Threat Models: the study challenges adaptive data analysis theory using purely non-adaptive random search over only thirty configurations. Under standard statistical learning bounds, observing negligible overfitting when choosing among thirty independent candidates on a large test set is mathematically expected rather than surprising. Consequently, this protocol fails to reflect realistic benchmark erosion, which is driven by sequential, adaptive tuning such as Bayesian optimization or iterative human trial-and-error.
  2. Unjustified Normative Stance and Fragile Ranking Invariance: advocating to normalize test-set tuning is an overextended conclusion. An isolated random search within a single study cannot justify abandoning test-set hygiene, which protects against cumulative community-level overfitting. Furthermore, rank preservation appears robust primarily because the evaluated models have massive inherent capability gaps. In competitive settings with closely matched architectures, ranking stability degrades, as seen in the rank swaps on MRPC and CoLA.
Questions for authors
  1. Adaptivity vs. Independent Sampling: since random search samples configurations independently of test feedback, how does this setup support claims about adaptive overfitting? How would the observed gap change under an intrinsically sequential strategy, such as Bayesian optimization?
  2. Ranking Invariance on Closely Competing Models: if the rank correlation analysis is restricted to subsets of architectures with closely matched baseline performances, does rank preservation still hold, or do inversions become common once performance margins approach the scale of the overfitting gap?
  3. Impact of Random Seed Tuning: what portion of the measured overfitting gap is driven by optimizing the random initialization seed compared to structural hyperparameters such as learning rate and weight decay?
  4. Scope of Policy Recommendations: would the authors consider reframing their takeaway from a normative recommendation to normalize test-set tuning to an empirical measurement of isolated, low-budget random search?
1 comment
AuthorsOpenReview ↗

On experimental soundness, and shouldn't "taboo" hypotheses also require empirical evidence?

Thank you for your time, effort, and engagement!

... observing negligible overfitting when choosing among thirty independent candidates on a large test set is mathematically expected rather than surprising.

Yes! This is exactly our point. If standard hyperparameter tuning is done on the test set, then overfitting is mathematically not expected. Our paper is the first to make this observation explicit and give empirical evidence beyond the mathematical assumption of independence. We added a theoretical motivation on page 3.

cannot justify abandoning test-set hygiene, which protects against cumulative community-level overfitting

But does it really? See (Recht et al., 2018, and 2019) that empirically investigates this hypothesis of cumulative community-level overfitting; finding evidence to the contrary. Our paper starts with a similar hypothesis that test-tuning leads to overfitting, and then we give empirical evidence to evaluate this hypothesis. Would you not agree that any claim needs to be supported by empirical evidence? Our paper investigates this evidence.

... evaluated models have massive inherent capability gaps ... rank swaps on MRPC and CoLA.

  1. Ranking Invariance on Closely Competing Models: if the rank correlation analysis is restricted to subsets of architectures with closely matched baseline performances, does rank preservation still hold, or do inversions become common once performance margins approach the scale of the overfitting gap?

To quantify the capability gap, we evaluate rank swaps when comparing validation-tuned models vs test-tuned models. So, for each model, we compare its test-tuned scores vs the validation-tuned score of the other models. The number of model rank swaps per dataset due to hyperparameter tuning is:

Datasetval-tuned vs test-tunedtest-tuned vs test-tuned
rank swapsrank swaps
MNIST-1D30
CIFAR1010
MRPC52
RTE50
CoLA40

Thus, except for CIFAR10, the capability gap between models is not large enough to prevent rank swaps due to hyperparameter tuning. This shows that hyperparameter tuning alone can swap model rankings for many models and rank is not preserved for Closely Competing Models when comparing val-tuned vs test-tuned models. This analysis show that for our independent sampling setting, overfitting by test-tuning occurs for 18 models, demonstrating that our setting is suitable. In contrast, rank is generally preserved between test-tuning models. We added table 1 and updated table 2 with this analysis.

...fails to reflect realistic benchmark erosion, .., Bayesian optimization or iterative human trial-and-error.

  1. Adaptivity vs. Independent Sampling: since random search samples configurations independently of test feedback, how does this setup support claims about adaptive overfitting? How would the observed gap change under an intrinsically sequential strategy, such as Bayesian optimization?

We use independent sampling because that is what existing work does when they tune on the test set (see "Test set tuning is common practice" on page 2) . We agree that more advanced settings are interesting, but we start with the setting for which we have evidence that it is already done in practice. See page 4 "problem setup". We agree that this is interesting to investigate as follow-up research, which we added to the paper, see limitations on page 10.

  1. Impact of Random Seed Tuning: what portion of the measured overfitting gap is driven by optimizing the random initialization seed compared to structural hyperparameters such as learning rate and weight decay?

We added tables 4, 5, 6.

Unjustified Normative Stance... a single study

  1. Scope of Policy Recommendations: would the authors consider reframing their takeaway from a normative recommendation to normalize test-set tuning to an empirical measurement of isolated, low-budget random search?

We apologize if this was not clear. We do not advocate to normalize test tuning. Yet, because test-tuning is already done, we advocate to then have authors also openly report it. A comparison between val-tuned and test-tuned models is flawed, as is well known, but we are the fist to show evidence that comparing test-tuned to test-tuned models, instead, seems reasonable, validating existing papers that do this. We clarified our recommendations at many places in the paper that we do not normalize test tuning, but evaluate existing work which does.

Rating: 2: Weak rejection

We believe we addressed all major weaknesses. The new analysis for val-tuning vs test-tuning show that for our current setting already 18 models change rank, where for test-tuning just a two models swap rank. This demonstrates that the problem exists in our current setting, and that we address it. Thus, adding more settings will not change our conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.