acceptodds
Under review as a conference paper at ICLR 2027

Beyond Final Regret: Stress-Testing Hyperparameter Optimization Conclusions Across Evaluation Regimes

Abstract

Hyperparameter optimization (HPO) methods are commonly compared on fixed collections of tasks using aggregate final performance, implicitly treating the benchmark and evaluation protocol as fixed. We study a different question: how sensitive are empirical conclusions about HPO methods to controlled changes in the evaluation regime? We introduce Beyond Final Regret, a configurable benchmark generator that exposes search-space structure, fidelity reliability, cost regime, distribution shift, evaluation protocol, and backend realization as explicit experimental variables. The framework generates reproducible benchmark instances and retains objective, fidelity, incumbent, and cost trajectories, enabling evaluation through final quality, anytime behavior, quality–cost trade-offs, conditional comparisons, and repeated-run distributions. We evaluate seven representative HPO methods across in-distribution, interpolation, extrapolation, cross-family, and targeted stress-test settings. The experiments show that several conclusions obtained from aggregate final performance are conditional on the evaluation regime. Relative performance changes with search-space structure, fidelity reliability, available resources, distribution shift, protocol, and backend, while other relationships remain stable across perturbations. In particular, methods with strong final quality can incur substantially larger computational costs, making recommendations dependent on the available resource budget rather than on a single cost-normalized scalar. These results position benchmark design itself as an experimental variable and provide an executable framework for measuring the sensitivity of HPO conclusions to that variable.

Primary area:
datasets and benchmarks
Keywords:
Benchmark generator, hyperparameter optimization, fidelity mismatch, budget sensitivity, robust optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YVoROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate whether HPO conclusions remain stable across systematic changes in search spaces, fidelity reliability, costs, distribution shifts, protocols and backends (synthetic or learned surrogate functions). The paper contributes a configurable benchmark generator, a common execution pipeline and analyses of final quality, anytime behavior, cost trade-offs, rank changes and run variability of the HPO process. In experiments, 7 optimizers are evaluated (including BO, TPE, Hyperband, ASHA, and BOHB). As expected BO leads in aggregate final quality and BOHB performs best under tight budgets. Multi-fidelity methods are cheaper but degrade when low-fidelity evaluations are misleading. Validation of the benchmark is primarily synthetic. The small real-task studies demonstrate applicability but not broader real-world representativeness.

Critical strengths

The paper in general addresses an important evaluation problem by treating benchmark conditions as experimental variables. Its trace-based framework supports final-quality, anytime, fixed-cost, conditional and repeated-run analyses. This allows for revealing behavior that final regret alone as a metric can hide.

Critical weaknesses

First, the realism and external validity of the generated benchmarks are not convincingly established. The authors do not show that they reproduce important properties of real HPO, such as noise of the objective function, seed and resampling effects, multimodality, conditional structure or realistic fidelity behavior. Since most experiments are synthetic and the real-task studies are small, the observed ranking changes could be specific to the generator rather than transferable to real workloads.

Second, the paper also omits closely related work on meta-surrogate benchmark generation, controllable multi-fidelity benchmarks, unified HPO evaluation, and protocol-sensitive analysis. This makes the novelty beyond combining existing ideas difficult to assess. Below are some references that in my opinion should be discussed in more detail:

Klein et al. (2019): https://proceedings.neurips.cc/paper_files/paper/2019/file/0668e20b3c9e9185b04b3d2a9dc8fa2d-Paper.pdf

Mallik et al. (2023): https://proceedings.neurips.cc/paper_files/paper/2023/file/1704fe7aaff33a54802b83a016050ab8-Supplemental-Conference.pdf

Eggensperger et al. (2019): https://www.jair.org/index.php/jair/article/download/11420/26488/21322

Egele et al. (2023): https://www.esann.org/sites/default/files/proceedings/2023/ES2023-84.pdf

Geburek et al. (2024): https://arxiv.org/html/2408.02533v1

Nagler et al. (2024): https://proceedings.neurips.cc/paper_files/paper/2024/file/47811ee68103bfcde7ca2223fccefb3a-Paper-Conference.pdf

Benjamins et al. (2025): https://arxiv.org/abs/2506.06143

Questions for authors

How are the objective, cost, noise and fidelity functions constructed and what evidence shows that they resemble real HPO workloads?

Are repeated evaluations stochastic through training seeds or resampling or is each generated function deterministic once instantiated (apologies if I missed that)?

How does the framework differ technically from Meta-Surrogate Benchmarking, MF-Prior-Bench/PriorBand benchmark functions, CARPS, and prior work on protocol-dependent HPO conclusions?

Reviewer_aQdROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a configurable HPO benchmark framework that varies search-space structure, fidelity reliability, cost, distribution shift, evaluation protocol, and backend. It evaluates seven optimizers and retains their execution traces to compare final quality, anytime behavior, fixed-cost performance, conditional ranks, and repeated-run variation. The main finding is that some aggregate conclusions persist, while others depend on resource limits and benchmark conditions.

Critical strengths
  1. Treating benchmark conditions as variables is a useful practical direction. The framework brings several relevant checks into one workflow, and retaining complete traces makes it possible to revisit conclusions without rerunning every optimizer.

  2. The cost-aware analysis is more informative than a final-quality ranking alone. Figure 1 and Table 2 show why an optimizer recommendation depends on the available resources, rather than assuming that one method is best under every budget.

Critical weaknesses
  1. The manuscript has visible presentation problems. There is an empty bullet immediately after “Our contributions are” on page 2. The main text contains only two figures, despite making claims across many controlled axes, and several important comparisons are mainly described through lists of numbers. The figures need more uncertainty information and direct examples of the manipulated conditions. The empty contribution item should certainly have been removed before submission.

  2. The benchmark generator, which is the main contribution, is not specified concretely enough in the paper. Section 3 names the controlled axes and writes the generator abstractly, but does not adequately show how particular settings change the objective landscape, fidelity response, and cost function. The appendix gives much more detail about evaluation metrics than about these generating mechanisms. Without this information, it is hard to judge what is being controlled and whether changes can really be attributed to an individual axis.

  3. The real-workload evidence is too small for the intended practical message. Appendix A.1 reports only three evaluated instances for the real-task extension and two for the external connector; the backend-diversity study has four. The paper acknowledges that these are diagnostic probes, which is appropriate, but then most of the contribution rests on generated settings whose relationship to real HPO workloads is not well established. The evidence does not yet make the benchmark's broader value convincing enough.

  4. The analysis introduces another source of benchmark dependence without examining it sufficiently. Equation 9 normalizes outcomes using the best and worst means among the compared methods. Changing the method set therefore changes the per-instance scaling and can affect aggregate conclusions. Likewise, the rank-shift heatmap is descriptive and does not quantify how much of a shift exceeds run-to-run uncertainty. Since conclusion sensitivity is the paper's central topic, sensitivity to these analysis choices should receive more attention.

Questions for authors
  1. Can you give the exact generating rules for one representative objective, fidelity-reliability intervention, and cost intervention, and identify what stays fixed when each axis changes? If these are already documented, please point to the relevant definitions.

  2. What existing validation shows that the generated fidelity and cost regimes resemble the real backends used here, beyond sharing an interface?

  3. Using the retained traces, how sensitive are the main aggregate comparisons to the normalization anchors and their method set? Which reported rank changes are clearly larger than seed variation?

Reviewer_NjxAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper takes a more holistic view on benchmarking HPO methods and studies the robustness of aggregated metrics across benchmarks wrt to design choices and task characteristics. The paper defines several scenarios and settings (misleading fidelity; transfer across families; evaluation protocol) and studies several baselines in this settings.

Critical strengths

The paper's idea and motivation are really interesting and provide a counterbalance to existing benchmarking efforts. After reading the Abstract and Intro, I was really looking forward to reading the rest of the paper.

Critical weaknesses

W1: Lack of Details. The benchmark generator is central to this approach. However, the main paper does not provide any insights into how benchmarks are generated (and neither does the appendix). Are they derived from real benchmarks? Or are they generated from a latent representation? How are they constructed and evaluated?

W2: Novelty. The most recent reference is from 2023, and the evaluated methods are also outdated (additionally, it is unclear what "surrogate Bayesian optimization" is); thus, the paper ignores recent progress in the field of HPO.

W3: Writing. The paper is written largely repetitively, and several phrases appear quite often in different variations (e.g., "avoid explicitly claiming novelty [..]"). Furthermore, several paragraphs are unnecessarily complex (without adding a lot of precision) and make the paper harder to understand than needed (e.g., Section 4.4: "Rank displacement is used descriptively to locate where an aggregate conclusion persists or changes")

Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.