- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors investigate whether HPO conclusions remain stable across systematic changes in search spaces, fidelity reliability, costs, distribution shifts, protocols and backends (synthetic or learned surrogate functions). The paper contributes a configurable benchmark generator, a common execution pipeline and analyses of final quality, anytime behavior, cost trade-offs, rank changes and run variability of the HPO process. In experiments, 7 optimizers are evaluated (including BO, TPE, Hyperband, ASHA, and BOHB). As expected BO leads in aggregate final quality and BOHB performs best under tight budgets. Multi-fidelity methods are cheaper but degrade when low-fidelity evaluations are misleading. Validation of the benchmark is primarily synthetic. The small real-task studies demonstrate applicability but not broader real-world representativeness.
Critical strengths
The paper in general addresses an important evaluation problem by treating benchmark conditions as experimental variables. Its trace-based framework supports final-quality, anytime, fixed-cost, conditional and repeated-run analyses. This allows for revealing behavior that final regret alone as a metric can hide.
Critical weaknesses
First, the realism and external validity of the generated benchmarks are not convincingly established. The authors do not show that they reproduce important properties of real HPO, such as noise of the objective function, seed and resampling effects, multimodality, conditional structure or realistic fidelity behavior. Since most experiments are synthetic and the real-task studies are small, the observed ranking changes could be specific to the generator rather than transferable to real workloads.
Second, the paper also omits closely related work on meta-surrogate benchmark generation, controllable multi-fidelity benchmarks, unified HPO evaluation, and protocol-sensitive analysis. This makes the novelty beyond combining existing ideas difficult to assess. Below are some references that in my opinion should be discussed in more detail:
Klein et al. (2019): https://proceedings.neurips.cc/paper_files/paper/2019/file/0668e20b3c9e9185b04b3d2a9dc8fa2d-Paper.pdf
Mallik et al. (2023): https://proceedings.neurips.cc/paper_files/paper/2023/file/1704fe7aaff33a54802b83a016050ab8-Supplemental-Conference.pdf
Eggensperger et al. (2019): https://www.jair.org/index.php/jair/article/download/11420/26488/21322
Egele et al. (2023): https://www.esann.org/sites/default/files/proceedings/2023/ES2023-84.pdf
Geburek et al. (2024): https://arxiv.org/html/2408.02533v1
Nagler et al. (2024): https://proceedings.neurips.cc/paper_files/paper/2024/file/47811ee68103bfcde7ca2223fccefb3a-Paper-Conference.pdf
Benjamins et al. (2025): https://arxiv.org/abs/2506.06143
Questions for authors
How are the objective, cost, noise and fidelity functions constructed and what evidence shows that they resemble real HPO workloads?
Are repeated evaluations stochastic through training seeds or resampling or is each generated function deterministic once instantiated (apologies if I missed that)?
How does the framework differ technically from Meta-Surrogate Benchmarking, MF-Prior-Bench/PriorBand benchmark functions, CARPS, and prior work on protocol-dependent HPO conclusions?