acceptodds
Under review as a conference paper at ICLR 2027

BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks

Abstract

Optimization of LLM training and inference configurations, such as hyperparameters, data mixtures, prompts, and parallelism strategies, is critical to performance. However, it is often done heuristically, which can lead to suboptimal outcomes. These tasks are noisy, expensive, and derivative-free, making them a natural fit for sample-efficient Bayesian optimization (BO) and other black-box optimization (BBO) methods. This direction remains underexplored, largely because LLM training and inference are prohibitively expensive for most BBO researchers, and new methods are often evaluated only on synthetic test functions and small-scale datasets that fail to capture the challenges of LLM optimization. This impedes the development of BBO methods and makes their effectiveness on modern LLM tasks difficult to assess. We introduce BoLT, the first LLM-centric benchmark that democratizes LLM research for the BBO community. BoLT covers a broad set of well-motivated LLM optimization problems, involving multi-fidelity, multi-objective, heteroscedastic noise, high-dimensional search spaces, and black-box constraints. Each problem is grounded in thousands of real LLM experiments, and is fully reproducible and accessible through lightweight emulators or tabular datasets. We benchmark a wide range of BO and BBO methods on BoLT. BO methods generally outperform other baselines, but no single method wins across tasks, and methods remain far from optimal on some problems. These gaps underscore the need for benchmarks grounded in modern LLM tasks. An anonymized version of BoLT is available at https://github.com/anonom799/bolt.

Primary area:
datasets and benchmarks
Keywords:
Bayesian optimization, blackbox optimization, LLM benchmark, AutoML

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_XEcUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Optimizing LLM training and inference requires an exhaustive search of many different design choices, such as what hyperparameters to tune, what data to train on, how to optimize prompts, etc. Each design choice is expensive to run, which highlights the need to predict results quickly to find the best configuration. This paper proposes that Bayesian-optimization (BO) approaches are well-suited for these tasks, and estabalishes a benchmark to test different BO strategies to predict outcomes of LLM design. They find that BO methods are generally useful, but no one method is clearly the best. The authors hope that this benchmark can help BO researchers to apply new methods to improve and facilitate the iterative process of LLM design.

Critical strengths

This paper establishes four kinds of tasks that are relevant in LLM training and development:

  1. LoRA hyperparameter finetuning;
  2. Data mixing ratios;
  3. Prompt engineering;
  4. Training parallelism configurations.

All four tasks are relevant to optimizing LLM training and/or inference. The paper presents a unique view that bayesian-optimization approaches could be used to help guide search over the different choices of designs to optimize LLM development.

This paper also purports to make LLM optimization more accessible without the access of GPUs. In order to accomplish this feature, they design emulators which can simulate experimental runs. The emulators are simple MLPs which can directly estimate a model's performance given its hyperparameter configuration, which is an interesting approach.

Critical weaknesses

The main emulator’s model choice feels insufficiently validated. I appreciate the authors' ablation on varying MLP width and depth to predict hyperparam configs → performance, but I am missing intuition on why this particular choice of architecture is suited for this task. Could we design simpler emulators, e.g. a linear regression fit, or even fit a scaling law, that takes as input the hyperparameter configs and predicts model accuracy? Given the vast literature on scaling laws, I would like to see more discussion on the relationship between emulation of LLM training and extrapolation of LLM training trends.

There is also the question of how emulators can affect the optimization landscape. For example, different emulator models could have the same prediction accuracy, but be much different in terms of the local smoothness, or interactions - and this could greatly affect the performance of bayesian-optimization methods. I think a worthy additional experiment would be to compare alternative emulator architectures and observe how the optimizer results could change.

While the LLM task coverage is broad, I would be more interested in learning what these tasks collectively teach us about choosing or designing optimizers for LLM workflows. The paper offers individual insights per task, e.g. local search helps prompt selection, domain priors help parallelism optimization, etc, but lacks a cohesive story about how to build a broad decision framework that could equip practitioners about how to improve LLM training.

Questions for authors

Can you discuss the relationship between your emulators and scaling laws? For example, could scaling law models, or linear regression fits, also achieve similar predictive performance as your MLP-architecture emulators? Would the models also be useful for BO?

Tables 4–5 compare MLP architectures, but do the relative rankings of optimizers remain consistent across different emulator model families? I believe this could help clarify whether the benchmark reflects the underlying LLM tasks, or if they reflect the properties of the emulator itself.

Although this benchmark is built primarily for optimization researchers, are there recommendations or guidance that one can make to practitioners when choosing to optimize their LLM development?

6 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

We thank the reviewer for their valuable review and for appreciating the relevance of our LLM optimization tasks, our perspective on BO for LLM development, and our MLP emulator approach. We address their concerns and questions below.

(W1) Could we design simpler emulators, e.g. a linear regression fit, or even fit a scaling law, that takes as input the hyperparameter configs and predicts model accuracy?

(Q1) could scaling law models, or linear regression fits, also achieve similar predictive performance as your MLP-architecture emulators?

Thank you for this suggestion. We have added a comparison of emulator models, including linear ridge regression, scaling-law/mixing-law models, kernel ridge regression, and Gaussian processes, using the same training data and held-out validation sets.

For HPO's scaling law, we evaluate a Chinchilla-inspired power-law baseline [1], adapted to predict downstream accuracy at fixed model size, with coefficients conditioned on the hyperparameters, acc(x,D)=a(x)−A(x)⋅D−α(x)\text{acc}(x, D) = a(x) - A(x) \cdot D^{-\alpha(x)}, where DD is in millions of training tokens and xx is the hyperparameter configuration. a(x)a(x), log⁡A(x)\log A(x), and log⁡α(x)\log \alpha(x) are linear in xx.

For DMO's mixing law, we adapt the form from [2], acc(x)=c+k⋅exp⁡(t⋅x)\text{acc}(x) = c + k \cdot \exp(t \cdot x), where xx here denotes the two-stage data mixture proportions, tt is a coefficient vector, and cc and kk are scalar coefficients.

We show results for a subset of emulator models, with the full ablation in the revision (Sec. B.6) including additional regression, kernel, GP, and tree-based models.

EmulatorHPO-4B R2R^2HPO-4B ρ\rhoHPO-8B R2R^2HPO-8B ρ\rhoDMO-4B R2R^2DMO-4B ρ\rho
MLP0.8350.9080.9270.9460.6470.786
Linear ridge regression0.3190.7660.5180.7970.4170.671
Scaling / mixing law0.8010.9090.8210.9120.4610.667
Matérn-5/2 GP0.8070.8950.9400.9580.6390.782

Linear regression falls substantially short of the MLPs, but the scaling-law baseline is competitive on HPO-4B, in terms of Spearman's ρ\rho. The Matérn-5/2 GP achieves the highest predictive metrics on HPO-8B, but our transferability comparison below (in our response to Q2 and W2) shows that this does not translate into better agreement with real optimizer rankings.

[1] Hoffmann et al. Training compute-optimal large language models. arXiv, 2022.

[2] Ye et al. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. ICLR, 2025.

(W1) Given the vast literature on scaling laws, I would like to see more discussion on the relationship between emulation of LLM training and extrapolation of LLM training trends. (Q1) Can you discuss the relationship between your emulators and scaling laws? ... Would the (scaling law) models also be useful for BO?

Scaling laws and our emulators both predict training outcomes without having to conduct every training run. The main difference is that scaling laws such as those used in Chinchilla [1] encode assumptions about how performance changes with training scale, which guide extrapolation beyond observed scales, while our emulators aim to reproduce the optimization landscape within the benchmark domain.

In our emulator comparisons, MLPs generally perform better than the scaling-law models. MLPs provide greater flexibility in modeling interactions and objective landscapes, which is an advantage in our setting, where we evaluate held-out configurations within the benchmark domain rather than extrapolation to larger models or longer training budgets.

Scaling-law models can also be useful within BO. For example, Kadra et al. [3] incorporate power-law learning-curve models into BO to inform configuration selection.

We have added this discussion of scaling laws and emulators in Sec. B.6.2 of our revision.

[3] Kadra et al. Scaling laws for hyperparameter optimization. NeurIPS, 2023.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

(W2) There is also the question of how emulators can affect the optimization landscape. For example, different emulator models could have the same prediction accuracy, but be much different in terms of the local smoothness, or interactions - and this could greatly affect the performance of bayesian-optimization methods. I think a worthy additional experiment would be to compare alternative emulator architectures and observe how the optimizer results could change.

(Q2) … do the relative rankings of optimizers remain consistent across different emulator model families? I believe this could help clarify whether the benchmark reflects the underlying LLM tasks, or if they reflect the properties of the emulator itself.

To assess how emulator choice affects optimizer rankings, we extend the real-versus-emulated comparison in Fig. 9 to additional emulator families. Specifically, we compare MLP, scaling/mixing-law, and Matérn GP models here, to illustrate different model families. The full experiment additionally includes quadratic ridge, kernel ridge, and gradient boosting (Sec. B.6.1).

On each emulator, we compare four BBO methods, with the same 10 initial evaluations, over 50 optimization iterations and five seeds. We compare these rankings against those obtained by running the same methods with real LLM evaluations, using matched seeds. We compute Spearman ρ\rho between seed-averaged method ranks at each iteration, then average over the indicated window:

EmulatorHPO–8B: iter. 1–50HPO–8B: iter. 31–50DMO–4B: iter. 1–50DMO–4B: iter. 31–50
MLP0.5810.9100.7430.946
Scaling / mixing law0.5120.9100.6020.553
Matérn-5/2 GP0.0410.6100.3350.454

Optimizer rankings are sensitive to the emulator family, even when predictive accuracy is comparable. Although the Matérn-5/2 GP has the highest predictive accuracy on HPO–8B, it has substantially lower ranking agreement with real runs. One possible explanation is that GP-based BO methods can model a GP emulator’s objective particularly well because of their shared modeling assumptions. This could disproportionately favor GP-based optimizers relative to non-GP baselines.

These results support the reviewer’s concern that predictive accuracy alone is insufficient for assessing whether an emulator preserves optimizer rankings. Comparing rankings against real runs is thus an important complementary check for potential emulator bias. Among the emulator families evaluated, our MLP emulators achieve the highest average ranking agreement over iterations 1–50 on both tasks, while the scaling law matches their late-stage agreement on HPO.

We have added this ablation study in Sec. B.6.1 of our revision, including full regret plots for each emulator.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

(W3) While the LLM task coverage is broad, I would be more interested in learning what these tasks collectively teach us about choosing or designing optimizers for LLM workflows. The paper offers individual insights per task, e.g. local search helps prompt selection, domain priors help parallelism optimization, etc, but lacks a cohesive story about how to build a broad decision framework that could equip practitioners about how to improve LLM training.

(Q3) Although this benchmark is built primarily for optimization researchers, are there recommendations or guidance that one can make to practitioners when choosing to optimize their LLM development?

While our experiments do not identify one optimizer that is best for every LLM workflow, they suggest that formulating the BBO problem appropriately is critical. Practitioners can use three broad questions to guide this formulation and narrow their optimizer choices.

(1) What is the search space, and what domain knowledge is available? For high-dimensional spaces, consider local search methods such as TuRBO, which performed well on our prompt-selection tasks. Where domain knowledge is available, use it to constrain the search space or inform the optimizer’s prior, as illustrated by our parallelism-optimization experiments. These are different ways to make optimization more efficient.

(2) What information do evaluations provide, and what objectives matter? If cheaper evaluations are informative about the target setting, consider multi-fidelity methods. If observations are very noisy, use methods that account for this, like LogNEI. When several objectives must be balanced, multi-objective methods can identify a Pareto frontier of trade-offs rather than requiring a single objective weighting in advance.

(3) What compute budget and resources are available? With a small evaluation budget, more exploitative settings may reach good configurations sooner, as illustrated by UCB with a smaller exploration parameter in our HPO experiments. With a larger budget, additional exploration may become worthwhile. If compute resources permit, consider batched methods to run multiple evaluations in parallel and reduce wall-clock time.

Within the appropriate formulation, LogNEI is a reasonable starting point for noisy single-objective optimization, while MES/GIBBON and their multi-fidelity variants are alternatives worth comparing. Their relative performance depends on the task and evaluation budget, so we recommend comparing a small set of suitable methods rather than adopting one acquisition function across all workflows.

This framework connects optimizer choices to the structure and resource constraints of an LLM workflow, and our benchmark provides evidence to help narrow these choices.

We have added this discussion to Sec. E of our revision.

We have compared alternative emulator models and their agreement with real optimizer rankings, clarified the relationship to scaling laws, and added guidance for practitioners. We hope these analyses and revisions address your concerns.

Reviewer_gx6eOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper provides a scalable benchmark for hyperparameter optimization for LLM by training a surrogate emulator that maps the hyperparameter choice to the outcomes. This emulator allows research to test BO or other black box optimization methods without the need to retrain LLM from scratch, avoiding a costly and time consuming bottleneck for research.

The paper then analyzes off-the-shelf BO and other black box optimization methods, and found that GP-based methods generally outperform non-GP-based methods.

Critical strengths
  • The main problem is well motivated and timely.
  • The paper provides a few important BBO settings in LLM, and a comprehensive analysis of BO and BBO methods on these settings.
  • The transferability to real LLM training experiment is a nice verification that the benchmark provides value for methodological development
Critical weaknesses
  • Qwen3-8B should typically have 80% accuracy on Math-500. It seems weird (on Figure 2) that the emulator only supports up to 51%. Could the authors explain why?
  • Minor:
    • Figures (axis labels, legends, etc.), are often illegible, which may be fixable with extra space, but as written do not fit the standard for publication
    • Positioned as a benchmark for BBO, the paper would benefit from expanding the testbed to experiments validation outside MATH-500 or Qwen models. Though I do understand the immense resource required for collecting these data
    • The use of emulator to becnhmark LLM BBO that the paper centers around seems first introduced by [1], which should be properly credited.

[1] Yen et al., Data mixture optimization: A multi-fidelity multi-scale Bayesian framework, NeurIPS 2025

Questions for authors
  • Are there experiments for different emulator model choices? The R^2 performance of the DMO-4B emulator seems low in particular. In training the emulator for HPO and DMO, there are options to include the the fidelity parameters in the emulator input space if the authors kept checkpoint data and trained on multiple model sizes. How was the emulator trained, and has the authors established benefits in including or excluding the fidelity parameter in the emulators
  • In providing multi-fidelity version of HPO, similar to [1], the authors have an option to provide another variant where model size and training token size are jointly optimized. Is there a reason this variant is not provided? This seems closest to the problem practitioners have to deal with.
  • Similar to the question above, why doesn't DMO have a MF-Cont setting?

[1] Yen et al., Data mixture optimization: A multi-fidelity multi-scale Bayesian framework, NeurIPS 2025

6 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

Thank you for the detailed feedback and for appreciating the motivation and timeliness of our work, the important BBO settings in LLM, our comprehensive analysis, and the verification of transferability to real LLM training experiments. We address your concerns and questions below.

(W1) Qwen3-8B should typically have 80% accuracy on Math-500. It seems weird (on Figure 2) that the emulator only supports up to 51%. Could the authors explain why?

The 51% number is from the DMO emulator, which is based on pretrained Qwen3-4B-Base fine-tuned using LoRA on 10M training tokens, rather than the larger, post-trained, Qwen3-8B. We used pretrained base models to allow for greater variations in scores, as post-trained models already perform well on MATH-500 and improvements from SFT can be marginal. We also used a small generation budget of 256 new tokens with no thinking. These model and evaluation differences mean our scores are not directly comparable to the cited 80% accuracy. HPO emulators also have lower MATH-500 scores, as the evaluation is zero-shot with an explicit instruction to output only the final answer, without reasoning or thinking, to facilitate faster evaluations.

As we noted in our limitations, we selected evaluation settings that provided faster inference to collect sufficient data to fit reliable emulators. Thus, the scores may not represent the best achievable performance of the evaluated models under unconstrained inference.

We have added these clarifications in Secs. C.1 and C.2 of our revision.

(Q1) Are there experiments for different emulator model choices?

We have added a comparison of various emulator models, and evaluated selected emulator models against real BBO method rankings in Sec. B.6.

For the comparison, we included MLPs (ours), linear and quadratic ridge regression, kernel ridge regression with RBF and Laplacian kernels, GPs with RBF and Matérn-5/2 kernels, random forests, gradient boosting trees, and scaling laws for HPO or mixing laws for DMO. For each task, all models were fitted on the same training data and evaluated on the same held-out set. A subset of the results in is the table below.

EmulatorHPO-4B R2R^2HPO-4B ρ\rhoHPO-8B R2R^2HPO-8B ρ\rhoDMO-4B R2R^2DMO-4B ρ\rho
MLP (ours)0.8350.9080.9270.9460.6470.786
Ridge (quadratic)0.7140.8470.7690.9000.5380.728
Kernel ridge (Laplacian)0.6840.9090.9040.9360.6350.779
GP (Matérn-5/2)0.8070.8950.9400.9580.6390.782
Scaling / mixing law0.8010.9090.8210.9120.4610.667

Our MLP emulators achieve the highest R2R^2 on HPO-4B and DMO-4B, while remaining competitive on HPO-8B.

For a subset of emulator models, we also compared BBO method rankings on the emulators against those from the real evaluations in Figs. 9–11. The table below reports Spearman’s ρ\rho, computed between method rankings averaged over five matched seeds and then averaged over the indicated iterations.

EmulatorHPO–8B: ρ\rho, iter. 1–50HPO–8B: ρ\rho, iter. 31–50DMO–4B: ρ\rho, iter. 1–50DMO–4B: ρ\rho, iter. 31–50
MLP0.5810.9100.7430.946
Scaling / mixing law0.5120.9100.6020.553
Matérn-5/2 GP0.0410.6100.3350.454

Optimizer rankings are sensitive to the emulator family, even when predictive accuracy is comparable. Although the Matérn-5/2 GP has the highest predictive accuracy on HPO–8B, it has substantially lower ranking agreement with real runs. One possible explanation is that GP-based BO methods can model a GP emulator’s objective particularly well because of their shared modeling assumptions. This could disproportionately favor GP-based optimizers relative to non-GP baselines.

These results show that predictive accuracy alone is insufficient for assessing whether an emulator preserves optimizer rankings. Comparing rankings against real runs is thus an important complementary check for potential emulator bias. Among the emulator families evaluated, our MLP emulators achieve the highest average ranking agreements on both tasks, while the scaling law matches their late-stage agreement on HPO.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

(Q1) The R^2 performance of the DMO-4B emulator seems low in particular.

DMO-4B's lower R2R^2 (0.63−0.710.63-0.71) may partly reflect noise across seeds, including randomness from training data sampling. Recall R2=1−RMSE2/σy2R^2=1-\text{RMSE}^2/\sigma_y^2, where σy\sigma_y measures variation in objective values across configurations in the evaluated dataset. In contrast, σpooled=1N∑iNσi2\sigma_\text{pooled}=\sqrt{\frac{1}{N}\sum^N_i \sigma^2_i} measures seed-to-seed variation at fixed configurations, where σi\sigma_i is estimated from 55 seeds at configuration ii, across N=100N=100 configurations. The table below shows that RMSE is comparable to σpooled\sigma_\text{pooled}, suggesting that seed noise contributes substantially to the observed prediction error.

For DMO-4B:

ObjectiveR2R^2 trainR2R^2 testRMSE trainRMSE testseed-to-seed σpooled\sigma_\text{pooled}
IFEval0.7130.6430.01290.01390.0146
MATH-5000.6420.6640.03170.03190.0331
MBPP+0.6700.6380.00910.00960.0111

Other emulator models we tested also do not improve R2R^2 for DMO-4B, as we have shown above. Nevertheless, in our transferability experiments (Fig. 9), our emulated DMO problem largely reproduces the method rankings from real evaluations, with a mean Spearman ρ\rho of 0.9460.946 between emulated and real method rankings over iterations 3131 to 5050, supporting the usefulness of our DMO emulator despite the lower R2R^2 values.

In our revision, we have added the analysis in Sec. B.4, and also corrected a minor error in Table 3 for the HPO-4B emulator. The updated test metrics show better predictive performance.

(Q1) In training the emulator for HPO and DMO, there are options to include the the fidelity parameters in the emulator input space if the authors kept checkpoint data and trained on multiple model sizes. How was the emulator trained, and has the authors established benefits in including or excluding the fidelity parameter in the emulators

For HPO, we do keep multiple checkpoints for each fine-tuning run and evaluate these checkpoints, so every run contributes several observations mapping (hyperparameters, training tokens) to a score. We collect these checkpoints for both 4B and 8B models. These data support both the token-fidelity HPO-MF-Cont setting and the model-size-fidelity HPO-MF-Disc setting.

Following the suggestion, we have added an ablation comparing HPO emulators trained (1) without fidelity inputs, separately for each model size, (2) with token fidelity per model size (ours), and (3) with both token and model-size fidelity in one shared emulator. For (1), models are trained only on full-budget data. For (3), we use the same training recipe as the separate 8B emulator, since 8B is the target fidelity. The table below shows the evaluation metrics, averaged over five seeds. The "(final)" metrics are based on the final checkpoints of 15 held-out runs per model size, while “all” means all held-out checkpoints, including at intermediate token budgets.

TaskTraining setupR2 ↑R^2\,\uparrow (all)ρ ↑\rho\,\uparrow (all)ρ25 ↑\rho_{25}\,\uparrow (all)R2 ↑R^2\,\uparrow (final)ρ ↑\rho\,\uparrow (final)RMSE ↓\downarrow (final)
HPO-4B(1) No fidelity input———0.5890.8760.0435
HPO-4B(2) Token input: per size (ours)0.8350.9080.6340.6870.9470.0379
HPO-4B(3) Token + model-size: shared0.8120.9140.6940.5740.8960.0443
HPO-8B(1) No fidelity input———0.7800.8220.0340
HPO-8B(2) Token input: per size (ours)0.9270.9460.8040.8910.9100.0239
HPO-8B(3) Token + model-size: shared0.9330.9470.8010.9250.9190.0198

Training on all checkpoints with token fidelity as an input (2) improves all final-checkpoint metrics over full-budget-only training (1) for both model sizes. Comparing (2) and (3), the shared emulator improves most HPO-8B metrics, while separate emulators achieve better HPO-4B accuracy at the final checkpoint. Thus, neither design uniformly dominates, and both are reasonable choices. We have added this study to Sec. B.7 of our revision.

We do not include fidelity parameters for DMO emulators due to data collection limitations, which we discuss in our response to Q3 next.

(Q3) why doesn't DMO have a MF-Cont setting?

We did not include a MF-Cont setting for DMO as DMO optimizes a two-stage data curriculum (one mixture for the first 5M tokens, another for 5M-10M), which complicates the use of training tokens as a fidelity axis. A checkpoint in the first stage does not depend on the second-stage mixture, so half of the input dimensions are inactive at low fidelity.

Also, DMO's cost is dominated by evaluation rather than training. Each checkpoint is scored on three benchmarks (IFEval, MATH-500, MBPP+), which takes about one hour on an L40. This makes collecting sufficient checkpoint evaluations for a continuous-fidelity DMO emulator too expensive for our compute budget.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

(Q2) In providing multi-fidelity version of HPO, similar to [1], the authors have an option to provide another variant where model size and training token size are jointly optimized. Is there a reason this variant is not provided? This seems closest to the problem practitioners have to deal with.

Thank you for this suggestion. We agree that jointly varying model size and training-token budget is practically relevant. The one-dimensional variants allow each fidelity axis to be studied separately, but both can easily be combined into a joint variant. We have implemented and evaluated a new joint-fidelity HPO variant, with discrete model size (4B or 8B) and continuous training-token budget (0.1M–9.1M tokens). Both fidelity parameters are selected alongside the hyperparameters, with the optimization target defined at the 8B model and full token budget.

We evaluated eight methods over ten seeds at a cumulative cost budget of 200, where evaluating the 8B model at the full token budget costs 1. The table below reports the final log simple regret as mean ± standard deviation across seeds.

MethodLog simple regret
LogEI−6.51 ± 1.45
UCB−6.25 ± 1.58
LogNEI−6.08 ± 0.76
MF-MES−6.03 ± 1.16
MF-GIBBON−5.79 ± 1.35
PES−5.75 ± 0.68
BOHB−4.65 ± 0.92
Random−4.04 ± 0.29

We have added this variant as HPO-MF-2D in the revision, introducing it in Sec. 3.1, with regret plot in Fig. 3d, and fidelity distribution plots in Fig. 21c. This addition also demonstrates that our emulators can readily serve as building blocks for new problem variants.

(W2) Minor:

  • Figures (axis labels, legends, etc.), are often illegible, which may be fixable with extra space, but as written do not fit the standard for publication
  • Positioned as a benchmark for BBO, the paper would benefit from expanding the testbed to experiments validation outside MATH-500 or Qwen models. Though I do understand the immense resource required for collecting these data
  • The use of emulator to becnhmark LLM BBO that the paper centers around seems first introduced by [1], which should be properly credited.

We have enlarged the figures, especially axis labels and legends, to improve readability in our revision.

Our benchmark does include tasks beyond MATH-500, including instruction following (IFEval), coding (MBPP+), and parallelism configuration optimization. Our real-versus-emulated DMO validation also includes IFEval and MBPP+, as reported in Sec. B.5. Additionally, PCO-16 uses a Llama model. We agree, nevertheless, that broader validation across models and tasks would strengthen the benchmark. Collecting additional real training evaluations remains constrained by compute costs, and we have released our data-generation code to support future expansion.

Thank you for pointing out the connection to [1]. We have credited its prior use of emulators to benchmark LLM BBO in Sec. 2 of our revision.

In summary, we have clarified the evaluation settings, compared emulator families, added fidelity-input ablations, and introduced a two-fidelity variant (HPO-MF-2D). We hope these address your concerns.

Reviewer_aV6vOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents TESTBED, a PyTorch toolkit that decomposes Bayesian experimental design (BED) into tasks, designers, objectives, and algorithms, with hooks for modifying the rollout. The authors implement several existing policy-based methods (DAD, RL-BOED, ALINE, ActionBED, JointVarPost, static and random baselines) within this framework and benchmark them on eight tasks under a shared Attention-based policy architecture and a 12-hour wall-clock budget. They report EIG bounds (sPCE and sNMC with L = 10⁹, N = 2,048) and downstream MSE/CMSE from a refitted action policy. Three controlled studies follow: (i) an architecture ablation on location finding (Attention vs. DeepSets, with and without time embeddings; Table 1), (ii) the effect of the number of contrastive samples L on DAD (Table 2), and (iii) a variational-family ablation for JointVarPost (Table 11). The paper also contributes REINFORCE-style estimators for JointVarPost and ActionBED on non-reparameterisable likelihoods (Appendix B).

Critical strengths
  • The benchmark has a strong motivation: configuration fine-tuning in modern LLMs. As the author claimed: this is the first LLM-centric benchmark that democratizes LLM research for the BBO community, which is beneficial for future research.

  • The paper provides a complete code repository that is ready for deployment. This provides strong reproductivity value.

Critical weaknesses
  1. The metric definition is inconsistent with the ranking criterion. Section 4 defines simple regret as log(f* − max_t′ f(x_t′)) using the observed point, while the appendix (Section D.11, Table 16) states that methods are ranked by “max_t y_t, where y_t = f(x_t) + ε_t,” meaning the maximum over noisy observations.

  2. Noise levels are inconsistent across task families. For HPO and DMO, σ_N is measured from five repeated evaluations (Table 14). For PO and PCO, σ_N is set to 0.001 without justification, and PCO throughput measurements are described as exhaustive single runs on real hardware. Since noise is one of the main challenge dimensions the benchmark claims to cover, the noise setting for the PO and PCO problems should be measured or justified, and a sensitivity analysis should show how the rankings change with σ_N.

  3. Baselines are compared with and without prior knowledge in the same tables. Several methods appear in both plain and “+prior” variants (LogNEIC+prior, UCB-C+prior, cKG+prior, ADMMBO+prior), where the prior encodes analytic throughput models from Hemachandra et al. (2026). Similarly, MLHGP is compared against a GP “with known ground-truth noise,” which is an oracle. Mixing these variants in the same ranking conflates method quality with access to domain knowledge. The benchmark should separate knowledge-free and knowledge-informed tracks.

Questions for authors

see weakness

2 comments

We thank the reviewer for their valuable feedback, and for recognizing the strong motivation, completeness of the code, and reproducibility value of our work.

We would also like to gently clarify that the review’s summary appears to refer to a different paper. Nevertheless, the strengths and weaknesses discussed correctly refers to our work, and we address the questions below.

  1. The metric definition is inconsistent with the ranking criterion. Section 4 defines simple regret as log(f* − max_t′ f(x_t′)) using the observed point, while the appendix (Section D.11, Table 16) states that methods are ranked by “max_t y_t, where y_t = f(x_t) + ε_t,” meaning the maximum over noisy observations.

Sorry for the confusion, there is no inconsistency in the metric definitions. The metric definition that appears to be inconsistent in Sec. D.11 (D.13 in the revision) actually refers to normalized regret, which uses the noisy observations yty_t. In comparison, simple regret defined in Sec. 4 uses the noiseless objective values f(xt′)f(x_{t'}). We discussed normalized regret in Sec. D.11 (now D.13) to compare the ranking stability of simple regret and normalized regret. We have made this clearer in our revision.

  1. Noise levels are inconsistent across task families. For HPO and DMO, σ_N is measured from five repeated evaluations (Table 14). For PO and PCO, σ_N is set to 0.001 without justification, and PCO throughput measurements are described as exhaustive single runs on real hardware. Since noise is one of the main challenge dimensions the benchmark claims to cover, the noise setting for the PO and PCO problems should be measured or justified, and a sensitivity analysis should show how the rankings change with σ_N.

For PO, evaluations used greedy decoding so re-evaluating a prompt is essentially deterministic, and σN\sigma_N was set to 0.0010.001 to reflect this. For PCO, estimating σN\sigma_N would require repeated runs over many configurations, each run using 16/32/64 GPUs in parallel, which was beyond our compute budget. We use σN=0.001\sigma_N=0.001 as a low-noise benchmark setting for PCO rather than an empirically calibrated estimate, and have evaluated PCO on a range of σN\sigma_N values below.

As suggested, we conducted a noise sensitivity analysis, by evaluating PCO and selected PO problems at higher noise levels (0.005,0.01,0.020.005, 0.01, 0.02). Methods are ranked by final log simple regret averaged over seeds, and we compare the method rankings with those at σN=0.001\sigma_N = 0.001. The table below reports Spearman's ρ\rho between the two rankings, and "–" indicates settings not evaluated for cost reasons.

ProblemMethodsσN=0.005\sigma_N=0.005σN=0.01\sigma_N=0.01σN=0.02\sigma_N=0.02
PCO-16150.770.560.45
PCO-32150.700.690.64
PCO-64150.830.650.69
PO-12811–0.810.77
PO-25611–0.96–

Across the tested noise levels, the rankings show good agreement with those at σN=0.001\sigma_N=0.001, with ρ≥0.64\rho \geq 0.64 on all problems except PCO-16, and particularly strong agreement on the selected PO problems (ρ≥0.77\rho \geq 0.77). We observe the largest sensitivity to noise in PCO-16, possibly due to its flatter near-optimal region (Fig. 24).

The noise sensitivity analysis and more details, including regret plots at each noise level, are in the revised paper (Sec. D.12). We have also added the justification for different noise levels across task families in Sec. D.1.

  1. Baselines are compared with and without prior knowledge in the same tables. Several methods appear in both plain and “+prior” variants (LogNEIC+prior, UCB-C+prior, cKG+prior, ADMMBO+prior), where the prior encodes analytic throughput models from Hemachandra et al. (2026). Similarly, MLHGP is compared against a GP “with known ground-truth noise,” which is an oracle. Mixing these variants in the same ranking conflates method quality with access to domain knowledge. The benchmark should separate knowledge-free and knowledge-informed tracks.

Following the suggestion, we have removed LogNEI+KN from the standard DMO-Het comparison (Fig. 4c) and presented it separately alongside LogNEI and LogNEI+MLHGP in Fig. 5c, emphasizing that +KN uses known ground-truth noise.

For PCO results, we also added separate knowledge-free and knowledge-informed tracks in Sec. D.10, for convenient comparison of methods within each track. The figure in the main paper (Fig. 8) retains both variants, distinguished by solid and dashed lines for each plain and +prior pair, as we want to compare across these knowledge settings to draw conclusions on the usefulness of prior knowledge. The plain and +prior variants use matched experimental settings and evaluation budgets, differing only in their GP prior-mean specification.

In summary, we have clarified the metric definitions, justified the noise settings and added sensitivity analyses, and provided separate oracle-noise comparison and separate PCO knowledge tracks. We hope these clarifications and revisions address your concerns.

Dear Reviewer aV6v,

As we're halfway through the discussion phase, please acknowledge the author's response and consider leaving them a comment on whether it has resolved some or all of your concerns.

Thanks!

Comments on the paper

1 comment

We thank all reviewers for their time and effort in providing valuable, constructive, and positive feedback. We are encouraged that the reviewers found:

  • BoLT to be the first LLM-centric benchmark for the BBO community (aV6v), well motivated and timely (gx6e, aV6v), with relevant tasks (XEcU) and important BBO settings (gx6e) for LLM development.
  • a comprehensive analysis of BO and BBO methods (gx6e), and the benchmark’s value supported by emulator-to-real transferability experiments (gx6e).
  • the code ready for deployment, with strong reproducibility value (aV6v).

Following the reviewers’ suggestions, we have included additional ablations, analyses, and clarifications in the revised paper. A summary of the additions:

  • emulator model family ablations (gx6e, XEcU; Sec. B.6). We compare several emulator models (MLPs, GPs, scaling laws, etc.) on both predictive metrics and emulator-to-real transferability, and find that MLPs (our current emulators) have the best overall transferability.
  • scaling law discussion (XEcU; Sec. B.6). In addition to the comparison above, we discuss how scaling laws and emulators serve different goals: extrapolation beyond observed scales versus reproducing the in-domain optimization landscape.
  • fidelity parameter experiments for HPO emulators (gx6e; Sec. B.7). Our choice of emulator fidelity inputs improves on or is comparable to the alternatives tested.
  • joint-fidelity HPO variant (gx6e; Sec. 3.1), which uses our existing emulators to give both continuous token fidelity and discrete model fidelity in one problem.
  • noise sensitivity analysis (aV6v; Sec. D.12) for PO and PCO experiments, where method rankings largely remain in good agreement across noise levels.

We are glad that all reviewers found our responses satisfactory, with Reviewer gx6e increasing their score. We thank the reviewers again for their feedback, which have improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.