- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Optimizing LLM training and inference requires an exhaustive search of many different design choices, such as what hyperparameters to tune, what data to train on, how to optimize prompts, etc. Each design choice is expensive to run, which highlights the need to predict results quickly to find the best configuration. This paper proposes that Bayesian-optimization (BO) approaches are well-suited for these tasks, and estabalishes a benchmark to test different BO strategies to predict outcomes of LLM design. They find that BO methods are generally useful, but no one method is clearly the best. The authors hope that this benchmark can help BO researchers to apply new methods to improve and facilitate the iterative process of LLM design.
Critical strengths
This paper establishes four kinds of tasks that are relevant in LLM training and development:
- LoRA hyperparameter finetuning;
- Data mixing ratios;
- Prompt engineering;
- Training parallelism configurations.
All four tasks are relevant to optimizing LLM training and/or inference. The paper presents a unique view that bayesian-optimization approaches could be used to help guide search over the different choices of designs to optimize LLM development.
This paper also purports to make LLM optimization more accessible without the access of GPUs. In order to accomplish this feature, they design emulators which can simulate experimental runs. The emulators are simple MLPs which can directly estimate a model's performance given its hyperparameter configuration, which is an interesting approach.
Critical weaknesses
The main emulator’s model choice feels insufficiently validated. I appreciate the authors' ablation on varying MLP width and depth to predict hyperparam configs → performance, but I am missing intuition on why this particular choice of architecture is suited for this task. Could we design simpler emulators, e.g. a linear regression fit, or even fit a scaling law, that takes as input the hyperparameter configs and predicts model accuracy? Given the vast literature on scaling laws, I would like to see more discussion on the relationship between emulation of LLM training and extrapolation of LLM training trends.
There is also the question of how emulators can affect the optimization landscape. For example, different emulator models could have the same prediction accuracy, but be much different in terms of the local smoothness, or interactions - and this could greatly affect the performance of bayesian-optimization methods. I think a worthy additional experiment would be to compare alternative emulator architectures and observe how the optimizer results could change.
While the LLM task coverage is broad, I would be more interested in learning what these tasks collectively teach us about choosing or designing optimizers for LLM workflows. The paper offers individual insights per task, e.g. local search helps prompt selection, domain priors help parallelism optimization, etc, but lacks a cohesive story about how to build a broad decision framework that could equip practitioners about how to improve LLM training.
Questions for authors
Can you discuss the relationship between your emulators and scaling laws? For example, could scaling law models, or linear regression fits, also achieve similar predictive performance as your MLP-architecture emulators? Would the models also be useful for BO?
Tables 4–5 compare MLP architectures, but do the relative rankings of optimizers remain consistent across different emulator model families? I believe this could help clarify whether the benchmark reflects the underlying LLM tasks, or if they reflect the properties of the emulator itself.
Although this benchmark is built primarily for optimization researchers, are there recommendations or guidance that one can make to practitioners when choosing to optimize their LLM development?
6 comments
Official Comment by Authors (2/3)
(W2) There is also the question of how emulators can affect the optimization landscape. For example, different emulator models could have the same prediction accuracy, but be much different in terms of the local smoothness, or interactions - and this could greatly affect the performance of bayesian-optimization methods. I think a worthy additional experiment would be to compare alternative emulator architectures and observe how the optimizer results could change.
(Q2) … do the relative rankings of optimizers remain consistent across different emulator model families? I believe this could help clarify whether the benchmark reflects the underlying LLM tasks, or if they reflect the properties of the emulator itself.
To assess how emulator choice affects optimizer rankings, we extend the real-versus-emulated comparison in Fig. 9 to additional emulator families. Specifically, we compare MLP, scaling/mixing-law, and Matérn GP models here, to illustrate different model families. The full experiment additionally includes quadratic ridge, kernel ridge, and gradient boosting (Sec. B.6.1).
On each emulator, we compare four BBO methods, with the same 10 initial evaluations, over 50 optimization iterations and five seeds. We compare these rankings against those obtained by running the same methods with real LLM evaluations, using matched seeds. We compute Spearman between seed-averaged method ranks at each iteration, then average over the indicated window:
| Emulator | HPO–8B: iter. 1–50 | HPO–8B: iter. 31–50 | DMO–4B: iter. 1–50 | DMO–4B: iter. 31–50 |
|---|---|---|---|---|
| MLP | 0.581 | 0.910 | 0.743 | 0.946 |
| Scaling / mixing law | 0.512 | 0.910 | 0.602 | 0.553 |
| Matérn-5/2 GP | 0.041 | 0.610 | 0.335 | 0.454 |
Optimizer rankings are sensitive to the emulator family, even when predictive accuracy is comparable. Although the Matérn-5/2 GP has the highest predictive accuracy on HPO–8B, it has substantially lower ranking agreement with real runs. One possible explanation is that GP-based BO methods can model a GP emulator’s objective particularly well because of their shared modeling assumptions. This could disproportionately favor GP-based optimizers relative to non-GP baselines.
These results support the reviewer’s concern that predictive accuracy alone is insufficient for assessing whether an emulator preserves optimizer rankings. Comparing rankings against real runs is thus an important complementary check for potential emulator bias. Among the emulator families evaluated, our MLP emulators achieve the highest average ranking agreement over iterations 1–50 on both tasks, while the scaling law matches their late-stage agreement on HPO.
We have added this ablation study in Sec. B.6.1 of our revision, including full regret plots for each emulator.
Official Comment by Authors (3/3)
(W3) While the LLM task coverage is broad, I would be more interested in learning what these tasks collectively teach us about choosing or designing optimizers for LLM workflows. The paper offers individual insights per task, e.g. local search helps prompt selection, domain priors help parallelism optimization, etc, but lacks a cohesive story about how to build a broad decision framework that could equip practitioners about how to improve LLM training.
(Q3) Although this benchmark is built primarily for optimization researchers, are there recommendations or guidance that one can make to practitioners when choosing to optimize their LLM development?
While our experiments do not identify one optimizer that is best for every LLM workflow, they suggest that formulating the BBO problem appropriately is critical. Practitioners can use three broad questions to guide this formulation and narrow their optimizer choices.
(1) What is the search space, and what domain knowledge is available? For high-dimensional spaces, consider local search methods such as TuRBO, which performed well on our prompt-selection tasks. Where domain knowledge is available, use it to constrain the search space or inform the optimizer’s prior, as illustrated by our parallelism-optimization experiments. These are different ways to make optimization more efficient.
(2) What information do evaluations provide, and what objectives matter? If cheaper evaluations are informative about the target setting, consider multi-fidelity methods. If observations are very noisy, use methods that account for this, like LogNEI. When several objectives must be balanced, multi-objective methods can identify a Pareto frontier of trade-offs rather than requiring a single objective weighting in advance.
(3) What compute budget and resources are available? With a small evaluation budget, more exploitative settings may reach good configurations sooner, as illustrated by UCB with a smaller exploration parameter in our HPO experiments. With a larger budget, additional exploration may become worthwhile. If compute resources permit, consider batched methods to run multiple evaluations in parallel and reduce wall-clock time.
Within the appropriate formulation, LogNEI is a reasonable starting point for noisy single-objective optimization, while MES/GIBBON and their multi-fidelity variants are alternatives worth comparing. Their relative performance depends on the task and evaluation budget, so we recommend comparing a small set of suitable methods rather than adopting one acquisition function across all workflows.
This framework connects optimizer choices to the structure and resource constraints of an LLM workflow, and our benchmark provides evidence to help narrow these choices.
We have added this discussion to Sec. E of our revision.
We have compared alternative emulator models and their agreement with real optimizer rankings, clarified the relationship to scaling laws, and added guidance for practitioners. We hope these analyses and revisions address your concerns.
Official Comment by Authors (1/3)
We thank the reviewer for their valuable review and for appreciating the relevance of our LLM optimization tasks, our perspective on BO for LLM development, and our MLP emulator approach. We address their concerns and questions below.
Thank you for this suggestion. We have added a comparison of emulator models, including linear ridge regression, scaling-law/mixing-law models, kernel ridge regression, and Gaussian processes, using the same training data and held-out validation sets.
For HPO's scaling law, we evaluate a Chinchilla-inspired power-law baseline [1], adapted to predict downstream accuracy at fixed model size, with coefficients conditioned on the hyperparameters, acc(x,D)=a(x)−A(x)⋅D−α(x), where D is in millions of training tokens and x is the hyperparameter configuration. a(x), logA(x), and logα(x) are linear in x.
For DMO's mixing law, we adapt the form from [2], acc(x)=c+k⋅exp(t⋅x), where x here denotes the two-stage data mixture proportions, t is a coefficient vector, and c and k are scalar coefficients.
We show results for a subset of emulator models, with the full ablation in the revision (Sec. B.6) including additional regression, kernel, GP, and tree-based models.
Linear regression falls substantially short of the MLPs, but the scaling-law baseline is competitive on HPO-4B, in terms of Spearman's ρ. The Matérn-5/2 GP achieves the highest predictive metrics on HPO-8B, but our transferability comparison below (in our response to Q2 and W2) shows that this does not translate into better agreement with real optimizer rankings.
[1] Hoffmann et al. Training compute-optimal large language models. arXiv, 2022.
[2] Ye et al. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. ICLR, 2025.
Scaling laws and our emulators both predict training outcomes without having to conduct every training run. The main difference is that scaling laws such as those used in Chinchilla [1] encode assumptions about how performance changes with training scale, which guide extrapolation beyond observed scales, while our emulators aim to reproduce the optimization landscape within the benchmark domain.
In our emulator comparisons, MLPs generally perform better than the scaling-law models. MLPs provide greater flexibility in modeling interactions and objective landscapes, which is an advantage in our setting, where we evaluate held-out configurations within the benchmark domain rather than extrapolation to larger models or longer training budgets.
Scaling-law models can also be useful within BO. For example, Kadra et al. [3] incorporate power-law learning-curve models into BO to inform configuration selection.
We have added this discussion of scaling laws and emulators in Sec. B.6.2 of our revision.
[3] Kadra et al. Scaling laws for hyperparameter optimization. NeurIPS, 2023.