- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1(−1)
Read review
Paper summary
Bayesian optimization (BO) is widely used for expensive experiment design, necessitating sample-efficient scheme. To make BO sample efficient, researchers use MAP estimation of kernel hyperparameters given observations. The issue posed by MAP estimation is essentially the ignorance of the uncertainty in the estimation. A fully Bayesian BO is known to be the remedy for this, but its computation is prohibitively expensive. This paper proposes to use the idling time during experiment execution to collect MC samples of the hyperparameters from the posterior, and reuse them via importance sampling. In combination with expensive experiments, the proposed method enjoys a fully Bayesian BO, while hiding the computational overhead behind the long wait time.
Critical strengths
Clear writing & presentation
I personally do not see any issues in the authors' scientific communication. For example, the figure captions are self-contained and easy to understand in combination with good visualization (Figs. 1, 2). Contents are tightly packed and no obvious redundancies are found from my side. So once important issues listed later are fixed, I am happy to increase the original score.
Additionally, the proposed method is very simple yet effective under some setups, so trying it out is not a big hurdle. It is also nice that they attached their implementations.
Limitations are honestly acknowledged
Since the proposed method is rather a practical method, it is important to share the limitations or downsides of the paper with the community.
Critical weaknesses
The motivation is somewhat contradictory to practical considerations
The first issue is that the authors claim FBO should be used for the low-budget regime where the hyperparameter distribution is uncertain, while this is exactly where the importance sampling can be often vulnerable due to a large posterior shift by informative observations. The authors repeatedly claim this to justify the use of FBO, but I do not see strong evidence, e.g., citations or experiments, of how stable EL-FBO is in this regime. Although the authors acknowledge that small can be used as a detector, this is not related to my point (, and the remedy for this point should also be presented). Whereas the presentation is very clear, the current limited contributions make the paper difficult to give acceptance recommendation.
Another point is indeed more ironic because we often would like to evaluate costly experiments in parallel, reducing the idling time significantly. Therefore, the lack of asynchronous evaluation substantially weakens the practical motivation and breadth of the empirical validation. The experiments nevertheless do not cover the asynchronous setup. The question is then how much speedup effect we can expect for each degree of parallelism. I'd like to see how much improvement we can expect over the parallel number growth. First of all, is FBO effective for such setup? Plus, how much runtime reduction can we expect with EL-FBO? If the asynchronous implementation is the bottleneck, OptunaHub provides a nice tool that enables benchmarking asynchronous setup without explicitly implementing this:
Additionally, the motivation saying that the planner is free during the experiments is also not necessarily true. For example, if we assume computational experiments, we do not want to continue running the BO routine behind the main experiment. Additionally, I doubt that people assign an exclusive planner process. In fact, both NePS and Optuna, widely known in the AutoML community, distribute planners across workers. So we need to explicitly run the planner concurrently in this case, increasing the code complexity. If this is not in the scope of this study, introduction should clearly state it.
Experiments (and their interpretation) can be fairer
Fig. 3 says FBO and EL-FBO achieve earlier convergence in many settings, but I cannot indeed see such behavior.
Even if I look at it favorably, Styblinski (5D), Six-Hump Camel (2D), and Egg Holder (2D) are the only ones that I can see marginally faster convergence.
This is only 3 out 15, so I'd say it is far from many.
The next point is the initialization strategy. Although the main text says they used samples from the Sobol sequence for the initialization, each method does not start from the same performance. I do not exactly know how the authors initialized each method, but for fair comparison, each method should share the same initialization samples. Looking at the top row of Figure 4, each curve does not start from the same point. For each trial, did MAP, FBO, and ELF-BO use exactly the same initial Sobol observations? Also, does iteration 0 in Figures 3, 4 denote the first BO-selected evaluation after the initial Sobol design? Is identical across methods within that trial before computing GAP? If not, please clarify whether GAP is normalized separately for each method run.
Most importantly, the asynchronous setup is entirely missing. As mentioned earlier, I'd say many people would think of using FBO under this setup.
Finally, some important baselines are missing. For example, why not using the FBO that reuse previous chain states, or adaptation information? Another baseline is FBO with reduced budget. Since it is obvious that the FBO with full budget incurs significant overhead, one would consider adapting the budget for FBO in the first place, making it natural to compare the proposed method with the reduced budget version. This baseline should be significantly faster than the current baseline, and if this baseline yield competitive performance, this adjustment is sufficient for most people. I'd say presenting why we still need to use the proposed method strengthens the paper.
Questions for authors
Suggestions
- Please see
Critical Weaknesses. - The asynchronous setup and the evidence for low-data regime are the most important points for me.
Questions
- Please see
Critical Weaknesses.
4 comments
Thank you for your comments.
In short, I would not use ELF-BO instead of FBO based on the provided data. In my honest opinion, your paper is very well written and the presentation is clear to me, but the reason why people should use the method in practice is very unclear based on the experimental results. Note that I am valuing empirical and practical benefits for this paper because of the limited methodological and conceptual novelties.
Low Performance in Async Setup
First of all, the absolute gain in Figs. 6 & 10 is too small for the additional complexity we incur in the whole system. The marginal gain is especially striking for async setup where the sleeping time would be shorter.
I'd suggest reframing the motivation of the paper and exploring setups where ELF-BO becomes more practical. There are also many other problem setups such as multi-objective optimization, constrained optimization, multi-fidelity optimization. Given the low performance, it'd be much more beneficial for communities if you find where ELF-BO is valuable. I did not unfortunately get such insight.
Another Point: Is MAP really weaker than FBO?
It is fine to cite old papers to say FBO outperforms MAP, but is it really true given recent works? All cited papers for this argument are older than the used prior by Hvarfner et al. (2024). The verification is necessary, e.g., on bbob.
Low Data Regime
For the Matérn kernel, which is the main setup, MCMC is anyways cheap in this regime, so I am still not sure about the benefit. The importance sampling bias is a real concern, but no proposition on Meff safeguard. The paper doesn't analyze estimator bias or error conditional on surprising observations either.
Overall, given the current motivation and results, I would not try the proposed method (but try lowering MCMC cost or using MAP instead). Although the paper is well written and the presentation is also good, the current framework is neither practical (marginal gain with a new critical failure mode) nor surprising to me. I'd be happy to raise the score if the authors find where the method is highly practical or why we should stop using FBO and MAP. Let me keep the score.
Please note that I recognize the speedup gain, but I'm wondering whether practitioners would choose a computationally intensive kernel for relatively cheap experiments (4--8 min). Aren't MAP & running more experiments sufficient in this scenario? For expensive experiments (hours to days), people run experiments in parallel, so the FBO's overhead is acceptable and the gain of ELF-BO is small with a critical failure mode.
I also noticed that different priors are used for MAP and FBO (Section 5). A controlled comparison should use the same prior, since otherwise the observed performance differences may be attributable to the prior rather than Bayesian marginalization.
This further strengthens my concern about whether FBO actually outperforms a properly configured MAP baseline. Given that ELF-BO's practical motivation relies heavily on preserving the sample-efficiency benefits of FBO while reducing its computational overhead, this confounding factor undermines a central empirical justification for the proposed approach.
I therefore lower my soundness score and rating from 2 to 1.
Thank you for the additional feedback. We have tried to address as much of it as we could in the limited time before the end of the discussion period.
Fully Bayesian vs. MAP
The goal of our paper is not to show that fully Bayesian optimization (FBO) consistently outperforms MAP on all tasks. Our objective is to provide practitioners with a lightweight version of FBO for problems in which being fully Bayesian offers advantages over a point estimate. We note that the cited work by Hvarfner et al. (2024) itself reports that, on certain problems, the fully Bayesian approach (SAASBO) outperforms their proposed scaled-RBF prior. In the caption of Figure 5 in Hvarfner et al., when talking about results in some synthetic functions, it states "Vanilla BO performs second best, beaten only by SAASBO on four tasks". This demonstrates that there are situations in which fully Bayesian outperforms MAP.
In line with this, we show real-world applications in which FBO and ELF-BO outperform MAP. On these problems, ELF-BO avoids the extra latency that Standard FBO would impose.
Prior Mismatch
We use a different prior because the original prior from Hvarfner et al. was designed with MAP in mind: its parameters are chosen to place its mode at the desired lengthscale. As a result, the heavy right tail of the log-normal distribution makes its mean and median too large for sampling. For this reason, we parametrize the log-normal by the location of its median instead.
We have updated the results in the paper to use the same prior for all methods; that is, MAP now also uses our modified prior. Although MAP's performance has improved on the synthetic functions, our goal of demonstrating that Standard FBO and ELF-BO agree still holds. Furthermore, the real-world results still show that ELF-BO matches Standard FBO's optimization performance at a decision latency similar to MAP's.
Faster evaluations
To illustrate the setting in which ELF-BO is most useful, we added experiments with a shorter evaluation time, s, on all five real-world tasks (Fig. 4, top). Since the datasets are queried as lookup tables, is a simulation parameter; we deliberately chose 20 s, just above Standard FBO's sampling time, to highlight the regime where decision latency matters most, rather than to represent the original experiments.
In this regime, having better and faster queries is crucial. ELF-BO runs as many experiments as MAP in the same wall-clock time, since both decide in under a second, while Standard FBO spends 12–16 s per decision, 60–80% of each evaluation, and completes only about 30 experiments in the time ELF-BO completes 50. At the end of its campaign, ELF-BO matches or exceeds MAP on four of the five tasks and Standard FBO on all five. With the original evaluation times (Fig. 4, bottom), the gap between ELF-BO and Standard FBO in wall-clock time is small (3–5%), which is consistent with your observation; ELF-BO's benefit grows as evaluations get closer to the cost of inference.
Response to Reviewer 5b13
Thank you for the helpful and thorough feedback! Below are our answers, with pointers to the revised manuscript. We hope these improvements convince you to reconsider your score, and we are happy to clarify anything further.
Low-data Regime
We agree that the low-data regime is where accounting for hyperparameter uncertainty matters most, and it is the regime we target throughout. Our real-world campaigns start from only 2d points and end with about 60 observations in 4–6 dimensions (e.g., 62 points in 6D for HPLC).
A natural concern is that the hyperparameter posterior shifts the most when data is scarce, so importance sampling from p(θ∣Dt−1) could be least reliable early on. Our new approximation analysis (Sec. 6, Figs. 7b, 7c) addresses this directly: at every step, including the earliest ones, ELF-BO tracks FBO on par with an independent FBO run. For t<10, W1=0.072±0.010 vs. 0.060±0.003 for the control.
In addition, on the real-world tasks the fully Bayesian methods, ELF-BO included, reach higher GAP than MAP within roughly the first 20 steps (Fig. 4), the most data-scarce part of the campaign.
Asynchronous Setting
Thank you for this suggestion. We extended ELF-BO to asynchronous batch BO (Sec. 5.2.1, App. D):
Planner Compute Resources
Our target applications are objectives evaluated on dedicated hardware separate from the planner, such as self-driving laboratories or remote simulations, where the planner's machine sits idle while the experiment runs. We now state this scope explicitly in the Introduction (L085-L088).
ELF-BO also requires no additional compute compared to Standard FBO: it runs the same MCMC sampler, but schedules it during otherwise idle time instead of after the observation arrives. The only added decision-time cost is computing the importance weights, which is negligible.
Finally, if the objective is a simulation sharing hardware with the planner, the two compete for resources, and ELF-BO's benefit depends on the spare capacity available. We added this to the Limitations (L536-L537).
Improvements over MAP
This point was also raised by Reviewer SSqY; we address it jointly in our Global Comment.
The synthetic benchmarks were designed to test whether ELF-BO reproduces Standard FBO, not to show gains over MAP. We agree that the original Fig. 3 caption overstated the results and revised it to match Sec. 5.1: ELF-BO tracks Standard FBO on most functions, and the benefit of full Bayes over MAP depends on the function (De Ath et al., 2021).
On all five real-world tasks, ELF-BO consistently outperforms MAP in area under the GAP curve, with the largest gains early; the same holds for the deep kernel and asynchronous experiments (Figs. 4–6). Since ELF-BO follows Standard FBO both when it outperforms MAP and when they are equivalent, practitioners need not know in advance whether a point estimate is "good enough": ELF-BO provides the fully Bayesian treatment at latency close to MAP's.
GAP Figure Clarification
For a fixed objective and seed, all methods start from the same D0 (L370-L371). The previous figures began after the first BO step, so starting points appeared to differ. The revised figures start every curve at GAP$_0 = 0$, and we clarified the definition of y0max (L374-L375).
References
George De Ath, Richard M Everson, and Jonathan E Fieldsend. How Bayesian Should Bayesian Optimisation Be? GECCO Companion, 2021.
Ben Riegler, James Odgers, and Vincent Fortuin. Standard Acquisition Is Sufficient for Asynchronous Bayesian Optimization. AISTATS, 2026.