acceptodds
Under review as a conference paper at ICLR 2027

Work While They Sleep: Exploiting Evaluation Latency for Fully Bayesian Optimization

Abstract

Black-box optimization problems are ubiquitous across science and engineering, often dealing with expensive objective functions. This objective latency has two consequences during optimization: (i) the objective evaluation dominates execution time, and (ii) sample-efficient algorithms are crucial to accelerate development and avoid wasting resources. Bayesian optimization (BO) methods are the de facto choice of planners for suggesting the next point to try. Standard BO fits the surrogate model's hyperparameters with a point estimate. Alternatively, a fully Bayesian approach uses model averaging to account for uncertainty over the hyperparameters, leading to better uncertainty estimates—useful in the low-data regime that is pervasive in BO. However, it is often prohibitively expensive and thus rarely used. In this work, we propose ELF-BO, an algorithm that uses the objective evaluation latency to headstart the computation of the next suggestion, allowing for fully Bayesian optimization without incurring substantial decision-time costs. This is done by sampling from the hyperparameter posterior _while_ the objective is being evaluated, only requiring reweighting of the samples once the objective value is observed. Across synthetic functions and real-world applications, we show that ELF-BO matches the performance of fully Bayesian methods while only incurring decision latency on par with or better than standard BO. Thus, ELF-BO makes fully Bayesian optimization practical in real-world use cases.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
bayesian optimization, fully bayesian optimization, markov chain monte carlo

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5b13OpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1(−1)
Read review
Paper summary

Bayesian optimization (BO) is widely used for expensive experiment design, necessitating sample-efficient scheme. To make BO sample efficient, researchers use MAP estimation of kernel hyperparameters given observations. The issue posed by MAP estimation is essentially the ignorance of the uncertainty in the estimation. A fully Bayesian BO is known to be the remedy for this, but its computation is prohibitively expensive. This paper proposes to use the idling time during experiment execution to collect MC samples of the hyperparameters from the posterior, and reuse them via importance sampling. In combination with expensive experiments, the proposed method enjoys a fully Bayesian BO, while hiding the computational overhead behind the long wait time.

Critical strengths

Clear writing & presentation

I personally do not see any issues in the authors' scientific communication. For example, the figure captions are self-contained and easy to understand in combination with good visualization (Figs. 1, 2). Contents are tightly packed and no obvious redundancies are found from my side. So once important issues listed later are fixed, I am happy to increase the original score.

Additionally, the proposed method is very simple yet effective under some setups, so trying it out is not a big hurdle. It is also nice that they attached their implementations.

Limitations are honestly acknowledged

Since the proposed method is rather a practical method, it is important to share the limitations or downsides of the paper with the community.

Critical weaknesses

The motivation is somewhat contradictory to practical considerations

The first issue is that the authors claim FBO should be used for the low-budget regime where the hyperparameter distribution is uncertain, while this is exactly where the importance sampling can be often vulnerable due to a large posterior shift by informative observations. The authors repeatedly claim this to justify the use of FBO, but I do not see strong evidence, e.g., citations or experiments, of how stable EL-FBO is in this regime. Although the authors acknowledge that small MM can be used as a detector, this is not related to my point (, and the remedy for this point should also be presented). Whereas the presentation is very clear, the current limited contributions make the paper difficult to give acceptance recommendation.

Another point is indeed more ironic because we often would like to evaluate costly experiments in parallel, reducing the idling time significantly. Therefore, the lack of asynchronous evaluation substantially weakens the practical motivation and breadth of the empirical validation. The experiments nevertheless do not cover the asynchronous setup. The question is then how much speedup effect we can expect for each degree of parallelism. I'd like to see how much improvement we can expect over the parallel number growth. First of all, is FBO effective for such setup? Plus, how much runtime reduction can we expect with EL-FBO? If the asynchronous implementation is the bottleneck, OptunaHub provides a nice tool that enables benchmarking asynchronous setup without explicitly implementing this:

Additionally, the motivation saying that the planner is free during the experiments is also not necessarily true. For example, if we assume computational experiments, we do not want to continue running the BO routine behind the main experiment. Additionally, I doubt that people assign an exclusive planner process. In fact, both NePS and Optuna, widely known in the AutoML community, distribute planners across workers. So we need to explicitly run the planner concurrently in this case, increasing the code complexity. If this is not in the scope of this study, introduction should clearly state it.

Experiments (and their interpretation) can be fairer

Fig. 3 says FBO and EL-FBO achieve earlier convergence in many settings, but I cannot indeed see such behavior. Even if I look at it favorably, Styblinski (5D), Six-Hump Camel (2D), and Egg Holder (2D) are the only ones that I can see marginally faster convergence. This is only 3 out 15, so I'd say it is far from many.

The next point is the initialization strategy. Although the main text says they used 2D2D samples from the Sobol sequence for the initialization, each method does not start from the same performance. I do not exactly know how the authors initialized each method, but for fair comparison, each method should share the same initialization samples. Looking at the top row of Figure 4, each curve does not start from the same point. For each trial, did MAP, FBO, and ELF-BO use exactly the same initial 2D2D Sobol observations? Also, does iteration 0 in Figures 3, 4 denote the first BO-selected evaluation after the initial Sobol design? Is y0max⁡y_0^{\max} identical across methods within that trial before computing GAP? If not, please clarify whether GAP is normalized separately for each method run.

Most importantly, the asynchronous setup is entirely missing. As mentioned earlier, I'd say many people would think of using FBO under this setup.

Finally, some important baselines are missing. For example, why not using the FBO that reuse previous chain states, or adaptation information? Another baseline is FBO with reduced budget. Since it is obvious that the FBO with full budget incurs significant overhead, one would consider adapting the budget for FBO in the first place, making it natural to compare the proposed method with the reduced budget version. This baseline should be significantly faster than the current baseline, and if this baseline yield competitive performance, this adjustment is sufficient for most people. I'd say presenting why we still need to use the proposed method strengthens the paper.

Questions for authors

Suggestions

  • Please see Critical Weaknesses.
  • The asynchronous setup and the evidence for low-data regime are the most important points for me.

Questions

  • Please see Critical Weaknesses.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 5b13

Thank you for the helpful and thorough feedback! Below are our answers, with pointers to the revised manuscript. We hope these improvements convince you to reconsider your score, and we are happy to clarify anything further.

Low-data Regime

We agree that the low-data regime is where accounting for hyperparameter uncertainty matters most, and it is the regime we target throughout. Our real-world campaigns start from only 2d2d points and end with about 60 observations in 4–6 dimensions (e.g., 62 points in 6D for HPLC).

A natural concern is that the hyperparameter posterior shifts the most when data is scarce, so importance sampling from p(θ∣Dt−1)p(\theta \mid \mathcal{D}_{t-1}) could be least reliable early on. Our new approximation analysis (Sec. 6, Figs. 7b, 7c) addresses this directly: at every step, including the earliest ones, ELF-BO tracks FBO on par with an independent FBO run. For t<10t < 10, W1=0.072±0.010W_1 = 0.072 \pm 0.010 vs. 0.060±0.0030.060 \pm 0.003 for the control.

In addition, on the real-world tasks the fully Bayesian methods, ELF-BO included, reach higher GAP than MAP within roughly the first 20 steps (Fig. 4), the most data-scarce part of the campaign.

Asynchronous Setting

Thank you for this suggestion. We extended ELF-BO to asynchronous batch BO (Sec. 5.2.1, App. D):

  • Method: ELF-BO starts MCMC on the data available when a batch is dispatched. If several evaluations finish in the meantime, it reweights the same samples with all of them at once: by Remark 2 (App. A), Proposition 1 extends directly, with the weight given by the joint predictive density of the k new observations. ELF-BO then fills the free workers via sequential greedy qLog-EI and resumes sampling. Following Riegler et al. (2026), pending evaluations are not modeled, keeping latency low (Alg. 2).
  • Setup: Evaluation durations are drawn randomly around each benchmark's mean evaluation time, under a fixed wall-clock budget. We use q = 4 workers in the main text (Fig. 6) and q = 2, 8 in App. D.4.
  • Results: With q = 4, ELF-BO matches or exceeds the best baseline in area under the GAP curve on all benchmarks, while reducing worker idle time by 7–9× compared to Standard FBO (from 2.9–5.9% to 0.3–0.9%). The reduction is largest with few workers: 14–18× at q = 2 and 4–5× at q = 8.

Planner Compute Resources

Our target applications are objectives evaluated on dedicated hardware separate from the planner, such as self-driving laboratories or remote simulations, where the planner's machine sits idle while the experiment runs. We now state this scope explicitly in the Introduction (L085-L088).

ELF-BO also requires no additional compute compared to Standard FBO: it runs the same MCMC sampler, but schedules it during otherwise idle time instead of after the observation arrives. The only added decision-time cost is computing the importance weights, which is negligible.

Finally, if the objective is a simulation sharing hardware with the planner, the two compete for resources, and ELF-BO's benefit depends on the spare capacity available. We added this to the Limitations (L536-L537).

Improvements over MAP

This point was also raised by Reviewer SSqY; we address it jointly in our Global Comment.

The synthetic benchmarks were designed to test whether ELF-BO reproduces Standard FBO, not to show gains over MAP. We agree that the original Fig. 3 caption overstated the results and revised it to match Sec. 5.1: ELF-BO tracks Standard FBO on most functions, and the benefit of full Bayes over MAP depends on the function (De Ath et al., 2021).

On all five real-world tasks, ELF-BO consistently outperforms MAP in area under the GAP curve, with the largest gains early; the same holds for the deep kernel and asynchronous experiments (Figs. 4–6). Since ELF-BO follows Standard FBO both when it outperforms MAP and when they are equivalent, practitioners need not know in advance whether a point estimate is "good enough": ELF-BO provides the fully Bayesian treatment at latency close to MAP's.

GAP Figure Clarification

For a fixed objective and seed, all methods start from the same D0\mathcal{D}_0 (L370-L371). The previous figures began after the first BO step, so starting points appeared to differ. The revised figures start every curve at GAP$_0 = 0$, and we clarified the definition of y0maxy_0^\text{max} (L374-L375).

References

George De Ath, Richard M Everson, and Jonathan E Fieldsend. How Bayesian Should Bayesian Optimisation Be? GECCO Companion, 2021.

Ben Riegler, James Odgers, and Vincent Fortuin. Standard Acquisition Is Sufficient for Asynchronous Bayesian Optimization. AISTATS, 2026.

Thank you for your comments.

In short, I would not use ELF-BO instead of FBO based on the provided data. In my honest opinion, your paper is very well written and the presentation is clear to me, but the reason why people should use the method in practice is very unclear based on the experimental results. Note that I am valuing empirical and practical benefits for this paper because of the limited methodological and conceptual novelties.

Low Performance in Async Setup

First of all, the absolute gain in Figs. 6 & 10 is too small for the additional complexity we incur in the whole system. The marginal gain is especially striking for async setup where the sleeping time would be shorter.

I'd suggest reframing the motivation of the paper and exploring setups where ELF-BO becomes more practical. There are also many other problem setups such as multi-objective optimization, constrained optimization, multi-fidelity optimization. Given the low performance, it'd be much more beneficial for communities if you find where ELF-BO is valuable. I did not unfortunately get such insight.

Another Point: Is MAP really weaker than FBO?

It is fine to cite old papers to say FBO outperforms MAP, but is it really true given recent works? All cited papers for this argument are older than the used prior by Hvarfner et al. (2024). The verification is necessary, e.g., on bbob.

Low Data Regime

For the Matérn kernel, which is the main setup, MCMC is anyways cheap in this regime, so I am still not sure about the benefit. The importance sampling bias is a real concern, but no proposition on Meff safeguard. The paper doesn't analyze estimator bias or error conditional on surprising observations either.


Overall, given the current motivation and results, I would not try the proposed method (but try lowering MCMC cost or using MAP instead). Although the paper is well written and the presentation is also good, the current framework is neither practical (marginal gain with a new critical failure mode) nor surprising to me. I'd be happy to raise the score if the authors find where the method is highly practical or why we should stop using FBO and MAP. Let me keep the score.

Please note that I recognize the speedup gain, but I'm wondering whether practitioners would choose a computationally intensive kernel for relatively cheap experiments (4--8 min). Aren't MAP & running more experiments sufficient in this scenario? For expensive experiments (hours to days), people run experiments in parallel, so the FBO's overhead is acceptable and the gain of ELF-BO is small with a critical failure mode.

I also noticed that different priors are used for MAP and FBO (Section 5). A controlled comparison should use the same prior, since otherwise the observed performance differences may be attributable to the prior rather than Bayesian marginalization.

This further strengthens my concern about whether FBO actually outperforms a properly configured MAP baseline. Given that ELF-BO's practical motivation relies heavily on preserving the sample-efficiency benefits of FBO while reducing its computational overhead, this confounding factor undermines a central empirical justification for the proposed approach.

I therefore lower my soundness score and rating from 2 to 1.

Thank you for the additional feedback. We have tried to address as much of it as we could in the limited time before the end of the discussion period.

Fully Bayesian vs. MAP

The goal of our paper is not to show that fully Bayesian optimization (FBO) consistently outperforms MAP on all tasks. Our objective is to provide practitioners with a lightweight version of FBO for problems in which being fully Bayesian offers advantages over a point estimate. We note that the cited work by Hvarfner et al. (2024) itself reports that, on certain problems, the fully Bayesian approach (SAASBO) outperforms their proposed scaled-RBF prior. In the caption of Figure 5 in Hvarfner et al., when talking about results in some synthetic functions, it states "Vanilla BO performs second best, beaten only by SAASBO on four tasks". This demonstrates that there are situations in which fully Bayesian outperforms MAP.

In line with this, we show real-world applications in which FBO and ELF-BO outperform MAP. On these problems, ELF-BO avoids the extra latency that Standard FBO would impose.

Prior Mismatch

We use a different prior because the original prior from Hvarfner et al. was designed with MAP in mind: its parameters are chosen to place its mode at the desired lengthscale. As a result, the heavy right tail of the log-normal distribution makes its mean and median too large for sampling. For this reason, we parametrize the log-normal by the location of its median instead.

We have updated the results in the paper to use the same prior for all methods; that is, MAP now also uses our modified prior. Although MAP's performance has improved on the synthetic functions, our goal of demonstrating that Standard FBO and ELF-BO agree still holds. Furthermore, the real-world results still show that ELF-BO matches Standard FBO's optimization performance at a decision latency similar to MAP's.

Faster evaluations

To illustrate the setting in which ELF-BO is most useful, we added experiments with a shorter evaluation time, Tf=20T_f = 20 s, on all five real-world tasks (Fig. 4, top). Since the datasets are queried as lookup tables, TfT_f is a simulation parameter; we deliberately chose 20 s, just above Standard FBO's sampling time, to highlight the regime where decision latency matters most, rather than to represent the original experiments.

In this regime, having better and faster queries is crucial. ELF-BO runs as many experiments as MAP in the same wall-clock time, since both decide in under a second, while Standard FBO spends 12–16 s per decision, 60–80% of each evaluation, and completes only about 30 experiments in the time ELF-BO completes 50. At the end of its campaign, ELF-BO matches or exceeds MAP on four of the five tasks and Standard FBO on all five. With the original evaluation times (Fig. 4, bottom), the gap between ELF-BO and Standard FBO in wall-clock time is small (3–5%), which is consistent with your observation; ELF-BO's benefit grows as evaluations get closer to the cost of inference.

Reviewer_SSqYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses the latency between successive evaluations in fully Bayesian optimization (FBO), particularly the computational cost of Markov chain Monte Carlo (MCMC) sampling required to update the posterior over GP hyperparameters after each observation. The proposed method, ELF-BO, exploits objective-evaluation time to perform MCMC sampling in parallel with function evaluations. Specifically, it samples hyperparameters from the posterior conditioned on the currently available data, $p(\theta \mid \mathcal{D}_{t-1}) $, while the objective function is being evaluated. Once the new observation becomes available, importance reweighting is applied to account for the updated posterior $ p(\theta \mid \mathcal{D}_t) $. This approach aims to reduce decision latency by moving most of the MCMC computation outside the interval between successive evaluations, particularly when objective evaluations take longer than posterior sampling.

Critical strengths
  1. The proposed method is conceptually simple and clearly presented. The importance-reweighting formulation provides a straightforward way to reuse hyperparameter samples obtained before the latest observation becomes available.
  2. The experimental evaluation is reasonably comprehensive, covering multiple synthetic benchmark functions and real-world experimental datasets.
Critical weaknesses
  1. The practical significance of the proposed latency reduction remains insufficiently established. In many real-world black-box optimization applications, each objective evaluation can take hours or even days. In such settings, the time required for MCMC sampling over GP hyperparameters may be negligible relative to the objective-evaluation time, potentially limiting the practical impact of reducing decision latency.
  2. The empirical results do not demonstrate a clear optimization-performance advantage over the existing baselines. In particular, the results in Figure 3 suggest that ELF-BO, Standard FBO, and MAP achieve broadly comparable performance on many benchmark functions. Although reducing decision latency is the primary goal of the paper, the limited performance gains over MAP make it difficult to assess the overall practical benefit of ELF-BO beyond its computational scheduling strategy.
Questions for authors
  1. In many practical black-box optimization applications, each objective evaluation can take hours or even days, making the time required for MCMC sampling over GP hyperparameters relatively negligible in comparison. In such settings, reducing decision latency may have little impact on the overall optimization time. Given that the primary contribution of ELF-BO is to reduce this latency, how significant is its practical benefit in real-world optimization scenarios where objective evaluations are extremely expensive?

  2. As shown in the experimental results, ELF-BO achieves broadly similar optimization performance to both MAP and Standard FBO on most benchmark functions. Thus, the results do not demonstrate a clear optimization advantage over either baseline. Given the additional importance-reweighting procedure and the reliance on fully Bayesian inference, what practical benefit does ELF-BO provide beyond reducing decision latency, particularly when the resulting reduction may have a limited impact on the total optimization time?

3 comments
AuthorsOpenReview ↗

Response to Reviewer SSqY

Thank you for the valuable comments and questions! Below are our answers and clarifications, which include pointers to their corresponding changes in the manuscript. We hope that the provided explanations and improvements in the paper will make you consider improving your score. We are eager to further discuss anything needed.

Extremely expensive objectives

We agree that ELF-BO's latency gains are most pronounced when TMCT_\text{MC} is a substantial fraction of TfT_f, and we have made this scope explicit in the revision (Sec. 3.1). To reinforce it, we added two real-world benchmarks (AM-ARES-Logo and AM-ARES-2DTS) with evaluation times under 10 minutes. In this regime, Standard FBO's decision latency is a more substantial fraction of each evaluation, making it more impractical. ELF-BO removes it almost entirely.

We also believe ELF-BO stays relevant when individual evaluations are much longer. Since ΔL=min⁡(TMC,Tf)−Tw\Delta L = \min(T_\text{MC}, T_f) - T_w, when TfT_f is large ELF-BO hides all of TMCT_\text{MC}. This cost is not negligible for richer surrogates: with a deep kernel, Standard FBO reaches 24 minutes per decision at 500 observations (Fig. 7), while ELF-BO stays under 3 seconds. ELF-BO therefore allows practitioners to spend more compute on inference (more samples, longer chains, richer surrogates such as deep kernels or BNNs) without paying for it at decision time if such added compute benefits their problem at hand. In other words, we see higher TfT_f as providing freedom for the practitioner to choose their surrogate model.

Improvements over baselines

This concern was also raised by Reviewer 5b13 (see "Improvements over MAP" in our response to them), and we address it jointly in our Global Comment. In short, the goal of ELF-BO is to match the optimization performance of Standard FBO at decision latency close to point-estimate BO, not to outperform Standard FBO. The revised results support this:

  • Pareto optimality: On all five real-world tasks, ELF-BO achieves area under the GAP curve on par with Standard FBO at more than 10×10\times lower latency (Matérn) and 60–130×60–130\times lower latency (deep kernel), similar to MAP's latency (Figs. 4, 5).
  • Gains over MAP: On all five real-world tasks, ELF-BO consistently outperforms MAP in area under the GAP curve, with the clearest gains in early optimization steps, where data is scarcest.
  • Asynchronous setting: ELF-BO matches or exceeds the best baseline in area under the GAP curve while reducing worker idle time by 7–9×7–9\times compared to Standard FBO (Fig. 6).
  • Synthetic functions: Here the benefit of a fully Bayesian treatment over MAP depends on the function, consistent with prior work (De Ath et al., 2021). These benchmarks were designed to test whether ELF-BO tracks Standard FBO, which it does on most functions. We analyze the exceptions through the effective sample size (Sec. 6).

References

George De Ath, Richard M Everson, and Jonathan E Fieldsend. How Bayesian Should Bayesian Optimisation Be? In Proceedings of the Genetic and Evolutionary Computation Conference Companion, 2021.

Reviewer_SSqYOpenReview ↗

Maintaining Original Score: Practical Impact Remains Limited in Typical Expensive BO Scenarios

I thank the authors for their thorough response and for making the corresponding changes in the manuscript. I understand that the core of this paper is to reduce decision latency while preserving optimization performance, and that the authors have appropriately limited the scope of application to objective functions with moderate evaluation times.

However, I still believe that the main purpose of BO is to tackle expensive and time-consuming objective functions (which is exactly why BO was originally developed), and in these cases, the number of evaluations is typically very small. Thus, while the proposed ELF-BO is useful and holds research merit for moderate evaluation times, its practical benefits will be diminished in most realistic BO problems.

Consequently, I will keep my original score.

Thank you for engaging with our response.

We agree that BO was originally developed for very expensive objectives, but it is now widely used at shorter timescales, notably in autonomous laboratories. Our benchmarks reflect this: the HPLC and AM-ARES datasets come from closed-loop autonomous campaigns, with evaluations of roughly 4 and 8 minutes. In such settings, machine time is valuable and planner downtime is a real cost.

We see ELF-BO as a method for this growing class of problems, much as multi-fidelity BO is valuable where cheap fidelities exist, even though many BO problems lack them. Within this scope, ELF-BO matches the performance of fully Bayesian BO at the decision latency of MAP. For very long evaluations, it remains useful: it tracks Standard FBO closely on almost all tasks (with exceptions analyzed in Sec. 6), and it hides the entire sampling cost, making richer surrogates such as deep kernels affordable at decision time.

Reviewer_8ek1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to use the time that BO trials are being evaluated to construct the next solution, where this time can be allocated to MCMC. In standard BO, one would wait until the trial's feedback arrives, and then start to do model fitting / acquisition optimization. The core insight seems to be that one can do MCMC on the data that already exists, and then when the new sample arrives, apply an importance sampling step to transform the approximation.

Critical strengths
  • This is a simple and elegant idea, which also would be useful in practice. I have not seen a BO implementation that makes use of this nice observation by the authors.
  • The importance sampling idea is clever (Prop 1).
  • The paper is also written in a way that is easy to read.
Critical weaknesses
  • Decision latency numbers are not provided for the continuous benchmarks, which would have been nice to see. In additional, I would like to see some plots where the x-axis is the wall time (or some variant) rather than the iteration. Given that this paper is meant to study a particular latency reduction, it feels apt to make these plots.
  • I'm curious about the approximation error that one gets from applying this procedure and more discussion on the trade-offs.
  • The latency formula of max(0, T_MC - T_f) <-- why do we do the max?
Questions for authors

See above

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8ek1

Thank you for the comments and feedback! We hope that the following points—and their corresponding changes in the new version of the paper—are able to address your concerns and convince you to raise your score. We will be glad to clarify any further discussion point if needed.

Latency and Time Results

Wall-clock in x-axis: We changed the existing real-world benchmarks plot to use wall-clock time in the x-axis and did the same for the two newly introduced real-world benchmarks. In addition, the async version of the algorithm suggested by Reviewer 5b13 is also evaluated with respect to wall-clock time.

Latency of continuous benchmarks: We included the decision latency for the synthetic continuous benchmarks (Fig. 3, bottom; Fig. 9). There is no significant difference when compared to the real-world pool-based benchmarks presented originally.

Approximation Error

We include an analysis on the approximation error introduced in ELF-BO in comparison to a reference standard FBO run on the HPLC dataset. To serve as control, we also evaluate the divergence of another standard FBO run (with different seed) to the reference run. Please refer to Fig. 7b and Fig. 7c in the updated paper.

Posterior approximation error: We include a comparison of the predictive posterior, which demonstrates that the Wasserstein-1 distance between ELF-BO and a reference FBO run is on par with FBO agreement across seeds.

Acquisition Function Values Agreement: We also compare if the acquisition function rankings over the input domain are well approximated using the Spearman correlation. Results indicate a correlation of 0.9650.965 while the second FBO seed has correlation of 0.9700.970.

Decision latency formula

The max in the decision latency exists in case the NUTS execution cannot be completely hidden behind the objective evaluation. Under this setting (TfT_f < TMCT_\text{MC}) we would have an addition of TMC−TfT_\text{MC} - T_f to the latency.

We included a clarification in the main text to address this issue (L264-L265).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their time and thoughtful feedback. We are glad that reviewers found ELF-BO to be "a simple and elegant idea, which also would be useful in practice" (Reviewer 8ek1), "conceptually simple and clearly presented" (Reviewer SSqY), and easy to adopt, since "trying it out is not a big hurdle" (Reviewer 5b13).

We have uploaded a revised version of the manuscript with all changes highlighted. The main changes are:

  1. Two additional real-world benchmarks (AM-ARES-Logo and AM-ARES-2DTS, 6D), bringing the total to five experimental datasets. This makes explicit the regime ELF-BO targets: objectives where fully Bayesian decision latency is a meaningful fraction of evaluation time (Sec. 5.2).
  2. Asynchronous batch BO (Sec. 5.2.1, App. D). Remark 2 generalizes the reweighting to any number of finished evaluations, so ELF-BO applies directly with multiple workers. ELF-BO matches or exceeds the best baseline in area under the GAP curve while reducing worker idle time by 7–9×7–9\times compared to Standard FBO.
  3. Approximation error analysis (Sec. 6, App. B.5). On HPLC, we compare ELF-BO's reweighted posterior at every step with a fresh NUTS fit on the same data, using a second, independent NUTS run as baseline. ELF-BO's Wasserstein-1 distance and acquisition rank agreement (Spearman 0.965 vs. 0.970 for the control) are on par with the noise floor.
  4. Wall-clock time, as requested, is now the x-axis for all real-world results, and decision latency is now also reported for the synthetic benchmarks (Fig. 3, bottom; Fig. 9).
  5. Clarifications: the decision latency formula (Sec. 3.1), GAP initialization (all curves now start at GAP$_0$ = 0 from the same D0\mathcal{D}_0), and a more careful Fig. 3 caption.

On improvements over baselines (raised by Reviewers SSqY and 5b13). On all five real-world tasks, ELF-BO consistently outperforms MAP in area under the GAP curve, with the largest gains in the early, most data-scarce steps (Fig. 4). This is the setting ELF-BO targets: expensive experiments with small evaluation budgets. The synthetic benchmarks, where gains over MAP depend on the function, were designed to test whether ELF-BO tracks Standard FBO, not to show improvements over MAP; we revised the Fig. 3 caption accordingly. Importantly, ELF-BO achieves this at decision latency close to MAP's: it matches Standard FBO at more than 10×10 \times lower latency with a Matérn kernel and 60–130×60–130 \times lower with a deep kernel, making it Pareto-optimal in area under the GAP curve vs. decision latency.

AuthorsOpenReview ↗

General Response After Extra Feedback

We thank the reviewers one more time for engaging in the discussion and, therefore, helping improve the quality of the paper.

Given that new round of feedback we have uploaded a second revision of the manuscript. The main changes are:

  • Same GP prior for all methods (Sec. 5, App. B.1). MAP now uses the same lengthscale prior as Standard FBO and ELF-BO, and all results have been updated. The new App. B.1 explains the choice. The prior of Hvarfner et al. (2024) is set by its mode, which suits MAP, but its heavy right tail puts the median at about 20× the mode, so a sampler would spend most of its draws on very long lengthscales. We therefore shift it so that the original mode becomes the median, and reduce the variance from 3 to 1.
  • Faster-evaluation regime (Fig. 4, top). On all five real-world tasks, we added runs with Tf=20T_f = 20 s, just above Standard FBO's sampling time (12–16 s). There, Standard FBO's latency takes up 60–80% of each evaluation, and it completes about 30 evaluations in the time ELF-BO completes 50. With the reported TfT_f (Fig. 4, bottom), Standard FBO finishes only 3–5% later.
  • Decision latency for the real-world tasks moved to the appendix (App. C, Fig. 8). Latency is measured per step, so the figure holds for both values of TfT_f.
  • More precise claims for asynchronous batch BO (Sec. 5.2.1, App. D.4). ELF-BO matches or exceeds Standard FBO in area under the GAP curve on all tasks and is the best method on four of five. On AM-ARES-2DTS, MAP is best and ELF-BO follows Standard FBO.
  • Wording. We replaced "Pareto dominance" with "Pareto optimality" and made the claims about MAP in the synthetic results more precise (Sec. 5.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.