acceptodds
Under review as a conference paper at ICLR 2027

BOING: OPTIMIZING BAYESIAN OPTIMIZATION WITH INFORMATION GEOMETRY

Abstract

We study Bayesian optimization (BO) through the lens of information geometry. We use insights from existing work that the surrogate, conventionally a GP, maps the input space to a lower-dimensional statistical manifold. The Acquisition Functions (AFs) in BO are defined on this manifold. Instead of conventional gradient-based optimization methods like L-BFGS or Adam, we propose : an iteration-free, trust-region-based method for AF optimization on a low-dimensional manifold that is parallelizable on a GPU. For a certain class of AFs in sequential BO, we show that the optimization of AF can be performed on a 1-D search space. We use the Fisher Information Matrix to construct the trust regions for . Our method achieves competitive or superior overall BO performance. Our method provides a - computational speedup for low-dimensional BO and reduces the total BO time by - %. For high-dimensional BO, we achieve a - speedup and reduce the total BO time by - %. We show speedups across multiple benchmarks and acquisition functions.

Primary area:
optimization
Keywords:
Bayesian Optimization, Constrained Optimization, Trust Region

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FyqeOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

One of the challenging parts of high-dimensional Bayesian optimization (BO) stems from the optimization acquisition function. The proposed method first identifies desirable posterior mean and variance. It then maps them back to the input space by limiting each local optimization within a certain radius. Since the transformed optimization is small enough to solve by grid search, the authors proposed to use the grid search internally, reducing the optimization runtime. The experiments showed that their method yields fast runtime, while retaining competitive performance compared to Bayesian optimization using L-BFGS-B.

Critical strengths

The idea of optimizing in the predictive distribution space is interesting

The idea itself is very interesting and it is true that the optimization in this space is much easier than that in the input space. Although I came up with several downsides, this methodology can be significantly enhanced with more careful design choice discussion. Additionally, exploiting the low-dimensional structure, embarrassing parallelization is possible.

Critical weaknesses

Design choice discussion is almost entirely skipped and not well motivated

First of all, the motivation is mixed up. In introduction, all cited work (Hvarfner et al. (2024); Xu et al. (2024); Papenmeier et al. (2025); Doumont et al. (2026)) discusses that the modeling part is bottlenecked but not the optimization part. They then claim that their propositions enhance the sample efficiency rather than the runtime. However, this paper claims its contribution as speedup rather than the sample efficiency. While this claim itself can be valid, it should be positioned more appropriately. Since the literature in q-batch setup [1,3,4] directly mentions that high-dimensional acquisition function optimization is challenging, citing these papers looks more natural to me, although q-batch setup seems to be unsupported by this work.

Another point is that the paper does not explain why they specifically choose Fisher information matrix (FIM) based constraint. The information geometric interpretation is valid, but it's unclear that it is essential to the proposed optimizer. In particular, the main KKT update and low-dimensional search hold even if we replace it with a positive definite matrix. The manuscript therefore needs to explain why FIM is preferred over simpler choices such as the identity or length scale-based metrics. This is especially important because the radius is not calibrated by the predictive distribution space quantity such as KL divergence. The radius is instead calibrated heuristically by the GP input length scales. I strongly believe that ablation study or empirical analysis is required to isolate the performance attribution whether information geometry is the source or not. The experiments nevertheless miss this obvious baseline. Furthermore, properties, which should naturally be examined in the context of information geometry, have remained largely unanalyzed. The invariance under coordinate transformations and the degenerate pullback geometry arising when Dx>DθD_x > D_\theta are concrete examples. Therefore, the authors should either rigorously analyze the geometry or soften the claims to something like a "Fisher-preconditioned parallel acquisition optimizer."

Ablation study is missing not only for the constraint geometry choice. For example, why do you use grid search for the 1D optimization routine instead of the Newton method? The derivative is very cheaply computable in the routine, and the Newton method mostly converges within a few function calls. It is, of course, fine to use the grid search, but no tradeoffs are discussed (runtime vs performance). What are the tradeoffs among grid search, the Newton method, and some other techniques such as Brent. At least, the Newton method can also be computed in parallel in the direction of the restart. Additionally, the boundary handling for the input is also not tested seriously, although this may affect the optimization robustness drastically.

The last small note is that the proposed method has an outer radius search, so it is not strictly 1D optimization, but broadly 2D. The statement around it should be fixed.

Overall, this paper uses a substantial fraction of the main body to literature context and standard geometric machinery, whereas some important details, e.g., Equations in Algorithm 1, are completely hidden in Appendix. In my opinion, significant revision is needed for this manuscript, so this manuscript is not ready for publication. Simply put, cut overly detailed machinery and add more mechanistic analysis or ablation study. The importance analysis may also help.

Experiment design does not isolate primary performance attributions and some results are overclaimed

My primary question is Why does your method perform well, even though the proposed method uses approximation under a strong assumption?. For example, what happens when both L-BFGS-B and the proposed method use the same number of restarts? If L-BFGS-B outperforms the proposed method with the same number of restarts, this explains that we need to use a sufficiently large number of restarts. If not, it implies the first-order approximation somehow contributes to the performance. If you get results that support the former, this justifies your speedup statement significantly. If the latter is supported, this probably needs some more theoretical explanations. For example, let's permit the first-order approximation assumption around the neighbor. It is true that we can get the inverse mapping under the certain conditions. However, the inverse mapping is practically (and mostly) not injective, because D>2D > 2. Then what is the solution quality or what is the theoretical upper bound of the optimum solution error? Does this error decrease as the number of restarts increases? Unfortunately, the current manuscript does not clarify this point. As mentioned earlier, ablation study on some control parameters is also missing.

As a sanity check on the runtime numbers, I benchmarked Optuna GPSampler, which performs the exact GP-based BO with batched acquisition evaluation [2], on MacBook M4. I optimized the Rastrigin function using LogEI with the same dimensionalities as in Figure 8 below (see Appendix. Benchmarking log). The worst runtimes for each setup among 50,100,150,200,250 trials are shown in the table below. I obtained the runtimes substantially below the BOING runtimes reported. While this is not an apples-to-apples comparison because the hardware and implementation may differ, it raises concern that the very large reported speedup factors can depend strongly on the particular baseline implementation. Since computational speed is a central contribution, hardware details and comparisons against Optuna GPSampler are needed.

12D60D102D
31ms55ms62ms

Table. The worst runtimes of Optuna GPSampler for each setup among 50,100,150,200,250 trials.

References

  • [1] Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective Bayesian optimization. In Advances in Neural Information Processing Systems, 2020.
  • [2] Irie, Kaichi, Shuhei Watanabe, and Masaki Onishi. Batch Acquisition Function Evaluations and Decouple Optimizer Updates for Faster Bayesian Optimization. arXiv preprint arXiv:2511.13625 (2025).
  • [3] Takashi Wada and Hideitsu Hino. Bayesian optimization for multi-objective optimization and multipoint search. arXiv preprint arXiv:1905.02370, 2019
  • [4] James Wilson, Frank Hutter, and Marc Deisenroth. Maximizing acquisition functions for Bayesian optimization. In Advances in Neural Information Processing Systems, 2018.
Questions for authors

Questions

  • See Critical Weaknesses.

Suggestions

  • See Critical Weaknesses
  • Since the paper discusses the speedup of acquisition function optimization by parallelization, the following paper is relevant and should be cited.
    • Irie, Kaichi, Shuhei Watanabe, and Masaki Onishi. Batch Acquisition Function Evaluations and Decouple Optimizer Updates for Faster Bayesian Optimization. arXiv preprint arXiv:2511.13625 (2025).
Reviewer_6R7TOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

BOING is an iteration-free constrained local optimization approach for acquisition functions in Bayesian optimization that uses Fisher information approaches. By considering a focused set of acquisition functions, the authors are able to have a significant, in wall-clock time, advantage over existing acquisition function optimizers, while retaining or beating them in BO performance.

Critical strengths
  • As a BO approach, I think BOING does well. Figures 4/5/6 speak for themselves.
  • The approach is very amenable to GPU paralelization
  • For large n/d regimes we desperately need more focus to make the global AF optimization problem fast to run, and this paper helps push that frontier.
  • BO experimental evaluation over a wide range of problems.
  • There is a clear geometric interpretation of their approach.
  • Drop-in optimizer to several popular AF choices.
Critical weaknesses

In strengths I wrote "as a BO approach". However, BOING is posited as an AF optimizer, and unfortunately the evaluations does not test it as one nor give sufficient justification why a single first-order step ought to beat or match the solutions found by Adam/L-BFGS. The reader is left thinking where this uplift/parity comes from. The core method itself appears to be a fast-in-wall-clock-time approach for local solutions to an acquisition function. To support the novelty/significance of the paper I think a deeper understanding is needed. For example, there are several existing works that have shown that such random, localized perturbations can perform well for BO, with two of them showing RAASP-style TS is enough [1-4]. It’s hard to tell whether the uplift you are getting is a result of your RAASP starting locations itself being sufficient as an optimizer, an indication that better/worse AF maximizer does not translate to better BO performance, the helpfulness of the geometry etc.

In terms of motivation, the paper offers a statistical manifold framing (GP maps inputs to distributions, AF lies only on the parameters such as μ\mu,Σ\Sigma) and the paper proposes information geometry as a natural framework. On that basis alone, the paper has motivated optimizing the AF through θ\theta, but I may have failed to see why this demands the Fisher metric. If the AF is monotone in μ\mu and Σ\Sigma any positive-definite quadratic constraint in θ\theta space places the optimizer on one quarter ellipse and can also be searched in one dimension. Prop 4.2 bounds the magnitude of the AF gradient but the authors have not shown this implies that a Fisher-shaped TR produces better local maximizers. The KL interpretation is meant to bound the change in the predictive distribution, but does not match what the TR does, which controls the error of the linear map. κ\kappa is also chosen by lengthscales, not information. Thus BOING's use of Fisher information is unsubstantiated.

Overall I feel this paper is lacking several key ablations and theoretical justifications that, to address, would require substantial changes to the paper.

My other concerns/observations:

  • The advantages lie primarily within Assumption 4.2 (and the AF depending on the input only through the parameter map). There are only a handful of acquisition functions that lie in this, although it contains several popular ones. Nevertheless, one needs to give up the generality that L-BFGS provides for super-fast optimization this method proposes.
  • The 1D search claim is a bit overstated. The authors instead have shown that, under a linearized posterior map, each Fisher-constrained subproblem around some point reduces to a 1D search. Linearization allows a cheap one-step map back, but thus is only first-order accurate.
  • There is no extension to the batch setting. One could greedily optimize a batch point. How does BOING do here?
  • The individual ingredients are not new; the contribution appears to lie in integrating them into a single-step batched candidate generator.
  • In the abstract you probably mean a x-times reduction instead of a >100% reduction. "30 - 20x computational speedup" is reversed
  • In some places you use \citet instead of \citep, such as line 121, 127.
  • Proposition 4.3 seems to be obvious under Assumption 4.3, I'm not sure what the significance is.

[1] Gramacy, Robert B., Annie Sauer, and Nathan Wycoff. "Triangulation candidates for Bayesian optimization." Advances in Neural Information Processing Systems 35 (2022): 35933-35945.

[2] Wycoff, Nathan, et al. "Voronoi candidates for bayesian optimization." arXiv preprint arXiv:2402.04922 (2024).

[3] Rashidi, Bahador, Kerrick Johnstonbaugh, and Chao Gao. "Cylindrical Thompson sampling for high-dimensional Bayesian optimization." International conference on artificial intelligence and statistics. PMLR, 2024.

[4] Fan, Donney, and Geoff Pleiss. "Adaptive Candidate Point Thompson Sampling for High-Dimensional Bayesian Optimization." arXiv preprint arXiv:2604.08891 (2026).

Questions for authors

I had these questions as I read the paper:

  1. How does BOING do as an AF optimizer? Does it find better optima than L-BFGS?
  2. It is easy to cap the number of iterates of Adam/L-BFGS. Given the same wall-clock time budget, how does BOING compare in both BO performance and AF optimization performance? What if we let them do a single gradient step?
  3. How much does BOING’s constrained optimization actually help vs RAASP? e.g. only evaluate at the starting locations.
  4. To isolate the impact of the Fisher information geometry, I think you could just take unit vectors in the direction of ∇xμ\nabla_x \mu and ∇xΣ\nabla_x \Sigma and sweep over a Euclidean ellipse around the starting locations.
  5. How much better could BOING do if you allow it the same wall-clock time budget as L-BFGS/Adam?
  6. What if you use BOING as an initializer to Adam/L-BFGS?
  7. In the appendix it is mentioned that you use clipping to bound points that leave the search cube. How much of your results of Figures 4/5 are a result that good solutions happen to lie on the boundary?
  8. What happens near the incumbent, where the gradient of the mean is close to zero and the gradient of the covariance may be colinear?
4 comments

Thank you for the detailed review. The common response addresses AF improvement beyond starts, one-update controls, Fisher versus Identity, local monotonicity, the search-cost comparisons, and the abstract speedup correction. We address the additional concerns below. References are to the revised manuscript.

Candidate-generation literature. Triangulation and Voronoi methods are now discussed at lines 88-92, and Cylindrical Thompson Sampling is cited in the starting-location context at lines 81-86. However, BOING searches the nonlinear local AF problem in predictive space, maps the selected update back to input space, and ranks the candidates using the actual posterior (Eqs. (5)-(7), Algorithm 1).

AF gains - The requested AF gains are in Figures 12-13, and BO performance is in Figures 4-6 and 17-19.

The boundary result, radius and generality. The boundary result is now Proposition 4.1. Its purpose is to make the constraint active under the stated local assumptions and support the direction search in Eqs. (11)-(12) (lines 220-225 and 246-250). Figure 3 and lines 282-287 explain the first-order limitation addressed in the common response.

Extension to quadratic constraints. We agree that we can use any quadratic constraint in BOING. We now address the motivation for BOING as a trust-region BO method that solves AF maximization in the trust region in manifold space rather than in input space. It takes an optimal step in the manifold space and uses the pullbacks to the input space. Since the manifold space is lower dimensional, we are able to achieve significant computational gains.

Additional optimization time and initialization. Appendix E.2 provides the one-update comparison, and Appendix E.4 tests extra search effort through Newton's method. Table 9 and Figure 20 show substantially more computation without consistent BO improvement. BOING candidates could initialize either optimizer, but we have not tested whether the additional AF or BO improvement justifies the combined computation time.

Collinear gradients. We assume the full row rank only at the centers of the TR. This helps in having a non-singular FIM. However, since we cannot guarantee full rank, we also use numerical stabilization using a small jitter for the inverse. This is also addressed in the revised manuscript now.

Dear Reviewer 6R7T,

Thank you very much for your help with reviewing for ICLR'27!

The authors have now responded to your reviews. As the discussion period is approaching the end, I'm wondering if you could have a look at the authors' responses and see whether they have addressed your concerns. And could you also leave a comment to let the authors know your thoughts? If any concerns remain, please explain which ones have not been addressed and why.

Warm regards, Your AC

Thank you for the response and additional ablations.

I think Appendix E muddies the results a bit. Here we see that Identity is about the same as Fisher. Figure 14 is also odd to me it seems, like in 10D Ackley, all methods find a point that is lower than the initial point? It's not surprising to me this would occur for Adam, but even in one step L-BFGS the single line search should satisfy the Wolfe condition. The curves here seem like we are testing different GP posteriors, while all methods should share one and optimize the same AF.

I may have missed it, but I do not see a follow up to my comment that the radius constraint is scaled using GP lengthscales. I do not see a justification for this, especially since we search over multiple radii. With Identity being about the same as Fisher, I remain unconvinced that the manuscript sufficiently justifies the benefits of predictive-space optimization and Fisher information here (although it doesn't seem to hurt).

Newton vs grid seems like it could go either way, but alas E.4 only measures BO performance.

Overall despite the mixed outcome the revision indeed establishes that BOING gives appreciable gain over the starting locations, even matching/beating the gradient based optimizers in certain cases.

We thank the reviewer for following up.

Comparison on the same GP. We also ran the comparison you suggested on 45 saved GP fits for Rastrigin in 12, 60, and 102 dimensions, using the same 32 starting locations and the same AF for every method on each fit. The median LogEI gain over the best start was 2.11 for one L-BFGS update, 2.41 for BOING-Identity, and 2.2 for BOING-Fisher. This separate experiment suggests that BOING indeed optimizes the AF better than L-BFGS for the same GP posterior.

Ackley10 and the starting locations. Kim, Liu and Chen (2025) showed that selecting the best acquisition value from a random grid can be sufficient for effective BO. We test whether selecting from the uniform and RAASP starting locations is also sufficient in higher dimensions. On Ackley10, one possible explanation is the bowl-shaped structure of the objective. When we maximize the negative Ackley function, even a GP that captures only the broad shape may predict a smooth, approximately Gaussian peak near the center. RAASP perturbations around good observations may therefore sample this region without further AF optimization. On Ackley10, we observe a similar decline in the selected AF values for LogEI, GIBBON and MES (Figures 14–16). This pattern does not hold consistently across the other benchmarks.

Fisher or Identity. We propose BOING as a generic method that solves a local trust-region acquisition problem on the predictive manifold and maps the selected candidate back to input space. Both constraints use this search. Identity measures the length of the input step, while Fisher measures local change in the predictive distribution.

For analytic LogEI, Identity often performs comparably to Fisher. Fisher's benefit in final BO performance is clearest for higher-dimensional GIBBON, while either constraint can do better for MES (Table 8, Figures 17–19). We use Fisher as the default only because it is local geometry-informed.

Both variants implement the same simplex search and multiple radii in parallel and improve the starting locations with one local search (Algorithm 1, Figure 12).

Newton versus grid. Figure 20 in E.4 also reports AF optimization time. To address AF quality directly, we checked the recorded Ackley10/LogEI runs with the Fisher constraint. Newton gives a mean AF gain of 0.70 over the current starts versus 0.59 for grid search, but takes 2.07 s per BO iteration versus 0.042 s for grid search on average. These gains are measured along each method's online trajectory. We use grid search because it keeps the inner optimization inexpensive, and Newton gives no consistent improvement in final BO performance. For a given start and radius, since both methods optimize the same AF over the same one-dimensional simplex, we do not observe any general difference/trend in AF value gain using Newton's method.

Why scale the radii by the GP lengthscales? Hvarfner et al. (2024) showed that the choice of lengthscale prior matters for high-dimensional BO, and Papenmeier et al. (2025) showed the benefit of local candidate generation with RAASP. Kiroriwal et al. (2026, Prop. 4.2) show that the squared AF gradient norm is bounded by the trace of the pullback Fisher matrix times an acquisition-dependent factor.

Away from the observations, as the GP posterior flattens towards its prior, this trace tends to zero. For bounded acquisition sensitivity, the AF gradient then vanishes. To avoid the constraint being in a region where the posterior reverts to the prior, we scale the radius grid by the mean fitted lengthscale. Similar to TuRBO, this is a heuristic, motivated by using GP lengthscales to set trust-region side lengths.

Reviewer_czKROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose a parallelizable algorithm to optimize the acquisition function not in the input space but rather in the predictive space for high-dimensional BO. In this predictive space, the authors use the Fisher information matrix to define a trust region that constrains the optimization. Under some assumptions, the authors show that the optimum of this constrained optimization problem lies on the boundary of this trust region which allowes them to reduce the problem to a simplex that they solve through grid search. The authors show on various high-dimensional BO problems significant speed ups in terms of total runtime as well as competitive and in parts improved performance compared to other baselines.

Critical strengths

S1) High-dimensional BO is a hot topic in research currently so the paper is very timely and the research direction is interesting and relevant.

S2) I think the paper is in general well written and I genuinely enjoyed reading it.

Critical weaknesses

W1) A lot of the story around the paper is framed in the direction of optimizing the AF but the algorithm and also the underlying assumptions imply that the main goal is actually not solving the original optimization problem. In fact, with assumptions 4.2 and 4.3, none of the candidate points can actually be a stationary point of alpha (Correct me if I am wrong here please!). Therefore, a lot of the additional gain especially in the high-dimensional setting could be just that the optimization is not actually solved to a stationary point but rather that a small local perturbation resulting from the constrained optimization problem just yield more exploitation and with this in the end better performance. As this, a lot if the performance gains could then be attributed to how to set up the initial set of candidates and the characterisation of the final algorithm could be more like a AF-ranked local candidate generation than to AF optimization.

W2) The core contribution of using the Fisher information is in the current empirical comparison not clear. This also touches on the point above. LBFGS and Adam optimize the AF to a stationary point of alpha. A fair baseline which is currently not included would be to just do a one step solve with a proper line search and/or with second order approximations and adding the result to the candidate set, using the exact same mechanism for the starting points. Note that such an algorithm could also easily be evaluated in parallel on a GPU and I would expect similar gains in total runtime. Or maybe even closer to the proposed setting would be to do a full natural gradient step as x_next=x_r + g_r^\dagger \grad_x \alpha(x_r) (maybe also with the same sweep over step length as you have?) which I believe would be the same as the full proposed algorithm but without the nonlinear search over omega. Please correct me if I misunderstood something here. Dont get me wrong, I think its elegant and nice to work in this induced manifold but in the current form I believe the paper does not provide sufficient evidence that this is necessary or that it explains the improved performance.

(Minor) I believe a stream of RW on local BO is missing namely the „purely“ local approaches such as GiBO [R1] (and follow up like [R2] or [R3] which also tries to use second order information for local exploitation).


[R1] Müller, Sarah, Alexander von Rohr, and Sebastian Trimpe. "Local policy search with Bayesian optimization." Advances in Neural Information Processing Systems 34 (2021): 20708-20720.

[R2] Nguyen, Quan, et al. "Local Bayesian optimization via maximizing probability of descent." Advances in neural information processing systems 35 (2022): 13190-13202.

[R3] Brunzema, Paul, and Sebastian Trimpe. "BayeSQP: Bayesian Optimization through Sequential Quadratic Programming." Advances in Neural Information Processing Systems 38 (2026): 59821-59848.

Questions for authors

Q1) In the abstract, you state a „total reduction by 300%-500%“ which is obviously not possible. Can you please state either what the actual reduction is or reframe it by how much faster you were? Also, I believe it would be good to already in abstract state the baseline to which you measure this speed-up against and/or reframe it as „up to …“

Q2) Can you clarify again which methods get what starting points? From the main text, it seems like the starting points across methods are not always the same but just their distribution but then the methods subsample from that distribution. Am I understanding this correctly? An additional ablation would be to give all optimizers the same number of starting points.

Q3) Could you rerun some of the examples and there also compute and store how large the gradient of alpha at the chosen location was for the different algorithms across iterations? I think that would further give some intuition on the above-mentioned points.

Q4) Does the proposed algorithm also transfer to low-dimensional settings? How does it compare on standard benchmark such as Hartmann6D or Ackley2D/10D/20D? Here the above mentioned local exploitation could hurt and demonstrating this could be valuable for practitioners.s

Q5) Could you add a baseline where you just use the starting points as the candidate set and then rank them based on the AF? This would again then cleanly ablate how much the additional optimization here helps or if the gains are largely based on specifying a good starting set (which seems plausible looking at the sophisticated heuristics in Table 3.).

4 comments

Thank you for the careful review and the suggestions for separating the effects of starting locations, the local update, and the constraint. The common response addresses these ablations, the AF-value comparisons, and the runtime wording in Q1. We address the additional concerns below. References are to the revised manuscript.

W1: Stationarity and the rank assumption. Assumption 4.2 requires full row rank only at the starting locations, not at all input locations or at the resulting candidates (lines 220-225). BOING solves a constrained local AF problem. Its optimum satisfies the boundary KKT condition in Eq. (9), rather than requiring a zero unconstrained AF gradient. The low-dimensional search is followed by a first-order map and actual-posterior candidate selection in Eqs. (5)-(7). The AF gains beyond starts are reported in Figure 12; they support the contribution of the local search without claiming convergence to an unconstrained stationary point.

Q2: Starting-point budget. In the main experiments, Adam, FSAM, and L-BFGS use ten restarts selected from 512 candidates drawn from the same starting-location distribution as BOING-H (lines 353-364; Appendix D.1, Table 2). We conduct ablations for the methods to have the same budget. We limit L-BFGS, Adam, and NGD to one step, and they use the same starting locations as BOING. (Appendix E.2).

Q3-Q5: Diagnostics and lower-dimensional behavior. The revised manuscript assesses the local update through AF value gain and BO performance (Figures 12-19). Hartmann6 and Ackley10 are included in Figures 14-19 and Tables 8-9. These comparisons show the lower-dimensional behavior as well as the effect of the constraint. The starts-only comparison requested in Q5 is addressed in the common response and Figure 12.

Suggested local BO references. GiBO, probability-of-descent BO, and BayeSQP are now discussed at lines 88-92. These methods guide evaluations using uncertain objective derivatives. BOING searches the chosen AF locally in predictive space.

Dear Reviewer czKR,

Thank you very much for your help with reviewing for ICLR'27!

The authors have now responded to your reviews. As the discussion period is approaching the end, I'm wondering if you could have a look at the authors' responses and see whether they have addressed your concerns. And could you also leave a comment to let the authors know your thoughts? If any concerns remain, please explain which ones have not been addressed and why.

Warm regards, Your AC

I thank the authors for the revision and the additional ablations. I think the new one-step and starts-only baselines have improved the paper. However, the new results do not resolve my main concerns. With matched starting points, the one-step baselines and even the starts-only baseline match or outperform BOING on several LogEI problems, and the identity constraint is often on par with Fisher, if I understand table 8 correctly, so the role of the information geometry remains unclear. The one-step baselines are also not fully specified as far as I can tell. I could not find a step size/learning rate or line search for Adam-1, L-BFGS-1, or NGD-1. Could you provide additional information on the design of these baselines? BOING selects the best of several radii per start, which makes the “same budget” comparison hard to interpret. Also note that the abstract still states “better AF optimization”. No need for an additional upload for this from my side so feel free to answer directly in the feed.

We thank the reviewer for following up.

The focus of the manuscript is not that the Fisher constraint is the optimal constraint. We show that BOING is a novel, general method that solves a local trust-region acquisition problem on the predictive manifold and maps the selected candidate back to the input space. This places BOING at the intersection of BO and AF optimization, and we show almost 2 orders of magnitude speedup in AF optimization in the inner loop in comparison to conventional methods like Adam and L-BFGS. This is why we still use the term "AF optimization".

Fisher or Identity. We agree with your earlier remark that an input Identity constraint can also be used. The same predictive-space search applies to either quadratic constraint (Appendix C.1). For analytic LogEI, where local monotonicity can be shown, Identity often performs comparably to Fisher and gives a higher mean final objective on several problems.

Experiments show that the Fisher constraint performs better on AFs known to be globally non-monotonic, and that the anisotropy of the AF needs to be considered for better optimization. As shown in MES and GIBBON. On the other hand, the Euclidean quadratic constraint (translation invariant), like the identity matrix, works roughly identically to the Fisher constraint when the AF can be proven to have local monotonicity, like LogEI. (Table 8, Figures 17-19) Kiroriwal et al. (2026, Table 2) report a similar pattern in input space, where TuRBO-REI sometimes performs better than FITR-REI.

Same Budget Ablation. We matched the number of starts and outer updates. At each BO iteration, each method uses 32 starts from the same seeded uniform/RAASP policy and selects one input for objective evaluation. Once the BO trajectories diverge, the locations around which the starts are drawn can differ (Appendix E.2).

  • Adam-1 takes one Adam update step per start, with learning rate 0.025 and no line search.
  • L-BFGS-1 uses BoTorch's bounded L-BFGS-B solver with the outer iteration limit set to one. Its built-in line search chooses the step length and can evaluate the AF and its gradient several times within that iteration.
  • NGD-1 takes one normalized natural-gradient direction per start. Along it, we test BOING's eight radius multipliers (0.05, 0.1, 0.2, 0.3, 0.4, 0.5, 0.8, 1), scaled by the mean fitted lengthscale. We keep the best AF value among those steps and the original start. There is no backtracking.
  • BOING uses the same starting locations and the eight radii per TR center.

We do not run full iteration-based L-BFGS, Adam, and NGD with as many starting locations, as it would be computationally expensive and would not be a fair comparison to BOING. Because in the time that the full iterative methods would take, BOING could search many more TRs and provide a much better optimization. Instead, we limit the number of update steps for the gradient-based methods and increase the number of starting locations to match BOING. The results show that BOING performs significantly better than one-step L-BFGS, Adam, and NGD on serious dimensional problems. All methods also use similar memory to have a fair comparison.

AF optimization. For the abstract's claim of "better AF optimization", Figure 13 gives the most direct evidence. It measures the AF gain over each run's best start under the posterior at that iteration. BOING gives better AF gains with both constraints (Figure 12 and Figure 13). Its advantage over the one-step baselines is clearest for GIBBON and MES, while LogEI is mixed.

This also reveals an interesting aspect about the comparison between AFs. Conventional AF optimization using L-BFGS for GIBBON or MES could show that LogEI is better than both. However, the results show that BOING consistently finds better AF values for MES and GIBBON. Moreover, in ANT(888D), SVM(388D), and MOPTA08(124D), GIBBON achieves almost the same BO performance as LogEI using only BOING. L-BFGS also proves to be a heuristic for complex AFs like MES and GIBBON. We believe that this novel method can open new perspectives on BO, and that we need more research in AF optimization rather than new AFs.

Comments on the paper

7 comments
Reviewer_FyqeOpenReview ↗

Could you update the manuscript directly?

Thank you for the comments, but would it be possible to prioritize some information? Three full response (+ 6 responses with numbers) in a row are excessive for a productive review discussion. Additionally, please update the manuscript first. As my concern claims that most information regarding design decisions are entirely skipped, postponing the updates until the camera ready version won't resolve the weaknesses.

AuthorsOpenReview ↗

Common response

We thank the reviewers for the detailed reviews and suggestions to improve the paper and the ablations. We address the common concerns below and the reviewer-specific points in separate replies. All references are to the revised manuscript.

Local optimization in predictive space. BOING performs the AF optimization steps on the low-dimensional statistical manifold and only maps the optimization result back into the high-dimensional input space. We use the local information geometry of the manifold to shape the trust region. The evaluation of the AF on the manifold is computationally very cheap. Further, our method is easily parallelizable on a GPU. This leads to - sometimes two orders of magnitude - improvement in wall-clock time with comparable or improved performance of the solution. The formulation and candidate selection are given in Eqs. (2)-(7) and Algorithm 1 (lines 180-198 and 465-466).

Constraint choices. The identity matrix gives a translation-invariant Euclidean input constraint, while the Fisher constraint accounts for direction-dependent changes in the predictive distribution. For analytic LogEI, the Euclidean constraint often performs comparably to Fisher. Fisher's benefit is clearest for higher-dimensional GIBBON and MES (lines 422-426; Appendix E.3, lines 1439-1447; Table 8 and Figures 17-19), where the TR needs to be locally geometry-informed. Appendix C.1 explains the generic formulation in detail.

AF value gain. We observe acquisition values approaching zero for GIBBON and MES, particularly for Adam-1, L-BFGS-1 and NGD-1 in some simulation environments. However, the decrease is the smallest in BOING. Together with the AF gains, this suggests that BOING can explore and optimize the acquisition function better in these settings under the same one-update budget (Figures 14-16; lines 454-457).

The focus of the manuscript is not that the Fisher constraint is the optimal constraint. It shows that BOING is a novel, generic method that solves a local trust-region acquisition problem on the predictive manifold and maps the selected candidate back to input space. This places BOING at an intersection of a BO method and AF optimization.

Improvement over starting locations. The improvement in BO performance does not stem from the starting locations; all methods in our benchmark draw the starting locations from the same RAASP policy. Figure 12 shows positive mean AF gain after one BOING step. The comparison uses the same 32-start uniform/RAASP policy, 20 initial observations and 200 adaptive evaluations (Discussion, lines 370-375; Appendix E.1, lines 1298-1305).

Comparison to iterative methods with the same budget. Figure 13 compares BOING with Adam-1, L-BFGS-1 and NGD-1 using 32 starts and one update per method. The benefit of the predictive-space search is clearest for GIBBON and MES, while the LogEI results are mixed. Figure 6 and Figures 17-19 give the BO performance (lines 427-433; Appendix E.2).

Local monotonicity. Local monotonicity benefits BOING by reducing the direction search at each radius to the one-dimensional simplex under either constraint. Although local monotonicity cannot always be guaranteed for GIBBON and MES, in practice BOING outperforms L-BFGS and Adam in both BO performance and AF gain. This is a novel finding, and future work should elucidate this. (lines 282-287 and 435-439; Appendix E.3, lines 1443-1447).

Direction and radius search. The one-dimensional search applies to a given starting location and radius. We sweep over many radii, so the complete sweep at each start is two-dimensional (Eqs. (11)-(12), Figure 3). Later selections favor the mean-gradient direction, suggesting a shift towards exploitation, while smaller radii continue to contribute (Figures 24-25; lines 449-453). The map back is only first-order accurate (lines 282-287).

Grid search versus Newton's method. Newton's method adds substantial computation time without consistent improvement in BO output. We therefore use grid search as the default (lines 441-445; Appendix E.4, Table 9 and Figure 20).

Boundary handling. Selected candidates frequently reach the input bounds, especially in higher dimensions. Figure 21 reports how often this occurs (Appendix E.5). This measures clipping frequency without isolating its contribution to BO performance.

Time and memory. BOING reuses the posterior Jacobian across directions and radii and provides multi-fold AF-optimization speedups over the conventional iterative baselines. Peak memory usage is of the same order (lines 458-461; Table 3, Figures 7, 9 and 11, and Tables 6-7). Appendix D.3, lines 1058-1068, gives the measurement definitions.

Wording correction The incorrect percentage reduction and reversed speedup range have been removed; the abstract names L-BFGS (lines 12-22). The current study covers scalar, sequential BO; parallel BO remains future work (lines 465-471).

AuthorsOpenReview ↗

Updated manuscript

Thank you for the detailed review and suggestions. We have updated the manuscript with the ablations and corrected the motivation.

The common response addresses the starting-location and one-update ablations, constraint choice, grid versus Newton search, boundary handling, and runtime findings. We address the additional concerns below. References are to the revised manuscript.

Motivation and presentation. We now use AF optimization as the inner problem and propose BOING as a BO method optimized in the manifold space. Wilson et al. is cited in the Introduction (p. 1). The predictive-space formulation, map back, and final AF-ranking step are now in Eqs. (5)-(7), with the direction search in Eqs. (11)-(12) and the operative steps in Algorithm 1. BOING reuses the posterior Jacobian across directions and radii, as stated in the Discussion at lines 458-461. The contribution concerns this local search, not batching acquisition evaluations alone.

What if input dimension Dx>2 BOING pushes the trust region in the input space onto the manifold. Since the transformation is local and first-order due to the Jacobian map, we do not assume that the map back is globally bijective. We do not claim invariance of the complete numerical algorithm under arbitrary coordinate transformations.

The first-order reconstruction is stated at lines 282-287 and in Appendix D.2, lines 1051-1054. Figure 3 shows that a larger predicted AF value need not give a larger AF value at the mapped input. We do not provide a finite-step approximation-error bound or a bound that decreases with the number of restarts. The outer-radius search is retained; the one-dimensional claim applies only to the direction search for each TR with radius κ\kappa.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.