- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
One of the challenging parts of high-dimensional Bayesian optimization (BO) stems from the optimization acquisition function. The proposed method first identifies desirable posterior mean and variance. It then maps them back to the input space by limiting each local optimization within a certain radius. Since the transformed optimization is small enough to solve by grid search, the authors proposed to use the grid search internally, reducing the optimization runtime. The experiments showed that their method yields fast runtime, while retaining competitive performance compared to Bayesian optimization using L-BFGS-B.
Critical strengths
The idea of optimizing in the predictive distribution space is interesting
The idea itself is very interesting and it is true that the optimization in this space is much easier than that in the input space. Although I came up with several downsides, this methodology can be significantly enhanced with more careful design choice discussion. Additionally, exploiting the low-dimensional structure, embarrassing parallelization is possible.
Critical weaknesses
Design choice discussion is almost entirely skipped and not well motivated
First of all, the motivation is mixed up. In introduction, all cited work (Hvarfner et al. (2024); Xu et al. (2024); Papenmeier et al. (2025); Doumont et al. (2026)) discusses that the modeling part is bottlenecked but not the optimization part. They then claim that their propositions enhance the sample efficiency rather than the runtime. However, this paper claims its contribution as speedup rather than the sample efficiency. While this claim itself can be valid, it should be positioned more appropriately. Since the literature in q-batch setup [1,3,4] directly mentions that high-dimensional acquisition function optimization is challenging, citing these papers looks more natural to me, although q-batch setup seems to be unsupported by this work.
Another point is that the paper does not explain why they specifically choose Fisher information matrix (FIM) based constraint. The information geometric interpretation is valid, but it's unclear that it is essential to the proposed optimizer. In particular, the main KKT update and low-dimensional search hold even if we replace it with a positive definite matrix. The manuscript therefore needs to explain why FIM is preferred over simpler choices such as the identity or length scale-based metrics. This is especially important because the radius is not calibrated by the predictive distribution space quantity such as KL divergence. The radius is instead calibrated heuristically by the GP input length scales. I strongly believe that ablation study or empirical analysis is required to isolate the performance attribution whether information geometry is the source or not. The experiments nevertheless miss this obvious baseline. Furthermore, properties, which should naturally be examined in the context of information geometry, have remained largely unanalyzed. The invariance under coordinate transformations and the degenerate pullback geometry arising when are concrete examples. Therefore, the authors should either rigorously analyze the geometry or soften the claims to something like a "Fisher-preconditioned parallel acquisition optimizer."
Ablation study is missing not only for the constraint geometry choice. For example, why do you use grid search for the 1D optimization routine instead of the Newton method? The derivative is very cheaply computable in the routine, and the Newton method mostly converges within a few function calls. It is, of course, fine to use the grid search, but no tradeoffs are discussed (runtime vs performance). What are the tradeoffs among grid search, the Newton method, and some other techniques such as Brent. At least, the Newton method can also be computed in parallel in the direction of the restart. Additionally, the boundary handling for the input is also not tested seriously, although this may affect the optimization robustness drastically.
The last small note is that the proposed method has an outer radius search, so it is not strictly 1D optimization, but broadly 2D. The statement around it should be fixed.
Overall, this paper uses a substantial fraction of the main body to literature context and standard geometric machinery, whereas some important details, e.g., Equations in Algorithm 1, are completely hidden in Appendix. In my opinion, significant revision is needed for this manuscript, so this manuscript is not ready for publication. Simply put, cut overly detailed machinery and add more mechanistic analysis or ablation study. The importance analysis may also help.
Experiment design does not isolate primary performance attributions and some results are overclaimed
My primary question is Why does your method perform well, even though the proposed method uses approximation under a strong assumption?.
For example, what happens when both L-BFGS-B and the proposed method use the same number of restarts?
If L-BFGS-B outperforms the proposed method with the same number of restarts, this explains that we need to use a sufficiently large number of restarts.
If not, it implies the first-order approximation somehow contributes to the performance.
If you get results that support the former, this justifies your speedup statement significantly.
If the latter is supported, this probably needs some more theoretical explanations.
For example, let's permit the first-order approximation assumption around the neighbor.
It is true that we can get the inverse mapping under the certain conditions.
However, the inverse mapping is practically (and mostly) not injective, because .
Then what is the solution quality or what is the theoretical upper bound of the optimum solution error?
Does this error decrease as the number of restarts increases?
Unfortunately, the current manuscript does not clarify this point.
As mentioned earlier, ablation study on some control parameters is also missing.
As a sanity check on the runtime numbers, I benchmarked Optuna GPSampler, which performs the exact GP-based BO with batched acquisition evaluation [2], on MacBook M4.
I optimized the Rastrigin function using LogEI with the same dimensionalities as in Figure 8 below (see Appendix. Benchmarking log).
The worst runtimes for each setup among 50,100,150,200,250 trials are shown in the table below.
I obtained the runtimes substantially below the BOING runtimes reported.
While this is not an apples-to-apples comparison because the hardware and implementation may differ, it raises concern that the very large reported speedup factors can depend strongly on the particular baseline implementation.
Since computational speed is a central contribution, hardware details and comparisons against Optuna GPSampler are needed.
| 12D | 60D | 102D |
|---|---|---|
| 31ms | 55ms | 62ms |
Table. The worst runtimes of Optuna GPSampler for each setup among 50,100,150,200,250 trials.
References
- [1] Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective Bayesian optimization. In Advances in Neural Information Processing Systems, 2020.
- [2] Irie, Kaichi, Shuhei Watanabe, and Masaki Onishi. Batch Acquisition Function Evaluations and Decouple Optimizer Updates for Faster Bayesian Optimization. arXiv preprint arXiv:2511.13625 (2025).
- [3] Takashi Wada and Hideitsu Hino. Bayesian optimization for multi-objective optimization and multipoint search. arXiv preprint arXiv:1905.02370, 2019
- [4] James Wilson, Frank Hutter, and Marc Deisenroth. Maximizing acquisition functions for Bayesian optimization. In Advances in Neural Information Processing Systems, 2018.
Questions for authors
Questions
- See
Critical Weaknesses.
Suggestions
- See
Critical Weaknesses - Since the paper discusses the speedup of acquisition function optimization by parallelization, the following paper is relevant and should be cited.
- Irie, Kaichi, Shuhei Watanabe, and Masaki Onishi. Batch Acquisition Function Evaluations and Decouple Optimizer Updates for Faster Bayesian Optimization. arXiv preprint arXiv:2511.13625 (2025).
Thank you for the detailed review. The common response addresses AF improvement beyond starts, one-update controls, Fisher versus Identity, local monotonicity, the search-cost comparisons, and the abstract speedup correction. We address the additional concerns below. References are to the revised manuscript.
Candidate-generation literature. Triangulation and Voronoi methods are now discussed at lines 88-92, and Cylindrical Thompson Sampling is cited in the starting-location context at lines 81-86. However, BOING searches the nonlinear local AF problem in predictive space, maps the selected update back to input space, and ranks the candidates using the actual posterior (Eqs. (5)-(7), Algorithm 1).
AF gains - The requested AF gains are in Figures 12-13, and BO performance is in Figures 4-6 and 17-19.
The boundary result, radius and generality. The boundary result is now Proposition 4.1. Its purpose is to make the constraint active under the stated local assumptions and support the direction search in Eqs. (11)-(12) (lines 220-225 and 246-250). Figure 3 and lines 282-287 explain the first-order limitation addressed in the common response.
Extension to quadratic constraints. We agree that we can use any quadratic constraint in BOING. We now address the motivation for BOING as a trust-region BO method that solves AF maximization in the trust region in manifold space rather than in input space. It takes an optimal step in the manifold space and uses the pullbacks to the input space. Since the manifold space is lower dimensional, we are able to achieve significant computational gains.
Additional optimization time and initialization. Appendix E.2 provides the one-update comparison, and Appendix E.4 tests extra search effort through Newton's method. Table 9 and Figure 20 show substantially more computation without consistent BO improvement. BOING candidates could initialize either optimizer, but we have not tested whether the additional AF or BO improvement justifies the combined computation time.
Collinear gradients. We assume the full row rank only at the centers of the TR. This helps in having a non-singular FIM. However, since we cannot guarantee full rank, we also use numerical stabilization using a small jitter for the inverse. This is also addressed in the revised manuscript now.