- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a scalable BO method, abbreviated “AMS-DVGP”. The authors distinguish “scalable” as meaning both in the number of observations (computational efficiency) and the number of dimensions (optimization efficiency). AMS-DVGP builds on the existing SVGP literature and adds on a “controller” that changes the search center, spatial scale, local/global behavior, and batch allocation. It also uses a Dirichlet-process Gaussian mixture summarize clusters of previously observed locations.
Critical strengths
The reported performance leadership over FocalBO is consistent
Trajectory-averaged regret is an interesting idea to measure early performance
Critical weaknesses
AMS-DVGP and FocalBO both share the same "improve BO performance by focusing SVGP training on local regions by weighting training data according to the active search region and using several levels of locality". The authors claim they differ by the controller by adding local/global search, choosing the centers, and batch allocation. Thus, the primary novelty introduced by this paper appears to be the integration of these ideas.
Unfortunately, several details are hidden inside a sparse Section 3.3. While Appendix A.2 gives the relevant hyperparameters, the paper itself offers no explanation as to how these parameters are chosen, how sensitive they are, and to what extent they are responsible for the experimental results.
For example, if we change the multi-scale allocation to (0.2, 0.4, 0.4), do we still preserve the performance gain and why? Some ablations to study how each piece of these ideas contribute to the performance of the method would help strengthen the paper.
The ingredients proposed lack significant details, handling, and justification. For example, take Section 3.2 on the progress. Here, progressed is classified into one of four categories. There is no explanation how this is done so. The appendix only offers hyperparameters. Even then, the threshold defines a which is nowhere else found in the paper.
Equation 1 is the standard ELBO for SVGPs which sums the log-likihood terms and adds a KL penalty. However, Equation 4 uses a weighted average, so there is an implicit scaling for the penalty.
Questions for authors
Figure 5 shows that the method is “robust” against choosing among UCB/EI/TS. In standard BO procedures vary due to inherent differences in locality bias and how the acquisition functions are optimized. e.g. TS is often quite exploratory and EI is often exploitative. While the paper makes no comment on this, it is not clear whether such a property is desirable or why such a behavior manifests in the approach. Some questions I ask are: are all acquisitions inherently local? One can answer this by sweeping the parameter in UCB, for example, and measuring the distance between two queries, between the query and the incumbent, etc.
How does the asymptotic complexity of your method compare to FocalBO or SVGPs? I have a feeling the full-covariance DP-GMM has a large effect on this.
Figure 4 shows that despite different objective functions and dimensions, the approach spends roughly the same time in each of the four categories the authors have binned. Is there an explanation, despite the different landscapes, the behavior is similar?
I think [1] is a very related paper, but since it was only released the start of this year I would consider it as concurrent work. I bring this up only to make authors aware of it for future versions of this work.
[1] Zhang, Yan, et al. "Regime-Adaptive Bayesian Optimization via Dirichlet Process Mixtures of Gaussian Processes." arXiv preprint arXiv:2601.20043 (2026).
9 comments
Q2. How were the controller parameters selected, how sensitive are the results to them, and does the performance gain persist if the multi-scale allocation is changed to (0.2,0.4,0.4)?
The controller constants were engineering choices reused across all eight tasks without task-specific controller tuning. We now state this explicitly, rather than suggesting that the particular numerical values were theoretically derived or selected by a validation sweep. Section 3 and Appendix A.2 define the progress statistic, thresholds, cluster gate, scale tuples, action priorities, and integer allocation rule. For example, the raw-incumbent progress statistic is , with thresholds 0.01, 0.003, and 0.001. The fourth accumulated stagnation event triggers global escape. We also explain the controller's sensitivity to response transformations.
We tested your proposed fine/medium/coarse allocation (0.2, 0.4, 0.4) on both 500-D tasks, with ten matched seeds and the original evaluation budgets and inducing capacities.
The endpoint reductions are 28.7% and 85.7%; all four paired comparisons have Holm-adjusted (Table 11). The initial response prefixes match for all twenty pairs. The archived files establish initial-y matching; the new controls additionally retain complete initial-X checks. Because these runs span execution environments, this comparison concerns evaluation efficiency.
Against the new full AMS baseline with the original allocation, the alternative allocation has raw p-values from 0.1602 to 0.3223, with all four adjusted values equal to 0.6406 (Table 10). Thus the observed advantage over archived FocalBO persists for the tested alternative; these tests do not prove equivalence between allocations or robustness to all controller constants. Both allocations retain all three scales, but that design rationale is not an experimentally isolated explanation. The remaining thresholds have not been exhaustively swept.
We also completed fixed-mixture and likelihood-only controls (Figure 8). They require more limited attribution: on Rosenbrock the fixed mixture has worse mean endpoint and trajectory regret, but its raw adjusts to 0.1094; the likelihood-only control is close to full AMS. The revised text distinguishes these interventions from the archived removal of the complete training mechanism.
The suggested allocation changes fine/medium/coarse shares from (0.4, 0.4, 0.2) to (0.2, 0.4, 0.4). The two relevant paired comparisons are reproduced together below; Rosenbrock values are in millions.
| Task / metric | Full AMS 40/40/20 | Suggested 20/40/40 | Raw vs full | vs full | Archived FocalBO | vs FocalBO |
|---|---|---|---|---|---|---|
| Rastrigin endpoint | 4703.1 | 4852.0 | 0.2324 | 0.6406 | 6803.2 | 0.0078 |
| Rastrigin trajectory | 5363.7 | 5550.3 | 0.1602 | 0.6406 | 7220.8 | 0.0078 |
| Rosenbrock endpoint | 1.374 | 1.561 | 0.1602 | 0.6406 | 10.921 | 0.0078 |
| Rosenbrock trajectory | 4.567 | 4.958 | 0.3223 | 0.6406 | 16.678 | 0.0078 |
Controller definitions are in Sections 3.1-3.3 and Appendix A.2, Table 4; complete paired results are in Appendix F.2, Tables 10-11 and Figure 8.
Q3. Why does Eq. (4) use a weighted average of the likelihood terms, which appears to introduce an implicit -dependent scaling of the KL penalty relative to Eq. (1)?
A. Your observation about the submitted expression is correct. Averaging the likelihood while retaining full KL gives , which is equivalent up to scale to . The written equation omitted the KL/N normalization used in the augmented training implementation.
The corrected Eq. (4), under equal weights and no auxiliary terms, gives . Omitting the stabilizer yields the standard ELBO divided by N. N denotes the total training-set size, including during minibatch training. Appendix F.6 additionally compares both normalizations on fixed histories and reports prediction error, latent/predictive standard deviations, coverage, and UCB selections. We explicitly report predictive overcoverage under KL/N and do not claim that this equal-weight diagnostic establishes calibration of the complete weighted model.
The fixed-history diagnostic is:
| Penalty | RMSE | Latent SD | Predictive SD | 95% coverage | NLPD | |
|---|---|---|---|---|---|---|
| 100 | KL/ | 0.960 | 1.533 | 2.241 | 0.9996 | 1.807 |
| 100 | KL | 4.441 | 1.386 | 3.739 | 0.9798 | 2.939 |
| 300 | KL/ | 1.322 | 1.743 | 2.658 | 0.9945 | 2.030 |
| 300 | KL | 7.167 | 1.759 | 4.755 | 0.9022 | 3.617 |
See Section 3.5, Eq. (4), and Appendix F.6, Table 15 and Figure 11.
Thank you for the specific suggestions on allocation, acquisition locality, computational cost, and related work. We have expanded the method specification, completed the targeted experiments, and narrowed the claims where the evidence does not support a stronger conclusion.
Q1. Given the overlap with FocalBO, what is the specific contribution of AMS-DVGP, and how does it differ from the concurrent DP-mixture work of Zhang et al.?
A. We agree that local weighting of sparse-GP training and a hierarchy of spatial scales are shared with FocalBO. We now present our contribution as a specific integration for adaptive spatial allocation. FocalBO adapts the depth of an incumbent-centered hierarchy; AMS additionally chooses local/global actions, can use a ranked cluster of visited inputs as a center, and allocates the next batch across regions and scales. Section 2 explicitly acknowledges these shared ingredients.
Thank you also for pointing us to Zhang et al., Regime-Adaptive Bayesian Optimization via Dirichlet Process Mixtures of Gaussian Processes. We have cited and discussed this work in a dedicated related-work subsection. RAMBO uses a DP mixture of GPs to represent latent objective regimes, with separate GP components. AMS uses one SVGP to predict the objective and a separate DP-GMM over visited input locations to guide proposals. Its progress categories describe improvement during a run, rather than latent components of a mixture predictive model. This is a distinction in modeling and decision roles, not a performance comparison with RAMBO; no head-to-head superiority is claimed. [Zhang et al., arXiv:2601.20043]
These distinctions are discussed in Sections 2.3-2.4.