- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In this work, the authors consider the use of isotropic Gaussian process models rather than the standard anisotropic counterpart for high-dimensional Bayesian optimization. This can be relevant in the regime where there are not enough observations compared to the input dimension to learn anisotropy properly. They analyze the difference over some experiments (training on isotropic or anisotropic problems) on the modeling before turning to optimization.
Critical strengths
The main idea is simple and there is an analysis based on the link between the likelihood and design locations (through distances)
Critical weaknesses
Analyzing the training of GP models in high-dimension is a recurrent topic, with many similar analyses over the year. Using an isotropic model is generally considered as a useful baseline before considering more expensive models, see e.g., Binois & Wycoff, 2022. Similar analyses can be found for instance in:
- Appriou, Rullière, Gaudrie, High-dimensional Bayesian optimization with a combination of Kriging models, Structural and Multidisciplinary Optimization, 2024.
- Blanchet-Scalliet, Helbert, Ribaud, Vial, Four algorithms to construct a sparse kriging kernel for dimensionality reduction, Computational Statistics, 2019.
Concerning the definition of bounds for hyperparameters and the use of the Fisher information matrix, a relevant baseline is Gu, Wang, Berger, Robust Gaussian stochastic process emulation, The Annals of Statistics, 2018.
The use of the Gaussian kernel is not recommended, see Xu et al. (2025), the corresponding results should be put to appendix and replaced by Matérn ones.
10 runs are not sufficient for drawing strong conclusions.
Questions for authors
In the discussion of Proposition 1, the outcome is that the likelihood surface is flat, hence that different hyperparameters can have similar MLE values. Why not considering a fully Bayesian framework here to appropriately take it into account?
L234: “This restriction does not create additional information.” Isn’t this obvious?
The use of Sobol sequences in high dimension may not be recommended, due to gaps and alignments. What happens with more robust methods for designs of experiments like Mak, S., & Joseph, V. R. (2025). Robust designs for Gaussian process emulation of computer experiments. arXiv preprint arXiv:2507.09156.?
For the samples from GPs, have you checked that the lengthscales are sufficiently large such that the complexity remains reasonable compared to iid noise (see Hvarfner et al., 2024)?
How do you set the noise variance in the experiments?
12 comments
Response to kernel choice and statistical evidence
Critical Weaknesses: "The use of the Gaussian kernel is not recommended ... the corresponding results should be ... replaced by Matérn ones."
We disagree that the squared exponential (SE) results should be set aside. Xu et al. (2025) do not show that SE kernels are inherently unsuitable for HDBO. They identify a particular failure mode that small, dimension-independent lengthscale initialization can lead to vanishing gradients during GP fitting. They also show that dimension-aware initialization greatly improves SE performance, and identify the dimension-scaled lognormal prior of Hvarfner et al. (2024) as another way to alleviate this problem. That is the prior we use for both Vanilla-MAP and Iso-MAP. Their findings are a reason to be careful about SE fitting, not to discard comparisons using it.
More importantly, the requested Matérn experiments are already in the paper. Appendix D.4 (Figures 23 to 25) repeats the Iso-MAP versus Vanilla-MAP comparison with Matérn 5/2 kernels on the synthetic, anisotropic GP-prior, and real-world problems. The central pattern remains on the synthetic and GP-prior benchmarks; on real-world tasks, the effect of kernel choice is more mixed, and we report that too. In several synthetic settings, Vanilla-MAP is actually more sensitive to the kernel family than Iso-MAP. We will make these results more prominent in the main text, but this is not a missing experiment. If you believe the Matérn results contradict our conclusions, which comparison concerns you?
Critical Weaknesses: "10 runs are not sufficient for drawing strong conclusions."
We strongly disagree with this blanket assessment. Ten run experiments are very common in recent HDBO work, including Xu et al. (2025) and Doumont et al. (AISTATS 2026). More importantly, we do not base our conclusions simply on the separation between median curves. Iso-MAP and Vanilla-MAP use paired initial designs, and Appendix D.1 (Table 2) reports the paired final-value differences, 95% bootstrap intervals, win counts, and one-sided Wilcoxon tests.
For example, Iso-MAP wins all 10 paired runs on 100D Styblinski–Tang () and 9 of 10 on HalfCheetah-v2 (). The anisotropic GP-prior comparisons at and also favor Iso-MAP ( and ). Across the 12 main problems, 11 median paired differences favor Iso-MAP and eight one-sided tests have (unadjusted); four remain below the Bonferroni threshold for 12 comparisons. We also report the cases where the evidence is inconclusive, and the SVM result that favors Vanilla-MAP. We have never claimed significant improvement on every problem.
Additional runs would improve precision, especially in the closer cases. But a blanket objection to 10 seeds does not engage with the statistics already provided. Which specific benchmark or conclusion do you believe is unsupported, and why are the paired results in Table 2 insufficient?
Response to Questions for Author 1 to 5
Q1: "Why not consider a fully Bayesian framework?"
A fully Bayesian ARD fit would not resolve the underlying ambiguity. If two lengthscale settings produce exactly the same training covariance, they have the same likelihood; integrating over them does not give the data a way to distinguish them. Bayesian inference can instead average over these settings, with their relative weights determined by the prior. That may improve predictions, but it answers a different question. Our paper deliberately studies the MAP-based Vanilla BO framework used in modern HDBO and asks whether fitting one shared lengthscale works better than fitting separate ones. A fully Bayesian method might perform differently, but that would not invalidate either the likelihood result or our matched MAP comparison.
We also include SAASBO, which uses fully Bayesian inference with a sparsity-inducing ARD prior, as a contextual baseline. It is substantially more expensive in our timing study (Table 3). Developing that comparison could be worthwhile, but we believe it is a separate investigation, not at all a flaw in what we have established.
Q2: "This restriction does not create additional information. Isn't this obvious?"
Yes. That sentence is a clarification, not a claimed result. Tying parameters reduces what must be estimated; it does not create information. The result is that different ARD models can fit the observed training data identically yet predict differently at new candidates (Appendix A.3). We can shorten the sentence.
Q3: Sobol sequences in high dimensions (Mak & Joseph, 2025)
Both methods start from the same 10 Sobol points; the other 990 evaluations are selected adaptively. This is standard practice in recent HDBO, including Hvarfner et al. (2024) and Papenmeier et al. (2025). More importantly, Proposition 1 holds for any fixed training design, and Appendix B.2 finds the predictive advantage on adaptively collected BO trajectories, not only Sobol designs.
A different initialization could certainly change the results, but Mak and Joseph study designs for GP emulation; they do not show that Sobol initialization systematically favors isotropic over ARD fitting in our comparison. What is the specific concern with using the same 10 Sobol points for both methods? We would be glad to consider evidence of such a bias.
Q4: Are the GP-prior objectives effectively iid noise?
No. Each objective is a fixed function drawn from an anisotropic SE GP prior and approximated using 4,096 random Fourier features (not independent noise at each evaluation). The generating lengthscales also scale with dimension (Appendix C.3). The functions are difficult globally but retain local structure: in 100D at , the expected generating-kernel correlation is about 0.02 between independent uniform points, but about 0.67 when only 10 coordinates are resampled. This is consistent with the BO methods improving far beyond Sobol sampling in Figure 3. We will add this simple correlation calculation to Appendix C.3.
Q5: "How do you set the noise variance in the experiments?"
No observation noise is added to the GP prior or deterministic synthetic objectives. The MuJoCo-v2 controller-tuning tasks average 10 rollouts with randomized initial states, while the v5 action-sequence tasks use fixed initial states. For the GP surrogates, the noise variance is fitted, not fixed to zero: both Vanilla-MAP and Iso-MAP standardize observations, fix the signal variance at one, and estimate the observation-noise parameter using the same MAP procedure (Appendix C.4). We will state the fitted noise prior explicitly there.
Response to prior work on GP modeling and Fisher information
We will address the review in separate comments to make the discussion easier to follow, starting with your concerns about prior work and Fisher information.
We agree that estimating GP lengthscales with limited high-dimensional data is a longstanding problem, and that isotropic GPs are not new - but this is our starting point, not at all the contribution we claim.
Our paper establishes something more specific. Proposition 1 characterizes which changes in ARD lengthscales the training likelihood cannot distinguish. Importantly, two ARD models can have exactly the same training covariance and likelihood yet different predictions at new candidate points (Corollary 2, Appendix A.3). We then test one of the simplest (limiting) response: tying the lengthscales while keeping the modern Vanilla BO pipeline matched. Remarkably, this can improve optimization even when the objective is generated from an anisotropic GP, where ARD is the more expressive model class. We also show where this advantage can reverse under extreme anisotropy (Section 6.2; Appendix D.2).
We do not believe these findings follow from the general observation that high-dimensional GP fitting is difficult. The cited papers address broadly related questions, but their actual contributions are quite different:
We are happy to acknowledge this earlier work, but citing papers on GP lengthscale estimation does not establish that our specific theoretical or empirical contributions are already known. If you believe one of these papers already proves our result or demonstrates the same empirical finding, could you identify the particular theorem or experiment? We would certainly want to address genuine overlap, but the review has not established such overlap.
Gu et al. (2018) study robust estimation of GP range parameters, including the use of Fisher information to construct objective priors and the importance of parameterization. We agree this is relevant background and will cite it. However, we are not proposing a new prior or claiming to introduce Fisher information. Our result addresses a different question.
Specifically, Proposition 2 establishes a bound connecting the information available about an ARD direction to how much that direction changes the weighted pairwise distances among the observed points. Under the stated assumptions, the directional Fisher information is bounded by a constant times ∥AXh∥22. If AXh=0, the Fisher information in that direction is exactly zero. This directly connects the training design to which coordinatewise lengthscale changes the likelihood can distinguish. Gu et al. study a different estimation problem; their use of Fisher information does not make this result redundant or invalidate it.
We are also unclear what you mean by "bounds for hyperparameters." Are you referring to the numerical bounds used during fitting, or to our theoretical Fisher-information bound? If you believe a particular choice or mathematical statement is problematic, please identify it and explain why. Similarly, if you believe Gu et al. already establish our result, please point us to the corresponding analysis.
As written, we see a relevant citation to add, but no identified flaw in our theory or evidence that the result is already known. We would appreciate a concrete explanation of what you believe is wrong or missing.