acceptodds
Under review as a conference paper at ICLR 2027

Isotropic Gaussian Processes Improve Vanilla Bayesian Optimization in High Dimensions

Abstract

High-dimensional Bayesian optimization (BO) often fits Gaussian process (GP) surrogates from far fewer observations than input dimensions. Modern Vanilla BO can perform well in this regime with dimension-aware priors, initialization, and acquisition optimization, but it typically retains automatic relevance determination (ARD), fitting one lengthscale per input coordinate. We study this modeling choice and propose Iso-BO, a controlled modification that replaces the ARD GP with an isotropic GP using one shared lengthscale while keeping the surrounding BO pipeline matched. For radial kernels, we show that the marginal log likelihood (MLL) depends on the inverse-squared ARD lengthscales only through weighted pairwise distances among the observed inputs. The current design can therefore leave some ARD directions exactly invisible or only weakly constrained by the MLL. Iso-BO removes coordinatewise reweighting and fits a single shared scale instead. Lengthscale-fitting and predictive-density diagnostics show that this finite-data effect appears in practice, including when the data-generating process is anisotropic. Across GP-prior, synthetic, and real-world benchmarks, Iso-BO often improves over matched modern Vanilla BO and remains competitive with the included high-dimensional BO baselines under the tested budgets. Stress tests also show the expected boundary wherein sufficiently strong, learnable anisotropy can favor the more flexible ARD model.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
High-Dimensional Bayesian Optimization, Gaussian Processes, Hyperparameter Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fTz7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this work, the authors consider the use of isotropic Gaussian process models rather than the standard anisotropic counterpart for high-dimensional Bayesian optimization. This can be relevant in the regime where there are not enough observations compared to the input dimension to learn anisotropy properly. They analyze the difference over some experiments (training on isotropic or anisotropic problems) on the modeling before turning to optimization.

Critical strengths

The main idea is simple and there is an analysis based on the link between the likelihood and design locations (through distances)

Critical weaknesses

Analyzing the training of GP models in high-dimension is a recurrent topic, with many similar analyses over the year. Using an isotropic model is generally considered as a useful baseline before considering more expensive models, see e.g., Binois & Wycoff, 2022. Similar analyses can be found for instance in:

  • Appriou, Rullière, Gaudrie, High-dimensional Bayesian optimization with a combination of Kriging models, Structural and Multidisciplinary Optimization, 2024.
  • Blanchet-Scalliet, Helbert, Ribaud, Vial, Four algorithms to construct a sparse kriging kernel for dimensionality reduction, Computational Statistics, 2019.

Concerning the definition of bounds for hyperparameters and the use of the Fisher information matrix, a relevant baseline is Gu, Wang, Berger, Robust Gaussian stochastic process emulation, The Annals of Statistics, 2018.

The use of the Gaussian kernel is not recommended, see Xu et al. (2025), the corresponding results should be put to appendix and replaced by Matérn ones.

10 runs are not sufficient for drawing strong conclusions.

Questions for authors

In the discussion of Proposition 1, the outcome is that the likelihood surface is flat, hence that different hyperparameters can have similar MLE values. Why not considering a fully Bayesian framework here to appropriately take it into account?

L234: “This restriction does not create additional information.” Isn’t this obvious?

The use of Sobol sequences in high dimension may not be recommended, due to gaps and alignments. What happens with more robust methods for designs of experiments like Mak, S., & Joseph, V. R. (2025). Robust designs for Gaussian process emulation of computer experiments. arXiv preprint arXiv:2507.09156.?

For the samples from GPs, have you checked that the lengthscales are sufficiently large such that the complexity remains reasonable compared to iid noise (see Hvarfner et al., 2024)?

How do you set the noise variance in the experiments?

12 comments
AuthorsOpenReview ↗

Response to prior work on GP modeling and Fisher information

We will address the review in separate comments to make the discussion easier to follow, starting with your concerns about prior work and Fisher information.

Critical Weaknesses: "Analyzing the training of GP models in high-dimension is a recurrent topic ... Using an isotropic model is generally considered as a useful baseline"

We agree that estimating GP lengthscales with limited high-dimensional data is a longstanding problem, and that isotropic GPs are not new - but this is our starting point, not at all the contribution we claim.

Our paper establishes something more specific. Proposition 1 characterizes which changes in ARD lengthscales the training likelihood cannot distinguish. Importantly, two ARD models can have exactly the same training covariance and likelihood yet different predictions at new candidate points (Corollary 2, Appendix A.3). We then test one of the simplest (limiting) response: tying the lengthscales while keeping the modern Vanilla BO pipeline matched. Remarkably, this can improve optimization even when the objective is generated from an anisotropic GP, where ARD is the more expressive model class. We also show where this advantage can reverse under extreme anisotropy (Section 6.2; Appendix D.2).

We do not believe these findings follow from the general observation that high-dimensional GP fitting is difficult. The cited papers address broadly related questions, but their actual contributions are quite different:

  • Binois & Wycoff (2022) survey high-dimensional GP modeling and discuss isotropy as one option. They do not establish our finite-design result or demonstrate the advantage of isotropy in a matched modern Vanilla BO comparison.
  • Blanchet-Scalliet et al. (2019) recognize that isotropy reduces the number of parameters, but regard full isotropy as too restrictive and develop groupwise kernels for GP emulation. They do not investigate the consequences for sequential BO or establish our theoretical result.
  • Appriou et al. (2024) address difficult lengthscale estimation by combining Kriging models with randomly chosen lengthscales, demonstrating improvements in BO. This is a different approach that neither characterizes the finite-design likelihood ambiguity nor compares tied versus independent lengthscales in an otherwise matched modern Vanilla BO pipeline.

We are happy to acknowledge this earlier work, but citing papers on GP lengthscale estimation does not establish that our specific theoretical or empirical contributions are already known. If you believe one of these papers already proves our result or demonstrates the same empirical finding, could you identify the particular theorem or experiment? We would certainly want to address genuine overlap, but the review has not established such overlap.

Critical Weaknesses: "Concerning the definition of bounds for hyperparameters and the use of the Fisher information matrix, a relevant baseline is Gu, Wang, Berger (2018)."

Gu et al. (2018) study robust estimation of GP range parameters, including the use of Fisher information to construct objective priors and the importance of parameterization. We agree this is relevant background and will cite it. However, we are not proposing a new prior or claiming to introduce Fisher information. Our result addresses a different question.

Specifically, Proposition 2 establishes a bound connecting the information available about an ARD direction to how much that direction changes the weighted pairwise distances among the observed points. Under the stated assumptions, the directional Fisher information is bounded by a constant times ∥AXh∥22\|A_Xh\|_2^2. If AXh=0A_Xh=0, the Fisher information in that direction is exactly zero. This directly connects the training design to which coordinatewise lengthscale changes the likelihood can distinguish. Gu et al. study a different estimation problem; their use of Fisher information does not make this result redundant or invalidate it.

We are also unclear what you mean by "bounds for hyperparameters." Are you referring to the numerical bounds used during fitting, or to our theoretical Fisher-information bound? If you believe a particular choice or mathematical statement is problematic, please identify it and explain why. Similarly, if you believe Gu et al. already establish our result, please point us to the corresponding analysis.

As written, we see a relevant citation to add, but no identified flaw in our theory or evidence that the result is already known. We would appreciate a concrete explanation of what you believe is wrong or missing.

AuthorsOpenReview ↗

Response to kernel choice and statistical evidence

Critical Weaknesses: "The use of the Gaussian kernel is not recommended ... the corresponding results should be ... replaced by Matérn ones."

We disagree that the squared exponential (SE) results should be set aside. Xu et al. (2025) do not show that SE kernels are inherently unsuitable for HDBO. They identify a particular failure mode that small, dimension-independent lengthscale initialization can lead to vanishing gradients during GP fitting. They also show that dimension-aware initialization greatly improves SE performance, and identify the dimension-scaled lognormal prior of Hvarfner et al. (2024) as another way to alleviate this problem. That is the prior we use for both Vanilla-MAP and Iso-MAP. Their findings are a reason to be careful about SE fitting, not to discard comparisons using it.

More importantly, the requested Matérn experiments are already in the paper. Appendix D.4 (Figures 23 to 25) repeats the Iso-MAP versus Vanilla-MAP comparison with Matérn 5/2 kernels on the synthetic, anisotropic GP-prior, and real-world problems. The central pattern remains on the synthetic and GP-prior benchmarks; on real-world tasks, the effect of kernel choice is more mixed, and we report that too. In several synthetic settings, Vanilla-MAP is actually more sensitive to the kernel family than Iso-MAP. We will make these results more prominent in the main text, but this is not a missing experiment. If you believe the Matérn results contradict our conclusions, which comparison concerns you?

Critical Weaknesses: "10 runs are not sufficient for drawing strong conclusions."

We strongly disagree with this blanket assessment. Ten run experiments are very common in recent HDBO work, including Xu et al. (2025) and Doumont et al. (AISTATS 2026). More importantly, we do not base our conclusions simply on the separation between median curves. Iso-MAP and Vanilla-MAP use paired initial designs, and Appendix D.1 (Table 2) reports the paired final-value differences, 95% bootstrap intervals, win counts, and one-sided Wilcoxon tests.

For example, Iso-MAP wins all 10 paired runs on 100D Styblinski–Tang (p=0.001p=0.001) and 9 of 10 on HalfCheetah-v2 (p=0.002p=0.002). The anisotropic GP-prior comparisons at γ=0.3\gamma=0.3 and 0.50.5 also favor Iso-MAP (p=0.010p=0.010 and 0.0070.007). Across the 12 main problems, 11 median paired differences favor Iso-MAP and eight one-sided tests have p<0.05p<0.05 (unadjusted); four remain below the Bonferroni threshold for 12 comparisons. We also report the cases where the evidence is inconclusive, and the SVM result that favors Vanilla-MAP. We have never claimed significant improvement on every problem.

Additional runs would improve precision, especially in the closer cases. But a blanket objection to 10 seeds does not engage with the statistics already provided. Which specific benchmark or conclusion do you believe is unsupported, and why are the paired results in Table 2 insufficient?

AuthorsOpenReview ↗

Response to Questions for Author 1 to 5

Q1: "Why not consider a fully Bayesian framework?"

A fully Bayesian ARD fit would not resolve the underlying ambiguity. If two lengthscale settings produce exactly the same training covariance, they have the same likelihood; integrating over them does not give the data a way to distinguish them. Bayesian inference can instead average over these settings, with their relative weights determined by the prior. That may improve predictions, but it answers a different question. Our paper deliberately studies the MAP-based Vanilla BO framework used in modern HDBO and asks whether fitting one shared lengthscale works better than fitting DD separate ones. A fully Bayesian method might perform differently, but that would not invalidate either the likelihood result or our matched MAP comparison.

We also include SAASBO, which uses fully Bayesian inference with a sparsity-inducing ARD prior, as a contextual baseline. It is substantially more expensive in our timing study (Table 3). Developing that comparison could be worthwhile, but we believe it is a separate investigation, not at all a flaw in what we have established.

Q2: "This restriction does not create additional information. Isn't this obvious?"

Yes. That sentence is a clarification, not a claimed result. Tying parameters reduces what must be estimated; it does not create information. The result is that different ARD models can fit the observed training data identically yet predict differently at new candidates (Appendix A.3). We can shorten the sentence.

Q3: Sobol sequences in high dimensions (Mak & Joseph, 2025)

Both methods start from the same 10 Sobol points; the other 990 evaluations are selected adaptively. This is standard practice in recent HDBO, including Hvarfner et al. (2024) and Papenmeier et al. (2025). More importantly, Proposition 1 holds for any fixed training design, and Appendix B.2 finds the predictive advantage on adaptively collected BO trajectories, not only Sobol designs.

A different initialization could certainly change the results, but Mak and Joseph study designs for GP emulation; they do not show that Sobol initialization systematically favors isotropic over ARD fitting in our comparison. What is the specific concern with using the same 10 Sobol points for both methods? We would be glad to consider evidence of such a bias.

Q4: Are the GP-prior objectives effectively iid noise?

No. Each objective is a fixed function drawn from an anisotropic SE GP prior and approximated using 4,096 random Fourier features (not independent noise at each evaluation). The generating lengthscales also scale with dimension (Appendix C.3). The functions are difficult globally but retain local structure: in 100D at γ=0.3\gamma=0.3, the expected generating-kernel correlation is about 0.02 between independent uniform points, but about 0.67 when only 10 coordinates are resampled. This is consistent with the BO methods improving far beyond Sobol sampling in Figure 3. We will add this simple correlation calculation to Appendix C.3.

Q5: "How do you set the noise variance in the experiments?"

No observation noise is added to the GP prior or deterministic synthetic objectives. The MuJoCo-v2 controller-tuning tasks average 10 rollouts with randomized initial states, while the v5 action-sequence tasks use fixed initial states. For the GP surrogates, the noise variance is fitted, not fixed to zero: both Vanilla-MAP and Iso-MAP standardize observations, fix the signal variance at one, and estimate the observation-noise parameter using the same MAP procedure (Appendix C.4). We will state the fitted noise prior explicitly there.

Reviewer_eBdqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

In this work, the authors propose not using in ARD in exponential kernels when performing BO with Gaussian Processes, but rather propose using an isotropic length-scale. The authors argue theoretically why ARD length-scales may overfit in small data regimes, and argue that for this reason isotropic length-scales should perform better in the BO regime. The authors then run experiments to validate their arguments; they first run BO on a number of synthetic problems where the synthetic optimisation problems are samples from GPs using ARD, which in theory should suit ARD based surrogates better. However, in their experiments the isotropic GP surrogates perform the best. The authors then run real-world experiments at budgets of 1000 points, during which the proposed isotropic GPs generally perform well, often outperforming other GP surrogates that are considered best practice in these settings.

Critical strengths
  1. This has practical implications for many BO users (of which there are many using exponential kernel GPs).
    • Whilst the recommendation and observation is simple (and perhaps an obvious one, given it (to me) appears to "just" be an overparametrization problem), it could still carry implications for practitioners.
Critical weaknesses
  1. The experimental results are for short horizons making it unclear if the observed performance increases are confounded due to the short BO campaigns.
  2. Some would argue that this could be a very incremental or lack of novelty paper due to the simplicity of the method (and it is possible that practitioners already know this due to it (in my opinion) being relative obvious that at small dataset sizes, ARD can lead to poor performance in large input spaces).
Questions for authors
  1. Could the authors discuss why they have such short experimental horizons? For example, in [1], one can see that it is possible to achieve much better objective values at longer horizons than what is run in this work.
  2. Whilst the held-out predictive density question is interesting it has been established in previous work that NLL (or NLPD) across the entire input space does not really tell one much about eventual BO performance (see [2] for a discussion of this). I think that the experiment the authors show therefore does not really underpin anything about eventual BO performance.
  3. Would the authors consider adding a ranking table of the surrogates across the synthetic and "real" optimisation problems in the main text? In my opinion, it is currently very difficult to evaluate which surrogate performs the best across all of the modalities, making it difficult to reach an objective conclusion on order of performance.

I do have one other comment that I do not know where to place, but that undeniably is affecting my assessment of the work - I see several places which I read as (I apologize for the term) AI slop, but that I cannot provide proof of being so. I strongly suggest the authors to amend this, as it to me makes the work significantly more difficult to understand and read. This is in particular true for much of Section 4 - I provide a couple of examples here:

  1. Line 196 "What the MLL sees" - what does this even mean?
  2. What is Proposition 1 supposed to tell me? It is possible that I simply do not understand the proposition, but the language is not helping me at all in understanding, which seems in stark contrast to the majority of the work. This makes me wonder what is causing the sudden change in style of writing?

Minor corrections:

  1. When reading the paper the first time SE is encountered, it is not explicitly written what this refers to (I assume it is squared-exponential).
  2. ψ\psi in Equation 1 is never defined.

[1]: Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search, Wang et. al, 2020 Neurips. [2]: On the Role of Model Uncertainties in Bayesian Optimization, Foldager et. al, UAI 2023.

8 comments
AuthorsOpenReview ↗

Response to evaluation budget (W1, Q1) and the role of NLPD (Q2)

Thank you for the review and for recognizing the practical relevance of this work. To keep the discussion manageable, we are responding in separate comments, beginning with your questions about the evaluation budget and predictive density.

Critical Weakness 1 / Question 1: "short experimental horizons"

We disagree that the evaluation horizon is a weakness or a confounding factor in this study. Limited evaluations are precisely the setting we want to understand, i.e., when observations are expensive and the BO dataset is initially small relative to the input dimension, is it worthwhile to fit DD separate lengthscales? A budget of 1000 evaluations is also entirely consistent with modern HDBO practice. Hvarfner et al. (2024), Xu et al. (2025), and Papenmeier et al. (2025) all study budgets of several hundred to around 1000 evaluations. Doumont et al. (AISTATS 2026, Best Student Paper Award) use 1000 evaluations for their main high-dimensional comparisons. In expensive black-box optimization, 1000 evaluations can already represent a considerable investment. We therefore do not see why this budget should be considered unusually short or inadequate for our question.

Wang et al. (2020; your [1]) show that better objective values can be found on MuJoCo tasks after much longer searches, including experiments with roughly 30,000 evaluations. But their approach also partitions the search space and uses local optimization within selected regions. Those results establish that longer searches with a different strategy can continue to improve, not that a 1000-evaluation comparison is confounded. Our central comparison gives Iso-MAP and Vanilla-MAP the same budget and otherwise matches their BO pipeline. We ask which surrogate parameterization works better with that amount of data. Moreover, refitting an exact GP after tens of thousands of observations is a different computational setting, given the O(n3)O(n^3) cost of a standard GP factorization.

Of course, the relative advantage could change as more data become available. We already acknowledge this possibility and show evidence of its limits: the predictive-density gap narrows as training size increases (Figure 2), and Vanilla-MAP can recover an advantage under extreme anisotropy (Appendix D.2). The exact null-space guarantee in Proposition 1 concerns very small designs; we do not claim it holds throughout a 1000-evaluation run. Proposition 2 instead characterizes how some ARD directions can remain weakly informed for a given design, with the experiments testing whether this matters in practice. A longer-horizon study would be interesting, but it is not needed to support our explicitly finite-budget conclusions.

Question 2: NLPD "does not really tell one much about eventual BO performance"

We agree with the caution, but think the conclusion goes too far. Foldager et al. (2023; your [2]) primarily examine the relationship between uncertainty calibration and BO regret, rather than NLPD as a diagnostic of GP lengthscale fitting. Doumont et al. (2026, Sec. 5.3) also show why predictive accuracy should not be treated as a proxy for optimization performance. We agree with that distinction, and do not use NLPD to predict regret.

Our NLPD experiments ask a different and important question: does the ambiguity in ARD fitting have consequences for predictions at unseen points? If different lengthscale settings can fit the training data equally well but predict differently at new points, we need to establish whether this theoretical ambiguity has a meaningful effect on predictive performance. Section 5 therefore tests held-out predictive density, including on anisotropic GP-generated objectives. We also go beyond predictions at uniformly sampled points: Appendix B.2 fits both surrogates to the first nn observations from the same Vanilla-MAP trajectories, then evaluates their predictions at points those trajectories select later. The isotropic model still gives lower NLPD on these adaptively chosen points, even though the data were collected by the ARD-based method.

Finally, Section 6 tests the outcome that NLPD cannot establish on its own: performance over the full sequential BO campaign. The predictive-density and BO experiments answer complementary questions; neither is offered as proof of the other.

AuthorsOpenReview ↗

Response to novelty (W2) and what Proposition 1 shows

Critical Weakness 2: "incremental or lack of novelty" ... "relatively obvious that at small dataset sizes, ARD can lead to poor performance"

We disagree that the simplicity of our approach makes the contribution incremental or obvious. In fact, we think the simplicity is one of its strengths. We are certainly not claiming to have invented isotropic GPs, and the general difficulty of fitting ARD with limited data is well known. But that is our starting point, not our contribution. Our paper establishes two things that we believe are new and important:

  1. A precise explanation of the problem. We show exactly how the observed design constrains ARD lengthscales through the training likelihood (Propositions 1 and 2). Crucially, two ARD models can have identical training covariances and likelihoods yet give different predictions at new candidate points (Corollary 2, Appendix A.3).
  2. A surprising practical result. Simply tying the lengthscales can improve modern Vanilla BO even when the objective is generated from an anisotropic GP, where ARD is the more expressive model class (Section 6.2). We also show that the advantage can reverse under extreme anisotropy (Appendix D.2). This reveals a meaningful tradeoff between model flexibility and what can reliably be learned from limited data.

We do not think either finding follows from a generic observation about overparameterization. Knowing that ARD can be difficult to fit does not establish which lengthscale parameterization will produce better predictions or BO decisions (especially when the underlying function is anisotropic). Moreover, recent work by Hvarfner et al. (2024), Xu et al. (2025), and Papenmeier et al. (2025b), as well as BoTorch's standard exact GP setup, continues to use ARD. We are challenging a modeling choice that remains standard in modern HDBO.

In our opinion, a simple intervention should not be confused with an obvious scientific result. If you believe our findings are already established, we would genuinely appreciate a reference demonstrating them. We think the contribution should be judged by what it establishes, rather than how complicated the proposed modification is.

Additional comment: "AI slop," "What the MLL sees," and Proposition 1

We also want to respond directly to the characterization of parts of our manuscript as "AI slop." We find this term inappropriate, particularly when the review acknowledges that there is no evidence to substantiate it. We openly disclosed our use of AI-assisted editing and technical checks (in accordance with the conference's disclosure requirements). The research ideas, mathematical analysis, writing, and experiments represent a substantial amount of work by the authors, and we take full responsibility for the paper. We certainly welcome criticism of unclear writing, but do not believe this characterization is a fair or constructive way to evaluate scientific work.

That said, we understand that our attempt to explain the theory in accessible language may not have worked well for you. Let us do our best to clarify exactly what Proposition 1 establishes and why we believe it matters.

For radial ARD kernels, the training covariance depends on the inverse-squared lengthscales η\eta only through the weighted pairwise distances AXηA_X\eta between observed points. Consider 10 observations in 100 dimensions. There are only 45 training pairs, so AXA_X is a 45×10045\times100 matrix with a null space of dimension at least 55. This means there are at least 55 independent directions in which the ARD parameters can change without changing any training covariance or the likelihood (provided the changes remain feasible and the other GP quantities are fixed).

The important part is what happens at a new candidate point. The distances from that candidate to the training points are generally not preserved by those same parameter changes. Corollary 2 (Appendix A.3) establishes that the train-test covariance changes for almost every candidate, meaning the posterior predictions and acquisition values can also change. Under MAP fitting, a prior can select among these training-equivalent models, but it cannot make the training likelihood distinguish them.

That is the central issue, i.e., the training data can be unable to distinguish between ARD models that would lead BO to make different decisions. This is why we believe Proposition 1 is important, rather than merely a restatement that ARD has many parameters.

We will revise the informal heading "What the MLL sees" to "What the training likelihood can distinguish" and improve the surrounding explanation. We respectfully welcome your thoughts on whether this clarification resolves your concern, or whether there is a particular mathematical point you still find unclear.

AuthorsOpenReview ↗

Response to ranking table (Q3) and minor points + revision summary and questions

Question 3: "Would the authors consider adding a ranking table...?"

Yes, we agree this would make the results easier to compare. We will add a compact table in the main text summarizing final performance and rankings across benchmarks, using the existing results. We will clearly distinguish methods with different evaluation budgets (particularly SAASBO), mark missing baselines, and only calculate aggregate rankings over comparable experiments. This will complement the paired statistical analysis already provided in Appendix D.1 (Table 2), where 11 of 12 median paired differences favor Iso-MAP and 8 of 12 one-sided Wilcoxon tests have p<0.05p<0.05.

Minor corrections: "SE" and the undefined symbol in Eq. (1)

SE is already introduced as "squared exponential" in the Introduction, but we will repeat it in Section 3. We believe the undefined symbol is ψ\psi (which denotes the radial kernel profile). We will make this explicit alongside Eq. (1).

Summary and remaining questions

We have carefully considered your review and, as explained in our previous responses, respectfully disagree with several of the criticisms. From our reading, your main concerns are that the evaluation budgets are too short, that our contribution is incremental or obvious, and that the predictive-density experiments do not establish improved BO performance. You also raised questions about the writing and presentation.

We believe we have addressed each of these points. The evaluation budgets are consistent with recent HDBO research and are precisely the regime our paper studies. The contribution is not simply using an isotropic GP, but explaining what finite training data can tell us about ARD lengthscales and demonstrating that a simpler model can improve BO even on anisotropic objectives. And we have never claimed that better NLPD guarantees better optimization; the predictive-density and BO experiments are designed to answer different questions.

We are also happy to make the presentation improvements discussed above, including the ranking table, a clearer explanation of Proposition 1, and the minor corrections. However, we do not believe the criticisms raised so far establish that our theoretical or empirical conclusions are unsupported.

We would therefore greatly appreciate your thoughts on three specific questions:

  1. Novelty: If our central finding is already obvious or established, could you point us to prior work that actually demonstrates it? We are not claiming to have invented isotropic GPs. We are claiming a specific finite-design explanation for the limitations of ARD fitting, together with the finding that isotropy can outperform modern ARD-based Vanilla BO even on anisotropic GP-generated objectives. We have not found these results in the existing literature. If we have overlooked something, we would certainly want to know.

  2. Evaluation budgets: Why do you consider 1000 evaluations insufficient to support our conclusions? Comparable budgets are widely used in the literature, and our claims concern performance under these finite budgets. We do not argue against studying longer campaigns, but would appreciate understanding which specific conclusion you believe requires a longer horizon, and why.

  3. Technical soundness: Is there a particular mathematical statement, assumption, experimental comparison, or inference that you believe is incorrect or inadequately supported? Your review gives a Soundness score of 2, but we have not been able to identify a technical problem from the comments. If one exists, we would genuinely appreciate knowing what it is so that we can address it.

We are very willing to correct mistakes and make meaningful improvements. But we think there is an important distinction between identifying a weakness in the evidence supporting our conclusions and preferring a different or broader contribution. At present, we believe the review's criticisms largely fall into the latter category, or concern presentation issues we can easily address.

If our responses resolve your concerns, we very much hope you will reconsider your assessment. We would very much appreciate your engagement on these points, particularly if there is a substantive issue that we have not adequately addressed.

Reviewer_FdCwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper rethinks the ARD lengthscale priors in Vanilla Bayesian Optimization. It shows that the exact DD ARD directions are invisible when the samples is limited by analyzing the MLL of a radial GP kernel. Instead, Iso-BO replaces the DD lengthscales with a shared scale, and demonstrates good experimental results.

Critical strengths
  1. The paper is written clearly.
  2. The modification of Iso-BO is elegant. And the motivation makes sense. Without user-specific priors, GP cannot effectively learn the underlying structure given limited observation. Thus, removing the degree of freedom in lengthscale to obtain better results is illustrative.
  3. The empirical analysis seems solid.
Critical weaknesses
  1. I think the main experiment does not fully disentangle the benefit of unifying lengthscales from the effect of regularization. ARD uses DD lengthscale-prior terms, whereas Iso uses only one, so their MAP objectives differ even along the isotropic restriction. Could the authors compare against a simple ARD baseline that shrinks coordinatewise lengthscales toward a shared value, while keeping the shared-scale prior and other BO settings matched (for example, lj=l0exp⁡(δj)l_j = l_0 \exp (\delta_j) with ∑jδj=0\sum_j \delta_j=0)? This would help clarify the benefits of the Iso GP.
  2. There is no regret analysis showing that Iso-BO can be better. I can understand the difficulty behind this. But maybe we can fix the BO history and compare the acquisition selected points to show the different behaviors of these methods.
Questions for authors

Please see the weaknesses above.

6 comments
AuthorsOpenReview ↗

Response to original review comments

Thank you for the careful review and for the positive assessment. Before responding to the two weaknesses, one small clarification on the summary: Iso-BO retains the dimension-scaled lengthscale prior of Hvarfner et al. (2024), applying it to one shared lengthscale rather than DD independent lengthscales. We do not introduce a new prior distribution; our idea is to change the lengthscale parameterization.

"the main experiment does not fully disentangle the benefit of unifying lengthscales from the effect of regularization. ARD uses (D) lengthscale-prior terms, whereas Iso uses only one"

We agree that the two MAP objectives differ, and we say so in Section 3 and Appendix A.2. But the paper does not rely on the MAP comparison alone. The theory in Section 4 and Appendix A is about the likelihood, not the prior. Specifically, along an exact null direction the training MLL is unchanged while predictions at new points still change (Appendix A.3), and a prior can regularize these directions, but cannot make them distinguishable to the training likelihood. Appendix D.4 also includes a comparison with no lengthscale prior at all. Iso-MLE and MSR both use Matérn-5/2 kernels and MLE fitting, so the prior asymmetry you describe is absent, and Iso-MLE still improves over MSR on most synthetic and GP-prior panels (Figs. 23 to 24). Finally, the DD prior terms do not keep ARD fits in check: in Figure 1, the L-BFGS-B ARD fits drift to average lengthscales 2-3x the true value in 50D and 100D, while the isotropic fits stay close to it.

"Could the authors compare against a simple ARD baseline that shrinks coordinatewise lengthscales toward a shared value"

This is a good idea, and it fits naturally between the two models we compare. Vanilla-MAP already applies the same prior to every ℓk\ell_k, which encourages shrinkage toward a fixed common center, whereas Iso-MAP fully pools the lengthscales at a learned scale. A partial-shrinkage model would introduce an additional parameter τ\tau controlling how strongly the coordinatewise lengthscales are tied. The important question then becomes how τ\tau should be selected, and whether that choice should change as more observations become available. Fixing one arbitrary value would give another comparison, but not answer this broader question. In our view, how much coordinatewise structure to allow given the current data is the central open question this paper raises, and Appendix F (BO-FIM) is our first step toward it. We will make this connection clearer in the revision, but see a proper study of adaptive shrinkage as follow-up work rather than a missing baseline.

"There is no regret analysis showing that Iso-BO can be better."

We very much agree this would be valuable, but we do think it is a separate paper. We do not claim a regret improvement anywhere (Section 7 says so explicitly). Existing regret bounds assume the kernel hyperparameters are known (Srinivas et al., 2010), or treat unknown or misspecified hyperparameters conservatively (Berkenkamp et al., 2019; Bogunovic & Krause, 2021). These frameworks do not directly capture the tradeoff that we study (i.e., the potential benefit of fitting fewer lengthscales when the available BO data cannot reliably determine the full ARD model). The effect we study comes from estimating DD lengthscales from a finite design, which those frameworks do not model. Building a regret theory that captures this would require new tools, and we do not think it is reasonable to expect it within this paper (though again this is an important topic).

"maybe we can fix the BO history and compare the acquisition selected points"

Part of this is already in the paper. Appendix B.2 fits both models to the same fixed Vanilla-MAP trajectories and compares their predictions on later observations. Appendix D.5 compares where the two methods sample and gives a geometric reason for the difference: around the incumbent, the high-correlation region is a Euclidean ball for the isotropic model but can stretch along long-lengthscale directions under ARD. Since we already have these fixed trajectories, we will add the direct version of your experiment: at checkpoints along each trajectory, both models propose the next point from the same data, and we report the true objective value at each proposal and its distance to the incumbent.

To summarize the revision: we will add the shrinkage discussion above, the fixed-history acquisition comparison, and a clearer pointer to the prior-free Iso-MLE vs. MSR comparison in the main text.

If these responses address your concerns, we hope you might consider strengthening your support for the paper. Thank you again for the constructive review.

Dear Reviewer FdCw,

As we're halfway through the discussion phase, please acknowledge the author's response and consider leaving them a comment on whether it has resolved some or all of your concerns.

Thanks!

I thank the authors for the detailed response, with the majority of my concerns solved. But as im not so familiar with this field, after carefully reading other reviewers' comments, I would like to maintain my rating currently. I'll also make a decision sooner or later on the authors' response to other reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.