acceptodds
Under review as a conference paper at ICLR 2027

Local Preferential Bayesian Optimization

Abstract

Bayesian optimization (BO) is a popular and effective approach for tuning expensive, noisy experiments, but requires the formulation of an explicit objective function. Preferential BO (PBO) removes this requirement by learning from pairwise human feedback, yet existing methods struggle to efficiently optimize beyond low- and medium-dimensional problems due to their global search approaches. We address this limitation by developing a family of local PBO methods that transfer key ideas from high-dimensional BO to the preferential setting. In particular, we introduce local PBO methods that adapt trust-region and derivative-informed local search to pairwise preference feedback, where the latter exploits first- and second-order derivatives of the Laplace-approximated Gaussian process (GP) posterior. To reduce sensitivity to initialization, we additionally consider global-to-local variants that transition from global exploration to local refinement. Our benchmarks on GP sample paths, standard optimization benchmark functions, and policy-search tasks show that local PBO methods are especially effective in high-dimensional and complex landscapes with steep optima. Compared with global preference-based baselines, they can substantially reduce cumulative regret, making them particularly useful for real-world preference-based optimization tasks such as policy search.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian Optimization, Preference Learning, Gaussian Processes

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jcbZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The article combines standard localised BO methods (trust regions, or local gradient moves) designed for high dimensional BO, with preferential Bayesian optimisation. An experimental validation is carried out, showing some strong performance. And an ablation study is carried out to determine what is actually important.

Critical strengths

The main subject of the paper is an obvious thing to do, and I don't think it's been done before

The article situates itself well in the literature

The derivation of the correct posterior distributions, derivatives and Hessians is what makes the article more than a plug and play of two concepts into a new article. This is correct so far as I can tell.

Critical weaknesses

The main subject of the paper is an obvious thing to do! It's just not very exciting. It's plugging together local methods with preferential BO, and a small number of necessary calculations to make it happen. While this is solid work, it doesn't feel sufficiently ground-breaking to be an ICLR paper. (I'm not one of those reviewers who insists everything must be revolutionary, but this one feels very much to be just plugging two things together.)

It is very difficult to gain much insight from the experimental results.

The case for cumulative regret is very poorly made.

Questions for authors

For the gradient based methods there seem to be very large requirements of the number of samples. Eg at l250 it is stated that GIBO carries out M experiments to evaluate the gradient at each iteration. Firstly, since we're doing local moves, can't we use previous observations as well to estimate the gradient? And if this is the case, do we need to take quite so many observations on each iteration?

Line 346, how does a line search work here? The referred appendix doesn't explain this.

Why are newly experimented points always (?) compared to the incumbent? I would have thought that more information could be gained at times by comparing two newly-sampled points.

Section 5.4. When we go global to local, wouldn't it make more sense to consider refining from any of the "good" points found by the global method, not just from the best one? Also, talk of finding a promising "region" is a little misleading I feel, unless I've missed something, because I think you refine a single point, not a region?

I could not parse Lines 398-400.

In line 402-403, I do not know what "GP sample-path problems" means.

Minor comments:

  • The very first reference, to Garnett (2023), is a very recent reference for a very established method...
  • Line 96. Starting with "Instead" doesn't make sense.
  • Line 311, I think you've used the preference operator to mean negative semi-definite?
  • Line 422, "smoother objectives from Sec 6" doesn't make much sense, since we're in Sec 6 already
4 comments
AuthorsOpenReview ↗

Answer to Reviewer jcbZ Weaknesses

Thank you for the time and effort invested in reviewing our paper. In the following, we address the stated weaknesses and answer the raised questions.

Weaknesses

W1: Limited novelty and contribution.

We agree that our methods build on existing local BO algorithms, but the calculations needed for the preferential setting are more than a small step. Deriving the joint predictive distribution of the latent utility, its gradient and its Hessian under the Laplace-approximated pairwise GP is what makes derivative-informed PBO possible, and to our knowledge GIPBO and PrefSQP are the first PBO methods to use this information. We believe this is useful because many applications where preference feedback is most natural, such as controller tuning or policy search, are high-dimensional, and our results show that local PBO can substantially improve cumulative performance there, which means fewer poor and potentially unsafe evaluations on the real system. In addition, our benchmarking of complementary local strategies against global PBO baselines across different problem settings gives an honest picture of when local PBO is effective and where it fails, which we believe is informative for both researchers and practitioners. Finally, the derivative framework opens the door to transferring further derivative-based local BO methods to the preferential setting.

W2: Difficult to interpret experimental results.

We agree that the number of methods and settings makes the results hard to digest. We have used part of the additional page to explain more clearly which method performs best in which problem setting and to derive recommendations for practitioners at the end of Section 6.

W3: Insufficient motivation for cumulative regret.

We agree that the motivation was too brief. Cumulative performance captures the quality of the full optimization trajectory, which matters in controller tuning and policy search, where every query is executed on the real system and poor intermediate evaluations can be costly or even crash the experiment (Menn et al., 2026). This is further amplified in typical PBO use cases such as exoskeleton gait tuning for user comfort (Tucker et al., 2020), where the user experiences and judges every queried gait. We have expanded this motivation in the introduction and now consistently use the term cumulative performance throughout the paper.

AuthorsOpenReview ↗

Answer to Reviewer jcbZ Questions

Questions

Q1: Sample requirements for gradient estimation.

Previous observations are indeed used, as the pairwise GP is fit on all comparisons collected so far. The gradient estimate at the current center therefore also draws on nearby earlier queries, and the GI acquisition in GIPBO accounts for the uncertainty reduction these already provide. The choice M=dM = d follows GIBO, and subsequent analysis identifies d+1d + 1 observations as optimal for the corresponding local gradient-estimation problem (Wu et al., 2023). We agree that this is a substantial part of the evaluation budget, and our results show that it does not always pay off. Gradient-based methods are most appropriate when accurate local refinement is valuable. A natural mitigation is to stop adaptively once the gradient estimate is sufficiently accurate, as studied for GIBO (He et al., 2025), which could be transferred to preferential feedback. We have added the cost trade-off to the practical recommendations in Section 6 and adaptive stopping as future work in Section 7.

Q2: Clarification of the line search.

The paragraph only referred to the ablations in Appendix C.8, but not to Appendix B.4, where the line search is described. We have reordered the paragraph to first explain the line search with a reference to Appendix B.4, renamed it to "Line Search", and made the description in Appendix B.4 more explicit. Given the search direction ptp_t, we discretize step lengths along the normalized direction into 100 candidates. Following Thompson sampling, we draw one joint sample from the pairwise GP posterior over all candidates and select the one with the highest sampled utility. This selection uses the surrogate only. The selected point is then queried and compared to the current iterate xtx_t, and it becomes the next iterate if it is preferred; otherwise, PrefSQP moves to the local sample with the highest posterior mean among those preferred over xtx_t. While revising this part, we also corrected our PrefSQP implementation to match Algorithm 2 (see the overview of changes).

Q3: Choice of comparison points.

In all our methods, a new point is compared against an already evaluated reference point, which is the local incumbent in GIPBO, the current center in PrefSQP, and the global incumbent in TuRPBO. We agree that comparing two new points can be more informative per comparison. However, since we consider the number of experiments to be the main cost, comparing against an already evaluated point yields one comparison per new experiment, whereas comparing two new points requires two experiments per comparison. In addition, comparing against the reference point directly indicates whether the new point is an improvement, which our methods use to update the incumbent and, in TuRPBO, to adapt the trust region.

Q4: Multiple starting points for local refinement and terminology.

Starting local refinement from several promising points, similar to the multiple trust regions in TuRBO (Eriksson et al., 2019), is a possible extension. Under our budget it involves a trade-off, since the local phase only has about five gradient steps and splitting them across several starting points would leave few steps for each. Note also that the local phase reuses all comparisons from the global phase, so the information about other good points is retained in the surrogate. We have added this as future work in Section 7. Regarding the second point, the global phase indeed provides a single starting point, and "region" refers to its neighborhood, which the local phase then explores. We have clarified this wording in Section 5.4.

Q5: Unclear description of additional experiments.

Due to limited space, this sentence was too condensed, and it also mentioned the Sushi and CarCab benchmarks without introducing them. We have rewritten it as follows. "Among the global-to-local variants, we report G2L-PrefSQP in the main text. Results for G2L-GIPBO and G2L-GIPBOr as well as full experimental details are in Appendix C, with ablations in Appendix C.8. The appendix further includes the two low-dimensional PBO benchmarks Sushi (Kamishima, 2003), built from human preference data, and CarCab (Tanabe & Ishibuchi, 2020), a multi-objective design problem with a nonlinear utility (Appendix C.7)."

Q6: Meaning of GP sample-path problems.

By GP sample-path problems, we mean objective functions drawn as random sample paths from a GP prior with a squared-exponential kernel, with a new sample drawn for each run. This setup allows a within-model comparison (Hennig & Schuler, 2012), in which the surrogate uses the hyperparameters of the generating GP, as well as an out-of-model comparison with learned hyperparameters. We have clarified this in Section 6.

AuthorsOpenReview ↗

Answer to Reviewer jcbZ Minor Comments

C1: Reference to Garnett (2023).

We cite Garnett (2023) as a comprehensive and widely used textbook reference on Bayesian optimization. Following your suggestion, we have additionally added the seminal work of Jones et al. (1998) and the review by Shahriari et al. (2015).

C2: Incorrect use of "Instead".

We have removed "Instead".

C3: Ambiguous use of the preference operator.

We intended to use the matrix inequality Ht≺0H_t \prec 0 to indicate that the Hessian is negative definite, rather than to express a preference relation. To avoid ambiguity with our preference notation, we have replaced this expression with the explicit statement that HtH_t is negative definite.

C4: Unclear reference to Section 6.

This sentence was removed when we revised the paragraph.

Thank you for your careful response. I still don't think this is super exciting, due to the fact that you've plugged two things together in a very obvious way. But the improved clarity means I am happy to raise my score to a weak accept.

Reviewer_giv3OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates high-dimensional pairwise preference Bayesian optimization (BO). It extends local methods originally developed for high-dimensional standard BO to the preference-based setting, providing a systematic study of local preference BO. Although several local methods, including GIBO, BayeSQP, and TuRBO, have been proposed to address high-dimensional optimization in standard BO, they cannot be directly applied to preference BO. This paper focuses on adapting these three approaches to the preference-based setting.

Critical strengths
  1. The paper is well organized and presents a comprehensive and systematic study of an important research problem. The proposed extensions are well motivated, and I believe this is a valuable contribution to the field.
  2. The experimental evaluation is extensive, and the experimental settings and implementation details are clearly described. The theoretical analysis is also presented clearly, which helps readers understand the methodological foundations of the proposed approaches.
Critical weaknesses

Overall, I find this to be a well-executed and comprehensive piece of work. I did not identify any major weaknesses that substantially undermine the paper's contributions.

Questions for authors
  1. Choice of the Laplace approximation. In Section 4, the authors use the Laplace approximation for posterior inference. Is this the standard choice in preference BO, or have alternative approaches, such as variational inference (VI) or expectation propagation (EP), also been explored in this setting? I would be interested to know whether the authors considered these alternatives when designing their method.

  2. Consistency of the posterior mean and covariance expressions. In Section 4, Joint Conditional Distribution, the posterior mean expressions for �0��0�, �1��1�, and $μ∇2fpref(x)\mu_{\nabla^2 f}^{\mathrm{pref}}(x) $involve the term �3��3�. In contrast, the posterior covariance expression,$ Σabpref(x)\Sigma_{ab}^{\mathrm{pref}}(x)$, involves �5��5�. Could the authors clarify why these expressions involve different matrix terms and explain how they are derived under the Laplace approximation? I may have misunderstood some details of the derivation, so I would appreciate further clarification.

1 comment
AuthorsOpenReview ↗

Answer to Reviewer giv3

Thank you for the very positive assessment of our work and for the time invested in reviewing our paper. We address your questions below.

Q1: Choice of the Laplace approximation

Thank you for this question. The Laplace approximation is the standard choice in PBO (Chu & Ghahramani, 2005; Brochu et al., 2010) and is the model implemented in BoTorch's PairwiseGP. Our derivative framework only requires a Gaussian approximation of the latent posterior, so EP or VI would also be possible in principle.

We chose Laplace for two reasons. First, the GI acquisition in GIPBO needs the effect of a hypothetical comparison on the gradient covariance for every candidate. In the Laplace approximation, as in EP, each comparison enters the posterior precision as an explicit additive term, so a hypothetical comparison can be incorporated directly by adding this term and updating the MAP estimate. Standard VI lacks this structure and would require re-optimizing the variational objective for every candidate and outcome.

Second, Laplace converges in a few Newton steps and is available in standard tooling. EP would have been a viable alternative, but we opted for the established BoTorch implementation, which is also used by our qEUBO baseline and keeps the surrogate identical across methods.

Q2: Predictive mean and covariance expressions

We thank the reviewer for pointing this out. The different matrix expressions arise from the laws of total expectation and total covariance and are mathematically consistent.

Specifically, the predictive means are obtained by averaging the GP conditional means over the Laplace-approximated posterior, yielding the factor KXX−1f^K_{XX}^{-1}\hat{f}. In contrast, the predictive covariance additionally accounts for uncertainty in the latent training values. Applying the law of total covariance and the Woodbury identity gives

= (K_{XX} + \hat{C}^{-1})^{-1},$$ where $S = (K_{XX}^{-1} + \hat{C})^{-1}$ is the regularized Laplace posterior covariance and $\hat{C} = C + \varepsilon_{\mathrm{reg}} I$. The same derivation applies to the gradient and Hessian covariance blocks using the corresponding kernel derivatives. We have added this explanation and the intermediate matrix identity to Section 4 of the revised manuscript.
Reviewer_oA84OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adapts existing local search mechanisms to high-dimensional preferential Bayesian optimization, introducing GIPBO, PrefSQP, and TuRPBO for gradient-guided, second-order, and adaptive trust-region search, respectively. The methods use a standard Pairwise GP surrogate, with G2L variants for gradient-based methods that divide the query budget equally between global exploration and local refinement. Experiments on GP sample paths, standard benchmark functions, and policy-search tasks show improvements in best-seen and cumulative utility across several high-dimensional settings, although the benefits depend on task characteristics and initialization.

Critical strengths
  1. Targeted adaptation to preference feedback: Equation (14) averages over both possible comparison outcomes to account for their effects on posterior gradient covariance, extending the gradient-information acquisition objective to preference feedback.

  2. Evaluation of both outcomes and search quality: Under matched budgets, identical initializations, and ten repeated runs, the experiments report best-seen and cumulative utility while examining the effects of dimensionality, smoothness, and hyperparameter learning.

Critical weaknesses
  1. Limited methodological novelty and insufficient attribution of gains: The main designs adapt existing local BO mechanisms to preference feedback, while PrefSQP and GIPBO also differ in sampling, comparison opponents, and step-size selection, leaving the independent value of second-order information beyond localization insufficiently established.

  2. Mismatch between acquisition definition and execution: Equation (14) constructs hypothetical comparisons against the fixed center xtx_t, whereas Equation (15) updates the local incumbent used as the actual comparison opponent, without explaining how outcome probabilities and hypothetical posteriors are adjusted accordingly.

  3. Insufficient evaluation of final recommendation quality: The best-seen and cumulative utility metrics in Section 6 do not establish whether the algorithm can identify and recommend a high-quality solution solely from noisy preferences, leaving the quality of its actual output insufficiently assessed.

Questions for authors
  1. Independent value of the proposed mechanisms: Could the authors clarify the technical advances beyond adapting existing local BO methods to preference feedback and isolate the benefit of second-order information by replacing the Hessian with a scale-matched negative identity matrix while keeping sampling, comparison opponents, and line search unchanged?

  2. Consistency of the acquisition rule: When the local incumbent differs from xtx_t, could the authors specify Equation (14)'s outcome probabilities and hypothetical posteriors for the actual comparison opponent while keeping the gradient-estimation center fixed, and clarify which definition was used in the experiments?

  3. Quality of the actual recommendation: Could the authors report the utility of the final recommendation selected solely from preference data under matched query budgets, and independently reevaluate the final policies in Figure 4 through repeated rollouts to determine whether best-seen gains translate into recommendation quality?

1 comment
AuthorsOpenReview ↗

Answer to Reviewer oA84

We thank the reviewer for the time and effort invested in the review. The detailed and on-point questions helped us improve and correct the paper.

In the following, we address the stated weaknesses and answer the raised questions.

W1/Q1: Methodological novelty and value of second-order information

We ran the suggested ablation (Appendix C.8). We replace the Hessian by cIcI while keeping local sampling, comparison opponents, incumbent, and step rule unchanged, with cc chosen either to match the Newton step length or the mean curvature, and combine both with the Thompson-sampling line search and a fixed step.

With 2% noise, no variant differs significantly from the Hessian (paired Wilcoxon, Holm-corrected). Without noise, the Hessian improves the final value on Levy 32-d (by 26%) and on GP samples with ℓ=0.1\ell = 0.1 (by 12%). In contrast, the scaled identity performs better on Rosenbrock 8-d/16-d and Styblinski-Tang 64-d.

We have revised our claims in Sections 6 and 8 accordingly: second-order information can help when comparisons are accurate, but its benefit is problem-dependent, and under moderate noise a well-scaled gradient step performs as well.

Regarding the methodological novelty, while our methods build on established local BO principles, we derive the joint predictive distribution of the latent utility, its gradient, and its Hessian under the Laplace-approximated pairwise GP (Section 4). This is what makes derivative-informed and second-order PBO possible, and to our knowledge GIPBO and PrefSQP are the first PBO methods to use this information.

The adaptations also go beyond a direct transfer, for example the preferential GI acquisition, which has to account for the outcome-dependent precision term C^\hat{C}.

W2/Q2: Consistency of the GIPBO acquisition rule

Thank you for identifying this inconsistency.

Estimating the gradient well from preferences would ideally require comparisons between all pairs of local queries. To keep the evaluation fair with respect to the other methods, which use one comparison per evaluation, we changed the comparison mechanism to the one described in the paper, where each local query is compared against the local incumbent xt,i−1⋆x^\star_{t,i-1}.

However, we kept the acquisition function from the original mechanism, which forms hypothetical comparisons against the center xtx_t. The two coincide for the first local query and differ afterwards.

We have corrected this so that the acquisition uses the actual comparison opponent while keeping xtx_t as the location of the gradient uncertainty (Eq. 15, Algorithm 1).

Appendix C.8 compares both versions on Levy 16-d with 20 seeds. Using the incumbent improves:

  • Final best-seen value: median −59.3-59.3 vs. −64.0-64.0 (paired Wilcoxon, p=0.021p = 0.021).
  • Final incumbent value: −67.7-67.7 vs. −74.6-74.6 (p=0.076p = 0.076, not significant).

W3/Q3: Quality of the final recommendation

We agree that best-seen performance alone does not show whether a method can identify a good point from preferences.

We now additionally report the true value of each method's incumbent, identified from comparisons alone (Appendix C.9, Figures 18–21).

The results show that the rankings are largely preserved. The ranking of the methods is identical in 14 of 24 problems and otherwise differs by a single swapped pair of methods (two in one problem; Spearman correlation at least 0.93). The best method is the same in 23 of 24 problems.

For the policy-search tasks, we re-evaluated the final recommendation of every run on 20 held-out reset seeds (Table 10). The held-out distance is on average 1.5% below the optimization objective for both robots, and the best method is unchanged in 7 of 8 tasks. We note that cumulative performance, our second metric, is not affected by this issue, as it measures the quality of all evaluations made during optimization.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of Changes

We thank all reviewers for their constructive feedback. We have uploaded a revised version. The main changes are:

  1. Consistent GIPBO acquisition (Reviewer oA84). The acquisition now uses the actual comparison opponent (Eq. 15, Algorithm 1), with an ablation in Appendix C.8.

  2. Value of second-order information (Reviewer oA84). New ablation of the posterior Hessian vs. a scale-matched identity (Appendix C.8), and correspondingly revised claims in Sections 6 and 8.

  3. Quality of the recommended point (Reviewer oA84). The true value of each method's incumbent for all experiments (Appendix C.9) and held-out re-evaluation of the final controllers (Table 10).

  4. Policy search. We replaced the policy-search experiment. The submitted objective (Gym reward over 300 steps, one random episode) was dominated by the alive bonus and did not require walking. We now adapt robust ARS controllers (Hopper, 33-d; HalfCheetah, 102-d) to changed robots, maximizing the forward distance over five fixed episodes with 2% preference noise. Walker2d was replaced by HalfCheetah because no robust Walker2d controller could be obtained. Final recommendations are re-evaluated on 20 unseen episodes (Reviewer oA84, Q3).

  5. Noise. The main experiments now use 2% observation noise instead of 10%, which was not applied correctly in the previous version. At 10% noise, 25–44% of the comparisons made by the local methods are wrong (Table 16), so the results largely reflect how strongly the comparisons are corrupted rather than how well the methods search. To make the effect of the noise level transparent, Appendix C.10 compares all methods on Levy 32-d across noise levels from 0 to 20%, and Table 16 reports the resulting fraction of wrong comparisons. TuRPBO performs best or among the best at every noise level, while the other local methods, which compare nearby points with small differences in utility, are more affected by noise than global methods such as qEUBO. The Sushi benchmark additionally tests all methods under strong preference noise (Appendix C.7).

  6. Clarifications. Derivation of the predictive covariance (Section 4, Reviewer giv3), the PrefSQP line search (Section 5.2, Appendix B.4, Reviewer jcbZ), the motivation for cumulative performance (Introduction, Reviewer jcbZ), the GP sample-path setup and the warm-starting description (Sections 5.4 and 6, Reviewer jcbZ), practical recommendations (Section 6), and future work (Section 7, Reviewer jcbZ).

  7. PrefSQP implementation. While revising the paper, we found that our PrefSQP implementation deviated from Algorithm 2: the iterate was reset to the point with the best noisy observation after each iteration, and the GP was refit after every local query. We corrected the implementation to match the description. The iterate xtx_t stays fixed while its dd local samples are compared to it, so the gradient and Hessian are evaluated where the samples were drawn. The GP is fit once per iteration. The line-search point becomes the next iterate only if it is preferred over xtx_t; otherwise, PrefSQP moves to the preferred local sample with the highest posterior mean. All PrefSQP and G2L-PrefSQP results were rerun. Performance on short length scales became slightly worse, while performance on longer length scales improved. Result changes on the synthetic functions are mainly caused by the adapted noise, which now corrupts local comparisons more.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.