- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies preferential Bayesian optimization (PBO) with the standard BoTorch setup. That setup fits a PairwiseGP using the Laplace approximation and picks queries with EUBO. The authors point out that EUBO usually picks two brand-new points for each query. As a result, the comparison graph splits into many disconnected pieces. They show that the likelihood Hessian is a weighted graph Laplacian whose rank drops by one for every disconnected piece. They argue that this makes the full Hessian ill-conditioned. Their fix, KappaSharp, adds a diagonal penalty to the MAP problem. The penalty is larger at points where the prior precision is low. Its overall strength is chosen so that the condition number of the Hessian drops to a target value. The penalty changes only the MAP estimate, not the covariance that the acquisition function uses. An adaptive variant turns the correction on only when the model has recently been confident about its comparisons. The paper supports the method with several propositions about the graph structure, the correction and its effect on EUBO. The experiments cover 11 benchmarks with 5 to 20 dimensions, including a plasma controller tuning problem. On the main initialization, the adaptive variant wins on 5 benchmarks, ties on 6 and loses on none against the standard setup.
Critical strengths
- The paper makes a clear and useful observation. It links EUBO's habit of picking fresh pairs to the Laplacian structure of the likelihood Hessian, and Figure 1 explains this link well. I think people who work with PairwiseGP will find it helpful. The fix is also cheap and easy to add to existing code. It only touches the MAP step and adds little runtime.
- Few BO papers use 60 paired seeds with common random numbers, five initialization schemes and 16,500 runs in total. The ablations in Table 2 are well designed. They separate the effect of moving the MAP from the effect of changing the Newton step or the covariance. The authors also report the losses of the static variant openly.
Critical weaknesses
- The paper's explanation of why the method works doesn't match its own evidence. The paper says a rank-deficient likelihood Hessian makes ill-conditioned, and that KappaSharp fixes this. But is full rank, so the missing likelihood directions don't make ill-conditioned on their own. In those directions, the posterior just falls back to the prior. The paper itself says this is the correct answer (Proposition 2 and Theorem 7).
- The ablations also point the same way. If conditioning were the problem, the MetricOnly variant should help, since it uses the corrected Hessian only to improve the Newton steps. Instead, it gets 1 win and 7 ties. The gains appear only when the MAP itself moves. So KappaSharp is really a change to the model rather than a numerical fix. It fits the mean under a stronger prior that pulls isolated points harder toward zero. Then it pairs that mean with the covariance of the original model. The result is not the posterior of any single model. The paper doesn't explain why this mix should lead to better queries. It also doesn't explain why the method hurts on Ackley-8D and never shows that is actually large in these runs either.
- The theory also does not address this gap. Propositions 4 and 5 are basic calculus. while the bound in Proposition 6 can never exceed . Theorem 9 compares MAP-only with full augmentation, not with the baseline. Its bound also grows as shrinks, which is exactly the ill-conditioned case the paper cares about.
- The experiments don't yet show the method beats reasonable alternatives. Almost all baselines are variants of KappaSharp itself. The connected-querying baseline fixes one candidate to the previous query point and loses badly. That makes me wonder whether it was set up fairly. The natural version fixes the previous winner, and BoTorch's analytic EUBO has a previous_winner option for this. The EP and Gibbs baselines use EI, which the authors admit is a poor fit.
- Simple alternatives that adjust the same knob are also missing. Examples are tuning the GP output scale or its prior, or running the adaptive rule with a uniform diagonal penalty. The uniform diagonal penalty already gets 3 wins and 2 losses, including +30.7% on DTLZ2-16D. So it isn't clear that the prior weighting is needed.
- The scoring makes the results hard to interpret. The metric is the percentage change in raw final utility, which depends on where the utility's zero point happens to sit. On Hartmann-6D, the known optimum is 3.32. Measured as regret against that optimum, the adaptive variant's reported −0.9% is about 7% worse than the baseline, and the static variant's −4.8% is about 39% worse. The win, loss and tie counts come from 55 uncorrected t-tests, so a few of the 18 wins could be chance.
- The paper doesn't say how the adaptive rule's four constants were chosen, and the sensitivity checks reuse the reported benchmarks. Finally, the abstract says "up to +10.9%" while Table 1 shows +24.0% on Levy-10D. The conclusion also calls the gains consistent, even though 6 of 11 benchmarks show no significant change.
- Please make the abstract and the conclusion match Table 1. The DTLZ2 problems are reduced to a single utility, so calling them multi-objective is confusing. DTLZ2 also comes from Deb et al. (2002), not González et al. (2017). Appendix G says the plasma utility is judged by human inspection, but the experiments use a simulated oracle, so please state clearly which one you used. Since every run uses σ = 0.1, it would help to test other levels of comparison noise as well.
Deb et al., "Scalable multi-objective optimization test problems", CEC, 2002.
Questions for authors
- Can you plot and the number of connected components over a run for the baseline? Can you also show that the shifted MAP gives a better surrogate, for example by ranking held-out points better, measured by Kendall's tau against the true utility?
- Is there a model whose posterior matches the MAP-only mix of mean and covariance? Did you run full augmentation, with R in both the MAP and the covariance, in the experiments? If so, how did it do?
- In connected querying, which point stays fixed? Is it the previous winner or one of the two previous points? How does the method compare when the previous winner is fixed?
- How does the adaptive rule perform with a uniform diagonal penalty? How does the baseline perform with a tuned or larger output-scale prior?
- Do your conclusions still hold if you report normalized regret against the known optima and correct for multiple testing?
- How did you choose , , the EMA rate and the minimum step? Did you test these values on any benchmarks you didn't also report?
- Why did you use BoTorch 0.6.0 and PyTorch 1.11? PairwiseGP has changed quite a bit since 2022, so I'd like to know whether the results hold on a current version.
8 comments
Response to Reviewer v87r (2/2)
Previous winner (Q3). The submitted connected querying followed Xu et al. (2024) and kept the new candidate of the previous query. The revision uses your version (App. M, p. 28): the winner of the last comparison is kept until it loses, and BoTorch's analytic EUBO chooses the other candidate given this winner (30 seeds, 6 benchmarks). At the same number of queries it is not significantly better than the baseline on any of the six, and PWR has a lower mean regret than it on all six. Counted in evaluated candidates it is competitive and significantly better than PWR on DTLZ2-20D and ZDT3-10D (uncorrected; Table 13, p. 29).
EP and Gibbs with expected improvement (EI). We replaced these with three EUBO variants with the same priors and seeds (App. N, p. 29): EP, and the exact posterior with the EP hyperparameters or with the baseline's own. With the baseline's hyperparameters, the exact posterior changes the regret significantly after Holm correction on no benchmark, and PWR has a lower regret than all three on DTLZ2-16D. The EP fits show the same collapse.
Outputscale and uniform correction (Q4). On development seeds, the outputscale prior SmoothedBox(0.25, 1) improves DTLZ2-16D by 3.6 points but worsens ZDT3-10D by 17.3, and SmoothedBox(4, 16) changes no benchmark significantly except Ackley-8D (-11.0, before correction; App. L, p. 28). A uniform with the same strength rule and refit is worse than PWR on DTLZ2-16D (+5.0) and DTLZ2-20D (+5.8), significantly after Holm correction, and better on ZDT3-10D (-10.7, significant only before correction; Table 3). With the same activation rule, uniform minus weighted is +3.7 on DTLZ2-16D (significant after Holm correction) and +4.3 on DTLZ2-20D, and no benchmark favors the uniform correction significantly (30 seeds; App. I, p. 26).
Scoring (Q5). We now report the normalized remaining regret , with mean curves (App. G, p. 23) and, at iteration 200, paired 95% intervals and Wilcoxon tests with Holm correction over the 13 benchmarks (Table 2); we removed the win/loss/tie counts. After Holm correction, DTLZ2-16D and DTLZ2-8D improve and Hartmann-6D, measured against the optimum as you suggested, is worse (+1.6 points, ). The mean regret is lower on all five problems with 14 to 20 dimensions (pooled Wilcoxon ).
Constants and sensitivity (Q6). We chose , the fixed outputscale, MAP-only and the correction at every iteration on development seeds; the first corrected iteration, , and the exponential moving average (EMA) weight of the gated version, 0.2, were not chosen on them. The 13 benchmarks (6 to 20 dimensions) include a robot-pushing simulator and a plasma-medicine problem. Every choice used only reported benchmarks, and five were not run on development seeds: DTLZ2-8D, RobotPush-14D, CarCab-7D and the new Styblinski-Tang-10D and Rosenbrock-10D (App. C, p. 19). On these five, PWR gains on DTLZ2-8D (significant after Holm correction) and loses on Styblinski-Tang-10D (significant only before correction); the other three are not significant. Vehicle-5D from the submission is shown only in App. G: 97% of the runs of both methods reach zero remaining regret within 20 iterations. App. K (p. 27) varies from 0.01 to 0.1, the first corrected iteration and, for the gated version, the threshold and the EMA weight, on test seeds 0 to 29, where the Plasma-16D gain is -7.7 (-3.3 over all 60). With another first corrected iteration (5, 15, 25) or EMA weight (0.1, 0.4), only one of 35 uncorrected tests differs significantly from the default. Every setting improves Plasma-16D and DTLZ2-16D significantly before correction, except one gated setting on each, and the losses on Styblinski-Tang-10D and Hartmann-6D grow with . We set the gated threshold without regret values, as the value at which the rule was on in about 40% of iterations in preliminary runs (App. I).
Abstract, conclusion, DTLZ2, Plasma and noise. The abstract and conclusion match Table 2, and Sec. 7.2 and the limitations state the losses. DTLZ2 is described as several outcomes that a piecewise-linear utility compares with a target, and is cited to Deb et al. (2002). The Plasma-16D comparisons are simulated from the utility, with no human in the loop (App. C). App. J (p. 26) tests lower and higher noise: with higher noise, PWR has a lower mean regret on ZDT3-10D and CarCab-7D (significant on CarCab-7D before correction), a similar one on DTLZ2-16D and no significant change on Plasma-16D.
Q7. BoTorch version. All revised results use BoTorch 0.18.1 and PyTorch 2.15.0 (App. C). We built the submission on the code of an earlier project of ours that used BoTorch 0.6.0 and PyTorch 1.11. The submitted results did not hold at 50 queries; item 3 of the general response explains why and why we now use 200.
We would be grateful for a reassessment of the revision.
Reply to Authors
Thank you for the revision. The authors replaced the conditioning explanation with lengthscale collapse, added the baselines I requested (previous-winner querying, output-scale priors, uniform penalty, lengthscale floors, EP, and exact posterior), switched to normalized remaining regret with Holm correction, and reported the losses. These changes address my main concerns.
However, two questions remain for me:
- The surrogate does not rank points more accurately (Kendall’s tau is mixed and often worse near the best points). The gains therefore come only from changes in the pairs that EUBO selects. The paper's factorial experiment shows that the refitted lengthscales pull the queries closer together while the shifted MAP pushes them farther apart, but the authors do not yet explain why these new pairs lower regret.
- MAP-only is not the posterior of any single model, and full augmentation performs similarly. It is therefore unclear whether the benefit comes from a deliberate bias in the mean or from an adjustment whose effect we do not fully understand.
If the authors can address the remaining questions above, I will consider increasing my score.
Response to Reviewer v87r (1/2)
Thank you for a detailed review. We agree with the central point of the comments, and addressing these comments led to the revision described in the general response. Results are for the revised method, Precision-Weighted Refit (PWR), on BoTorch 0.18.1 with 200 queries and 60 paired test seeds, unless stated otherwise.
Explanation vs. evidence, and κ (Q1, first part). The revision replaces the conditioning explanation with the lengthscale collapse of Sec. 4.2. Fig. 2 plots, for baseline runs of four benchmarks (test seeds 0 to 4), the number of connected components, κ(H), the largest and smallest eigenvalues of H, and the smallest distance between candidates. The median κ(H) reaches 1011 to 1016, but through λmax(K−1); λmin(H) falls by less than two orders of magnitude and stays bounded by the prior, as you argued. The revision uses Proposition 2 to show that, under EUBO and given the within-component differences, the Gaussian process prior decides the relation between components through its lengthscales, the shortest of which collapse (Sec. 4.2, Table 1).
MetricOnly: a change to the model. We agree. The revision describes PWR as a change to how PairwiseGP is fitted, with the effect running through the lengthscale refit (Sec. 5.2), and drops MetricOnly. Without the refit the gain disappears on DTLZ2-16D, DTLZ2-20D and Ackley-8D (Table 3, relative to PWR: +7.8, +7.7 and +11.7 points, all significant after Holm correction).
Not the Laplace posterior of a single model (Q2). Sec. 5.3 states this, and we know of no model whose posterior is the MAP-only mix. Full augmentation, with R also in the precision, is the Laplace posterior of the probit model with prior precision K−1+R, up to the clipping in BoTorch. We ran it on all 13 benchmarks (App. B.8, p. 18): minus PWR, it is significant after Holm correction only on CarCab-7D (-1.8 points, full augmentation better); on ZDT3-10D it is -11.1 (not significant after correction), and on the other eleven benchmarks the two are within 1.8 points. We chose MAP-only on development seeds by a rule fixed in advance (App. C, p. 19).
How the mix changes the queries. Sec. 6 tests this with a factorial experiment on baseline states (eight benchmarks, test seeds 0 to 19; Table 7, p. 22). The refitted hyperparameters make EUBO choose points closer to the observed candidates and to each other on every benchmark. R in the MAP estimate does the opposite. One reason: EUBO is positively homogeneous of degree one in the posterior means and standard deviations, so shrinking all means by a common factor gives the same maximizer as inflating all standard deviations by its inverse; we expect the non-uniform shrinkage by R to act similarly. With both of its changes, PWR chooses a tighter pair than the baseline on Plasma-16D and DTLZ2-16D and points farther from the observed candidates on ZDT3-10D, Levy-10D, Ackley-8D and Hartmann-6D. R in the precision has smaller effects.
A better surrogate? (Q1, second part). Not consistently. On the data of baseline runs (test seeds 0 to 9), Kendall's rank correlation with the true utility changes by -0.15 to +0.07 globally and is lower near the best points on four low-dimensional problems (uncorrected Wilcoxon p<0.05; Table 9, p. 23). The gains come with changes in where EUBO queries, and a complete account of why these changes lower the regret is still open.
Ackley-8D. The mean now improves (20.4% to 10.9%) because the baseline ends above 40% remaining regret on 19 of the 60 seeds (PWR on 6), but PWR is worse on 60% of the seeds. On Ackley-8D and Hartmann-6D, PWR explores more and refines less (Tables 7 and 8). Hartmann-6D is the one significant loss after Holm correction (+1.6 points). Sec. 7.2 states both losses, and the limitations state the one on Hartmann-6D.
Theory. Submitted Propositions 4, 5 and 6 and Theorem 9 are now Propositions 5, 6 and 8 and Theorem 11 in App. B (pp. 16 and 17), as properties of the correction. Proposition 8 bounds the shift of the MAP estimate relative to ∥f^0∥Hˉ, and the bound of Theorem 11 is of order η/λmin(Hη), as you note, and we use neither as an argument for the method. App. B.8 states that its results do not show that MAP-only is better than the baseline or than full augmentation; the experiments decide this. The main-text theory is Propositions 1 and 2 and what they imply for the kernel.
Baselines beyond variants of the method. The revision adds lengthscale floors of 0.2 and 0.3, freezing the hyperparameters after 50 iterations, a random second point in every fifth query, two outputscale priors, previous-winner querying, expectation propagation (EP) and an exact posterior (Table 3; App. L, M, N). The floors and the freezing worsen ZDT3-10D by about 9 points, and the floors do not reproduce the gain on DTLZ2-16D (-0.1 and -0.5 points).