- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper combines expensive experimental evaluations with pairwise comparisons from a subject expert with the goal of optimizing a known utility function over an unknown objective , where the structure of is known and assumed to be linear. Authors propose a mechanism for efficiently combining the two sources of information, such that there maybe misalignment in the projected preference model and the actual objective . Authors provide experimental analysis across a range of tasks and compare with a range of baselines as well to demonstrate empirical merit of approach.
Critical strengths
- As opposed to naive preference optimization works, authors make a distinction between the emergent model from pairwise data and the actual function to be optimized. This is potentially very useful in settings where there is a lot of cheap pairwise data, such as self driving applications, where user data maybe available in the form of feedback, and actual tests are extremely costly.
- Authors have provided an extensive literature review, that captures several existing domains that ask related questions, and the modeling framework also leverages inspiration in the right way from these fields, such as multi-fidelity BO and preferential optimization, resulting in a very intuitive mathematical framework.
Critical weaknesses
- A strong limitation of this work is lack of actual expert in experiments for generating reliable pairwise data. Authors rely on synthetic framework for pairwise generation, while it is understandable that human studies may not be possible, it is fairly straightforward to use an LLM as a proxy source of cheap pairwise data, or leverage existing human datasets. In the absence of human studies, LLM proxy or use of existing datasets/benchmarks to construct pairwise comparisons, it is difficult to verify the practical merit of this work.
- It seems like an inefficient operation to use a global in modeling the correlation because in regions where the data is locally aligned, good pairwise demos will remain less trustworthy as a consequence of low global , similarly high overall correlation does not imply locally sound correlation, disturbing the accuracy of model. Please see questions for more clarification needed on this and other topics.
Questions for authors
- Can authors provide successful replication using real world pairwise data to demonstrate the robustness of framework? If not, what remains the key limitation in doing so, and what can be said about real world applicability from synthetic data?
- Does a global suffice in modeling the misalignment between pairwise and real data? What motivates a global instead of a spatially informed one?
- In the experiment results in Figure 1 and 2, UCB and logEI outperform the approach as they remain uncontaminated by the misaligned pairwise data, this raises questions about whether the pairwise data is being used effectively at all or not. Can authors clarify if this is due to the global used in modeling, or is there another reason?
4 comments
Weakness 1 & Question 2: Modelling the spatially variable correlation
The Reviewer suggests that using a global correlation parameter is inefficient, and asked why we chose it instead of a spatially informed one. To address this issue, we extended EA-BO so that it can model a spatially varying correlation; see Appendix F.2 (for output-specific correlations, see Appendix F.1 and our response to the Reviewer aYBp).
Specifically, we model the expert reliability by the spatially varying correlation parameter , where has a GP prior with a linear kernel and is a probit function. We tested the extended model on the Hartmann6 data set with a linear with true reliability at the neighborhood of the global optimum and at the neighborhood of the local optimum (and the reverse setting).
Including the spatially varying correlation in the model slightly improved the optimization performance (see table below for mean AUCs), suggesting the utility of enabling the correlation to depend on the input when an expert that is reliable in one region of the design space, but unreliable in the other.
Mean AUC for EA-BO wt. G and L denote the global and local optima.
| Problem | True reliability | EA-BO (global ) | EA-BO (local ) |
|---|---|---|---|
| Hartmann6 | 0.777 | 0.791 | |
| Hartmann6 | 0.610 | 0.652 |
The reason that we originally modelled the correlation parameter by a global constant is that the objective in BO is optimization, not parameter estimation. Instead of spending a too large proportion of the budget towards learning a complicated correlation structure, the EA-BO with the global just quickly learns whether the expert is in general reliable or not, and allocates the rest of the budget accordingly.
However, the above results demonstrate that—as the Reviewer suggested—estimating the spatially varying correlation structure can improve the optimization performance, at least when the correlation structure is simple enough so that it can be learned within a limited budget. Thus, we have included the extended version of the EA-BO that can model the spatially varying into the revised article.
Question 3: The performance of EA-BO when the expert is misaligned
The Reviewer asks why the evaluation-only UCB and logEI outperform EA-BO in the misaligned results of Fig. 1 and Fig. 5, and, specifically, if this is due to pairwise data being used inefficiently or due to EA-BO using a global correlation parameter .
When the expert is aligned (), EA-BO outperforms the evaluation-only methods UCB and logEI, since its mixed (both comparisons and evaluations) strategy can utilize cheaper and informative comparisons. However, when the expert is misaligned (), the evaluation-only strategy is optimal, and thus the performance of the UCB and logEI is the upper limit: it is not possible to use pairwise queries efficiently, since they are completely uninformative. Further, having a spatially varying in the model is not useful, since the true is globally zero.
In the case of the misaligned expert, EA-BO is able to quickly learn that the expert is not reliable, and thus switch from the mixed strategy to the optimal evaluation-only strategy: the small performance difference between the evaluation-only methods and EA-BO is equal to the proportion of the budget that EA-BO has to spend for detecting an unreliable expert. EA-BO outperforms the earlier HITL BO method CoExBO (Adachi et al., 2024), which uses uncertainty weighting and scheduled decay to handle unreliable experts instead of estimating reliability.
References
Ahneman, Derek T., et al. "Predicting reaction performance in C–N cross-coupling using machine learning." Science 360.6385 (2018): 186-190.
Boobier, Samuel, Anne Osbourn, and John BO Mitchell. "Can human experts predict solubility better than computers?." Journal of cheminformatics 9.1 (2017): 63.
McCoy, Stephen A., et al. "Large language models as proxies for human experts in synthesis optimization." Digital Discovery 5.10 (2026): 4058-4062.
Thanks for answering the questions and weaknesses. I appreciate the authors looking into a spatially variable correlation. While I appreciate the inclusion of human and LLM expert results, they amplify my concerns pertaining to performance of EA-BO compared to evaluation-only baselines. Overall, the question of how to incorporate an expert's pairwise judgement is difficult to answer universally precisely because the degree of alignment can vary considerably across tasks.
In the newly added results, the Buchwald–Hartwig: briefed LLM result seems to be the only one where EA-BO clearly outperforms evaluation-only baselines (please correct me if I am wrong). To fundamentally resolve this issue, we either need (1) better modeling paradigms (for eg the spatially variable correlation and potentially more) to combat the sensitivity of the approach to an unaligned expert, or, (2) we need a theoretical explanation supporting that EA-BO is indeed able to quickly learn when an expert is not reliable, without wasting too many queries. The present scope of the paper offers neither, and the spatially variable correlation is included as an additional ablation in the paper, whereas in my opinion, it should be a standard choice, and authors must explore the issue of robustness to weakly aligned experts in greater detail. I agree that the objective in BO is optimization and not parameter estimation, but in the absence of good quality data, parameter estimation can be essential in informing an accurate cost landscape to optimize over, because currently the expert data maybe corrupting the optimization globally. In the absence of either strong modeling choices, empirical results supporting robustness to expert data, or theoretical explanation (for eg regret analysis explaining the fallback to evaluation-only paradigm for weakly aligned experts), I would keep my original score. I am open to increasing the score if the authors believe the paper addresses one of these issues clearly and can point me to specific parts in the paper for that.
Thank you for your thorough response. We have revised the article to address two out of three main concerns mentioned in your response (Providing additional theoretical results was unfortunately not feasible within a remaining day of the author response period.). In particular, we have: (1) strengthened modelling choices by making the spatially variable correlation model the standard model (See Section 3 for the updated model definition, Figure 1–3 for the updated main results, and Appendix F.2 for the ablation comparing the original and updated model.); (2) provided empirical results supporting robustness to data from weakly aligned experts (See Appendix E.6.).
Thank you for insisting on these revisions. They make the article stronger. In particular, modelling the spatially variable correlation improves the optimization performance when the true reliability is spatially variable (Figure 20, Table 20), while causing no observable performance penalty when it is global (Figure 19).
Update: the model with spatially varying correlations is now the standard choice
As you suggested in your response, we have now revised the article so that modelling a spatially variable expert reliability parameter is the default choice (See Section 3, where we define the model.). We have re-ran the main experiments of the article using the spatially variable version (Figures 1 and 3) and added a comparison where also the ground truth reliability is spatially variable (Figure 2).
See also the updated Appendix F.2., where we compare the performance of the original model and the updated model. When the true reliability is globally constant, there is no observable performance penalty from using a more complicated spatially variable model (Figure 19). However, when the true reliability is spatially variable, the updated model is able to leverage it to outperform the original model, in particular on the Hartmann6 data set (Figure 20 and Table 20).
Empirical results supporting robustness to data from weakly align experts
We provide additional empirical evidence that EA-BO is able to quickly learn when an expert is not reliable, without wasting too many queries in the regime where the expert is weakly aligned (see Appendix E.6).
We have reproduced the main results in the table below. Entries are mean comparisons performed by EA-BO (the updated version with spatially variable correlation parameter) in successive intervals of 20 budget units. When the expert is misaligned, EA-BO buys significantly fewer comparisons already in the second interval. By the third interval, i.e. into the total budget, only few unreliable comparisons are bought. Thus the detection and identification happens while there is substantial optimization budget left.
Further, Figures 7–10 indicate that the drop in the amount of queries performed coincides with the drop of the posterior mean of the reliability parameter , verifying that the algorithm works in a principled fashion.
| Problem | Expert / true reliability | 0–20 | 20–40 | 40–60 | 60–80 |
|---|---|---|---|---|---|
| Ackley5 | Constant, | 10.0 | 16.4 | 12.7 | 11.5 |
| Ackley5 | Constant, | 10.0 | 0.0 | 0.0 | 0.0 |
| Hartmann6 | Constant, | 10.0 | 20.0 | 14.4 | 7.7 |
| Hartmann6 | Constant, | 10.0 | 8.2 | 1.7 | 0.2 |
| B-H | Briefed LLM | 10.0 | 18.0 | 18.0 | 14.4 |
| B-H | Unbriefed LLM | 10.0 | 12.8 | 3.5 | 1.1 |
Could previously collected comparisons continue to harm optimization?
In Figures 14-16 we examine the effect of removing the comparisons from EA-BO’s dataset and computing the exact posterior using only the evaluations. This allows us to measure the incurred harm from allowing misspecified data into the dataset.
We see that in all of the misaligned cases the incurred harm is close to , suggesting that the only negative effects from querying a misspecified expert are the opportunity cost of not spending the budget spent on queries on evaluations instead and the locations of bought evaluations. Thus, there is negligible residual harm from the misalignment to the posterior.
Thank you for your review. We have uploaded the revision in which we address both of the critical weaknesses you pointed out: (1) To verify that EA-BO works also on real preferential data, have conducted additional experiments where the comparisons are provided either by an human expert or by an LLM expert (Section 6.1., Figure 6, Appendix E.2), as you suggested. (2) To enable modelling the expert reliability ρ locally, we have extended EA-BO by adding a spatially varying correlation parameter, and verified experimentally that this extended version can learn and utilize a simple spatially varying correlation structure (Appendix F.2).
Weakness 1 & Question 1: Replication using real-world pairwise data
The concern of the Reviewer was that the lack of actual expert in experiments for generating reliable pairwise data. The Reviewer suggested either using an LLM expert or existing human datasets to generate pairwise comparisons.
We tried both of these suggestions by adding two tasks: (1) Optimizing the solubility of drug-like molecules (Boobier et al., 2017). The pairwise comparisons are based on the estimates of the two human experts collected by Boobier et al. (2017). The average reliabilities of these experts were 83 and 62. (2) Optimizing the catalytic activity for Buchwald-Hartwig reaction (Ahneman et al., 2018). The pairwise comparisons are provided either by an LLM that is briefed by 80 measured reactions from a previous related experiment, or by an unbriefed LLM, following the approach by McCoy et al. (2026) (see Section 6.1 and Appendix E.2. for additional details).
See the table below of the summary of the results. The full results can be found in Figure 6 and Table 5. On the solubility task, there was not a significant difference between EA-BO and the evaluation-only methods. EA-BO outperformed the comparison-only methods for the less reliable expert (ρ=0.62).
The results on the Buchwald-Hartwig data set suggest that an LLM can be used instead of the human expert to improve the cost-efficiency of BO: when the LLM was briefed, both EA-BO and the comparison-only methods outperformed the evaluation-only methods. This means that the LLM expert alone outperformed the experiments-only methods. When the LLM was not briefed, it acted basically as an unreliable expert, and thus the comparison-only methods failed. However, as with the synthetic data sets, EA-BO was able to identify the unreliability of the expert, and, consequently, to reach the similar final utility as the evaluation-only methods.
Mean AUC / Final utility on realistic-feedback settings. Solubility percentages denote expert comparison accuracy.