acceptodds
Withdrawn from ICLR 2027

CALIPER: EVIDENCE-CALIBRATED MULTI-EXPERT LLM PRIORS FOR MULTI-OBJECTIVE BAYESIAN OPTIMIZATION

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

External priors can accelerate Bayesian optimization (BO) precisely when evaluation budgets are too small to recover easily from bad guidance. LLM committees make this opportunity inseparable from a reliability problem: expertise varies by objective, multiple experts may share the same bias, and stated confidence may not predict accuracy. To address this issue, we introduce Counterfactual Adaptive LLM-Integrated Prior Evidence Replay (CALIPER), a plug-in calibration layer for multi-objective BO that learns two complementary forms of trust from newly observed objectives. Relative reliability routes each objective toward the experts that predict it well; absolute reliability controls whether the committee should influence the optimizer at all. CALIPER combines objective-wise reputation, counterfactual validation of confidence, prior-amplitude control, and an explicit no-prior arm. It injects the resulting signal as a residual-GP mean shift, preserving the downstream qLogEHVI/qLogNEHVI acquisition pipeline. We evaluate every prior-control strategy with the same frozen LLM information and a locked, disjoint development/evaluation protocol on ESOL, FreeSolv, and Lipophilicity. The real-prior benchmarks show an asymmetric payoff: CALIPER captures useful guidance on ESOL and Lipophilicity while recovering the no-prior final hypervolume on FreeSolv, where unconditional injection causes negative transfer. Controlled stress tests isolate the mechanism: CALIPER gains 8.1% trajectory AUC when all experts are useful, routes objective-specialized expertise for a 6.3% gain, and stays within 0.6% of no-prior BO when every expert is misleading, compared with a 10.1% loss from fixed injection. These results establish evidence calibration as a practical interface for extracting value from uncertain external knowledge without binding the optimizer to it.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian optimization, multi-objective optimization, large language models, expert priors, evidence calibration, uncertainty calibration, Gaussian processes

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_69WrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is concerned with constructing a strong prior distribution for multi-objective Bayesian optimization over m criteria. The construction is based on E experts, considered frozen (and usually given by prompts to an LLM). Each expert e provides a score mu_{e j}(x) for candidate x and criterion j, as well as some confidence statement c_e(x). Neither is necessarily well-calibrated. The work aims to turn this input into a useful prior, which is used in two ways. First, it provides a prior mean function which is added to the residual (prior 0 mean) GP. Second, it provides some "counterfactual" confidence estimate, based on which the overall confidence/reliability of experts is judged and taken into account.

There is quite some prior literature on prior construction for BO, also MOBO, and what is proposed here is nothing very novel given. The presentation of this work is really quite poor, which makes it hard to say much more. From a lot of undefined lingo over undefined notation to totally unmotivated expressions (containing sigmas in sigmas, softmax, clipping, and a large number of free parameters, which the authors likely tuned until they got good results). The presentation does essentially nothing to structure this development in any sensible way or guide the reader.

Critical strengths

The overall problem addressed here is potentially important and timely.

I cannot detect any other strength of this work.

Critical weaknesses

The paper has a poor presentation. It uses a lot of undefined lingo like "markets, capital, reputation, market trust, real-market capital, and even "shadow capital". It is not even completely clear what the inputs are. I guess these are mu_{e j} and c_e. But where are K_{e j, t} coming from? They are then combined in some complex, unmotivated expressions, involving lots of things from the ML toolbox, like sigmas nested in sigmas, clipping, or softmax. The chain is xi_j -> alpha_{e j, t} -> v_{e j} -> bar{mu}_j, chi_j -> G_j. This ends up with 3 "arms" based on using G_j with different values for d_e(x). Just from this vocabulary (!!), I am guessing there is some bandit vote involved here. Crucially, it is certainly not clear from the writing why this complex construction would amend anything about the inputs possibly being uncalibrated, biased, or whatever else the authors are saying in their overall motivation.

At a later point (maybe also in the Appendix), they talk about "shadow", and in the abstract I see the term "replay". None of this is explained or connected. Reading this is quite a frustrating exercise.

And by the way, the expression come with a large number of free parameters:

  • a_xi, tau_xi
  • T
  • alpha_min, alpha_max
  • eta, lambda, eta_p, ...
  • s_min, ...

In Table 5, the authors list the values they chose for all these many free parameters. I have no doubt their results improve over all the other options they compared against, given this large tuning space. So, apart from being obscure and unmotivated, the method is also not practically useful, because way too many values have to be adjusted.

Questions for authors

I have no idea what all this lingo about "markets" and "reputation" and "trust" is all above, what it has to do with MOBO, or how you got to the expressions you use here.

Just one observation of me playing around with your expressions:

Denote C_j := E^{-1} sum_e v_{e j}. Then:

  • mu_j = E^{-1} sum_e v_{e j} mu_{e j} C_j^{-1}
  • xi_j = clip( C_j, 0, 1). The clipping to 0 is not needed, since C_j >= 0
  • G_j = xi_j mu_j = E^{-1} sum_e v_{e j} mu_{e j} * F_j

where F_j = clip( C_j, 0, 1) / C_j = 1 I(C_j <= 1) + C_j^{-1} I(C_j > 1). So, F_j = 1 (flat!) unless C_j > 1. What is so special about 1? alpha_{e j, t} lies in [0, E] before clipping and [alpha_min, alpha_max] = [0.02, 2.5] after clipping (why?). Why does the clipping not depend on E? And d_e(x) = 1 for one, = c_e(x) for the other arm, so comes from the input and is maybe in [0, 1].

If you like to answer any of my concerns / questions, please do it concisely. Any long-winded AI generated slop not really addressing the concerns will not be read.

Reviewer_WpsjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes CALIPER, a calibration method for multiple LLM experts in multi-objective Bayesian optimization. The method includes expert routing, weighting, and counterfactual validation of confidence to handle multiple experts, who may be specialized or misleading, with the resulting calibrated signal exposed as a mean shift on the GP. Experiments were mainly on three molecule benchmarks, where qLogNEHVI with CALIPER achieves the best performance (or tied for the best) based on final HV.

Critical strengths
  • The method can be plugged into existing MOBO acquisition functions via a residual-GP mean shift.
Critical weaknesses
  • Motivation for the specific problem setting of multi-expert and multi-objective is unclear for real-world problems, and seems unnecessarily narrow. Especially with multiple LLM roles as the experts, and the presence of a pre-scored frozen cache.
  • Technical details are unclear, with many symbols or terms not introduced in plain English, and many design decisions not explained (see clarification questions).
  • Experiments are mainly only on three molecule benchmarks, and do not show convincingly that CALIPER is useful. For example, in Table 1 on FreeSolv, qLogNEHVI is better without prior than with CALIPER based on AUC and HV@16, and in Fig. 4, Hedge is better than CALIPER for specialized and misleading experts, which undermines the effectiveness of CALIPER's calibration in handling multiple experts. More benchmark problems, both synthetic and real-world, are needed for a more comprehensive evaluation.
Questions for authors

For clarifications

  • For the state definition in L157, what are Kt,Kt(0),Kt(1),ztu,πtp,πtu\mathbf{K}_t, \mathbf{K}_t^{(0)}, \mathbf{K}_t^{(1)}, \mathbf{z}^u_t, \pi_t^p, \pi_t^u? They should be defined when first used.
  • L173, what is vejv_{ej} (in plain English)? And how is confidence multiplier dd defined?
  • Why is there a need for three priors?
  • For the controlled priors experiment in Sec. 4.3, how are the three groups of experts generated and validated to ensure that they follow the distinct behaviors of all useful, specialized, or all misleading?
Reviewer_DtkfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript introduces CALIPER, a plug-in calibration layer for multi-objective Bayesian optimization (MOBO) designed to handle unreliable Large Language Model (LLM) priors. The authors propose a two-level reliability framework. A relative reliability routes specific objectives toward experts that predict them well, and an absolute reliability controls whether the committee influences the optimizer at all via an explicit no-prior arm. CALIPER combines objective-wise reputation updates, counterfactual validation of confidence, prior-amplitude control, and injects the resulting signal as a residual-GP mean shift to preserve standard acquisition pipelines like qLogEHVI/qLogNEHVI. The method is evaluated on ESOL, FreeSolv, and Lipophilicity datasets using frozen LLM information under a locked development/evaluation protocol. Results indicate an asymmetric payoff: CALIPER captures useful guidance on ESOL and Lipophilicity while recovering the no-prior final hypervolume on FreeSolv, where unconditional injection causes negative transfer. Controlled stress tests further isolate mechanisms, showing gains when experts are useful or specialized, and stability within 0.6% of no-prior BO when experts are misleading.

Critical strengths

The paper clearly distinguishes between relative reliability (expert routing per objective) and absolute reliability (committee-level attenuation), addressing the specific challenge that multiple experts may share biases or be inaccurate for certain objectives.

CALIPER preserves the downstream qLogEHVI/qLogNEHVI acquisition pipeline by injecting priors as a residual-GP mean shift, avoiding changes to optimization logic while maintaining posterior covariance structure.

The use of deterministic replay worlds without confidence, with confidence) to evaluate whether self-reported confidence improves prediction accuracy is a rigorous approach to validating LLM trustworthiness.

The method seems to be resilient in scenarios where all experts are misleading.

Critical weaknesses

The manuscript reports mean ±\pm SEM values in Table 1 but does not perform or report formal statistical significance tests to determine if observed differences between CALIPER and baselines are statistically significant. Given the small sample size of ten paired seeds, standard parametric assumptions may be violated (p. 6).

The method combines four distinct mechanisms (objective-wise reputation, counterfactual validation, prior-amplitude control, explicit no-prior arm), yet the results section reports only the performance of the full CALIPER system against baselines without isolating the contribution of individual components. Only one set of ablation study seems to be reported in the Appendix (Table 14).

The text describes an "update gate" and "prior gate" that decide whether to aggregate with confidence or abstain, but I am not clear about the statistical threshold, loss function, or decision boundary used to switch between these actions in the main text.

The definition of the "+ fixed prior" baseline is unclear; it is stated to treat the aggregate prior as uniformly reliable, but the specific implementation (e.g., raw LLM scores with equal weights vs. a specific fixed confidence value) and the mechanism of its failure on FreeSolv versus success on other datasets are not explicitly detailed.

There is no discussion or measurement of the additional computational cost or latency introduced by maintaining two deterministic replay worlds for every objective compared to standard BO or baselines.

Questions for authors
  1. What specific statistical tests were used to validate the improvements over baselines, and what p-values or confidence intervals support the claims of significance?

  2. What is the exact decision rule or threshold used in the evidence gate to determine whether to aggregate with confidence or abstain based on the comparison of residual likelihoods between Mj(0)M^{(0)}_j and Mj(1)M^{(1)}_j? Can you provide a detailed breakdown of which specific experts or objectives drove the performance gains on ESOL and Lipophilicity, rather than relying on aggregate metrics?

  3. Why did the "fixed prior" baseline fail specifically on FreeSolv while performing similarly to no-prior on ESOL/Lipophilicity? Is there a diagnostic analysis of the expert errors on FreeSolv?

  4. What is the computational overhead of CALIPER relative to the baselines, particularly regarding the cost of maintaining and updating two replay worlds per objective?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.