acceptodds
Under review as a conference paper at ICLR 2027

PHEBO: Predictive Feedback-Driven Hypothesis Evolution for Bayesian Optimization

Abstract

Bayesian optimization (BO) offers a sample-efficient framework for scientific optimization with costly evaluations. Large language models (LLMs) can enrich this framework by proposing domain-informed hypotheses, but how to assess their reliability, translate them into numerical guidance, and revise them as evidence accumulates remains a key challenge. This paper introduces PHEBO, an LLM-enhanced BO framework in which predictive feedback drives the evolution of hypotheses population. These hypotheses are encoded in conditional Gaussian process models that quantify their numerical effects. Prospective forecast evaluation updates hypothesis credits, which govern acquisition and hypothesis selection. At scheduled interventions, the LLM uses accumulated observations and credits to revise existing hypotheses, recombine them, and propose new ones, closing the loop between domain reasoning and numerical optimization. Rather than relying on LLMs for in-context numerical prediction or self-reported confidence, PHEBO focuses the LLM on identifying and revising qualitative trends and relationships, guided by empirical predictive feedback. Across four offline scientific optimization tasks spanning analytical chemistry, materials design, and organic photovoltaics, PHEBO achieves better overall performance against GP-based and LLM-enhanced BO baselines. Beyond numerical optimization, the resulting domain beliefs, refined through observation and reasoning, have the potential to inform scientific discovery and support other LLM-enhanced workflows.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Bayesian Optimization, Large Language Model, Gaussian Process, Hypothesis Evolution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WpwuOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PHEBO, an LLM-enhanced Bayesian optimization framework that uses predictive feedback to evaluate and evolve domain hypotheses. Structured hypotheses are encoded as conditional GP experts, with their numerical effects inferred from observations. Prospective predictive scores update hypothesis credits, which determine expert participation, mixture-based acquisition, and subsequent LLM-guided hypothesis revision. Experiments on four offline scientific optimization tasks demonstrate competitive performance against GP-based and LLM-enhanced baselines. Component ablations and additional analyses examine the contributions of belief injection, credit weighting, and hypothesis evolution, while a transfer experiment explores whether the resulting hypotheses can assist another optimizer.

Critical strengths

**Well-motivated and coherent framework. **The paper addresses an important question in LLM-assisted optimization: how to evaluate qualitative guidance and revise it as evidence accumulates. The connection between executable hypotheses, prospective predictive assessment, and hypothesis evolution is clearly defined. Predictive feedback influences both numerical experiment selection and subsequent hypothesis generation, providing a meaningful integration of language-based reasoning and statistical inference.

Careful mathematical formulation. The conditional expert model appropriately accounts for the GP residual correction and uncertainty in the hypothesis coefficient. Forecasts are frozen before outcomes are observed, and newly generated hypotheses must receive a prospective score before participating in acquisition. The paper also carefully distinguishes the propriety of the censored scoring rule from the properties of the clipped credit update.

**Relevant comparisons and informative empirical analysis. ** The evaluation includes relevant baselines such as LGBO, CAKE, and LLAMBO, with favorable results on several tasks. Component ablations, multiple LLM backbones, sensitivity analysis, and cost analysis provide useful supporting evidence. The detailed trajectory analysis and discussion of unsuccessful interventions and candidate-set limitations further help explain the method’s behavior.

Critical weaknesses

1.Lack of comparison with direct history-conditioned prompting. PHEBO maintains and evolves a population of hypotheses using predictive credits, while the LLM already receives historical observations during hypothesis generation. However, the current ablation study does not establish whether this additional evolutionary mechanism provides substantial benefits over directly generating new hypotheses from the same observation history. A baseline using the same hypothesis representation, conditional GP experts, and acquisition mechanism, but without credit-guided evolution, would help clarify the necessity of the proposed approach.

2.Potential cold-start limitations of predictive credit. One important motivation for LLM-assisted BO is to alleviate data scarcity during early optimization. However, PHEBO relies on subsequent experimental observations to evaluate hypothesis reliability, potentially introducing an additional learning cost precisely when observations are scarce. Although hypotheses become eligible after a single prospective evaluation, it remains unclear whether such limited evidence is sufficient to establish reliable credits. Further analysis of early-stage performance and credit reliability under small evaluation budgets would strengthen the paper.

Questions for authors

:In addition to the aforementioned weaknesses, I have a question regarding the relationship between predictive reliability and optimization utility. While I appreciate the novelty of PHEBO's predictive-feedback mechanism, I wonder whether the authors have considered alternative approaches to reliably incorporating LLM guidance into BO, such as LLINBO [1] and LABO [2], have also explored different approaches to reliably incorporating LLM guidance into BO. What are the potential advantages of PHEBO over these approaches? I believe a brief discussion of their differences methods would be valuable.

[1] Chang, C.-Y., Azvar, M., Okwudire, C., & Al Kontar, R. (2025). LLINBO: Trustworthy LLM-in-the-Loop Bayesian Optimization. arXiv:2505.14756.

[2] Chen, Z., et al. (2026). LABO: LLM-Assisted Bayesian Optimization. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026).

5 comments
AuthorsOpenReview ↗

Response to Reviewer Wpwu - Weakness 1

Thank you for this suggestion. We have added a HISTORY-REFRESH baseline that directly generates new hypotheses conditioned on accumulated observations. We summarize the comparison in the main-text ablation study (Sec. 5.3) and provide the detailed protocol and results in Appendix C.6 (Table 13 and Fig. 8). This baseline retains the same hypothesis representation, conditional GP experts, predictive-credit weighting, and acquisition policy, but removes parent selection, credit-guided revision, and inherited credit from hypothesis generation. Initial observations and hypotheses, generation frequency, and proposal limits are matched to the full method.

HPLC ablation results, reported as mean ± sample standard deviation. Higher is better for both metrics. Bold indicates the highest mean in each column.

VariantFinal relative objective (%)rAUC (%)
Pure GP84.34 ± 5.6974.34 ± 5.29
+Beliefs87.31 ± 1.8380.04 ± 4.27
+Credit88.09 ± 0.9581.15 ± 3.31
HISTORY-REFRESH86.28 ± 2.3880.85 ± 3.55
Full (PHEBO)88.58 ± 4.4581.30 ± 4.01

On HPLC, HISTORY-REFRESH achieves a higher mean rAUC with lower variability across seeds than the fixed initial-belief variant (+Beliefs): 80.85 ± 3.55% versus 80.04 ± 4.27% (mean ± sample standard deviation). This suggests more consistent performance over the optimization budget, although its mean final objective is lower. HISTORY-REFRESH falls below both +Credit and Full PHEBO in mean rAUC (81.15% and 81.30%, respectively). Full PHEBO also achieves a higher mean final relative objective than HISTORY-REFRESH (88.58% versus 86.28%).

These results support the benefit of credit-guided hypothesis evolution beyond direct history-conditioned refresh in this setting. They are also consistent with our motivation for predictive scoring: access to observation history alone does not ensure that newly generated hypotheses provide useful guidance. Predictive credit provides empirical feedback for assessing their influence on optimization and guiding subsequent revision.

AuthorsOpenReview ↗

Response to Reviewer Wpwu - Weakness 2

We agree that becoming eligible after one prospective evaluation does not establish that a hypothesis has reliable credit. We have added an analysis of early-budget performance and credit accumulation in Appendix F.5, supported by the controlled hypothesis experiments in Appendix F.3–F.4.

Evaluation budget and early performance. Credit assessment uses the same observations collected during optimization and requires no additional objective evaluations. The first query uses the baseline GP; its outcome supplies both an objective observation and the first prospective scores. Hypotheses can participate in acquisition from the second query, with their credits updated after each subsequent observation. This avoids a separate evaluation phase, although learning which hypotheses are useful can still affect early optimization performance.

To examine this effect, we compare the first ten rounds of GP-only and full PHEBO on HPLC using five paired seeds and the same five initial observations. Table 17 reports the following results (mean ± sample standard deviation); T counts evaluations after initialization, and prefix rAUC is computed over the corresponding budget.

TGP best relative objective (%)PHEBO best relative objective (%)GP prefix rAUC (%)PHEBO prefix rAUC (%)
156.19 ± 0.0056.19 ± 0.0056.19 ± 0.0056.19 ± 0.00
372.15 ± 5.8072.01 ± 10.5862.55 ± 4.5261.61 ± 4.29
572.15 ± 5.8074.26 ± 6.9566.39 ± 4.9266.26 ± 5.62
1074.37 ± 4.4381.57 ± 8.6869.83 ± 4.6071.26 ± 6.74

PHEBO does not improve on GP at every early checkpoint. At round 3, both means are slightly lower; at round 5, its mean best objective is higher but prefix rAUC remains slightly lower. By round 10, full PHEBO achieves higher mean values on both metrics than GP-only in this HPLC comparison. These results suggest that useful guidance can emerge within a small budget, while retaining the limitation that gains are not immediate or uniform across seeds.

What early credit can establish. In the controlled tubing-volume experiment, the data-supported and data-conflicting hypotheses receive positive and negative score differences, respectively, on the identical first query and outcome in all five seed pairs (Appendix F.3). Their mean credits remain separated after three observations (0.823 versus −0.213) and five observations (0.768 versus −0.116), as reported in Appendix F.5. This provides evidence that early observations can differentiate predictive guidance in this controlled setting. The data-conflicting hypothesis initially remained active, with its first exit from the active set occurring between rounds 11 and 19 across the five seeds (Appendix F.4). We therefore treat the first prospective score as provisional evidence for weighting hypotheses and update their credits as further outcomes are observed.

AuthorsOpenReview ↗

Response to Reviewer Wpwu - Question 1

Thank you for suggesting this comparison. PHEBO's distinguishing advantage is that it turns LLM knowledge into explicit hypotheses that can be evaluated, reused, and refined throughout optimization. We have clarified this distinction in Sec. 2.2.

Compared with LLINBO [1], which regulates LLM-proposed designs through participation decay, acquisition-based acceptance, or surrogate constraints, PHEBO assigns predictive evidence to individual domain hypotheses. This allows useful relationships to guide multiple decisions and gives the LLM hypothesis-specific feedback for subsequent revision. Compared with LABO [2], which uses LLM numerical predictions as low-fidelity evaluations with discrepancy correction and uncertainty-based gating, PHEBO elicits qualitative relationships and learns their numerical effects through GP inference from observations. It can therefore use domain knowledge expressed as trends or interactions without requiring the LLM to supply objective values.

Predictive reliability and optimization utility play complementary roles in this process. Each hypothesis is assessed using forecasts frozen before the new outcome is observed. The improvement-censored score evaluates the predicted probability of non-improvement, or the predictive density of the observed value when improvement occurs (Sec. 4.3). This focuses feedback on outcomes relevant to optimization. Relative scores against the baseline GP update hypothesis credits, which determine expert participation and acquisition weights.

The next experiment maximizes the credit-weighted expected improvement over the candidate set, as defined in Eq. (14) (Sec. 4.5). This acquisition combines the baseline GP with the active hypothesis experts using their respective credit-derived weights. Appendix A.7 further shows that this weighted acquisition is exactly the expected improvement under the corresponding predictive mixture. Thus, predictive evidence determines how much each hypothesis contributes to the decision, while EI evaluates the expected gain of each candidate. This connection does not imply that better predictive scores guarantee greater realized optimization gains. The same feedback also guides hypothesis revision, connecting evidence from completed experiments to both the next experiment and the knowledge used to select it.

[1] Chang, C.-Y., Azvar, M., Okwudire, C., & Al Kontar, R. (2025). LLINBO: Trustworthy LLM-in-the-Loop Bayesian Optimization. arXiv:2505.14756.

[2] Chen, Z., et al. (2026). LABO: LLM-Assisted Bayesian Optimization. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026).

Reviewer_LiYaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose PHEBO, an LLM-enhanced BO framework that uses predictive feedback to evaluate and evolve scientific hypotheses. LLM-generated hypotheses are incorporated into GP models, while their influence on optimization is adjusted based on predictive performance. The framework continuously refines hypotheses through mutation, crossover, and innovation. Experiments on four scientific optimization tasks demonstrate competitive performance against existing GP-based and LLM-enhanced BO methods.

Critical strengths
  1. Interesting Methodological Contribution: The paper introduces a framework that integrates LLM-generated scientific hypotheses into BO through conditional GP models and predictive feedback.

  2. Adaptive Hypothesis Evolution: The proposed method continuously evaluates and refines hypotheses through mutation, crossover, and innovation, allowing domain knowledge to evolve as new observations become available.

  3. Reliable Hypothesis Evaluation: Instead of relying on LLM self-reported confidence, the framework uses empirical predictive performance to determine the influence of each hypothesis.

  4. Competitive Experimental Performance: PHEBO achieves strong results on four scientific optimization tasks, outperforming the strong baselines on three of them.

  5. Ablation and Transferability Studies: The authors investigate the contributions of individual components and demonstrate that evolved hypotheses can improve the performance of another BO framework.

Critical weaknesses
  1. Limited Experimental Evaluation: The method is evaluated on only four offline scientific optimization tasks, which limits the evidence for its generalizability.

  2. Limited Methodological Novelty: Several components, including GP-based hypothesis modeling, predictive scoring, and genetic evolution, build upon existing techniques. The paper could better clarify its methodological contributions beyond integrating these components.

  3. Insufficient Ablation Studies: The main component ablation study is conducted only on the HPLC task. Additional ablations across all four tasks would provide stronger evidence for the effectiveness of predictive credit and hypothesis evolution.

  4. Limited Robustness Analysis: The paper does not systematically investigate how the framework behaves when LLM-generated hypotheses are incorrect, misleading, or contradictory. Such experiments would help demonstrate the reliability of the proposed credit mechanism.

  5. Computational Overhead: The framework requires repeated LLM interactions and maintains multiple GP experts, potentially introducing substantial computational costs. A clearer comparison of computational efficiency against existing methods would strengthen the practical contribution.

  6. Potential Initialization Bias: PHEBO and some LLM-based baselines use LLM-generated initial points, while GPBO and CAKE use Sobol initialization. Additional experiments with identical initial designs across methods would help ensure a fair comparison.

Questions for authors
  1. How does PHEBO perform when LLM-generated hypotheses are incorrect, misleading, or contradictory? Could the authors provide controlled experiments to demonstrate whether the predictive credit mechanism can effectively identify and suppress unreliable hypotheses?

  2. The main component ablation study is conducted only on HPLC. Could the authors extend the ablation experiments to the other three tasks to demonstrate that predictive credit and hypothesis evolution consistently contribute to performance improvements?

  3. PHEBO and some LLM-based baselines use LLM-generated initial points, while GPBO and CAKE use Sobol initialization. Could the authors provide comparisons using identical initial designs across all methods, along with wall-clock time and LLM inference costs? Such results would strengthen the fairness of the evaluation and increase my confidence in the practical effectiveness of PHEBO.

6 comments
AuthorsOpenReview ↗

Response to Reviewer LiYa - Weakness 1

We agree that evaluation on four offline tasks provides limited evidence of generalizability. To broaden the evaluation, we have added experiments on the 20-dimensional Sandwich recipe-optimization task. The task description and experimental analysis are provided in Appendix B.5 and Appendix C.7, respectively.

Additional task and experimental setup. Sandwich optimizes the masses of 20 ingredients using a deterministic objective that combines nutritional quality, measured by the Healthy Eating Index-2020 [1], with a reward for a preferred energy level. The objective couples ingredient proportions and total recipe size (Appendix B.5, Eq. (46)). We compare PHEBO, GPBO, LGBO, LLAMBO, and CAKE using the same five initial recipes and 25 subsequent evaluations, with five seeds per method (Appendix C.7).

Results. The table below reports the final relative objective and rAUC from Table 14. Values are means ± sample standard deviations over five seeds. The final relative objective is the final score normalized by the theoretical upper bound of 110 and expressed as a percentage. Higher is better for both metrics, and bold denotes the highest mean.

MethodFinal relative objective (%)rAUC
GPBO92.00 ± 1.320.8923 ± 0.0153
LGBO92.73 ± 1.150.8907 ± 0.0078
LLAMBO88.22 ± 2.830.8672 ± 0.0240
CAKE93.46 ± 0.850.8956 ± 0.0105
PHEBO93.74 ± 0.670.9004 ± 0.0190

PHEBO achieves the highest mean final relative objective and rAUC among the five methods. Together with the trajectories in Fig. 9, these results provide additional evidence of its effectiveness on a 20-dimensional task beyond the four original benchmarks, covering both final performance and progress over the evaluation budget.

Remaining limitation. We were unable to extend the evaluation to a wet-lab setting within the available time. The additional Sandwich experiment remains an offline evaluation with a deterministic oracle, so it does not establish performance under laboratory noise and operational constraints. We acknowledge this limitation in the revised manuscript's Limitations section and identify prospective wet-lab validation as a direction for future work.

[1] Shams-White, M. M., et al. Healthy Eating Index-2020: Review and update process to reflect the Dietary Guidelines for Americans, 2020–2025. Journal of the Academy of Nutrition and Dietetics, 123(9), 2023.

AuthorsOpenReview ↗

Response to Reviewer LiYa - Weakness 2

Thank you for encouraging us to clarify the methodological contribution. We agree that GP modeling, predictive scoring, and genetic operators have established foundations. We have added an explicit statement in the Introduction to clarify that PHEBO's contribution is a hypothesis-level feedback mechanism that couples experiment selection with the evolution of domain beliefs.

Specifically, each hypothesis is instantiated as a conditional GP expert and scored prospectively against a baseline GP, distinguishing improvement over the incumbent from non-improvement. The resulting predictive credits control hypothesis participation and weighting in acquisition, and guide the LLM's revision and recombination of hypotheses. This mechanism connects how LLM-generated knowledge is used and how it evolves through the same predictive assessment, providing the methodological contribution beyond the individual components.

AuthorsOpenReview ↗

Response to Reviewer LiYa - Weakness 3 and Question 2

Following your suggestion, we have extended the component ablation study to Cross-barrel, Concrete, and PCE10. Together with the original HPLC study, the ablations now cover all four main tasks. The additional trajectories and detailed analysis are provided in Appendix C.5 and Fig. 7, with a corresponding reference added to Sec. 5.3.

Controlled ablation setup. We use the same four incremental variants as in the HPLC study: Pure GP, +Beliefs, +Credit, and +Evolution (Full). +Beliefs adds fixed hypotheses with fixed mixture weights; +Credit introduces predictive-credit weighting while retaining the fixed hypothesis population; and Full additionally enables credit-guided hypothesis evolution. Each run starts with five initial observations and receives 25 subsequent BO evaluations.Within each seed, all four variants share the same initial observations, and the belief-based variants additionally share the same initial hypothesis population(Appendix C.5).

Final relative objective (%) at BO round 25, reported as mean ± sample standard deviation. Bold indicates the best result in each column.

VariantHPLCCross-barrelConcretePCE10
Pure GP84.34 ± 5.6982.18 ± 5.6376.38 ± 2.9192.97 ± 0.98
+Beliefs87.31 ± 1.8381.83 ± 8.6576.83 ± 0.0893.26 ± 1.10
+Credit88.09 ± 0.9583.06 ± 10.0977.17 ± 0.5193.26 ± 1.10
+Evolution (Full)88.58 ± 4.4585.15 ± 4.0878.18 ± 6.6694.32 ± 0.51

Results across tasks. Full PHEBO achieves the highest mean final relative objective among the four variants on all three additional tasks (Fig. 7). On Cross-barrel, +Credit improves the final mean over +Beliefs, and enabling evolution further improves the final mean with lower variability. On Concrete, the mean final objective increases modestly across the four variants. On PCE10, +Beliefs and +Credit perform similarly, while Full attains a higher final objective.

These results strengthen the evidence for the contributions of predictive credit and hypothesis evolution beyond HPLC. They also show that the incremental benefit varies by task: credit weighting provides little additional improvement on PCE10 in this comparison, whereas the full framework achieves the highest mean final performance across the added tasks.

Reviewer_ThvoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PHEBO, a BO framework augmented by LLMs. The LLMs are used to generate qualitative scientific hypotheses, coupled with Gaussian Processes (GPs) to quantify their numerical contributions to BO campaigns. As opposed to prior LLM-enhanced frameworks, which used LLMs for direct numerical prediction or candidate generation, PHEBO uses LLMs to propose and iteratively revise qualitative hypotheses, and uses GPs to model the numerical effects of the proposed hypotheses. Using a credit system, hypotheses whose predictive models outperform a baseline GP model are assigned higher priority, while poorly performing ones are weighted lower. The framework additionally uses LLM-guided mutation, crossover, and innovation to evolve the hypothesis population throughout the optimization campaign.

Critical strengths
  1. The paper is well-motivated and presents a clever approach to leveraging the power of LLMs for BO campaigns. Their “division of labor” between LLM hypotheses generation and GP quantification is intuitive and addresses some limitations of using LLMs in BO frameworks.
  2. The authors have performed comprehensive benchmarking along with ablations studies to show the performance of their proposed framework on four different relevant scientific tasks. The inclusion of multiple LLM backbones further solidifies their empirical results.
Critical weaknesses
  1. In Section 5.1, it is stated that PHEBO, LLAMBO, and LGBO use the same LLM-suggested initial points while GPBO and CAKE use Sobol initialization. This could explain the difference between the curves at BO Round 0 of Figure 2. However, this raises the concern that the comparison of the algorithms might not be fair, since they started from different initial points. While the choice of the authors to use LLM-suggested initial points is understandable in the context of the work, it does introduce potential bias in the comparison of the methods and, thus, it is highly recommended to re-run the experiments with either all initialized with LLM-suggested points or Sobol initialization. However, if the authors insist on keeping the initializations as is, I would recommend that the authors also report the GAP metric introduced in [1], which normalizes performance with respect to the initial set of each method. The biggest concern would be that using the GAP metric, from visual inspection of the panels on Figure 2, the proposed method might not be outperforming the baselines to the extent the current version of Figure 2 is suggesting. For example, Figure 2B shows that CAKE and GPBO are more than 10 points below PHEBO at initialization, but at the end of the optimization campaign, they’re behind by about 10 points. Figure 2C also shows that while CAKE and GPBO are more than 5 points behind PHEBO at initialization, they are about 2-3 points behind at the end of the optimization. This is sort of supported in Figure 2D, the only benchmark where all methods seem to have the same initial best value (even though they weren’t initialized the same), where we see that PHEBO underperforms CAKE.
  2. In Section 5.3 (line 450), it is stated that “enabling thinking allows Kimi-K2.6 to reach 90% of the best performance in fewer rounds, suggesting that explicit reasoning can improve optimization efficiency.” However, after a closer look at Figure 3c, it looks like while the average of the five replicates for Kimi-K2.6 with thinking mode enabled is lower than the rest, the individual points tell a different story: there’s a high degree of variation in the performance of Kimi-K2.6. To that end, I think Figure 3c would substantially benefit from including standard error bars (in addition to what’s already plotted), and the interpretation would require from further statistical testing.

[1] Jiang, S., Chai, H., Gonzalez, J., and Garnett, R. BINOCULARS for efficient, nonmyopic sequential experimental design. In ICML, 2020.

Questions for authors
  1. Can the authors please provide results of Figure 2 with identical initializations for all methods? If possible, please try initializing all methods with Sobol, and also with LLM-suggested points as two different initialization strategies, to measure the effect of initialization on the optimization campaign.
  2. Can the authors further clarify whether the differences between Kimi-K2.6 thinking-enabled vs. disabled models were evaluated using paired random seeds with identical initial observations? Are the observed differences between the two modes statistically significant?
  3. Can the authors also report AUC values for the optimization curves? In Section 5.2 (line 377), it is mentioned that “these trajectories indicate earlier gains and better final performance on these two tasks …”; it would be helpful to have a table of final values as well as the area under the optimization curves to better quantify the performance (instead of visual inspection).
5 comments
AuthorsOpenReview ↗

Response to Reviewer Thvo - Weakness 1 and Question 1

Thanks for your nice suggestion. We agree that this controlled comparison strengthens our empirical evaluation and provides readers with further insight into the role of initialization in optimization performance.

Shared-initialization experiments. Following your suggestion, we have evaluated all five methods (PHEBO, GPBO, LGBO, LLAMBO, and CAKE) under both Sobol and LLM-suggested initialization on HPLC, Cross-barrel, and Concrete. Within each task and initialization strategy, all methods use the same five initial points and their observations, followed by 25 sequential evaluations. The experimental setup and optimization trajectories are provided in Appendix C.4 and Fig. 6 of the revised manuscript.

The results are summarized in the table below. Both metrics are expressed as percentages and reported as means. Bold indicates the highest mean within each task, initialization, and metric.

MethodHPLCCross-barrelConcrete
LLM-suggested initialization
Final relative objective (%)
PHEBO88.3584.5579.15
GPBO83.0782.1877.16
LGBO85.1281.3876.96
LLAMBO69.3873.9967.41
CAKE76.6781.1479.08
Relative AUC (%)
PHEBO79.8678.1074.98
GPBO77.0477.1274.24
LGBO77.4375.6274.84
LLAMBO65.1170.9667.22
CAKE67.7671.9474.12
Sobol initialization
Final relative objective (%)
PHEBO87.9984.8977.13
GPBO80.3873.2476.00
LGBO84.1781.2376.65
LLAMBO55.9074.7374.49
CAKE79.6676.0176.72
Relative AUC (%)
PHEBO71.8576.8874.32
GPBO61.9368.6273.11
LGBO69.3271.3372.88
LLAMBO53.7569.2273.14
CAKE61.3767.9874.85

PHEBO achieves the highest mean final relative objective on all three tasks under both initialization strategies. On HPLC and Cross-barrel, PHEBO has the highest mean rAUC under both strategies. On Concrete, the methods are more closely matched: PHEBO has the highest mean rAUC with LLM-suggested initialization, while CAKE has the highest mean rAUC with Sobol initialization. Thus, the additional experiments support PHEBO's competitiveness under matched initialization, with gains that vary by task and metric.

Why we chose matched initialization rather than GAP for this comparison. GAP measures the fraction of the initial optimality gap closed during optimization [1]. However, a lower initial best value does not necessarily imply a lower final value, and normalizing improvement does not make different initial observation sets equivalent. Those observations also differ in their locations and the information they provide to the surrogate. To address the fairness concern directly, we therefore followed your recommendation to repeat the comparisons with shared initial observations under both strategies. This controls the starting data while allowing us to compare the final relative objective and progress over the same evaluation budget.

PCE10 in Fig. 2(d). We acknowledge that CAKE performs better on this task. Appendix C.1 and Fig. 5 identify candidate-set coverage as a contributing limitation: PHEBO selects from a fixed set of 2,048 Sobol points, whereas CAKE continuously optimizes its acquisition function and can reach boundary compositions outside that set. All five fixed candidate sets have ceilings below the score of the pure-PCBM vertex under the benchmark oracle, and CAKE reaches that vertex in three runs. PHEBO reaches its candidate-set ceiling in two runs. The shared-initialization experiments above directly test that concern and show that PHEBO remains competitive on HPLC, Cross-barrel, and Concrete.

[1] Jiang, S., Chai, H., Gonzalez, J., and Garnett, R. BINOCULARS for efficient, nonmyopic sequential experimental design. In ICML, 2020.

AuthorsOpenReview ↗

Response to Reviewer Thvo - Weakness 2 and Question 2

We agree that the lower mean number of rounds should be interpreted alongside the variation across runs. We have revised the discussion to distinguish the observed average difference from evidence of an improvement in optimization efficiency.

Paired experimental setup. Yes, the thinking-enabled and thinking-disabled Kimi-K2.6 configurations were evaluated using five matched BO random seeds and identical initial observations. We have made this explicit in Sec. 5.3. Thus, each pair starts from the same observed data, controlling for differences in initialization.

Variability in Fig. 3(c). The revised figure includes error bars indicating ±1 standard error of the mean (SEM), while retaining the mean bars and individual-run markers. The caption specifies that the results are summarized over five seeds (lines 484–485). It also clarifies that the efficiency metric is the first BO round reaching 90% of each run's own final objective. The individual points show the variation across runs, while the SEM bars summarize uncertainty in the estimated means.

Statistical conclusion and revised interpretation. Although Thinking On reaches this threshold earlier on average, the current comparison does not establish a reliable efficiency advantage over Thinking Off. We have therefore replaced the earlier suggestion that explicit reasoning improves optimization efficiency with a description of the observed trend.

AuthorsOpenReview ↗

Response to Reviewer Thvo - Question 3

Following your suggestion, we have added Table 1 to the main text to report both the final relative objective and the area under the optimization curves in Fig. 2. Sec. 5.2 now discusses these quantitative results alongside the trajectories.

Metric and normalization. To facilitate comparison, we report relative AUC (rAUC): the area under the normalized best-so-far curve divided by the total reference area over the evaluation budget. Specifically, each run's normalized best-so-far objective lies in [0, 1], so the reference area is the rectangle of height 1 and width T = 25 rounds. We compute the curve's area using the trapezoidal rule, including round 0 after the five initial evaluations, and divide by T. Table 1 expresses rAUC as a percentage. Higher rAUC indicates better performance across the evaluation budget.

Quantitative results. The table below reproduces Table 1. Values are means ± sample standard deviations over five runs. Higher is better; bold indicates the highest mean for each task and metric.

MethodHPLCCross-barrelConcretePCE10
Final relative objective (%)
PHEBO88.35 ± 3.8984.55 ± 3.7878.59 ± 5.8594.25 ± 0.46
GPBO80.38 ± 9.5173.24 ± 2.2776.00 ± 1.3193.18 ± 0.96
LGBO85.12 ± 7.7981.38 ± 8.8176.96 ± 0.4193.42 ± 0.97
LLAMBO69.38 ± 3.8473.99 ± 2.3567.97 ± 1.9888.13 ± 3.86
CAKE79.66 ± 10.5976.01 ± 9.1676.99 ± 2.3695.60 ± 1.43
Relative AUC (%)
PHEBO79.86 ± 4.7378.10 ± 1.5274.31 ± 3.4091.24 ± 0.91
GPBO61.93 ± 4.1668.60 ± 2.8173.11 ± 2.2991.04 ± 0.97
LGBO77.43 ± 6.8675.56 ± 6.6874.84 ± 1.7191.92 ± 0.99
LLAMBO65.11 ± 3.1070.96 ± 3.1467.65 ± 1.9486.51 ± 3.45
CAKE61.37 ± 4.0667.98 ± 7.7375.03 ± 1.4891.46 ± 2.28

On HPLC and Cross-barrel, PHEBO achieves both the highest mean final relative objective (88.35% and 84.55%) and the highest mean rAUC (79.86% and 78.10%). The final values quantify its better terminal performance, while rAUC summarizes performance throughout the budget. Together with the trajectories in Fig. 2, where PHEBO's advantage is apparent by rounds 5 and 10 on these tasks, respectively, these results support the statement about earlier gains and better final performance in Sec. 5.2. These are the main-experiment results; comparisons controlling for initialization are reported separately in Appendix C.4 and Table 12, as discussed in our response to W1 and Q1.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Update Manuscript

We would like to thank all of the reviewers again for their constructive comments and suggestions, which have helped us clarify the contribution and strengthen the empirical evaluation. We uploaded a revised version of our paper and highlight the major modifications in red for visibility. In short,

  1. Method presentation in the introduction. Revised the description of our method to make its overall workflow and key contributions clearer and more accessible to readers (Introduction).
  2. Contribution and related work. Clarified the hypothesis-level feedback mechanism and its differences from related approaches, including LLINBO and LABO (Introduction and Sec. 2.2).
  3. Fair initialization comparisons. Added experiments with shared Sobol and LLM-suggested initializations across five methods on HPLC, Cross-barrel, and Concrete (Appendix C.5).
  4. Performance and variability reporting. Added final objective and rAUC summaries to the main text, and clarified the paired setup, error bars, and statistical interpretation of the thinking-mode comparison (Table 1 and Sec. 5.3).
  5. Additional ablations. Extended component ablations to the other three tasks and added a direct history-conditioned hypothesis-refresh baseline (Sec. 5.3 and Appendices C.6–C.7).
  6. Higher-dimensional evaluation. Added the 20-dimensional Sandwich task, including its description and experimental results (Appendices B.5 and C.8).
  7. Robustness and cold-start analysis. Added controlled experiments with misleading and contradictory hypotheses, together with analyses of early-budget performance and credit accumulation (Appendix F, especially F.5).
  8. Computational costs. Expanded the cost analysis with measured LLM calls, token usage, GP fitting costs, and wall-clock time across methods (Appendix C.3, Table 9).

Thank you all again for your precious and insightful suggestions. Please let us know if you have additional questions or ideas for improvement.

Kind regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.