- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces PHEBO, an LLM-enhanced Bayesian optimization framework that uses predictive feedback to evaluate and evolve domain hypotheses. Structured hypotheses are encoded as conditional GP experts, with their numerical effects inferred from observations. Prospective predictive scores update hypothesis credits, which determine expert participation, mixture-based acquisition, and subsequent LLM-guided hypothesis revision. Experiments on four offline scientific optimization tasks demonstrate competitive performance against GP-based and LLM-enhanced baselines. Component ablations and additional analyses examine the contributions of belief injection, credit weighting, and hypothesis evolution, while a transfer experiment explores whether the resulting hypotheses can assist another optimizer.
Critical strengths
**Well-motivated and coherent framework. **The paper addresses an important question in LLM-assisted optimization: how to evaluate qualitative guidance and revise it as evidence accumulates. The connection between executable hypotheses, prospective predictive assessment, and hypothesis evolution is clearly defined. Predictive feedback influences both numerical experiment selection and subsequent hypothesis generation, providing a meaningful integration of language-based reasoning and statistical inference.
Careful mathematical formulation. The conditional expert model appropriately accounts for the GP residual correction and uncertainty in the hypothesis coefficient. Forecasts are frozen before outcomes are observed, and newly generated hypotheses must receive a prospective score before participating in acquisition. The paper also carefully distinguishes the propriety of the censored scoring rule from the properties of the clipped credit update.
**Relevant comparisons and informative empirical analysis. ** The evaluation includes relevant baselines such as LGBO, CAKE, and LLAMBO, with favorable results on several tasks. Component ablations, multiple LLM backbones, sensitivity analysis, and cost analysis provide useful supporting evidence. The detailed trajectory analysis and discussion of unsuccessful interventions and candidate-set limitations further help explain the method’s behavior.
Critical weaknesses
1.Lack of comparison with direct history-conditioned prompting. PHEBO maintains and evolves a population of hypotheses using predictive credits, while the LLM already receives historical observations during hypothesis generation. However, the current ablation study does not establish whether this additional evolutionary mechanism provides substantial benefits over directly generating new hypotheses from the same observation history. A baseline using the same hypothesis representation, conditional GP experts, and acquisition mechanism, but without credit-guided evolution, would help clarify the necessity of the proposed approach.
2.Potential cold-start limitations of predictive credit. One important motivation for LLM-assisted BO is to alleviate data scarcity during early optimization. However, PHEBO relies on subsequent experimental observations to evaluate hypothesis reliability, potentially introducing an additional learning cost precisely when observations are scarce. Although hypotheses become eligible after a single prospective evaluation, it remains unclear whether such limited evidence is sufficient to establish reliable credits. Further analysis of early-stage performance and credit reliability under small evaluation budgets would strengthen the paper.
Questions for authors
:In addition to the aforementioned weaknesses, I have a question regarding the relationship between predictive reliability and optimization utility. While I appreciate the novelty of PHEBO's predictive-feedback mechanism, I wonder whether the authors have considered alternative approaches to reliably incorporating LLM guidance into BO, such as LLINBO [1] and LABO [2], have also explored different approaches to reliably incorporating LLM guidance into BO. What are the potential advantages of PHEBO over these approaches? I believe a brief discussion of their differences methods would be valuable.
[1] Chang, C.-Y., Azvar, M., Okwudire, C., & Al Kontar, R. (2025). LLINBO: Trustworthy LLM-in-the-Loop Bayesian Optimization. arXiv:2505.14756.
[2] Chen, Z., et al. (2026). LABO: LLM-Assisted Bayesian Optimization. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026).
5 comments
Response to Reviewer Wpwu - Weakness 2
We agree that becoming eligible after one prospective evaluation does not establish that a hypothesis has reliable credit. We have added an analysis of early-budget performance and credit accumulation in Appendix F.5, supported by the controlled hypothesis experiments in Appendix F.3–F.4.
Evaluation budget and early performance. Credit assessment uses the same observations collected during optimization and requires no additional objective evaluations. The first query uses the baseline GP; its outcome supplies both an objective observation and the first prospective scores. Hypotheses can participate in acquisition from the second query, with their credits updated after each subsequent observation. This avoids a separate evaluation phase, although learning which hypotheses are useful can still affect early optimization performance.
To examine this effect, we compare the first ten rounds of GP-only and full PHEBO on HPLC using five paired seeds and the same five initial observations. Table 17 reports the following results (mean ± sample standard deviation); T counts evaluations after initialization, and prefix rAUC is computed over the corresponding budget.
| T | GP best relative objective (%) | PHEBO best relative objective (%) | GP prefix rAUC (%) | PHEBO prefix rAUC (%) |
|---|---|---|---|---|
| 1 | 56.19 ± 0.00 | 56.19 ± 0.00 | 56.19 ± 0.00 | 56.19 ± 0.00 |
| 3 | 72.15 ± 5.80 | 72.01 ± 10.58 | 62.55 ± 4.52 | 61.61 ± 4.29 |
| 5 | 72.15 ± 5.80 | 74.26 ± 6.95 | 66.39 ± 4.92 | 66.26 ± 5.62 |
| 10 | 74.37 ± 4.43 | 81.57 ± 8.68 | 69.83 ± 4.60 | 71.26 ± 6.74 |
PHEBO does not improve on GP at every early checkpoint. At round 3, both means are slightly lower; at round 5, its mean best objective is higher but prefix rAUC remains slightly lower. By round 10, full PHEBO achieves higher mean values on both metrics than GP-only in this HPLC comparison. These results suggest that useful guidance can emerge within a small budget, while retaining the limitation that gains are not immediate or uniform across seeds.
What early credit can establish. In the controlled tubing-volume experiment, the data-supported and data-conflicting hypotheses receive positive and negative score differences, respectively, on the identical first query and outcome in all five seed pairs (Appendix F.3). Their mean credits remain separated after three observations (0.823 versus −0.213) and five observations (0.768 versus −0.116), as reported in Appendix F.5. This provides evidence that early observations can differentiate predictive guidance in this controlled setting. The data-conflicting hypothesis initially remained active, with its first exit from the active set occurring between rounds 11 and 19 across the five seeds (Appendix F.4). We therefore treat the first prospective score as provisional evidence for weighting hypotheses and update their credits as further outcomes are observed.
Response to Reviewer Wpwu - Question 1
Thank you for suggesting this comparison. PHEBO's distinguishing advantage is that it turns LLM knowledge into explicit hypotheses that can be evaluated, reused, and refined throughout optimization. We have clarified this distinction in Sec. 2.2.
Compared with LLINBO [1], which regulates LLM-proposed designs through participation decay, acquisition-based acceptance, or surrogate constraints, PHEBO assigns predictive evidence to individual domain hypotheses. This allows useful relationships to guide multiple decisions and gives the LLM hypothesis-specific feedback for subsequent revision. Compared with LABO [2], which uses LLM numerical predictions as low-fidelity evaluations with discrepancy correction and uncertainty-based gating, PHEBO elicits qualitative relationships and learns their numerical effects through GP inference from observations. It can therefore use domain knowledge expressed as trends or interactions without requiring the LLM to supply objective values.
Predictive reliability and optimization utility play complementary roles in this process. Each hypothesis is assessed using forecasts frozen before the new outcome is observed. The improvement-censored score evaluates the predicted probability of non-improvement, or the predictive density of the observed value when improvement occurs (Sec. 4.3). This focuses feedback on outcomes relevant to optimization. Relative scores against the baseline GP update hypothesis credits, which determine expert participation and acquisition weights.
The next experiment maximizes the credit-weighted expected improvement over the candidate set, as defined in Eq. (14) (Sec. 4.5). This acquisition combines the baseline GP with the active hypothesis experts using their respective credit-derived weights. Appendix A.7 further shows that this weighted acquisition is exactly the expected improvement under the corresponding predictive mixture. Thus, predictive evidence determines how much each hypothesis contributes to the decision, while EI evaluates the expected gain of each candidate. This connection does not imply that better predictive scores guarantee greater realized optimization gains. The same feedback also guides hypothesis revision, connecting evidence from completed experiments to both the next experiment and the knowledge used to select it.
[1] Chang, C.-Y., Azvar, M., Okwudire, C., & Al Kontar, R. (2025). LLINBO: Trustworthy LLM-in-the-Loop Bayesian Optimization. arXiv:2505.14756.
[2] Chen, Z., et al. (2026). LABO: LLM-Assisted Bayesian Optimization. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026).
Response to Reviewer Wpwu - Weakness 1
Thank you for this suggestion. We have added a HISTORY-REFRESH baseline that directly generates new hypotheses conditioned on accumulated observations. We summarize the comparison in the main-text ablation study (Sec. 5.3) and provide the detailed protocol and results in Appendix C.6 (Table 13 and Fig. 8). This baseline retains the same hypothesis representation, conditional GP experts, predictive-credit weighting, and acquisition policy, but removes parent selection, credit-guided revision, and inherited credit from hypothesis generation. Initial observations and hypotheses, generation frequency, and proposal limits are matched to the full method.
HPLC ablation results, reported as mean ± sample standard deviation. Higher is better for both metrics. Bold indicates the highest mean in each column.
On HPLC, HISTORY-REFRESH achieves a higher mean rAUC with lower variability across seeds than the fixed initial-belief variant (+Beliefs): 80.85 ± 3.55% versus 80.04 ± 4.27% (mean ± sample standard deviation). This suggests more consistent performance over the optimization budget, although its mean final objective is lower. HISTORY-REFRESH falls below both +Credit and Full PHEBO in mean rAUC (81.15% and 81.30%, respectively). Full PHEBO also achieves a higher mean final relative objective than HISTORY-REFRESH (88.58% versus 86.28%).
These results support the benefit of credit-guided hypothesis evolution beyond direct history-conditioned refresh in this setting. They are also consistent with our motivation for predictive scoring: access to observation history alone does not ensure that newly generated hypotheses provide useful guidance. Predictive credit provides empirical feedback for assessing their influence on optimization and guiding subsequent revision.