- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper uses least-squares fitting to adaptively learn the trust of LLM-derived priors on Bayesian optimization. The methods are evaluated on six gene perturbation screens, 184 ProteinGym protein fitness assays, and 43 FLAb antibody developability assays.
Critical strengths
The figures and tables are visually polished and clearly presented.
Critical weaknesses
- Limited novelty. Previous work, including LLINBO: Trustworthy LLM-in-the-Loop Bayesian Optimization(https://arxiv.org/abs/2505.14756) and LABO (ICML 2026), has explored trustworthy use of LLMs in BO. The paper lacks both methodological and empirical comparisons with these approaches.
- Incremental contribution. Beyond updating the trust coefficient, could the prior itself evolve with experimental feedback?
Questions for authors
- Could the authors evaluate generalization to other domains, such as AutoML?
- How does the method perform in high-dimensional optimization settings?
- Can prior reliability be estimated accurately from very few observations? I remain unconvinced and would appreciate further evidence.
12 comments
Response to Reviewer an8o (2/3)
Comparison with a LABO-style adaptation.
LABO is the closest method to ours: like , its KOH scale is fitted to labels and free in sign. The difference is how that scale is fitted. LABO regresses on the prior alone, whereas HARIBO fits jointly with the features (Prop. 1(a)).
Table R2. LABO-style vs. HARIBO with LLM priors, Recall@R10 (10 seeds).
protein (protein_matched), LLM prior | features | LABO-style | HARIBO | |
|---|---|---|---|---|
| Qwen2.5-32B (212) | PCA-50 | 0.580 | 0.592 | 0.02 |
| Qwen2.5-32B (212) | full one-hot | 0.604 | 0.616 | 0.15 |
| Claude Sonnet 5 (184) | PCA-50 | 0.630 | 0.634 | 0.14 |
| Claude Sonnet 5 (184) | full one-hot | 0.653 | 0.663 |
HARIBO is ahead in every cell. With ESM2 and on gene the two agree within 0.006 (n.s.), and both survive the flipped prior, unlike LLINBO.
Cost. This comparison removes LABO's in-loop LLM. As published, LABO uses the LLM as a low-fidelity simulator and evaluates it at its initial design and at the candidates it selects. HARIBO needs one initial elicitation round before the first experiment (5.9k tokens per gene screen) and a closed-form ridge fit on a CPU (39 ms for ten rounds).
The key distinction is where adaptation occurs: LLINBO filters LLM proposals, and LABO calibrates a low-fidelity signal before fitting a discrepancy. HARIBO fits the prior coefficient jointly with the candidate features, estimating the prior's value beyond them, so even a fixed prior can be exploited, ignored, or inverted as evidence accumulates, without further LLM queries.
Added in Sec. 2 and "LLM-in-the-loop BO baselines" (new Appendix E, blue).
[W2. Can the prior itself evolve with experimental feedback?]
Thank you for this suggestion. Yes, and we see three ways, two of which we tested.
- Re-elicit the prior after measurement. Qwen2.5-32B sees its first measured batch, with or without tools (gene search, pathways, the BO surrogate), and returns a revised prior. The revision improves prior quality on proteins, antibodies and an AutoML benchmark (LCBench), but lowers it on genes (hit rate 0.088 → 0.056). HARIBO's ten-round recall moves by at most 0.02 either way (gene 0.131 → 0.129, protein 0.595 → 0.598, antibody 0.615 → 0.633, LCBench 0.622 → 0.619 without tools; tools do not add to this), while the revision costs 113k–122k extra tokens against one-shot elicitation's 5.9k.
- Re-query every round with measured feedback. This is the closed-loop co-scientist we already compare against (§4.4; 20–32× the requests). From round two its prompt contains the measured score and hit status of every candidate it proposed. Yet the feedback buys little:
- Qwen2.5-32B: after ten rounds of feedback the agent is below one query plus HARIBO in all three domains (gene 0.109 vs. 0.131, protein 0.505 vs. 0.590, antibody 0.464 vs. 0.599), and below knowledge-free BO on protein and antibody.
- Claude Sonnet 5: the agent leads on gene (0.169), but one wider call plus HARIBO, with no feedback to the LLM, reaches 0.163 (App. C.8), so the remaining recall gap is 0.006. Its gene gain sits on the three best-covered screens (App. C.6). On protein, the difference from one-round HARIBO is not statistically significant, and on antibody the agent ends below BO.
- BioDiscoveryAgent's critic and tools every round do not help either: 0.095 with tools vs. 0.098 without, against 0.128 for HARIBO (3 gene screens).
- Let the evolving prior enter as a new column. HARIBO's construction already accommodates this, because prior columns get learned coefficients (§3). Its weight is again learned from the labels, so an updated prior is not trusted blindly either. We now state this as an extension for in-the-loop agents.
In short, letting the prior evolve can improve the prior, but the tested prior revisions produced modest changes in downstream recall while multiplying the cost. One-shot elicitation stays the default; the evolving variants are reported as options.
Added in "Prior revision with data and tools" (new Appendix E, blue).
[Q1. Generalization to other domains such as AutoML]
We added an AutoML benchmark, LCBench (Zimmer et al., 2021; MLP hyperparameters on OpenML datasets). The prior comes from one LLM request per dataset, which returns a recommended range and importance for each hyperparameter, converted into a score per configuration.
The pattern is the same as in biology. The prior is weak on its own (0.177 vs. 0.514 for BO), yet HARIBO turns it into a gain over BO at both budgets: 0.605 vs. 0.514 at 10% of the pool, and 0.388 vs. 0.275 at 5% (both ). With the prior's sign flipped, the fixed-weight schemes collapse (fixed blend 0.157, BO-style 0.141 vs. BO 0.514 at 10%). HARIBO stays near BO at 10% (0.502) and above it at 5% (0.317 vs. 0.275).
Added in Sec. 4 and the appendix (blue).
Response to Reviewer an8o (3/3)
[Q2. High-dimensional settings]
We evaluate high-dimensional candidate representations within finite pools. The paper compresses features with PCA-50; we reran without compression.
- Protein: the full one-hot encoding of each mutant (position and residue indicators, 75–707 dimensions, median 450), on 212 assays.
- Gene: raw STRING-800 + scGPT-512 + DepMap profiles concatenated, 2,120 dimensions, with no PCA. The pool is restricted to the genes all three sources cover.
Table R3. Recall@R10, compressed vs. uncompressed features.
| domain, prior | features (dim) | pure BO | HARIBO | HARIBO − BO |
|---|---|---|---|---|
| protein, +ESM2 | PCA (50) | 0.553 | 0.700 | +0.147 |
| one-hot (~450) | 0.594 | 0.717 | +0.124 | |
| protein, −ESM2 | PCA (50) | 0.553 | 0.684 | +0.131 |
| one-hot (~450) | 0.594 | 0.706 | +0.112 | |
| protein, Sonnet 5 | PCA (50) | 0.559 | 0.634 | +0.076 |
| one-hot (~450) | 0.599 | 0.663 | +0.065 | |
| gene, Sonnet 5 | STRING-PCA (50), same pool | 0.191 | 0.215 | +0.024 |
| raw (2,120) | 0.251 | 0.291 | +0.041 | |
| gene, Qwen2.5-32B | STRING-PCA (50), same pool | 0.191 | 0.209 | +0.017 |
| raw (2,120) | 0.251 | 0.279 | +0.028 |
HARIBO stays well above BO in high dimensions. On gene the richer 2,120-dimensional features lift BO from 0.191 to 0.251, and HARIBO's margin grows rather than shrinks (+0.024 → +0.041 with Sonnet 5, where it also beats the fixed blend, 0.291 vs. 0.272). On protein the margin stays large, including under the adversarial prior, where the fixed blend and BO-style fall to 0.421, 0.17 below BO. (Gene numbers are on the ~9k genes all three sources cover, so ten batches of 128 screen ~14% of the pool; all arms share that pool.)
The construction scales with dimension because the prior is a single extra column. Its coefficient is left unpenalized, while the feature coordinates share a ridge penalty chosen by leave-one-out cross-validation, which matters most when approaches or exceeds the number of labels (App. A.2). The fit stays closed-form: for labels and features it costs per round, so it runs on a CPU at either dimensionality.
Added in "Budget and feature strength" (new Appendix E, blue).
[Q3. Can prior reliability be estimated from very few observations?]
Thank you for raising this. It is central to the method, because round one has no labels and the trust must be learned on the fly. We tested it directly by shrinking the batch and recording, each round, whether the sign of matches the population coefficient computed from all labels. Protein: 184 assays, 10 seeds. "Labels" is the number of measurements behind the estimate.
Table R4. Recall@R10 and sign accuracy of with tiny batches.
| prior | batch | labels at 1st / 2nd / 3rd estimate | sign of correct | BO | fixed blend | BO-style | HARIBO |
|---|---|---|---|---|---|---|---|
| +ESM2 | 2 | 4 / 6 / 8 | 81% / 84% / 85% | 0.065 | 0.099 | 0.107 | 0.110 |
| +ESM2 | 4 | 4 / 8 / 12 | 71% / 84% / 89% | 0.124 | 0.205 | 0.206 | 0.210 |
| +ESM2 | 8 | 8 / 16 / 24 | 77% / 88% / 93% | 0.240 | 0.353 | 0.377 | 0.379 |
| −ESM2 | 2 | 4 / 6 / 8 | 78% / 80% / 82% | 0.065 | 0.029 | 0.030 | 0.102 |
| −ESM2 | 4 | 4 / 8 / 12 | 79% / 80% / 84% | 0.124 | 0.070 | 0.061 | 0.189 |
| −ESM2 | 8 | 8 / 16 / 24 | 84% / 86% / 89% | 0.240 | 0.152 | 0.141 | 0.359 |
With only four labels, the learned coefficient already has the right sign on 71–81% of runs, and the accuracy keeps rising as labels accrue. That is enough to matter: with two measurements per round (20 in total), HARIBO under an adversarial prior reaches 0.102, against 0.065 for BO and 0.03 for the fixed weights. A weight fixed in advance cannot recover from a wrong sign at any budget.
On genes (6 screens), HARIBO beats BO on all six screens at every batch size from 8 to 64 for Sonnet 5 (, the minimum attainable with six screens), e.g. 0.064 vs. 0.026 at batch 32. At batch 8 the sign is close to chance (57%), because round one contains just 4 nominated genes.
These results provide empirical evidence that useful prior weighting can emerge from very few observations in the tested settings. Equation 3 motivates collecting contrasting prior values in the initial batch under its stated OLS assumptions; it is not an exact variance guarantee for the regularized estimator in the few-label regime. Reliability remains task-dependent, as the gene result at batch 8 shows.
Added in "Few observations" (new Appendix E, blue).
We hope these clarifications and experiments address your concerns, and we would be glad to clarify anything further.
Response to Reviewer an8o (1/3)
We thank Reviewer an8o for the review, for finding our figures and tables "visually polished and clearly presented", and for the pointers to LLINBO and LABO. We address each point with new experiments, all run in the paper's loop with the same features, seeds and paired Wilcoxon tests over datasets.
[W1. Comparison with LLINBO and LABO]
Thank you for these pointers. Both are relevant, and we now discuss them in §2 and add empirical comparisons using adaptations to our setting. Our contribution is a specific mechanism for incorporating an unreliable prior: fitting its signed coefficient jointly with candidate features, allowing the surrogate to exploit, ignore, or invert the prior without further LLM queries during the discovery loop.
What they do.
How HARIBO differs.
Empirical comparison. Both methods need an LLM inside the loop, so we adapted their decision rules to our one-shot prior, with the LLM's "suggestion" at each round being its top-ranked unmeasured candidates:
Table R1. LLINBO vs. HARIBO, Recall@R10 (protein: 184 assays, PCA-50; gene: 6 screens; 10 seeds). All methods use the same cached prior; these comparisons evaluate adapted decision rules rather than the full original LLM-in-the-loop systems.
What the table shows. LLINBO's rules guarantee no-regret asymptotically, but in this short-horizon, batch regime they are conservative in both directions.