acceptodds
Under review as a conference paper at ICLR 2027

Biological Context Does Not Reliably Improve LLM-Guided Experimental Optimization

Abstract

Autonomous labs increasingly deploy large language models (LLMs) as closed-loop biology optimizers that propose a design, read the outcome, then propose again. This is partly because of an assumption that biological knowledge helps an LLM choose better designs than Bayesian optimization, which has no access to biological context. We test that assumption on 45 biology tasks constructed from 2,485 experimental rows across 341 published papers, scored by oracles fitted to those rows, each run over 30 iterations under two prompt conditions. The domain-aware condition provides the published parameter names, units, and labels, and the domain-agnostic renames them so they are not biologically meaningful, with search space and feedback identical. Across 19 LLMs, 14 sit at or below a GP-UCB reference in both conditions, and biological context costs further ground, domain-agnostic outperforming domain-aware on 58.7% of tasks by trajectory and 61.9% by endpoint. The failure is specific to biology, reversing on education tasks built the same way. Domain-aware prompting tends to help models propose better initial designs, while making them respond worse to feedback. Part of the reason is anchoring, with models keeping the conventional categorical choice even when the oracle scores a different (published best) value higher. Post-training Llama 3.1 8B to frontier-level raw optimization does not transfer to biology-aware optimization improvements. Biology-aware optimization therefore looks like a separate capability that autonomous laboratory loops should target directly.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
biology awareness, LLMs as optimizers, closed-loop experiments

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_huTGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether biological context can improve experimental optimization guided by LLMs. The authors constructed 45 tasks from 2,485 experimental rows in 341 published papers, providing scalar feedback through fitted surrogate evaluation models (oracles). Under a fixed 30-round query budget, 19 LLMs were compared under two prompt conditions: one preserving biological semantics (domain-aware) and the other anonymizing parameter names (domain-agnostic).

Pooled across models, the domain-aware condition achieved average win rates of 41.3% and 38.1% on the optimization trajectory (bsf-AUC@30) and endpoint (bsf-Outcome@30), respectively. The authors further analyzed the quality of the initial design, the model's tendency to adhere to conventional categories, the observed opposite patterns on educational tasks, and the effect of post-training Llama 3.1 8B on anonymized optimization trajectories. The paper argues that biological priors may interfere with the model's use of feedback during the optimization process, and that optimization within a biological context may constitute an independent capability distinct from abstract optimization.

Critical strengths

S1. This paper tests an important hypothesis across multiple tasks and models. The paper explores whether biological context can help language models make better decisions in sequence experiment optimization, which has significant practical implications for autonomous experimental systems. Experiments cover 45 tasks and 19 models, using the same search space, scoring function, and query budget under different conditions. This broad coverage makes negative results more compelling than a few selected failures. Even without proposing new optimization algorithms, this systematic empirical study contributes to the problem.

S2. The authors examine multiple alternative explanations and point out the limitations of the correlation analysis. The paper analyzes oracle-aligned subsets and reports statistical results at both the task and model levels; it also sets up educational domain controls, controls with fixed initial designs, and conducts post-training experiments. These analyses help identify which factors might contribute to the observed performance gaps. The appendix also clearly illustrates the uncertainties at the task level and the limited scope of the anchoring to convention analysis, enabling readers to distinguish which findings are experimentally supported and which explanations remain to be verified.

Critical weaknesses

W1. The core experiments do not adequately isolate the effect of biological semantics.

Prompt confounding. In §A.2, the domain-aware prompts ask the model to explain causal patterns and to propose and test a theory (including stating the specific causal claim under examination), whereas the anonymized prompts ask it to optimize the objective value and provide only a one-sentence explanation. The experiment thus varies semantic labels, reasoning demands, and output format simultaneously, so the performance difference cannot be attributed to biological context alone.

Existing controls do not remove this confounding. The control that restates the optimization direction on every turn addresses only a single prompt difference, and the effect reversal education tasks does not rule out an interaction between domain and prompt template. The critical missing control is one that keeps the instructions and output structure identical while replacing only the semantic labels.

The "separate capability" inference is overstrong. All training in §6 uses anonymized representations, whereas test-time evaluation switches to biological labels and a different prompt. Although the between-condition difference in training gains is indeed significant, it remains explainable by the train–test distribution shift, and is insufficient to establish that biology-aware optimization must be trained as an independent capability.

W2. Oracle validation and the mechanistic metrics are insufficient to support the stronger biological interpretation.

Limited scope of oracle validation. The row-wise leave-one-out validation and the alignment checks against the best published literature row do not directly verify ranking reliability for the novel combinations proposed by the optimizer. A shared scoring function ensures a consistent optimization target, but it cannot rule out mis-scoring of biologically plausible designs by the surrogate; equally, the current evidence does not establish that such bias has actually occurred.

The aligned subset is insufficient to rule out oracle artifacts. In §A.4, the mixed-effects endpoint win rate on the 18 aligned tasks is 41.5% [24.6%, 58.4%] (p = 0.33). Task variance is substantially larger than model variance, so the main text cannot claim to have ruled out oracle misspecification solely on the basis of the narrower model-level interval. The endpoint result over all tasks still supports a negative effect, but this support should be explicitly qualified as applying to the surrogate tasks evaluated here.

The mechanistic interpretation exceeds what the metrics measure. The +7.7 and −10.3 figures in §5.2 are percentage-point changes in matching the parameter values of the best published literature row, not direct measures of design quality or of the ability to respond to feedback. In the presence of parameter interactions, a given value from the best row need not transfer to other configurations; at this stage, the anchoring analysis is therefore more appropriately regarded as suggestive evidence.

Questions for authors

Q1. Prompt control (W1). Apart from the objective-direction restatement experiment, have the authors compared the two arms using the same instruction template, reasoning requirements, and output format, changing only the semantic labels and units? If so, does the performance gap remain?

Q2. Oracle construction and validation (W2). How are differences between source studies and missing parameter values handled when fitting and evaluating the oracles? In particular, how is the best published design scored if some of its parameters are missing? Do leave-one-source-study-out tests, or checks in the regions actually queried by the optimizers, support reliable design rankings? How does the claim in §4 that the result “is not an artifact of a misspecified oracle” fit with the uncertainty in the aligned subset’s task-level results?

Q3. Interpreting parameter match rates (W2). Why should changes in how often parameters match the best published design indicate changes in design quality or the ability to use feedback? The fixed-initial-design analysis in §B.2 is helpful because it measures objective values, but it does not establish this link for the match-rate metric. What evidence supports the stronger interpretations in §5.2 and §8? If that link remains uncertain, the claims should be limited to the behavior measured.

Q4. Interpreting transfer (W1). Could the transfer gap in §6 reflect sensitivity to changes in representation or instructions after anonymized training? A useful control would keep the prompt template fixed and use previously unseen, non-biological parameter names. What evidence separates these explanations? Without it, I would limit the conclusion to the finding that training gains transfer differently across the prompt conditions tested.

1 comment
AuthorsOpenReview ↗

Prompt design, oracle validation, and post-training transfer

Thank you for your detailed and constructive review. Your distinction between the findings and their interpretation helped us make several concrete revisions.

Q1. Prompt comparison. The instructions intentionally ask the aware model to use biological knowledge through mechanistic hypotheses. We have revised the abstract, introduction, and Section 3.1 to describe both the label and instruction differences. We have not run a matched-template control, so we do not attribute the gap to labels alone. The comparison of the two strategies under common tasks, scores, and budgets remains supported.

Q2. Oracle construction and statistical support. Appendix A.3 now explains missing-value handling and source-paper encoding. Missing numerical values use stored medians, missing categories are represented explicitly, and unreported fields in a published-best design invoke the saved preprocessing. Source-paper identifiers can enter as target-encoded features, with a fallback when absent.

Our audit identified target encoding before the outer validation split. Section 3.2 and Appendix A.3 explain why we no longer use the original R2R^2 as evidence of predictive validity. This does not change the recorded optimization outcomes. We have no study-held-out or trajectory-level measured validation.

Section 4 now puts the statistical analyses side by side. The full-benchmark endpoint disadvantage remains significant after accounting for task and model variation (38.4%38.4\%, 95% CI 28.8%,48.0%28.8\%,48.0\%; p=0.017p=0.017). The aligned-subset estimate points in the same direction but is less precise. We no longer claim that this subset rules out oracle artifacts.

Q3. Behavioral interpretation. We retain Figure 3 and specify its calculation in Appendix A.3. We revised Section 5.2 and the conclusion so categorical agreement is not presented as complete-design quality or causal feedback use. Section 5.2 also reports the existing fixed-start objective comparison. Claude Sonnet 4.6 favors the agnostic condition on 7 of 8 tasks (p=0.039p=0.039), while the Llama comparison is inconclusive. This provides separate evidence about performance beyond the opening design.

Q4. Transfer. Section 6 and the conclusion now describe unequal transfer rather than independent capabilities. Changes in representation and instructions could contribute, but the difference in trajectory gains is itself significant, +0.359+0.359 [0.124,0.6550.124,0.655], p=0.0065p=0.0065. This is stronger than significance in one condition and nonsignificance in the other. Better raw optimization did not produce comparable gains when the model was asked to use biological knowledge.

Reviewer_3PGsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates 19 LLMs on 45 fixed-parameter biological optimization tasks using tree-based oracles fitted to literature-derived measurements. Each run receives scalar feedback over 30 rounds under domain-aware or domain-agnostic prompting. The aware condition retains biological parameter names, units, and labels; the agnostic condition uses abstract labels. The paper reports lower aware-condition performance on average, examines initial proposals and subsequent feedback adaptation, and studies post-training transfer in Llama 3.1 8B. The two conditions also differ in role and explanation instructions, and the optimization outcomes are measured on fitted oracles rather than new laboratory experiments.

Critical strengths
  • The study evaluates a consequential assumption across many models and tasks. Examining both initial proposals and adaptation over repeated feedback provides useful evidence about where the tested prompting strategies differ. The breadth of the evaluation makes the observed pattern worth investigating.

  • The post-training experiment provides a complementary observation. Improvements trained under agnostic prompting transfer differently to the aware condition on held-out tasks. This strengthens the case for studying condition-dependent optimization behavior, although it does not independently isolate a biological-semantic mechanism.

Critical weaknesses
  • The prompt contrast does not isolate biological semantics. The aware and agnostic conditions change role framing and explanation instructions as well as names and labels. Holding the search space, oracle, and budget fixed controls other parts of the experiment, but cannot identify which prompt changes cause the performance difference. The education comparison and objective-direction reminder do not substitute for wording-matched prompts. This matters because biological context is the central explanatory variable of the paper.

  • The connection between oracle outcomes and actual biological optimization remains insufficiently established. The feedback landscape is learned from sparse literature records. The alignment check establishes where a published best design falls in the fitted oracle’s score range, but does not validate the landscape encountered along generated optimization trajectories. I need more direct evidence that the observed prompt difference corresponds to measured biological outcomes before assigning the main conclusion the significance claimed for experimental optimization.

Questions for authors
  • Can you compare prompts that are identical in role, length, rationale instructions, and objective wording, differing only in biological names, units, and labels? Does the observed gap persist?
  • What independent measured-data validation supports the oracle rankings along the regions explored by the optimization trajectories? Is the prompt-dependent outcome difference reproduced against held-out experimental measurements or another evaluation that directly checks those outcomes?
1 comment
AuthorsOpenReview ↗

Scope of the strategy comparison and surrogate evaluation

Thank you for your thoughtful review and for recognizing the value of the broad evaluation and post-training study.

Prompt comparison. The aware condition was designed to use biological knowledge through mechanistic hypotheses. Our experiment asks whether that approach improves optimization over an anonymized alternative. We have revised the abstract, introduction, and Section 3.1 to state that both labels and reasoning instructions differ. We have not run the requested matched-template experiment because of the high inference costs needed to run ~103,000 additional model calls. Thus we have revised language in the paper to not attribute the gap to biological semantics alone. Section 5.1 also clarifies that the education reversal does not isolate a labels-only effect.

The comparison still answers a practical question. Giving the model biological information and instructions for using it did not reliably improve performance under the same search spaces, feedback, and budgets.

Oracle validity. Section 3.2 and Appendix A.3 now distinguish evaluation against a fixed surrogate from validation against biological measurements. We do not have independent measured-outcome validation along generated trajectories, and the published-best alignment check does not provide it.

Our audit also identified target encoding before the outer validation split. We document this issue and retain the original R2R^2 for transparency, rather than as evidence of predictive validity. This correction does not change the recorded optimization outcomes on the fixed benchmark.

Supporting evidence. Section 4 now reports the task-aware estimates prominently. The endpoint disadvantage remains significant after accounting for task and model variation (38.4%38.4\%, 95% CI 28.8%,48.0%28.8\%,48.0\%; p=0.017p=0.017). Section 6 and the conclusion retain the unequal-transfer finding. The difference in post-training gains is itself significant (p=0.0065p=0.0065), although it does not identify a biological-semantic mechanism. These results support testing biology-aware optimization directly rather than assuming that domain knowledge or better raw optimization will improve it.

Reviewer_RKk9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates whether providing biologically meaningful parameter names, units, and categorical labels improves LLM-guided experimental optimization. The authors construct 45 biology optimization tasks from 2,485 experimental records across 341 papers and evaluate 19 LLMs under domain-aware and domain-agnostic prompting conditions, with identical parameter spaces, oracle feedback, and optimization budgets. They further evaluate education tasks, analyze the evolution of literature-optimum matching over iterations, and post-train Llama 3.1 8B to investigate whether improvements in raw optimization transfer to biology-aware optimization. The main findings are that biological context does not reliably improve optimization performance and that domain-aware prompting may encourage persistent preference for conventional choices despite feedback favoring alternatives.

Critical strengths

1. Important research question and potentially valuable negative findings. The paper investigates whether providing domain-specific biological context actually improves LLM-guided experimental optimization, challenging an important assumption in autonomous experimentation. The question is relevant to the community, and the finding that domain-aware prompting does not consistently improve optimization is potentially informative for the design of LLM-based scientific systems.

2. Broad empirical investigation. The evaluation covers 45 biological optimization tasks and 19 LLMs, complemented by education tasks, iteration-level analyses, and post-training experiments. The authors make a substantial effort to investigate not only whether the performance gap exists, but also when it emerges and what behavioral patterns may explain it.

Critical weaknesses

1. The experimental design does not isolate biological context from differences in reasoning instructions. The paper attributes the observed performance gap to biological semantics, but Appendix A.2 reveals that the two conditions differ in more than parameter names and labels. The domain-aware prompt explicitly asks models to identify patterns, formulate causal hypotheses, and propose configurations to test or exploit these hypotheses, whereas the domain-agnostic prompt primarily asks models to improve their objective based on previous feedback. These differences may independently affect exploration and feedback responsiveness, potentially explaining the behavior later interpreted as anchoring to conventional biological choices. Consequently, the central causal claim is not adequately supported by the current comparison. A matched-prompt experiment that holds reasoning instructions constant and varies only biological semantics is needed to distinguish these explanations.

2. The operationalization and statistical support for the proposed mechanism are insufficiently clear. Figure 3 is central to the claim that biological context improves initial designs but reduces responsiveness to feedback. However, the paper does not sufficiently clarify how the match rate is computed across mixed numerical and categorical parameter spaces, including which parameters contribute to the metric and how proposals and repeated runs are aggregated. This makes it difficult to determine precisely what the reported changes measure. In addition, Appendix A.4 reports a significant endpoint effect on oracle-aligned tasks under the model-level analysis but a nonsignificant result under the task-level mixed-effects analysis. These issues limit confidence in the quantitative evidence supporting the proposed mechanism.

Questions for authors

1. Critical experimental question: Can the authors provide a controlled comparison in which the two conditions use identical reasoning instructions, output requirements, and feedback prompts, differing only in whether parameter names and categorical labels carry biological meaning? If such a comparison is unavailable, how can the authors distinguish the effect of biological context from the effect of the additional hypothesis-generation and causal-reasoning instructions in the domain-aware condition?

2. Clarification of the central metric: Could the authors specify the exact computation of the Figure 3 match rate, including which parameters are evaluated, how continuous and categorical parameters are handled, and how proposals and repeated runs are aggregated? Which specific implementation reproduces the reported estimates, and how should the mechanism claim be interpreted given the differing statistical conclusions under model-level and task-level analyses?

2 comments
AuthorsOpenReview ↗

Scope of the comparison and supporting evidence

Thank you for your time and careful review. We appreciate your recognition of the question’s importance and the breadth of the evaluation.

Prompt comparison. The instruction difference was intentional. The aware condition asks the model to use biological knowledge through mechanistic hypotheses; the agnostic condition asks it to optimize from numerical feedback. We have revised the abstract, introduction, and Section 3.1 to state both differences explicitly. Our experiment tests whether this use of biological knowledge improves optimization under identical search spaces, feedback, and budgets. We have not run a matched-template comparison, so we do not separate the effects of labels and instructions.

Figure 3. We retain the plotted trend and clarify its calculation in Appendix A.3. It compares first and last stored proposals on one categorical parameter per task, selected by the spread in mean oracle scores across literature categories and requiring a reported published-best value. Numerical parameters are excluded. Matching uses normalized categorical equality, excluding missing or undecodable values. Agnostic final-observation counts supply the weights for both conditions. The saved table reproduces the +7.7+7.7 and −2.6-2.6 percentage-point differences.

We now describe this as categorical agreement rather than complete-design quality or causal feedback use. Section 5.2 also reports the existing fixed-start objective comparison. For Claude Sonnet 4.6, the agnostic trajectory advantage persists with the opening design held fixed on 7 of 8 tasks (p=0.039p=0.039); the Llama comparison is inconclusive.

Statistical support. The endpoint result is supported under both model-level and task-aware analyses. Accounting for task and model variation gives an aware-condition win rate of 38.4%38.4\% [95% CI 28.8%,48.0%28.8\%,48.0\%], significantly below 50%50\% (p=0.017p=0.017). The aligned-subset estimate points in the same direction but is less precise, so we do not use that subset alone to establish the effect. We also document a target-encoding issue in oracle validation and no longer rely on the original R2R^2 to establish biological generalization. That correction does not change the recorded benchmark comparisons.

Thank you for the detailed response and the revisions.

The clarification of the match rate in Appendix A.3 and the additional task- and model-aware analysis address my concerns on these points.

However, my main concern remains. The revision now states that labels and reasoning instructions vary together, and the authors acknowledge that the two effects cannot be separated. Although this makes the claims more accurate, it also narrows the contribution. So the central question is still not directly answered. Without a matched-template comparison, the conclusions about biological context and anchoring remain difficult to interpret.

Therefore, I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.