- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether biological context can improve experimental optimization guided by LLMs. The authors constructed 45 tasks from 2,485 experimental rows in 341 published papers, providing scalar feedback through fitted surrogate evaluation models (oracles). Under a fixed 30-round query budget, 19 LLMs were compared under two prompt conditions: one preserving biological semantics (domain-aware) and the other anonymizing parameter names (domain-agnostic).
Pooled across models, the domain-aware condition achieved average win rates of 41.3% and 38.1% on the optimization trajectory (bsf-AUC@30) and endpoint (bsf-Outcome@30), respectively. The authors further analyzed the quality of the initial design, the model's tendency to adhere to conventional categories, the observed opposite patterns on educational tasks, and the effect of post-training Llama 3.1 8B on anonymized optimization trajectories. The paper argues that biological priors may interfere with the model's use of feedback during the optimization process, and that optimization within a biological context may constitute an independent capability distinct from abstract optimization.
Critical strengths
S1. This paper tests an important hypothesis across multiple tasks and models. The paper explores whether biological context can help language models make better decisions in sequence experiment optimization, which has significant practical implications for autonomous experimental systems. Experiments cover 45 tasks and 19 models, using the same search space, scoring function, and query budget under different conditions. This broad coverage makes negative results more compelling than a few selected failures. Even without proposing new optimization algorithms, this systematic empirical study contributes to the problem.
S2. The authors examine multiple alternative explanations and point out the limitations of the correlation analysis. The paper analyzes oracle-aligned subsets and reports statistical results at both the task and model levels; it also sets up educational domain controls, controls with fixed initial designs, and conducts post-training experiments. These analyses help identify which factors might contribute to the observed performance gaps. The appendix also clearly illustrates the uncertainties at the task level and the limited scope of the anchoring to convention analysis, enabling readers to distinguish which findings are experimentally supported and which explanations remain to be verified.
Critical weaknesses
W1. The core experiments do not adequately isolate the effect of biological semantics.
Prompt confounding. In §A.2, the domain-aware prompts ask the model to explain causal patterns and to propose and test a theory (including stating the specific causal claim under examination), whereas the anonymized prompts ask it to optimize the objective value and provide only a one-sentence explanation. The experiment thus varies semantic labels, reasoning demands, and output format simultaneously, so the performance difference cannot be attributed to biological context alone.
Existing controls do not remove this confounding. The control that restates the optimization direction on every turn addresses only a single prompt difference, and the effect reversal education tasks does not rule out an interaction between domain and prompt template. The critical missing control is one that keeps the instructions and output structure identical while replacing only the semantic labels.
The "separate capability" inference is overstrong. All training in §6 uses anonymized representations, whereas test-time evaluation switches to biological labels and a different prompt. Although the between-condition difference in training gains is indeed significant, it remains explainable by the train–test distribution shift, and is insufficient to establish that biology-aware optimization must be trained as an independent capability.
W2. Oracle validation and the mechanistic metrics are insufficient to support the stronger biological interpretation.
Limited scope of oracle validation. The row-wise leave-one-out validation and the alignment checks against the best published literature row do not directly verify ranking reliability for the novel combinations proposed by the optimizer. A shared scoring function ensures a consistent optimization target, but it cannot rule out mis-scoring of biologically plausible designs by the surrogate; equally, the current evidence does not establish that such bias has actually occurred.
The aligned subset is insufficient to rule out oracle artifacts. In §A.4, the mixed-effects endpoint win rate on the 18 aligned tasks is 41.5% [24.6%, 58.4%] (p = 0.33). Task variance is substantially larger than model variance, so the main text cannot claim to have ruled out oracle misspecification solely on the basis of the narrower model-level interval. The endpoint result over all tasks still supports a negative effect, but this support should be explicitly qualified as applying to the surrogate tasks evaluated here.
The mechanistic interpretation exceeds what the metrics measure. The +7.7 and −10.3 figures in §5.2 are percentage-point changes in matching the parameter values of the best published literature row, not direct measures of design quality or of the ability to respond to feedback. In the presence of parameter interactions, a given value from the best row need not transfer to other configurations; at this stage, the anchoring analysis is therefore more appropriately regarded as suggestive evidence.
Questions for authors
Q1. Prompt control (W1). Apart from the objective-direction restatement experiment, have the authors compared the two arms using the same instruction template, reasoning requirements, and output format, changing only the semantic labels and units? If so, does the performance gap remain?
Q2. Oracle construction and validation (W2). How are differences between source studies and missing parameter values handled when fitting and evaluating the oracles? In particular, how is the best published design scored if some of its parameters are missing? Do leave-one-source-study-out tests, or checks in the regions actually queried by the optimizers, support reliable design rankings? How does the claim in §4 that the result “is not an artifact of a misspecified oracle” fit with the uncertainty in the aligned subset’s task-level results?
Q3. Interpreting parameter match rates (W2). Why should changes in how often parameters match the best published design indicate changes in design quality or the ability to use feedback? The fixed-initial-design analysis in §B.2 is helpful because it measures objective values, but it does not establish this link for the match-rate metric. What evidence supports the stronger interpretations in §5.2 and §8? If that link remains uncertain, the claims should be limited to the behavior measured.
Q4. Interpreting transfer (W1). Could the transfer gap in §6 reflect sensitivity to changes in representation or instructions after anonymized training? A useful control would keep the prompt template fixed and use previously unseen, non-biological parameter names. What evidence separates these explanations? Without it, I would limit the conclusion to the finding that training gains transfer differently across the prompt conditions tested.
Prompt design, oracle validation, and post-training transfer
Thank you for your detailed and constructive review. Your distinction between the findings and their interpretation helped us make several concrete revisions.
Q1. Prompt comparison. The instructions intentionally ask the aware model to use biological knowledge through mechanistic hypotheses. We have revised the abstract, introduction, and Section 3.1 to describe both the label and instruction differences. We have not run a matched-template control, so we do not attribute the gap to labels alone. The comparison of the two strategies under common tasks, scores, and budgets remains supported.
Q2. Oracle construction and statistical support. Appendix A.3 now explains missing-value handling and source-paper encoding. Missing numerical values use stored medians, missing categories are represented explicitly, and unreported fields in a published-best design invoke the saved preprocessing. Source-paper identifiers can enter as target-encoded features, with a fallback when absent.
Our audit identified target encoding before the outer validation split. Section 3.2 and Appendix A.3 explain why we no longer use the original R2 as evidence of predictive validity. This does not change the recorded optimization outcomes. We have no study-held-out or trajectory-level measured validation.
Section 4 now puts the statistical analyses side by side. The full-benchmark endpoint disadvantage remains significant after accounting for task and model variation (38.4%, 95% CI 28.8%,48.0%; p=0.017). The aligned-subset estimate points in the same direction but is less precise. We no longer claim that this subset rules out oracle artifacts.
Q3. Behavioral interpretation. We retain Figure 3 and specify its calculation in Appendix A.3. We revised Section 5.2 and the conclusion so categorical agreement is not presented as complete-design quality or causal feedback use. Section 5.2 also reports the existing fixed-start objective comparison. Claude Sonnet 4.6 favors the agnostic condition on 7 of 8 tasks (p=0.039), while the Llama comparison is inconclusive. This provides separate evidence about performance beyond the opening design.
Q4. Transfer. Section 6 and the conclusion now describe unequal transfer rather than independent capabilities. Changes in representation and instructions could contribute, but the difference in trajectory gains is itself significant, +0.359 [0.124,0.655], p=0.0065. This is stronger than significance in one condition and nonsignificance in the other. Better raw optimization did not produce comparable gains when the model was asked to use biological knowledge.