- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces ReproBAIT, a benchmark for evaluating whether coding agents can reproduce biological AI tools from limited descriptions. The benchmark covers three tools with different levels of complexity and compares several coding agents under different information conditions. The results show that agents can reproduce some functions of simpler tools, while struggling with more complex ones. The paper also examines the possible effects of memorization and agent harnesses.
Critical strengths
- The benchmark includes both open-source and closed-source tools, providing different evaluation settings.
- The experiments consider information access, model complexity, and memorization, which helps explain the observed results.
- The evaluation environment is reasonably controlled, with fixed compute budgets and held-out test data.
Critical weaknesses
- Only three biological AI tools are evaluated. This is too small to support broad conclusions about the reproduction capabilities of coding agents.
- Each tool is mainly evaluated using one specific metric. Achieving a similar score does not necessarily mean that the original tool has been successfully reproduced.
- Some target tools may have appeared in the training data of the evaluated LLMs. Although the authors investigate this issue, the experiments cannot clearly separate memorization from actual reproduction ability.
- Different models use different agent harnesses, and the main results report the best of three attempts. This makes it difficult to compare the agents fairly and assess the stability of their performance.
- The main contribution is an evaluation framework based on existing coding agents and biological tools. The paper does not introduce a new agent method or learning algorithm.
Questions for authors
- Can the authors explain why only three tools were selected and whether the findings would hold for a larger set of biological AI tools?
- How can the benchmark distinguish genuine tool reproduction from achieving a high score on a limited evaluation metric?
- Can the authors provide stronger evidence that the results are not mainly influenced by memorization?
- Why are the best results from three attempts used instead of reporting mean performance with standard deviations?
- Can the authors provide more controlled comparisons using the same agent harness across different models?
Thank you for the helpful feedback. We address each of your questions below:
1. Three tools and the benchmark contribution
We agree that only three tools cannot support conclusive statements about general BAIT reproducibility. We selected three illustrative use cases to gain initial insights into what is feasible and to guide future research. We revised the abstract (lines 20–23) and introduction (lines 97–99) to narrow the claims accordingly. We also expect this framework to be most interesting when applied to closed source tools.
We view a primary contribution of our paper as the framework itself, allowing a scalable approach to evaluate agentic capabilities for replicating biological AI tools—a task that has not, to our knowledge, been reported elsewhere in the literature. We note that the paper was submitted in the datasets and benchmarks track. We believe benchmark infrastructure for assessing emerging capabilities is a valuable contribution, consistent with other papers in this track, and will catalyze further investigation of these capabilities. Based on the results of the paper and our limited experimentation on further BAITs, we expect future runs will find further evidence of BAIT reproducibility across a range of biological functions.
2. Task scope and metrics
We acknowledge that a high score on a single metric does not establish that the original BAIT has been reproduced. Our primary metrics are deliberately narrow so that scoring can be automated and repeated.
We have revised the text to state each task's scope explicitly and have now added all calculated metrics per attempt in Appendix B. These provide additional context for any discrepancy between reproducing a specific functionality and reproducing the broader tool. For example, on RFdiffusion3 the highest designability scores come from attempts whose backbones are near-duplicates of known PDB entries, so the primary score alone would have been misleading. We highlight this in the results.
3. Memorisation evidence
We agree that our experiments cannot definitively separate memorisation from reproduction ability. Our transcript diagnostics detect evidence consistent with prior recall but cannot fully quantify its contribution. Our code similarity check can detect verbatim code regurgitation, but cannot detect rephrased implementations. However, we argue that the combination of the two checks provides a useful diagnostic signal for the overall memorisation of the given BAIT. Given the secrecy around training datasets for frontier models, we hope that future research will help shed light onto the effects of memorisation.
That said, we would like to emphasise that CMD’s training pipeline is closed-source, so there is no code for an agent to have memorised. Agents still match the published validity score. Further evidence to rule out memorization would involve the addition of further closed-source tools or those past the training-data cutoff of the evaluated model; but this is beyond the scope of the paper’s initial exploration.
4. Best-of-three
The best-of-three framing was intended to represent a motivated user who can retry several times and use a successful result (where success criteria are known in advance as they are here). However, we agree that this limited a full reading of the results distribution. We have updated the manuscript to report all three replicates with means and standard deviations for each task.
5. Harness comparisons
The original harnesses were included to correspond to practical usage for the evaluated models, but we agree this may have confounded the comparison between models. We now use the same ReAct harness across all four models and all three tasks.
The paper-only results under the common ReAct harness are shown below. Each cell gives mean fidelity (% of the task reference) ± sample SD over three attempts, each with a 12-hour allowance.
We include a harness comparison in § 3.3 as evidence that ReAct does not appear to disadvantage the agents we tested on ProteinMPNN compared to the proprietary harnesses.