acceptodds
Withdrawn from ICLR 2027

TadA-Bio: A Replayable Biomedical AI-Agent Benchmark for Budgeted Directed Evolution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scientific agents in biological discovery campaigns must coordinate sequential operations: choosing which branch to open, how much evidence to collect, which candidates to validate, and when to stop under finite time and budget. Existing scientific evaluations in protein engineering largely emphasize sequence prediction or candidate ranking, leaving the role of an agent as the operator of a budgeted, long-horizon campaign underexplored. We introduce TadA-Bio, a replayable benchmark for evaluating such decision policies. It is built from measured sequences, activities, and provenance from a historical 31-round TadA directed-evolution campaign, but adds a controlled replay layer: each episode exposes a partial public state, hides future candidates and activities, and uses fixed benchmark costs, times, screening noise, and reward rules. An agent can prepare seeds, open libraries, screen, confirm hits, acquire BioSkills, or stop; terminal cash is the primary outcome. Across heuristic policies, three LLM-agent harnesses (Codex, OpenCode, and Claude Code), budget sweeps, and BioSkill settings, the benchmark separates productive operation from cash preservation, wasteful exploration, runtime failures, and unproductive skill purchase. TadA-Bio is a playable, discriminative, and diagnostic offline benchmark, providing a grounded example and potential foundation for future automated biological laboratories.

Primary area:
datasets and benchmarks
Keywords:
AI agents, scientific discovery, biomedical benchmarks, directed evolution, budgeted planning, tool use

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YXJLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a replayable benchmark for evaluating AI agents on budget-constrained, long-horizon tasks, using data from a 31-round TadA directed-evolution campaign. It evaluates three LLM-agent harnesses on these tasks.

Critical strengths

The benchmark combines real experimental data, long-horizon decision-making, and replayability, with a valuable focus on budget constraints that matter in laboratory practice.

Critical weaknesses

The paper provides insufficient background on the TadA task and assumes too much domain-specific knowledge. It also does not clearly explain how the recorded campaign is converted into an interactive environment. In particular, how the environment handles situations not covered by the campaign remains unclear—an essential issue for a long-horizon benchmark.

Questions for authors
  1. Informative figures. Figures 1 and 2 do not adequately explain how the benchmark works. Figure 1 resembles an AI-generated promotional poster, occupying substantial space while conveying little technical information. Figure 2 lacks a clear sequence of steps and an adequate explanation of the environment. Please revise both figures to communicate the task and its execution more clearly.
  2. Replayability and experimental validity. I understand “replayable” to mean that the benchmark can be rerun. However, the recorded campaign covers only a limited set of trajectories. How does the environment respond when an agent chooses an action outside the recorded data, and how do you ensure that its feedback is consistent with real experimental outcomes? Please address this explicitly in the response and revise the main text accordingly.
Reviewer_QJB3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TadA-Bio is an offline benchmark that evaluates agents as operators of budget-constrained protein-engineering campaigns. It converts records from a historical 31-round TadA directed-evolution campaign into a partially observed environment with branch preparation, library screening, candidate confirmation, optional BioSkill purchases, and stopping decisions. Performance is measured primarily by terminal profit under a predefined cost and reward schedule. The task formulation is useful, but the experiments show system-level performance differences more clearly than they show that the benchmark reflects biologically meaningful experimental planning.

Critical strengths
  1. Relevant evaluation objective. Evaluating complete experimental trajectories captures resource allocation and stopping decisions that static sequence-ranking tasks miss.
  2. Grounding in measured data. Historical sequence–activity records provide a stronger empirical foundation than entirely synthetic outcomes.
  3. Controlled evaluation. A host-side judge separates hidden outcomes from agent observations, while matched replay seeds support comparisons across configurations.
Critical weaknesses
  1. The central capability claim is insufficiently validated. The environment may primarily measure generic resource allocation over a fixed search space. Screening already reveals activity with bounded multiplicative noise, and agents cannot propose arbitrary new sequences. The paper does not establish whether biological sequence information materially improves decisions beyond candidate identifiers, observed scores, costs, and branch structure. A sequence-masking control would directly test this issue.

  2. Historical grounding does not establish realistic decision dynamics. The measurements are real, but the replay layer defines branch unlocking, costs, perturbations, and rewards. It is unclear whether agents must identify and confirm the correct parent or simply access a predefined library.

  3. The baselines do not establish the difficulty of the planning problem. Information equivalence ensures fairness but not strong baseline performance. The paper lacks detail on heuristic tuning and does not include stronger public-information planners. It is therefore unclear how much agents gain from sophisticated reasoning beyond a well-designed screening and budget-allocation policy.

  4. The BioSkill results lack a clear mechanism. The skills are mainly sequence-analysis workflows, but their contents, execution requirements, and actual use are not sufficiently documented. The experiments do not distinguish added computational capability from workflow guidance or context changes. Lower market performance could reflect cost, delay, poor selection, or later misuse.

Questions for authors
  1. How does performance change when sequences are replaced with anonymous identifiers while scores, costs, and topology remain available?
  2. Can the authors provide complete BioSkill examples and traces showing how skill outputs changed subsequent actions?
  3. Do the principal findings hold against stronger planners and under alternative cost, reward, and noise calibrations?
Reviewer_Jwt1OpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Disclaimer: I am not a biomedical AI expert, AC has been notified, my review will be associated with a low confidence score. I tried to provide an honest reading and interpretation of this work to the best of my capabilities. I can’t judge the ecological validity of this work.

I was able to get a general gist of the paper, which tries to implement a re-playable benchmark for protein engineering for agents in long horizon tasks. The idea is to observe how different agents (referred to as “budgeted campaign operators”) take different paths that account for different costs.

Critical strengths

This could be a very useful benchmark. I like the replayability idea a lot. The idea I think is incredibly sound and I haven't seen a benchmark like this.

Critical weaknesses

The presentation is too high level and never goes deep enough where it matters: Since TadA-Bio is derived from TadA one would expect a more explicit description of what a TadA campaign is, with possibly a fully or partially unrolled example. Concepts are introduced and basically never defined, the intro says “confirms hits, purchases BioSkills” and I don’t know what both of these are (I googled BioSkills but not sure if what I have found is actually correct). The information provided is not - at least for me - enough to understand how this benchmark is set up and then unrolled. Design choices do not often seem to be justified. Several sentences, even after re-reading, do not seem to help me understand or are using terms that I don't think were carefully defined.

Figure 2 seems AI generated, this would not be a problem if there were no errors in the image. Like the dice with two “one” sides. The figure also mentions “Action Writer” or just “Writer” that I am not sure what it means and it’s never mentioned again (I could go on, like what is “JudgeCore”?). Same for Figure 1 where I have no idea what “high titness” means (actually I just realized it’s “fitness”, but still…). The very same figure also says “TadA-Bio evaluates agents as operators of a budgeted BioFoundry campaign”, no idea what “BioFoundry” is, it’s probably a name that was never updated. Again, I do not have a problem with AI generated images, but these contain many errors and undefined concepts. I don’t like over-indexing on this aspect, but the big issue is it kind of leaves me a bit concerned, even more so considering that I wanted to rely on figures to better understand this paper.

This suggests that, contrary to what the authors’ LLM statement says, the paper was not carefully reviewed after generation (which could also affect other parts of this work).

Questions for authors

A question I have is that it’s very hard to measure the significance of this work when compared to the TadA benchmark (which is cited). I think it would be good to have a little bit more clarity on how data was extracted and used. I would argue that this might partially affect the novelty - again hard for me to judge - but not realistically the usefulness of the contribution itself.

Many decisions are not justified - unless some of these are standard in the field - for example “mine_library returns reproducible noisy observations for latent activities using multiplicative clipped Gaussian noise with zero mean, standard deviation 0.10, and clipping at plus/minus 0.20.” why?

Re code: seems authors run containers with --cap-add SYS_ADMIN --cap-add NET_ADMIN --security-opt seccomp=unconfined --security-opt apparmor=unconfined. I feel like this is pretty risky (even more so if - can’t really understand this from a quick skim - agents run inside the containers).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.