- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a replayable benchmark for evaluating AI agents on budget-constrained, long-horizon tasks, using data from a 31-round TadA directed-evolution campaign. It evaluates three LLM-agent harnesses on these tasks.
Critical strengths
The benchmark combines real experimental data, long-horizon decision-making, and replayability, with a valuable focus on budget constraints that matter in laboratory practice.
Critical weaknesses
The paper provides insufficient background on the TadA task and assumes too much domain-specific knowledge. It also does not clearly explain how the recorded campaign is converted into an interactive environment. In particular, how the environment handles situations not covered by the campaign remains unclear—an essential issue for a long-horizon benchmark.
Questions for authors
- Informative figures. Figures 1 and 2 do not adequately explain how the benchmark works. Figure 1 resembles an AI-generated promotional poster, occupying substantial space while conveying little technical information. Figure 2 lacks a clear sequence of steps and an adequate explanation of the environment. Please revise both figures to communicate the task and its execution more clearly.
- Replayability and experimental validity. I understand “replayable” to mean that the benchmark can be rerun. However, the recorded campaign covers only a limited set of trajectories. How does the environment respond when an agent chooses an action outside the recorded data, and how do you ensure that its feedback is consistent with real experimental outcomes? Please address this explicitly in the response and revise the main text accordingly.