- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper evaluates coding agents on DREAM computational biology challenges, using historical expert-team leaderboards as a reference. It introduces a containerized evaluation framework and studies repeated-run performance, additional task context, and test-time optimization. It also audits unusually high scores for leakage and compares human and LLM judgments of biological appropriateness. Under guided prompts, no agent’s mean score exceeds the historical human best on the 14 matched tasks, although some individual runs do.
Critical strengths
- Historical competition results provide a meaningful reference for agent performance. The paper asks whether agents produce competitive scientific models, beyond merely completing a workflow. Repeated runs and task-specific scoring make this comparison more informative than isolated demonstrations, and the framework could be useful for tracking progress.
- The leakage analysis is concrete and valuable. Appendix F distinguishes direct retrieval of test labels, test-label use during model selection, and shortcuts through available metadata. I appreciated the authors’ acknowledgment that an underspecified task can permit unintended shortcuts without implying deliberate deception. This analysis shows why impressive scores need inspection before being treated as evidence of scientific capability.
Critical weaknesses
- The optimization experiment does not preserve an independent final test set. AutoResearch receives scores on newly sampled 80% subsets of the host holdout during optimization, then is evaluated on the full same holdout (App. D). Resampling overlapping subsets does not remove adaptive reuse of test information. Combined with different agent models and unequal budgets across methods, this makes Figure 4 difficult to interpret as evidence of generalization gains from a particular optimization strategy.
- The biological-judgment calibration is too small to support a reliable correction claim. The correction is fitted on 37 solutions and evaluated on only seven (App. E). The pooled mean moves slightly closer to the human mean, but individual corrected means move farther away, and the uncertainty is substantial. This is an interesting exploratory study of rater disagreement, but it does not yet establish that calibration makes LLM judgments more trustworthy.
Questions for authors
- Can the optimization results be evaluated on a final set never used for intermediate feedback, with a comparable-budget repeated-single-run baseline?
- Is leakage screening applied consistently to all retained runs, or mainly triggered by above-human scores? Please clarify how less obvious leakage in ordinary-scoring runs is handled.
- Given the seven-solution calibration evaluation and wide uncertainty, would you narrow the claim to an exploratory analysis of agreement and calibration rather than evidence of improved judgment?
5 comments
Thanks authors for new added experiments and revised version of manuscript, I have checked them and have some follow up questions regrading to the replies of my previous questions:
-
Since in Figure 4, you ran only 1 campaign per setting, and the score is statistically not different from single runs. Does this mean the benefit of TTO is still inconclusive on this task?
-
I am still conerned about the initial leakage screen, how do you make sure the LLM rater did not miss subtle metadata shortcuts?
We also thank the reviewer for the very fast reflexes in their review!
The reviewer is correct, at the current level of experimentation, the benefit of TTO over independent runs is not established and at an N=1, we are unable to make any statistical claims. For the camera ready version, we plan on increasing this experiment to an N=3, 3 independent runs per roster set for AutoResearch and 3 for GEPA and RSA and try to compare their advantage then, and also extend to another challenge (BEAT-AML). Our current replays simply assess variability in the produced code, not in the process itself.
About their second point: We can never guarantee that an LLM based screen for cheating/leakage will be complete and will detect the potentially nuanced ways in which an LLM could cheat during its modeling process. We have not, and there is obviously survivor bias here, seen cases where the LLM maliciously fetched something pertinent to the answer, realized that was against the rules, and subsequently hid its behavior. Usually, it is a problem of the task definition not being sufficiently strict, being an example of intent drift.
We conducted a random review of obvious web searches and found that they do not occur (e.g. simply fetching results). On the MOD Q1 and our observed shortcuts, all known issues (taking advantage of metadata to enhance performance) were flagged by an LLM, even in runs where the score is below human performance.
Of course, we do not know how many false negatives may be hiding there. To attempt to address this, we will perform a screen of runs flagged as negatives (results were cheating was not identified) to understand the actual sensitivity of this method. We were focused mostly on the superhuman performing runs as we see those as an important step in establishing agents' performance in biomedical data science tasks.
we thank the reviewer for the thorough review and their comments! We understand and take their concerns very seriously and have designed extra experiments to address them.
AutoResearch:
We added a new comparison with disjoint fitting search selection sets of sizes 220/73/74 from the original training samples, making it so that the final 368 are never seen before evaluation. The historical exposure though does remain a limitation. we performed mixed autoresearch and single agent autoresearch with 50 steps, split 36 across searching, 12 across selection and 3 for the final replays. We also run single model campaigns (four collaborators of 1 model) with more modern models. While this is simply one campaign per each arm (mixed an 4 single agent ones), lowering our statistical power, we did not observe any mean superhuman performance. However, we are toning down our claim about AutoResearch being worse than individual runs, as it is statistically indistinguishable from the best expected runs.
Manuscript locations: Figure 1 caption, p. 2; Appendix F, “Audit coverage,” p. 32, source-check examples, pp. 32–37, and MOD mechanisms, p. 39; corrected MOD Q1a inputs and limits, Appendix A, p. 15.
We appreciate the concerns about budget and echo them. Due to the methods working very differently to each other, it is hard to truly equalize the budgets. We added figures that report Agent hours taken and will be performing token and cost calculations for the final version.
Manuscript locations: Appendix A, Figure A.2 and caption, p. 16; historical design and its limits, Section 5, p. 7, Figure 4, p. 9, and Appendix D, pp. 26–27.
Calibration
We have revised the claims about judge calibration and still urge the community to consider this. We also mention in our future work that we will scale up our own analysis.
Leakage Screens
We specify in Appendix F what our initial screen covers and how we apply additional scrutiny to superhuman runs. As agents have been repeatedly proven to be at least competent in biomedical tasks, we prioritized a secondary audit from the authors of superhuman runs. In that audit, the authors checked the trace, the challenge setup, the produced code to understand whether any kind of leakage occured.