acceptodds
Under review as a conference paper at ICLR 2027

Can agents outperform human expert crowdsourcing? Insights from DREAM

Abstract

LLM-powered coding agents are increasingly claimed to be able to perform the work of a human scientist on open-ended, real-world tasks in computational biology. Existing benchmarks with pre-specified answers can only establish whether an agent is correct, but not whether it is competitive with a domain expert who attacked the same open problem. Here, we present a living, extensible benchmark of commercial and open-source AI agents on DREAM (Dialogue on Reverse Engineering Assessment and Methods) Challenges, which are crowdsourced competitions with hidden test data, expert-crafted scoring, and human team leaderboards. Our tasks evaluate end-to-end computational biology workflows against expert teams from crowdsourced challenges. Under guided prompt settings, agents' mean score never surpasses the human best, though individual runs did on 2 tasks. Surprisingly, in some cases where agent received less context, agents outperformed the best human solutions and we did not find statistically significant gains with increased context. Worryingly, in some cases when agents outperform the top historical human solution, careful examination of the traces can reveal nuanced ways of exploiting leaked data attributes, which we exclude from our comparisons. Additionally, we compare three test-time optimization methods for improving agent performance on one of our tasks, juxtaposing approaches that optimize the prompt, the code, and agent swarms in an exploratory case study on one task. Agent swarms show the largest increase in performance, consistent with other test-time scaling findings, though careful guards should be implemented to avoid gaming leaderboards. Finally, we introduce a multi-layered evaluation pipeline combining ground-truth metrics, LLM-as-judge scoring, and human calibration to characterize agent's domain know-how. Our open-source framework and leaderboard aim to support the responsible adoption of AI agents in the life sciences, and we invite the community to submit both agents and challenges.

Primary area:
datasets and benchmarks
Keywords:
Evaluation, Responsible AI, Agents, Crowdsourcing, Challenges

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_nPtZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates coding agents on DREAM computational biology challenges, using historical expert-team leaderboards as a reference. It introduces a containerized evaluation framework and studies repeated-run performance, additional task context, and test-time optimization. It also audits unusually high scores for leakage and compares human and LLM judgments of biological appropriateness. Under guided prompts, no agent’s mean score exceeds the historical human best on the 14 matched tasks, although some individual runs do.

Critical strengths
  1. Historical competition results provide a meaningful reference for agent performance. The paper asks whether agents produce competitive scientific models, beyond merely completing a workflow. Repeated runs and task-specific scoring make this comparison more informative than isolated demonstrations, and the framework could be useful for tracking progress.
  2. The leakage analysis is concrete and valuable. Appendix F distinguishes direct retrieval of test labels, test-label use during model selection, and shortcuts through available metadata. I appreciated the authors’ acknowledgment that an underspecified task can permit unintended shortcuts without implying deliberate deception. This analysis shows why impressive scores need inspection before being treated as evidence of scientific capability.
Critical weaknesses
  1. The optimization experiment does not preserve an independent final test set. AutoResearch receives scores on newly sampled 80% subsets of the host holdout during optimization, then is evaluated on the full same holdout (App. D). Resampling overlapping subsets does not remove adaptive reuse of test information. Combined with different agent models and unequal budgets across methods, this makes Figure 4 difficult to interpret as evidence of generalization gains from a particular optimization strategy.
  2. The biological-judgment calibration is too small to support a reliable correction claim. The correction is fitted on 37 solutions and evaluated on only seven (App. E). The pooled mean moves slightly closer to the human mean, but individual corrected means move farther away, and the uncertainty is substantial. This is an interesting exploratory study of rater disagreement, but it does not yet establish that calibration makes LLM judgments more trustworthy.
Questions for authors
  1. Can the optimization results be evaluated on a final set never used for intermediate feedback, with a comparable-budget repeated-single-run baseline?
  2. Is leakage screening applied consistently to all retained runs, or mainly triggered by above-human scores? Please clarify how less obvious leakage in ordinary-scoring runs is handled.
  3. Given the seven-solution calibration evaluation and wide uncertainty, would you narrow the claim to an exploratory analysis of agreement and calibration rather than evidence of improved judgment?
5 comments

we thank the reviewer for the thorough review and their comments! We understand and take their concerns very seriously and have designed extra experiments to address them.

AutoResearch:

We added a new comparison with disjoint fitting search selection sets of sizes 220/73/74 from the original training samples, making it so that the final 368 are never seen before evaluation. The historical exposure though does remain a limitation. we performed mixed autoresearch and single agent autoresearch with 50 steps, split 36 across searching, 12 across selection and 3 for the final replays. We also run single model campaigns (four collaborators of 1 model) with more modern models. While this is simply one campaign per each arm (mixed an 4 single agent ones), lowering our statistical power, we did not observe any mean superhuman performance. However, we are toning down our claim about AutoResearch being worse than individual runs, as it is statistically indistinguishable from the best expected runs.

Manuscript locations: Figure 1 caption, p. 2; Appendix F, “Audit coverage,” p. 32, source-check examples, pp. 32–37, and MOD mechanisms, p. 39; corrected MOD Q1a inputs and limits, Appendix A, p. 15.

We appreciate the concerns about budget and echo them. Due to the methods working very differently to each other, it is hard to truly equalize the budgets. We added figures that report Agent hours taken and will be performing token and cost calculations for the final version.

Manuscript locations: Appendix A, Figure A.2 and caption, p. 16; historical design and its limits, Section 5, p. 7, Figure 4, p. 9, and Appendix D, pp. 26–27.

Calibration

We have revised the claims about judge calibration and still urge the community to consider this. We also mention in our future work that we will scale up our own analysis.

Leakage Screens

We specify in Appendix F what our initial screen covers and how we apply additional scrutiny to superhuman runs. As agents have been repeatedly proven to be at least competent in biomedical tasks, we prioritized a secondary audit from the authors of superhuman runs. In that audit, the authors checked the trace, the challenge setup, the produced code to understand whether any kind of leakage occured.

Thanks authors for new added experiments and revised version of manuscript, I have checked them and have some follow up questions regrading to the replies of my previous questions:

  1. Since in Figure 4, you ran only 1 campaign per setting, and the score is statistically not different from single runs. Does this mean the benefit of TTO is still inconclusive on this task?

  2. I am still conerned about the initial leakage screen, how do you make sure the LLM rater did not miss subtle metadata shortcuts?

We also thank the reviewer for the very fast reflexes in their review!

The reviewer is correct, at the current level of experimentation, the benefit of TTO over independent runs is not established and at an N=1, we are unable to make any statistical claims. For the camera ready version, we plan on increasing this experiment to an N=3, 3 independent runs per roster set for AutoResearch and 3 for GEPA and RSA and try to compare their advantage then, and also extend to another challenge (BEAT-AML). Our current replays simply assess variability in the produced code, not in the process itself.

About their second point: We can never guarantee that an LLM based screen for cheating/leakage will be complete and will detect the potentially nuanced ways in which an LLM could cheat during its modeling process. We have not, and there is obviously survivor bias here, seen cases where the LLM maliciously fetched something pertinent to the answer, realized that was against the rules, and subsequently hid its behavior. Usually, it is a problem of the task definition not being sufficiently strict, being an example of intent drift.

We conducted a random review of obvious web searches and found that they do not occur (e.g. simply fetching results). On the MOD Q1 and our observed shortcuts, all known issues (taking advantage of metadata to enhance performance) were flagged by an LLM, even in runs where the score is below human performance.

Of course, we do not know how many false negatives may be hiding there. To attempt to address this, we will perform a screen of runs flagged as negatives (results were cheating was not identified) to understand the actual sensitivity of this method. We were focused mostly on the superhuman performing runs as we see those as an important step in establishing agents' performance in biomedical data science tasks.

Reviewer_FQGBOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a new agentic benchmark, HeadlessAgents, which evaluates agents on computational biology tasks drawn from DREAM Challenges. It includes 23 tasks from 12 challenge families. Each task is an ML-style challenge, although not all of them are supervised learning problems. The authors compare agents’ solutions with those of the human teams that originally competed in each challenge. The paper includes an extensive evaluation:

  • The authors test many agents, both general-purpose and biology-specific.
  • They find that no agent’s mean score across its multiple runs exceeds the best human team’s score on any task with a matched human leaderboard.
  • They find that the variance in scores across runs of the same agent is high.
  • They evaluate several test-time optimization methods, such as GEPA, on one task.
  • They examine how performance depends on prompt context and find no significant improvement from adding descriptions of the winning solutions to the prompts.
  • The authors audit runs that score above the best human for cheating and leakage. On MOD Q1 they found agents cheated and removed these runs from analysis.
  • They use the DeLeAn rubrics to annotate the task prompts, showing the types of abilities each task demands.
  • They judge the “biological appropriateness” of solutions using both human raters and LLM judges. The authors design the benchmark to be extensible to new tasks or agents.
Critical strengths

The paper introduces a large benchmark of genuine scientific value. The tasks are real computational biology tasks, not toy tasks. Matching or beating the human teams on these challenges would mean doing as well as skilled researchers who worked on the same open problem. Placing an agent’s score within the full population of competing teams is especially informative: it shows where agents land relative to the range of human performance.

The evaluation is broad and informative, going beyond a single leaderboard number. In particular, the evaluation of the following was unique and informative/surprising:

  • Prompt context: varying the amount of context, from a bare challenge link to the winning teams’ write-ups, tests whether agents can benefit from the domain knowledge and insights that helped human teams succeed. It is surprising that they do not. Further investigation as to why agents ignore this would be helpful.
  • Test-time optimization: for GEPA, AutoResearch, and RSA, this is the first honest evaluation I have seen of these techniques of whether spending more compute on iterative refinement beyond what agents do on their own helps on scientific tasks.
  • Biological appropriateness: rating solutions with both human experts and LLM judges examines whether agents use sound domain reasoning, which is one thing that separates a biology task from a standard ML task.
Critical weaknesses
  • The exclusion of 8/22 tasks from the human comparison needs justification
  • Web search confounds the context manipulation, and this should be audited. Agents had web search enabled in every prompt mode, so even low-context runs could retrieve the challenge papers and winning teams’ write-ups.
  • The test-time optimization results are not convincing and may amount to a negative result:
    • A compute-matched baseline is missing. The paper does not compare the TTO methods with simply taking the best of N independent runs at matched compute.
    • A validation-based baseline is missing. Nor does it compare against selecting among runs using a validation split of the training data.
    • The best single run beats all three methods. That run is itself picked using test scores, so it relies on roughly the same information as the TTO results, yet it outperforms all methods. As presented, the experiment does not show that TTO is useful here.
    • The TTO methods use the test set, which seems unfair. AutoResearch received feedback from subsets of the test holdout across many rounds of iteration, and the extensive testing this involves makes overfitting to the test set a real concern. It would be more convincing to run the whole procedure with a holdout split carved from the training data and score the final result once on the test set.
  • The human judging analysis does not account for rater differences. One human rater (Rater 3) disagreed with nearly everyone, which pulls down the reported human–human agreement. Normalizing for each rater’s tendencies, for example with a model that includes rater effects, would give a fairer estimate of inter-human agreement. Because the scale is ordinal, a weighted or chance-corrected measure would also be more appropriate than exact agreement.
Questions for authors

Minor weaknesses:

  • The figures are hard to interpret, and more informative legends would help. In several cases it was difficult to work out exactly what was plotted. In Fig 1b, for example, “mean agent rank” could mean the average of the agents’ individual ranks or the rank of the agents’ average score. The composition of the ranking pool is also unclear: which human teams are included, which score represents each team, and how many teams each task has.
  • The hard-prompt results that beat humans are surprising, since hard prompts provide the least context, and these results are not shown in the main human comparison, which is restricted to guided prompts. The authors should explain why the least-informed setting produces the strongest results on these tasks.
  • The claim of “hundreds of expert teams” is overstated. Many of these challenges had only a a few tens of teams. Challenges are also open to anyone, so participants are self-selected rather than vetted domain experts.
  • The DeLeAn analysis is not convincing. I would expect added useful information to make a task easier, but the annotated demand does not decrease with more context: the paper and write-up prompts are rated as more demanding than the guided prompt. This likely happens because the annotator saw only the prompt text and not the attached materials. Only averages over tasks are reported, so the spread across tasks is not visible. The demand scores are also never related to agent performance, so it is unclear which attributes, if any, predict success. From the limited data available, I do not see DeLeAn as obviously useful here.
  • Fig 4b would be clearer as a trajectory. I would prefer to see RMSE over rounds, with each attempt colored by whether it was a leap or a tweak and linked to the attempt it built on. This would show whether leaps contributed to later improvements. Suggestion:
  • Fig 3d may be easier to compare to 3a-c if it was also a plot rather than a table
2 comments

We thank the reviewer for their kind and extremely thoughtful review! We understand their concerns and seek to address them.

Leaderboard for 14/22 tasks - There are specific compatibility reasons that we do not some of the extra 8 tasks. We add explanations for each in Table A.5 - It boils down to we were unable to reproduce the exact scoring mechanism for the historical human runs and used a surrogate metric to evaluate agent performance instead. As such, the agent performance metric and the metric that the human runs were evaluated for were different and these 8 are excluded. We will be expanding our metrics and renewing runs with newer models too for the final version and it will be posted on our living benchmark page

Manuscript locations: Section 4, “Ground-truth metrics,” p. 6; Appendix A, Table A.5 and compatibility discussion, p. 20; NeTParInf sensitivities, Figures A.4–A.5, pp. 21–22.

Web access - The reviewer is entirely correct. We allowed the agents to access to internet in order to make sure they do not get stuck in quagmires for documentation and troubleshooting code and to mirror the everyday usage that practitioners employ. We clarify this limitation in the manuscript and introduce an analysis that will evaluate the traces for unintended web searches that can influence results. This analysis is currently underway though, and not present in the rebuttal version currently.

Optimization interpretation - We added some extra comparisons to try and identify the gains owed to TTO. We isolated the training and testing cohort completely to ensure no adaptive overfitting occurred. We also launch 1 agent AutoResearch copies (4 copies of the same and a mixed roster of Grok 4.7, GPT-6.1 Sol, Gemini-3.8, Claude Sonnet 5.5) at 50 steps each, alongside new GEPA and RSA runs at the same number of steps. We also add a best expected out of N independent runs and use bootstrapping to identify what kind of score would be achieved. However, these runs are still not entirely compute matched. We report agent hours, which are broadly similar, but due to the methods working very differently, it is quite hard to truly match the compute across them. We are currently perfoming analyses for token and money spend.

Appendix A, Figure A.2 and caption, p. 16, for the new outcomes, independent control and oracle curves; Section 5, p. 7, Figure 4, p. 9, and Appendix D, pp. 26–27, for the qualified historical result.

Raters - We now also report quadratic weighted Cohen's kappa for all pairs with 10k bootstrap resamples in Table E.1 and Table E.2. We are scaling up the analyses to be able to fit something like a rated fixed effect models, but we are hesitant to do so in such a low sample regime. We also tone down some of the agreement claims in response to reviewers' comments and are actively working on judging for the final camera ready version and clarify some of this in our future work paragraph.

Appendix E, “Inter-rater analysis,” “Ordinal agreement and rater sensitivity,” and Tables E.1–E.2, p. 29; Figure 5, p. 10; calibration interpretation, pp. 30–31.

Figure interpretation - The author is correct and the figure is trying to cram too much information. Mean agent rank is the rank of the agents' average score. We hope that these will become much clearer in the web version of the benchmark and will amend the figure with the number of human teams.

Hard Context - The reviewer is correct in this. We are currently exploring the runs. Our first step was to vet that was no leakage, and our second is understanding whether newer methods were used

Expert teams - We will tone down this claim, the teams are not all experts, but due to the wisdom of crowd adage, we believe that the best team would effectively come up with an expert human solution. This is corroborated by the fact that, at least for now, the agents rarely outperform said solution.

Figures 4b and 3d will be amended

After letting the reviewer's comments stew for a few days, we were inspired to design new experiments that will hopefully be ready for the camera ready version, or if not, definitely our online benchmark release. We once again thank the reviewer for their extreme thoroughness and thoughtfulness in their review and are collectively very thankful for the thought provoking discussion.

Leaderboard - We have contacted the creators of most tasks to try and get any of the 8 tasks we could not correctly calculate the metric for.

Web access - We will design URL blacklists for some of these tasks, implemented within Docker, in our future evaluations to make sure the models do not access URLs they should not. While this is hardly exhaustive, it will at least make sure we do not waste tokens on obvious cheating issues like just fetching the results and posting them as a "run"

TTO - For this and another reviewer's comments, we will also probably be running further replications of each setup, to at least attempt to make a statistical claim for the performance of TTO versus just completing N independent runs. Right now, as we only have an N=1 for each TTO, it is quite hard to say whether one is statistically better than the other, though it does seem that running independently is currently favoured.

We will pass over the figures to try and clear up the legends

For the hard prompt results, our hypothesis, consistent with our initial judging results that the agents do not really take biological context into account, is that these models simply treat everything as an ML engineering task, whereas historically, teams taking part in DREAM challenges were quite inventive in their feature engineering.

Re DeLeAn, we apologize for not previously addressing this point. The annotator read the prompt text, without having the attached data, as the reviewer inferred. Our current data averages all tasks in the each of the prompt modes and weighs them equally. We will refine this analysis so that we can see if any tasks are unusually easy/hard and try to relate this to performance. We will also probably have to evaluate how exactly we annotate the results with DeLeAn to ensure we are being fair in this. We once again thank the reviewer for their point

Reviewer_SCLxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduce is a reusable, extensible framework that runs agents on DREAM computational-biology challenges and compares their scores with historical expert teams. The paper studies repeated runs, prompt context, test-time optimization, and leakage, and compares human and LLM ratings of biological appropriateness.

Critical strengths
  1. Real biological tasks and historical leaderboards provide useful reference scores. Repeated runs show substantial variation, so one successful run does not represent an agent's typical performance.

  2. The audits distinguish test-label retrieval, model selection using test labels, and shortcuts from supplied metadata. These examples show how high scores can reflect leakage or mistakes in task design rather than prediction skill.

Critical weaknesses
  1. AutoResearch repeatedly uses the final test set for feedback. It receives scores from random 80% subsets of the same 368-sample holdout used for final scoring (Appendix D). The gains may reflect overfitting to this set, rather than better performance on unseen data. Changing the subset does not fix this; an untouched test set is needed. The exploratory Figure 4 also changes models and budgets, so it cannot establish which optimization method is better.

  2. The MOD Q1 setup allows a shortcut through visit-timing metadata. Agents received these fields without being told not to use them (Appendix F). Excluding runs that use them is justified by the original expression-only challenge, but leaves performance under the intended rules unknown. Correct the inputs and instructions, then rerun the affected comparisons; deleting runs alone does not fix the task.

Questions for authors

Questions

  1. How many runs were attempted, failed, excluded and retained for each task, agent and prompt? can you state the exclusion reasons and which runs enter each figure's means and failure rates.

  2. How were retained runs below the human best checked for leakage? Figure 1 says all shown runs passed an audit, but Appendix F specifies a secondary audit of above-human runs. MOD test labels were accessible online, so low scores alone do not rule out leakage.

Minor comments

  1. The pooled mean moves closer to the human mean, but the category distribution moves farther away: total variation increases from approximately 0.12 to 0.28 (Tables E.2–E.3). Seven evaluation solutions with broad uncertainty do not show better agreement across rating categories. Limit p.6's positive claim to the mean agreement observed here.

  2. The descriptions on pp.36 and 56–59 treat Pearson correlation as ranking, although identical rankings can have different Pearson scores. Correct these passages, identify which were used in the evaluated judging prompts, and check whether the affected ratings need updating.

  3. The independent-reproduction claim is too broad. The eleven Olfaction wins were rescored; signaling predictions were unavailable for independent rescoring (Table F.1). State this limit in p.24's reproduction claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.