acceptodds
Under review as a conference paper at ICLR 2027

OmicsBench: A Large-Scale Verifiable Benchmark for Agentic Life Sciences Capabilities

Abstract

Progress in the life sciences relies on the collection and analysis of complex data modalities. Increasingly capable AI agents have shown promise in revolutionizing the latter by automating bioinformatics. To realize this progress, the field has devised a number of agentic life sciences benchmarks. These benchmarks often test agent capabilities broadly, with a few tasks per field across many life sciences domains. Consequently, they do not exhaustively explore capabilities in a particular bioinformatics domain. We present OmicsBench, a collection of 784 expert-constructed, short- and long-horizon tasks that comprehensively explore six data-intensive life sciences domains (spatial-omics, single-cell transcriptomics, epigenomics, variant interpretation and discovery, microbial surveillance, and protein function). Each task is composed of a real-world dataset, a task prompt, and a deterministic grader. We show that across 31 model-harness pairs, no agent scores higher than 50% on the benchmark and conduct detailed analysis on failure modes and model behavior. It is our hope that our work will inform model capability development and future construction of benchmark and training tasks.

Primary area:
datasets and benchmarks
Keywords:
computational biology, benchmarks, bioinformatics, agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rh88OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OmicsBench, a benchmark for testing whether AI agents can carry out bioinformatics analyses. It contains 784 tasks across six life-science domains, including 741 short-horizon tasks and 43 long-horizon tasks. Each task uses real data and includes an execution environment, a task prompt, and a deterministic grader. The benchmark builds on the authors’ earlier benchmarks, with additional quality checks and improvements to the execution environment. The authors test 31 combinations of models and agent frameworks. The best average pass rates are 49.4% on short-horizon tasks and 32.8% on long-horizon tasks. The paper also studies why agents fail. Its main contribution is the benchmark and the experimental analysis.

Critical strengths
  1. Clear and easy to follow. The paper is well organized. It explains the gap in existing benchmarks, introduces the task construction process, and then presents the results.
  2. The paper offers a useful view of how agents perform on real bioinformatics tasks. The failure analysis also raises interesting questions.
Critical weaknesses
  1. Section 4.2 reports common problems in failed runs, but it does not show how often the same analysis choices appear in successful runs. The results therefore describe what happens in failed runs more clearly than they show which choices separate success from failure. Comparing successful and failed attempts by the same model on the same task would provide stronger support for the explanations in Section 4.2.
  2. The two task sets cover different domains, and the paper does not report a matched comparison between separate steps and an end-to-end version of the same workflow. The appendix provides results by domain, but these still do not separate harder individual steps from additional errors that arise when the agent connects those steps. Since the benchmark already includes data snapshots and intermediate checkpoints, a matched analysis would help show more clearly what the long-horizon tasks measure.
Questions for authors
  1. Section 4.2 reports 4,556 failed runs, while the caption of Figure 5 reports 6,347. Please explain what each number refers to and which set of runs was used to calculate the results in the figure.
  2. Why is the Table A5 example labeled as unnecessary task abandonment? The quoted agent states that there is only one sample per group and that the response is completely confounded with batch. It therefore declines to make a claim based on cohort-level significance testing. Could you provide the full task requirements and explain why this is unnecessary abandonment?
  3. Do the reported analysis choices appear more often in failed runs? For tasks where the same model has both successful and failed attempts, could you use the same criteria to compare choices, such as selecting thresholds without checking them? Are these choices clearly more common in failed attempts?
  4. How much of the low pass rate comes from inconsistent performance? Using the existing repeated trials, could you report the fraction of tasks solved at least once under the same number of attempts, such as pass@3 for both horizons? You can show some examples.
1 comment

[Critical weakness 1] Comparing successful and failed attempts by the same model on the same task would provide stronger evidence in the failure mode analysis rather than just analyzing failed attempts

We agree with the reviewer that this would make a stronger analysis, and have updated the manuscript accordingly. We conducted an analysis on the tasks where a model-harness configuration both passed and failed. Every failure mode identified in a failing run was checked by an LLM judge against every other run of that configuration on the task. As a sanity check, we found that the judge could correctly identify a failure mode in its failing run of origin in 99.5% of trials. Figure A1 shows the results of this analysis stratified across the benchmarks. In general, we find that 56.8% of failing runs make mistakes shared with other failing runs, while only 6.8% of passing runs exhibit this behavior. We believe this strongly validates that the failure modes we identify are linked to failing trajectories.

[Critical weakness 2] The paper does not report a matched comparison between separate steps and an end-to-end version of the same workflow.

We agree this would be the cleanest comparison, but it cannot be built for these tasks. Long-horizon tasks do not prescribe an analysis path, so testing one step alone would mean handing the agent partly analyzed data in which the earlier decisions are already made. Those decisions are a large part of what the task tests.

Our wording in Sec 3.1 invited this reading, and we have corrected it. Anchors are not intermediate checkpoints. They are the steps an agent must work through to reach the correct final conclusion, and they are graded in the final answer.

Instead, we have added a comparison of failure patterns across horizons for spatial-omics and single-cell transcriptomics (Sec 4.3). Short-horizon tasks test one analytical choice. Long-horizon tasks require all or nearly all anchors to be correct, so an agent can get most of the answer right and still fail.

[Q1] Mismatch in number of failed runs between Section 4.2 and Figure 5

We thank the reviewer for catching this mistake. In an early version of the manuscript, we ran fewer trials, then increased the number of trials to improve our statistical power. The lower number represents our initial results, whereas the higher number represents the actual number of trials.

[Q2] Mismatch between failure mode and the given example in Table A5

We thank the reviewer for their comment. We have extensively redesigned the failure mode analysis and addressed several points that we believe strengthen its validation. First, we recomputed the analysis without truncation of outputs or tool calls, and provided the LLM judge with the reference answers and evaluation notes. Second, we more carefully implemented a Docent-based system that is well-known for exposing failure modes in complex traces. Third, we implemented a check on tasks that had both passing and failing trajectories (Figure A1), validating that we often observed failure modes drawn from failing trajectories in other failing trajectories from that same task, but not in other passing trajectories. Finally, we conducted manual review of the failure modes, eliminating those where we could not find strong evidence of correct judgement

In doing so we have unified the analysis to address a consistent pool of 6,090 trajectory failure mode reviews and provided much stronger and more consistent examples from trajectories that expose these failure modes in Table A6.

[Q3] How represented are the selected failure modes in successful runs

We thank the reviewer for this, and agree that this is an important plot to report! We’ve included discussion of this in lines 378-383 and figure A1.

[Q4] How much of low pass rate comes from inconsistent performance

We agree that reporting pass@k and pass^k are important to measure performance across trials, and now include these metrics and related analysis in Appendix F (Inconsistency Performance Analysis, Figures A13 and A14).

Reviewer_xD7TOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present OmicsBench, a benchmark containing 741 short tasks and 43 longer workflows across six life sciences domains. Each task provides a real dataset, a software environment, a question, and a deterministic grader. The authors describe expert task review and safeguards against shortcuts, then evaluate 31 combinations of models and agent harnesses over repeated trials. The best reported pass rates are 49.4% on short tasks and 32.8% on long tasks. They also compare costs and refusals, and use an LLM judge to analyze scientific errors and undesirable behavior.

Critical strengths
  1. The problem is well motivated, and the benchmark covers a useful range of real bioinformatics / comp bio analyses, from individual steps to complete workflows.

  2. The evaluation across 31 model and harness combinations is substantial. Repeated trials, costs, and refusal breakdowns make the comparisons informative.

Critical weaknesses
  1. It is unclear whether the graders consistently accept different scientifically valid analyses. The authors describe careful task review, but the manuscript does not show complete prompts and graders, acceptance tolerances, or quantitative validation against independent expert judgments. This matters for tasks whose answers depend on analysis choices. For example, Table A1 treats differential expression based on FDR without a fold-change cutoff as an error, although the appropriate criterion depends on the question. The current evidence does not establish whether failures consistently reflect incorrect science or disagreement with a particular reference analysis.

  2. The failure analysis needs stronger validation. The LLM judge sees shortened tool outputs without the reference answer, and no quantitative agreement with independent expert labels is reported. A second LLM review is helpful but does not establish accuracy. In Table A5, an agent’s concern about one sample per group and complete batch confounding is labelled “unnecessary task abandonment.” That may be appropriate scientific caution; the excerpt does not explain why it is wrong. The analysis populations also differ between Figure 5 and Table A3. These issues limit confidence in the claimed causes of failure and comparisons of undesirable behavior.

Questions for authors
  1. Grading: Could the authors provide complete prompts, reference analyses, and graders for representative tasks, including the epigenomics example in Table A1? Please specify accepted alternatives, numerical tolerances, and how intermediate anchors determine whether a long task passes.

  2. Failure labels: Why is the one-sample-per-group, batch-confounded example in Table A5 considered unnecessary abandonment? Does an independent expert audit of full trajectories support the reported scientific-error and undesirable-behavior labels?

  3. Analysis counts: Which runs underpin the final analyses? Please reconcile the 6,347 failures in Figure 5 with the 4,556 in Table A3, and the different behavior-analysis populations in Figure A1 and Table A2, including judge refusals and other exclusions.

2 comments

Comment [1/2]

We thank the reviewer for their careful and constructive review!

[Critical weakness 1] It is unclear whether the graders consistently accept different scientifically valid analyses.

We agree that we should more clearly explain how we design our graders to accept different valid analyses, so we have added to our appendix. Most of our tasks grade a biological result (for example, a percentage or a gene count) within a stated tolerance, rather than checking implementation details. For analysis-dependent tasks, the task author and other human reviewers recomputed the answer with several defensible approaches during construction and set the tolerance to accept them. We now include an Appendix B, which includes details on graders and several examples of task tolerance, including the epigenomics example shown in Table A1. We also enclose four additional task examples and tolerance here:

  1. Epigenomics, non-canonical methylation–expression fraction (ESCC). A per-gene t-test gives 37.5%, the source paper's DMR + edgeR pipeline gives 37.0%, and the paper's published cluster file gives 28.3%. The grader accepts 37 ± 10 percentage points, so all three pass. A looser methylation FDR (30–35%) also passes, as does a ±2 kb promoter window when combined with the expression fold-change filter (33–38%).

  2. Single-cell, pseudobulk DE (IFN-β vs. PBS, CD14 monocytes). The grader accepts 1,150 ± 350 DEGs. With DESeq2 on donor pseudobulks, donor cell-count filters from 3 to 10 cells give 911–1,468 DEGs, all inside the range; we will confirm edgeR and limma-voom in the revision. The grader rejects only cell-level tests that treat cells as independent replicates (t-test 3,194; Wilcoxon 625).

  3. Function, polymerase mutational spectrum from raw reads. The author swept the conventions a solver must choose: quality threshold, denominator, pooled vs. per-position averaging, and flooring corrected values at zero. Every combination that filters low-quality bases lands within ±1.5 percentage points on each of the four graded substitution types.

  4. Microbial surveillance, avian shotgun taxonomy. The ground truth keeps only three abundant families that should not depend on the classifier, and the task passes at family-set Jaccard ≥ 0.5. The tolerance is designed to accept standard classifiers (Kraken2, Centrifuge, Kaiju, alignment-based profiling).

[Critical weakness 2] Failure analysis needs stronger validation.

We thank the reviewer for their comment. We have extensively redesigned the failure mode analysis and addressed several points that we believe strengthen its validation. First, we recomputed the analysis without truncation of outputs or tool calls, and provided the LLM judge with the reference answers and evaluation notes. Second, we more carefully implemented a Docent-based system that is well-known for exposing failure modes in complex traces. Third, we implemented a check on tasks that had both passing and failing trajectories (Figure A1), validating that we often observed failure modes drawn from failing trajectories in other failing trajectories from that same task, but not in other passing trajectories. Finally, we conducted manual review of the failure modes, eliminating those where we could not find strong evidence of correct judgement.

In doing so we have unified the analysis to address a consistent pool of 6,090 trajectory failure mode reviews and provided much stronger and more consistent examples from trajectories that expose these failure modes in Table A6.

[Question: Grading] Could the authors provide complete prompts, reference analyses, and graders for representative tasks, including the Table A1 example? Please specify accepted alternatives, tolerances, and how intermediate anchors determine whether a long task passes.

We added an appendix section, "Grading and Alternative Analyses", that describes the grader types, how tolerances were set, and worked examples with the analyses that pass and fail, including the Table A1 task.

As for Anchors, they are the steps an agent must work through in a long-horizon task to reach the correct conclusion. They are not graded as checkpoints during a run; they are checked in the final structured answer. Each check is a gate (must be correct), an additive check (contributes points), or a hard-fail check (reporting a specific incorrect interpretation fails the run).

Comment [2/2]

Question: Failure labels] Why is the one-sample-per-group, batch-confounded example in Table A5 considered unnecessary abandonment? Does an independent expert audit of full trajectories support the reported scientific-error and undesirable-behavior labels?

We thank the reviewer for this comment. We have redesigned the failure mode analysis as specified above, with several steps included to ensure robustness of the scientific-error and undesirable-behavior labels, including manual examination of dozens of full trajectories. Updated examples of how the trajectories map onto failure modes are provided in Tables A5 and A6.

[Question: Analysis counts] Which runs underpin the final analyses? Please reconcile 6,347 vs 4,556, and the different behavior-analysis populations in Figure A1 and Table A2, including judge refusals and exclusions.

We thank the reviewer for catching this inconsistency. We have double-checked our numbers throughout the paper and report the following:

Our analysis focused on seven model-harness pairs, as specified in Figure 5. Across these model-harness pairs, we had 16,404 runs

Out of these, Figure 5, Table A4, and Table A6 are based upon 6,738 non-refusal failures in the 2,105 all-fail cells, covering 496 tasks. 378 refusals were removed from analysis. Out of the 6,738 non-refusal failures, 6,090 had failure mode labels that were successfully extracted by the two judge LLM system. Subsequent failure mode analysis was based on these 6,090 failures and the 43 failure modes derived from clustering them.

Separately, Figure A1 is based on 1,480 non-refusal failing runs and 1,359 passing runs derived from tasks with both passing and failing trajectories.

Finally, the behavioral analysis showcased in Figure A2, Table A3, and Table A5 was based on 2,468 runs from 734 tasks. These tasks and trajectories were sampled from 15,965 non-refused runs across the 7 model-harness configurations such that all model harness pairs would be equally well represented.

We have detailed all of these numbers with consistency in the new version of the paper that we have uploaded.

Reviewer_rtaPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present omicsbench: a large scale verifiable benchmarks that cover short and long horizon agentic tasks for life science tasks. It includes 784 tasks from 6 (short horizon) + 2 (long horizon) domains, and has been used for evaluate a large amount of frontier models from the closed and open-source communities. The authors conclude the competitive performances of closed models and also did an analysis on failure modes and agents' misbehaviors.

Critical strengths

The paper is well written, clearly scoped and motivated, especially for readers who don't have domain expertise like me. the wide coverage of the tested models and harnesses also help consolidate the conclusions and findings.

Critical weaknesses

My major concern is on the contributions and originality of this work. considering

OmicsBench is a composite benchmark featuring tasks from six short-horizon and two long-horizon
benchmarks that we previously published in preprint

if the benchmarks are composed of tasks that have been curated before and published in separate preprints, this work is more of an evaluations of existing models, which may have limited contributions to the community. if this got addressed, i am happy to reconsider my score.

Besides, if the authors can share more details / statistics on the human reviews on the benchmarks (e.g. the consistency of llm and human judgement), that will help prove the quality and soundness of the benchmarks.

Questions for authors
  1. as a non-expert in the domain, i wonder if the coverage of the benchmarks is large enough to claim that it's testing AGENTIC LIFE SCIENCES CAPABILITIES. is it too general? also most of the tasks are on the data-driven discovery side, right? (i expect none of them involve wet labs, like an agent operating some lab equipments. correct me if i am wrong please)
  2. line 374 is an interesting discussion on model variance, how does harness affect the agent behaviors?
  3. in line 203, the experiments set all models to their highest reasoning efforts, how does that compare with the default settings for each api? this could be helpful for practitioners.
  4. what are the time horizon of these different tasks? also does the time taken from each models vary much?
1 comment

We thank the reviewer for their time and helpful comments!

[Critical Weakness 1] If the benchmark is composed of tasks that have been curated before and published in separate preprints, the work is more of an evaluation of existing models

We thank the reviewer for raising this point. We agree that we should have been clearer about what this paper adds to our earlier preprints. OmicsBench builds on eight benchmarks that we developed and shared on arXiv over the course of this year. For this paper, we revised the tasks and added human verification for all long-horizon tasks and 158 short-horizon tasks. We also updated the prompts and sandboxes, restricted internet access, and removed dataset identifiers that agents could use to look up published answers. We describe these changes in the OmicsBench Updates appendix. We also spent a substantial part of this work comparing model performance and failure modes across domains. Each preprint reported results for one benchmark whereas here we evaluated the same 31 model–harness pairs across all eight. This cross-domain analysis is one of the main contributions of the paper. The compilation of cross-domain failure modes and undesirable behaviors was only possible after combining tasks from the various sub-benchmarks. Finally, we respectfully ask the reviewer to consider that just because a work has already been published as a preprint does not necessarily diminish its impact in peer-reviewed settings: to our knowledge, it is common for work to be published in both preprint and later in journal or conference proceedings.

[Critical Weakness 2] Sharing more details/statistics on the human review will help prove the quality and soundness of the benchmark

We appreciate the reviewers' questions on human review. We did not use LLMs to review benchmark tasks. Every task was reviewed by an expert holding an advanced degree in the life sciences with relevant domain experience for verifiability, durability, and anti-shortcut properties. In addition to this review process, 158 of the tasks were human verified by an additional expert uninvolved in task construction, adding an additional layer of guarantee on the quality and soundness of the tasks.

[Q1] Is the coverage large enough to call this a life sciences capability benchmark?

We appreciate the reviewers' concern with the scope, and agree that a more appropriate title for the benchmark would be “Agentic Bioinformatics Capabilities.” We have updated the text accordingly.

[Q2] How does the harness effect model variance?

We believe the reviewer brought up an interesting point, and we have included more details on how harnesses affect agent behavior in the main text. Briefly, the harness has a much smaller effect than the model. Failure-mode shares shifted by 2.3 points on average between harnesses of the same model, compared with 4.5 points between models, and choice of harness explained under 2% of the variance in pass rate.

[Q3] How does reasoning effort change the results?

We agree with the reviewer that having the default reasoning efforts would be helpful for practitioners, especially given that scientists often do not adjust the effort levels of the models. We have added this analysis in Appendix H (Effort Sweeps, Figure A17). We find that decreasing effort generally decreased cost, tokens, turns, and run time for nearly every configuration but it did not always decrease pass rate. One reason for this was a difference between refusal behavior, as observed in Claude Opus 5.5.

[Q4] What are the time horizons of the different tasks and does this differ between models?

We agree with the reviewers that adding this would be useful. We have added run times and timeouts in Appendix G (Timeout Analysis, Figures A15 and A16). Short-horizon tasks have a 6-hour limit and long-horizon tasks a 10-hour limit. For a broad overview, run time varies between models; the median short horizon task, for instance, ranged from 3.1 minutes (GPT-6 Astra / Pi) to 87.5 minutes (Gemini 3.8 Flash / Pi). Timeout behavior, median runtimes, and Pareto frontier are now present in Appendix G.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.