- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents OmicsBench, a benchmark for testing whether AI agents can carry out bioinformatics analyses. It contains 784 tasks across six life-science domains, including 741 short-horizon tasks and 43 long-horizon tasks. Each task uses real data and includes an execution environment, a task prompt, and a deterministic grader. The benchmark builds on the authors’ earlier benchmarks, with additional quality checks and improvements to the execution environment. The authors test 31 combinations of models and agent frameworks. The best average pass rates are 49.4% on short-horizon tasks and 32.8% on long-horizon tasks. The paper also studies why agents fail. Its main contribution is the benchmark and the experimental analysis.
Critical strengths
- Clear and easy to follow. The paper is well organized. It explains the gap in existing benchmarks, introduces the task construction process, and then presents the results.
- The paper offers a useful view of how agents perform on real bioinformatics tasks. The failure analysis also raises interesting questions.
Critical weaknesses
- Section 4.2 reports common problems in failed runs, but it does not show how often the same analysis choices appear in successful runs. The results therefore describe what happens in failed runs more clearly than they show which choices separate success from failure. Comparing successful and failed attempts by the same model on the same task would provide stronger support for the explanations in Section 4.2.
- The two task sets cover different domains, and the paper does not report a matched comparison between separate steps and an end-to-end version of the same workflow. The appendix provides results by domain, but these still do not separate harder individual steps from additional errors that arise when the agent connects those steps. Since the benchmark already includes data snapshots and intermediate checkpoints, a matched analysis would help show more clearly what the long-horizon tasks measure.
Questions for authors
- Section 4.2 reports 4,556 failed runs, while the caption of Figure 5 reports 6,347. Please explain what each number refers to and which set of runs was used to calculate the results in the figure.
- Why is the Table A5 example labeled as unnecessary task abandonment? The quoted agent states that there is only one sample per group and that the response is completely confounded with batch. It therefore declines to make a claim based on cohort-level significance testing. Could you provide the full task requirements and explain why this is unnecessary abandonment?
- Do the reported analysis choices appear more often in failed runs? For tasks where the same model has both successful and failed attempts, could you use the same criteria to compare choices, such as selecting thresholds without checking them? Are these choices clearly more common in failed attempts?
- How much of the low pass rate comes from inconsistent performance? Using the existing repeated trials, could you report the fraction of tasks solved at least once under the same number of attempts, such as pass@3 for both horizons? You can show some examples.
[Critical weakness 1] Comparing successful and failed attempts by the same model on the same task would provide stronger evidence in the failure mode analysis rather than just analyzing failed attempts
We agree with the reviewer that this would make a stronger analysis, and have updated the manuscript accordingly. We conducted an analysis on the tasks where a model-harness configuration both passed and failed. Every failure mode identified in a failing run was checked by an LLM judge against every other run of that configuration on the task. As a sanity check, we found that the judge could correctly identify a failure mode in its failing run of origin in 99.5% of trials. Figure A1 shows the results of this analysis stratified across the benchmarks. In general, we find that 56.8% of failing runs make mistakes shared with other failing runs, while only 6.8% of passing runs exhibit this behavior. We believe this strongly validates that the failure modes we identify are linked to failing trajectories.
[Critical weakness 2] The paper does not report a matched comparison between separate steps and an end-to-end version of the same workflow.
We agree this would be the cleanest comparison, but it cannot be built for these tasks. Long-horizon tasks do not prescribe an analysis path, so testing one step alone would mean handing the agent partly analyzed data in which the earlier decisions are already made. Those decisions are a large part of what the task tests.
Our wording in Sec 3.1 invited this reading, and we have corrected it. Anchors are not intermediate checkpoints. They are the steps an agent must work through to reach the correct final conclusion, and they are graded in the final answer.
Instead, we have added a comparison of failure patterns across horizons for spatial-omics and single-cell transcriptomics (Sec 4.3). Short-horizon tasks test one analytical choice. Long-horizon tasks require all or nearly all anchors to be correct, so an agent can get most of the answer right and still fail.
[Q1] Mismatch in number of failed runs between Section 4.2 and Figure 5
We thank the reviewer for catching this mistake. In an early version of the manuscript, we ran fewer trials, then increased the number of trials to improve our statistical power. The lower number represents our initial results, whereas the higher number represents the actual number of trials.
[Q2] Mismatch between failure mode and the given example in Table A5
We thank the reviewer for their comment. We have extensively redesigned the failure mode analysis and addressed several points that we believe strengthen its validation. First, we recomputed the analysis without truncation of outputs or tool calls, and provided the LLM judge with the reference answers and evaluation notes. Second, we more carefully implemented a Docent-based system that is well-known for exposing failure modes in complex traces. Third, we implemented a check on tasks that had both passing and failing trajectories (Figure A1), validating that we often observed failure modes drawn from failing trajectories in other failing trajectories from that same task, but not in other passing trajectories. Finally, we conducted manual review of the failure modes, eliminating those where we could not find strong evidence of correct judgement
In doing so we have unified the analysis to address a consistent pool of 6,090 trajectory failure mode reviews and provided much stronger and more consistent examples from trajectories that expose these failure modes in Table A6.
[Q3] How represented are the selected failure modes in successful runs
We thank the reviewer for this, and agree that this is an important plot to report! We’ve included discussion of this in lines 378-383 and figure A1.
[Q4] How much of low pass rate comes from inconsistent performance
We agree that reporting pass@k and pass^k are important to measure performance across trials, and now include these metrics and related analysis in Appendix F (Inconsistency Performance Analysis, Figures A13 and A14).