- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes OmicsBench, a benchmark for evaluating the ability of large language model (LLM) agents to complete long-horizon, multi-turn bioinformatics research tasks. The authors point out that existing bioinformatics benchmarks mainly focus on single-step analyses or relatively independent tasks, making it difficult to evaluate multiple interdependent analysis steps in real research workflows and the impact of preceding results on subsequent tasks. To address this, OmicsBench models a complete research task as a directed acyclic graph (DAG), in which nodes represent specific bioinformatics subtasks and edges represent data or result dependencies between tasks. The benchmark contains 97 research-study-scale tasks and 662 subtasks, covering 11 biological domains, 18 analysis types, and 10 organisms, and it includes 58 classes of deterministic graders to automatically evaluate outputs such as gene lists, ranked results, pathways, genomic regions, and numerical values.
The authors further test 12 models and 3 types of agent harnesses, and compare two execution modes: in single-pass mode, the agent completes tasks sequentially according to a predefined workflow; in gated mode, if a subtask scores below 0.7, the agent is allowed up to three retries. Experimental results show that different agents achieve average scores of approximately 0.43–0.57 in single-pass mode, while gated mode can raise scores to approximately 0.54–0.67, indicating that current models still have clear deficiencies in reliability and task execution on complex, long-horizon bioinformatics workflows, while failure feedback and retries can bring some performance improvement. The authors also further analyze agent execution trajectories, summarize and identify multiple failure modes, including incorrect method selection or parameter configuration, result sets that are too broad or too narrow, insufficient use of input data, premature termination, replacing formal statistical analysis with keyword matching, and failure to perform multiple testing correction. Overall, the main contribution of this paper is to construct a benchmark that is closer to actual computational biology research workflows and to use it to systematically evaluate the capabilities and main failure modes of current AI agents in long-horizon, multi-turn, task-dependent bioinformatics research tasks.
Critical strengths
The benchmark design has strong novelty and practical significance. OmicsBench does not merely evaluate the ability to complete a single bioinformatics task; instead, it models a real research workflow as a DAG composed of multiple interdependent subtasks, thereby enabling evaluation of an agent's context retention, result passing, and sequential task execution over long-horizon tasks. Compared with traditional single-step benchmarks, this design is closer to actual computational biology research workflows and is one of the paper's most important contributions.
The evaluation scale is relatively large and combines quantitative performance with failure-mode analysis. The authors construct a benchmark containing 97 research-study-scale tasks and 662 subtasks, and systematically evaluate it across multiple models and agent harnesses. At the same time, the paper not only reports final task scores but also further analyzes agent trajectories and summarizes multiple specific failure modes. This allows the benchmark not only to answer how current models perform, but also to reveal where and in what ways agents fail in bioinformatics analyses, giving it good diagnostic value and potential to drive future research.
Critical weaknesses
Gated mode cannot cleanly demonstrate that the “feedback mechanism” itself brings performance improvement. The paper mainly interprets the performance improvement between single-pass and gated modes as the agent being able to self-correct after receiving failure feedback. However, gated mode also allows the agent to retry up to 3 times, so the improvement may come from additional computational budget and multiple sampling rather than from feedback itself. In particular, if the final score is taken as the best result among multiple attempts, then this setting is effectively close to best-of-3 evaluation. The absence of a control such as “allowing multiple attempts but providing no feedback” makes the paper’s conclusion about the effectiveness of feedback-driven interaction still insufficiently convincing. This directly affects our assessment of the gated evaluation results and their scientific significance.
The correspondence between the benchmark’s scoring method and “true scientific ability” remains highly questionable. OmicsBench’s ground truth and deterministic graders are largely constructed based on the specific analysis pipelines and results in the original published studies, so whether an agent can reproduce the predefined reference analysis will significantly affect the final score. However, in real bioinformatics research, different but reasonable analysis methods, parameter choices, or statistical strategies may produce different results, and this does not mean those results are scientifically wrong. Therefore, the current benchmark score may simultaneously reflect bioinformatics analysis ability, the ability to strictly follow a specified analysis pipeline, and the ability to reproduce a reference answer, rather than being fully equivalent to scientific task-solving ability. Since this concerns the core capability that the benchmark claims to measure, it will substantially affect our evaluation of the paper’s main contribution.
Questions for authors
Questions for Authors
Regarding the role of feedback versus additional retries in Gated mode: The paper interprets the performance improvement of Gated mode over Single-pass mode as the agent being able to use feedback for self-correction. Did the authors conduct a control experiment in which the agent is given the same number of retries and the same computational budget but no specific failure feedback? If so, what is the performance improvement under this control? This result would help determine whether the observed gains come from feedback or mainly from additional attempts.
Regarding whether deterministic graders measure scientific correctness: The paper’s ground truth and scoring criteria are mainly based on the specific analysis pipelines and results in the original studies. Have the authors conducted independent human expert validation of these graders, especially to assess the proportion of answers that are inconsistent with the reference answer but may be equally reasonable biologically or statistically yet are judged incorrect? If such cases exist, how do the authors ensure that the benchmark score represents the agent’s actual scientific reasoning / bioinformatics capability rather than merely its ability to reproduce a specific reference workflow?
3 comments
Weakness: The correspondence between the benchmark’s scoring method and “true scientific ability” remains highly questionable. OmicsBench’s ground truth and deterministic graders are largely constructed based on the specific analysis pipelines and results in the original published studies, so whether an agent can reproduce the predefined reference analysis will significantly affect the final score. However, in real bioinformatics research, different but reasonable analysis methods, parameter choices, or statistical strategies may produce different results, and this does not mean those results are scientifically wrong. Therefore, the current benchmark score may simultaneously reflect bioinformatics analysis ability, the ability to strictly follow a specified analysis pipeline, and the ability to reproduce a reference answer, rather than being fully equivalent to scientific task-solving ability. Since this concerns the core capability that the benchmark claims to measure, it will substantially affect our evaluation of the paper’s main contribution.
We thank the reviewer for raising this concern, and we completely agree that a given dataset and associated scientific question can often be analyzed in more than one reasonable way, which is part of what makes bioinformatics analysis hard to evaluate.
We chose to design OmicsBench tasks as fixed sequences of subtasks that each follow one study's analysis, because this is how scientists tend to work with agentic coding tools in practice. A scientist typically settles on one high-level approach out of several workable ones and then directs the tool through it one step at a time, giving feedback and letting it retry a step until the result is satisfactory, a workflow that OmicsBench mirrors as closely as a deterministic benchmark can.
Even so, OmicsBench tasks do not reduce to following a specified pipeline, since each subtask fixes the result to produce, not how the agent produces it. The agent decides how to read and prepare the data, which tools and methods to use, and how to check its own intermediate results, the same choices that a scientist directing a coding agent leaves to it in practice.
Because the agent picks its own tools and methods within the bounds of each subtask's instructions, it may take a different but equally valid route from the one the source study took, and arrive at slightly different numbers. Each subtask's instructions therefore fix the choices that would change the result, and the graders' tolerances absorb the smaller differences that remain.
Question: Regarding whether deterministic graders measure scientific correctness: The paper’s ground truth and scoring criteria are mainly based on the specific analysis pipelines and results in the original studies. Have the authors conducted independent human expert validation of these graders, especially to assess the proportion of answers that are inconsistent with the reference answer but may be equally reasonable biologically or statistically yet are judged incorrect? If such cases exist, how do the authors ensure that the benchmark score represents the agent’s actual scientific reasoning / bioinformatics capability rather than merely its ability to reproduce a specific reference workflow?
On the reviewer's question of whether the graders measure scientific correctness, OmicsBench's benchmark construction pipeline grounds each reference answer in its source study through the following steps:
- The writer agent takes the reference answer for each subtask from a result that the source study reports in its text, tables, figures, or supplementary files.
- As we have added to Section 3.2, the writer agent confirms that each reference answer can be replicated from the study's published data.
- As Section 3.2 also now describes in more detail, the human experts who proposed each source paper manually confirm that each reference answer faithfully translates the study's results into its subtask.
As for answers that differ from a reference answer but are equally reasonable, each subtask's instructions leave the agent real scientific choices to make, but constrain the analysis enough that any reasonable set of choices arrives at nearly the same answer. Where reasonable choices still leave small numerical differences, the graders' built-in tolerances absorb them, so that if two reasonable normalization methods put a gene's fold change at 2.1 and 2.3, for example, a numeric grader with a relative tolerance credits both.
Once again, thank you for your thoughtful review and valuable feedback! We hope our replies and revisions address most of your concerns. Please consider raising your review score if you feel they have strengthened our paper.
I thank the authors for their response, which has clarified several aspects of the benchmark. However, some concerns remain. The improvement in the gated setting cannot be clearly attributed to feedback, as the effect of additional retries has not been isolated. Furthermore, although the authors have clarified the reference curation and scoring tolerance mechanisms, it remains unclear whether the scorers can reliably recognize alternative but scientifically valid analytical results. These limitations are worth considering when interpreting the benchmark results.
Thank you for taking the time to carefully review our paper! We'll address your comments below:
Since this weakness and the question that follows it concern the same control, we answer them together. We thank the reviewer for pointing this out, and we agree that gated mode's gains reflect the in-context feedback and the additional attempts together. To clarify, the feedback itself is minimal, since an agent whose answer fails is told only that its attempt fell below the required threshold and receives no other information about its answer.
Moreover, we have revised the limitations section to state that the gated gains are an upper bound on what in-context feedback contributes, and the future work section now takes up the reviewer's suggested control, retries without feedback, as the direct way to isolate that contribution.