acceptodds
Under review as a conference paper at ICLR 2027

OmicsBench: a long-horizon multi-turn benchmark of AI agents on research study-scale bioinformatics

Abstract

Computational biologists increasingly delegate substantial analyses to AI agents in long-horizon, multi-turn sessions, yet most benchmarks of agents' bioinformatics capabilities pose a single short task with a single-shot prompt. We present OmicsBench, a long-horizon, multi-turn bioinformatics benchmark for AI agents. Each of its 97 tasks links dependent subtasks into a directed acyclic graph, for 662 underlying subtasks in all. Each subtask is posed as one user turn and graded deterministically against an answer key, with optional in-context feedback on incorrect answers. Across 12 models on three harnesses, the slowest agent averages 3.6 hours per task, and no combination comes near saturating the benchmark, with mean scores of 0.43 to 0.57 out of a possible 1 without in-context feedback and 0.54 to 0.67 with it. Multi-turn evaluation also uncovers behavior that single-shot prompts cannot, as agents score lower on subtasks that depend on earlier subtasks, and in-context feedback helps agents recover some of that loss. Because deterministic scoring gives no qualitative account of how an agent failed, we also introduce an unsupervised discovery pipeline that describes, embeds, and clusters agents' failed attempts to surface emergent failure modes. Applied to OmicsBench, this pipeline exposes qualitative shortcomings in how agents carry out bioinformatics analyses, ranging from agents applying their own statistical filters in place of the specified ones to agents writing whole answers from memory.

Primary area:
datasets and benchmarks
Keywords:
Genomics, Omics, Computational Biology, Bioinformatics, AI, LLM, Benchmark, Evaluation, Dataset, Long-Horizon, In-Context Feedback, Multi-Turn, Biology

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_duLoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OmicsBench, a benchmark for evaluating the ability of large language model (LLM) agents to complete long-horizon, multi-turn bioinformatics research tasks. The authors point out that existing bioinformatics benchmarks mainly focus on single-step analyses or relatively independent tasks, making it difficult to evaluate multiple interdependent analysis steps in real research workflows and the impact of preceding results on subsequent tasks. To address this, OmicsBench models a complete research task as a directed acyclic graph (DAG), in which nodes represent specific bioinformatics subtasks and edges represent data or result dependencies between tasks. The benchmark contains 97 research-study-scale tasks and 662 subtasks, covering 11 biological domains, 18 analysis types, and 10 organisms, and it includes 58 classes of deterministic graders to automatically evaluate outputs such as gene lists, ranked results, pathways, genomic regions, and numerical values.

The authors further test 12 models and 3 types of agent harnesses, and compare two execution modes: in single-pass mode, the agent completes tasks sequentially according to a predefined workflow; in gated mode, if a subtask scores below 0.7, the agent is allowed up to three retries. Experimental results show that different agents achieve average scores of approximately 0.43–0.57 in single-pass mode, while gated mode can raise scores to approximately 0.54–0.67, indicating that current models still have clear deficiencies in reliability and task execution on complex, long-horizon bioinformatics workflows, while failure feedback and retries can bring some performance improvement. The authors also further analyze agent execution trajectories, summarize and identify multiple failure modes, including incorrect method selection or parameter configuration, result sets that are too broad or too narrow, insufficient use of input data, premature termination, replacing formal statistical analysis with keyword matching, and failure to perform multiple testing correction. Overall, the main contribution of this paper is to construct a benchmark that is closer to actual computational biology research workflows and to use it to systematically evaluate the capabilities and main failure modes of current AI agents in long-horizon, multi-turn, task-dependent bioinformatics research tasks.

Critical strengths

The benchmark design has strong novelty and practical significance. OmicsBench does not merely evaluate the ability to complete a single bioinformatics task; instead, it models a real research workflow as a DAG composed of multiple interdependent subtasks, thereby enabling evaluation of an agent's context retention, result passing, and sequential task execution over long-horizon tasks. Compared with traditional single-step benchmarks, this design is closer to actual computational biology research workflows and is one of the paper's most important contributions.

The evaluation scale is relatively large and combines quantitative performance with failure-mode analysis. The authors construct a benchmark containing 97 research-study-scale tasks and 662 subtasks, and systematically evaluate it across multiple models and agent harnesses. At the same time, the paper not only reports final task scores but also further analyzes agent trajectories and summarizes multiple specific failure modes. This allows the benchmark not only to answer how current models perform, but also to reveal where and in what ways agents fail in bioinformatics analyses, giving it good diagnostic value and potential to drive future research.

Critical weaknesses

Gated mode cannot cleanly demonstrate that the “feedback mechanism” itself brings performance improvement. The paper mainly interprets the performance improvement between single-pass and gated modes as the agent being able to self-correct after receiving failure feedback. However, gated mode also allows the agent to retry up to 3 times, so the improvement may come from additional computational budget and multiple sampling rather than from feedback itself. In particular, if the final score is taken as the best result among multiple attempts, then this setting is effectively close to best-of-3 evaluation. The absence of a control such as “allowing multiple attempts but providing no feedback” makes the paper’s conclusion about the effectiveness of feedback-driven interaction still insufficiently convincing. This directly affects our assessment of the gated evaluation results and their scientific significance.

The correspondence between the benchmark’s scoring method and “true scientific ability” remains highly questionable. OmicsBench’s ground truth and deterministic graders are largely constructed based on the specific analysis pipelines and results in the original published studies, so whether an agent can reproduce the predefined reference analysis will significantly affect the final score. However, in real bioinformatics research, different but reasonable analysis methods, parameter choices, or statistical strategies may produce different results, and this does not mean those results are scientifically wrong. Therefore, the current benchmark score may simultaneously reflect bioinformatics analysis ability, the ability to strictly follow a specified analysis pipeline, and the ability to reproduce a reference answer, rather than being fully equivalent to scientific task-solving ability. Since this concerns the core capability that the benchmark claims to measure, it will substantially affect our evaluation of the paper’s main contribution.

Questions for authors

Questions for Authors

Regarding the role of feedback versus additional retries in Gated mode: The paper interprets the performance improvement of Gated mode over Single-pass mode as the agent being able to use feedback for self-correction. Did the authors conduct a control experiment in which the agent is given the same number of retries and the same computational budget but no specific failure feedback? If so, what is the performance improvement under this control? This result would help determine whether the observed gains come from feedback or mainly from additional attempts.

Regarding whether deterministic graders measure scientific correctness: The paper’s ground truth and scoring criteria are mainly based on the specific analysis pipelines and results in the original studies. Have the authors conducted independent human expert validation of these graders, especially to assess the proportion of answers that are inconsistent with the reference answer but may be equally reasonable biologically or statistically yet are judged incorrect? If such cases exist, how do the authors ensure that the benchmark score represents the agent’s actual scientific reasoning / bioinformatics capability rather than merely its ability to reproduce a specific reference workflow?

3 comments

Thank you for taking the time to carefully review our paper! We'll address your comments below:

Weakness: Gated mode cannot cleanly demonstrate that the “feedback mechanism” itself brings performance improvement. The paper mainly interprets the performance improvement between single-pass and gated modes as the agent being able to self-correct after receiving failure feedback. However, gated mode also allows the agent to retry up to 3 times, so the improvement may come from additional computational budget and multiple sampling rather than from feedback itself. In particular, if the final score is taken as the best result among multiple attempts, then this setting is effectively close to best-of-3 evaluation. The absence of a control such as “allowing multiple attempts but providing no feedback” makes the paper’s conclusion about the effectiveness of feedback-driven interaction still insufficiently convincing. This directly affects our assessment of the gated evaluation results and their scientific significance.

Question: Regarding the role of feedback versus additional retries in Gated mode: The paper interprets the performance improvement of Gated mode over Single-pass mode as the agent being able to use feedback for self-correction. Did the authors conduct a control experiment in which the agent is given the same number of retries and the same computational budget but no specific failure feedback? If so, what is the performance improvement under this control? This result would help determine whether the observed gains come from feedback or mainly from additional attempts.

Since this weakness and the question that follows it concern the same control, we answer them together. We thank the reviewer for pointing this out, and we agree that gated mode's gains reflect the in-context feedback and the additional attempts together. To clarify, the feedback itself is minimal, since an agent whose answer fails is told only that its attempt fell below the required threshold and receives no other information about its answer.

Moreover, we have revised the limitations section to state that the gated gains are an upper bound on what in-context feedback contributes, and the future work section now takes up the reviewer's suggested control, retries without feedback, as the direct way to isolate that contribution.

Weakness: The correspondence between the benchmark’s scoring method and “true scientific ability” remains highly questionable. OmicsBench’s ground truth and deterministic graders are largely constructed based on the specific analysis pipelines and results in the original published studies, so whether an agent can reproduce the predefined reference analysis will significantly affect the final score. However, in real bioinformatics research, different but reasonable analysis methods, parameter choices, or statistical strategies may produce different results, and this does not mean those results are scientifically wrong. Therefore, the current benchmark score may simultaneously reflect bioinformatics analysis ability, the ability to strictly follow a specified analysis pipeline, and the ability to reproduce a reference answer, rather than being fully equivalent to scientific task-solving ability. Since this concerns the core capability that the benchmark claims to measure, it will substantially affect our evaluation of the paper’s main contribution.

We thank the reviewer for raising this concern, and we completely agree that a given dataset and associated scientific question can often be analyzed in more than one reasonable way, which is part of what makes bioinformatics analysis hard to evaluate.

We chose to design OmicsBench tasks as fixed sequences of subtasks that each follow one study's analysis, because this is how scientists tend to work with agentic coding tools in practice. A scientist typically settles on one high-level approach out of several workable ones and then directs the tool through it one step at a time, giving feedback and letting it retry a step until the result is satisfactory, a workflow that OmicsBench mirrors as closely as a deterministic benchmark can.

Even so, OmicsBench tasks do not reduce to following a specified pipeline, since each subtask fixes the result to produce, not how the agent produces it. The agent decides how to read and prepare the data, which tools and methods to use, and how to check its own intermediate results, the same choices that a scientist directing a coding agent leaves to it in practice.

Because the agent picks its own tools and methods within the bounds of each subtask's instructions, it may take a different but equally valid route from the one the source study took, and arrive at slightly different numbers. Each subtask's instructions therefore fix the choices that would change the result, and the graders' tolerances absorb the smaller differences that remain.

Question: Regarding whether deterministic graders measure scientific correctness: The paper’s ground truth and scoring criteria are mainly based on the specific analysis pipelines and results in the original studies. Have the authors conducted independent human expert validation of these graders, especially to assess the proportion of answers that are inconsistent with the reference answer but may be equally reasonable biologically or statistically yet are judged incorrect? If such cases exist, how do the authors ensure that the benchmark score represents the agent’s actual scientific reasoning / bioinformatics capability rather than merely its ability to reproduce a specific reference workflow?

On the reviewer's question of whether the graders measure scientific correctness, OmicsBench's benchmark construction pipeline grounds each reference answer in its source study through the following steps:

  • The writer agent takes the reference answer for each subtask from a result that the source study reports in its text, tables, figures, or supplementary files.
  • As we have added to Section 3.2, the writer agent confirms that each reference answer can be replicated from the study's published data.
  • As Section 3.2 also now describes in more detail, the human experts who proposed each source paper manually confirm that each reference answer faithfully translates the study's results into its subtask.

As for answers that differ from a reference answer but are equally reasonable, each subtask's instructions leave the agent real scientific choices to make, but constrain the analysis enough that any reasonable set of choices arrives at nearly the same answer. Where reasonable choices still leave small numerical differences, the graders' built-in tolerances absorb them, so that if two reasonable normalization methods put a gene's fold change at 2.1 and 2.3, for example, a numeric grader with a relative tolerance credits both.


Once again, thank you for your thoughtful review and valuable feedback! We hope our replies and revisions address most of your concerns. Please consider raising your review score if you feel they have strengthened our paper.

I thank the authors for their response, which has clarified several aspects of the benchmark. However, some concerns remain. The improvement in the gated setting cannot be clearly attributed to feedback, as the effect of additional retries has not been isolated. Furthermore, although the authors have clarified the reference curation and scoring tolerance mechanisms, it remains unclear whether the scorers can reliably recognize alternative but scientifically valid analytical results. These limitations are worth considering when interpreting the benchmark results.

Reviewer_yWjPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose OmicsBench as a benchmark for long-horizon multi-turn bioinformatics agents, with each task structured as a DAG where each node comprises sub-tasks along with a suite of graders. They also use OmicsBench to evaluate multiple LLMs and harnesses, identify the best-performing setups, and decompose agent responses and failure modes.

Critical strengths

Strengths

  • The paper is well-placed as a new benchmark at the intersection of single-pass and long-range tasks.
  • The benchmark overview describes the relevant information, such as tasks, modality, data size distributions, etc., concisely.
  • The evaluation is comprehensive as it covers 12 models and 3 harnesses.
  • The failure analyses (including unsupervised new failures) provide interesting insights into the typical failure modes of current bioinformatics agents.
Critical weaknesses
  • Major concerns include clarifications in benchmark construction and its relevance to real-world bioinformatics agent use:
    • How are the DAGs of sub-tasks constructed?
    • If a DAG is entirely up to the agents (writer, reviewer, etc.), then how do human experts verify the generated DAGs?
    • How much effort is required by domain experts during the verification stage (e.g., average time taken to evaluate one sub-task/task/DAGs)?
    • How do you ensure that the DAG is complete and correct for a task?
    • Since the bioinformatics tasks for research are open-ended, the DAG might not be unique, and the evaluation constrains the agent to a particular DAG. What if multiple DAGs (multiple ways) can solve a high-level task?
    • Regarding the evaluators, how is their correctness (e.g., all possible similarities of biological entities/naming is handled) ensured? Next, how is their completeness ensured (i.e., does the set of evaluators cover all aspects of that sub-task)?
    • Can existing benchmarks (e.g., BixBench-Hypothesis) be formulated into the proposed definition of tasks, in particular, convert protocols into nodes and corresponding rubric criteria into node-specific evaluators?
    • Since the tasks are created from existing works (papers), some of the performance might (especially when the agent answers without doing the analysis) be explained from their pre-training - is there a potential way in which your evaluators that can be used to address this or reduce the score by adding a penalty (e.g., a low score if the evaluator passes but there is not the correct analysis)?
    • In the real world, it is rare that human experts would design the entire DAG and the evaluators for each step (e.g., maybe they are not sure of the exact analysis)? How closely does strict step-by-step evaluation align with real-world use of such agents?
  • Can you describe the final set of sub-tasks (662) and the final set of graders (57)? How much overlap is there between the tasks and graders across the 97 tasks?
  • The benchmark derives tasks and answers from published studies, but these studies may have been part of the model's training data and some correct answers could reflect recall of published results. Keeping benchmark tasks private does not exclude exposure to these published studies. Moreover, the reviewer-agent check for data leakage is insufficiently described to assess whether it addresses this concern. No publication-date range is provided or the list of papers as the authors decided to keep 94 /97 tasks private. Because of this it is difficult to evaluate how truly useful and well designed this benchmark is. Even if all these papers are very recent, the future models will be trained on them so I am not convinced of the usefulness of this benchmark in the future and I believe the authors should release their tasks.
  • What does the multi-turn formulation changes? The authors motivate the benchmark through multi-turn interaction, but do not compare the same tasks presented as one complete prompt versus successive requests under matched budgets.
  • Some relevant benchmarks in related work are missing like HeurekaBench and ScienceAgentBench.
Questions for authors

See weaknesses above

3 comments

Thank you for taking the time to carefully review our paper! We'll address your comments below:

How are the DAGs of sub-tasks constructed?

We have added a description of how the task DAGs are built to Section 3.2 of the revised paper, and we agree that the original text said too little about it.

Each DAG follows the analysis path that its source study carried out, with each subtask asking for one result that the study reports and each edge marking where the study used an earlier result to reach a later one. We tie the DAG to the study because the study's reported results are the evidence for each answer key, and because a published study's order of analyses reflects a workflow that scientists chose for that data.

If a DAG is entirely up to the agents (writer, reviewer, etc.), then how do human experts verify the generated DAGs?

The human experts who proposed the source papers for the OmicsBench tasks also verified that the DAG generated for each task matched the analysis in that task's source paper, since each expert had already read and understood the paper they proposed. Each expert checked the subtasks and dependencies of every DAG built from their paper against that paper's analysis, and edited or rejected any task whose DAG did not match.

How much effort is required by domain experts during the verification stage (e.g., average time taken to evaluate one sub-task/task/DAGs)?

We have added to Section 3.2 that human experts spent approximately 8 to 10 hours on each task, to better assure readers of the tasks' quality.

How do you ensure that the DAG is complete and correct for a task?

The human experts who verify each DAG are the final check that it is complete and correct. Each expert had proposed the source paper for the task they verified, so they had read and understood that paper's analysis and could check every subtask and dependency in the DAG against it.

Before the experts' review, the writer agent's instructions forbid it from adding dependencies that the paper's analysis lacks or from flattening real dependencies into a list, and, as we now describe in Section 3.2, the reviewer agent checks each task's dependencies against the paper's analysis.

Since the bioinformatics tasks for research are open-ended, the DAG might not be unique, and the evaluation constrains the agent to a particular DAG. What if multiple DAGs (multiple ways) can solve a high-level task?

We thank the reviewer for raising this, and we agree that more than one DAG can often solve the same high-level task, which is part of what makes bioinformatics analysis hard to evaluate.

We chose to fix each OmicsBench task to one DAG to mirror how scientists often work in practice, choosing one approach among several workable ones and then directing an agentic coding tool through it one request at a time.

Each DAG therefore poses its source study's approach as a series of requests with the high-level choices already made, so the benchmark measures how well an agent carries out a plan that a scientist has settled on.

Regarding the evaluators, how is their correctness (e.g., all possible similarities of biological entities/naming is handled) ensured? Next, how is their completeness ensured (i.e., does the set of evaluators cover all aspects of that sub-task)?

We thank the reviewer for this question, which the original text did not answer clearly. As we now describe in Section 3.2, we verified, for every reference answer across the 97 benchmark tasks, that each term eligible for database matching, such as a gene, organism, or metabolite, can be matched through its reference database. As a result, an agent earns credit for an entity in a reference answer no matter which of that entity's recorded names it uses, so it cannot lose credit for naming an entity differently from the reference answer.

As for completeness, the writer agent gives each subtask one grader for every part of its result that the source study reports, such as which genes change and in which direction, and the human experts who verified each task checked that no reported part of the result was left without a grader. Parts of the analysis whose outputs the study does not report, or does not describe fully, have no reference output to grade against, so they are left to our failure mode analysis pipelines, which read the agent's trajectory.

Can existing benchmarks (e.g., BixBench-Hypothesis) be formulated into the proposed definition of tasks, in particular, convert protocols into nodes and corresponding rubric criteria into node-specific evaluators?

Yes, and the framework is built to allow this, since any study that can be expressed as a DAG of deterministically gradable steps can become an OmicsBench task (see Section 1). The protocol steps of a BixBench-Hypothesis task would become subtasks, with an edge wherever a step uses an earlier step's output, and its final verdict on the hypothesis would become a closed-choice subtask. The one requirement is that each rubric criterion be restated as an answer key that a deterministic grader can score, since OmicsBench grades without an LLM judge.

Since the tasks are created from existing works (papers), some of the performance might (especially when the agent answers without doing the analysis) be explained from their pre-training - is there a potential way in which your evaluators that can be used to address this or reduce the score by adding a penalty (e.g., a low score if the evaluator passes but there is not the correct analysis)?

We thank the reviewer for this suggestion. Our failure mode analysis pipelines address this directly, since they read agents' trajectories, and as Section 4.5 shows, they surfaced answering from memorized knowledge as a failure mode of its own. Both the a priori annotation pipeline and the unsupervised discovery pipeline can flag any answer that an agent produced by regurgitating its training data rather than computing it from the dataset. Those attempts can then be discarded and the agent's score adjusted accordingly, which provides the penalty that the reviewer describes.

Furthermore, the no-data baseline that we ran at the request of Reviewer iwpt (see Section 4.2 and the new Figure A5) also suggests that such answers account for little of the benchmark's scores, since agents scored a mean of 0.09 on tasks without their datasets, compared to 0.80 with them.

In the real world, it is rare that human experts would design the entire DAG and the evaluators for each step (e.g., maybe they are not sure of the exact analysis)? How closely does strict step-by-step evaluation align with real-world use of such agents?

We thank the reviewer for this question. In the authors' opinion, the OmicsBench evaluation model emulates the most common way that scientists, and users of agentic coding tools more generally, work with these tools. The DAG emulates how a scientist would pick their way through obtaining insights from biological data with a coding agent, by choosing high-level analyses, letting the coding agent carry them out, reviewing the results, and if necessary asking the coding agent to revise its work, before giving the coding agent the next analysis to carry out.

We agree with the reviewer that a scientist would not usually write an evaluator for each step, and our graders are not meant to mirror how a scientist checks a result. Instead, along the analysis path that the source study took, the study's reported results tell us what each subtask should return, which lets us assess the agent fairly at every step.

Can you describe the final set of sub-tasks (662) and the final set of graders (57)? How much overlap is there between the tasks and graders across the 97 tasks?

We thank the reviewer for this question, and we would like to clarify how the benchmark's tasks and graders relate. The subtasks cover a broad range of analyses, from differential expression and enrichment analysis to cell type annotation, while the graders are far fewer because each is built for one shape of answer, such as a set of genes, rather than for one task or type of analysis. The same graders are therefore shared across many tasks and subtasks, and for a fuller picture, Figure 3C breaks down the subtasks by analysis type and Appendix F lists every grader.

The benchmark derives tasks and answers from published studies, but these studies may have been part of the model's training data and some correct answers could reflect recall of published results. Keeping benchmark tasks private does not exclude exposure to these published studies. Moreover, the reviewer-agent check for data leakage is insufficiently described to assess whether it addresses this concern. No publication-date range is provided or the list of papers as the authors decided to keep 94 /97 tasks private. Because of this it is difficult to evaluate how truly useful and well designed this benchmark is. Even if all these papers are very recent, the future models will be trained on them so I am not convinced of the usefulness of this benchmark in the future and I believe the authors should release their tasks.

We thank the reviewer for raising this, and we agree that data leakage is a real risk for any benchmark that draws on published science, since models may have been trained on the very studies whose results the benchmark asks them to reproduce. To measure this risk, we ran a no-data baseline, which Reviewer iwpt also requested, to check whether agents actually rely on a task's data, and agents scored a mean of 0.09 on tasks without their datasets, compared to 0.80 with them (see Section 4.2 and the new Figure A5).

With the same aim, the reviewer agent in our benchmark construction pipeline sends a task back to the writer agent whenever it reveals details that could identify its source study, such as its title, authors, DOI, or accession numbers, so that agents must work from the data. We have now added text describing this check in Section 3.2.

Finally, to catch any leakage that gets past these checks, anyone evaluating agents on OmicsBench can run the failure mode analysis pipelines of Section 4.5 on agents' trajectories to identify answers written from memorized knowledge rather than computed from the data.

What does the multi-turn formulation changes? The authors motivate the benchmark through multi-turn interaction, but do not compare the same tasks presented as one complete prompt versus successive requests under matched budgets.

We thank the reviewer for this question. In the authors' opinion, the multi-turn format matters for reasons that hold irrespective of any difference in scores.

First, it makes OmicsBench's gated mode possible. In gated mode, when an agent's answer to a subtask fails, we tell the agent and let it retry before giving it the next subtask, much as a scientist would send back a flawed result. That feedback has to arrive between one subtask and the next, which is only possible when each subtask is its own turn.

Second, as the revised conclusion now states, posing each subtask as its own turn in the session divides agents' trajectories cleanly into sections corresponding to the different subtasks that make up a task. Unlike a single prompt, in which the work on different subtasks is interleaved in one trajectory, this division lets those running the benchmark relate an agent's work on one subtask to its work on the subtasks it depends on, and analyze its failures subtask by subtask.

Some relevant benchmarks in related work are missing like HeurekaBench and ScienceAgentBench.

We thank the reviewer for pointing out these two relevant benchmarks, and we have added both to the related work in Section 2 of the revised paper.


Thank you again for your detailed questions and valuable feedback! We hope our answers have addressed most of your concerns, and we would be grateful if you would consider raising your score if you feel the revisions have improved our paper.

Reviewer_iwptOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper develops omicsbench, a collection of 97 bioinformatics tasks curated from previous papers. They evaluated LLMs on this benchmark, finding that the models still have trouble completing these tasks.

Critical strengths

Evaluating agentic performance in bioinformatics is an important direction and this benchmark is generally well designed. The failure analysis in the paper is interesting. Overall, with some improvements, I think this could be a useful benchmark for the community.

Critical weaknesses

I have several concerns and questions.

  1. Since the answers are curated from published papers, there is a real chance of models memorizing the answers. A no-data baseline (no access to the dataset, same questions) would be helpful to add, but doesn't rule out memorization.
  2. How do we know that the "ground truth" answers in the benchmark are actually correct?
  3. There could be multiple reasonable ways of analyzing the same data. How would the benchmark capture this ambiguity?
Questions for authors

See above.

2 comments

Thank you for taking the time to carefully review our paper! We'll address your comments below:

Since the answers are curated from published papers, there is a real chance of models memorizing the answers. A no-data baseline (no access to the dataset, same questions) would be helpful to add, but doesn't rule out memorization.

We thank the reviewer for this point, and we agree that models may have seen the source studies during training, which is an important concern for a benchmark built from published work.

We have run the no-data baseline that the reviewer suggests, on a sample of 20 OmicsBench tasks with three model/harness combinations, and agents scored a mean of 0.09 on tasks without datasets, compared to 0.80 with datasets (see Section 4.2 and the new Figure A5).

As we now describe in more detail in Section 3.2, OmicsBench also makes it harder for an agent to answer from memorized results by sanitizing every task of references to its source study within the writer and reviewer agent loop, where the reviewer agent sends a task back to the writer agent for revision if it reveals details that could identify the source study, such as its title, authors, DOI, accession numbers, quotes, or figure and table references.

Additionally, since, as the reviewer notes, a no-data baseline cannot rule out memorization by itself, we note that anyone using the OmicsBench framework can run our failure mode analysis pipelines on agents' trajectories to find agents that wrote answers from memorized knowledge instead of computing them from the data (see Section 4.5).

How do we know that the "ground truth" answers in the benchmark are actually correct?

We primarily ensure that each reference answer is correct by drawing it from the results that the source study reports in its text, tables, figures, or supplementary files.

Additionally, as we now describe in Section 3.2, the writer agent also confirms that the study's results can be replicated from the study's published data, which we pass to the agent alongside the prompts, so that a correct analysis earns full credit.

Finally, as Section 3.2 also now describes in more detail, the human experts who verified each task, the same experts who proposed its source paper, checked each reference answer against the study to ensure that its results were faithfully translated into subtasks and their reference answers.

There could be multiple reasonable ways of analyzing the same data. How would the benchmark capture this ambiguity?

We thank the reviewer for raising this, and we agree that the same data can often be analyzed in more than one reasonable way, which is part of what makes bioinformatics analysis hard to evaluate.

We chose to settle each OmicsBench task on one analysis to mirror how scientists often work in practice, choosing one approach among several workable ones and then directing an agentic coding tool through it one request at a time.

Each task therefore poses its source study's approach as a series of requests with the high-level choices already made, so the benchmark measures how well an agent carries out an analysis that a scientist has settled on.


Once again, thank you for your thoughts and valuable feedback! We hope we have addressed most of your concerns. Please consider raising your review score if you feel this process has improved the quality of our paper.

I thank the authors for their response, which has addressed some of the questions. But other questions remain. The benchmark rely on published results, but we know that there are errors in many published results. Moreover if there are multiple valid analyses, then it seems that the benchmark could wrongly penalize a correct analysis different from the chosen one.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Revisions

We thank all three reviewers for their careful and constructive reviews. We have uploaded a revised version of the paper, and we summarize its changes below.

New experiment, as requested by Reviewer iwpt

In response to Reviewer iwpt, we ran a sample of OmicsBench tasks with the same prompts but with each task's dataset removed (the "no-data baseline" that Reviewer iwpt proposed), and report the results in Section 4.2 and a new Figure A5. We found that agents scored far lower without the data, showing that they cannot answer the tasks from their knowledge of the published studies alone.

In response to Reviewers iwpt and yWjP

  • Section 3.2 describes how the reviewer agent sanitizes each task of details that could identify its source study.

In response to Reviewers iwpt and duLo

  • Section 3.2 states that the writer agent confirms each reference answer can be replicated from the study's published data.

In response to Reviewer yWjP

  • Section 2 adds HeurekaBench and ScienceAgentBench to the related work.
  • Section 3.2 describes how each task's DAG follows its source study's analysis.
  • Section 3.2 states that every term in the reference answers that is eligible for database matching resolves in its reference database.
  • Section 3.2 reports the time that human experts spent on each task.

In response to Reviewer duLo

  • The limitations section states that the gated gains are an upper bound on what in-context feedback contributes, and the future work section proposes a control with retries that receive no feedback.

Other changes

  • The discussion expands the implications of the study for training.
  • Minor wording and clarity edits throughout.

Dear Reviewers,

As the discussion phase draws to a close, please acknowledge the author's response and let them know whether it has addressed some or all of your concerns.

Thank you for the time and effort you and the other reviewers have put into this discussion.

Dear Reviewers,

Thank you again for your reviews. As the discussion period closes today, we wanted to check whether our responses and the revised paper have addressed your concerns. We would be glad to answer any further questions before the deadline, and if your concerns are resolved, we would be grateful if you would consider updating your score.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.