Beyond Benchmark Scores: An Analysis of Bioinformatics Agent Evaluations

As AI agents become capable of performing complex biological analyses, evaluating what they actually understand has become as important as evaluating what they can do. Bioinformatics benchmarks have followed this progression, moving from question answering towards evaluating if agents can conduct open-ended computational biology research tasks. This increased realism comes at the cost of verifiability. As more of the scientific process is delegated to the agent, it is not enough to evaluate just the results of the analysis, but to understand if we can trust the scientific process through which the agent arrives at the results. To get to the bottom of this question, this paper tries to systematically analyze components of a good evaluation and how to find out if we're actually evaluating biological reasoning.