acceptodds
Withdrawn from ICLR 2027

BioReasonBench: A Diagnostic Benchmark for Multimodal Reasoning with Structured Visual Evidence in Biology

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) can describe scientific images fluently, but it remains unclear whether they can use biological images as structured evidence across a complete reasoning chain. Existing evaluations are often text-only, recognition-oriented, or summarized by a single accuracy that obscures where reasoning fails. We introduce BioReasonBench, a diagnostic benchmark of biology questions, spanning three question types: single-select, multiple-select, and open-ended. Of these, (\(87.0%\)) are grounded in images of pedigrees, pathways, experimental setups, quantitative curves, and other biological content. Unlike the single-answer choice and single-target open responses common in multimodal benchmarks, \(193/194\) multiple-select items have two or three correct options, and all open-ended items are multi-blank ( blanks in total). Each item is organized along five diagnostic axes and source/year splits, and is evaluated with question-type-aware exact, partial-credit, and LLM-judge protocols. Across MLLMs, mean accuracy falls from \(61.2%\) on single-select to \(28.3%\) on exact multiple-select and \(11.6%\) on fully correct open-ended questions. Relation graphs, tables, and genetics are especially difficult. Partial option sets and partially correct blanks are common, revealing failures of correspondence, state tracking, calibration, and causal completeness rather than uniform factual ignorance. On set0, the strongest model scores \(71.0/100\), matching the lowest of Grade-12 students, who average . We hope BioReasonBench provides actionable insights into fine-grained visual-reasoning mechanics and guides the development of next-generation MLLMs capable of trustworthy scientific inquiry.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Multimodal Reasoning Benchmark, AI for Biology

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rfHJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BioReasonBench evaluates multimodal reasoning on Chinese high-school biology examination and practice questions. The benchmark contains 1027 questions, including 894 with images or tables, and covers single-select, multiple-select, and multi-blank open-ended formats. Questions are annotated along five axes: biological organization, organism group, mechanism field, reasoning mode, and visual type. The evaluation distinguishes exact from partial multiple-select performance and blank-level from whole-question open-answer correctness, using Kimi-K2.5 to grade open responses.

The authors evaluate 13 multimodal models and report mean accuracies of 61.2% on single-select, 28.3% on exact multiple-select, and 11.6% on fully correct open questions. They analyze performance across biological and visual categories and use an LLM to attribute errors to mechanisms such as visual-symbol binding and hypothesis cascades. A comparison on one 25-question examination finds that ten students from a top-tier high school average 88.1/100, versus 71.0 for the strongest model. The principal contributions are the biology-focused collection, evaluation of answer completeness, and proposed diagnostic analyses.

Critical strengths

The benchmark evaluates a practically meaningful aspect of scientific answering: completeness. Almost all multiple-select questions require two or three correct options, and the 298 open questions contain 2,238 answer blanks. Reporting both partial and complete correctness makes it possible to distinguish models that recover some required information from models that deliver a fully correct answer. Table 2 demonstrates that this distinction materially changes the interpretation of performance. This is a useful evaluation contribution regardless of whether the proposed explanation for the completion gap is ultimately confirmed.

The domain-specific organization and empirical coverage support useful analysis beyond an aggregate leaderboard. The five annotation axes, source divisions, evaluation of 13 models, and directly observable output-error summaries provide several ways to inspect model weaknesses. The student comparison also supplies a concrete reference using the same examination and point allocation. Although that comparison should remain limited to the selected cohort and examination, it helps establish that the tasks have a meaningful human performance reference.

Critical weaknesses

The central diagnosis of structured visual reasoning is not sufficiently isolated from other explanations. Section 4.4 interprets the decline from single-select to exact multiple-select and fully correct open answers as evidence that assembling evidence is the primary bottleneck. These formats contain different questions and require different numbers of correct components. Open questions contain 3–14 blanks; even under an illustrative model with independent per-blank success probability (p), complete-question success decreases as (p^k) for (k) blanks. The completion gap is meaningful, but its existence alone does not identify a distinct reasoning failure.

Similarly, image presence does not establish image necessity. The supplied manuscript does not report matched image-removal or equivalent textual-evidence controls. Although Figure 3 separates question formats, the remaining relationships among biological topic, visual type, and item difficulty are not disentangled. Consequently, the results support descriptive difficulty patterns more strongly than the conclusion that failures primarily arise from visual binding or state tracking.

Establishing the diagnostic contribution is particularly consequential because CMMU already includes Chinese multimodal school-exam questions, biology, multiple-response questions, and fill-in-the-blank answers. BioReasonBench’s biological depth and preservation of complete multi-blank problems are potentially valuable distinctions, but the related-work discussion should position them against this close precedent.

The reliability of the evaluation and mechanism labels is insufficiently established in the available material. Kimi-K2.5 grades open answers, assigns primary error mechanisms to 4,208 incorrect responses, and is itself an evaluated model that leads the reported open-answer metrics. This makes independent validation consequential; it does not by itself establish that the judgments are biased.

The main text mentions agreement with expert-graded student responses, but the supporting Appendix E is absent from the supplied PDF. Agreement on student-answer scoring would not, by itself, validate grading of model-generated answers or the substantially different task of assigning error mechanisms. The reported dominance of binding and cascade errors over knowledge gaps therefore needs independent support. Categories inferred from generated reasoning traces should also be distinguished from established causal explanations of model behavior.

Questions for authors

Does “image-grounded” indicate that an image is present, or was its necessity independently verified? What evidence separates visual-reasoning failures from underlying biology difficulty and the number of required answer components? In particular, are there matched image-removal or textual-evidence results, and do the completion findings persist when stratified by blank count?

Please provide the referenced grading and error-analysis prompts and the validation results from Appendices B, C, and E. What agreement with independent experts is obtained on model-generated open answers, including both Kimi and non-Kimi outputs? Is there separate expert validation of the L2 error categories, and how sensitive are the principal conclusions to the grader or annotator?

What are the evaluated question and blank counts for each model, and how are missing responses, API failures, and unparsable answers handled? What explains the missing GPT-4.1-mini set0 results, and do the main comparisons hold on common items? Please also clarify the unusually low GPT-5.4 multiple-select result. Section 4.7 describes its exact-to-partial gap as large, whereas Table 2 reports 6.7% versus 9.8%, a gap of only 3.1 percentage points.

How does the contribution differ from CMMU’s biology questions and mixed response formats, particularly regarding preserved multi-blank structure, annotation quality, and the diagnostic conclusions enabled by the benchmark? A precise comparison would help establish the significance of the proposed resource without requiring an extensive new leaderboard comparison.

Reviewer_XkcpOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces BioReasonBench, a diagnostic benchmark of 1,027 biology questions from Chinese Gaokao and mock examinations, 87% of which contain images or tables. It covers single-select, multiple-select, and multi-blank open-ended formats, with five diagnostic annotation axes and scoring protocols that distinguish partial from complete correctness. Evaluations of 13 MLLMs report mean accuracies of 61.2%, 28.3%, and 11.6% across these formats, respectively. A student comparison, error-attribution analyses, and ensemble experiments further examine model limitations in structured visual reasoning and answer completeness.

Critical strengths

(1) The benchmark explicitly evaluates answer completeness through genuine multiple-select and multi-blank questions, with partial-credit and blank-level scores distinguishing local correctness from complete answers.

(2) The evaluation combines diagnostic slices, a human baseline, error attribution, and ensemble analyses. The finding that majority voting can reduce multiple-select accuracy is practically informative.

Critical weaknesses

(1) Publicly circulated examination materials create a plausible training-data exposure risk that remains insufficiently quantified, limiting confidence in interpreting the reported performance as generalization to unseen questions.

(2) The paper interprets lower accuracy on multiple-select and multi-blank questions as evidence of an answer-completeness bottleneck. However, requiring multiple components to be correct can itself lower the probability of a fully correct answer. The lower accuracy could simply reflect the need to get more components right at once. It therefore does not demonstrate a specific weakness in the models’ ability to assemble complete answers.

(3) The human comparison involves ten students from one top-tier school and a single 25-question paper. Without uncertainty estimates or broader replication, the reported 17.1 points gap has limited generalizability.

Questions for authors

See Weaknesses.

Reviewer_oa6WOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BioReasonBench, a multimodal benchmark designed to evaluate reasoning over structured visual evidence in biology. The benchmark comprises 1,027 biology questions, of which 894 involve images or tables. The authors further categorize the questions along five dimensions, including biological organization level, organism group, biological mechanism, reasoning mode, and visual type, enabling a fine-grained analysis of MLLMs across different biological reasoning tasks. The experiments evaluate 13 MLLMs. The results indicate that current models perform substantially worse on multiple-select and open-ended questions than on single-select questions, with particularly pronounced difficulties in genetics and relational graph reasoning. Additionally, the authors compare the performance of several models with that of high school students, highlighting the remaining limitations of current MLLMs in complex biological reasoning tasks.

Critical strengths
  1. A Valuable Multimodal Benchmark Resource for Biology. This paper presents a biology benchmark comprising 1,027 questions across multiple biological domains, including genetics, metabolism, and neurobiology, and incorporates various types of biological images and structured information. Compared with benchmarks that primarily focus on single-select questions, the inclusion of multiple-select and multi-blank open-ended questions enables the benchmark to assess the completeness of model-generated answers to some extent.
  2. Fine-Grained Error Analysis. Beyond reporting overall accuracy, the paper attempts to characterize different types of errors, including visual-symbol binding failures, hypothesis propagation errors, and incomplete answers. It further examines model performance across five diagnostic dimensions. These analyses provide useful empirical insights into the capabilities and limitations of multimodal models in biological reasoning tasks.
Critical weaknesses
  1. Limited Research Contributions and Novelty, with Insufficient Comparisons to Existing Biology Benchmarks. First, the primary contribution of this paper lies in the collection, categorization, and organization of existing biology examination questions into an evaluation dataset. The overall contribution remains largely an extension of existing dataset construction practices and statistical analyses of model performance. For a contribution at the level of ICLR, the authors should further demonstrate that the proposed benchmark can reveal important limitations that existing evaluation frameworks fail to capture, or establish a biologically grounded visual reasoning evaluation framework with clear methodological novelty. Currently, the paper primarily reports performance differences across question types and biological categories, without introducing a systematic methodology for validating biological reasoning capabilities, such as interventions on visual evidence, verification of reasoning correctness, or more rigorous causal attribution analyses.

  2. Insufficient Reliability of the Evaluation Protocol and Inadequate Validation on Recent Frontier Models. First, the paper employs Kimi-K2.5 as an LLM-as-Judge for open-ended questions. However, biological reasoning involves complex mechanisms, specialized terminology, and causal relationships, making binary judgments by a general-purpose LLM potentially unreliable. More importantly, Kimi-K2.5 is also one of the evaluated models and achieves strong performance, raising concerns about self-preference and evaluation bias. Although the authors report human agreement validation, the evidence remains insufficient. More independent evaluation, such as expert-validated judges or cross-model assessments, is needed to ensure fairness and reliability. Second, the paper attributes the performance gap between single-select, multiple-select, and open-ended questions primarily to reasoning incompleteness. However, these question types differ substantially in answer complexity and scoring criteria. In particular, open-ended questions contain approximately 7.5 blanks on average, making question-level accuracy inherently lower under exact-match scoring. Therefore, the observed performance gap does not necessarily demonstrate deficiencies in structured visual reasoning. Finally, despite evaluating 13 MLLMs, the paper omits several recent frontier models, including the GPT-6 series and Claude family, particularly GPT-6 Astra and Claude Opus 5.5. Given recent advances in scientific and biological reasoning capabilities, these omissions limit the representativeness and generalizability of the findings. Additional evaluations on frontier models are necessary to establish whether the reported difficulty patterns, model rankings, and error mechanisms remain valid.

Questions for authors

Question 1: Benchmark Novelty and Comparison with Existing Work. Compared with existing biological visual reasoning benchmarks, such as the FigQA task in LAB-Bench and SORBE, what unique contributions does BioReasonBench offer in terms of evaluation capabilities, scientific question design, and diagnostic mechanisms? Can the authors provide cross-benchmark evaluations or ablation studies to quantitatively demonstrate that the proposed five-dimensional taxonomy identifies reasoning failures that existing evaluation frameworks cannot capture? Question 2: Evaluation Reliability and Representativeness of Frontier Models. Why do the authors select Kimi-K2.5, which is also one of the evaluated models, as the primary judge for open-ended questions? Can they provide independent human evaluations, inter-judge agreement results, and an analysis of how judge selection affects model rankings? Furthermore, can the authors explain why recent frontier models, such as the GPT-6 series and Claude Opus 5.5, were excluded and verify whether the reported conclusions generalize to these models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.