- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BioReasonBench evaluates multimodal reasoning on Chinese high-school biology examination and practice questions. The benchmark contains 1027 questions, including 894 with images or tables, and covers single-select, multiple-select, and multi-blank open-ended formats. Questions are annotated along five axes: biological organization, organism group, mechanism field, reasoning mode, and visual type. The evaluation distinguishes exact from partial multiple-select performance and blank-level from whole-question open-answer correctness, using Kimi-K2.5 to grade open responses.
The authors evaluate 13 multimodal models and report mean accuracies of 61.2% on single-select, 28.3% on exact multiple-select, and 11.6% on fully correct open questions. They analyze performance across biological and visual categories and use an LLM to attribute errors to mechanisms such as visual-symbol binding and hypothesis cascades. A comparison on one 25-question examination finds that ten students from a top-tier high school average 88.1/100, versus 71.0 for the strongest model. The principal contributions are the biology-focused collection, evaluation of answer completeness, and proposed diagnostic analyses.
Critical strengths
The benchmark evaluates a practically meaningful aspect of scientific answering: completeness. Almost all multiple-select questions require two or three correct options, and the 298 open questions contain 2,238 answer blanks. Reporting both partial and complete correctness makes it possible to distinguish models that recover some required information from models that deliver a fully correct answer. Table 2 demonstrates that this distinction materially changes the interpretation of performance. This is a useful evaluation contribution regardless of whether the proposed explanation for the completion gap is ultimately confirmed.
The domain-specific organization and empirical coverage support useful analysis beyond an aggregate leaderboard. The five annotation axes, source divisions, evaluation of 13 models, and directly observable output-error summaries provide several ways to inspect model weaknesses. The student comparison also supplies a concrete reference using the same examination and point allocation. Although that comparison should remain limited to the selected cohort and examination, it helps establish that the tasks have a meaningful human performance reference.
Critical weaknesses
The central diagnosis of structured visual reasoning is not sufficiently isolated from other explanations. Section 4.4 interprets the decline from single-select to exact multiple-select and fully correct open answers as evidence that assembling evidence is the primary bottleneck. These formats contain different questions and require different numbers of correct components. Open questions contain 3–14 blanks; even under an illustrative model with independent per-blank success probability (p), complete-question success decreases as (p^k) for (k) blanks. The completion gap is meaningful, but its existence alone does not identify a distinct reasoning failure.
Similarly, image presence does not establish image necessity. The supplied manuscript does not report matched image-removal or equivalent textual-evidence controls. Although Figure 3 separates question formats, the remaining relationships among biological topic, visual type, and item difficulty are not disentangled. Consequently, the results support descriptive difficulty patterns more strongly than the conclusion that failures primarily arise from visual binding or state tracking.
Establishing the diagnostic contribution is particularly consequential because CMMU already includes Chinese multimodal school-exam questions, biology, multiple-response questions, and fill-in-the-blank answers. BioReasonBench’s biological depth and preservation of complete multi-blank problems are potentially valuable distinctions, but the related-work discussion should position them against this close precedent.
The reliability of the evaluation and mechanism labels is insufficiently established in the available material. Kimi-K2.5 grades open answers, assigns primary error mechanisms to 4,208 incorrect responses, and is itself an evaluated model that leads the reported open-answer metrics. This makes independent validation consequential; it does not by itself establish that the judgments are biased.
The main text mentions agreement with expert-graded student responses, but the supporting Appendix E is absent from the supplied PDF. Agreement on student-answer scoring would not, by itself, validate grading of model-generated answers or the substantially different task of assigning error mechanisms. The reported dominance of binding and cascade errors over knowledge gaps therefore needs independent support. Categories inferred from generated reasoning traces should also be distinguished from established causal explanations of model behavior.
Questions for authors
Does “image-grounded” indicate that an image is present, or was its necessity independently verified? What evidence separates visual-reasoning failures from underlying biology difficulty and the number of required answer components? In particular, are there matched image-removal or textual-evidence results, and do the completion findings persist when stratified by blank count?
Please provide the referenced grading and error-analysis prompts and the validation results from Appendices B, C, and E. What agreement with independent experts is obtained on model-generated open answers, including both Kimi and non-Kimi outputs? Is there separate expert validation of the L2 error categories, and how sensitive are the principal conclusions to the grader or annotator?
What are the evaluated question and blank counts for each model, and how are missing responses, API failures, and unparsable answers handled? What explains the missing GPT-4.1-mini set0 results, and do the main comparisons hold on common items? Please also clarify the unusually low GPT-5.4 multiple-select result. Section 4.7 describes its exact-to-partial gap as large, whereas Table 2 reports 6.7% versus 9.8%, a gap of only 3.1 percentage points.
How does the contribution differ from CMMU’s biology questions and mixed response formats, particularly regarding preserved multi-blank structure, annotation quality, and the diagnostic conclusions enabled by the benchmark? A precise comparison would help establish the significance of the proposed resource without requiring an extensive new leaderboard comparison.