acceptodds
Under review as a conference paper at ICLR 2027

DeepVisualMath: Comprehensive Evaluation of Visual Mathematical Reasoning in Large Multimodal Models

Abstract

Visual mathematical reasoning requires integrating visual evidence with mathematical concepts, yet its evaluation in multimodal large language models (MLLMs) remains fragmented across benchmarks targeting limited domains and skills. We introduce ***DeepVisualMath***, a unified benchmark and evaluation framework for holistic assessment of visual mathematical reasoning. The benchmark comprises **10,298 questions**, each paired with one image, organized into **7 categories, 18 subcategories, and 80 tasks**, spanning applied sciences, continuous mathematics, relational geometry across dimensions, discrete mathematics and formal logic, plane geometry, contextual mathematics, and solid geometry. Beyond the benchmark, we provide an automated curation pipeline, a reproducible evaluation harness, and an automated error analysis framework for detailed diagnosis across the task hierarchy. Evaluation of 10 MLLMs reveals substantial and uneven performance gaps: overall accuracy ranges from 15.50% to 66.67%, with even the strongest model leaving approximately one third of questions unresolved. Performance also varies markedly across mathematical domains; the leading model achieves 77.36% in continuous mathematics but only 51.85% in plane geometry, highlighting limitations obscured by aggregate scores. These findings demonstrate that strong performance in one mathematical domain does not imply broad visual mathematical competence. ***DeepVisualMath*** provides a systematic foundation for evaluating these capabilities and identifying priorities for improving mathematical reasoning grounded in visual evidence.

Primary area:
datasets and benchmarks
Keywords:
Visual mathematical reasoning, Multimodal large language models, Benchmarking, Holistic evaluation, Geometric reasoning, Automated error analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y62eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DeepVisualMath, a benchmark for visual mathematical reasoning comprising 10,298 instances spanning 7 categories, 18 subcategories, and 80 tasks. This benchmark was constructed from existing datasets through a process involving deterministic filtering, visual deduplication and balancing, and multimodal quality review. Furthermore, the authors propose a framework for automated evaluation and error analysis; and evaluated the performance of ten multimodal models and humans, and compared three AI agent systems using a subset containing 350 challenging problems.

Critical strengths

Deepvisualmath covers a broad scope while explicitly focusing on the reliance on visual information. It integrates multiple mathematical domains, including continuous mathematics, discrete reasoning, and spatial geometry. The "visual dependency" filtering mechanism addresses a significant issue: problems involving images can sometimes be solved using only textual cues. Once validated, this carefully constructed dataset will serve as a valuable resource for evaluation.

This work aims to move beyond the limitations of focusing solely on overall accuracy. Through category-level evaluation, problem-level coverage analysis, and error analysis, it distinguishes between the various causes of failure. Experiments with intelligent agents demonstrate that certain failures to provide a direct answer can be rectified through alternative reasoning processes, offering insights for further analysis.

Critical weaknesses
  1. Regarding the value of the benchmark itself, the fact that DeepVisualMath was created by filtering existing datasets somewhat diminishes its contribution. Although the paper notes that existing benchmarks are fragmented, it fails to adequately demonstrate which critical evaluation issues would remain unresolved if a unified classification system and evaluation protocol were applied to existing resources.
  2. While the paper's conclusions are sound, they do not offer significant new insights.
  3. There are conflicting and inconsistent data reported in the paper. Discrepancies exist in the category statistics between Table 1 and Table 7; furthermore, the model used for scoring is labeled as Gemma-4-31B-it in Section 4 and Appendix F.2, but as Gemini 3.1 in Appendix D.2. The number of valid evaluation samples for Kimi is only 1,796. These inconsistencies undermine the reliability of the paper.
Questions for authors

1.see Weaknesses 2. The paper lacks evaluation results for certain smaller-scale models, such as Qwen3.5-9B and Qwen3.8-27B; it is recommended to include evaluation results for these models.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Y62e [Part 1/3]

We thank the reviewer for a balanced review and for judging the paper's conclusions sound. The reviewer highlights the breadth of DeepVisualMath across continuous, discrete, and geometric domains, its explicit visual-dependency filter against text-only shortcuts, and its move beyond overall accuracy through category-level evaluation, item-level coverage, error analysis, and agents that recover some failures. We answer each question with quantitative evidence and add three smaller models as requested.


Weaknesses

  • W1. Value of a benchmark built from existing datasets.

    • Response: Applying a unified taxonomy and protocol to the raw sources would leave three problems that only item-level curation solves, and curation produces measurable properties that no source has. Table R1. What curation adds over the 21 sources. | Problem in raw sources | Size | What DeepVisualMath does | |---|---|---| | Items answerable without the image | 6,018 (32.3% of Stage 2 pool) | Removed by M3 | | Cross-source near-duplicate images | 3,475 (CLIP cosine > 0.97) | Removed in Stage 2 | | Incomplete coverage | Broadest source 10/18 subcategories; median 3; best pair 13; 2 subcategories in only one source | All 18 in one set | | 21 scoring protocols | 21 prompt formats and extractors | One harness, one judge (κ = 0.930) | | Unknown joint reliability | n/a | Split-half τ = 0.958; leave-one-source-out τ ≥ 0.889 | The failure mode is documented: GeminiPro reaches 42.7% on MMMU with no image input [2], and MMMU-Pro filtered text-answerable questions from MMMU to obtain a valid test [3]. Table R2. Curation changes what is measured: the same source, scored by category.
    SourceHighest categoryLowest categorySpread
    MMMUContinuous 57.6Discrete 18.838.8
    OlympiadBenchContinuous 46.8Applied 12.834.0
    STELAR-topoContinuous 56.3Solid 27.728.6
    MathVisionContinuous 36.1Solid 15.320.8

    Source-level reporting merges these differences into one number. Across all items, category explains correctness beyond source identity (logistic regression, χ²(6) = 1,686.4, p < 10⁻³⁰⁰). Table R3. Accepted benchmarks built by curating existing data.

    BenchmarkVenueBuilt from
    MathVista [1]ICLR 20245,405 of 6,141 items from 28 existing datasets
    MMStar [2]NeurIPS 20241,500 items selected from 6 benchmarks
    MMMU-Pro [3]ACL 2025Filtered and augmented MMMU
    MMLU-Pro [4]NeurIPS 2024 D&BFiltered MMLU plus existing sets

    Each was accepted because curation yields a more valid measurement than its sources.

AuthorsOpenReview ↗

Response to Reviewer Y62e [Part 2/3]

  • W2. The conclusions do not offer new insights.

    • Response: We agree that the paper should state its new findings explicitly. Table R4 lists them; none is observable in any single source benchmark. Table R4. Findings and the evidence for each.
    #FindingEvidence
    1Applied mathematics is the shared bottleneckLowest category for every model (2.16% to 15.48%), below the next-lowest with p ≤ 6.4 × 10⁻⁶ per model; Continuous highest for every model
    2Domain gaps within one model are largeBest-minus-worst category gap from 23.97 (Qwen3.7 Flash) to 57.05 points (Gemini 3.5 Flash)
    3Overall rank does not transfer across domainsGemini 3.1 Flash Lite, third overall, leads all models on Discrete, Solid, and Applied; model × category effect χ²(48) = 193.3, p = 2.4 × 10⁻¹⁹
    4The leading models are statistically tiedMcNemar separates 32 of 36 pairs, none among the top three; top-three ranks interchange in bootstrap resamples while ranks 6 to 9 never change
    5Error diversity determines ensemble coverageBest pair solves 66.1% of items, 14.7 points above the best single model; all models together solve 79.75%
    6Failures arise before computationDominant failure is loss of visual or task grounding: fluent algebra on a misread chart value, relation, or option (Section 5.4)
    7Tool use does not recover most hard failuresNewer-generation agents (Claude Sonnet 6, GPT 5.6 Luna Max) with code execution solve 15.43% and 17.71% of items every model fails
    8The human gap exceeds the model spreadBest model 34.38 points below high school, close to the 35.86-point model spread; every human level beats every model by ≥ 20.89 points in every category
    9Parameter count does not determine the profileQwen3-VL-8B exceeds the 31B Gemma model on Solid and on Discrete plus Socio-Econ. (Q2)

    Table R5. Human difficulty mirrors model difficulty (accuracy %).

    AppliedContinuous
    High school (human)77.26 (lowest)92.44 (highest)
    Best model15.48 (lowest)70.94 (highest)

    Humans and models find the same category hardest and the same category easiest, which supports the taxonomy as a measure of mathematical content.

AuthorsOpenReview ↗

Response to Reviewer Y62e [Part 3/3]

  • W3. Consistency of reported data.
    • Response: All three points are resolved in the revised paper (General Response G1 to G3).

      • (a) Table 1 vs. Table 7. Table 1 and Table 7 now share the final taxonomy, in which each subcategory belongs to exactly one category: Probability, Statistics and Analytics (512 items) belongs to Continuous, and Numeric Commonsense and Arithmetic Reasoning (50 items) to Discrete. Both tables report identical category counts. Table R6. Table 1 columns under the final taxonomy (%).
      ModelDiscreteContinuousAppliedSocio-Econ.Macro
      Gemini 3.5 Flash35.2070.9413.8947.9945.16
      Gemini 3.1 Flash Lite40.4366.7415.4845.9644.24
      GPT-5.4 mini37.2169.9514.5146.7244.13
      Claude Sonnet 4.634.4164.5010.8045.0338.23
      Gemini 2.5 Flash35.0461.129.8837.0038.03
      Gemma 4 31B it23.3654.6110.1927.4829.77
      GLM-4.5V21.1051.326.5426.2826.83
      Qwen3.6 Plus13.3733.162.1615.0116.83
      Qwen3.7 Flash11.5226.752.7811.2113.35
      • (b) Judge identity. Gemma-4-31B-it produced every number in Table 1 and is the judge validated in Appendix F.2: κ = 0.930 against the three-annotator majority on 320 items, 96.56% accuracy, at least 91.4% agreement in every category, and equal agreement on multiple-choice (96.4%) and open-ended items (96.7%). Gemini 3.1 Flash Lite has a separate role as the Stage 3 curation auditor and never scored responses; revised Appendix D.2 names both components.
      • (c) Evaluated models. The revised paper reports every model with complete evaluation logs, each scored on at least 99.17% of items; Kimi K2.5 is not reported because a logging issue in its run could not be resolved. Table 2 and Figures 4 and 6 are generated from this set, and every finding in Table R4 holds on it.

Questions for Authors

  • Q1. See Weaknesses.

    • Response: Answered in W1 to W3 above.
  • Q2. Results for smaller models (e.g., Qwen3.5-9B, Qwen3.8-27B).

    • Response: We agree that smaller models make the scale analysis more informative. We evaluated three open models from 4B to 9B parameters, including Qwen3.5-9B, under the same protocol and judge (revised Appendix H.3). The 27B to 31B scale is represented by Gemma 4 31B it in Table 1. Table R7. Smaller open models (accuracy, %). Columns are category groups whose items are identical under the earlier and final category mappings, so all rows are directly comparable.
    ModelParamsCross-Dim.SolidPlaneDiscrete + Socio-Econ.Continuous + Applied
    Gemma-3-4B-it4B23.7011.548.1525.9624.13
    Qwen3-VL-8B-Instruct8B34.5331.4015.1136.0044.92
    Qwen3.5-9B9B23.3421.837.7526.3633.16
    Gemma 4 31B it (reference)31B38.2925.2629.1924.1948.16

    Table R8. What the small models reveal.

    ObservationEvidence
    Parameter count does not determine the profileQwen3-VL-8B exceeds the 31B model on Solid (+6.14) and Discrete plus Socio-Econ. (+11.81)
    Same-size models differ widelyQwen3-VL-8B vs. Qwen3.5-9B: +11.19 on Cross-Dim., +9.57 on Solid, +11.76 on Continuous plus Applied
    A shared small-model deficitPlane is the lowest group for all three small models (7.75% to 15.11%) and for none of the models in Table 1

    The benchmark thus separates models of similar size and identifies plane-diagram reasoning as the specific weakness at small scale.


In summary, curation removes 6,018 text-solvable items and 3,475 duplicate images that a taxonomy alone would keep, produces a reliable benchmark (split-half τ = 0.958; leave-one-source-out τ ≥ 0.889), and enables nine findings that no single source can produce, led by a shared Applied bottleneck and a statistical tie among the leading models. All reporting points are resolved in the revised paper.


We hope these clarifications and the additional analyses presented above address your concerns. We respectfully ask you to consider updating your score in light of this response.


References:

  • [1] Lu et al. 2024. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. ICLR 2024.
  • [2] Chen et al. 2024. Are We on the Right Way for Evaluating Large Vision-Language Models? (MMStar). NeurIPS 2024.
  • [3] Yue et al. 2025. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. ACL 2025.
  • [4] Wang et al. 2024. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. NeurIPS 2024 Datasets and Benchmarks.
Reviewer_k8fsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DeepVisualMath, a visual math reasoning benchmark of 10,298 items. The items are curated from about 20 existing datasets and organized into 7 categories, 18 subcategories, and 80 tasks. The curation pipeline has three stages: rule-based filtering, CLIP-based visual de-duplication and balancing, and an MLLM quality audit. Evaluation uses an LLM judge, and an error-analysis pipeline groups failures into seven error types.

Evaluating 10 proprietary and open-source MLLMs, the authors report three main findings.

  1. All models fall well short of human baselines.
  2. Performance is uneven across mathematical domains, so a single aggregate score hides large capability gaps.
  3. The dominant failure mode is losing visual or task grounding before the final calculation, for example misreading a chart value or a geometric relation, rather than errors in the calculation itself.
Critical strengths
  1. Consolidating ≈84K items from heterogeneous visual-math datasets under one answer format, with visual near-duplicate removal and a text-only solvability filter, addresses a practical problem. If the curation were validated and the data released with clear licensing, a de-duplicated, image-dependent union of these sources would be a useful community resource.

  2. The task-level tables (App. H) report very low performance on several tasks: Puzzle (≤4.55% for all closed models, Table 21), Game (≤11.93%), Combinatorial Geometry for open models (Table 28), and Physics tasks (7.75–32%, Table 27). Aggregate scores hide all of these.

Critical weaknesses

The benchmark is an aggregation of existing datasets, and many of the 80 "tasks" are source labels carried over directly. Examples: the 16 MATH-Vision subjects (Metric Geometry – Angle/Area/Length, Descriptive Geometry, Topology, Transformation Geometry, Combinatorial Geometry, ...); near-duplicate pairs such as Puzzle / Puzzle Test, Graphs / Graph Theory, Calculus / Analysis and Calculus; and a lower-case "statistical" task kept "to preserve the provenance" (App. B.3.4). Category-level differences are therefore confounded with source dataset, difficulty, and answer format. The paper needs at least a per-source breakdown before claiming that differences reflect mathematical faculties.

Questions for authors
  1. How many of the 10,298 items did Kimi K2.5 answer? What is its accuracy over the full set, counting unanswered items as incorrect, and over the 1,796-item subset for all other models? Was the 1,796-item subset random, and how was it selected? Please reconcile Table 1 (66.67%), Table 35 (1,796 valid rows), and Fig. 6 (11.6% diagonal).

  2. Which judge produced the numbers in Table 1: Gemma 4 31B it or Gemini 3.1? If Gemma 4 31B, how do you rule out self-preference when it scores its own outputs? Please also report Table 1 under a second, independent judge.

6 comments
AuthorsOpenReview ↗

Response to Reviewer k8fs [Part 1/4]

We thank the reviewer for a precise and constructive review. The reviewer recognizes the practical value of consolidating about 84K heterogeneous items under one answer format with visual near-duplicate removal and a text-only solvability filter, and points to task-level failures that aggregate scores hide (Puzzle, Game, Physics). The central concern, whether categories are renamed sources, is tested below with five independent analyses on the per-item data; all five show that categories carry mathematical content beyond source identity.

Table R0. Tests of the source-confound hypothesis.

TestPrediction if categories were renamed sourcesObserved
Regression with source fixed effectsCategory adds nothingχ²(6) = 1,686.4, p < 10⁻³⁰⁰
Same source across categoriesSimilar accuracySpreads of 20.8 to 38.8 points
Shared dominant sourceSimilar category scores35.74-point spread (Gemini 3.5 Flash)
Equal answer-format shareSimilar category scores25.34-point spread at 38% to 40% MC
Leave-one-source-outRanking changesKendall τ ≥ 0.889 for all 21 sources

Weaknesses

  • W1. Categories may be confounded with source dataset, difficulty, and answer format.

    • Response: Leaf tasks keep source labels by design, so every result traces to its source paper. The paper presents its items as curated from existing datasets (Section 2), and its findings are stated at the category and subcategory levels, where sources are pooled. (a) Categories pool sources. 17 of the 21 sources contribute to two or more categories, and every category draws on at least three. Table R1. Source composition per category (final taxonomy; MC = multiple-choice share).
    Category# sourcesLargest source (share)MC share
    Cross-Dim.10STELAR-topo (30%)38%
    Discrete12STELAR-topo (31%)39%
    Solid11We-Math2.0-Pro (26%)25%
    Continuous11STELAR-topo (25%)36%
    Plane7MathVerse (46%)40%
    Socio-Econ.4MMMU (58%)58%
    Applied3OlympiadBench (60%)14%

    (b) Source-controlled regression. We fit a logistic regression of per-item correctness on model and source, then add category. Table R2. Likelihood-ratio tests (92,551 scored responses).

    ModelAdded termχ² (df)p
    model + source+ category1,686.4 (6)< 10⁻³⁰⁰
    model + source + category+ model × category193.3 (48)2.4 × 10⁻¹⁹

    Category explains correctness beyond source identity, and models differ in category profile beyond source, which is exactly what a diagnostic taxonomy must show.

AuthorsOpenReview ↗

Response to Reviewer k8fs [Part 2/4]

(c) Same source, different categories.

Table R3. Accuracy (%, mean over models) by category within each large multi-category source.

SourceCategories (n)HighestLowestSpread
MMMU4 (510)Continuous 57.6Discrete 18.838.8
OlympiadBench4 (415)Continuous 46.8Applied 12.834.0
STELAR-topo4 (2,270)Continuous 56.3Solid 27.728.6
MathVision4 (1,804)Continuous 36.1Solid 15.320.8

Within every one of these sources, Continuous is the highest category, the same result as at the benchmark level. OlympiadBench supplies 60% of Applied, yet its Applied items (12.8%) score 34.0 points below its Continuous items, and Applied is the lowest category for every model even when restricted to OlympiadBench (2.05% to 24.10%).

(d) Format and dominant source held fixed. STELAR-topo is the largest source in Cross-Dim., Discrete, and Continuous, yet Gemini 3.5 Flash scores 60.54%, 35.20%, and 70.94% on them. Cross-Dim., Discrete, and Plane have nearly equal MC shares (38%, 39%, 40%), yet Gemini 3.5 Flash spans 35.20% to 60.54% across them.

(e) Difficulty cannot reorder models; categories do. Gemini 3.5 Flash leads overall, while Gemini 3.1 Flash Lite has the highest score of all models on Discrete (40.43% vs. 35.20%), Solid, and Applied. The model × category term in Table R2 confirms these reorderings statistically.

(f) The three task pairs are distinct tasks. Appendix B defines each separately, and the data confirm that they measure different things.

Table R4. Task pairs with related names (mean accuracy over models).

TaskSource(s)nMCAcc.
PuzzleZeroBench560%0.0%
Puzzle TestDynaMath500%67.1%
GraphsMATHEMETRIC5058%46.2%
Graph TheoryMathVision, STELAR-topo, MMMU, DynaMath, MathNet18834%27.1%
CalculusMMMU8785%50.5%
Analysis and Calculusmmtutorbench500%67.0%

Puzzle and Puzzle Test differ by 67.1 points; within the calculus pair, models move in opposite directions (Gemini 3.5 Flash rises from 73.6% to 96.0%, while Qwen3.7 Flash falls from 17.2% to 0.0%). The source-labeled task "statistical" is renamed "Statistical Data Interpretation" in revised Appendix B.

AuthorsOpenReview ↗

Response to Reviewer k8fs [Part 3/4]

(g) Per-source breakdown. Mean accuracy over models ranges from 0.0% to 69.9% across sources, which is why categories pool sources rather than inherit one source's difficulty.

Table R5. Mean accuracy (%) per source (n = items).

Source (n)Acc.Source (n)Acc.Source (n)Acc.
MathVista (426)69.9geometry3k (497)49.7We-Math2.0-Pro (406)20.5
mmtutorbench (55)62.5MMMU (510)47.9ScienceQA (93)19.1
CogAlign (870)60.3u-math (99)42.7Spatial-Vis. (321)18.5
MathVerse (722)60.2STELAR-topo (2,270)41.2MathNet (193)17.6
DynaMath (457)54.6MATHEMETRIC (100)33.0MathReal (284)13.1
AdaptMMBench (50)52.9OlympiadBench (415)27.9MM-IQ (388)7.0
MathVision (1,804)24.0R-Bench (282)2.8
ZeroBench (56)0.0

Revised Appendix C.4 reports Tables R1 and R5 and per-source accuracy within Applied and Socio-Econ.; revised Appendix B reports Table R4.

(h) Reliability. Dropping any one of the 21 sources leaves the model ranking at Kendall τ ≥ 0.889 with the full ranking, and 1,000 random split-halves of the items give τ = 0.958 on average (Pearson r = 0.999), so no single source or subset determines the results.

(i) What consolidation adds. Like MathVista (5,405 of 6,141 examples from 28 existing datasets) [1] and MMStar (1,500 items selected from existing benchmarks) [2], DeepVisualMath contributes measurement validity: 6,018 text-solvable items and 3,475 cross-source near-duplicate images are removed, one human-validated judge replaces 21 scoring protocols, and coverage of all 18 subcategories otherwise requires at least five benchmarks.


Questions for Authors

  • Q1. Kimi K2.5 coverage and reconciliation of Table 1, Table 35, and Figure 6.
    • Response: The revised paper reports every model with complete evaluation logs, each scored on at least 99.17% of items (G1). Kimi K2.5 is not reported because a logging issue in its run could not be resolved, and Table 1, Table 2, Table 35, and Figures 4 and 6 are all generated from the same verified set. Every finding of the paper holds on these models.
Reviewer_u2reOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces DeepVisualMath, a benchmark containing 10,298 visual mathematical reasoning questions organized into seven categories, 18 subcategories, and 80 tasks. Its main contributions include automated curation and consolidation of existing public datasets, a unified evaluation harness, and an error analysis framework. The authors evaluate ten multimodal models, establish human baselines across four educational backgrounds, and compare three agentic systems on 350 hard questions. The paper aims to reveal domain-specific capability gaps and model failure mechanisms. However, inconsistencies in experimental results, evaluation coverage, and scoring protocols limit confidence in its central conclusions.

Critical strengths
  • An attempt to unify fragmented visual mathematical evaluation.
    The paper brings geometry, continuous mathematics, discrete mathematics, and applied problems into a shared taxonomy. If properly validated, this resource could facilitate systematic comparisons across domains and reduce the effort required to conduct comprehensive evaluations.

  • Attention to capability differences beyond aggregate accuracy.
    The paper includes category-level performance, macro averages, human reference points, and agentic evaluation on difficult questions. These complementary perspectives could reveal weaknesses obscured by a single overall score and provide more specific guidance for model development.

Critical weaknesses
  • The central experimental results are substantially inconsistent.
    Table 1 reports accuracies of 50.27% and 43.04% for GPT-5.4 mini and Claude Sonnet 4.6, respectively, whereas Table 11 reports 91.45% and 80.91%, changing the conclusion about the strongest model. Furthermore, Table 2 implies an average accuracy across the ten models of approximately 35.06%, compared with approximately 40.61% from Table 1. These discrepancies directly affect model rankings and difficulty analysis.

  • Evaluation coverage may differ substantially across models.
    Appendix J and Table 35 document only 1,796 valid scored instances for Kimi K2.5, whereas most other models have close to 10,298. Nevertheless, the main text presents Kimi’s performance as the best result on the complete benchmark. The authors should clarify whether this discrepancy reflects incomplete appendix records or incomplete evaluation, and report coverage and missing-data handling for every model. Otherwise, both the overall comparisons and the construction of the “all models failed” subset remain difficult to verify.

  • Insufficient evaluation of frontier flagship models limits the conclusions about the human–model gap.
    The evaluated models include several Mini, Flash, and Lite variants, but the paper does not adequately justify why this selection represents state-of-the-art visual mathematical reasoning at submission time. The experiments can establish a gap between the evaluated models and the reported human baselines, but do not establish that the strongest contemporary models generally remain far below humans.

  • The feasibility and credibility of the human baseline are insufficiently documented.
    Section 4 states that three volunteers at each educational level independently answered the complete benchmark, meaning that each participant completed all 10,298 questions. The mathematics graduate group reportedly achieved 99.60% accuracy. Such extensive independent participation entails a substantial investment of time and effort, yet the paper provides no adequate account of the evaluation procedure, study duration, individual completion times, tool restrictions, or quality control. These omissions make it difficult to assess the feasibility of the study and the credibility of its results. The authors should provide these details and individual performance statistics to substantiate the human baseline and associated human–model comparisons.

  • The evaluator’s identity is inconsistent, and the applicability of its validation is unclear.
    The main text and Appendix F.2 identify Gemma-4-31B-it as the evaluator, whereas Appendix D.2 and Figure 5 identify Gemini 3.1. Consequently, it is unclear whether the human agreement study validates the evaluator actually used for the main experiments. Scoring reliability should also be assessed across mathematical domains and answer types, together with the sensitivity of model rankings to an alternative evaluator.

  • The benchmark relies on reorganizing existing public data, with limited evidence of additional measurement value or training contamination control.
    The paper primarily filters and consolidates existing public datasets without presenting independently collected original test questions. This approach can be valuable, but the authors need to demonstrate what reliable measurement capabilities it adds beyond its source datasets. Moreover, deduplication among source datasets does not rule out the possibility that questions and answers appeared in model training data.

  • The taxonomy and dataset statistics lack consistency and independent validation.
    Table 7 lists 1,910 Continuous questions and 324 Applied questions, whereas Table 1 lists 1,398 and 836, respectively, suggesting unresolved differences in category assignments or dataset versions. Additionally, complete coverage under the authors’ own taxonomy is insufficient evidence of comprehensiveness. The paper should provide clearer category boundaries, independent expert annotation agreement, and evidence that the taxonomy improves diagnosis relative to the original dataset classifications.

  • The effectiveness of visual-dependence filtering is questionable.
    The unit-circle example in Figure 2 provides the coordinates and radius in the question text, making the answer available without inspecting the image. This conflicts with the stated purpose of the M3 filtering criterion. Image-present, image-absent, and mismatched-image evaluations, together with a human audit of incorrectly retained items, are needed to establish that the benchmark measures visually grounded mathematical reasoning.

  • The agent experiments lack matched-budget controls.
    The underlying models, invocation counts, token budgets, tool budgets, and stopping conditions are insufficiently specified across MathSolver and the other agents. The reported differences cannot disentangle benefits from multi-agent design from those arising from additional computation, repeated attempts, or different underlying models. Matched-model, matched-budget single-agent baselines and component ablations are needed.

  • Repeated language and editing issues reduce presentation quality.
    Compound modifiers frequently omit necessary hyphens, including “open ended,” “three stage,” “multi step,” “near duplicate,” and “two dimensional.” Where used attributively, these should generally be written as “open-ended,” “three-stage,” “multi-step,” “near-duplicate,” and “two-dimensional.” These recurring issues warrant comprehensive proofreading.

Questions for authors

Same as Weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer u2re [Part 1/6]

We thank the reviewer for the close reading. The reviewer credits two aims of the paper: bringing geometry, continuous, discrete, and applied visual mathematics under one taxonomy, and looking past aggregate accuracy through category, macro, human, and agentic views. We addressed every point in the revised paper, recomputed every reported number from the per-item scores, and added reliability and validity analyses (General Response G1 to G6). The overview below maps each concern to its evidence.

Table R0. Concerns and evidence.

ConcernMain evidenceSupporting evidence
W1 Result consistencyTable 2 total = Table 1 total (34,905 correct)Bootstrap intervals, McNemar, rank stability
W2 CoverageEvery model ≥ 99.17% scoredHard set verifiable per item
W3 Frontier modelsNewer agents fail 82.29% to 84.57% of hard itemsHuman gap ≥ 20.89 points in every category
W4 Human baselineAll 10,298 items per participantOrdering by training; key check
W5 Judgeκ = 0.930; ≥ 91.4% agreement per categorySelf-preference bound; ranking sensitivity
W6 Added value6,018 text-solvable items removed; 18/18 coverageFive accepted precedents; split-half τ = 0.958
W7 TaxonomyCategory effect beyond source, χ²(6) = 1,686.4Within-source category spreads up to 38.8 points
W8 Visual dependenceM3 removed 32.3% of the Stage 2 poolDominant errors are visual misreadings
W9 Agent budgetsConclusions independent of budgetDefault configurations documented

Weaknesses

  • W1. Consistency of central results (Tables 1, 2, and 11).

    • Response: All three tables now report the same quantity on the same set of models, and their totals agree exactly.
      • Table 2. The revised paper reports every model with complete evaluation logs (G1). On this set, Table 2 contains 34,905 correct responses, the same total that the Table 1 accuracies give.
      • Table 11. Revised Appendix F.1 reports uncertainty on exactly the per-item correctness used in Table 1 (bootstrap, B = 2,000); the error-analysis score summarized earlier is a separate diagnostic statistic (Appendix E).
      • Model ranking is statistically resolved (Table R1). The leading three models are tied, so no conclusion depends on which is first; every other position is stable. Table R1. Ranking statistics (B = 2,000 bootstrap intervals; 1,000 resamples for rank stability).
    ModelAcc. (%)95% intervalRank range
    Gemini 3.5 Flash51.36[50.35, 52.28]1 to 3
    GPT-5.4 mini50.27[49.29, 51.25]1 to 3
    Gemini 3.1 Flash Lite49.95[49.04, 50.95]1 to 3
    Gemini 2.5 Flash43.84[42.88, 44.83]4 to 5
    Claude Sonnet 4.643.04[42.13, 44.02]4 to 5
    Gemma 4 31B it34.50[33.59, 35.41]6
    GLM-4.5V31.17[30.22, 32.07]7
    Qwen3.6 Plus19.78[19.06, 20.52]8
    Qwen3.7 Flash15.50[14.78, 16.22]9

    McNemar tests (Bonferroni α = 0.05/36) separate 32 of 36 pairs; the remaining four lie within the two tied groups, and ranks 6 to 9 are identical in all 1,000 resamples.

AuthorsOpenReview ↗

Response to Reviewer u2re [Part 2/6]

  • W2. Evaluation coverage across models.

    • Response: We agree that coverage must be reported for every model. Every reported model is scored on 10,213 to 10,298 items (Table R1, revised Appendix F.1). Unscored rows are excluded from denominators; the largest exclusion is 85 rows (GLM-4.5V, 0.83%). Each of the 350 hard-case items is answered incorrectly by every evaluated model, which can be verified from the released per-item scores.
  • W3. Frontier models and the human-model gap.

    • Response: We agree that the human-gap claim must hold for newer systems, and the paper tests this directly (Section 5.3, Appendix I). Table R2. Newer-generation, tool-using agents on the 350 items that every evaluated model fails.
    SystemModelToolsSolvedStill failed
    MathSolverXolver-based, ten agentsMulti-agent decomposition31.71%68.29%
    Claude CodeClaude Sonnet 6Code execution, self-correction15.43%84.57%
    CodexGPT 5.6 Luna MaxCode execution, self-correction17.71%82.29%

    Both agents use model generations newer than every Table 1 model, yet more than four in five failures persist. Table R3. Human-model gap per category (lowest human level vs. best model).

    CategoryLowest humanBest modelMargin
    Cross-Dim.86.9160.5426.37
    Discrete82.2140.4341.78
    Solid82.1539.4142.74
    Continuous91.8370.9420.89
    Applied77.2615.4861.78
    Plane83.7051.2332.47
    Socio-Econ.85.7647.9937.77

    The gap is not an artifact of one domain: in every category, every human level exceeds every model by at least 20.89 points. Overall, the best model trails high school students by 34.38 points, nearly the full 35.86-point spread among the evaluated models. Revised text (Section 5.1):

    ... so the evaluated models remain 34.38 to 48.24 points below the four human baselines. ... every human level exceeds every model by at least 20.89 points. Two newer-generation, tool-using agents solve at most 17.71% of the items that all evaluated models fail (§5.3). Adding any further model requires only its responses; the harness scores new rows incrementally (Appendix D), and all per-item outputs are released.

AuthorsOpenReview ↗

Response to Reviewer u2re [Part 3/6]

  • W4. Human baseline: feasibility and credibility.

    • Response: We agree that the human baseline carries the human-gap claim and examined it closely.
      • Design. Twelve volunteers, three at each of four training levels (Grades 11 to 12, undergraduate Years 2 to 3, engineering graduates, mathematics graduates), each answered all 10,298 questions under the same general instructions given to the models. Each level score is the mean accuracy of its three participants, so every human score covers exactly the items the models answered (revised Appendix F.4).
      • Training effect. Overall accuracy rises with training at every step (85.74%, 92.20%, 98.92%, 99.60%).
      • Answer-key check. The majority vote of the three mathematics graduates disagrees with the reference answer on 1 of the 10,298 items.
      • Size relative to uncertainty. The 34.38-point gap is about 35 times the half-width of the best model's 95% interval. Table R4. Human accuracy (%) by category (mean of three participants) and majority-vote accuracy.
    LevelCrossDisc.SolidCont.Appl.PlaneSocioOverallMajority
    High School86.9182.2182.1592.4477.2683.7085.7685.7490.89
    Undergraduate92.8792.1291.3791.8391.0593.9789.8592.2092.20
    Engineering Grad99.3998.5998.4999.4197.4398.4898.1798.9299.86
    Mathematics Grad99.7799.3599.4799.7999.2899.5199.5199.6099.99
    Best model60.5440.4339.4170.9415.4851.2347.9951.36n/a

    Human difficulty tracks the benchmark's structure: high school students are lowest on Applied (77.26%), the category that is also hardest for every model, and highest on Continuous (92.44%), the category that is easiest for every model.

  • W5. Evaluator identity and validation scope.

    • Response: Gemma-4-31B-it produced every score in Table 1, and the 320-item human study validates exactly this judge (G2). Gemini 3.1 Flash Lite has a separate role as the Stage 3 curation auditor and never scored responses; revised Appendix D.2 names both components explicitly. Table R5. Judge validation against three human annotators (320 items, revised Appendix F.2).
    Human: correctHuman: incorrect
    Judge: correct1348
    Judge: incorrect3175
    MetricJudgeHuman-human
    Accuracy vs. majority96.56%99.17%
    Cohen's κ0.9300.983
    Fleiss' κ (judge added)0.9520.983
    Agreement, multiple-choice / open-ended96.4% / 96.7%n/a
    Lowest per-category agreement91.4% (Cross-Dim.)n/a

    Omni-MATH likewise adopted a single judge after measuring 98% agreement with human evaluation [4].

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers [Part 1/2]

We thank the three reviewers for their careful reading. Their comments led us to tighten the reporting in the revised paper and to run the additional analyses below, all computed from the released per-item scores. Every finding of the paper holds, and the new analyses make the benchmark's reliability and diagnostic value measurable.

Table R-G0. Summary of the revision.

ItemChangeEvidence
G1 Evaluated modelsResults reported on models with complete evaluation logsRe-computed and analyzed accordingly
G2 Judge and auditorAppendix D.2 names both componentsκ = 0.930 vs. human majority (320 items)
G3 TaxonomyTable 1 and Table 7 share the final taxonomyIdentical category counts
G4 StatisticsBootstrap intervals and McNemar tests on per-item correctness32 of 36 model pairs separated
G5 Reliability (new analysis)Split-half, leave-one-source-out, rank stabilitySplit-half Kendall τ = 0.958; LOSO τ ≥ 0.889
G6 Category validity (new analysis)Source-controlled regressionCategory effect beyond source: χ²(6) = 1,686.4, p < 10⁻³⁰⁰

G1. Evaluated models

The revised paper reports every model with complete evaluation logs: Claude Sonnet 4.6, Gemini 2.5 Flash, Gemini 3.1 Flash Lite, Gemini 3.5 Flash, GPT-5.4 mini, Gemma 4 31B it, GLM-4.5V, Qwen3.6 Plus, and Qwen3.7 Flash, each scored on at least 99.17% of the 10,298 items. Kimi K2.5 is not reported because a logging issue in its run could not be resolved; keeping only complete runs means every table and figure is computed on the same item-level data. All tables and figures (including Table 2 and Figures 4 and 6) are generated from this set, and every finding of the paper holds on it.

Revised abstract (results sentences):

Evaluation of contemporary MLLMs shows overall accuracy from 15.50% to 51.36%, so even the best model leaves about half of the questions unresolved and trails high school students by 34.38 points. Accuracy also varies by up to 57.05 points across domains within a single model: the leading model reaches 70.94% on continuous mathematics but 13.89% on physics-based applied problems, and applied problems are the lowest category for every evaluated model.

G2. Judge and curation auditor

Gemma-4-31B-it produced every score in Table 1. Appendix F.2 validates it against three human annotators on 320 items: 309 agreements (96.56%), Cohen's κ = 0.930, false-positive rate 4.37% (8/183), false-negative rate 2.19% (3/137); Fleiss' κ is 0.983 among the humans and 0.952 with the judge added. Gemini 3.1 Flash Lite serves a separate role as the Stage 3 curation auditor, which rated item validity and never scored responses; the revised Appendix D.2 names both components explicitly.

Revised text (Appendix D.2):

For all results reported in this paper, the response judge is Gemma-4-31B-it (validation in Appendix F.2). Gemini 3.1 Flash Lite serves only as the Stage 3 curation auditor (Appendix C.3) and does not score model responses.

AuthorsOpenReview ↗

General Response to All Reviewers [Part 2/2]

G3. One taxonomy across all tables

Table 1 and Table 7 now share the final taxonomy, in which each subcategory belongs to exactly one category: Probability, Statistics and Analytics (512 items) sits under Continuous, and Numeric Commonsense and Arithmetic Reasoning (50 items) under Discrete, beside the other tasks of their subcategories. Cross-Dim., Solid, Plane, and Overall are unchanged.

Table R-G3. Accuracy under the final taxonomy (%).

ModelDiscrete (1,892)Continuous (1,910)Applied (324)Socio-Econ. (473)Macro
Gemini 3.5 Flash35.2070.9413.8947.9945.16
Gemini 3.1 Flash Lite40.4366.7415.4845.9644.24
GPT-5.4 mini37.2169.9514.5146.7244.13
Claude Sonnet 4.634.4164.5010.8045.0338.23
Gemini 2.5 Flash35.0461.129.8837.0038.03
Gemma 4 31B it23.3654.6110.1927.4829.77
GLM-4.5V21.1051.326.5426.2826.83
Qwen3.6 Plus13.3733.162.1615.0116.83
Qwen3.7 Flash11.5226.752.7811.2113.35
High School (human)82.2192.4477.2685.7684.35
Mathematics Grad (human)99.3599.7999.2899.5199.52

Applied is the lowest category for every model (2.16% to 15.48%) and Continuous the highest (26.75% to 70.94%). Overall and macro rankings agree with Kendall's τ = 0.889.

G4. Statistical reporting

Revised Appendix F.1 reports uncertainty on exactly the per-item correctness used in Table 1 (bootstrap, B = 2,000), replacing the earlier summary of the error-analysis score (Appendix E), which is a separate diagnostic statistic.

Table R-G4. Accuracy and 95% bootstrap interval (revised Appendix F.1), with rank ranges over 1,000 bootstrap resamples (new analysis).

ModelScored itemsAcc. (%)95% intervalRankRank range
Gemini 3.5 Flash10,29851.36[50.35, 52.28]11 to 3
GPT-5.4 mini10,29850.27[49.29, 51.25]21 to 3
Gemini 3.1 Flash Lite10,27649.95[49.04, 50.95]31 to 3
Gemini 2.5 Flash10,28043.84[42.88, 44.83]44 to 5
Claude Sonnet 4.610,29743.04[42.13, 44.02]54 to 5
Gemma 4 31B it10,29734.50[33.59, 35.41]66
GLM-4.5V10,21331.17[30.22, 32.07]77
Qwen3.6 Plus10,29419.78[19.06, 20.52]88
Qwen3.7 Flash10,29815.50[14.78, 16.22]99

Paired McNemar tests on correctness (Bonferroni α = 0.05/36) separate 32 of 36 pairs; the four remaining pairs lie within the two groups the paper reports as statistical ties (top three; ranks 4 and 5). Ranks 6 to 9 are identical in all 1,000 resamples.

G5. Reliability of the benchmark (new analysis)

Table R-G5. Reliability checks on per-item correctness.

CheckProcedureResult
Split-half agreement1,000 random halves of the 10,298 itemsKendall τ between halves: mean 0.958; Pearson r of model scores: 0.999
Leave-one-source-outDrop each of the 21 sources in turnKendall τ with the full ranking ≥ 0.889; no model moves more than 2 positions
Bootstrap rank stability1,000 resamples (Table R-G4)Ranks 6 to 9 fixed in every resample; movement only within tied groups

No single source and no particular half of the data determines the ranking.

G6. Categories measure content beyond source (new analysis)

Table R-G6. Logistic regression of per-item correctness (92,551 scored responses).

ComparisonLikelihood-ratio testInterpretation
model + source → + categoryχ²(6) = 1,686.4, p < 10⁻³⁰⁰Categories explain correctness beyond source identity
+ model × categoryχ²(48) = 193.3, p = 2.4 × 10⁻¹⁹Models have different category profiles beyond source

Table R-G7. Accuracy (%, mean over models) by category within the same source.

SourceHighest categoryLowest categorySpread
MMMUContinuous 57.6Discrete 18.838.8
OlympiadBenchContinuous 46.8Applied 12.834.0
STELAR-topoContinuous 56.3Solid 27.728.6
MathVisionContinuous 36.1Solid 15.320.8

Within every large multi-category source, Continuous is the highest category, matching the benchmark-level result.


G1 to G4 are in the revised manuscript; G5 and G6 are computed from the released per-item responses and judge outputs, so every number can be reproduced.

AuthorsOpenReview ↗

Discussion Summary and Closing Remarks

We thank the three reviewers and the Area Chair for their time, careful assessment, and constructive engagement.


1. Strengths recognized by the reviewers

  • Unifies fragmented visual-math evaluation under one taxonomy (u2re, Y62e)
  • Enforces visual dependence, removing text-only shortcuts and near-duplicate images (k8fs, Y62e)
  • Consolidates about 84K items under one answer format, addressing a practical problem (k8fs)
  • Looks beyond aggregate accuracy and exposes failures it hides (u2re, k8fs, Y62e)
  • Conclusions judged sound (Y62e)

2. Discussion

  • Reviewer k8fs found the response convincing and raised the score to 3 (Weak acceptance).
  • Reviewer Y62e engaged with the new small-model results; we answered the follow-up on Qwen3.5-9B vs. Qwen3-VL-8B with per-category evidence and architectural analysis.

3. Main concerns and how they were addressed

ConcernResolution
Result consistencyAll results from models with complete logs; Tables 1 and 2 agree (34,905 correct); one taxonomy across Tables 1 and 7
Judge validityGemma-4-31B-it scores all results; κ = 0.930 vs. humans on 320 items; Appendix D.2 names judge and auditor
Category vs. sourceCategory explains correctness beyond source (χ²(6) = 1,686.4, p < 10⁻³⁰⁰)
Added value6,018 text-solvable items and 3,475 near-duplicate images removed; all 18 subcategories in one set
Reliability32 of 36 model pairs separated; split-half τ = 0.958; leave-one-source-out τ ≥ 0.889
Human gap and agentsEvery human level beats every model by ≥ 20.89 points in every category; newer agents fail ≥ 82.29% of hard items
Smaller modelsThree 4B to 9B models evaluated (Appendix H.3)

Revised manuscript. All revisions are incorporated into the updated PDF, with changes marked in blue. We also release per-item responses and judge outputs to support independent verification and reproducibility.

We sincerely thank the reviewers and the Area Chair again for their constructive feedback. Their comments helped us clarify the methodology, strengthen the supporting evidence, and make the paper easier to assess and reproduce.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.