- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DeepVisualMath, a benchmark for visual mathematical reasoning comprising 10,298 instances spanning 7 categories, 18 subcategories, and 80 tasks. This benchmark was constructed from existing datasets through a process involving deterministic filtering, visual deduplication and balancing, and multimodal quality review. Furthermore, the authors propose a framework for automated evaluation and error analysis; and evaluated the performance of ten multimodal models and humans, and compared three AI agent systems using a subset containing 350 challenging problems.
Critical strengths
Deepvisualmath covers a broad scope while explicitly focusing on the reliance on visual information. It integrates multiple mathematical domains, including continuous mathematics, discrete reasoning, and spatial geometry. The "visual dependency" filtering mechanism addresses a significant issue: problems involving images can sometimes be solved using only textual cues. Once validated, this carefully constructed dataset will serve as a valuable resource for evaluation.
This work aims to move beyond the limitations of focusing solely on overall accuracy. Through category-level evaluation, problem-level coverage analysis, and error analysis, it distinguishes between the various causes of failure. Experiments with intelligent agents demonstrate that certain failures to provide a direct answer can be rectified through alternative reasoning processes, offering insights for further analysis.
Critical weaknesses
- Regarding the value of the benchmark itself, the fact that DeepVisualMath was created by filtering existing datasets somewhat diminishes its contribution. Although the paper notes that existing benchmarks are fragmented, it fails to adequately demonstrate which critical evaluation issues would remain unresolved if a unified classification system and evaluation protocol were applied to existing resources.
- While the paper's conclusions are sound, they do not offer significant new insights.
- There are conflicting and inconsistent data reported in the paper. Discrepancies exist in the category statistics between Table 1 and Table 7; furthermore, the model used for scoring is labeled as Gemma-4-31B-it in Section 4 and Appendix F.2, but as Gemini 3.1 in Appendix D.2. The number of valid evaluation samples for Kimi is only 1,796. These inconsistencies undermine the reliability of the paper.
Questions for authors
1.see Weaknesses 2. The paper lacks evaluation results for certain smaller-scale models, such as Qwen3.5-9B and Qwen3.8-27B; it is recommended to include evaluation results for these models.
6 comments
Response to Reviewer Y62e [Part 2/3]
-
W2. The conclusions do not offer new insights.
- Response: We agree that the paper should state its new findings explicitly. Table R4 lists them; none is observable in any single source benchmark. Table R4. Findings and the evidence for each.
# Finding Evidence 1 Applied mathematics is the shared bottleneck Lowest category for every model (2.16% to 15.48%), below the next-lowest with p ≤ 6.4 × 10⁻⁶ per model; Continuous highest for every model 2 Domain gaps within one model are large Best-minus-worst category gap from 23.97 (Qwen3.7 Flash) to 57.05 points (Gemini 3.5 Flash) 3 Overall rank does not transfer across domains Gemini 3.1 Flash Lite, third overall, leads all models on Discrete, Solid, and Applied; model × category effect χ²(48) = 193.3, p = 2.4 × 10⁻¹⁹ 4 The leading models are statistically tied McNemar separates 32 of 36 pairs, none among the top three; top-three ranks interchange in bootstrap resamples while ranks 6 to 9 never change 5 Error diversity determines ensemble coverage Best pair solves 66.1% of items, 14.7 points above the best single model; all models together solve 79.75% 6 Failures arise before computation Dominant failure is loss of visual or task grounding: fluent algebra on a misread chart value, relation, or option (Section 5.4) 7 Tool use does not recover most hard failures Newer-generation agents (Claude Sonnet 6, GPT 5.6 Luna Max) with code execution solve 15.43% and 17.71% of items every model fails 8 The human gap exceeds the model spread Best model 34.38 points below high school, close to the 35.86-point model spread; every human level beats every model by ≥ 20.89 points in every category 9 Parameter count does not determine the profile Qwen3-VL-8B exceeds the 31B Gemma model on Solid and on Discrete plus Socio-Econ. (Q2) Table R5. Human difficulty mirrors model difficulty (accuracy %).
Applied Continuous High school (human) 77.26 (lowest) 92.44 (highest) Best model 15.48 (lowest) 70.94 (highest) Humans and models find the same category hardest and the same category easiest, which supports the taxonomy as a measure of mathematical content.
Response to Reviewer Y62e [Part 3/3]
- W3. Consistency of reported data.
-
Response: All three points are resolved in the revised paper (General Response G1 to G3).
- (a) Table 1 vs. Table 7. Table 1 and Table 7 now share the final taxonomy, in which each subcategory belongs to exactly one category: Probability, Statistics and Analytics (512 items) belongs to Continuous, and Numeric Commonsense and Arithmetic Reasoning (50 items) to Discrete. Both tables report identical category counts. Table R6. Table 1 columns under the final taxonomy (%).
Model Discrete Continuous Applied Socio-Econ. Macro Gemini 3.5 Flash 35.20 70.94 13.89 47.99 45.16 Gemini 3.1 Flash Lite 40.43 66.74 15.48 45.96 44.24 GPT-5.4 mini 37.21 69.95 14.51 46.72 44.13 Claude Sonnet 4.6 34.41 64.50 10.80 45.03 38.23 Gemini 2.5 Flash 35.04 61.12 9.88 37.00 38.03 Gemma 4 31B it 23.36 54.61 10.19 27.48 29.77 GLM-4.5V 21.10 51.32 6.54 26.28 26.83 Qwen3.6 Plus 13.37 33.16 2.16 15.01 16.83 Qwen3.7 Flash 11.52 26.75 2.78 11.21 13.35 - (b) Judge identity. Gemma-4-31B-it produced every number in Table 1 and is the judge validated in Appendix F.2: κ = 0.930 against the three-annotator majority on 320 items, 96.56% accuracy, at least 91.4% agreement in every category, and equal agreement on multiple-choice (96.4%) and open-ended items (96.7%). Gemini 3.1 Flash Lite has a separate role as the Stage 3 curation auditor and never scored responses; revised Appendix D.2 names both components.
- (c) Evaluated models. The revised paper reports every model with complete evaluation logs, each scored on at least 99.17% of items; Kimi K2.5 is not reported because a logging issue in its run could not be resolved. Table 2 and Figures 4 and 6 are generated from this set, and every finding in Table R4 holds on it.
-
Questions for Authors
-
Q1. See Weaknesses.
- Response: Answered in W1 to W3 above.
-
Q2. Results for smaller models (e.g., Qwen3.5-9B, Qwen3.8-27B).
- Response: We agree that smaller models make the scale analysis more informative. We evaluated three open models from 4B to 9B parameters, including Qwen3.5-9B, under the same protocol and judge (revised Appendix H.3). The 27B to 31B scale is represented by Gemma 4 31B it in Table 1. Table R7. Smaller open models (accuracy, %). Columns are category groups whose items are identical under the earlier and final category mappings, so all rows are directly comparable.
Model Params Cross-Dim. Solid Plane Discrete + Socio-Econ. Continuous + Applied Gemma-3-4B-it 4B 23.70 11.54 8.15 25.96 24.13 Qwen3-VL-8B-Instruct 8B 34.53 31.40 15.11 36.00 44.92 Qwen3.5-9B 9B 23.34 21.83 7.75 26.36 33.16 Gemma 4 31B it (reference) 31B 38.29 25.26 29.19 24.19 48.16 Table R8. What the small models reveal.
Observation Evidence Parameter count does not determine the profile Qwen3-VL-8B exceeds the 31B model on Solid (+6.14) and Discrete plus Socio-Econ. (+11.81) Same-size models differ widely Qwen3-VL-8B vs. Qwen3.5-9B: +11.19 on Cross-Dim., +9.57 on Solid, +11.76 on Continuous plus Applied A shared small-model deficit Plane is the lowest group for all three small models (7.75% to 15.11%) and for none of the models in Table 1 The benchmark thus separates models of similar size and identifies plane-diagram reasoning as the specific weakness at small scale.
In summary, curation removes 6,018 text-solvable items and 3,475 duplicate images that a taxonomy alone would keep, produces a reliable benchmark (split-half τ = 0.958; leave-one-source-out τ ≥ 0.889), and enables nine findings that no single source can produce, led by a shared Applied bottleneck and a statistical tie among the leading models. All reporting points are resolved in the revised paper.
We hope these clarifications and the additional analyses presented above address your concerns. We respectfully ask you to consider updating your score in light of this response.
References:
- [1] Lu et al. 2024. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. ICLR 2024.
- [2] Chen et al. 2024. Are We on the Right Way for Evaluating Large Vision-Language Models? (MMStar). NeurIPS 2024.
- [3] Yue et al. 2025. MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark. ACL 2025.
- [4] Wang et al. 2024. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. NeurIPS 2024 Datasets and Benchmarks.
Response to Reviewer Y62e [Part 1/3]
We thank the reviewer for a balanced review and for judging the paper's conclusions sound. The reviewer highlights the breadth of DeepVisualMath across continuous, discrete, and geometric domains, its explicit visual-dependency filter against text-only shortcuts, and its move beyond overall accuracy through category-level evaluation, item-level coverage, error analysis, and agents that recover some failures. We answer each question with quantitative evidence and add three smaller models as requested.
Weaknesses
W1. Value of a benchmark built from existing datasets.
Source-level reporting merges these differences into one number. Across all items, category explains correctness beyond source identity (logistic regression, χ²(6) = 1,686.4, p < 10⁻³⁰⁰). Table R3. Accepted benchmarks built by curating existing data.
Each was accepted because curation yields a more valid measurement than its sources.