acceptodds
Under review as a conference paper at ICLR 2027

MathVis: Can AI Reason through Drawing? A Benchmark for Mathematical Figure Generation

Abstract

This paper introduces MathVis, a benchmark designed to evaluate LLMs' ability to visualize frontier mathematical problems and perform diagram-assisted reasoning. The benchmark consists of 512 tasks, each comprising a mathematical statement, a reference diagram, executable TikZ/SVG code, a Gold Intermediate Representation (IR), atomic rubrics, and error codes. Through rigorous evaluation, GPT-6-astra, Claude Fable 5, and Gemini 3.1 Pro achieve Pass@1 scores of merely31.25%,26.56%,and22.85%,respectively. We found that although the tasks require visualization, LLMs still reason primarily through formal languages and symbols, rarely relying on visual diagrams. Furthermore, while the solutions generated by LLMs may contain all the necessary elements, they are often visually confusing to humans. Ultimately, this benchmark aims to visualize mathematical problems and promote research on enabling LLMs to simplify their reasoning via visual diagrams.

Primary area:
datasets and benchmarks
Keywords:
Evaluation Benchmark, MathVis, Large Language Models, Mathematical Diagram Generation, Implicit Geometric Reasoning, TikZ/SVG

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_1yPbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces MathVis, a 512-task benchmark for generating executable mathematical figures across 27 mathematical sub-disciplines. The main idea is to combine TikZ/SVG generation with Gold IR and atomic constraints, so that technical and mathematical failures can be separated. I think the move beyond elementary geometry is a reasonable and genuinely new benchmark direction. My main concern is more basic: I do not trust the benchmark evaluation in its current form. The paper's own Rouché examples contain grading decisions that directly contradict the shown figures, while the paper does not report aggregate evaluator-reliability statistics. Since the evaluator is the main contribution, this is the main reason I lean reject.

Critical strengths
  • The benchmark targets a real gap. Most existing mathematical figure-generation benchmarks focus on elementary geometry or educational diagrams, while MathVis tries to cover more advanced structures from complex analysis, differential geometry, topology, etc.

  • The Gold IR + atomic-predicate design is potentially useful. Separating executability, mathematical constraints, and presentation could provide much more informative diagnostics than image similarity or a single pass/fail score.

Critical weaknesses
  • Correctness. I do not think the paper has established that its evaluator is reliable. In the Rouché example, fig 9 explicitly states that both functions have three zeros inside C, yet the paper marks this conclusion as missing. fig 8 also visibly contains the radius-two circle that the paper says is absent. These are not minor presentation issues. If the showcased grading decisions are already wrong, it is hard to know how much confidence to place in the benchmark-level results.

  • Scope/significance. The paper makes claims about how LLMs use visual versus symbolic reasoning, but the main experiment is essentially single-round text-to-code generation. It does not test whether models actually use generated diagrams for subsequent reasoning, or whether diagrams improve reasoning relative to a symbolic baseline. So I think the benchmark contribution is narrower than the paper currently claims.

Questions for authors
  • Have the authors independently re-audited all 512 tasks and their predicate-level judgments for errors like those in the Rouché example? If so, how many judgments changed?

  • Can the authors report actual inter-rater agreement or evaluator error rates for the hard semantic predicates? This seems necessary before the model score differences are interpretable.

  • Can the authors report mathematical hard-predicate accuracy separately from strict delivery? Otherwise it is difficult to tell whether the low scores mainly come from mathematical errors or presentation constraints.

  • Do the authors have an experiment where generated diagrams are actually used for a subsequent reasoning task? If not, I think the claims about visual versus symbolic reasoning should be substantially narrowed.

Reviewer_VaEcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a new benchmark for generating diagrams in higher level mathematical problems (research, undergraduate and graduate math). They curate a comprehensive benchmark consisting of 512 tasks, each task equipped with (statement, diagram, TikZ/SVG, Gold IR, rubrics, error codes). They do an evaluation of frontier LLMs on the task, and show that the SOTA is quite weak. They propose a framework for mathematical figures consisting of four well-thought out capabilities.

Critical strengths

S1. The dataset itself is a very good contribution to the field. It is well-designed, has a lot of items, each item is equipped with a lot of metadata to assist with the task, and has good coverage of advanced topics.

S2. The task itself is very important. While most benchmarks focus on elementary or high school level math problems, this work focuses on more advanced figure drawing.

Critical weaknesses

W1. The central weakness is that the evaluation protocol reliability is not established in the paper. For example, how often do annotators agree on scores? How often do automated judges agree with human scoring? The task is inherently subjective, but these things need to be reported in the paper. I would be happy to increase my score if the evaluation protocol is more thoroughly tested.

W2. There needs to be a clearer delineation when reporting the scores between correctness of the argument and clarity of presentation.

Questions for authors

Q1. Can you include agreement and evaluation statistics more comprehensively?

Q2. How much of the scores are related to mathematical accuracy vs. clarity of exposition?

Reviewer_BcqnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MathVis, a benchmark for evaluating mathematical figure generation using executable TikZ/SVG programs. It contains 512 tasks across 27 mathematical sub-disciplines and employs Gold IR and atomic constraints to assess mathematical correctness, technical validity, and visual quality. Experiments across 16 model configurations reveal substantial limitations in current models' figure-generation capabilities.

Critical strengths
  • The benchmark addresses an interesting and relatively underexplored problem, extending mathematical diagram generation beyond elementary geometry.

  • The combination of executable graphics, Gold IR, and atomic constraints provides a useful framework for distinguishing technical failures from mathematical errors.

  • The evaluation covers a diverse set of models and mathematical domains, offering useful insights into current limitations.

Critical weaknesses
  • The benchmark relies heavily on expert-defined predicates and automated validators, but does not report finalized inter-annotator agreement or quantitative validator accuracy. This makes the reliability of the reported scores difficult to assess.

  • Question-level evaluation logs are available for only 3 of the 16 model configurations, and several model-specific execution settings are not fully documented. This limits independent verification of the model comparisons.

  • The experiments primarily evaluate diagram generation rather than whether generated diagrams improve mathematical reasoning. The connection between benchmark performance and actual diagram-assisted reasoning remains unclear.

Questions for authors
  • Can the authors report inter-annotator agreement and validate the accuracy of automated mathematical predicates against independent expert judgments?

  • Can the complete question-level results and execution configurations for all evaluated models be released to support reproducibility?

  • Does better performance on MathVis translate into improved mathematical problem-solving or proof understanding when models use their generated diagrams?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.