- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces PICTUREPROOF, a benchmark for evaluating visual mathematical reasoning in vision-language models through “proofs without words.” Rather than asking models to extract values from diagrams and produce numerical answers, the benchmark requires models to explain how a given figure proves a stated theorem. The dataset contains 50 source proofs, represented in TikZ, and is expanded to 246 images using 100 proof-preserving and 96 proof-breaking variants. These controlled variants are intended to test whether models actually follow the argument encoded in the displayed figure rather than relying on a familiar theorem or memorized proof. Model responses are evaluated using problem-specific expert-written rubrics for proof correctness and visual grounding, together with an eloquence score, with grading performed at scale by an LLM judge. The authors evaluate nine closed- and open-weight VLMs on proof reconstruction and proof validation, and report that even strong models become substantially less reliable under controlled visual modifications, often accepting invalid constructions or rejecting valid ones. The paper further analyzes annotation agreement, categorizes common failure modes, and explores whether visual revisiting and attention to localized error regions are associated with better visual reasoning.
Critical strengths
-
The benchmark targets an important and underexplored aspect of visual mathematical reasoning. PICTUREPROOF moves beyond standard visual-math tasks where the figure mainly supplies numerical values or constraints, and instead requires models to reconstruct the mathematical argument encoded by a proof-without-words figure. This is consequential because it evaluates whether a VLM can connect visual structure to multi-step mathematical reasoning, rather than only solve a math problem that happens to contain an image.
-
The proof-preserving and proof-breaking variants provide a strong diagnostic for whether models actually use the displayed figure. By keeping the theorem fixed while systematically modifying the visual construction, the benchmark can distinguish explanations that are grounded in the current image from responses that rely on a familiar theorem or canonical proof. This controlled perturbation design is, in my view, the strongest methodological contribution of the paper, because it enables more informative failure analysis than performance on the original figures alone.
Critical weaknesses
The evidence does not sufficiently isolate memorization as the cause of the observed degradation. The proof-preserving and proof-breaking variants show that models often fail to faithfully follow the displayed construction, but performance drops under modified figures do not uniquely indicate memorization. They may also arise from limited robustness to visual perturbations, failures to recognize fine-grained geometric or symbolic changes, or difficulties in integrating visual information into subsequent reasoning. Recent studies have identified related bottlenecks in visual perception, fine-grained diagram grounding, and perception-to-reasoning integration for visual mathematical problems [1–3]. Thus, the current experiments establish brittleness to controlled visual changes more directly than memorization as the underlying mechanism.
[1] CHEN S, YUAN H, XU Y, et al. MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics. 2026.
[2] ZHANG S, CHEN A, SUN Y, et al. Primitive Vision: Improving Diagram Understanding in MLLMs[C]//Proceedings of the 42nd International Conference on Machine Learning. 2025: 74732–74755.
[3] CHEN S, XU Y, XIE J, et al. CogFlow: Bridging Perception and Reasoning through Knowledge Internalization for Visual Mathematical Problem Solving[C]//The Fourteenth International Conference on Learning Representations. 2026.
Questions for authors
How do the authors distinguish memorization of a canonical proof from failures in visual perception or visual-information integration? For example, do the authors have results from a theorem-only/no-image condition, or from an intermediate evaluation of whether the model correctly recognizes the modified visual relation before reasoning about proof validity? Such evidence would materially affect how strongly I interpret the paper’s memorization claim.
2 comments
Awaiting feedback on rebuttal
Dear reviewer,
Since we will not be able to address issues after the 18th, we look forward to know if our rebuttal adequately addresses all your concerns. We agree with your concerns about distinguishing memorization from results on the modified figures, and have done exploratory analysis of different failure modes. Thank you very much for your hard work, your feedback has improved the quality of our paper, and we hope to get to hear from you soon.
Best, Authors
Thank you for taking the time to review our work! We address your questions and concerns inline:
W1 & Q1. Distinguishing canonical-proof memorization from visual perception, perturbation robustness and information-integration failures through theorem-only baselines and recognition of modified visual relations
Thank you for honing in on this important nuance and for pointing us to these studies! We agree that performance on modified figures alone cannot tell us whether a model departs from the displayed argument due to memorization or for some other mix of reasons. Some of the other hypotheses we had discussed among ourselves on the topic were sycophancy and language dominance, and although the current manuscript does not capture this complexity well, we will definitely edit our final manuscript to communicate this. Your suggestions in your question give a useful starting point here. We will revise RQ2 to focus on whether models follow the visual construction when the theorem is held fixed, and discuss memorization alongside other failures of visual perception and integration.
The different model behaviors leading to this failure mode might be possible to tease apart through careful manipulations of the system prompt and ways that image proofs are presented. However, we believe this careful causal analysis would be a downstream project and is out of scope of our work, which succeeds in its stated goal of introducing a mathematical task that elegantly dissociates image-reasoning from language-based reasoning.
In any case, following your suggestions, we ran two exploratory probes with Grok 4.3 and GPT-6 Astra on five problems selected for clear, verifiable visual edits. We asked each model to identify the edited relation directly and, in a separate call, to prove the theorem without the image. We compared these answers with the models’ saved RQ2 responses.
Grok correctly identified the edited relation when asked directly in two of the three cases where it had accepted an invalid figure. It correctly read the reversed graph levels and counted three shaded strips, yet its RQ2 explanations relied on the original relations in both cases. Grok missed two of the five relations even under direct questioning. Astra identified all five and rejected all five invalid figures in RQ2. The two Grok examples show that a model can report the relevant visual relation when prompted directly and still give a proof explanation that contradicts it. Whether Grok noticed the relation during its earlier proof reasoning remains unknown.
In the theorem-only condition, both models used the auxiliary-function argument underlying the eπ>πe figure. On the other four problems, both used strategies different from the pictured proofs. The shared argument is compatible with either memorization or independent derivation.
In the revised manuscript, we will report these probes as case studies and discuss how they relate to MathFlow, Primitive Vision, and CogFlow [1–3], particularly their findings on visual perception, diagram grounding, and how models use visual evidence in mathematical reasoning. We will also deepen the discussion on the alternative mechanisms of sycophancy and language-dominance that could contribute, and how these can be studied further.