acceptodds
Under review as a conference paper at ICLR 2027

PictureProof: Evaluating Mathematical Reasoning in Vision-Language Models with Visual Proofs

Abstract

Human mathematical reasoning is profoundly visual at the level of intuition, discovery, and communication. However, existing visual math benchmarks evaluate on shallow calculation-based tasks rather than deeper modes of reasoning. To address this benchmarking gap, we introduce , a benchmark of visual mathematical reasoning based on translating visual math proofs into natural language. Because the grading for this task is qualitative, we implement a framework in which VLM outputs are graded at scale with an LLM-judge using human expert-crafted rubrics for each benchmark item. Our evaluations reveal dramatic failures in visual mathematical reasoning both on open-source as well as frontier VLMs. We further demonstrate that some of the failures are caused by a decay in attention to visual tokens during reasoning. We open-source our benchmark — specifically, the visual math proofs in TikZ code format, rubrics, and LLM-judge prompts — in order for the community to run further experiments on mathematical reasoning in VLMs.

Primary area:
datasets and benchmarks
Keywords:
Mathematical Reasoning, Vision Language Models, Visual Reasoning, Proofs

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XEFqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces PICTUREPROOF, a benchmark for evaluating visual mathematical reasoning in vision-language models through “proofs without words.” Rather than asking models to extract values from diagrams and produce numerical answers, the benchmark requires models to explain how a given figure proves a stated theorem. The dataset contains 50 source proofs, represented in TikZ, and is expanded to 246 images using 100 proof-preserving and 96 proof-breaking variants. These controlled variants are intended to test whether models actually follow the argument encoded in the displayed figure rather than relying on a familiar theorem or memorized proof. Model responses are evaluated using problem-specific expert-written rubrics for proof correctness and visual grounding, together with an eloquence score, with grading performed at scale by an LLM judge. The authors evaluate nine closed- and open-weight VLMs on proof reconstruction and proof validation, and report that even strong models become substantially less reliable under controlled visual modifications, often accepting invalid constructions or rejecting valid ones. The paper further analyzes annotation agreement, categorizes common failure modes, and explores whether visual revisiting and attention to localized error regions are associated with better visual reasoning.

Critical strengths
  1. The benchmark targets an important and underexplored aspect of visual mathematical reasoning. PICTUREPROOF moves beyond standard visual-math tasks where the figure mainly supplies numerical values or constraints, and instead requires models to reconstruct the mathematical argument encoded by a proof-without-words figure. This is consequential because it evaluates whether a VLM can connect visual structure to multi-step mathematical reasoning, rather than only solve a math problem that happens to contain an image.

  2. The proof-preserving and proof-breaking variants provide a strong diagnostic for whether models actually use the displayed figure. By keeping the theorem fixed while systematically modifying the visual construction, the benchmark can distinguish explanations that are grounded in the current image from responses that rely on a familiar theorem or canonical proof. This controlled perturbation design is, in my view, the strongest methodological contribution of the paper, because it enables more informative failure analysis than performance on the original figures alone.

Critical weaknesses

The evidence does not sufficiently isolate memorization as the cause of the observed degradation. The proof-preserving and proof-breaking variants show that models often fail to faithfully follow the displayed construction, but performance drops under modified figures do not uniquely indicate memorization. They may also arise from limited robustness to visual perturbations, failures to recognize fine-grained geometric or symbolic changes, or difficulties in integrating visual information into subsequent reasoning. Recent studies have identified related bottlenecks in visual perception, fine-grained diagram grounding, and perception-to-reasoning integration for visual mathematical problems [1–3]. Thus, the current experiments establish brittleness to controlled visual changes more directly than memorization as the underlying mechanism.

[1] CHEN S, YUAN H, XU Y, et al. MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics. 2026.

[2] ZHANG S, CHEN A, SUN Y, et al. Primitive Vision: Improving Diagram Understanding in MLLMs[C]//Proceedings of the 42nd International Conference on Machine Learning. 2025: 74732–74755.

[3] CHEN S, XU Y, XIE J, et al. CogFlow: Bridging Perception and Reasoning through Knowledge Internalization for Visual Mathematical Problem Solving[C]//The Fourteenth International Conference on Learning Representations. 2026.

Questions for authors

How do the authors distinguish memorization of a canonical proof from failures in visual perception or visual-information integration? For example, do the authors have results from a theorem-only/no-image condition, or from an intermediate evaluation of whether the model correctly recognizes the modified visual relation before reasoning about proof validity? Such evidence would materially affect how strongly I interpret the paper’s memorization claim.

2 comments

Thank you for taking the time to review our work! We address your questions and concerns inline:

W1 & Q1. Distinguishing canonical-proof memorization from visual perception, perturbation robustness and information-integration failures through theorem-only baselines and recognition of modified visual relations

Thank you for honing in on this important nuance and for pointing us to these studies! We agree that performance on modified figures alone cannot tell us whether a model departs from the displayed argument due to memorization or for some other mix of reasons. Some of the other hypotheses we had discussed among ourselves on the topic were sycophancy and language dominance, and although the current manuscript does not capture this complexity well, we will definitely edit our final manuscript to communicate this. Your suggestions in your question give a useful starting point here. We will revise RQ2 to focus on whether models follow the visual construction when the theorem is held fixed, and discuss memorization alongside other failures of visual perception and integration.

The different model behaviors leading to this failure mode might be possible to tease apart through careful manipulations of the system prompt and ways that image proofs are presented. However, we believe this careful causal analysis would be a downstream project and is out of scope of our work, which succeeds in its stated goal of introducing a mathematical task that elegantly dissociates image-reasoning from language-based reasoning.

In any case, following your suggestions, we ran two exploratory probes with Grok 4.3 and GPT-6 Astra on five problems selected for clear, verifiable visual edits. We asked each model to identify the edited relation directly and, in a separate call, to prove the theorem without the image. We compared these answers with the models’ saved RQ2 responses.

ModelDirect relation correctInvalid figures accepted in RQ2Correct direct answers among false acceptances
Grok 4.33/53/52/3
GPT-6 Astra5/50/5—

Grok correctly identified the edited relation when asked directly in two of the three cases where it had accepted an invalid figure. It correctly read the reversed graph levels and counted three shaded strips, yet its RQ2 explanations relied on the original relations in both cases. Grok missed two of the five relations even under direct questioning. Astra identified all five and rejected all five invalid figures in RQ2. The two Grok examples show that a model can report the relevant visual relation when prompted directly and still give a proof explanation that contradicts it. Whether Grok noticed the relation during its earlier proof reasoning remains unknown.

In the theorem-only condition, both models used the auxiliary-function argument underlying the eπ>πee^\pi > \pi^e figure. On the other four problems, both used strategies different from the pictured proofs. The shared argument is compatible with either memorization or independent derivation.

In the revised manuscript, we will report these probes as case studies and discuss how they relate to MathFlow, Primitive Vision, and CogFlow [1–3], particularly their findings on visual perception, diagram grounding, and how models use visual evidence in mathematical reasoning. We will also deepen the discussion on the alternative mechanisms of sycophancy and language-dominance that could contribute, and how these can be studied further.

AuthorsOpenReview ↗

Awaiting feedback on rebuttal

Dear reviewer,

Since we will not be able to address issues after the 18th, we look forward to know if our rebuttal adequately addresses all your concerns. We agree with your concerns about distinguishing memorization from results on the modified figures, and have done exploratory analysis of different failure modes. Thank you very much for your hard work, your feedback has improved the quality of our paper, and we hope to get to hear from you soon.

Best, Authors

Reviewer_F6DFOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PictureProof, a benchmark comprising 50 visual proofs, 100 proof-preserving variants, and 96 proof-breaking variants. It combines expert-written rubrics with an LLM judge to evaluate proof reconstruction, visual grounding, validity judgments, and flaw diagnosis. It further investigates models’ reliance on visual evidence and their failure modes.

Critical strengths
  1. The variants have clear mathematical significance. Valid variants preserve the relations required by the proof, while invalid variants break essential relations. This design can test both stability under transformations that preserve proof validity and sensitivity to changes in mathematically essential relations.

  2. Separating proof correctness from visual grounding is well motivated. A mathematically correct proof may not explain the displayed construction, while an accurate description of the diagram may still omit essential reasoning. Evaluating these dimensions separately helps distinguish these failure modes.

  3. The editable TikZ constructions have value as a research resource. Programmatic representations facilitate geometric modifications, variant generation, and defect localization. Fully releasing the constructions, generation scripts, and rubrics would support reproducibility and further research.

Critical weaknesses
  1. The related-work discussion contains inaccuracies, and the contribution needs more precise positioning. The characterization of existing benchmarks as computational tasks that do not evaluate reasoning processes is overly broad. UniGeo already includes 9,543 proving problems with annotated multistep proofs [1]; MathVerse includes stepwise reasoning evaluation [2]; and OlympiadBench includes proof problems and human evaluation of sampled proof responses [3]. Moreover, DynaMath’s variants do not invariably preserve the correct answer [4].

GSM8K-V [5] and SeePhys Pro [6] also conduct controlled modality diagnostics. The paper should clarify the additional value of reconstructing a specified visual proof and diagnosing its defects, rather than relying on broad contrasts with previous benchmarks.

  1. The automatic evaluation framework lacks sufficient independent validation. The authors acknowledge that agreement among human annotators does not validate the judge, and annotators could see one another’s scores. The reported results therefore do not establish independent agreement between human experts and the judge. The judge also receives CANDIDATE_LABEL, introducing an uncontrolled risk of bias associated with model identity. Furthermore, the number and final disposition of cases flagged for human review are insufficiently documented; the diagnosis scores differ by 16.6 and 14.3 percentage points for Opus 5 and GPT-5.5, respectively, depending on how these cases are handled.

  2. The comparison between RQ1 and RQ2 is confounded. These experiments change not only the images but also the task prompts, the candidates’ reasoning-effort parameter settings, and the judge’s reasoning-effort setting. RQ2 additionally assigns zero explanation credit when a model rejects a valid diagram. Consequently, the score difference combines changes in task requirements, validity judgments, and proof quality, and cannot be directly attributed to visual perturbations or memorization.

  3. The causal claim about attention decay lacks direct support. The abstract attributes some failures to declining attention to visual tokens during reasoning, but the paper does not directly demonstrate this temporal change. The localized attention associations in Table 6 do not establish causality, and the visual-revisiting interventions in Table 5 do not yield consistent, statistically supported improvements.

  4. The validity evaluation lacks necessary baselines and a complete analysis of both error types. The dataset contains 150 valid and 96 invalid diagrams. Given this composition, an always-YES baseline achieves approximately 61.0% precision, 100% recall, and 75.8% F1, despite failing to reject any invalid diagram. Constant-prediction baselines, full confusion matrices, class-specific recall, and balanced accuracy should be reported.

  5. The category counts in Table 7 do not match the reported totals. The listed counts sum to 71 for Opus 5, whereas the reported total is 74; for Grok 4.3, they sum to 174 rather than 175. This is inconsistent with the statement that every failure is assigned to exactly one category. The unaccounted-for cases and statistical denominators require clarification.

  6. The theorem statement in Figure 1 omits a necessary condition. The original result counts equilateral triangles [7], but “equilateral” is missing from the displayed statement. Using the original definition of (n) as the number of lattice points along each side, (n=3) gives six points forming (\binom63-3=17) nondegenerate triangles, whereas (\binom54=5) counts only the equilateral triangles. The authors should verify whether this omission also appears in the actual model inputs and rubrics.

[1] Jiaqi Chen et al. UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression. EMNLP, 2022.

[2] Renrui Zhang et al. MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? ECCV, 2024.

[3] Chaoqun He et al. OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems. ACL, 2024.

[4] Chengke Zou et al. DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models. ICLR, 2025.

[5] Fan Yuan et al. GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts? arXiv:2509.25160, 2025.

[6] Kun Xiang et al. SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoning. arXiv:2605.09266, 2026.

[7] Peter Kagey. A Proof Without Words: Triangles in the Triangular Grid. arXiv:2211.00186, 2022.

Questions for authors
  1. Can the authors provide an independent, anonymized human–judge agreement evaluation on responses not used for rubric calibration, and test whether hiding or swapping model labels changes the scores?

  2. Can the authors align the candidates’ and judge’s reasoning settings, perform paired comparisons between original diagrams and valid variants under the same RQ2 prompt, and separately report validity judgments and proof quality conditional on acceptance?

  3. Can the authors provide direct measurements of visual attention across reasoning positions and controlled experiments supporting the causal claim, or revise the abstract to reflect the available evidence?

  4. How many cases in Table 3 were flagged for human review, and how were they ultimately adjudicated? Why is UNCLEAR, mentioned in the appendix, absent from the table, and how are such outputs included in the metrics?

  5. Does the missing “equilateral” condition in Figure 1 also occur in the actual model inputs or rubrics? If so, have the affected responses and scores been reassessed?

  6. How do the 206 expert scorecards relate to the 144 complete scorecards in RQ3, and which analyses use the additional ratings? How should the discrepancies between category counts and totals in Table 7 be explained?

5 comments

Thank you for taking the time to review our work and for sharing your detailed insights! We address the main concerns and questions inline:

W1. Related-work accuracy and positioning: proof reasoning, answer-changing variants, modality diagnostics, reconstructing a specified visual proof and diagnosing visual defects

Thank you for pointing us to these papers! We agree that our original description was too broad. Several prior benchmarks do indeed evaluate intermediate reasoning, include proof tasks, or use controlled variants to probe reliance on visual information. We will revise the Related Work section to discuss these contributions explicitly and correct our description of DynaMath, whose variants can change the correct answer.

This table demonstrates how we plan to more precisely position our paper in the final version. Please let us know if there are further corrections you would like to make.

BenchmarkWhat is gradedVariantsReconstructs pictured proofDiagnoses proof defects
UniGeoAnswers + proof sequences❌❌❌
MathVerseAnswers + reasoning steps (subset)✅ Text/image balance❌❌
OlympiadBenchAnswers + sampled proofs❌❌❌
DynaMathFinal answers✅ Answers can change❌❌
GSM8K-VFinal answers✅ Text vs. image sequences❌❌
SeePhys ProFinal answers✅ Text-to-image transfer❌❌
PictureProofProof, grounding, validity, diagnosis✅ Diagram edits, fixed theorem✅✅

W2 & Q1. Independent blinded human–judge agreement on responses held out from rubric calibration, annotator score visibility, candidate-label hiding or swapping, and flagged-case sensitivity

Thank you for raising these concerns! We evaluated the judge against independent expert ratings on new responses and tested whether hiding the candidate model’s name changes the judge’s scores.

Blinded evaluation on held-out responses. We sampled ten problems from the annotated set and generated 20 new responses with GPT-6 Astra and Claude Opus 5. These responses were held out from rubric calibration and covered 6 original figures, 6 valid variants, and 8 invalid variants. Two experts independently graded all 20 responses using the fixed rubrics, with model names, judge scores, and each other’s scores hidden.

For each response, we compared the LLM judge’s score with the average of the two expert scores. Proof correctness and visual grounding are scored out of 5, and defect diagnosis is scored out of 2.

Measure#AnswersPearson correlationMean absolute error (score pts)
Proof correctness120.8550.521
Visual grounding120.9710.313
Defect diagnosis80.9870.063

Judge–expert agreement was generally high on these held-out responses, particularly for visual grounding and defect diagnosis.

Annotator score visibility. In the original study, we instructed annotators to apply the rubric using only their own judgment. We realized after grading that the interface had made other annotators’ scores visible. In this new blinded study, the two experts’ ratings correlated at 0.948 for proof correctness and 0.826 for visual grounding.

Candidate-label bias. For a follow-up study, we sampled 45 saved answers to original figures, one per source problem and five from each of nine models. We graded each answer twice with the same judge prompt and settings, once with the actual model name and once with the label “Anonymous candidate.” We randomized the order of the two grading calls, for a total of 90 judgments.

All scores are out of 5. We compute the change for each answer by subtracting the named score from the anonymous score. Mean absolute change measures how much scores moved in either direction.

MeasureNamed meanAnonymous meanMean change (named - anonymous score)95% confidence intervalMean absolute change
Proof correctness3.7223.817+0.094[−0.006, +0.206]0.228
Visual grounding3.8443.922+0.078[−0.006, +0.161]0.189

Hiding the model name raised average scores by less than 0.1 points on both measures. The average effect was small on the original answers we sampled.

Flagged cases. We resolved all 158 flagged response cases in the updated RQ2 benchmark. Our response to Q4 details the adjudication and reports the revised Table 3 scores.

W3 & Q2. Controlling RQ1–RQ2 prompt and reasoning-setting differences, pairing originals with valid variants under RQ2, and separating validity judgments from proof quality conditional on acceptance

Thank you for raising this! RQ2 already reruns the original images as well as the proof-preserving variants to get rid of this confound. We compared each of the 50 original figures with its two proof-preserving variants within RQ2, holding the task prompt and inference settings fixed across originals and variants. We also held the judge’s grading prompt and reasoning effort fixed. The prompts had to change between RQ1 and RQ2 in order to capture additional reasoning failure modes of the models.

Each table entry shows the result for originals followed by the result for variants (original → variant). We report the percentage of valid figures the model accepts and the mean proof and grounding scores among accepted responses. The final scores average over all responses, assigning zero to rejections. All explanation scores are out of 5.

ModelAcceptance %Accepted proofAccepted groundingFinal proofFinal grounding
GPT-6 Astra94 → 934.62 → 4.564.62 → 4.574.34 → 4.244.34 → 4.25
Gemini 3.1 Pro88 → 844.36 → 4.524.53 → 4.653.83 → 3.803.99 → 3.90
Claude Opus 580 → 734.70 → 4.684.89 → 4.803.76 → 3.423.91 → 3.50
GPT-5.566 → 594.30 → 4.564.30 → 4.502.84 → 2.692.83 → 2.65
Grok 4.376 → 793.18 → 3.363.34 → 3.752.42 → 2.652.54 → 2.96
Llama 4 Maverick36 → 323.24 → 2.412.78 → 2.931.17 → 0.771.00 → 0.94
Qwen3-VL-235B20 → 204.12 → 3.704.60 → 4.150.82 → 0.740.92 → 0.83
Qwen3-VL-8B2 → 04.00 → —4.00 → —0.08 → 0.000.08 → 0.00

GPT-6 and Opus have similar accepted-proof scores on originals and valid variants. Opus and GPT-5.5 accept fewer valid variants, which lowers their final scores. These results show how validity decisions and explanation quality contribute separately to each model’s performance.

W4 & Q3. Direct measurements of visual attention across reasoning positions, controlled evidence for causality, and revising the abstract’s attention-decay claim

This was an error on our part. We have revised the abstract to state that the attention results are observational experiments.

The original statement came from early experiments with a smaller model (Gemma 4 12B). There, attention to the image fell steadily as reasoning grew, and increasing image attention by re-inserting the image mid-reasoning improved visual grounding on a representative pilot set. Closer analysis showed that the attention decay was largely an architectural issue. Gemma 4 uses a 1024-token sliding window in 5 of every 6 layers, so once the reasoning was more than 1024 tokens past the image, those layers did not see the image, and image attention averaged over all layers was 1/6th of the full-attention layers’ value.

In follow-up analyses of the reported models, we accounted for such hybrid-attention layers as well as for the growing length of the reasoning, and found no strong evidence of a systematic decline in attention to the image.

W5. Always-YES and always-NO validity baselines, full confusion matrices, valid- and invalid-class recall, balanced accuracy and F1 interpretation

Thank you for pointing out how the class imbalance affects valid-class F1! We report confusion matrices for every model and for the always-YES and always-NO baselines. Our updated benchmark contains 152 valid diagrams (50 originals, 100 proof-preserving variants, and two variants reclassified during review) and 88 invalid diagrams. We first report counts by ground-truth validity and predicted verdict.

ModelValid → YESValid → NOInvalid → YESInvalid → NO
GPT-6 Astra14210484
Gemini 3.1 Pro130222860
Claude Opus 5114381474
GPT-5.594581573
Grok 4.3118345731
Llama 4 Maverick521001771
Qwen3-VL-235B311211177
Qwen3-VL-8B1151187
Always YES1520880
Always NO0152088

Next, we report recall for each class. Valid recall measures how often a model accepts a valid diagram. Invalid recall measures how often it rejects an invalid diagram. Balanced accuracy averages those two rates, giving each class equal weight. The always-YES and always-NO baselines each score 50% balanced accuracy.

ModelValid recall %Invalid recall %Balanced accuracy %Valid F1 %
GPT-6 Astra93.495.594.495.3
Gemini 3.1 Pro85.568.276.983.9
Claude Opus 575.084.179.581.4
GPT-5.561.883.072.472.0
Grok 4.377.635.256.472.2
Llama 4 Maverick34.280.757.447.1
Qwen3-VL-235B20.487.553.932.0
Qwen3-VL-8B0.798.949.81.3
Always YES100.00.050.077.6
Always NO0.0100.050.00.0

Validity judgments are only one part of our evaluation, and we also score proof correctness, visual grounding, and flaw diagnosis.

Q4. Counts and adjudication of Table 3 review flags, effects on updated scores, and the extraction, reporting and metric treatment of UNCLEAR verdicts

Thank you for bringing this up! Since submission, we manually resolved all 170 flagged cases. For some context, a judge may flag a response for various reasons, typically if the response identifies a different defect than intended. This can sometimes require a careful check of the diagram.

We then updated the benchmark to revise some diagrams and remove variants that still supported a valid proof. We generated and graded new responses for the revised figures. Importantly, the judge no longer flagged responses, which is important for scalability.

The original extractor assigned UNCLEAR when it could not find a YES or NO verdict in the requested format. We recovered 82 verdicts through improved parsing and five through human review. All 87 now count as YES or NO in the validity metrics.

The updated Table 3 reflects expert adjudication, figure revisions, and verdict recovery across 240 diagrams (152 valid and 88 invalid).

ModelProof scoreFlaw diagnosisValid precisionValid recallValid F1
GPT-6 Astra85.593.697.393.495.3
Gemini 3.1 Pro76.260.882.385.583.9
Claude Opus 570.375.589.175.081.4
GPT-5.555.061.386.261.872.0
Grok 4.351.315.467.477.672.2
Llama 4 Maverick18.28.575.434.247.1
Qwen3-VL-235B15.523.873.820.432.0
Qwen3-VL-8B0.534.050.00.71.3

W7 & Q5. The missing equilateral-triangle condition in Figure 1, verification of model inputs and rubrics, and whether responses or scores require reassessment

Thank you for catching this! That was an omission from the picture, we will update it in the revision. The equilateral condition is present in the actual model inputs and rubrics, and so no responses are affected by the omission.

W6 & Q6. Reconciling 206 expert scorecards with the 144 used in RQ3, the use of additional ratings, and Table 7’s missing categories, failure totals and percentage denominators

Thank you for pointing this out. The 206 expert scorecards are the full set summarized in Appendix B, Table 8. Of these, 144 cover the 40 responses used to compute inter-annotator agreement in RQ3. The remaining 62 contribute to the score distribution in Table 8 and fall outside the RQ3 agreement analysis.

In the submitted Table 7, we omitted the “Other” row while including its cases in the totals. That row accounts for three Opus 5 failures and one Grok 4.3 failure. We will include it in the revision. After adjudication and dataset revisions, Opus 5 and Grok 4.3 had 71 and 167 failures, respectively, across the 240 updated diagrams. Each model’s category percentages use its total number of failures as the denominator.

Reviewer_tN7YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PictureProof evaluates whether vision-language models can reconstruct the mathematical argument encoded in a visual proof, rather than merely produce a correct answer or an independent proof of the stated theorem. The benchmark contains 50 source problems, with editable TikZ constructions yielding 50 original figures, 100 proof-preserving variants, and 96 proof-breaking variants.

The paper evaluates proof reconstruction, visual grounding, validity classification, and flaw diagnosis. Open-ended explanations are graded by an LLM judge using expert-written rubrics. Additional experiments investigate visual revisiting during generation, attention to localized errors, and qualitative failure modes.

Critical strengths
  1. The benchmark targets a meaningful distinction between mathematical correctness and fidelity to visual evidence. A model can know that a theorem is true while failing to explain the argument represented by a particular diagram. PictureProof explicitly distinguishes these abilities and evaluates proof correctness separately from visual grounding. Requiring explanations of the displayed construction provides a useful complement to final-answer evaluation and conventional theorem-proving tasks.

  2. The proof-preserving and proof-breaking variants provide a valuable diagnostic design. Keeping the theorem fixed while modifying the construction tests whether a model follows the supplied visual argument. Including both valid and invalid modifications is particularly useful: a model cannot succeed simply by rejecting unusual figures. The editable TikZ representations and annotated broken relations also support controlled investigations of specific visual errors. The authors invest meaningful expert effort in rubric construction and calibration and transparently disclose limitations of their human-agreement study.

Critical weaknesses

Section 2.3 describes an expert-feedback loop that revises rubrics until the judge’s decisions align with expert assessments. This is a useful calibration procedure, but it does not establish performance on responses held out from calibration.

RQ3 reports agreement among human annotators, rather than agreement between the LLM judge and independent expert judgments. Moreover, annotators could see one another’s scores. The manuscript explicitly acknowledges both limitations; my concern is therefore not a lack of disclosure, but the absence of the validation needed to support the automated scores. High human consistency under this protocol does not establish that the judge correctly assesses mathematical reasoning or visual grounding.

This is particularly consequential for flaw diagnosis. Table 3 reports appreciable changes when credit is withheld for cases flagged for human review—for example, Opus 5 changes from 72.3 to 55.7. This sensitivity analysis is helpful, but it does not replace expert adjudication of the flagged cases or an independent audit of unflagged ones. The RQ1 grading prompt also exposes the candidate model’s identity, introducing an avoidable potential source of bias.

A blinded, held-out expert evaluation covering original figures, valid variants, and invalid variants is needed, with judge–human agreement, score errors, and the stability of model comparisons reported separately.

Questions for authors
  1. What causes the judge to flag a case for human review, how many cases are flagged for each model, and were they subsequently adjudicated? How do expert-adjudicated scores affect Table 3?

  2. Within RQ2, how do the 50 original figures compare with their 100 proof-preserving variants under identical prompts and inference settings? Please distinguish validity rejection from explanation quality conditional on acceptance.

  3. What reported evidence supports the abstract’s specific claim about attention decay causing failures? If no additional evidence is available, would the authors revise this to an observational finding about attention and error diagnosis?

3 comments

Thank you for taking the time to review our work! We address your questions and concerns:

W1. Held-out response validation after rubric calibration

Thank you for this suggestion! To test whether our format of rubric calibration allows an LLM-judge to accurately judge new responses, we sampled 10 problems from our annotated set, generated 20 new responses with GPT-6 Astra and Opus 5, and separately graded them.

The set included 6 responses to original figures, 6 to valid variants, and 8 to invalid variants. Two experts independently graded all 20 responses using the fixed rubrics, with model names, judge scores, and each other’s scores hidden.

For each response, we averaged the two expert scores and compared that average with the LLM judge’s score. We report Pearson correlation across responses and mean absolute error.

Measure#AnswersPearson correlationMean absolute error (score pts out of 55)
Proof correctness120.8550.521
Visual grounding120.9710.313
Defect diagnosis80.9870.063

We find that judge–expert agreement was high on held-out responses.

W2. Independent human–judge agreement on proof correctness and visual grounding, and annotators’ access to one another’s scores

Thank you for pointing out this omission! Below, we report the agreement between humans and judges on the original answers.

We asked the LLM judge to score the same 40 answers the experts had already graded. For each of the 20 problems, we included one answer from GPT-5.5 and one from Qwen3-VL-235B. Each answer had at least two expert ratings and was scored three times by the judge. On each 0–5 scoring axis, we computed the Pearson correlation across answers between the median expert score and the mean judge score.

Measure#AnswersPearson correlation
Proof correctness400.876
Visual grounding400.846

Annotator score leak. After the initial grading, we realized that the interface had made other annotators’ scores visible. We had instructed annotators to apply the rubric using only their own judgment. We check for this in the held-out study described in W1,where two experts independently graded fresh responses with each other’s scores hidden. The correlations between their ratings were 0.948 for proof correctness and 0.826 for visual grounding, indicating high agreement.

W3 & Q1. Reasons and per-model counts for review flags, expert adjudication and updated Table 3 scores, auditing unflagged answers, and candidate-identity bias

Review flags. Thank you for bringing this up! Since submission, we manually resolved all 170 flagged cases and updated our benchmark to address problematic flagged entries. For context, a judge may flag a response if it identifies a different defect than intended. This can require a careful check of the diagram. We revised some diagrams and removed variants that still supported a valid proof. We generated and graded new responses for the revised figures. The judge no longer flags responses at this point, which matters for scalability.

Below, we report the updated table covering 240 images from 50 source problems, with 152 valid and 88 invalid images.

ModelProof scoreFlaw diagnosisValid precisionValid recallValid F1
GPT-6 Astra85.593.697.393.495.3
Gemini 3.1 Pro76.260.882.385.583.9
Claude Opus 570.375.589.175.081.4
GPT-5.555.061.386.261.872.0
Grok 4.351.315.467.477.672.2
Llama 4 Maverick18.28.575.434.247.1
Qwen3-VL-235B15.523.873.820.432.0
Qwen3-VL-8B0.534.050.00.71.3

Candidate identity bias. This is an important observation! We agree that showing the candidate’s name could influence the judge’s scores. To test this, we sampled 45 saved answers to original figures, one per source problem and five from each of nine models. We graded each answer twice with the same judge prompt and settings, once with the actual model name and once with the label “Anonymous candidate.” We randomized the order of the two grading calls, for a total of 90 judgments.

All scores are on a 0–5 scale. For each answer, we subtract the named score from the anonymous score. The mean change shows the direction of the shift, and the mean absolute change shows how much scores moved in either direction.

MeasureNamed meanAnonymous meanMean change95% confidence intervalMean absolute change
Proof correctness3.7223.817+0.094
−0.006,+0.206-0.006, +0.206

| 0.228 | | Visual grounding | 3.844 | 3.922 | +0.078 |

−0.006,+0.161-0.006, +0.161

| 0.189 |

Hiding the model name raised average scores by less than 0.1 points on both measures. Overall, the candidate label had a small average effect on the original answers we sampled.

W4. Blinded held-out expert evaluation of originals, valid variants and invalid variants: judge–human agreement, score errors and stability of model comparisons

Thank you for suggesting this evaluation. As described in W1, two experts independently graded 20 fresh responses from GPT-6 Astra and Claude Opus 5, with model identities, judge scores, and each other’s scores hidden. Because checking each response against its figure and rubric takes substantial expert time, we prioritized two independent ratings per response and coverage of all three figure types. As reported in W1, judge–expert agreement was generally high, particularly for visual grounding and defect diagnosis.

We believe the structure of the task helps explain this agreement. Two explanations can describe the same visual argument in very different words, yet the steps that make the argument work are often concrete enough to check. For instance, if a step relies on two lengths being equal, an evaluator can ask whether the diagram establishes that equality and whether the response explains how it is used. Our rubric authors drew on their experience summarizing proofs into key steps to turn these judgments into explicit scoring criteria.

Assessing the stability of the full model ranking would require more costly model calls and expert ratings than we could collect during the rebuttal period.

Q2. Paired comparisons of originals and valid variants under identical RQ2 prompts and inference settings, separating validity rejection from proof quality conditional on acceptance

Within RQ2, we compared each of the 50 original figures with its two planned proof-preserving variants, for a total of 100 variants. For each model, we used the same task instructions and inference settings for originals and variants. The judge also used the same grading instructions and reasoning setting for both.

Each table entry shows the result for originals followed by the result for variants (original → variant). The acceptance rate is the percentage of responses that say YES. We average proof and grounding scores over YES responses for the accepted scores and over all responses for the final scores, assigning zero to rejections. All explanation scores are out of 5.

ModelAcceptance %Accepted proofAccepted groundingFinal proofFinal grounding
GPT-6 Astra94 → 934.62 → 4.564.62 → 4.574.34 → 4.244.34 → 4.25
Gemini 3.1 Pro88 → 844.36 → 4.524.53 → 4.653.83 → 3.803.99 → 3.90
Claude Opus 580 → 734.70 → 4.684.89 → 4.803.76 → 3.423.91 → 3.50
GPT-5.566 → 594.30 → 4.564.30 → 4.502.84 → 2.692.83 → 2.65
Grok 4.376 → 793.18 → 3.363.34 → 3.752.42 → 2.652.54 → 2.96
Llama 4 Maverick36 → 323.24 → 2.412.78 → 2.931.17 → 0.771.00 → 0.94
Qwen3-VL-235B20 → 204.12 → 3.704.60 → 4.150.82 → 0.740.92 → 0.83
Qwen3-VL-8B2 → 04.00 → —4.00 → —0.08 → 0.000.08 → 0.00

Notably, Opus 5 accepted 80% of originals and 73% of valid variants, while its mean proof score among accepted responses stayed nearly unchanged (4.70 vs. 4.68).

Q3. Evidence for attention decay causing failures, and revising the claim to an observational finding about attention and error diagnosis

Thank you for raising this and suggesting a fix! This was an error on our part and we have revised the abstract to an observational finding about attention and error diagnosis. Please see our response to Reviewer F6DF (W4 & Q3) for additional details.

I thank the authors for their responses. I am inclined to maintain this positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Manuscript and Discussion Period Summary

Dear Reviewers and AC’s,

Thank you again for your time and thoughtful suggestions! We would like to re-summarize our paper’s strengths and our comprehensive response to the cited weaknesses.

We created a benchmark based on explaining visual math proofs, which probes the intersection of proof-based reasoning and visual grounding---a capability that existing visual math benchmarks fail to evaluate. We developed expert-curated rubrics for an LLM judge-based evaluation framework for the community to automatically evaluate model performance on 50 core visual proofs with 102 proof-preserving variants and 88 proof-breaking variants. This work will be broadly useful for advancing multi-modal math models for education and scientific research as well as ML research on robustness and mech interp of VLMs.

We believe we addressed all of the weaknesses described by the reviewers. Most importantly:

1 - Held-out response validation after rubric calibration; blinded human expert judge validation. In our original paper, we did not have a section demonstrating human-LLM judge alignment on held-out responses. Also, human experts were able to see model name, LLM judge scores, and other human annotator scores while they scored the model proof explanations. Although our annotators are extremely experienced with math olympiad training and university math course grading, the setup could bias their evaluations.

We demonstrated generalization of our LLM judge rubrics and annotator independence with an additional round of human expert grading on 20 newly-generated proofs (10 problems each from GPT-6 Astra and Claude Opus 5).

For relevant discussion, please refer to: tN7Y (W1, W2, W4); F6DF (W2, Q1)

2 - Candidate model identity and potential judge bias. Reviewers were concerned that seeing the evaluated model’s name could influence the LLM judge’s scores. We demonstrated that hiding model names changed average proof correctness and visual grounding LLM judge scores by less than 0.1 points on a 5-point scale. We ran this follow-up experiment with 45 saved responses spanning nine models, grading each response once with its actual model name and once with an anonymous label. For relevant discussion, please refer to: tN7Y (W3 & Q1); F6DF (W2 & Q1)

3 - Expert review of flagged variants and more complete validity evaluation. Our original results included model responses to proof-breaking variants that were flagged for human review as having a possibly correct interpretation. We resolved all flagged and unclear variant cases, fixed 8 variants with genuine issues (out of 198 variants) , and reran evaluation on a cleaned 240-item benchmark set. For validity evaluation completeness, we report valid-class F1, confusion matrices, per-class recall, balanced accuracy, and always-YES/NO baselines. Please refer to: tN7Y (W3 & Q1); F6DF (W2, W5, Q4)

4 - Controlled comparisons of original figures and proof-preserving variants. Reviewers noted that RQ1 and RQ2 combine changes in task instructions, reasoning settings, and the treatment of rejected proofs. We controlled for experimental differences and showed that validity decisions and explanation quality contribute separately to each model’s performance. Specifically, we compared 50 original proofs against their 100 proof-preserving variants under identical model and judge settings, and reported acceptance conditional explanation quality, and final scores separately to distinguish rejecting a valid construction from failing to explain the proof. For example, Claude Opus 5 accepted fewer valid variants (73% vs. 80%), yet its proof score on accepted ones barely changed (4.68 vs. 4.70/5). Please refer to: tN7Y (Q2); F6DF (W3 & Q2)

5 - Non-structural corrections. We addressed a variety of positioning and wording issues. We removed a problematic, non-foundational interpretation about attention decay (see tN7Y W4, Q3; F6DF W4, Q3). We agreed that memorization may not be the sole cause of failures on modified figures. We ran pilot experiments that found some models recognize edits while reciting unedited proofs and we will update our writeup to reflect the larger variety of possible VLM failure modes, such as sycophancy and language dominance (see XEFq W1, Q1). We addressed the literature positioning concerns raised (F6DF W1). Finally, we clarified figure captions, annotation counts, and failure reporting (F6DF W6, W7, Q5, Q6).

Thank you again for the careful feedback! Overall, we hope these discussions demonstrate the value of PictureProof as a benchmark and we are excited to polish our paper into a final version based on all of your help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.