acceptodds
Withdrawn from ICLR 2027

GraphTheory-VL: Visual Instances for Exact Mathematical Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A mathematical drawing specifies an instance, while a solution must recover that instance and compute its requested target. A final answer alone can hide an incorrect structural interpretation, and removing the drawing can make a reasonable refusal indistinguishable from a failed solution. We introduce GraphTheory-VL, a collection of 62 visual mathematics problems with an embedded 20-problem hard subset, linking graph, algebraic, and topological targets to reference checks and complete model responses. We evaluate three model families with four responses per problem and compare original-image and text-only inputs on the hard subset. Mean accuracy on the 62-problem collection ranges from 10.89% to 31.85%, while the hard subset remains at 1.25% to 5.00%. Across models, at least one response succeeds on 43 of 62 problems, but only five of the 20 hard problems. An analysis of all 120 first responses in the image ablation distinguishes reasonable uncertainty, unsupported instance assumptions, and observable mathematical errors. In particular, a text-only response can match the target after guessing an omitted parameter, whereas a response that correctly identifies a graph can still compute the wrong critical group. These results connect repeated correctness to the mathematical and visual evidence needed to interpret it.

Primary area:
datasets and benchmarks
Keywords:
visual mathematical reasoning, vision-language models, multimodal benchmarks, graph theory, instance recovery, exact mathematical targets, repeated correctness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZoFKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents Graphtheory-VL, a benchmark for testing visual mathematical understanding and reasoning in VLMs. Each benchmark item is a mathematical diagram + question + answer + facts needed to derive the answer. The paper evaluates several frontier VLMs on the dataset, showing that it is challenging for modern VLMs. One of the paper's central theses is that evaluations must take into account the visual dependence more explicitly, rather than relying solely on text scores.

Critical strengths

S1. The benchmark is difficult and valuable for understanding failure modes of VLMs in mathematical reasoning.

S2. The content card is a good use of rubric-based grading system. It is useful to separate correctness from the visual grounding.

S3. The failure analysis and characterization in Table 4 is very useful.

Critical weaknesses

W1. It is not clear how the dataset is constructed, what the source of the original ~600 problems is, and what the unifying theme is in the dataset. There appears to be some themes of group theory, graph theory, and geometry but this needs to be reported more concretely.

W2. Since the benchmark is based on rubrics, there needs to be experiments showing that the LLM-judge is good at following the rubric. Additionally, there needs to be human studies showing how often they agree with the LLM-judge. More broadly, there needs to be more human validation to show that the automated evaluation methodology is accurate and scalable.

W3. It would be good to revisit the no-image condition for the other problems. Since the Hard20 dataset is already very hard when the image is provided, it makes it harder to differentiate the failures when no image is provided. Furthermore, for the text-only condition, the instruction still refers to the supplied image. This makes one likely cause of the model hallucinations to be sycophancy, since it's been told implicitly that the image exists. This is fine but perhaps the interpretation needs to be narrower.

W4. There is insufficient evaluations of a diverse set of VLMs on the task. For this reason, it would be also good to retain some of the easier problems from the initial ~600 problem set to retain different difficulty levels for different model sizes/classes.

Questions for authors

Q1. Can the authors describe the family of mathematical tasks they are attempting to capture with this dataset?

Q2. On the Hard20 dataset, the performance of VLMs is very low. Can the authors clarify if this can be traced to poor visual understanding, as opposed to the difficulty of the problems themselves? One way to measure difficulty of the problem would be by providing a structured description of the image to the VLM.

Reviewer_eXwuOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GraphTheory-VL introduces 62 visual mathematics problems selected from 623 candidates, including a nested Hard20 subset selected for persistent model failures. The benchmark combines graph-theoretic, algebraic, and topological targets with reference explanations and archived responses.

Three model families are evaluated with four responses per problem. Original-image mean accuracy ranges from 10.89% to 31.85% on Challenge62 and from 1.25% to 5.00% on Hard20. Image-removal experiments and an analysis of 120 first responses examine the relationship between target correctness, missing information, and unsupported assumptions. The main contribution is a curated collection of difficult instances with supporting diagnostic evidence.

Critical strengths
  1. The mathematical targets support meaningful examination of model errors. Structured outputs such as group invariant factors and symbolic polynomials expose distinctions that simpler numerical checks can miss. The worked cases illustrate both incorrect instance reconstruction and incorrect mathematical computation, making the collection useful for targeted investigation.
  2. The evaluation is transparent and goes beyond aggregate accuracy. The paper reports uncertainty, repeated-attempt success, and response-level behavioral categories. Its separation of justified uncertainty from unsupported answers is valuable, and the accompanying records make individual judgments inspectable.
Critical weaknesses
  1. The small, model-conditioned benchmark has limited statistical precision and external validity.

    Challenge62 contains only 62 problems, while the central image-ablation study uses 20. Table 1 illustrates the resulting imprecision: GPT achieves 31.85% with a 95% interval of [23.0, 41.1], while Gemini achieves 28.63% with [19.4, 38.3]. More directly, their paired difference is 3.23 percentage points with an interval of [-6.85, 13.31]. These results do not establish their relative ordering. This is a limitation of the benchmark’s ability to distinguish comparable systems, rather than evidence that the statistical calculation is incorrect. Membership is also explicitly conditioned on failures of selected models. Separate evaluation responses and the inclusion of Claude, which was not used for screening, provide useful validation, but only three model families are evaluated. Repeated responses do not increase problem diversity. Without broader evaluation on models excluded from selection or a qualified-human baseline, it remains unclear how broadly the observed difficulty generalizes. The paper acknowledges these limitations, but they still constrain the significance and practical utility of the resource.

  2. The additional diagnostic insight into visual dependence is limited and insufficiently independently validated.

    Visual dependence and reasoning quality have already been investigated by benchmarks such as MathVerse and GraphVerse. The additional mathematical targets are useful, but stronger evidence is needed to establish a substantial diagnostic advance. The paper acknowledges text-sufficient cases within Challenge62. On Hard20, identifying missing instance information supports dependence on the supplied image, but does not establish how effectively models use that information. Both image conditions are near the accuracy floor, and all paired image-ablation intervals in Table 3 include zero. A verified structural-input condition is missing, so the experiments cannot adequately separate failures to recover the instance from failures in subsequent mathematical reasoning. Table 4 provides quantitative error analysis, including 51 instance-statement conflicts among 60 original-image first responses. However, these AI-assisted categories lack independent human calibration and do not establish the causal bottleneck. Likewise, the scoring pipeline relies heavily on DeepSeek judgments. Access to reference answers makes judging different from solving, but its reliability still requires independent measurement. Checking positive labels does not estimate false negatives among rejected answers or rule out shared reference errors. These uncertainties directly affect the reported difficulty and diagnostic conclusions.

Questions for authors
  1. Diagnostic validation: How does performance change when the same instance is supplied through a verified adjacency list or equivalent structural description? Such a comparison would help establish which failures concern instance extraction and which concern subsequent reasoning.

  2. Human difficulty: Can you provide a qualified-human baseline, specifying participants’ expertise, time limits, and tool access? This would help distinguish mathematical difficulty from ambiguous or demanding visual interpretation.

  3. Generalization: Can the difficulty findings be corroborated using a broader set of strong models that did not participate in screening, with benchmark membership fixed before evaluation?

Reviewer_9jUNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces GraphTheory-VL, a benchmark for evaluating mathematical reasoning over visual graph-theoretic instances. The motivation is to evaluate whether the model could utilize the visual information properly and derive correct information and reasoning from text-image pairs. The benchmark consists of 62 selected problems (Challenge62), including a harder subset of 20 problems (Hard20), obtained from an initial pool of 623 question–image pairs through model-based screening and reference verification.

Critical strengths
  1. The targeted evaluation problem is meaningful. A model may respond to a correct answer without sufficient visual evidence by guess. Only checking the final answer is also not sufficient to establish the correctness because the reasoning trace may include incorrect prerequisite and evidence. This paper provides a collection of challenging visual mathematics problems with per-question reference explanations and supporting evidence for possible deeper analysis.

  2. The experiments provide some detailed information. The authors provide repeated-attempt results, image-removal experiments, individual problem results, reference-correction records, and several concrete failure analyses.

Critical weaknesses
  1. The reliability of the mathematical references and evaluation labels is not clear. Where are these tasks derived and how about the distribution? Since the benchmark is initially generated with AI assistant, how to make sure the query and answer are correct? Also, it’s not clear how reliability of the model-based semantic judgments is in scoring. Whether using Deepseek-V4-Flash is enough for evaluating such complex math reasoning trace? More information is suggested.

  2. To evaluate the model without images, the experiments are conducted on Hard20 subset. Why? The original-image accuracy is already extremely low. The experiment has limited ability to quantify the contribution of visual information.

  3. The main benchmark contains only 62 questions, while the image-removal experiment is restricted to 20 questions. The small size results in wide uncertainty intervals.

  4. The proposed contribution relative to existing visual-reasoning benchmarks needs better illustration. Previous work, including MathVerse, VCBench, and GraphVerse, has already investigated explicit visual dependence and evaluation beyond final-answer accuracy. GraphVerse includes graph-image interventions and process-sensitive evaluation. It’s not clear the advantageous features of the proposed benchmark.

  5. The presentation and organization are not easy to follow.

Questions for authors

See Weakness 1, Weakness 2 and Weakness 4

Moreover, Can the authors provide a more systematic evaluation of visual grounding, such as intermediate visual-fact verification or controlled changes to visual instances? Although the paper provides references, the current evaluation does not systematically verify individual visual facts and mathematical reasoning steps in model responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.