- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents Graphtheory-VL, a benchmark for testing visual mathematical understanding and reasoning in VLMs. Each benchmark item is a mathematical diagram + question + answer + facts needed to derive the answer. The paper evaluates several frontier VLMs on the dataset, showing that it is challenging for modern VLMs. One of the paper's central theses is that evaluations must take into account the visual dependence more explicitly, rather than relying solely on text scores.
Critical strengths
S1. The benchmark is difficult and valuable for understanding failure modes of VLMs in mathematical reasoning.
S2. The content card is a good use of rubric-based grading system. It is useful to separate correctness from the visual grounding.
S3. The failure analysis and characterization in Table 4 is very useful.
Critical weaknesses
W1. It is not clear how the dataset is constructed, what the source of the original ~600 problems is, and what the unifying theme is in the dataset. There appears to be some themes of group theory, graph theory, and geometry but this needs to be reported more concretely.
W2. Since the benchmark is based on rubrics, there needs to be experiments showing that the LLM-judge is good at following the rubric. Additionally, there needs to be human studies showing how often they agree with the LLM-judge. More broadly, there needs to be more human validation to show that the automated evaluation methodology is accurate and scalable.
W3. It would be good to revisit the no-image condition for the other problems. Since the Hard20 dataset is already very hard when the image is provided, it makes it harder to differentiate the failures when no image is provided. Furthermore, for the text-only condition, the instruction still refers to the supplied image. This makes one likely cause of the model hallucinations to be sycophancy, since it's been told implicitly that the image exists. This is fine but perhaps the interpretation needs to be narrower.
W4. There is insufficient evaluations of a diverse set of VLMs on the task. For this reason, it would be also good to retain some of the easier problems from the initial ~600 problem set to retain different difficulty levels for different model sizes/classes.
Questions for authors
Q1. Can the authors describe the family of mathematical tasks they are attempting to capture with this dataset?
Q2. On the Hard20 dataset, the performance of VLMs is very low. Can the authors clarify if this can be traced to poor visual understanding, as opposed to the difficulty of the problems themselves? One way to measure difficulty of the problem would be by providing a structured description of the image to the VLM.