- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces MathVis, a 512-task benchmark for generating executable mathematical figures across 27 mathematical sub-disciplines. The main idea is to combine TikZ/SVG generation with Gold IR and atomic constraints, so that technical and mathematical failures can be separated. I think the move beyond elementary geometry is a reasonable and genuinely new benchmark direction. My main concern is more basic: I do not trust the benchmark evaluation in its current form. The paper's own Rouché examples contain grading decisions that directly contradict the shown figures, while the paper does not report aggregate evaluator-reliability statistics. Since the evaluator is the main contribution, this is the main reason I lean reject.
Critical strengths
-
The benchmark targets a real gap. Most existing mathematical figure-generation benchmarks focus on elementary geometry or educational diagrams, while MathVis tries to cover more advanced structures from complex analysis, differential geometry, topology, etc.
-
The Gold IR + atomic-predicate design is potentially useful. Separating executability, mathematical constraints, and presentation could provide much more informative diagnostics than image similarity or a single pass/fail score.
Critical weaknesses
-
Correctness. I do not think the paper has established that its evaluator is reliable. In the Rouché example, fig 9 explicitly states that both functions have three zeros inside C, yet the paper marks this conclusion as missing. fig 8 also visibly contains the radius-two circle that the paper says is absent. These are not minor presentation issues. If the showcased grading decisions are already wrong, it is hard to know how much confidence to place in the benchmark-level results.
-
Scope/significance. The paper makes claims about how LLMs use visual versus symbolic reasoning, but the main experiment is essentially single-round text-to-code generation. It does not test whether models actually use generated diagrams for subsequent reasoning, or whether diagrams improve reasoning relative to a symbolic baseline. So I think the benchmark contribution is narrower than the paper currently claims.
Questions for authors
-
Have the authors independently re-audited all 512 tasks and their predicate-level judgments for errors like those in the Rouché example? If so, how many judgments changed?
-
Can the authors report actual inter-rater agreement or evaluator error rates for the hard semantic predicates? This seems necessary before the model score differences are interpretable.
-
Can the authors report mathematical hard-predicate accuracy separately from strict delivery? Otherwise it is difficult to tell whether the low scores mainly come from mathematical errors or presentation constraints.
-
Do the authors have an experiment where generated diagrams are actually used for a subsequent reasoning task? If not, I think the claims about visual versus symbolic reasoning should be substantially narrowed.