acceptodds
Under review as a conference paper at ICLR 2027

GeoPerturb: Probing Structural Vulnerability in Multimodal Geometry Reasoning

Abstract

Large Multimodal Models achieve strong results on visual geometry benchmarks, yet it remains unclear whether this performance reflects geometric reasoning that remains stable across equivalent drawings. Rather than relying on low-level pixel corruptions, we introduce GeoPerturb, a matched diagnostic framework that reconstructs each problem in TikZ and generates four answer-preserving variants that change orientation, symbol identity, visual appearance, or irrelevant structure. The resulting benchmark contains 531 matched problem suites. Across proprietary and open-source LMMs, we find that the same problem is often answered correctly in one equivalent view and incorrectly in another. This instability is selective rather than generic: rotation and relabeling expose the clearest failures, while appearance changes and irrelevant clutter are less disruptive for frontier models. Marginal accuracy can further conceal these failures because gains on previously incorrect problems can offset losses on problems already solved; canonical-conditioned retention makes these solved-to-failed transitions visible. A controlled fine-tuning study further shows that explicit derivation supervision, rather than exposure to additional views alone, drives cross-view improvement. GeoPerturb therefore shifts geometry evaluation from asking whether a model can answer one drawing to asking whether it preserves the same solution across equivalent drawings. The central weakness is not simply low accuracy, but unstable correctness.

Primary area:
datasets and benchmarks
Keywords:
vision question answering, cross-modal application, multimodality, AI4Math, Math Reasoning, Geometry Problem Solving

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CVLEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

GeoPerturb evaluates whether multimodal models preserve correctness across equivalent representations of geometry problems. It constructs 531 matched suites using TikZ reconstruction and four perturbation families: rotation, relabeling, appearance changes, and irrelevant visual additions. Paired accuracy changes and canonical-conditioned retention reveal failures hidden by aggregate accuracy. A fine-tuning study further compares derivation supervision with answer-only supervision.

Critical strengths

S1. An editable representation for controlled geometric transformations.

Reconstructing diagrams in TikZ provides a practical way to manipulate geometric elements and labels explicitly. This representation makes it easier to generate interpretable transformations while preserving the intended problem structure.

S2. A clear taxonomy of geometry-specific invariances.

The four perturbation families organize representation changes around orientation, symbolic identity, appearance, and irrelevant content. Synchronized relabeling across the diagram and text is particularly well motivated, as it tests whether a solution depends on arbitrary entity names.

Critical weaknesses

W1. Experimental reliability and the fine-tuning conclusions need stronger support.

The public repository contains scoring and label-synchronization errors, including accepting 1/2 as 2, but the actual experimental artifacts are unavailable to determine whether these affect the reported results. In the fine-tuning study, AnswerOnly reduces perturbation accuracy from 60.1% to 29.4%, leaving severe training degradation unexplained. A shared evaluation prompt does not control generated reasoning length, and improved average accuracy alone does not establish improved cross-view consistency. The conclusions would be stronger with the actual evaluation pipeline, an explanation of the AnswerOnly collapse, and post-training retention results across multiple seeds.

W2. The intervention taxonomy does not yet justify the claimed hierarchy of vulnerabilities.

Rotation and relabeling modify the task-relevant representation, whereas Redundancy is restricted to detached, unlabeled objects without proof-relevant alignment (section 3.2). The latter therefore tests a narrow form of distractor rejection; its small effect does not establish that irrelevant structure is generally less disruptive. The proposed hierarchy also depends on the metric: in Table 3, relabeling retention exceeds Redundancy retention for both Opus (88.1% versus 85.7%) and Sonnet (93.5% versus 90.9%). These point estimates do not establish the reverse ordering, but they weaken a consistent ranking across families. The framework convincingly identifies failures under selected redraws; its broader diagnostic interpretation would benefit from distinguishing those observations from claims about the relative difficulty of different invariances.

Questions for authors

Q1. Could you clarify whether the experimental scorer evaluates only the extracted final answer?

The released scorer can mark a response as correct when the gold number appears in intermediate reasoning, even if the final answer is incorrect. Could you clarify whether this implementation was used for the results in the paper? If so, please report the corrected results after rescoring the affected outputs, particularly the CoT–AnswerOnly comparisons in Table 4, and indicate whether the main conclusions still hold. If a different scorer was used, a brief explanation of how it avoids this error would resolve the concern.

Q2. How should the relative vulnerability of the four families be interpreted?

Is the ordering in section 4.2 intended to describe net accuracy loss or the loss of previously solved cases? Table 3 does not consistently place relabeling below Redundancy in retention. Given the deliberately peripheral distractors used for Redundancy, it would help to clarify which conclusions are specific to the chosen transformations and which are expected to generalize across each family.

Q3. Could you clarify what the fine-tuning comparison establishes about the source of improvement?

Appendix H.4's CoT targets include visual analysis, distractor checks, and element mapping alongside geometric derivations. The comparison therefore changes several forms of supervision together. What evidence distinguishes the contribution of derivation supervision from these additional perceptual and grounding annotations, particularly given the substantial AnswerOnly degradation in Table 4?

If the authors can adequately address these concerns, I would consider raising my score.

Reviewer_CSacOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GeoPerturb, a diagnostic benchmark for evaluating whether multimodal models preserve correct answers across equivalent geometry diagrams. It constructs 531 matched problem suites using TikZ reconstruction and four answer-preserving transformations: rotation, relabeling, appearance changes, and irrelevant-object insertion. Experiments across 11 models combine paired accuracy changes with canonical-conditioned retention, showing that aggregate accuracy can conceal correct-to-incorrect transitions. An additional Qwen3-VL-8B fine-tuning study reports improved performance on perturbed views with derivation supervision.

Critical strengths

The matched evaluation provides an interpretable way to examine whether previously solved problems remain solved after equivalent redraws. Separating lost solutions from newly gained ones is consequential: the transition analysis demonstrates that stable or improving aggregate accuracy can conceal failures on previously correct examples.

The dataset construction includes substantial quality control, including independent annotation of the constructed views and scoring validation. Separately analyzing Original-to-Canonical changes also strengthens the evaluation by acknowledging that reconstruction itself can affect model performance.

Critical weaknesses

The contribution relative to closely related work is insufficiently established. MathLens already evaluates answer-preserving geometry modifications, including rotations with upright labels, point relabeling, and additional geometric elements, alongside cross-variant consistency. The current paper does not discuss this direct overlap. Although GeoPerturb’s correctness-conditioned analysis provides a useful perspective, the manuscript needs to establish more clearly what additional methodological or empirical insight it contributes beyond existing geometry robustness benchmarks.

Important gaps remain in the evidence supporting the robustness conclusions. Table 3 and Figure 3 use model-specific complete-case pools of 45 or 93 problems from the 96-problem Source-Test split, without adequately explaining the missing cases. Comparisons remain paired within each model, but the differing pools limit cross-model comparability and leave possible selection effects unresolved. Furthermore, Table 4 reports accuracy without the proposed retention metrics. It therefore demonstrates improved performance on perturbed views, but does not directly establish improved correctness preservation—the distinction emphasized throughout the paper.

Questions for authors

What substantive differences from MathLens establish the added contribution of GeoPerturb, particularly in its transformations, evaluation protocol, and resulting findings?

Why do the complete-case Source-Test pools differ substantially across models? Please clarify the missing-case criteria and whether the main conclusions hold on a common set of problems.

For the fine-tuning conditions in Table 4, do the improvements also hold under canonical-conditioned family retention and all-four retention? Reporting these metrics from the existing predictions would directly address whether the intervention improves the paper’s central notion of robustness.

Reviewer_jG8pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether the Multimodal LLM will produce matched answers under diagram variants covering rotation, relabeling, appearance changes and irrelevant visual elements. The authors construct a benchmarking dataset including 531 geometry problems. After evaluating 11 Multimodal models, the authors find that rotation and relabeling produces substantial instability. The authors also proposed some fine-tuning methods to improve performance on perturbed diagrams.

Critical strengths
  1. Careful check of the benchmark by three annotators. The authors also evaluate the effect of redrawing separately fro the later perturbations.

  2. The analysis of individual prediction changes is informative. Table 3 show that average accuracy can stay the same even when a model loses some of the problems. Tracking these changes can give a better view of how stable the models answers actually are.

Critical weaknesses

1 I am not convinced that the paper established a clear contribution beyond MathLens which already tests geometry models using answer-preserving rotations, point relabeling, and added auxiliary elements, and measures consistency across the resulting diagrams. The author also does not discuss this closely related work in Section 2. See Chung et al., What MLLMs Learn about When they Learn about Multimodal Reasoning.

  1. I could not find a clear explanation for the different sample sizes used in the retention analysis. There are 96 Source-Test problems, but Table 10 lists only 45 complete cases for GPT-5.4-High and Claude-Opus, compared with 93 for Gemini and Claude-Sonnet. Without knowing why cases were excluded, it is difficult to judge how representative the results are or to compare models evaluated on different questions. The authors should explain the exclusions and check whether the main findings hold on the problems shared across models.
Questions for authors
  1. What does GeoPerturb add beyond MathLens’s answer-preserving diagram modifications and consistency analysis? Please identify the main new capability or finding.
  2. Why are there only 45 complete Source-Test cases for several models, compared with 93 for others? How were missing or unscorable responses handled? Using the existing predictions, do the main findings hold on the shared set of problems?
  3. For Table 4, what training instructions were given to the AnswerOnly models? Were the number of training examples and optimization steps matched across the Original, Canonical, and Perturbation conditions? These details would help explain the large AnswerOnly performance drop and the gains from perturbation training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.