- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
GeoPerturb evaluates whether multimodal models preserve correctness across equivalent representations of geometry problems. It constructs 531 matched suites using TikZ reconstruction and four perturbation families: rotation, relabeling, appearance changes, and irrelevant visual additions. Paired accuracy changes and canonical-conditioned retention reveal failures hidden by aggregate accuracy. A fine-tuning study further compares derivation supervision with answer-only supervision.
Critical strengths
S1. An editable representation for controlled geometric transformations.
Reconstructing diagrams in TikZ provides a practical way to manipulate geometric elements and labels explicitly. This representation makes it easier to generate interpretable transformations while preserving the intended problem structure.
S2. A clear taxonomy of geometry-specific invariances.
The four perturbation families organize representation changes around orientation, symbolic identity, appearance, and irrelevant content. Synchronized relabeling across the diagram and text is particularly well motivated, as it tests whether a solution depends on arbitrary entity names.
Critical weaknesses
W1. Experimental reliability and the fine-tuning conclusions need stronger support.
The public repository contains scoring and label-synchronization errors, including accepting 1/2 as 2, but the actual experimental artifacts are unavailable to determine whether these affect the reported results. In the fine-tuning study, AnswerOnly reduces perturbation accuracy from 60.1% to 29.4%, leaving severe training degradation unexplained. A shared evaluation prompt does not control generated reasoning length, and improved average accuracy alone does not establish improved cross-view consistency. The conclusions would be stronger with the actual evaluation pipeline, an explanation of the AnswerOnly collapse, and post-training retention results across multiple seeds.
W2. The intervention taxonomy does not yet justify the claimed hierarchy of vulnerabilities.
Rotation and relabeling modify the task-relevant representation, whereas Redundancy is restricted to detached, unlabeled objects without proof-relevant alignment (section 3.2). The latter therefore tests a narrow form of distractor rejection; its small effect does not establish that irrelevant structure is generally less disruptive. The proposed hierarchy also depends on the metric: in Table 3, relabeling retention exceeds Redundancy retention for both Opus (88.1% versus 85.7%) and Sonnet (93.5% versus 90.9%). These point estimates do not establish the reverse ordering, but they weaken a consistent ranking across families. The framework convincingly identifies failures under selected redraws; its broader diagnostic interpretation would benefit from distinguishing those observations from claims about the relative difficulty of different invariances.
Questions for authors
Q1. Could you clarify whether the experimental scorer evaluates only the extracted final answer?
The released scorer can mark a response as correct when the gold number appears in intermediate reasoning, even if the final answer is incorrect. Could you clarify whether this implementation was used for the results in the paper? If so, please report the corrected results after rescoring the affected outputs, particularly the CoT–AnswerOnly comparisons in Table 4, and indicate whether the main conclusions still hold. If a different scorer was used, a brief explanation of how it avoids this error would resolve the concern.
Q2. How should the relative vulnerability of the four families be interpreted?
Is the ordering in section 4.2 intended to describe net accuracy loss or the loss of previously solved cases? Table 3 does not consistently place relabeling below Redundancy in retention. Given the deliberately peripheral distractors used for Redundancy, it would help to clarify which conclusions are specific to the chosen transformations and which are expected to generalize across each family.
Q3. Could you clarify what the fine-tuning comparison establishes about the source of improvement?
Appendix H.4's CoT targets include visual analysis, distractor checks, and element mapping alongside geometric derivations. The comparison therefore changes several forms of supervision together. What evidence distinguishes the contribution of derivation supervision from these additional perceptual and grounding annotations, particularly given the substantial AnswerOnly degradation in Table 4?
If the authors can adequately address these concerns, I would consider raising my score.