acceptodds
Under review as a conference paper at ICLR 2027

GeoFlip: Do MLLMs Know Whether Geometry Problems Are Underdetermined?

10 views

Abstract

Advances in supervised fine-tuning and reinforcement learning have substantially improved the mathematical reasoning capabilities of multimodal large language models (MLLMs), enabling them to solve geometry problems when sufficient evidence is provided. However, whether MLLMs can reliably recognize underdetermination induced by removing essential visual evidence remains unclear. Existing reliability evaluations mainly focus on missing textual conditions, leaving the role of visual evidence in multimodal mathematical reasoning insufficiently explored. To address this gap, we evaluate MLLMs on paired solvable and underdetermined geometry problems, jointly assessing problem-solving ability and underdetermination recognition. Specifically, we propose Key-Information Removal (KIR), which uses source solutions to guide the removal of essential visual evidence from geometry images. To ensure the validity of the resulting problems, automatic problem auditing, targeted repair, and human verification are further applied so that the remaining conditions no longer uniquely determine the target. Through this process, we construct GeoFlip, a benchmark comprising hundreds of solvable–underdetermined geometry problem pairs. Experiments on GeoFlip reveal a substantial gap between problem-solving ability and underdetermination recognition, with models frequently repeating source answers even after necessary evidence has been removed. Moreover, underdetermination recognition varies substantially with the type and modality of removed evidence, while model scaling and reasoning modes do not consistently improve underdetermination recognition. Finally, to improve reasoning reliability in both solvable and underdetermined settings, we develop a unified symbolic reasoning pipeline that combines multimodal formalization with geometric deduction and algebraic computation. Compared with direct answering, our pipeline raises pair success rate from 0.3% to 59.5% with Claude Opus 5 and from 35.1% to 57.2% with GPT-5.5.

Primary area:
datasets and benchmarks
Keywords:
mathematical reasoning, multimodal large language models, unsolvable problems, reasoning reliability, neuro-symbolic reasoning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_GbRsOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper raises an interesting question: can MLLMs identify geometric underdeterminedness—situations where a problem lacks a unique solution due to the removal of critical visual evidence? The authors introduce a custom data construction pipeline and discover that even powerful models do not reliably detect such underdetermined cases. Furthermore, they demonstrate that combining multimodal analysis with symbolic reasoning improves success rates.

Critical strengths
  1. This paper addresses a significant problem; unlike previous studies that focused solely on accuracy, this work examines the conditions underlying geometric reasoning. This approach is practically valuable for assessing whether a model remains faithful to the provided visual evidence.
  2. The study is robust and yields meaningful conclusions, highlighting issues within current Multimodal Large Language Models (MLLMs) that had previously gone unnoticed.
  3. Based on experimental observations from the benchmark, the authors propose a method to address these specific issues.
Critical weaknesses

In my view, this article has no major weaknesses. A small suggestion: you could include a schematic diagram of the solution method presented in this paper in the appendix, which would help readers’ understanding.

Questions for authors
  1. I am not very familiar with PyEuclid. Could you explain how it differs from other symbolic solvers like InterGPS or AutoGPS, and why this paper chose PyEuclid as its symbolic solver?
  2. The paper reports the overall Precision (P), Recall (R), and F1-score for the multimodal formalization. I would like to know: how many problems were parsed completely and correctly? For underdetermined problems, how many of the parsed results incorrectly reconstructed the omitted conditions? And how do these critical errors impact the final results?
  3. How are final answers ultimately classified into the categories NV, US, and EF? Specifically, for the identification of US, is the process based on rule matching, manual annotation, or an LLM judge?
2 comments
AuthorsOpenReview ↗

Response Letter for Reviewer GbRs (Part 1)

We thank the reviewer for the positive assessment and the helpful suggestions. We clarify the existing method schematic, our choice of symbolic solver, and the formalization and evaluation results below.

Suggestion: Schematic diagram of the solution method.

Thank you for the suggestion. Figure 3 in Section 4 (p. 6) illustrates the complete pipeline: multimodal formalization of each problem, initialization of the problem state, interaction between the deductive database (DD) and algebraic system (AS), and the SOLVED/UNDETERMINED outputs. We will make the explanation of this figure more explicit and add a cross-reference in the appendix to help readers follow the solution procedure.

Q1: Differences among PyEuclid, Inter-GPS, and AutoGPS, and the choice of PyEuclid.

We selected PyEuclid for its strong geometry-solving accuracy and simple integration as a purely symbolic backend. We compared Inter-GPS, AutoGPS, and PyEuclid, as summarized in Table 1, and found that both AutoGPS and PyEuclid report high accuracy with GT formal language. AutoGPS includes multimodal formalization and iterative feedback/refinement, while PyEuclid directly performs geometric deduction and algebraic computation from formal language, without LLM calls during reasoning. We also tried the AutoGPS pipeline. In our trials, its PGDP-Net diagram parser showed limited parsing accuracy on the underdetermined counterparts. We therefore use an MLLM for multimodal formalization, followed by PyEuclid for purely symbolic reasoning. This design keeps parsing and symbolic reasoning separate and lets us use the same backend with either predicted or GT formal language.

Table 1. Method comparison and reported Geometry3K accuracy with GT formal language (%). Scores follow the original evaluation protocols.

AspectInter-GPSAutoGPSPyEuclid
InputTextual question and geometry imageTextual question and geometry imageFormal language
MethodText/diagram parsing and symbolic reasoningPGDP-Net diagram parsing, MLLM formalization/refinement, and symbolic reasoningSymbolic reasoning
Accuracy78.394.594.7

We will add this comparison and the rationale for choosing PyEuclid to the main text.

AuthorsOpenReview ↗

Response Letter for Reviewer GbRs (Part 2)

Q2: Problem-level formalization accuracy and reconstruction of omitted conditions.

We supplement fact-level P/R/F1 with problem-level exact-match accuracy and an analysis of reconstructed omitted facts on all 587 GeoFlip pairs.

Problem-level exact match. We compare the predicted and GT formal languages as fact sets after deduplication, canonicalization of equivalent expressions, arithmetic normalization, and alignment of point identifiers by geometric role. A problem matches only if the normalized fact sets are identical and the target gg is correct. Table 2 shows exact-match accuracy ranging from 34.58% to 41.57%. Complete, correct formalization remains challenging despite higher fact-level F1 scores.

Table 2. Formalization exact-match accuracy (count and percentage).

ProblemOurs (GPT-5.5)Ours (Claude Opus 5)
Source GP\mathrm{GP}213 (36.29%)244 (41.57%)
Counterpart GP∗\mathrm{GP}^{\ast}203 (34.58%)203 (34.58%)

Reconstruction of omitted facts. For each pair, we compare the GT fact sets of the source GP\mathrm{GP} and its counterpart GP∗\mathrm{GP}^{\ast} to identify omitted facts. A GP∗\mathrm{GP}^{\ast} is considered a reconstruction case if at least one omitted fact is reconstructed in its predicted fact set. Table 3 summarizes reconstruction cases and their solver outcomes, with percentages computed over all counterparts. Among reconstruction cases, 87.1% (54/62) for GPT-5.5 and 92.4% (73/79) for Claude Opus 5 yield numerical answers despite counterpart underdetermination, contributing to FailRGP∗\mathrm{FailR}_{\mathrm{GP}^{\ast}}.

Table 3. Reconstruction of omitted facts and associated solver outcomes.

OutcomeOurs (GPT-5.5)Ours (Claude Opus 5)
Total62 (10.56%)79 (13.46%)
EFRGP∗\mathrm{EFR}_{\mathrm{GP}^{\ast}}5 (0.85%)2 (0.34%)
FailRGP∗\mathrm{FailR}_{\mathrm{GP}^{\ast}}54 (9.20%)73 (12.44%)
SuccRGP∗\mathrm{SuccR}_{\mathrm{GP}^{\ast}}3 (0.51%)4 (0.68%)

We will include this analysis in the revised manuscript.

Q3: How NV, US, and EF are assigned.

We assign NV, US, and EF automatically using boxed-answer extraction and numerical parsing. For each counterpart GP∗\mathrm{GP}^{\ast}, the procedure is:

  1. Extract the final answer. The parser extracts the last complete, non-empty boxed answer. If none can be extracted, the response is EF.
  2. Classify the extracted answer. The math-verify-based numerical parser attempts to extract a finite real value from the boxed content. The response is classified as NV if parsing succeeds and as US otherwise.

To check for incorrect US labels, we audited 574 responses originally classified as US from the 14 MLLMs in the main results. GPT-6.1 Sol reviewed only the text inside each final answer box, and all reclassifications were confirmed by human review. Table 4 summarizes the audit: only one GPT-4o response required reclassification as NV, while 573 were retained as US. The US misclassification rate is therefore low (0.17%).

Table 4. Reviewed labels for US predictions on underdetermined counterparts.

ModelNVUS
Claude Sonnet 4.6029
Qwen3-VL-235B-A22B-Instruct020
InternVL3.5-241B-A28B016
Claude Opus 505
GPT-4o1142
GLM-5.3-Flash021
GPT-5.50242
Claude Haiku 4.5010
Gemini 3.8 Flash012
Kimi K3029
DeepSeek-V4-Flash-Vision-Exp025
Qwen3-VL-235B-A22B-Thinking04
Qwen3.8-Flash03
Qwen3.8-Max015
Total1573

We will document this classification procedure and the supplementary audit in Section 5.1 and the evaluation appendix.

We thank the reviewer again for the constructive suggestions. The clearer figure references and these additional analyses will improve the reproducibility and accessibility of the paper.

Reviewer_cKhmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GeoFlip constructs 587 solvable--underdetermined geometry problem pairs from Geometry3K, where a solvable question is changed by removing solution-relevant visual evidence and text, making the counterpart non-uniquely solvable. Key-Information Removal (KIR), which uses source solutions to guide the removal of essential visual evidence from geometry images. To ensure the validity, automatic problem auditing, targeted repair, and human verification are used. The benchmark reports severe failures in models’ recognition of underdetermination. THe developed unified symbolic reasoning pipeline raises pair success to 57.2% with GPT-5.5 and 59.5% with Claude Opus 5.

Critical strengths

1、Well-targeted paired benchmark. Human-approved data and directly test whether visual evidence is used rather than whether the model merely obtains a numerical answer. Separate image-only, symbol-only, and joint removals reveal a hierarchy of potential failure modes.

2、Constructive comparisons. Pair Success Rate requires both correct source solving and correct rejection of an underdetermined case, the symbolic baseline and ground-truth-formalization upper bound is helpful to distinguish perception or formalization errors from solver-side limits (Table 2).

3、A detailed failure taxonomy. Distinguishing image-only, question-only, joint, and symbol-versus-text deletions exposes how mathematical vision dependence changes abstention behavior.

Critical weaknesses

1、Failure to derive maybe not a proof of underdetermination. Section 4 returns UNDERDETERMINED when PyEuclid can no longer derive facts while the target is unresolved. This only establishes failure of the implemented rule system to solve the problem, unless completeness for the restricted geometry language is shown. Is it possible that a solved but hard problem could be falsely regarded as unresolved ?

2、Diversity and coverage of data. All 587 pairs derive from the Geometry3K test split, with mostly local notation removal (83.8% image-only), the diversity and number of coverage case maybe not so enough to support all experiments and analysis. Could the experiments and pair constructing extand to more datasets or data categories?

3、The novelty claims. MM-AQA (Madhusudhan et al., 2026) , SpatialUncertain (Zhang et al., 2026) already constructs unanswerable multimodal tasks from answerable instances. GeoFlip's defensible novelty should be claimed more clearly, it maybe not the first discovery of visual-evidence insufficiency or multimodal abstention .

Questions for authors

1、Can the authors characterize the completeness of PyEuclid for the accepted problem grammar, or provide a constructive multi-solution cases for a large representative subset of the underdetermined counterparts? How many genuinely solvable examples are returned UNDERDETERMINED?

2、Compared with MM-AQA and SpatialUncertain, what precisely is new in data quality, causal isolation of missing visual evidence, or evaluation? Would other existing abstention-capable baselines perform differently on matched GeoFlip pairs?

5 comments
AuthorsOpenReview ↗

Response Letter for Reviewer cKhm (Part 1)

We thank the reviewer for the thoughtful comments on mathematical validity, benchmark coverage, and positioning relative to related work. We respond to each concern below.

W1: Risk of misclassifying solvable problems as underdetermined.

We thank the reviewer for raising this concern. The risk of misclassifying a solvable problem as underdetermined depends on the solver's deductive coverage and the accuracy of the input formalization. To quantify this risk in our benchmark setting, we evaluated all 587 solvable source geometry problems (GPs) (Table 1).

When the solver returns SOLVED with a numerical value, we classify the result as Correct if it matches the ground-truth target and Incorrect otherwise. Every UNDETERMINED output on this set is a false-positive underdetermination decision.

Table 1. Solver outcomes on solvable GPs.

ModelCorrectIncorrectUndeterminedTimeout
Ours (GT)94.2%0.0%1.0%4.8%
Ours (GPT-5.5)80.6%5.1%3.9%10.4%
Ours (Claude Opus 5)87.2%2.6%2.4%7.8%

Ours (GT) evaluates the symbolic solver's problem-solving ability with human-verified formalizations, while the other two settings evaluate the full pipeline with MLLM-generated formalizations. Ours (GT) correctly solves 94.2% of the source problems and returns UNDETERMINED for only 1.0%. With formalizations from GPT-5.5 and Claude Opus 5, the false-positive underdetermination rates increase to 3.9% and 2.4%, respectively. Thus, Ours (GT) rarely misclassifies solvable problems as underdetermined, while formalization quality also affects our pipeline. More challenging problems, such as Olympiad-level geometry, may require a stronger symbolic solver. Developing such a solver is beyond the scope of this work.

We will add this analysis to Section 5.

Q1: Constructive multi-solution cases.

We have completed manual annotation for all retained counterparts. The annotation procedure focused on constructing two admissible geometric configurations that satisfy the same retained conditions but yield different values for the requested target.

Formally, for counterpart ii, let CiC_i contain all conditions retained in the textual question Qi∗Q_i^{\ast} and geometry image Ii∗I_i^{\ast}, including geometric relations and numerical constraints. Let gig_i denote the requested target. We accepted two candidate configurations Mi(1)\mathcal{M}_i^{(1)} and Mi(2)\mathcal{M}_i^{(2)} only after manually verifying that

\mathcal{M}_i^{(2)} \models C_i,\qquad g_i(\mathcal{M}_i^{(1)}) \ne g_i(\mathcal{M}_i^{(2)}).$$ Each verified pair establishes non-uniqueness of the requested target independently of PyEuclid's termination. Annotators used two methods to construct candidate configurations: 1. **Varying the removed condition.** Annotators varied the removed angle, length, or geometric relation while preserving all retained conditions, and checked whether the target value changed. 2. **Coordinate construction.** Annotators introduced a coordinate system and expressed all retained geometric conditions as equations to constrain the point coordinates. They assigned coordinates to a subset of points, solved for the remaining point coordinates, and computed the target value. They then repeated the procedure with a different initial coordinate assignment and checked whether the two target values differed. **Example (`prob_6189` in the supplementary material).** The source problem gives a right triangle $ABC$ with $\angle C=90^\circ$, $BC=24$, and $AB=74$. The task is to find $CA$. KIR removes only the text notation $74$ indicating the length of $AB$, leaving $\angle C=90^\circ$ and $BC=24$ unchanged. Since $AB$ is now unspecified, we can set $AB=26$ or $AB=40$, which yield $CA=10$ or $CA=32$, respectively. Both satisfy the retained conditions and form Pythagorean triples, but give different values of $CA$. Thus, $CA$ is not uniquely determined. We will include the annotation procedure and representative constructions in the revised manuscript and appendix.
AuthorsOpenReview ↗

Response Letter for Reviewer cKhm (Part 2)

W2: Evidence diversity and dataset coverage.

We would like to clarify that GeoFlip covers multiple types of geometric evidence, target quantities, and diagrams.

Despite this predominance of image-only edits, GeoFlip covers diverse types of visual evidence. Text notations (Appendix A, Figure 8(b)) include point, line, and angle labels, together with numerical or variable annotations for lengths, angles, and areas. Symbol notations (Figure 8(a)) include marks for perpendicularity, equal angles, equal lengths, and parallelism, as well as annotation leaders and length indicators. These two notation types yield the three removal groups summarized in Table 2.

Table 2. Types of removed visual evidence.

Text-onlySymbol-onlyBoth
66.8%18.7%14.5%

Diversity also extends to the requested targets and diagram types, as summarized in Tables 3 and 4. Categories can overlap, so percentages need not sum to 100%.

Table 3. Target quantities in GeoFlip.

Target quantityShare
Length50.3%
Angle39.4%
Area9.0%
Ratio2.0%
Other0.5%

Table 4. Diagram types in GeoFlip.

Diagram typeShare
Triangles47.9%
Circles24.2%
Lines13.1%
Parallelograms10.7%
Trapezoids4.6%
Rectangles4.6%
Other4.4%
Rhombi2.6%
Squares1.4%

Although the main benchmark focuses on missing visual evidence, our ablation experiments also cover different removal modalities. We randomly sample 250 source problems and use the same KIR pipeline to construct three variants of each problem by removing evidence from the question, the image, or both.

Our experiments currently cover only Geometry3K because KIR requires each geometry image II and textual question QQ to be accompanied by a solution program SP\mathrm{SP} and visual evidence annotations L={(vk,bk)}k=1m(I)\mathcal{L}=\{(v_k,b_k)\}_{k=1}^{m(I)}. Each annotation pairs a visible text or symbol notation vkv_k with its bounding box bkb_k. These annotations help maintain data quality by grounding evidence selection in the source solution and enabling precise edits to the intended regions. However, datasets such as GeoQA and SolidGeo do not provide the localized annotations required by KIR, so we have not conducted experiments on them. We acknowledge that the benchmark's limited scale and single-dataset coverage remain limitations. Cross-dataset generalization remains untested. In future work, we will explore reducing reliance on some of these annotations to extend the pipeline to larger and more diverse datasets.

AuthorsOpenReview ↗

Response Letter for Reviewer cKhm (Part 3)

W3 / Q2: Novelty and abstention baselines.

We do not claim to be the first to construct unanswerable multimodal tasks. Our first systematic study concerns underdetermination recognition after visual evidence removal in multimodal geometry reasoning. Our Related Work discusses MM-AQA [1]. Table 5 extends this comparison to SpatialUncertain [2].

  • Visual evidence control. MM-AQA includes broad visual transformations such as cropping, masking, and blurring. SpatialUncertain introduces occluders or changes camera viewpoints. For geometry diagrams, broad cropping or masking can damage the geometric structure or obscure unrelated conditions. KIR uses source solutions to locate and erase individual text or symbol notations. Compared with broad cropping or masking, this fine-grained, solution-guided removal targets specific geometric conditions while preserving the geometric primitives and unrelated annotations, keeping the diagram intact and meaningful.
  • Data quality. Our manual annotation provides rigorous mathematical proofs of non-uniqueness by verifying two admissible geometric configurations with different target values (see Q1).

Table 5. Comparison of MM-AQA, SpatialUncertain, and GeoFlip.

AspectMM-AQASpatialUncertainGeoFlip
DomainSTEM and document QA3D spatial QAPlane geometry
Visual evidence controlCoarse, question-aware visual transformationsObject occlusion and camera-viewpoint changesFine-grained, solution-guided visual evidence removal with diagram preservation
Data qualityDual-VLM consensus filtering and stratified human validationGeometry- and projection-derived labels with human quality controlAutomatic auditing and human review. Rigorous mathematical proofs of non-uniqueness (see Q1)
MethodMulti-agent reasoning with verificationStructured prompting and LoRA fine-tuningUnified symbolic reasoning
EvaluationAnswerable and unanswerable accuracy, abstention rate, and Matthews correlation coefficientAnswerable, unanswerable, and overall accuracy. ViewSelect and Abstain-then-ViewSelect accuracySuccess Rate on source and counterpart problems, Pair Success Rate, and Answer Repetition Rate

Following your suggestion, we additionally evaluated three existing abstention-capable baselines on GeoFlip pairs, all using GPT-5.5 (Table 6).

  • Self-Consistency Abstention [1]. For each problem, we independently sample 10 responses. Valid answers include numerical values and statements of underdetermination or abstention. With at least six valid answers, we retain an answer receiving six or more votes, otherwise abstaining. Fewer than six valid answers yield EF.
  • P(True)-style self-evaluation [3]. The same model estimates its candidate answer's correctness probability pp given the problem. We retain the candidate answer when p≥0.5p \geq 0.5 and abstain when p<0.5p < 0.5.
  • TUBench [4]. We adapt its prompt for source GPs and their underdetermined counterparts. The model first returns Unanswerable if information is insufficient or a numerical answer otherwise, then explains its response.

Table 6. Abstention baselines on GeoFlip. All metrics are percentages.

SettingSuccRGP↑\mathrm{SuccR}_{\mathrm{GP}}\uparrowSuccRGP∗↑\mathrm{SuccR}_{\mathrm{GP}^{\ast}}\uparrowPSR↑\mathrm{PSR}\uparrowARR↓\mathrm{ARR}\downarrow
GPT-5.586.7%41.2%35.1%26.7%
Self-Consistency Abstention88.1%55.2%46.2%26.9%
P(True)71.7%60.8%47.5%18.1%
TUBench49.9%77.7%43.3%20.3%
Ours (GPT-5.5)80.6%65.9%57.2%11.9%

As shown in Table 6, Self-Consistency improves both problem solving on source GPs and underdetermination recognition on their counterparts over direct answering. TUBench achieves the highest SuccRGP∗\mathrm{SuccR}_{\mathrm{GP}^{\ast}}. Despite their strong underdetermination recognition, P(True) and TUBench yield lower SuccRGP\mathrm{SuccR}_{\mathrm{GP}} than direct answering. PSR captures this trade-off by requiring success on both members of each pair. Our method achieves the highest PSR and lowest ARR.

In the revised manuscript, we will discuss SpatialUncertain in Related Work and include these baseline experiments in the main results.

  1. Madhusudhan, Nishanth, et al. "Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems." arXiv:2604.14799 (2026).
  2. Zhang, Yue, et al. "Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?" arXiv:2605.30557 (2026).
  3. Kadavath, Saurav, et al. "Language Models (Mostly) Know What They Know." arXiv:2207.05221 (2022).
  4. He, Xingwei, et al. "TUBench: Benchmark Large Vision-Language Models on Trustworthiness With Unanswerable Questions." IEEE Transactions on Dependable and Secure Computing (2026).
Reviewer_4xq9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether multimodal large language models can identify insufficient conditions in geometry problems—that is, determine when the answer is no longer uniquely determined due to the absence of key evidence. The authors propose the Key Information Removal (KIR) method, which involves removing numerical values or geometric relationship labels from diagrams and examining other possible solution paths to construct a benchmark comprising 587 pairs of “solvable–underdetermined” problems. They also propose an inference scheme that combines multimodal formalization with symbolic solvers. The main contribution is the extension of geometric reasoning evaluation to the assessment of condition sufficiency, using paired evaluations to simultaneously examine both correct problem-solving and the ability to identify insufficient information. For validation, all problem pairs were reviewed by three evaluators, and experiments covered 18 baseline models. The results show that strong problem-solving ability does not guarantee reliable underdetermination recognition, and neither scaling up the model nor increasing the length of the reasoning chain led to consistent improvements. The authors’ method improved the paired success rates of GPT-5.5 and Claude Opus 5 from 35.1% and 0.3% to 57.2% and 59.5%, respectively, indicating that explicit condition extraction and symbolic reasoning help improve reliability, though the effectiveness remains constrained by the accuracy of the formalization.

Critical strengths

This paper extends the focus of geometric reasoning assessment to whether models can identify cases of insufficient information, which holds clear research value. Its “solvable–underdetermined” paired design allows for the simultaneous evaluation of correct solutions and appropriate refusals to answer, thereby preventing models from achieving high scores by uniformly responding with “insufficient information.” The data construction not only removes conditions that rely on the original solution path but also examines alternative solution paths; furthermore, all 587 sets of problems underwent a three-person review, which enhances the benchmark’s credibility. Experiments across multiple model types reveal a significant gap between problem-solving ability and the ability to identify underdetermination, while methods combining formal and symbolic reasoning further demonstrate feasible directions for addressing this issue, endowing this work with both evaluation value and methodological insights.

Critical weaknesses

The main issue is that the proposed method classifies a state—where the symbolic solver cannot continue its derivation and the goal remains unsolved—as “underdetermined”; however, the paper does not adequately explain the scope of its rules or provide corresponding proofs of non-uniqueness. Therefore, it cannot yet be ruled out that insufficient solving ability might be mistakenly interpreted as insufficient conditions. This implies that some of the performance improvements may stem from conservative outputs following a solving failure, and further verification is needed to determine whether the method truly achieves reliable underdetermination detection. Furthermore, the paper explains the phenomenon of the system outputting the same answer even after conditions are removed as “memory”; however, the existing evidence is insufficient to distinguish between causes such as memorization of training data, visual misreading, and the completion of conditions based on graphical appearance. Consequently, while the experiments provide strong support for reliability defects at the behavioral level, the explanation of the underlying mechanisms and principles for improvement still needs to be strengthened.

Questions for authors

Section 4 derives a state where the process has terminated but the objective remains unsolved, labeling it as UNDETERMINED. Does this determination provide sufficient assurance within the scope of the problems under study, or does it merely indicate that the current rules cannot solve the problem? For the 587 modified problems ultimately retained, did the manual review document evidence proving that the objective was not unique—such as two geometric configurations that satisfy the remaining conditions but have different objective values, or free parameters that can alter the objective value? Some models in Table 2 exhibit a high failure rate in extracting answers for underdetermined problems, and the evaluation counts answers not provided in the specified boxed format as extraction failures. Of these responses, how many correctly pointed out the insufficient conditions in the body of the text? If grading were based on the actual semantic meaning of the responses, would the comparisons among the main models and the performance gaps still hold?

3 comments
AuthorsOpenReview ↗

Response Letter for Reviewer 4xq9 (Part 1)

We thank the reviewer for the careful assessment.

W1 / Q1: Solver termination versus mathematical underdetermination.

We thank the reviewer for raising this concern. The risk of misclassifying a solvable problem as underdetermined depends on the solver's deductive coverage and the accuracy of the input formalization. To quantify this risk in our benchmark setting, we evaluated all 587 solvable source geometry problems (GPs) (Table 1).

When the solver returns SOLVED with a numerical value, we classify the result as Correct if it matches the ground-truth target and Incorrect otherwise. Every UNDETERMINED output on this set is a false-positive underdetermination decision.

Table 1. Solver outcomes on solvable GPs.

ModelCorrectIncorrectUndeterminedTimeout
Ours (GT)94.2%0.0%1.0%4.8%
Ours (GPT-5.5)80.6%5.1%3.9%10.4%
Ours (Claude Opus 5)87.2%2.6%2.4%7.8%

Ours (GT) evaluates the symbolic solver's problem-solving ability with human-verified formalizations, while the other two settings evaluate the full pipeline with MLLM-generated formalizations. Ours (GT) correctly solves 94.2% of the source problems and returns UNDETERMINED for only 1.0%. With formalizations from GPT-5.5 and Claude Opus 5, the false-positive underdetermination rates increase to 3.9% and 2.4%, respectively. Thus, Ours (GT) rarely misclassifies solvable problems as underdetermined, while formalization quality also affects our pipeline.

We will add this analysis to Section 5.

W2: Interpretation of answer repetition and the term “memorization.”

Thank you for highlighting the need to clarify the interpretation of answer repetition and our use of the term “memorization.” We have conducted a response-level analysis to examine the behaviors associated with answer repetition. We identified 996 pairs of model responses across seven models in which a model returned the same numerical answer to the source GP\mathrm{GP} and its counterpart GP∗\mathrm{GP}^{\ast}. We defined the following four behavior categories:

  • Visual Recognition Errors (VRE). The response misinterprets text, numerical values, geometric relations, objects, or annotation locations in the geometry image and uses the misinterpreted information in its solution.
  • Appearance-Based Condition Completion (ABCC). The response supplies unstated conditions based on the geometry image's appearance, such as line orientation, shape, relative lengths, proportions, or apparent geometric relations.
  • Pattern-Based Assumptions (PBA). The response treats the current problem as a familiar problem type, special configuration, or common numerical pattern and adopts additional conditions normally associated with that pattern.
  • Other Errors (OE). The response makes other errors in deriving relations, applying theorems or formulas, or performing numerical calculations.

We then used GPT-6.1 Sol to classify each response pair. The inputs included the textual questions, complete model responses, geometry images, and their formal languages for both GP\mathrm{GP} and GP∗\mathrm{GP}^{\ast}, together with the removal rationale RR\mathrm{RR}. Multiple labels were allowed when a response exhibited more than one behavior.

Table 2 reports category counts and percentages for each model. The Total row uses all response pairs as the denominator. Categories may overlap, so percentages need not sum to 100%. Across all response pairs, PBA (46.9%) and ABCC (43.6%) are the most frequent behaviors, followed by VRE (33.4%) and OE (7.9%). The most frequent behavior also varies across models. VRE accounts for 66.9% of GPT-4o cases, while ABCC and PBA each occur in more than 70% of the response pairs for both Qwen3.8-Flash and Qwen3.8-Max. These results associate answer repetition with several observable behaviors.

Table 2. Behaviors associated with answer repetition.

Model (nn)VREABCCPBAOE
GPT-4o (130)87 (66.9%)25 (19.2%)37 (28.5%)18 (13.8%)
Claude Sonnet 4.6 (160)45 (28.1%)75 (46.9%)54 (33.8%)20 (12.5%)
Kimi K3 (149)42 (28.2%)60 (40.3%)37 (24.8%)11 (7.4%)
DeepSeek-V4-Flash-Vision-Exp (142)56 (39.4%)28 (19.7%)64 (45.1%)9 (6.3%)
Qwen3-VL-235B-A22B-Instruct (144)45 (31.3%)51 (35.4%)75 (52.1%)14 (9.7%)
Qwen3.8-Flash (107)22 (20.6%)75 (70.1%)77 (72.0%)3 (2.8%)
Qwen3.8-Max (164)36 (22.0%)120 (73.2%)123 (75.0%)4 (2.4%)
Total (996)333 (33.4%)434 (43.6%)467 (46.9%)79 (7.9%)

We will add the classification procedure and results to the manuscript and replace “memorization” with answer repetition to clarify the interpretation of ARR.

AuthorsOpenReview ↗

Response Letter for Reviewer 4xq9 (Part 2)

Q2: Non-uniqueness evidence for the underdetermined counterparts.

We have completed manual annotation for all counterparts. The annotation procedure focused on constructing two admissible geometric configurations that satisfy the same retained conditions but yield different values for the requested target.

Formally, for counterpart ii, let CiC_i contain all conditions retained in the textual question Qi∗Q_i^{\ast} and geometry image Ii∗I_i^{\ast}, including geometric relations and numerical constraints. Let gig_i denote the requested target. We accepted two candidate configurations Mi(1)\mathcal{M}_i^{(1)} and Mi(2)\mathcal{M}_i^{(2)} only after manually verifying that

\mathcal{M}_i^{(2)} \models C_i,\qquad g_i(\mathcal{M}_i^{(1)}) \ne g_i(\mathcal{M}_i^{(2)}).$$ Annotators used two methods to construct candidate configurations: 1. **Varying the removed condition.** Annotators varied the removed angle, length, or geometric relation while preserving all retained conditions, and checked whether the target value changed. 2. **Coordinate construction.** Annotators introduced a coordinate system and expressed all retained geometric conditions as equations to constrain the point coordinates. They assigned coordinates to a subset of points, solved for the remaining point coordinates, and computed the target value. They then repeated the procedure with a different initial coordinate assignment and checked whether the two target values differed. **Example (`prob_6189` in the supplementary material).** The source problem gives a right triangle $ABC$ with $\angle C=90^\circ$, $BC=24$, and $AB=74$. The task is to find $CA$. KIR removes only the text notation $74$ indicating the length of $AB$, leaving $\angle C=90^\circ$ and $BC=24$ unchanged. Since $AB$ is now unspecified, we can set $AB=26$ or $AB=40$, which yield $CA=10$ or $CA=32$, respectively. Both satisfy the retained conditions and form Pythagorean triples, but give different values of $CA$. Thus, $CA$ is not uniquely determined. We will include the annotation procedure and representative constructions in the revised manuscript and appendix.
AuthorsOpenReview ↗

Response Letter for Reviewer 4xq9 (Part 3)

Q3: Semantic correctness among extraction failures.

We appreciate this suggestion and have conducted a supplementary semantic re-evaluation of the responses originally classified as extraction failures (EFs).

We randomly selected nine models and reviewed their 1,576 original EF responses from the format-based evaluation. We then used GPT-6.1 Sol to assess each response individually, providing the textual question Q∗Q^{\ast}, geometry image I∗I^{\ast}, formal language of I∗I^{\ast}, removal rationale RR\mathrm{RR}, and complete model response as inputs. We incorporated the following two criteria into the assessment instructions:

  1. Correct recognition of underdetermination. The response recognizes that the available information is insufficient, for example by identifying a missing condition relevant to the requested target.
  2. Format-independent assessment. No fixed answer format is required. A missing \boxed{} statement, or an unfinished response does not automatically disqualify a response that satisfies the semantic criterion above.

Through this semantic evaluation followed by human verification, we recovered 197 responses that correctly recognized underdetermination. We reclassified these responses as correct US predictions and recomputed the evaluation metrics on the same problem pairs with unchanged denominators. Table 3 reports the results. Original EF denotes the original extraction-failure count, and Recovered EF denotes the count accepted after semantic review. Δ\Delta denotes the change from the original format-based evaluation in percentage points.

Table 3. Changes after semantic-guided evaluation of extraction failures

ModelOriginal EFRecovered EFΔEFRGP∗\Delta \mathrm{EFR}_{\mathrm{GP}^{\ast}}ΔSuccRGP∗\Delta \mathrm{SuccR}_{\mathrm{GP}^{\ast}}ΔPSR\Delta \mathrm{PSR}
Qwen3.8-Flash35947−8.0-8.0+8.0+8.0+5.5+5.5
DeepSeek-V4-Flash-Vision-Exp14835−6.0-6.0+6.0+6.0+4.6+4.6
InternVL3.5-241B-A28B28633−5.6-5.6+5.6+5.6+3.6+3.6
Qwen3.8-Max28830−5.1-5.1+5.1+5.1+4.3+4.3
Qwen3-VL-235B-A22B-Instruct9227−4.6-4.6+4.6+4.6+3.4+3.4
GPT-4o2011−1.9-1.9+1.9+1.9+1.0+1.0
Kimi K3319−1.5-1.5+1.5+1.5+0.7+0.7
Gemini 3.8 Flash3514−0.7-0.7+0.7+0.7+0.7+0.7
Claude Sonnet 4.611−0.2-0.2+0.2+0.2+0.2+0.2
Total1576197

The original format-based evaluation thus underestimates underdetermination recognition for these models. After semantic recovery, SuccRGP∗\mathrm{SuccR}_{\mathrm{GP}^{\ast}} increases by 0.2--8.0 percentage points, and PSR increases by 0.2--5.5 percentage points. The gap between problem-solving ability and underdetermination recognition remains substantial. These results preserve our main conclusion that strong problem-solving ability does not guarantee reliable underdetermination recognition.

The key model comparisons also hold after this supplementary EF review, although some rankings change. GPT-5.5 remains the strongest direct-answering MLLM on SuccRGP∗\mathrm{SuccR}_{\mathrm{GP}^{\ast}} (41.2%) and PSR (35.1%). Ours (GPT-5.5) achieves 65.9% and 57.2%, retaining advantages of 24.7 and 22.1 percentage points, respectively. Ours (Claude Opus 5) also retains a higher PSR (59.5%).

We will add the semantic evaluation results and review criteria to the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.