- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper raises an interesting question: can MLLMs identify geometric underdeterminedness—situations where a problem lacks a unique solution due to the removal of critical visual evidence? The authors introduce a custom data construction pipeline and discover that even powerful models do not reliably detect such underdetermined cases. Furthermore, they demonstrate that combining multimodal analysis with symbolic reasoning improves success rates.
Critical strengths
- This paper addresses a significant problem; unlike previous studies that focused solely on accuracy, this work examines the conditions underlying geometric reasoning. This approach is practically valuable for assessing whether a model remains faithful to the provided visual evidence.
- The study is robust and yields meaningful conclusions, highlighting issues within current Multimodal Large Language Models (MLLMs) that had previously gone unnoticed.
- Based on experimental observations from the benchmark, the authors propose a method to address these specific issues.
Critical weaknesses
In my view, this article has no major weaknesses. A small suggestion: you could include a schematic diagram of the solution method presented in this paper in the appendix, which would help readers’ understanding.
Questions for authors
- I am not very familiar with PyEuclid. Could you explain how it differs from other symbolic solvers like InterGPS or AutoGPS, and why this paper chose PyEuclid as its symbolic solver?
- The paper reports the overall Precision (P), Recall (R), and F1-score for the multimodal formalization. I would like to know: how many problems were parsed completely and correctly? For underdetermined problems, how many of the parsed results incorrectly reconstructed the omitted conditions? And how do these critical errors impact the final results?
- How are final answers ultimately classified into the categories NV, US, and EF? Specifically, for the identification of US, is the process based on rule matching, manual annotation, or an LLM judge?
2 comments
Response Letter for Reviewer GbRs (Part 2)
Q2: Problem-level formalization accuracy and reconstruction of omitted conditions.
We supplement fact-level P/R/F1 with problem-level exact-match accuracy and an analysis of reconstructed omitted facts on all 587 GeoFlip pairs.
Problem-level exact match. We compare the predicted and GT formal languages as fact sets after deduplication, canonicalization of equivalent expressions, arithmetic normalization, and alignment of point identifiers by geometric role. A problem matches only if the normalized fact sets are identical and the target is correct. Table 2 shows exact-match accuracy ranging from 34.58% to 41.57%. Complete, correct formalization remains challenging despite higher fact-level F1 scores.
Table 2. Formalization exact-match accuracy (count and percentage).
| Problem | Ours (GPT-5.5) | Ours (Claude Opus 5) |
|---|---|---|
| Source | 213 (36.29%) | 244 (41.57%) |
| Counterpart | 203 (34.58%) | 203 (34.58%) |
Reconstruction of omitted facts. For each pair, we compare the GT fact sets of the source and its counterpart to identify omitted facts. A is considered a reconstruction case if at least one omitted fact is reconstructed in its predicted fact set. Table 3 summarizes reconstruction cases and their solver outcomes, with percentages computed over all counterparts. Among reconstruction cases, 87.1% (54/62) for GPT-5.5 and 92.4% (73/79) for Claude Opus 5 yield numerical answers despite counterpart underdetermination, contributing to .
Table 3. Reconstruction of omitted facts and associated solver outcomes.
| Outcome | Ours (GPT-5.5) | Ours (Claude Opus 5) |
|---|---|---|
| Total | 62 (10.56%) | 79 (13.46%) |
| 5 (0.85%) | 2 (0.34%) | |
| 54 (9.20%) | 73 (12.44%) | |
| 3 (0.51%) | 4 (0.68%) |
We will include this analysis in the revised manuscript.
Q3: How NV, US, and EF are assigned.
We assign NV, US, and EF automatically using boxed-answer extraction and numerical parsing. For each counterpart , the procedure is:
- Extract the final answer. The parser extracts the last complete, non-empty boxed answer. If none can be extracted, the response is EF.
- Classify the extracted answer. The
math-verify-based numerical parser attempts to extract a finite real value from the boxed content. The response is classified as NV if parsing succeeds and as US otherwise.
To check for incorrect US labels, we audited 574 responses originally classified as US from the 14 MLLMs in the main results. GPT-6.1 Sol reviewed only the text inside each final answer box, and all reclassifications were confirmed by human review. Table 4 summarizes the audit: only one GPT-4o response required reclassification as NV, while 573 were retained as US. The US misclassification rate is therefore low (0.17%).
Table 4. Reviewed labels for US predictions on underdetermined counterparts.
| Model | NV | US |
|---|---|---|
| Claude Sonnet 4.6 | 0 | 29 |
| Qwen3-VL-235B-A22B-Instruct | 0 | 20 |
| InternVL3.5-241B-A28B | 0 | 16 |
| Claude Opus 5 | 0 | 5 |
| GPT-4o | 1 | 142 |
| GLM-5.3-Flash | 0 | 21 |
| GPT-5.5 | 0 | 242 |
| Claude Haiku 4.5 | 0 | 10 |
| Gemini 3.8 Flash | 0 | 12 |
| Kimi K3 | 0 | 29 |
| DeepSeek-V4-Flash-Vision-Exp | 0 | 25 |
| Qwen3-VL-235B-A22B-Thinking | 0 | 4 |
| Qwen3.8-Flash | 0 | 3 |
| Qwen3.8-Max | 0 | 15 |
| Total | 1 | 573 |
We will document this classification procedure and the supplementary audit in Section 5.1 and the evaluation appendix.
We thank the reviewer again for the constructive suggestions. The clearer figure references and these additional analyses will improve the reproducibility and accessibility of the paper.
Response Letter for Reviewer GbRs (Part 1)
We thank the reviewer for the positive assessment and the helpful suggestions. We clarify the existing method schematic, our choice of symbolic solver, and the formalization and evaluation results below.
Suggestion: Schematic diagram of the solution method.
Thank you for the suggestion. Figure 3 in Section 4 (p. 6) illustrates the complete pipeline: multimodal formalization of each problem, initialization of the problem state, interaction between the deductive database (DD) and algebraic system (AS), and the SOLVED/UNDETERMINED outputs. We will make the explanation of this figure more explicit and add a cross-reference in the appendix to help readers follow the solution procedure.
Q1: Differences among PyEuclid, Inter-GPS, and AutoGPS, and the choice of PyEuclid.
We selected PyEuclid for its strong geometry-solving accuracy and simple integration as a purely symbolic backend. We compared Inter-GPS, AutoGPS, and PyEuclid, as summarized in Table 1, and found that both AutoGPS and PyEuclid report high accuracy with GT formal language. AutoGPS includes multimodal formalization and iterative feedback/refinement, while PyEuclid directly performs geometric deduction and algebraic computation from formal language, without LLM calls during reasoning. We also tried the AutoGPS pipeline. In our trials, its PGDP-Net diagram parser showed limited parsing accuracy on the underdetermined counterparts. We therefore use an MLLM for multimodal formalization, followed by PyEuclid for purely symbolic reasoning. This design keeps parsing and symbolic reasoning separate and lets us use the same backend with either predicted or GT formal language.
Table 1. Method comparison and reported Geometry3K accuracy with GT formal language (%). Scores follow the original evaluation protocols.
We will add this comparison and the rationale for choosing PyEuclid to the main text.