- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies which explicit geometry representation is the most useful for solid-geometry problem solving with vision-language models (VLMs). To figure out this question, the paper introduces GeoText, which is a structured representation for diagnostic analysis over four geometric input information: view elements, 3D shape structure, given measurements and constraints, and spatial relations. To evaluate on this question, a new benchmark called OrthoMindBench containing mostly three-view drawing questions is constructed. The paper evaluates on the constructed benchmarks across different VLM models, and comes to the conclusion that explicit geometry information given to the model can substantially reduce the contribution of images for solid-geometry problem solving.
Critical strengths
-
The paper presents a valuable study of analyzing how much each geometric component contribute to the solid-geometry problem solving question with VLMs. The conclusion of "explicit geometry information dominates the benefits from images" seems to be natural, but can also be informative to provide guidance for future works on solving mathematical geometry problems.
-
The paper constructs OrthoMindBench, which can be useful for future research on solid-geometry problems, especially for the controlled experiments for structured representations to study the individual component contribution to the performance of solving mathematical geometry problems.
Critical weaknesses
-
Compared to existing benchmarks on solid-geometry tasks like SolidGeo [1], the questions in the constructed OrthoMind-Bench seems to have limited diversity. In Lines 147-148, the paper states that over 80% of the questions are three-view drawings, and the rest of the questions being in unspecified categories. In comparison, SolidGeo states in its Section 3.1 that the questions have 8 fine-grained domains, and are categorized into three levels of difficulty. Therefore, the constructed benchmark seems to be less comprehensive and carefully curated than the existing benchmark.
-
The details of the proposed benchmark is also not clear presented. For example, in Lines 149-150, the paper states that "two batches of 1,420 and 3,937 problems have disjoint identifiers and are not difficulty strata". However, I feel confused about how are these two batches of benchmarking data separated, and why we even need two batches of data. The illustration of the dataset breakdown in Appendix C demonstrates that these two batches have very different coverage of the properties like "Relations", as one batch is all covered while the other only has less than 10% covered, which makes people feel even more confused on the purpose of having two batches. Also, there is seldom an example of how the question looks like. The only visual available in the paper is the three-view diagram in Figure 1. Other than that, for example, for the example question presented in Section A.2, the problem only has text description without any illustration, which makes audience very hard to get a taste of what the example questions really look like.
The above two points are the major weaknesses of the paper. Other weaknesses include that the title of the paper is confusing and kind of misleading, as the title kind of suggest 3D-aware spatial reasoning within VLMs, which is nowhere close to the actual topic studied in the paper, which is basically reasoning on geometry for AI-for-Math problems. This can attract many wrong audience to this paper, and I think the keywords like "math" needs to be added in the title.
[1] Wang et al. SolidGeo: Measuring Multimodal Spatial Math Reasoning in Solid Geometry. NeurIPS 2025.
Questions for authors
-
What is the specific question breakdown for the proposed OrthoMind-Bench? For example, for the rest of 20% questions beyond three-view drawings, what question types do they belong to? Also, how is the question diversity compared to existing benchmarks on K-12 solid-geometry questions including SolidGeo?
-
What is the purpose of having two batches for the proposed benchmark? What is each split of the benchmark tailored for?
Thank you for the clarification and revisions. Your response has largely addressed my concerns. However, I still find the empirical validation insufficient and do not consider the work ready for acceptance. I will therefore maintain my original rating.