- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SolidGeoGen, a framework for constructing solid-geometry and spatial-reasoning data through verifiable symbolic generation, aiming to address the lack of reliable 3D structures, text-diagram alignment, and verifiable reasoning processes in existing multimodal geometry datasets. The framework establishes a formal system comprising 472 geometry description symbols and 680 theorem families. Through geometric structure instantiation, theorem-driven deduction, text-diagram generation, and symbolic verification, it automatically constructs problems with formal representations, geometric diagrams, answers, and reasoning traces. Building on this framework, the authors construct S2Bench, a benchmark containing 2,000 problems across 10 task categories, and S2Train, a training corpus containing 10,000 examples. They further propose VISTA, a training framework that uses structured visual hints and symbolic reasoning traces to generate supervised training data, followed by supervised fine-tuning and reinforcement learning to improve model capabilities. Experiments evaluate 21 language and multimodal models, with the best-performing model achieving 67.2% accuracy on S2Bench. Through VISTA training, the accuracy of Qwen3.5-9B improves from 14.0% to 42.4%, while Qwen3.5-27B improves from 19.4% to 50.9%. The trained models also achieve improvements on several external mathematical and spatial-reasoning benchmarks.
Critical strengths
-
Section 3 establishes a formal description language and an executable theorem system for solid geometry, and generates questions, diagrams, symbolic conditions, and reasoning traces from shared geometric parameters. Compared with approaches that generate text, images, and answers separately, this design explicitly represents geometric objects and their relationships during data construction, while using symbolic replay and deterministic computation to verify the consistency between answers and formal conditions. The paper implements 558 parameterized generation schemas covering tasks such as solid-geometry calculations, cross-sections, folding, projections, and voxel-based spatial relationships, providing a valuable technical foundation for constructing structured geometry data at scale.
-
Table 2 systematically evaluates 21 models, revealing substantial performance differences across 10 categories of solid-geometry and spatial-reasoning tasks. Table 3 further shows that VISTA-SFT improves the accuracy of the 9B and 27B models to 37.6% and 43.7%, respectively, while subsequent reinforcement learning further increases their accuracy to 42.4% and 50.9%. Table 4 also reports experimental results on seven external benchmarks, with relatively substantial improvements on MindCube and MMSI. These results demonstrate that the constructed data can effectively improve model performance on related tasks and provide some evidence of cross-benchmark transferability.
Critical weaknesses
-
Section 3 presents theorem-driven symbolic generation as a core contribution of SolidGeoGen, but many of the spatial-reasoning tasks constructed in the paper do not directly rely on the proposed theorem deduction system. According to Appendix B.2, the complete theorem library contains 680 theorem families, whereas the reasoning traces in S2Bench instantiate only 76 of them. Meanwhile, tasks such as voxel counting, dice-orientation tracking, and state transformation are primarily solved through deterministic executors rather than geometric theorem search. Although the paper integrates these two computational approaches into a unified formal interface, it does not report the specific proportions of theorem-driven and executor-driven examples, nor does it analyze the actual contributions of different parts of the 680 theorem families to data generation and model performance. Therefore, the current results support the broad task coverage of the framework, but do not sufficiently demonstrate that the large-scale theorem system is a key factor underlying these tasks and the observed performance improvements.
-
Section 3.3 validates generated examples by replaying the recorded theorem steps and checking the consistency of the final answers. However, this process mainly ensures consistency between the generated results and the internal formal system, rather than fully guaranteeing the correctness of every reasoning trace. According to Equations (7) and (8), when the premises of a theorem rule are not satisfied, the system can skip that rule and continue replaying. As long as the target can still be derived and the final answer agrees, the example may pass validation. This leaves a gap between the implemented verification procedure and the step-by-step verifiable reasoning emphasized in the paper. Furthermore, the generator and verifier rely on the same theorem definitions, making it possible for errors shared by both components to remain undetected. Although Appendix D describes independent mathematical audits and Appendix C.4 reports manual diagram inspection on 100 samples, the paper does not further report the coverage of independent verification across all theorems and tasks. Therefore, stronger empirical evidence is needed to support the claims regarding data reliability and the verifiability of reasoning steps.
-
Section 5.2 uses the low accuracy of models on S2Bench to demonstrate the difficulty of solid geometry and spatial reasoning. However, some benchmark tasks simultaneously involve complex numerical combinations, long chains of computation, and strict final-answer matching. For example, the SGMR case in Figure 9 of the appendix requires models to compute cross-sectional areas, volumes, distances between planes, and polygon side counts, then substitute the numerators and denominators of multiple rational numbers into a weighted expression to produce a large integer. Similar multi-variable combination problems also appear in FGVGR and NFMC. Under this evaluation setting, even if a model correctly understands the 3D structure, an error in any intermediate numerical value or final calculation causes the entire problem to be marked incorrect. The paper does not provide stage-wise accuracy for visual structure recognition, spatial relationship inference, and subsequent numerical computation, nor does it include controlled experiments accounting for computation length and answer complexity. Consequently, the current results cannot clearly distinguish whether model failures arise from limitations in 3D visual understanding or from long-chain computation and strict answer formats. This affects the assessment of what capabilities the benchmark actually measures and where its difficulty originates.
-
Section 5.3 analyzes the main bottlenecks of the models using structured visual hints and reasoning hints, concluding that fine-grained visual perception and 3D structural understanding are important factors limiting model performance. However, the visual hints are derived from complete formal geometric conditions and may directly provide object relationships, spatial states, and key structural information. Therefore, their effect may not be limited to improving visual perception, but may also substantially simplify subsequent reasoning. In Appendix G, providing only reasoning hints increases the overall accuracy of the 9B and 27B models from 14.0% to 14.8% and from 19.4% to 20.5%, respectively, whereas visual hints account for most of the improvements. The authors also acknowledge that structural hints in voxel-counting tasks may directly reveal the answers and therefore exclude this category from the average reported in Figure 5. The current experiments demonstrate that models benefit from explicit structural information, but do not sufficiently distinguish failures in visual perception, spatial state construction, and symbolic reasoning. Since this diagnosis directly motivates the subsequent training framework, the lack of independent analysis of these capabilities leaves the central argument of the paper insufficiently supported.
-
The VISTA framework proposed in Section 4 primarily uses structured hints to assist a teacher model in generating correct answers, followed by standard supervised fine-tuning and post-training with GRPO or GSPO. Although Table 3 compares direct reinforcement learning with reinforcement learning initialized from SFT, it does not adequately isolate the respective contributions of structured visual hints and symbolic reasoning traces to the final training improvements. In particular, Appendix G already shows that reasoning hints alone provide relatively limited benefits, yet Section 5 does not further compare supervised training data generated using only visual structure hints, only symbolic reasoning traces, or both. Appendix H also shows that only 6,422 teacher-generated responses out of 10,000 training instances pass answer filtering and enter the SFT dataset, meaning that the data selection process itself may also influence model performance. Without a conventional supervised fine-tuning baseline using the same number of samples, the same teacher model, and the same training budget, it remains difficult to determine whether the improvements of VISTA primarily come from structured symbolic supervision, improved data quality through answer filtering, or the combined effects of conventional SFT and reinforcement learning.
-
Section 4.1 filters SFT data solely based on whether the final answer generated by the teacher matches the symbolic reference answer, without independently verifying each step of the natural-language reasoning process. This creates a gap with the paper's objective of providing reliable process supervision through verifiable symbolic reasoning traces. A correct final answer does not necessarily imply that the intermediate reasoning steps are correct. Figure 32 in the appendix provides a noteworthy example: although VISTA-RL-27B obtains the correct final answer of 2100, its intermediate reasoning interprets the corresponding angles as 45 degrees and 135 degrees, whereas the task explicitly defines angle categories of 60 degrees and 120 degrees. This indicates that the model may still obtain the correct final answer despite mathematical errors in its intermediate explanations. The paper does not report step-level accuracy for generated reasoning traces, nor does it analyze the proportion of cases where the final answer is correct but the reasoning process contains errors. Therefore, the current results mainly demonstrate improvements in final-answer accuracy, but are insufficient to support the stronger claim that the models have learned reliable and verifiable symbolic reasoning processes.
-
Section 3.4 and Appendix E describe formal deduplication and structural equivalence checks between S2Bench and S2Train, which help prevent identical problems from appearing directly in both the training and test sets. However, both datasets are still generated from the same collection of 558 parameterized generation schemas. The current split primarily evaluates the ability of models to solve different instances within the same generation system, without testing unseen generation schemas, geometric structure combinations, or theorem combinations. For a framework that emphasizes structural diversity and scalable symbolic generation, whether models can transfer learned rules to new geometric constructions is an important consideration when assessing generalization. Although Table 4 reports results on seven external benchmarks, the average accuracy improvements of the 9B and 27B models are only 4.6 and 5.0 percentage points, respectively, with substantial variation across tasks. For example, the improvements on SolidGeo are only 0.8 and 1.4 percentage points, respectively. Therefore, the existing evidence supports a certain degree of external transfer, but more targeted validation is needed to determine whether the capabilities learned from symbolic training data can generalize to unseen complex geometric constructions.
Questions for authors
Please refer to Critical Weaknesses.
Thank you for the detailed responses and additional analyses. The clarification of predefined versus automatically instantiated choices, the coverage and construction-path statistics for S²Bench and S²Train, the capability-level comparison with Hilbert-Geo, and the per-task acceptance counts with 9B student comparisons partially address my concerns. Under identical problem IDs, the differences between supervision sources remain small (18.2% to 19.8%), so much of the full-set gain appears to come from broader accepted coverage rather than from the hint content itself. I am raising my score to 3.
I also encourage the authors to publicly release the generation code, the 558 schemas, the theorem library, and the S²Bench/S²Train datasets. As the paper’s main value lies in serving as a reusable resource, a public release would substantially strengthen its impact and reproducibility.