acceptodds
Under review as a conference paper at ICLR 2027

SolidGeoGen: Advancing Multimodal Solid-Geometry and Spatial Reasoning through Verifiable Symbolic Generation

11 views

Abstract

Solid geometry requires multimodal large language models (MLLMs) to jointly interpret 3D structures and perform mathematical and spatial reasoning, yet high-quality data with aligned diagrams and verifiable solutions remains scarce. Existing data-generation methods focus primarily on plane geometry, while solid-geometry problem generation remains limited in both diversity and reliability. We present SolidGeoGen, a theorem-driven symbolic framework for generating diverse solid-geometry problems with aligned diagrams, formal representations, and verifiable reasoning traces. This design ensures high-quality, internally consistent data for both training and evaluation. Building on SolidGeoGen, we construct SBench, a Solid-Spatial Benchmark comprising 2,000 problems across ten solid-geometry and spatial-reasoning tasks. We further build STrain, a training corpus containing 10,000 examples. Leveraging STrain, we propose VISTA, a Visual-structure-Informed and Symbolic-Trace-Assisted training framework. Extensive evaluation highlights the difficulty of SBench: all evaluated open-source MLLMs achieve below 30% average accuracy, while even the strongest proprietary model reaches only 67.2%. VISTA substantially improves accuracy on SBench at both 9B and 27B scales, while achieving consistent gains on external benchmarks for multimodal math reasoning and spatial intelligence. These results demonstrate that symbolic generation can provide scalable and reliable supervision for advancing complex multimodal reasoning in MLLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
math reasoning, multimodal reasoning, data-generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_f3sCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SolidGeoGen, a framework for constructing solid-geometry and spatial-reasoning data through verifiable symbolic generation, aiming to address the lack of reliable 3D structures, text-diagram alignment, and verifiable reasoning processes in existing multimodal geometry datasets. The framework establishes a formal system comprising 472 geometry description symbols and 680 theorem families. Through geometric structure instantiation, theorem-driven deduction, text-diagram generation, and symbolic verification, it automatically constructs problems with formal representations, geometric diagrams, answers, and reasoning traces. Building on this framework, the authors construct S2Bench, a benchmark containing 2,000 problems across 10 task categories, and S2Train, a training corpus containing 10,000 examples. They further propose VISTA, a training framework that uses structured visual hints and symbolic reasoning traces to generate supervised training data, followed by supervised fine-tuning and reinforcement learning to improve model capabilities. Experiments evaluate 21 language and multimodal models, with the best-performing model achieving 67.2% accuracy on S2Bench. Through VISTA training, the accuracy of Qwen3.5-9B improves from 14.0% to 42.4%, while Qwen3.5-27B improves from 19.4% to 50.9%. The trained models also achieve improvements on several external mathematical and spatial-reasoning benchmarks.

Critical strengths
  1. Section 3 establishes a formal description language and an executable theorem system for solid geometry, and generates questions, diagrams, symbolic conditions, and reasoning traces from shared geometric parameters. Compared with approaches that generate text, images, and answers separately, this design explicitly represents geometric objects and their relationships during data construction, while using symbolic replay and deterministic computation to verify the consistency between answers and formal conditions. The paper implements 558 parameterized generation schemas covering tasks such as solid-geometry calculations, cross-sections, folding, projections, and voxel-based spatial relationships, providing a valuable technical foundation for constructing structured geometry data at scale.

  2. Table 2 systematically evaluates 21 models, revealing substantial performance differences across 10 categories of solid-geometry and spatial-reasoning tasks. Table 3 further shows that VISTA-SFT improves the accuracy of the 9B and 27B models to 37.6% and 43.7%, respectively, while subsequent reinforcement learning further increases their accuracy to 42.4% and 50.9%. Table 4 also reports experimental results on seven external benchmarks, with relatively substantial improvements on MindCube and MMSI. These results demonstrate that the constructed data can effectively improve model performance on related tasks and provide some evidence of cross-benchmark transferability.

Critical weaknesses
  1. Section 3 presents theorem-driven symbolic generation as a core contribution of SolidGeoGen, but many of the spatial-reasoning tasks constructed in the paper do not directly rely on the proposed theorem deduction system. According to Appendix B.2, the complete theorem library contains 680 theorem families, whereas the reasoning traces in S2Bench instantiate only 76 of them. Meanwhile, tasks such as voxel counting, dice-orientation tracking, and state transformation are primarily solved through deterministic executors rather than geometric theorem search. Although the paper integrates these two computational approaches into a unified formal interface, it does not report the specific proportions of theorem-driven and executor-driven examples, nor does it analyze the actual contributions of different parts of the 680 theorem families to data generation and model performance. Therefore, the current results support the broad task coverage of the framework, but do not sufficiently demonstrate that the large-scale theorem system is a key factor underlying these tasks and the observed performance improvements.

  2. Section 3.3 validates generated examples by replaying the recorded theorem steps and checking the consistency of the final answers. However, this process mainly ensures consistency between the generated results and the internal formal system, rather than fully guaranteeing the correctness of every reasoning trace. According to Equations (7) and (8), when the premises of a theorem rule are not satisfied, the system can skip that rule and continue replaying. As long as the target can still be derived and the final answer agrees, the example may pass validation. This leaves a gap between the implemented verification procedure and the step-by-step verifiable reasoning emphasized in the paper. Furthermore, the generator and verifier rely on the same theorem definitions, making it possible for errors shared by both components to remain undetected. Although Appendix D describes independent mathematical audits and Appendix C.4 reports manual diagram inspection on 100 samples, the paper does not further report the coverage of independent verification across all theorems and tasks. Therefore, stronger empirical evidence is needed to support the claims regarding data reliability and the verifiability of reasoning steps.

  3. Section 5.2 uses the low accuracy of models on S2Bench to demonstrate the difficulty of solid geometry and spatial reasoning. However, some benchmark tasks simultaneously involve complex numerical combinations, long chains of computation, and strict final-answer matching. For example, the SGMR case in Figure 9 of the appendix requires models to compute cross-sectional areas, volumes, distances between planes, and polygon side counts, then substitute the numerators and denominators of multiple rational numbers into a weighted expression to produce a large integer. Similar multi-variable combination problems also appear in FGVGR and NFMC. Under this evaluation setting, even if a model correctly understands the 3D structure, an error in any intermediate numerical value or final calculation causes the entire problem to be marked incorrect. The paper does not provide stage-wise accuracy for visual structure recognition, spatial relationship inference, and subsequent numerical computation, nor does it include controlled experiments accounting for computation length and answer complexity. Consequently, the current results cannot clearly distinguish whether model failures arise from limitations in 3D visual understanding or from long-chain computation and strict answer formats. This affects the assessment of what capabilities the benchmark actually measures and where its difficulty originates.

  4. Section 5.3 analyzes the main bottlenecks of the models using structured visual hints and reasoning hints, concluding that fine-grained visual perception and 3D structural understanding are important factors limiting model performance. However, the visual hints are derived from complete formal geometric conditions and may directly provide object relationships, spatial states, and key structural information. Therefore, their effect may not be limited to improving visual perception, but may also substantially simplify subsequent reasoning. In Appendix G, providing only reasoning hints increases the overall accuracy of the 9B and 27B models from 14.0% to 14.8% and from 19.4% to 20.5%, respectively, whereas visual hints account for most of the improvements. The authors also acknowledge that structural hints in voxel-counting tasks may directly reveal the answers and therefore exclude this category from the average reported in Figure 5. The current experiments demonstrate that models benefit from explicit structural information, but do not sufficiently distinguish failures in visual perception, spatial state construction, and symbolic reasoning. Since this diagnosis directly motivates the subsequent training framework, the lack of independent analysis of these capabilities leaves the central argument of the paper insufficiently supported.

  5. The VISTA framework proposed in Section 4 primarily uses structured hints to assist a teacher model in generating correct answers, followed by standard supervised fine-tuning and post-training with GRPO or GSPO. Although Table 3 compares direct reinforcement learning with reinforcement learning initialized from SFT, it does not adequately isolate the respective contributions of structured visual hints and symbolic reasoning traces to the final training improvements. In particular, Appendix G already shows that reasoning hints alone provide relatively limited benefits, yet Section 5 does not further compare supervised training data generated using only visual structure hints, only symbolic reasoning traces, or both. Appendix H also shows that only 6,422 teacher-generated responses out of 10,000 training instances pass answer filtering and enter the SFT dataset, meaning that the data selection process itself may also influence model performance. Without a conventional supervised fine-tuning baseline using the same number of samples, the same teacher model, and the same training budget, it remains difficult to determine whether the improvements of VISTA primarily come from structured symbolic supervision, improved data quality through answer filtering, or the combined effects of conventional SFT and reinforcement learning.

  6. Section 4.1 filters SFT data solely based on whether the final answer generated by the teacher matches the symbolic reference answer, without independently verifying each step of the natural-language reasoning process. This creates a gap with the paper's objective of providing reliable process supervision through verifiable symbolic reasoning traces. A correct final answer does not necessarily imply that the intermediate reasoning steps are correct. Figure 32 in the appendix provides a noteworthy example: although VISTA-RL-27B obtains the correct final answer of 2100, its intermediate reasoning interprets the corresponding angles as 45 degrees and 135 degrees, whereas the task explicitly defines angle categories of 60 degrees and 120 degrees. This indicates that the model may still obtain the correct final answer despite mathematical errors in its intermediate explanations. The paper does not report step-level accuracy for generated reasoning traces, nor does it analyze the proportion of cases where the final answer is correct but the reasoning process contains errors. Therefore, the current results mainly demonstrate improvements in final-answer accuracy, but are insufficient to support the stronger claim that the models have learned reliable and verifiable symbolic reasoning processes.

  7. Section 3.4 and Appendix E describe formal deduplication and structural equivalence checks between S2Bench and S2Train, which help prevent identical problems from appearing directly in both the training and test sets. However, both datasets are still generated from the same collection of 558 parameterized generation schemas. The current split primarily evaluates the ability of models to solve different instances within the same generation system, without testing unseen generation schemas, geometric structure combinations, or theorem combinations. For a framework that emphasizes structural diversity and scalable symbolic generation, whether models can transfer learned rules to new geometric constructions is an important consideration when assessing generalization. Although Table 4 reports results on seven external benchmarks, the average accuracy improvements of the 9B and 27B models are only 4.6 and 5.0 percentage points, respectively, with substantial variation across tasks. For example, the improvements on SolidGeo are only 0.8 and 1.4 percentage points, respectively. Therefore, the existing evidence supports a certain degree of external transfer, but more targeted validation is needed to determine whether the capabilities learned from symbolic training data can generalize to unseen complex geometric constructions.

Questions for authors

Please refer to Critical Weaknesses.

Reviewer_jxSFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SolidGeoGen, a symbolic generation framework for solid-geometry and spatial-reasoning problems. Using 558 parameterized generation schemas, it constructs formal conditions, natural-language questions, rendered diagrams, answers, and structured reasoning traces from shared geometric states. Generated answers are checked through symbolic replay or task-specific deterministic procedures. The authors construct S²Bench, containing 2,000 problems across ten task categories, and S²Train, containing 10,000 problems. They further propose VISTA, which uses privileged visual-structure and reasoning-trace hints to synthesize SFT data through rejection sampling, followed by RLVR. On-policy distillation from the resulting RL models into base-model students is explored separately. Evaluation of 21 models shows that all evaluated off-the-shelf open-source MLLMs in Table 2 score below 30% on S²Bench. VISTA improves Qwen3.5-9B/27B from 14.0%/19.4% to 42.4%/50.9%, with positive transfer to seven external benchmarks.

Critical strengths

S1. Substantial engineering for 3D multimodal synthesis, supported by validation and decontamination.

The framework integrates geometric representation, rendering, theorem-based reasoning, and deterministic spatial computations across ten task categories. Constructing these components from shared geometric states is a sensible approach to reducing inconsistencies between questions, diagrams, and answers. Appendix E describes symmetry-aware canonicalization that goes beyond lexical deduplication, while Appendix C.4 reports a 100-instance visual audit. Appendix D also distinguishes symbolic replay from independent checks of executor computations. These components support the value of the system as a reusable resource for 3D spatial reasoning.

S2. Demonstrated training utility and informative structural-hint analysis.

The training gains occur at both model scales and extend beyond the generated benchmark: Table 4 reports average improvements of 4.6 and 5.0 percentage points across seven external benchmarks, with substantial gains on MindCube and MMSI. The hint analysis provides a useful complementary observation. Averaging Table 19 over the nine tasks excluding VC, whose visual hints may expose the answer, gives Qwen3.5-27B accuracies of 18.1% without hints, 64.4% with visual-structure hints, and 19.2% with reasoning hints alone. This suggests that recovering and using 3D structure is an important difficulty for these models, although the intervention does not isolate visual perception from subsequent structural inference.

Critical weaknesses

W1. The generation mechanism and its distinctive contribution would benefit from clarification.

The schema constraints and worked example in Appendices C.2 and C.5 provide a useful starting point, but it remains unclear which target and theorem-sequence choices are predefined and which are selected automatically from instantiated conditions. A concise account of rule selection, alternative derivations, and termination would help assess the generation mechanism beyond the Construct_T abstraction.

The validation results and downstream gains are encouraging, although actual structural diversity could be characterized more directly. Appendix B.2 reports 76 theorem families in S²Bench out of a library of 680; this does not imply limited diversity, but library size alone does not establish dataset coverage. Schema frequencies and theorem-composition distributions would be informative. It would also help to distinguish the formal foundation from Hilbert-Geo (Xu et al., CVPR 2026), particularly the new representational or executable capabilities beyond library scale, while acknowledging its different focus on solving rather than generation.

W2. The roles of construction paths and supervision sources could be more clearly distinguished.

The combination of theorem-based construction and deterministic execution is reasonable. However, their distribution across tasks is not reported, and the role of geometric deductions beyond the bridge rules in executor-based instances remains unclear. This limits assessment of how the theorem library contributes to the spatial tasks.

The training results demonstrate the value of the complete pipeline, but attribution to individual supervision sources remains open. Table 19 evaluates hints at inference time, rather than students trained under different synthesis conditions; reasoning hints also derive from full traces that may include executor computations. A focused no-hint, visual-only, and combined-hint synthesis comparison would clarify these contributions. Since filtering retains 6,422 of 10,000 problems, reporting per-task/schema acceptance would also help distinguish better responses from changes in training-set composition. These clarifications would strengthen the interpretation of the gains without diminishing the demonstrated utility of the resource.

Questions for authors

Q1. Could you clarify how target selection and theorem composition operate within a schema?

Using Appendix C.2's example, which choices are predefined and which follow automatically from instantiated conditions? A brief explanation or pseudocode covering rule selection, alternative derivations, and termination would make the procedure easier to assess.

Q2. Could you summarize the released datasets' coverage and construction paths?

Beyond the separate audit of all 558 schemas, what are the actual schema frequencies and theorem-sequence diversity in S²Bench and S²Train? What proportion of each task uses theorem-based or executor-based construction? A representative executor-based example would help clarify whether geometric deductions occur beyond the bridge rules in Appendix D.3.

Q3. Could you clarify how supervision and filtering contribute to the training gains?

Please provide per-task/schema acceptance counts for the 6,422-example SFT set. If feasible, a one-scale comparison of no-hint, visual-only, and combined-hint synthesis, using the same problem pool, comparable generation budgets, and matched student training-token budgets, would be informative. Reporting acceptance coverage alongside performance would help interpret differences between the retained training sets.

If the authors can adequately address these concerns, I would consider raising my score.

2 comments

Thank you for the detailed responses and additional analyses. The clarification of predefined versus automatically instantiated choices, the coverage and construction-path statistics for S²Bench and S²Train, the capability-level comparison with Hilbert-Geo, and the per-task acceptance counts with 9B student comparisons partially address my concerns. Under identical problem IDs, the differences between supervision sources remain small (18.2% to 19.8%), so much of the full-set gain appears to come from broader accepted coverage rather than from the hint content itself. I am raising my score to 3.

I also encourage the authors to publicly release the generation code, the 558 schemas, the theorem library, and the S²Bench/S²Train datasets. As the paper’s main value lies in serving as a reusable resource, a public release would substantially strengthen its impact and reproducibility.

Thank you for carefully considering our responses and additional analyses, and for raising your score. We appreciate your constructive feedback and emphasis on the value of a publicly available resource.

To our knowledge, SolidGeoGen is the first symbolically grounded system for verified multimodal solid-geometry problem generation. It combines geometric diagrams, formal conditions, symbolic reasoning traces, and verifiable reference answers within a shared representation, providing a reusable foundation for benchmark construction, training-data generation, and research on spatial reasoning.

We commit to publicly releasing the SolidGeoGen generation code, all 558 generation schemas, the theorem library, and the S²Bench and S²Train datasets. We agree that making these resources available is central to the contribution, enabling the community to reproduce, inspect, and extend our framework.

Thank you again for your thoughtful assessment and helpful suggestions, which have strengthened both the presentation and the planned release of our work.

Reviewer_WKYkOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Current AI models really struggle with 3D geometry and spatial reasoning. A big reason why is that it's incredibly hard to find good training data where the text question, the 3D diagram, the math, and the step-by-step solution all perfectly match up.

To fix this, the authors built a system called SolidGeoGen. It uses strict mathematical rules (theorems) to automatically generate flawless solid geometry problems, making sure the picture, the wording, and the solution are perfectly aligned.

Using this generator, they created three main things:

  1. A new test called S²Bench (2,000 challenging geometry and spatial reasoning problems).
  2. A new training dataset called S²Train (10,000 examples with step-by-step hints).
  3. A training method called VISTA, which teaches AI models how to use those hints to solve 3D problems.

When they tested existing AI models on their new benchmark, the models performed poorly (most open-source models scored under 30%, and even the best model, GPT-5.6-Sol, only hit 67%). However, when they trained models using their new VISTA method, performance skyrocketed. For example, a 27B model jumped from a 19.4% accuracy to 50.9%. Better yet, these trained models didn't just memorize the new data—they also got noticeably better at other standard math and spatial tests.

Critical strengths

The first major strength is the creation of a highly reliable, automated data generator. By building a strict formal system with executable mathematical theorems, the authors can guarantee that every generated problem has perfectly aligned text, images, answers, and step-by-step proofs. This is incredibly consequential because getting clean, perfectly aligned 3D data is currently a massive bottleneck in AI, and their system solves it at scale.

The second critical strength is that their training method proves this data actually teaches models how to reason better. Instead of just introducing a new benchmark, they show that fine-tuning models on their data causes massive accuracy jumps. Most importantly, the models didn't just memorize the test set; they got significantly better at completely different, out-of-domain spatial tests. This proves the data is teaching genuine 3D understanding rather than just fitting a synthetic pattern.

Critical weaknesses
  1. The training method essentially works by distilling a teacher model that was already given the answers. The authors use "privileged hints"—which include the formal geometric structure and step-by-step solution traces—to guide a teacher model. The student model then learns to copy the teacher's output on the raw image. While this yields great results, it is a well-known distillation trick rather than a fundamental breakthrough in spatial reasoning. It doesn't prove the model learned to genuinely "see" 3D space; it just proves the model can memorize how a teacher talks about a problem when the teacher was handed the solution.

  2. The second critical weakness is the closed-loop nature of the evaluation. Both the S²Bench (the test set) and S²Train (the training set) are generated by the exact same symbolic engine. This means the models are being tested on the exact same synthetic distribution they were trained on. The model might simply be learning the specific formatting quirks and rules of the SolidGeoGen engine rather than developing generalizable 3D reasoning. While the authors do show some out-of-domain generalization, the core benchmark risks rewarding models for fitting the generator's distribution rather than real spatial intelligence.

Questions for authors

First, I would like to understand the true impact of the "privileged hints" during training. The VISTA framework uses a teacher model conditioned on formal, step-by-step hints to generate the training data. To help me assess whether the student model is learning genuine spatial reasoning or just mimicking the teacher's reasoning format, could you provide the accuracy of the teacher model (Qwen3.5-27B) on the S²Bench evaluation set when it is also given these privileged hints? If the teacher with hints scores near 100%, it suggests the bottleneck is purely visual extraction. If it scores much lower, it implies the symbolic hints themselves are incomplete.

Second, regarding the closed-loop evaluation: since both the S²Bench test set and S²Train dataset are generated by the exact same SolidGeoGen engine, how do you ensure the models aren't just learning the specific stylistic quirks of your generator? While Table 4 shows out-of-domain (OOD) generalization, could you clarify whether those OOD datasets (like SolidGeo or HilbertGeo) are human-authored or synthetically generated? Clarifying this would help establish whether the VISTA training method truly transfers to real-world, human-crafted spatial reasoning tasks.

As a future direction, it would be highly valuable to explore a neuro-symbolic test-time approach. Currently, the model outputs natural language, and correctness is judged by an external evaluator. A powerful extension of SolidGeoGen would be to train models to output the formal Geometry Description Language (GDL) directly. This would allow for real-time, exact symbolic verification of the model's reasoning steps during inference, rather than just checking the final answer.

1 comment

Dear reviewer WKYk, thank you again for your thoughtful review and constructive feedback. Your questions have helped us clarify the contributions of SolidGeoGen and strengthen the supporting evidence.

As the rebuttal period draws to a close, we would greatly appreciate your consideration of our responses above, particularly the clarifications on external benchmark provenance and the evidence of transfer to real-world spatial understanding.

We hope these responses address your concerns. If you find that the clarifications and additional evidence resolve them, we would be grateful if you could take this into account when reconsidering your overall assessment. Thank you for your time and careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.