- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ShapeMind, a framework that generates low-poly 3D assets with explicit semantic part structure and, where applicable, kinematic structure. It couples ShapeVLM, which handles 3D semantic understanding, with Semantic PartGen, which performs native part-level mesh generation.
Critical strengths
(1) The paper introduces ShapeCodec, a compact contour-based representation that expresses 3D geometry within a VLM’s existing text vocabulary. It pairs this with ShapeVLM, a two-turn supervised VLM that couples semantic structure prediction with part geometry reconstruction.
(2) The paper introduces an octree forest with part-local topology latents, designed for native generation of multi-part meshes.
(3) Experiments on multiple benchmarks show state-of-the-art performance in both 3D semantic understanding and part-mesh generation.
Critical weaknesses
(1) Much of the framework builds on existing techniques: (a) ShapeVLM fine-tunes an existing VLM (Qwen3-VL-8B) with standard supervised training. Its main novelty lies in how the geometry is encoded rather than in the model itself. (b) Semantic PartGen adopts Nexus’s vertex–topology factorization, frozen mesh VAE, and face recovery algorithm. (c) The scoped part/object attention follows CubePart. The authors should state more precisely what is new, such as the octree forest, the tri-stream conditioning, and the box re-fitting in stage 2. And also should show the individual impact of each component by providing the ablation experimental results.
(2) The paper provides Figure 4 on Semantic PartGen, but it doesn’t provide detailed structural information about MMDiT. It would help to explain: (a) how the MMDiT blocks differ from standard DiT blocks in this design; (b) how part-scope and object-scope attention are implemented and arranged across layers; (c) how both differ from standard full self-attention. An ablation comparing scoped attention with standard full attention would also clarify its benefit.
(3) The paper would benefit from a discussion of whether the approach could extend to more complex settings, such as multi-object compositions or indoor and outdoor scenes.
(4) The comparisons could be broadened. For example, X-Part is discussed in the related work but not compared experimentally. Comparing with native mesh generators combined with part segmentation would also better support the low-poly mesh quality claims.
(5) The 64³ grid limits the geometric detail available to ShapeVLM, and the average of about 6K tokens per object will grow at higher resolutions. In addition, the baseline comparisons use only objects with at most eight parts. The authors should discuss how the method scales with resolution and part count, and include representative failure cases.
Questions for authors
(1) How do the Table 2 results change when the coarse geometry G₀ comes from an image-to-3D model rather than ground-truth geometry?
(2) How sensitive is Semantic PartGen to noise in the input boxes and to the vertex-count condition N?
(3) What is the end-to-end performance on PartObjaverse-Tiny when using ShapeVLM-predicted boxes and descriptions instead of ground-truth conditions?
3 comments
I thank the authors for their detailed rebuttal and the additional experiments. These have addressed several of my concerns. However, the following issues remain:
W1 (Novelty and component contributions). The response lists ShapeCodec, the octree forest, and the tri-stream MMDiT as the main contributions, but the revised manuscript still does not delineate which components are new and which are inherited from Nexus and CubePart. More importantly, the octree forest, presented as a core contribution, is not ablated. A comparison with a direct extension of Nexus (e.g., per-part generation with box conditioning) is needed to establish its benefit.
W2 (Architecture clarity). Appendix B.1 clarifies the block design and attention masks. As a suggestion, a figure that contrasts the MMDiT block with a standard DiT block, and part-scope with object-scope attention, would make these differences easier to follow.
W5 (Failure cases). Errors are mentioned briefly in Appendices D.2 and F, but there is no systematic failure-case analysis. Such an analysis is important for understanding the method’s limitations, especially for ShapeVLM’s under-segmentation of complex objects.
Q1 (Image-generated geometry). The new results are informative. However, the HY2.1 block uses updated PartGen weights, so the drop in Part-CD/F cannot be attributed to the input geometry alone.
Overall, the rebuttal improves the clarity of the paper, but the contribution of the key novel component remains unverified. I therefore maintain my original score.
We appreciate your thorough and informative reviews, which help to clarify our work.
With respect to the remaining issues:
W1 (Novelty and component contributions)
Thank you for highlighting the need to distinguish our contributions more clearly. Our contributions are ShapeCodec, the octree-forest representation, and tri-stream MMDiT; These are not inherited from Nexus or CubePart. We reuse Nexus’s vertex–topology factorization, mesh VAE, and face recovery, and follow CubePart’s scoped attention design. We will explicitly distinguish these foundations from our contributions in the manuscript.
For the octree-forest comparison, independent box-conditioned part generation has foreseeable issues: limited cross-part context can lead to missing or duplicated geometry and inconsistent interfaces, complicating assembly. These motivate our joint representation. We have not conducted this comparison primarily due to limited computational resources and lack of access to Nexus’s official code and checkpoints.
W2 (Architecture clarity)
Thank you for this helpful suggestion. We will add a schematic contrasting our MMDiT block with a standard DiT block and illustrating part-scope versus object-scope attention in the camera-ready version.
W5 (Failure cases)
Thank you for emphasizing the need for a more systematic failure analysis. ShapeVLM can predict coarser decompositions than the reference annotations, merging fine components, as illustrated by the vehicle and tank in Figures 11–12. When semantic cues are ambiguous or unclear, PartGen may also interpret part descriptions inconsistently, potentially generating duplicated geometry across parts. We will expand these observations into a structured failure-case analysis in the camera-ready version.
Q1 (Image-generated geometry)
We have updated Table 2 in the main paper, containing two settings: with GT and HY2.1 coarse mesh. The HY2.1 coarse mesh numbers are updated with the same old checkpoint.
Thank you for recognizing our representation and native part generation, and for your constructive suggestions. All figures refer to the revised paper.
Novelty and component ablations (W1).
Our contributions are ShapeCodec, the octree forest, and tri-stream MMDiT; Appendix B.1 provides implementation details. Revised-paper Table 1 ablates ShapeCodec, while the octree forest extends holistic mesh representations to multiple parts. The table below compares our tri-stream design with a standard dual-stream MMDiT that concatenates text with other conditioning tokens. We also ablate box re-fitting—using boxes fitted to S1 vertices for S2—a design choice rather than a core contribution.
Architecture/box conditioning: 197 PartObjaverse-Tiny objects.
MMDiT implementation (W2).
Revised Appendix B.1 details the token organization and attention mechanisms.
Scene extensions (W3).
Appendix F (Figures 17–18) shows eight PartGen scenes from synthetic images and supplied boxes/text. Complex scenes retain coherent layouts. Extending object-trained ShapeVLM to scenes is future work.
Broader comparisons (W4).
ArtLLM uses XPart for geometry generation; we have revised the paper to make this clearer. Paper Table 3 includes HY2.1 as native 3D generator, and PatchAlign3D + HoloPart and OmniPart as part segmentation and completion.
Resolution and part count (W5).
The 64³ grid balances efficiency and accuracy: voxel precision suffices for object-level articulation/bboxes; finer grids increase ShapeCodec tokens.
Ours leads all quality metrics on 137 objects with >8 parts; generation supports up to 32 parts.
137 PartObjaverse-Tiny objects with >8 parts.
Image-generated coarse geometry (Q1).
Geometry-dependent methods share HY2.1 meshes; PartGen uses predicted conditions and Nᵥ = 4000. Pose/scale discrepancies reduce spatial scores; type/graph/range remain relatively stable. Ours leads five of eight metrics, including both mesh scores.
PartNet-Mobility: GT vs. HY2.1. Bold/italic: best/second per block.
Sensitivity to boxes and vertex count (Q2).
Halving/doubling vertex counts changes quality metrics by at most 1.3%. Our model is sensitive to bbox changes, enabling control over part positions and extents (Figure 15).
PartObjaverse-Tiny condition sensitivity.
Predicted conditions on PartObjaverse-Tiny (Q3).
ShapeVLM predicts coarser parts than GT due to its training annotations. PartGen preserves coherent geometry across different decompositions (Figures 11–12).
GT/predicted part counts.