acceptodds
Withdrawn from ICLR 2027

CoRe3D: Collaborative Reasoning through Octant-Based Chain-of-Thought for Unified 3D Generation and Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent progress in multimodal foundation models shows that explicit reasoning mechanisms can improve reliability, cross-modal alignment, and generation, yet comparable mechanisms for 3D remain limited. We present CoRe3D, a unified framework for 3D understanding and generation, that performs collaborative reasoning over two coupled traces: (i) a semantic CoT that expands a prompt into an interpretable plan (parts, attributes, relations), and (ii) an octant-based geometric CoT that realizes this plan as localized 3D tokens, enabling compositional and interpretable generation. To jointly optimize both traces end-to-end under heterogeneous, partly non-differentiable 3D objectives, we propose Co-GRPO, a trace-aware group-relative preference optimization that integrates multi-critic 3D rewards. Experiments show that CoRe3D improves 3D understanding (e.g. +27% BLEU-1, +23% ROUGE-L, +13% METEOR over the strongest baseline) and achieves state-of-the-art text-to-3D alignment (+38% CLIP over the next best method), while also supporting a diverse set of tasks spanning general multimodal reasoning, image-to-3D, reasoning-based 3D generation over complex prompts, and fine-grained 3D part editing within a single model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
unified 3D model, 3D reasoning, text-to-3D, 3D understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CZ6qOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

CoRe3D extends a unified 3D language model to connect semantic planning with spatial construction: it generates a textual plan describing an object’s attributes and relations, then autoregressively predicts 512 spatially indexed 3D tokens conditioned on that plan. Its main contributions are semantic–geometric co-attention, which gives geometric predictions continued access to the plan; an octant-based token representation for local construction; and Co-GRPO, a GRPO-based training framework combining preference, attribute correctness, text–3D alignment, and physical-coherence rewards. The rewards reference each candidate’s semantic plan, while adaptive sampling balances generation and understanding updates.

Critical strengths

The paper is well motivated: explicit semantic cot planning offers a reasonable way to guide 3D generation and connect linguistic intent with geometric construction.

  • Clear and intuitive methodology: Semantic–geometric co-attention allows each geometric decoding step to access the semantic plan, providing continued guidance during construction. Co-GRPO combines complementary preference, semantic alignment, attribute correctness, and physical-coherence rewards to jointly train generation and understanding. Together, these components form an easy-to-follow framework addressing both semantic faithfulness and geometric quality.
  • Strong presentation: The paper is well organized and readable, with polished figures, effective layouts, and informative qualitative comparisons that clearly illustrate the framework and its capabilities.
Critical weaknesses

Figure 1 may mislead readers about the actual generation process. It appears to suggest that successive semantic CoT stages guide the construction of corresponding object parts, potentially through explicit stage-to-region supervision. However, Section 3.3 describes generating the complete semantic trace first, followed by autoregressive geometric decoding that accesses this trace through co-attention. The method does not specify interleaved semantic–geometric generation, stage-to-part assignments, or explicit supervision of these correspondences. The authors should distinguish conceptual guidance from the implemented decoding procedure and revise Figure 1 to reflect the actual temporal flow.

The paper does not directly show whether using the semantic trace to compute rewards improves performance. Table 5 tests different critics, but does not test whether those critics benefit from reading the trace. Removing Semantic CoT in Table 6 also changes how the object is generated, so it cannot isolate the effect on reward computation. The missing comparison is straightforward: keep the semantic trace available during generation, and only vary whether the reward evaluators use it.

In Table1 and Table3, The baseline comparisons could be more up to date. Keeping ShapeLLM-Omni makes sense because it is the starting point, but newer models such as Qwen3-VL-8B and Qwen3.5-9B would provide useful comparisons for captioning and language reasoning. A particularly informative baseline would use these models to expand the prompts while keeping the 3D generator fixed. This would help show whether the gains come from the proposed joint framework or simply from stronger semantic planning.

Questions for authors

Evidence for trace-aware rewards: What evidence isolates the benefit of using the semantic trace in reward computation? Can the authors provide a controlled comparison that keeps semantic-trace generation, co-attention, critics, and training budget fixed, and only changes whether rewards use the original prompt alone or also the generated trace? This would directly test the claimed contribution of trace-aware evaluation.

Evidence beyond the current baselines: Have the authors evaluated comparable-size recent models, such as Qwen3-VL-8B or Qwen3.5-9B, on the tasks in Tables 1 and 3 under matched evaluation conditions? In particular, is there evidence that CoRe3D outperforms using a stronger model to produce semantic plans while keeping the 3D generator fixed? This would help separate the benefit of the proposed framework from improvements due to stronger semantic planning.

Clarification of evaluation settings: Why does the full model achieve a METEOR score of 24.98 in Tables 1, 4, and 5, but only 17.03 under the default settings in Tables 6–8? Were the same datasets, splits, checkpoint, and evaluation protocol used? If not, please specify the differences and indicate which results are directly comparable.

Reviewer_ViFWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper integrates language reasoning, 3D geometry generation, and reinforcement learning into a unified framework. By designing Semantic CoT and Geometric CoT, and introducing Co-GRPO to solve two problems: how to enable the model to understand complex or implicit language instructions and generate reasonable 3D objects accordingly; and how to complete 3D generation and understanding simultaneously through the same model.

Critical strengths
  1. The paper addresses an important challenge of bridging high-level semantic understanding and low-level 3D generation within a unified framework.
  2. The co-attention mechanism provides a reasonable way to condition geometric generation on semantic descriptions, with supporting ablation results.
  3. The method achieves substantial improvements in text-to-3D alignment and 3D captioning over several competitive baselines.
Critical weaknesses
  1. The geometric CoT largely resembles conventional autoregressive 3D token generation, while Co-GRPO mainly combines existing GRPO with multiple reward functions. The distinction from prior methods is insufficiently established.
  2. Although the paper includes reward-critic ablations, it lacks direct comparisons with standard GRPO, fixed task sampling, and independently optimized generation and understanding. The specific benefit of the proposed optimization remains unclear.
  3. The evidence for true reasoning is rather weak. These reasoning-oriented generation experiments mainly rely on carefully selected qualitative examples. It is not yet clear whether these improvements are due to actual semantic-geometric reasoning, or simply because of the enhancement of semantic recognition and conditioning capabilities.
Questions for authors
  • Why do the full-model results differ between the main paper and Appendix Table 6? Were different evaluation datasets or protocols used?
Reviewer_9xm2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CoRe3D, an LLM-based framework for unified 3D object generation and understanding. It integrates semantic chain-of-thought (CoT) reasoning with autoregressive 3D token generation through a semantic–geometric co-attention mechanism, which conditions geometric token prediction on the semantic reasoning trace. The framework is further optimized using Co-GRPO, a GRPO-based post-training strategy that combines multiple 3D reward signals to jointly improve generation and understanding.

Critical strengths

The paper has two main strengths:

  • The framework integrates explicit semantic chain-of-thought reasoning with autoregressive 3D token generation through semantic–geometric co-attention, providing a direct connection between textual planning and geometric construction.
  • The experiments demonstrate improvements in both 3D generation quality and 3D understanding, with the latter evaluated through object captioning. Improved captioning is particularly valuable given the often limited quality of textual annotations in existing 3D datasets.
Critical weaknesses

The paper has the following weaknesses:

  • The central recipe of semantic CoT-guided autoregressive generation optimized through GRPO with multiple reward signals closely follows T2I-R1 [1]. Although CoRe3D introduces 3D-specific rewards and jointly trains understanding tasks, its methodological contribution beyond adapting this recipe to 3D is not sufficiently clear.
  • The evaluation lacks comparisons with recent methods, including TRELLIS.2 [2] for image-to-3D generation and AR3D-R1 [3] for autoregressive text-to-3D generation. AR3D-R1 is particularly relevant because it also investigates reasoning-guided 3D generation with GRPO-based post-training. These omissions limit the evidence supporting the paper’s claims of state-of-the-art performance.
  • The paper does not clearly describe how the final meshes and textures are generated. If it follows the TRELLIS-based pipeline used by ShapeLLM-Omni, the text conditioning for the second-stage generation is insufficiently specified: does it use the original user prompt, the generated semantic CoT, or a condensed description? This choice could substantially affect the final geometry and appearance, particularly because TRELLIS is trained on concise object descriptions, whereas CoT traces may be considerably longer and differ in style. Without clarifying this conditioning, it is difficult to assess how much of the reported improvement comes from the proposed 3D model versus the text supplied to the downstream generator.

References

  • [1] Dongzhi Jiang et al. T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoT. NeurIPS, 2025. Paper
  • [2] Jianfeng Xiang et al. Native and Compact Structured Latents for 3D Generation. CVPR, 2026. Paper
  • [3] Yiwen Tang et al. Are We Ready for RL in Text-to-3D Generation? A Progressive Investigation. CVPR, 2026. Paper
Questions for authors

Several points would benefit from clarification:

  • Could the authors clarify how the intermediate geometries in Figures 1 and 6 correspond to the Morton-ordered token sequence? The figures appear to show a front-to-back progression; please specify the coordinate convention and traversal order, and clarify whether these intermediate shapes are decoded from actual token prefixes or are schematic illustrations. Beyond these implementation details, what motivates this ordering for geometric CoT? Some prior autoregressive mesh generators use a bottom-to-top progression, which can be intuitively related to constructing an object from its supporting structure upward. How does the chosen traversal relate to this modeling intuition, and is it intended to capture a meaningful construction process or primarily preserve spatial locality?
  • Could the authors explain the differences between the experimental results and implementation details reported in the main paper and the appendix? For example, the full model achieves a captioning METEOR score of 24.98 in Table 1 but 17.03 in Table 6. Section 4 also reports a rollout group size of 8, whereas Appendix C specifies 4 candidates per prompt. Do these results correspond to different training or evaluation settings?
  • Table 4 demonstrates the benefit of co-attention, but the direction of information exchange remains unclear. Equation (1) updates geometric hidden states using the semantic trace. Does the implemented mechanism also allow semantic representations to incorporate previously generated geometry, or is generation conditioned on a fixed semantic trace throughout? Given the reported improvement in 3D understanding, could the authors discuss whether geometry-to-semantic feedback could help identify and address errors during generation? If such feedback is already supported by the architecture, a targeted ablation of the interaction direction would further clarify the role of CoT in 3D generation and understanding. Otherwise, this could be discussed as a potential extension rather than an additional experiment required for the current submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.