acceptodds
Withdrawn from ICLR 2027

Cog2Gen3D: Sculpturing 3D Semantic-Geometric Cognition for 3D Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generative models have achieved success in producing semantically plausible 2D images, but it remains challenging in 3D generation due to the absence of spatial geometry constraints. Typically, existing methods utilize geometric features as conditions to enhance spatial awareness. However, these methods can only model relative relationships and are prone to scale inconsistency of absolute geometry. Thus, we argue that semantic information and absolute geometry empower 3D cognition, thereby enabling controllable 3D generation for the physical world. In this work, we propose Cog2Gen3D, a 3D cognition-guided diffusion framework for 3D generation. Our model is guided by three key designs: 1) Cognitive Feature Embeddings. We encode different modalities into semantic and geometric representations and further extract logical representations. 2) 3D Latent Cognition Graph. We structure different representations into dual-stream semantic-geometric graphs and fuse them via common-based cross-attention to obtain a 3D cognition graph. 3) Cognition-Guided Latent Diffusion. We leverage the fused 3D cognition graph as the condition to guide the latent diffusion process for 3D Gaussian generation. Under this unified framework, the 3D cognition graph ensures the physical plausibility and structural rationality of 3D generation. Moreover, we construct a validation subset based on the Marble World Labs. Extensive experiments demonstrate that our Cog2Gen3D significantly outperforms existing methods in semantic fidelity and geometric plausibility. Our code will be released upon acceptance.

Primary area:
generative models
Keywords:
3D scene generation, diffusion model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xq9uOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents Cog2Gen3D, a latent diffusion model for 3D Gaussian generation conditioned on a learned “3D cognition graph”. The 3D cognition graph is formed by pre-trained ResNet50 (semantic tokens), VGGT (geometric tokens), and CLIP (logical tokens). The semantic and geometric tokens form two separate latent graphs which are then fused via cross-attention with the CLIP tokens acting as queries. The denoiser is conditioned on this fused representation and during training the diffusion, node classification and multi-view rendering losses are all combined. CogSG-3D is constructed from publicly available 3D datasets as well as Marble generated worlds.

Critical strengths
  1. Table 6 compares ResNet50, CLIP ViT-L, and VGGT as geometric encoders, with VGGT giving the best 3D-Front results. This provides a useful empirical result for conditioning 3D generation on a geometry foundation model.
  2. The evaluation covers text-to-3D generation and image-to-3D generation of objects and scenes. The ablations cover graph construction, encoder choice, fusion, positional encoding, node count, inference steps, and loss weights.
  3. Table 11 reports inference time and memory use for the method and baselines, providing useful information about practical cost.
Critical weaknesses
  1. I found it difficult to reconcile the latent-graph conditioning in the main text (Eqs. 5–8) with the warped frames and masks described in Appendix A.2. The connection between these representations is not explained, and the appendix’s temporal-modeling claims contradict Sec. 5.4 and Appendix B.6. The text-only path is also unclear: the semantic and geometric branches require images, but their inputs for T3Bench are unspecified. There are also no experiments varying the graph or comparing multiple outputs for a fixed input to demonstrate graph-level control and sampling diversity.
  2. VGGT is trained at a normalized canonical scene scale, but the paper does not explain how its features or the learned positional embeddings establish calibrated physical scale. Tables 6 and 8 demonstrate benefits from geometric features and positional encoding, respectively, but do not directly validate scale recovery or consistency. Describing EchoScene and Layout2Scene as limited to “2D relative” geometry is also inaccurate, since both explicitly use 3D geometry.
  3. Table 2’s OmniObject3D FID/MMD values match DiffTF’s class-conditional results, but it is unclear how these are comparable to image-conditioned generation. Table 1’s combined T3Bench quality/alignment scores are called quality scores, and the DreamFusion row repeats SJC’s surroundings/multi-object scores and overall average. The procedure used to obtain the proposed method's scores is unspecified. Table 3 does not explain image-to-graph/layout conversion for EchoScene and Layout2Scene or alignment/cropping for LucidDreamer. Using LucidDreamer as an object baseline in Fig. 15 also needs justification. Missing comparisons include TRELLIS[1], Hunyuan3D 2.0[2] and TripoSG[3] for objects; GALA3D[4], DecompDreamer[5] and LayoutDreamer[6] for composition; and DreamMesh[7] and VP3D[8] on T3Bench.
  4. The qualitative comparisons show one rendering per output, and the supplementary video pans around the frontal view without revealing the objects’ sides or backs. This makes it difficult to assess the claimed structural quality and multi-view consistency, particularly on surfaces unseen in single-image inputs. The views used for Table 2’s image metrics are also unspecified.

References:

[1] Xiang, Jianfeng, et al. “Structured 3D Latents for Scalable and Versatile 3D Generation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 21469–21480.

[2] Zhao, Zibo, et al. “Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation.” arXiv, 2025, arXiv:2501.12202.

[3] Li, Yangguang, et al. “TripoSG: High-Fidelity 3D Shape Synthesis Using Large-Scale Rectified Flow Models.” arXiv, 2025, arXiv:2502.06608.

[4] Zhou, Xiaoyu, et al. “GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-Guided Generative Gaussian Splatting.” Proceedings of the 41st International Conference on Machine Learning, PMLR, vol. 235, 2024, pp. 62108–62118.

[5] Nath, Utkarsh, et al. “DecompDreamer: A Composition-Aware Curriculum for Structured 3D Asset Generation.” Transactions on Machine Learning Research, 2026.

[6] Zhou, Yang, et al. “LayoutDreamer: Physics-Guided Layout for Text-to-3D Compositional Scene Generation.” Pattern Recognition, vol. 178, 2026, article 113427.

[7] Yang, Haibo, et al. “DreamMesh: Jointly Manipulating and Texturing Triangle Meshes for Text-to-3D Generation.” Computer Vision – ECCV 2024, Lecture Notes in Computer Science, vol. 15117, Springer, 2025, pp. 162–178.

[8] Chen, Yang, et al. “VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 4896–4905.

Questions for authors

Refer to Weaknesses

Reviewer_eajzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Cog2Gen3D, which generates 3D Gaussian objects and indoor scenes from images and text. Semantic, geometric and logical tokens are extracted with ResNet50, VGGT and CLIP, organized into a semantic graph and a geometric graph, and fused into a "3D cognition graph" that conditions a latent diffusion model. The motivation is that scene-graph and layout-based methods only model relative relations and therefore get absolute scale wrong, while semantics-guided methods often produce physical violations such as intersecting objects (Fig. 1). The authors also assemble a dataset, CogSG-3D, from several public 3D datasets and self-built data, and evaluate on T3Bench (text-to-3D), ShapeNet and OmniObject3D (image-to-3D objects), and 3D-Front (image-to-3D scenes).

Critical strengths
  1. The two model-level contributions are concrete and supported by ablations: VGGT as the geometry encoder, and graph-structured fusion of semantic and geometric features through logical tokens. Tables 4–9 vary the token types, graph vs. flattened tokens, the geometry backbone, the fusion strategy, the 3D positional embedding and the number of nodes, and each part contributes to the final result. This is the part of the paper that I find convincing on its own terms.

  2. The benchmark numbers are strong: 56.6 average on T3Bench vs. 45.7 for GaussianDreamer (Table 1), and the best CD / F-Score / IoU on 3D-Front among all compared methods (Table 3). If these results hold up, the method is a useful step for text/image-to-3D generation. My concerns below are about whether they show what the paper claims, and how far they can be trusted.

Critical weaknesses
  1. The central claim is not tested. The paper is motivated by scale inconsistency (Abstract, Sec. 1, Conclusion) and physical violations such as object intersections (Fig. 1), but I could not find an experiment that measures either. Figure 3 shows that VGGT features give better cross-view correspondence than ResNet50 features, which is not the same as recovering absolute scale. CD, F-Score and IoU in Table 3 measure overall shape agreement, and since the alignment/normalization protocol is not stated, scale errors may be largely normalized away. The user-study score is subjective, and object intersections are mentioned only in the Fig. 1 caption. This is decisive for my rating: the experiments show that the model scores well, not that it fixes the problems the paper is built around. Addressing this needs new experiments, e.g. relative-size error and an object-intersection rate on 3D-Front against the Table 3 baselines, not a clarification.

  2. The reported results are hard to trust. In Table 10, CD, F-Score and IoU are identical (0.063 / 58.43 / 0.683) for 50, 75, 100 and 200 sampling steps, while inference time grows from 132 s to 274 s, and the full-model IoU is 0.682 in Table 3 but 0.683 in Table 10. No variance is reported. CogSG-3D, which is listed as a contribution, comes without a description of how the sources are merged, which categories are kept, or what the train/test splits are, and ShapeNet, OmniObject3D and 3D-Front are used both inside CogSG-3D and for evaluation with no stated exclusion. Sec. 5.2 also says scene generation is evaluated on CogSG-3D, but Table 3 only has 3D-Front. This matters because the strong numbers in Tables 1–3 are the paper's main support, and with these issues I can't tell how far they can be relied on.

Questions for authors
  1. How are scale consistency and object intersections measured? Could the authors report a direct metric for each on 3D-Front, compared with the baselines in Table 3?

  2. Eqs. (1)–(3) and (8) describe the denoiser as conditioned on the fused graph G_cog, but Appendix A.2 says the cognition graph "is explicitly instantiated as a sequence of rendered warped frames and their corresponding valid masks", which are concatenated along the channel dimension, and it also mentions "long-range temporal dependencies" and "scene dynamics". What actually conditions the diffusion model? If it is the warped frames, where do depth and camera poses come from? And what replaces the image input for the text-only prompts in Table 1?

  3. What are the train/test splits for each dataset, and how were evaluation assets excluded from CogSG-3D? How are predictions aligned/normalized before computing CD, F-Score and IoU? Table 2 is presented as image-to-3D but reports only FID/KID/MMD and includes EG3D and GET3D, which are unconditional generators: how is agreement with the input image measured, and how are these baselines conditioned? For Table 3, how are the scene graphs and layouts for EchoScene and Layout2Scene obtained?

  4. Could the authors re-check Table 10 and explain why the metrics are identical from 50 to 200 steps?

  5. Minor: please report the number of participants and examples, the rating scale and the uncertainty for the user study (Table 4). The inference time at the default 50 steps is 132 s in Table 10 but 3.5 min in Table 11, both on 3D-Front. The AI use statement (p. 10) says no generative AI tools were used, but Sec. 4.1 uses Gemini to generate text descriptions for the dataset.

Reviewer_siL5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Cog2Gen3D for 3D scene generation from text or image prompts. It combines ResNet50, VGGT, and CLIP features through semantic and geometric latent graphs, which condition a latent diffusion model to generate 3D Gaussians. An auxiliary node classification loss uses entity labels from explicit scene graphs.

Critical strengths
  1. The paper is generally readable, and the figures provide a useful overview of the framework.

  2. I find the idea of using a cognition graph to organize semantic and geometric information for 3D scene generation a little bit interesting.

Critical weaknesses
  1. I am not fully convinced of the need for the graph. The auxiliary loss only supervises entity categories, so similar supervision could also be applied to ordinary latent tokens. Table 5 reports an improvement from graph construction, but the baseline is only described as “flattening the graph into a token sequence.” It is unclear which graph operations are removed and how the tokens are subsequently processed. A clearly specified Transformer baseline with the same encoder features, semantic supervision, and diffusion backbone would help distinguish the contribution of relational modeling from that of the features and auxiliary loss.

  2. The text-to-3D comparisons mainly cover early methods such as LatentNeRF, SJC, and GaussianDreamer, including the room-scale examples in Fig. 14. Comparisons with more recent scene generation methods, such as Lyra or DreamScene, would help establish the framework’s competitiveness. Differences in training data and input conditions should be clearly stated. Such comparisons would contextualize overall generation quality, while controlled ablations should establish the contribution of the graph.

Questions for authors
  1. In Table 5, what replaces graph construction and message passing? How are the flattened tokens processed, and is the same node classification loss retained?

  2. For the reported text-to-3D experiments, what inputs are supplied to ResNet and VGGT? Is a reference image first generated from the text?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.