acceptodds
Withdrawn from ICLR 2027

UniGaussianGPT: Composable Autoregressive Semantic 3D Gaussians Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in 3D generative modeling have enabled the synthesis of detailed scenes with coherent geometry and appearance, mostly through diffusion models that refine an entire scene holistically. Autoregressive formulations offer a complementary paradigm that constructs Gaussian scenes step by step, bringing a compositional inductive bias under which generation and completion follow from the same next-token prediction. However, these advances largely focus on producing renderable scenes, while recognizing their contents remains a separate task. Semantic information links geometry to the objects and regions it represents, providing a foundation for scene understanding and downstream applications such as robotics and AR/VR. We therefore present , an autoregressive framework that unifies 3D Gaussian scene generation and semantic understanding by learning to predict scene contents and their semantics from shared contextual representations. To support this learning, we build labeled training scenes by composing Gaussian assets converted from complete furniture meshes with separately fitted empty rooms, and supervise Gaussian attributes directly so that occluded surfaces survive tokenization. Generated scenes are thus composable, consisting of semantically labeled and individually complete components. Extensive experiments on composed 3D-FRONT scenes show that achieves state-of-the-art performance in both semantic understanding and scene generation and completion, while better retaining occluded content through tokenization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Autoregressive Generation, Semantic Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wiAYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends GaussianGPT with semantic prediction. A semantic head reads the hidden state after each feature token. It is trained jointly with next-token prediction. The authors also construct composed Gaussian scenes and add direct attribute supervision to the tokenizer. The model supports scene generation, completion, and semantic labeling.

Critical strengths
  1. The motivation for preserving occluded surfaces is clear. Rendering losses alone provide weak supervision for these regions.
  2. The data construction pipeline provides complete furniture assets with semantic labels.
  3. The reported improvements are substantial. The semantic baselines are also fine-tuned on the proposed data.
Critical weaknesses
  1. The contribution is incremental. The overall backbone largely follow GaussianGPT. The main addition of network is a two-layer semantic head.
  2. The contribution of semantic learning to generation is not clear. Table 4 reports validation cross-entropy of 2.213 with semantic supervision and 2.221 without it.
Questions for authors

1.Does semantic supervision improve generation metrics when the data and tokenizer are fixed? 2. How are separate instances recovered when nearby objects share a category 3. How accurate are the semantic labels on freely generated scenes?

Reviewer_XpVLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds on GaussianGPT to jointly learn scene generation and semantic understanding. It reads the Transformer state after each feature token to predict the category of the corresponding spatial cell, then assigns that label to the decoded Gaussians. Category predictions are not fed back as conditioning inputs, leaving the original autoregressive sampling procedure unchanged. The authors also construct training scenes by composing Gaussian assets converted from complete furniture meshes with empty-room backgrounds, and introduce direct attribute supervision to better preserve occluded content during tokenization. Experiments on these composed 3D-FRONT scenes evaluate semantic understanding, generation, completion, and content retention, alongside examples of component removal, rotation, and duplication.

Critical strengths

1.The task is meaningful, and the semantic readout is well motivated: reading the state after the current feature token gives access to both the realized local content and the preceding context, without changing the sampling procedure. In Table 1, the method achieves a 3D mIoU of 0.578 versus SceneSplat’s 0.385, and a 2D mIoU of 0.547 versus 0.441 for the strongest baseline, 3DGS-GBP. All baselines were fine-tuned on the same dataset.

2.Constructing complete assets and empty rooms separately, then directly supervising Gaussian attributes, addresses a practical problem: correct-looking renderings do not necessarily mean that occluded content has been preserved. In Table 3, recall of occluded Gaussians increases from 83.32% to 92.96%, while the recall loss due to the opacity threshold drops from 9.95 to 0.38 percentage points. These results support the value of the combined data and tokenizer design, which also provides a more suitable representation for downstream editing.

Critical weaknesses

1.The source of the generation gains is unclear. Table 4 shows only a small validation CE difference with and without semantic supervision (2.213 vs. 2.221), without corresponding generation metrics or variability. It is also unclear whether the generation baselines were adapted to the composed data, and Table 3 lacks a matched control without the attribute loss. These gaps make it difficult to separate the contributions of semantic training, data construction, and tokenizer fine-tuning.

2.Generated semantics and component editability need stronger validation. Table 1 and Figure 5 evaluate recognition on existing scenes, not label accuracy in newly generated regions. Editing is demonstrated qualitatively, but the spatial-connectivity rules, manual intervention, and handling of nearby same-category objects are unclear. Table 3’s reconstruction results do not directly establish the completeness or editability of generated components.

Questions for authors

1.Were GaussianGPT and WorldGrow in Table 2 adapted to the composed training data? Please clarify their training settings and report generation metrics for the w/o semantic head model using the same tokenizer, data, and training budget, including the number of generated samples and variability across runs.

2.Which checkpoint is frozen in the “frozen GPT” variant in Table 4? Had it already been trained for generation on the current tokenizer’s tokens? If not, please add a control that freezes the trained w/o-head backbone and trains only the semantic head.

3.Was the GaussianGPT VAE in Table 3 also fine-tuned on the composed data? Please report results with the same data and training budget but without , and clarify how hidden and visible Gaussians are defined.

4.What fine-tuning budgets and inputs were used for the semantic baselines in Table 1? How are image-based predictions mapped to 3D cells, and are all methods’ 2D labels rendered using the same geometry? These details would help interpret the large performance gaps.

5.Could you independently assess semantic accuracy on randomly selected generated or completed scenes, focusing on labels in newly generated regions? Evaluating labels only in the given context would not answer this question.

6.How exactly are category predictions and spatial connectivity used to identify editable components? Does this involve manual selection or correction? Please clarify how well the procedure separates nearby objects of the same category, and report success rates over a set of samples or representative failure cases.

Reviewer_uXuGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes UniGaussianGPT, an autoregressive framework that jointly learns semantic recognition and 3D Gaussian scene generation in a shared representation. The same contextual state that predicts the next scene token also predicts the semantic category of the current latent cell, and predicted categories are transferred to decoded Gaussians without being inserted into or fed back into the sequence. Scenes are built from complete furniture meshes and separately fitted empty rooms, with direct attribute supervision to preserve occluded Gaussian content through tokenization. Experiments report state-of-the-art semantic understanding, generation, and completion, plus better retention of hidden content.

Critical strengths

The same autoregressive contextual state predicts both the next scene token and the semantic category at the current location, integrating semantic recognition into the 3D Gaussian scene representation without inserting category tokens or changing the original sampling procedure. And this paper supports semantic component-level operations such as removal, rotation, and duplication.

Critical weaknesses
  • Limited novelty. The method essentially adds a semantic prediction head and a joint semantic loss to GaussianGPT, so that the same autoregressive state predicts both the next token and the category. There are few new algorithmic mechanisms; it is closer to an engineering extension of an existing autoregressive 3D Gaussian model than a substantive methodological advance.

  • Insufficient comparison. Although the paper emphasizes semantic understanding, it mainly compares with a limited set of methods such as SceneSplat and 3DGS-GBP/IGGT/SegVGGT, some of which were not designed for the composed 3D-FRONT setting. A direct baseline—first generating a 3D Gaussian scene, then applying an off-the-shelf segmentation/understanding model and back-projecting the results to Gaussians—is missing, even though such a post-hoc pipeline could also achieve strong segmentation performance.

  • Limited experimental generalization. Semantic evaluation is concentrated on 480 held-out chunks from composed 3D-FRONT with 23 folded categories, without validation across datasets, real scans, different room types, or category-distribution shifts. The ablations only cover a few variants such as w/o semantic head, frozen GPT, and Lovász, lacking systematic analysis of the semantic head design, attribute-loss weight, data composition strategy, occlusion ratio, and class imbalance. Failure cases of semantic prediction and their impact on downstream generation/editing are also not reported.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.