acceptodds
Under review as a conference paper at ICLR 2027

DistScene: Object-to-Scene Distillation for 3D Scene Generation

Abstract

We present DistScene, a framework for single-image compositional 3D scene generation by jointly modeling the environment and individual objects. Unlike existing methods that represent scenes primarily as collections of objects, we model the environment as an explicit scene component to provide geometric context for object placement. Specifically, we introduce Scene-Frame Generation, which jointly generates separate environment and object components in a shared coordinate frame, allowing their geometry and relative placement to be learned together. Then we introduce Object-Centric Refinement to refine each object in a local frame with scene context. Finally, we develop Object-to-Scene Distillation to transfer pretrained object-generation priors to scene generation through automatically composed and rendered synthetic scenes. Evaluations on indoor and outdoor benchmarks demonstrate improved scene-level geometric accuracy over the evaluated baselines.

Primary area:
generative models
Keywords:
3D scene generation, 3D generative models, compositional 3D generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_zG7wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles decomposable 3D scene generation from a single image. The goal is to keep object identity while recovering environment geometry object layout and local detail. The method builds on pretrained TRELLIS.2. It first jointly generates the environment and object components in a shared scene coordinate frame. Each object is then normalized to a local frame refined with image and fixed scene latent conditioning and placed back into the scene. For component-aligned supervision the authors use a pretrained generator to produce object and environment assets and construct training scenes through procedural composition geometric validity checks and rendering. The paper reports geometry improvements on indoor and outdoor benchmarks. Ablations analyze the roles of environment components synthetic training data and scene-conditioned refinement.

Critical strengths

1.The method addresses a real and important tension between global layout and local detail. It jointly generates the environment and separate objects in a shared scene frame and then refines each object with scene-conditioned local refinement so scene-level organization and object-level resolution are handled separately but stay linked. This is a clearly motivated and practically useful design compared with generating objects alone and relying on external assembly and the data pipeline keeps scene-object coordinate correspondences for matched supervision.

2.The ablations give useful evidence. They test the environment component and the refinement context separately. The results support the core motivation that local refinement should keep scene context.

Critical weaknesses

1.The distinction from closely related compositional generation methods needs to be developed more clearly. PartCrafter already jointly generates multiple separate components from a single image without pre-segmentation and in a shared global frame. So joint component generation independent identity and shared coordinates are not the main novelty of DistScene. DistScene cites PartCrafter but treats it mainly as object-level part generation and does not explain the relation to the multi-object scene setting or give a direct comparison.

2.It is unclear whether inference starts from a coarse latent or random noise and whether the coarse geometry is an initial state a condition or only sparse coordinate support. It is also unclear how the Eq. (8) correspondence is built when training targets and inputs have different sparse coordinates.

Questions for authors

Q1. Please clarify the relationship between Eq. (8), Figure 7, and Appendix A.1. How are z_0,z_1, and the intermediate state z_t constructed during refinement training? What is the initial state at inference?

Q2. Please provide a focused technical comparison between Scene-Frame Generation and PartCrafter's scene-generation formulation, distinguishing their shared design elements from the additional environment and refinement mechanisms.

1 comment
AuthorsOpenReview ↗

Response for zG7w

We thank Reviewer zG7w for the constructive feedback and for recognizing our design for balancing global layout and local detail, as well as the supporting ablations. We address the overlapping concerns together below.

W1 / Q2. Technical distinction from PartCrafter and direct comparison

We agree that joint component generation, separate component identities, and shared scene coordinates are shared design elements. The main differences between Scene-Frame Generation and PartCrafter’s scene-generation formulation are:

  1. Explicit environment generation. PartCrafter jointly generates separate object meshes in a shared scene coordinate frame. Our Scene-Frame Generation instead jointly generates an explicit environment component and multiple object components, allowing objects to exchange information with the surrounding environment during generation.

  2. Object-count specification. PartCrafter conditions generation on a predetermined number of components and initializes the corresponding number of noisy latent sets. In contrast, DistScene uses a fixed maximum number of sparse-voxel component slots and determines the final object count by retaining non-empty slots, without requiring a per-image object count as input.

Beyond Scene-Frame Generation, DistScene introduces scene-conditioned Object-Centric Refinement for local geometric detail and Object-to-Scene Distillation for component-aligned training supervision.

We have revised Related Work to acknowledge PartCrafter’s support for both decomposable objects and multi-object scenes. We also add direct qualitative comparisons in Figures 6 and 7 (Sec. 5.2), and demonstrate refinement of PartCrafter’s outputs in Figure 12 (Appendix A.8).

W2 / Q1. Refinement training, inference initialization, and sparse-coordinate correspondence

Thank you for pointing out this ambiguity. We have corrected the Method section in Secs. 4.3 and 4.4, replacing the description of a degraded coarse latent as the flow’s initial state.

During training, z1z_1 is the canonical object target retained during Object-to-Scene Distillation, and z0z_0 is Gaussian noise defined on the same local sparse support. The intermediate state is constructed as

zt=(1−t)z0+tz1.z_t=(1-t)z_0+t z_1.

Because z0z_0 and z1z_1 share sparse coordinates, interpolation and the velocity target z1−z0z_1-z_0 are computed token-wise within the object-local representation. No point-wise correspondence between scene-frame and object-local tokens is required.

At inference, the predicted coarse object mesh is normalized and voxelized to define the local sparse support. Latent features on this support are initialized with Gaussian noise and generated conditioned on the input image and fixed scene-frame latents. Thus, the coarse geometry provides the support and condition, rather than the initial latent features.


Thank you again for your comments. We are happy to continue the discussion if anything remains unclear.

Reviewer_rmszOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposed and interesting approach for image to compositional 3D scene generation with disentangled objects. The method consists of two stages, where in the first Scene Frame Generation stage objects and environment are jointly modelled via sparse voxel latent representation and a flow model is trained to jointly predict the coarse latent representation for N object and environment conditioned on the input image. In the second Object-Centric Refinement stage the model performs refinement of individual 3D objects generated in the scene while being conditioned on the coarse complete scene structured latents along with the input image. The model is trained as a LoRA on state-of-the-art image-to-3D Trellis.2 model and with a synthetically created dataset consisting of compositions of synthetically generated 3D assets placed in the generated 3D environments. The results of the proposed approach is promising and generalizes to be various scenes both for indoor and outdoor environments.

Critical strengths

[Simple and End2End approach]: I really liked the end the end approach of image to 3D scene generation with minimal cascading of models. Existing approaches have a cascade of 3D reconstruction, segmentation and refiner modules making the pipeline complex and error prone. The proposed method has a very simple and intuitive pipeline - jointly generate coarse object and scene representations and refine the object representations thereafter. The objective is simple flow objective but used in two different context of 3D latent generation and refiner.

[Promising Results]: The results presented in the paper are very good with close resemblance to the input image components - individual object shapes and appearances, consistent object placements and background extent. The quantitative results further substantiate the superiority of the proposed method over the baselines across all the metrics and significantly better in the user study. I think the End2End nature of the proposed method has provided the gains as compared to the baselines that accumulate the errors from different models.

Critical weaknesses

I don't have any major concerns in the paper, but want to understand the proposed method with some ablation studies:

[Scalability of Data]: The model is trained with 125K synthetic scenes, generated with composition of synthetic objects, how does the model's performance change with data scaling. An ablation with using different fraction of the training samples should be provided.

[Generalization to in-the-wild image]: The qualitative results are mostly focused on images of 3D renderings or synthetic images generated using image generation model. Both of which have polished and clean appearance with clean object boundaries and somewhat simpler backgrounds. How does the model perform in real world images that are messy for example an image of students dorm room, or heavy traffic road scene. Further, one assumption in the paper is about the objects being visible in the image for 3D reconstruction, I am curious to know how does the method perform in heavy occlusion scenarios.

[Generalization of the refiner]: As the refiner is trained independently, can it be also used to improve coarse 3D reconstructions generate by other image-to-3D generation models? Demonstrating that the refiner can be plugged with other 3D reconstructions can be an useful application of the paper.

[Refiner module analysis]: The refiner model is quite interesting, as it learns a flow between the coarse mesh representation to finer representation with the velocity prediction. It would be nice to see the transition trajectory path that the refiner model takes from the coarse mesh to finer mesh. And how does the number of intermediate steps affects the quality of the predicted mesh.

Questions for authors

[Why does the image is being conditioned in the refiner]: As the refiner already has the coarse structured scene latent along with the object mesh, what additional information that the input image is providing to the refiner?

L243: How is the learned embedding e_i used to mark the region for corresponding object O_i?


I want to understand the opinion of the authors about this upcoming trend in 3D generation space: Recent LLM models such as GPT-Astra can generate plausible looking 3D scenes from text and even from images. I want to understand the authors thoughts about how the proposed methods and individual models distinguish and have advantage over such 3D generation capabilities of large foundational models. I know in the current form they are extremely costly in generating an accurate image to 3D reconstruction.

2 comments
AuthorsOpenReview ↗

Response for rmsz [1/2]

We thank Reviewer rmsz for the constructive feedback and for appreciating our simple pipeline and promising results in geometry, appearance, and scene layout. We address each concern below.

W1. Scalability of training data

We have added experiments using 25%, 50%, and the full training set in Table 4 and discuss them in the new “Effect of training data scale” paragraph in Sec. 5.3.

Training dataMIDI CD_S ↓MIDI FS_S ↑MIDI CD_O ↓MIDI FS_O ↑MIDI IoU ↑Gen3DSR CD_S ↓Gen3DSR FS_S ↑
25%0.094468.960.198740.760.28460.141462.00
50%0.090569.600.197340.810.30140.136664.31
Full0.087771.590.152951.640.36880.095879.68

Increasing the training data improves all reported metrics across these configurations. These results suggest that Object-to-Scene Distillation provides an effective way to expand training supervision and improve scene generation quality.

W2. Generalization to in-the-wild image

We have added qualitative comparisons on cluttered scenes and occluded inputs in Figures 6 and 7, discussed in Sec. 5.2. Figure 6 shows coherent scene structures and plausible layouts under complex object arrangements, while Figure 7 illustrates plausible completion of partially observed content, including a vehicle beneath a pedestrian bridge and walls extending beyond the image boundary.

However, it is still very difficult for our current model to handle extremely challenging scenes, such as densely cluttered dorm rooms or heavily occluded traffic. Expanding our training data to include more complex layouts and stronger occlusions may help address this limitation in future work.

W3. Generalization of the refiner

We have added before-and-after refinement examples for MIDI, SceneGen, PartCrafter, 3D-Fixer, and SAM3D in Appendix A.8, Figure 12. Our independently trained Object-Centric Refinement module improves local object structures in these examples, including wheel spokes, tire treads, and vehicle body contours. These results illustrate its potential to enhance object geometry produced by other scene-generation pipelines.

W4. Refiner module analysis

We have added intermediate refinement outputs at sampling steps 1, 4, 8, and 12 in Appendix A.9, Figure 13. In these examples, recognizable vehicle surfaces emerge by step 8, while step 12 produces cleaner surfaces and more distinct wheels, body contours, and roof structures. We also clarify in the Method section that refinement generates local object geometry from Gaussian noise, using the coarse object to define local sparse support and the image and scene latents as conditions. Thus, Figure 13 visualizes intermediate states along a sampling trajectory, rather than directly transforming coarse latent features into fine features.

AuthorsOpenReview ↗

Response for rmsz [2/2]

Q1. Why does the image is being conditioned in the refiner

The image supplies object-specific details that guide refinement beyond the coarse geometry and scene context. Our comparison in Appendix A.10, Figure 14 shows that removing direct image conditioning produces rough surfaces and incomplete local structures.

Q2. How does the learned embedding eie_i mark the target object?

To identify the scene component corresponding to the object being refined, we assign the same learned slot embedding eie_i to that object’s scene-frame tokens and object-local refinement tokens, while other components use different slot embeddings. This embedding is added to each token’s projected features before the Transformer blocks, together with a role embedding distinguishing scene-frame from object-local tokens. We will further release our code to support reproducibility.

Q3. Relationship to large foundation models

Recent LLMs can already generate plausible 3D scenes from text and images, but two challenges remain. First, generating high-resolution, detailed 3D scenes can be slow and require substantial token consumption, resulting in high costs. Second, recovering accurate 3D layouts from reference images remains challenging; depth estimation provides useful geometric guidance, but does not directly yield a complete mesh scene.

We therefore envision a mutually reinforcing loop between DistScene and LLM-based agents. On one side, DistScene could provide agents with a dedicated, potentially lower-cost tool for 3D scene generation: Scene-Frame Generation produces the environment together with coarse object geometry and layouts, while Object-Centric Refinement improves object-level geometric detail. On the other side, agents could participate in scene planning and data construction through Object-to-Scene Distillation, to build larger, higher-quality, more diverse datasets with plausible layouts. In this loop, agents help improve the training data, the data strengthen DistScene, and the improved DistScene provides agents with more capable 3D scene generation tools.


Thank you again for your comments. We are happy to continue the discussion if anything remains unclear.

Reviewer_qrkNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on the generation of decomposable 3D scenes from a single image. Building on TRELLIS.2, the authors jointly generate the environment and independent objects in a shared coordinate system, then transform the objects into local coordinate frames and use the original image and scene context to recover fine details. To address the scarcity of scene-level training data, the authors use a pretrained generator to produce object and environment assets.

Critical strengths
  1. The motivation is clear. Environment modeling addresses spatial relationships, local refinement improves object-level detail, and automatic data construction addresses the scarcity of supervision.
  2. The method balances scene-level coherence with object-level independence. Objects share a coordinate system with the environment while retaining separate identities, facilitating subsequent object-level operations.
  3. The geometric results are strong, and the ablation studies support the contributions of the main components. The method achieves SOTA geometric results on both indoor and outdoor benchmarks. Ablation studies examine the contributions of environment modeling, training data, local refinement, and scene conditioning.
Critical weaknesses
  1. Table 4 compares the proposed dataset with the MIDI dataset using the same architecture, demonstrating the effectiveness of the synthetic dataset as a whole. However, it is unclear whether the two datasets are matched in size. Consequently, the source of the performance gains remains unclear.
  2. The CD reported for TRELLIS.2 on MIDI-test differs between Table 2 (0.1444) and Table 4 (0.1500 with masks and 0.1610 without masks).
  3. One of the paper’s key motivations is to improve object–environment relationships. However, CD, F-score, and IoU cannot directly assess these relationships (see Questions For Authors for details).
Questions for authors
  1. Could the authors clarify the issues 1 and 2 raised in the Critical Weaknesses section?
  2. Could the authors explain the rationale behind the chosen evaluation metrics, particularly how they assess object–environment relationships? For example, how do these metrics detect interpenetration and floating objects?
  3. Could the authors provide a comparison with other methods in terms of time and GPU memory usage?
1 comment
AuthorsOpenReview ↗

Response for qrkN

We thank Reviewer qrkN for the constructive feedback and for recognizing our motivation, the balance between scene coherence and object independence, and the strong geometric results. We address the concerns below, grouping overlapping questions together.

W1 / Q1. Dataset size and the source of performance gains.

Thank you for highlighting this ambiguity. The original comparison did not match the dataset sizes. We now add SF w/o Env*, trained on our synthetic data with the same number of scenes as the MIDI training set, to Table 4.

MethodMIDI CD_S ↓MIDI FS_S ↑MIDI CD_O ↓MIDI FS_O ↑MIDI IoU ↑Gen3DSR CD_S ↓Gen3DSR FS_S ↑
SF w/o Env0.092267.940.244429.980.17980.136762.04
SF w/o Env*0.098565.250.247929.150.18050.149261.16
SF w/o Env (MIDI)0.098165.740.234733.030.20840.193646.68

At equal size, the two variants (SF w/o Env* and SF w/o Env (MIDI)) achieve comparable scene-level CD on MIDI-test (0.0985 versus 0.0981), while our data achieves lower CD on Gen3DSR-test (0.1492 versus 0.1936). This suggests that the cross-dataset benefit is not explained solely by having more training scenes. We additionally add results using 25%, 50%, and the full training set; increasing the data scale improves all seven reported metrics. These comparisons are discussed in the new “Effect of training data scale” paragraph in Sec. 5.3.

W2 / Q1. Inconsistent TRELLIS.2 results between Tables 2 and 4.

We apologize for the inconsistency. Table 2 inadvertently retained results from an earlier subset evaluation and was not updated with the complete evaluation results. We have corrected the TRELLIS.2 row in Table 2 and explicitly labeled its mask setting. Its MIDI-test scene-level CD is now 0.1500 with masks, consistent with Table 4; the 0.1610 without masks result remains a separate configuration.

W3 / Q2. Evaluation of object–environment relationships.

We agree that CD, F-score, and bounding-box IoU measure geometric accuracy and do not directly assess collisions or support. We have clarified their scope and added object collision rate COL_{ob} and support rate SUP, following SceneEval, in Appendix A.7, Table 8. COL_{ob} measures the percentage of objects colliding with other objects, while SUP measures the percentage supported by other objects or architectural surfaces.

For a fair comparison, all methods use their generated objects with the same ground-truth environments. We additionally evaluate DistScene with its own generated environment as a separate configuration.

MethodMIDI COLob ↓MIDI SUP ↑Gen3DSR COLob ↓Gen3DSR SUP ↑
MIDI23.4486.3782.8188.22
SAM3D29.6380.8571.5184.34
SceneGen43.6082.7878.0586.37
3D-Fixer19.3784.0261.5683.40
DistScene (GT env)18.0682.1923.5683.33
DistScene (own env)18.0696.5223.5689.18

DistScene achieves the lowest object collision rate on both benchmarks. Its jointly generated objects and environment also achieve the highest support rates, consistent with joint object–environment modeling and the geometric validity checks used in our data construction.

Q3. Inference time and GPU memory.

We have added inference-time and peak GPU-memory comparisons in Appendix A.6, Table 7.

MethodMIDI-test CD ↓Gen3DSR-test CD ↓Time (s/scene)Peak GPU memory (GiB)
MIDI0.15500.283019.5112.35
SAM3D0.14860.205517.7220.83
SceneGen0.15020.204054.4219.47
3D-Fixer0.12950.102763.5124.70
DistScene (512)0.09340.097438.5218.00
DistScene (1024)0.08770.0958424.5021.82
Under the evaluated pipelines, our 512-resolution configuration runs faster and uses less GPU memory than SceneGen and 3D-Fixer. The 1024-resolution configuration prioritizes reconstruction quality and achieves the lowest scene-level CD on both benchmarks.

Thank you again for your comments. We are happy to continue the discussion if anything remains unclear.

Reviewer_My2BOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DistScene, a framework for single-image compositional 3D scene generation that jointly models the environment and individual objects to improve scene-level geometric and spatial consistency. Specifically, the method introduces Scene-Frame Generation to jointly generate the environment and objects in a shared coordinate frame, Scene-aware Object-Centric Refinement to enhance local geometric details, and Object-to-Scene Distillation to construct synthetic scene-level supervision from a pretrained object generator. Experiments on indoor and outdoor benchmarks demonstrate that DistScene outperforms existing methods in scene-level geometric accuracy, object-level reconstruction quality, and spatial alignment.

Critical strengths
  1. The paper is well-motivated by the observation that explicitly modeling the environment jointly with objects can improve spatial coherence in compositional 3D scene generation.
  2. The paper is well-written and well-organized, with clear explanations and intuitive illustrations of the proposed framework.
Critical weaknesses
  1. Evaluation details. Unlike previous methods that primarily focus on object generation, DistScene explicitly generates the environment as a separate component. Could the authors clarify how the scene-level CD and F-score in Table 2 are computed, particularly whether environment geometry is included in the evaluation and how a fair comparison with baselines that do not explicitly generate environments is ensured?
  2. Object count accuracy. The joint sparse-structure generation module predicts a variable number of objects through non-empty component slots. It would be helpful if the authors could report the accuracy of the predicted object counts compared with the ground-truth counts on the test sets, to better assess the reliability of this generation process.
Questions for authors
  1. Code availability and reproducibility. The current Reproducibility Statement section appears to contain only the template instructions, without providing any specific information about reproducibility. Could the authors consider releasing the implementation code to facilitate reproducibility and help verify the experimental settings?
3 comments
AuthorsOpenReview ↗

Response for My2B

We thank Reviewer for the constructive feedback and for recognizing the motivation of joint object–environment modeling. We address each concern below.

W1. Evaluation details and comparison with baselines.

The scene-level CD and F-score in Table 2 evaluate only object geometry, excluding the environment, for all methods. We remove our generated environment and follow the same object-geometry evaluation protocol as 3D-Fixer. Thus, all methods are compared on the same geometric scope, ensuring a fair comparison without giving DistScene additional credit for environment generation. We have clarified this in Sec. 5.1.

To separately assess scene relationships, we additionally report object collision and support rates in Appendix A.7, Table 8. All methods are evaluated using their generated objects with ground-truth environments; we also report DistScene with its own generated environment as a separate configuration.

W2. Object-count accuracy.

We have added object-count evaluation in Appendix A.5, Table 6. The predicted count is the number of non-empty object slots remaining after empty-support filtering, excluding the environment, and is compared with the annotated object count.

DatasetExact countOver-countUnder-count
MIDI-test43.6%5.9%50.4%
Gen3DSR-test24.0%51.0%25.0%

These results reveal different counting tendencies: Under-counting is more frequent on MIDI-test, where small objects occupy few scene-frame voxels and may merge with neighboring objects or the environment. On Gen3DSR-test, over-counting is more frequent, as visually separable regions without separate object annotations may be generated as individual components.

Q1. Code availability and reproducibility.

Thank you for identifying the incomplete statement. We have replaced the template text with a concrete commitment to release the trained models, training and inference code, synthetic dataset, and Object-to-Scene Distillation data-generation code upon acceptance.


Thank you again for your comments. We are happy to continue the discussion if anything remains unclear.

Thanks for clarifying how Table 2 compares object geometry across methods and for adding the object-count analysis. The updated release plan also addresses my reproducibility question. I am maintaining my positive rating.

AuthorsOpenReview ↗

Thank you for your feedback

Thank you for your follow-up and continued support. Your suggestions helped us clarify the evaluation protocol, examine object-count accuracy, and strengthen our reproducibility statement. We are glad that these revisions address your concerns and remain committed to releasing the models, code, and data as described.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.