acceptodds
Under review as a conference paper at ICLR 2027

WorldSculpt: Generating Compositional Worlds from Grounded Videos

Abstract

Generating large-scale cluttered scenes as individually editable object meshes is challenging. Existing compositional methods largely target simple scenes. Our key insight is that, with appropriate design, a single-object generative prior can scale to large-scale, densely cluttered scenes without scene-level training. We introduce , which realizes this idea through canonical-space multi-view conditioning, training-time augmentation, and object-wise scene composition. Finetuned entirely on single-object data, it generates hundreds of mutually occluding objects as complete, spatially aligned meshes in a shared world frame. We further introduce a photorealistic benchmark of densely cluttered scenes with per-object annotations. Evaluations demonstrate superior geometric accuracy and observation alignment over prior approaches. We also demonstrate broader applicability by converting 3DGS worlds from Marble into compositional scenes.

Primary area:
generative models
Keywords:
Mesh Generation, Compositional Scene Generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_A7beOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents WorldSculpt, which generates individually editable object meshes from posed multi-view images, instance masks, and coarse 3D boxes. It adapts Pixal3D through anchor-aligned canonicalization, learned multi-view feature aggregation, and augmentation-based finetuning using only single-object data. The paper also introduces UE-MeshyScene, containing 2,299 objects across six cluttered environments. Validation includes comparisons on four datasets, component ablations, grounding perturbations, and qualitative conversion of Marble-generated worlds.

Critical strengths
  • Strong empirical performance. Improvements extend across four datasets and multiple geometry and alignment metrics. On UE-MeshyScene, [email protected] increases from 0.359 for ShapeR to 0.738. Comparisons with Pixal3D plus the same assembly pipeline also demonstrate benefits beyond scene assembly alone.

  • Useful evidence for scaling object-level priors. Handling up to 701 objects per scene using only single-object finetuning is practically meaningful. The new benchmark and augmentation ablations help substantiate this contribution, particularly under substantial occlusion.

Critical weaknesses
  • The proposed method is somehow incremental. Pixal3D already provides projected feature conditioning, multi-view aggregation, and object-separated scene generation; ShapeR already studies robust object-centric reconstruction from casual captures. WorldSculpt’s distinction therefore rests mainly on its adaptation recipe and demonstrated scale. In Table 5, learned aggregation provides only a modest gain over averaging on UE-MeshyScene, limiting the strength of the architectural contribution.

  • Validation of realistic scenes remains limited. The large-scale benchmark contains six synthetic environments, while the real HouseCat6D evaluation has only 0.34% average occlusion by the paper’s measure. Ground-truth grounding and separately applied synthetic perturbations do not establish robustness to jointly occurring segmentation, association, localization, and camera errors. The Marble demonstration is encouraging but qualitative.

Questions for authors
  • How much improvement remains over native Pixal3D-MV when it receives the same finetuning data, augmentation, and optimization budget? Can the authors quantify performance with jointly estimated grounding on cluttered real captures, for example the ShapeR benchmark?

  • How are the provided ground-truth 3D boxes incorporated into MV-SAM3D? Does it still predict object poses while WorldSculpt uses box-derived placement? A comparison with matched placement, or separate shape and pose metrics, would help clarify the source of the improvement.

1 comment

Thanks for the rebuttal. The authors have addressed part of my concerns, I would like to maintain my rating as weak accept.

Reviewer_8nAvOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper addresses turning a multi-view capture of a large, densely cluttered scene into a set of complete, separately editable 3D meshes, one per object, placed in a shared world frame. The authors' core idea is that a 3D generative model trained only on individual objects can handle such scenes without any scene-level training, as long as each object's observations are brought into the model's object-centric frame and the model learns to cope with occluded and imperfect views. Concretely, given camera poses, per-object masks and 3D boxes that the paper describes as coarse, they fuse image features from the views in which an object appears (up to a cap of 20), use them to condition a pretrained single-object generator that is lightly fine-tuned with simulated occlusion and noise, and place each generated mesh back into the scene. They also build a new synthetic benchmark of six large scenes containing 93 to 701 objects each (2,299 in total), and report large gains over many single- and multi-view baselines on it and on three existing datasets.

Critical strengths
  1. A real problem handled with a simple idea. Current 3D world generators give one fused scene, not separate objects that can be moved or edited. The paper reuses a model trained only on single objects and handles the scene one object at a time, moving each object's views into that model's own object-centered frame, so no training on scenes is needed. This matters to me because the output is directly editable, the approach scales naturally to scenes with hundreds of objects, and it should keep improving as single-object models improve.

  2. Clear gains on the most heavily occluded benchmark. Training with simulated occlusions and noisy cameras and masks is a simple and sensible way to bridge clean object renders and messy scene views. The margin over baselines is largest on the new benchmark, whose occlusion ratio of 48% is far above that of the existing datasets. This matters because heavy occlusion is exactly where current scene reconstruction breaks.

Critical weaknesses
  1. The contribution of the new conditioning is not isolated. Of the conditioning components, Table 5 isolates only the learned aggregator, which plain averaging nearly matches, and without the augmentation the model drops below the base model's own multi-view checkpoint on several metrics. What is missing is the base model's multi-view version trained with the same augmentation but without the new conditioning. The gain is also never related to the level of occlusion within a dataset, which is where the paper's main claim lies.

  2. The scene-level claims rest on idealized inputs and are never checked at the scene level. All quantitative results use ground-truth masks, boxes and cameras; the only example with predicted inputs is qualitative. The robustness test perturbs each annotation type separately and mildly, not in the larger, correlated way a real detector or camera tracker would fail. Objects are generated independently, but nothing measures whether neighbours overlap or float, and silhouette agreement is reported only on the input views. The number of views given to each multi-view baseline is also not stated.

Questions for authors
  1. How would the base model's multi-view version do if it were trained with the same augmentation but without your conditioning? This seems the cleanest way to show what the new conditioning adds.
  2. Within one dataset, does the gain over the baselines grow with how occluded an object is? And between one view and the 20-view cap, how does your method's quality change with only a few views?
  3. On 3D-FRONT the boxes come from the complete meshes, so the box size already tells the model the full extent of a hidden object. How do results change when boxes come only from visible parts, or when box errors are larger and camera errors are shared across a whole frame? Could the predicted-mask-and-box pipeline of the application section be run on one scene that has ground truth, for example one UE-MeshyScene scene, and evaluated with the same metrics?
  4. How often do neighbouring objects overlap or float, compared with the same statistics on the ground-truth meshes, and how well do the shapes match frames not used as input?

If these are answered, I will be glad to raise my score.

Reviewer_HjBoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces WorldSculpt, a framework for generating large-scale, densely cluttered 3D scenes as collections of individually editable object meshes. WorldSculpt adapts a single-object 3D generative prior (Pixal3D) to multi-view scene observations without requiring scene-level training. It does this by extracting and mapping multi-view image features into an anchor-aligned canonical space, fusing them with a learned multi-view aggregator, and finetuning the model purely on single-object data using heavy augmentations (like simulated occlusion and camera noise) to mimic real-world scene degradation. The framework is rigorously evaluated on three small-scale public datasets (3D-FRONT, Toys4k-Scene, HouseCat6D) and a newly proposed large-scale UE-MeshyScene benchmark.

Critical strengths
  • Decomposing large-scale 3D generation into local, anchor-aligned generation tasks using a single-object generative prior offers a highly scalable approach to a difficult problem. It demonstrates a practical way to bypass the need for massive, perfectly annotated scene-level 3D datasets.

  • The methodology is sensible and requires lightweight adaptation of single object priors.

  • Extensive Quantitative Evaluation: The paper presents highly robust numerical evaluations across four different datasets (including real-world, synthetic, and large-scale environments) and compares against 16 distinct baselines.

  • The UE-MeshyScene benchmark provides a much-needed rigorous evaluation environment, featuring up to 701 objects per scene and nearly 50% occlusion, pushing the field beyond simple tabletop datasets.

Critical weaknesses
  • Restrictive input assumptions + overclaiming about depth: The method relies on having multi-view 2D instance masks and 3D world-space localization boxes for every object in the scene. The authors repeatedly claim their method "does not require depth as input". However, obtaining a metric 3D bounding box in a shared world frame inherently requires depth estimation, multi-view stereo, or a pre-existing 3D reconstruction. . Claiming the method does not require depth is an overclaim that masks the complexity of obtaining its prerequisite 3D input.

  • Lack of texture generation and methodological justification: While the paper aims to generate "Compositional Worlds" for downstream applications, the current pipeline is limited to untextured 3D structural geometry. Although the authors acknowledge this omission in the limitations section, it remains a significant gap given that their chosen generative prior (Pixal3D) explicitly supports texture generation. The paper should methodologically justify why texture generation was excluded (e.g., the complexities of fusing multi-view textures under severe occlusion) upfront in the main method section to contextualize the scope of the work, rather than relegating it to a brief admission at the very end.

  • Limited qualitative examples in the submission: The supplementary materials lack an extensive qualitative gallery (e.g., explicitly showing the fragmented, isolated input crops used at inference for various objects) or structural ablations. This makes it difficult to fully assess the model's ability to handle severe occlusion based purely on the static text.

Questions for authors

Please aim to address the issues raised in the main weakness, and specifically answer the question about texture.

How can this method be extended to handle dynamic objects? and can it be applied to in-the-wild image collections of existing places? Can it handle outdoor scenes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.