acceptodds
Under review as a conference paper at ICLR 2027

SpatialFactory: Unleashing MLLM Spatial Intelligence for Compositional Scene Synthesis

Abstract

High-fidelity and diverse compositional 3D scene synthesis is essential for AR/VR, robot simulation, and the curation of training data for scene understanding and generation. However, existing agentic scene synthesis workflows typically rely on simple in-context linguistic priors while lacking visual planning and imagination, severely limiting the realism and diversity of the generated scenes. To address this issue, we present SpatialFactory, an end-to-end agentic compositional scene synthesis framework built on recent MLLM advances in spatial intelligence. It introduces Multimodal Generation LMs as global top-down visual-spatial imagination engines to leverage vast 2D priors for enhanced generalization and realism, while leveraging Multimodal Understanding LMs and a 3D lifting tool to support a more stable scene parsing and reasoning process from visual hypotheses to compositional 3D scenes without the need for any specialized visual understanding models. Furthermore, to evaluate the effectiveness of the proposed scene synthesis method and the spatial reasoning performance of advanced MLLMs, we introduce a benchmark comprising 111 high-quality, professionally crafted scenes with ground-truth meshes across 12 common indoor categories, accompanied by a comprehensive evaluation pipeline. SpatialFactory significantly outperforms existing agentic methods on this benchmark, while also demonstrating strong generalization and controllability under complex floorplan constraints and detailed textual descriptions. Code and generated scenes will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Compositional Scene Synthesis, Spatial Intelligence Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_SPDwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpatialFactory is an agentic framework that generates compositional 3D scenes from text and floorplan instructions through top-down visual planning, geometric lifting, asset generation, and iterative spatial refinement. Its contributions include this integrated pipeline, a benchmark of 111 professionally crafted scenes across 12 indoor categories, and a toolkit for rendering and exporting generated scenes. Validation includes baseline comparisons using a common foundation model, spatial-understanding experiments across seven models, component ablations, and a preference study, with reported improvements in instruction following and perceived scene quality.

Critical strengths

Strengths are:

  1. Effective integration of visual planning and 3D refinement. The framework combines a global top-down visual plan with geometric lifting and iterative object/layout correction. Its experiment results suggest that this combination is useful for generating detailed scenes from complex spatial requirements.

  2. Substantial evaluation and a useful benchmark. The 111-scene benchmark includes structured layouts and ground-truth meshes, supporting evaluation beyond visual appearance. Comparisons using a common foundation model, spatial-understanding ablations, and a 43-participant human study provide complementary evidence for the reported improvements.

Critical weaknesses

My major concersn are as follows:

  1. The contribution of the proposed framework is not sufficiently isolated. The shared-backbone comparisons and lifting ablations provide some evidence of benefit, but they do not establish how much of the full scene-generation improvement comes from the proposed design rather than the capabilities of pretrained models. This limits confidence in the framework’s methodological contribution and its value as foundation models improve.

  2. The positioning against closely related methods is incomplete. The paper should more clearly distinguish its contributions from agentic approaches and discuss recent image-based scene-generation methods such as I-Scene. A comparison with I-Scene under compatible inputs would help evaluate the image-to-3D realization stage. Without this discussion and appropriate comparisons, the claimed advances in compositional generation and generalization remain insufficiently established.

  3. Insufficient validation of physical stability and difficult relations such as containment and placement underneath objects and lack of runtime estimates by object counts.

Questions for authors
  1. The introduction states that the pipeline does not require specialized detection, segmentation, or depth-estimation models. However, the proposed method use cropped object image for asset generation. Please clarify which stages require segmentation and whether this optional procedure was used in the reported experiments.
  2. The paper describes orientation, scale, and layout correction, but their individual effectiveness remains insufficiently evaluated. How much does each component improve spatial relations, and physics such as collision / stability? Please also clarify whether collisions during refinement are detected through geometric computations or inferred from visual feedback.
  3. Although the representation supports “inside,” “below,” and support relations, their correctness is not evaluated separately. The metric use 3DBB/2DBB to evalaute, how valid containment and support contacts are distinguished from collisions? Mesh-based collision and floating metrics are useful, but do the generated scenes remain physically stable after simulation?
  4. Table 3 supports a benefit from the proposed pipeline, but the improvement from 3D lifting becomes smaller with GPT-6 Astra. Do the other proposed components still provide meaningful gains with stronger foundation models?
  5. The paper acknowledges missed small or occluded objects and distortions in generated assets. How reliably does the crop-to-image-to-3D process preserve the intended object’s shape and details?
  6. Using the same Gemini backbone makes the main comparison more controlled. However, asset-generation components differ, and SceneWeaver’s retrieval initialization was removed. How do these changes affect the results, and how much of the improvement can be attributed specifically to the proposed framework?
Reviewer_i95COpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

this paper presents spatial factory which is an agentic framework for 3d scene generation from language and image descriptions. SpatialFactory first use image generation model to generate a BEV image of the floorplan , then use multimodal LLM to parse the floorplan into object attributes including bounding boxes/spatial relationships/orientations and a 3d lifting procedure to create coarse 3d layouts. Then scene is then finalized with asset generation and iterative refinement. comprehensive experiments show significant improvement compared with baselines

Critical strengths

This framework combine the spatial reasoning capability of image generation model/ MLLM and geometry based tools in different steps which is effective for complex scene generations. Experiment results show significant improvement compared with baselines

Critical weaknesses

while the method is effective for 3d scene generation, the main idea of first generating a 2D scene image and then converting it to 3D scenes has be explored in many existing works, which follow a similar procedure (image generation -> scene parsing -> asset generation -> final refinement). From my perspective, the major difference with existing works is the proposed geometry-based 3D lifting procedure which builds on existing foundation models. therefore, the overall novelty and technical contribution is limited

Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.