acceptodds
Under review as a conference paper at ICLR 2027

High-Fidelity Single Image to 3D Scene Generation with Foundation Model Orchestration

Abstract

Single-image 3D scene generation supports immersive content creation and embodied-AI simulation. High-fidelity generation must provide complete object structure under occlusion, detailed geometry and appearance, and metrically consistent layout. Training one end-to-end model for all these goals would require samples that jointly encompass natural occlusion, high-quality geometry and texture, and metric-scale layout, making such data difficult to collect at scale. Existing methods therefore often trade off these goals: occlusion-aware generators may lack local detail, detail-rich object generators are sensitive to incomplete observations, and their visually plausible layouts can still have poor metric-scale accuracy. Our key observation is that structured latents naturally separate these roles: occlusion awareness is primarily reflected in the global structure of active voxel coordinates, while feature fields encode local detail and appearance. We exploit this data–capability decomposition through foundation-model orchestration: a heterogeneous latent infusion scheme conditions a detail-rich generator on active voxel coordinates supplied by an occlusion-aware model, to generate high-quality 3D assets under occlusion. To achieve accurate metric-scale layouts, we develop constraints on metric-related pose components based on visible-region boundaries and visible depth, which can be solved by an efficient iterative linear solver. Despite its simple design and straightforward implementation, extensive experiments on the large-scale 3D-FRONT and M3DLayout benchmarks show that our method significantly improves object fidelity and metric-scale layout accuracy, while retaining competitive computational efficiency. These results validate the effectiveness of foundation-model orchestration as a practical approach towards high-fidelity 3D scene generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D scene generation, training-free

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GkGDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FOMO, a training-free pipeline that builds a 3D scene from a single image. For each object, an occlusion-aware generator such as SAM3D gives a coarse mesh and its active voxels. An image editor then turns the mesh rendering and the scene image into a complete object image. TRELLIS or TRELLIS.2 takes the transferred voxels and this image and only synthesizes the local features. For layout, the method keeps the upstream rotation and fits scale and translation from 2D box edges and mean visible depth. Experiments show better object fidelity.

Critical strengths

The motivation is intuitive. SAM3D completes hidden parts but smooths details, while TRELLIS.2 keeps details but breaks under occlusion. Using voxel coordinates as the handoff between two pretrained models is efficient and needs no training or feature alignment. Fixing rotation and fitting only scale and translation keeps calibration cheap. It helps all eight generators and stays stable.

Critical weaknesses
  1. It is not clear that the voxel structure is what carries the occlusion awareness, because Seedream also fills in hidden parts in image space. The gain seems to come from a closed commercial editor, which also hurts reproducibility. The paper also does not say what happens when the edited image disagrees with the voxels.

  2. The core mechanism is closer to existing works than the framework suggests. What passes between the models is a plain occupancy grid on the same 64³ lattice, and TRELLIS itself builds this grid by voxelizing meshes. The TRELLIS paper also already regenerates local details on a fixed coarse structure. So any generator's mesh could be voxelized and refined in the same way, and the paper does not show what the claimed latent compatibility adds over this simple pipeline.

Questions for authors
  1. How much of the gain remains if Seedream is replaced by an open-source editor? How often does the edited image disagree with the transferred voxels? The editor prompt also asks Seedream to add realistic surface detail, so the fine details may already be in the edited image. What happens if SAM3D keeps its own voxels and its own second stage but receives the same edited image? If this comes close to FOMO, the gain comes from a better input image rather than from combining two models.
Reviewer_TLLxOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FOMO, a training-free framework for high-fidelity 3D scene generation from a single image. It combines complementary pretrained models by transferring active voxel coordinates from an occlusion-aware 3D generator to a detail-rich generator, enabling improved geometric details and appearance while preserving complete object structures. Additionally, it introduces a visibility-based calibration method that optimizes object scales and translations using projected bounding boxes and visible depth, improving metric-scale accuracy and scene layout without requiring additional training.

Critical strengths

The proposed framework is reasonable and combines complementary pretrained models in an intuitive manner. It first uses an occlusion-aware 3D generator to recover complete object structures, then employs image inpainting to provide a more informative visual condition for a detail-rich 3D generator, followed by metric-scale calibration. The whole pipeline sounds reasonable for me and it makes improvements over baselines.

Critical weaknesses
  1. Limited methodological novelty: Although the overall pipeline is reasonable, the methodological novelty appears somewhat limited, as the framework primarily orchestrates existing pretrained models rather than introducing a fundamentally new 3D generation mechanism. And this framework is not that novel because almost all existing works follow such an inpainting and generation scheme. The most significant change is that this method first generates the 3D shapes and then uses the coarse shape to improve the 2D inpaintings.
  2. Inference overhead and error propagation: The proposed framework relies on multiple sequential components, which may introduce additional inference overhead and make the overall pipeline sensitive to errors in earlier stages. In particular, inaccurate structural predictions from the initial 3D generator may constrain the subsequent refinement process, while artifacts introduced by the image inpainter may propagate into the final 3D asset. The final calibration stage only adjusts object scale and translation and cannot directly correct such geometric errors. Although the paper reports competitive runtime, it would be helpful to clarify the end-to-end latency under realistic workloads and provide further analysis of failure cases and error propagation across the pipeline.
  3. [Major concern] Insufficient justification for the need for model orchestration: The motivation of the paper relies on the observation that existing occlusion-aware 3D generators may lack fine geometric details, whereas detail-rich generators may struggle with incomplete observations. While the experiments demonstrate improvements over selected baselines, it remains unclear how broadly this trade-off holds and whether the proposed framework offers a substantial practical advantage over strong single-model alternatives. Theoretically and practically speaking, there is no evidence that completion and fine details conflict with each other. Training a better 3D generation method seems to be a more promising direction.
  4. Limited analysis of model selection and generalizability The paper evaluates several combinations of upstream occlusion-aware generators and downstream detail-rich generators, along with ablations of image conditioning and heterogeneous latent infusion. However, the rationale for selecting these particular models and the generalizability of the proposed interface remain insufficiently explored. A more systematic analysis of how the choice of upstream generator and downstream refiner affects the results would help clarify whether the improvements arise from the proposed design itself or depend heavily on specific model combinations. It would also be interesting to discuss whether the proposed approach can be extended to other 3D representations beyond the particular structured latent interface considered in this work.
Questions for authors

Refer to weakness.

Reviewer_fC3TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies single-image 3D scene generation. Due to the difficulty of collecting sufficiently large-scale training data for end-to-end scene generation, existing methods often face trade-offs among recovering occluded structures, preserving detailed geometry and appearance, and maintaining a coherent scene layout.

The key insight is that a structured 3D latent representation separates different types of information: active voxel coordinates capture global structure, while the associated feature fields encode local geometry and appearance details. Based on this observation, the paper proposes FOMO, a training-free foundation-model orchestration framework. Through a heterogeneous latent injection scheme, FOMO uses active voxel coordinates from an occlusion-aware generator (e.g., SAM3D) to guide a detail-rich generator (e.g., TRELLIS), with the goal of generating high-quality 3D assets under occlusion.

For metric-scale scene layout, the authors further observe that only scale and translation depend on metric scale, and therefore optimize only these variables for efficient calibration. Experiments on 3D-FRONT and M3DLayout show improvements in object fidelity and metric-layout accuracy while maintaining relatively high computational efficiency.

Critical strengths
  • The paper studies a practically meaningful image-to-3D scene generation problem.
  • The proposed framework is training-free, which is attractive given the difficulty of collecting large-scale paired training data for 3D scene generation.
Critical weaknesses
  • The initialization and treatment of object rotation are unclear. How is the initial rotation (R_i) obtained? Since the subsequent calibration optimizes only scale and translation, it is unclear how rotational misalignment is handled and how the final object pose is guaranteed to correspond to the input image.

  • The technical contribution appears relatively limited. Optimizing only scale and translation seems more like an engineering simplification than a substantial methodological contribution. In addition, the overall pipeline appears to combine two existing systems: SAM3D first generates a coarse mesh from which active voxels are extracted, and TRELLIS then performs detail-rich generation conditioned on these voxels. This also makes the method less efficient than SAM3D alone. From the qualitative results in Figure 2, the improvement appears somewhat incremental: some objects are still not fully recovered, and fine-grained appearance details, such as the puzzle texture, are not well preserved.

  • The presentation needs substantial improvement. The method section is not sufficiently clear or self-contained, and several important components are introduced without adequate context. For example, around Line 180, G_VLM and the identity-preserving inpainter appear without sufficient explanation of what models they refer to or how they are used in the pipeline. In addition, the practical assumptions are not clearly discussed. The method appears to require camera intrinsics, object masks, and metric depth, but it is unclear how these quantities are obtained in real-world settings and how errors from off-the-shelf estimators affect the final results. I strongly recommend moving the concrete implementations of the occlusion-aware generator and detail-rich generator from the appendix to the main paper. Without explicitly stating that these correspond to models such as SAM3D and TRELLIS, the current terminology is overly abstract and makes the method difficult to understand. Similarly, the descriptions of comparison methods such as CUPID, MIDI, SceneGen, and others should be moved or at least briefly summarized in the main paper. In the current version, it is difficult to interpret the experimental comparisons without first consulting the appendix.

Questions for authors
  • How is the initial rotation (R_i) obtained? Why is rotation not optimized during calibration, and how sensitive is the method to rotation errors?

  • How are camera intrinsics, object masks, and metric depth obtained in practical settings? How robust is the method to errors in these estimated inputs?

  • Please move, or at least briefly summarize, the concrete models used for the occlusion-aware generator and detail-rich generator, as well as the main competing methods such as CUPID, MIDI, and SceneGen, in the main paper so that the method and experiments can be understood without relying heavily on the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.