acceptodds
Withdrawn from ICLR 2027

Easy-3Dfy: Object-Level Geometric Composition for Single-Image 3D Scene Construction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Single-image 3D scene construction is a fundamental problem in 3D vision. Recent methods such as SAM-3Dfy enable end-to-end reconstruction of 3D assets and their scene layouts, but still struggle with fine-grained scene geometry and asset quality. Meanwhile, specialized 3D object generators provide detailed object geometry, while 3D foundation models offer strong and generalizable spatial understanding. We propose **Easy-3Dfy**, a compositional workflow that decouples object generation from scene layout estimation, allowing specialized models to fully exploit their respective strengths. An RPF-based aligner then bridges these capabilities by establishing geometric correspondence between object-local meshes and partial scene pointmaps, recovering object layout for scene composition. Without requiring complete scene-level supervision, the proposed workflow consistently outperforms SAM-3Dfy across multiple geometric metrics on Scene-Maker and 3D-FUTURE200. Our results demonstrate that explicit geometric composition can effectively combine high-quality object generation with strong spatial understanding, while enabling the workflow to benefit independently from advances in both 3D object generation and foundation models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
scene generation, 3D generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_1HU5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Easy-3Dfy separates single-image 3D scene construction into object generation and layout estimation. It extracts partial object point clouds from a predicted pointmap and uses an RPF-based registration module to establish correspondences with complete object meshes. It then estimates a rotation, translation, and uniform scale for each object to assemble the scene. The registration module is trained on object-level data without complete scene supervision. Experiments on Scene-Maker and 3D-FUTURE200 show improved layout metrics and compatibility with different upstream models.

Critical strengths

1.The modular design has practical value. Mesh and point-cloud interfaces connect object generation with layout estimation, making it easy to replace generators, including commercial APIs. Experiments with multiple generators support this flexibility. 2.The main comparison isolates the contribution of layout estimation. Using the same object meshes as SAM-3Dfy allows improvements in scene metrics to be attributed to layout estimation rather than asset differences. Scene-CD, Scene-FScore, and BBox IoU improve on both datasets.

Critical weaknesses

1.The advantages over existing registration methods remain unclear. The method follows the RPF architecture and training objective, while CAST already generates corresponding points and then fits similarity transforms. The modular design is useful, but comparisons with other registration methods under matched conditions are missing. This makes it difficult to assess what the task-specific adaptation adds. The method also uses 50 flow inference steps without clearly reporting the inference hardware, batch size, or peak GPU memory usage. Without comparisons, it remains unclear whether the accuracy gains justify the computational cost. 2.The effects of input errors, generated-asset differences, and inter-object relationships are not sufficiently evaluated. The paper includes pointmap and generator comparisons, geometric augmentation, and failure cases, but does not systematically test different levels of depth distortion and shape mismatch. The final uniform scaling also cannot correct differences in object aspect ratios. 3.The registration module discards RGB to avoid interference from appearance differences. Table 8 supports this choice under the current geometric metrics, but does not assess the cost of losing appearance cues—for example, whether geometrically symmetric objects are oriented consistently with the input image. 4.Objects are also registered separately, without explicit constraints between them. Claims of reduced floating and interpenetration rely mainly on qualitative examples, without quantitative results on contact, support, or penetration. Although the authors discuss metric limitations, these claims still need more direct validation.

Questions for authors

1.What specific improvements does the task-specific adaptation provide over CAST and other registration methods with appropriate scale handling? Can they be compared using the same meshes, pointmaps, and evaluation protocol? 2.What are the registration module’s latency, throughput, and peak GPU memory usage under specified hardware, batch size, and input point count? Compared with other registration methods, do its accuracy gains justify any additional computational cost? 3.Can the effects of pointmap errors, aspect-ratio differences, and local shape changes on layout accuracy be quantified? For geometrically symmetric objects with distinct textures, does discarding RGB affect orientation estimation? 4.Are there quantitative results on penetration, contact, or support relationships that support the claimed improvements in inter-object spatial relations?

Reviewer_SAoVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Easy-3Dfy is a compositional workflow for single-image 3D scene construction. Its central move is to decouple object asset generation from scene-level spatial reconstruction, reformulating single-image scene construction as an object-level point cloud registration problem. Given a complete object asset and the corresponding partial scene point cloud extracted from an image, the goal is to estimate the transformation from the asset's local coordinate system into the scene's pointmap coordinate system.

Critical strengths

Easy-3Dfy's separation of object-asset generation from scene-level placement is enforced by three explicit, typed interfaces (object-local meshes, scene pointmap, local-to-scene transforms), and the method discards every provider pose/layout output and estimates composition purely from scene geometry. The RPF aligner is trained on asset instances from rendered meshes with pixel-aligned depth for ground-truth correspondences, which seems to be more stable than the training-free methods.

Critical weaknesses

The gain over SAM-3Dfy is small, and the experiment does not show that the trained aligner produced it. In the main table both methods use the same SAM-3Dfy meshes, so the new pieces are the MoGeV2 pointmap and RPF. If even one step of the process of completing, meshing or point clouding is incorrect, it will all be included in the same scaling, rotation and translation. There will be no chance to make corrections later. Objects do not correct each other either, and failure scenarios are not included in the Chamfer of the main table. Therefore, the cumulative errors that are built up layer by layer cannot be seen from the numbers.

Questions for authors

What is the scale and pose error of Easy-3Dfy and of SAM-3Dfy before the evaluation alignment? The published metrics first scale each scene to the ground-truth box and then rigidly align it. If Easy-3Dfy is already much closer in the pointmap’s metric frame, Table 1 understates the layout result. If the two methods are close before that alignment as well, the table is the whole result. For each method, how many scenes were left out of Scene-CD, and on what fraction of scenes does Easy-3Dfy have a higher box IoU than SAM-3Dfy when both use the same meshes?

Reviewer_MDhcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a modular workflow for single-image 3D scene reconstruction, decomposing the task into 2D amodal completion, object-level 3D generation, and point cloud registration. The first two stages use existing image-editing and image-to-3D models, while an RPF-based geometric aligner, trained on object-level data, estimates the transformations needed to assemble the generated assets into a scene. Experiments on Scene-Maker and 3D-FUTURE200 demonstrate improvements in scene-level geometric accuracy.

Critical strengths
  1. Modular and flexible design. The framework separates asset generation from layout estimation, allowing upstream components to be independently replaced or upgraded. This flexibility makes the pipeline practically useful for integrating advances in different foundation models.

  2. Reduced dependence on scene-level supervision. The registration module is trained using object-level data and subsequently applied to scene reconstruction. Given the scarcity of high-quality scene-level annotations, this training strategy offers a practical advantage.

Critical weaknesses
  1. Unclear positioning of the contribution. Decomposing scene reconstruction into object generation and layout estimation is an established approach, with related frameworks such as CAST and SceneMaker already exploring similar strategies. Although the proposed workflow is reasonable, the paper does not sufficiently clarify what new methodological insights or capabilities it introduces beyond these existing approaches.

  2. Insufficient evidence for the claimed benefits of decoupling. The paper motivates its design partly by suggesting that jointly modeling asset generation and layout estimation limits reconstruction quality. However, the presented comparisons do not adequately substantiate this explanation. For example, the close-up comparisons in Figure 4 highlight the inferior appearance quality of SAM-3Dfy assets relative to those produced by specialized generators. These models also differ in their representations, generation resolutions, and training procedures, which could explain the appearance gap independently of whether layout estimation is coupled with generation. Consequently, these comparisons demonstrate differences in upstream asset quality but do not isolate the benefits of the proposed decomposition.

  3. Limited methodological novelty and insufficient validation of the registration adaptation. The registration module adopts the architecture and training formulation of the existing RPF framework, with the main task-specific adaptation lying in the construction of object-level training pairs. While the shared-mesh comparison demonstrates improvements over SAM-3Dfy’s layout predictions, it does not establish the advantages of this adaptation over alternative registration approaches. Without such comparisons or targeted ablations of the training protocol, the paper does not convincingly establish a substantial contribution beyond applying an existing registration framework to this task.

Questions for authors

Please refer to the weakness.

Reviewer_3FxnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Easy-3Dfy decouples single-image 3D scene construction into object generation and layout estimation. It uses an RPF-based aligner to establish correspondences between generated meshes and monocular scene geometry, followed by Umeyama fitting. Experiments on Scene-Maker and 3D-FUTURE200 demonstrate improved layout using identical SAM-3Dfy meshes and compatibility with different asset providers.

Critical strengths
  1. Improvements in table.1 provide meaningful evidence of better layout estimation, independently of asset quality.
  2. The geometry-based interface accommodates multiple asset providers
  3. Alignment training requires object-level data rather than complete scene supervision
Critical weaknesses
  1. The workflow primarily integrates existing generators, monocular geometry estimation, RPF, and Umeyama fitting. Section 3.4 explicitly retains the original RPF architecture and training formulation; the main additions are task-specific training and system integration. The methodological advance is therefore limited.
  2. Table 7 shows that alternative generated-mesh providers do not outperform the SAM-3Dfy-mesh configuration on layout metrics. Thus, the experiments demonstrate provider compatibility more convincingly than additional quality gains from the modular design.
  3. Table 1 compares only MIDI, SceneGen, and SAM-3Dfy, omitting relevant approaches such as CAST, Gen3DSR, 3D-Fixer. This leaves advancement over closely related methods unclear.
Questions for authors
  1. Could you elaborate on the main methodological contribution beyond the integration of existing components?
  2. Given the results in Table 7, what quantitative evidence supports additional scene-quality gains from alternative asset generators, beyond compatibility with different providers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.