- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Scene-SAM3D extends a pretrained SAM3D model to reconstruct object-level assets from calibrated multi-view observations without additional training. The method selects informative views using occupancy coverage, combines their information through conflict-aware fusion during generation, and refines the placement of reconstructed objects through Gaussian-based layout optimization. Experiments on Replica and ScanNet++ demonstrate improvements in object reconstruction and scene assembly over single-view and multi-view SAM3D baselines.
Critical strengths
-
The paper presents a practical, training-free approach to extending SAM3D to multi-view scene reconstruction. Coverage-based view selection and conflict-aware fusion address complementary challenges: choosing informative observations and combining them while handling inconsistent predictions.
-
The experimental results show improvements at both the object and scene levels. The ablations help explain the contributions of the proposed components, and the layout optimization substantially improves the consistency of the assembled scene.
Critical weaknesses
-
Limited comparison with scene-level reconstruction methods. The experimental comparisons mainly focus on single-image 3D generation methods and multi-view extensions of SAM3D. While these comparisons help evaluate the proposed view selection and fusion strategy, they provide a limited assessment of the overall scene-level reconstruction capability. The paper lacks sufficient discussion and experimental comparison with relevant scene-level methods, such as InstaScene [1], SimRecon [2], and ShapeR [3]. Such comparisons would help establish the advantages of Scene-SAM3D in reconstructing complete scenes as collections of individual assets.
-
Insufficient evaluation of interacting objects and fine-grained scene decomposition. The presented examples mainly contain relatively independent objects, with limited evaluation of close contact, substantial mutual occlusion, or complex object relationships. Some examples also appear to reconstruct a sofa and its cushions as a single asset, leaving the achievable granularity of scene decomposition unclear. It would be useful to evaluate whether closely interacting components can be reconstructed as separate assets with consistent geometry and placement. Although the layout optimization includes a penetration loss, its effectiveness in these challenging cases is not directly demonstrated through dedicated experiments or quantitative measurements of interpenetration and contact quality.
[1] Zesong Yang et al. InstaScene: Towards Complete 3D Instance Decomposition and Reconstruction from Cluttered Scenes. ICCV, 2025.
[2] Chong Xia et al. SimRecon: SimReady Compositional Scene Reconstruction from Real Videos. CVPR, 2026.
[3] Yawar Siddiqui et al. ShapeR: Robust Conditional 3D Shape Generation from Casual Captures. CVPR, 2026.
Questions for authors
-
Could the authors clarify how Scene-SAM3D differs from InstaScene, SimRecon, and ShapeR in input requirements and output capabilities, and provide a comparison with applicable methods under a compatible evaluation setting?
-
If a sofa and its cushions are provided with separate instance masks, can the method reconstruct and place them as separate assets? Please show such an example, including the resulting contact relationships and any interpenetration.
-
Could the authors provide an ablation of the penetration loss on scenes containing closely interacting objects, reporting both penetration reduction and reconstruction accuracy? Does reducing penetration ever introduce visible gaps between objects that should remain in contact?
5 comments
Response to Reviewer 3GGQ (Part 2)
Due to the character limit, we have divided our response into two consecutive comments. This is Part 2 of 2.
W2 & Q3. Penetration Loss Analysis in Challenging Cases
Thank you for highlighting this. We analyzed penetration loss and reconstruction accuracy in a challenging case containing a sofa and four separately generated cushions. Our Gaussian-based layout refinement runs for 200 iterations. During the first 100 iterations, we optimize the reconstruction objectives, including mask and RGB losses, while monitoring penetration without penalizing it. We then enable the penetration penalty for the remaining 100 iterations.
Stage 1 Stage 2
Steps 0–99 100–199
Penetration penalty OFF ON
Penetration loss ↓ █▆▇▇▇▆▆▆▆▆ ▃▂▁▁▁▁▁▁▁▁
Chamfer distance ↓ █▂▂▃▃▃▃▃▃▃ ▁▁▁▁▂▂▂▂▂▂
Each block represents a 10-step mean.
| Checkpoint | Penetration loss ↓ | CD-L1 (cm) ↓ |
|---|---|---|
| Initialization | 0.374189 | 7.355 |
| After 100 updates, before penetration optimization | 0.119367 | 5.846 |
| After 200 updates | 0.011728 | 5.653 |
Over the 200 optimization steps, penetration loss decreases by 96.9%, while scene-level Chamfer distance decreases by 23.1%. In the cases shown in the supplementary material and on the anonymous project page linked in the abstract, optimization reduces interpenetration without introducing visible gaps between objects that should remain in contact.
References:
[1] DP-Recon [Ni et al., CVPR 2025]
[2] InstaScene [Yang et al., ICCV 2025]
[3] MV-SAM3D [Li et al., Arxiv 2026]
We thank the reviewer for their positive assessment and constructive suggestions! We will incorporate these updates into the revised manuscript and hope our responses address any remaining concerns.
Looking Forward to Your Further Feedback
Dear Reviewer 3GGQ,
Thank you for your thoughtful comments and for raising clear and important questions. Your feedback is helping us strengthen the paper, and we sincerely appreciate your time and effort.
In our rebuttal, we have responded to the points you raised. We would appreciate hearing whether our responses have addressed your concerns and whether you have any remaining questions. We look forward to further discussion and suggestions, as they will help us improve the quality of the paper.
If your concerns have been satisfactorily addressed, we would be grateful if you would consider updating your score accordingly.
Best regards,
The Authors
Response to Reviewer 3GGQ (Part 1)
Due to the character limit, we have divided our response into two consecutive comments. This is Part 1 of 2.
We are sincerely grateful to the reviewer for their positive assessment of our work and deeply appreciate their insightful and constructive feedback. We have carefully considered every comment and provide detailed responses below:
Thank you for suggesting these comparisons. Our original scene-level experiments focused on SAM3D and its multi-view variants to isolate the benefits of our layout refinement under the same pretrained backbone. We agree that broader scene-level comparisons would better establish the capabilities of the full pipeline.
The main differences between our method with InstaScene, SimRecon, and ShapeR are summarized below:
We will incorporate this comparison and discussion into our Related Work section.
Reproduction also requires care. InstaScene's public release does not include its in-situ 3D completion module and scene assembly. SimRecon releases both viewpoint optimization and object-relation graphs modules, but leaves asset generation, alignment, and scene integration to users. These gaps prevent a direct reproduction of the complete published pipelines without additional implementation choices.
For ShapeR, the main contribution is a learned instance-level geometry generator. Our work instead extends a frozen single-image generator to produce textured scene assets and refine their placement. We therefore prioritized methods that recover both geometry and appearance in the additional experiments.
To broaden the evaluation, we reproduced applicable methods as faithfully as possible and added scene-level comparisons with DP-Recon, InstaScene, and MV-SAM3D on the Replica dataset.
* InstaScene denotes our reproduction. Our method achieves the best scene reconstruction quality and the shortest runtime among the compared methods (33.5× and 15.3× speedups, respectively).
Thank you for raising this point. Yes, given separate, consistent instance masks across views, Scene-SAM3D can reconstruct and place closely interacting components, such as a sofa and its cushions, as separate assets. The granularity of our generated assets follows granularity of the input masks.
For the relevant examples in our Replica and ScanNet++ evaluation, these components are grouped in the gt annotations. Simply splitting the merged meshes would leave incomplete reference assets. We therefore follow the benchmark granularity for consistent instance-level evaluation.
We have added fine-grained examples showing separately generated meshes and their placement after our rigid gaussian layout refinement to the anonymous project page linked in the abstract (https://anonymousauthors7903.github.io/Scene-SAM3D/). These examples illustrate our method’s ability to reconstruct closely interacting components as separate scene assets.