acceptodds
Under review as a conference paper at ICLR 2027

Scene-SAM3D: Seeing More of the Whole via View Selection and Conflict-Aware Fusion

Abstract

High-quality 3D scene assets are critical for embodied applications such as robotic manipulation, navigation, and simulation. Despite their strong object priors, recent single-image 3D generation models such as SAM3D remain insufficient for real-world scenes, where severe occlusions, redundant observations, and cross-view inconsistencies make reliable scene generation challenging. To help SAM3D see more of the whole, we introduce Scene-SAM3D, a training-free framework that extends SAM3D from single-view object generation to calibrated multi-view scene asset generation. For faithful assets from observations of varying quality, Scene-SAM3D selects a compact set of complementary views, reducing redundancy while providing additional evidence for regions occluded in individual views. Based on the selected views, it performs step-efficient latent velocity fusion to integrate multi-view evidence and suppress cross-view conflicts in canonical space. Finally, a lightweight rigid-object Gaussian optimization refines the scene layout within 200 iterations while preserving the generated object geometry. Experiments on Replica and ScanNet++ demonstrate consistent improvements at both instance and scene levels, with our method reducing scene-level CD by 43.8% on Replica and 30.9% on ScanNet++, while cutting flow-model sampling FLOPs and wall-time latency by nearly 20% under the same multi-view setting. Our anonymous project page is available at https://anonymousauthors7903.github.io/Scene-SAM3D/.

Primary area:
generative models
Keywords:
3D Generation, Multi-View Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3GGQOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Scene-SAM3D extends a pretrained SAM3D model to reconstruct object-level assets from calibrated multi-view observations without additional training. The method selects informative views using occupancy coverage, combines their information through conflict-aware fusion during generation, and refines the placement of reconstructed objects through Gaussian-based layout optimization. Experiments on Replica and ScanNet++ demonstrate improvements in object reconstruction and scene assembly over single-view and multi-view SAM3D baselines.

Critical strengths
  1. The paper presents a practical, training-free approach to extending SAM3D to multi-view scene reconstruction. Coverage-based view selection and conflict-aware fusion address complementary challenges: choosing informative observations and combining them while handling inconsistent predictions.

  2. The experimental results show improvements at both the object and scene levels. The ablations help explain the contributions of the proposed components, and the layout optimization substantially improves the consistency of the assembled scene.

Critical weaknesses
  1. Limited comparison with scene-level reconstruction methods. The experimental comparisons mainly focus on single-image 3D generation methods and multi-view extensions of SAM3D. While these comparisons help evaluate the proposed view selection and fusion strategy, they provide a limited assessment of the overall scene-level reconstruction capability. The paper lacks sufficient discussion and experimental comparison with relevant scene-level methods, such as InstaScene [1], SimRecon [2], and ShapeR [3]. Such comparisons would help establish the advantages of Scene-SAM3D in reconstructing complete scenes as collections of individual assets.

  2. Insufficient evaluation of interacting objects and fine-grained scene decomposition. The presented examples mainly contain relatively independent objects, with limited evaluation of close contact, substantial mutual occlusion, or complex object relationships. Some examples also appear to reconstruct a sofa and its cushions as a single asset, leaving the achievable granularity of scene decomposition unclear. It would be useful to evaluate whether closely interacting components can be reconstructed as separate assets with consistent geometry and placement. Although the layout optimization includes a penetration loss, its effectiveness in these challenging cases is not directly demonstrated through dedicated experiments or quantitative measurements of interpenetration and contact quality.

[1] Zesong Yang et al. InstaScene: Towards Complete 3D Instance Decomposition and Reconstruction from Cluttered Scenes. ICCV, 2025.

[2] Chong Xia et al. SimRecon: SimReady Compositional Scene Reconstruction from Real Videos. CVPR, 2026.

[3] Yawar Siddiqui et al. ShapeR: Robust Conditional 3D Shape Generation from Casual Captures. CVPR, 2026.

Questions for authors
  1. Could the authors clarify how Scene-SAM3D differs from InstaScene, SimRecon, and ShapeR in input requirements and output capabilities, and provide a comparison with applicable methods under a compatible evaluation setting?

  2. If a sofa and its cushions are provided with separate instance masks, can the method reconstruct and place them as separate assets? Please show such an example, including the resulting contact relationships and any interpenetration.

  3. Could the authors provide an ablation of the penetration loss on scenes containing closely interacting objects, reporting both penetration reduction and reconstruction accuracy? Does reducing penetration ever introduce visible gaps between objects that should remain in contact?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3GGQ (Part 1)

Due to the character limit, we have divided our response into two consecutive comments. This is Part 1 of 2.

We are sincerely grateful to the reviewer for their positive assessment of our work and deeply appreciate their insightful and constructive feedback. We have carefully considered every comment and provide detailed responses below:


W1 & Q1. Comparison with Scene-level Reconstruction Methods

Thank you for suggesting these comparisons. Our original scene-level experiments focused on SAM3D and its multi-view variants to isolate the benefits of our layout refinement under the same pretrained backbone. We agree that broader scene-level comparisons would better establish the capabilities of the full pipeline.

  • Scene-level Methods Discussion

The main differences between our method with InstaScene, SimRecon, and ShapeR are summarized below:

MethodInputs and preprocessingOutputsFocus
Scene-SAM3DMulti-view images, 2D masks, and depthTextured object meshes, refined scene placementTraining-free extension of SAM3D, with lightweight pose and scale refinement
InstaScenePosed images and masks; initial 2DGS reconstructionCompleted object geometry and appearance represented by GaussiansPer-scene Gaussian reconstruction and decomposition, generative completion of partial objects
SimReconVideo, SfM, Gaussian reconstruction, and 2D masksOptimized object views and a scene graph; manual asset placementViewpoint optimization and object-relation graphs
ShapeRPosed images, metric point clouds, detected 3D instances, and generated captionsMeshes without textureA trained multi-modal model for instance-level untextured geometry generation

We will incorporate this comparison and discussion into our Related Work section.

  • Scene-level Method Comparison

Reproduction also requires care. InstaScene's public release does not include its in-situ 3D completion module and scene assembly. SimRecon releases both viewpoint optimization and object-relation graphs modules, but leaves asset generation, alignment, and scene integration to users. These gaps prevent a direct reproduction of the complete published pipelines without additional implementation choices.

For ShapeR, the main contribution is a learned instance-level geometry generator. Our work instead extends a frozen single-image generator to produce textured scene assets and refine their placement. We therefore prioritized methods that recover both geometry and appearance in the additional experiments.

To broaden the evaluation, we reproduced applicable methods as faithfully as possible and added scene-level comparisons with DP-Recon, InstaScene, and MV-SAM3D on the Replica dataset.

MethodCD ↓[email protected] ↑NC ↑Time (h)/scene ↓
MV-SAM3D7.730.550.69—
DP-Recon6.110.630.725.36
InstaScene*5.770.720.662.45
Scene-SAM3D (ours)4.860.730.730.16

* InstaScene denotes our reproduction. Our method achieves the best scene reconstruction quality and the shortest runtime among the compared methods (33.5× and 15.3× speedups, respectively).


W2 & Q2. Fine-grained Scene Decomposition Generation

Thank you for raising this point. Yes, given separate, consistent instance masks across views, Scene-SAM3D can reconstruct and place closely interacting components, such as a sofa and its cushions, as separate assets. The granularity of our generated assets follows granularity of the input masks.

For the relevant examples in our Replica and ScanNet++ evaluation, these components are grouped in the gt annotations. Simply splitting the merged meshes would leave incomplete reference assets. We therefore follow the benchmark granularity for consistent instance-level evaluation.

We have added fine-grained examples showing separately generated meshes and their placement after our rigid gaussian layout refinement to the anonymous project page linked in the abstract (https://anonymousauthors7903.github.io/Scene-SAM3D/). These examples illustrate our method’s ability to reconstruct closely interacting components as separate scene assets.

AuthorsOpenReview ↗

Response to Reviewer 3GGQ (Part 2)

Due to the character limit, we have divided our response into two consecutive comments. This is Part 2 of 2.


W2 & Q3. Penetration Loss Analysis in Challenging Cases

Thank you for highlighting this. We analyzed penetration loss and reconstruction accuracy in a challenging case containing a sofa and four separately generated cushions. Our Gaussian-based layout refinement runs for 200 iterations. During the first 100 iterations, we optimize the reconstruction objectives, including mask and RGB losses, while monitoring penetration without penalizing it. We then enable the penetration penalty for the remaining 100 iterations.

                       Stage 1       Stage 2
Steps                  0–99          100–199
Penetration penalty    OFF           ON
Penetration loss ↓     █▆▇▇▇▆▆▆▆▆    ▃▂▁▁▁▁▁▁▁▁
Chamfer distance ↓     █▂▂▃▃▃▃▃▃▃    ▁▁▁▁▂▂▂▂▂▂

Each block represents a 10-step mean.

CheckpointPenetration loss ↓CD-L1 (cm) ↓
Initialization0.3741897.355
After 100 updates, before penetration optimization0.1193675.846
After 200 updates0.0117285.653

Over the 200 optimization steps, penetration loss decreases by 96.9%, while scene-level Chamfer distance decreases by 23.1%. In the cases shown in the supplementary material and on the anonymous project page linked in the abstract, optimization reduces interpenetration without introducing visible gaps between objects that should remain in contact.

References:

[1] DP-Recon [Ni et al., CVPR 2025]

[2] InstaScene [Yang et al., ICCV 2025]

[3] MV-SAM3D [Li et al., Arxiv 2026]

We thank the reviewer for their positive assessment and constructive suggestions! We will incorporate these updates into the revised manuscript and hope our responses address any remaining concerns.

AuthorsOpenReview ↗

Looking Forward to Your Further Feedback

Dear Reviewer 3GGQ,

Thank you for your thoughtful comments and for raising clear and important questions. Your feedback is helping us strengthen the paper, and we sincerely appreciate your time and effort.

In our rebuttal, we have responded to the points you raised. We would appreciate hearing whether our responses have addressed your concerns and whether you have any remaining questions. We look forward to further discussion and suggestions, as they will help us improve the quality of the paper.

If your concerns have been satisfactorily addressed, we would be grateful if you would consider updating your score accordingly.

Best regards,

The Authors

Reviewer_nsqHOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
1
Presentation
3
Soundness
1(−1)
Read review
Paper summary

This paper extends a pretrained single-image 3D object generation foundation model (SAM3D) to multi-view, object-level 3D scene generation in a training-free manner. The proposed framework consists of three stages. First, Occupancy-Coverage View Selection identifies anchor and helper views based on their occupancy coverage. Second, Step-Efficient Conflict-Aware Multiview Fusion integrates flow velocities from multiple views during the early denoising stages, employing Sign Consensus to suppress conflicting contributions. Finally, an additional Rigid-Object Gaussian Layout Refinement stage optimizes the rotation, scale, and translation (RST) of individual objects using multi-view rendering losses.

Critical strengths

The paper addresses a meaningful problem: extending pretrained single-view 3D generation models to multi-view scene reconstruction without additional training.

Critical weaknesses

The primary weakness of this paper lies in its limited algorithmic novelty. Most of the proposed components are closely related to well-established techniques and offer limited new methodological insights. Moreover, I have substantial concerns regarding the generalizability of the overall framework to challenging real-world scenes.

Limited novelty of occupancy-based view selection. Constructing an occupancy grid to estimate view visibility and select optimal observations appears to be a relatively straightforward application of established techniques. Similar strategies have already been explored in prior works, such as DPRecon and InstaScene. However, these highly relevant works are missing from the discussion, making it difficult to assess the novelty of the proposed view selection strategy.

Insufficient exploitation of calibrated multi-view geometry during generation. The framework assumes calibrated multi-view observations but does not fully exploit the available geometric relationships during the generative fusion process. While camera poses and depth maps are utilized for view selection, these geometric priors are not effectively incorporated into the core multi-view velocity fusion mechanism. Given that calibrated camera poses and depth estimates are already available, a more principled approach would explicitly leverage cross-view geometric correspondences, visibility constraints, or reprojection consistency to guide the generative fusion process.

Post-hoc layout refinement despite SAM3D's native pose estimation. SAM3D already predicts object rotation, scale, and translation (RST), and the proposed framework further benefits from calibrated multi-view inputs. Nevertheless, the generated objects still require an additional offline per-object layout optimization stage to achieve spatial consistency. This raises a fundamental concern about whether the calibrated multi-view information is being effectively exploited during generation. Ideally, a geometry-aware, calibrated multi-view-conditioned generation framework should directly produce spatially consistent object layouts without requiring additional post-hoc optimization. Prior works such as InstaScene and DecomVoxel have already demonstrated relevant approaches to addressing this problem.

Reliance on ground-truth depth for scene refinement limits practical applicability. The method relies on ground-truth depth maps and camera parameters for world-space object placement and Gaussian refinement. However, accurate ground-truth depth is generally unavailable in practical scene reconstruction scenarios, particularly when the input consists solely of RGB observations. Furthermore, the Gaussian refinement stage explicitly relies on depth supervision, raising concerns about its robustness to inaccurate depth estimates. It would be important to evaluate how the framework performs when ground-truth depth is replaced with estimated depth, especially under realistic depth estimation errors.

Insufficient evaluation on challenging scenes and fine-grained instance decomposition. The evaluations on Replica and ScanNet++ primarily involve relatively simple scenes with clearly separable object instances. I strongly encourage the authors to further evaluate the framework on more challenging real-world scenes, including difficult subsets of ScanNet++ and scenes from ZipNeRF, to better assess its generalizability and robustness.

Additionally, I am interested in whether the proposed framework supports fine-grained instance decomposition and reconstruction. For example, in the Replica sofa reconstruction, can the method reconstruct individual cushions as separate 3D assets while preserving their relative spatial relationships? Such an evaluation would provide further insights into the framework's ability to handle complex object structures and fine-grained scene decomposition.

Unclear end-to-end computational efficiency. The overall pipeline appears computationally expensive, involving depth estimation with Depth Anything 3, 3D occupancy construction, multiple SAM3D evaluations, Gaussian proxy construction, and iterative multi-view layout refinement. However, the computational overhead of these components is not sufficiently analyzed. I strongly recommend providing a comprehensive runtime breakdown for each evaluated scene, including depth estimation, view selection, SAM3D generation, Gaussian proxy construction, and Gaussian layout optimization. This would help clarify the practical efficiency and scalability of the proposed framework.

Questions for authors

Insufficient exploitation of calibrated multi-view geometry during generation. The current multi-view fusion process does not effectively leverage the known camera poses and depth maps, resulting in misaligned object layouts that require an additional optimization stage. Could the authors better exploit these available multi-view geometric priors to guide the generation process and achieve spatially consistent object layouts directly?

Insufficient evaluation on challenging real-world scenes. The scenes presented in the paper are relatively simple, and the layout alignment process additionally relies on ground-truth depth maps. I strongly encourage the authors to evaluate the framework on more challenging real-world scenes, such as full-scene reconstruction on ScanNet++ and ZipNeRF. The qualitative results presented in DecomVoxel could serve as a useful reference for evaluating complex scene reconstruction.

8 comments
AuthorsOpenReview ↗

Response to Reviewer nsqH (Part 1)

Due to the character limit, we have divided our response into two consecutive comments. This is Part 1 of 2.

We appreciate the reviewer’s careful assessment and constructive comments. We respond to each concern below and hope our clarifications and additional evidence will support a positive reassessment.


W1. Novelty of View Selection Strategy.

We thank the reviewer for highlighting these relevant works and will expand the discussion. Our selection strategy differs in both its objective and mechanism. DP-Recon uses a learned visibility grid to filter floaters and determine object bounds, then samples novel views around these bounds for SDS optimization. InstaScene selects minimally occluded rendered views based on depth differences between object-only and full-scene renderings as conditions for object completion.

Our method instead selects complementary observations directly from the input images, without first optimizing a scene representation. We choose an anchor view with the largest observed 3D coverage, then greedily select helper views that maximize additional coverage beyond the selected set. This explicitly accounts for redundancy across views and provides complementary evidence for anchor-guided latent fusion. Its effectiveness is supported by Table 5: with MultiDiffusion fixed, coverage selection reduces CD by 10.58% compared with random selection and 13.64% compared with k-means++.


W2(a). & Q1(a). Geometric Guidance in Multi-view Fusion

We thank the reviewer for this valuable suggestion. Geometric information already directly guides our velocity fusion. As described in §3.2 (lines 252–260), we use depth and camera intrinsics to determine anchor visibility in SAM3D’s canonical latent space. This visibility guides multi-view fusion: observed regions follow the anchor’s sign, while unobserved regions use multiview sign consensus. And our fusion also exploits cross-view consistency by aggregating sign-consistent velocity updates from multiple views.

W2(b). & W3. & Q1(b). Clarification of Post-hoc Scene Layout Refinement

Below, we clarify the need for lightweight layout refinement and highlight our method’s advantages over InstaScene and DecomVoxel in pipeline simplicity and computational efficiency.

1. Why we utilize lightweight layout refinement?

Regarding the need for additional layout optimization, SAM3D predicts each object’s canonical-to-camera pose from a single view. Assets reconstructed from different views, including objects that never appear together in one image, must therefore be assembled in a shared world coordinate system using camera extrinsics and depth. Errors in the predicted poses and scales, together with inaccuracies in depth-based initialization, can lead to inconsistent object placement after assembly. We therefore apply lightweight scene-level refinement, jointly optimizing object-level rotation, translation, and scale using calibrated multiview observations while keeping asset geometry and appearance fixed. This takes only 200 iterations and under five minutes per scene on average.

2. Comparison with InstaScene and DecomVoxel.

Our method offers a simpler and more efficient pipeline than InstaScene and DecomVoxel. We will expand the related-work discussion to clearly explain these advantages.

InstaScene first reconstructs and decomposes a Gaussian scene, then completes objects through geometry-guided generation and further Gaussian optimization. DecomVoxel first reconstructs a global sparse-voxel scene with another model GeoSVR, partitions it using instance masks, and performs spatially constrained in-situ completion. Both therefore separate the scene layout prediction and 3D complete mesh generation, establishing object placement through prior scene reconstruction.

Our pipeline instead obtains object assets and initial poses from SAM3D, followed by a lightweight correction of their transforms. This elegantly avoids a separate stage of global scene reconstruction and 3D instance decomposition. In our Replica reproduction, the total runtime is 0.16 hours per scene, compared with 2.45 hours for InstaScene, demonstrating the efficiency of this design even with layout refinement included.

AuthorsOpenReview ↗

Response to Reviewer nsqH (Part 2)

Due to the character limit, we have divided our response into two consecutive comments. This is Part 2 of 2.


W4. Clarification of Using GT Depth and Evaluation with Estimated Depth

Scene Refinement with Estimated Depth.

We thank the reviewer for raising this important concern. We use GT depth in our original scene-level experiments for two reasons:

  1. Controlled evaluation of lightweight layout refinement. Our refinement runs for only 200 iterations. GT depth provides stable initialization and supervision, allowing us to assess the effectiveness of layout optimization under controlled conditions.

  2. Consistent scale and coordinates. GT depth and camera parameters place scene initialization and the evaluation reference in the same metric coordinate frame, allowing us to evaluate layout refinement separately from depth-estimation errors.

To assess practical applicability without GT depth, we additionally evaluate Depth Anything 3 (DA3) estimates.

MethodDepth sourceGS iterationsCD ↓[email protected] ↑NC ↑
Scene-SAM3DGT2004.860.730.73
Scene-SAM3DDA32007.730.620.68
Scene-SAM3DDA34006.470.670.69

With the same 200-iteration budget, using DA3-estimated depth yields lower reconstruction quality than using GT depth. Extending refinement to 400 iterations reduces CD by 16.3%, consistent with less accurate depth-based initialization requiring more optimization. These results demonstrate that our framework can reconstruct scene layouts without GT depth, with additional refinement improving accuracy when estimated depth is used.


W5. & Q2. Evaluation on Fine-grained Instance Decomposition and Real-world Scenes.

We thank the reviewer for these valuable suggestions.

  1. Fine-grained instance decomposition

Yes. Given separate, consistent instance masks across views, Scene-SAM3D can reconstruct closely interacting components, such as a sofa and its cushions, as separate assets with consistent relative placement. The granularity of the generated assets follows that of the input masks.

In the relevant Replica and ScanNet++ examples, these components are grouped in the GT annotations. Simply splitting the reference meshes would leave incomplete component-level assets. We therefore follow the annotation granularity for consistent instance-level evaluation.

We have added fine-grained examples to the anonymous project page linked in the abstract (https://anonymousauthors7903.github.io/Scene-SAM3D/), showing separately generated meshes and their relative placement after Gaussian layout refinement. These examples demonstrate our ability to reconstruct and arrange closely interacting components as individual scene assets.

  1. Evaluation on more challenging real-world scenes.

We thank the reviewer for suggesting additional real-world experiments to strengthen the evidence for our method’s generalizability. To clarify, in our experiments, we use the same Replica and ScanNet++ scene subsets as DP-Recon (Ni et al., 2025) to ensure comparability.

Neither the official ScanNet++ benchmark nor the prior work we follow defines a dedicated difficult-scene subset. To broaden the evaluation as suggested, we have therefore added qualitative results on Zip-NeRF scenes to the anonymous project page linked in the abstract (https://anonymousauthors7903.github.io/Scene-SAM3D/). The released Zip-NeRF dataset does not provide reference meshes for evaluating our geometric reconstruction metrics, so we assess these scenes qualitatively. These additional examples broaden the evaluation beyond the original benchmarks and illustrate our method’s applicability to complex real-world scenes.


W6. End-to-end Computational Efficiency

We thank the reviewer for requesting a detailed runtime analysis. Our full pipeline averages approximately 9.6 minutes per scene. The breakdown below reports the average runtime on an NVIDIA A6000 Pro GPU.

StageTime (s/scene)
Depth estimation and point-map preparation27.11
View selection18.32
SAM3D generation and mesh export232.33
Gaussian proxy initialization, layout refinement, and export293.98
Scene assembly3.76
Total575.50

With an average end-to-end runtime of 9.6 minutes per scene, our framework offers an efficient and practical solution for multiview scene asset generation.

We appreciate the reviewer’s suggestions, which helped clarify our method’s strengths and practical value. We will incorporate these updates into the revision and hope our responses address the concerns and support a positive reassessment.

AuthorsOpenReview ↗

Looking Forward to Your Further Feedback

Dear Reviewer nsqH,

Thank you for your thoughtful comments and for raising clear and important questions. Your feedback is helping us strengthen the paper, and we sincerely appreciate your time and effort.

In our rebuttal, we have responded to the points you raised. We would appreciate hearing whether our responses have addressed your concerns and whether you have any remaining questions. We look forward to further discussion and suggestions, as they will help us improve the quality of the paper.

If your concerns have been satisfactorily addressed, we would be grateful if you would consider updating your score accordingly.

Best regards,

The Authors

Reviewer_2Hb8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends Sam3d from a single-image 3D object generator to a multi-view scene asset generation without any training. There are three parts. First, for each object it selects one anchor view and a few helper views based on how much new 3D occupancy a view adds after projecting masked depth into the world frame. Second, it fuses the view-conditioned velocities of the flow model. Third, the generated objects are kept fixed and only their pose and scale are optimized with Gaussian splatting rendering losses over all views.

Results on Replica and ScanNet++ show gains at both object and scene level with lower flow model cost than fusing all views at every step.

Critical strengths

The paper is practical. Sam3D is strong on single images but using it on real multi-view images is not obvious, and the paper gives a simple training free recipe that works with the frozen model. All the view selection, fusion, alignment split makes sense.

The view selection is simple but it matters a lot, a single well-chosen view already beats several multi view baselines with random views. The sign agreement rule is a cheap way to avoid averaging updates that do not agree, and the Gaussian refinement gives a consistent gain over ICP on both datasets without touching the meshes.

The paper is clearly written and figures are helpful with nice presentation.

Critical weaknesses

The paper claims three things, but I think only one of them is supported properly. my concern is that occupancy coverage view selection plus the conflict-aware fusion giving consistent object level gains is only partly supported. The rigid object Gaussian refinement claim is supported only against very simple baslines. For the view selection, it clearly helps, but most of the object level gain comes from it, and greedy coverage selection is a classic next-best-view idea. For the conflict-aware fusion, which is the early stopping plus the sign agreement rule, the gap to MultiDiffusion fusion and the concurrent MV-SAM3D is a couple of CD points, there is no standard deviation even though random-view results are averaged over multiple seeds, and on some metrics the method is not the best.

So I cannot tell how much of this is real. The sign rule itself is a heuristic with little analysis, and the fusion-step ablation shows that starting late hurts but not that fusing until the end actually causes conflicts, as the paper says.

The paper cites several scene-level methods, including MV-SAM3D and the Ni et al. 2025 work whose protocol it follows, but none of them appear in the scene-level tables. The refinement also uses ground-truth depth and cameras, while view selection uses estimated depth, so we do not know how it behaves without ground truth.

Questions for authors
  • how should I read the scene-level numbers? They are against your own pipeline with ICP, so what would you expect against a method like MV-Sam3D or Ni et al. 2025 under the same protocol?

  • how much does the layout refinement depend on ground-truth depth? Do you have a feeling for what happens with the estimated depth that is already used for selection?

  • random view results are averaged over 5 seeds. How big is the variation, and how many scenes are in each dataset? I ask because the gaps to some baselines are small.

  • what does the efficiency gain look like at 4 views, and how much time do depth estimation, selection and the Gaussian refinement add on top of the flow model?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2Hb8 (Part 1)

Due to the character limit, we have divided our response into two consecutive comments. This is Part 1 of 2.

We thank the reviewer for the constructive feedback and careful reading! Below we responded all concerns point-by-point and sincerely hope for a positive reassessment.


Q1 & W3(a). Comparison with Scene-level Reconstruction Methods

Thank you for question about scene-level comparison. Our original experiments emphasized instance-level comparisons under multi-view input, since object-centric generators such as TRELLIS do not provide a dedicated scene assembly strategy.

The scene-level tables primarily assess the gains from our lightweight layout refinement while keeping the pretrained SAM3D backbone fixed. Comparing ICP and Gaussian refinement within the same generation pipeline helps isolate the contribution of layout optimization.

We agree that broader scene-level comparisons would better establish the capabilities of the full pipeline. We therefore reproduced applicable methods as faithfully as possible and added comparisons with DP-Recon (Ni et al., 2025), InstaScene, and MV-SAM3D on Replica.

MethodCD ↓[email protected] ↑NC ↑Time (h/scene) ↓
MV-SAM3D7.730.550.69—
DP-Recon6.110.630.725.36
InstaScene*5.770.720.662.45
Scene-SAM3D (ours)4.860.730.730.16

* InstaScene denotes our reproduction.

Our method achieves the best reported reconstruction metrics and is substantially faster (33.5× and 15.3× speedups, respectively) than other methods.


Q2 & W3(b). Clarification of Using GT Depth and Evaluation with Estimated Depth

We thank the reviewer for raising this important concern. We use GT depth and camera parameters in our original scene-level experiments for two reasons:

  1. Controlled evaluation of lightweight layout refinement. Our refinement runs foronly 200 iterations. GT depth provides stable initialization and supervision, allowing us to assess the effectiveness of layout optimization under controlled conditions.

  2. Consistent scale and coordinates. GT depth and camera parameters ensure that scene initialization and the evaluation reference share the same metric scale and coordinate frame, avoiding confounding the evaluation of layout refinement with errors from Depth Anything V3 depth estimation.

We also compare results using estimated depth (Depth Anything V3) with GT depth.

MethodDepth sourceGS iterationsCD ↓[email protected] ↑NC ↑
Scene-SAM3DGT2004.860.730.73
Scene-SAM3DDA32007.730.620.68
Scene-SAM3DDA34006.470.670.69

With the same 200-iteration budget, using DA3-estimated depth yields lower reconstruction quality than using GT depth. Extending refinement to 400 iterations reduces CD by 16.3%, consistent with less accurate depth-based initialization requiring more optimization. These results support our use of GT depth to evaluate lightweight layout refinement without confounding its effectiveness with depth-estimation errors.

AuthorsOpenReview ↗

Response to Reviewer 2Hb8 (Part 2)

Due to the character limit, we have divided our response into two consecutive comments. This is Part 2 of 2.


Q3 & W1. Performance Gains and Random-View Variability

Thank you for raising this point. Both MV-SAM3D and our reproduced SAM3D + MultiDiffusion baseline build on the strong pretrained SAM3D model, making further improvements nontrivial. Under the same four-view coverage setting on Replica, our method reduces CD by 4.4% relative to MV-SAM3D and 3.9% relative to MultiDiffusion, without additional training. These gains represent meaningful improvements over strong baselines.

We acknowledge that our method does not lead on every metric. But indoor scene annotations contain incomplete instance meshes due to limited visibility and occlusion. Symmetric CD can therefore over-penalize plausible geometry completed in unobserved regions. Completeness provides an important complementary measure, assessing coverage of the available GT surfaces without directly penalizing completed geometry absent from the reference. On Replica, our method reduces completeness error by 4.9% relative to MV-SAM3D and 4.4% relative to MultiDiffusion.

For appearance metrics, Fig. 4 shows that GT mesh renderings differ from the input images in illumination, brightness, contrast, and material appearance. Image-based metrics consequently reflect these rendering discrepancies as well as reconstruction quality. We therefore regard CD, completeness, and F-score as the main quantitative evidence of improvement, supported by the qualitative comparisons.

To quantify variation under random view selection, we provide mean ± standard deviation over five seeds below. Following the evaluation subsets used by DP-Recon (Ni et al., 2025), we evaluate 7 scenes with 73 instances on Replica and 6 scenes with 87 instances on ScanNet++. The following results are for Replica.

MethodViewsCD ↓Comp ↓NC ↑[email protected] ↑PSNR ↑SSIM ↑LPIPS ↓
MV-SAM3D4 (random)44.70 ± 1.6535.59 ± 1.5872.62 ± 0.530.69 ± 0.0113.13 ± 0.040.71 ± 0.000.42 ± 0.00
MV-SAM3D4 (coverage)41.1732.2475.210.7213.200.710.40
Ours4 (coverage)39.3730.6575.170.7313.200.710.40

The metrics show limited variation across seeds, and our method achieves better results than even the best random-view results across the five seeds.


Q4. Efficiency Study and Full Computational Cost Report

Thank you for requesting this breakdown. Compared with MultiDiffusion, our method reduces both flow-model FLOPs and sampling latency by nearly 20% at 10 views, and achieves a 15.24% efficiency gain at 4 views.

We additionally report the average runtime per scene for each pipeline stage below on Nvidia A6000 Pro.

StageTime (s/scene)
Depth Estimation27.11
View Selection18.32
Flow Model Generation and Mesh Export232.33
Gaussian Refinement293.98
Scene Assembly3.76
Total≈575.5

On average, the remaining pipeline modules add only 343.17 seconds per scene beyond the 3D generation stage.


W2. Analysis of Sign Rule in Conflict-aware Multiview Fusion

We thank the reviewer for this constructive feedback. We will add further analysis of the sign rule.

  1. Motivation of using sign rule. On Replica, we found that 15.66% of velocity elements disagree in sign across views, motivating explicit conflict resolution. When choosing a conflict-resolution strategy, we explored several alternatives in preliminary experiments, including cosine-similarity-based fusion. However, cosine similarity proved insufficiently discriminative: mean voxel-wise cross-view cosine similarity is 0.9033 in Stage 1 and 0.9897 in Stage 2. Such high overall alignment can mask disagreements in individual channels, motivating our element-wise sign rule for resolving conflicts in SAM3D canonical latent space.

  2. Clarification of fusion-step ablation.We would also like to clarify a possible misunderstanding of the results in Figure 6 fusion-step ablation. This ablation evaluates fusion schedules with our sign rule already enabled. Therefore, it does not establish that continuing fusion to the end causes conflicts. Instead, it shows that the benefits of our multiview fusing are concentrated in the early steps. Extending fusion beyond the first 15 steps provides no meaningful quality gain while increasing computation. We therefore fuse only during these initial steps to retain reconstruction quality at lower cost, and will revise the text to make this rationale explicit.

Thanks again for the thoughtful feedback! We will revise the paper to better demonstrate our method’s strengths and practicality, and hope these clarifications address your concerns and encourage a positive reassessment.

AuthorsOpenReview ↗

Looking Forward to Your Further Feedback

Dear Reviewer 2Hb8,

Thank you for your thoughtful comments and for raising clear and important questions. Your feedback is helping us strengthen the paper, and we sincerely appreciate your time and effort.

In our rebuttal, we have responded to the points you raised. We would appreciate hearing whether our responses have addressed your concerns and whether you have any remaining questions. We look forward to further discussion and suggestions, as they will help us improve the quality of the paper.

If your concerns have been satisfactorily addressed, we would be grateful if you would consider updating your score accordingly.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.