acceptodds
Under review as a conference paper at ICLR 2027

SPOON: Towards Coherent Compositional 3D Scene Generation from Uncalibrated Multi-view Images

Abstract

Compositional 3D scene generation aims to recover complete 3D object shapes and their spatial arrangement from visual observations. Recent image-conditioned 3D generators provide strong priors for producing high-quality object geometry, making the generation of complex scenes increasingly practical. A central challenge is therefore to spatially organize these generated assets into a globally coherent scene while remaining consistent with multi-view observations. Existing approaches either entangle scene layout with object generation or separately estimate spatial placement from view-specific observations, where pose hypotheses may remain ambiguous and inconsistent across views, often resulting in an incoherent object-camera soup. We introduce SPOON, a framework that reformulates multi-view compositional 3D generation as scene-level, geometry-grounded pose reasoning. Rather than treating view-specific object pose hypotheses independently, SPOON coordinates them using reconstruction-derived multi-view geometry through a Guide-Route-Reconcile paradigm. This progressively organizes object poses and camera configurations into a coherent scene-level spatial arrangement. Extensive experiments on ARSG-110K and MIDI-3D-Front demonstrate consistent improvements in object placement and scene composition across varying numbers of input views. On ARSG-110K, SPOON reduces scene-level and object-level Chamfer distances by 12.7% and 17.7%, respectively, compared with a strong baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D scene reconstruction and generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RyeYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a pose-reasoning pipeline for multi-view compositional 3D generation. Instead of treating view-specific object pose hypotheses independently, the proposed method coordinates the hypotheses using multi-view geometry through a Guide-Route-Reconcile paradigm. It first uses VGGT-omega to estimate the camera poses to guide the object pose generation, then it improves object layout by selecting the most plausible layout based on the geometric observability. Lastly, it uses differentiable Gaussian Splatting to jointly optimize the camera poses and object poses. Results show that the proposed method can produce a more accurate object placements and scene layouts compared to the baselines methods.

Critical strengths
  1. The idea overall is clearly presented, and the paper is also well-motivated.
  2. The obtained scene layouts and object placements are better than the baseline models on ARSG-110K and MIDI-3D-Front.
Critical weaknesses

The whole system seems to be a combination of multiple existing methods / techniques, which raises the concerns of 1) novelty, 2) robustness if individual components fail, and 3) computational overhead. I would like to hear how the authors defend their ideas, and would like to see a robustness analysis.

Questions for authors

There are quite some post-processing refinement steps like in sec 4.2 and 4.3. Can you apply the proposed Route and Reconcile steps to the output of MV-SAM3D or other methods? This is to see if the proposed components themselves are universally useful, and which parts of the system bring most of the gains.

5 comments
AuthorsOpenReview ↗

Response to Reviewer RyeY

1. Novelty and Technical Contributions.

The central contribution of SPOON lies in geometry-grounded pose reasoning for coherent compositional scene generation. While existing models provide reconstructed scene geometry and generated object assets, accurately placing these assets into a coherent scene-level configuration remains challenging.

SPOON addresses this challenge by making reconstruction-derived geometry directly inform three stages of object placement: Guide establishes a reconstruction-to-canonical surface bridge and incorporates geometry-derived pose guidance into the generative flow. Route evaluates the geometric reliability of view-conditioned pose hypotheses, and Reconcile jointly refines object placements and camera poses while preserving the generated asset geometry.

Importantly, these contributions are evaluated under the same foundational models as MV-SAM3D, including its existing layout injection and pose refinement. The controlled ablations in Table 4 further demonstrate the benefits of the proposed mechanisms beyond the inherited components.

2. Robustness to Component Failures.

SPOON incorporates an explicit fallback mechanism to reduce the risk of unreliable geometric guidance during pose generation. Specifically, Guide is disabled when fewer than four valid views are available or the surface-alignment RMSE exceeds 0.15, reverting to the original pose generation process while retaining Route and Reconcile.

We further analyze Guide's reliability on ARSG-110K (4 views) by ranking objects according to their geometric reconstruction error, categorizing the worst 20% as Poor and the remaining 80% as Normal. As shown below, the overall fallback rate increases from 37.70% to 59.78% for the Poor group. Notably, the insufficient-view rates remain comparable (28.42% vs. 29.35%), whereas the conditional RMSE fallback rate rises substantially from 12.98% to 43.08%. This indicates that the surface-alignment reliability check more frequently rejects guidance for objects with larger reconstruction errors.

Guide fallback under varying reconstruction quality on ARSG-110K (4 views).

Reconstruction QualityObjectsN_valid < 4 (%)RMSE Fallback (%)Total Fallback (%)
Normal36628.4212.9837.70
Poor9229.3543.0859.78

RMSE fallback is measured among objects with at least four valid views; the other rates are computed over all objects in each group.

Furthermore, Table 4 evaluates the effect of removing individual components on MIDI-3D-Front (4 views). Removing Guide, Route, or Reconcile yields IoUs of 0.677, 0.669, and 0.617, respectively, compared with 0.684 for the complete pipeline and 0.597 for MV-SAM3D. These results show that SPOON retains improvements over the baseline even when an individual stage is omitted.

3. Computational Overhead.

We further profile the computational overhead of SPOON across 4/8/12 input views and 2/4/6 objects, reporting the runtime of each stage and peak GPU memory below.

Guide and Route introduce modest additional costs, accounting for only 3.8%–6.1% of the MV-SAM3D baseline runtime across the evaluated settings. Reconcile constitutes the primary additional runtime due to joint object–camera optimization. Importantly, all stages execute sequentially, and none exceeds the peak GPU memory of the baseline generation process. Thus, SPOON maintains the same peak GPU memory footprint as MV-SAM3D across the evaluated configurations.

These results characterize the computational trade-off: geometry-guided generation and pose routing introduce limited overhead, while scene-level reconciliation incurs additional optimization time to improve object placement and spatial coherence.

Stage-wise runtime and peak GPU memory under varying numbers of views and objects.

ViewsObjectsBaseline (s)Guide (s)Route (s)G+R OverheadReconcile (s)Peak Mem. (GB)
4271.893.500.515.58%65.3621.88
44136.216.381.025.43%88.8630.98
46204.9010.781.656.07%127.9332.98
82144.485.440.984.44%134.3921.98
84309.739.741.993.79%189.9730.75
86393.3815.613.244.79%240.7432.83
122199.207.041.414.24%195.5822.20
124427.3913.263.013.81%270.8529.94
126637.7122.336.404.51%344.3635.28

G+R Overhead is measured relative to the MV-SAM3D baseline runtime. Peak memory is unchanged from the baseline, as all stages are executed sequentially.

AuthorsOpenReview ↗

Response to Reviewer RyeY

Q1. Applicability of Route and Reconcile to Existing Methods.

Yes. Route and Reconcile can be applied to the generated object assets and view-conditioned pose hypotheses of MV-SAM3D. This is supported by our controlled ablations. On MIDI-3D-Front (4 views), applying Route alone improves layout IoU from 0.597 to 0.617 (Table 2), while further incorporating Reconcile increases it to 0.677 (Table 4, w/o Guide), compared with 0.684 for the full SPOON pipeline. These results show that the downstream pose coordination stages contribute effective gains independently of Guide.

Our additional few-view experiments further support this finding. On ARSG-110K, Route improves IoU from 0.5020 to 0.5216 (2 views) and from 0.5760 to 0.5883 (3 views). Adding Reconcile further increases IoU to 0.5476 and 0.6364, respectively. These results demonstrate the effectiveness of Route and Reconcile across different datasets and input-view settings.

Few-view comparison on ARSG-110K.

ViewsMethodCD-S ↓F-S ↑CD-O ↓F-O ↑IoU-B ↑
2MV-SAM3D0.095473.360.198459.290.5020
2+ Route0.0885 (+7.25%)74.85 (+2.02%)0.1817 (+8.39%)60.92 (+2.74%)0.5216 (+3.90%)
2+ Route + Reconcile0.0823 (+13.80%)76.87 (+4.78%)0.1709 (+13.86%)63.95 (+7.86%)0.5476 (+9.09%)
3MV-SAM3D0.086380.380.142468.600.5760
3+ Route0.0841 (+2.58%)81.24 (+1.07%)0.1372 (+3.67%)69.71 (+1.61%)0.5883 (+2.13%)
3+ Route + Reconcile0.0807 (+6.50%)84.21 (+4.76%)0.1327 (+6.82%)75.25 (+9.69%)0.6364 (+10.47%)

Relative improvements over MV-SAM3D are shown in parentheses. Guide is disabled in both settings.

These results demonstrate the complementary contributions of the three stages: Route improves pose hypothesis selection, Reconcile further enhances scene-level layout coherence, and Guide provides additional benefits through geometry-grounded pose generation. Beyond MV-SAM3D, Route and Reconcile are not specific to a particular generative backbone, suggesting broader applicability to methods with compatible representations.

Dear Reviewer RyeY,

As the discussion period is closing soon, we would like to kindly follow up. Our detailed responses are above; in case it helps, we highlight three results particularly relevant to your concerns:

  • Independent effectiveness of Route and Reconcile (W1, Q1): On MIDI-3D-Front, applying Route and Reconcile to MV-SAM3D without Guide improves layout IoU from 0.597 to 0.617 and 0.677, respectively. Additional 2/3-view experiments further demonstrate their effectiveness when Guide is disabled.
  • Robustness under inaccurate reconstruction (W2): Guide fallback increases from 37.70% to 59.78% for the highest-reconstruction-error group, with conditional RMSE fallback rising from 12.98% to 43.08%. Component ablations also show that SPOON retains improvements over MV-SAM3D when individual stages are removed.
  • Computational overhead (W3): Our additional experiments cover 4–12 views and 2–6 objects. Guide and Route introduce only 3.8%–6.1% additional runtime, with no increase in peak GPU memory, while Reconcile accounts for the primary additional runtime.

We hope these results and clarifications address your concerns and would appreciate your feedback if any questions remain.

Reviewer_1LoLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the layout inconsistency problem in multi-view compositional 3D scene generation: existing methods generate high-quality object assets through multi-view fusion, but the object poses estimated independently from each view conflict with one another in a shared world frame, which the authors call the object–camera soup. The paper proposes SPOON, a three-stage framework: Guide, which steers pose generation using reconstructed scene geometry through a surface-based coordinate bridge; Route, which selects the pose hypothesis from the most geometrically reliable view; and Reconcile, which jointly optimizes object Sim(3) transformations and camera poses via differentiable rendering.

Critical strengths
  1. Precise problem definition. The object–camera soup phenomenon is real but has not been explicitly named or studied in isolation before. The root cause is accurately identified: multi-view fusion operates only on the shape branch, while the pose branch still evolves independently per view. The experimental evidence directly supports this motivation: MV-SAM3D degrades with more views on MIDI, whereas the proposed method remains stable.

  2. Technically sound design of the Guide stage. Rather than aligning the unstable decoded intermediate pose states, the authors use the canonical object surface as a cross-frame bridge. By anchoring cross-frame alignment to a stable geometric quantity instead of noisy intermediate states, this design avoids injecting alignment errors into the sampling trajectory.

Critical weaknesses

W1. Essentially system integration with limited mechanistic novelty. Guide is point-cloud alignment plus flow guidance; Route is a heuristic product of two simple ratios; and Reconcile is, by the authors' own admission, "conceptually analogous to Bundle Adjustment." The contribution lies in identifying the right problem and organizing these pieces into a coherent pipeline, but there is no new technical tool at the mechanism level. Route in particular is over-named: Table 2 shows the scoring formula barely outperforms single cues (IoU 0.617 vs. 0.598/0.614), and it is mainly the truncation threshold filtering that does the work, making the name "geometry-aware pose routing" an overstatement.

W2. Unknown robustness boundaries due to upstream dependencies. The Guide stage assumes the reconstructed point cloud is accurate enough to align with the canonical surface. Although the appendix provides a fallback, the trigger frequency is never quantified, and there is no failure analysis under poor reconstruction quality. The reliance on ground-truth instance masks is also understated in the main text. These factors determine the practical deployment boundary of the method, yet no stress tests are provided.

W3. Limited strength of the core evidence. Comparisons with single-view baselines in the same table carry little weight; the only meaningful comparison is against MV-SAM3D. Moreover, the IoU gain grows with the number of views, coinciding with MV-SAM3D's own increasing degradation, making it hard to distinguish whether the gains come from the method's inherent advantage or from repairing the baseline's specific deficiencies. In addition, CAST and SceneGen, both single-image methods with layout as a core contribution, are not compared against, with no justification given.

Questions for authors

Please add comparisons with CAST and SceneGen under the single-view setting, or clearly justify why they are not comparable (e.g., input requirements, camera assumptions, evaluation protocols).

The bridge acceptance condition (Appendix A.2) requires at least 4 valid views, so the Guide stage is disabled by design below 4 views. Please: (a) report results with 2–3 input views to isolate the remaining contributions of Route and Reconcile, and state the minimum number of views at which the full pipeline produces gains; (b) report the actual activation rate of the Guide stage at each view setting (4/6/8), and the fallback trigger frequency and failure cases under poor reconstruction quality.

Please provide the per-scene metric distribution to distinguish whether the gains mainly come from fixing a small number of catastrophic layout failures or from across-the-board improvements.

On MIDI-3D-Front, IoU completely saturates from 4 to 8 views, while ARSG-110K continues to improve. Please analyze the reason for this discrepancy between the two datasets and describe how input views are sampled in each setting.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 1LoL

W1(a). Technical Contributions beyond System Integration.

SPOON addresses scene-level layout coherence by coordinating object pose hypotheses across generation, selection, and refinement. The key contribution of Guide is not point-cloud alignment alone, but establishing a stable reconstruction-to-canonical coordinate bridge through object surfaces and incorporating geometry-derived pose endpoints into the flow trajectory through time-consistent guidance, rather than relying on unstable intermediate pose estimates. Route evaluates the geometric observability of view-specific pose hypotheses using aggregated multi-view geometry, combining frame completeness and visible coverage for reliability-aware selection. Reconcile extends the principle of joint optimization to compositional scene generation, jointly refining instance-level Sim(3) transformations and camera poses through differentiable multi-view rendering, without requiring explicit cross-view point or feature correspondences, while preserving the generated canonical object geometry. Together, these designs enable reconstruction-derived geometry to inform pose generation, hypothesis selection, and scene-level refinement while preserving the underlying object generative prior.

W1(b). Effectiveness of Geometry-Aware Pose Routing.

Our routing contribution lies in geometry-aware reliability assessment of view-specific pose hypotheses. Specifically, we project aggregated multi-view object geometry into each input view to evaluate frame completeness and visible geometric coverage, providing a geometry-grounded criterion for reliable pose selection beyond 2D mask size or boundary filtering.

Table 2 demonstrates the effectiveness of this design. Compared with the original reference-view selection, our routing reduces scene-level Chamfer distance by 13.3% (0.083 → 0.072), while improving F-score from 87.84 to 90.08 and layout IoU from 0.597 to 0.617. Combining both geometric cues further reduces Chamfer distance by 4.0% (0.075 → 0.072) over Max Visible Coverage alone, while improving F-score (89.94 → 90.08) and IoU (0.614 → 0.617). These results support the effectiveness of geometry-aware observability assessment for pose hypothesis selection.

W2(a) & Q2(b). Guide Activation, Fallback, and Reconstruction Quality.

Guide Activation and Fallback Analysis. We quantify the object-level Guide activation and fallback frequencies across 4/6/8 input views on both ARSG-110K and MIDI-3D-Front.

Guide activation and fallback statistics across input views.

DatasetViewsActivation (%)Total Fallback (%)N_valid < 4 (%)RMSE Fallback (%)
ARSG457.8642.1428.6013.54
ARSG675.3324.678.3016.38
ARSG885.8114.193.4910.70
MIDI477.7522.252.0120.24
MIDI678.7921.210.0021.21
MIDI883.1916.810.0016.81

Fallback occurs due to either insufficient valid views (<4) or bridge RMSE fallback. All percentages are computed over the evaluated objects.

On ARSG-110K, the activation rate increases from 57.86% to 75.33% and 85.81%, while the fallback rate decreases from 42.14% to 24.67% and 14.19%, respectively. This improvement is primarily driven by fewer cases with insufficient valid views. Among objects with sufficient valid views, the alignment fallback rate further decreases, suggesting more reliable geometric support with additional observations. On MIDI-3D-Front, the activation rate similarly increases from 77.75% to 83.19%, with fallback primarily attributed to alignment RMSE fallback rather than insufficient views. These results characterize the practical activation conditions of Guide across different view settings.

AuthorsOpenReview ↗

Response to Reviewer 1LoL

W2(b). Failure Analysis under Poor Reconstruction Quality.

We further analyze Guide reliability on ARSG-110K (4 views) by ranking objects according to their geometric reconstruction error, categorizing the worst 20% as the poor-reconstruction group and the remaining 80% as the normal group.

Guide activation and fallback under varying reconstruction quality on ARSG-110K (4 views).

MetricNormal (N=366)Poor (N=92)
Guide Activation (%)62.3040.22
Total Fallback (%)37.7059.78
N_valid < 4 (%)28.4229.35
Conditional RMSE Fallback (%)12.9843.08

Objects are ranked by geometric reconstruction error, with the highest-error 20% categorized as Poor and the remaining 80% as Normal. This grouping is used only for robustness analysis. Conditional RMSE fallback is computed among objects with at least four valid views.

For the poor-reconstruction group, the overall fallback rate increases from 37.70% to 59.78%, while insufficient-view rates remain comparable. Notably, the conditional RMSE fallback rate rises from 12.98% to 43.08%, showing that the surface-alignment reliability check more frequently rejects guidance for objects with larger reconstruction errors.

We also added representative cases in Fig. 9 of the supplementary material, illustrating both a fallback triggered by excessive surface alignment error (RMSE = 0.1807 > 0.15) and a case where the bridge passes validation despite imperfect reconstruction errors (RMSE = 0.1291 < 0.15). These cases highlight the fallback behavior and potential failure modes of surface-bridge validation under inaccurate reconstruction.

W2cc. Robustness to Imperfect Instance Masks.

Following the instance-mask-conditioned setting of existing scene generation methods (e.g., 3D-Fixer and MV-SAM3D), our main experiments use benchmark-provided ground-truth instance masks to ensure consistent segmentation inputs and isolate the effectiveness of scene-level pose reasoning.

To evaluate robustness to imperfect masks, we conduct additional experiments on ARSG-110K with four input views, replacing GT masks with SAM2-predicted masks. For each object in each visible view, we sample two positive point prompts within its GT mask and independently run the SAM2 image predictor.

Instance mask quality on ARSG-110K.

Mask SourceIoU (%) ↑Dice (%) ↑Precision (%) ↑Recall (%) ↑
SAM2 (2-point prompts)75.8882.5196.8377.77

SAM2 is prompted with two positive points sampled from each visible object's GT mask. Metrics are computed against GT masks.

Performance with SAM2-predicted instance masks on ARSG-110K (4 views).

MethodCD-S ↓F-S (%) ↑CD-O ↓F-O (%) ↑IoU-B (%) ↑
MV-SAM3D0.0960574.710.1853561.4048.62
SPOON0.0836881.680.1554270.1857.81

Both methods use the same SAM2-predicted instance masks.

The SAM2-predicted masks achieve 75.88% IoU and 82.51% Dice against GT masks, with 96.83% precision and 77.77% recall. Despite these segmentation inaccuracies, SPOON consistently outperforms MV-SAM3D across all reconstruction metrics under the same SAM2 masks. In particular, SPOON improves bounding-box IoU by 9.19 percentage points and reduces scene- and object-level Chamfer distances by 12.88% and 16.15%, respectively.

These results demonstrate that SPOON's improvements are retained under imperfect instance segmentation. We note that the current experiment uses GT-derived point prompts and therefore does not establish fully annotation-free deployment. The relatively limited mask recall also suggests potential room for improvement through more effective prompting or mask refinement.

AuthorsOpenReview ↗

Response to Reviewer 1LoL

W3 & Q1. Additional Comparisons with SceneGen and CAST.

We evaluate SceneGen on ARSG-110K and MIDI-3D-Front using its official pretrained weights and inference pipeline, following the same evaluation protocol as our main experiments.

Comparison with SceneGen on ARSG-110K.

MethodViewsCD-S ↓F-S ↑CD-O ↓F-O ↑IoU-B ↑
SAM3D10.14665.320.25748.480.445
SceneGen10.41615.701.0886.380.073
MV-SAM3D40.06382.370.11369.480.585
SPOON (Ours)40.05587.080.09378.020.662

Comparison with SceneGen on MIDI-3D-Front.

MethodViewsCD-S ↓F-S ↑CD-O ↓F-O ↑IoU-B ↑
SAM3D10.08477.060.05392.620.430
SceneGen10.10562.360.08385.000.249
MV-SAM3D40.08387.840.02096.760.597
SPOON (Ours)40.06991.730.01697.660.684

SceneGen achieves layout IoUs of 0.073 and 0.249 on ARSG-110K and MIDI-3D-Front, respectively. Although SceneGen jointly predicts object geometry and relative poses through a scene-aware position head, these results highlight the challenges of accurate spatial placement from single-view observations. Its authors also acknowledge limitations in modeling object contact relationships, which may lead to overlaps and geometric inconsistencies (Appendix E.1).

For CAST, the official implementation and pretrained models are not publicly available, preventing a faithful quantitative reproduction. CAST addresses single-view scene composition through geometric alignment and physics-aware refinement, whereas SPOON focuses on coordinating object poses using shared multi-view geometry.

W3 & Q4. View-Scaling Trends and Geometric Coverage. To clarify whether our improvements primarily arise from correcting MV-SAM3D's degradation, we examine the performance trends of both methods across input views. On MIDI, the increasing gap partly reflects MV-SAM3D's degradation, while SPOON maintains stable IoU. Importantly, on ARSG, MV-SAM3D itself improves from 0.585 to 0.615, yet SPOON achieves larger improvements from 0.662 to 0.723, widening the absolute IoU gain from +0.077 to +0.108. This demonstrates that our gains are not solely attributable to baseline degradation.

Furthermore, we analyze geometric coverage under the respective view sampling protocols. ARSG uses the first 4/6/8 frames in sequence, whereas MIDI employs camera-center farthest point sampling (FPS) from 11 candidate views. With four views, MIDI already achieves 93.63% geometric coverage, compared with 70.18% on ARSG. Increasing to eight views adds only 5.16 percentage points of coverage on MIDI, versus 17.72 points on ARSG. These observations suggest that higher initial coverage and diminishing additional geometric evidence may partly explain the IoU saturation on MIDI, whereas additional views provide greater coverage gains on ARSG.

View-scaling performance and geometric coverage on ARSG-110K and MIDI-3D-Front.

DatasetViewsMV-SAM3D IoU ↑SPOON IoU ↑Δ IoU ↑Coverage (%)
ARSG40.5850.662+0.07770.18
ARSG60.6060.695+0.08980.79
ARSG80.6150.723+0.10887.90
MIDI40.5970.684+0.08793.63
MIDI60.5680.684+0.11695.92
MIDI80.5720.684+0.11298.79

Δ IoU denotes the absolute IoU improvement of SPOON over MV-SAM3D.

Reviewer_nf1TOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The goal of this paper is to generate a coherent compositional 3D scene from uncalibrated multi-view images. A scene-level, geometry-grounded pose reasoning framework, i.e., SPOON, is proposed via a Guide-Route-Reconcile paradigm for multi-view compositional 3D generation. By doing so, camera geometry and scene structure can be jointly recovered in a shared coordinate system. The experiments demonstrate the effectiveness of the proposed framework in object placement and scene composition across views.

Critical strengths

The proposed Guide-Route-Reconcile paradigm for compositional 3D scene generation is reasonable for addressing the object-camera soup problem. The evaluations cover two benchmarks across varying views, showing strong performance with increasing views.

Critical weaknesses
  1. While the proposed paradigm provides a reasonable workflow, the individual modules are largely adopted from prior works (Wang et al. 2026a, Li et al. 2026a, Matsuki et al. 2024). The performance gains brought by these models should be disentangled from the proposed technical contributions.
  2. As the proposed framework contains several stages, i.e., Guide-Route-Reconcile, errors introduced in earlier stages can affect downstream optimization. A detailed evaluation of intermediate stage outputs and error propagation analysis is necessary.
  3. Although the authors claim that the proposed framework scales robustly with more views, it seems that the performance of the proposed method decreases when increasing the number of views on MIDI-3D-Front.
  4. Compared with MV-SAM 3D, the running time of the proposed method is much higher. It would be better to add a discussion of how runtime and memory scale to scenes with more objects or views.
Questions for authors
  1. How about the computational cost and running time for larger object counts or more than 8 views?
  2. How about the performance of the proposed method on real-world multi-object scenes like MV-SAM3D (Li et al. 2026a)?
5 comments
AuthorsOpenReview ↗

Response to Reviewer nf1T

W1. Technical Contributions and Performance Attribution.

Our methodological contribution concerns how reconstruction-derived geometry guides pose generation and coordinates object–camera hypotheses within a shared scene frame. Notably, we evaluate SPOON and MV-SAM3D using the same SAM3D backbone and VGGT-Ω reconstruction front end, with MV-SAM3D's official layout injection and post-generation pose refinement enabled (Appendix A.3). Thus, the improvements cannot be attributed simply to substituting stronger reconstruction or generation models.

While VGGT-Ω provides shared scene geometry and MV-SAM3D generates canonical object assets with view-conditioned pose hypotheses, reliably organizing these assets into a coherent scene-level spatial arrangement remains challenging. SPOON addresses this challenge through a surface-based coordinate bridge that enables flow-based geometric guidance during pose generation, followed by geometry-aware routing and joint object–camera reconciliation.

The controlled ablations further support these contributions. Table 4 shows that SPOON achieves 0.684 IoU compared with 0.597 for MV-SAM3D, while removing Guide, Route, or Reconcile reduces IoU to 0.677, 0.669, and 0.617, respectively. Table 2 further shows that Route alone improves IoU from 0.597 to 0.617 without Guide or Reconcile. Separately, Table 3 shows that joint object–camera refinement improves IoU from 0.651 to 0.662 over object-only refinement. These results support the effectiveness of the proposed pose coordination mechanisms under shared foundational components.

W2. Scene-level refinement of initial layout proposals.

Our framework consists of two complementary steps: Guide and Route establish object-level layout proposals through geometry-grounded pose generation and hypothesis selection, while Reconcile jointly refines object transformations and camera poses to enforce scene-level consistency. To investigate whether layout errors from the preceding stages persist or can be corrected during downstream optimization, we rank scenes by their pre-reconciliation layout IoU and analyze the bottom 20% on each benchmark, comparing the same scenes before and after Reconcile.

DatasetIoU Before ↑IoU After ↑Mean Gain (pp)Median Gain (pp)
ARSG-110K0.42400.5175+9.35+11.36
MIDI-3D-Front0.35360.4788+12.52+12.84

As shown, Reconcile improves scene-level layout IoU by 9.35 pp on ARSG-110K and 12.52 pp on MIDI-3D-Front, with median paired gains of 11.36 pp and 12.84 pp, respectively. These results demonstrate that scene-level reconciliation can substantially improve lower-IoU initial layouts, thereby mitigating residual layout errors from the preceding stages.

W3. View-Scaling Performance on MIDI-3D-Front.

We clarify that SPOON exhibits scene-level performance saturation rather than degradation as the number of input views increases on MIDI-3D-Front. Specifically, SPOON maintains a layout IoU of 0.684 across 4/6/8 views, while scene-level CD remains nearly unchanged (0.069/0.070/0.069). In comparison, MV-SAM3D's IoU decreases from 0.597 to 0.568/0.572, indicating that SPOON maintains more stable scene-level layout accuracy across the evaluated view settings.

The observed saturation may be partly explained by the limited additional geometric coverage. MIDI-3D-Front employs camera-center farthest point sampling from 11 candidate views, with four views already achieving 93.63% geometric coverage. Increasing to eight views adds only 5.16%, suggesting that additional observations provide limited complementary geometric evidence.

View-scaling performance and geometric coverage on MIDI-3D-Front.

ViewsCoverage (%)MV-SAM3D IoU ↑SPOON IoU ↑Δ IoU ↑
493.630.5970.684+0.087
695.920.5680.684+0.116
898.790.5720.684+0.112

Δ IoU denotes the absolute IoU improvement of SPOON over MV-SAM3D.

AuthorsOpenReview ↗

Response to Reviewer nf1T

W4 & Q1. Runtime and Memory Scalability. We conduct scalability experiments with 2–6 objects and 4–12 input views. The table below reports the stage-wise runtime and peak GPU memory.

Runtime and memory scalability across input views and object counts.

ViewsObjectsBaseline (s)Guide (s)Route (s)G+R OverheadReconcile (s)Peak Mem. (GB)
4271.893.500.515.58%65.3621.88
44136.216.381.025.43%88.8630.98
46204.9010.781.656.07%127.9332.98
82144.485.440.984.44%134.3921.98
84309.739.741.993.79%189.9730.75
86393.3815.613.244.79%240.7432.83
122199.207.041.414.24%195.5822.20
124427.3913.263.013.81%270.8529.94
126637.7122.336.404.51%344.3635.28

G+R Overhead denotes the additional runtime of Guide and Route relative to MV-SAM3D. Peak GPU memory is identical for MV-SAM3D and SPOON under the evaluated settings.

Runtime scalability. Guide and Route incur only 3.8%–6.1% additional runtime relative to MV-SAM3D across all configurations. Even with 12 views and 6 objects, they require only 22.33 s and 6.40 s, respectively, compared with 637.71 s for the underlying generation process. The remaining overhead primarily arises from Reconcile, which performs iterative joint refinement of object layouts and camera poses.

Memory efficiency. Guide, Route, and Reconcile execute sequentially, and their measured memory requirements remain below the peak of the underlying generation process. Consequently, SPOON introduces no additional peak GPU memory overhead, requiring 35.28 GB even with 12 views and 6 objects.

Together with the improvements in object placement and scene-level coherence demonstrated in our main experiments, these results show that SPOON introduces modest Guide and Route overhead while maintaining the baseline's peak GPU memory footprint across the evaluated settings.

Q2. Qualitative Evaluation on Real-World Multi-Object Scenes.

To assess SPOON on real-world multi-object scenes, we provide additional qualitative comparisons with MV-SAM3D on ScanNet++ in Fig. 10 of the supplementary material. Given eight uncalibrated input views, SPOON produces more coherent object placements and spatial arrangements than MV-SAM3D in the presented examples. These results provide further qualitative evidence of SPOON's applicability to real-world scenes.

Dear Reviewer nf1T,

As the discussion period is closing soon, we would like to kindly follow up. Our detailed responses are above; in case it helps, we highlight three results particularly relevant to your concerns:

  • Shared backbones and controlled ablations (W1): Under the same SAM3D and VGGT-Ω models and official MV-SAM3D baseline settings, our controlled ablations isolate the effects of Guide, Route, and Reconcile, with layout IoU improving from 0.597 to 0.684.
  • Improvement of initial layouts (W2): For the bottom 20% of scenes ranked by initial layout IoU, Reconcile improves IoU by 9.35 pp on ARSG-110K and 12.52 pp on MIDI-3D-Front, demonstrating its effectiveness in improving less accurate initial layouts.
  • View-scaling stability (W3): SPOON maintains 0.684 layout IoU across 4/6/8 views on MIDI-3D-Front, indicating scene-level performance saturation rather than degradation.

We also provide additional runtime and memory scalability analyses and real-world ScanNet++ comparisons (W4, Q1–Q2).

We would appreciate knowing whether these results address your concerns and would be happy to clarify anything that remains.

Comments on the paper

1 comment

We sincerely thank all reviewers for their careful reading and constructive feedback. We have provided detailed responses and additional experiments, and hope these new results and clarifications help resolve your concerns. If so, we would greatly appreciate your consideration of updating your rating. We welcome any further questions and would be happy to address them during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.