acceptodds
Under review as a conference paper at ICLR 2027

Trace2Stable: Learning from Physical Search for 3D Scene Refinement

Abstract

Image-based 3D reconstruction often produces visually plausible but physically invalid scenes: objects may fall or interpenetrate when simulated. Physical refinement must resolve these violations without replacing the recovered geometry or unnecessarily changing the observed arrangement. Existing per-scene search repeatedly simulates candidate pose corrections to find a stable, collision-free layout. We propose Trace2Stable, which amortizes physical search by learning reusable repair hypotheses from optimization traces. A graph-conditioned flow proposes multiple pose corrections, and candidate support relations represent ambiguous contacts. Simulation verifies the proposed repairs; measured outcomes guide residual refinement, while candidate ordering, alternative support hypotheses, and adaptive rollout continuation direct the remaining verification budget. Prior searches thus guide which repairs to test, while physical tests choose a valid arrangement or preserve an already valid input. We compare against the cross-entropy method (CEM), which searches for pose corrections from scratch in each scene. On 32 controlled layouts with matched simulation objects, Trace2Stable increases the proportion of stable outputs without illegal collisions from 21.88 to 59.38 percent while using 46.95 percent fewer candidate-environment simulation steps than this per-scene search baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Scene Refinement, Amortized Physical Inference, Budgeted Simulation Verification

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_HroFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper introduces Trace2Stable, a framework designed to tackle the physical invalidity (e.g., collisions, interpenetrations, and gravitational collapse) commonly found in image-based 3D scene reconstructions. Rather than running sample-inefficient per-scene searches from scratch (such as the Cross-Entropy Method used in REST3D), the authors propose amortizing physical search by learning reusable repair hypotheses from optimization trajectories.

The framework employs a graph-conditioned flow to propose multi-modal pose corrections under ambiguous support relationships, followed by budgeted physical simulation verification and residual refinement. On a benchmark of 32 controlled layouts, Trace2Stable improves the proportion of stable, collision-free configurations from 21.88% to 59.38% while reducing candidate-environment simulation steps by 46.95%. Transfer capabilities are further probed on datasets including ReplicaCAD, HSSD, and ScanNet++.

Critical strengths
  1. Conceptual novelty of amortizing physical search: Formulating 3D physical scene refinement as an amortized learning problem over historical optimization trajectories is an insightful and principled direction. It effectively addresses the cold-start inefficiency of standard black-box heuristic search methods like CEM.
  2. Synergistic framework design: Coupling generative multi-modal hypothesis generation (via graph-conditioned flow) with budgeted simulator-in-the-loop verification strikes an appealing balance between leveraging learned physical priors and guaranteeing physical validity through simulation.
Critical weaknesses
  1. The core quantitative claims in Section 4 heavily hinge on an evaluation over only 32 controlled layouts. In absolute terms, improving the collision-free stability rate from 21.88% to 59.38% corresponds to successfully resolving merely 12 additional scenes (an increase from 7/32 to 19/32). Indoor scene arrangements present an immense combinatorial configuration space characterized by multi-object contact topologies, mass distributions, and geometric friction boundaries. Evaluating amortized learning—whose foundational premise is the cross-scene transferability of optimization traces—on a test set of just 32 layouts is statistically fragile. Such a small sample size cannot rule out sampling variance, object-category bias, or overfitting to specific canonical arrangements, leaving the generalization capability of the graph-conditioned flow unsubstantiated.

  2. A foundational premise established in the Introduction (Section 1) is that physical refinement must eliminate collisions and instability "without replacing the recovered geometry or unnecessarily changing the observed arrangement." However, the paper creates a significant evaluation disconnect by focusing almost exclusively on simulation stability and candidate-environment step counts, while omitting rigorous metrics quantifying visual fidelity and pose drift. In a rigid-body physics simulator, it is trivially easy to achieve "stability" and "zero collision" by pushing objects far apart, shifting them out of the camera's line of sight, or snapping them directly onto the floor plane. Without measuring translation drift (Delta T in cm), rotation drift (Delta R in degrees), 2D mask/depth reprojection errors, or Chamfer Distance relative to the initial reconstruction, there is no empirical guarantee that physical stability was not achieved at the cost of catastrophic layout distortion.

  3. The paper touts a 46.95% reduction in "candidate-environment simulation steps" compared to CEM. While reducing simulator rollout queries is desirable, it paints an incomplete picture of computational complexity. Trace2Stable introduces considerable auxiliary overhead: graph neural network message passing, multiple ODE integration steps for the conditional flow to generate multi-modal candidates, support-tree hypothesis branching, and candidate re-ranking. The Cross-Entropy Method, while sample-inefficient in query counts, involves negligible per-sample neural latency. Without reporting full end-to-end wall-clock time (GPU forward sampling + CPU/GPU simulation verification + residual optimization), it remains unverified whether this method actually accelerates overall scene processing or merely transfers the computational bottleneck from the physics engine to the neural network.

  4. Despite combining learned flow priors, support branching, candidate bank selection, and residual physics refinement, the terminal success rate plateaus at only 59.38%. This indicates that more than 40% of scenes in the controlled synthetic test suite still suffer from interpenetration or gravitational collapse, reflecting a high failure rate for an iterative refinement pipeline with simulator-in-the-loop residual search. Furthermore, when transferring to real-world/scanned datasets such as ScanNet++ and HSSD, the paper noticeably downgrades its evaluation to "support selection and layout preservation" rather than executing full rigid-body dynamic rollouts to measure terminal stability and penetration. Real-world meshes contain scanning artifacts, partial occlusions, and non-convex concavities where physics engines notoriously fail; bypassing end-to-end simulation rollouts on ScanNet++ sidesteps the core challenge of real-to-sim transfer.

Questions for authors
  1. Can the authors provide experimental results across a substantially larger benchmark (e.g., hundreds of scenes from ProcTHOR, Habitat-Matterport3D, or the complete ReplicaCAD suite)? Specifically, what are the mean success rates, standard deviations across multiple random seeds, and confidence intervals to demonstrate that the amortized flow prior generalizes beyond the 32 controlled layouts?

  2. To verify that the refinement does not achieve physical stability by drastically altering the scene geometry or violating the observed layout: (a) What are the mean/median translation drifts (in cm) and rotation drifts (in degrees) of objects after refinement relative to the unrefined initial reconstruction? (b) Could the authors report 2D visual reprojection metrics (e.g., multi-view rendering PSNR/SSIM, segmentation mask IoU, or depth L1 error) comparing the refined 3D layout against the input single-view image?

  3. Could the authors provide a comprehensive wall-clock time breakdown (in seconds per scene) comparing Trace2Stable and the baseline CEM? Specifically, please dissect:

  • Neural proposal generation time (GNN feature extraction + Flow sampling of K candidates).
  • Physical verification time (simulator rollout steps).
  • Residual refinement optimization time. Is Trace2Stable strictly faster in wall-clock execution than CEM on identical hardware?
  1. Given that the terminal success rate is 59.38% on the controlled benchmark, what are the primary root causes for the remaining 40.62% failure cases? Does the graph-conditioned flow generate out-of-distribution initial proposals that trap the local residual search in poor local minima, or does the budgeted rollout prematurely terminate before convergence? A detailed quantitative failure mode breakdown (e.g., percentage failed due to unresolvable contact cycles, severe mesh penetration, or unstable stacking) would be highly illuminating.

  2. Why was terminal dynamic simulation omitted in the ScanNet++ transfer experiments in favor of surrogate metrics like support selection? What is the actual collision-free stability rate when Trace2Stable is deployed on reconstructed ScanNet++ meshes inside Isaac Gym / PyBullet, and how does the method handle non-watertight geometry and noisy reconstructed surfaces?

1 comment

Hi Reviewer HroF,

We sincerely apologize for the additional reminder, and we greatly appreciate your time and consideration throughout the discussion. As the discussion period is approaching its end, with only 6 hours remaining, we would like to kindly follow up on our responses and revisions.

In particular, regarding your concerns about the generalizability and statistical reliability of our results, the preservation of observed visual layouts during physical refinement, the actual end-to-end runtime, and the validity of dynamic simulation on raw reconstructed meshes, we have conducted additional evaluations and substantially expanded our analyses. Specifically, we have added independent source-level evaluations, further examined input-layout preservation and relative object arrangements, provided more detailed runtime measurements and failure analyses, and supplemented the physical validation on scan-derived meshes, including collision representations and physical-property consistency.

We have also revised the manuscript and supplementary materials to make these results and evaluation protocols clearer. We sincerely hope that these additional experiments, analyses, and clarifications have adequately addressed the concerns you raised. If you feel that our revisions have resolved these issues, we would be very grateful if you could reconsider the paper's rating.

Thank you again for your thoughtful feedback, constructive suggestions, and careful evaluation. We truly appreciate the time and effort you have devoted to helping us improve this work.

Best regards,
Authors of Submission 1727

Reviewer_pPZ7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies physical refinement of single-image 3D scene reconstructions, which often contain falling, interpenetrating or unsupported objects. Instead of running per-scene CEM search from scratch as in REST3D, Trace2Stable learns from CEM optimization traces. It has three parts: a set-Transformer support posterior that keeps multiple candidate parent relations; a graph-conditioned flow-matching model that proposes multiple pose corrections, trained on the clean or low-risk candidates found by CEM; a budgeted verification stage that uses learned candidate ordering, prefix and second-bank opening, and recovery-based rollout continuation, with simulation making the final acceptance decision.

Critical strengths

1 Reusing search traces is a sensible idea. Training a multi-hypothesis generator on the set of acceptable CEM candidates, rather than only the final solution, is a reasonable way to amortize physical search while keeping simulation as the verifier. The ReplicaCAD comparison (Table 2, Table 5) supports this: the proposal-only flow already beats deterministic GNN, VAE and diffusion proposals, and has better candidate coverage. 2 Physically grounded acceptance. Verifying every returned layout by simulation, and keeping the original placement as a candidate, means the method can repair invalid scenes while leaving valid ones unchanged (Fig. 4, Fig. 9). This is a practical property for reconstruction pipelines.

Critical weaknesses

1 The evaluation is too small to support most claims. The main quantitative results rest on 21, 32 and 48 scenes, and the real-image evaluation has only 9 reconstructions. 2 The method saves simulation steps, but wall time is the same as CEM (63.66 s vs. 63.68 s, Table 3), and the offline cost of generating traces is not counted. 3. Many of the added verification components give little or no measurable gain: learned ordering vs. generation ordering (0.00 difference, Sec. G.3)

Questions for authors

1 Can you report the main comparisons (Table 3, Table 4) on a substantially larger set of layouts, e.g. several hundred procedural layouts? 2 What is a single, fixed Trace2Stable configuration, and how does it perform across all benchmarks compared with “flow + residual CEM” alone? 3 Under what conditions do the step savings translate into wall-time savings, and what is the total offline cost of generating traces and training?

2 comments

Hi Reviewer pPZ7,

We sincerely apologize for the additional reminder, and we greatly appreciate your time and consideration throughout the discussion. As the discussion period is approaching its end, with only 6 hours remaining, we would like to kindly follow up on our responses and revisions.

In particular, regarding your concerns about the scalability and fairness of our evaluation, the actual computational efficiency, and the effectiveness of the physical verification components, we have conducted additional large-scale matched-budget evaluations, provided independent fixed-budget validation, and further clarified the complete runtime and verification comparisons. We have also revised the presentation of our experimental configurations to make the comparisons and contributions of each component clearer.

We sincerely hope that these additional experiments and clarifications have adequately addressed your concerns. If you feel that our responses have resolved the issues you raised, we would be very grateful if you could reconsider the paper's rating.

Thank you again for your constructive feedback, careful evaluation, and valuable time. We truly appreciate your help in improving our work.

Best regards,
Authors of Submission 1727

Reviewer_pPZ7OpenReview ↗

raise score

Dear author,

I have raised my score to 3. The rebuttal addressed most of my concerns.

Reviewer_yzJXOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Trace2Stable studies physical refinement of reconstructed indoor scenes with fixed object geometry. It learns a graph-conditioned distribution of joint pose corrections from CEM optimization traces, retains alternative support hypotheses, and uses physical simulation to evaluate candidate placements. Additional components prioritize candidates, allocate proposal-bank prefixes, open alternative banks, and continue unresolved trajectories. The furniture configuration combines flow proposals with one residual-CEM update. Evaluation includes source-disjoint ReplicaCAD groups, controlled layouts, released reconstructions, and component-level transfer studies on ScanNet++, HSSD, ProcTHOR, and iTHOR.

Critical strengths
  1. Practical relevance. The paper addresses the gap between visually plausible reconstruction and physically valid scenes. Improving stability and contact consistency is valuable for using reconstructed environments in simulation.
  2. A well-motivated use of search experience. Learning alternative repairs from previous searches and verifying them through simulation provides a coherent refinement strategy. The controlled experiments demonstrate its potential to reduce repeated physical search and improve the returned arrangements.
Critical weaknesses
  1. Limited evidence of generalization across reconstruction models and datasets. Reusing physical-search experience is the central motivation, but the main reconstruction evaluation uses REST3D outputs and does not show whether the complete method remains effective with another image-to-3D model. The cross-dataset experiments also do not clearly distinguish direct reuse of the complete model from recalibration or retraining. Appendix H includes dedicated ScanNet++ value-model training and sequential proposal-model fits, while the unchanged-model iTHOR experiment in Table 6 evaluates only support ranking. Thus, reuse of the complete method across reconstruction models and indoor-scene datasets remains unclear. The current experiments do not sufficiently establish the robustness of a fixed complete model to unseen data distributions and OOD inputs. The adaptation requirements should be specified, alongside cross-dataset evaluation of the same complete model.
  2. The incremental benefits of the additional verification and support-branching mechanisms are insufficiently established. The paper presents budgeted verification and support reasoning as major contributions. Existing ablations isolate some of these components, but the results do not convincingly establish consistent, substantial gains for the complete refinement procedure over simpler ordering and support strategies. On identical candidate banks, learned and generation ordering both return 18/32 valid layouts. Mean queries decrease from 3.25 to 2.47, but total steps including Local work change only from 4070.00 to 4066.09. Exhaustive selection returns 19/32 valid layouts at 4093.75 steps. Thus, the query reduction yields little reduction in total simulation work. Support branching also has uncertain gains. Table 4 reports an improvement of 12.50 percentage points, with a 95% interval of [−8.33, 33.33]. Further replications yield a pooled gain of 11.25 points, with an interval of [−6.25, 28.75]; the last replication reverses sign. The independent contribution of structural branching is therefore not yet conclusive.
Questions for authors
  1. Can the same complete model transfer to a different reconstruction system or indoor-scene dataset? Please distinguish direct reuse, recalibration, and retraining in each transfer experiment.
  2. Given the small total-step saving from learned ordering and the mixed support-branching results, under what task conditions do these mechanisms provide consistent independent gains? What explains the differences across replications?
  3. For the nine released reconstructions, what is the scene-level physical success rate under the stated replay protocol? What causes the remaining unstable objects and the five initially stable objects that regress?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.