acceptodds
Under review as a conference paper at ICLR 2027

PhysRecon: Simulation-in-the-Loop Scene Reconstruction for Physically Credible Embodied Data Generation from Single Images

Abstract

Single-view 3D reconstruction and 6D pose estimation often produces visually plausible yet physically infeasible scenes, such as floating objects, unstable stacking and unreasonable object scales, while suffering from incomplete geometry caused by visual occlusion. To address these issues, we propose PhysRecon, a physics-guided closed-loop framework for physically credible single-image 3D scene reconstruction. Different from conventional purely visual reconstruction methods, our pipeline integrates occlusion-aware visual completion and physics-based iterative refinement. Specifically, we adopt an occlusion-aware inpainting module to recover complete object geometry under occluded observations. Instead of directly predicting object mass, our framework estimates physical density and calculates mass via reconstructed mesh volume, enabling physically consistent parameter modeling. We further design a failure-driven local sampling strategy combined with quasi-static physical simulation and physical violation auditing, which iteratively corrects unreasonable object scales and poses while freezing well-reconstructed objects. Evaluated on HomebrewedDB and LM-O benchmarks, our method outperforms state-of-the-art visual reconstruction baselines in both geometric reconstruction accuracy and 6D pose estimation. Extensive quantitative ablation and qualitative visualization further validate the effectiveness of each core component, demonstrating that PhysRecon can generate visually faithful and physically reasonable 3D scenes from a single RGB image.

Primary area:
applications to robotics, autonomy, planning
Keywords:
3D Scene Reconstruction, Physical Parameter Estimation, Vision-Language Prior, Agent-Based Simulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FgJUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to solve single-image scene reconstruction whose outputs are usable in physics simulators with no floating, interpenetration or wrong scale. The authors propose a training-free pipeline via "perception-then-physics-validation" rather than physics losses. Specifically, a frozen single-view reconstruction backbone, occlusion-aware inpainting, VLM-predicted physical parameters, then Isaac Sim used as a black-box auditor that resamples per-object scale/pose until the scene is statically stable and still matches the input masks.

Critical strengths

The paper has strengths in the following perspectives:

  1. A reasonable and simple decomposition. The proposed pipeline involves frozen perception + black-box simulator audit + failure-driven local resampling with a hard visual-consistency gate, which avoids differentiable-physics training and keeps outputs tied to the image.
  2. Protocol choices are good in principle. The authors design identical per-(frame, object) target lists, baselines re-run locally with official weights. RecGen's self-reported HB numbers are close to the paper's re-run, which lends credibility.
  3. Honest statistics on the one ablation that exists (paired sign tests, CIs), conceding inpainting gains are "moderate but not universally consistent" (Sec. 4.4).
  4. Qualitative results (Figs. 2, 3, 6, 7) show visibly cleaner meshes than RecGen/Any6D on textured boxes and curved objects, showcasing the performance of the proposed method.
Critical weaknesses

The majority of the weaknesses is related to scope and evaluation, summarized as follows:

  1. The main claim (physical credibility) is never measured, especially quantitatively. There is no stability rate under simulation, penetration depth, settle displacement or floating-object rate, for the method or the baselines.
  2. Input modality is ambiguous. The comparison may be RGB vs RGB-D. The paper says "single RGB image" throughout but never names the reconstruction backbone. RecGen conditions on depth-derived pointmaps (RGB-D input) and Any6D needs an RGB-D input. Moreover, it is unclear whether instance masks are ground truth or predicted, and whether the "visib_fract" used for occlusion reasoning (Sec. 2) is the GT annotation of that name, which would be test-time data leakage.
  3. The core contribution (physics loop) is never ablated. Table 3 ablates only inpainting. There is no "no physics refinement" row, so Table 1 gains cannot be attributed to the loop vs inpainting vs backbone choice.
  4. Missing baselines. 1) REST3D (cited, closest baseline) is not compared. 2) SAM3D and Any6D+TRELLIS, and 3) the weaker Any6D+InstantMesh is used instead.
Questions for authors

Besides the above weaknesses, I have the following questions for the authors:

  1. Which exact models are the reconstruction backbone, inpainting model, VLM and segmentation source?
  2. Are the instance masks and visib_fract from ground-truth annotations or predicted at test time?
  3. Please add a "no physics loop" ablation and a "VLM priors replaced by constants" ablation on the full target sets, with paired tests.
  4. Where does the method support curved contact surfaces, given Sec. 3.2 and Eq. 8 only model planar supports? Any experiment with a non-planar support?
  5. What are ε_v, ε_ω, τ_iou, τ_pen, T_sim, and what is output when I_max is reached with no violation-free candidate?
Reviewer_GsFnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

PhysRecon addresses single-image 3D scene reconstruction with an emphasis on physical plausibility. The pipeline combines occlusion-aware image inpainting, object-wise mesh reconstruction, VLM-estimated physical priors, and simulation-guided pose and scale refinement. Object mass is computed from estimated density and mesh volume. With mesh geometry fixed, a failure-driven loop samples bounded pose-scale perturbations for objects that fail the audit, filters candidates by penetration and visual consistency, and evaluates them through quasi-static simulation. Experiments on HomebrewedDB and LM-O compare reconstruction and pose accuracy with RecGen and Any6D.

Critical strengths

The paper addresses a practical problem by combining visual reconstruction with simulation-based physical validation.

Critical weaknesses
  1. Poor writing and uninformative figures. The method description leaves essential components unspecified. Sections 3.2 and 4.2 refer to an inpainting module, a 3D reconstruction module, and a VLM without identifying the actual models or adequately explaining how occlusion completion and initial scene reconstruction are implemented. The pipeline diagram in Figure 1 largely lists module names and connections, without clarifying candidate selection or scene updates, providing insufficient information to understand the refinement procedure. The result visualizations are similarly limited: the paper provides no close-up views of contact regions, annotated visualizations of specific physical violations, or motion trajectories during simulation. This makes it difficult to assess whether the claimed physical corrections actually occur.
  2. The novelty relative to closely related work is not convincingly established. The paper characterizes REST3D [1] as using differentiable physical energy terms (p. 3, lines 134–143), whereas REST3D already uses Gaussian pose sampling, forward physics simulation, and cross-entropy method updates. This substantially overlaps with the central optimization strategy of PhysRecon. Updating only failed objects, searching over scale, and applying a visual gate may distinguish the implementation, but the paper provides neither a direct comparison with REST3D nor targeted evidence that these changes yield a meaningful advantage. The claimed distinction between the two frameworks therefore needs substantial revision.

[1] https://arxiv.org/abs/2605.30338

Questions for authors
  1. Which specific models and configurations are used for segmentation, inpainting, 3D reconstruction, and physical-prior estimation? Please also clarify the candidate-selection and scene-update procedure after filtering, including what happens if no feasible candidate is found within the iteration budget.
  2. After correcting the comparison with REST3D, what evidence isolates the benefit of PhysRecon's proposed modifications?
  3. What scene state is actually returned? Page 3, line 141 states that simulator-settled poses are never adopted as outputs, whereas Section 3.3.2, line 215 says to output the steady-state scene, and Section 3.3.3, line 220 centers sampling on the previous steady state. Please reconcile these descriptions and clarify which state is used for the reported evaluations.
Reviewer_bqTFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PhysRecon, a single-image 3D reconstruction pipeline that combines occlusion-aware inpainting with physics-based pose and scale refinement. The method reconstructs object meshes, estimates physical properties using a VLM, and uses Isaac Sim to identify unstable or visually inconsistent object configurations. Failed objects are refined through bounded pose-scale sampling, followed by penetration checks, visual consistency checks, and further simulation.

The method is evaluated on HomebrewedDB and LM-O against RecGen and Any6D, using geometry reconstruction and 6D pose estimation metrics. The authors also provide an inpainting ablation and qualitative examples of physics-based refinement.

Critical strengths

1. The physics refinement loop is straightforward and interpretable.

I like that the method explicitly checks physical violations instead of relying only on visual reconstruction. The failure-driven sampling also avoids modifying objects that already pass the checks. The use of both physical and visual constraints is reasonable, since a physically stable scene can still differ from the input image.

2. The paper evaluates on established reconstruction benchmarks.

The authors evaluate against RecGen and Any6D using the same target lists and evaluation driver. The results show improvements in normalized Chamfer distance and average pose error on both HomebrewedDB and LM-O. The inpainting ablation also includes paired comparisons and statistical testing rather than relying only on qualitative examples.

Critical weaknesses

1. The paper does not quantitatively evaluate its main claim of physical credibility.

The main contribution is the simulation-in-the-loop refinement, but Tables 1 and 2 only report geometry and pose estimation metrics, including Chamfer distance, ADD-SB, and depth recall. These metrics cannot directly establish whether the reconstructed scenes are physically stable or suitable for embodied simulation.

Figure 4 shows a few examples of correcting floating or unstable objects, but there are no quantitative results for physical violation rate, penetration depth, object displacement after simulation, or the percentage of scenes that pass the physics audit.

This is especially important because recent work such as REST3D (Ma et al., 2026), NeoWorld-Pro (He et al., 2026), and Scenethesis (Ling et al., 2026) has also explored physically constrained scene construction or refinement.

I would like to see quantitative stability evaluations before and after refinement, with comparisons against existing physics-based methods. Without these experiments, the central claim is not adequately tested.

2. The contribution of the physics refinement module is not isolated.

The only quantitative component ablation in Table 3 removes occlusion-aware inpainting. There is no corresponding quantitative ablation that removes the physics refinement loop, despite this being the main contribution of the paper.

The improvements over RecGen and Any6D could come from multiple components, including inpainting, changes in object initialization, or the pose-scale refinement procedure. The current experiments do not establish how much improvement is specifically due to simulation feedback.

I would like to see comparisons between the initial reconstruction, reconstruction with geometric pose optimization only, and reconstruction with the complete simulation-in-the-loop refinement. A simple random or local pose search without physics feedback would also be a useful baseline.

3. The physical parameter modeling and scale refinement need clarification.

Section 3.2 states that object mass is computed as m = ρV, where V is the mesh volume precomputed at initialization. However, the refinement loop subsequently modifies object scale.

If scale changes, the physical volume should also change, and mass should be updated accordingly. It is unclear whether this is handled by the implementation. If the initial mass remains fixed while the object is rescaled, the simulation may use inconsistent physical parameters.

The paper also does not evaluate the accuracy of the VLM-predicted density, friction, and stiffness. These quantities affect simulation outcomes, so it would help to understand how sensitive the method is to inaccurate physical property estimates.

4. The evaluation is limited relative to the claimed simulation-ready application.

The experiments focus on static reconstruction and pose estimation, while the title and motivation emphasize physically credible embodied data generation.

There is no downstream robot interaction experiment, trajectory replay, or policy evaluation showing that the reconstructed environments can reproduce physical interactions.

In addition, Section 3.2 describes planar supporting surfaces, while the paper claims support for curved contact surfaces. The implementation and evaluation of this capability are not sufficiently clear.

I would like to see either a downstream simulation experiment or a more focused evaluation of contact behavior, especially for curved, stacked, and mutually supporting objects.

Questions for authors
  1. Can the authors quantitatively evaluate the physical refinement loop? What percentage of scenes pass the physics audit before and after refinement? Could the authors report penetration depth, displacement after settling, and failure rates, and compare against REST3D or other 2026 physics-aware reconstruction methods?

  2. What is the contribution of simulation feedback itself? Table 3 only ablates inpainting. Could the authors compare against the same initialization without physics refinement and against local pose-scale search using only visual or geometric constraints?

  3. Is mass recomputed when object scale changes? Section 3.2 defines mass using the initial mesh volume, but Section 3.3 optimizes scale. How is consistency between scale, volume, mass, and inertia maintained?

  4. How accurate are the predicted physical properties? Have the authors evaluated sensitivity to errors in density, friction, and contact stiffness?

  5. How often does the refinement fail to converge? Could the authors report the average number of iterations, candidate evaluations, runtime, and the fraction of scenes that fail to satisfy all constraints within the maximum iteration budget?

  6. How are curved contact surfaces handled? The method describes planar supporting surface reconstruction. Could the authors clarify how non-planar supports are represented and provide evaluation cases that specifically test curved contacts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.