acceptodds
Under review as a conference paper at ICLR 2027

ScenePlacer: Reformulating Scene-Level Object Pose Estimation as Visibility Field Learning

Abstract

We present ScenePlacer, a framework for scene-level pose estimation of objects generated by state-of-the-art native shape generation models, placing generated 3D objects into scenes given a partial geometric observation without modifying the underlying generative pipeline. Existing scene generation approaches either generate objects directly in scene coordinates, compromising their canonical-space priors, or regress poses from complete generated shapes in the shape generation process — the latter being a poorly conditioned problem that is sensitive to scale ambiguity and partial observations. ScenePlacer decouples object generation from pose estimation by reformulating alignment as conditional visibility-field learning. Given an observed partial point cloud and a generated mesh, it predicts a visibility field defined in object space, then estimates the rigid transformation by aligning the partial observation to visible parts on the mesh. We propose a pretrained 3D shape VAE as our backbone, preserving its mesh prior while conditioning it on a separately encoded observation. We further introduce visibility-aware, radius-constrained supervision and synthesize training pairs using only object-level meshes and rendered views, eliminating the need for scene-level pose annotations. On 3D-FUTURE, ScenePlacer consistently outperforms existing placement and pose-refinement baselines under both ground-truth and monocular depth estimation. With generated objects, it achieves a Chamfer distance of 0.0077 and an F-score of 92.40 under estimated depth and improves to 0.0023 and 98.27, respectively, under ground-truth geometry.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
scene-level object pose estimation, visibility field learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cPsqOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper places generated 3D objects back into a scene. It takes a generated mesh and a partial point cloud of the same object. Instead of regressing the pose, it predicts which part of the mesh is visible in the observation. It then aligns this visible part to the observed points to get the pose. The model builds on a pretrained shape VAE and is trained only on object meshes with rendered depth. On 3D-FUTURE, it beats SAM3D, Any6D and 3DFixer under both ground truth and estimated depth.

Critical strengths

The method leaves the object generator untouched. Shape quality is therefore not sacrificed for pose accuracy, and the module can be paired with any generator or depth model. Predicting the visible part of the mesh is a simple and natural way to frame the problem. It replaces full-to-partial alignment with partial-to-partial alignment, and training needs only object-level data rather than scene-level pose labels. Reusing a pretrained shape VAE is a sensible choice. Freezing the mesh encoder and letting the observation modulate it are both supported by the ablations, and the gains over the baselines are large and consistent.

Critical weaknesses
  1. Predicting visibility from an unaligned point cloud already requires the network to infer the relative pose. The output is a mask rather than point matches, so the final pose still depends on a Chamfer fit that needs a good initialization. In this sense, the hard part is not removed. The idea is also close to overlap prediction in point cloud registration. Furthermore, the paper does not explain how the pose solver is initialized or how scale is recovered.

  2. The training input is always a clean subset of the same mesh, with no shape gap, depth noise or occlusion. The model is thus never trained on the very difficulty that motivates it. In addition, each object is placed independently, so the scene-level claim involves no reasoning about support or collision.

Questions for authors
  1. How is Eq. 4 solved? How do you initialize it and recover scale? How much better is the result if the solver gets the ground truth visible region?

  2. In real scenes the viewing direction is known, and it largely decides what is visible. Why is it removed during training?

  3. How does accuracy change as the generated mesh drifts away from the real shape?

Reviewer_q2RTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces ScenePlacer, a method for scene-level object spatial arrangement estimation from a single image. The object shapes are first estimated via off-the-shelf method (SAM3D). Then, the scene placement is recast as visibility field learning. To this end, for each of the scene objects, every point on the reconstructed mesh surface is mapped to a probability for it being visible in the observed viewpoint. This mapping is parametrized using a neural network, and is built upon the Hunyuan3D-2.1 shape VAE [1] (3DShape2VecSet-style [2]), with an additional encoder. Namely, mesh/reconstruction latents from the frozen encoder cross-attend, in the decoding, to tokens produced by a trainable encoder copy from the observed partial point cloud (unprojected GT or MoGe depth). The pose is then a rigid transformation that minimizes the Chamfer distance between the predicted visible surface and the observed point cloud. The model is trained on the Objaverse [3] dataset, without the need for scene-level pose labels. When evaluated on 1,000 3D-FUTURE [4] test scenes, ScenePlacer outperforms 3D-Fixer [6], SAM3D [7] and Any6D [8], both using GT or MoGe [5] depth.

[1] Hunyuan3D Team, Hunyuan3D 2.1: From Images to High-Fidelity 3D Assets with Production-Ready PBR Material, arXiv 2025.

[2] Zhang et al., 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models, ACM TOG (SIGGRAPH) 2023.

[3] Deitke et al., Objaverse: A Universe of Annotated 3D Objects, CVPR 2023.

[4] Fu et al., 3D-FUTURE: 3D Furniture Shape with TextURE, IJCV 2021.

[5] Wang et al., MoGe: Unlocking Acfcurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision, CVPR 2025.

[6] Yin et al., 3D-Fixer: Coarse-to-Fine In-place Completion for 3D Scenes from a Single Image, CVPR 2026.

[7] Chen et al., SAM 3D: 3Dfy Anything in Images, CVPR 2026.

[8] Lee et al., Any6D: Model-free 6D Pose Estimation of Novel Objects, CVPR 2025.

Critical strengths

S1. Training needs only object-level meshes with rendered views, no scene-level pose labels, and the lead in performance holds under, potentially noisy, MoGe depth even though the method is only trained with GT depth renders.

S2. The main comparison is controlled where it matters. Namely, "Ours (SAM3D obj.)", SAM3D and Any6D all place the same SAM3D meshes, so the CD drop from 0.98 to 0.36 (GT depth) and from 1.31 to 0.77 (MoGe depth) in Tab. 1 can be attributed to the visbility field and placement algorithm.

Critical weaknesses

W1. No pose metric, and a single generator. All experiments use scene-level CD and F-score after GT-bounding-box normalization (L321-L323, L378-L380). This however mixes pose error with the generator's shape error. Additionally, only SAM3D is used for shape estimation, against the claim of straightforward integration with diverse generators (L097-L098). Please report pose-based metrics (per-object rotation, translation and scale error) and results with a second shape generator, e.g. Hunyuan3D-2.1 [1].

W2. The pose solver is not described sufficiently. As described, the network input is normalized to a unit sphere (Eq.11), so the method never sees the scene scale. At inference time, the predicted visible patch is in mesh scale [-1, 1] and the observed partial point cloud (GT or MoGe unprojected depth) is in scene scale, this mismatch cannot be solved with a single rigid T (Eq. 4), without also recovering scale parameter. However, this step is not described.

W3. The closest prior work is cited but never discussed. CAST [9] (L132) similarly places a generated object by mapping the scene-space partial cloud into the object's canonical frame with a point-cloud diffusion model. SceneMaker [10] (L131) similarly decouples pose from generation. Moreover, estimating the visible or corresponding part of a shape before alignment is also established via canonical coordinates [11] or image-to-3D feature space matching [12]. The paper does not say what predicting visibility gains over these formulations. Please discuss the difference explicitly.


[9] Yao et al., CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image, ACM TOG (SIGGRAPH) 2025.

[10] Shi et al., SceneMaker: Open-set 3D Scene Generation with Decoupled De-occlusion and Pose Estimation Model, CVPR 2026.

[11] Wang et al., Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation, CVPR 2019.

[12] Antić et al., SDFit: 3D Object Pose and Shape by Fitting a Morphable SDF to a Single Image, ICCV 2025.

Questions for authors
  1. (W1) How is its depth brought to scene scale, and are the instance masks ground truth or predicted?

  2. (W1, clarification) How are 3D-Fixer failed generations (L403-L404) scored in the evaluation protocol? Namely, the reported CD of 23.35 is roughly 20x worse than any other method, while it's [email protected] of 85.65 is best among baselines. Median results might make more sense, because this shows that a few ``exploded'' results can dominate CD.

  3. (W2) Is the solver rigid or non-rigid, ie. with or without scale param? How are scale and rotation initialized, which optimizer is used, how many iterations? Please explain this further.

  4. (W2) What is the per-object/per-scene runtime of ScenePlacer, Any6D and SAM3D+PO?

  5. (W3) What does predicting visibility gain over predicting canonical coordinates for the observed points, as CAST's AlignGen and NOCS do?

  6. (clarification) Was the 130k Objaverse training subset (L305) checked for instance overlap with 3D-FUTURE, and do the Hunyuan3D-2.1 ShapeVAE or SAM3D training sets include 3D-FUTURE meshes? This matters most for the Ours (GT obj.) column.

Reviewer_jupoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the placement step in compositional single-image scene generation: given a canonical-space mesh from a generator (e.g. SAM 3D) and the object's partial point cloud from depth, recover its pose and scale in the scene. Rather than regressing the pose or registering the full mesh to the partial observation, the authors predict which part of the mesh is visible from the camera. A frozen Hunyuan3D-2.1 ShapeVAE encodes the mesh, a second trainable encoder encodes the observation, and the decoder outputs a per-point visibility probability on the mesh surface. The pose is then solved by Chamfer alignment between the predicted visible part and the observation. Training uses Objaverse renders: depth maps are rendered and rotation augmentation is used to construct paired data. On 3D-FUTURE, with SAM 3D meshes, CD drops from 0.98 (SAM 3D's own layout prediction) to 0.36 with GT depth, and from 1.31 to 0.77 with MoGe depth. The experiments support the effectiveness of the method.

Critical strengths
  1. Learning a decoupled placer is practical. The generator is untouched, training needs no scene-level pose labels, and the module can take different generators and depth sources. Partial-complete pairs built from synthetic data alone are enough to learn this placer.

  2. The gain on experiments is clear and the variables are well controlled. SAM 3D, SAM 3D+PO, Any6D and ScenePlacer place the same meshes, so the gap in Tab. 1 really comes from the placement method (GT depth: CD 0.36 vs. 0.98 / 1.51 / 1.04). Tab. 3 shows that the same solver without the visibility field degrades to CD 1.32. This supports the paper's key claim.

Critical weaknesses
  1. The evaluation doesn't support a "pose estimation" contribution. Everything is measured on one synthetic dataset (3D-FUTURE) with a single scene-level CD/F-score, with no variance reported. There are no rotation, translation or scale errors, so I can't tell what is actually being fixed. The mean CD also looks outlier-driven: 3D-Fixer has CD 23.35 but the best baseline [email protected]. More importantly, there is no registration baseline (Predator, GeoTransformer, FPFH+RANSAC) and no model-based pose estimator (FoundationPose / SAM-6D), even though these take exactly the same inputs. Predator in particular also predicts overlap before registering. Without these comparisons I can't judge whether the visibility-field formulation is better than existing tools. I hope the authors can add the missing experiments.

  2. The solver is unspecified, and the robustness claim isn't backed by the results. Eq. 4 says "rigid", but the mesh is normalized, so scale must also be estimated. The initialization, optimizer, number of hypotheses, handling of symmetric objects and runtime are all missing, so the "efficient" claim can't be checked. Training observations are always exact subsets of the same mesh, with no depth noise, no scale gap, no inter-object occlusion and no shape mismatch, which is precisely the generated-vs-observed mismatch used to motivate the paper. Unsurprisingly, the method degrades more than SAM 3D under MoGe depth: CD ×2.1 vs. ×1.34, and [email protected] drops 11.9 vs. 3.6 points.

Questions for authors

1.Please carefully specify the pose solver: transform type and how scale is recovered, initialization and restarts, symmetry handling, and runtime vs. Any6D / SAM 3D+PO. Is the solver in the Tab. 3 "Only Pose Solver" ablation identical?

2.Can you report rotation, translation and scale errors (or success rates at thresholds), along with median CD and how failed generations are counted?

3.How does ScenePlacer compare with Predator/GeoTransformer and FoundationPose/SAM-6D given the same mesh and observation?

4.Does adding depth noise, occlusion and mesh perturbation during training reduce the GT→MoGe drop? A breakdown by occlusion ratio would also help.

5.Is any quantitative evaluation on real data possible?

If the authors can adequately address the above during the discussion period, I would consider raising my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.