acceptodds
Withdrawn from ICLR 2027

Model-Free 6D Pose Estimation via 3D Registration with Distance-Color Fields

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We address model-free 6D pose estimation, which recovers the relative pose, i.e., 3D translation and orientation, of an arbitrary object from a pair of reference and query RGB-D images, without requiring a pre-built CAD model. Unlike existing approaches, we tackle this problem directly in 3D space through two steps: (i) reconstructing a 3D mesh of the reference object using a generative reconstruction framework, and (ii) registering the query instance to the reconstructed mesh via 3D registration based on distance–color fields. Our amodal generator effectively reconstructs a complete 3D mesh aligned with the occluded reference view. Our Distance-Color Field Sample Consensus (DCF-SAC) efficiently performs 3D registration using distance-color fields of the reference mesh as fast lookup tables for sample-consensus scoring. On six benchmarks, the proposed method matches or outperforms reconstruction-based and reconstruction-free methods while running about four times faster than the leading reconstruction-based method.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Model-free 6D pose estimation, 3D Registration, Point Cloud, RANSAC

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hGaqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a model-free 6D pose estimation framework that combines generative 3D reconstruction with distance-color field-based registration. The core idea is to reconstruct a complete 3D mesh from a single reference RGB-D image and then evaluate and refine pose hypotheses without repeated rendering by the proposed DCF-SAC. Experiments on six benchmarks demonstrate competitive accuracy and improved efficiency.

Critical strengths
  1. The paper addresses a meaningful problem in model-free 6D pose estimation and combining generative reconstruction with direct 3D registration is reasonable.
  2. The proposed DCF-SAC provides an efficient way to evaluate and refine pose hypotheses using distance and color fields.
  3. The experiment is relatively comprehensive, covering six benchmarks, representative baselines, and extensive ablation studies. The reported results are strong.
Critical weaknesses
  1. The contribution seems incremental. The proposed method largely builds upon existing generative reconstruction, correspondence-based registration, distance-field optimization, and sample-consensus techniques.
  2. The paper emphasizes amodal reconstruction as an important component, while the overall AR improves only marginally from 74.75 to 74.92 after finetuning. Moreover, the occluder-aware conditioning introduced during training is disabled at inference. How the model benefits from occlusion-aware training without this conditioning?
  3. The improvement over CoordAR in mean BOP AR decreases from 2.9 to 0.8 points when using the original symmetry annotations, which should be further discussed.
  4. Although generating the mesh in a known reference coordinate frame removes the need for a separate pose alignment step, reconstruction inaccuracies may still introduce geometric or scale misalignment.
Questions for authors

Please refer to the Weaknesses.

Reviewer_4PTYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors address the problem of model-free 6D pose estimation meaning in robotic scenarios with no model provided as CAD object. The authors reconstruct a 3D mesh using a generative reconstruction framework and register the 3D shape to given RGB-D scan.

Critical strengths

The work is an interesting approach to solving the model-free 6d pose estimation problem. In general, the approach is engineering heavy but rigors in the object pose aspects, thus an interesting work for real-world applications.

Critical weaknesses

I think the work mainly lacks in novelty for the level of this conference. The work is a very nice engineering contribution but at the end of the day just combines two approaches and spends extensive effort in optimizing it from an engineering perspective (but with lack of additional insights or observations that match this conference - while still being a very good paper but more focused on the application itself)

  • The method combines two approaches thus lack novelty
  • Since its a generative method not only 6d pose estimation should be evaluated but additionally the generation quality (chamfer distance etc.). The reasoning behind this is as the generation becomes worse how does it impact the pose estimation process and the end-to-end grasping process. As the 3D generative model becomes better (especially Sam3D-objects, GPT-6-Astra for example and newer methods) does the pose estimation process become obsolete or are there still relevant improvements.
  • Does the runtime also consider the generation time?
  • I think the strengths of the paper are in its runtime (at least assumed runtime for now) and this also comes from the DCF-SAC, which precomputes a certain set of values to reduce corresponds computation time
Questions for authors

See under weakness, but additionally for the tested scenarios how relevant is a model-free approach in reality? Just a general question, as most bin picking scenario assumes a model is given which seems quite fair for a factory setting, and for real-world humanoid examples the problem are different in nature and more dynamic? (not an attack on the paper genuine interest on if its actually extendable to real-world humanoid scenarios?)

Reviewer_9kx1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses model-free 6D pose estimation from a single reference and a set of query RGB-D views, in two stages: (1) an amodal generative reconstructor, built on TRELLIS and its depth-aware variant Affostruction, which produces a complete textured mesh of the reference object directly in a chosen frame so that the reference pose is known by construction; and (2) DCF-SAC, a sample-consensus registration in which the mesh's unsigned distance and surface color are baked once into a shared lattice, so each pose hypothesis is scored by one lookup per query point instead of by rendering, with the same truncated cost reused as a point-to-field ICP objective. Results are reported on six benchmarks at 74.9 mean BOP AR and 324 ms. These are my thoughts after my first full read; I did not check the implementation.

Critical strengths

The useful idea is that dense verification should be paid once per object rather than once per hypothesis: baking the mesh into a distance and color field replaces a per-hypothesis neighbor query with one shared bake plus lookups, and the same truncated cost then serves for hypothesis scoring and for refinement. The paper also re-runs all baselines under a single protocol rather than quoting published numbers.

Critical weaknesses
  1. I am not an expert in 6DoF pose estimation, but I am curious how well current geometric foundation models already do on this problem, since registering an RGBD observation to another RGBD view of the same rigid object should be quite similar to multi-view geometry estimation. Against that, the subsequent optimization in this paper (hypothesis generation, cost evaluation, ICP optimization and early stopping) appears too complicated to me. Given a reconstructed mesh, a direct pose estimation method such as VGGT applied to the two views is the obvious comparison, and it is absent. Without it I cannot tell how much of the pipeline is necessary.

  2. The real-time claim appears confusing. TRELLIS generation should take more than two seconds, while the reported runtime is 324 ms, and Tab. 7 and Tab. 13 report that same figure as a registration-only, per-query-frame cost. It is therefore unclear whether generation time is counted.

Questions for authors
  1. Why would introducing 3D generation help, given that the other views are hallucinated? Relatedly, since hallucinated surface can only matter where the reference and query views do not overlap, could you report accuracy stratified by relative viewing angle or surface overlap?

  2. VGGT + relative pose extraction should be a baseline. Fundamentally, isn't registering one RGBD frame to another RGBD frame of the same object exactly the object pose estimation problem? Since depth is provided on both sides, scale can be fixed directly. Please also report VGGT + 3D generation (rendering multiple views) followed by the same registration, if multiple views are helpful.

  3. Can you also compare with off-the-shelf TRELLIS + FoundationPose? Tab. 4 evaluates FoundationPose on the paper's own reconstruction, but the version without the amodal finetuning and canonical-pose conditioning is not reported, so it is not possible to see whether retraining the generator was necessary.

  4. Was the method compared against real-time 6-DoF pose tracking built on multi-view geometry networks, e.g. KV-Tracker [1], which operates from monocular RGB without depth, object priors, or any generated mesh?

  5. Is the generation time counted in the 324 ms of Tab. 3, and is the reconstructed mesh cached per reference view? Please report end-to-end latency per benchmark, with reconstruction and registration listed separately. Relatedly, how does this method compare to FoundationPose once configuration is accounted for?

[1] KV-Tracker: Real-Time Pose Tracking with Transformers, CVPR 2026. https://marwan99.github.io/kv_tracker/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.