acceptodds
Under review as a conference paper at ICLR 2027

InstantRefine: One-Reference 6D Pose Refinement for Novel Objects by Direct Regression

Abstract

The task of 6D object pose estimation plays an important role in downstream robotics and augmented reality applications. To broaden its practical applicability, recent research increasingly leverages a single reference view rather than requiring a complete 3D object model. A common paradigm first estimates a coarse pose and subsequently refines it using a dedicated module. However, existing refiners primarily focus on establishing fine-grained 3D–3D correspondences, where pose updates are derived indirectly and remain susceptible to suboptimal predictions due to the misalignment between correspondence metrics and pose accuracy. To address this, we propose InstantRefine, a simple yet effective plug-and-play refiner that directly regresses pose updates. By representing both the coarse pose and the observations as a set of point tokens within a Transformer architecture, our method bypasses explicit correspondence matching, 3D model reconstruction, and geometric pose solvers (e.g., RANSAC/SVD) entirely. Beyond the one-reference setting, our framework naturally extends to multi-view references, significantly enhancing pose accuracy. Extensive experiments across multiple benchmarks demonstrate that InstantRefine consistently outperforms existing refiners under diverse pose initializations and extreme viewpoint variations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
6D object pose estimation, computer vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_u2roOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

InstantRefine is a one-reference 6D pose refinement pipeline: given a query RGB-D image, a reference RGB-D image, masks, and an initial coarse pose, it back-projects and samples both views, transforms the query points into a Normalized Anchor Frame (NAF), tokenizes each view by concatenating some off-the-shelf features, and feeds the resulting point tokens into a Transformer. Two linear heads regress the delta rotation and delta translation, and the pose is updated iteratively. The core contribution is a Transformer-based delta-pose estimation network operating on NAF-anchored tokens, trained on synthetic data and evaluated on LINEMOD, YCB-V, and TUD-L against ICP, UNOPose, COG, and FoundationPose.

Overall, the pipeline is coherent and the results are promising, but the paper still contains some unclear claims and analysis. The multi-reference contribution is not discussed in detail. The two core components, NAF and tokenization, are defined rather than analyzed, with no ablation over normalization schemes, token counts, or individual feature sources. The training procedure is underspecified—the loss, rotation parameterization, and data scale are missing. Overall, several issues are fixable with additional experiments, but the multi-reference claim and the ablation inconsistency affect core credibility.

Critical strengths

1.A complete, modular one-reference refinement pipeline that removes reconstruction, explicit matching, and geometric solvers from the loop. 2. An end-to-end Transformer delta-pose network with decoupled [R]/[T] tokens and 4D positional encoding, naturally extensible to extra reference views. 3. Credible single-reference gains over correspondence- and reconstruction-based refiners, with reconstruction overhead eliminated, across three datasets and three upstream coarse-pose sources.

Critical weaknesses
  1. Pipeline, Network, and Training Are Only Partially Presented The training objective is not mathematically specified: the rotation parameterization (6D/quaternion/axis-angle), whether the rotation loss is geodesic, and what λmax = 10 actually weights are all missing. The target delta pose is expressed in NAF (Eq. 4, Fig. A7), but the network's output parameterization within NAF—specifically how the 6-vector output maps to the NAF translation component—is not specified. The scale of the training data is never quantified: there are no object/scene/view-pair counts, no split, and no deduplication, so the 400K iterations cannot be judged. Moreover, the training data are synthetic, and there is no discussion of whether the synthetic-to-real gap influences performance; no mixed or adapted training is reported. The single-view vs. multi-view recipe is unclear: the single-reference setting is "trained with V = 2," whereas the multi-view setting uses V = 2–9; yet testing uses 16 references with no explanation.

  2. NAF and Tokenization Are Defined, Not Analyzed Sensitivity is reported only at the end-to-end level (Table A6 for SAM masks, Fig. A9 for depth noise), not at the level of NAF's center and diameter estimates. How d and the frame origin respond to mask misses, depth bias, and missing regions is not analyzed; the robustness boundary of the frame itself is never discussed. NAF–initial-pose coupling is quantified only through binned analysis over COG's natural error distribution (Fig. A10); there is no controlled sweep over initial rotation/translation error, so the rotational error threshold beyond which refinement degrades is unknown. There is no ablation over alternative normalization schemes (unit sphere, PCA, learned). The per-view token count (128) has no ablation or justification, and the per-view budget is never varied. Multi-view token scaling is not discussed; 16 references would yield ~2210 tokens with quadratic attention cost, yet Table 2 reports FPS and memory without decomposing the token-count contribution to accuracy or cost. Tokens are the core of the pipeline; Table 3 provides only partial evidence—B1 removes PointNet geometry while DINOv2/SD-VAE remain, and A1 removes SD-VAE—but there is no independent removal of DINOv2 and no controlled comparison of feature-source contributions under a fixed token budget. Which of PointNet, DINOv2, or SD-VAE contributes most is therefore not answered and should be discussed.

Questions for authors
  1. Are the 16-reference results (Tables 1–2) produced by the same model trained with V ≤ 8, or by a separately trained multi-view model? If the former, what allows the model to handle 16 references unseen during training (e.g., permutation-invariant attention over views)? If the latter, please give its training configuration. This directly determines whether the multi-reference contribution stands.

  2. In Table 3, A1 (no SD-VAE) leaves YCB-V ADD(-S) unchanged and drops AR by only 0.3, and D1 (entangled translation) differs from Full by ≤1.5 AR, yet the text claims "larger degradation" and a "consistent performance drop." Is this a table error, a metric error, or overstatement? If the gaps are within noise, please report multi-seed variance.

  3. Please state the training data scale and model capacity of the UNOPose, COG, and FoundationPose pretrained models, and whether they are comparable to InstantRefine. Can a same-class direct-regression refiner (DeepIM-style or an adaptation with the same interface) be added?

  4. Please explain the basis for 128 per-view tokens, and whether token count versus accuracy and attention cost was analyzed at ~2210 total tokens; whether PointNet, DINOv2, and SD-VAE contributions can be isolated under a fixed token budget, since Table 3's B1/A1 only partially separate them.

  5. Please state whether any analysis shows the synthetic-to-real gap is not a bottleneck (e.g., synthetic-only vs. mixed training). If no such data exist, this is future work and will not count against the current assessment.

Reviewer_VUJdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents InstantRefine, a framework for refining the 6D relative pose of previously unseen objects from a single reference view. Given reference and query RGB-D images with object masks and camera intrinsics, the method refines an initial relative pose supplied by an external estimator by directly predicting rotation and translation updates, without explicit correspondence matching, object reconstruction, or geometric pose solvers. It first transforms the reference and query point clouds into a normalized coordinate frame anchored to the reference observation. Local geometry is then combined with image features extracted using DINOv2 and a Stable Diffusion VAE to form point tokens. A Transformer uses spatial positional encoding to process these tokens, while dedicated rotation and translation tokens aggregate evidence for pose regression. The updates can be applied iteratively, and additional reference views can be incorporated as aligned observation tokens. Trained on synthetic data, the model is evaluated on LINEMOD, YCB-Video, and TUD-L with initial poses from three external estimators. Experiments compare its refinement accuracy and computational efficiency with correspondence-based and reconstruction-based approaches. Ablation studies examine the contributions of individual components, while further analyses assess how performance depends on initialization quality and responds to depth noise and predicted segmentation masks.

Critical strengths
  1. InstantRefine directly predicts pose updates from reference and query RGB-D observations, avoiding explicit correspondence matching, object reconstruction, and geometric pose solvers. This formulation enables refinement for novel objects without requiring complete 3D models at inference.

  2. The evaluation covers three benchmarks and three external pose initializers. In the single-reference, single-iteration setting, InstantRefine consistently outperforms the compared UNOPose and COG refinement modules on both ADD(-S) and AR, supporting its applicability across different coarse estimation pipelines.

  3. The token-based architecture accommodates additional aligned reference observations, while predicted pose updates can be applied iteratively. The experiments demonstrate further accuracy gains from these extensions, providing options for adjusting the balance between accuracy and computational cost.

Critical weaknesses
  1. Although qualitative examples illustrate challenging viewpoint changes, the paper does not report performance stratified by reference–query viewpoint difference or visible surface overlap. The initialization analysis in the appendix examines coarse pose errors, which is a distinct factor. Consequently, the evaluation provides limited evidence for the claimed advantage under extreme viewpoint changes and low overlap.

  2. The paper motivates its approach through the mismatch between correspondence supervision and pose accuracy. However, comparisons with existing refiners also involve differences in architecture and feature representations. A controlled comparison using the same backbone and inputs would be needed to establish how much of the improvement comes specifically from replacing correspondence estimation with direct pose regression.

  3. The multi-reference accuracy results in Table 1 use five refinement iterations, whereas the corresponding runtime in Table 2 uses one iteration. These results therefore describe different operating settings. Moreover, the 16-reference configuration requires 5.8 GB of GPU memory and runs at 5.0 FPS for one iteration. Reporting accuracy and runtime under matched settings would clarify the computational cost of the reported performance gains.

Questions for authors
  1. Can the authors report performance by reference–query viewpoint difference and visible surface overlap to substantiate the claimed robustness to extreme viewpoint changes and low overlap?

  2. Can the authors provide a controlled comparison using the same backbone and inputs to isolate the benefit of direct pose regression over correspondence estimation?

  3. Can the authors report multi-reference accuracy and runtime using the same number of refinement iterations to clarify the computational cost of the reported gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.