InstantRefine: One-Reference 6D Pose Refinement for Novel Objects by Direct Regression

The task of 6D object pose estimation plays an important role in downstream robotics and augmented reality applications. To broaden its practical applicability, recent research increasingly leverages a single reference view rather than requiring a complete 3D object model. A common paradigm first estimates a coarse pose and subsequently refines it using a dedicated module. However, existing refiners primarily focus on establishing fine-grained 3D–3D correspondences, where pose updates are derived indirectly and remain susceptible to suboptimal predictions due to the misalignment between correspondence metrics and pose accuracy. To address this, we propose InstantRefine, a simple yet effective plug-and-play refiner that directly regresses pose updates. By representing both the coarse pose and the observations as a set of point tokens within a Transformer architecture, our method bypasses explicit correspondence matching, 3D model reconstruction, and geometric pose solvers (e.g., RANSAC/SVD) entirely. Beyond the one-reference setting, our framework naturally extends to multi-view references, significantly enhancing pose accuracy. Extensive experiments across multiple benchmarks demonstrate that InstantRefine consistently outperforms existing refiners under diverse pose initializations and extreme viewpoint variations.