acceptodds
Withdrawn from ICLR 2027

Calib-X: Training-Free Camera-to-Robot Pose Estimation via Dense Feature Matching

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Camera-to-robot pose estimation is a fundamental yet challenging problem for vision-based robotic manipulation, requiring accurate recovery of the rigid transformation between a camera and a robot. Conventional calibration approaches often rely on fiducial markers or manual procedures, limiting scalability in real-world deployments. Recent learning-based methods alleviate this burden by directly predicting poses from images, but they typically require large-scale pose annotations and costly retraining when transferring to new robots or environments. In this paper, we introduce , a framework that reformulates camera-to-robot calibration as an problem. Leveraging robot kinematics, Calib-X first generates multi-view robot renderings from predefined viewpoints and establishes dense correspondences between rendered templates and an uncalibrated camera image using semantic features from the foundation model DINOv3. These correspondences enable direct estimation of the initial camera pose without any task-specific optimization or data collection. Furthermore, we propose an iterative pose refinement strategy that continuously re-renders the robot under the updated pose and performs dense feature alignment with the query image, progressively improving calibration accuracy. Extensive experiments on four public benchmarks demonstrate that Calib-X achieves accurate and robust camera-to-robot pose estimation across diverse robotic platforms and environments. Without requiring markers, annotations, or retraining, Calib-X provides a scalable and generalizable solution for real-world robot calibration. The code will be released publicly upon publication.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Camera-to-robot pose estimation, Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_j9nFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a training-free method for relative camera-to-robot pose estimation. Hand-eye calibration is a real and well-known problem in robotics, and typical solutions involve using a calibration board. The authors propose leveraging a robot pose estimation pipeline based on robot renders in a given joint state with a two-stage process: first, dense feature matching against rendered templates using a pretrained matching model, and then local refinement via iterative re-rendering and matching.

Critical strengths
  1. The proposed pipeline is simple to use and doesn't require any model training.
Critical weaknesses
  1. Fundamentally, this is still a 6-DoF pose estimation pipeline, so it is not clear what the value of the custom method is—why not use existing CAD-based 6-DoF pose estimation pipelines?
  2. Conceptually, there is nothing novel in the method. The value for the community is fairly limited.
  3. Practically, the quality of the method fully depends on an accurate digital twin for both the robot and the gripper.
  4. One of the advantages of calibration-based methods is that you can run relative pose estimation across a set of robot poses, compensating for any kinematic imperfections. Authors do mention a multi-frame variant, but if I understand correctly, it still describes a fixed robot pose and multiple rendered images.
Questions for authors
  1. Did the authors try to compare the method to regular training-free CAD-based 6-DoF object pose estimation methods (e.g., FoundPose, MegaPose, Pos3R)?
  2. How is the multi-frame Calib-X† in Table 3 formulated?
Reviewer_PH7ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Calib-X estimates camera-to-robot pose from a single RGB image without robot-specific training. Given camera intrinsics, robot geometry, and joint states, it renders templates at sampled azimuths with a fixed elevation and a configuration-dependent viewing distance. SAM suppresses the observation's background, and pretrained RoMa v2 establishes rendered-to-real correspondences. Rendered position maps convert these into 3D-to-2D matches for PnP-RANSAC. The selected coarse pose is refined by repeatedly re-rendering the robot and updating the correspondences and pose.

Quantitative experiments on DREAM-real and Panda Manipulation compare with trained calibration methods and examine refinement, foreground masking, and matcher choice. On DREAM-real, Calib-X achieves an ADD AUC of 87.704% versus 86.231% for CtRNet-X. Three matching-based refinement iterations take approximately six seconds in the reported experiment. The paper also includes a multi-frame variant and qualitative examples on DROID and RoboChallenge Table30v2 to illustrate performance in varied environments and on additional robot platforms.

Critical strengths
  1. Marker-free calibration without robot-specific training is practically useful, and the pipeline combines pretrained matching with known robot geometry in a straightforward way.
  2. DREAM-real results are competitive with trained methods. Table 2 also shows that feature-matching refinement is substantially faster than the evaluated differentiable-rendering alternative, approximately 6 versus 120 seconds.
Critical weaknesses
  1. The methodological contribution is insufficiently distinguished. Rendered-template matching and PnP are established ideas, including in FoundPose. The paper should identify the robot-specific advances and clarify comparisons with existing training-free calibration methods under comparable input assumptions.
  2. Viewpoint coverage is not adequately tested. Templates sample azimuth on an orbit with fixed elevation and one distance per robot configuration. Robustness to top-down, close-range, or otherwise poorly covered views remains unclear, as does sensitivity to template count and placement.
  3. Partial-visibility robustness is overstated. In Table 3, single-frame Calib-X improves AUC over CtRNet-X (69.977 versus 59.828) but has worse mean ADD (0.091 versus 0.056 m). This mixed result does not support an unqualified superiority claim and warrants error-distribution analysis.
  4. The multi-frame result is insufficiently specified. Table 3 reports Calib-X† without explaining frame count or aggregation, making comparison with CtRNet-X† difficult to assess. Key template and correspondence-selection settings are also missing.
Questions for authors
  1. Contribution and comparable baselines. Which components are new or materially modified relative to FoundPose-style template matching combined with robot kinematics? A direct implementation under matched inputs would help establish the contribution. For other training-free calibration methods, please distinguish single-image methods from those requiring multiple frames, exploratory motion, or manual annotations when defining a fair comparison.
  2. Template coverage and reproducibility. Please specify the template count, elevation, configuration-dependent viewing distance, correspondence threshold, and PnP-RANSAC settings. How does initialization behave for top-down or close-range partial views far from the template orbit? A sensitivity study of template placement and count would clarify whether refinement can reliably recover from poor initial coverage.
  3. Partial-visibility error distribution. In Table 3, higher AUC but worse mean ADD suggests that the advantage varies across the error distribution. Can you report median and upper-quantile ADD, failure rates at a stated threshold, and representative failures? Please examine whether large errors correlate with visibility, viewpoint, masking, or correspondence quality.
  4. Multi-frame estimation protocol. How many frames does Calib-X† use, how are they selected, and how are observations with different joint configurations combined? Please explain whether correspondences are jointly optimized or independently estimated poses are aggregated, and whether CtRNet-X† uses the same frame budget. Reporting the added runtime would clarify the practical benefit over single-frame calibration.
Reviewer_ytFiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Calib-X estimates the camera-to-robot transformation from a single RGB image, given camera intrinsics, a robot model, and the current joint configuration. It matches the observation against multiple rendered robot templates using pretrained RoMa v2, lifts the matches to 2D–3D correspondences, and estimates the pose with PnP-RANSAC. Re-rendering and re-matching then refine the estimate. The pipeline avoids robot-specific training and manual calibration targets.

Critical strengths
  • The approach is clear and practically motivated. Combining pretrained matching with known robot geometry provides an understandable calibration pipeline, and avoiding robot-specific training is useful when adapting to another robot or repositioning a camera.

  • The reported results support the usefulness of the complete system. On DREAM-real, the overall AUC is 87.7, slightly above CtRNet-X. The refinement and foreground-masking ablations also provide evidence that these components help within the proposed pipeline.

  • The authors state that the code will be released upon publication.

Critical weaknesses
  • I would appreciate a clearer account of the contribution beyond existing model-based 6D pose estimation pipelines. Given the robot model and joint configuration, the robot’s 3D geometry is known, making the connection to model-based 6D pose estimation particularly relevant. FoundPose and Pos3R already establish image-to-model correspondences using foundation features and rendered templates, followed by geometric pose estimation. GenFlow also uses iterative rendering and correspondence-based pose refinement, although its training requirements differ. The paper cites the first two works, but I am still unsure which robot-specific difficulty requires a new design here, and which capability goes beyond adapting these pipelines with RoMa v2. A more explicit comparison of the operations, assumptions, and added benefits would help establish the independent contribution.

  • Tables 2 and 5 are helpful, but they do not fully separate the benefit of the stronger matcher from that of the proposed refinement strategy. A controlled comparison of MASt3R and RoMa v2, before and after refinement, would help clarify this. The foreground processing, templates, and geometric estimation should be comparable, with the matching resolution, sampling, and confidence filtering explained. The broader conclusion that dense matching is more suitable than sparse matching also seems stronger than this experiment establishes, since the compared methods differ in features and training as well as correspondence density.

  • Partial views can leave fewer distinctive correspondences or weaker geometric constraints, even when the underlying camera-to-base transformation remains fixed. With this in mind, I am wondering where the method becomes unreliable. In Table 3, the single-frame robot-in-and-out result has a higher AUC than CtRNet-X, but its mean ADD is worse: 91 mm versus 56 mm. These aggregates do not reveal whether a small number of severe failures or a broader error distribution accounts for this difference. Representative failure cases and the frequency of large errors, including cases where refinement worsens the estimate, would help assess the robustness claim.

Questions for authors
  • My main concern is Weakness 1. Could the authors clarify which robot-specific challenges require the proposed design and what Calib-X adds beyond the cited 6D pose estimation pipelines? Please also explain how the contribution statement and related-work discussion would be revised. If the response establishes this contribution with convincing evidence, I would be willing to reconsider my current rating.

  • Section 3.2 explains viewpoint sampling and SAM masking, but gives little detail on rendering appearance. Beyond the reported masking, are standard robot-model renders sufficient for reliable RoMa v2 matching, or are specific materials, lighting, or additional preprocessing needed? Please clarify the setup and whether these choices materially affect correspondence quality and pose accuracy. If special choices are essential, a comparison with a simple default rendering would help establish their role. This would also clarify whether successful rendered-to-real matching comes mainly from RoMa v2 or from additional design in Calib-X.

  • Could the authors discuss the conditions under which Calib-X would be preferable to conventional target-based calibration, considering accuracy requirements and setup effort? A discussion grounded in the reported accuracy and required inputs would help clarify its intended use cases.

Reviewer_yZfKOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Calib-X estimates the camera-to-robot pose from one RGB image, given the intrinsics, the robot model and the joint state. Given the joint state, the shape of the robot is known. So the pose estimation can be done by a template matching and PnP refinement with a known-pose rendered views (so as the 3D point positions). Calib-X renders T templates on an orbit at fixed elevation and distance, masks the query with SAM, and runs the pretrained RoMa v2 matcher between each template and the query. Rendered position maps lift the matches to 3D. PnP-RANSAC gives one pose per template, and the best by inlier count and reprojection error becomes the coarse pose. Refinement then repeats re-rendering at the current pose, re-matching and PnP. "Training-free" here means no robot-specific training, in the same sense as FoundPose or Pos3R.

The claimed contributions are the training-free framework, sim-to-real correspondences without calibration annotations or fine-tuning, and the iterative refinement. Quantitative validation covers two datasets, both using the Franka Panda: DREAM-real (overall AUC 87.70 and mean ADD 12 mm, against 86.23 and 14 mm for CtRNet-X) and Panda Manipulation (single-frame AUC 85.38 robot-in-view and 69.98 in-and-out, against 60.32 and 59.83). A batch variant reaches 88.54 and 83.49. Ablations cover refinement, foreground masking and the matcher. DROID and RoboChallenge Table30 v2 (ARX5, UR5) appear only as overlay figures.

Critical strengths

Calib-X is a simple but competitive method based on the presented experimental results. It's a training-free framework without the requirement of calibration annotations for fine pose estimation. It's also fast: 3 iterations at about 6 s beat 3,000 steps of differentiable rendering at about 120 s on partial views.

Critical weaknesses
  1. The generalisation claim has no numbers, and no training-free method is compared. The quantitative results come from one robot, the Panda, for which trained models already exist. The abstract claims "extensive experiments on four public benchmarks" and "diverse robotic platforms", but DROID and Table30 v2 get only overlay figures.

  2. The method contribution is not clearly distinguished. The two-stage design, template correspondences and PnP followed by re-rendering and re-matching, is the structure of Co-op[CVPR 2025], which, though, aims for object pose estimation.

  3. Viewpoint coverage is a separate gap. Most DREAM-real frames are Panda-Orb, whose cameras sit on an orbit at about 30° and 45° elevation according to the DREAM paper. The template generator in the paper uses the same geometry. The paper does not state the template elevation, so I cannot tell whether its largest benchmark tests off-template viewpoints at all.

  4. The number of the template and the radius of orbit of the rendering, two key numbers, are not given. If the radius is already close to the robot and the templates are fine-grained, the coarse pose estimation can already be very good.

  5. The reliability claims go beyond the tables. The text says Calib-X "clearly outperforms" CtRNet-X in single-frame estimation and gives "reliable calibration under both full and partial robot visibility". On in-and-out, Table 3 shows better AUC (69.98 against 59.83) but worse mean ADD (91 mm against 56 mm). And the 'reliable calibration' claim requires reports success rate or failure detection, etc.

  6. Two statements in the text are wrong or overstated. The 26.964 margin over the "best DREAM variant" is the gap to DREAM-F, while to the best variant is DREAM-H at 19.12. "Best overall performance on the four scenes" holds only for the avg, since CtRNet wins AK (89.928 against 85.224) and RS (90.789 against 87.953).

  7. The ablation conclusions are also stronger than the data or not clear. With SAM and three iterations, Table 4 gives in-and-out AUC 66.415 where Table 3 gives 69.977 for what appears to be the same configuration.

Questions for authors
  1. How is SAM prompted on each dataset for the robot? The masks shown in the paper are so perfect. Can SAM generate such good masks during the 1st stage?

  2. What are T, the template elevation, and the orbit distance rule? Are they the same for each dataset? Need to finetune the parameters to get the current speed and accuracy?

  3. Why do Tables 3 and 4 differ for the same configuration?

  4. Can you also report the performance of each step: coarse pose, each stage of the refinement, and the number of templates and orbit distance to generate the templates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.