acceptodds
Withdrawn from ICLR 2027

CalibAll: Offline Eye-to-Hand Calibration for Heterogeneous Robot Datasets

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cross-embodiment robot learning requires action representations with consistent semantics across robot platforms. Existing representations suffer from platform-specific inconsistencies, while current solutions often maintain embodiment-specific action heads or learn latent action spaces without explicitly establishing a shared geometric reference. Recent approaches adopt camera-centric motion representations to address this issue, but missing camera extrinsic annotations limit their coverage across existing datasets. We present CalibAll, an optimization-based, robot-independent pipeline for offline camera extrinsic annotation. CalibAll initializes extrinsics via temporal Perspective-n-Point (PnP) using a tracked end-effector point, then refines the estimate by aligning rendered and observed robot masks through differentiable rendering. It additionally derives camera-frame tool center point (TCP) poses from robot and gripper URDFs to support action unification. We apply CalibAll to 16 datasets spanning four robot platforms, producing approximately 97K calibrated episodes across single-arm and bimanual systems. Downstream experiments show that using the annotated dataset for cross-embodiment pretraining improves policy performance in both simulation and real-world tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
hand-to-eye caliberation, dataset, annotation pipeline

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tB8COpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The proposed CalibAll is a training-free pipeline for estimating camera extrinsics from existing robot manipulation datasets. It converts robot end-effector poses into standardized camera-frame tool center point (TCP) actions.

Critical strengths

The problem is important for cross-embodiment robot learning, and the approach is practical because it reduces dependence on physical robot access and robot-specific training data. The reported calibration results are strong: on DREAM, CalibAll achieves an AUC of 97.642 and an average distance error of 8 mm, outperforming the compared methods. Simulation and real-robot experiments also provide encouraging evidence of downstream utility, including a 77.5% average real-world success rate across four tasks.

Critical weaknesses

There is a gap between the paper’s broad scalability claims and its calibration-failure analysis. In the DROID evaluation, tracking fails in 5 of 20 scenes, and only 14 scenes yield successful refinement.

The dataset-construction procedure also requires manual intervention when tracking or segmentation fails, limiting the claim of fully automatic annotation.

In addition, the calibration benchmark is primarily based on DREAM; evaluation across additional platforms with independently verified ground-truth extrinsics would better substantiate robot independence.

Finally, real-world results are based on only 10 trials per task and setting. The reported differences are promising but have substantial sampling uncertainty, and the paper provides limited statistical analysis.

Last point, the single-arm/bimanual action convention—assigning arms according to their left-to-right image position—may be sensitive to camera viewpoint and could produce inconsistent action semantics across scenes.

Questions for authors

Here are some suggestions for improvements: It would be nice to report calibration success and failure rates across all evaluated datasets, robot platforms, and challenging conditions, including occlusion and clutter.

Also, it would be good to evaluate on more datasets with independent calibration ground truth and report translation/rotation errors alongside uncertainty or confidence intervals. Also provide ablations or sensitivity analyses for camera-intrinsic errors, tracking noise, segmentation quality, and manual intervention.

Finally, clarify how the arm-ordering convention behaves under changing viewpoints and test whether it preserves consistent action semantics.

Reviewer_SQs2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CalibAll, an optimization-based, robot-independent pipeline for annotating camera extrinsics in existing offline robot manipulation datasets. The pipeline operates in three stages: (1) coarse initialization via temporal PnP, where an end-effector tracking point is located using a vision foundation model and tracked through the video, then aligned with the forward-kinematics-derived 3D trajectory; (2) refinement via differentiable rendering that aligns rendered robot masks with observed masks obtained from a promptable segmentation model; and (3) postprocessing that converts flange poses to standardized camera-frame TCP poses and produces a unified 14-dimensional action layout to enable joint training of single-arm and bimanual data.

Critical strengths

The paper is clearly written and well-organized, making it easy to follow. The structure is logically arranged, and the work demonstrates a certain degree of novelty.

Critical weaknesses
  1. The experimental evaluation lacks sufficiently strong comparisons against state-of-the-art methods, which makes it difficult to judge the paper's claimed advances. (As shown in Table 1, the latest comparative method is the paper from 2025.)
  2. The headline claim rests on downstream gains (RoboTwin simulation, real-robot ALOHA), yet the Camera-Frame variant differs from baselines in multiple ways simultaneously: the action representation and the pretraining corpus (the annotated datasets vs. the original datasets), plus auxiliary annotations (masks, bounding boxes, 2D/3D TCP trajectories) that baselines do not receive. As presented, one cannot tell whether the improvements come from camera-frame actions, from the additional geometric annotations, or simply from a cleaner/larger pretraining mixture.
  3. The paper reports calibration accuracy only on DREAM (a Franka-only benchmark) and 20 DROID scenes; for the 16 heterogeneous datasets that constitute the actual contribution — including UR5e, XArm7, and bimanual ALOHA — there is no ground truth and no quantitative accuracy estimate, only qualitative renderings. Since these annotations are the paper's central deliverable, the absence of any error characterization on the released corpus leaves the correctness of the resource essentially unverified.
  4. The DROID experiment shows the pipeline failing on 5/20 scenes due to tracking loss, i.e., a 25% failure rate under realistic clutter and motion. This is consequential because it directly bounds the method's applicability to the broader OXE-scale corpus it targets: the very datasets most in need of retroactive calibration (in-the-wild, cluttered, dynamic) are those where the tracking component is least reliable. The paper would be substantially stronger with a per-dataset failure/effort breakdown and ideally an automatic re-initialization mechanism, rather than deferring full automation to future work.
  5. Too many of the cited references are arXiv preprints. Such papers have not undergone peer review and therefore lack authority/credibility.
Questions for authors
  1. The experimental evaluation lacks sufficiently strong comparisons against state-of-the-art methods, which makes it difficult to judge the paper's claimed advances.
  2. The headline claim rests on downstream gains (RoboTwin simulation, real-robot ALOHA), yet the Camera-Frame variant differs from baselines in multiple ways simultaneously: the action representation and the pretraining corpus (the annotated datasets vs. the original datasets), plus auxiliary annotations (masks, bounding boxes, 2D/3D TCP trajectories) that baselines do not receive. As presented, one cannot tell whether the improvements come from camera-frame actions, from the additional geometric annotations, or simply from a cleaner/larger pretraining mixture.
  3. The paper reports calibration accuracy only on DREAM (a Franka-only benchmark) and 20 DROID scenes; for the 16 heterogeneous datasets that constitute the actual contribution — including UR5e, XArm7, and bimanual ALOHA — there is no ground truth and no quantitative accuracy estimate, only qualitative renderings. Since these annotations are the paper's central deliverable, the absence of any error characterization on the released corpus leaves the correctness of the resource essentially unverified.
  4. The DROID experiment shows the pipeline failing on 5/20 scenes due to tracking loss, i.e., a 25% failure rate under realistic clutter and motion. This is consequential because it directly bounds the method's applicability to the broader OXE-scale corpus it targets: the very datasets most in need of retroactive calibration (in-the-wild, cluttered, dynamic) are those where the tracking component is least reliable. The paper would be substantially stronger with a per-dataset failure/effort breakdown and ideally an automatic re-initialization mechanism, rather than deferring full automation to future work.
  5. Too many of the cited references are arXiv preprints. Such papers have not undergone peer review and therefore lack authority/credibility.
Reviewer_FgzaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The papers introduces a system that can retrieve camera-extrinsics (among other auxiliary information that can be used for downstream applications) from a given robotics dataset and annotate the same so as to be used to pretrain a generalist policy that can work across several fixed-base manipulator embodiments. The method itself shows significant gains over other related works on two camera-pose estimation metrics. The policy, when finetuned task-specifically, is then shown to produce small advantages in simulation benchmarks and substantial advantages in real-life tasks over other methods such as - task-specific training from scratch, cross-embodiment pretraining with the raw action representations mixed, using robot-specific embodiment information in prompts and using robot-specific action heads in pretraining. The real life tasks further show a small advantage over the foundation model, pi0.5. Overall, I view the main contributions of this paper as an offline pipeline for camera-extrinsics annotation on already collected data and the key insights that follow the same, like getting a coarse annotation and refining it helps and that the specific combination of methods used here works empirically.

Critical strengths

Real-robot evaluations. The authors have demonstrated higher success rates on both simulation and in real-life robotics tasks. It shows that the same pretraining method could transfer to real-life. The evaluation against a VLA like pi0.5 in the real tasks, apart from the self-designed ablations, also gives me a good anchor to judge the results on.

Intuitive pipeline. For the specified problem of retrieving camera-extrinsics in an offline manner that the paper has set out to solve, the pipeline is very intuitive. A practitioner can gain several overall insights from the design of the coarse-to-fine pipeline while reading the paper as it is presented. I also appreciate the sections under 4.1 - “Ablation study”, “Convergence analysis of refinement” as it further reinforces the overall intuition for the workflow. The same pipeline also producing auxiliary information like masks and bounding boxes for both the robot and gripper, 3D TCP trajectories in the camera frame, and corresponding 2D TCP trajectories in the image space is added strength.

Critical weaknesses

The extensive effort for little returns. From a thorough read of the paper, I was able to make out that unquantified human-engineering effort went into annotation of the dataset, with several parts requiring quality control. The pipeline too, though intuitive, appears to be a very expensive combination of prior methods. As the paper itself currently frames it, if the problem is missing camera extrinsics in currently curated datasets in the field of robotic manipulation (i.e. with robotic arms like Franka, UR5e + Robotiq, AgileX ALOHA, XArm7, etc.,), it begs a question about whether just collecting new data with the camera extrinsics annotation done in a disciplined manner could solve the issue. This is a major worry because the paper also appears highly specific to only manipulation datasets (and not, for example, internet-scale datasets like human videos) and since, currently the field is only at its infancy, a simple solution of disciplined cross-embodiment-compatible data collection, following its open-source release, could be more impactful. Coming to the little returns part of my point, I could see that in simulation, despite pretraining for 200k steps and finetuning for 100k steps, the method only gains around 6 percent improvement over the from-scratch baseline which as I understand is only trained for 100k steps task-specifically (without any 200k step pre-training). In real-life, despite the substantial gain from other baselines, again I could see only a small absolute improvement over pi0.5. (I look at absolute scores here as the number of evaluation trials is less). This reinforces my worry that the paper appears to do a lot, for a little advantage. While this is not bad inherently, the value of CalibAll as a tool is dependent on its efficacy, quick setup and usage and downstream advantage.

Unclear attribution to what makes the method work. Building on the prior point, there is already a question of how much of the advantage should be attributed to human effort and careful quality control. Apart from this, the implementation choices of the pipeline’s individual components are undermotivated or underexplained as presented. The experiments (for which the number of training seeds are not reported) also show noisy results: the simulation evaluation of the main method has a sudden comparatively sharp dip in the finetuning at 80k timesteps and the real-robot task gains look overblown compared to the simulation gains for which I couldn’t find any deep investigation or explanation. Coming to robustness, Appendix D reports successful calibration on 14/20 scenes, with tracking failing in five cases. This further strengthens my concern on the method’s robustness itself.

Questions for authors

Clarification-based:

  1. While I understand it is hard to retrieve lost engineering experience when working on this project, I would be more assured if the authors could provide a honest write-up on more details with respect to the pre and post processing steps. Such as what was automated, how much quality control had to be done, what other human-engineering had to be done while curating the dataset. Only if time allows, I would be much more assured if the authors can do a smaller-scale controlled and quick experiment demonstrating that the same method without significant engineering can provide gains on a smaller cross-embodiment dataset.
  2. Would the datasets curated with camera extrinsics be open-sourced? And could the paper be edited and framed in a way that removes the stronger claims of solving cross-embodiment representations?
  3. Could the authors explain the noisy finetuning results and/or statistically prove through another way that the camera-frame pretraining and finetuning is as reliable as advertised? Similarly, could the authors explain the discrepancy between the simulation gains and the real robot task gains (i.e. around 6 percent improvement over from-scratch in sim benchmark vs. around 43 percent improvement over from-scratch in real-life tasks)

Suggestion-based:

  1. Could the from-scratch baselines be extended to match the overall training steps used for other baselines and the main method? Is there a way to transfer the insights or even the pipeline onto human-to-robot data pipelines? Could the authors illustrate this using a small experiment, if time permits?
  2. I found several of the segmented image data annotation examples in the appendix very interesting. A section in the main paper with such an illustration and some description or qualitative / quantitative analyses of the same would strengthen the presentation.

To summarize my position on this paper, if my questions are answered and I can be convinced of the impact of this work despite the effort-vs-returns trade-off, then I would be willing to move my score up.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.