- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to effectively reuse large-scale heterogeneous open-source robot data across embodiments. The authors introduce CamX, a corpus of roughly 20k hours of robot demonstrations from 37 sources converted into a unified camera-centric representation, and CamUVA, a unified video-action model that identifies visual and action streams through camera poses and jointly models video and actions. Beyond presenting the resulting model, the paper uses this setup to study several design choices for heterogeneous robot learning, including action representation, camera-pose encoding, multi-objective training, and model scaling. The final model is evaluated with a single checkpoint across multiple robot embodiments and settings, including cross-embodiment and unseen-embodiment transfer.
Critical strengths
Evidence at a scale commensurate with pretraining-level conclusions. The empirical scale is valuable because the breadth and heterogeneity of both the training data and downstream evaluation are well matched to the claims being made. CamX aggregates data across many sources and embodiments, while the design studies repeatedly evaluate on the full 120-task RoboLab benchmark and the final model is tested across simulated and real-world settings with both seen and unseen embodiments. This makes the resulting observations more indicative of large-scale heterogeneous robot pretraining behavior, rather than effects specific to a small number of source-target pairs.
Key methodological choices are well interrogated. The paper separately examines several decisions that are central to the proposed framework, including the action representation, pose-based view identification, organization of the video-action objectives, and model/compute scaling. The largely controlled experimental setup makes these studies useful beyond the final model itself, and provides a more interpretable empirical picture of which choices appear to matter when learning from heterogeneous robot data.
Critical weaknesses
The current evidence does not fully disentangle the benefit of canonicalization from the benefit of using the camera frame itself. The main action-representation experiment shows that camera-pose actions are substantially more effective than joint-space actions under cross-embodiment training, supporting the value of a shared geometric action representation. Appendix C.1 further illustrates that dataset-provided end-effector frames follow inconsistent conventions, whereas camera frames follow a standardized convention. However, these comparisons couple two distinct factors: canonicalizing the coordinate convention across embodiments and using the camera pose itself as the canonical action. The current evidence therefore does not clearly identify which of these factors is primarily responsible for the observed improvement. For example, an embodiment-specific fixed transform could in principle map each native EEF convention to a common TCP convention, with semantically aligned axes such as a consistent approach direction and gripper-opening direction. Since a rigidly mounted wrist camera is itself related to such a TCP by a fixed transform, this leaves open whether the benefit comes mainly from adopting a consistent coordinate convention or from the additional visual-action alignment specific to the camera frame. Clarifying this distinction would substantially sharpen the paper's central representation claim.
Questions for authors
Camera frame versus a canonicalized TCP frame. Could the authors clarify what is the essential source of the gain from the camera-centric action representation: the use of a consistent canonical coordinate system, or the choice of the camera frame specifically? Since a wrist camera and the corresponding end effector are rigidly related within each embodiment, such a baseline would help distinguish whether the advantage comes mainly from coordinate canonicalization or from the additional inductive bias of expressing perception and action in naturally aligned frames.
Coverage of camera-centric canonicalization across heterogeneous datasets.
The appendix clearly describes the validity checks used to filter datasets whose camera calibration or coordinate relationships cannot be reliably recovered. What remains less clear to me is the practical coverage cost of these requirements. Across the original candidate datasets (Table 2 in Appendix C), what fraction of the data can ultimately be canonicalized reliably, and what are the dominant reasons for exclusion (e.g., missing or inconsistent camera-to-robot calibration, inaccurate nominal extrinsics, or unavailable transforms between actuator groups)? This seems particularly relevant for datasets such as AgiBot, where only a subset of the available data appears to be retained. A quantitative characterization of this attrition would help clarify how readily the proposed interface can scale to additional heterogeneous open-source robot data.
4 comments
I would like to thank the authors for their thorough rebuttal. The response has addressed my main concerns with strong empirical evidence.
Specifically, for W1/Q1, the authors' large-scale comparison (5,000 GPU-hours) is highly convincing. The 7.1% performance gain on DROID tasks provides strong evidence that the camera frame itself offers a unique advantage beyond simple coordinate alignment. This clarifies the central claim of the paper.
Furthermore, the detailed accounting of data being excluded with corresponding reason, and the clarification regarding the AgiBot dataset effectively resolve my concerns about the scalability and practical applicability of the CamX interface.
Given these clarifications, I am satisfied with the response and have increased my score to 4.
Reply to the Reviewer's Update
Thank you for your quick response!
As the large-scale comprison keeps running (now we have trained 7k gpu hours), we notice that the performance gap between canonical TCP (57.8%) and camera (59.6%) becomes substantially smaller (1.8%). Our hypothesis is that given limited number of embodiments (we only have 77 sources in the training data), sufficient compute, and a large model size, the model is still able to memorize all TCP definitions per embodiment and performs similarly to the more aligned camera pose control, just like the result we have observed in the 3-embodiment training. RoboLab DROID is also a seen embodiment so it does not require zero-shot embodiment generalization.
That said, the gap before 5k gpu hours is persistent, and we would modify our claim as: Camera pose provides a less ambiguous cross-embodiment action representation over canonical TCP pose with faster convergence speed, but does not substantially improve performance under sufficient training compute. It also provides an intuitive approach to visually check the alignment of both camera and TCP frame canonicalization, avoiding potential transform mistakes.
We are sorry for the incomplete conclusion in the previous reply. Please feel free to adjust your recommendation score accordingly.
I thank the authors for their transparent and rigorous response.
I am impressed by the authors' commitment to scientific integrity, specifically their decision to run additional large-scale comparisons to ensure the accuracy of their conclusions. Although the additional experiments show a narrower performance gap between camera-pose and canonical TCP, I believe this does not undermine the paper's fundamental contribution. The revised focus on the benefits of reduced ambiguity and faster convergence actually makes the claims more robust.
The substantial value of the work—namely the large-scale data infrastructure and the systematic empirical framework—remains intact. I will maintain my score of 4.
Response to Reviewer stGq
W1 / Q1 (Camera frame vs. a canonicalized TCP frame)
A fingertip-based TCP convention does not directly extend to head and body motion, whereas an attached-camera frame provides a common convention across these moving bodies. In the practical process of canonicalization, the camera frame also provides an explicit and visually verifiable alignment check through CAD model projection (see camx-anonymous.github.io/camx-dataset), helping detect misaligned camera-frame definitions and avoid conversion mistakes.
We define the canonical TCP at the midpoint between the closed fingertips, with z along the approach direction and x/y following the wrist-camera convention. We did not see a clear difference when training on only three embodiments (DROID, AgiBot and YAM). Therefore, we ran a head-to-head comparison using the full CamX mixture with 5,000 GB300 GPU-hours per model, keeping the training recipe and data mixture identical and varying only the action representation. On the 120 RoboLab DROID tasks, the success rate with camera pose (58.8%) is higher than with canonical TCP (51.7%) by 7.1 percentage points. We also evaluate bimanual YAM on the same 120 RoboLab tasks, and the success rate with camera pose (23.8%) is similar to canonical TCP (24.2%). Since both arms use a consistent canonical convention, the 7.1-point gain on DROID comes from the camera frame itself rather than from canonicalization, while on YAM the two are similar. Together with the practical benefits of a standardized camera-frame convention, these results support our choice of the camera-pose action space.
Q2 (Coverage cost of camera-centric canonicalization)
Thank you for the careful review and for raising the question of AgiBot dataset coverage: the download was interrupted, and now we have expanded Beta from 71 tasks (1,080 h) to 193 tasks (2,623 h), with 2,601 h in the current training pool. Sources absent from the submitted mix now contribute 5,969 training hours after conversion and validation.
For the original candidate-source list, the updated inventory contains 27,210 h. The latest training recipe retains 24,501 h (90.0%) for training and 152 h for validation. The remaining omissions from the inventory are:
The largest identified losses are temporal misalignment and truncated videos. The recovery of previously omitted sources demonstrates that submission-time coverage understated compatibility with the interface. We will add this accounting to Appendix C.2, separating operational omissions, technical exclusions, and validation holdouts.