acceptodds
Under review as a conference paper at ICLR 2027

What Did We Learn from 20k Hours of Open-Source Robot Data?

Abstract

Robot learning is often constrained by the perceived scarcity of data, yet the community has already accumulated tens of thousands of hours of publicly available demonstrations. The key challenge is thus not collecting more data, but effectively reusing heterogeneous datasets collected across different robot platforms. We propose the camera-centric action representation as a unified interface for robot data. This representation captures the relation between vision and action while abstracting away embodiment-specific kinematic differences. Building on this idea, we present CamX, a unified corpus of 20k hours of open-source robot demonstrations from 37 sources, converted to a common camera-centric representation. To leverage CamX, we train CamUVA, a unified video-action model that uses the camera-centric action representation and identifies every camera view and action stream by its camera pose. This design allows the model to extract value from cross-embodiment datasets and generalize to unseen robot embodiments out of the box. Together, CamX and CamUVA provide a foundation for systematically studying both what matters—the design choices that enable effective scaling—and what is possible—the capabilities that emerge from scaling. Ultimately, through our empirical study, we hope to show a path toward scaling robot learning by effectively reusing the growing body of heterogeneous open-source robot data.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Cross-Embodiment Training, Robotic Manipulation, Video-Action Models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_stGqOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to effectively reuse large-scale heterogeneous open-source robot data across embodiments. The authors introduce CamX, a corpus of roughly 20k hours of robot demonstrations from 37 sources converted into a unified camera-centric representation, and CamUVA, a unified video-action model that identifies visual and action streams through camera poses and jointly models video and actions. Beyond presenting the resulting model, the paper uses this setup to study several design choices for heterogeneous robot learning, including action representation, camera-pose encoding, multi-objective training, and model scaling. The final model is evaluated with a single checkpoint across multiple robot embodiments and settings, including cross-embodiment and unseen-embodiment transfer.

Critical strengths

Evidence at a scale commensurate with pretraining-level conclusions. The empirical scale is valuable because the breadth and heterogeneity of both the training data and downstream evaluation are well matched to the claims being made. CamX aggregates data across many sources and embodiments, while the design studies repeatedly evaluate on the full 120-task RoboLab benchmark and the final model is tested across simulated and real-world settings with both seen and unseen embodiments. This makes the resulting observations more indicative of large-scale heterogeneous robot pretraining behavior, rather than effects specific to a small number of source-target pairs.

Key methodological choices are well interrogated. The paper separately examines several decisions that are central to the proposed framework, including the action representation, pose-based view identification, organization of the video-action objectives, and model/compute scaling. The largely controlled experimental setup makes these studies useful beyond the final model itself, and provides a more interpretable empirical picture of which choices appear to matter when learning from heterogeneous robot data.

Critical weaknesses

The current evidence does not fully disentangle the benefit of canonicalization from the benefit of using the camera frame itself. The main action-representation experiment shows that camera-pose actions are substantially more effective than joint-space actions under cross-embodiment training, supporting the value of a shared geometric action representation. Appendix C.1 further illustrates that dataset-provided end-effector frames follow inconsistent conventions, whereas camera frames follow a standardized convention. However, these comparisons couple two distinct factors: canonicalizing the coordinate convention across embodiments and using the camera pose itself as the canonical action. The current evidence therefore does not clearly identify which of these factors is primarily responsible for the observed improvement. For example, an embodiment-specific fixed transform could in principle map each native EEF convention to a common TCP convention, with semantically aligned axes such as a consistent approach direction and gripper-opening direction. Since a rigidly mounted wrist camera is itself related to such a TCP by a fixed SE(3)SE(3) transform, this leaves open whether the benefit comes mainly from adopting a consistent coordinate convention or from the additional visual-action alignment specific to the camera frame. Clarifying this distinction would substantially sharpen the paper's central representation claim.

Questions for authors

Camera frame versus a canonicalized TCP frame. Could the authors clarify what is the essential source of the gain from the camera-centric action representation: the use of a consistent canonical coordinate system, or the choice of the camera frame specifically? Since a wrist camera and the corresponding end effector are rigidly related within each embodiment, such a baseline would help distinguish whether the advantage comes mainly from coordinate canonicalization or from the additional inductive bias of expressing perception and action in naturally aligned frames.

Coverage of camera-centric canonicalization across heterogeneous datasets.
The appendix clearly describes the validity checks used to filter datasets whose camera calibration or coordinate relationships cannot be reliably recovered. What remains less clear to me is the practical coverage cost of these requirements. Across the original candidate datasets (Table 2 in Appendix C), what fraction of the data can ultimately be canonicalized reliably, and what are the dominant reasons for exclusion (e.g., missing or inconsistent camera-to-robot calibration, inaccurate nominal extrinsics, or unavailable transforms between actuator groups)? This seems particularly relevant for datasets such as AgiBot, where only a subset of the available data appears to be retained. A quantitative characterization of this attrition would help clarify how readily the proposed interface can scale to additional heterogeneous open-source robot data.

4 comments
AuthorsOpenReview ↗

Response to Reviewer stGq

W1 / Q1 (Camera frame vs. a canonicalized TCP frame)

"...what is the essential source of the gain from the camera-centric action representation: the use of a consistent canonical coordinate system, or the choice of the camera frame specifically?"

A fingertip-based TCP convention does not directly extend to head and body motion, whereas an attached-camera frame provides a common convention across these moving bodies. In the practical process of canonicalization, the camera frame also provides an explicit and visually verifiable alignment check through CAD model projection (see camx-anonymous.github.io/camx-dataset), helping detect misaligned camera-frame definitions and avoid conversion mistakes.

We define the canonical TCP at the midpoint between the closed fingertips, with z along the approach direction and x/y following the wrist-camera convention. We did not see a clear difference when training on only three embodiments (DROID, AgiBot and YAM). Therefore, we ran a head-to-head comparison using the full CamX mixture with 5,000 GB300 GPU-hours per model, keeping the training recipe and data mixture identical and varying only the action representation. On the 120 RoboLab DROID tasks, the success rate with camera pose (58.8%) is higher than with canonical TCP (51.7%) by 7.1 percentage points. We also evaluate bimanual YAM on the same 120 RoboLab tasks, and the success rate with camera pose (23.8%) is similar to canonical TCP (24.2%). Since both arms use a consistent canonical convention, the 7.1-point gain on DROID comes from the camera frame itself rather than from canonicalization, while on YAM the two are similar. Together with the practical benefits of a standardized camera-frame convention, these results support our choice of the camera-pose action space.

Q2 (Coverage cost of camera-centric canonicalization)

"...what fraction of the data can ultimately be canonicalized reliably, and what are the dominant reasons for exclusion...?"

Thank you for the careful review and for raising the question of AgiBot dataset coverage: the download was interrupted, and now we have expanded Beta from 71 tasks (1,080 h) to 193 tasks (2,623 h), with 2,601 h in the current training pool. Sources absent from the submitted mix now contribute 5,969 training hours after conversion and validation.

For the original candidate-source list, the updated inventory contains 27,210 h. The latest training recipe retains 24,501 h (90.0%) for training and 152 h for validation. The remaining omissions from the inventory are:

Recorded reason and representative datasetHoursShare of candidate hours
Missing/inconsistent calibration or unrecoverable pose references: Dexora and OpenNeo Flexiv1360.50%
Inaccurate nominal extrinsics: affected OpenNeo ARX5 and UR5 tasks1470.54%
Unavailable transforms between actuator groups: FastUMI-100K dual-arm and Vista-UMI3151.16%
No usable wrist imagery: affected Bridge V2 recordings910.33%
Pose/video timing mismatch, predominantly GenRobot8813.24%
Truncated video files: GenRobot4691.72%
Other filtering and training-selection omissions5191.91%

The largest identified losses are temporal misalignment and truncated videos. The recovery of previously omitted sources demonstrates that submission-time coverage understated compatibility with the interface. We will add this accounting to Appendix C.2, separating operational omissions, technical exclusions, and validation holdouts.

I would like to thank the authors for their thorough rebuttal. The response has addressed my main concerns with strong empirical evidence.

Specifically, for W1/Q1, the authors' large-scale comparison (5,000 GPU-hours) is highly convincing. The 7.1% performance gain on DROID tasks provides strong evidence that the camera frame itself offers a unique advantage beyond simple coordinate alignment. This clarifies the central claim of the paper.

Furthermore, the detailed accounting of data being excluded with corresponding reason, and the clarification regarding the AgiBot dataset effectively resolve my concerns about the scalability and practical applicability of the CamX interface.

Given these clarifications, I am satisfied with the response and have increased my score to 4.

AuthorsOpenReview ↗

Reply to the Reviewer's Update

Thank you for your quick response!

As the large-scale comprison keeps running (now we have trained 7k gpu hours), we notice that the performance gap between canonical TCP (57.8%) and camera (59.6%) becomes substantially smaller (1.8%). Our hypothesis is that given limited number of embodiments (we only have 77 sources in the training data), sufficient compute, and a large model size, the model is still able to memorize all TCP definitions per embodiment and performs similarly to the more aligned camera pose control, just like the result we have observed in the 3-embodiment training. RoboLab DROID is also a seen embodiment so it does not require zero-shot embodiment generalization.

That said, the gap before 5k gpu hours is persistent, and we would modify our claim as: Camera pose provides a less ambiguous cross-embodiment action representation over canonical TCP pose with faster convergence speed, but does not substantially improve performance under sufficient training compute. It also provides an intuitive approach to visually check the alignment of both camera and TCP frame canonicalization, avoiding potential transform mistakes.

We are sorry for the incomplete conclusion in the previous reply. Please feel free to adjust your recommendation score accordingly.

I thank the authors for their transparent and rigorous response.

I am impressed by the authors' commitment to scientific integrity, specifically their decision to run additional large-scale comparisons to ensure the accuracy of their conclusions. Although the additional experiments show a narrower performance gap between camera-pose and canonical TCP, I believe this does not undermine the paper's fundamental contribution. The revised focus on the benefits of reduced ambiguity and faster convergence actually makes the claims more robust.

The substantial value of the work—namely the large-scale data infrastructure and the systematic empirical framework—remains intact. I will maintain my score of 4.

Reviewer_hGwGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to reuse heterogeneous robot demonstrations for cross-embodiment learning. It introduces CamX, a corpus assembled from multiple sources containing approximately 20k hours of demonstrations, standardized into a camera-centric action representation. Building on this corpus, CamUVA combines camera-pose encoding, flexible sequence packing, and joint video-action training to support different robot and camera configurations. The paper evaluates architectural and training choices through simulation ablations, and demonstrates deployment of a single checkpoint on single-arm, bimanual, and mobile manipulation systems, including transfer to unseen gripper–camera configurations.

Critical strengths

The strongest aspect is the substantial integration and empirical effort. Converting heterogeneous datasets into a consistent representation requires careful handling of coordinate conventions, calibration, temporal alignment, and data quality. The resulting pipeline and corpus could be useful resources for the community.

The evaluation covers complementary settings, i.e., a 120-task simulation benchmark, several real-world platforms, and controlled changes to grippers and cameras. Deploying the same checkpoint across these configurations without task-specific fine-tuning is a meaningful practical result. The model achieves performance comparable to the strongest evaluated simulation baseline while enabling efficient action-only inference.

The paper also provides useful implementation details and discusses failures rather than presenting only successful demonstrations. The ablations on geometric view identification and training-objective sampling help explain the final system design.

Critical weaknesses

My main concern is that the conceptual novelty and new insights are relatively limited compared with the breadth of the engineering effort. Camera-frame action representations, geometric attention encodings, unified video-action objectives, and data-semantic alignment have substantial precedents. The paper should more clearly establish its incremental contribution over some baselines such as Qwen-RobotManip. In particular, comparing camera trajectories primarily against joint-space actions leaves the advantage over closely related relative end-effector representations unresolved.

The experiments demonstrate a capable integrated system, but provide weaker evidence for the central scaling argument. Most design studies use DROID or a small mixture of sources, and the paper does not sufficiently isolate the benefit of the full CamX mixture from backbone initialization, compute, and other training improvements. In Figure 5, adding embodiments reduces DROID performance for both action representations; this supports reduced negative transfer rather than positive transfer from heterogeneous data.

The cross-embodiment evaluation also lacks competing methods under the same transfer protocol. Its results establish that CamUVA can transfer, but not that its proposed interface transfers better than existing alternatives. Moreover, the five configurations share the same ARX5 arm, so this experiment primarily tests gripper and camera variation. Performance remains uneven, including complete failure on one task–configuration pair.

Finally, some conclusions require statistical caution. Real-world conditions typically contain only 10–20 trials, several baseline results are taken from prior publications, and most ablation uncertainties reflect selected checkpoints rather than independent training runs.

Questions for authors
  • What is the precise advantage of predicting camera trajectories over existing camera-frame relative end-effector actions? Please clarify the relationship to Qwen-RobotManip formulations and identify which differences are substantive beyond coordinate parameterization.

  • What evidence isolates the value of the full CamX mixture? Can the authors provide a controlled comparison or existing training analysis separating data diversity and scale from compute and backbone effects?

  • How does CamUVA compare with a closely related method under the same cross-embodiment protocol? A matched comparison would strengthen the claim beyond demonstrating transfer within the proposed model.

  • How broadly should the zero-shot embodiment claim be interpreted? Please distinguish unseen kinematics, unseen gripper geometry, unseen camera models, and unseen combinations of otherwise familiar components.

  • How robust are the smaller reported improvements? Please clarify sensitivity to checkpoint selection, training seeds, and deployment settings, and distinguish directly reproduced baseline comparisons from comparisons using published numbers.

5 comments
AuthorsOpenReview ↗

Response to Reviewer hGwG (1/3)

W1 / Q1 (Camera trajectories vs. camera-frame end-effector actions, and Qwen-RobotManip)

"...the conceptual novelty and new insights are relatively limited..." ... "What is the precise advantage of predicting camera trajectories over existing camera-frame relative end-effector actions?"

We agree that the individual components have precedents; our contribution is unifying them through one camera-centric interface and the empirical findings this enables.

Qwen-RobotManip predicts end-effector motion expressed in the reference-camera axes (Eq. 5). This removes dependence on the EEF frame orientation convention, but retains dependence on the chosen EEF frame origin: different definitions can yield different translational targets for the same rigid-body rotation. Their preprocessing adjusts TCP definitions to reconcile logged poses with robot kinematics, but does not specify a universal EEF-origin convention across embodiments (e.g., the gripper tip or flange). They also discuss a full-transform alternative (Eq. 6), which corresponds to relative camera motion for a camera rigidly attached to the EEF, but adopt Eq. 5 because of concerns about long-tailed translation distributions and calibration sensitivity.

In contrast, our method directly predicts camera trajectories as actions, using the well-defined camera optical center as the origin and a consistent camera-axis convention across all datasets. Therefore, camera trajectories provide an operational advantage: an explicit convention that can be visually verified through CAD model projection (see camx-anonymous.github.io/camx-dataset), helping detect misaligned camera-frame definitions and avoid conversion mistakes.

In our empirical results, we did not see a clear difference when training on 3 embodiments (DROID, AgiBot and YAM) and comparing EEF (or, equivalently, TCP) actions with camera actions. Therefore, we ran a head-to-head comparison using the full CamX mixture with 5k GB300 GPU-hours per model, keeping the training recipe and data mixture identical and comparing only canonicalized TCP and camera-pose actions. We define the canonical TCP at the midpoint between the closed fingertips, with z along the approach direction and x/y following the wrist-camera convention. On the 120 RoboLab DROID tasks, the success rate with camera pose (58.8%) is higher than with canonical TCP (51.7%) by 7.1 percentage points. We also evaluate bimanual YAM on the same 120 RoboLab tasks, where the success rate with camera pose (23.8%) is similar to canonical TCP (24.2%). Since both arms use a consistent canonical convention, the 7.1-point gain on DROID comes from the camera frame itself rather than from canonicalization, while on YAM the two are similar. Together with the practical benefits of a standardized camera-frame convention, these results support our choice of the camera-pose action space.

W2 / Q2 (What isolates the value of the full CamX mixture)

"What evidence isolates the value of the full CamX mixture?" ... "...this supports reduced negative transfer rather than positive transfer from heterogeneous data."

To isolate the effect of the training mixture, we compare DROID-only and full-CamX models trained for 5k GB300 GPU hours with the same initialization, number of tokens per iteration, and training recipe. We evaluate the same two checkpoints on the original Franka setup in RoboLab and a modified setup using bimanual YAM, with 1,200 evaluation episodes per setup.

On the original Franka setup, DROID-only and full-CamX achieve 60.2% and 58.8% success, respectively, despite DROID accounting for only 11.6% of the mixed model’s training tokens. On bimanual YAM (16.8% of training tokens in the full mixture), the corresponding success rates are 3.5% and 23.8%. Thus, at a matched training budget, the full mixture substantially improves performance on the changed embodiment while largely retaining performance on the original setup. We agree that Fig. 5 likewise shows reduced negative transfer rather than positive transfer.

This comparison isolates the effect of replacing DROID-only training with the full mixture, but does not separately identify the contributions of data scale, diversity, or individual sources. Both runs use a later training recipe and evaluation setting, so their absolute scores should not be compared directly with Fig. 9.

AuthorsOpenReview ↗

Response to Reviewer hGwG (2/3)

W3 / Q3 (A competing method under the same cross-embodiment protocol)

"How does CamUVA compare with a closely related method under the same cross-embodiment protocol?"

We do not yet have a competing method evaluated under the same gripper–camera transfer protocol. We compare CamUVA with the cross-embodiment methods LAP and CLOAK on DROID, which is included in both baselines’ training data. CamUVA outperforms both by at least 32.2 points. This provides a comparison on a shared data domain, although the full training mixtures differ. Evaluating these baselines on other robots and tasks absent from their training data would confound method differences with training coverage. Retraining them on the full CamX mixture would provide a more controlled comparison, but exceeds our available compute budget. We acknowledge that our cross-embodiment experiment establishes CamUVA’s transfer capability, but does not isolate a comparative advantage of its camera-centric interface over existing alternatives. We offer CamUVA as a reference baseline for future evaluations across a broader range of embodiments.

W3 / Q4 (How broadly the zero-shot claim should be read)

"How broadly should the zero-shot embodiment claim be interpreted?" ... "Performance remains uneven, including complete failure on one task–configuration pair."

In this cross-embodiment experimental setup, our policy follows UMI, taking only wrist-camera images as input and predicting camera trajectories as output. Robot arm kinematics are abstracted out of the policy interface, and the policy can be deployed on robot arms with different IK controllers. Therefore, our zero-shot cross-embodiment result mainly concerns unseen gripper geometry and unseen gripper-camera combinations. In the reviewer's categories:

CategoryEvidence
Unseen arm kinematicsWe additionally deploy the same checkpoint on a UR7 arm absent from training. It achieves 16/20 successes on cup arrangement, similar to the result on ARX5.
Unseen gripper geometryWe evaluate the policy with a newly designed gripper finger: 30/40 successful trials.
Unseen camera modelOur training data covers a broad range of camera models used in public robot datasets, including the D405. We did not test unseen camera models.
Unseen combination of seen componentsUMI fingers paired with a D405: 28/40 successful trials. During training, these fingers were paired only with GoPro and iPhone cameras.

Following prior work (LAP and CLOAK), we use “zero-shot” to mean deployment without training or fine-tuning on the target embodiment configuration. This does not require matching the exact performance on seen embodiments.

The one complete failure is GenRobot on Cup Flipping (0/10): GenRobot differs from the demonstration embodiment (iPhUMI) in both gripper geometry and camera type (fisheye stereo instead of a rectilinear iPhone camera), and the policy consistently overshoots the cup before closing the gripper (Appendix E.4.2). The other 19 task–configuration pairs succeed in 30–100% of trials, and 16 of them in at least 70%.

AuthorsOpenReview ↗

Response to Reviewer hGwG (3/3)

W4 / Q5 (Robustness of the smaller improvements)

"How robust are the smaller reported improvements? Please clarify sensitivity to checkpoint selection, training seeds, and deployment settings..."

Checkpoint selection. Our design-study scores average the results of the three best evaluated checkpoints under the same selection rule for all variants. Replacing these with the final three evaluated checkpoints preserves all method rankings in Figures 5 and 7. Success rates decrease by only 0.51 percentage points on average across the 12 bars. In particular, Difficulty-Aware’s improvements over Token-Matching remain positive at 3.21 and 4.54 points in policy and policy-planner modes, respectively. These results support robustness to checkpoint selection.

Training seeds. Each full-DROID ablation requires approximately 4,000 GB300 GPU-hours to converge, limiting most design choices to one training run. The Difficulty-Aware ablation has a second seed under the same checkpoint-selection and evaluation protocol. Relative to the first seed, its success rate decreases by 2.6 percentage points in policy mode and increases by 1.0 percentage point in policy-planner mode. The new seed also outperforms the other methods, indicating a persistent improvement.

Deployment and trials. In RoboLab simulation, performance depends on model-specific rollout settings. We therefore evaluate alternative configurations and report each model under its best-performing tested configuration, with settings and sensitivity documented in Appendix D. For real-world cross-embodiment evaluation, we conduct 200 trials across four tasks and five configurations. Each configuration is evaluated on the same ten initial states per task, varying background, object position, and object identity or pose. This matching reduces differences in task conditions when comparing embodiments. We acknowledge that ten trials per condition provide limited precision: the results support successful transfer across the tested configurations, while small differences between embodiments should be interpreted cautiously.

Baseline result sources. We directly evaluated all five RoboLab models and the ABC-VLA baseline. The scores for UVA, Diffusion Policy, and ModPack come from their respective publications, with adapted evaluation protocols (matching objects, initial configs and backgrounds in each episode). We will label Fig. 9 to distinguish directly evaluated baselines from published results.

Reviewer_C9KKOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work construct CamX by converting 20k open-source robot demonstrations into a unified camera-centric format and evaluate CamUVA trained on this dataset. The key distinction between CamUVA and prior UVA/WAM approaches is its camera-centric action representation, which captures the relationship between visual observations and the robot’s physical movements. Through this approach, the authors develop a UVA that generalizes across embodiments, including grippers with different geometries and new camera configurations. They present analyses culminating in Findings 1-4 in the paper. Based on the proposed pipeline, experiments demonstrate improvements from incorporating camera poses and multiple objectives, while the cross-embodiment results further support the effectiveness of the proposed approach.

Critical strengths
  • CamUVA uses CaPE based on camera poses in SE(3) and is designed to flexibly support different camera configurations. This enables CamUVA to build a pipeline for jointly training on 20k hours of data spanning diverse embodiments.

  • The empirical findings, including the benefits of camera poses, the advantages of using different UVA training objectives, and scaling experiments across model size and training data, offer meaningful insights for the field.

  • CamUVA is trained from a pretrained Cosmos checkpoint, providing a way to inherit the upstream model’s rich understanding of dynamics and expressive motion representations.

  • The authors conduct ablation studies on various techniques incorporated into CamUVA, including difficulty-aware token balancing and CaPE, and provide cross-embodiment evaluations across diverse embodiments.

Critical weaknesses
  • A core assumption of the proposed camera-centric representation is that wrist-camera motion serves as a proxy for end-effector motion. This assumption may make joint training with human data challenging.

  • Token balancing relies on several heuristic choices whose justification is unclear.

  • The model does not condition future predictions on an extended history of past observations. Consequently, it may struggle with tasks requiring memory or repeated actions, and its applicability may be limited to tasks whose state is fully determined by the current image.

Questions for authors

Key distinction from prior work using CaPE is that this study assigns identical 3D positional indices to tokens at the same timestep and resolution, relying solely on CaPE to distinguish between views. Does this design yield meaningful improvements over Simplex RoPE or spatial concatenation on a broader range of tasks beyond 'carry a box' task? If so, what underlying mechanism explains these improvements?

1 comment
AuthorsOpenReview ↗

Response to Reviewer C9KK

W1 (Camera-centric actions and human data)

"This assumption may make joint training with human data challenging."

For human hand data with a wrist camera, we can still use the camera-pose representation, while adding a new action MLP encoder and decoder for hand joint angles or fingertip poses, as with dexterous hand datasets. For purely egocentric human data without wrist cameras, video could enter the training recipe through the action-free video-generation mode for task planning and language grounding.

W2 (Heuristic choices in token balancing)

"Token balancing relies on several heuristic choices whose justification is unclear."

Token matching addresses a measurable imbalance: equal sample counts give policy training fewer tokens because a policy sample contains only one observation frame. At fixed compute, matching token counts raises policy success from 32.4 to 43.7 and policy-planner success from 39.8 to 46.0 (Fig. 7).

The additional difficulty-aware weighting is a heuristic. We prioritize policy and policy-planner modes because they must choose task-relevant actions; forward and inverse dynamics receive additional conditioning, and policy-planner training already predicts video. The hand-set sampling ratio is 10:1:0.25:0.25:0.05 for policy, policy-planner, video generation, forward dynamics, and inverse dynamics, based on the convergence speed observed during token-matching training. It adds 3.7 and 4.0 points on top of the token-matching baseline; we do not claim this ratio is optimal.

Source shares are also hand-set to prevent corpus size from determining exposure. We adjust the sampling ratio based on task diversity and data quality.

W3 (No extended observation history)

"The model does not condition future predictions on an extended history of past observations."

The submitted checkpoint indeed uses only the current observation. The architecture supports observation history through attention to earlier chunks, whose keys and values remain cached at inference. In our updated training run, half of the training tokens are used for chunk-wise autoregressive generation, with an attention mask similar to those used in Lingbot-VA and DreamZero. The model supports up to 60 chunks of history (32 actions per chunk, thus at most 120 s of history when executed at 16 Hz). We evaluated our model on tasks that require memory (see camx-anonymous.github.io/camuva-rollouts/#long-context) and achieved an overall success rate of 13/15. The cup task appears in the training data (Gated Memory Policy), and the other two tasks are completely unseen throughout training. This demonstrates that the model is able to refer to observation history.

Q1 (CaPE beyond the box-carrying task, and the mechanism)

"Does this design yield meaningful improvements over Simplex RoPE or spatial concatenation on a broader range of tasks beyond 'carry a box' task? If so, what underlying mechanism explains these improvements?"

Beyond box carrying, we evaluate action-conditioned video generation as the camera moves randomly around a central pose. We compare two protocols: teacher-forced prediction uses ground-truth conditioning frames at every chunk; autoregressive rollout uses ground-truth context for the first 10 chunks, then feeds generated frames back as conditioning. CaPE achieves higher PSNR under both protocols, with a 3.4–3.5 dB gain over the alternatives during autoregressive rollout:

View encodingTeacher-forced PSNRAutoregressive PSNR
CaPE22.9 dB22.6 dB
Concatenation21.3 dB19.1 dB
Simplex21.7 dB19.2 dB

With identical 3D positional indices across views, CaPE supplies relative camera geometry directly to attention. We hypothesize that this helps retrieve relevant past observations from similar viewpoints, improving consistency during rollout. In bimanual control, fixed camera mounts also make relative wrist-camera poses informative about relative gripper poses. Concatenation and Simplex distinguish views without explicitly encoding this geometry.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.