acceptodds
Withdrawn from ICLR 2027

Zeva-EgoAction: Scalable Egocentric Mid-Training for Robot Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Egocentric human videos contain manipulation experience, but their lack of action annotations limits their use in robot policy training. We propose Zeva-EgoAction, which extracts supervision from action-unlabeled egocentric videos to enhance robot-pretrained vision-language-action (VLA) models. Its Action-Centric Encoder (ACE) uses annotated trajectories to learn a physically grounded latent space through environment distillation and action-neighborhood matching, then generates action pseudo-labels for unseen and unlabeled Ego data. On an unseen dataset, ACE labels all evaluated samples without fine-tuning and improves action-representation fidelity, measured by action-distance correlation, by 37.6% relative to HaMeR with retargeting on the common valid subset. Scaling Ego mid-training data to 10,000 hours raises RoboTwin success from 63.8% to 75.3%; under this setting, approximately 4–5 hours of Ego video yield training gains comparable to one hour of robot demonstrations. Real-robot post-training further shows that task-matched Ego videos improve average success by 35 percentage points when Robot demonstrations are scarce. These results demonstrate that physically grounded pseudo-labels can convert large-scale human videos into effective supervision for robot policy training.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Egocentric Video, Vision-Language-Action Models, Action Representation Learning, Robot Manipulation, Data-Efficient Robot Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_t4DbOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Zeva-EgoAction learns an Action-Centric Encoder from annotated human and robot trajectories. An action-conditioned teacher models environmental variation, and a visual-only student learns continuous action tokens through environment distillation and trajectory-neighborhood matching. The frozen encoder labels additional human videos for VLA mid-training. The paper reports RoboTwin improvement from 46.00% to 51.98% under clean-to-random evaluation and from 63.8% to 75.3% in a separate mixed-training scaling study.

Critical strengths
  1. Leveraging large-scale human egocentric videos to improve robotic manipulation is highly relevant to the robotics community. The optional ICCL extension further explores adaptation through accumulated interaction experience without updating policy parameters during deployment.
  2. The combination of offline learning from diverse human videos and online adaptation through robot interaction experience is well motivated. Human videos can provide general physical interaction priors, while the optional online extension can help address local execution errors, such as failed grasps or inaccurate placement.
  3. The reported results are promising. In particular, the method achieves strong clean-to-random generalization on RoboTwin 2.0, outperforming the compared methods.
Critical weaknesses
  1. Several methodological details are insufficiently defined, making the presentation difficult to follow. For example, L_{cam} is introduced in Eq. (8) without a detailed definition in the manuscript; the values or estimation procedure for s_g in Eq. (12) are unspecified; and the conditioning variable c in R^2(a | c, Z_{env})-R^2(a | c), on page 7 around lines 373–374, is not explicitly defined.
  2. The related work section does not sufficiently cover prior approaches that share this paper’s motivation of leveraging human egocentric data to train or improve robotic policies and VLAs [1–4]. Discussing these works would help clarify how the proposed approach differs in its use of human data, supervision construction, and transfer to robot control.
  3. This paper proposes a trajectory-neighborhood KL matching method for continuous action tokens as its core contribution; however, the incremental contribution of this method compared to existing physics-based representation learning methods has not yet been fully substantiated. RS-CL [5] has already established a soft contrast supervision based on the distances between a robot’s proprioceptive states. Meanwhile, CLAP [6] aligns visual transitions with physical action representations through cross-modal contrastive learning based on a robot action dictionary.
  4. The physical distances used for training and fidelity evaluation require further clarification. Equation (12) appears to apply mean squared error (MSE) to each action component group (including quaternion-based rotations). In the absence of a consistent quaternion sign convention or a sign-invariant distance metric, equivalent rotations represented by qq and −q-q would result in non-zero distances. This could distort the evaluation of neighborhood supervision and representation fidelity. [1] In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Data,arXiv:2511.15704 [2] H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation,AAAI 2026 [3] PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence,arXiv:2512.16793 [4] Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments,arXiv:2606.32009 [5] Contrastive Representation Regularization for Vision-Language-Action Models [6] CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos
Questions for authors
  1. My main concern is the incremental methodological contribution. Could the authors clarify the distinctions and advantages of the proposed approach relative to RS-CL and CLAP, and provide controlled comparisons to support these claims?
  2. How is quaternion sign ambiguity handled in the physical distance defined in Eq. (12)? If no sign canonicalization or sign-invariant distance is used, how does this affect neighborhood supervision, fidelity evaluation, and downstream policy performance?
3 comments

Thank you for recognizing the relevance of Ego mid-training and the clean-to-random results.

1. Distinction from RS-CL and CLAP, and comparative evidence

Highlight. RS-CL [1] uses existing Robot-state annotations to improve policy representations; ZEVA-EGOACTION learns to label Ego videos without state or action annotations. Unlike CLAP [2], it preserves similar actions across recordings, reaching 51.98% RoboTwin success—3.86 points above CLAP—and outperforming CLAP and LAPA [3] in action-geometry preservation.

ZEVA-EGOACTION extends physical supervision to action-unannotated Ego video through a separately trained labeling encoder. RS-CL [1] uses existing proprioceptive-state annotations recorded in Robot demonstration datasets, rather than recovering states from RGB video. Distances between these annotated states provide soft supervision for policy representations. It studies visual robustness, including background changes, but does not provide an egocentric action-labeling pipeline that explicitly separates head-camera motion from manipulation. Our encoder learns from action-annotated human and robot data, then labels additional RGB transitions without action annotations.

ZEVA-EGOACTION preserves similar actions across recordings rather than treating them as negatives. CLAP [2] grounds visual transitions through a robot action dictionary and contrastive alignment, but its instance-based human-video contrast can treat different recordings of similar actions as negatives. Our physical neighborhoods instead provide graded relationships between annotated trajectories.

ZEVA-EGOACTION achieves higher downstream success than LAPA and CLAP under matched policy-training settings. The comparison uses the same π0.5\pi_{0.5} base initializer, Ego corpus, source mixture, policy-training budget, and RoboTwin post-training configuration. All methods are evaluated under Table 1's clean-to-random protocol over all 50 tasks.

MethodRoboTwin 2.0 Hard SR (%)
π0.5\pi_{0.5}46.00
π0.5\pi_{0.5} + LAPA mid-training41.48
π0.5\pi_{0.5} + CLAP mid-training48.12
ZEVA-EGOACTION51.98

ZEVA-EGOACTION also preserves physical-action geometry better than both baselines across all three datasets. A complementary encoder comparison on common GT-annotated transitions yields:

EncoderEgoDexAgiBotEgoVerse
LAPA0.1810.1940.199
CLAP0.1240.2870.149
ZEVA-EGOACTION0.8130.7880.747

Entries are episode-balanced Pearson correlations between physical-action and token distances, includes 200 EgoDex, AgiBot, EgoVerse episodes from distinct source sessions; respectively. EgoVerse is unseen during ZEVA-EGOACTION training. These results support ZEVA-EGOACTION as a labeling method; the released encoders have different pretraining histories, so this is not an isolated loss ablation. We have not conducted a direct RS-CL comparison and do not claim experimental superiority over it.

2. Quaternion ambiguity and the implemented rotation distance

Highlight. Our implementation resolves quaternion sign ambiguity before computing rotation-vector MSE; it does not compare raw quaternion coordinates. On the new evaluation sets, replacing this term with an SO(3) geodesic alternative changes total physical distance by less than 1% for at least 99% of pairs in each dataset.

Eq. (12) operates on canonicalized, chunk-start-relative rotation vectors, not raw quaternions. The manuscript omitted this conversion. The implementation composes rotations, aligns quaternion signs before interpolation, canonicalizes the scalar sign, and converts to rotation vectors before computing group MSE. This addresses the ordinary q/−qq/-q ambiguity, although the principal rotation-vector representation still has a boundary at π\pi and is not an SO(3) geodesic metric.

The sampled rotations remain away from the principal-vector boundary, and most pairwise distances are insensitive to the metric replacement. The maximum relative angles were 154.7° on EgoDex, 69.1° on AgiBot, and 144.1° on EgoVerse, with no sampled point reaching 170°. Replacing the rotation term with squared SO(3) geodesic distance, divided by three to match the original per-coordinate MSE convention, while retaining the other groups and scales, changes the total physical distance by less than 1% for at least 99% of evaluated pairs in each dataset. A small number of outliers remain more sensitive.

3. Definitions of LcamL_{\mathrm{cam}}, sgs_g, and cc

Highlight. We have clarified the implementation: LcamL_{\mathrm{cam}} supervises normalized camera-motion prediction, sgs_g provides fixed calibration scales for each action group, and cc conditions the leakage analysis on camera motion.

  • LcamL_{\mathrm{cam}}: masked MSE for six-dimensional camera-motion prediction—translation and rotation-vector increments expressed in the chunk's first-camera frame, normalized using source-specific statistics. We average over six coordinates, valid time steps within each sample, and then samples. Samples without camera supervision contribute no camera-loss term.
  • sgs_g: fixed group-MSE normalization scales for translation, rotation, and gripper: 3.95216×10−43.95216\times10^{-4}, 1.49381×10−21.49381\times10^{-2}, and 1.35501×10−21.35501\times10^{-2}, respectively. They were estimated as group-MSE medians from equal-duration EgoDex/AgiBot calibration pairs and remain frozen during evaluation.
  • cc: the camera-motion descriptor comprising start-to-end translation and rotation vector. The conditional leakage statistic measures the additional action predictability provided by ZenvZ_{\mathrm{env}} beyond that explained by camera motion alone.

We have added these definitions alongside the corresponding equations.

4. Coverage of related human-video approaches

Highlight. H-RDT [4] and In-N-On [5] rely on existing pose/motion annotations; Human-as-Humanoid [6] relies on pose recovery, whose errors can directly propagate into action-expert training; PhysBrain [7] provides semantic rather than physical-action labels. ZEVA-EGOACTION supplies physical priors to the VLM through action tokens rather than precise control targets, reducing reliance on high-precision pose recovery and enabling RGB-only labeling after encoder training.

We have expanded the related-work discussion around the following distinctions.

ZEVA-EGOACTION extends action supervision to Ego videos without existing motion annotations. H-RDT [4] and In-N-On [5] rely on existing pose/motion annotations for policy learning and human-to-robot transfer. Our encoder learns from annotated human and Robot trajectories, then labels additional videos from RGB transitions alone.

ZEVA-EGOACTION supplies physical priors to the VLM without requiring high-precision pose recovery from each human video. Human-as-Humanoid [6] reconstructs and retargets human motion into robot-action targets for flow-matching training, so recovery errors can directly propagate into action-expert training. Our action tokens instead support VLM mid-training, while Robot demonstrations provide precise control supervision.

ZEVA-EGOACTION provides physical-action supervision beyond semantic descriptions. PhysBrain [7] derives structured language descriptions from egocentric video for embodied understanding and planning. Our encoder instead uses physical-trajectory relationships to shape action-token geometry, allowing RGB video to supply action-level physical priors.

References

[1] T. Kim et al. Contrastive Representation Regularization for Vision-Language-Action Models. In International Conference on Machine Learning (ICML), 2026.

[2] C. Zhang et al. CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos. arXiv preprint arXiv:2601.04061, 2026.

[3] S. Ye et al. Latent Action Pretraining from Videos. In International Conference on Learning Representations (ICLR), 2025.

[4] H. Bi et al. H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation. arXiv preprint arXiv:2507.23523, 2025.

[5] X. Cai et al. In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Data. arXiv preprint arXiv:2511.15704, 2025.

[6] X. Lin et al. Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments. arXiv preprint arXiv:2606.32009, 2026.

[7] X. Lin et al. PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence. arXiv preprint arXiv:2512.16793, 2025.

Reviewer_t4DbOpenReview ↗

Response to Author

I am very grateful to the author for the detailed response, which has largely addressed my concerns. Therefore, I maintain my positive view of this paper.

Reviewer_jToSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ZEVA-EGOACTION, a method for converting action-unlabeled egocentric human videos into supervision for robot vision-language-action policies. Its Action-Centric Encoder (ACE) is trained in two stages: an action-conditioned teacher separates environment-related visual changes from action information, and a visual-only student learns action tokens whose neighborhood structure matches that of annotated physical trajectories. The frozen encoder then generates pseudo-labels for unseen egocentric data. Experiments report improved action-distance correlation on EgoVerse, gains on RoboTwin and real-robot ChemLab-Evo tasks, and improved performance as the amount and diversity of egocentric training data increase.

Critical strengths

The method directly addresses a meaningful representation-learning problem. Rather than treating visual reconstruction alone as evidence of action understanding, ACE uses annotated physical trajectories to shape the geometry of its action-token space. The ablations support the value of these design choices: HSIC reduces measured action leakage into environment tokens, while neighborhood matching improves action-distance correlations across Ego and robot data. This gives the central technical contribution a clear rationale and empirical support.

Critical weaknesses
  1. The real-robot evaluation covers only three atomic ChemLab-Evo tasks, with 20 trials per reported setting; the added human videos are task-matched and recorded from the robot’s main-camera viewpoint. This is useful evidence for the tested setup, but it leaves uncertainty about how well the approach transfers to more varied tasks, viewpoints, and operating conditions. Consequently, the real-world data-efficiency claims should be interpreted as specific to this limited evaluation.
  2. The paper reports point estimates for policy success and Pearson correlations, but does not provide confidence intervals or multi-seed variation for the main results. In the EgoVerse geometry analysis, 2,647 pairwise comparisons are formed from 378 samples; pairs sharing samples are statistically dependent, so the effective evidence is less than 2,647 independent comparisons. This makes it difficult to judge the robustness of the reported margins, particularly for the downstream gains and the action-geometry comparison.
Questions for authors
  1. Can the authors provide uncertainty estimates for the principal policy-success and action-geometry results? In particular, were the main policy results repeated across training seeds, and can the EgoVerse correlation comparison be accompanied by confidence intervals computed with resampling at the sample level rather than treating all pairs as independent?
  2. For the real-robot data-efficiency claims, how were the reported average gains and the approximate 1.5:1 Ego-to-robot demonstration ratio calculated across tasks and demonstration budgets? Please clarify the aggregation and interpolation procedure, and provide the per-setting trial counts or variability if available.
2 comments

Thank you for highlighting evaluation coverage and statistical robustness.

1. Expanded real-robot evaluation and training variability

Highlight. Across six additional real-robot tasks and three independent training seeds, ZEVA-EGOACTION achieves 73.89% mean success, compared with 65.28% for Zeva and 55.83% for π0.5\pi_{0.5}. These results extend the task coverage and provide multi-seed evidence of downstream gains under the fixed-camera setup.

ZEVA-EGOACTION achieves the highest mean success on five added tasks and ties for the highest on the sixth. We added six individually evaluated tasks from a laboratory dissolution workflow, extending the original three-task study. Beaker placement uses a different background and beaker from the original experiment. Water transfer uses a graduated cylinder rather than the original pouring vessel, making controlled pouring more challenging.

All methods use 100 Robot demonstrations per task, without additional task-matched Ego data in this comparison. The π0.5\pi_{0.5} and Zeva baselines initialize from π0.5\pi_{0.5} base; ZEVA-EGOACTION initializes from the Ego-mid-trained checkpoint. Each method is trained with three independent seeds, with 20 evaluation trials per task per seed. Methods share the same placement groups, varied within the training distribution; cameras and object instances are fixed, and retries are not allowed.

Taskπ0.5\pi_{0.5}ZevaZEVA-EGOACTION
Press Button33.33 ± 7.6438.33 ± 5.7758.33 ± 2.89
Place Beaker on Scale66.67 ± 7.6476.67 ± 5.7795.00 ± 5.00
Add Salt31.67 ± 10.4150.00 ± 8.6650.00 ± 10.00
Move Beaker to Table76.67 ± 10.4180.00 ± 10.0083.33 ± 12.58
Add Water65.00 ± 10.0073.33 ± 2.8978.33 ± 7.64
Mix Solution61.67 ± 2.8973.33 ± 11.5578.33 ± 16.07
Macro-average55.83 ± 0.8365.28 ± 5.0973.89 ± 1.92

Entries are success rates (%) as mean ± sample standard deviation across training seeds. For the macro-average, we first average tasks within each seed, then calculate statistics across seeds. ZEVA-EGOACTION improves the mean by 18.06 percentage points over π0.5\pi_{0.5} and 8.61 points over Zeva.

2. Dependence-aware action-geometry analysis

Highlight. ZEVA-EGOACTION's correlation advantage over both baselines remains positive across all three datasets: all paired 95% confidence intervals for the differences exclude zero. We bootstrap episodes rather than individual pairs, accounting for the dependence between comparisons sharing source episodes.

The new evaluation supports ZEVA-EGOACTION's advantage while accounting for shared-sample dependence. Because the original evaluation data could not be recovered, we conducted a new comparison on 200 EgoDex, AgiBot, EgoVerse episodes from distinct source sessions, using identical selected transitions and GT physical distances across encoders.

We resample episodes, not individual transition pairs. In each bootstrap replicate, all transitions from an episode share its resampling multiplicity, and each pair receives the product of its two endpoint episodes' multiplicities. Pairs sharing an episode are therefore reweighted together, rather than treated as independent observations. We use the same 2,000 resamples for all encoders to obtain episode-balanced Pearson estimates and paired confidence intervals for their differences.

EncoderEgoDexAgiBotEgoVerse
LAPA [1]0.181 [0.072, 0.292]0.194 [0.138, 0.252]0.199 [0.172, 0.225]
CLAP [2]0.124 [0.027, 0.226]0.287 [0.233, 0.335]0.149 [0.118, 0.179]
ZEVA-EGOACTION0.813 [0.748, 0.885]0.788 [0.746, 0.826]0.747 [0.717, 0.771]

Brackets denote 95% confidence intervals. EgoVerse is unseen during ZEVA-EGOACTION training. Its analysis compares equal-duration pairs across multiple horizons.

3. Data efficiency and the empirical demonstration ratio

Highlight. Ego-supported ZEVA-EGOACTION improves success at every tested low-Robot-data point. The original 1.5:1 ratio is an empirical dataset-level estimate. We collected additional Ego demonstrations in new real-robot settings to more comprehensively evaluate their potential to substitute for Robot demonstrations.

ZEVA-EGOACTION effectively incorporates Ego data during post-training, achieving comparable or better performance with fewer costly Robot demonstrations. We interpret the Ego-to-Robot ratio as a dataset-level empirical summary, dependent on task and operating point.

The supplementary sweep compares Robot-only π0.5\pi_{0.5} against Ego-initialized ZEVA-EGOACTION at Robot budgets of 10, 20, and 100 demonstrations. ZEVA-EGOACTION adds respectively 90, 80, and 0 independently collected task-matched Ego episodes, keeping its total downstream demonstration count at 100.

TaskRobot demosAdditional Ego demos in ZEVA-EGOACTIONπ0.5\pi_{0.5} success/trialsZEVA-EGOACTION success/trials
Press Button10900/201/20
Press Button20801/203/20
Press Button10009/2010/20
Place Beaker on Scale10902/208/20
Place Beaker on Scale20805/2011/20
Place Beaker on Scale100013/2018/20
Mix Solution10900/207/20
Mix Solution20803/209/20
Mix Solution100010/2016/20

This sweep uses one training seed (4) and 20 trials per point, separate from the three-seed experiment above.

We linearly interpolate the Robot-only curve within its measured range to estimate the Robot budget matching an Ego-supported result, then divide the added Ego count by the Robot demonstrations saved. For example, mixing solution achieves 35% success with 10 Robot and 90 Ego demonstrations; the interpolated Robot-only budget for the same success rate is approximately 65.7 demonstrations, giving an empirical ratio of 90/(65.7−10)≈1.62:190/(65.7-10)\approx1.62:1.

References

[1] S. Ye et al. Latent Action Pretraining from Videos. In International Conference on Learning Representations (ICLR), 2025.

[2] C. Zhang et al. CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos. arXiv preprint arXiv:2601.04061, 2026.

Reviewer_bGZpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper address the insufficiency of action labels. The authors propose a method called Zeva-EgoAction, which generates action pseudo-labels from unlabeled egocentric videos to mid-train a robot-pretrained π0.5 policy. Its Action-Centric Encoder learns action-aware representations from annotated human and robot data, then labels unseen videos using RGB frame pairs alone. Mid-training on egocentric data improves RoboTwin 2.0 performance, with several hours of human video providing gains comparable to robot demonstrations. Real-world experiments on an ARX X5 further demonstrate strong task success and substantial benefits from adding task-matched egocentric demonstrations when robot data is limited.

Critical strengths
  • provides analysis on action geometry
Critical weaknesses
  • The proposed approaches are highly similar to previous latent action methods [1,2,3], and some robot foundation models already exploit such action labels during training [4]

    • These papers [1,2,3] also derive pseudo labels (latent actions) for the game play or egocentric human videos to utilize them for better robot training
    • If their core contribution lies in the details of the labeling recipe, it is marginal
  • Though the authors provide ablation study on their components, they did not directly compared with previous action labeling approaches.

  • Comparisons with other human video mid-training recipes are not provided.

[1] D. Schmidt, M. Jiang, “Learning to Act without Actions,” ICLR 2024. [2] J. Bruce, M. Dennis, A. Edwards, J. Parker-Holder, et al., “Genie: Generative Interactive Environments,” ICML 2024. [3] S. Ye, J. Jang, B. Jeon, S. Joo, et al., “Latent Action Pretraining from Videos,” ICLR 2025. [4] J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, et al., “GR00T N1: An Open Foundation Model for Generalist Humanoid Robots,” arXiv 2025.

Questions for authors
  • The authors should clarify the uniqueness of the proposed method beyond previous action labeling approaches. Please refer weaknesses #1
  • Check the alignments of each parts in Fig.2
2 comments

Thank you for recognizing the action-geometry analysis and for highlighting the need to clarify our contribution and compare against prior action-labeling and human-video mid-training approaches.

1. Contribution beyond existing latent-action labeling

Highlight. Frequent camera motion in Ego video can contaminate prediction/reconstruction-based action labels, as used in LAPO [1], Genie [2], LAPA [3], and GR00T N1 [4], limiting supervision quality at scale. CLAP [5] pushes different human-video recordings apart even when their physical actions are similar. ZEVA-EGOACTION separates camera/environmental variation from manipulation and preserves graded physical-action similarity across recordings.

ZEVA-EGOACTION explicitly constrains camera and environmental variation to reduce their contamination of manipulation labels. Prediction or reconstruction objectives [1–4] must explain visual changes caused by both manipulation and head-camera motion, without necessarily separating their contributions. Our action-conditioned teacher learns environment tokens under an action-dependence penalty. The RGB-only student then distills these tokens, allowing reconstruction to account for environmental variation separately while physical-neighborhood supervision grounds the action tokens in manipulation. Once trained, the student labels additional Ego videos without action annotations.

ZEVA-EGOACTION uses physically similar actions across recordings as shared supervision instead of pushing them apart as negatives. In CLAP’s human-video branch [5], an augmented transition and its teacher representation form a positive pair, while other batch samples serve as negatives regardless of their physical-action similarity. Our neighborhood-matching objective instead uses physical-trajectory distances to distinguish similar, moderately different, and dissimilar actions. Repeated manipulations across people and scenes can therefore reinforce a shared action representation as the Ego corpus grows.

We compare ZEVA-EGOACTION with LAPA and CLAP at two levels: how faithfully their token representations preserve ground-truth physical-action geometry, and how using their labels for Ego mid-training affects downstream RoboTwin success rates under matched policy-training settings.

2. Direct comparison of action-labeling quality

Highlight. ZEVA-EGOACTION's environment–action separation outperforms LAPA's non-separated labeling and CLAP's alternative separation in preserving action geometry across all three datasets, reaching 0.747 correlation on unseen EgoVerse versus 0.199 from LAPA.

ZEVA-EGOACTION's token distances more closely track ground-truth physical-action differences than either baseline. We evaluate all three encoders on the same action-annotated transitions within each dataset, using episode-balanced Pearson correlation between physical-action and token distances. Token distances are computed from learned vectors, not numerical code indices; higher correlations indicate stronger preservation of physical-action geometry.

EncoderEgoDexAgiBotEgoVerse
LAPA0.181 [0.072, 0.292]0.194 [0.138, 0.252]0.199 [0.172, 0.225]
CLAP0.124 [0.027, 0.226]0.287 [0.233, 0.335]0.149 [0.118, 0.179]
ZEVA-EGOACTION0.813 [0.748, 0.885]0.788 [0.746, 0.826]0.747 [0.717, 0.771]

The uncertainty estimates account for transition pairs sharing episodes. Brackets denote 95% confidence intervals from 2,000 episode-level bootstrap replicates, paired across methods. The evaluation includes 200 EgoDex, AgiBot, EgoVerse episodes from distinct source sessions; individual pairs are not treated as independent observations.

3. End-to-end comparison with human-video mid-training baselines

Highlight. ZEVA-EGOACTION reaches 51.98% success across all 50 RoboTwin 2.0 Hard tasks, exceeding LAPA by 10.50 points, CLAP by 3.86 points, and the original π0.5\pi_{0.5} baseline by 5.98 points. The three mid-training variants use the same Ego corpus and policy-training settings, showing that the labeling method matters for downstream transfer.

ZEVA-EGOACTION achieves higher downstream success than both LAPA and CLAP under matched policy-training settings. We use LAPA, CLAP, and ZEVA-EGOACTION labels within the same π0.5\pi_{0.5} pipeline. All three variants share the released base checkpoint, Ego corpus, source mixture, policy-training budget, and RoboTwin post-training configuration.

Specifically, mid-training uses approximately 10.3K hours of Ego video, with EgoDex, Ego4D, EgoVerse, and Egocentric-10K sampled at a fixed 2:2:1:12:2:1:1 ratio. The shared optimization recipe uses a global batch size of 4,096 and AdamW, with a 2,000-step warmup to 5×10−55\times10^{-5} followed by cosine decay toward 5×10−65\times10^{-6}. Downstream training and evaluation follow the main comparison in Table 1: 50 clean demonstrations per task, followed by 100 randomized Hard evaluation episodes for each of all 50 RoboTwin 2.0 tasks.

MethodRoboTwin 2.0 Hard success rate (%)
π0.5\pi_{0.5}46.00
π0.5\pi_{0.5} + LAPA mid-training41.48
π0.5\pi_{0.5} + CLAP mid-training48.12
ZEVA-EGOACTION51.98

4. Alignment and notation in Figure 2

Thank you for pointing this out. We have standardized the notation and improved the alignment of the modules and connections in Figure 2, ensuring consistency with the definitions in the main text.


[1] Dominik Schmidt and Minqi Jiang. Learning to Act without Actions. International Conference on Learning Representations (ICLR), 2024.

[2] Jake Bruce, Michael D. Dennis, Ashley Edwards, et al. Genie: Generative Interactive Environments. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:4603–4623, 2024.

[3] Seonghyeon Ye, Joel Jang, Byeongguk Jeon, et al. Latent Action Pretraining from Videos. International Conference on Learning Representations (ICLR), 2025.

[4] Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, et al. GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv preprint arXiv:2503.14734, 2025.

[5] Chubin Zhang, Jianan Wang, Zifeng Gao, et al. CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos. arXiv preprint arXiv:2601.04061, 2026.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.