- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Zeva-EgoAction learns an Action-Centric Encoder from annotated human and robot trajectories. An action-conditioned teacher models environmental variation, and a visual-only student learns continuous action tokens through environment distillation and trajectory-neighborhood matching. The frozen encoder labels additional human videos for VLA mid-training. The paper reports RoboTwin improvement from 46.00% to 51.98% under clean-to-random evaluation and from 63.8% to 75.3% in a separate mixed-training scaling study.
Critical strengths
- Leveraging large-scale human egocentric videos to improve robotic manipulation is highly relevant to the robotics community. The optional ICCL extension further explores adaptation through accumulated interaction experience without updating policy parameters during deployment.
- The combination of offline learning from diverse human videos and online adaptation through robot interaction experience is well motivated. Human videos can provide general physical interaction priors, while the optional online extension can help address local execution errors, such as failed grasps or inaccurate placement.
- The reported results are promising. In particular, the method achieves strong clean-to-random generalization on RoboTwin 2.0, outperforming the compared methods.
Critical weaknesses
- Several methodological details are insufficiently defined, making the presentation difficult to follow. For example, L_{cam} is introduced in Eq. (8) without a detailed definition in the manuscript; the values or estimation procedure for s_g in Eq. (12) are unspecified; and the conditioning variable c in R^2(a | c, Z_{env})-R^2(a | c), on page 7 around lines 373–374, is not explicitly defined.
- The related work section does not sufficiently cover prior approaches that share this paper’s motivation of leveraging human egocentric data to train or improve robotic policies and VLAs [1–4]. Discussing these works would help clarify how the proposed approach differs in its use of human data, supervision construction, and transfer to robot control.
- This paper proposes a trajectory-neighborhood KL matching method for continuous action tokens as its core contribution; however, the incremental contribution of this method compared to existing physics-based representation learning methods has not yet been fully substantiated. RS-CL [5] has already established a soft contrast supervision based on the distances between a robot’s proprioceptive states. Meanwhile, CLAP [6] aligns visual transitions with physical action representations through cross-modal contrastive learning based on a robot action dictionary.
- The physical distances used for training and fidelity evaluation require further clarification. Equation (12) appears to apply mean squared error (MSE) to each action component group (including quaternion-based rotations). In the absence of a consistent quaternion sign convention or a sign-invariant distance metric, equivalent rotations represented by and would result in non-zero distances. This could distort the evaluation of neighborhood supervision and representation fidelity. [1] In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Data,arXiv:2511.15704 [2] H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation,AAAI 2026 [3] PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence,arXiv:2512.16793 [4] Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments,arXiv:2606.32009 [5] Contrastive Representation Regularization for Vision-Language-Action Models [6] CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos
Questions for authors
- My main concern is the incremental methodological contribution. Could the authors clarify the distinctions and advantages of the proposed approach relative to RS-CL and CLAP, and provide controlled comparisons to support these claims?
- How is quaternion sign ambiguity handled in the physical distance defined in Eq. (12)? If no sign canonicalization or sign-invariant distance is used, how does this affect neighborhood supervision, fidelity evaluation, and downstream policy performance?
3 comments
3. Definitions of , , and
Highlight. We have clarified the implementation: supervises normalized camera-motion prediction, provides fixed calibration scales for each action group, and conditions the leakage analysis on camera motion.
- : masked MSE for six-dimensional camera-motion prediction—translation and rotation-vector increments expressed in the chunk's first-camera frame, normalized using source-specific statistics. We average over six coordinates, valid time steps within each sample, and then samples. Samples without camera supervision contribute no camera-loss term.
- : fixed group-MSE normalization scales for translation, rotation, and gripper: , , and , respectively. They were estimated as group-MSE medians from equal-duration EgoDex/AgiBot calibration pairs and remain frozen during evaluation.
- : the camera-motion descriptor comprising start-to-end translation and rotation vector. The conditional leakage statistic measures the additional action predictability provided by beyond that explained by camera motion alone.
We have added these definitions alongside the corresponding equations.
4. Coverage of related human-video approaches
Highlight. H-RDT [4] and In-N-On [5] rely on existing pose/motion annotations; Human-as-Humanoid [6] relies on pose recovery, whose errors can directly propagate into action-expert training; PhysBrain [7] provides semantic rather than physical-action labels. ZEVA-EGOACTION supplies physical priors to the VLM through action tokens rather than precise control targets, reducing reliance on high-precision pose recovery and enabling RGB-only labeling after encoder training.
We have expanded the related-work discussion around the following distinctions.
ZEVA-EGOACTION extends action supervision to Ego videos without existing motion annotations. H-RDT [4] and In-N-On [5] rely on existing pose/motion annotations for policy learning and human-to-robot transfer. Our encoder learns from annotated human and Robot trajectories, then labels additional videos from RGB transitions alone.
ZEVA-EGOACTION supplies physical priors to the VLM without requiring high-precision pose recovery from each human video. Human-as-Humanoid [6] reconstructs and retargets human motion into robot-action targets for flow-matching training, so recovery errors can directly propagate into action-expert training. Our action tokens instead support VLM mid-training, while Robot demonstrations provide precise control supervision.
ZEVA-EGOACTION provides physical-action supervision beyond semantic descriptions. PhysBrain [7] derives structured language descriptions from egocentric video for embodied understanding and planning. Our encoder instead uses physical-trajectory relationships to shape action-token geometry, allowing RGB video to supply action-level physical priors.
References
[1] T. Kim et al. Contrastive Representation Regularization for Vision-Language-Action Models. In International Conference on Machine Learning (ICML), 2026.
[2] C. Zhang et al. CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos. arXiv preprint arXiv:2601.04061, 2026.
[3] S. Ye et al. Latent Action Pretraining from Videos. In International Conference on Learning Representations (ICLR), 2025.
[4] H. Bi et al. H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation. arXiv preprint arXiv:2507.23523, 2025.
[5] X. Cai et al. In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Data. arXiv preprint arXiv:2511.15704, 2025.
[6] X. Lin et al. Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments. arXiv preprint arXiv:2606.32009, 2026.
[7] X. Lin et al. PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligence. arXiv preprint arXiv:2512.16793, 2025.
Response to Author
I am very grateful to the author for the detailed response, which has largely addressed my concerns. Therefore, I maintain my positive view of this paper.
Thank you for recognizing the relevance of Ego mid-training and the clean-to-random results.
1. Distinction from RS-CL and CLAP, and comparative evidence
ZEVA-EGOACTION extends physical supervision to action-unannotated Ego video through a separately trained labeling encoder. RS-CL [1] uses existing proprioceptive-state annotations recorded in Robot demonstration datasets, rather than recovering states from RGB video. Distances between these annotated states provide soft supervision for policy representations. It studies visual robustness, including background changes, but does not provide an egocentric action-labeling pipeline that explicitly separates head-camera motion from manipulation. Our encoder learns from action-annotated human and robot data, then labels additional RGB transitions without action annotations.
ZEVA-EGOACTION preserves similar actions across recordings rather than treating them as negatives. CLAP [2] grounds visual transitions through a robot action dictionary and contrastive alignment, but its instance-based human-video contrast can treat different recordings of similar actions as negatives. Our physical neighborhoods instead provide graded relationships between annotated trajectories.
ZEVA-EGOACTION achieves higher downstream success than LAPA and CLAP under matched policy-training settings. The comparison uses the same π0.5 base initializer, Ego corpus, source mixture, policy-training budget, and RoboTwin post-training configuration. All methods are evaluated under Table 1's clean-to-random protocol over all 50 tasks.
ZEVA-EGOACTION also preserves physical-action geometry better than both baselines across all three datasets. A complementary encoder comparison on common GT-annotated transitions yields:
Entries are episode-balanced Pearson correlations between physical-action and token distances, includes 200 EgoDex, AgiBot, EgoVerse episodes from distinct source sessions; respectively. EgoVerse is unseen during ZEVA-EGOACTION training. These results support ZEVA-EGOACTION as a labeling method; the released encoders have different pretraining histories, so this is not an isolated loss ablation. We have not conducted a direct RS-CL comparison and do not claim experimental superiority over it.
2. Quaternion ambiguity and the implemented rotation distance
Eq. (12) operates on canonicalized, chunk-start-relative rotation vectors, not raw quaternions. The manuscript omitted this conversion. The implementation composes rotations, aligns quaternion signs before interpolation, canonicalizes the scalar sign, and converts to rotation vectors before computing group MSE. This addresses the ordinary q/−q ambiguity, although the principal rotation-vector representation still has a boundary at π and is not an SO(3) geodesic metric.
The sampled rotations remain away from the principal-vector boundary, and most pairwise distances are insensitive to the metric replacement. The maximum relative angles were 154.7° on EgoDex, 69.1° on AgiBot, and 144.1° on EgoVerse, with no sampled point reaching 170°. Replacing the rotation term with squared SO(3) geodesic distance, divided by three to match the original per-coordinate MSE convention, while retaining the other groups and scales, changes the total physical distance by less than 1% for at least 99% of evaluated pairs in each dataset. A small number of outliers remain more sensitive.