acceptodds
Under review as a conference paper at ICLR 2027

Bridging Human and Robot Experiences via a Shared Hierarchical Semantic Latent Space

Abstract

Large-scale egocentric human videos provide rich and scalable manipulation experience for robot policy learning, yet substantial embodiment gaps in visual appearance, kinematic structure, and action spaces hinder effective knowledge transfer. Existing methods often decouple cross-embodiment knowledge alignment from robot policy learning, making it difficult for human manipulation knowledge to directly and fully contribute to robot action learning. To address this issue, we propose Nexus-VLA, a hybrid policy learning framework that enables cross-embodiment knowledge transfer through end-to-end human-robot joint training, allowing human manipulation experience to directly contribute to robot policy learning. We first introduce a hierarchical cross-embodiment semantic alignment paradigm that progressively establishes shared manipulation semantics from task intent to fine-grained interactions, enabling more complete use of the manipulation knowledge contained in human videos. Building on this hierarchy, we design a shared manipulation latent representation that jointly encodes multi-level shared knowledge into an internal representation directly used for continuous action generation, improving policy learning efficiency while effectively leveraging human knowledge. Across RoboCasa-GR1, OOD settings, and real-world robot experiments, Nexus-VLA achieves an 80.5% task success rate on RoboCasa-GR1 and demonstrates strong advantages in human-data utilization, learning efficiency, and cross-scenario generalization.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FJdMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the high cost of collecting robot demonstrations and the difficulty of leveraging human videos without robot action labels, the paper proposes Nexus-VLA, an end-to-end framework for joint training on unpaired human and robot demonstrations. Four types of semantic supervision—intent, spatial, motion, and interaction—shape a shared semantic latent within a Qwen2.5-VL-3B-Instruct backbone. A flow-matching action model is jointly trained using human wrist-motion supervision and robot wrist-motion and joint-action supervision. At inference time, the policy generates action chunks conditioned on visual-language features, the semantic latent, and the robot’s proprioceptive state, and executes the robot joint-action component without requiring the offline annotation tools or the four semantic prediction heads.

Critical strengths

The paper addresses an important problem of broad interest: robot demonstrations are costly to collect, while human manipulation videos are abundant but typically lack robot action labels(lines 33-42).

The finding(Table 3) that a unified latent representation outperforms separate latents for the four semantic levels is interesting, suggesting that part of the performance gain comes from the unified representation itself.

Critical weaknesses

The idea of bridging human and robot experiences through four types of information—intent, spatial, motion, and interaction—is reasonable. However, these semantic factors have already been explored to varying degrees in prior cross-embodiment learning research. Therefore, the novelty of this work primarily depends on how these factors are jointly organized, rather than on the four types of information themselves.

The paper adopts Qwen2.5-VL-3B-Instruct as its vision-language backbone but does not sufficiently justify this choice. Although the newer, similarly sized Qwen3-VL-4B-Instruct had been publicly available for a considerable period before submission, the proposed method is evaluated on only a single backbone. Controlled experiments with a newer backbone would help determine whether the reported gains depend on the specific backbone and whether they persist with a stronger foundation model.

Questions for authors

Why does the paper use Qwen2.5-VL-3B-Instruct as its vision-language backbone rather than Qwen3-VL-4B-Instruct?

Reviewer_sfz8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Nexus-VLA, a framework for transferring manipulation knowledge from unpaired egocentric human videos to robot policies. The main idea is to construct a shared hierarchical semantic latent representation encompassing four levels of manipulation knowledge: Intent, Spatial, Motion, and Interaction.

The proposed framework uses frozen structured manipulation experts to generate semantic supervision for human and robot demonstrations. A shared vision-language backbone aggregates these signals into a unified semantic latent, which is subsequently used to condition action generation. The authors also introduce a dual-space action representation, combining shared wrist/end-effector motion with robot-specific joint actions, to enable end-to-end co-training using human and robot demonstrations.

Experiments are conducted on RoboCasa-GR1, six simulated OOD settings, and real-world manipulation tasks. Nexus-VLA achieves an 80.5% average success rate on RoboCasa-GR1, outperforming the strongest reported competing method by 7.7 percentage points. The results also demonstrate improvements in generalization and learning efficiency.

Critical strengths
  1. Strong empirical performance across diverse evaluation settings. The paper provides extensive evaluations covering 24 simulated manipulation tasks, multiple OOD scenarios, and real-world robot experiments. The improvements over existing methods are substantial, particularly under distribution shifts. The component ablations also show consistent benefits from incorporating semantic supervision and unified latent representations. These results suggest that the framework is practically useful and that its improvements are not restricted to a single experimental setting.

  2. A practical framework for integrating unpaired human and robot experiences. The proposed dual-space action representation provides a reasonable way to jointly utilize human wrist motion and robot joint-control demonstrations without requiring paired human–robot trajectories. Moreover, the structured manipulation experts are only needed during training, avoiding explicit semantic prediction at inference time. This design is appealing from a practical policy-learning perspective.

Critical weaknesses
  1. The novelty and necessity of the proposed hierarchical semantic alignment are insufficiently established.

The main conceptual contribution is the hierarchical semantic latent representation. However, according to Section 3.1, the four semantic levels are supervised through separate prediction heads operating on the same latent token, with their losses combined through a weighted sum.

It is unclear what explicitly enforces hierarchical relationships or consistency across these semantic levels beyond shared representation learning. Similarly, the paper does not introduce an explicit cross-embodiment alignment objective, making it difficult to determine whether the learned representation is genuinely embodiment-invariant.

Consequently, the proposed mechanism appears closely related to multi-task semantic supervision with a shared latent representation rather than a distinctly hierarchical alignment mechanism. Although the unified-versus-independent latent ablation is useful, additional analysis would be needed to establish the claimed hierarchical structure and distinguish the contribution from conventional shared-representation learning.

This issue is important because hierarchical cross-embodiment alignment is presented as the primary methodological novelty of the paper.

  1. The experimental evidence does not sufficiently isolate the contribution of human experience from semantic supervision and additional annotation resources.

Table 3 shows that hierarchical alignment and unified latent modeling improve robot-only success from 72.42% to 78.25%. Adding human–robot co-training further improves performance to 80.16%, corresponding to an additional gain of only 1.91 percentage points.

These results suggest that a substantial portion of the improvement comes from structured semantic supervision rather than human experience itself. While the OOD experiments provide encouraging evidence of cross-embodiment transfer, the paper does not sufficiently disentangle these effects.

Furthermore, the semantic supervision relies on several pretrained experts, including Qwen3-VL, SAM 3, CoTracker3, and HaMeR. The associated supervision-generation cost and sensitivity to annotation quality are not quantitatively evaluated. The construction of the shared human wrist/robot end-effector action space also deserves more explanation, particularly regarding coordinate alignment and pose-estimation uncertainty.

These omissions make it difficult to fully assess the practical efficiency and robustness of the proposed knowledge-transfer mechanism.

Questions for authors
  1. What explicitly makes the learned semantic representation hierarchical? Beyond jointly supervising four prediction heads from the same latent token, are there mechanisms that enforce dependencies or consistency among Intent, Spatial, Motion, and Interaction? Could the authors provide quantitative evidence of such hierarchical relationships or compare against an equally supervised flat multi-task representation?

  2. How is the shared wrist/end-effector action space constructed across embodiments? Human wrist poses are estimated from egocentric videos, whereas robot end-effector poses are obtained from robot states. How are these representations aligned in coordinate systems, scale, and orientation? How sensitive is policy performance to inaccuracies in human pose estimation?

  3. Can the authors provide stronger evidence isolating the benefits of human demonstrations? Given that human–robot co-training adds only 1.91 percentage points over the robot-only unified-latent variant in Table 3, could the authors report additional experiments controlling for training compute, robot-data exposure, and semantic annotation resources? Repeated-run statistics and confidence intervals would also help determine the reliability of the improvements.

Reviewer_2w9VOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Nexus-VLA trains one Qwen2.5-VL policy on unpaired EgoDex human videos and RoboCasa-GR1 robot demonstrations. A single semantic token is appended to the vision-language sequence, and four lightweight heads read that token to predict a local subtask, 2D object and target boxes, future wrist traces, and contact points. The four heads and the external experts that build their labels are used only during training. Universal performance boost is seen across simulation and real-robot benchmarks.

Critical strengths

On the standard GR1 split, Nexus-VLA is ahead of every method listed in Table 1. The average is 80.5%, which is 7.7 points above ACE-Ego-0 at 72.8%. Table 8 gives it the best or a tied-best score on 18 of the 24 tasks, including articulated Close tasks and object-transfer groups. The margin is therefore spread across the suite rather than produced by one task family.

Critical weaknesses

Human video accounts for a small slice of the reported GR1 gain, while the paper presents that gain as cross-embodiment transfer. Table 3 trains the hierarchy on robot data alone. The robot-only baseline is 72.42%. Hierarchical alignment reaches 75.67%. Independent latents reach 76.92%. One unified latent reaches 78.25%. Adding EgoDex co-training on top of that unified latent reaches 80.16%, a further 1.91 points. Human–robot co-training without the hierarchy is 71.75% at the checkpoint in the table. The text says this run is still climbing and reaches 76.32% at 200K steps, above a robot-only peak of 72.92%. Even that later number is a 3.4-point gain from mixing human data, against a 5.8-point gain from the unified latent with no human data. Table 1’s headline 80.5% also does not match Table 3’s 80.16%. Table 2’s 20.7-point OOD gap, including the 28.6-point Ego-Object gap, is measured only against the internal baseline. ACE-Ego-0 and the other human-video methods in Table 1 are absent from those shifts. The experiments therefore show that multi-level auxiliary supervision helps a robot-only policy. They do not show that human video is what carries the 80.5% result.

The auxiliary targets themselves are prior ideas, so packing them into one token is an implementation choice rather than a new supervision scheme. The Motion Projector predicts future 2D wrist traces over 16 frames (Eq. 17, Appendix A.1). HAMSTER already trains a VLM to predict a coarse 2D end-effector path, including where the gripper opens and closes, and builds those paths from point tracking on human video and from robot proprioception [1]. TraceVLA feeds a multi-point visual trace of the end-effector trajectory into a VLA so that action prediction is conditioned on that motion [2]. The Grounding Projector predicts 2D boxes of the manipulated object and the target (Eq. 15). ECoT trains a VLA to emit object names and bounding-box coordinates before the action [3]. RoboPoint trains a VLM to point at image-space affordances specified by language [4]. The Subtask Projector autoregressively predicts the current local subtask (Eq. 14). ECoT already predicts a high-level plan, the current subtask, and a low-level move command in the same chain as the boxes and the gripper position [3]. Emma-X then adds the look-ahead piece that ECoT lacked: a grounded subtask label, the gripper’s future 2D position, and the 3D movement required to reach it [5]. What Nexus-VLA adds is a contact head and the decision to read all four targets from one semantic token. Table 3 shows that this shared token beats four independent latents by 1.33 points on robot data. It does not remove the trace, the boxes, or the subtask one at a time, and it does not compare the token with the HAMSTER path or the ECoT and Emma-X chains under the same backbone.

[1] Li, Yi, et al. “HAMSTER: Hierarchical Action Models for Open-World Robot Manipulation.” International Conference on Learning Representations, 2025.

[2] Zheng, Ruijie, et al. “TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies.” International Conference on Learning Representations, 2025.

[3] Zawalski, Michał, et al. “Robotic Control via Embodied Chain-of-Thought Reasoning.” Conference on Robot Learning, 2024.

[4] Yuan, Wentao, et al. “RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics.” Conference on Robot Learning, 2024.

[5] Sun, Qi, et al. “Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning.” arXiv preprint arXiv:2412.11974 (2024).

Questions for authors

Table 1 reports 80.5% and Table 3 reports 80.16% for the full model, while the robot-only unified latent already reaches 78.25%. Which of these numbers is the human-transfer result, and which existing table or figure separates EgoDex from that robot-only latent on the Ego-Object and Fully Unseen shifts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.