acceptodds
Withdrawn from ICLR 2027

Learning Beyond Bodies: Joint Learning and Transfer for Robust Humanoid Motion Tracking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Conventional humanoid motion tracking approaches often condition control policies on retargeted robot motion, potentially carrying retargeting artifacts into motion execution. Beyond reference quality, a central question is how to share motion knowledge across robots and reuse it when adapting to a new embodiment with limited motion data. We address these challenges by separating shared human-motion conditioning from embodiment-specific control. Our motion encoder takes body-centric human keypoints as input, avoiding direct dependence on retargeted joint references while retaining retargeted trajectories for training supervision. On our primary motion benchmark, this input yields average tracking success and position accuracy comparable to robot joint-reference conditioning. Controlled single-embodiment comparisons additionally show smoother execution on this benchmark and selected cases with retargeting-induced joint discontinuities. This common input enables joint reinforcement learning of a shared encoder across robots, while dedicated control heads combine its features with proprioception and reference orientation. Across heterogeneous simulated humanoids, joint training improves position tracking on the same benchmark over corresponding single-robot training. This separation also supports adaptation: an embodiment excluded from pretraining reuses the encoder and learns a new control head. In a low-data setting, freezing the transferred encoder improves position tracking on an external motion dataset relative to training from scratch. These results demonstrate that our framework supports more robust motion execution, effective joint learning across humanoids, and the transfer of motion knowledge to new embodiments.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Humanoid Motion Tracking, Cross-Embodiment Learning, Motion Retargeting, Transfer Learning.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DJcNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a common human-motion reference interface for heterogeneous humanoid trackers. A shared MLP encodes two frames of pelvis-centered human keypoints, while robot-specific policy heads combine this representation with proprioception and reference orientation to produce joint-position targets. Joint PPO training shares the reference encoder across robots; adaptation trains a new target head with the encoder frozen or fine-tuned. Importantly, robot-specific retargeting remains necessary for rewards, initialization, and reference orientation. Evaluation includes seven simulated humanoids, BONES-SEED and external AMASS motions, four-robot joint training, transfer to two excluded embodiments, and qualitative deployment on G1 and K1. Human-keypoint conditioning gives comparable average primary-benchmark position tracking to robot-reference conditioning. Joint training yields modest position-error improvements, and frozen transfer improves success rates under restricted target-motion data.

Critical strengths
  1. A practical separation of shared motion description from embodiment-specific feedback control. The same fixed-dimensional human-keypoint interface supports native robot state/action spaces without padding or joint-layout matching. The paper explicitly retains retargeted supervision rather than overstating the approach as retargeting-free. This makes the proposed reuse boundary technically clear.
  2. Transfer is evaluated on excluded bodies and an external motion dataset. Table 2 tests G1 and X2 after pretraining on four different robots, under both uniformly reduced data and locomotion-only target data. Frozen transfer improves AMASS success by 2.15-4.55 percentage points in Uniform-10K and 1.60-3.40 points in Locomotion-10K.
Critical weaknesses
  1. The paper lacks direct comparisons with the most relevant cross-embodiment motion tracking and transfer methods, making it difficult to assess its contribution beyond existing work.

  2. The real-world evidence in the provided materials consists primarily of static images, with no verifiable videos of continuous execution or quantitative real-robot evaluation results.

  3. The added value of cross-embodiment representation learning is not adequately isolated. Cross4 improves BONES-SEED MPKPE by only 3.3-4.6%, with mixed AMASS outcomes, while aggregate interactions and compute differ from Solo. The transfer experiment also compares a broadly pretrained encoder with a target trained on only 10K motions; it does not include an equally pretrained single-source encoder or a frozen random-feature control.

Questions for authors

See weaknesses.

Reviewer_2ooLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework that uses a common human-motion reference to share motion knowledge across humanoid robots and to adapt to new robots with limited motion data. The key idea is to separate motion description from physical execution. A shared encoder reads body-centric human keypoints and is trained jointly across robots, while robot-specific control heads handle execution. To transfer to a new robot, the encoder is frozen and a new control head is trained. In simulation, the approach tracks about as well as conditioning on retargeted robot joints, lowers source-position error near retargeting jumps on a selected set of G1 clips, and gives higher success than training from scratch on new robots.

Critical strengths
  • The goal is clear and the architecture design is simple. Motion description is separated from robot-specific execution, and each robot keeps its native observation and action spaces without joint padding.
  • The evaluation is broad. It covers seven simulated humanoids, a held-out external dataset (AMASS), and qualitative deployment on two physical robots.
  • The paper states its limits openly. It notes the unmatched compute in joint training, the look-ahead and noise confound in the reference comparison, and that the discontinuity analysis uses a selected set of clips. The discontinuity analysis also reports a paired bootstrap interval.
Critical weaknesses

W1. The novelty over prior cross-embodiment work is unclear, and no external baseline is compared.

  • Using human motion as a shared reference, with a shared encoder and robot-specific execution, is already common. X-WBC trains human-centered command tokens jointly across robots, with robot-specific modules for execution. Yan & Lee (2026) learn a latent space shared by humans and several humanoids, and add a new robot by training only robot-specific layers. Their setting is kinematic rather than physics-based tracking, but the structure is the same. SONIC maps human motion directly to control commands. EAGLE (2026) and VENOM (2026) also train one controller across multiple humanoids. Yan & Lee, EAGLE, and VENOM are not cited.
  • The paper's stated difference from X-WBC is that it uses no alignment loss (Sec. 2). This is a direct result of using a single input type, not a new technique. X-WBC aligns several command sources (human motion, robot motion, sparse VR) in one space. Here every robot receives the same human keypoints, so there is nothing to align. The cost is that the policy can no longer take robot references as input. The paper does not test whether this trade-off is worth it.
  • All experiments are internal comparisons (Human Keypoints vs. Robot Motion, Cross4 vs. Solo, frozen vs. scratch). There is no comparison with an X-WBC-style aligned encoder or with Any2Any-style transfer, so the benefit over existing approaches is unknown.

W2. The joint-training and transfer gains are not shown to come from other bodies.

  • Joint training. Each robot runs 4,096 environments. So at the same 150k iterations, the shared encoder in Cross4 learns from four times as many rollouts as in Solo. The authors acknowledge this (Sec. 5.4), but it means contribution 2 is not cleanly tested. The gains are also small: BONES-SEED success rises by 0.33 to 1.13 points, and on AMASS success drops slightly for T1 (−0.05) and T800 (−0.60) (Fig. 5, Table 6).
  • Transfer. The frozen encoder was pretrained on 100k motions, while the scratch baseline sees only the 10k target motions. The frozen encoder has therefore seen ten times more motion, including categories such as dance that the pretraining pool covers but Locomotion-10K excludes (Sec. 5.5). The authors themselves describe Uniform-10K as testing the effect of pretraining on more motions.
  • The encoder input is the same for every robot, so it is unclear whether learning it needs other bodies at all. The missing control is an encoder trained on the same 100k motions with a single robot, or with the target robot itself. If it performs similarly, the gain comes from motion coverage rather than from other bodies, and the "Beyond Bodies" claim does not hold.
  • No seeds or variance are reported. Several gaps are within one point, and the AMASS transfer curves move by several points between checkpoints (Fig. 6c, 6g).

W3. The claims about what is shared go beyond what is actually shared or measured.

  • The shared part is a small MLP that maps a 132-D keypoint vector to a 256-D feature (Table 4). Balance, contact, and feedback control all live in the robot-specific head. In transfer, this head is trained from scratch for the same 150k iterations as the scratch baseline (Sec. 5.5).
  • So what transfers is mainly how human pose is summarized, while the control itself is learned again. Yet the conclusion claims that controllers accumulate motor skill rather than relearn it (Sec. 7).
  • The paper gives the encoder the role of motion description (Sec. 4.2), but does not define what this description should contain, such as body-independent motion meaning. No experiment checks what the encoder actually learns, so this role is claimed but not shown.

W4. The interface still depends on per-robot retargeting, global motion is missing from both input and metric, and key details are unclear.

  • The actor observes a relative reference-anchor orientation taken from the retargeted trajectory (Sec. 4.2, Fig. 2d). This suggests that retargeting is still needed at deployment, which conflicts with the claim that new motions need no per-robot re-specification (Sec. 7). The paper does not say what the real-robot runs used for this input (Sec. 5.6).
  • Each frame is expressed in its own pelvis frame (Eq. 1), and the observation includes anchor orientation but not anchor position. So how far and in which direction the body should move is not in the input. MPKPE also aligns the horizontal anchor position and heading before computing error (Sec. 5.1), so errors in walking speed or direction are not counted. This applies to both inputs, so the relative comparison is fair, but the claim of accurate tracking leaves out the global trajectory.
  • Figures 1 and 3 are conceptual, as their captions state. Some details needed to understand the setup are missing, such as what each critic observes.

References

  • Y. Yan and D. Lee. Learning a Unified Latent Space for Cross-Embodiment Robot Control. IEEE Robotics & Automation Magazine, 2026. arXiv:2601.15419.
  • Q. Peng, Y. Lin, Y. Xue, J. Pang, and W. Zhang. Embodiment-Aware Generalist Specialist Distillation for Unified Humanoid Whole-Body Control (EAGLE). arXiv:2602.02960, 2026.
  • S. Padmanabhan, K. Miyazawa, and T. Horii. VENOM: Versatile Embodied Network for Omni-bodied Motion tracking. arXiv:2606.16696, 2026.
Questions for authors
  1. How does the method compare with an X-WBC-style aligned encoder and with Any2Any-style transfer, using the same robots and data? How does it differ from prior cross-embodiment works?
  2. How does transfer perform with an encoder trained on the same 100k motions using only one robot, or only the target robot?
  3. Can you compare Cross4 and Solo under matched interaction, for example by giving Solo four times the environments or iterations? How many seeds were used, and what is the variance?
  4. How does transfer perform with a frozen, randomly initialized encoder, or with the 132-D keypoints fed directly to the head? This would show whether the pretrained encoder adds anything.
  5. At deployment, where does the reference-anchor orientation come from? Is retargeting still run for each robot, including in the real-robot experiments?
  6. The reward tracks anchor position (App. A.2), but neither the reference nor the observation contains it. How does the policy know the target walking speed and direction? Can you report root position or velocity error without horizontal alignment?
  7. What does each critic observe? Does it receive privileged information, such as the global anchor position or velocity, that the actor lacks? This would explain how the policy learns the anchor-position reward without seeing the anchor position.
Reviewer_MCaZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether body-centric human keypoints can serve as a shared motion-reference interface for humanoid motion tracking across different robot embodiments. It proposes a shared human-motion encoder with robot-specific control heads, jointly trained across multiple robots using reinforcement learning. For adaptation to unseen embodiments, the pretrained encoder is frozen while a new robot-specific control head is trained. Experiments compare human-keypoint and robot-motion conditioning, evaluate cross-embodiment joint learning and transfer under limited motion data, and demonstrate motion tracking on two physical humanoids.

Critical strengths
  1. Decoupling motion representation and embodiment-specific execution. Sections 4.1–4.4 introduce a common 132-dimensional human-keypoint reference, a shared motion encoder, and robot-specific control heads operating in their native observation and action spaces, without requiring joint padding or explicit joint correspondence across robots. This provides a accessible design for cross-embodiment transfer by separating reusable motion-reference features from robot-specific feedback control.

  2. Comprehensive empirical evaluation. Table 1 demonstrates broadly comparable average tracking success and positional accuracy to robot-motion conditioning across seven humanoids. Table 2 evaluates frozen-encoder transfer, fine-tuning, and training from scratch on previously unseen embodiments, while Figure 7 demonstrates the same human-keypoint reference driving two physical humanoids. The paper also appropriately limits its conclusions about retargeting discontinuities to a screened G1 cohort and acknowledges potential confounding factors, making the interpretation of these results more credible.

Critical weaknesses
  1. Limited methodological novelty and insufficient comparison with closely related work. The paper proposes a shared human-keypoint encoder with embodiment-specific control heads to enable joint learning and transfer across humanoids. However, this design is conceptually close to existing keypoint-guided control and cross-embodiment learning approaches, such as AnyBody, TRACE, X-WBC, and Any2Any. The paper does not sufficiently distinguish its methodological contributions from these approaches or provide direct comparisons demonstrating its advantages. The existing baselines primarily evaluate variations of the proposed framework rather than its competitiveness with prior methods. A clearer technical distinction and comparisons under matched experimental settings would help establish the novelty and significance of the contribution.

  2. The main experimental comparisons do not adequately isolate the benefits of the proposed framework. Several important comparisons contain confounding factors that weaken the central claims. First, the Cross4 experiments use more aggregate training experience than the single-robot baselines, making it difficult to attribute the improvements specifically to cross-embodiment knowledge sharing. Second, the transfer experiments compare an encoder pretrained on 100,000 motions against scratch training on only 10,000 motions. Without a single-source-robot pretraining baseline using the same motion data and training budget, it remains unclear whether the benefits arise from cross-embodiment learning or simply from additional motion-representation pretraining. Third, the comparison between Human Keypoints and Robot Motion also introduces future look-ahead and input-noise augmentation, preventing the benefits from being attributed solely to the proposed reference representation. More carefully controlled comparisons and ablations are needed to substantiate the paper's three primary contributions.

  3. Limited statistical evidence and generalization evaluation to support the reported improvements. The main results in Tables 1, 2, and 6 lack independent training seeds, uncertainty estimates, or statistical significance analysis, despite several improvements being relatively small. For example, Table 2 reports success-rate gains of only 0.28 and 0.92 percentage points for G1 and X2 under Uniform-10K training. Furthermore, the transfer experiments cover only two unseen humanoid embodiments, while the real-world demonstrations provide qualitative evidence of tracking feasibility rather than quantitative validation of cross-embodiment transfer. Although human-keypoint conditioning achieves comparable positional accuracy, Table 1 also shows consistently larger average orientation and joint tracking errors than robot-motion conditioning. Together, these limitations make it difficult to assess the robustness, generality, and practical significance of the reported improvements.

Questions for authors
  1. Statistical reliability of the reported improvements: For Tables 1, 2, and 6, how many independent training runs were performed per condition, and can the authors report the mean and variation across seeds? In particular, do the relatively small Uniform-10K frozen-transfer gains in Table 2 (0.28 and 0.92 percentage points for G1 and X2) remain consistently positive across independent runs?

  2. Comparison with existing cross-embodiment transfer methods: How does the proposed frozen-encoder transfer compare with existing cross-embodiment adaptation approaches, such as Any2Any or TRACE, that transfer a controller or latent policy rather than only a reference encoder? Given the conceptual overlap, what are the specific methodological advantages of transferring the shared reference encoder? A comparison under matched source-target configurations and training budgets, or a clearer explanation of the differences in experimental settings, would substantially affect the assessment of the transfer contribution.

  3. Benefits of joint learning under matched training budgets: Section 5.4 acknowledges unmatched aggregate compute and interaction budgets between Cross4 and Solo. Can the authors provide a comparison with matched total training interactions and encoder optimization budgets? In particular, do the reported Cross4 improvements persist when controlling for the additional training experience accumulated across multiple robots? This would help clarify whether the improvements arise from cross-embodiment knowledge sharing rather than increased training experience.

  4. Contribution of cross-embodiment pretraining to transfer: In Section 5.5, the pretrained encoder has been trained on 100,000 motions across four source robots, whereas the target policies are trained using only 10,000 motions. How much of the improvement comes specifically from cross-embodiment pretraining rather than general motion-representation pretraining on a larger dataset? Could the authors compare against an encoder pretrained on a single source robot using the same 100,000 motions and a matched training budget, then transferred to the same target embodiments? This distinction is important for validating the claimed benefit of shared learning across embodiments.

  5. Isolating the effect of human-keypoint conditioning: Section 5.2 compares Human Keypoints against Robot Motion conditioning, but the former also incorporates future look-ahead and input-noise augmentation. Can the authors clarify whether these design choices were independently evaluated? In particular, how does human-keypoint conditioning compare against robot-motion conditioning when both use equivalent temporal information and augmentation? Such a comparison would help establish whether the improvements in tracking and robustness to retargeting discontinuities originate from the proposed reference representation itself.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.