acceptodds
Withdrawn from ICLR 2027

SMR: Learning Streaming Motion Retargeting for Humanoids from Human Semantic

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retargeting human motion to humanoids requires preserving motion semantics while accounting for the robot's physical constraints. Kinematic pipelines prescribe a robot trajectory through geometric matching, without establishing whether it can be executed under the robot's dynamics. Prior physics-supervised neural methods train bidirectional full-sequence models on physics-refined kinematic trajectories, remaining tied to prescribed kinematic solutions and requiring noncausal context. We propose SMR, a streaming neural motion retargeter learned from semantic-conditioned physics experts. We convert human motion into robot-aware task-space targets for body landmark positions, orientations, and velocities. Physics experts track these targets directly in simulation under contact, dynamics, and actuator constraints. We train a single strictly causal Transformer on their executed state trajectories that complete the motion and pass semantic and physical validity checks. At each incoming frame, the student uses only current and past human observations to generate a robot motion reference for a downstream whole-body controller. Experiments demonstrate that semantic-conditioned physics experts generate motion with higher semantic fidelity and physical validity than direct optimization-based retargeting, while the resulting streaming retargeter achieves improved closed-loop performance over kinematic and physics-refined baselines under a common whole-body controller.

Primary area:
applications to robotics, autonomy, planning
Keywords:
motion retarget, whole body control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1XQkOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SMR, a real-time streaming retargeting pipeline for human-to-humanoid retargeting. The authors propose to first learn groups of expert motion trackers that follow landmarks from human data. Next, they gather a dataset of expert rollouts that demonstrate high-quality retargeting. Those rollouts are then used to learn a causal transformer that consumes human motion and predicts robot motions. The authors show that this pipeline improves physical validity and supports real-time teleoperation. The authors show that SMR follows human motion semantics better and leads, in general, to more robust tracking behavior.

In summary, the authors show two main contributions

  • a semantic extraction method to define targets for the expert policies
  • a causal transformer trained on human-to-humanoid trajectories

The general idea aligns with NMR; however, the authors skip the initial IK retargeting and instead use pre-computed landmarks as goals for the expert policies, similar to the objective used in methods such as ReActor. Using a causal neural model is also similar to other retargeting work with paired data.

Nevertheless, in its configuration and as a whole, the method is clean and well put together. To improve my score, I'd like a more honest comparison with other work in the field, clearly stating the method as a whole as the key contribution and acknowledging that parts are conceptually and practically very similar to prior work. Furthermore, I would like to see some challenging motions that push the method to its limits.

Critical strengths
  • The qualitative and quantitative results are clear and convincing.

  • The method is simple and intuitive.

Critical weaknesses
  • The novelty is limited and incremental. It follows ideas from NMR and ReActor. The key differences need to be stated more precisely.

  • Challenging results are missing and should be added, e.g., backflips, cartwheels, expressive dancing, etc. I would be interested to see the transformer output for those cases.

  • Table 1. Once trained, both IKMR and ReActor can be applied to real-time applications.

Questions for authors
  1. I find one big limitation of this paper to be the transformer model. Starting from a dataset that respects all physical constraints, the neural method adds back artifacts such as ground penetration or limit violations. Instead of distilling states, could you distill expert actions, like in BeyondMimic, and get a final retargeting policy similar to ReActor? This would also work in real-time and map directly from states to actions.

  2. I'm missing information about the NMR evaluation. Did you retrain the method on all the data, or did you simply pick a pre-trained checkpoint? While Figure 4 shows saturation at 50% data, data diversity might be a key reason for SMR's better quality.

  3. For challenging motions like backflips and cartwheels, expert rollouts might be below the required retargeting quality and therefore not part of the training for the transformer. Yet if you ask the transformer for such a motion, what would happen? I would like to see some limitations or challenging motions.

  4. How tedious is the "semantic extraction" step? What if defining correspondences is difficult because the morphology is very different?

  5. Figure 5 shows a sudden jump in reward at the last 600 iterations for all methods and a very unexpected fall in reward after; why is this the case?

Reviewer_AzcEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SMR, a streaming framework for retargeting human motion to humanoid robots. Its central idea is to train physics experts to track morphology-aware human task-space targets directly, allowing robot trajectories to emerge through simulated execution. Accepted expert rollouts then supervise a causal Transformer that generates robot motion references from current and past human observations. A downstream whole-body controller executes these references.

Critical strengths
  • Directly tracking human task-space targets gives physics experts freedom to resolve embodiment differences without following a prescribed robot joint trajectory. The comparison between semantic-conditioned and GMR-conditioned experts supports the value of this design.
  • The frozen-controller evaluation is useful, which increases completion and reduces both reference-tracking and human-target execution errors.
Critical weaknesses
  • Some physical quality is lost during distillation. In Table 3, self-collision increases from 0.01% for the semantic expert to 1.79% for SMR, while foot slip increases from 0.017 to 0.030 m/s.
  • The hardware results consist of selected examples from independent recordings, without repeated-trial success rates, measured tracking errors, or response delays.
Questions for authors
  • What causes the increase in self-collision and foot slip after distillation?
  • The quantitative hardware comparisons between SMR and GMR over repeated trials on matched motion tasks.
Reviewer_ZMjsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper learns a causal Transformer for humanoid motion retargeting, where large-scale human motions are first transformed into robot-aware task-space landmark trajectories using a deterministic geometric transformation. Motion features are then extracted from these transformed trajectories, and the motions are clustered into groups, especially for large-scale datasets such as BONES-SEED. For each motion group, a motion-tracking expert is trained using PPO in physics simulation. A training dataset is then collected by rolling out these motion experts and retaining trajectories that satisfy semantic and physical validity checks. The causal Transformer is subsequently trained on this dataset in a supervised manner. To demonstrate the effectiveness of the proposed method, the authors compare the quality of the generated motion references against GMR, NMR, and other learning-based baselines. They further evaluate closed-loop motion tracking performance on 400 motions using the same frozen MimicLite controller on Unitree G1, reporting success rates and tracking errors. The proposed method achieves a 97% success rate, compared with 90% for GMR and 92% for NMR. Finally, the authors report real-time inference latency and provide qualitative demonstrations of streaming teleoperation on a physical Unitree G1 robot.

Critical strengths
  • The proposed learning-based motion retargeting framework proposed in this is extenadable, further improved can be applied to either motion feature extraction, motion expert training, and the final causal transformer training.

  • The semantic-conditioned motion expert is well empirically studied. Section 3.2 makes the expert track transported landmark positions, orientations, and velocities directly rather than a retargeted GMR joint trajectory. In Table 3, the motion expert achieves substantially lower errors than GMR and the GMR-conditioned expert while retaining very low joint-limit and collision violations.

Critical weaknesses

The major weakness is the motivation and formulation of the proposed method. The paper does not clearly explain why the proposed framework is formulated as a motion retargeting method rather than a direct motion-tracking policy. Specifically, the trained PPO motion experts produce both control actions and physically executed robot trajectories. However, the authors choose to distill the expert trajectories into a causal Transformer that predicts robot motion references, which are subsequently executed by a separate MimicLite motion-tracking policy. It is unclear why the authors do not directly distill the expert actions into a causal motion-tracking policy conditioned on human motion and robot proprioception, thereby avoiding the need for an additional tracking controller.

This weakness is particularly important because the proposed 12.7M-parameter causal Transformer is itself substantially larger than some existing MLP-based motion-tracking policies, while the overall system still requires an additional learned controller. Although separating retargeting from tracking may offer advantages in modularity, the paper does not sufficiently justify this architectural choice or provide experiments comparing it with direct policy distillation. Such a comparison would help establish whether the proposed two-stage framework offers meaningful benefits in closed-loop performance, computational efficiency, or robustness. Without this justification, the necessity and advantages of the proposed formulation remain unclear, weakening the motivation and significance of the paper's central contribution.

Questions for authors
  • As discussed in the major weakness, could the authors further clarify the motivation for formulating the proposed method as a motion retargeter rather than directly distilling the physics experts into a causal motion-tracking policy? Since the trained experts provide both control actions and physically executed robot trajectories, what are the specific advantages of distill the motion trajectory only beyond the flexibility to use different downstream motion trackers?

  • The proposed method is evaluated exclusively using a frozen MimicLite controller. To what extent do the reported improvements depend on the specific tracking policy? If a different high-performance motion tracker, such as SONIC, were used, would the relative improvements over existing retargeting baselines still hold?

  • Appendix A.4 states that the experts are trained before the train/validation/test split, while the test split is subsequently used to evaluate student distillation on held-out motion groups. Were the test motions included in the training pools of the experts that generated their labels? If so, is it correct that Tables 3 and 4 primarily evaluate the student's generalization to held-out motions, rather than the generalization of the complete expert–student pipeline to motions unseen by the experts? Could the authors clarify this distinction and its implications for interpreting the reported results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.