acceptodds
Under review as a conference paper at ICLR 2027

SetMimic: Policy-in-the-Loop Interpretation of Motion Intent for Humanoid Imitation from Imperfect Motion References

12 views

Abstract

Humanoid imitation commonly relies on high-quality motion capture of professional performers, treating reference joint trajectories as reliable targets for tracking rewards. Internet videos offer a much broader source of motion data, but video-based motion capture often produces physical artifacts such as foot skating, ground penetration, and inconsistent contacts. Directly rewarding joint-level tracking of these references can encourage policies to reproduce artifacts or pursue infeasible targets. Inspired by how professional animators repair motion by preserving the intended action while correcting local artifacts, our key insight is that an imperfect reference can still convey reliable motion intent even when its exact joint trajectories are unsuitable for execution. We present **SetMimic**, a policy-in-the-loop interpretation framework that learns a humanoid policy for each imperfect retargeted motion by preserving reliable intent and resolving uncertain details through robot execution feedback. SetMimic estimates uncertainty to build a structured Motion Intent Representation, then samples and refines intent-consistent trajectories into a reference tube. Short policy rollouts and a learned feasibility critic guide refinement toward realizations that the robot can execute. An uncertainty-conditioned set-tracking objective tracks reliable components precisely while allowing uncertain components to vary within admissible regions, where physical objectives guide execution without a set-tracking penalty. This formulation couples motion interpretation with policy learning to preserve the intended action while accommodating imperfect video-derived references.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Imperfect Motion References, Humanoid Imitation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_baUBOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

SetMimic learns motion tracking from humanoid references imperfectly retargeted from internet videos, estimating how reliable each part of the reference is and tracking reliable components tightly while letting uncertain ones vary within a tube, with the final reference chosen among sampled candidates through short policy rollouts. It is evaluated on dynamic routines on the AgiBot X2 against BeyondMimic with GMR and NMR references and generalist trackers such as BFMTrack and SONIC, with qualitative hardware results.

Critical strengths
  • Learning from noisy video references is a relevant problem, and relaxing tracking where the reference is unreliable is a simple and intuitive idea. The ablations consistently support each component on the physical metrics.
Critical weaknesses

Overall, I don't think the main claims are supported by the current evaluation.

  • What if the motion is just more difficult at some points, rather than corrupted? SetMimic seems to loosen tracking in those parts too, since velocity spikes and joint-limit proximity also flag highly dynamic segments. And since Clean MPJPE only covers reliable components, the evaluation can't even tell the two cases apart. This leaves the claim that SetMimic preserves the intended motion while improving physical execution without evidence, as the only fidelity metric excludes exactly the parts the method changes.
  • The comparisons are hard to read due to some inconsistent writing. It is not clear which retargeter SetMimic uses (contradicting sentences mention both GMR and NMR in the text), and the generalist trackers were not trained for the X2, with no explanation of how they were adapted. No variances are reported either, and the Clean MPJPE differences are very close to NMR + BeyondMimic, so it's hard to judge which differences in Table 1 are meaningful.
Questions for authors
  1. Could you report MPJPE on all components, including the most dynamic segments, to show that SetMimic preserves the difficult parts of the motion rather than smoothing them out?
  2. It is not clear whether SetMimic uses NMR or GMR references in Table 1. Sec. 4.1 says NMR, lines 354 and 413 say GMR, and its Table 1 row is identical to the GMR-based Full SetMimic in Table 2. This matters for the claim that the gains go beyond retargeting. Could you report SetMimic with both?
  3. Could you provide more details on how SONIC was retrained for the X2 embodiment? On which datasets? SONIC is trained on the Unitree G1, while the paper states that the pretrained models were retained. The same applies to BFMTrack.
  4. Line 357 states "any embodiment-specific adaptations must be documented before reporting results". Is this a leftover from a previous draft?
  5. Hardware results are qualitative. The video is not synchronized and doesn't highlight what the reader should look at, so from the current presentation I couldn't clearly tell whether the stumbling could be fixed with better tuning of the baselines. How were the baselines tuned on the X2, and could you report a quantitative hardware measure (e.g., stumbles per motion over repeated trials)?
  6. SetMimic also ends up committing to a single selected reference, so a constrained retargeter (e.g., KDMR or OmniRetarget) could in principle achieve the same improvements. How does SetMimic compare against such a pipeline?
  7. Where do the 2-second candidate rollouts start, and how many rollouts are run per candidate?
  8. Termination conditions for motion tracking pipelines like BeyondMimic are usually restrictive. How do you make sure they are not triggered when relaxing tracking to the tube?
Reviewer_yoETOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SetMimic, a framework for learning humanoid motion imitation policies from imperfect references. Instead of treating reference trajectories as exact targets, SetMimic estimates motion uncertainty, constructs multiple candidate trajectories, uses policy rollouts to evaluate their feasibility, and introduces uncertainty-conditioned set tracking. Experiments on 22 video-derived motions demonstrate improvements in tracking fidelity and physical execution metrics over several baselines.

Critical strengths
  1. Meaningful problem formulation. The paper addresses an important limitation of conventional motion tracking: imperfect references may contain physically infeasible artifacts. Treating references as uncertain motion constraints rather than exact targets is a promising direction.

  2. Principled integration of reference adaptation and policy learning. Combining uncertainty-conditioned set tracking with policy-in-the-loop feasibility feedback provides a coherent approach. The reported improvements in foot sliding, torque violations, and jerk, together with component ablations, support its practical effectiveness.

Critical weaknesses
  1. Insufficient validation of the core claim. The paper claims to preserve motion intent while correcting unreliable references, but the experiments primarily evaluate physical execution quality. There is no direct evaluation of uncertainty estimation accuracy, motion intent preservation, or recovery from truly infeasible references. Improvements in smoothness and stability alone do not establish that the proposed method correctly interprets motion intent.

  2. Unclear experimental fairness. The paper contains inconsistent descriptions of whether GMR or NMR references are used in the main comparisons. Furthermore, the adaptation of pretrained generalist baselines to the AgiBot X2 embodiment is insufficiently explained. These ambiguities undermine confidence in the reported comparisons.

Questions for authors
  1. Which reference source (GMR or NMR) is used for each method in Table 1? How are SONIC and BFMTrack adapted to AgiBot X2?

  2. How are the reliable reference components used in Clean MPJPE determined? Are they independently annotated or derived from the proposed uncertainty estimator?

  3. Can the authors evaluate the method using controlled reference corruption with clean motion ground truth, demonstrating that uncertainty estimation correctly identifies corrupted components and that SetMimic preserves the original motion intent? A comparison against uncertainty-weighted point tracking would also help establish the necessity of set tracking.

Reviewer_wmfvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SetMimic, a framework that learns humanoid policy from imperfect retargeted motion. It (1) estimates per-feature uncertainty and contact probabilities from the retargeted motion, (2) samples and refines a reference tube which includes coherent candidate trajectories, (3) uses short policy rollouts and a feasibility critic to select an executable candidate, (4) and leverages a set-tracking term to penalize deviation only outside uncertainty-scaled regions. Compared to baselines, it reduces foot sliding, torque violation and jerk, with marginal MPJPE gains.

Critical strengths
  • Learning from imperfectly retargeted motion is an important, well-motivated problem.
  • SetMimic consistently outperforms retargeting baselines and generalist trackers.
  • The authors ablated all components to show the necessity of proposed pipeline.
  • The authors included real world results.
Critical weaknesses

Experiments

  • The paper states that no method fell, yet averages only fall-free rollouts. If all baselines complete every motion, the benchmark does not test the robustness the paper emphasizes.
  • It was only tested on one robot, one motion-capture model and 22 clips of one genre; hardware results are qualitative only.
  • Key baselines are missing. RL tracking with artifact penalties (foot sliding, penetration, torque) and physics-aware retargeting such as KDMR or PhysDiff followed by tracking are cited but not compared, although the paper argues both are insufficient.
  • Computational cost is unreported. Does SetMimic require more time to train compared to baselines?

Method

  • It has many hyperparameters (δ_g, α_g, k_g, λ_phys, N, K, refinement and score weights, number of interpretation rounds), most of which are unreported, with no sensitivity analysis.
Questions for authors

Please check weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.