acceptodds
Under review as a conference paper at ICLR 2027

InterMimicGen: Scaling Humanoid Loco-Manipulation through Self-Evolving Motion Imitation

Abstract

Captured human-object interactions provide rich supervision for humanoid loco-manipulation, but they are sparse, heterogeneous, and not directly executable by robots. We introduce InterMimicGen, a self-evolving motion-imitation framework in which robot motion data and a tracking policy improve each other. First, we consolidate motion-captured human-object interaction datasets and retarget them into humanoid robot references while preserving whole-body coordination and dexterous hand-object relationships. This produces a large and diverse humanoid robot reference collection for dexterous whole-body loco-manipulation. Second, we train a physics-based generalist tracker that executes these references in simulation on a humanoid with dexterous hands, covering a scale and diversity beyond prior humanoid tracking systems for loco-manipulation. Third, we close a data flywheel: each round makes small, task-preserving changes to where an interaction takes place and how the body performs it, fine-tunes the tracker on them, and keeps only the variants whose simulated execution completes the task, which seed the next round. With more iterations, these small edits compound into broader coverage around the sparse original demonstrations while preserving task semantics and motion quality. Experiments show contact-preserving retargeting across robot configurations, broad tracking with a single generalist policy, executable motions that keep growing over augmentation rounds, and transfer to real robots. InterMimicGen provides a unified path from heterogeneous human demonstrations to a continually expanding motion resource for humanoid robot learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
human-object interaction, humanoid loco-manipulation, motion imitation, motion retargeting, motion generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UP4sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work presents InterMimicGen, a data generation and verification framework for human-object interaction (HOI). The authors consolidate heterogeneous interaction captures, propose a two-stage dexterous whole-body retargeter to construct robot interaction references, and train an RL policy that serves as both tracker and verifier. Building on these components, the proposed ``self-evolving'' loop iteratively rolls out verified references and fine-tunes the tracker. Experiments show that the retargeter produces cleaner dexterous references than prior work, and that over five augmentation rounds the verified reference set grows by more than 100$\times$ while the evolved tracker executes ~98% of the generated references.

Critical strengths

The ablation studies demonstrate that the self-evolving loop expands executable whole-body dexterous interactions. Table 3 shows the tracker improves from 52-64% to 98% success on the generated references, which grow over 100X in five rounds. Table 11 further shows this growth depends on in-loop learning rather than filtering, as a frozen tracker yields only 0.4% versus 72.9% in round 2.

Critical weaknesses

The claimed capability growth is validated only within the generation loop itself. Accepted references are filtered by the execution success of the fine-tuned tracker and then included in its further training. However, essential validation steps are missing, such as testing the tracker on held-out objects or out-of-distribution editing operations, as well as comparing the performance of downstream policy learning when using augmented data versus using only the initial seed data.

Questions for authors
  1. Is the "self-evolution improves the tracker" result a circular evaluation with train/test leakage? Can you report the evolved tracker's success on held-out objects or on edits beyond the training range, and a downstream policy trained on augmented data versus seed data alone?

  2. Scalability of the flywheel. The hundredfold growth densifies around existing seeds along a few hand-designed edit axes, and per-round gains are already decelerating. It's unclear whether the loop scales or saturates. Can you characterize its asymptotic behavior, such as acceptance rate and cost over more rounds, and show whether it can expand into interaction types where seed coverage and tracking success are poor, rather than only thickening what already works?

5 comments
AuthorsOpenReview ↗

Response to Reviewer UP4s

We thank the reviewer for the positive assessment and constructive feedback. We appreciate the recognition that the self-evolving loop expands executable whole-body dexterous interactions, and that this growth depends on in-loop learning rather than filtering.

W1: Validation beyond the generation loop. We agree that Table 3 measures the executability of the generated archive rather than generalization. Generalization holds beyond the loop: the tracker improves on edits beyond its training range, and on held-out objects, training with augmented data outperforms training on seed data alone. Per reviewer’s request, we present two additional evaluations.

W1.1: Edits beyond the training range. Beyond the loop, the evolved tracker learns interaction skills rather than memorizing its archive. Each round extends competence past the range it was trained on: trackers from rounds 1 and 2, which never trained on round-3 references, already execute far more than the original tracker, as shown below. This is exactly what allows the next round to push further.

TrackerOriginalRound 1Round 2Round 3Round 4Round 5
SR on round-3 references (%) ↑40.085.289.696.398.599.3

W1.2: Held-out objects and downstream learning. The gain also transfers to held-out OmniContact motions, boxes of sizes never seen in training with motions never used in any round. Under matched compute (we train the original checkpoint longer to match the compute), the tracker trained with augmented data outperforms the one trained on seed data alone. The augmented data itself, not only the archive it is evaluated on, therefore improves downstream learning.

Tracker (matched compute)OmniContact SR (%), Single rollout ↑OmniContact SR (%), Best-of-3 ↑
Continued training (seed data only)40.055.0
Self-evolution (ours)69.486.7

Q1: Circular evaluation. We thank the reviewer for the suggestion. We find the improvement holds on data the tracker was never trained on: edits beyond the training range (W1.1), and held-out objects from OmniContact, where training with augmented data outperforms training on seed data alone (W1.2).

Q2: Scalability of the flywheel. Per the reviewer's request, we run three more rounds. The archive keeps growing: each round adds more references than the one before at a lower cost per reference, and the acceptance rate declines only gradually. The displacement range, however, levels off after round 6, so later rounds mainly densify the coverage around each seed, likely within what the robot can physically execute. The hand-designed edit axes set only the directions of variation, and coverage is measured independently of them (L292).

RoundGrowth (×)Acceptance rate (%)Compute (GPU h)Accepted refs per GPU hMean max displacement (cm)
1–5150.585.225.727923.5
6213.485.45.653929.9
7331.080.97.476330.1
8459.477.58.077131.7

Q2.2: Interaction types with poor seed coverage or tracking success. Per reviewer’s request, we run the loop on a trash can, an object outside the paper's experiments on which the original tracker succeeds on only 17.4% of clips. Starting from 11 seeds, the loop expands the executable references from 775 after round 1 to 10,434 after round 5, with a displacement p90 of 16.8 cm and 76.9% of executed candidates accepted (videos in 04.1 of the supplementary webpage, Grasp × Inspire, Motions 5–6), though the displacement is smaller than the augmentation with in-distributional trajectories.

Thank you for the clarifications! I still have two minor questions: 1) the gains seem to plateau at iterations 6–8. Does this suggest the loop is beginning to saturate? 2) are there quantitative metrics showing how the humanoid motion style or domain changes across iterations?

AuthorsOpenReview ↗

Response to Reviewer UP4s’s Follow-up Questions

We sincerely thank the reviewer for the thoughtful follow-up questions and for the careful reading of our responses. We address the remaining questions below with new results.

1) Per-seed displacement saturates but the flywheel keeps scaling. We would like to first clarify what saturates: it is the displacement range around each seed; that is, how far its augmented references move away from the seed. 

Why a bound exists. Each augmented reference keeps the starting point of its seed and its goal. The edits leave the motion before the first hand-object contact untouched and move the object only during contact, on the seed's own timing, and the augmented reference must still reach the same type of outcome, such as placement on a support (L238–240, Sec. D.2). 

The more the motion must change within the same semantic, the more challenging it is to complete the interaction in time. This is a physical constraint. Saturation is therefore expected: it reflects the physical limit of each seed rather than a limit of the loop.

We further show that the accepted references keep the interaction in the original distribution (point 2 below).

Within the bound, the loop keeps densifying the distribution. Each round still adds more verified references than the one before. 

 

2) Motion style and domain across rounds. Across rounds, the interaction stays within the distribution of the retargeted human captures, while the motion distribution shifts away from the original postures (Sec. H and Table 19 of the revision).

  • Metric. 
    • We use precision (Kynkäänniemi et al., NeurIPS 2019), a standard metric for comparing generated data with a reference distribution. Here it is the share of each round's augmented references that fall inside the manifold of the retargeted human captures of the same object. The manifold is the union of balls centered at these captures, each with a radius equal to the distance to its third nearest neighbor. 
    • We compute this metric with the two feature definitions of Sec. E.1, evaluated on the robot trajectories. Body motion is described by the body keypoints relative to the pelvis and the pelvis trajectory, and interaction by the object's position relative to the pelvis and the wrists and by the object's own motion.
  • The interaction keeps its semantics. The metric is 0.72–0.80 in every round with no downward trend, close to the 0.83 of the retargeted human captures themselves. This aligns with our intention: every accepted reference keeps the interaction type, intended contacts, and object of its seed (L238–240). 
  • The motion distribution shifts. The metric falls from 0.70 in round 1, the level of the retargeted human captures themselves, to 0.54 in round 8, showing the deviation from the original references. The augmentation is designed for this: it adds new ways of performing the same interaction (L234–235), and each accepted edit is executed successfully in simulation.
FeaturesRetargeted human capturesSeedsR1R2R3R4R5R6R7R8
Interaction0.830.850.720.750.770.780.780.800.790.78
Body motion0.710.770.700.690.670.640.640.560.540.54
Body motion, spatial edits––0.670.640.600.600.610.600.570.58
Body motion, body edits––0.730.730.730.680.680.510.510.49

Retargeted human captures: each capture tested against all the others (leave-one-out). Seeds: the original references before any round. R1–R8: the augmented references of each round. The last two rows split the augmented references by edit type, which does not apply to the retargeted human captures and the seeds (denoted as ‘–’).

We would greatly appreciate any other metrics the reviewer would find informative for measuring how the distribution changes, and we would be more than happy to compute and report them.

 

Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved Precision and Recall Metric for Assessing Generative Models. NeurIPS 2019.

Reviewer_ogDROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces InterMimicGen, a framework combining contact-aware retargeting, generalist interaction tracking, and iterative augmentation to expand human-object demonstrations into executable robot references.

Critical strengths

The paper addresses an important data-scarcity problem in dexterous humanoid loco-manipulation through a coherent combination of retargeting, physics-based tracking, and iterative augmentation.

The implementation demonstrates substantial engineering breadth, supporting multiple robot configurations, articulated hands, diverse objects, and real-world execution.

The frozen-tracker ablation provides useful evidence that continued policy learning contributes to reference expansion beyond filtering what the initial tracker can already execute.

Critical weaknesses
  1. The benefit of iterative self-evolution is not sufficiently isolated. The evolved tracker is evaluated primarily on references used in training and retained through successful execution. Moreover, Table 3 counts a reference as successful if any of three rollouts succeeds. These results establish archive executability, but provide limited evidence of generalization. Comparisons against continued training and one-shot augmentation under matched compute budgets, evaluated on independently sampled targets, would strengthen the central claim.

  2. Task-preservation criteria require clarification. Table 9 does not check whether spatially edited placement tasks leave the object stably resting on its support. Completing the reference without falling and matching terminal contact labels may not guarantee the intended task outcome. Please explain how task semantics are verified for spatial edits, particularly when executed trajectories are relabeled and retained.

  3. Novelty over related data-generation approaches remains insufficiently demonstrated. The paper should more clearly establish its advantages over Humanoid-DART and HumanoidMimicGen through common-task comparisons of coverage, generation yield, computational cost, and downstream learning benefits. The body-motion range comparison with ULTRA alone does not establish these advantages.

  4. The contact metrics do not fully establish fine-grained grasp quality. Contact preservation uses a 2 cm proximity threshold and counts interpenetration as contact, while the reported object-penetration rate remains 37.2%. Finger-level contact accuracy, grasp stability, and ablations separating finger opposition from geometry cleanup would better support the dexterous-retargeting contribution.

  5. Hardware validation is only indirectly connected to the learned generalist and augmentation loop. G1 deployment uses SONIC for body control and finger-trajectory replay. Quantitative comparisons between original and augmented trajectories, with repeated trials and object-placement perturbations, would clarify the practical benefits of the generated data.

Questions for authors
  1. Could the iterative loop be compared with continued training and one-shot augmentation under matched computational budgets, using independently sampled evaluation targets? Could single-rollout success rates also be reported alongside the current best-of-three metric?

  2. How is task completion guaranteed for spatial edits, particularly placement tasks, when Table 9 does not require the object to rest stably on its support?

  3. Could comparisons with Humanoid-DART and HumanoidMimicGen quantify task-space coverage, generation yield, computational cost, and downstream learning benefits on shared tasks?

  4. Could the authors separately ablate finger opposition and geometry cleanup, and evaluate finger-level contact accuracy and grasp stability beyond the current proximity-based contact metric?

  5. Which hardware demonstrations use references generated in later augmentation rounds? Could repeated real-world trials compare original and augmented references, including sensitivity to object-placement perturbations?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ogDR (1/3)

We thank the reviewer for the detailed and constructive feedback. We appreciate the recognition that our work addresses the data-scarcity problem in dexterous humanoid loco-manipulation, the substantial engineering breadth of our paper, and that the continued policy learning contributes to reference expansion.

W1: Isolating the benefit of self-evolution. The benefit of self-evolution holds beyond the archive it builds: it makes the policy generalize better without capturing new recordings.

W1.1: Benefit of iterative self-evolution.

  • Evolution is what makes large edits executable. Physics-based tracking can repair a small mismatch from the kinematic edit, and repeating such small edits accumulates into large variations, whereas a large edit applied at once moves the reference too far for tracking to correct (videos in 04.2 of the supplementary webpage).
  • With one-shot augmentation sampling the same edit range as our round 5, only 29.5% of its edits are corrected by tracking, versus 82.4% in our loop, which also reaches a larger displacement range, as large displacement in one shot cannot be tracked.
MethodDisplacement p90 (cm) ↑Executed (%) ↑
One-shot augmentation15.829.5
Self-evolution (ours)28.682.4

Inspire, bimanual.

W1.2: Matched-compute comparison on held-out motions. 

  • Per the reviewer's request, we compare self-evolution with (a) continued training of the original tracker without new data (under the same budget) and (b) a tracker fine-tuned once on one-shot augmentation without evolution (W1.1), all under matched compute. As independently sampled targets, we use motions from OmniContact that are never used in training or produced by any round. Self-evolution outperforms both baselines.
Tracker (matched compute)OmniContact SR (%), Single rollout ↑OmniContact SR (%), Best-of-3 ↑
Continued training40.055.0
One-shot augmentation51.166.7
Self-evolution (ours)69.486.7

 

  • In a separate test, the evolved tracker also extends past its own training range: trackers from rounds 1 and 2 already execute 85.2% and 89.6% of the references accepted in round 3, which they never trained on, versus 40.0% for the original tracker.

W1.3: Best-of-three success. Per reviewer’s request, we report single-rollout success rates alongside the best-of-three metric in Table 3 of the revision. The gap between the original and evolved trackers becomes even larger.

SettingAccepted edits, Orig. → Evol. (%), Best-of-3 ↑Accepted edits, Orig. → Evol. (%), Single rollout ↑Original refs, Orig. → Evol. (%), Best-of-3 ↑Original refs, Orig. → Evol. (%), Single rollout ↑
Inspire, bimanual52.1 → 98.429.3 → 92.9100 → 10092.4 → 98.6
Dex3, bimanual59.3 → 98.741.8 → 95.9100 → 10090.3 → 100
Inspire, grasping64.2 → 98.943.6 → 93.6100 → 100100 → 100
AuthorsOpenReview ↗

Response to Reviewer ogDR (2/3)

W2: Task semantics for spatial edits. For spatial edits, our acceptance criteria ensure that the task semantics are preserved.

W2.1: Resting on the support. The object is ensured to rest stably on its support even under spatial edits. The "–" entry in Table 9 only reflects that the support itself is relabeled and can change its position. This is an option, and our augmentation keeps both: with a fixed support, the object rests on the original support, and with a relocated support, the support moves together with the object, so the object rests on the relocated support. Both are shown in 04.1 of the supplementary webpage (Grasp × Inspire, Motions 7–8). We clarify this in Table 9 of the revision (L1094–1107).

W2.2: Verification of task semantics. The two support options in W2.1 correspond to the two interpretations of task preservation: (i) the original task goal is achieved, or (ii) the motion semantics are preserved under a new goal. A fixed support realizes (i): the object is placed on the original support, so the original goal is preserved. A relocated support realizes (ii): the goal changes, but the interaction type, intended contacts and object are preserved (L238–240), so a placement remains a placement, a push remains a push, and a pull remains a pull.

W2.3: Relabeled and retained trajectories. All checks run on the executed rollout itself, which is exactly the trajectory that is relabeled and retained. Only accepted trajectories can seed later rounds, and a failed proposal is never edited again, so a semantic failure cannot propagate across rounds (L288–290).

W3: Comparison with related data-generation approaches. InterMimicGen matches or exceeds Humanoid-DART and ULTRA in coverage, yields more physically valid motions, and gives a higher downstream success rate.

MethodCoverage† ↑Diameter‡ ↑Yield (%) ↑Compute (GPU h) ↓Downstream SR (%) ↑
Humanoid-DART41.811.8×8.5416.966.7
ULTRA0.74.3×––63.3
Ours42.914.7×85.225.786.7

†Distinct whole-body motions per seed: motions differing by more than four times the replay noise in joint-angle space, each method measured against its own replay noise and subsampled to the same number of motions, averaged over seeds. ‡Largest distance between two motions of the same seed, in units of the replay noise. Downstream SR: best-of-3 success on held-out OmniContact motions (W1.2; details in the revised Sec. E.4).

W3.1: Humanoid-DART. Humanoid-DART generates from a group of similar seeds within one task space with rigid hands, whereas InterMimicGen applies to diverse interaction types with dexterous hands at the same time. We reproduce it on the same task following their official release and report the four requested measures in the table above.

Our loop reaches comparable coverage over a wider range and an order of magnitude higher yield, although the two methods propose candidates differently, and our data gives a higher downstream success rate. Ours uses more compute because it fine-tunes the tracker iteratively.

W3.2: HumanoidMimicGen. Its trajectory-synthesis code is not publicly released, but its data is bounded by its released whole-body controller, since every trajectory, including the teleoperated source demonstrations, is an execution of this controller. This controller limits how far the torso can bend: on our references, with hand targets adapted to the object as in its skill adaptation, it executes only 0.9% of them, versus 70.7% for our tracker, because it cannot bend over to a box near the floor (median maximum torso tilt 50° versus 86° in our references).

W3.3: ULTRA. We use ULTRA's officially released augmented data for coverage. ULTRA also uses rigid hands without fingers. As ULTRA only scales the scene, its coverage on body motion remains limited.

AuthorsOpenReview ↗

Response to Reviewer ogDR (3/3)

W4: Fine-grained grasp quality. We agree and add all four requested evaluations. When penetrating contacts are not counted, our method shows even stronger improvement compared to baseline Weave at both the hand and the finger level and yields more stable grasps. The ablations show that the geometry cleanup accounts for the contact and penetration gains, while the finger opposition term forms the grasp.

W4.1: Fine-grained contact metrics and ablations.

  • Contact without penetration. Counting only contacts that penetrate no more than 0.5 cm, ours preserves far more contact than Weave at both the hand and the finger level, and stays ahead at looser tolerances (revised Sec. E.2, Penetration-aware contact preservation and Per-finger contact preservation).
  • Grasp stability. Measuring how far each finger’s contact point drifts on the object surface during contact, ours drifts less than Weave (revised Sec. E.2, Grasp stability).
MethodContact, 0.5 cm (%), CP ↑Contact, 0.5 cm (%), Per-finger CP ↑Obj. penetration (%) ↓Contact drift, median (cm) ↓Contact drift ≤ 2 cm (%) ↑
Weave8.045.967.43.2825.6
Ours w/o cleanup6.338.972.82.6038.5
Ours51.458.037.22.3044.4

W4.2: Ablations of the cleanup and the opposition term.

  • The cleanup accounts for the contact and penetration gains: without it, the hands close deeper into the object, so penetration rises and few contacts stay within 0.5 cm (W4.1).
  • The opposition term forms the grasp: with it, the thumb and a finger first touch the object from opposite sides far more often (revised Sec. E.2; videos in 02.4 of the supplementary webpage), without this, the reference tends to penetrate the object then grasp.
MethodCP (%), penetration ≤ 0.5 cm ↑CP (%), penetration ≤ 1 cm ↑CP (%), penetration ≤ 2 cm ↑CP (%), any penetration ↑
Weave8.018.143.796.1
Ours w/o cleanup6.311.432.599.0
Ours51.455.969.398.2

 

MethodOpposed at first contact (%), Tripod ↑Opposed at first contact (%), Clothes stand ↑
Weave36.425.2
Ours w/o opposition38.451.6
Ours64.375.4

W4.3: Object-penetration rate. Our 37.2% is already far below Weave's 67.4% under the same protocol, and the remaining penetration can be removed downstream with the simulator, since only executed rollouts are retained as our data collection.

W5: Hardware validation. Augmented references achieve a success rate comparable to the original references in real-robot trials. Per the reviewer's request, we run repeated trials on two sets, each with one original reference and three augmented references edited from it.

OriginalAugmented
Set 18/97/9
Set 29/99/9
Total17/1816/18

Each set has one original reference (9 trials) and three augmented references (3 trials each).

Q1: Matched-compute comparison and single-rollout success. Please see W1.2 for the comparison with continued training and one-shot augmentation under matched compute on independently sampled targets, and W1.3 for single-rollout success rates.

Q2: Task completion for spatial edits. Please see W2.1 and W2.2.

Q3: Comparison with Humanoid-DART and HumanoidMimicGen. Please see W3.1 and W3.2.

Q4: Ablations and fine-grained grasp metrics. Please see W4.1 and W4.2.

Q5: Hardware demonstrations from later rounds. Yes. All hardware demonstrations of augmented references use references generated in rounds 4–5 of the self-evolving loop, and the round of each demonstration is listed in 01.1 of the supplementary webpage. The repeated real-world comparison between original and augmented references is reported in W5.

Reviewer_NU4DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents InterMimicGen, a framework for scaling humanoid loco-manipulation from human–object demonstrations. It combines contact-preserving whole-body retargeting, a generalist tracking policy, and iterative motion augmentation. The framework modifies object trajectories and body postures, trains the tracker on these variations, and retains successful executions for subsequent augmentation. This process expands executable motion coverage while preserving task semantics. Experiments evaluate retargeting quality, generalist tracking, iterative data expansion, and real-robot execution across multiple robot configurations.

Critical strengths

1.The paper consolidates a large and diverse collection of 16,059 interaction motions and trains a shared policy across different objects and interaction types. This provides a useful foundation for studying scalable humanoid loco-manipulation. 2.The retargeting pipeline shows substantial improvements in penetration, motion smoothness, and joint-limit compliance. Demonstrations across multiple robot configurations, including real-robot executions, further support the practical value of the generated motions.

Critical weaknesses

1.The novelty of the retargeting algorithm is limited. Its main contribution is optimizing fingertip and thumb directions, which addresses only a narrow aspect of interaction and seems less relevant to many demonstrated tasks, such as box carrying and sitting on chairs. Moreover, the paper does not show whether this optimization improves downstream tracking success rates, leaving its practical contribution to the overall framework unclear. 2.The motivation for the self-evolving component is unclear. The augmentation procedure still relies on manually specified rules, while robustness to minor trajectory variations should be a basic capability of a well-trained policy. Why is iterative reference augmentation necessary instead of standard domain randomization? The authors should demonstrate that this procedure improves generalization beyond the augmented reference trajectories, rather than simply encouraging the policy to fit a larger set of predefined motions.

Questions for authors

1.Regarding the quantitative retargeting comparison in Table 1, the core contribution of the proposed retargeting method lies in hand optimization. However, improvements over the baseline in hand-contact metrics appear limited, whereas improvements in physical plausibility metrics, including penetration, motion, joint acceleration, and joint-limit violations, are substantial. The authors need to clarify the reasons for this discrepancy; otherwise, the contribution of the retargeting component is further weakened. 2.If the augmented references involve only small trajectory changes relative to the original data, why not incorporate these variations, such as offsets in object or humanoid trajectories, through domain randomization during policy training? This is often a more effective approach for sim-to-real transfer.

3 comments
AuthorsOpenReview ↗

Response to Reviewer NU4D (1/3)

We thank the reviewer for the constructive feedback. We appreciate the acknowledgment of our large-scale collection of diverse interaction motions, the improvement of our retargeting in motion quality, and our multi-robot support with real-world execution.

W1: Novelty and downstream effect of retargeting. Please see our new videos in 02.1 of the supplementary webpage for challenging dexterous examples. To the best of our knowledge, we present one of the few methods that produce high-quality dexterous whole-body interactions from diverse challenging MoCap data. Extending non-dexterous retargeting to our collection is non-trivial.

W1.1: Novelty of the retargeting algorithm.

  • Problem 1: Prior whole-body interaction retargeting, such as OmniRetarget and UMR, uses rigid hands and can only approximate a grasp, with the hand sometimes floating (Figures 10 and 11). Our hands hold the object at a consistent grasp position, promoted by the finger opposition.
  • Problem 2: Within one joint optimization, the body and the fingers need constraints at very different scales, so the body terms overshadow the hand terms, and a single objective tends to either loosen contact or over-constrain the body (L189–192).
  • Our solution is simple yet effective: the two-stage optimization (whole-body solve, then geometry cleanup) addresses the imbalance optimization, while the finger opposition term directs the thumb and the fingers to approach the object from opposite sides, whereas without it the fingers tend to penetrate the object to form a grasp (new videos in 02.4 of the supplementary webpage). It is critical for contact-rich grasping, which makes up 8 of our 15 benchmark objects (W1.2), and it improves downstream tracking (W1.3).
  • We also note that the retargeting algorithm is not claimed as a main contribution of our paper. Our three contributions (L087–093) are the most diverse humanoid reference collection for dexterous whole-body loco-manipulation, a generalist tracker that executes it at scale, and the self-evolving loop that expands it. Retargeting serves the first: it is how we turn heterogeneous human data into this collection. The reviewer recognizes this result, noting that our 16,059 interaction motions and shared policy "provide a useful foundation for studying scalable humanoid loco-manipulation".

W1.2: Relevance to the demonstrated tasks.

  • Dexterous grasping is a substantial part of our results. We show that non-dexterous approaches like OmniRetarget and UMR fail on our tasks (Figures 10 and 11), with significantly more contact floating.
  • 8 of the 15 objects in our tracking benchmark are dexterous tasks, involving objects like tripod, racket and dumbbell (Sec. E.3). A G1 with rigid hands cannot perform any of them (L183–184, Table 5). The self-evolving loop also runs on Inspire grasping (Table 3).
  • In the revised supplementary webpage, we show more challenging dexterous examples from datasets including ARCTIC, HUMOTO and ParaHome in 02.1.

W1.3: Downstream tracking success.

  • The finger opposition term improves downstream tracking success. Due to the limited rebuttal period, we could not train the trackers to convergence, but under the same training setting and compute budget, references retargeted with the term already reach a grasping success rate of 59.2%, versus 55.3% without it.
  • The gain comes from better-formed grasps. As references retargeted without the term penetrate to form the grasp (reference video in 02.4 of the supplementary webpage), in physics it directly pulls the object down without grasping.
AuthorsOpenReview ↗

Response to Reviewer NU4D (2/3)

W2: Motivation of the self-evolving component. Evolution is necessary because physics-based tracking can repair a small mismatch from the kinematic edit. And repeating small edits accumulates into large variations (L078–082). A large edit applied at once moves the reference too far for tracking to correct, so most of it cannot be executed (W2.1, videos in 04.2 of the supplementary webpage). As a result, our loop reaches a larger displacement range than one-shot augmentation, with edits far beyond minor variations that domain randomization cannot achieve (Q2), and the co-evolved tracker generalizes to held-out OmniContact motions (W2.5).

W2.1: Why iterative augmentation. We let one-shot augmentation try the same range of our iterative editing. Our loop retains far more edits and reaches a larger displacement range with physics validation, than one-shot augmentation.

MethodDisplacement p90 (cm) ↑Corrected by tracking (%) ↑
One-shot augmentation15.829.5
Self-evolution (ours)28.682.4

Inspire, bimanual.

W2.2: Manually specified edit rules. The rules specify only the directions of variation, not the motions themselves, so the policy does not fit a set of predefined motions. We hand-specify a few generic coordinates, object placement (planar translation and yaw) and body posture (pelvis, stance and arm nullspace), much as domain randomization specifies which parameters to randomize. They are defined in each seed's own interaction frame (L1042–1044) and shared by every object, interaction type and embodiment. The motions themselves, and the range they reach, are decided by physical execution (L288–292).

W2.3: Robustness to minor variations.

  • Even our generalist tracker, trained on 16,059 motions, executes only 52–64% of the accepted edits whereas the fine-tuned tracker can execute over 98% (Table 3). These edits are therefore far beyond the minor variations a well-trained policy can absorb.
  • Such robustness holds for locomotion, but loco-manipulation is harder: contact makes its dynamics highly non-smooth, and we find that a policy cannot simply interpolate between demonstrations (L059–063). Iteratively filling in the space between them is therefore what lets the policy generalize beyond memorizing its data (W2.5).

W2.4: Comparison with domain randomization. Please see our response to Q2.

W2.5: Generalization beyond the augmented references. The co-evolved tracker generalizes well to data it has never been trained on. Under the same compute budget, it outperforms both continued training of the original tracker and the tracker finetuned on one-shot augmentation without evolution, on held-out motions from OmniContact that are never used in training.

Tracker (matched compute)OmniContact SR (%), Single rollout ↑OmniContact SR (%), Best-of-3 ↑
Continued training40.055.0
One-shot augmentation51.166.7
Self-evolution (ours)69.486.7
AuthorsOpenReview ↗

Response to Reviewer NU4D (3/3)

Q1: Limited hand-contact gain versus large plausibility gains. Hand contact is best assessed together with penetration, since a hand can also keep contact by sinking into the object. Viewed jointly, our hand-contact gain is large.

  • The baseline trades penetration for contact. Weave penetrates the object in 67.4% of frames versus 37.2% for ours (Table 1). When any penetration counts as contact, both methods preserve almost all contacts (96.1% versus 98.2%), so the gap looks small.
  • Without penetration, our contact preservation is far higher. When contacts deeper than 0.5 cm are not counted, ours preserves 51.4% versus 8.0% for Weave, and it stays ahead at every tolerance (table below; metrics in the revised Sec. E.2).

Finally, to the best of our knowledge, no prior work performs dexterous whole-body retargeting. We therefore go beyond prior work and also compare with Weave, the strongest available reference, which should be viewed as concurrent work.

MethodCP (%), penetration ≤ 0.5 cm ↑CP (%), penetration ≤ 1 cm ↑CP (%), penetration ≤ 2 cm ↑CP (%), any penetration ↑
Weave8.018.143.796.1
Ours51.455.969.398.2

Q2: Why not domain randomization. Domain randomization (DR) varies the robot's state around a fixed reference, whereas our augmentation varies the reference itself, which DR cannot replace.

  1. DR varies states and observations, not references. DR usually randomizes physical properties such as object mass and friction, sensor noise, and external pushes, all around a fixed reference, so the policy learns to recover to that same reference. Our augmentation instead changes the reference: where the object is placed and the whole-body motion that realizes it, verified in physics and pushed further from verified results in each round.
  2. DR induces much smaller trajectory variation. DR’s perturbations do make the executed trajectory deviate, but only locally around the reference: successful DR executions reach a displacement p90 of 13.4 cm, versus 28.6 cm for our loop.
  3. Only reference augmentation generalizes to new references. Under the same compute budget, we train a tracker with object-property randomization and random pushes, and run it with our loop's own collection and relabeling on edits sampled at once over a range comparable to round 5. The DR tracker executes 23.5% of these new references, versus 82.4% for ours.

We agree with the reviewer that DR remains useful for sim-to-real robustness, which our augmentation complements rather than replaces, since it targets data generation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their careful and constructive reviews. The reviewers appreciated the large and diverse reference collection with a shared policy as a useful foundation for scalable humanoid loco-manipulation (NU4D), the retargeting improvements in motion quality (NU4D), the support of multiple robot configurations with real-world execution (NU4D, ogDR), the importance of the data-scarcity problem (ogDR), and the self-evolving loop that expands executable dexterous interactions through in-loop learning (ogDR, UP4s). We answer the remaining points under each review. All page and line references refer to the revised paper, with modified parts highlighted in blue. References such as 02.4 refer to sections of the updated supplementary webpage.

What experiments are added? Each is answered in full under the review that asked for it.

  • Generalization to held-out motions (NU4D W2.5, ogDR W1.2, UP4s W1.2). On OmniContact motions never used in training, the evolved tracker reaches 86.7% success under matched compute, versus 55.0% for continued training and 66.7% for one-shot augmentation.
  • Iterative versus one-shot augmentation, and domain randomization (NU4D W2.1 and Q2, ogDR W1.1). Our loop reaches a displacement p90 of 28.6 cm, versus 15.8 cm for one-shot augmentation over the same edit range and 13.4 cm for domain randomization.
  • Single-rollout success (ogDR W1.3). The gap between the original and evolved trackers becomes larger than under the best-of-three metric.
  • Fine-grained contact and grasp metrics (NU4D Q1, ogDR W4). We add hand- and finger-level contact preservation without penetration, contact drift for grasp stability, and opposed grasp at onset. Our retargeting outperforms Weave on all of them: when contacts penetrating deeper than 0.5 cm are not counted, it preserves 51.4% of hand contacts versus 8.0%, and it also preserves more finger contacts, drifts less, and forms opposed grasps more often. Ablations show that the geometry cleanup accounts for the contact and penetration gains, while the finger opposition term forms the grasp.
  • Downstream effect of the finger opposition term (NU4D W1.3). Under the same training setting and compute budget, references retargeted with the term reach a grasping success rate of 59.2%, versus 55.3% without it.
  • Comparison with related data-generation approaches (ogDR W3). InterMimicGen generates more distinct whole-body motions per seed over a wider range compared with Humanoid-DART (coverage 42.9 versus 41.8, diameter 14.7× versus 11.8×) and ULTRA (42.9 versus 0.7, 14.7× versus 4.3×). HumanoidMimicGen cannot cover such motions, since the whole-body controller that bounds its data executes only 0.9% of our references.
  • Edits beyond the training range (UP4s W1.1). Trackers from earlier rounds generalize to the edits of later rounds, which they never trained on, much better than the original tracker.
  • Scalability (UP4s Q2). Three more rounds grow the verified references from 150.5× to 459.4×, and the loop expands an object on which the original tracker succeeds on only 17.4% of clips.
  • Repeated real-robot trials (ogDR W5). Augmented references show a comparable success rate to the original references.

What changes in the revision? The revised PDF includes every change below, highlighted in blue.

  • Table 3 (p. 9, L433–447): single-rollout success rates (ogDR W1.3).
  • Table 9 (p. 21, L1094–1107): clarification of the support for spatial edits (ogDR W2.1).
  • Sec. E.2 (pp. 22–23, L1174–1195), Sec. F.2 (p. 24, L1288–1295), Table 11 (p. 26, L1395–1403) and Table 12 (p. 27, L1405–1409): fine-grained contact and grasp metrics, with ablations of the geometry cleanup and the finger opposition term (NU4D Q1, ogDR W4).
  • Sec. E.4 (p. 24, L1255–1262), Sec. F.5 (p. 28, L1489–1493) and Table 15 (p. 28, L1496–1501): coverage and diameter compared with ULTRA (ogDR W3).
  • Sec. E.4 (p. 24, L1263–1266), Sec. F.6 and Table 16 (pp. 28–29, L1505–1517): matched-compute comparison on held-out OmniContact motions (NU4D W2.5, ogDR W1.2, UP4s W1.2).
  • Sec. G and Table 17 (p. 29, L1530–1542): repeated real-robot trials of original and augmented references (ogDR W5).
  • Sec. H, Table 18 and Table 19 (p. 30, L1571–1611): held-out evaluation on a dexterous task (ogDR follow-up) and precision of the augmented references across rounds (UP4s follow-up).
  • Supplementary webpage:
    • 01.1: hardware demonstrations of references from later rounds, with the round of each (ogDR Q5).
    • 02.1: new dexterous retargeting examples (NU4D W1.2).
    • 02.4: retargeting with and without the finger opposition term (NU4D W1.1, ogDR W4.2).
    • 04.1: augmentation of a new object (UP4s Q2) and of fixed and relocated supports (ogDR W2.1).
    • 04.2: self-evolution versus one-shot augmentation (NU4D W2.1, ogDR W1.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.