- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work presents InterMimicGen, a data generation and verification framework for human-object interaction (HOI). The authors consolidate heterogeneous interaction captures, propose a two-stage dexterous whole-body retargeter to construct robot interaction references, and train an RL policy that serves as both tracker and verifier. Building on these components, the proposed ``self-evolving'' loop iteratively rolls out verified references and fine-tunes the tracker. Experiments show that the retargeter produces cleaner dexterous references than prior work, and that over five augmentation rounds the verified reference set grows by more than 100$\times$ while the evolved tracker executes ~98% of the generated references.
Critical strengths
The ablation studies demonstrate that the self-evolving loop expands executable whole-body dexterous interactions. Table 3 shows the tracker improves from 52-64% to 98% success on the generated references, which grow over 100X in five rounds. Table 11 further shows this growth depends on in-loop learning rather than filtering, as a frozen tracker yields only 0.4% versus 72.9% in round 2.
Critical weaknesses
The claimed capability growth is validated only within the generation loop itself. Accepted references are filtered by the execution success of the fine-tuned tracker and then included in its further training. However, essential validation steps are missing, such as testing the tracker on held-out objects or out-of-distribution editing operations, as well as comparing the performance of downstream policy learning when using augmented data versus using only the initial seed data.
Questions for authors
-
Is the "self-evolution improves the tracker" result a circular evaluation with train/test leakage? Can you report the evolved tracker's success on held-out objects or on edits beyond the training range, and a downstream policy trained on augmented data versus seed data alone?
-
Scalability of the flywheel. The hundredfold growth densifies around existing seeds along a few hand-designed edit axes, and per-round gains are already decelerating. It's unclear whether the loop scales or saturates. Can you characterize its asymptotic behavior, such as acceptance rate and cost over more rounds, and show whether it can expand into interaction types where seed coverage and tracking success are poor, rather than only thickening what already works?
5 comments
Thank you for the clarifications! I still have two minor questions: 1) the gains seem to plateau at iterations 6–8. Does this suggest the loop is beginning to saturate? 2) are there quantitative metrics showing how the humanoid motion style or domain changes across iterations?
Response to Reviewer UP4s’s Follow-up Questions
We sincerely thank the reviewer for the thoughtful follow-up questions and for the careful reading of our responses. We address the remaining questions below with new results.
1) Per-seed displacement saturates but the flywheel keeps scaling. We would like to first clarify what saturates: it is the displacement range around each seed; that is, how far its augmented references move away from the seed.
Why a bound exists. Each augmented reference keeps the starting point of its seed and its goal. The edits leave the motion before the first hand-object contact untouched and move the object only during contact, on the seed's own timing, and the augmented reference must still reach the same type of outcome, such as placement on a support (L238–240, Sec. D.2).
The more the motion must change within the same semantic, the more challenging it is to complete the interaction in time. This is a physical constraint. Saturation is therefore expected: it reflects the physical limit of each seed rather than a limit of the loop.
We further show that the accepted references keep the interaction in the original distribution (point 2 below).
Within the bound, the loop keeps densifying the distribution. Each round still adds more verified references than the one before.
2) Motion style and domain across rounds. Across rounds, the interaction stays within the distribution of the retargeted human captures, while the motion distribution shifts away from the original postures (Sec. H and Table 19 of the revision).
- Metric.
- We use precision (Kynkäänniemi et al., NeurIPS 2019), a standard metric for comparing generated data with a reference distribution. Here it is the share of each round's augmented references that fall inside the manifold of the retargeted human captures of the same object. The manifold is the union of balls centered at these captures, each with a radius equal to the distance to its third nearest neighbor.
- We compute this metric with the two feature definitions of Sec. E.1, evaluated on the robot trajectories. Body motion is described by the body keypoints relative to the pelvis and the pelvis trajectory, and interaction by the object's position relative to the pelvis and the wrists and by the object's own motion.
- The interaction keeps its semantics. The metric is 0.72–0.80 in every round with no downward trend, close to the 0.83 of the retargeted human captures themselves. This aligns with our intention: every accepted reference keeps the interaction type, intended contacts, and object of its seed (L238–240).
- The motion distribution shifts. The metric falls from 0.70 in round 1, the level of the retargeted human captures themselves, to 0.54 in round 8, showing the deviation from the original references. The augmentation is designed for this: it adds new ways of performing the same interaction (L234–235), and each accepted edit is executed successfully in simulation.
| Features | Retargeted human captures | Seeds | R1 | R2 | R3 | R4 | R5 | R6 | R7 | R8 |
|---|---|---|---|---|---|---|---|---|---|---|
| Interaction | 0.83 | 0.85 | 0.72 | 0.75 | 0.77 | 0.78 | 0.78 | 0.80 | 0.79 | 0.78 |
| Body motion | 0.71 | 0.77 | 0.70 | 0.69 | 0.67 | 0.64 | 0.64 | 0.56 | 0.54 | 0.54 |
| Body motion, spatial edits | – | – | 0.67 | 0.64 | 0.60 | 0.60 | 0.61 | 0.60 | 0.57 | 0.58 |
| Body motion, body edits | – | – | 0.73 | 0.73 | 0.73 | 0.68 | 0.68 | 0.51 | 0.51 | 0.49 |
Retargeted human captures: each capture tested against all the others (leave-one-out). Seeds: the original references before any round. R1–R8: the augmented references of each round. The last two rows split the augmented references by edit type, which does not apply to the retargeted human captures and the seeds (denoted as ‘–’).
We would greatly appreciate any other metrics the reviewer would find informative for measuring how the distribution changes, and we would be more than happy to compute and report them.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved Precision and Recall Metric for Assessing Generative Models. NeurIPS 2019.
Response to Reviewer UP4s
We thank the reviewer for the positive assessment and constructive feedback. We appreciate the recognition that the self-evolving loop expands executable whole-body dexterous interactions, and that this growth depends on in-loop learning rather than filtering.
W1: Validation beyond the generation loop. We agree that Table 3 measures the executability of the generated archive rather than generalization. Generalization holds beyond the loop: the tracker improves on edits beyond its training range, and on held-out objects, training with augmented data outperforms training on seed data alone. Per reviewer’s request, we present two additional evaluations.
W1.1: Edits beyond the training range. Beyond the loop, the evolved tracker learns interaction skills rather than memorizing its archive. Each round extends competence past the range it was trained on: trackers from rounds 1 and 2, which never trained on round-3 references, already execute far more than the original tracker, as shown below. This is exactly what allows the next round to push further.
W1.2: Held-out objects and downstream learning. The gain also transfers to held-out OmniContact motions, boxes of sizes never seen in training with motions never used in any round. Under matched compute (we train the original checkpoint longer to match the compute), the tracker trained with augmented data outperforms the one trained on seed data alone. The augmented data itself, not only the archive it is evaluated on, therefore improves downstream learning.
Q1: Circular evaluation. We thank the reviewer for the suggestion. We find the improvement holds on data the tracker was never trained on: edits beyond the training range (W1.1), and held-out objects from OmniContact, where training with augmented data outperforms training on seed data alone (W1.2).
Q2: Scalability of the flywheel. Per the reviewer's request, we run three more rounds. The archive keeps growing: each round adds more references than the one before at a lower cost per reference, and the acceptance rate declines only gradually. The displacement range, however, levels off after round 6, so later rounds mainly densify the coverage around each seed, likely within what the robot can physically execute. The hand-designed edit axes set only the directions of variation, and coverage is measured independently of them (L292).
Q2.2: Interaction types with poor seed coverage or tracking success. Per reviewer’s request, we run the loop on a trash can, an object outside the paper's experiments on which the original tracker succeeds on only 17.4% of clips. Starting from 11 seeds, the loop expands the executable references from 775 after round 1 to 10,434 after round 5, with a displacement p90 of 16.8 cm and 76.9% of executed candidates accepted (videos in 04.1 of the supplementary webpage, Grasp × Inspire, Motions 5–6), though the displacement is smaller than the augmentation with in-distributional trajectories.