- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Based on a fact that HOI generators (OMOMO, CHOIS) train on small motion capture datasets and overfit to a few object instances, the paper explores an idea that treat physics-based RL policies as an upstream data generator rather than a downstream corrector. It proposes a framework that rolls out task-specific RL experts (carry / sit / climb, AMP-style rewards, a TokenHSI-type humanoid in Isaac Gym) over randomized objects, retargets the rigid-body rollouts to SMPL with a learned transformer plus contact-aware optimization, and retrains the unchanged CHOIS diffusion model on synthetic sequences. On the evaluation side, gains on the headline benchmarks are small, the big long-horizon win is against a baseline that never saw the tasks, and no control separates physics data from more data.
Critical strengths
The paper shows promising positioning with good evaluation setup. I summarize the strengths of this paper as follows:
- Clean framing. The diffusion backbone is CHOIS unchanged, so Tables 2/4/5 isolate the effect of data (Sec. 4.1).
- The retargeting bridge is a real engineering need and is evaluated on its own with promising settings.
- The performance boosts point the same way across four evaluations with variance reported.
- Table 5 is the right ablation and is honest: the sim-only model collapses on real test.
- Long-horizon qualitative results (Fig. 5) show a kinematic model composing skills absent from OMOMO with stable contacts.
Critical weaknesses
- No data-matched control: physics data vs simply more data. Every comparison is CHOIS-on-OMOMO vs CHOIS-on-OMOMO +4.4K sim sequences. Nothing separates physics-grounded trajectories from any additional geometry-diverse data.
- Unseen-object protocol under-specified. Sec. 4.3 says the 10/5 instance split follows Li et al. (2024), but CHOIS's published unseen-object test used 17 3D-FUTURE meshes. Which 5 objects are held out is not stated.
- Novelty overstated relative to PARC and TokenHSI. PARC already uses physically based tracking policies to generate augmented training data for a kinematic generator in a loop, so the claim to be the "first" needs narrowing to task-oriented experts for HOI. The humanoid, task set (follow/carry/climb/sit), AMP rewards and object counts coincide with TokenHSI's splits, yet the appendix calls the tables/chairs "procedurally generated". If TokenHSI's released policies and assets were used, contribution 1 reduces to running an existing controller and the technical content is the retargeter.
- Synthetic diversity narrower than the narrative. Boxes vary only in [0.3, 0.4]m per side and sim text conditioning is 3 fixed templates with 3 task types.
- Missing baselines and discussions. GenHOI and Lou et al. 2025 (cited unseen-object HOI generators) not compared. InterMimic (CVPR 2025, physics-based whole-body HOI imitation) not cited.
Questions for authors
Besides the above weaknesses, I have the following questions for the authors:
- Were the RL experts and chair/table assets TokenHSI's released policies and 3D-FRONT objects?
- Which 5 OMOMO objects are held out in Table 2, and which category in Table 7? Do they share categories with D_sim? Why is P_hand ~4× lower for all methods on the unseen split?
- Were Table 4 baselines retrained or copied? Why is HOI-HFLI absent from Table 4, and why does OMOMO score F1 0.05 on unseen objects when it conditions on ground-truth object motion?
- Why does Fig. 6 saturate at 10% of D_sim? Does adding task types or object categories keep helping?
- How is sit/climb retargeting validated, given refinement touches only arm joints? Any pelvis/foot penetration numbers?
11 comments
Response to Reviewer g6pk (Part 2)
Weakness 3: Novelty overstated relative to PARC and TokenHSI. PARC already uses physically based tracking policies to generate augmented training data for a kinematic generator in a loop, so the claim to be the "first" needs narrowing to task-oriented experts for HOI. The humanoid, task set (follow/carry/climb/sit), AMP rewards and object counts coincide with TokenHSI's splits, yet the appendix calls the tables/chairs "procedurally generated". If TokenHSI's released policies and assets were used, contribution 1 reduces to running an existing controller and the technical content is the retargeter.
Response: Thank you for highlighting these connections. PAD-HOI shares with PARC the broader idea of using physics-based policies to provide training data for a kinematic motion generator, and our expert-policy setup builds on TokenHSI. We have clarified the relationship to these works and the provenance of the reused policies and assets in the revised manuscript.
Relationship to PARC. The distinction concerns the role of the physics policies and the source of supervision. PARC uses physics-based tracking to improve motions produced by a generator for terrain traversal and feeds the resulting motions back into training. PAD-HOI uses task-oriented physics experts to supply human–object interaction examples that are missing or scarce in the target MoCap dataset. These interactions are converted into SMPL supervision and combined with real MoCap data to train a conditional HOI generator. Our contribution statement specifically refers to this setting: “To the best of our knowledge, we are the first to use task-oriented physics policies as an upstream source of training data for generative HOI synthesis.” We have made the relationship to PARC’s broader physics-based augmentation strategy more explicit in Section 2.
TokenHSI policies and asset sources. We use TokenHSI’s released policy checkpoints to collect interaction trajectories, with chair and table assets from 3D-FRONT following TokenHSI’s splits. We have corrected “procedurally generated” to distinguish these assets from the size-randomized boxes and clarified their provenance in Section 4.1 and Appendix C.2.
Technical contribution and demonstrated value. Our contribution is to establish task-oriented physics policies as a practical source of supervision for generative HOI synthesis, enabling skills available in physics controllers to supplement those represented in scarce MoCap data. Making this possible requires resolving the mismatch between reduced-body policy rollouts and SMPL motions with consistent surface contacts. Our learned reconstruction and contact-aware refinement provide this bridge: Table 1 shows that contact F1 increases from 0.13 to 0.83 over Mapping+IK, while refinement reduces hand penetration from 0.18 to 0.02 without degrading contact F1. The resulting supervision improves unseen-object HOI generation with an unchanged CHOIS backbone (Table 2) and extends the task coverage used in long-horizon generation (Table 3). Together, these results establish both the technical adaptation needed to use task-oriented policy data and its demonstrated value for training HOI generators.
Weakness 4: Synthetic diversity narrower than the narrative. Boxes vary only in [0.3, 0.4]m per side and sim text conditioning is 3 fixed templates with 3 task types.
Response: Thank you for raising this point. Our diversity claim primarily concerns object geometry and task execution. The box size range describes only one subset of the synthetic data. The carry subset also includes 30 distinct table and 90 distinct chair instances, as detailed in Appendix C.2. Thus, geometric variation extends beyond box scaling to different object instances and shapes.
We explicitly specify the three task types, carry, sit, and climb in Section 4.1. The fixed text templates serve as task annotations for conditioning, rather than as a source of linguistic diversity. We have clarified these distinctions in Appendix C.2 and corrected the asset descriptions to distinguish size randomized boxes from 3D-FRONT chair and table instances.
Response to Reviewer g6pk (Part 3)
Weakness 5: Missing baselines and discussions. GenHOI and Lou et al. 2025 (cited unseen-object HOI generators) not compared. InterMimic (CVPR 2025, physics-based whole-body HOI imitation) not cited.
Response: Thank you for pointing out these relevant works. We have added GenHOI as a baseline in Section 4.3, Table 2, introduced cross-generator augmentation ablations in Section 4.6, Table 6, and added a discussion of InterMimic in Section 2.
GenHOI comparison and augmentation ablation. Table 2 now includes GenHOI under the same unseen-object evaluation protocol. To further examine whether the benefits of our simulated supervision extend beyond CHOIS, the new Section 4.6 (Ablation Studies) compares generators trained with and without simulation augmentation. Within each generator, the architecture and other training settings remain unchanged, only the training data differs. All augmented models use the same additional simulation dataset.
The CHOIS and GenHOI results from Table 6 are reproduced below. Both use the unseen-object protocol of Table 2.
| Generator | Training data | ↑ | ↑ | ↑ | ↓ | MPJPE ↓ | ↓ | ↓ | ↓ | |
|---|---|---|---|---|---|---|---|---|---|---|
| CHOIS | OMOMO | 0.59 | 0.76 | 0.64 | 0.56 | 0.13 | 16.50 | 28.78 | 14.29 | 1.04 |
| CHOIS | OMOMO + simulation | 0.64 | 0.83 | 0.68 | 0.59 | 0.14 | 16.18 | 27.76 | 14.24 | 1.03 |
| GenHOI | OMOMO | 0.62 | 0.79 | 0.67 | 0.58 | 0.18 | 16.00 | 26.75 | 14.98 | 1.20 |
| GenHOI | OMOMO + simulation | 0.68 | 0.86 | 0.71 | 0.60 | 0.19 | 16.06 | 26.45 | 14.88 | 1.21 |
For GenHOI, simulation augmentation improves contact recall from 0.62 to 0.68, precision from 0.79 to 0.86, and F1 from 0.67 to 0.71. Root and object translation errors also decrease. Together with the CHOIS results, this demonstrates that the contact-generation benefits extend beyond a single backbone. Section 4.6 additionally evaluates HOI-HFLI under the standard protocol of Table 4, showing improved motion accuracy with simulation augmentation.
InterMimic discussion. We recognize InterMimic as an influential milestone in physics-based whole-body HOI imitation and appreciate its valuable contribution to the community. Its omission from the original submission was an oversight. We have now added the citation and discussed its relationship to our work in Section 2, including its use of reference human–object motions to learn interaction control and our use of task-oriented physics policies to provide training data for HOI generation.
Comparison with Lou et al. Lou et al. also addresses MoCap data scarcity, using an image generation model to obtain HOI data. This provides a complementary data source to our policy-generated interactions. Its reported evaluation does not directly match our Table 2 protocol, and its code was not publicly available when we conducted the experiments. We therefore provide a qualitative comparison here, while the added GenHOI experiments provide a quantitative comparison under our evaluation protocol.
Question 1: Were the RL experts and chair/table assets TokenHSI's released policies and 3D-FRONT objects?
Response: Yes. We use TokenHSI’s released policy checkpoints to collect interaction trajectories, and the chair/table assets are 3D-FRONT objects following TokenHSI’s splits. We have corrected the “procedurally generated” description and clarified these sources in Section 4.1 and Appendix C.2. As detailed in our response to Weakness 3, our contribution is to make policy-generated interactions usable as training supervision for generative HOI models and empirically demonstrate their value alongside real MoCap data.
Response to Reviewer g6pk (Part 1)
Weakness 1: No data-matched control: physics data vs simply more data. Every comparison is CHOIS-on-OMOMO vs CHOIS-on-OMOMO +4.4K sim sequences. Nothing separates physics-grounded trajectories from any additional geometry-diverse data.
Response: Thank you for suggesting this control. We conducted an additional experiment under the Table 2 protocol, augmenting OMOMO with real HOI data from BEHAVE rather than simulated trajectories. We selected table-, chair-, and box-related sequences from BEHAVE Date01 to align the object categories with our simulated data. This produced 578 additional training trajectories, covering five chair, two box, and two table instances.
To provide a comparable-scale control, we also evaluate augmentation using only 10% of our simulation data, approximately 440 sequences. This uses fewer additional sequences than the BEHAVE augmentation and tests whether the benefits persist at a similar data scale. All variants use the same CHOIS backbone, unseen-object split, and evaluation protocol, with other training settings unchanged.
Penetration and position errors are reported in cm.
Adding BEHAVE improves contact F1 from 0.64 to 0.67, confirming that additional real data also benefits unseen-object generalization. At a comparable augmentation scale, the 10% simulation subset achieves the same contact F1 and precision as BEHAVE, with higher recall (0.63 vs. 0.60), lower hand penetration (0.14 vs. 0.22 cm), and lower human and object motion errors than BEHAVE augmentation. Expanding to the full simulation dataset further improves contact recall and F1.
These results demonstrate the practical value of simulated supervision even with approximately 440 additional sequences. We have included the experimental setup and results in Appendix K, Table 9.
Weakness 2: Unseen-object protocol under-specified. Sec. 4.3 says the 10/5 instance split follows Li et al. (2024), but CHOIS's published unseen-object test used 17 3D-FUTURE meshes. Which 5 objects are held out is not stated.
Response: Thank you for catching this attribution error. We follow OMOMO’s object split (Li et al., 2023), rather than CHOIS’s 3D-FUTURE unseen-object protocol. Following OMOMO’s preprocessing, the multi-part objects vacuum (training) and mop (test) are excluded from the original 10/5 split, leaving nine training objects and four held-out test objects.
The four held-out test instances are excluded from both real and simulated training data. We have corrected the attribution and effective split counts in Section 4.3 and provided the complete object lists and preprocessing details in Appendix C.2.