acceptodds
Under review as a conference paper at ICLR 2027

Policy-as-Data: Learning Generalizable HOI Diffusion Models from Simulated Physics

Abstract

Synthesizing realistic Human-Object Interactions (HOI) is critical for creating embodied avatars and functional virtual environments. However, current data-driven approaches primarily rely on motion capture datasets, which are expensive to scale and limited in functional diversity. Models trained with these datasets struggle to generalize to unseen objects and maintain physical consistency over long horizons. In this paper, we propose a novel framework that leverages a physics simulator to overcome the data-scarcity bottleneck in HOI generation. Specifically, we propose a scalable pipeline, called \ours, which leverages policies trained with reinforcement learning in a physics simulator for task-oriented data generation and trains a generative model on the augmented dataset for generalizable HOI generation. To seamlessly utilize the synthetic data, we introduce a coarse-to-fine retargeting process that bridges the representation gap between the simplified model used in the physics simulator and the standard parametric body models required for generative training. Validated through comprehensive experiments, our method demonstrates improved generalization to unseen objects and supports long-horizon generation, while providing broader task and object coverage and improved contact consistency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human-Object Interaction, Motion Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_g6pkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Based on a fact that HOI generators (OMOMO, CHOIS) train on small motion capture datasets and overfit to a few object instances, the paper explores an idea that treat physics-based RL policies as an upstream data generator rather than a downstream corrector. It proposes a framework that rolls out task-specific RL experts (carry / sit / climb, AMP-style rewards, a TokenHSI-type humanoid in Isaac Gym) over randomized objects, retargets the rigid-body rollouts to SMPL with a learned transformer plus contact-aware optimization, and retrains the unchanged CHOIS diffusion model on synthetic sequences. On the evaluation side, gains on the headline benchmarks are small, the big long-horizon win is against a baseline that never saw the tasks, and no control separates physics data from more data.

Critical strengths

The paper shows promising positioning with good evaluation setup. I summarize the strengths of this paper as follows:

  1. Clean framing. The diffusion backbone is CHOIS unchanged, so Tables 2/4/5 isolate the effect of data (Sec. 4.1).
  2. The retargeting bridge is a real engineering need and is evaluated on its own with promising settings.
  3. The performance boosts point the same way across four evaluations with variance reported.
  4. Table 5 is the right ablation and is honest: the sim-only model collapses on real test.
  5. Long-horizon qualitative results (Fig. 5) show a kinematic model composing skills absent from OMOMO with stable contacts.
Critical weaknesses
  1. No data-matched control: physics data vs simply more data. Every comparison is CHOIS-on-OMOMO vs CHOIS-on-OMOMO +4.4K sim sequences. Nothing separates physics-grounded trajectories from any additional geometry-diverse data.
  2. Unseen-object protocol under-specified. Sec. 4.3 says the 10/5 instance split follows Li et al. (2024), but CHOIS's published unseen-object test used 17 3D-FUTURE meshes. Which 5 objects are held out is not stated.
  3. Novelty overstated relative to PARC and TokenHSI. PARC already uses physically based tracking policies to generate augmented training data for a kinematic generator in a loop, so the claim to be the "first" needs narrowing to task-oriented experts for HOI. The humanoid, task set (follow/carry/climb/sit), AMP rewards and object counts coincide with TokenHSI's splits, yet the appendix calls the tables/chairs "procedurally generated". If TokenHSI's released policies and assets were used, contribution 1 reduces to running an existing controller and the technical content is the retargeter.
  4. Synthetic diversity narrower than the narrative. Boxes vary only in [0.3, 0.4]m per side and sim text conditioning is 3 fixed templates with 3 task types.
  5. Missing baselines and discussions. GenHOI and Lou et al. 2025 (cited unseen-object HOI generators) not compared. InterMimic (CVPR 2025, physics-based whole-body HOI imitation) not cited.
Questions for authors

Besides the above weaknesses, I have the following questions for the authors:

  1. Were the RL experts and chair/table assets TokenHSI's released policies and 3D-FRONT objects?
  2. Which 5 OMOMO objects are held out in Table 2, and which category in Table 7? Do they share categories with D_sim? Why is P_hand ~4× lower for all methods on the unseen split?
  3. Were Table 4 baselines retrained or copied? Why is HOI-HFLI absent from Table 4, and why does OMOMO score F1 0.05 on unseen objects when it conditions on ground-truth object motion?
  4. Why does Fig. 6 saturate at 10% of D_sim? Does adding task types or object categories keep helping?
  5. How is sit/climb retargeting validated, given refinement touches only arm joints? Any pelvis/foot penetration numbers?
11 comments
AuthorsOpenReview ↗

Response to Reviewer g6pk (Part 1)

Weakness 1: No data-matched control: physics data vs simply more data. Every comparison is CHOIS-on-OMOMO vs CHOIS-on-OMOMO +4.4K sim sequences. Nothing separates physics-grounded trajectories from any additional geometry-diverse data.

Response: Thank you for suggesting this control. We conducted an additional experiment under the Table 2 protocol, augmenting OMOMO with real HOI data from BEHAVE rather than simulated trajectories. We selected table-, chair-, and box-related sequences from BEHAVE Date01 to align the object categories with our simulated data. This produced 578 additional training trajectories, covering five chair, two box, and two table instances.

To provide a comparable-scale control, we also evaluate augmentation using only 10% of our simulation data, approximately 440 sequences. This uses fewer additional sequences than the BEHAVE augmentation and tests whether the benefits persist at a similar data scale. All variants use the same CHOIS backbone, unseen-object split, and evaluation protocol, with other training settings unchanged.

Training dataCrecC_{\mathrm{rec}} ↑CprecC_{\mathrm{prec}} ↑CF1C_{\mathrm{F1}} ↑C%C_{\%}PhandP_{\mathrm{hand}} ↓MPJPE ↓TrootT_{\mathrm{root}} ↓TobjT_{\mathrm{obj}} ↓
OMOMO0.590.760.640.560.1316.5028.7814.29
OMOMO + BEHAVE (578 sequences)0.600.830.670.560.2216.4128.3215.14
OMOMO + 10% simulation (≈440 sequences)0.630.830.670.580.1416.3327.8214.45
OMOMO + full simulation (4.4K sequences)0.640.830.680.590.1416.1827.7614.24

Penetration and position errors are reported in cm.

Adding BEHAVE improves contact F1 from 0.64 to 0.67, confirming that additional real data also benefits unseen-object generalization. At a comparable augmentation scale, the 10% simulation subset achieves the same contact F1 and precision as BEHAVE, with higher recall (0.63 vs. 0.60), lower hand penetration (0.14 vs. 0.22 cm), and lower human and object motion errors than BEHAVE augmentation. Expanding to the full simulation dataset further improves contact recall and F1.

These results demonstrate the practical value of simulated supervision even with approximately 440 additional sequences. We have included the experimental setup and results in Appendix K, Table 9.


Weakness 2: Unseen-object protocol under-specified. Sec. 4.3 says the 10/5 instance split follows Li et al. (2024), but CHOIS's published unseen-object test used 17 3D-FUTURE meshes. Which 5 objects are held out is not stated.

Response: Thank you for catching this attribution error. We follow OMOMO’s object split (Li et al., 2023), rather than CHOIS’s 3D-FUTURE unseen-object protocol. Following OMOMO’s preprocessing, the multi-part objects vacuum (training) and mop (test) are excluded from the original 10/5 split, leaving nine training objects and four held-out test objects.

  • Training: largetable, woodchair, plasticbox, largebox, smallbox, trashcan, monitor, floorlamp, and clothesstand.
  • Held-out test: smalltable, whitechair, suitcase, and tripod.

The four held-out test instances are excluded from both real and simulated training data. We have corrected the attribution and effective split counts in Section 4.3 and provided the complete object lists and preprocessing details in Appendix C.2.

AuthorsOpenReview ↗

Response to Reviewer g6pk (Part 2)

Weakness 3: Novelty overstated relative to PARC and TokenHSI. PARC already uses physically based tracking policies to generate augmented training data for a kinematic generator in a loop, so the claim to be the "first" needs narrowing to task-oriented experts for HOI. The humanoid, task set (follow/carry/climb/sit), AMP rewards and object counts coincide with TokenHSI's splits, yet the appendix calls the tables/chairs "procedurally generated". If TokenHSI's released policies and assets were used, contribution 1 reduces to running an existing controller and the technical content is the retargeter.

Response: Thank you for highlighting these connections. PAD-HOI shares with PARC the broader idea of using physics-based policies to provide training data for a kinematic motion generator, and our expert-policy setup builds on TokenHSI. We have clarified the relationship to these works and the provenance of the reused policies and assets in the revised manuscript.

Relationship to PARC. The distinction concerns the role of the physics policies and the source of supervision. PARC uses physics-based tracking to improve motions produced by a generator for terrain traversal and feeds the resulting motions back into training. PAD-HOI uses task-oriented physics experts to supply human–object interaction examples that are missing or scarce in the target MoCap dataset. These interactions are converted into SMPL supervision and combined with real MoCap data to train a conditional HOI generator. Our contribution statement specifically refers to this setting: “To the best of our knowledge, we are the first to use task-oriented physics policies as an upstream source of training data for generative HOI synthesis.” We have made the relationship to PARC’s broader physics-based augmentation strategy more explicit in Section 2.

TokenHSI policies and asset sources. We use TokenHSI’s released policy checkpoints to collect interaction trajectories, with chair and table assets from 3D-FRONT following TokenHSI’s splits. We have corrected “procedurally generated” to distinguish these assets from the size-randomized boxes and clarified their provenance in Section 4.1 and Appendix C.2.

Technical contribution and demonstrated value. Our contribution is to establish task-oriented physics policies as a practical source of supervision for generative HOI synthesis, enabling skills available in physics controllers to supplement those represented in scarce MoCap data. Making this possible requires resolving the mismatch between reduced-body policy rollouts and SMPL motions with consistent surface contacts. Our learned reconstruction and contact-aware refinement provide this bridge: Table 1 shows that contact F1 increases from 0.13 to 0.83 over Mapping+IK, while refinement reduces hand penetration from 0.18 to 0.02 without degrading contact F1. The resulting supervision improves unseen-object HOI generation with an unchanged CHOIS backbone (Table 2) and extends the task coverage used in long-horizon generation (Table 3). Together, these results establish both the technical adaptation needed to use task-oriented policy data and its demonstrated value for training HOI generators.


Weakness 4: Synthetic diversity narrower than the narrative. Boxes vary only in [0.3, 0.4]m per side and sim text conditioning is 3 fixed templates with 3 task types.

Response: Thank you for raising this point. Our diversity claim primarily concerns object geometry and task execution. The box size range describes only one subset of the synthetic data. The carry subset also includes 30 distinct table and 90 distinct chair instances, as detailed in Appendix C.2. Thus, geometric variation extends beyond box scaling to different object instances and shapes.

We explicitly specify the three task types, carry, sit, and climb in Section 4.1. The fixed text templates serve as task annotations for conditioning, rather than as a source of linguistic diversity. We have clarified these distinctions in Appendix C.2 and corrected the asset descriptions to distinguish size randomized boxes from 3D-FRONT chair and table instances.

AuthorsOpenReview ↗

Response to Reviewer g6pk (Part 3)

Weakness 5: Missing baselines and discussions. GenHOI and Lou et al. 2025 (cited unseen-object HOI generators) not compared. InterMimic (CVPR 2025, physics-based whole-body HOI imitation) not cited.

Response: Thank you for pointing out these relevant works. We have added GenHOI as a baseline in Section 4.3, Table 2, introduced cross-generator augmentation ablations in Section 4.6, Table 6, and added a discussion of InterMimic in Section 2.

GenHOI comparison and augmentation ablation. Table 2 now includes GenHOI under the same unseen-object evaluation protocol. To further examine whether the benefits of our simulated supervision extend beyond CHOIS, the new Section 4.6 (Ablation Studies) compares generators trained with and without simulation augmentation. Within each generator, the architecture and other training settings remain unchanged, only the training data differs. All augmented models use the same additional simulation dataset.

The CHOIS and GenHOI results from Table 6 are reproduced below. Both use the unseen-object protocol of Table 2.


GeneratorTraining dataCrecC_{\mathrm{rec}} ↑CprecC_{\mathrm{prec}} ↑CF1C_{\mathrm{F1}} ↑C%C_{\%}PhandP_{\mathrm{hand}} ↓MPJPE ↓TrootT_{\mathrm{root}} ↓TobjT_{\mathrm{obj}} ↓OobjO_{\mathrm{obj}} ↓
CHOISOMOMO0.590.760.640.560.1316.5028.7814.291.04
CHOISOMOMO + simulation0.640.830.680.590.1416.1827.7614.241.03
GenHOIOMOMO0.620.790.670.580.1816.0026.7514.981.20
GenHOIOMOMO + simulation0.680.860.710.600.1916.0626.4514.881.21

For GenHOI, simulation augmentation improves contact recall from 0.62 to 0.68, precision from 0.79 to 0.86, and F1 from 0.67 to 0.71. Root and object translation errors also decrease. Together with the CHOIS results, this demonstrates that the contact-generation benefits extend beyond a single backbone. Section 4.6 additionally evaluates HOI-HFLI under the standard protocol of Table 4, showing improved motion accuracy with simulation augmentation.

InterMimic discussion. We recognize InterMimic as an influential milestone in physics-based whole-body HOI imitation and appreciate its valuable contribution to the community. Its omission from the original submission was an oversight. We have now added the citation and discussed its relationship to our work in Section 2, including its use of reference human–object motions to learn interaction control and our use of task-oriented physics policies to provide training data for HOI generation.

Comparison with Lou et al. Lou et al. also addresses MoCap data scarcity, using an image generation model to obtain HOI data. This provides a complementary data source to our policy-generated interactions. Its reported evaluation does not directly match our Table 2 protocol, and its code was not publicly available when we conducted the experiments. We therefore provide a qualitative comparison here, while the added GenHOI experiments provide a quantitative comparison under our evaluation protocol.


Question 1: Were the RL experts and chair/table assets TokenHSI's released policies and 3D-FRONT objects?

Response: Yes. We use TokenHSI’s released policy checkpoints to collect interaction trajectories, and the chair/table assets are 3D-FRONT objects following TokenHSI’s splits. We have corrected the “procedurally generated” description and clarified these sources in Section 4.1 and Appendix C.2. As detailed in our response to Weakness 3, our contribution is to make policy-generated interactions usable as training supervision for generative HOI models and empirically demonstrate their value alongside real MoCap data.

Reviewer_ehonOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

PAD-HOI aims to generate HOI motion data using physics-based RL policies and combine the resulting synthetic data with existing mocap data to train an HOI diffusion model. To this end, the method introduces a coarse-to-fine retargeting scheme that converts reduced rigid-body motions into SMPL representations. The authors report that this data augmentation improves generalization to unseen objects and generation performance on long-horizon, multi-task sequences.

Critical strengths

S1. The authors propose a practical approach to addressing the limited coverage of existing mocap datasets by generating HOI motions using physics-based RL. The proposed method is interesting and easy to follow.

S2. The coarse-to-fine retargeting method achieves high contact consistency, while the contact refinement stage effectively reduces hand-object penetration.

S3. Extensive experiments, including unseen-object generalization, real/sim data ablations, and synthetic data scaling, demonstrate the effectiveness of the proposed data augmentation approach.

Critical weaknesses

W1. Insufficient justification for the diffusion-based generation stage. Existing physics-based RL controllers can already generate various interaction motions with objects of different sizes. The authors leverage these controllers to obtain physically plausible motion sequences, which are then retargeted from a reduced rigid-body representation to SMPL. However, the benefit of training a diffusion model to generate full motion sequences from these data remains unclear. In particular, motions generated by the diffusion model may be less physically stable than the original controller outputs. A direct comparison between “RL + SMPL retargeting” and “diffusion generation” would therefore help clarify the added value of the diffusion stage.

W2. Unclear waypoint definitions for Sit/Climb tasks. While the original CHOIS uses object-centric waypoints, PAD-HOI evaluates goal success based on the human root position in its long-horizon evaluation. However, it remains unclear how waypoints are defined for the Sit and Climb tasks in both the authors' implementation of CHOIS and PAD-HOI, given that the interacting objects (e.g., chairs and platform boxes) remain stationary during these interactions.

W3. Insufficient contact handling and physical evaluation for Sit/Climb tasks. The proposed contact-aware retargeting mainly focuses on hand-object contact and upper-body motion, significantly improving hand-contact and hand-penetration metrics. However, Sit and Climb account for half of the tasks in the long-horizon evaluation and rely primarily on pelvis-object and foot-object contact rather than hand contact. Therefore, the authors should clarify how PAD-HOI handles these contacts and report the body-penetration score PbodyP_{body} to evaluate physical plausibility beyond hand-object interactions.

Questions for authors

Please refer to the concerns raised in the weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ehon (Part 1)

Question 1: Insufficient justification for the diffusion-based generation stage. Existing physics-based RL controllers can already generate various interaction motions with objects of different sizes. The authors leverage these controllers to obtain physically plausible motion sequences, which are then retargeted from a reduced rigid-body representation to SMPL. However, the benefit of training a diffusion model to generate full motion sequences from these data remains unclear. In particular, motions generated by the diffusion model may be less physically stable than the original controller outputs. A direct comparison between “RL + SMPL retargeting” and “diffusion generation” would therefore help clarify the added value of the diffusion stage.

Response: Thank you for raising this question. The diffusion stage provides a unified conditional generator that learns from both policy-generated interactions and real MoCap data. The task-specific RL controllers supply interaction examples, while PAD-HOI combines these examples with real human motions in a shared distribution conditioned on text, object geometry, initial states, and waypoints. Table 5 demonstrates the complementary value of real and simulated supervision.

Following your suggestion, we have added a direct comparison between RL + retargeting and PAD-HOI diffusion generation in Appendix J and Figure 7, using an OMOMO floorlamp-carrying task. We execute TokenHSI’s released multitask checkpoint in carry mode and convert its output using our coarse retargeter, preserving the simulated pelvis and object trajectories. PAD-HOI is conditioned on the same object geometry, the first retargeted human–object state, a carrying instruction, and an object-waypoint plan with the same destination.

In the RL rollout, the lamp falls during pickup and remains far from the target, with a terminal object-position error of 3.513 m. Three PAD-HOI diffusion samples under the same conditions achieve errors of 0.033, 0.019, and 0.044 m, respectively. Figure 7 shows the second sample, selected for its highest hand–object proximity while the lamp is elevated. This example illustrates the practical value of the learned generator: PAD-HOI produces carrying sequences in a case where direct execution of the selected controller followed by retargeting fails. Together with Table 5, this supports using policy-generated interactions as complementary training supervision for a generative model.

We agree that diffusion generation does not automatically preserve the physical stability of controller trajectories. Goal proximity alone does not establish physically valid manipulation, and the diffusion outputs in this comparison are not replayed in the physics simulator. We therefore present this as an illustrative task-completion comparison, rather than evidence of superior physical stability or aggregate gains in success rate or diversity. Physical quality is evaluated separately through the reported interaction metrics, including the additional pelvis, foot, and whole-body penetration analyses in Appendix L.

AuthorsOpenReview ↗

Response to Reviewer ehon (Part 2)

Question 2: Unclear waypoint definitions for Sit/Climb tasks. While the original CHOIS uses object-centric waypoints, PAD-HOI evaluates goal success based on the human root position in its long-horizon evaluation. However, it remains unclear how waypoints are defined for the Sit and Climb tasks in both the authors’ implementation of CHOIS and PAD-HOI, given that the interacting objects (e.g., chairs and platform boxes) remain stationary during these interactions.

Response: Thank you for raising this question. Both our implementation of CHOIS and PAD-HOI use sparse object waypoints as conditioning for all tasks. For Sit and Climb, these waypoints specify a constant object position, since the chair or box should remain stationary.

The evaluation distinguishes human-root goal success from object-waypoint adherence:

  • Human-root goal success (SfinalS_{\mathrm{final}}): This measures the proportion of rollouts in which the human root reaches within 10 cm of its final target at sequence end. For Sit, this target is located 14 cm above the geometric center of the chair’s seat. For Climb, it is located 95 cm above the box’s top surface, corresponding to the standing root height of the rendered character. These human-root targets are used only for evaluation and are not supplied as conditioning to either model.
  • Object-waypoint adherence (SwpS_{\mathrm{wp}}): This measures the proportion of rollouts in which the object reaches all intermediate object waypoints within 10 cm at the corresponding stage boundaries. It does not measure human-root positions. For stationary-object tasks, it assesses whether the object remains at its prescribed location at these boundaries. This is relevant because the baseline can generate unintended object floating or displacement even under constant object-waypoint conditions.

We have clarified the conditioning and evaluation distinction in Section 4.3 and provided the target definitions and metric computations in Appendix F, explicitly distinguishing object-waypoint conditioning, human-root goal success, and object-waypoint adherence.

AuthorsOpenReview ↗

Response to Reviewer ehon (Part 3)

Question 3: Insufficient contact handling and physical evaluation for Sit/Climb tasks. The proposed contact-aware retargeting mainly focuses on hand-object contact and upper-body motion, significantly improving hand-contact and hand-penetration metrics. However, Sit and Climb account for half of the tasks in the long-horizon evaluation and rely primarily on pelvis-object and foot-object contact rather than hand contact. Therefore, the authors should clarify how PAD-HOI handles these contacts and report the body-penetration score to evaluate physical plausibility beyond hand-object interactions.

Response: Thank you for highlighting the need to evaluate contacts beyond the hands. Our contact-aware refinement explicitly optimizes upper-body joints for hand–object contact. Pelvis and foot contacts are transferred from the simulated trajectories through the coarse retargeter, without dedicated contact refinement. We have clarified this distinction in Section 3.2.

We now evaluate physical quality beyond hand interactions at both the retargeting and final generation stages. For retargeting, Appendix L, Table 10 reports results on 100 retargeted simulation sequences per task. Contact-conditioned pelvis penetration for Sit is 0.1944 cm, and foot penetration for Climb is 0.9346 cm, directly assessing the quality of the retargeted training trajectories.

For final generation, we compare CHOIS and PAD-HOI under the same evaluation conditions using 100 Climb and 100 Sit sequences per model. Following the same penetration computation used for PhandP_{\mathrm{hand}}, we compute PfootP_{\mathrm{foot}} for Climb, PpelvisP_{\mathrm{pelvis}} for Sit, and Pwhole bodyP_{\mathrm{whole\ body}} for both tasks. The corresponding vertex sets contain 502, 164, and 10,475 vertices, respectively.

We report both the all-frame mean and the contact-conditioned mean. The latter averages the same per-frame penetration measure only over frames in which at least one vertex in the evaluated region is within 2 cm of the object surface or penetrates it. We also report the percentage of such contact frames.

TaskMetric ↓MethodAll-frame mean (cm)Contact-conditioned mean (cm)Contact frames (%)
ClimbPfootP_{\mathrm{foot}}CHOIS3.0807.46041.26
ClimbPfootP_{\mathrm{foot}}PAD-HOI0.2680.86530.95
ClimbPwhole bodyP_{\mathrm{whole\ body}}CHOIS2.2004.32051.01
ClimbPwhole bodyP_{\mathrm{whole\ body}}PAD-HOI0.0420.08847.64
SitPpelvisP_{\mathrm{pelvis}}CHOIS0.000*N/A*0.00
SitPpelvisP_{\mathrm{pelvis}}PAD-HOI0.0350.18219.03
SitPwhole bodyP_{\mathrm{whole\ body}}CHOIS1.1503.47033.20
SitPwhole bodyP_{\mathrm{whole\ body}}PAD-HOI0.0110.03728.77

For Sit, CHOIS produces no pelvis–object contact frames, so its contact-conditioned pelvis penetration is undefined (N/A). Its zero all-frame pelvis penetration reflects the absence of the intended contact rather than successful sitting. Contact-frame percentages are descriptive statistics, not metrics to maximize.

For Climb, PAD-HOI reduces contact-conditioned foot penetration from 7.460 to 0.865 cm and whole-body penetration from 4.320 to 0.088 cm. For Sit, contact-conditioned whole-body penetration decreases from 3.470 to 0.037 cm. Qualitative inspection shows that CHOIS approaches the chair and performs grasping-like motions rather than sitting, explaining its zero pelvis-contact rate. PAD-HOI establishes pelvis–object contact, with contact-conditioned pelvis penetration of 0.182 cm. These results highlight why penetration must be interpreted together with the intended interaction: avoiding contact can yield zero penetration without completing the task.

We have included these evaluations in Appendix L, Tables 10 and 11, with a reference in Section 4.3, providing direct measurements of penetration beyond hand contacts in both the retargeted training trajectories and final generated motions.

I appreciate the authors' additional experiments and clarifications, which have addressed some of my concerns.

I agree that PAD-HOI can serve as a useful framework for integrating simulated RL trajectories with real MoCap data. The floorlamp example supports this perspective, as the released TokenHSI policy was not specifically trained for floorlamp carrying, whereas such interactions are included in the diffusion model's training data.

However, this comparison is double-edged. While it highlights the benefit of data integration, it does not necessarily justify the use of diffusion over RL + retargeting, since the two methods have different training distributions for the evaluated task. In commonly trained tasks such as box carrying, this discrepancy would not exist.

Moreover, data integration is not exclusive to diffusion models. Goal-conditioned RL methods using AMP can also incorporate real MoCap references, potentially with relatively few motion clips, although learning new tasks may require additional policy training. Therefore, the advantage of using diffusion rather than incorporating real MoCap into physics-based controllers remains unclear, particularly considering the potential trade-off in physical consistency.

For these reasons, I maintain my original rating.

Reviewer_iBcEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper leverages a physics simulator to overcome the data-scarcity bottleneck in HOI generation. Trajectories are generated in the simulator, and then retargeted to SMPL format to be used as training data.

Critical strengths
  • The paper introduces more simulated data to train HOI model to improve generalization. Processes are introduced to make sure the simulated data could be used along with the real data to train HOI model.
Critical weaknesses
  • The contribution is in using simulated data for model training. This contribution is quite limited as retargeting is not new for this purpose and the advantage of the proposed one is not well demonstrated in this paper.
  • The description of the long-horizon multi-task generation and the associated evaluation setting are not clear. This makes the results in Table 3 inconvincible.
Questions for authors
  • I was not able to locate (a), (b), and (c) in Figure 2 though they are described in the caption.
  • It is unclear how adding data without changing the model could make a model 'supports' long-horizon generation, assuming the original model does not support such generation.
2 comments
AuthorsOpenReview ↗

Response to Reviewer iBcE

Weakness 1: The contribution is in using simulated data for model training. This contribution is quite limited as retargeting is not new for this purpose and the advantage of the proposed one is not well demonstrated in this paper.

Response: We thank the reviewer for raising this concern. We agree that simulation-based augmentation and motion retargeting are established techniques. However, our contribution is not motion retargeting in isolation. To the best of our knowledge, we are the first to use task-oriented physics policies as a source of training data for generative HOI synthesis. Realizing this requires bridging a representation gap between reduced-body physics controllers and SMPL-based HOI generators while preserving human–object contacts. Our contribution therefore includes both introducing this source of supervision and developing the adaptation needed to make it usable for HOI training.

Table 1 directly evaluates the benefit over Mapping+IK: learned reconstruction increases contact F1 from 0.13 to 0.83, and subsequent refinement reduces hand penetration from 0.18 to 0.02 without reducing contact F1. These results show that the proposed adaptation substantially improves contact fidelity over the evaluated mapping-based baseline. Table 2 establishes its practical downstream value: adding the resulting simulated interactions to the training data, while keeping the CHOIS backbone unchanged, improves unseen-object contact recall from 0.59 to 0.64, precision from 0.76 to 0.83, and F1 from 0.64 to 0.68.


Weakness 2: The description of the long-horizon multi-task generation and the associated evaluation setting are not clear. This makes the results in Table 3 inconvincible.

Response: Thank you for raising this concern. We acknowledge that the original main-text description did not make the long-horizon generation procedure and evaluation setting sufficiently clear. We have expanded the generation description in Section 3.3 and the evaluation discussion in Section 4.3, with detailed protocols provided in Appendix F.

The long-horizon sequences combine navigation, carrying, climbing, and sitting. Both CHOIS and PAD-HOI use the same autoregressive sliding-window procedure, with 120-frame windows and approximately 10 frames of overlap. Within each task, overlapping human–object states provide constraints for the next window; when assembling the sequence, we retain the current window’s output in the overlapping region. At each task transition, the preceding task’s terminal human pose supplies the next task’s first-frame human condition, while the object state, object geometry, text instruction, and sparse object waypoints are updated for the next task.

For evaluation, we generate 400 rollouts in total across three random seeds, with an average sequence length of 600 frames. Table 3 reports the mean and standard deviation of the three per-seed aggregate results. In addition to the motion and interaction metrics used for short sequences, we evaluate goal-reaching success and transition-specific errors. Specifically:

  • SfinalS_{\mathrm{final}} measures the proportion of rollouts whose human pelvis reaches within 10 cm of the final target.
  • SwpS_{\mathrm{wp}} measures the proportion in which the object reaches all intermediate object waypoints within 10 cm at the corresponding stage boundaries.
  • MPJPEtrans\mathrm{MPJPE}_{\mathrm{trans}} measures mean per-joint position error relative to retargeted RL expert reference motions within a 20-frame window centered on each task transition.
  • TobjtransT_{\mathrm{obj}}^{\mathrm{trans}} measures the Euclidean distance between generated and reference object positions over the same intervals.

These revisions clarify the generation mechanism in Section 3.3, the evaluation setting and metric definitions accompanying Table 3 in Section 4.3, and the window assembly, reference-motion construction, and statistical reporting protocol in Appendix F.


Question 1. I was not able to locate (a), (b), and (c) in Figure 2 though they are described in the caption.

Response: Thank you for spotting this inconsistency. The labels (a), (b), and (c) in the caption correspond to stages (1), (2), and (3) in Figure 2, respectively. We have corrected the caption to match the labels in the figure.

AuthorsOpenReview ↗

Response to Reviewer iBcE (Continued)

Question 2. It is unclear how adding data without changing the model could make a model 'supports' long-horizon generation, assuming the original model does not support such generation.

Response: Thank you for raising this point. We agree that adding training data does not itself introduce a new mechanism for long-horizon generation. As described in Section 3.3 and Appendix F, we use the sliding-window mechanism already supported by the baseline, without changing its network architecture. However, the ability to generate successive windows does not ensure successful execution of the tasks within them. The additional tasks considered here are absent from or sparsely represented in the original training data, limiting the baseline’s performance when these tasks are composed into long-horizon sequences.

Our contribution is to address this training-data limitation. The simulated interactions provide additional contact-rich supervision for carrying, climbing, and sitting, improving task coverage and generation quality under the same long-horizon generation mechanism. Table 3 and Figure 5 demonstrate the resulting improvements in long-horizon task completion and transition quality.

We have clarified these complementary roles in Sections 3.3 and 4.3, and Appendix F: the sliding-window mechanism constructs long sequences, while the additional training supervision provides task-specific interaction examples that improve multi-task generation quality, as demonstrated in Table 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewers and Summary of Revisions

We sincerely thank the reviewers and the AC for their time and careful consideration. We appreciate the reviewers’ recognition of our practical approach to enriching limited MoCap coverage with simulated HOI data (iBcE, ehon), the effectiveness of our retargeting bridge in preserving contacts and reducing hand–object penetration (ehon, g6pk), and the extensive evaluations supporting simulation-based augmentation (ehon, g6pk). We also appreciate the recognition of our controlled evaluation with an unchanged generation backbone and the promising composition of skills absent from OMOMO (g6pk).

Following the reviewers’ suggestions, we have expanded the experiments with a comparable-scale comparison of real and simulated data augmentation, evaluations on additional generator backbones, and pelvis, foot, and whole-body penetration analyses for both retargeted training trajectories and final generated motions. We have also added an illustrative comparison between direct RL execution followed by retargeting and diffusion generation to clarify the role of the diffusion stage. In addition, we have clarified the long-horizon generation and evaluation protocols, the unseen-object split, and the sources of our physics controllers and object assets. Finally, we have expanded the related-work discussion, corrected the OMOMO results and associated comparisons, and clarified the scope of our approach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.