acceptodds
Under review as a conference paper at ICLR 2027

TriFactorHOI: Structured Diffusion for Controllable and Contact–Consistent Human–Object Interaction Synthesis

15 views

Abstract

Recent diffusion models have advanced human–object interaction (HOI) generation. However, generating coordinated human–object motion from sparse waypoints while following user-specified routes and maintaining stable contacts remains challenging. We propose TriFactorHOI, a structured diffusion framework that organizes HOI generation through object path, part, and contact factors. For trajectory control, a reference path interpolated from sparse waypoints serves both as a generation condition and as the baseline for planar object translation, with learned residuals accommodating interaction-related deviations. A part-based spatiotemporal denoiser preserves separate temporal and part axes to model within-part dynamics and cross-part human–object coordination. Hand–object and foot–ground guidance refine contacts during denoising. After sampling, a full-sequence foot-refinement network predicts support probabilities and correction confidence for world-space foot anchoring and temporal inverse kinematics. Experiments on FullBodyManipulation demonstrate improved waypoint adherence, motion-distribution quality, and contact stability. With the reference-path condition held fixed, residual prediction reduces waypoint error and FID relative to absolute planar translation. Tests on unseen 3D-FUTURE objects without retraining further support generalization to new object geometries. The code and processed datasets will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human–Object Interaction, Diffusion Models, Physically Plausible Motion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4tGtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents TriFactorHOI, a structured diffusion framework for generating coordinated human–object motion conditioned on text, object geometry, initial states, and sparse object waypoints. It combines waypoint-relative object translation, a part-based spatiotemporal denoiser, and contact-aware refinement. During sampling, geometric guidance improves hand–object and foot–ground contact; afterward, a learned support predictor and temporal inverse kinematics reduce foot sliding. Experiments on FullBodyManipulation and unseen 3D-FUTURE objects demonstrate improved trajectory adherence, motion quality, and contact stability, supported by component ablations and a user study.

Critical strengths

The framework is well motivated, with complementary designs addressing trajectory control, coordinated human–object motion, and contact stability. Carefully controlled ablations—including matched-token comparisons and applying the proposed refinement to a strong baseline—support the contributions of individual components.

Quantitative evaluation, testing on unseen object geometries, and a blinded user study provide broad evidence of improved generation quality.

Critical weaknesses

The framework combines waypoint-relative prediction, part-based attention, contact guidance, and inverse kinematics, all of which build on established techniques. The paper should clarify its technical advances beyond their integration.

The interpolated reference path passes through the supplied waypoints, where waypoint error is evaluated. Although the controlled ablation supports residual prediction, stronger evaluation between waypoints and under challenging user-defined routes would better establish controllability.

The contact objectives enforce geometric consistency but do not assess balance, contact forces, friction, or object dynamics. Improved contact and reduced sliding therefore do not establish dynamically feasible interactions.

The 3D-FUTURE experiment primarily tests transfer to new object geometries under the same conditioning protocol. Moreover, the 20 reported seeds vary diffusion sampling with fixed network weights, leaving training variability unmeasured.

Numerical inconsistencies require correction. Appendix A.3.6 states that applying guidance and refinement to MaMi-HOI increases contact F1 from 0.75 to 0.79, whereas Table 9 reports 0.77. Appendix A.3.8 describes the reduction in foot sliding from 0.30 to 0.23 as 21.3%, while the displayed values imply approximately 23.3%.

Questions for authors

see above

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4tGt (1/2)

We appreciate the reviewer's careful reading and constructive feedback. We respond to each point below:

W1. Contribution statement

We thank the reviewer for this comment. We clarify here, point by point, how the details of our method differ from prior work:

(1) Waypoint-relative prediction. Prior work interpolates the waypoints into a reference trajectory and then predicts the absolute object motion (CHOIS and Omomo fall in this group), so the route itself is a generation target. We use the reference path both as a condition and as the baseline of the planar displacement: the network predicts only the residual to that baseline, normalized by sxy\mathbf{s}_{xy}, the 99.9th percentile of the training residuals, and clipped to [−8,8][-8,8] with the scale fixed at training and inference. Capacity therefore goes into local correction rather than into redrawing the whole path, and waypoint error need not be forced down by a loss term.

(2) Part-based attention. Unlike AnyTop, OpenT2M and related methods, our work covers both the human and the object together. Our partitioning is semantic, and the manipulated object is the seventh part: it exchanges information with the six body parts inside the same part attention rather than in a separate stream; the axis itself is fixed by ablation rather than given by skeleton topology.

(3) Contact guidance. Contact guidance is widely used in HOI generation and we adopt it. More importantly, unlike common contact guidance we add a palm-orientation term: it aligns the palm normal to the local surface normal of the object, estimated by a local fit with 32 neighbors, and is applied only to hands within 8 cm of the surface. It targets a failure mode in which the hand reaches the object but with the wrong orientation, so that the interaction does not read as a grasp.

(4) Inverse kinematics. IK is widely used in this field to remove foot penetration and sliding, and we use it as well; unlike the usual per-frame IK, ours is driven by learned quantities and is temporally consistent. The foot refinement network predicts a support-phase probability and a correction confidence per frame; together with the generated world-space foot positions they define persistent contact anchors, target positions and constraint weights. The solve then penalizes horizontal displacement between adjacent support frames in addition to the position constraints, so the foot stays put over a support interval instead of being corrected frame by frame.

We have added more discussion of prior work in Sec. 2.1 and Sec. 2.2 of the main text to clarify how our approach differs from previous methods.

W2. Waypoints

We thank the reviewer for this question. The concern has two parts: (a) whether the waypoints seen at test time follow the same distribution as those seen in training, and (b) whether the waypoints are sampled from ground-truth trajectories rather than drawn by a user. We address both in the three steps below.

(1) Train and test waypoints. FullBodyManipulation is split by sequence, so the waypoints used at test time are drawn from the object trajectories of held-out sequences; no test waypoint trajectory appears during training.

(2) Cross-dataset evaluation. We further evaluate on 3D-FUTURE, with 17 unseen meshes paired with held-out motions and without retraining (A.2). Both the object geometries and the waypoint trajectories there lie outside the training distribution.

(3) Perturbation. To probe the setting the benchmarks do not cover, a route drawn by a user, we perturbed the input waypoints with independent Gaussian noise per axis (sigma = 15 cm, frame 0 unchanged), at a magnitude that makes the inputs differ substantially from the ground-truth routes, as a hand-drawn route would. The planar displacement is Rayleigh-distributed, with a mean of 18.86 cm over 678 cases and 2,712 waypoints, about nineteen times our own waypoint error.

Table R1: Out-of-distribution waypoint test on 3D-FUTURE.

SettingTsT_s↓TeT_e↓TxyT_{xy}↓HfeetH_{\mathrm{feet}}↓FSFS↓RprecR_{\mathrm{prec}}↑FIDFID↓C_{\\%}↑PhandP_{\mathrm{hand}}↓
Original waypoints3.185.530.992.500.250.691.460.730.14
Perturbed, σ\sigma = 15 cm3.135.281.012.640.250.681.430.750.15

Three observations follow: the model still reaches the perturbed waypoints; it follows the new route, since the deviation from the original ground-truth route is 18.87 cm, essentially the input offset of 18.86; and nothing degrades by an order of magnitude.

In the revision, this perturbation study is summarized in the appendix.

AuthorsOpenReview ↗

Response to Reviewer 4tGt (2/2)

W3. Physical feasibility

We thank the reviewer for raising this point, and we would like to clarify the setting our method targets.

There are two lines of work in this field, and they are complementary. We explain them here:

(1) Our task and objective. The input is a text instruction, the object geometry and sparse object waypoints; the output is a full-body human motion together with the corresponding object motion. The requirement is that the result look natural, follow the instruction and respect the geometry of the object. This is the standard objective of text-driven HOI generation, which serves animation and virtual reality rather than the direct control of a physical robot.

What does physical plausibility mean in this setting? In this line of work, plausibility is usually assessed visually: the hand reaches the object surface, the feet neither slide nor penetrate the ground, and the body stays balanced. Simulator-level dynamic feasibility is a different objective, since it requires contact forces, friction, joint torque limits and a control policy. Generative models can rarely satisfy both at the same time: fully physically consistent motion and smooth, natural movement are difficult to achieve simultaneously in generative models. Our contact guidance and foot refinement target the former.

(2) Work that targets dynamic feasibility. A separate line does pursue absolute physical feasibility, and it typically sits downstream of generation, adapting a produced motion for robotic or simulated execution: PhysHOI and InterMimic track a given reference with physics-based control, SONIC builds humanoid tracking on a motion prior learned from capture and bridges it with a kinematic planner, and GenMimic retargets generated motion and absorbs its artifacts with a policy. Our output is the kind of reference such pipelines consume, and the Conclusion already states that our refinement is kinematic.

W4. Training-seed variance

We thank the reviewer for this comment. To cover training variance, we trained three additional models with different training seeds and evaluated each under 20 sampling seeds; the table reports the mean and standard deviation over all training-by-inference runs on the 3D-FUTURE evaluation.

Table R2: Mean and standard deviation over training and sampling seeds on the 3D-FUTURE evaluation.

TsT_s↓TeT_e↓TxyT_{xy}↓HfeetH_{\mathrm{feet}}↓FSFS↓
Mean3.17575.53560.98562.50300.2512
Std0.00370.03230.00930.01700.0020
Mean/Std± 0.12%± 0.58%± 0.94%± 0.68%± 0.79%

Both the training-seed and the sampling variance are small relative to the gaps between methods reported in Tables 1 and 2.

W5. Two numerical issues

Thank you very much to the reviewer for the careful reading. Here are two questions regarding the numerical values:

We recomputed every derived percentage in the paper from the unrounded 20-seed means, which shows where the two figures come from. In Table 9 the CF1C_{\mathrm{F1}} entry read 0.77 while App. A.3.6 states 0.79; the unrounded means are 0.7462 and 0.7891, so 0.77 was a transcription error.

The 21.3% foot-sliding reduction in A.3.8 is correct: (0.2963-0.2333)/0.2963 = 21.3%, computed from unrounded means while the table displays two decimals.

In the revision, Table 9 now reads 0.79, Table 11 states that percentages are computed from unrounded means, and A.3.8 quotes the unrounded pair. The remaining derived figures follow the same convention and are unchanged, and no measured value changes. We thank the reviewer again for the helpful comments on this paper.

Dear Reviewer 4tGt,

We would like to kindly follow up on our response to your review, which we posted in two parts because of the length limit. Part 1 covers W1 and W2, and part 2 covers W3, W4 and W5.

Since posting our reply we have also updated the manuscript. Table 9 now reads 0.79. Table 11 states the convention used for the percentages quoted in the text, and A.3.8 quotes the unrounded pair. Additional prior work is discussed in the related work part. A.3.9 now includes the out-of-distribution waypoint test.

We would greatly appreciate it if you could revisit our response and the revised PDF, and let us know whether any concerns remain before the discussion period ends.

Thank you again for your time and constructive feedback.

Best regards, The Authors

Thanks for the rebuttal.

Reviewer_y9UNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose TriFactorHOI, a method for generating stable and natural human-object interaction motion sequences conditioned on user-specified waypoints. To improve trajectory controllability, human-object coordination, and contact consistency, they introduce three core factors: Path, Part, and Contact. The Path factor predicts planar translation residuals relative to an interpolated dense reference path, while the Part factor generates full-body interaction sequences using spatiotemporal attention with semantic tokens. The Contact factor further improves hand and foot contact through geometry-aware guidance and foot refinement. The proposed method outperforms existing baselines on most evaluation metrics across the FullBodyManipulation and 3D-FUTURE datasets.

Critical strengths

S1. The problem formulation and the proposed solution are well aligned. The authors address controllability, human-object coordination, and contact consistency by introducing three distinct factors. The overall method is straightforward and easy to follow.

S2. TriFactorHOI outperforms prior baselines on most evaluation metrics, demonstrating the effectiveness of combining the three factors.

S3. The qualitative results presented in the main paper and appendix show clear improvements over prior baselines, further supporting the quantitative results. Additionally, the supplementary videos demonstrate the promising performance of TriFactorHOI.

Critical weaknesses

W1. The necessity of spatiotemporal attention factorization is not sufficiently justified. The authors factorize spatiotemporal attention into separate temporal and part attention modules to model human-object motion. However, this factorized design limits direct attention across body parts at different time steps. In contrast, full spatiotemporal self-attention allows direct interactions among all body parts and time steps, although at a higher computational cost. Comparing the factorized design with full self-attention in terms of the trade-off between performance and computational efficiency would provide stronger evidence for this design choice.

W2. The contributions of semantic part modeling and contact guidance are not sufficiently disentangled. Table 4 shows that semantic part assignment significantly improves foot height and foot sliding metrics, yet all variants are evaluated with interaction guidance and FR/IK enabled. Meanwhile, Table 7 shows that removing guidance also degrades foot height performance, suggesting that guidance may contribute to these improvements. Table 9 only partially addresses this issue, as it does not evaluate semantic and free tokens without guidance. Therefore, further experiments isolating the contributions of semantic part modeling and interaction guidance would help clarify the source of these improvements.

W3. The choice of temporal patch size is not sufficiently justified. The proposed part-aware denoiser represents a 120-frame sequence using 30 temporal tokens per part (4 frames per token), yet no analysis of this design choice is provided. An ablation study varying the temporal patch size (e.g., 2, 4, and 8 frames per token) would help justify the chosen temporal granularity.

Questions for authors

Please refer to the concerns raised in the weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer y9UN (1/2)

We appreciate the reviewer's careful reading and constructive feedback. We respond to each point below:

W1. Comparison with full space-time attention

Our token count is 30 temporal tokens × 7 parts = 210, so joint self-attention over all tokens is affordable at this scale. Keeping width, block count, heads, FFN and training budget fixed, we replaced the factorized attention with 210-way joint self-attention and retrained; both models are evaluated with 20 sampling seeds.

Table R1: Comparison with full space-time attention on FullBodyManipulation (means over 20 sampling seeds).

ModelRprecR_{\mathrm{prec}}↑FIDFID↓HfeetH_{\mathrm{feet}}↓FSFS↓CF1C_{\mathrm{F1}}↑MPJPEMPJPE↓TxyT_{\mathrm{xy}}↓
Joint space-time attention0.5870.6733.350.2640.73015.140.119
TriFactorHOI0.7390.3442.990.2330.79913.280.130

The two models are almost equally expensive: joint attention is 2% slower and needs 9.9% more peak memory. Attention is not the end-to-end bottleneck. The comparison is therefore one of quality at comparable cost, not a quality-for-efficiency trade-off.

Ours is better on six of the seven metrics. Joint attention wins only on TxyT_{\mathrm{xy}}, 0.119 against 0.130 cm: 8.4% better but 0.11 mm in absolute terms. From the experimental results, factorized space-time performs better than full space-time attention. We believe the possible reason is that although full space-time attention has a larger representational range, with a limited amount of data this makes optimization more difficult.

W2. Separating semantic parts from contact guidance

Thanks for your constructive suggestion. We completed a 3×2 comparison. Guidance is applied at inference time, in the last 25 reverse steps, so the on/off axis needs no retraining: each model is evaluated under both settings with its own weights.

Table R2: Part-token ablation with and without inference-time guidance on FullBodyManipulation.

ModelGuidanceFSFS↓HfeetH_{\mathrm{feet}}↓CF1C_{\mathrm{F1}}↑FIDFID↓TxyT_{\mathrm{xy}}↓RprecR_{\mathrm{prec}}↑
Global 1 tokenNo0.365.780.450.470.340.73
Free 7 tokensNo0.375.920.320.540.230.64
TriFactorHOINo0.244.850.630.650.130.70
Global 1 tokenYes0.404.280.800.570.360.74
Free 7 tokensYes0.373.970.800.450.210.72
TriFactorHOIYes0.232.990.800.340.130.74

Ours is best with and without guidance. Foot sliding is 0.23 against 0.37 with guidance and 0.24 against 0.37 without it, so the gap barely moves, from 0.14 to 0.13; this gain is attributable to semantic tokenization rather than to guidance. Foot height is strongly affected by guidance, from 4.85 to 2.99 and from 5.92 to 3.97, but the semantic advantage persists in both settings and is of similar size, 0.98 and 1.07 cm, so the two effects are close to additive. Contact F1 separates the variants only without guidance, where ours retains 0.63 against 0.45 and 0.32; with guidance all three saturate at 0.80. One result works against us and we state it: without guidance our FID (0.65) is higher than free tokens (0.54) and the global token (0.47). We therefore claim the semantic axis for foot stability and contact, not for distribution quality.

In the paper, Table 4 has been extended to list the three generators under both settings, and App. A.3.1 now discusses this comparison.

AuthorsOpenReview ↗

Response to Reviewer y9UN (2/2)

W3. Temporal block size

We added 2-frame (60 tokens) and 8-frame (15 tokens) settings.

Table R3: Temporal-block ablation on FullBodyManipulation (means over 20 sampling seeds).

SettingFSFS↓RprecR_{\mathrm{prec}}↑MPJPEMPJPE↓HfeetH_{\mathrm{feet}}↓FIDFID↓
2 frames (60 tokens)0.2540.62715.572.920.354
4 frames, ours (30 tokens)0.2330.73913.282.990.344
8 frames (15 tokens)0.2680.56815.472.980.344

Both directions degrade, and 4 frames is best on 12 of 16 metrics; the 2-frame arm doubles the token count and the attention cost without buying quality. Three exceptions: end error is best at 8 frames (1.344 vs 1.455); HfeetH_{\mathrm{feet}} and contact ratio are best at 2 frames, but contact recall drops 10.9% and CF1C_{\mathrm{F1}} drops 8.7% there; and FID is nearly identical across the three. Taking prior work and our own results together, we consider 4 frames the most reasonable choice.

In the paper, we have added a presentation and discussion of the results of this ablation study in Sec. A.3.10.

We thank the reviewer again for the helpful comments on this paper.

Thank you for the detailed rebuttal and additional experiments. The new results substantially address my concerns and provide stronger empirical support for the proposed design. I encourage the authors to include the full spatiotemporal attention comparison in the revised manuscript and clearly specify the architectural configurations, parameter counts, and training budgets to ensure a fair comparison. I will maintain my original rating.

We sincerely thank you for taking the time to provide insightful feedback and for your positive evaluation of our work. We greatly appreciate your recognition of our work and your continued support of our paper!

Reviewer_9DgsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a generative framework for human–object interaction and motion from text and scene. The generation is done is steps, where a path is computed first, and serves a blueprint to a consecutive model that generates object and human motion together. Finally, foot and hand contacts are further refined in the last step. The approach achieves decent results, and makes perfect engineering sense, but it is hard to identify exactly the amount of novelty introduced.

Critical strengths

The strongest aspect of the paper is that the overall pipeline is thoughtfully designed, with each component serving a clear purpose and fitting naturally into the generation process. While a bit cumbersome with terms, the exposition does a good job of giving reason to the design.

The empirical evaluation is convincing. The numerical results are conclusive, and the qualitative results seem decent, with clear advantages over the competition. The cross-dataset evaluation on unseen objects strengthens this further.

I also found the joint treatment of hand–object and foot–ground contact interesting.

Critical weaknesses

The primary concern is that the paper does not clearly distinguish its contributions from ideas directly inspired from previous work. For example, decomposing interaction representations into object trajectory, motion, and contact-related components has been explored in the past (eg Blueprint2Motion, HOIDiNi). Likewise, separate spatial and temporal attention layers has appeared in prior architectures (eg AnyTop). On the other hand, the specific variant of the representation with body parts bears differences, and so do the repeated spatiotemporal blocks in the architecture. A similar question arises around the DLS-IK step. The paper does not clearly articulate where the conceptual novelty begins and where it is primarily adapting existing ideas.

As a side note, Section 3.2.2 describes the inference pipeline through text, when this is a central and elaborate process in the paper. A figure would be useful.

Questions for authors

I would urge the authors to distinguish well the innovation from the existing ideas, and emphasize the effects of these aspects. This includes the decomposed representations, the architecture, and the fundumental difference to the missing works mentioned in the weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9Dgs

We appreciate the reviewer's careful reading and constructive feedback. We respond to each point below:

W1 & Q1: Contribution statement

Prior work typically divides HOI (human–object interaction) generation into two consecutive stages, whereas we integrate the designed three factors into a single stage. Specifically, the state is factorized along three axes that also determine the computation. The path factor turns the interpolated reference path into both a condition and the baseline of planar object translation, so the denoiser predicts only a residual; the requested route therefore lives in the representation itself rather than in a post-hoc constraint. The part factor adds a third, semantic axis beside time and space, with seven part tokens that include the manipulated object, so that body and object exchange information through part attention inside each patch. The contact factor acts in two places: geometry-aware guidance during reverse diffusion, and, after sampling, a learned support state and correction confidence that set world-space foot anchors, targets and weights for temporal inverse kinematics.

These factors are not independent add-ons. Each addresses a distinct failure mode, and the ablations show the coupling. Forcing the waypoint residual to zero removes the flexibility of the path factor and zeroes waypoint error. Replacing the semantic part axis with free tokens of the same count and parameter budget raises foot sliding from 0.23 to 0.37 and foot height from 2.99 to 3.97. Removing interaction guidance raises foot height from 2.99 to 4.85 and drops contact F1 from 0.80 to 0.63. Together they are what makes transfer to unseen objects without retraining work (Table 2); taken alone, each is a local improvement.

Compared with Blueprint2Motion and HOIDiNi, both split the problem into two stages around an explicit intermediate: a limb-object blueprint that a second stage must realize, or an object-centric then human-centric optimization in the noise space of a frozen model. Our method has neither an intermediate artifact nor a second stage. The reference path is simply the interpolation of the supplied waypoints, the part axis is a trained architectural component rather than a test-time search, and the three factors are optimized jointly against the final objective.

Compared with AnyTop, which separates temporal from skeleton-topology axes for arbitrary skeletons, we apply the same pair of axes to semantic parts, with the manipulated object as the seventh part; human and object exchange information through part attention inside each patch. The deeper difference is what defines the axis: AnyTop's axis is given by the skeleton topology, whereas ours is defined by the semantic decomposition and validated by ablation.

Against standard DLS-IK, the solver is identical; what differs is what drives it - the learned support probability and correction confidence determine the world-space anchors, targets and weights, and the solver penalizes horizontal displacement across adjacent support frames.

We polished our manuscript to further clarify our contribution, and we have added discussion of and citations to Blueprint2Motion, AnyTop, and HOIDiNi in the Sec. 2.1 and Sec. 2.2.

W2: Figure for the inference pipeline

We thank the reviewer for this suggestion. We have added a figure to Sec. 3.2.2 of the main text, which shows the inference pipeline: how the text, object geometry and reference path are encoded once, how the motion is projected into part tokens and denoised, and where interaction guidance and foot refinement act on the sampled sequence.

We thank the reviewer again for the helpful comments on this paper.

Dear Reviewer 9Dgs,

We would like to kindly follow up on our response to your review.

Since posting our reply we have also updated the manuscript. The related work now discusses Blueprint2Motion, AnyTop and HOIDiNi, which addresses the point raised in W1. Sec. 3.2.2 has a new figure of the inference pipeline, as suggested in W2.

We would greatly appreciate it if you could revisit our response and the revised PDF, and let us know whether any concerns remain before the discussion period ends.

Thank you for taking the time to provide helpful feedback and for your positive evaluation of and support for our work.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.