- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents TriFactorHOI, a structured diffusion framework for generating coordinated human–object motion conditioned on text, object geometry, initial states, and sparse object waypoints. It combines waypoint-relative object translation, a part-based spatiotemporal denoiser, and contact-aware refinement. During sampling, geometric guidance improves hand–object and foot–ground contact; afterward, a learned support predictor and temporal inverse kinematics reduce foot sliding. Experiments on FullBodyManipulation and unseen 3D-FUTURE objects demonstrate improved trajectory adherence, motion quality, and contact stability, supported by component ablations and a user study.
Critical strengths
The framework is well motivated, with complementary designs addressing trajectory control, coordinated human–object motion, and contact stability. Carefully controlled ablations—including matched-token comparisons and applying the proposed refinement to a strong baseline—support the contributions of individual components.
Quantitative evaluation, testing on unseen object geometries, and a blinded user study provide broad evidence of improved generation quality.
Critical weaknesses
The framework combines waypoint-relative prediction, part-based attention, contact guidance, and inverse kinematics, all of which build on established techniques. The paper should clarify its technical advances beyond their integration.
The interpolated reference path passes through the supplied waypoints, where waypoint error is evaluated. Although the controlled ablation supports residual prediction, stronger evaluation between waypoints and under challenging user-defined routes would better establish controllability.
The contact objectives enforce geometric consistency but do not assess balance, contact forces, friction, or object dynamics. Improved contact and reduced sliding therefore do not establish dynamically feasible interactions.
The 3D-FUTURE experiment primarily tests transfer to new object geometries under the same conditioning protocol. Moreover, the 20 reported seeds vary diffusion sampling with fixed network weights, leaving training variability unmeasured.
Numerical inconsistencies require correction. Appendix A.3.6 states that applying guidance and refinement to MaMi-HOI increases contact F1 from 0.75 to 0.79, whereas Table 9 reports 0.77. Appendix A.3.8 describes the reduction in foot sliding from 0.30 to 0.23 as 21.3%, while the displayed values imply approximately 23.3%.
Questions for authors
see above
4 comments
Response to Reviewer 4tGt (2/2)
W3. Physical feasibility
We thank the reviewer for raising this point, and we would like to clarify the setting our method targets.
There are two lines of work in this field, and they are complementary. We explain them here:
(1) Our task and objective. The input is a text instruction, the object geometry and sparse object waypoints; the output is a full-body human motion together with the corresponding object motion. The requirement is that the result look natural, follow the instruction and respect the geometry of the object. This is the standard objective of text-driven HOI generation, which serves animation and virtual reality rather than the direct control of a physical robot.
What does physical plausibility mean in this setting? In this line of work, plausibility is usually assessed visually: the hand reaches the object surface, the feet neither slide nor penetrate the ground, and the body stays balanced. Simulator-level dynamic feasibility is a different objective, since it requires contact forces, friction, joint torque limits and a control policy. Generative models can rarely satisfy both at the same time: fully physically consistent motion and smooth, natural movement are difficult to achieve simultaneously in generative models. Our contact guidance and foot refinement target the former.
(2) Work that targets dynamic feasibility. A separate line does pursue absolute physical feasibility, and it typically sits downstream of generation, adapting a produced motion for robotic or simulated execution: PhysHOI and InterMimic track a given reference with physics-based control, SONIC builds humanoid tracking on a motion prior learned from capture and bridges it with a kinematic planner, and GenMimic retargets generated motion and absorbs its artifacts with a policy. Our output is the kind of reference such pipelines consume, and the Conclusion already states that our refinement is kinematic.
W4. Training-seed variance
We thank the reviewer for this comment. To cover training variance, we trained three additional models with different training seeds and evaluated each under 20 sampling seeds; the table reports the mean and standard deviation over all training-by-inference runs on the 3D-FUTURE evaluation.
Table R2: Mean and standard deviation over training and sampling seeds on the 3D-FUTURE evaluation.
| ↓ | ↓ | ↓ | ↓ | ↓ | |
|---|---|---|---|---|---|
| Mean | 3.1757 | 5.5356 | 0.9856 | 2.5030 | 0.2512 |
| Std | 0.0037 | 0.0323 | 0.0093 | 0.0170 | 0.0020 |
| Mean/Std | ± 0.12% | ± 0.58% | ± 0.94% | ± 0.68% | ± 0.79% |
Both the training-seed and the sampling variance are small relative to the gaps between methods reported in Tables 1 and 2.
W5. Two numerical issues
Thank you very much to the reviewer for the careful reading. Here are two questions regarding the numerical values:
We recomputed every derived percentage in the paper from the unrounded 20-seed means, which shows where the two figures come from. In Table 9 the entry read 0.77 while App. A.3.6 states 0.79; the unrounded means are 0.7462 and 0.7891, so 0.77 was a transcription error.
The 21.3% foot-sliding reduction in A.3.8 is correct: (0.2963-0.2333)/0.2963 = 21.3%, computed from unrounded means while the table displays two decimals.
In the revision, Table 9 now reads 0.79, Table 11 states that percentages are computed from unrounded means, and A.3.8 quotes the unrounded pair. The remaining derived figures follow the same convention and are unchanged, and no measured value changes. We thank the reviewer again for the helpful comments on this paper.
Dear Reviewer 4tGt,
We would like to kindly follow up on our response to your review, which we posted in two parts because of the length limit. Part 1 covers W1 and W2, and part 2 covers W3, W4 and W5.
Since posting our reply we have also updated the manuscript. Table 9 now reads 0.79. Table 11 states the convention used for the percentages quoted in the text, and A.3.8 quotes the unrounded pair. Additional prior work is discussed in the related work part. A.3.9 now includes the out-of-distribution waypoint test.
We would greatly appreciate it if you could revisit our response and the revised PDF, and let us know whether any concerns remain before the discussion period ends.
Thank you again for your time and constructive feedback.
Best regards, The Authors
Thanks for the rebuttal.
Response to Reviewer 4tGt (1/2)
We appreciate the reviewer's careful reading and constructive feedback. We respond to each point below:
W1. Contribution statement
We thank the reviewer for this comment. We clarify here, point by point, how the details of our method differ from prior work:
(1) Waypoint-relative prediction. Prior work interpolates the waypoints into a reference trajectory and then predicts the absolute object motion (CHOIS and Omomo fall in this group), so the route itself is a generation target. We use the reference path both as a condition and as the baseline of the planar displacement: the network predicts only the residual to that baseline, normalized by sxy, the 99.9th percentile of the training residuals, and clipped to [−8,8] with the scale fixed at training and inference. Capacity therefore goes into local correction rather than into redrawing the whole path, and waypoint error need not be forced down by a loss term.
(2) Part-based attention. Unlike AnyTop, OpenT2M and related methods, our work covers both the human and the object together. Our partitioning is semantic, and the manipulated object is the seventh part: it exchanges information with the six body parts inside the same part attention rather than in a separate stream; the axis itself is fixed by ablation rather than given by skeleton topology.
(3) Contact guidance. Contact guidance is widely used in HOI generation and we adopt it. More importantly, unlike common contact guidance we add a palm-orientation term: it aligns the palm normal to the local surface normal of the object, estimated by a local fit with 32 neighbors, and is applied only to hands within 8 cm of the surface. It targets a failure mode in which the hand reaches the object but with the wrong orientation, so that the interaction does not read as a grasp.
(4) Inverse kinematics. IK is widely used in this field to remove foot penetration and sliding, and we use it as well; unlike the usual per-frame IK, ours is driven by learned quantities and is temporally consistent. The foot refinement network predicts a support-phase probability and a correction confidence per frame; together with the generated world-space foot positions they define persistent contact anchors, target positions and constraint weights. The solve then penalizes horizontal displacement between adjacent support frames in addition to the position constraints, so the foot stays put over a support interval instead of being corrected frame by frame.
We have added more discussion of prior work in Sec. 2.1 and Sec. 2.2 of the main text to clarify how our approach differs from previous methods.
W2. Waypoints
We thank the reviewer for this question. The concern has two parts: (a) whether the waypoints seen at test time follow the same distribution as those seen in training, and (b) whether the waypoints are sampled from ground-truth trajectories rather than drawn by a user. We address both in the three steps below.
(1) Train and test waypoints. FullBodyManipulation is split by sequence, so the waypoints used at test time are drawn from the object trajectories of held-out sequences; no test waypoint trajectory appears during training.
(2) Cross-dataset evaluation. We further evaluate on 3D-FUTURE, with 17 unseen meshes paired with held-out motions and without retraining (A.2). Both the object geometries and the waypoint trajectories there lie outside the training distribution.
(3) Perturbation. To probe the setting the benchmarks do not cover, a route drawn by a user, we perturbed the input waypoints with independent Gaussian noise per axis (sigma = 15 cm, frame 0 unchanged), at a magnitude that makes the inputs differ substantially from the ground-truth routes, as a hand-drawn route would. The planar displacement is Rayleigh-distributed, with a mean of 18.86 cm over 678 cases and 2,712 waypoints, about nineteen times our own waypoint error.
Table R1: Out-of-distribution waypoint test on 3D-FUTURE.
Three observations follow: the model still reaches the perturbed waypoints; it follows the new route, since the deviation from the original ground-truth route is 18.87 cm, essentially the input offset of 18.86; and nothing degrades by an order of magnitude.
In the revision, this perturbation study is summarized in the appendix.