- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MAMHOI, a factorized framework for scene-aware human–object interaction generation. It introduces motion affordance as an intermediate representation to connect scene understanding and HOI motion synthesis, allowing the two components to be trained independently on human–scene and human–object datasets. Experiments demonstrate improved scene compatibility and interaction quality compared with existing methods.
Critical strengths
- Meaningful problem. Learning scene-aware HOI from separate HSI and HOI datasets addresses the scarcity of paired training data.
- Intuitive factorization. Motion affordance provides a reasonable interface between scene understanding and interaction generation.
- Strong quantitative results. The method improves penetration, motion smoothness, and contact metrics on the constructed scene-aware benchmark.
- Additional validation. The paper includes ablations, an auxiliary evaluation on TRUMANS, and a perceptual user study.
Critical weaknesses
- Insufficient validation of the learned affordance representation. The paper lacks comparisons against simpler alternatives, such as direct scene occupancy conditioning or path-based affordance construction. It is unclear whether the improvements come from the learned affordance representation or simply from introducing scene-aware conditioning.
- Definition of affordance. There is a domain gap between the affordance used in training (oracle) and inference (prediction). There is no quantitative evaluation of this gap.
- Loss of 3D geometric information. Appendix D.3 describes vertically pooling the local occupancy voxels and broadcasting the resulting 2D representation along the height dimension. This discards height-dependent geometric information and may limit the ability to handle complex 3D obstacles.
- Dependence on external path planning. The method requires a planned object path, and the evaluation uses manually selected feasible start–goal configurations with A* planning. Its robustness to imperfect or physically infeasible paths remains unclear.
Questions for authors
- Height collapse. Appendix D.3 min-pools each local volume along the vertical axis and broadcasts it back along height, so the condition appears to carry no height information. How does the model then distinguish passing under an obstacle from routing around it?
- Oracle vs. predicted affordance. Training uses an affordance built from the observed motion, inference uses the predicted one (Section 3.3.2). Could you report Table 2 with the GT-derived affordance as an oracle, to quantify this gap?
- Planner contribution. What happens with the path removed (text + scene only)?
- Affordance semantics. The affordance is an AABB around LINGO human-only trajectories (Section 4.1) but is defined as the joint human–object support (Section 3.3.1). How does it capture the wider clearance large objects need?
- User study scope. All four pairings in Appendix F are MAMHOI vs. CHOIS variants. Could you add InfBaGel, and report significance for the 884 votes?
- Inference cost. Appendix D.5 specifies 1,000 denoising steps over autoregressive windows. Could you report wall-clock time and memory, given the characterization of HOSIG as expensive in Section E.3?
2 comments
Q4: Affordance semantics and object clearance
We have revised Section 3.3.1 to clarify that Factor I predicts coarse human-centric spatial support relative to the planned object path. Because it learns from LINGO without object geometry, it cannot explicitly capture object-specific clearance requirements. Factor II combines this soft condition with object geometry, the planned path, and the learned HOI prior to generate coordinated motion.
Our new per-object evaluation (Appendix E.2, Table 5) compares generation with and without predicted affordance guidance. Across all six objects, this guidance reduces human-scene penetration rates and object-scene penetration depths while maintaining similar path errors.
Entries show Without → With (relative change). Affordance is removed only at inference, with the generator and paths fixed. These results provide empirical evidence that HSI-learned affordances offer useful guidance for HOI generation across different objects, although they do not establish object-specific clearance.
| Object | Human-scene penetration rate (%) ↓ | Object-scene penetration depth (×10⁻³) ↓ | Object-path error ↓ |
|---|---|---|---|
| clothesstand | 31.084 → 12.200 (-60.75%) | 3.590 → 1.193 (-66.77%) | 0.1527 → 0.1482 (-2.95%) |
| floorlamp | 29.291 → 13.279 (-54.67%) | 1.651 → 1.535 (-7.03%) | 0.1553 → 0.1511 (-2.70%) |
| largebox | 28.641 → 14.312 (-50.03%) | 1.666 → 0.944 (-43.34%) | 0.1530 → 0.1482 (-3.14%) |
| largetable | 32.180 → 16.164 (-49.77%) | 3.425 → 3.152 (-7.97%) | 0.1487 → 0.1471 (-1.08%) |
| smallbox | 26.518 → 15.657 (-40.96%) | 1.432 → 0.511 (-64.32%) | 0.1562 → 0.1544 (-1.15%) |
| whitechair | 33.691 → 18.327 (-45.60%) | 1.512 → 1.303 (-13.82%) | 0.1505 → 0.1463 (-2.79%) |
We have clarified this limitation in lines 508 to 516 and will explore object-aware affordances in future work.
Q5: Expanded user study and statistical significance
We have expanded the user study to include InfBaGel and report all six pairings in Appendix F, Table 9. All comparisons are significant at p < 0.001 using two-sided exact binomial tests against a preference probability of 0.5, under an independent-judgment assumption. The test is described in the caption (lines 1245 to 1249), and the main-text results are updated in lines 494 to 507.
Votes and preference percentages refer to Method A. SDF denotes test-time SDF guidance.
| Method A | Method B | A votes / total | A preference | Two-sided p-value |
|---|---|---|---|---|
| Ours | CHOIS | 195 / 225 | 86.7% | 8.01 × 10⁻³¹ |
| Ours + SDF | CHOIS + SDF | 182 / 216 | 84.3% | 1.19 × 10⁻²⁵ |
| Ours | CHOIS + SDF | 204 / 221 | 92.3% | 6.92 × 10⁻⁴² |
| Ours + SDF | CHOIS | 180 / 222 | 81.1% | 1.53 × 10⁻²¹ |
| Ours | InfBaGel | 195 / 229 | 85.2% | 1.23 × 10⁻²⁸ |
| Ours + SDF | InfBaGel | 188 / 230 | 81.7% | 3.05 × 10⁻²³ |
Q6: Inference speed and memory usage
Following InfBaGel, we report FPS as generated motion frames per second of wall-clock computation. Results on a single NVIDIA RTX 3090 are included in Appendix E.4, Table 7. All methods use approximately 6 to 9 GB of peak GPU memory. Runtime is mainly affected by guidance and optimization where required. InfBaGel achieves higher throughput through consistency-based few-step denoising.
| Method | FPS ↑ |
|---|---|
| CHOIS | 3.234 |
| MAMHOI (Ours) | 2.933 |
| InfBaGel | 24.039 |
| HOSIG | 0.23 |
Thank you for highlighting our strong quantitative results and for your helpful suggestions. We address your comments and questions below.
W1: Learned affordance versus simpler conditions
Without affordance conditioning during training, the generator reduces to the CHOIS baseline, which is already included in our comparisons. To isolate the contribution of learned affordance within our trained model, we remove it or replace it with an A* corridor or raw scene occupancy at inference, keeping the generator and planned paths fixed. These variants yield higher scene penetration despite similar path errors, supporting the benefit of learned affordance beyond path control (Appendix E.2, Table 4).
Object-scene penetration depth is averaged per frame.
W2 & Q2: Affordance transfer and the oracle gap
At inference, affordance is predicted from the distribution learned on LINGO, while the generator is trained with affordances derived from OMOMO motion. Our factorization transfers coarse spatial support without requiring identical affordances across domains. As a soft condition, it guides motion without requiring strict adherence to its boundaries. The generator combines it with object geometry, the planned path, and the learned HOI prior. The distribution gap remains a limitation, and we will explore better alignment in future work.
The ablation in W1 (Appendix E.2, Table 4) supports the usefulness of this transfer, although it does not quantify the oracle gap.
A matched oracle evaluation for Table 2 is unavailable because the test scenes lack ground-truth HOI sequences corresponding to the planned paths. GT-derived affordances would also expose target-motion information, introducing ground-truth leakage and preventing a like-for-like comparison with standard inference. We have added this clarification in Appendix E.2 (lines 1104 to 1109).
W3 & Q1: Height-dependent scene constraints
We agree that 2D affordances lose height-dependent information, limiting distinctions such as passing under an obstacle versus routing around it. This is a limitation of the current representation, not a fundamental limitation of our factorization. We have clarified this in lines 508 to 516 and will explore richer height-aware representations in future work.
W4 & Q3: Dependence on external path planning
Our formulation assumes a planner-provided object path and does not perform high-level route planning. Routes incompatible with object geometry should be rejected or adjusted by an object-aware planner, as our generator is not guaranteed to correct them. Our evaluation therefore uses feasible start and goal configurations. We clarify this assumption and limitation in lines 512 to 513.
Both factors are trained with the path as an input, so removing it only at inference would test a missing required condition. A meaningful comparison would require training for generation without a prescribed path and evaluating task completion rather than path adherence. We leave joint route planning and interaction synthesis to future work.