- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents AW3D, which introduces action-stage planning between textual instructions and frame-by-frame human-object interaction generation to address omitted actions, ordering, and timing in complex interactions. The method first segments ground-truth interactions into stages based on changes in contact and relative motion, then constructs 64 action prototype classes through joint-embedding predictive learning and a Gaussian mixture model. Given text, object geometry, and the target length, it predicts prototype distributions and stage boundaries. A recurrent interaction-state model then refines the plan and constructs separate framewise conditions for the body, hands, and object to guide diffusion-based generation. The paper reports improvements in semantic alignment and the quality of the generated motion distribution on InterAct, and includes experiments integrating the method with different generators, generating long sequences, handling unseen objects, evaluating on independent datasets, and measuring user preferences.
Critical strengths
- Organizing action stages to improve the execution of complex instructions is a worthwhile direction and provides a useful perspective on the relationship between semantic planning and continuous motion generation.
- The authors learn reusable prototypes from changes in human-object relationships and use them as planning units. Soft prototype distributions, condition-dependent state adjustments, and role-specific guidance connect coarse plans to the motions of the body, hands, and object.
- Beyond the main experiments on InterAct, the authors examine integration of the prototype pathway with different generators, long sequences, and unseen objects, and conduct separate training and evaluation on OMOMO and BEHAVE.
Critical weaknesses
- The paper is primarily motivated by improving action content and temporal order. However, in Table 4, shuffling the prototype order even yields slightly better mean values for both metrics. It will be better to explain such observations.
- These prototypes are learned mainly from contact and relative-motion features. It remains unclear to what extent they represent the semantic action stages in textual instructions and whether 64 classes are sufficient to preserve important semantic distinctions. The limited semantic gains in the ablations raise this question, but do not establish that information loss in the prototype representation is the cause.
- The current results reflect a trade-off among different objectives, and I recommend further analysis of these visible issues and possible directions for improvement.
Questions for authors
- How do the authors explain the small changes in semantic metrics after removing the prototype pathway in Table 4, and the slight improvement in some semantic metrics after shuffling the prototype order?
- Are there more direct evaluations of event completion or ordering that demonstrate the independent contribution of stage planning?
- Have the authors analyzed the semantic action content associated with different prototypes and how the number of prototypes affects semantic discrimination?
4 comments
Response to Reviewer HC2z (2/2)
W3: Trade-offs among evaluation objectives
We thank the reviewer for encouraging us to examine these trade-offs. In Table 1, AW3D achieves better text–motion alignment and lower penetration than Text2HOI, while Text2HOI has less foot sliding and higher contact F1.
-
Text2HOI's higher contact F1 indicates better agreement with the reference contact labels. This can coexist with greater penetration because a hand joint close to the surface may count as contact even when it lies inside the object. Text2HOI also completes fewer requested stages in the direct evaluation (Table R2). Matching contact labels and following the requested sequence are therefore distinct objectives.
-
AW3D uses an ordered prototype plan to make the progression of requested actions explicit. Its stronger text–motion alignment and stage completion are consistent with this design, while its lower penetration is a separate advantage. The remaining gaps in contact F1 and foot sliding show where execution within each stage can still improve.
These observations suggest two improvements during sampling: guide the hand to the object surface at the planned contact times while penalising penetration separately, and limit foot motion during stance. Both constraints target execution within each stage without changing the planned action order.
Q2: Direct evaluation of stage completion and ordering
We appreciate the suggestion to measure whether the final motion completes each requested stage. Before reviewing model outputs, we selected 44 multi-stage prompts from the held-out InterAct split; 37 have at least three annotated stages. We manually annotated stage intervals in the reference motions and stage presence and timing in anonymised videos. All five methods received the same prompt, object geometry, and target length for each case. We have added the stage-level results and discussion to Sec. 4.3 of the revised manuscript.
Stage recall measures the fraction of required stages that occur. A stage pair counts as correct only if both stages occur and their onsets follow the requested order. For completed stages, duration error is the absolute difference from the annotated duration, normalised by the reference sequence length.
Table R2: Stage-level evaluation of final generated motions on 44 held-out InterAct prompts.
| Method | Stage recall (%) ↑ | Ordered-pair accuracy (%) ↑ | Norm. duration error ↓ (n) |
|---|---|---|---|
| HOI-Diff | 34.0 | 18.9 | 0.212 (27) |
| Text2HOI | 41.0 | 25.6 | 0.172 (31) |
| CHOIS | 52.8 | 34.8 | 0.197 (36) |
| LIGHT | 62.7 | 39.8 | 0.197 (43) |
| AW3D | 85.8 | 74.5 | 0.106 (44) |
For duration error, n denotes the number of prompts with at least one completed stage.
Table R2 shows that AW3D leads on all three stage metrics. AW3D completes every required stage in order on 28 of 44 prompts, compared with 13 for LIGHT. On the 79 stages both methods complete, AW3D also has lower mean duration error (0.085 vs. 0.146). These results show that the full system completes more of the requested interaction and better matches its timing.
Q3: Prototype content and granularity
We appreciate the reviewer's question about what different prototypes represent and how the bank size affects the distinctions they capture. We use a matched example to illustrate what the prototypes distinguish, then refer to the overall results in W2 to explain our choice of bank size.
-
Consider two stages involving the same monitor, prompted by “move the monitor” and “rotate the monitor”. Both occur at similar points in their sequences, with both hands close to the monitor. The first stage mainly changes its position, moving it by about 30 cm, while the second mainly changes its orientation, turning it by about 38°. The classifier using only hand proximity and phase predicts transport for both. Adding the prototype distributions lets it distinguish the two. This example illustrates how a prototype distribution can encode what happens to the object even when hand proximity and timing are similar.
-
We assess the effect of bank size across the full set of matched pairs in W2. The scores are close across the three bank sizes, so we use K=64 as a practical middle ground, with more prototype components than K=32 and fewer candidates for the planner to score than K=128. The purpose is to keep the plan compact while distinguishing changes in the interaction. The prototype distribution describes what changes, and the stage boundary specifies when it should occur.
Thanks for the authors' response. The explanation is to some extent reasonable to me. However, it is more like a trade-off rather than a necessary option. Are there alternative approaches that can achieve simultaneous improvements?
Response to Reviewer HC2z
Thank you for this thoughtful follow-up. We agree that better instruction following need not come at the cost of physical quality. In Table 1, AW3D achieves better text alignment and lower penetration than Text2HOI, while Text2HOI achieves higher contact F1. A hand joint close to the object surface can count as contact even if it lies inside the object, so higher contact F1 alone does not imply more plausible manipulation. Text2HOI uses contact maps to guide hand placement, while AW3D specifies which interaction stages should occur and when. This action-planning pathway can be integrated into different HOI backbones, including Text2HOI itself. In both cross-backbone evaluations (Tables 2 and 3), adding AW3D raises R@3 and lowers MM Dist and FID for Text2HOI, CHOIS, and LIGHT. Contact F1 improves in most comparisons and remains comparable in the others. Compared with the variant without the prototype pathway, full AW3D achieves lower FID and higher contact F1(Table 4). In the blind study, viewers more often preferred animations generated by backbones with AW3D for both instruction following and natural human–object coordination (Appendix E.1).
Following our earlier suggestion to combine action planning with contact-map guidance, we tested a Text2HOI-style contact map for the entire instruction. We trained the contact predictor and hand adapter while keeping AW3D's planner and motion generator frozen. We evaluated the resulting model on the full InterAct test split using the Table 1 protocol.
| Method | R@1 ↑ | R@2 ↑ | R@3 ↑ | FID ↓ | MM Dist ↓ | Div. → | FSR ↓ | Pene ↓ | Contact → | Int. Prec. ↑ | Int. Rec. ↑ | Int. F1 ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AW3D | 0.485 | 0.723 | 0.826 | 0.101 | 2.401 | 7.852 | 0.088 | 0.1490 | 0.126 | 0.756 | 0.603 | 0.630 |
| AW3D + contact map | 0.494 | 0.720 | 0.828 | 0.117 | 2.400 | 7.844 | 0.085 | 0.1541 | 0.128 | 0.747 | 0.607 | 0.632 |
Compared with AW3D alone, adding the Text2HOI-style contact map slightly improves R@3 and contact F1, but raises FID from 0.101 to 0.117. The combined model improves R@3, FID, MM Dist, and contact F1 over the same backbone without either addition (Table 4). Overall, AW3D improves instruction following across different HOI backbones while maintaining competitive physical quality, so the two objectives need not come at the expense of each other. Its action-planning pathway can also be combined with methods that target physical execution, with the potential to improve physical quality further.
Response to Reviewer HC2z (1/2)
We appreciate the reviewer's careful reading and constructive feedback. We respond to each point below:
W1 & Q1: Interpreting the prototype ablations
We thank the reviewer for raising these points. We believe the small changes in the retrieval scores in Table 4 have two main explanations. Removing the prototype pathway leaves the original text condition in place. Shuffling the plan retains the planned action content but changes its order. In both cases, the generated motion can still match the overall action described by the text, which is what whole-clip retrieval primarily measures.
When the prototype pathway is removed, the diffusion generator still receives the full instruction and can infer the main action from it. This explains why R@3 and MM Dist change little even though FID worsens. The text can guide the generator towards the intended action, while the prototype pathway makes the progression of that action explicit.
Shuffling keeps the same prototype segments and time-slot boundaries but changes which segment occupies each slot. For an instruction to lay a clothes stand flat and then pick it up, the shuffled output performs both actions but lays the stand flat later than AW3D and the reference. The main actions are still recognisable, so the retrieval scores can remain close even when a transition is delayed. Across ten additional paired evaluations with fixed checkpoints, each variant achieves higher R@3 in five pairs, and mean MM Dist is similar between the two variants. Keeping the planned order, however, lowers FID and raises contact F1 in all ten pairs.
We also examined why retrieval remains close after shuffling. We use a parameter-matched GRU control that takes the previous planned prototype as history in place of the adjustment predicted for the previous stage. Table R1 compares ordered and shuffled plans under this control.
Table R1: Effect of prototype order in the GRU control without residual feedback.
Bold marks the better value. For Diversity and Contact, values closer to the ground truth (7.843 and 0.202) are better.
In this GRU control, shuffling worsens both retrieval metrics. With recurrent refinement, the scores remain close across the paired runs. This contrast suggests that feeding back predicted interaction states helps retain the overall action when the plan order is perturbed.
Taken together, these results explain why removing the prototype pathway or shuffling the plan can leave whole-clip retrieval largely unchanged. In both cases, the generator still has enough information to produce a recognisable interaction. The contribution of the ordered prototype plan is to guide when each requested action occurs and how it follows the previous one. This is why the slight retrieval advantage of shuffling in Table 4 does not indicate better execution of the requested sequence.
W2: Prototype semantics and capacity
We thank the reviewer for asking how the prototypes relate to the action stages in the instructions. We first quantify what action information the prototypes retain, then examine how bank size affects that result.
To quantify what action information the prototypes preserve, we tested whether they distinguish two types of object motion that instructions can specify: moving an object to a different position and changing its orientation. We used 56 matched pairs of reference windows, controlling for object geometry, hand proximity, and temporal phase. The labels were based on displacement and rotation and checked against the instructions and motions. In cross-validation grouped by object geometry, adding the default prototype distributions to hand proximity and phase raises macro F1, the average F1 across transport and reorientation, from 52.2% to 74.2%. This measures how much the prototypes help distinguish transport from reorientation beyond contact and timing.
To assess whether K=64 is sufficient for this distinction, we vary the number of prototypes available to represent a stage. Each stage is described by a distribution over those prototypes, so increasing K adds components to that representation, not action labels. The same test gives macro F1 scores of 72.3%, 74.2%, and 74.9% for K=32, 64, and 128, respectively. Contact-state NMI is also similar across these bank sizes (0.344, 0.346, and 0.333, respectively). For the tested distinction, the 64-prototype bank achieves comparable performance while requiring fewer candidates to be scored than K=128.