acceptodds
Under review as a conference paper at ICLR 2027

AW3D: Action Prototype Learning with World Modelling for Text-Driven 3D Human-Object Interaction Generation

Abstract

Realistic three-dimensional human–object interaction (HOI) animation is important for character animation, embodied simulation and humanoid control. Although existing text-conditioned HOI generators can produce physically plausible interactions, they remain limited in generating motions that faithfully follow textual instructions, often omitting required actions or executing them with incorrect timing or ordering. Here we present AW3D, an action-prototype framework with interaction world modelling, whose central idea is to explicitly organize high-level interaction intent into an ordered sequence of variable-duration action stages before dense motion generation. AW3D incorporates three key designs: Joint-Embedding Action Prototype Learning, which constructs a data-driven vocabulary of semantically meaningful interaction stages; Distribution-Guided Action Sequence Decomposition, which decomposes high-level intent into an ordered sequence of action prototypes and estimates their durations; and Interaction World Modelling for Role-Guided Dense Motion Generation, which adapts the resulting sequence to coordinate body, hand and object motions according to their roles in the interaction. Extensive experiments demonstrate that AW3D achieves state-of-the-art semantic alignment and motion quality, while remaining competitive in physical plausibility and interaction consistency. Code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D human-object interaction, human motion generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HC2zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AW3D, which introduces action-stage planning between textual instructions and frame-by-frame human-object interaction generation to address omitted actions, ordering, and timing in complex interactions. The method first segments ground-truth interactions into stages based on changes in contact and relative motion, then constructs 64 action prototype classes through joint-embedding predictive learning and a Gaussian mixture model. Given text, object geometry, and the target length, it predicts prototype distributions and stage boundaries. A recurrent interaction-state model then refines the plan and constructs separate framewise conditions for the body, hands, and object to guide diffusion-based generation. The paper reports improvements in semantic alignment and the quality of the generated motion distribution on InterAct, and includes experiments integrating the method with different generators, generating long sequences, handling unseen objects, evaluating on independent datasets, and measuring user preferences.

Critical strengths
  1. Organizing action stages to improve the execution of complex instructions is a worthwhile direction and provides a useful perspective on the relationship between semantic planning and continuous motion generation.
  2. The authors learn reusable prototypes from changes in human-object relationships and use them as planning units. Soft prototype distributions, condition-dependent state adjustments, and role-specific guidance connect coarse plans to the motions of the body, hands, and object.
  3. Beyond the main experiments on InterAct, the authors examine integration of the prototype pathway with different generators, long sequences, and unseen objects, and conduct separate training and evaluation on OMOMO and BEHAVE.
Critical weaknesses
  1. The paper is primarily motivated by improving action content and temporal order. However, in Table 4, shuffling the prototype order even yields slightly better mean values for both metrics. It will be better to explain such observations.
  2. These prototypes are learned mainly from contact and relative-motion features. It remains unclear to what extent they represent the semantic action stages in textual instructions and whether 64 classes are sufficient to preserve important semantic distinctions. The limited semantic gains in the ablations raise this question, but do not establish that information loss in the prototype representation is the cause.
  3. The current results reflect a trade-off among different objectives, and I recommend further analysis of these visible issues and possible directions for improvement.
Questions for authors
  1. How do the authors explain the small changes in semantic metrics after removing the prototype pathway in Table 4, and the slight improvement in some semantic metrics after shuffling the prototype order?
  2. Are there more direct evaluations of event completion or ordering that demonstrate the independent contribution of stage planning?
  3. Have the authors analyzed the semantic action content associated with different prototypes and how the number of prototypes affects semantic discrimination?
4 comments
AuthorsOpenReview ↗

Response to Reviewer HC2z (1/2)

We appreciate the reviewer's careful reading and constructive feedback. We respond to each point below:

W1 & Q1: Interpreting the prototype ablations

We thank the reviewer for raising these points. We believe the small changes in the retrieval scores in Table 4 have two main explanations. Removing the prototype pathway leaves the original text condition in place. Shuffling the plan retains the planned action content but changes its order. In both cases, the generated motion can still match the overall action described by the text, which is what whole-clip retrieval primarily measures.

When the prototype pathway is removed, the diffusion generator still receives the full instruction and can infer the main action from it. This explains why R@3 and MM Dist change little even though FID worsens. The text can guide the generator towards the intended action, while the prototype pathway makes the progression of that action explicit.

Shuffling keeps the same prototype segments and time-slot boundaries but changes which segment occupies each slot. For an instruction to lay a clothes stand flat and then pick it up, the shuffled output performs both actions but lays the stand flat later than AW3D and the reference. The main actions are still recognisable, so the retrieval scores can remain close even when a transition is delayed. Across ten additional paired evaluations with fixed checkpoints, each variant achieves higher R@3 in five pairs, and mean MM Dist is similar between the two variants. Keeping the planned order, however, lowers FID and raises contact F1 in all ten pairs.

We also examined why retrieval remains close after shuffling. We use a parameter-matched GRU control that takes the previous planned prototype as history in place of the adjustment predicted for the previous stage. Table R1 compares ordered and shuffled plans under this control.

Table R1: Effect of prototype order in the GRU control without residual feedback.

MethodR@3 ↑FID ↓MM Dist ↓Div. →FSR ↓Pene ↓Contact →Int. Prec. ↑Int. Rec. ↑Int. F1 ↑
GRU with ordered plan0.8260.1162.4137.8100.0930.1600.1250.7490.5920.616
GRU with shuffled plan0.8150.1302.4377.8020.0890.1660.1270.7330.5840.614

Bold marks the better value. For Diversity and Contact, values closer to the ground truth (7.843 and 0.202) are better.

In this GRU control, shuffling worsens both retrieval metrics. With recurrent refinement, the scores remain close across the paired runs. This contrast suggests that feeding back predicted interaction states helps retain the overall action when the plan order is perturbed.

Taken together, these results explain why removing the prototype pathway or shuffling the plan can leave whole-clip retrieval largely unchanged. In both cases, the generator still has enough information to produce a recognisable interaction. The contribution of the ordered prototype plan is to guide when each requested action occurs and how it follows the previous one. This is why the slight retrieval advantage of shuffling in Table 4 does not indicate better execution of the requested sequence.

W2: Prototype semantics and capacity

We thank the reviewer for asking how the prototypes relate to the action stages in the instructions. We first quantify what action information the prototypes retain, then examine how bank size affects that result.

  1. To quantify what action information the prototypes preserve, we tested whether they distinguish two types of object motion that instructions can specify: moving an object to a different position and changing its orientation. We used 56 matched pairs of reference windows, controlling for object geometry, hand proximity, and temporal phase. The labels were based on displacement and rotation and checked against the instructions and motions. In cross-validation grouped by object geometry, adding the default prototype distributions to hand proximity and phase raises macro F1, the average F1 across transport and reorientation, from 52.2% to 74.2%. This measures how much the prototypes help distinguish transport from reorientation beyond contact and timing.

  2. To assess whether K=64 is sufficient for this distinction, we vary the number of prototypes available to represent a stage. Each stage is described by a distribution over those prototypes, so increasing K adds components to that representation, not action labels. The same test gives macro F1 scores of 72.3%, 74.2%, and 74.9% for K=32, 64, and 128, respectively. Contact-state NMI is also similar across these bank sizes (0.344, 0.346, and 0.333, respectively). For the tested distinction, the 64-prototype bank achieves comparable performance while requiring fewer candidates to be scored than K=128.

AuthorsOpenReview ↗

Response to Reviewer HC2z (2/2)

W3: Trade-offs among evaluation objectives

We thank the reviewer for encouraging us to examine these trade-offs. In Table 1, AW3D achieves better text–motion alignment and lower penetration than Text2HOI, while Text2HOI has less foot sliding and higher contact F1.

  1. Text2HOI's higher contact F1 indicates better agreement with the reference contact labels. This can coexist with greater penetration because a hand joint close to the surface may count as contact even when it lies inside the object. Text2HOI also completes fewer requested stages in the direct evaluation (Table R2). Matching contact labels and following the requested sequence are therefore distinct objectives.

  2. AW3D uses an ordered prototype plan to make the progression of requested actions explicit. Its stronger text–motion alignment and stage completion are consistent with this design, while its lower penetration is a separate advantage. The remaining gaps in contact F1 and foot sliding show where execution within each stage can still improve.

These observations suggest two improvements during sampling: guide the hand to the object surface at the planned contact times while penalising penetration separately, and limit foot motion during stance. Both constraints target execution within each stage without changing the planned action order.

Q2: Direct evaluation of stage completion and ordering

We appreciate the suggestion to measure whether the final motion completes each requested stage. Before reviewing model outputs, we selected 44 multi-stage prompts from the held-out InterAct split; 37 have at least three annotated stages. We manually annotated stage intervals in the reference motions and stage presence and timing in anonymised videos. All five methods received the same prompt, object geometry, and target length for each case. We have added the stage-level results and discussion to Sec. 4.3 of the revised manuscript.

Stage recall measures the fraction of required stages that occur. A stage pair counts as correct only if both stages occur and their onsets follow the requested order. For completed stages, duration error is the absolute difference from the annotated duration, normalised by the reference sequence length.

Table R2: Stage-level evaluation of final generated motions on 44 held-out InterAct prompts.

MethodStage recall (%) ↑Ordered-pair accuracy (%) ↑Norm. duration error ↓ (n)
HOI-Diff34.018.90.212 (27)
Text2HOI41.025.60.172 (31)
CHOIS52.834.80.197 (36)
LIGHT62.739.80.197 (43)
AW3D85.874.50.106 (44)

For duration error, n denotes the number of prompts with at least one completed stage.

Table R2 shows that AW3D leads on all three stage metrics. AW3D completes every required stage in order on 28 of 44 prompts, compared with 13 for LIGHT. On the 79 stages both methods complete, AW3D also has lower mean duration error (0.085 vs. 0.146). These results show that the full system completes more of the requested interaction and better matches its timing.

Q3: Prototype content and granularity

We appreciate the reviewer's question about what different prototypes represent and how the bank size affects the distinctions they capture. We use a matched example to illustrate what the prototypes distinguish, then refer to the overall results in W2 to explain our choice of bank size.

  1. Consider two stages involving the same monitor, prompted by “move the monitor” and “rotate the monitor”. Both occur at similar points in their sequences, with both hands close to the monitor. The first stage mainly changes its position, moving it by about 30 cm, while the second mainly changes its orientation, turning it by about 38°. The classifier using only hand proximity and phase predicts transport for both. Adding the prototype distributions lets it distinguish the two. This example illustrates how a prototype distribution can encode what happens to the object even when hand proximity and timing are similar.

  2. We assess the effect of bank size across the full set of matched pairs in W2. The scores are close across the three bank sizes, so we use K=64 as a practical middle ground, with more prototype components than K=32 and fewer candidates for the planner to score than K=128. The purpose is to keep the plan compact while distinguishing changes in the interaction. The prototype distribution describes what changes, and the stage boundary specifies when it should occur.

Thanks for the authors' response. The explanation is to some extent reasonable to me. However, it is more like a trade-off rather than a necessary option. Are there alternative approaches that can achieve simultaneous improvements?

AuthorsOpenReview ↗

Response to Reviewer HC2z

Thank you for this thoughtful follow-up. We agree that better instruction following need not come at the cost of physical quality. In Table 1, AW3D achieves better text alignment and lower penetration than Text2HOI, while Text2HOI achieves higher contact F1. A hand joint close to the object surface can count as contact even if it lies inside the object, so higher contact F1 alone does not imply more plausible manipulation. Text2HOI uses contact maps to guide hand placement, while AW3D specifies which interaction stages should occur and when. This action-planning pathway can be integrated into different HOI backbones, including Text2HOI itself. In both cross-backbone evaluations (Tables 2 and 3), adding AW3D raises R@3 and lowers MM Dist and FID for Text2HOI, CHOIS, and LIGHT. Contact F1 improves in most comparisons and remains comparable in the others. Compared with the variant without the prototype pathway, full AW3D achieves lower FID and higher contact F1(Table 4). In the blind study, viewers more often preferred animations generated by backbones with AW3D for both instruction following and natural human–object coordination (Appendix E.1).

Following our earlier suggestion to combine action planning with contact-map guidance, we tested a Text2HOI-style contact map for the entire instruction. We trained the contact predictor and hand adapter while keeping AW3D's planner and motion generator frozen. We evaluated the resulting model on the full InterAct test split using the Table 1 protocol.

MethodR@1 ↑R@2 ↑R@3 ↑FID ↓MM Dist ↓Div. →FSR ↓Pene ↓Contact →Int. Prec. ↑Int. Rec. ↑Int. F1 ↑
AW3D0.4850.7230.8260.1012.4017.8520.0880.14900.1260.7560.6030.630
AW3D + contact map0.4940.7200.8280.1172.4007.8440.0850.15410.1280.7470.6070.632

Compared with AW3D alone, adding the Text2HOI-style contact map slightly improves R@3 and contact F1, but raises FID from 0.101 to 0.117. The combined model improves R@3, FID, MM Dist, and contact F1 over the same backbone without either addition (Table 4). Overall, AW3D improves instruction following across different HOI backbones while maintaining competitive physical quality, so the two objectives need not come at the expense of each other. Its action-planning pathway can also be combined with methods that target physical execution, with the potential to improve physical quality further.

Reviewer_41w6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AW3D learns a bank of object-centric interaction-stage prototypes, predicts a variable-duration ordered prototype sequence from text, and uses an interaction world model plus role-guided dense generator to produce body, hand, and object motion. It is evaluated on InterAct and paired with CHOIS, Text2HOI, and LIGHT backbones; the appendix adds object-disjoint and dataset-specific tests.

Critical strengths
  1. The stage-based formulation gives the model an explicit way to represent action order and duration. This targets a common failure in text-driven HOI generation: producing plausible motion while skipping or reordering parts of the instruction.

  2. The InterAct results are strong, particularly for text alignment. Compared with guided LIGHT, R@3 rises from 0.748 to 0.826 and FID falls from 0.171 to 0.101 in Table 1. The authors also add the prototype pathway to three different generators and test it on longer sequences, so the result is not confined to one backbone.

Critical weaknesses
  1. The main claim is that action prototypes keep the requested stages in the right order. I could not find a test of whether the generated motions actually do that. R-Precision and FID would not necessarily catch a skipped or reordered action. Figure 5 evaluates the predicted plan, but not whether the final motion follows it. Could the authors annotate a set of multi-stage prompts and report stage recall and ordering accuracy on the generated results?

  2. Table 4 leaves me unsure how much the order of the prototypes matters. Shuffling them gives a slightly higher R@3 than the full model (0.833 vs. 0.826), although the full model does better on FID and contact F1. How consistent are these differences across runs, and what kinds of errors does shuffling actually cause?

Questions for authors
  • Can you report stage recall, ordering accuracy, and duration error on a held-out set with human-labeled action stages, alongside prompt-level examples where the plan fails?
  • In Sec. 3.4, the “interaction world model” predicts a residual for each prototype embedding and uses it to condition the motion generator. It does not predict the next human or object state. How does this module compare with a similarly sized GRU that simply encodes the planned prototype sequence and passes it to the motion generator? The static-memory ablation in Table 4 is quite close to the full model. A comparison with a parameter-matched temporal conditioner would make its contribution clearer.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 41w6 (1/2)

We appreciate the reviewer's careful reading and constructive feedback. We respond to each point below:

W1 & Q1: Stage completion and temporal order in generated motions

We thank the reviewer for this helpful suggestion. We agree that evaluating the final motions stage by stage is important for determining whether the requested actions are completed in the intended order. We have added the stage-level results and discussion to Sec. 4.3 of the revised manuscript. We also examine cases where the decoded plan fails to separate requested stages and how this affects the final motion. We address these points in three parts.

For the evaluation, we selected a fixed set of 44 temporally demanding prompts from the held-out InterAct split before inspecting the model outputs. The text screen required at least three action-lexicon groups, followed by human inspection of the prompt and reference motion to identify observable stages. The resulting set includes 33 long sequences and 37 prompts with at least three annotated stages. For example, a push–release–pull instruction requires releasing hand contact and reversing object motion; a lift–rotate–place instruction requires an orientation change before placement. These prompts test distinct transitions within one interaction.

We used human annotations to mark stage intervals in the reference motions and to record whether and when each stage occurred in the anonymized generated videos. The five methods share the same prompt, object geometry, and target length per case. Table R1 covers all 44 prompts and 220 reviewed generated videos. Supplement S1 (Stage_Evaluation.zip) provides the full prompt manifest, reference and generated clips, stage annotations, and a script that recomputes the table.

We average scores across prompts. Stage recall counts the fraction of required stages that occur. Ordered-pair accuracy counts a required pair as correct when both stages occur and their onsets follow the requested order. Normalized duration error is the absolute difference from the annotated stage duration, divided by the ground-truth sequence length and averaged over completed stages within each prompt.

In Table R1, AW3D achieves the highest stage recall and ordered-pair accuracy, improving over LIGHT, the strongest baseline on both metrics, by 23.1 and 34.7 percentage points, respectively. On the 51 pairs that both methods realize, both preserve the requested order on almost all pairs. The main gain is in completing more of the requested sequence in order: AW3D completes every required stage in order on 28/44 prompts, compared with 13/44 for LIGHT. AW3D also has the lowest duration error; on the 79 stages completed by both methods, the stage-averaged error is 0.085 vs. 0.146 for LIGHT. Together, these results support our central claim at the level of the final motion: AW3D realizes more of the requested interaction in the intended sequence.

Table R1: Stage-level evaluation of final generated motions on 44 held-out InterAct prompts.

MethodStage recall (%) ↑Ordered-pair accuracy (%) ↑Norm. duration error ↓ (n)
HOI-Diff34.018.90.212 (27)
Text2HOI41.025.60.172 (31)
CHOIS52.834.80.197 (36)
LIGHT62.739.80.197 (43)
AW3D85.874.50.106 (44)

For duration error, n denotes the number of prompts with at least one completed stage.

AuthorsOpenReview ↗

Response to Reviewer 41w6 (2/2)

To examine why some requested stages are still missed despite the overall gains, we compared the decoded plans used for generation with the human-annotated stage boundaries and the corresponding generated videos.

Prompt (abridged)Decoded planFinal motion
Sit on the table, hold its edges, lean back, then sit upright (case 41).The plan has only two events, with the sole boundary at frame 30. The annotated transitions into holding, leaning back, and sitting upright (frames 43, 76, and 166) all fall in the second event.The person sits and holds the edges; the lean-back and return are not realized as distinct stages.
Grasp the trolley handle, pull back, push forward, pull again, then set the trolley flat (case 53).One event spans frames [160, 230), covering both boundaries of the annotated forward push (frames 195 and 220).The person grasps and pulls the trolley back, but skips the forward push, second pull, and final flat placement.

We observe that AW3D can struggle with instructions that require successive changes in posture or object motion. In these examples, the decoded plan groups several requested transitions into a single event, and the final motion misses later actions. In the seated example, AW3D better captures the requested hand contact with the table edges than the baselines, but still misses the lean back and return to an upright pose. In the trolley example, AW3D completes the initial grasp and pull, but the subsequent push, second pull, and flat placement are missing from our result as well as the baseline outputs. These cases suggest that the planner should represent successive requested actions as separate stages.

W2: Does prototype order matter?

We thank the reviewer for this helpful comment. We address both the stability of the scores and what changes in the generated motions.

  1. Across ten additional paired evaluations with fixed checkpoints, each variant achieves higher R@3 in five pairs, and mean MM Dist is similar. Preserving the planned order, however, lowers FID and raises contact F1 in all ten pairs, by 0.013 and 0.023 on average, respectively. Our paired stage-level comparison also finds lower stage recall (76.9% vs. 85.8%) and ordered-pair accuracy (60.9% vs. 74.5%) for the shuffled variant than for Full AW3D.

  2. To understand why retrieval can remain close, we examined how the shuffled plan changes the motion. Shuffling keeps the same prototype segments and slot boundaries but reassigns the segments to different slots. For example, when asked to lay a clothes stand flat and then pick it up, the shuffled output performs both actions but lays the stand flat later than AW3D and the reference motion. Because both actions still appear, whole-clip retrieval can recognise them without detecting the delay.

  3. We further examined whether the planned order helps without residual feedback. Table R2 compares ordered and shuffled plans using the same Plan-only GRU. Preserving the order raises R@3 by 0.011 and lowers MM Dist and FID, showing that the ordered plan provides useful guidance even without residual feedback.

Together, these results show that close retrieval scores can coexist with missed stages or delayed transitions, while preserving prototype order improves stage completion and hand–object contact.

Q2: Does residual feedback help beyond encoding the plan?

We appreciate the suggestion to include a parameter-matched GRU control. Plan-only uses the planned prototype sequence as its recurrent history, while AW3D feeds back its predicted latent interaction states. Shuffled + Plan-only uses the same Plan-only GRU with the shuffled plan.

With the same ordered plan and GRU architecture, Full AW3D achieves lower FID and MM Dist and higher contact F1 than Plan-only (Table R2). In AW3D, the GRU receives the adjustment predicted for the previous stage, so its next prediction builds on how the plan has already been adapted to the current interaction. These results show that residual feedback adds value beyond encoding the plan by helping the generator coordinate hand–object contact as it executes the planned stages.

Table R2: Effect of prototype order and residual feedback on the final motion.

MethodR@3 ↑FID ↓MM Dist ↓Div. →FSR ↓Pene ↓Contact →Int. Prec. ↑Int. Rec. ↑Int. F1 ↑
Full AW3D0.8260.1012.4017.8520.0880.1490.1260.7560.6030.630
Plan-only0.8260.1162.4137.8100.0930.1600.1250.7490.5920.616
Shuffled + Plan-only0.8150.1302.4377.8020.0890.1660.1270.7330.5840.614

Full AW3D uses the Table 1 reference. Bold marks the better completed value for metrics with a preferred direction.

Dear Reviewer 41w6,

Thank you again for taking the time to review our work and for your helpful feedback. In response to your questions, we evaluated stage completion and ordering on held-out InterAct prompts using human annotations (Table R1) and compared AW3D with a plan-only GRU to examine the role of residual feedback (Table R2). We would be grateful to hear whether these results address your concerns. We would be happy to clarify any remaining points.

Best regards,
The authors

Reviewer_mZUgOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents AW3D, a hierarchical framework for text-driven 3D human-object interaction (HOI) generation. The proposed framework consists of three components: (1) joint-embedding action prototype learning, which discovers reusable interaction representations through masked relational prediction and Gaussian mixture modeling; (2) distribution-guided action sequence decomposition, which predicts ordered action prototypes and their durations; and (3) interaction world modeling, which produces context-dependent latent states and role-specific guidance for dense HOI generation. Comprehensive evaluation demonstrates the effectiveness of the framework

Critical strengths
  • The action prototype self-supervised learning is well established and well motivated
  • Comprehensive evaluations are included and the state-of-the-art results are reported. Also, the integration of the proposed action-prototype module into several HOI generation methods suggests that the proposed representation can serve as a generalizable prior.
Critical weaknesses
  • Given that the action prototypes are learned in a self-supervised manner, could the authors further evaluate how well these learned prototypes can generalize, especially to unseen objects?
Questions for authors

As listed in the weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer mZUg

We appreciate the reviewer's careful reading and constructive feedback. We respond to each point below:

W1 & Q1: Generalization to unseen objects

We sincerely thank the reviewer for suggesting that we examine whether the learned prototypes transfer to unseen objects. We address this in three parts.

We froze the encoder and the 64-prototype bank learned on the object-disjoint training split and applied them to 18,725 reference stages from 22 held-out object instances, without any refitting. All 64 prototypes were used, and their assignments were more strongly associated with contact state than with temporal phase (NMI: 0.366 vs. 0.045). This shows that the learned prototypes capture contact-related structure on new object geometries.

To complement this analysis, we also evaluate the generated motions on the same split (Appendix D.1). Adding the action-prototype pathway raises interaction F1 from 0.627 to 0.672 for LIGHT and from 0.564 to 0.575 for our backbone, while reducing FID in both comparisons. Together, these results show that the prototypes remain informative on unseen object instances and that the pathway improves the resulting interactions.

To test transfer beyond InterAct, we evaluated six methods on the 17 3D-FUTURE [1] meshes from the CHOIS unseen-object evaluation. We recomputed the geometry features for each mesh and predicted new plans for the AW3D variants. We used the standard InterAct checkpoints and each method's usual sampling settings. All methods shared the same instructions, meshes, target lengths, and sampling seeds across 34 cases.

Table R1 shows that both AW3D variants exceed the baselines in carry-stage occurrence and ordered completion. LIGHT + AW3D completes the most sequences in order, while Full AW3D achieves the highest completion rate with hand proximity during carry. Together, these results show that prototype-guided generation better preserves the action sequence specified by the text on unseen object geometries.

Table R1: Stage completion and hand--object proximity on unseen 3D-FUTURE meshes (%).

MethodCarry stag↑Ordered completion↑Completion + carry contact↑Carry contact coverage↑Contact frames
HOI-Diff14.75.92.95.938.5
Text2HOI32.429.426.523.961.0
CHOIS29.414.711.818.553.9
LIGHT50.041.232.435.968.9
LIGHT + AW3D73.570.652.955.460.5
Full AW3D67.655.955.961.381.5

Carry requires the object to move horizontally while lifted; ordered completion also requires it to be set down. We count a frame as contact when a wrist or finger joint is within 5 cm of the object surface. Completion with carry contact requires contact in at least half of the airborne frames in a lift--move--place episode. Carry contact coverage is averaged across all 34 cases, with zero assigned when no carry is detected. Contact frames are measured over the full sequence. Bold indicates the highest value in each column.

[1] Fu, H., Jia, R., Gao, L. et al. 3D-FUTURE: 3D Furniture Shape with TextURE. Int J Comput Vis 129, 3313–3337 (2021). https://doi.org/10.1007/s11263-021-01534-z

Dear Reviewer mZUg,

Thank you again for taking the time to review our work and for your helpful feedback. Our response discusses the evaluation on unseen InterAct objects (Appendix D.1) and the matched comparison on 3D-FUTURE meshes (Table R1). We would be grateful to hear whether these results address your question about generalization to unseen objects. We would be happy to clarify any remaining points.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.