acceptodds
Under review as a conference paper at ICLR 2027

MAD-HOI: Masked Autoregressive Diffusion for Generating Articulated Hand Object Interactions from Text

Abstract

Methods for text-based generation of hand-object interaction (HOI) sequences primarily focus on producing smooth, physically plausible trajectories. A truly utilitarian method should additionally support variable-length generation, composite motion sequences, motion completion and infilling, and reliable termination without compromising physical plausibility. Standard diffusion models for HOI generation are typically trained only for text-to-motion generation on atomic motions and require the motion length to be specified a-priori. Autoregressive (AR) methods provide greater sequence-level flexibility, but commonly depend on discrete motion codes, which can lose contact-sensitive motion detail. To address these key limitations, we present a model performing Masked Autoregression with Diffusion for HOI generation (MAD-HOI). Our method starts by encoding hand and object motions in a continuous latent space while keeping them disentangled to maintain stream-wise control. This is followed by a masked autoregressive transformer to predict context features that condition a flow-matching head. MAD-HOI is capable of motion generation for atomic and composite articulated sequences, conditioned motion completion and infilling, as well as EOM (End of Motion) prediction from a single training objective. We provide comprehensive evaluations for these capabilities and benchmark our method on the ARCTIC and GRAB datasets. Our experiments show that MAD-HOI achieves the strongest text–motion alignment and distributional generation quality against baselines on both ARCTIC and GRAB, while maintaining competitive geometric accuracy and physical plausibility. Code will be made public.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hand Object Interaction, Motion Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vm1mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MAD-HOI, a text-conditioned generative framework for articulated hand-object interactions. The method combines a cascaded variational autoencoder with a masked autoregressive transformer and a flow-matching generation head. By representing object motion and left/right hand motions in separate continuous latent streams, the framework supports variable-length motion generation, handedness control, motion completion, infilling, and composite interactions within a unified architecture.

The authors evaluate MAD-HOI on ARCTIC and GRAB, comparing it with existing text-to-HOI generation approaches. Results demonstrate strong performance in text-motion alignment and distributional metrics, while achieving competitive geometric and physical plausibility. Additional experiments examine motion composition, sequence termination, and conditional completion.

Critical strengths
  1. A unified framework with useful generation capabilities. The paper goes beyond conventional text-to-HOI generation by supporting motion completion, infilling, composite sequences, handedness control, and automatic termination within one framework. These capabilities are practically meaningful, especially for generating complex articulated interactions without requiring separately designed modules for individual tasks.

  2. Promising empirical results and structured motion representation. The use of separate continuous latent streams for object and hand motions is well motivated. The tokenizer comparison provides evidence that continuous representations better preserve contact-sensitive motion details than the evaluated discrete alternative. Furthermore, the results on ARCTIC and GRAB demonstrate strong text-motion alignment and distributional quality, supporting the effectiveness of the overall framework.

Critical weaknesses
  1. The methodological novelty relative to existing masked autoregressive diffusion approaches is not sufficiently established. The proposed framework combines continuous motion latents, masked autoregressive prediction, and flow matching, all of which have been explored in prior generative modeling research. More importantly, the paper acknowledges that the concurrent HO-Flow method adopts a closely related formulation involving continuous temporal latents and masked autoregressive latent diffusion. Although MAD-HOI introduces separate modality streams and additional generation capabilities, the paper does not sufficiently isolate which improvements arise from these specific design choices rather than the underlying generative architecture. The absence of a direct comparison with HO-Flow, while understandable given the lack of public code, weakens the evidence supporting the claimed methodological distinction.

  2. The evaluation protocol raises concerns about the reliability of the reported generation improvements. The paper reports substantial improvements in FID, retrieval accuracy, and matching score using a newly trained text-motion evaluator. Although all baselines are evaluated uniformly, the evaluator jointly encodes hand and object motions and differs from those used in previous work. Consequently, it is difficult to determine whether the large improvements reflect genuinely better motion generation or favorable properties of the newly constructed embedding space. For example, Table 2 reports an ARCTIC retrieval accuracy of 0.991 for MAD-HOI, identical to the ground-truth value, and an FID of 0.071. Such strong results would benefit from additional independent validation, including evaluation under an established protocol or more direct physical and semantic measures.

  3. Physical plausibility and long-horizon compositional generalization remain insufficiently demonstrated. Although MAD-HOI achieves strong distributional generation metrics, Table 3 shows that its physical plausibility is not consistently superior. On GRAB, the proposed method obtains a penetration rate of 34.60% and a physical plausibility score of 87.01%, both worse than some competing methods. Furthermore, composite generation is evaluated only on ARCTIC, with sequences limited to 152 frames and combinations of at most four atomic motions. Therefore, the current evidence does not fully establish robust long-horizon compositional generation or reliable physical interaction across more challenging manipulation scenarios.

Questions for authors
  1. Methodological novelty: Could the authors clarify the main technical differences between MAD-HOI and HO-Flow, particularly regarding masked autoregressive generation and continuous latent modeling? Which architectural components are primarily responsible for the improvements?

  2. Evaluation reliability: Since the text-motion evaluator is trained specifically for the proposed evaluation protocol, have the authors examined whether the strong FID and retrieval results remain consistent under an alternative evaluator or independent evaluation setting?

  3. Physical plausibility: Table 3 shows relatively high penetration rates and lower physical plausibility on GRAB. Could the authors clarify the main failure modes and explain why the improvements in text alignment do not consistently translate into physically plausible interactions?

  4. Compositional generalization: How well does MAD-HOI generalize to unseen combinations of atomic actions? Could the authors clarify the proportion of unseen composite combinations in the evaluation and the effect of the 152-frame context limitation?

2 comments
AuthorsOpenReview ↗

Independent evaluation and comparison with HO-Flow

Thank you for the careful review. This comment covers evaluation and HO-Flow; the next covers physical plausibility and composition.

Q2 (W2). Do the results hold under an independent evaluator? Yes. Table 15 (new) re-scores every method with three evaluators that are independent of ours, and MAD-HOI is best on 15 of their 20 scores.

A Text2HOI-style action recognizer and a TMR-style retrieval model. We trained both from scratch with 3 seeds, following Text2HOI's recognizer protocol and TMR (Petrovich et al., ICCV 2023). Neither evaluator reads the MANO parameters that our evaluator reads. They both work on hand joints, and the TMR model also works on object pose. MAD-HOI has the best text-alignment score under both models on both datasets. MAD-HOI reaches a Top-3 accuracy of 0.94 on ARCTIC and 0.98 on GRAB, against 0.83 and 0.93 for DiffH2O, the next-best method. The scores MAD-HOI does not lead are distribution scores, not alignment scores, where the top methods are within a small margin of each other.

HOIGPT's released ARCTIC evaluator. MAD-HOI has the best R@1 (0.400; DiffH2O 0.374; GT 0.417) and the best FID (0.27; DiffH2O 0.58). We encode each clip at its generated length, because HOIGPT's rule of padding to the GT length penalizes a method that predicts its own length. Under that padding rule DiffH2O has the better FID (0.56 against 2.38) and R@1 is a tie. Appendix A.10 reports both protocols.

On the Top-3 of 0.991, equal to GT. Top-3 saturates on ARCTIC: GT itself reaches 0.99, so the strongest methods tie there. R@1 and FID separate them, and Table 19 reports both. A learned evaluator can also score generated clips above held-out GT, because it rewards motion that is typical of the training distribution. Real training clips with the test prompts reach a recognizer accuracy of 0.91 on ARCTIC, against 0.58 for test GT, and DiffH2O exceeds GT in the same way (0.656). A score above GT therefore reflects the evaluator, not motion that is better than real.

Q1 (W1). Differences from HO-Flow, and which components matter. Both methods use a masked autoregressive transformer with a flow-matching head. They differ in how they tokenize an interaction and represent motion in generation. HO-Flow fuses the object and both hands into one token per time step. It generates motions in a frame anchored at the first object pose, and it takes the motion length, the active hands and the initial object pose as inputs. MAD-HOI gives the object, the left hand and the right hand their own tokens with modality embeddings, and it learns idle streams and an end-of-motion (EOM) token. It predicts the length and the active hands itself, in the global frame. HO-Flow released its code after our submission, so we trained it on our splits, without its GraspXL pretraining, with its authors' sampling settings (Tables 15 and 19).

Three takeaways:

MAD-HOI is better aligned with the text while receiving less GT inputs during inference. It is ahead of HO-Flow on 10 of the 11 ARCTIC scores (tied on the other) and on 6 of the 9 GRAB scores of Table 15.

HO-Flow depends on its GT inputs. It has no stopping mechanism, so when scored without the GT length, its GRAB FID becomes more than five times worse (Table 19).

HO-Flow has the better contact on GRAB (Phy 91 against 87) and MAD-HOI on ARCTIC (90 against 77), where we had to add an articulation channel to HO-Flow's released code.

On which components matter:

Ablations (Table 2). Without modality embeddings the generator does not converge (ARCTIC FID 0.071 to 56.7). Without EOM prediction, FID rises from 0.071 to 1.13.

Stream-wise conditioning (Table 7, new). MAD-HOI fixes any stream and generates the others by masking, without retraining. Given the GT object trajectory, hand-joint error falls from 16.0 to 6.5 cm on ARCTIC and from 50.7 to 15.2 cm on GRAB. A fused token cannot express this conditioning by masking. The direction matters for robotics, where retargeting starts from an object and articulation trajectory.

AuthorsOpenReview ↗

Physical plausibility and composition

Q3 (W3). Physical plausibility on GRAB. You are right, and we corrected the paper. On GRAB, MAD-HOI's Phy is 87.0, below Text2HOI (94.5) and DiffH2O (93.2; Table 3).

The gap comes from lost contact, not from penetration. MAD-HOI penetrates in 34.6% of GRAB frames, second to DiffH2O (31.3%) and close to GT (30.6%); Text2HOI is at 45.1%. MADHOI’s penetrations are the shallowest of all methods (2.07 cm; GT 2.16 cm).

Most failures are near misses (Table 17, new). Phy asks whether a hand touches the object in the frames where the object moves. In MAD-HOI's failed frames, the nearest hand vertex is a median 7.5 mm from the object surface, and 57% are within 1 cm: the hand hovers just off the surface. In 10% of them it is more than 5 cm away. Text2HOI and DiffH2O miss by less (median 3.3 and 3.1 mm) likely because Text2HOI adds a contact-refinement network, and DiffH2O generates the grasp in a dedicated stage.

Why strong text alignment does not carry over: The disentangled latent streams provide us with compositional benefits but do not tie the hands and object into a single entity, meaning that they are not constricted to be generated in a tight region. As such, when the interaction naturally goes to a region in the 3D frame which is underrepresented in the dataset, the model’s predictions lose the homogeneous penetration-less contact. We have not applied any refinement to our predictions in table 2 and 3 but can consider applying geometric refinement post-hoc.

Q4 (W3). Unseen combinations and the 152-frame window.

Proportion of unseen combinations. Only 10% of sequences in the composite ARCTIC test set use an ordered action pair not seen in training. Thus, the provided experiment results demonstrate systematic composition, though for a small sample. We removed the sentence that said many compound combinations are unseen, and the paper no longer describes these results as long-horizon. The composite results show that one model generates one-action and two-action prompts of variable length within a window and ends them itself. They do not show composition of unseen pairs.

Explicit chaining beyond the maximum window: One pass generates at most 152 frames. Longer sequences come from chaining (Table 6, new): We generate a single atomic motion with its own EOM, crop this motion and pass the latents from a few frames at its end to the generator as conditions for completion of a new text intent for a new atomic motion. Here the model will impart continuity between the atomic motions but we explicitly control the generation through two inference steps. We test this explicit chaining on unseen combinations of two motions for ARCTIC and GRAB. We see that while the first motion is similar in quality to our base generation, the second chained motion suffers some degradation. However, this degradation is in part due to the lack of matching GT motions as these unseen combined motions correspond to two demonstrations or different humans or different object starting points in the raw mocap data.

We hope the independent evaluation, the HO-Flow comparison and the corrected claims address your soundness concerns. We are glad to run further checks during the discussion.

Reviewer_1DSsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents MAD-HOI, a masked autoregressive diffusion framework for text-driven hand-object interaction generation. The main motivation is to combine the sequence-level flexibility of autoregressive generation with the fine-grained motion quality of continuous latent representations. The method first learns a cascaded VAE that separately represents object, left-hand, and right-hand motion streams, while conditioning hand decoding on object motion. A masked autoregressive transformer then predicts contextual features for a flow-matching head to generate the masked continuous motion latents.

Beyond standard text-to-motion generation, MAD-HOI supports variable-length generation with learned end-of-motion prediction, motion completion and infilling, composite interaction generation, and explicit control of handedness. The method is evaluated on both ARCTIC and GRAB, and shows strong results in text-motion alignment, distributional metrics, geometric accuracy, and physical plausibility. The paper also provides extensive additional evaluations on EOM prediction, completion/infilling, user preference, motion-handedness generalization, and downstream robot retargeting.

Critical strengths
  1. The paper addresses an important limitation of existing HOI generation methods by going beyond atomic text-to-motion generation. MAD-HOI supports variable-length generation, completion, infilling, composite interactions, and handedness control within a unified framework.

  2. The continuous and disentangled motion representation is well designed for hand-object interaction. Separating object, left-hand, and right-hand streams provides useful modality-level control, while the cascaded VAE and geometric losses help preserve interaction-sensitive details.

  3. The motivation for continuous latents is supported by a meaningful tokenizer comparison. The proposed VAE shows substantially lower penetration, higher contact, and smoother motion than the VQ-VAE counterpart.

  4. The experimental evaluation is comprehensive and convincing. MAD-HOI achieves strong text-motion alignment and distributional quality on both ARCTIC and GRAB, while also reporting detailed geometric and physical-plausibility metrics.

  5. The paper provides extensive additional validation, including EOM prediction, completion and infilling, composite motion generation, ablation studies, user evaluation, generalization analysis, and a downstream robot-retargeting demonstration.

Critical weaknesses

Major Weaknesses

  1. Limited generalization to unseen objects and actions. The paper demonstrates promising generalization across handedness, but the appendix reports that similar splits excluding specific actions or objects do not show robust generalization. This is an important limitation for practical text-driven HOI generation. I would encourage the authors to discuss this issue more prominently and clarify whether the limitation mainly comes from data scale, the CLIP-based action representation, or the PointNet-based object representation.
  2. The evidence for compositional generalization could be stronger. Composite motion generation is an attractive capability of MAD-HOI, but the model is trained using 30% of the available compound ARCTIC sequences, including motions composed of multiple atomic interactions. Therefore, it is not fully clear whether the model learns systematic composition or mainly generalizes among similar compound patterns observed during training. A controlled split that holds out specific combinations or orderings of known atomic actions would provide stronger evidence for compositional generalization.
  3. Computational efficiency is insufficiently evaluated. MAD-HOI performs iterative masked generation and uses 18 refinement steps in the flow-matching head. However, I could not find inference-time, throughput, or computational-cost comparisons with the baseline methods. Since the proposed framework introduces additional flexibility through masked autoregressive generation, reporting parameter counts and inference latency would help readers better understand the practical cost of the method.

Minor Weaknesses

  1. The EOM prediction results are useful, but the wording around reliable termination may be slightly strong. The dedicated Text2HOI length predictor still achieves lower raw length prediction errors. MAD-HOI is appealing because termination is integrated into the generation framework without a separate predictor, but the claim could be phrased more carefully.
  2. The VAE versus VQ-VAE comparison could provide more implementation details. Since this experiment supports one of the main motivations of the paper, it would be helpful to report the VQ-VAE codebook size, latent capacity, compression ratio, and whether the same geometric reconstruction losses are used for both tokenizers.
  3. The user study is relatively small. The evaluation protocol is well controlled, but only three raters are collected per dataset. Reporting inter-rater agreement or uncertainty would make the human evaluation more convincing.
  4. Some terminology and presentation could be clarified. The use of "autoregressive" may be slightly confusing because the model uses iterative masked bidirectional prediction rather than conventional strictly causal next-token generation. It would also help to clarify that the "single training objective" refers to the generation stage, since the VAE is trained separately with reconstruction, KL, and geometric losses.
Questions for authors

See weakness

4 comments
AuthorsOpenReview ↗

Generalization and composition

Thank you for the thorough and encouraging review. We agree with all three major points: W1 and W2 led us to narrow the claims, and W3 to add measurements.

W1. Generalization to unseen objects and actions. We agree that this is an important limitation, and Section A.10 now states it and discusses the bottlenecks. “For robustness to object geometries, a stronger geometry conditioning would be required for generating grasps (something different from the simple pointnet-based conditioning). However, scaling up data would be necessary to unlock time-varying grasps and wrist motions.” MAD-HOI generalizes across handedness (Level-2 split: FID 0.071 to 0.137, Top-3 0.991 to 0.914; Table 14), but not robustly to held-out actions or objects. We believe data scale is the main cause: ARCTIC has 11 objects and GRAB 51, with limited action diversity, and we train one model per dataset.

W2. Compositional generalization. Only 10% of sequences in the composite ARCTIC test set use an ordered action pair not seen in training. Thus, the provided experiment results demonstrate systematic composition, though for a small sample. What we can show without retraining is composition by explicit chaining through conditioned completion (Table 6, new). We generate a single atomic motion with its own EOM, crop this motion and pass the latents from a few frames at its end to the generator as conditions for completion of a new text intent for a new atomic motion. Here the model will impart continuity between the atomic motions but we explicitly control the generation through two inference steps. We test this explicit chaining on unseen combinations of two motions for ARCTIC and GRAB. We see that while the first motion is similar in quality to our base generation, the second chained motion suffers some degradation. However, this degradation is in part due to the lack of matching GT motions as these unseen combined motions correspond to two demonstrations or different humans or different object starting points in the raw mocap data.

AuthorsOpenReview ↗

Computational cost analysis

W3. Computational cost. Table 18 (new) reports parameters, network evaluations and inference times on one H200 GPU at batch size 1.

MethodParams (M)Inference Time (s), ARCTIC / GRAB
Text2HOI42.14.1 / 4.1
LatentHOI227.47.4 / 7.4
DiffH2O234.033.1 / 32.9
HO-Flow310.98.4 / 7.5
MAD-HOI320.07.1 / 8.0

MAD-HOI's inference time is in the range of the other latent-space methods: it is slower than Text2HOI, comparable to LatentHOI and HO-Flow, and more than four times faster than DiffH2O. The cost comes from the flow-matching head, which holds 270.9M of the generator's 290.8M parameters. Generation runs 18 unmasking iterations, each with an adaptive ODE solve of the head, about 1,900 head evaluations per ARCTIC sequence. The "18 refinement steps" of the submission referred to these iterations, and Section 4.2 now says so.

AuthorsOpenReview ↗

Addressing minor points

Minor 1. Wording on termination. We agree and changed it. Text2HOI's dedicated length predictor is more accurate than EOM (Table 5). The abstract now says "automatic termination", the conclusion no longer says "reliable", and Section 5 reports the mean error to the range of demonstrated lengths (2.3 frames on ARCTIC, 4.9 on GRAB) in place of "miss it by only 2 to 5 frames". The claim is that termination is part of generation, with no separate predictor.

Minor 2. VAE and VQ-VAE details. Appendix A.2 now gives the configuration. The VQ-VAE has one codebook of 512 codes of dimension 512 for the object and one shared by both hands; the VAE latents are also 512-dimensional. Both tokenizers use the same cascade (object first, then hands decoded given the object), the same temporal compression (4 frames per token, 38 tokens per stream for 152 frames) and the same contact, penetration and distance-map losses; the VQ-VAE adds the commitment loss.

Minor 3. User study. We agree that three raters per dataset support only a coarse conclusion, and the appendix now reports agreement and uncertainty per dataset. Agreement is moderate: Kendall's W for alignment is 0.75 on both datasets, and the mean pairwise Spearman correlation is 0.54. On ARCTIC, MAD-HOI has the best mean rank (2.08) and is preferred over Text2HOI, OpenHOI and LatentHOI in 71% to 88% of trials, with 95% bootstrap intervals above 50%. Against DiffH2O it is preferred in 58.7% of trials, which is not significant but still a majority.

Minor 4. Terminology. Both points are fixed. Section 4.2 states that we use "autoregressive" in the sense of MAR (Li et al., 2024): iterative masked prediction with bidirectional attention, not causal next-token decoding. The abstract and introduction now say "a single generator objective", since the VAE is trained separately with reconstruction, KL and geometric losses. Two results added since your review support strengths you noted. Three independent evaluators confirm the text-alignment ranking (Table 15), and stream-wise conditioning shows the modality-level control directly (Table 7).

We are glad to answer further questions.

Thanks for the author's clarification on my concern. The authors addressed my concerns about the unseen objects, computational cost, termination token, VAE details, and user study. Therefore, I would like to keep my score as weak accept.

Reviewer_aNVMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MAD-HOI models text-conditioned hand-object interactions with continuous latents for the object and the two hands. These latents are interleaved and generated using a masked autoregressive model with a flow-matching prediction head. Modality tokens identify the streams, while learned end-of-motion tokens support variable lengths within a fixed context window. The same model supports generation, completion, and infilling. The paper evaluates on ARCTIC and GRAB using a joint hand-object embedding evaluator, geometric metrics, component ablations, and a user study, with additional experiments on composite ARCTIC motions.

Critical strengths
  1. The structured representation is well suited to controllable HOI generation. Separate object and hand streams allow selective conditioning, while object-conditioned hand decoding provides a useful dependency between their motions.

  2. Integrating generation, completion, infilling, and learned termination in one continuous model has practical value. Including object motion and articulation in the text-motion evaluator also better reflects the interaction being generated.

Critical weaknesses
  1. The methodological novelty appears limited. MARDM already combines continuous motion latents with masked autoregressive diffusion, while HOIGPT already separates hand and object tokens. The cascaded decoder, modality embeddings, and idle/EOM handling are useful adaptations, but the paper does not clearly identify the interaction-modeling advance beyond combining these established ideas.

  2. The contribution to compositional interaction modeling is less clear than the capability list suggests. Completion and infilling largely follow from changing the mask, while EOM selects an endpoint within a fixed context. Composite generation additionally uses compound training sequences within the same 152-frame window. These capabilities support a useful unified system, but leave the claimed advance in long-horizon interaction modeling underdeveloped.

Questions for authors
  1. What specific interaction-modeling limitation of a decomposed HOI representation combined with an existing continuous masked generator is addressed by the cascaded VAE and modality design?

  2. What is the intended scope of the composition contribution: unseen action combinations within the trained window, or a mechanism for extending interaction sequences? Please clarify which aspect goes beyond masked conditional generation and training on compound sequences.

  3. Does EOM represent the end of an annotated clip, sustained inactivity, or completion of the instructed manipulation? How does the formulation distinguish a pause within a composite interaction from its final termination?

3 comments
AuthorsOpenReview ↗

What the per-stream formulation contributes

We thank you for your careful and constructive review. We uploaded a revised PDF in which every change is in blue. The new results are in Section 6 (one added page) and Appendix A.10. Kindly review them in conjunction with the following:

  1. Prior works have either shown a masked generator being applied to a single motion latent for the full SMPL model or have performed hand-object decoupling but in discrete VQ-tokens. (Section 3 shows what this quantization costs, there is penetration in 24.7% of the reconstructed frames, against 2.1% of our VAE.) We show disentanglement of the hand and object streams in the continuous latent space with the intuition that object motion affects the hand motion (cascade) and then apply masking in our generator to enable stream-wise completion.

MAD-HOI is a continuous per-stream formulation of an interaction, one object stream, hand streams decoded given the object (cascaded design), one token per stream with modality embeddings, and learned idle and EOM tokens. We agree that the building blocks for our design choice exists, the contribution is this formulation and the evidence that this design choice matters.

Four results:

A masked generator on decomposed streams is not enough by itself. Without modality embeddings the generator does not converge: ARCTIC FID rises from 0.071 to 56.7, and GRAB FID from 1.08 to 11.0 (Table 2).

MAD-HOI generates handedness and length; the other methods receive them. Every baseline is given the GT active-hand flags, and DiffH2O, LatentHOI and HO-Flow are also given the motion length. MAD-HOI generates the unused hand as an idle stream and ends the motion with EOM. Learning termination also improves generation: without EOM prediction, ARCTIC FID rises from 0.071 to 1.13 (Table 2).

The hands can be generated for a given object, without retraining (Table 7, new). The model is trained only with masks over whole time steps. Fixing the object stream to GT and generating both hands lowers hand-joint error from 16.0 to 6.5 cm on ARCTIC and from 50.7 to 15.2 cm on GRAB.

HO-Flow shares our generator family but fuses all streams into one token. MAD-HOI outperforms the fused-motion latent design as seen in Table 15. Trained on our splits, it trails MAD-HOI on 10/11 ARCTIC scores and 5/9 GRAB scores, although it receives the GT length, active hands and initial object pose.

The third result is the capability we find most useful in practice, especially for robot retargeting pipelines that start from an object and articulation trajectory, and need the hands that follow it.

AuthorsOpenReview ↗

Scope of composition

The purpose of introducing compound sequences in training was to get the model to recognize text that means compositing and automatically detect a sequence end. This is an implicit compound generation. The model needed to get acclimated to words such as ‘and’ that refer to multiple motions. The conditional generation nature of our method also means that explicit chaining is possible and we can generate motions longer than the context length of 152 frames. Mechanism: We can generate a single atomic motion with its own EOM, crop this motion and pass a few frames from its end to the generator as conditions for completion of a new text intent for a new atomic motion. Here the model will impart continuity between the atomic motions but we explicitly control the generation through two inference steps. The metrics for compound motion in Tables 2 and 3 are from the implicit compound generation on unseen arctic compound sequences.

AuthorsOpenReview ↗

EOM semantics

In the tokenizer (VAE), we only learn idle latents that indicate sustained inactivity that comes from padding. In the generator, it corresponds to the end of the intended manipulation and is detected from sustained inactivity. The model has no notion of task success. We apply a threshold on token/latent distance from the learned EOM latents and detect the end of the motion when the distance first falls under 0.5.

Nuances:

An idle hand does not end the motion, because the object and the other hand are still active.

A pause inside a composite motion ends it only if all three streams pause together. This is rare in ARCTIC: at the boundary between two actions the hands keep moving (median 1.8 mm per frame), and 2.6% of our two-action generations stop at that boundary. Overall, 11.0% of them end before 70% of the GT length.

Limitation. The model cannot tell a long joint pause from the end of the task, and Appendix A.10 says so.

We are glad to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.