- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes MAD-HOI, a text-conditioned generative framework for articulated hand-object interactions. The method combines a cascaded variational autoencoder with a masked autoregressive transformer and a flow-matching generation head. By representing object motion and left/right hand motions in separate continuous latent streams, the framework supports variable-length motion generation, handedness control, motion completion, infilling, and composite interactions within a unified architecture.
The authors evaluate MAD-HOI on ARCTIC and GRAB, comparing it with existing text-to-HOI generation approaches. Results demonstrate strong performance in text-motion alignment and distributional metrics, while achieving competitive geometric and physical plausibility. Additional experiments examine motion composition, sequence termination, and conditional completion.
Critical strengths
-
A unified framework with useful generation capabilities. The paper goes beyond conventional text-to-HOI generation by supporting motion completion, infilling, composite sequences, handedness control, and automatic termination within one framework. These capabilities are practically meaningful, especially for generating complex articulated interactions without requiring separately designed modules for individual tasks.
-
Promising empirical results and structured motion representation. The use of separate continuous latent streams for object and hand motions is well motivated. The tokenizer comparison provides evidence that continuous representations better preserve contact-sensitive motion details than the evaluated discrete alternative. Furthermore, the results on ARCTIC and GRAB demonstrate strong text-motion alignment and distributional quality, supporting the effectiveness of the overall framework.
Critical weaknesses
-
The methodological novelty relative to existing masked autoregressive diffusion approaches is not sufficiently established. The proposed framework combines continuous motion latents, masked autoregressive prediction, and flow matching, all of which have been explored in prior generative modeling research. More importantly, the paper acknowledges that the concurrent HO-Flow method adopts a closely related formulation involving continuous temporal latents and masked autoregressive latent diffusion. Although MAD-HOI introduces separate modality streams and additional generation capabilities, the paper does not sufficiently isolate which improvements arise from these specific design choices rather than the underlying generative architecture. The absence of a direct comparison with HO-Flow, while understandable given the lack of public code, weakens the evidence supporting the claimed methodological distinction.
-
The evaluation protocol raises concerns about the reliability of the reported generation improvements. The paper reports substantial improvements in FID, retrieval accuracy, and matching score using a newly trained text-motion evaluator. Although all baselines are evaluated uniformly, the evaluator jointly encodes hand and object motions and differs from those used in previous work. Consequently, it is difficult to determine whether the large improvements reflect genuinely better motion generation or favorable properties of the newly constructed embedding space. For example, Table 2 reports an ARCTIC retrieval accuracy of 0.991 for MAD-HOI, identical to the ground-truth value, and an FID of 0.071. Such strong results would benefit from additional independent validation, including evaluation under an established protocol or more direct physical and semantic measures.
-
Physical plausibility and long-horizon compositional generalization remain insufficiently demonstrated. Although MAD-HOI achieves strong distributional generation metrics, Table 3 shows that its physical plausibility is not consistently superior. On GRAB, the proposed method obtains a penetration rate of 34.60% and a physical plausibility score of 87.01%, both worse than some competing methods. Furthermore, composite generation is evaluated only on ARCTIC, with sequences limited to 152 frames and combinations of at most four atomic motions. Therefore, the current evidence does not fully establish robust long-horizon compositional generation or reliable physical interaction across more challenging manipulation scenarios.
Questions for authors
-
Methodological novelty: Could the authors clarify the main technical differences between MAD-HOI and HO-Flow, particularly regarding masked autoregressive generation and continuous latent modeling? Which architectural components are primarily responsible for the improvements?
-
Evaluation reliability: Since the text-motion evaluator is trained specifically for the proposed evaluation protocol, have the authors examined whether the strong FID and retrieval results remain consistent under an alternative evaluator or independent evaluation setting?
-
Physical plausibility: Table 3 shows relatively high penetration rates and lower physical plausibility on GRAB. Could the authors clarify the main failure modes and explain why the improvements in text alignment do not consistently translate into physically plausible interactions?
-
Compositional generalization: How well does MAD-HOI generalize to unseen combinations of atomic actions? Could the authors clarify the proportion of unseen composite combinations in the evaluation and the effect of the 152-frame context limitation?
2 comments
Physical plausibility and composition
Q3 (W3). Physical plausibility on GRAB. You are right, and we corrected the paper. On GRAB, MAD-HOI's Phy is 87.0, below Text2HOI (94.5) and DiffH2O (93.2; Table 3).
The gap comes from lost contact, not from penetration. MAD-HOI penetrates in 34.6% of GRAB frames, second to DiffH2O (31.3%) and close to GT (30.6%); Text2HOI is at 45.1%. MADHOI’s penetrations are the shallowest of all methods (2.07 cm; GT 2.16 cm).
Most failures are near misses (Table 17, new). Phy asks whether a hand touches the object in the frames where the object moves. In MAD-HOI's failed frames, the nearest hand vertex is a median 7.5 mm from the object surface, and 57% are within 1 cm: the hand hovers just off the surface. In 10% of them it is more than 5 cm away. Text2HOI and DiffH2O miss by less (median 3.3 and 3.1 mm) likely because Text2HOI adds a contact-refinement network, and DiffH2O generates the grasp in a dedicated stage.
Why strong text alignment does not carry over: The disentangled latent streams provide us with compositional benefits but do not tie the hands and object into a single entity, meaning that they are not constricted to be generated in a tight region. As such, when the interaction naturally goes to a region in the 3D frame which is underrepresented in the dataset, the model’s predictions lose the homogeneous penetration-less contact. We have not applied any refinement to our predictions in table 2 and 3 but can consider applying geometric refinement post-hoc.
Q4 (W3). Unseen combinations and the 152-frame window.
Proportion of unseen combinations. Only 10% of sequences in the composite ARCTIC test set use an ordered action pair not seen in training. Thus, the provided experiment results demonstrate systematic composition, though for a small sample. We removed the sentence that said many compound combinations are unseen, and the paper no longer describes these results as long-horizon. The composite results show that one model generates one-action and two-action prompts of variable length within a window and ends them itself. They do not show composition of unseen pairs.
Explicit chaining beyond the maximum window: One pass generates at most 152 frames. Longer sequences come from chaining (Table 6, new): We generate a single atomic motion with its own EOM, crop this motion and pass the latents from a few frames at its end to the generator as conditions for completion of a new text intent for a new atomic motion. Here the model will impart continuity between the atomic motions but we explicitly control the generation through two inference steps. We test this explicit chaining on unseen combinations of two motions for ARCTIC and GRAB. We see that while the first motion is similar in quality to our base generation, the second chained motion suffers some degradation. However, this degradation is in part due to the lack of matching GT motions as these unseen combined motions correspond to two demonstrations or different humans or different object starting points in the raw mocap data.
We hope the independent evaluation, the HO-Flow comparison and the corrected claims address your soundness concerns. We are glad to run further checks during the discussion.
Independent evaluation and comparison with HO-Flow
Thank you for the careful review. This comment covers evaluation and HO-Flow; the next covers physical plausibility and composition.
Q2 (W2). Do the results hold under an independent evaluator? Yes. Table 15 (new) re-scores every method with three evaluators that are independent of ours, and MAD-HOI is best on 15 of their 20 scores.
A Text2HOI-style action recognizer and a TMR-style retrieval model. We trained both from scratch with 3 seeds, following Text2HOI's recognizer protocol and TMR (Petrovich et al., ICCV 2023). Neither evaluator reads the MANO parameters that our evaluator reads. They both work on hand joints, and the TMR model also works on object pose. MAD-HOI has the best text-alignment score under both models on both datasets. MAD-HOI reaches a Top-3 accuracy of 0.94 on ARCTIC and 0.98 on GRAB, against 0.83 and 0.93 for DiffH2O, the next-best method. The scores MAD-HOI does not lead are distribution scores, not alignment scores, where the top methods are within a small margin of each other.
HOIGPT's released ARCTIC evaluator. MAD-HOI has the best R@1 (0.400; DiffH2O 0.374; GT 0.417) and the best FID (0.27; DiffH2O 0.58). We encode each clip at its generated length, because HOIGPT's rule of padding to the GT length penalizes a method that predicts its own length. Under that padding rule DiffH2O has the better FID (0.56 against 2.38) and R@1 is a tie. Appendix A.10 reports both protocols.
On the Top-3 of 0.991, equal to GT. Top-3 saturates on ARCTIC: GT itself reaches 0.99, so the strongest methods tie there. R@1 and FID separate them, and Table 19 reports both. A learned evaluator can also score generated clips above held-out GT, because it rewards motion that is typical of the training distribution. Real training clips with the test prompts reach a recognizer accuracy of 0.91 on ARCTIC, against 0.58 for test GT, and DiffH2O exceeds GT in the same way (0.656). A score above GT therefore reflects the evaluator, not motion that is better than real.
Q1 (W1). Differences from HO-Flow, and which components matter. Both methods use a masked autoregressive transformer with a flow-matching head. They differ in how they tokenize an interaction and represent motion in generation. HO-Flow fuses the object and both hands into one token per time step. It generates motions in a frame anchored at the first object pose, and it takes the motion length, the active hands and the initial object pose as inputs. MAD-HOI gives the object, the left hand and the right hand their own tokens with modality embeddings, and it learns idle streams and an end-of-motion (EOM) token. It predicts the length and the active hands itself, in the global frame. HO-Flow released its code after our submission, so we trained it on our splits, without its GraspXL pretraining, with its authors' sampling settings (Tables 15 and 19).
Three takeaways:
MAD-HOI is better aligned with the text while receiving less GT inputs during inference. It is ahead of HO-Flow on 10 of the 11 ARCTIC scores (tied on the other) and on 6 of the 9 GRAB scores of Table 15.
HO-Flow depends on its GT inputs. It has no stopping mechanism, so when scored without the GT length, its GRAB FID becomes more than five times worse (Table 19).
HO-Flow has the better contact on GRAB (Phy 91 against 87) and MAD-HOI on ARCTIC (90 against 77), where we had to add an articulation channel to HO-Flow's released code.
On which components matter:
Ablations (Table 2). Without modality embeddings the generator does not converge (ARCTIC FID 0.071 to 56.7). Without EOM prediction, FID rises from 0.071 to 1.13.
Stream-wise conditioning (Table 7, new). MAD-HOI fixes any stream and generates the others by masking, without retraining. Given the GT object trajectory, hand-joint error falls from 16.0 to 6.5 cm on ARCTIC and from 50.7 to 15.2 cm on GRAB. A fused token cannot express this conditioning by masking. The direction matters for robotics, where retargeting starts from an object and articulation trajectory.