- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Dex2HOI method for Hand-object interaction synthesis from text. The method supports human interaction with up to 2 objects. The paper propose a network with 2 object-specific streams exchanging information through bidirectional cross-attention and combines their predictions by a motion fusion network, while using hand-object relation representations to encourage coherent interactions.
Critical strengths
-
The method jointly generates full-body motion, articulated fingers, and object motion without additional test-time contact optimization, offering a useful balance between interaction quality and inference efficiency.
-
Hand-relative representation and learned fusion improves performance.
Critical weaknesses
-
The conceptual novelty appears limited despite the empirical gains. HIMO already supports full-body multi-object generation with articulated fingers, while InterGen uses weight-sharing cooperative denoisers with cross-stream attention. The main additions appear to be the learned mixture of left-wrist-relative, right-wrist-relative, and global object translations, and the fusion of the two human-motion predictions. The ablations support their usefulness, but I do not find the technical advance over prior work strong enough. Figure 2 could also illustrate the mixture representation and fusion operation more clearly.
-
The contact results are mixed. In Table 4, Dex2HOI has better CR than LatentHOI, but its right-hand ID is 1.94 cm versus 0.77 cm, and Phy is 81.98 versus 96.16. LatentHOI only models hands, so the tasks differ, but the paper should explain these weaker results. Figure 10 does not clearly show better contact either. These results do not give strong enough support for reliable contact on unseen objects.
Questions for authors
-
What is the main new contribution over HIMO and InterGen? Please explain what the hand-relative object representation and motion fusion add, and show these two parts more clearly in Figure 2.
-
Why are ID and Phy worse than LatentHOI in Table 4, although CR is higher? How do the examples in Figure 10 support better contact on unseen objects?
We thank the reviewer for the valuable feedback and for acknowledging the hand-relative object representations and efficiency balance as our contributions. All concerns raised are valid and we aim to address them:
1. Contributions
Our key contributions in Dex2HOI are the following:
Comparison with other works:
Dex2HOI indeed adopts InterGen’s weight-shared cooperative denoiser - a well established method for coordinating information from independent streams - and we cite it as such. However, how we approach HOI generation for multiple objects is fundamentally different to HIMO: HIMO-Gen independently denoises human and object motion, usually resulting in contact inconsistencies as seen in our evaluation (Table 2, Figure 5). It is also bounded by the number of objects present in the scene: they train different networks for 2 or 3 objects, while Dex2HOI shares the same weights for all settings.
Ablation evidence. Replacing the wrist-relative representation with global object trajectories reduces GRAB R-Prec@3 from 0.593 to 0.437. On HUMOTO, the same ablation worsens FID from 0.655 to 0.950 and increases penetration from 9.8 to 13.8 mm (Table 3). These ablations isolate the contribution of the object representation within Dex2HOI’s architecture.
2. Evaluation against LatentHOI
Dex2HOI is a framework for single-shot, full-body HOI generation. It also models the entire duration of an object interaction: approach, grasp and release. In contrast, LatentHOI is a grasping-only method with a VAE dedicated to static grasp generation, i.e. it always generates only valid grasp frames.
Comparing the two is a challenging test and serves as an evaluation baseline on unseen geometry; We perform this experiment because other full-body baselines (HIMO-Gen, HOIDiNi, CoDA) do not show evaluation on unseen object geometries.
Phy is a metric to assess physical plausibility by calculating the ratio of contact frames in which the object remains off the ground, which affects the approach and place frames, justifying the drop. Recomputing the Phy metric on grasp-only frames of Dex2HOI sequences significantly increases that metric, achieving similar performance as LatentHOI:
To further support our generalization claims, we test grab-trained checkpoints of LatentHOI and Dex2HOI on the DexYCB dataset:
On GRAB, Dex2HOI’s higher contact ratio co-occurs with deeper interpenetration because CR measures contact coverage, while ID measures how far the surfaces overlap. On DexYCB, Dex2HOI has lower CR for both hands, and the ID difference is also smaller against LatentHOI (1.73 vs 1.50 cm for the right hand respectively), which implies that Dex2HOI’s grasping ability transfers to unseen DexYCB objects without retraining better than LatentHOI.
Revised Manuscript
We thank the reviewer for their constructive feedback, especially for Figure 2 (method) and Figure 10 (LatentHOI comparison) and we point to the following updates on the manuscript:
p4, Figure 2: updated method figure now includes detailed motion fusion network architecture, object representation breakdown, and how the diffusion process handles the object motion streams.Further evaluation on unseen geometries against LatentHOI:
p19, L958-L1020: Figures 10 and 11 and Table 5. Updated figures show hand-object contact maps at a 5 mm threshold. Penetrations become clearer that way.We kindly welcome the reviewer to view these changes. Additional video comparisons will be included in the final version of our paper upon publication. We hope this rebuttal strengthens our position and we await the reviewer’s response.