acceptodds
Under review as a conference paper at ICLR 2027

Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation

Abstract

Recent advances in 4D Human-Object Interaction (HOI) generation have enabled increasingly realistic motion synthesis, particularly for single-object manipulation. Yet current research overlooks an inherent property of human behavior: people naturally coordinate both hands and manipulate multiple objects simultaneously. To address this gap, we present Dex2HOI, a unified diffusion model for single- and two-object HOI synthesis from text. At its core, Dex2HOI employs a Dual-Stream Diffusion approach, where each object is processed in a dedicated interaction stream and coordinated through bidirectional cross-attention. To synthesize the final motion, we introduce a Motion Fusion Network integrated with novel hand-relative object representations and contact-aware conditioning applied across the whole sequence. By sampling the diffusion process autoregressively over prefix-conditioned windows, Dex2HOI generates arbitrarily long sequences at real-time speed omitting redundant test-time optimization, achieving up to ×540 inference speed-up over prior state-of-the-art methods. Extensive evaluation on both single- and two-object benchmarks demonstrates state-of-the-art quantitative results, marking a step beyond conventional single-object HOI generation and toward expressive multi-object manipulation. Code and models will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human-Object Interaction, Human Motion Generation, Diffusion Models, Dexterous Manipulation, Text-to-Motion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DeUTOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Dex2HOI method for Hand-object interaction synthesis from text. The method supports human interaction with up to 2 objects. The paper propose a network with 2 object-specific streams exchanging information through bidirectional cross-attention and combines their predictions by a motion fusion network, while using hand-object relation representations to encourage coherent interactions.

Critical strengths
  1. The method jointly generates full-body motion, articulated fingers, and object motion without additional test-time contact optimization, offering a useful balance between interaction quality and inference efficiency.

  2. Hand-relative representation and learned fusion improves performance.

Critical weaknesses
  1. The conceptual novelty appears limited despite the empirical gains. HIMO already supports full-body multi-object generation with articulated fingers, while InterGen uses weight-sharing cooperative denoisers with cross-stream attention. The main additions appear to be the learned mixture of left-wrist-relative, right-wrist-relative, and global object translations, and the fusion of the two human-motion predictions. The ablations support their usefulness, but I do not find the technical advance over prior work strong enough. Figure 2 could also illustrate the mixture representation and fusion operation more clearly.

  2. The contact results are mixed. In Table 4, Dex2HOI has better CR than LatentHOI, but its right-hand ID is 1.94 cm versus 0.77 cm, and Phy is 81.98 versus 96.16. LatentHOI only models hands, so the tasks differ, but the paper should explain these weaker results. Figure 10 does not clearly show better contact either. These results do not give strong enough support for reliable contact on unseen objects.

Questions for authors
  1. What is the main new contribution over HIMO and InterGen? Please explain what the hand-relative object representation and motion fusion add, and show these two parts more clearly in Figure 2.

  2. Why are ID and Phy worse than LatentHOI in Table 4, although CR is higher? How do the examples in Figure 10 support better contact on unseen objects?

1 comment

We thank the reviewer for the valuable feedback and for acknowledging the hand-relative object representations and efficiency balance as our contributions. All concerns raised are valid and we aim to address them:


1. Contributions

Our key contributions in Dex2HOI are the following:

  1. We propose an object motion representation which can handle all manipulation phases at once (approach, bimanual, single-hand - as seen in Figure 7) without requiring any further adaptation for each stage of the manipulation. Each object's translation is a learned, per-frame mixture of left-wrist-relative, right-wrist-relative and global components (Eq. 3), so which hand controls the object is a predicted quantity and the object trajectory inherits that hand's motion.
  2. We sample motion in a single-shot pass, which increases efficiency in contrast to other works.
  3. We train a single HOI Diffusion model that can handle both single and multiple HOI. The motion fusion network merges the two stream full-body predictions into one body.

Comparison with other works:

Dex2HOI indeed adopts InterGen’s weight-shared cooperative denoiser - a well established method for coordinating information from independent streams - and we cite it as such. However, how we approach HOI generation for multiple objects is fundamentally different to HIMO: HIMO-Gen independently denoises human and object motion, usually resulting in contact inconsistencies as seen in our evaluation (Table 2, Figure 5). It is also bounded by the number of objects present in the scene: they train different networks for 2 or 3 objects, while Dex2HOI shares the same weights for all settings.

Ablation evidence. Replacing the wrist-relative representation with global object trajectories reduces GRAB R-Prec@3 from 0.593 to 0.437. On HUMOTO, the same ablation worsens FID from 0.655 to 0.950 and increases penetration from 9.8 to 13.8 mm (Table 3). These ablations isolate the contribution of the object representation within Dex2HOI’s architecture.


2. Evaluation against LatentHOI

Dex2HOI is a framework for single-shot, full-body HOI generation. It also models the entire duration of an object interaction: approach, grasp and release. In contrast, LatentHOI is a grasping-only method with a VAE dedicated to static grasp generation, i.e. it always generates only valid grasp frames.

Comparing the two is a challenging test and serves as an evaluation baseline on unseen geometry; We perform this experiment because other full-body baselines (HIMO-Gen, HOIDiNi, CoDA) do not show evaluation on unseen object geometries.

Phy is a metric to assess physical plausibility by calculating the ratio of contact frames in which the object remains off the ground, which affects the approach and place frames, justifying the drop. Recomputing the Phy metric on grasp-only frames of Dex2HOI sequences significantly increases that metric, achieving similar performance as LatentHOI:

MethodIV R (cm³) ↓IV L (cm³) ↓ID R (cm) ↓ID L (cm) ↓CR R (%) ↑CR L (%) ↑Phy (%) ↑Phy grasp-only (%) ↑
LatentHOI6.381.660.770.2911.941.1196.1697.8
Dex2HOI7.392.591.940.7515.027.3581.9896.3

To further support our generalization claims, we test grab-trained checkpoints of LatentHOI and Dex2HOI on the DexYCB dataset:

MethodIV R ↓IV L ↓ID R ↓ID L ↓CR R % ↑CR L % ↑Phy (%) ↑Phy grasp-only (%) ↑
LatentHOI12.100.891.500.1218.241.7695.896.0
Dex2HOI11.210.461.730.0713.730.6585.094.5

On GRAB, Dex2HOI’s higher contact ratio co-occurs with deeper interpenetration because CR measures contact coverage, while ID measures how far the surfaces overlap. On DexYCB, Dex2HOI has lower CR for both hands, and the ID difference is also smaller against LatentHOI (1.73 vs 1.50 cm for the right hand respectively), which implies that Dex2HOI’s grasping ability transfers to unseen DexYCB objects without retraining better than LatentHOI.


Revised Manuscript

We thank the reviewer for their constructive feedback, especially for Figure 2 (method) and Figure 10 (LatentHOI comparison) and we point to the following updates on the manuscript:

  • p4, Figure 2: updated method figure now includes detailed motion fusion network architecture, object representation breakdown, and how the diffusion process handles the object motion streams.

  • Further evaluation on unseen geometries against LatentHOI: p19, L958-L1020: Figures 10 and 11 and Table 5. Updated figures show hand-object contact maps at a 5 mm threshold. Penetrations become clearer that way.

We kindly welcome the reviewer to view these changes. Additional video comparisons will be included in the final version of our paper upon publication. We hope this rebuttal strengthens our position and we await the reviewer’s response.

Reviewer_vzp2OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Dex2HOI, a unified diffusion-based framework for text-conditioned full-body human-object interaction generation. The proposed approach introduces a hand-relative object representation that decomposes object translation into relative component to the end-effectors, and global components, with learnable mixture weights.

Critical strengths
  • Proposed hand-relative representation is simple yet effective in multiple object interaction synthesis
  • State-of-the-art results are reported.
  • Unified multiple-object interaction generation method, and has the potential to extend to further more object numbers.
Critical weaknesses
  • Novelty is somewhat limited, as some of the distance designs and dual-stream for motion generation are widely used in previous methods [1,2,3].
  • I appreciate the motivation behind combining hand-relative and global object representations, as this design captures both local hand-object relationships and global motion. However, I am concerned about whether this representation remains robust in more complex multi-object manipulation scenarios, particularly those involving prolonged reaching or approaching phases before contact is established, and those long-horizon tasks. I would really appreciate if the author can provide some evaluation on complex benchmarks [4], and some long-horizon task.

[1] Ron, Roey, et al. "Hoidini: Human-object interaction through diffusion noise optimization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026. [2]. Peng, Xiaogang, et al. "Surface Keypoint Representation for Multi-Object and Articulated Human-Object Interaction Generation." arXiv preprint arXiv:2608.03158 (2026). [3]. Liang, Han, et al. "Intergen: Diffusion-based multi-human motion generation under complex interactions." International Journal of Computer Vision 132.9 (2024): 3463-3483. [4]. Kim, Jeonghwan, et al. "Parahome: Parameterizing everyday home activities towards 3d generative modeling of human-object interactions." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Questions for authors

Listed in the weaknesses.

2 comments

We sincerely thank the reviewer for their very thoughtful feedback. We appreciate their recognition of the simplicity and effectiveness of our method for multi-object HOI. We would like to clarify how Dex2HOI builds on prior work and to discuss how the hand-relative object motion representations handle prolonged approach phases and longer sequences using autoregressive generation:


1. Novelty

Our key contributions are:

  1. We propose an object motion representation that can handle all manipulation phases at once (approach, bimanual, single-hand as seen in Figure 7) without requiring any further adaptation for each stage of the manipulation.
  2. We sample motion in a single-shot pass, which increases efficiency in contrast to other works.
  3. We train a single HOI Diffusion model that can handle both single and multiple HOI.

Comparison with other works

Dex2HOI indeed adopts InterGen’s weight-shared cooperative denoiser - a well established method for coordinating information from independent streams - and we cite it as such. However, how we approach HOI generation for multiple objects is fundamentally different to HIMO: HIMO-Gen independently denoises human and object motion, usually resulting in contact inconsistencies as seen in our evaluation (Table 2, Figure 5). It is also bounded by the number of objects present in the scene: they train different networks for 2 or 3 objects, while Dex2HOI shares the same weights for all settings.

On the other hand, both SK-HOI and HOIDiNi are methods that rely on post-hoc optimization. SK-HOI reports 4 minutes per 124-frame sequence (100 optimization steps) while HOIDiNi takes on average 2 minutes per 60-frame sequence (200 optimization steps). In contrast, Dex2HOI samples motion in single-shot passes, achieving fast inference without sacrificing plausibility (useful for downstream tasks for HOI like robotics). Please note that, SK-HOI has not open-sourced training weights and we therefore rely on HIMO-Gen for multi-object evaluation.


2. Generation on ParaHome

We agree that the ParaHome dataset is a more complex motion dataset, including bimanual motion combined with useful in-scene context. We retrain both models (HIMO-Gen and Dex2HOI) on ParaHome bimanual scenarios:

Test set selection: We hold out every 10th capture from the 207 ParaHome captures, yielding 20 test captures. From those, we select eligible bimanual two-object segments and after filtering, 146 training captures and 16 test captures contribute 391 and 52 segments, respectively, as shown in the following table:

MetricTrainTest
Two-object segments39152
Tasks (distinct text prompts)2813
Objects involved1410
Object pairs149
Source captures14616

Long-horizon generation. Dex2HOI generates each test segment autoregressively at its full length. The longest segment spans 555 frames (18.5s, 30 fps), approximately three times the mean duration of a bimanual HUMOTO segment (while median episode length for HUMOTO is 4 sec and ParaHome 8 sec). To assess whether interaction quality degrades as generation progresses, we report contact F1 and object floating in the following table. Contact accuracy does not degrade over the horizon: contact F1 is 0.69, 0.63 and 0.70 over 0–4 s, 4–8 s and 8+ s. The hand-relative representation retains grasps: the generated hand stays within 5 cm of the object on 88.8% of ground-truth hold frames as seen in the table below (Contact Ret. is contact retention after 8 s (≤2 cm / ≤5 cm)):

MethodContact F1 (P/R)Contact ret.Object floatingObject position errorFID ↓MPJPE ↓
Ground Truth1.000100%/100%1.0%0 cm——
HIMO-Gen0.520 (0.42/0.69)61.1%/86.2%28.1%41.7 cm0.18022.8 cm
Dex2HOI0.676 (0.67/0.68)69.0%/88.8%12.3%18.6 cm0.02614.6 cm

Overall, our hand-object representation is specifically designed to model long approach and placement of objects in complex scenarios. In our formulation, object movement is tied to the wrists, and prompt action information like ‘grasps’ and ‘lifts’ conditions the model to limit approach distances between hands and objects (relative to the first frame, as initialization).


Revised Manuscript

We hope that our rebuttal answers the reviewer’s concerns on Dex2HOI's contribution and on more challenging long horizon generation. We invite the reviewer to view the following updates on the pdf:

  • Related work section mentions sk-hoi as concurrent research L131-L135.

  • Appendix now includes evaluation on ParaHome bimanual scenarios: p23: Tables 8 and 9, Figure 15.

  • Additional video comparisons will be included in the final version of our paper upon publication.

We thank the reviewer for their valuable remarks and we await any further feedback.

Thanks the authors for the detailed rebuttal. I think the rebuttal has resolved my concerns.

Reviewer_sgweOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Dex2HOI generates full-body human–object interactions, including articulated hand motion and trajectories for one or two objects, conditioned on text, object geometry, and a motion prefix. It combines dual-stream diffusion, hand-relative object representations, and a motion fusion network, with contact-aware training losses that avoid additional test-time optimization. Autoregressive sampling supports longer sequences. Experiments on GRAB, HUMOTO, and HIMO evaluate generation quality, contact accuracy, and efficiency.

Critical strengths
  1. The paper addresses a meaningful problem by jointly modeling full-body motion, detailed hand articulation, and two-object manipulation within a unified framework supporting both single- and two-object interactions.
  2. The hand-relative representation and dual-stream architecture provide useful inductive biases for coordinated interactions. Table 3 supports their effectiveness, showing improvements in FID and penetration when incorporating the proposed representation, dual-stream design, and motion fusion.
Critical weaknesses
  1. Generality across object counts remains limited. Dex2HOI models at most two independent objects per window, while the extensions in Section 9 rely on sequential switching or rigid grouping.
  2. The HIMO evaluation compares only against MDM and HIMO-Gen, omitting recent strong baselines such as PAMotion (Di et al., CVPR 2026) and MP-HOI (Wang et al., 2026). These omissions limit the evidence supporting the claimed state-of-the-art performance.
Questions for authors
  1. The authors should clarify whether the single- and two-object results use the same jointly trained checkpoint and whether support for more than two objects is limited to sequential switching or rigid grouping.
  2. The omission of PAMotion and MP-HOI from the HIMO comparison should be explained, including any differences in task settings or limitations in implementation availability that prevent a fair comparison.
  3. The higher right-hand penetration and lower Phy score than LatentHOI in Table 4 require further explanation. Their implications for physically plausible grasping on unseen objects should be discussed in light of Dex2HOI’s broader full-body task.
5 comments

We sincerely thank the reviewer for their constructive feedback and for recognizing Dex2HOI’s contribution to multi-object full-body HOI generation. We aim to address each question, clarifying our contributions and embedding suggested changes into the revised manuscript.


1. Interacting with more than two objects

Dex2HOI predicts motion for two objects per window, and sequential switching allows interaction with more objects over time, following contact intent and natural adaptation to more cluttered scenes. The hand-relative representations allow for diverse object control: One hand can move multiple objects, both hands can work together, or each hand can move a different object (as seen in Fig. 7). Cases with simultaneous interaction with three or more objects however are bounded by the scarcity of available 4D HOI datasets:

Dataset evidence. In HIMO’s three-object split, 45.7% of frames involve exactly one moving object, 8.2% involve two, and only 0.13% involve three, during brief placements such as setting a cube onto a cylinder. Thus, all frames involve at most two moving objects. In HUMOTO, rigid grouping handles objects that move together, such as items carried in a basket.

However, to show that Dex2HOI generalizes to n-objects per hand we generate 3-object examples: Since each slot predicts its object relative to the wrists, a third object can be added at inference time without retraining: As shown in Fig. 17, we first generate a two object interaction (grasped by the same hand), and a second pass inpaints the interaction with the third object.

Training. Dex2HOI’s architecture supports unified training: an unused object is masked in both input and loss. In our ablations, we train different variants: one- object checkpoints, two -object checkpoints with unified training across HUMOTO. We also transfer a grasping prior from GRAB (SMPL-X) to HUMOTO (Mixamo). The unified training variant outperforms baselines (Table 2):

MethodFID ↓Diversity→MM ↑Pen ↓R-Pre@3 ↑MMdist ↓
HIMO-Gen0.8860.6060.01911.70.2951.281
GRAB Transfer0.5760.7660.10911.40.2751.259
Unified Training0.6550.7800.0569.80.2751.260

2. Recent Baselines

We have deliberately tried to evaluate Dex2HOI against all SOTA open-sourced full-body HOI works. Both PAMotion and MP-HOI have not open sourced their code. Similarly, SK-HOI hasn’t open sourced their checkpoints and evaluation.

To our knowledge, no other recent paper releases evaluation baselines on multiple objects, so we rely on HIMO-Gen for multi-object evaluation (while also comparing against single-object SOTA). We plan to fully release code and models upon publication, so future research can rely on more diverse evaluation.

We’ve added the aforementioned works in our related works section in the revised pdf (L131–L135), due to their high relevance to Dex2HOI.

3. LatentHOI Comparison

Dex2HOI is a framework for single-shot, full-body HOI generation. It also models the entire duration of an object interaction: approach, grasp and release. In contrast, LatentHOI is a grasp-only method with a dedicated VAE to static grasp generation, i.e. it always generates valid grasp frames.

Comparing the two works is a challenging stress test and serves as an evaluation baseline on unseen geometry; We perform this experiment because other full-body baselines (HIMO-Gen, HOIDiNi, CoDA) do not show evaluation on unseen object geometries. That being said, the Phy metric is a heuristic to assess physical plausibility by calculating the ratio of contact frames in which the object remains off the ground, which affects the approach and place frames, justifying the 10% drop.

Recomputing the Phy metric on grasp-only frames of Dex2HOI sequences significantly raises that metric, achieving similar performance as LatentHOI:

MethodIV R/L ↓ID R/L ↓CR R/L % ↑Phy % ↑Phy grasp only % ↑
LatentHOI6.38/1.660.77/0.2911.94/1.1196.1697.8
Dex2HOI7.39/2.591.94/0.7515.02/7.3581.9896.3

To further support our generalization claims, we test grab-only checkpoints of LatentHOI and Dex2HOI on DexYCB objects:

MethodIV R/L ↓ID R/L ↓CR R/L % ↑Phy % ↑Phy grasp only % ↑
LatentHOI12.10/0.891.50/0.1218.24/1.7695.896.0
Dex2HOI11.21/0.461.73/0.0713.73/0.6585.094.5

DexYCB results are included in the revised manuscript along with Figures 10 (GRAB) and 11 (DexYCB) showing hand-object contact maps at 5mm thresholds. Penetrations become clearer that way.


Revised Manuscript

We welcome the reviewer to view the following changes on the updated PDF:

  • L1270-L1288 Sequential 3-object HOI
  • L131-L135 Updated Related Work
  • L958-L1020 Figures 10,11 and Table 5. Extended LatentHOI comparisons

We hope our response addresses the reviewer’s concerns adequately and we kindly welcome all further feedback.

While I acknowledge that these works are not open-source, you should still account for these stronger baselines in your table to more convincingly validate your method.

We thank the reviewer for responding to our rebuttal and we want to address their concern:

Both baselines (MP-HOI and PAMotion) use their own internally pretrained - not open sourced - motion evaluator, so metrics such as FID, R-Precision and Diversity are computed in different feature spaces and cannot be compared directly.

Without released evaluators and checkpoints or access to the generated motions of each baseline, we cannot evaluate these methods with the same evaluator used in our table. We have included an elaborate section in the Related Works section (L131-L135) where we thoroughly compare Dex2HOI with both baselines.

Could you please clarify how you would suggest including these baselines in our tables given that evaluators, checkpoints, generated outputs, and inference code are unavailable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.