acceptodds
Withdrawn from ICLR 2027

DuetHOI: Language-Guided Bimanual Hand–Object Motion Generation with Articulation Planning and Contact Refinement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Bimanual articulated-object interaction generation requires a model to capture the evolution of object articulation, coordination between the two hands, and fine-grained hand–object contact. Existing methods typically encode object and hand motion as a unified high-dimensional sequence, making it difficult to explicitly accommodate the different scales of manipulation progress, relative bimanual motion, and local hand pose. We propose **DuetHOI**, a structured framework for bimanual articulated-object interaction generation. DuetHOI first uses ContactVAE to predict contact intent on the object surface from language and object geometry. Conditioned on this spatial intent, ArtPlanner provides a trajectory-level articulation reference, and DualFormer generates the global interaction using object states, object-centered hand positions, and compact hand-pose tokens learned by ManiVAE. After global generation, ProxiRefine fixes the object trajectory and residually corrects the two hands using current hand–object proximity, improving local surface alignment. Experiments across articulated and rigid objects and single- and bimanual settings show that DuetHOI outperforms three adapted baselines on most contact and hand–object consistency metrics, with particularly strong results in bimanual articulated interaction generation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Bimanual Hand-Object Interaction, Articulated Object Manipulation, Motion Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RL4rOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses text-conditioned generation of hand–object interactions with rigid and articulated objects. It represents each frame with lightweight tokens and a diffusion transformer alternates attention across tokens within each frame and across frames within each token stream.

Critical strengths

Compact and structured representations for motion diffusion: The paper introduces a compact representation for bimanual hand–object motion generation by separating object states, object-centered hand positions, and low-dimensional hand-pose latents. This avoids directly diffusing high-dimensional MANO poses and provides a structured way to model the different components of the interaction sequence.

Critical weaknesses
  1. The pipeline has several components that have precedents in HOI generation: ContactVAE resembles Text2HOI's contact-map VAE, ManiVAE resembles LatentHOI's GraspVAE, and ProxiRefine builds on existing contact-based pose refinement. The paper names all these components giving the impression that these are standalone contributions but these have limited novelty and perhaps the strength is the light-weight representations in the pipeline. These components (ContactVAE, ManiVAE and ArtPlanner) should be compared against the baselines where possible, not just against ground truth. And the paper should focus more on comparing network sizes or inference times to support the light-weight aspect, as the novelty is limited.

  2. Limited diversity from design choices: The pipeline is largely deterministic - ArtPlanner always decodes its prior mean, DualFormer uses deterministic DDIM (noise adds a little variation), and ProxiRefine is deterministic. As a conseqeunce, DuetHOI has the lowest SD in the primary Bim-Art. setting and the lowest SD/OD ratio in 5 of 6 modes (from Tables 1 and 5). It could be that the method is seeking a mode instead of learning a good distribution. FID, prevision and recall over multiple generates trajectories would be necessary to understand the behavior.

  3. The reported gains in contact and hand accuracy are not consistently accompanied by better object motion reconstruction and this is evident in the GRAB ORE results. I would like to see some analysis/explanation specifically for this metric and why its an outlier compared to baselines for GRAB.

  4. I think DiffH2O from SIGGRAPH Asia 2024 is also a very relevant baseline that should be included. Further, the qualitative results should include visualizations from all baselines.

Questions for authors
  1. Adding DiffH2O would be nice but even without that, model size and efficiency could be quantified against baselines

  2. How can you dissect weakness #2 to see if the generations are merely sticking to ground truth trajectories and if repeated generations are yielding simialr trajectories?

Clarifications:

  1. It would help to understand GRAB ORE numbers with some explanation/visualizations. I want to understand how Phy is the best when ORE is the worst.
Reviewer_vB8SOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets language-guided generation of bimanual hand--object motion, particularly with articulated objects, arguing that encoding object motion, hand trajectories, and hand poses as one unified sequence entangles manipulation progress, bimanual coordination, and local contact.

DuetHOI predicts an object-side contact map with a ContactVAE, plans a scalar articulation trajectory with a conditional VAE (ArtPlanner), encodes MANO poses into 32-D tokens (ManiVAE), generates object state, object-centered hand positions, and hand-pose tokens with a slot/temporal factorized diffusion Transformer (DualFormer), and finally refines the hands with the object trajectory fixed using joint-level proximity cues (ProxiRefine).

With its components trained in stages on all single- and bimanual, rigid and articulated interaction modes of ARCTIC and GRAB, it outperforms adapted Text2HOI, LatentHOI, and MDM baselines on most contact, hand-accuracy, and relative-interaction metrics (e.g., C-F1 0.834 vs. 0.767 on ARCTIC bimanual-articulated), with ablations of each module and module-level evaluations in the appendix.

Critical strengths
  1. A clean, inspectable decomposition of bimanual articulated interaction. Separating object state, object-centered hand positions, and compact hand-pose tokens (Sec. 3.1, Eq. 17), planning articulation explicitly, and refining hands with the object trajectory fixed (Sec. 3.6) gives each sub-problem its own representation and makes the pipeline interpretable. The appendix evaluates the intermediate modules directly (ContactVAE AP/F1 in Table 6, ArtPlanner trajectory error in Table 7, ManiVAE reconstruction of ~2 mm in Table 8), which is more thorough than typical for this line of work.

  2. Consistent gains in contact and hand accuracy across interaction modes with one jointly trained model. On ARCTIC bimanual-articulated, DuetHOI leads eight of ten metrics, improving C-F1 from 0.767 to 0.834, MPJPE_A by 13.4%, and NN-Rel by 41.4% (Table 1), and similar gains hold in the bimanual-rigid and single-hand settings on ARCTIC and GRAB (Tables 1, 2a), all with the same evaluation code and splits for every method. The ablations show that ContactVAE in particular matters (C-F1 0.786 →\rightarrow 0.679 without it, Table 2b).

Critical weaknesses
  1. The novelty relative to Text2HOI is not clearly delineated, and the Text2HOI comparison may not reflect its native design. As described in its original paper, Text2HOI (Cha et al., 2024) already predicts an object contact map from text and object geometry with a VAE, conditions a diffusion generator on it, and refines the hands to improve contact, on ARCTIC and GRAB; DuetHOI uses the text annotations prepared by Text2HOI for both datasets (Appx. D). The related work characterizes it only as a unified-sequence method (Sec. 2.1). In the experiments, Text2HOI is "adapted" by retaining "its direct diffusion formulation for hand--object motion" (Appx. E), and it is not stated whether its contact predictor and hand refiner were kept; if they were removed, the baseline lacks exactly the components that DuetHOI re-introduces as ContactVAE and ProxiRefine. The remaining differences (ArtPlanner, ManiVAE tokens, factorized attention, object-centered positions) give modest gains in the ablation on the primary setting (e.g., MPJPE_A 12.458 without ManiVAE and 12.429 with flattened attention vs. 12.243; Table 2b). This matters because it determines how much of the improvement is due to the paper's new contributions.

    • Suggestion: position DuetHOI against Text2HOI's contact prediction and refinement in the related work, state whether Text2HOI's contact predictor and hand refiner were kept, and report the native Text2HOI if they were not.
  2. Physical plausibility is not fully evaluated, and some reported trade-offs are underexplored. All contact metrics (Phy, C-F1, C-Rate) use an unsigned distance threshold of 5 mm on hand vertices or joints (Appx. G.1, G.3), so hand points that penetrate the object near its surface also count as contact, and no penetration metric is reported, even though the failure case shows interpenetration (Fig. 10) and penetration measures are standard for this task. Phy is close to saturation for DuetHOI in the ARCTIC bimanual settings (0.997--1.000), which limits its discriminative power. DuetHOI also has a larger object rotation error than Text2HOI and MDM on GRAB (ORE 48.12 vs. 31.06/31.24 bimanual; 46.26 vs. 31.75/31.56 single-hand) and the lowest sample diversity on ARCTIC bimanual-articulated (SD 4.489), the data splits (including the "unseen oven" category in Fig. 5) are not specified, and no variance across runs is reported. This matters because the paper's main claims concern contact quality and physical plausibility.

    • Suggestion: report a penetration metric, use a signed distance for the contact metrics, discuss the ORE and diversity trade-offs, specify the data splits, and report variance.
Questions for authors

Clarifications (could change my assessment):

  1. In the Text2HOI baseline, were its contact-map predictor and hand refiner retained? How does DuetHOI compare with Text2HOI run in its original configuration and evaluation protocol on ARCTIC and GRAB?
  2. What penetration depth/volume do DuetHOI and the baselines produce? How do C-F1 and C-Rate change if penetrating vertices are excluded from contact?
  3. Why is DuetHOI's object rotation error on GRAB substantially higher than that of Text2HOI and MDM?
  4. Which train/validation/test splits are used for ARCTIC and GRAB, and which object categories (e.g., the "unseen oven" in Fig. 5) are held out? Are the reported numbers averaged over multiple samples or seeds, and what is the variance?

I am open to revisiting my rating if the rebuttal addresses these points well.

Reviewer_KWa5OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a framework for language-conditioned bimanual hand–object interaction generation, particularly for articulated objects. This is an active line or research with important implications and clearly requires more progress. The main insight of the approach is to decompose the problem. In consecutive steps, the method performs contact prediction, object articulation planning, interaction generation, and a final refinement. While this is claimed as a central contribution, it is not a new idea. Experiments are run on established small datasets (ARCTIC and GRAB), and are compared against dated methods from over two years ago. Qualitative results are limited, and indicate decent performance.

Critical strengths
  • Separating articulation planning, and motion generation into steps is a reasonable approach, which is portrayed nicely in the writing.
  • The experimental results are decent. Some of the qualitative motions portrayed are of high quality
  • Disregarding its lack of recent literature, the structure of the evaluation is done well. It has sufficient ablations and experiments
Critical weaknesses
  • The central problem of the paper is its novelty and relation to previous work. The main insight is decomposing interaction representations and generation into steps. But this has been explored extensively in recent HOI-generation literature, and is often referred to as blueprint HOI. The paper critically misses all of this recent literature that already explores this direction, and how it is distinguished from them. A few examples of such papers are JointHOI, HOiDini, MOCHI, and Blueprint2Motion.

  • This immediately projects on the quality of the evaluation - without comparing to contemporary literature, the proposed evaluation, while nicely executed, does not inform the reader regarding the method's merits.

  • Similarly, the paper employed dated architectures. The approach relies on relatively dated architectural choices, and does not offer a discussion on alternatives or justification for such choices.

Questions for authors

Unfortunately, without a much more thorough literature review, a discussion of differences, and a full evaluation against more recent work, this method cannot be assessed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.