acceptodds
Withdrawn from ICLR 2027

DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Physics-based character control for dexterous Human-Object Interaction (HOI) relies heavily on motion-capture data, which limits its applicability beyond captured scenarios. Meanwhile, video generative models can synthesize realistic HOI videos for unseen objects and scenarios directly from text prompts, but 3D interactions learned from such synthetic data remain kinematic without ensuring physical plausibility. Imitating these videos with physics-based control is a promising direction, yet unreliable object tracking and hand-object misalignment make it difficult to obtain accurate imitation targets. To address these challenges, we present DeVI (Dexterous Video Imitation), a novel framework that learns physics-based character control for dexterous HOI by imitating text-conditioned synthetic videos. To bypass unreliable 3D object motion reconstruction, we introduce hybrid imitation targets that combine reconstructed 3D human motion with tracked 2D object trajectories. Our Visual HOI Alignment further refines the human reference to align with both the generated video and the initial 3D object configuration, making it suitable for physical interaction. Extensive experiments show that DeVI's hybrid representation is more effective than those used by the baselines and that Visual HOI Alignment improves reconstruction quality and imitation success. We further demonstrate physically plausible functional manipulation from text instructions, including target-aware interactions and articulated object manipulation, without requiring motion-capture demonstrations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Character Animation, Human-Object Interaction, Video Imitation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aMkMOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

DeVI learns a physics-based interaction policy per synthetic video using reconstructed 3D human motion and tracked 2D object vertices. Visual HOI Alignment refines human references without recovering explicit 3D object trajectories.

Critical strengths
  1. Hybrid targets address incompatible human and object reconstructions.
  2. Alignment raises clip success from 10% to 90%, while ARCTIC provides ground-truth-based evidence.
  3. Appendix baselines, stage-wise analysis, and failures strengthen evaluation transparency.
Critical weaknesses
  1. A 263-frame video requires approximately 20 GPU hours of policy training, leaving any cost advantage over MoCap pipelines unestablished without comparisons of total effort, functional success, and 3D accuracy.
  2. VHA shares reprojection and contact optimization ideas with prior HOI reconstruction, but its novelty over GHOST and CARI4D remains unclear without isolating the contribution of initial object anchoring.
  3. The 10-clip comparative subset and 15-video stage-wise evaluation provide limited evidence of generalization across objects, tasks, and environments, despite auxiliary benchmarks and qualitative transfer examples.
Questions for authors

See Weaknesses

Reviewer_5MhdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes the Dexterous Video Imitation (DeVI) framework that learns physics-based dexterous human-object interaction (HOI) without requiring 3D motion-capture demonstrations. The proposed method combines several different algorithms and optimizations. First, DeVI uses a video diffusion model to generate HOI videos and extracts hybrid imitation targets by combining estimated 3D human motion and 2D object trajectories. Additionally, the authors introduce Visual HOI alignment to refine reconstructed human motion and improve alignment with the generated video and initial 3D object configuration. These hybrid targets are used to train a humanoid control policy through reinforcement learning with a hybrid tracking reward. Experiments on synthetic videos and the GRAB dataset demonstrate improved imitation accuracy and success rates compared to existing methods. Diverse functional manipulation experiments show the capability of the method to support real-life applicability.

Critical strengths

** The proposed approach is a practical approach that combines reconstructed 3D human motion with 2D object trajectories, avoiding the challenging reconstruction of full 3D object motion. This is a interesting because inaccurate 3D object trajectories can significantly degrade human-object interaction imitation. And this alignment process clearly explained and demonstrated with examples in the supplementary material. The findings and proposed solution could be very useful to the research community.

** In Table 1, DeVI able to increase success rate %10 to %90 percent. This substantial improvement demonstrates the effectiveness of the proposed method. Qualitative comparision also shows capabiltiy of the model compare to other models. While other models cannot complete task while during the video.

**Demonstrating the method's effectiveness in real-life scenarios is practical with usuing robot arm. Additionally, making synthesized data generation more similar to real-life examples could help bridge the gap between simulation and real-world applications.

Critical weaknesses

** The main comparison with other methods is limited to only 10 different generated videos. Although the authors provide additional evaluations on 15 videos in the appendix, this is still a relatively small evaluation set and may not be sufficient for a comprehensive comparison. The number and diversity of test videos should be increased to provide a more reliable evaluation of the proposed method. Otherwise, the paper/ experiment seems incomplete.

** In the qualitative examples, some baseline methods appear to grasp the object initially but then lose control of their joints and fail to complete the task. Some of them even struggle to maintain the initial standing position. Although the authors state that the baseline methods were trained under comparable settings, these results raise questions about their training quality and convergence. Providing training curves and more details about the convergence of the baseline methods would help clarify the fairness of the comparisons.

** The framework requires training a separate reinforcement learning policy for each generated video. In the Appendix A.11, training on a 263-frame video takes approximately 20 hours on an NVIDIA A6000 GPU. This significantly limits practical scalability. The paper does not demonstrate a unified policy capable of generalizing across substantially different interaction sequences without additional training.

Questions for authors

Minor Weaknesses

** In Section 3, the notation should be clarified better, as some notation explanations are missing. For example, I assume that (F) represents the total number of frames, but it is not explained in the text.

** In Figure 11, the zoomed-in region in the bottom-right image appears to be incorrectly placed.

** The authors claim that the method can support multi-object interactions. However, they do not provide supporting evidence or demonstrate any examples.

Questions:

** Could you also provide more object interaction examples from the GRAB dataset? GRAB contains 51 different objects, but the Table 1 evaluation uses only 10 different objects. Could you please provide success rates for more of the remaining objects and interaction scenarios?

** In Figure 4, the front-view and reference-view results for other models appear to be misaligned, especially in the second row. Could you clarify why this happens? Would applying additional HOI alignment improve the performance of the other models?

** If the object's initial orientation or placement changes, how does this affect the success rate of the proposed method? Could you provide some examples?

Reviewer_U24MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DeVI, a framework for learning physics-based dexterous human–object interaction from text-conditioned synthetic videos. Given an initial 3D scene and a text instruction, the method generates an interaction video and extracts hybrid imitation targets comprising reconstructed 3D human motion and tracked 2D object points. A Visual HOI Alignment stage refines the body and hand motion using image-space observations, temporal consistency, and contact with the initial object geometry. A separate control policy is then trained for each video using human tracking, projected object tracking, and contact rewards.The main contribution is the use of 2D object tracks as supervision for physical interaction, avoiding explicit reconstruction of potentially unreliable 3D object trajectories.

Critical strengths
  1. The paper addresses an important gap between visually plausible synthetic HOI videos and physically executable dexterous interactions. The proposed hybrid representation provides a practical approach to handling inaccurate 3D object reconstruction and human–object misalignment. The motivation is well explained, and the overall pipeline is clear.

  2. The experiments demonstrate substantial improvements over the evaluated baselines. In particular, the synthetic-video experiments show strong imitation performance. The ablation study further confirms the importance of Visual HOI Alignment for downstream policy learning.

Critical weaknesses
  1. Although DeVI reduces the reliance on 3D HOI motion-capture data, its video generation process still requires an initialized 3D scene with known object geometry and poses. This dependence may constrain the diversity of objects and scenes that can be used to generate interactions, particularly when suitable 3D assets are unavailable.Furthermore, the method trains a separate policy for each video, making it unclear how efficiently the framework can scale to a substantially larger and more diverse collection of interactions.

  2. In the GRAB experiments, the 2D and 6D object tracking methods use different early-termination criteria (94 pixels vs. 0.15 m), which may lead to different rollout lengths and affect policy learning. Since GRAB provides ground-truth 6D object trajectories, both methods could use the same 3D object tracking error as the early-termination criterion, regardless of their respective reward formulations. Using different termination conditions makes it difficult to determine whether the reported performance improvements arise solely from the proposed 2D supervision.

  3. The paper presents non-tabletop interaction experiments to demonstrate the generalization capability of DeVI. However, these results are primarily qualitative, and the non-tabletop experiments use motion references from the FullBodyManip dataset rather than evaluating the complete pipeline from synthetic video generation to physics-based control.

Questions for authors
  1. Visual HOI Alignment primarily combines standard 2D reprojection and temporal consistency losses with a Chamfer-distance-based HOI contact constraint. Since these are relatively conventional optimization techniques, what is the main technical novelty of this module beyond their combination? Could the authors provide further ablations to clarify the contribution of the HOI contact constraint?

  2. Could the authors adopt the same 3D object tracking error threshold for both methods and examine whether the advantage of 2D supervision still holds?

  3. The non-tabletop experiments in the appendix primarily provide qualitative results using FullBodyManip references. Could the authors evaluate the complete pipeline, from synthetic video generation to physics-based control, on non-tabletop interactions and report quantitative metrics such as tracking errors and task success rates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.