Reconstructing Deformable Hand-Object Manipulation via Vision-Physics Inverse Modeling

Reconstructing dynamic hand-object interaction from visual observations is fundamental for enabling embodied agents to acquire dexterous manipulation skills. While existing methods have achieved promising results under rigid-body assumptions or template-based representations, they struggle to capture the complex, spatially varying deformations arising in interactions with deformable objects, especially in bimanual settings. In this paper, we present a novel vision-physics inverse modeling framework for reconstructing bimanual dynamic deformable hand-object interactions from multi-view videos. Our key insight is that object deformation is a direct physical response to hand-applied forces, which is implicitly encoded in temporal appearance variations across views. Building on this observation, our method jointly leverages geometry, appearance, and interaction cues from multi-view observations to support physically grounded non-rigid reconstruction. Specifically, we introduce a physics-aware deformation model that estimates both a time-varying force field and a time-invariant material field, allowing fine-grained deformations to be recovered via differentiable simulation. To facilitate evaluation, we further construct a real-world multi-view dataset of bimanual interactions with diverse deformable objects. Extensive experiments demonstrate that our method significantly outperforms prior approaches in both geometric accuracy and physical plausibility, particularly in challenging deformable interaction scenarios. The code and dataset will be made publicly available.