ConAct: Contact as Action for Multi-Fingered Robot Hand Policies from Human Videos

Policies for multi-fingered robot hands are typically trained on teleoperated demonstrations, which are slow and costly to collect, especially for bimanual tasks. Human videos offer a scalable alternative, yet they show where the hands move but not how firmly they press. We argue that this missing contact is a key reason why policies pretrained on human videos still rely on many teleoperated demonstrations. To address this limitation, we introduce ConAct (Contact as Action), a bimanual policy for multi-fingered hands that learns from human videos with few teleoperated demonstrations. Specifically, our key idea is to extend the point-based action representation shared by human and robot hands with contact, i.e., a contact intensity at each 3D keypoint, which we label automatically on about 1M egocentric video episodes without any tactile sensor or manual annotation. This design not only lets the policy learn from human videos how firmly to press, but also lets us turn the predicted contact into force along the normals of the robot hand at deployment, without any force sensing. On a real bimanual robot with two 20-DoF hands, ConAct outperforms three Vision-Language-Action models pretrained on human videos on all three tasks at every amount of teleoperated demonstrations. For example, with only 20 teleoperated demonstrations in total, it more than doubles the average success rate of the strongest baseline (17.2% $\to$ 41.7%) and already matches a baseline trained with ten times more. Our ablations further show that both the contact intensity and the force it drives contribute substantially to this gain. Moreover, on a bimanual task, ConAct more than quadruples the success rate of the best baseline with only five teleoperated demonstrations (12.5% $\to$ 53.1%). Videos are available at https://anonymous39306.github.io/conact.