Videos to Forces: Physics Based Climbing Control From Online Videos

Climbing motions are especially challenging to reconstruct from videos because of their dense, complex contacts. Existing video-based reconstructions recover kinematics but not the required dynamics; physics-based motion tracking with reinforcement learning (RL) offers an alternative, still relies on these imperfect kinematic references and requires the policy to discover viable contact forces through exploration. We reinterpret a fictitious force that prior methods use for partial assistance, instead using it to fully support the motion as an intermediate dynamic reference. Although residual assistance must ultimately be removed, this force reveals how much support must ultimately come from physical contacts, and we use it to densely guide the policy to learn to clime using only the physical contact forces. Our method reconstructs challenging climbing behaviors from video-derived motion references. Across diverse wall geometries and climbing techniques, it reduces penetration, improves motion quality, and makes policy learning more reliable and efficient than prior kinematic and physics-based approaches. To our knowledge, this is the first method to reconstruct complex human climbing motions from video using only physical contact forces at test time.