GEARS: Seeing Geometry, Diffusing Actions for Zero-Shot Sim-to-Real Dexterous Manipulation

Simulation provides scalable interaction data and privileged geometric labels for dexterous manipulation, yet zero-shot deployment remains limited by coupled visual and physical mismatches. Appearance changes alter the student's visual observations, while hidden simulator states and randomized physical parameters can make multiple expert actions plausible from the same deployable observation. We present Geometric Action Representation for Sim-to-Real (GEARS), a three-stage framework that uses privileged information only during training. GEARS first fine-tunes a visual foundation model with joint inverse-depth and surface-normal prediction and a cross-task consistency objective. This produces a frozen geometric encoder that consumes only RGB at deployment. GEARS then trains a privileged reinforcement learning expert with Virtual Particle Stein Variational Gradient Descent (VP-SVGD), which uses an estimated failure landscape to adapt ten object-pose and physical randomization dimensions. Finally, a diffusion student conditioned on the frozen geometric features and proprioception learns the expert action distribution after hidden simulator information is marginalized. The central design links perception and action modeling: geometric supervision preserves contact-relevant scene structure in the RGB representation, while diffusion represents the action variation induced by unobserved state and physics. Across four chemistry-laboratory tasks on a fixed dual-arm platform, GEARS achieves a 67.5% real-world success rate without real-world policy training, substantially outperforming the strongest controlled baseline at 42.5%, with gains on all four tasks. These results demonstrate the effectiveness of jointly addressing visual domain shift through geometric representation learning and action ambiguity through diffusion-based policy modeling for zero-shot sim-to-real dexterous manipulation.