DexSeed: Scalable Physics-Grounded Demonstration Generation for Sim-to-Real Dexterous Manipulation

Learning visuomotor policies for contact-rich dexterous manipulation requires demonstrations that are diverse and physically valid, yet teleoperation is costly and embodiment-specific, while human videos are not directly executable by robot hands. Existing data-generation methods either replay kinematically transformed demonstrations, which break under physics variation, or track reference trajectories in simulation, which inherit the source timing and contact schedule and limit diversity. We introduce DexSeed, a physics-grounded pipeline that converts a handful of human RGB-D videos into large-scale dexterous manipulation datasets under joint randomization of layout, geometry, visual, and physics. Rather than treating a human demonstration as a trajectory to copy, DexSeed uses it as an object-centric, phase-aligned prior over task-relevant contact structure, combining temporal style augmentation, task-conditioned objective, and sampling-based optimization through full physical rollouts. Across four contact-rich tasks, DexSeed achieves the highest data-generation success from both teleoperated demonstrations and human videos, improves simulated downstream policy success by 33.2% on average, and achieves a 45% zero-shot real-world success rate using only 3 human videos per task. Videos are available at dexseed.github.io.