DexPrior: Bootstrapping Online Dexterous Adaptation with Multi-Task Latent Priors

Dexterous manipulation is challenging to learn from scratch due to its contact-rich interactions and long-horizon dynamics. While demonstrations can provide a useful initialization for online adaptation, conventional behavioral policies are typically task-specific and may degrade when trained on heterogeneous multi-task data. We present DexPrior, a framework that learns task-agnostic latent skills from multi-task demonstrations, providing a reusable prior for online adaptation. DexPrior adopts an Encoder–Prior–Decoder architecture to learn reusable skills from shared state representations and privileged information. The pretrained skill prior is then reused for online adaptation, where a diffusion-based action decoder and an additional latent-skill transition enable skill-level exploration within the two-layer MDP formulation. On DexJoCo, we evaluate online adaptation against from-scratch and pretrained-policy RL baselines, demonstrating the effectiveness of multi-task skills for long-horizon dexterous manipulation.