Learning Latent Atomic Movements for Generalizable and Adaptable Robot Policies

Adapting robot policies from limited demonstrations is challenging when task-dependent movement selection and precise control are learned jointly. We propose an atomic-movement interface that enables adaptation through movement selection while keeping the policy fixed. A latent action model learns compact movement representations as sparse combinations of shared codebook entries, supervised by action reconstruction. A latent prior predicts these representations from the instruction and observation history, and an observation-conditioned policy generates the corresponding actions. To encourage the policy to follow the movement condition, we introduce Latent Override, which pairs an observation with a compatible donor movement and its action target during training. After learning this interface, adaptation updates only the lightweight prior adapter while keeping the movement representation and policy fixed. Using Cosmos-Policy as the policy backbone, our method achieves 65.5% average success across ten real-world base tasks. Under few-shot adaption setting, it achieves 58.33% average success across six adaptation tasks, compared with 50.0% for Cosmos-Policy and 55.0% for ${\pi }_{0.5}$ under full-model fine-tuning, with lower measured adaptation time and GPU requirements. These results support reusing a learned movement interface and a fixed policy for few-shot robot policy adaptation.