Training with Borrowed Eyes: Decentralized Multi-Robot Manipulation via Privileged Information Distillation

Learning decentralized policies provides a scalable paradigm for multi-robot manipulation by enabling flexible collaboration without centralized observations or communication at deployment. However, agents operating solely on local observations often struggle to infer the intentions and behaviors of their partners, resulting in degraded coordination. In this work, we present a decentralized manipulation framework that transfers coordination knowledge from a privileged centralized policy to locally conditioned agents. During training, a privileged teacher with access to global observations captures coordinated behaviors, while privileged information distillation transfers its action distribution to decentralized students conditioned only on local observations. At deployment, each agent independently predicts its own actions without access to the states, observations, or actions of partners. This formulation retains the scalability and robustness of fully decentralized execution while exploiting privileged global information during training to foster implicit coordination. Experiments on real-world multi-arm manipulation tasks show that our approach consistently outperforms centralized baselines and prior decentralized methods, providing a practical approach for transforming pretrained centralized policies into scalable decentralized robotic systems.