Faster Policy Learning from Multiple Experts: Stitch, Imitate, Optimize

How can we train a new policy from a set of expert policies, when none of the experts fully solve the task and none are available at deployment? We propose algorithms to learn to stitch the best experts across states, and to further improve upon the stitched experts. We jointly train two policies: a *student*, which will ultimately be deployed, and a *selector*, which at each state selects an expert or the student to act in the environment. The selector is updated with standard reinforcement learning (RL) to maximize returns. The student is updated with standard RL on its own actions, and with behavior cloning (BC) on actions taken by selected experts. Because the selector tends to select experts only if they outperform the student, the BC can automatically stop as the student outperforms the experts. Overall, this simple mechanism allows the student to combine the strengths of multiple experts, learn quickly from them in the early phase, and continue improving beyond them in the later phase, with the selector automatically guiding the phase transition. Empirically, we measure returns on all combinations of eight environments (four classic control environments, four higher-dimensional MuJoCo environments) with up to twelve expert sets. On all expert sets, averaging over the environments, our most robust algorithm's return CIs overlap or exceed both max-aggregation's (Cheng et al., 2020) and max${}^{+}$-aggregation's (Liu et al., 2024), exceeding them by up to $$\begin{gathered} 5 \\ times \end{gathered}$$.