RoboTaskonomy: Discovering Transfer-Relevant Kinship among Robotic Skills

Real-world robots are expected to continually acquire new skills to adapt to changing demands and environments. The ability of prior skills to facilitate new skill acquisition, i.e., skill transferability, is critical for improving learning efficiency. However, skill transferability remains poorly understood, and it is unclear which relationships among skills are relevant to transfer and how they should be characterized. In this paper, we present RoboTaskonomy, a framework for characterizing transfer-relevant relationships among robotic skills and discovering how these relationships are structured. We first construct skill relationships from task attributes, fine-grained trajectories, and local motion distributions, then assess their correspondence with measured transfer effects. This analysis identifies a transfer-relevant representation space in which skill relationships align more closely with measured transfer effects. Building on this representation space, we discover a set of behavioral primitives shared across skills. Skills can be represented through combinations of these primitives, with stable structure across task contexts and unseen skills emerging as novel compositions. We further leverage this structure to guide prior-skill selection in continual learning, significantly improving new-skill acquisition in both simulation and real-world tasks.