Building World Models from Intrinsic Motivation

In the absence of extrinsic rewards or task supervision, human behavior is guided by intrinsic motivation—neural signals that drive curiosity, creativity, play, and other essential facets of intelligence. Intuitively, intrinsic motivation holds a fundamental role in developing robust internal models of the world that enable generalization to new tasks from little to no task-directed experience. However, this hypothesis has not been carefully explored: existing approaches often focus on model-free reinforcement learning, compare only a small number of intrinsic motivations, use inconsistent reinforcement learning algorithms, and fail to isolate the benefits of representation learning from those of policy learning. This makes it impossible to compare different hypotheses for intrinsic motivation, and more importantly, ignores the interplay between intrinsic motivation and world-model building. To close these gaps, we integrate 24 state-of-the-art self-supervised exploration algorithms—spanning a breadth of methods across curiosity, entropy, and skill-discovery—with a common model-based reinforcement learning backbone, and evaluate how well these intrinsic motivations learn an effective world model. We test our agents on zero-shot, few-shot, and multi-task control tasks in locomotion, manipulation, and open-ended gaming environments using three competing world-model representation learning hypotheses: observation reconstruction, latent prediction, and redundancy reduction. Across all experiments, we find that curiosity-driven exploration with observation reconstruction has the strongest aggregate family profile across representations, but the best individual pairing can depend on the specific experiment. Reconstruction is particularly effective for visual zero-shot control, while the reconstruction-free models are achieve competitive transfer and few-shot adaptation. These findings motivate evaluating intrinsic drives by the reusable world models they build and designing intrinsic motivation and representation learning jointly. Our approach demonstrates that world models provide a powerful tool to measure the quality of intrinsically motivated experience, and identify intrinsic motivation as a fundamental ingredient for building world models just as animals do—autonomously.