Learning Modular World Models by Composing Reusable Dynamics

Massively multitask world models promise to turn broad interaction data into reusable dynamics for control, but such reuse is not guaranteed by pooling tasks into a single dense conditional model. In novel tasks, a held-out environment may combine familiar objects, goals, initial states, or interaction patterns in a configuration never observed during training. We formulate this challenge as concept accumulation and recombination: each environment constrains only part of a latent concept library, while evaluation requires composing the learned parts under new task contexts. As a guiding analysis, we study this problem in a sparse latent-dynamics setting, where sparse predictive supports and a shared interaction rule are sufficient for recombination under explicit structural assumptions. Motivated by this principle, we introduce CWM, a compositional task-routing and dynamics-masking framework that separates task-expert routing from entity and interaction masking, identifying which factors are predictively relevant before modeling how they evolve under actions. This simple structural bias encourages the world model to share reusable transition rules while suppressing task-irrelevant factors and environment-specific co-occurrence shortcuts. On MM-Bench, which spans more than 200 tasks across 10 domains, CWM preserves the scalability of massively multitask training, yielding 32% overall gain over the state-of-the-art. It also provides stronger adaptation to held-out compositional tasks, showing that a sparse task-routing and dynamics-masking inductive bias can improve both large-scale multitask learning and generalization to new task recombinations.