BRICKS-WM: Building Reusable Dynamics Networks for Compositional World Models

Model-based reinforcement learning (MBRL) has achieved remarkable success in continuous control by leveraging latent world models. However, monolithic world models couple distinct dynamics in a single transition dynamics model, making it difficult to reuse learned dynamics separately from the original predictor. To address this, we introduce BRICKS-WM (Building Reusable dynamICs networKS for compositional World Models), a framework for composing world models from reusable dynamics modules. Motivated by the insight that the physical world is composed of entities, we model global dynamics as a composition of distinct dynamics modules interacting via latent interfaces. As a minimal instantiation, we factorize the environment dynamics into an actuated Agent dynamics module and an external Background dynamics module, bridged by a learned latent interface. For reuse, we recompose the world model by combining a previously learned Background dynamics module with a newly learned Agent dynamics module through an adapted latent interface. Empirically, BRICKS-WM achieves control performance comparable to strong monolithic baselines when trained from scratch, and enables the reuse of background dynamics across scenarios.