Fast Residual Memory for Test-Time Training of Latent World Models

Latent world models are typically deployed as static predictors, even though every prediction is eventually confronted with a realized future state that provides direct self-supervised feedback. We study how such feedback can be reused at test time without modifying the pretrained world model itself. We introduce Fast Residual Memory, a trajectory-local fast-weight state that stores recent prediction residuals and retrieves context-dependent corrections for subsequent predictions. Observed residuals are written into the fast state through a lightweight delta-rule update, while learned write and read projections are trained with a support-query objective so that residuals collected from past transitions improve held-out future predictions rather than merely memorizing observed errors. At deployment, the pretrained world model and residual-memory parameters remain fixed, while only the fast state evolves through a causal predict-observe-update process. Across LeWM and DINO-WM, Fast Residual Memory reduces latent prediction error under held-out dynamics shifts, including both interpolation and extrapolation settings. Further analysis shows that the fast state is best understood as a context-dependent, trajectory-local memory of recent model error rather than as a global representation that can be reliably reused across trajectories.