Knowing What to Adapt: Continual Test-Time Adaptation of World Models

A world model that adapts during deployment can repair today's forecast while learning the wrong dynamics. A pushed object that moves less than expected may reflect a shifted camera or increased friction: both corrections fix the prediction, but only the friction should be remembered. We cast continual test-time adaptation as a question of correction lifetime: what should the model use now, and what should it keep? Conditionally Resolved Dynamics (CoRD) splits every correction into a temporary cache and a candidate dynamics memory. The candidate holds only the part of the correction that no observation change can produce, and an exact least-squares identity shows that the split itself costs no local fit. Because equally good fits can call for different memories, unresolved directions never persist, and a candidate persists only if the model left after cache expiry beats the incumbent on fresh outcomes under anytime-valid comparisons. Across pushing, navigation, and three control tasks with appearance shifts, overlapping dynamics shifts, and returns to nominal, CoRD lowers 10-step forecast error by 12–27% over the strongest baseline, cuts first-return error by 34–47% versus carried adapters while recovering faster than episode-reset ones, and retains nothing before the dynamics change, whereas validating only the full fit commits spurious corrections in 42% of streams.