Before the Next Outcome: History-Aware Test-Time Adaptation for Latent World Models

Test-time adaptation of latent world models typically relies on real transitions observed after the agent acts, leaving the model unadapted at the beginning of deployment. We study pre-outcome test-time adaptation: adapting a pretrained world model using only the target-domain history that is already available, be- fore the next real state is observed. This setting is challenging because historical variation mixes appearance nuisance with genuine dynamics changes, while repre- sentation and dynamics adaptation are coupled through a shared latent space. We propose HATTA (History-Aware Test-Time Adaptation), which exploits the two forms of evidence already present in the observed history. HATTA uses historical transition compatibility to select counterfactual visual hypotheses and construct a dynamics-aware representation correction, while observed local motion is used to calibrate the action-conditioned dynamics predictor. A latent-space coordina- tion step controls the accepted representation update and re-encodes the original history before dynamics adaptation. HATTA requires no new future outcome, source-domain data, reward, goal, or shift label. Experiments on PushT visual shifts and PointMaze dynamics shifts show that short histories provide useful pre- outcome signals for both visual and dynamics adaptation. The unified method retains most of the gains of the two specialized branches and achieves overall point estimates close to a feedback-based AdaJEPA reference without using new real transition feedback. These results suggest that target-domain history can sup- port meaningful world-model adaptation before additional interaction feedback becomes available.