Filtering Frozen World Models: Covariance-Aware Surprise and Better Forecasts at Test Time

Trained world models are usually run as fixed maps: each observation is encoded at face value and rolled forward, so sensor noise passes straight into the latent state. We show that classical state estimation recovers much of the lost accuracy without retraining. KalWM (Kalman World Model) applies an extended Kalman filter to the latent state of a frozen predictor, leaving every weight unchanged, and provides two outputs: a posterior that seeds the model's own rollout, and a covariance-weighted innovation that scores unexpected observations. Ours results shows that on released checkpoints of a trained visual world model for planning tasks, filtering the same three-frame history reduces six-step prediction error by $2.2\%$, $31.0\%$, and $1.5\%$ under Gaussian image noise of standard deviation $0.1$, and covariance-weighted scoring raises sensor-change detection AUROC. Ablations with fixed innovations show that these two benefits have distinct sources: filtering improves the prediction stream, while covariance weighting improves how its errors are ranked.