acceptodds
Under review as a conference paper at ICLR 2027

Filtering Frozen World Models: Covariance-Aware Surprise and Better Forecasts at Test Time

Abstract

Trained world models are usually run as fixed maps: each observation is encoded at face value and rolled forward, so sensor noise passes straight into the latent state. We show that classical state estimation recovers much of the lost accuracy without retraining. KalWM (Kalman World Model) applies an extended Kalman filter to the latent state of a frozen predictor, leaving every weight unchanged, and provides two outputs: a posterior that seeds the model's own rollout, and a covariance-weighted innovation that scores unexpected observations. Ours results shows that on released checkpoints of a trained visual world model for planning tasks, filtering the same three-frame history reduces six-step prediction error by , , and under Gaussian image noise of standard deviation , and covariance-weighted scoring raises sensor-change detection AUROC. Ablations with fixed innovations show that these two benefits have distinct sources: filtering improves the prediction stream, while covariance weighting improves how its errors are ranked.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Post-training, Self-supervised Learning, Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7F6BOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is proposing a post-processing phase for world models which they call it Kalman World Model. They take a frozen world model and apply Kalman filtering on its latent state to address noisy sensor inputs. Experiments use LeWM as the frozen world model and consider three tasks and add gaussian noise to the image observations.

Critical strengths

None

Critical weaknesses

The paper’s motivation is somewhat unclear, and the problem the authors are trying to solve is not clearly explained. The writing is very unclear and uses non-standard terminology.

There is no background section on Kalman filtering, and the reader is assumed to be an expert in the field. Even so, the paper uses a particular form of extended Kalman filter that is never discussed in the text (the observation function is identity, and a particular form of memory-fading noise is assumed), which raises the concern that the authors may not fully understand what their method is doing. The text is lengthy but adds very little information; for example, “Appendix G gives a numerical example.” What is the purpose of this numerical example, and how does it help the reader better understand the method? The method section mainly describes implementation details, which is a common pattern in AI-generated text produced with minimal supervision. The experiments are disappointing. Although the authors added Gaussian noise to the input image observations (which is itself a limitation), the results in Table 1 show that the base WM without the filtering phase predicts better in many cases, and in the remaining cases the gap is very small; the results are therefore inconclusive.

There are many more issues with this paper; however, the ones I have raised are sufficient for rejection.

Questions for authors

None

Reviewer_a67eOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies test-time filtering of frozen latent world models using an extended Kalman filter. The resulting posterior initializes future predictions, while covariance-weighted innovations support change detection. Experiments on LeWM checkpoints show improved forecasting under strong observation noise, but benefits vary across tasks and clean-input predictions can deteriorate.

Critical strengths

Improving frozen world models without retraining is practically appealing, and the comparisons using identical innovations help separate filtering gains from covariance-scoring gains.

Critical weaknesses
  1. Technical novelty is limited: the method primarily applies established extended Kalman filtering and innovation scoring to a frozen JEPA-style world model, with relatively modest methodological adaptation.
  2. The experiments provide limited generalizable insight into when the approach is useful. Findings such as covariance scoring contributing differently across tasks remain largely descriptive, without sufficiently explaining the underlying causes or guiding method selection.
  3. Section 3 interleaves standard Kalman-filter machinery with implementation-specific choices, making it difficult to identify the paper’s distinct technical contribution despite its acknowledgment that the estimator itself is classical.
Questions for authors
  1. Could you more clearly distinguish the standard EKF components from the adaptations required for frozen world models, and explain which adaptations are essential?
  2. Can you identify measurable task or representation properties that explain the differing benefits of covariance scoring and help predict when it will be useful?
Reviewer_eCdKOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper wraps a frozen latent world model in an extended Kalman filter ("KalWM"). The encoder output is treated as a noisy measurement with a diagonal covariance fitted from innovations. The frozen predictor is linearized by autodiff, with singular values floored at 0.1 and a covariance discount γ. The filter provides (i) a posterior that replaces raw encodings as the history fed to the native rollout, and (ii) a Gaussian innovation score ("surprise") for change detection. On released LeWM checkpoints for Push-T, TwoRoom and OGBench cube-single, with Gaussian pixel noise σ=0.1, six-step latent prediction error drops by 2.2%, 31.0% and 1.5%, but clean one-step error increases on all tasks. For a sensor-noise increase from 0.03 to 0.1, matched-time AUROC improves on Push-T (0.951 → 0.991) and cube-single (0.979 → 0.988), but not on TwoRoom. Controlled point-mass and lifted 192-D synthetic studies compare scoring rules and covariance structures (dense/diagonal/low-rank EKF, UKF, EnKF). The authors state that they do not introduce a new estimator; the contribution is framed as an empirical account of when post-hoc filtering helps.

Critical strengths
  1. Honest, well-controlled evaluation design. Fitting, discount selection, alarm calibration and testing use disjoint episodes. The identical-innovation ablation cleanly separates "better state estimate" from "better scoring rule" (Sec. 4.2). Matched-time AUROC plus a no-change temporal control avoid a common pitfall in change-detection evaluation (App. B, Table 6). Negative results (clean-input degradation, TwoRoom detection) are reported prominently.
  2. Zero retraining and model-agnostic. The wrapper changes only inference and could in principle be applied to any frozen latent predictor with a tractable Jacobian. The structured-covariance variants (diagonal via the residual Jacobian identity, low-rank via Woodbury) give large CPU speedups at higher latent widths (Sec. 4.3).
Critical weaknesses
  1. The methodological contribution is minimal, and the main visual gains are small and lack simple baselines. Applying a textbook EKF to a learned latent is explicitly not new (the paper says so). Related learned-filtering work (KVAE, DKF, RKN) already combines filtering with learned latents, so the value must come from the empirical findings. Those findings are modest. Forecast gains under heavy noise are 2.2% and 1.5% on two of three tasks, and clean-input predictions get worse everywhere. Most importantly, there is no comparison to trivial alternatives that also need no retraining: averaging or exponential smoothing of the three encoded frames, image-space denoising (Gaussian/median filtering or an off-the-shelf denoiser) before encoding, or averaging encodings over test-time augmentations. Without these, it is impossible to tell whether the Kalman machinery (Jacobians, covariance propagation) matters, or whether any temporal smoothing yields the same 31% on TwoRoom. Discount selection also uses paired clean images and a known noise level on validation episodes (Sec. 5), which is a privileged setup.
  2. The evaluation does not test what world models are for, and the detection task is nearly trivial. No result evaluates planning or closed-loop success with the visual world models. The forecasting metric is distance to clean latent encodings, which need not correlate with planning quality. The Panda control study (App. H) uses a different, linear-Gaussian model, as the authors note. The visual detection task is an increase in i.i.d. pixel-noise variance. Detecting it requires no world model (a pixel-statistics detector would likely be near-perfect), and baseline AUROCs are already 0.95–0.998, so the reported gains (+0.009 to +0.040) are at the ceiling. In the synthetic RQ3 study, the dense EKF reaches only 0.61 AUROC, versus 0.84 for the raw residual and 0.97 for low-rank. This suggests the dense filter performs worse than not filtering in that setting, which undermines the method's reliability rather than illuminating a tradeoff. Thresholds there are also fitted in-sample.
Questions for authors
  1. How does KalWM compare to (a) an EMA or mean over the three encoded frames with the coefficient selected on the same validation data, (b) image denoising before encoding, and (c) test-time-augmentation averaging of encodings? These would show whether covariance propagation through the predictor Jacobian is responsible for the gains.
  2. Does filtering improve planning success (e.g., LeWM's CEM/MPC success rate) on any task under σ = 0.1 noise? This is the most important missing result.
  3. Can you evaluate detection on changes that are dynamics-relevant and not trivially visible in pixel statistics (e.g., object mass or friction changes, occluders, or a swapped object in Push-T)?
  4. Why does the dense EKF perform so much worse than the raw residual in Table 4 (0.61 vs 0.84 AUROC)? Is this caused by the singular-value floor or by γ = 0.1 inflating the covariance until surprise is insensitive?
  5. The visual filter uses LeWM's one-step interface while the native predictor is history-windowed (Sec. 3.1). How much is lost by this mismatch, and would filtering with the windowed predictor as the transition change the clean-input degradation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.