acceptodds
Withdrawn from ICLR 2027

Filter Before Adapting Latent World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time adaptation helps latent world models remain effective under deployment shift. However, direct adaptation treats every prediction–observation mismatch as a learning signal, even when the discrepancy arises from a transiently unreliable observation. Such erroneous updates can persist after the observation recovers, degrading subsequent prediction and planning. We formulate this problem as latent residual attribution, which seeks to recover the clean-target prediction-side residual from an observed discrepancy that also contains observation-side error. We introduce BayesJEPA, which estimates prediction-side and observation-side uncertainty and uses their relative reliability to filter the residual before adapting the predictor. Under second-moment assumptions, this uncertainty-weighted rule is optimal among linear attribution rules; under Gaussian uncertainty, it becomes the posterior mean with a Kalman-style gain and induces the posterior-optimal estimate of the corresponding clean-target adaptation gradient. On PointMaze, BayesJEPA improves recovery of the clean-target residual, reduces harmful updates and parameter drift under transient visual corruption, and preserves useful adaptation under persistent dynamics shifts. Under mixed dynamics and observation shifts, it further improves planning over direct adaptation. Together, our theoretical and empirical results establish filtering before adapting as a principled and effective approach to robust test-time adaptation of latent world models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
latent world models, JEPA, test-time adaptation, Bayesian filtering, Kalman filter

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tjUwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies test-time adaptation of JEPA-style latent world models and asks a good question: When the model’s prediction disagrees with the next observation, how much of that disagreement should the model learn from. The proposed method, BayesJEPA, estimates uncertainty on both the prediction side and the observation side, and uses a Kalman-style gain to scale the residual before the gradient step. The gain is the standard linear MMSE solution, and the paper shows it is optimal under second-moment assumptions. Experiments on PointMaze show that filtering reduces harmful updates and parameter drift under transient visual corruption, keeps adaptation working under persistent dynamics shift, and gives the largest benefit when both kinds of shift occur together.

Critical strengths

The residual-attribution framing is a useful way to think about this problem, and it is stated cleanly. Separating “the model is wrong” from “the target is wrong” is a distinction that most TTA work ignores. The diagnostics built around it (harmful-update rate against a clean-target oracle, drift from the pre-burst predictor, gain response across phases) are more informative than success rate alone, and I would like to see them used in other TTA papers.

The paper is careful about what it claims. The theory is explicitly a local, per-step statement, the deployed method is described as a plug-in approximation, and the limitations of the oracle definition are spelled out.

Critical weaknesses

The main experimental support for the filter is the mixed setting. Under transient corruption BayesJEPA is still behind the frozen model, and under persistent shift the numbers are close to AdaJEPA. Given that the method amounts to weighting the adaptation loss per transition, it should be compared with simpler ways of doing the same thing, e.g. thresholding the reliability score, lowering the learning rate, or periodically resetting the predictor. None of these are in the paper. Without them it is hard to say whether the Kalman-style gain matters.

The optimality results assume the uncertainties are known. In the experiments the observation-side uncertainty is predicted by a network trained on clean/corrupted frame pairs, using the same corruption types that appear at test time. In effect the update is skipped when a known corruption is detected. There is no experiment with a corruption type the network was not trained on, so it is unclear how the method behaves beyond that setting.

Questions for authors

(weakness 1) Could you add comparisons with simpler selective-adaptation baselines under the same conditions, e.g. a hard gate on the reliability score, direct adaptation with a lower learning rate, or periodic reset to the pre-burst predictor?

(weakness 2) What happens with a corruption type the reliability network was not trained on? Holding out one of OCC/DARK/SNP/BLUR would be the easiest test.

Reviewer_vgeiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

BayesJEPA filters latent prediction residuals using estimated prediction and observation uncertainty before adapting a world model. The paper derives conditional optimality results and evaluates the method on PointMaze under visual corruption, dynamics shifts, and their combinations.

Critical strengths
  1. The residual-attribution formulation offers an interpretable approach to adaptation under unreliable observations.

  2. Experiments combine visual corruption with dynamics shifts and include diagnostics of residual error, harmful updates, and parameter drift.

Critical weaknesses
  1. Figure 1 does not explain the persistent gap in cumulative success or establish that prediction quality remains impaired after observations recover. Over-adaptation to the recent replay buffer, possibly causing catastrophic forgetting even without corruption, remains an alternative explanation. BayesJEPA's gains may also reflect reduced update magnitudes under corruption.

  2. Assumption 1 requires the clean-target prediction residual and current observation error to be conditionally uncorrelated, Cov⁡(δk,εk∣Hk−1)=0\operatorname{Cov}(\delta_k,\varepsilon_k\mid\mathcal H_{k-1})=0. The paper provides insufficient modeling justification or empirical evidence for this assumption. Although the optimality results hold under the stated model, their applicability to the residuals encountered during deployment remains unclear.

  3. The reference z⋆z^\star is defined through a clean rendering of the same simulator state. Its interpretation is less clear for some extreme cases (e.g., reflective or transparent objects, whose appearance depends on illumination and viewpoint). Even a clean image contains only partial information about the state. Under these perceptual ambiguities unresolved, the definition of z⋆z^\star might not be rigour.

  4. Evaluation is confined to PointMaze. AdaJEPA also studies PushT/PushObj, with broader coverage of visual, object-shape, dynamics, and maze-layout shifts. This paper omits contact-rich manipulation and changes in object geometry, where useful adaptation may be harder to distinguish from unreliable supervision. The narrower evaluation leaves generalization beyond maze navigation insufficiently tested.

Questions for authors
  1. Can the authors add an entirely clean run under nominal dynamics and evaluate checkpoints before, during, and after the burst on a fixed held-out clean probe set? Disabling updates only during the burst would help distinguish corruption-induced damage from over-adaptation or forgetting that occurs without corruption.

  2. What modeling justification or correlation measurements support Assumption 1? With nonzero conditional cross-covariance, how would the optimal gain and Theorem 3's risk comparison change, and what guarantees would remain for the implemented gain?

  3. What should z⋆z^\star preserve, which variations should count as nuisance corruption, and does the method address uncertainty from partial observability?

  4. Can the authors evaluate FrozenJEPA, AdaJEPA, and BayesJEPA on PushT or PushObj under matched evaluation and adaptation protocols, including visual and object-shape shifts, to test whether useful adaptation is preserved in manipulation tasks?

Reviewer_KkGqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how latent world models should adapt when prediction errors may arise from either changed environment dynamics or unreliable observations. It proposes BayesJEPA, which estimates prediction-side and observation-side uncertainty and uses their relative magnitudes to scale the residual before updating the model.

The paper derives an optimal affine estimator under second-moment assumptions and connects it to Bayesian estimation under Gaussian assumptions. Experiments on PointMaze show improved residual recovery and planning performance over direct adaptation under mixed dynamics and observation shifts.

Critical strengths
  1. The problem is relevant. Distinguishing genuine dynamics changes from temporary sensing failures is important for reliable online adaptation.

  2. The experiments include useful diagnostics. Residual recovery, harmful-update frequency, uncertainty estimates, and parameter drift help examine the proposed mechanism.

  3. The mixed-shift results are encouraging, with BayesJEPA outperforming AdaJEPA by a large margin.

Critical weaknesses
  1. The paper is very dense and difficult to follow. The relatively simple central idea is obscured by extensive notation, repeated definitions, and closely related theorems and corollaries. The relationship between the ideal statistical model, the learned uncertainty estimates, and the implemented update is difficult to reconstruct. This presentation makes it unnecessarily challenging to identify and assess the actual contribution.

  2. The experiments do not establish the necessity of the proposed method. In the default scalar implementation, BayesJEPA simply scales each transition’s direct-adaptation gradient. Yet the paper does not compare against basic alternatives such as a tuned smaller learning rate, gradient clipping, hard reliability gating, or observation-only weighting. Without these controls, the reported improvements cannot be attributed specifically to the proposed prediction/observation uncertainty ratio rather than to suppressing updates on corrupted observations.

  3. The empirical evidence is too narrow to support the broader claims. Evaluation is restricted to one PointMaze environment, one pretrained representation, and a small set of shifts. Planning results lack confidence intervals or paired significance analysis. BayesJEPA remains below the frozen model under occlusion alone and slightly below direct adaptation under persistent dynamics shifts. These results suggest a useful trade-off in this particular setting, but do not establish a generally effective approach to robust world-model adaptation.

  4. The theoretical contribution is limited and does not validate the deployed estimator. The main derivation follows the standard linear minimum-mean-square-error/Kalman gain calculation, while the gradient result follows from applying a fixed Jacobian to the conditional mean. The implementation uses learned error-scale proxies rather than the conditional covariances assumed by the theory. Although the paper acknowledges this gap, it does not demonstrate that the practical estimator behaves close to the theoretical optimum. The formal analysis therefore provides motivation rather than a guarantee for the implemented method.

  5. Reproducibility and computational costs are insufficiently documented. The paper does not clearly specify the number of MC-dropout samples, dropout configuration, reliability-network architecture and training schedule, or calibration sample sizes. It also omits runtime and memory measurements despite adding stochastic predictor passes and a separate network.

Questions for authors
  1. What evidence shows that the proposed uncertainty ratio is necessary? Could a simpler method using the same observation-reliability network, such as hard gating or observation-only weighting, achieve comparable results?

  2. Does the reliability network generalize beyond its training corruptions? Please specify the corruption types and severities used for training and calibration, and evaluate on excluded corruption families. This would clarify whether the method handles unfamiliar observation failures or mainly detects familiar synthetic perturbations.

  3. What does the theory establish about the implemented method? The guarantees assume conditional moments, whereas the implementation uses learned error-scale proxies. Can you quantify the effect of this mismatch and show that the practical gain improves on simpler heuristic weighting?

  4. How strong and general are the planning improvements? Given the restriction to one PointMaze environment and the absence of confidence intervals, can you provide paired uncertainty estimates and evidence from another environment or representation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.