- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies test-time adaptation of JEPA-style latent world models and asks a good question: When the model’s prediction disagrees with the next observation, how much of that disagreement should the model learn from. The proposed method, BayesJEPA, estimates uncertainty on both the prediction side and the observation side, and uses a Kalman-style gain to scale the residual before the gradient step. The gain is the standard linear MMSE solution, and the paper shows it is optimal under second-moment assumptions. Experiments on PointMaze show that filtering reduces harmful updates and parameter drift under transient visual corruption, keeps adaptation working under persistent dynamics shift, and gives the largest benefit when both kinds of shift occur together.
Critical strengths
The residual-attribution framing is a useful way to think about this problem, and it is stated cleanly. Separating “the model is wrong” from “the target is wrong” is a distinction that most TTA work ignores. The diagnostics built around it (harmful-update rate against a clean-target oracle, drift from the pre-burst predictor, gain response across phases) are more informative than success rate alone, and I would like to see them used in other TTA papers.
The paper is careful about what it claims. The theory is explicitly a local, per-step statement, the deployed method is described as a plug-in approximation, and the limitations of the oracle definition are spelled out.
Critical weaknesses
The main experimental support for the filter is the mixed setting. Under transient corruption BayesJEPA is still behind the frozen model, and under persistent shift the numbers are close to AdaJEPA. Given that the method amounts to weighting the adaptation loss per transition, it should be compared with simpler ways of doing the same thing, e.g. thresholding the reliability score, lowering the learning rate, or periodically resetting the predictor. None of these are in the paper. Without them it is hard to say whether the Kalman-style gain matters.
The optimality results assume the uncertainties are known. In the experiments the observation-side uncertainty is predicted by a network trained on clean/corrupted frame pairs, using the same corruption types that appear at test time. In effect the update is skipped when a known corruption is detected. There is no experiment with a corruption type the network was not trained on, so it is unclear how the method behaves beyond that setting.
Questions for authors
(weakness 1) Could you add comparisons with simpler selective-adaptation baselines under the same conditions, e.g. a hard gate on the reliability score, direct adaptation with a lower learning rate, or periodic reset to the pre-burst predictor?
(weakness 2) What happens with a corruption type the reliability network was not trained on? Holding out one of OCC/DARK/SNP/BLUR would be the easiest test.