- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors propose JEPA-TTT, a latent world model that supports test-time training to adapt to changes in environment dynamics. Unlike conventional JEPA models, which remain frozen after training, JEPA-TTT collects data in a replay buffer at test time and uses these data to update the model, particularly its predictor. Across eight dynamics-shift conditions, JEPA-TTT reduces latent prediction error by up to 83% and improves performance by 153% compared with a frozen JEPA model.
Critical strengths
- The paper extends JEPA to support test-time training and demonstrates that the model can be updated stably using data collected at test time.
- The paper is clearly written and easy to follow, from the problem formulation and methodology to the experiments.
- The experiments evaluate the method under various dynamics shifts and examine how different replay strategies affect performance.
Critical weaknesses
-
Limited generality of the assumed setting. The proposed method assumes that the task (reward function) and observation distribution remain fixed, while only the dynamics governing the effects of actions change. In my view, this does not appear to be a particularly general setting. It seems more common for observations to change while the underlying dynamics remain broadly similar to those encountered previously: for example, a dropped ball still bounces, and an object still moves when pushed.
-
Insufficient comparison and analysis relative to AdaJEPA. AdaJEPA appeared on arXiv approximately three months before this paper’s submission. My understanding is that ICLR considers work published in a peer-reviewed venue within two months of submission to be concurrent work. Since AdaJEPA has not undergone peer review, there may nevertheless be grounds for treating it as concurrent work. Even so, if this paper is published, the two works will occupy closely related positions in the literature, and a more detailed comparison would be valuable. In particular, although the experiments include a comparison with AdaJEPA, a more thorough analysis is needed to explain which methodological differences account for the observed performance gap.
-
Exclusion of continual learning and catastrophic forgetting. Test-time training is closely related to continual learning. However, as acknowledged in the conclusion, this paper does not address continual adaptation or catastrophic forgetting. Adapting to a single shift using a substantial amount of data, without considering continual adaptation, may amount to overwriting the model’s existing knowledge with knowledge of the new environment.
Questions for authors
-
Could the authors describe the methodological similarities and differences between JEPA-TTT and AdaJEPA in more detail, particularly regarding their objective functions? For example, the paper states that AdaJEPA requires a goal image. However, my reading of Section 3.2 of AdaJEPA suggests that the adaptation objective itself does not use a goal image.
-
In Figure 5, JEPA-TTT generally outperforms AdaJEPA. Have the authors conducted a more detailed analysis of the reasons for this performance gap?
-
The images in Figure 3 alone do not provide enough context to understand the circumstances under which JEPA-TTT produces predictions close to the ground truth. Could the authors also show the planned trajectories and explain where the frozen model fails and how JEPA-TTT improves upon it?
-
Is the visual encoder in Equation 2 frozen? How is the visual encoder’s training objective formulated?
-
I am not sure I understand the statement in Lines 391–393, “It is not a deployable baseline…” Could the authors clarify its meaning?