- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes AdaJEPA, a test-time adaptation framework for JEPA-based latent world models used in model predictive control.
Critical strengths
- The paper demonstrates substantial practical benefits from a lightweight adaptation procedure. The comparison against the same frozen model provides convincing evidence that online updates can improve planning in the evaluated settings.
- The empirical study goes beyond a single favorable configuration and provides useful guidance about applicability. The paper evaluates multiple world-model implementations and examines layer selection, LoRA, learning rates, update counts, replay buffers, and pretraining data scale.
Critical weaknesses
-
The fine-tuning comparison does not sufficiently establish the claimed sample-efficiency advantage. In Section 4.2, the main offline fine-tuning baselines receive only one epoch over the target-domain trajectories, whereas AdaJEPA repeatedly updates on a small rolling buffer throughout execution. Matching learning rates and trainable parameters does not match optimization effort, data reuse, or the relevance of the adaptation data to the current trajectory. The number of optimizer updates and the convergence behavior of these baselines are not reported.
-
It remains unclear whether local adaptation reduces compounding errors in multi-step rollouts. Reporting prediction error across rollout horizons would better support the claimed connection between adaptation and improved planning.
Questions for authors
-
AdaJEPA addresses domain shift, but compounding errors may persist in multi-step rollouts. Even introducing rollout-reliability estimation and adaptive termination, as in MBRL-LDN [1], does not eliminate this problem. Have the authors considered approaches that avoid error accumulation altogether?
-
JEPA-based planning is constrained by compounding errors and the need for action-space alignment, making it difficult to scale. Have the authors considered having the model plan directly in state space?
[1] Improving Model‐Based Deep Reinforcement Learning with Learning Degree Networks and Its Application in Robot Control
4 comments
Thank you again for helping us improve our paper! In summary, we have updated the pdf (in blue) according to your suggestions. We show that step-matched, learning-rate-tuned offline FT baselines still underperform AdaJEPA (App. B.3). We also found that adaptation helps to decrease the rollout error on the transitions the agent is about to execute (Fig. 12). Error still grows with the horizon, and avoiding accumulation is a common challenge in world models that requires future work. We hope the revision addresses your concerns and would be grateful if you would consider raising your score.
Thank you for the authors' response. The authors may have misunderstood "plan directly in state space." What I mean is that both image space and latent space are state space. My question is: why not do video generation or latent-space sequence generation? Because explicit planning with MPC in JEPA is actually very inefficient.
Sorry for the misunderstanding and thank you for clarifying!
If we understand correctly, we think you're suggesting an image or latent trajectory conditioned on the current observation and goal. This is a promising alternative, though a controller or inverse-dynamics model would still be needed to translate the generated states into actions (e.g. a video of picking an apple does not itself specify the robot’s actions). We agree that planning with WMs can be slow, but trajectory generation is not necessarily faster or free of accumulated errors: diffusion involves iterative computation, and autoregressive video generation can still accumulate errors. Generated trajectories may also violate physical constraints[1], making them difficult to execute reliably.
Our focus is on improving world models through online adaptation within JEPA-based MPC. Extending this idea to adapting the generator or controller using execution feedback is an interesting future direction. Incorporating policy to accelerate and improve planning is also another future direction.
[1] VideoPhy-1 (ICLR 2025) and VideoPhy-2 (ICLR 2026)
Thanks for your positive recommendation and experiment suggestions! We have added all requested experiments to address your concerns.
Thanks for the suggestion! We updated the offline fine-tuning baseline and find the sample-efficiency advantage is still robust.
[Table 1] Offline finetuning baseline (success in %, three seeds):
At the same step count, offline fine-tuning on one trajectory provides only modest gains. More steps on one trajectory cause overfitting: the training loss keeps falling with no gain in success. With more trajectories (N=32) and updates, offline finetuning still significantly below AdaJEPA that only sees one trajectory.
Thanks for the suggestion! We found that AdaJEPA reduces multi-step prediction error on recent, untrained transitions from the agent's executed trajectory.
We evaluate Adapt on the PushObj shapes (GD, 150 episodes per shape). Starting from an observed frame, we roll the model forward autoregressively using the recorded actions and measure latent MSE against the observed frame at each horizon, encoded by the corresponding frozen or adapted encoder. We use two diagnostics:
[Table 2] Open-loop rollout error under frozen → adapted weights (adapted / frozen ratio of means), MSE x1e3.
On recent untrained transitions, the adapted model has lower mean prediction error at every measured horizon for both seen and unseen shapes. Held-out errors increase slightly on seen shapes and change little on unseen shapes. These results are consistent with local correction along the executed trajectory.
The results above show that AdaJEPA reduces multi-step prediction error along the executed trajectory, though error still grows with rollout horizon. Avoiding error accumulation remains an open problem. Combining AdaJEPA with rollout-reliability estimation as studied in MBRL-LDN is a promising future direction.
Yes, AdaJEPA can also be applied directly in state space when the underlying state is accessible. However, we consider the more challenging setting where the model only has access to images.
We are happy to further revise the paper if you have more suggestions!