acceptodds
Under review as a conference paper at ICLR 2027

Adaptation-Ready JEPA: Source Training for Test-Time Adaptive World Models

Abstract

Latent world models often require test-time adaptation under deployment shifts, while existing methods mainly rely on self-supervised objectives that can be con- structed directly at deployment. In contrast, supervision that explicitly shapes control-related representations, such as physical states, is often available only during source training. Such source-only supervision can improve the initial rep- resentation, but its constraints do not automatically carry over once the model is further updated by a different self-supervised objective. We therefore study a supervision-availability mismatch under a fixed deployment update: how can training-only supervision improve the model produced by that update? We pro- pose Adaptation-Ready JEPA (AR-JEPA), which keeps AdaJEPA’s deployment update unchanged, differentiably simulates it during source training, and opti- mizes the source model through the resulting adapted model. A meta-learning- style post-update physical objective alone is insufficient in this setting, because the adapted object is itself the world model that must continue to support dy- namics prediction and MPC. AR-JEPA therefore uses Post-Phys to preserve control-related physical information in the updated representation and Post-Dyn to preserve future latent-dynamics prediction of the same model, jointly main- tainingrepresentation content and predictive functionality after adaptation. After training, physical labels and the auxiliary head are removed, while real deploy- ment retains AdaJEPA’s original adaptation loss, updatable parameters, and MPC procedure. On Push-Object, AR-JEPA achieves the highest 13-OOD adapted success among the compared configurations (40.56%) and shows smaller future- dynamics and physical-representation degradation under an identical AdaJEPA update. PointMaze further shows that the relative roles of pre- and post-adaptation physical supervision depend on the environment

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World Models, JEPA, Test-Time Adaptation, Meta-Learning, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KZXiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

AR-JEPA studies how supervision available only during source training can improve a world model that subsequently adapts using a different, self-supervised objective. Building on AdaJEPA, it simulates the deployment update during source training and differentiates through that update. Physical supervision is applied before and after the simulated update, while an additional post-update dynamics objective supervises prediction on later transitions. Physical labels and the auxiliary head are removed at deployment, leaving AdaJEPA’s adaptation procedure and planner unchanged. Experiments on Push-Object and PointMaze compare eight combinations of these objectives. Full AR-JEPA achieves the highest adapted success on Push-Object, while a simpler variant performs better on PointMaze.

Critical strengths
  1. The paper identifies a clear and practically relevant mismatch: physical supervision can shape a representation during training, but its benefits may not survive subsequent self-supervised updates. Evaluating the updated world model during source training is a sensible way to address this problem without adding deployment-time labels or auxiliary components.

  2. The factorized experiments are a useful strength. Comparing all eight combinations of pre-update physical supervision, post-update physical supervision, and post-update dynamics supervision provides a more informative analysis than a comparison against the base model alone. Keeping the deployment updater and planner fixed also helps isolate the effects of source training. The paper openly reports that the best configuration differs across environments.

Critical weaknesses

Important implementation and comparison details are missing. Section 3.4 directs readers to the appendix for loss weights and training schedules, but the appendix defers these values to a future code release and leaves the rule for limiting post-adaptation gradients unspecified. These choices could materially affect the comparisons, especially given the different preferred configurations across environments.

Questions for authors
  1. Can you provide paired confidence intervals for the differences between AR-JEPA and the strongest relevant baselines, including the effect of adding Post-Dyn?

  2. How should we interpret the adaptation-specific contribution when most of the Push-Object advantage exists before adaptation and frozen Auxiliary-only achieves the highest PointMaze success? What evidence supports choosing the full method over the simpler alternatives in these settings?

  3. Does differentiating through the simulated update improve performance over a control using the same physical and future-prediction supervision, data, and comparable training budget, but without the simulated update?

  4. Please provide the exact loss weights, training durations, ramp schedules, gradient-limiting rule, and validation procedure used to select them. Were these settings held fixed across the factorized comparisons?

2 comments
AuthorsOpenReview ↗

Response to the Reviewer’s Insightful Comments on AR-JEPA

Dear Reviewer,

Thank you for your careful and insightful review and for recognizing the value of our work. Your questions helped us focus the new comparisons.

Training settings and reproducibility (weakness and Q4)

Appendix A now gives the exact loss weights, training lengths, ramp schedules, gradient limit, and validation procedure. The eight configurations within each environment share these settings. A disabled objective has zero weight. We calibrate weights on source training data, use source validation only to monitor training, and take the checkpoint at the fixed training endpoint.

We expanded Push-Object evaluation from three to six seeds for all eight methods. The anonymous GitHub repository contains the code, the checkpoint for the original 40.56% result, and test records for the original and added seeds.

Our reading is that your remaining questions focus on two points. First, how much does AR-JEPA help, and where do the gains come from? Second, how do these gains change across deployment conditions?

A. Push-Object performance and sources of gains

1. Paired confidence intervals and the effect of Post-Dyn (Q1). AR-JEPA has the highest mean adapted success under the fixed deployment update. Its lead over the strongest ablation, Pre+Meta-Phys, is 1.31 pp (paired 95% CI [−0.01, 2.63]).

BaselineAR-JEPA minus baseline (pp)Paired 95% CI
AdaJEPA (V0)+2.31[0.98, 3.64]
Auxiliary-only+4.46[2.11, 6.82]
AR-Dyn+1.51[0.04, 2.99]
Meta-Phys+2.13[0.88, 3.37]
Pre+Meta-Phys+1.31[−0.01, 2.63]
AR-PreAux+2.05[0.15, 3.95]
AR-PostAux+1.90[−0.59, 4.38]

Adding Post-Dyn contributes +1.19 pp on average (95% CI [0.61, 1.76]). For each seed, we average the differences in adapted success with and without Post-Dyn across the four Pre-Phys/Post-Phys combinations. We compute the mean and 95% tt interval across six seeds.

2. Matched control for the simulated update (Q3). As you suggested, Post-at-W applies the same physical and future prediction losses directly to ww without simulating the update. We match data, initialization, base weights, ramps, gradient limits, outer update counts, and deployment. Full AR-JEPA reaches 39.82% adapted success versus 39.15% for Post-at-W.

We also added First-Order, which keeps the simulated update in the forward pass and uses a first-order meta-gradient. It reaches 37.92%. Both controls exceed Adajepa at 37.51%, showing a benefit from extra source supervision. Post-update supervision and the full meta-gradient further improve the point estimates.

3. Initialization and performance after adaptation (first part of Q2). We agree that most of the lead exists before adaptation. AR-JEPA starts 2.80 pp ahead of Adajepa, and their adaptation gains are 4.74 and 5.23 pp. Auxiliary-only leads Adajepa before adaptation but falls behind afterward. AR-JEPA remains 2.31 pp ahead under the same update rule. Keeping that control advantage through adaptation is the value of AR-JEPA on Push-Object.

4. Physical readouts and joint objectives. Your question about the source of the gains led us to examine physical readouts alongside task success. In §4.3, we track readouts through repeated updates on shared offline trajectories in place of the one-step loss diagnostic DLD_L. In §4.4, Post-Phys and Post-Dyn raise adapted success by 1.25 and 1.19 pp on average, with both 95% CIs above zero.

In §4.5, readout quality and success are positively related across Push-Object and six PointMaze shift levels (mean Pearson r=0.686r=0.686). AR-JEPA maintains strong readouts and achieves the highest mean adapted success on Push-Object. The rankings can differ. With the same physical supervision and strong readouts, AR-JEPA with Post-Dyn achieves higher mean adapted success than Pre+Meta-Phys. Physical readouts show what information is available, and planning also needs predictions of action outcomes. Post-Phys and Post-Dyn address these two needs in the same updated model.

B. Performance under different conditions (second part of Q2)

Your PointMaze comment led us to compare six dynamics shift levels (§4.6). At ×5, AR-JEPA ranks first among eight methods with 37.33% mean adapted success, ahead of Auxiliary-only (36.00%) and AR-PreAux (35.67%). Its physical readout rank rises from fifth in the original setting to second at ×5, ahead of both.

We agree that a simpler configuration is reasonable in the original setting. Frozen Auxiliary-only reaches 92.67%, compared with 88.33% for adapted AR-JEPA, and AR-PreAux leads after adaptation. Under milder shifts, direct state supervision and dynamics inferred from three frames, as discussed by AdaJEPA, may already support control. Joint supervision after the update starts to show its value under stronger shifts.

Thank you for helping us reorganize the experiments and link each conclusion to its evidence.

Thank you for the detailed rebuttal. The response has addressed most of my concerns. I recommend highlighting the changes in the revised manuscript to make them easier to identify and assess. Overall, I am satisfied with the rebuttal and will maintain my positive rating.

Reviewer_9VsSOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies test-time adaptation as a way to address out-of-distribution (OOD) generalization when a world model is deployed on tasks outside its training distribution. The goal is to improve the performance of world-model approaches on OOD tasks. The paper focuses on a specific setting in which privileged information is available during training but not at test time. The method is evaluated in simulated environments against AdaJEPA and a V0 baseline.

Critical strengths
  • Because world models do not rely on expert data, they promise better generalization than policies learned through imitation learning. However, current world models are known to perform worse on tasks that deviate from their training distribution. Recent work such as AdaJEPA suggests that test-time training is a promising way to address this.
  • Preserving representations through test-time updates is an interesting problem that previous work has largely overlooked.
Critical weaknesses
  • The paper uses the term "source" without defining it clearly. It seems to refer to the first training phase; an explicit definition would make the paper easier to follow.
  • The setting is not well motivated. It combines privileged training with test-time adaptation, but the paper does not explain why the privileged information would be unavailable at deployment. Moreover, AR-JEPA is compared to AdaJEPA only in settings where the privileged information could be available at test time: all experiments are in simulation, where the underlying states are always accessible. A more convincing setting would be a robotics scenario in which the world model is trained in simulation, with access to the full state, and then deployed on a real robot where only sensor observations are available.
  • The V0 baseline is never defined, which makes the experimental setup hard to understand and the contributions hard to assess. Is V0 meant to reproduce AdaJEPA? If so, its numbers do not appear to match those reported in the original AdaJEPA paper on the same tasks.
  • More generally, the other baselines and ablations are not clearly defined or motivated, which weakens the claimed contributions.
Questions for authors
  • What exactly is the V0 baseline, and why do its results differ from those reported in the original AdaJEPA paper?
3 comments
AuthorsOpenReview ↗

Response to the Reviewer’s Insightful Comments on AR-JEPA

Dear Reviewer,

Thank you for your careful review. Your comments helped us improve the paper’s presentation and explain more clearly why we included each comparison and what it tests.

1. What is V0, and why do its results differ from AdaJEPA's?(Q1)(W3)

V0 is our AdaJEPA baseline. It uses AdaJEPA's architecture, base training objective, deployment update, and planner, with Pre-Phys, Post-Phys, and Post-Dyn off. We clarified this in the revision.

AdaJEPA's public PushObj release contains only evaluation tasks, so we use different source data. We use the Temporal Straightening code linked by AdaJEPA and the public DINO-WM pusht_noise data specified in its PushT config. All eight configurations share these data. To strengthen the comparison, we expanded their paired Push-Object evaluation from three to six seeds. AdaJEPA data · Wang et al., ICML 2026 · PushT config · Zhou et al., ICML 2025

2. What do we mean by source training?(W1)

Following prior work on test-time adaptation, source training uses offline source-domain trajectories to learn the world model's initial parameters. AR-JEPA simulates the update at this stage. Test-time updates use transitions from closed-loop interaction. We will define the term at first use. Boudiaf et al., ICML 2023 · Park et al., ICCV 2025

3. Privileged supervision and deployment(W2)

Our Related Work (§2.2) discusses training-only privileged supervision. As you point out, we should explain why the labels are unavailable at deployment. Training uses simulator state labels. A robot with only sensor observations cannot access those labels directly, so test-time updates use observations and actions.

We use AdaJEPA's Push-Object and PointMaze simulation benchmarks. As you note, the simulator can still access ground truth at test time. Both methods update and plan using only observations and actions. Scaffolder also trains and tests in simulation, with extra sensor information used only in training. Hu et al., ICLR 2024

Thank you for suggesting full-state training in simulation and sensor-only deployment on a real robot. This would test the practical value of our setting more directly. We will consider adding real-robot experiments.

4. What do the ablations test?(W4)

V0 is the common control with only AdaJEPA's base objective. Pre-Phys and Post-Phys supervise physical representations before and after the simulated update, respectively. Post-Dyn supervises the updated model's predictions of later transitions.

To test whether standard physical supervision is enough and whether post-update supervision adds value, Auxiliary-only uses Pre-Phys, Meta-Phys uses Post-Phys, and Pre+Meta-Phys uses both.

The updated model must also predict the effects of actions for planning. Adding Post-Dyn to V0, Auxiliary-only, Meta-Phys, and Pre+Meta-Phys yields AR-Dyn, AR-PreAux, AR-PostAux, and full AR-JEPA, respectively. These pairs test whether Post-Dyn improves control on its own or with physical supervision. Each changes one source objective under the same deployment update and planner.

5. Revised experiments and additional comparisons

Your point that the draft left the purpose of the comparisons unclear led us to revisit §§4.3–4.6. In §4.3, physical readouts on shared trajectories replace the single-update loss diagnostic DLD_L to track what survives repeated updates. Section 4.4 extends the Post-Dyn analysis to all three objectives, with average effects and paired confidence intervals. Since other variants also gained on both tasks, §4.5 now uses ablations and readouts to explain why the updated model needs physical information and predictions for planning. In §4.6, we vary the dynamics shifts on PointMaze and find that AR-JEPA can outperform simpler variants under stronger shifts.

Appendix B.1 adds two controls with matched supervision data and outer-loop budgets. The full method leads at 39.82% adapted success, with point estimates favoring post-update supervision and the full meta-gradient. In B.2, existing one-step curves and a new three-step comparison show that AR-JEPA leads AdaJEPA late in both matched settings. B.3 shows a gain from increasing test updates per feedback from one to two with the same one-step checkpoint.

AuthorsOpenReview ↗

Follow-up to the Reviewer’s Insightful Comments on AR-JEPA

Dear Reviewer 9VsS,

Thank you again for your review. We clarified that V0 is our AdaJEPA baseline and that its Push-Object results differ from the published numbers because we used different source training data. All eight configurations share these data, and we expanded the paired evaluation from three to six seeds.

We also defined source training, explained why physical state labels are used only during training and how our simulation protocol enforces this restriction at test time, and clarified what each ablation tests.

When you have a chance, could you let us know whether our response addresses your concerns? We would be happy to clarify any points that remain open.

AuthorsOpenReview ↗

Thank You for the Review — A Brief Follow-up

Dear Reviewer,

Sorry to trouble you again as the discussion deadline approaches. We wanted to briefly follow up on our response and ask whether any concerns remain.

Our response explains what V0 is and why its results differ from AdaJEPA's published results. It also clarifies the use of physical labels only during training and what each ablation tests.

Could you kindly let us know whether these explanations address your concerns? If so, we would appreciate your taking them into account in your final assessment. We would be happy to clarify any remaining points before the discussion closes.

Thank you very much for your time and consideration.

Reviewer_q9NrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a supervision-availability mismatch in test-time adaptive world models: physical labels are available during source training but not at deployment, where the update rule is fixed. AR-JEPA leaves deployment unchanged and differentiates through a simulated AdaJEPA update during source training, using Pre-Phys, Post-Phys, and Post-Dyn objectives. Experiments on Push-Object and PointMaze report favorable results and AR-JEPA achieves the highest adapted success on Push-Object.

Critical strengths
  1. The problem is real and underexplored: a source-stage representation constraint need not survive a different self-supervised deployment update.

  2. Full factorial ablation. All eight on/off combinations of Pre-Phys, Post-Phys, and Post-Dyn are evaluated under the same deployment update, which allows the effect of each term to be isolated.

  3. Physical labels and the auxiliary head are removed after training, so any improvement comes without additional test-time supervision or latency.

Critical weaknesses
  1. Gains reflect initialization, not the update. AR-JEPA’s Frozen success is ~3 pp above V0 while adaptation gains are nearly identical (+3.95 vs. +3.74). The only evidence for a better update is cross-domain consistency, which is fragile at 13 domains × 3 seeds; D$_L$ does not resolve the attribution.

  2. PointMaze does not support the method. Auxiliary-only frozen (92.67%) beats AR-JEPA adapted (88.33%), and the top adapted variant omits Post-Phys. Calling this “placement sensitivity” sidesteps the implication that the meta-update is unnecessary when physical supervision is strong.

  3. The complementarity argument (§4.5) fails. Positive gains on both benchmarks also hold for V0, Meta-Phys, and Pre+Meta-Phys; this property does not distinguish the joint family.

Questions for authors
  1. How much of the gain remains if Post-Phys and Post-Dyn are evaluated on the un-updated model w rather than w⁺, or if the meta-gradient is replaced by a first-order approximation?

  2. Source training simulates a single update on 1–5 transitions, whereas deployment applies one update per feedback event across an entire episode. Does the advantage over V0 persist, grow, or vanish as feedback events accumulate, and would a multi-step inner loop during training change the conclusions?

6 comments
AuthorsOpenReview ↗

Response to the Reviewer’s Insightful Comments on AR-JEPA

Dear Reviewer,

Thank you for your careful and insightful review. Your questions get to the heart of our study and helped us choose which comparisons to add. Your comments also helped us reorganize the experiments and connect AR-JEPA's design more clearly to the evidence.

1. How much of the gain remains if we evaluate the Post objectives before the update or use a first-order approximation? (Q1)

Both ablations retain some of the gain (Appendix B.1). Post-at-W and First-Order reach 39.15% and 37.92% success after adaptation, respectively, compared with 37.51% for V0 (Adajepa) and 39.82% for the full method. Extra source supervision already helps, and post-update supervision and the full meta-gradient further improve the point estimates.

2. Does the advantage hold as feedback accumulates or with a multi-step training inner loop? (Q2)

The advantage generally grows as feedback accumulates and holds late in the task. Source training simulates the single update that AdaJEPA makes per feedback event at TTA=1. “1–5 transitions” refers to the number of transitions used for that step. At deployment, updated parameters carry over within a task.

The existing evaluation curves in Appendix B.2 show that the methods start close in the one-step setting. AR-JEPA leads by about 2.2–2.6 percentage points over the last ten feedback events, with a final gap of 2.31 percentage points. We also trained AR-JEPA with a three-step inner loop and used three updates per feedback event for both methods at test time. AR-JEPA still leads late, with final success rates of 40.74% versus 39.49%.

We are also training models with other inner-loop lengths. Each run takes about a day. We wanted to share these results early and hear your thoughts. We will add the rest once training is complete.

Responses to the weaknesses

1. Sources of the gains, retention of physical information, and cross-domain evaluation (W1)

Your comments prompted us to examine frozen success more closely and expand Push-Object evaluation from three to six seeds for all eight methods. The code, Push-Object weights, and test results for all six seeds are available online. Anonymous repository

As you noted, AR-JEPA leads V0 before adaptation and retains its lead under the same deployment update. Auxiliary-only also starts ahead of V0 but falls behind after adaptation. Retaining a stronger model's control advantage through adaptation is one benefit of our source training.

We agree that DLD_L, which measures the change in prediction loss from a single update, cannot separate the contributions of initialization and adaptation. We replaced it in §4.3 with physical readouts that track whether information learned during source training survives repeated updates.

2. Evidence for combining Post-Phys and Post-Dyn (W3)

As you pointed out, several configurations gain on both tasks. To test what each objective contributes, §4.4 compares configurations that differ in one objective. Post-Phys and Post-Dyn improve adapted success by 1.25 and 1.19 percentage points on average, respectively. Both 95% confidence intervals are above zero. This supports directly supervising the updated model during source training.

Section 4.5 explains the role of each objective. Physical readout quality and task success are related, but their rankings sometimes differ. Planning also requires the model to predict the outcomes of candidate actions. AR-JEPA and Pre+Meta-Phys use the same physical supervision. Adding Post-Dyn raises mean adapted success from 38.51% to 39.82%. Together with the ablations, these results support using Post-Phys to retain physical information and Post-Dyn to improve the predictions used for planning.

3. PointMaze performance at different shift levels (W2)

Your comments prompted us to compare six levels of dynamics shift (§4.6). At ×5, AR-JEPA ranks first among eight methods with 37.33% mean adapted success, ahead of Auxiliary-only at 36.00% and AR-PreAux at 35.67%. Its physical readout rank also rises from fifth in the original setting to second at ×5, ahead of both configurations.

This helps explain the original results. Under milder shifts, direct state supervision and the ability to infer dynamics from a three-frame history, as discussed by AdaJEPA, may already provide enough information for control. Under stronger shifts, AR-JEPA begins to benefit from supervising both physical information and predictions after the update.

4. Reorganized experiments

The revised manuscript presents the control results, physical readouts, effects of individual and combined objectives, and the PointMaze shift analysis in that order. Appendices B.1–B.3 report the additional comparisons and results for repeated feedback and multiple updates. Thank you for helping us clarify the evidence for each conclusion and the conditions under which it holds.

AuthorsOpenReview ↗

Follow-up on Multi-Step Adaptation in AR-JEPA

Dear Reviewer ,

Thank you again for your review. Following up on Q2, we have completed the two-step and five-step experiments on Push-Object. The table includes the earlier one-step and three-step results for comparison.

The table shows the difference in cumulative success between AR-JEPA and V0 (AdaJEPA), in percentage points. The first number in each column is the number of simulated updates in AR-JEPA’s training inner loop. The second is the number of test-time updates per feedback event for both methods.

Feedback events1 / 12 / 23 / 35 / 5
2+0.15+0.13+0.18+0.26
4+0.72+0.72+1.21+1.31
6+1.56+1.28+1.18+2.21
8+1.00+2.49+1.13+0.69
10+1.79+2.18+0.62+1.00
12+2.51+2.08+0.44+0.38
14+2.26+1.79+1.13+0.44
16+2.18+2.03+0.95+0.51
18+2.36+2.03+1.15+0.44
20 (final)+2.31+2.38+1.26+0.77

Across the four settings, AR-JEPA’s lead grows early and holds late in the task, consistent with our response to Q2. Averaged equally across the settings, the lead rises from 0.18 percentage points after two feedback events to 1.40 after ten and stays within 1.35–1.68 points from the tenth through the final event. AR-JEPA finishes ahead of V0 in all four settings.

When you have a chance, could you let us know whether these results and our earlier response address your concerns? We would be happy to clarify any remaining points.

Reviewer_q9NrOpenReview ↗

Response to the authors

Thank you for the response, which addresses my concern about the complementarity. I am also curious about where the gain comes from. Without simulating the update the model already obtains most of the improvement over V0, while the full method adds a relatively small further gain. Does this suggest that adding these objectives during source training is already effective by itself, and that the contribution of simulating the update is relatively limited?

In addition, the paper does not include any visualizations. Since both Push-Object and PointMaze use image observations, and AdaJEPA shows planning trajectories and decoded rollouts on the same benchmarks, could the authors explain why qualitative results are not provided?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.