- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
AR-JEPA studies how supervision available only during source training can improve a world model that subsequently adapts using a different, self-supervised objective. Building on AdaJEPA, it simulates the deployment update during source training and differentiates through that update. Physical supervision is applied before and after the simulated update, while an additional post-update dynamics objective supervises prediction on later transitions. Physical labels and the auxiliary head are removed at deployment, leaving AdaJEPA’s adaptation procedure and planner unchanged. Experiments on Push-Object and PointMaze compare eight combinations of these objectives. Full AR-JEPA achieves the highest adapted success on Push-Object, while a simpler variant performs better on PointMaze.
Critical strengths
-
The paper identifies a clear and practically relevant mismatch: physical supervision can shape a representation during training, but its benefits may not survive subsequent self-supervised updates. Evaluating the updated world model during source training is a sensible way to address this problem without adding deployment-time labels or auxiliary components.
-
The factorized experiments are a useful strength. Comparing all eight combinations of pre-update physical supervision, post-update physical supervision, and post-update dynamics supervision provides a more informative analysis than a comparison against the base model alone. Keeping the deployment updater and planner fixed also helps isolate the effects of source training. The paper openly reports that the best configuration differs across environments.
Critical weaknesses
Important implementation and comparison details are missing. Section 3.4 directs readers to the appendix for loss weights and training schedules, but the appendix defers these values to a future code release and leaves the rule for limiting post-adaptation gradients unspecified. These choices could materially affect the comparisons, especially given the different preferred configurations across environments.
Questions for authors
-
Can you provide paired confidence intervals for the differences between AR-JEPA and the strongest relevant baselines, including the effect of adding Post-Dyn?
-
How should we interpret the adaptation-specific contribution when most of the Push-Object advantage exists before adaptation and frozen Auxiliary-only achieves the highest PointMaze success? What evidence supports choosing the full method over the simpler alternatives in these settings?
-
Does differentiating through the simulated update improve performance over a control using the same physical and future-prediction supervision, data, and comparable training budget, but without the simulated update?
-
Please provide the exact loss weights, training durations, ramp schedules, gradient-limiting rule, and validation procedure used to select them. Were these settings held fixed across the factorized comparisons?
2 comments
Thank you for the detailed rebuttal. The response has addressed most of my concerns. I recommend highlighting the changes in the revised manuscript to make them easier to identify and assess. Overall, I am satisfied with the rebuttal and will maintain my positive rating.
Response to the Reviewer’s Insightful Comments on AR-JEPA
Dear Reviewer,
Thank you for your careful and insightful review and for recognizing the value of our work. Your questions helped us focus the new comparisons.
Training settings and reproducibility (weakness and Q4)
Appendix A now gives the exact loss weights, training lengths, ramp schedules, gradient limit, and validation procedure. The eight configurations within each environment share these settings. A disabled objective has zero weight. We calibrate weights on source training data, use source validation only to monitor training, and take the checkpoint at the fixed training endpoint.
We expanded Push-Object evaluation from three to six seeds for all eight methods. The anonymous GitHub repository contains the code, the checkpoint for the original 40.56% result, and test records for the original and added seeds.
Our reading is that your remaining questions focus on two points. First, how much does AR-JEPA help, and where do the gains come from? Second, how do these gains change across deployment conditions?
A. Push-Object performance and sources of gains
1. Paired confidence intervals and the effect of Post-Dyn (Q1). AR-JEPA has the highest mean adapted success under the fixed deployment update. Its lead over the strongest ablation, Pre+Meta-Phys, is 1.31 pp (paired 95% CI [−0.01, 2.63]).
Adding Post-Dyn contributes +1.19 pp on average (95% CI [0.61, 1.76]). For each seed, we average the differences in adapted success with and without Post-Dyn across the four Pre-Phys/Post-Phys combinations. We compute the mean and 95% t interval across six seeds.
2. Matched control for the simulated update (Q3). As you suggested, Post-at-W applies the same physical and future prediction losses directly to w without simulating the update. We match data, initialization, base weights, ramps, gradient limits, outer update counts, and deployment. Full AR-JEPA reaches 39.82% adapted success versus 39.15% for Post-at-W.
We also added First-Order, which keeps the simulated update in the forward pass and uses a first-order meta-gradient. It reaches 37.92%. Both controls exceed Adajepa at 37.51%, showing a benefit from extra source supervision. Post-update supervision and the full meta-gradient further improve the point estimates.
3. Initialization and performance after adaptation (first part of Q2). We agree that most of the lead exists before adaptation. AR-JEPA starts 2.80 pp ahead of Adajepa, and their adaptation gains are 4.74 and 5.23 pp. Auxiliary-only leads Adajepa before adaptation but falls behind afterward. AR-JEPA remains 2.31 pp ahead under the same update rule. Keeping that control advantage through adaptation is the value of AR-JEPA on Push-Object.
4. Physical readouts and joint objectives. Your question about the source of the gains led us to examine physical readouts alongside task success. In §4.3, we track readouts through repeated updates on shared offline trajectories in place of the one-step loss diagnostic DL. In §4.4, Post-Phys and Post-Dyn raise adapted success by 1.25 and 1.19 pp on average, with both 95% CIs above zero.
In §4.5, readout quality and success are positively related across Push-Object and six PointMaze shift levels (mean Pearson r=0.686). AR-JEPA maintains strong readouts and achieves the highest mean adapted success on Push-Object. The rankings can differ. With the same physical supervision and strong readouts, AR-JEPA with Post-Dyn achieves higher mean adapted success than Pre+Meta-Phys. Physical readouts show what information is available, and planning also needs predictions of action outcomes. Post-Phys and Post-Dyn address these two needs in the same updated model.
B. Performance under different conditions (second part of Q2)
Your PointMaze comment led us to compare six dynamics shift levels (§4.6). At ×5, AR-JEPA ranks first among eight methods with 37.33% mean adapted success, ahead of Auxiliary-only (36.00%) and AR-PreAux (35.67%). Its physical readout rank rises from fifth in the original setting to second at ×5, ahead of both.
We agree that a simpler configuration is reasonable in the original setting. Frozen Auxiliary-only reaches 92.67%, compared with 88.33% for adapted AR-JEPA, and AR-PreAux leads after adaptation. Under milder shifts, direct state supervision and dynamics inferred from three frames, as discussed by AdaJEPA, may already support control. Joint supervision after the update starts to show its value under stronger shifts.
Thank you for helping us reorganize the experiments and link each conclusion to its evidence.