acceptodds
Under review as a conference paper at ICLR 2027

CUG-JEPA: Continual Test-Time Adaptation of Latent World Models under Visual Condition Shifts

Abstract

Latent world models can exploit interactive feedback to improve planning, yet single-episode adaptation does not amount to reliable learning under continual deployment: resetting the model after each episode discards previously acquired corrections, whereas unconstrained continual updates may accumulate negative transfer and catastrophic forgetting. To address this challenge, we propose CUG-JEPA, a continual test-time adaptation framework for sequential shifts in visual conditions. Within the closed loop of model predictive control, CUG-JEPA jointly combines non-monotonic uncertainty weighting, online parameter-importance protection, and conditional local restoration, preserving model state across episodes while balancing adaptation to the current environment against long-term knowledge retention. At deployment time, the model uses real transitions naturally observed after action execution as self-supervision, without requiring additional expert demonstrations. Under the primary continual visual-stream protocol on PushObj, spanning three unseen shapes and five random seeds, CUG-JEPA performs one gradient update per feedback step and improves AdaJEPA's overall success rate from 37.3% to 42.9%, an absolute gain of 5.6 percentage points. Results on disjoint task batches further reveal that the gains depend on the task set, providing practical evidence for moving world models beyond short-term error correction toward controlled self-updating under continual deployment.

Primary area:
applications to robotics, autonomy, planning
Keywords:
continual test-time adaptation, latent world models, continual learning, model predictive control, visual distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_U4jdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CUG-JEPA, a continual test-time adaptation method that removes the episode reset in AdaJEPA and carries parameters and adaptation state across episodes. To control accumulated updates, the paper combines non-monotonic uncertainty weighting, online parameter-importance protection, and conditional local restoration within the MPC loop. On the continual visual stream of PushObj, the paper reports an improvement over AdaJEPA from 37.3% to 42.9%; however, on a disjoint task batch the ranking is reversed, and the aggregate gain decreases to +3.0 percentage points.

Critical strengths

Continual TTA of a world model in a closed-loop setting is a meaningful problem formulation. Addressing adaptation and retention simultaneously in a scenario where updates alter subsequent actions and feedback distinguishes this problem from existing classification-centric CTTA, and I agree with the authors that it is an important problem.

Critical weaknesses

1. The effectiveness of the proposed mechanisms is not confirmed by the experiments in the paper itself.

  • In "Value of online adaptation" (Fig. 4a), no benefit from UG is observed. Under Fixed SNP1, the success rate is the same as that of AdaJEPA (47.5%), and under PushT rapid switch it is even lower (80.3% vs. 81.5%). Based on these results, it is difficult to conclude that uncertainty weighting is effective.
  • In Fig. 4(b), shuffled weights (53.5%) achieve a higher success rate than online calibration (51.5%). If performance does not decline even when the correspondence between samples and weights is randomly shuffled, this implies that the weighting does not utilize sample-specific information, which casts doubt on the effectiveness of online calibration itself.
  • In Fig. 4(c), the no-restoration condition (56.1%) outperforms all restoration conditions. There is no explanation as to why the condition without restoration consistently performs better. Furthermore, while the main protocol uses a fixed threshold (KL ≥ 2000 / CV ≥ 1.0), this experiment employs statistical thresholds such as P95, P99, and P99.5, so the two settings do not correspond to each other.
  • According to L261–264, the weighted implementation disables the baseline's contiguous-segment merging. Therefore, the difference between AdaJEPA and UG cannot be attributed solely to the effect of the weighting function. The exact difference between the two implementations is not isolated.

2. The main claim and the comparative design do not support the motivation.

  • The paper's motivation is that simply removing the episode reset causes continual updates to accumulate negative transfer and forgetting (L045–046). However, the main continual-stream protocol does not jointly compare a frozen model, episode-reset AdaJEPA, and naive persistent (no-reset) AdaJEPA, and §4.2 does not specify which of the latter two the reported AdaJEPA baseline corresponds to. Without this comparison, it is impossible to fairly attribute where the benefits of the proposed method come from.
  • Fig. 1 is labeled as a conceptual curve. It is unclear whether this drift accumulation was actually observed; if it was not, the phenomenon should be demonstrated in the experimental results. As it stands, Fig. 1 does not serve as a main figure.
  • The authors themselves state that improvements are not consistent across all task batches, which weakens the claim itself. There is no analysis of why the ranking was reversed in the non-overlap batch (51.1% vs. 41.1%). Although the authors refer to "practical evidence," the computational cost should also be reported; however, there is no report on the cost.
  • There is no performance comparison with state-of-the-art TTA baselines, including CoTTA. This comparison is essential.
  • It is unclear what Fig. 4(d) is intended to show. The figure reports only a "continual update" condition and full fine-tuning, and the paper does not state whether the "continual update" condition corresponds to CUG-JEPA or to naive continual updates. The connection to the primary claim is not explained.
Questions for authors
  1. Is the cumulative drift shown in Fig. 1(a) an actually observed phenomenon? If so, please provide the corresponding experimental results.
  2. Can you present a comparison in the main continual-stream protocol that includes a frozen model, episode-reset AdaJEPA, and naive persistent AdaJEPA?
  3. Apart from disabling segment merging, what are the differences between the AdaJEPA and UG implementations? Are there any results for AdaJEPA with only merging disabled?
  4. How do you interpret the fact that shuffled weights yield higher success than online calibration in Fig. 4(b)?
  5. In Fig. 4(c), why does the no-restoration condition perform better than all restoration conditions? Which of the percentile thresholds in this experiment does the fixed threshold of the main protocol correspond to?
  6. Have you analyzed why the ranking was reversed in the non-overlap task batch?
  7. Regarding the claim of "practical evidence," how does the computational cost compare to that of AdaJEPA?
  8. Can you provide a comparison with state-of-the-art TTA baselines such as CoTTA?
  9. What does Fig. 4(d) aim to demonstrate regarding the primary claim?
3 comments
AuthorsOpenReview ↗

Response to Reviewer U4jd

Thank you for the careful review. We clarify the baseline and motivation and add matched controls, ablations, costs, and a CoTTA-style comparison. Success-rate differences are in percentage points.

Q1. Conceptual drift

Thank you for identifying this ambiguity. We narrowed the Introduction (pp. 1–2) and clarified that Fig. 1 shows hypothetical trajectories (p. 2). A 24-stream diagnostic did not detect progressive control decline after initial adaptation: persistent adaptation’s episode-24 minus episode-1 difference was +2.08 points, with a 95% interval spanning zero. Its SR@20 budget and external clock differ from the primary protocol; it does not establish cumulative control collapse or demonstrate that CUG-JEPA resolves forgetting.

Q2. Baseline controls

Our primary comparator is naive persistent AdaJEPA; the historical 37.3% baseline is persistent, not reset. Matched SR@30 evaluations cover three shapes, five seeds, and 450 episodes per method. Frozen, reset, persistent, and CUG-JEPA achieve 28.4%, 43.8%, 37.3%, and 42.9%. Persistent is significantly below reset. CUG-JEPA’s +5.56-point advantage over persistent does not survive Holm correction across four comparisons, and superiority over reset is not established. See Table 1 (p. 8) and Appendix E.1 (pp. 18–19).

Q3. Segment merging and UG

Thank you for noting the attribution issue. Beyond disabling merging, UG uses MC-dropout prediction-loss variance, non-monotonic weighting, filtering, and weighted averaging (§3.2, p. 5). In six matched SR@20 streams, disabling merging alone changes success from 32.6% to 34.0%, a +1.39-point difference with 95% interval [−3.47, +6.94]. The mechanism subset does not establish independent UG or restoration benefits. These results are separate from SR@30. See p. 8 and Appendix E.2 (pp. 19–20).

Q4. Shuffled weights

Thank you for noting this distinction. Shuffling permutes sample-to-weight assignments, not model parameters. Online calibration achieves 51.5% versus 53.5% for shuffled weights, a −2.00-point difference with 95% interval [−4.53, +0.53]. This does not establish a control benefit from calibration or systematic superiority of shuffling. Prediction-error risk may not identify useful control updates, but that explanation remains unverified. See pp. 8–9 and Fig. 4(b).

Q5. Restoration thresholds

Thank you for noting the unclear correspondence. No restoration achieves 56.1%, above P95/P99/P99.5 at 49.7%/49.1%/53.0% (Fig. 4(c), p. 9); a restoration benefit is not established. Removing useful adaptation is a possible, unverified explanation. KL = 2000 matches none of these percentiles and exceeds the calibration maximum of 601.66. The primary rule also includes loss CV ≥ 1.0 as an OR condition and a 20-step cooldown, so it cannot equal a single KL percentile (§3.2, p. 6; Appendix B.1, pp. 13–14).

Q6. Ranking reversal

Our audit of 36 runs verified matched ordered task states; reversal persists when comparing seed 100 in both batches. “Non-overlap” applies to the original two-seed collection; two task-state pairs per shape overlap with the expanded five-seed collection. Interaction-driven visual timing can change exposure histories, but no clock-by-batch intervention has established causality. The cause remains unresolved, and the negative batch demonstrates task-dependent gains. See §4.1 (p. 6), p. 8, and §5.1 (p. 10).

Q7. Computational cost

On an RTX 5080 under SR@30, median planning-plus-adaptation time increases from 0.7950 s for persistent AdaJEPA to 1.0401 s for CUG-JEPA (+30.8%); adaptation alone increases from 0.0316 s to 0.2746 s (8.7 times). These trajectory-dependent measurements exclude full simulation, loading, and probes; they are not shared-input microbenchmarks. See p. 9 and Appendix E.3 (p. 20, Table E5).

Q8. CoTTA comparison

The added CoTTA-style world-model adapter achieves 28.0% versus CUG-JEPA’s 42.9% across 15 matched streams and 450 episodes per method. The difference is +14.89 points, with 95% interval [+10.44, +19.56] and Holm-adjusted p=0.00037p = 0.00037 across three comparisons. Its classification objective is adapted to transition-prediction and teacher-consistency MSE. This supports superiority over this specified adapter, not unchanged official CoTTA or TTA methods generally. See Table 1 (p. 8) and Appendix D (pp. 16–17).

Q9. Fig. 4(d)

Thank you for noting the unclear label. “Continual update” is naive persistent adaptation on LeWorldModel, not CUG-JEPA. Under combined visual/dynamics shifts, persistent adaptation and full fine-tuning obtain MSEs of 0.0142/0.0078 but both achieve only 4% control success. This motivates directly measuring control and retention; it neither establishes CUG-JEPA’s effectiveness nor substitutes for primary-protocol forgetting evidence. See Fig. 4(d), p. 9, and Appendix A.3 (p. 12).

AuthorsOpenReview ↗

Response to Reviewer U4jd

Thank you again for your feedback. We have conducted additional experiments and revised the manuscript to address the weaknesses you identified. Could you please let us know whether these additions and revisions address your main concerns? If any concerns remain, we would be happy to provide further clarification and conduct additional experiments to address them.

I thank the authors for the additional experiments and the clear response. The new results confirm my concerns, and I keep my rating.

(1) The method does not outperform the reset baseline. Episode-reset AdaJEPA has a higher point estimate than CUG-JEPA, and the comparator in the submission was the persistent variant rather than the reset method.

(2) The drift accumulation that motivates the paper is not observed. Persistent adaptation shows no progressive decline, and Fig. 1 is hypothetical.

(3) None of the three mechanisms has an attributable benefit. Disabling segment merging alone produces a gain comparable to UG, so no independent benefit of uncertainty weighting is established. Also calibration and restoration have no effect, and the added CoTTA-style baseline performs at the level of the frozen model.

I do not think these issues can be fixed by revision under the current framing.

Reviewer_Ne4GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates continual test-time adaptation of latent world models under sequential visual shifts. Based on AdaJEPA, the paper proposes CUG-JEPA, which maintains adaptation state across episodes and combines uncertainty-based sample weighting, online parameter-importance regularization, and conditional encoder restoration. Experiments on PushObj with three unseen object shapes demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper is well-writtent and easy to follow.

  2. The investigated problem, continual adaptation of latent world model, is interesting and important.

  3. The experiments and reports are transparent and show some interesting findings.

Critical weaknesses
  1. The paper combines tes-time adaptation of latent world models with several continual test-time adaptation strategies, such as uncertainty-based selection, parameter regularization, and restoration toward pretrained weights, which has already been proposed by previous TTA research like EATA, CoTTA, etc. It is not clear why these methods benefit and which of them are specific to the closed-loop setting, which limits the contribution of the paper.

  2. The compared baseline methods are mainly about the uncertainty estimators rather than continual adaptation methods. More continual test-time adaptation baselines should be considered for comparisons.

  3. The ablation studies are not sufficient. A uniform-weight baseline with identical segmentation is essential. Moreover, shuffled weights achieve a higher point estimate than loss-variance weighting in the calibration study, making it unclear whether the utility of the sample–uncertainty correspondence works.

  4. The improvement of the proposed method is not consistent. Although the primary improvement is +5.6 percentage points, but it does not remain significant under the reported multiple-comparison correction. Moreover, the performance of CUG-JEPA on the disjoint task batch is even worse than AdaJEPA.

Questions for authors

Please refer to the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ne4G

Thank you for the constructive review. Your concerns helped us sharpen the contribution and improve the evaluation. We have clarified the motivation, added a continual-TTA comparison and matched ablations, and narrowed our claims. Success-rate differences below are in percentage points.

1. Contribution and the closed-loop setting

We agree that sample selection, parameter regularization, and restoration have precedents in EATA, CoTTA, and related TTA work. Section 2 positions our framework relative to these approaches. We have revised the Introduction, Fig. 1 caption, and method framing to separate update rules from demonstrated control-retention benefits (pp. 1–2, Section 3). Adapting the latent model changes subsequent MPC plans and actions, which changes the transitions available for later learning. Prediction-error reduction therefore need not imply better control. Our implementation combines non-monotonic prediction-loss-variance weighting, online source-anchored parameter protection, and conditional encoder restoration across episodes. These specify how feedback and persistent updates are regulated; they do not establish independent component benefits. The contribution is their integration and evaluation in continual latent planning. We do not claim the individual mechanisms as new or uniquely necessary for closed-loop control, and improved control retention remains unestablished.

2. Additional continual-adaptation baseline

Following your suggestion, we added a CoTTA-style world-model adapter under the original SR@30 protocol, matching task states across 15 shape-by-seed streams and 450 episodes per method (Table 1, p. 8; Appendix D, pp. 16–17). It retains an independent EMA teacher used by MPC, 32-view augmentation-averaged update targets, stochastic restoration, and persistent Adam. Classification objectives are replaced by observed-transition and teacher-consistency MSE; augmentations preserve action geometry, and the classification-confidence gate is omitted. We disclose these departures rather than claim an unchanged official CoTTA reproduction. CUG-JEPA achieves 42.9% versus 28.0%, a +14.89-point difference with unadjusted 95% paired CI [+10.44, +19.56] and Holm p=0.00037p = 0.00037 across three prespecified comparisons. Inference uses whole streams rather than independent episodes. Appendix D also discloses unequal anchor access and optimizer semantics. CUG uses clean initial observations including later tasks, while AdaJEPA/CUG recreate Adam per adaptation call. The result concerns this specified adapter and cannot establish general CTTA superiority or isolate a component. We have not added EATA; baseline coverage remains limited.

3. Identical segmentation and shuffled weights

Thank you for identifying the segmentation confound. Section 3.2 now explicitly states that UG also disables merging; Appendix E.2 adds an unmerged uniform-weight baseline and component-removal results (pp. 19–20). In six matched SR@20 streams, disabling merging alone changes success from 32.6% to 34.0%, a +1.39-point difference with 95% CI [−3.47, +6.94]. On the identical four-stream I/square subset, unmerged uniform adaptation and UG achieve 25.00% and 26.04%; the difference is +1.04 points with 95% CI [−6.25, +8.33]. Independent UG and restoration benefits are not established. These controls use a separate SR@20 protocol, not the primary SR@30 setting. We also agree that Fig. 4(b) does not demonstrate useful sample-to-weight correspondence. Loss-variance weighting, online calibration, and shuffled weights achieve 50.4%, 51.5%, and 53.5%. Online minus shuffled is −2.00 points with 95% CI [−4.53, +0.53]. Shuffling permutes sample-to-weight assignments. Predictive risk may differ from control-update value, but this is an unverified interpretation. The new ablation does not overturn this negative calibration evidence.

4. Inconsistent improvements and revised claims

We agree that robust superiority is not established. The original +5.6-point comparison has Holm p=0.3350p = 0.3350 across eight comparisons. On the additional task batch, CUG-JEPA achieves 41.1% versus persistent AdaJEPA's 51.1%; the deduplicated aggregate difference is +3.0 points with 95% CI [−2.0, +7.8] (Section 5, p. 8). The ranking reversal remains unresolved. We added Frozen, episode-reset, and persistent controls in Table 1 and Appendix E.1, achieving 28.4%, 43.8%, and 37.3%, versus CUG-JEPA's 42.9%. This rerun establishes neither superiority over reset nor corrected superiority over persistent (Holm p=0.09570p = 0.09570 across four contrasts). The revised Introduction and Fig. 1 distinguish conceptual trajectories from empirical evidence; the Conclusion qualifies control-retention claims, and Section 5.1 retains the task-sampling limits. We appreciate the opportunity to make the contribution and its uncertainties more precise.

I thank the authors for the additional experiments. The episode-reset baseline and the matched-segmentation ablation address some of my concerns about the original comparisons. However, on the primary task batch, CUG-JEPA achieves a higher success rate than AdaJEPA without episode resets (42.9% vs. 37.3%), but its advantage does not carry over to the additional task batch, where the ranking reverses. The primary results also do not establish an advantage over episode-reset AdaJEPA (43.8%), and the matched-segmentation ablation leaves the benefit of uncertainty weighting unresolved. Given the remaining uncertainty about whether the proposed combination offers a consistent benefit, I keep my original weak reject rating.

AuthorsOpenReview ↗

Response to Reviewer Ne4G

Thank you for emphasizing this comparison. Our goal is to improve success when adaptation updates persist across episodes. This setting matters because task boundaries need not coincide with environmental changes. For example, a factory robot may complete many container transfers while the same camera defocus or lighting change persists. Resetting after each transfer can remove harmful updates, but also discards useful corrections and requires adapting again. We therefore investigate how to retain useful experience while limiting harmful updates. In the primary batch, CUG-JEPA improves success over no-reset AdaJEPA from 37.3% to 42.9%. However, we acknowledge that this remains below episode-reset AdaJEPA’s 43.8%, and the additional-batch reversal leaves consistency unresolved. The factory example motivates persistent adaptation; it does not establish its superiority. Our current results have not demonstrated an advantage over episode resets.

Reviewer_BDrcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CUG-JEPA combines uncertainty weighting, parameter-importance protection, and conditional restoration for continual adaptation of latent world models. Simulated visual-shift experiments show a 5.6-percentage-point improvement over AdaJEPA in the primary evaluation, alongside mixed results on additional tasks.

Critical strengths

The paper addresses an important deployment problem: adapting from interaction while preserving knowledge across episodes. Paired evaluations, confidence intervals, and negative controls help characterize both benefits and failure conditions.

Critical weaknesses

The improvement is not robust: the primary comparison fails Holm correction, the disjoint batch reverses the ranking, and the aggregate confidence interval includes zero. Component benefits remain unclear. Shuffled weighting and no restoration perform strongly in separate diagnostics, while UG also changes segment merging, complicating attribution.

Questions for authors

Clarification: Does the primary AdaJEPA baseline retain parameters and optimizer state across episodes? Clarification: How do configuration differences between the main experiments and mechanism diagnostics explain their conflicting results?

1 comment
AuthorsOpenReview ↗

Response to Reviewer BDrc

Thank you for the thoughtful review and for recognizing the value of paired evaluations and negative controls. Your questions identified ambiguities in our baseline and protocol descriptions. We have clarified state lifetimes, added matched controls and ablations, and narrowed the interpretation. Success-rate differences below are in percentage points.

1. Baseline parameters and optimizer state

Thank you for highlighting the unclear baseline definition. The primary 37.3% comparator is naive persistent AdaJEPA. It carries adapted parameters across episodes, but Adam is recreated at every adaptation call, including within an episode. The recent-5 transition buffer is cleared at episode boundaries. CUG-JEPA also uses fresh Adam per call. It carries importance and drift-monitoring state across episodes; restoration clears the corresponding importance/EMA state but retains detector statistics. We have clarified this in Section 4.2, the Fig. 3 caption, and Appendix E.1 (Table E1 and accompanying text, p. 18). The added SR@30 controls cover three shapes, five seeds, and 450 episodes per method. Frozen, episode-reset AdaJEPA, persistent AdaJEPA, and CUG-JEPA achieve 28.4%, 43.8%, 37.3%, and 42.9% (Table 1, p. 8). Superiority over episode-reset AdaJEPA is not established.

2. Main experiments and mechanism diagnostics

We agree that the protocol distinctions needed to be clearer. Section 4.2 separates the primary continual stream from the targeted diagnostics, and the added Appendix E.2 specifies the supplementary ablation protocol. The main experiment uses 30 episodes per stream, at most 30 MPC iterations per episode, and switches every 100 actual interaction steps. The new segmentation ablation instead uses 24 episodes, at most 20 MPC iterations, a fixed external visual clock, per-episode seeding, and Monte Carlo RNG isolated from planning. The original implementation shares that RNG stream. The older return-to-clean diagnostic uses 40 episodes, 50-step switches, and protection strength 1 rather than 10; its percentile restoration settings differ from the primary KL/CV trigger. Fig. 4(b) is a separate calibration study. These experiments answer different questions and cannot be pooled. The configuration differences do not establish which factor caused the conflicting outcomes. No matched intervention across these factors has resolved the discrepancy, and the unfavorable within-diagnostic comparisons remain limitations.

3. Component attribution and negative controls

We agree that the original AdaJEPA-to-UG comparison confounds weighting with segment handling. We have made this explicit in Section 3.2 and added an unmerged uniform-weight control and component-removal results in Appendix E.2 (pp. 19–20). Disabling merging alone changes SR@20 from 32.6% to 34.0% over six matched streams, a +1.39-point difference with 95% CI [−3.47, +6.94]. On the same four-stream I/square subset, UG and unmerged uniform adaptation achieve 26.04% and 25.00%. Independent UG and restoration benefits are not established. We also retain the original negative evidence. Online calibration achieves 51.5% versus 53.5% for shuffled weights; their paired difference is −2.00 points with 95% CI [−4.53, +0.53]. No restoration achieves 56.1% versus 49.7%, 49.1%, and 53.0% for P95, P99, and P99.5 (Fig. 4, p. 9). These diagnostics establish neither a control benefit from sample-specific weighting nor a restoration benefit. The overall CUG-JEPA result cannot substitute for component-level evidence.

4. Robustness and revised scope

We agree that the improvement is task dependent. The original +5.6-point comparison has Holm-adjusted p=0.3350p = 0.3350 across eight comparisons. The additional batch reverses the ranking (CUG-JEPA 41.1%, persistent AdaJEPA 51.1%); the deduplicated aggregate gain is +3.0 points with 95% CI [−2.0, +7.8] (Section 5, p. 8). The added state-control rerun also does not establish corrected superiority over persistent AdaJEPA (Holm p=0.09570p = 0.09570 across four prespecified contrasts, Table E2). We have narrowed the Introduction, identified Fig. 1's trajectories as hypothetical, and qualified the control-retention claims in the Conclusion (pp. 1–2, 10). Section 5.1 retains the task-sampling and incomplete-ablation limits. The supported finding is a favorable primary point estimate with unresolved robustness and component attribution. Thank you for helping us make these distinctions explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.