- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CUG-JEPA, a continual test-time adaptation method that removes the episode reset in AdaJEPA and carries parameters and adaptation state across episodes. To control accumulated updates, the paper combines non-monotonic uncertainty weighting, online parameter-importance protection, and conditional local restoration within the MPC loop. On the continual visual stream of PushObj, the paper reports an improvement over AdaJEPA from 37.3% to 42.9%; however, on a disjoint task batch the ranking is reversed, and the aggregate gain decreases to +3.0 percentage points.
Critical strengths
Continual TTA of a world model in a closed-loop setting is a meaningful problem formulation. Addressing adaptation and retention simultaneously in a scenario where updates alter subsequent actions and feedback distinguishes this problem from existing classification-centric CTTA, and I agree with the authors that it is an important problem.
Critical weaknesses
1. The effectiveness of the proposed mechanisms is not confirmed by the experiments in the paper itself.
- In "Value of online adaptation" (Fig. 4a), no benefit from UG is observed. Under Fixed SNP1, the success rate is the same as that of AdaJEPA (47.5%), and under PushT rapid switch it is even lower (80.3% vs. 81.5%). Based on these results, it is difficult to conclude that uncertainty weighting is effective.
- In Fig. 4(b), shuffled weights (53.5%) achieve a higher success rate than online calibration (51.5%). If performance does not decline even when the correspondence between samples and weights is randomly shuffled, this implies that the weighting does not utilize sample-specific information, which casts doubt on the effectiveness of online calibration itself.
- In Fig. 4(c), the no-restoration condition (56.1%) outperforms all restoration conditions. There is no explanation as to why the condition without restoration consistently performs better. Furthermore, while the main protocol uses a fixed threshold (KL ≥ 2000 / CV ≥ 1.0), this experiment employs statistical thresholds such as P95, P99, and P99.5, so the two settings do not correspond to each other.
- According to L261–264, the weighted implementation disables the baseline's contiguous-segment merging. Therefore, the difference between AdaJEPA and UG cannot be attributed solely to the effect of the weighting function. The exact difference between the two implementations is not isolated.
2. The main claim and the comparative design do not support the motivation.
- The paper's motivation is that simply removing the episode reset causes continual updates to accumulate negative transfer and forgetting (L045–046). However, the main continual-stream protocol does not jointly compare a frozen model, episode-reset AdaJEPA, and naive persistent (no-reset) AdaJEPA, and §4.2 does not specify which of the latter two the reported AdaJEPA baseline corresponds to. Without this comparison, it is impossible to fairly attribute where the benefits of the proposed method come from.
- Fig. 1 is labeled as a conceptual curve. It is unclear whether this drift accumulation was actually observed; if it was not, the phenomenon should be demonstrated in the experimental results. As it stands, Fig. 1 does not serve as a main figure.
- The authors themselves state that improvements are not consistent across all task batches, which weakens the claim itself. There is no analysis of why the ranking was reversed in the non-overlap batch (51.1% vs. 41.1%). Although the authors refer to "practical evidence," the computational cost should also be reported; however, there is no report on the cost.
- There is no performance comparison with state-of-the-art TTA baselines, including CoTTA. This comparison is essential.
- It is unclear what Fig. 4(d) is intended to show. The figure reports only a "continual update" condition and full fine-tuning, and the paper does not state whether the "continual update" condition corresponds to CUG-JEPA or to naive continual updates. The connection to the primary claim is not explained.
Questions for authors
- Is the cumulative drift shown in Fig. 1(a) an actually observed phenomenon? If so, please provide the corresponding experimental results.
- Can you present a comparison in the main continual-stream protocol that includes a frozen model, episode-reset AdaJEPA, and naive persistent AdaJEPA?
- Apart from disabling segment merging, what are the differences between the AdaJEPA and UG implementations? Are there any results for AdaJEPA with only merging disabled?
- How do you interpret the fact that shuffled weights yield higher success than online calibration in Fig. 4(b)?
- In Fig. 4(c), why does the no-restoration condition perform better than all restoration conditions? Which of the percentile thresholds in this experiment does the fixed threshold of the main protocol correspond to?
- Have you analyzed why the ranking was reversed in the non-overlap task batch?
- Regarding the claim of "practical evidence," how does the computational cost compare to that of AdaJEPA?
- Can you provide a comparison with state-of-the-art TTA baselines such as CoTTA?
- What does Fig. 4(d) aim to demonstrate regarding the primary claim?
3 comments
Response to Reviewer U4jd
Thank you again for your feedback. We have conducted additional experiments and revised the manuscript to address the weaknesses you identified. Could you please let us know whether these additions and revisions address your main concerns? If any concerns remain, we would be happy to provide further clarification and conduct additional experiments to address them.
I thank the authors for the additional experiments and the clear response. The new results confirm my concerns, and I keep my rating.
(1) The method does not outperform the reset baseline. Episode-reset AdaJEPA has a higher point estimate than CUG-JEPA, and the comparator in the submission was the persistent variant rather than the reset method.
(2) The drift accumulation that motivates the paper is not observed. Persistent adaptation shows no progressive decline, and Fig. 1 is hypothetical.
(3) None of the three mechanisms has an attributable benefit. Disabling segment merging alone produces a gain comparable to UG, so no independent benefit of uncertainty weighting is established. Also calibration and restoration have no effect, and the added CoTTA-style baseline performs at the level of the frozen model.
I do not think these issues can be fixed by revision under the current framing.
Response to Reviewer U4jd
Thank you for the careful review. We clarify the baseline and motivation and add matched controls, ablations, costs, and a CoTTA-style comparison. Success-rate differences are in percentage points.
Q1. Conceptual drift
Thank you for identifying this ambiguity. We narrowed the Introduction (pp. 1–2) and clarified that Fig. 1 shows hypothetical trajectories (p. 2). A 24-stream diagnostic did not detect progressive control decline after initial adaptation: persistent adaptation’s episode-24 minus episode-1 difference was +2.08 points, with a 95% interval spanning zero. Its SR@20 budget and external clock differ from the primary protocol; it does not establish cumulative control collapse or demonstrate that CUG-JEPA resolves forgetting.
Q2. Baseline controls
Our primary comparator is naive persistent AdaJEPA; the historical 37.3% baseline is persistent, not reset. Matched SR@30 evaluations cover three shapes, five seeds, and 450 episodes per method. Frozen, reset, persistent, and CUG-JEPA achieve 28.4%, 43.8%, 37.3%, and 42.9%. Persistent is significantly below reset. CUG-JEPA’s +5.56-point advantage over persistent does not survive Holm correction across four comparisons, and superiority over reset is not established. See Table 1 (p. 8) and Appendix E.1 (pp. 18–19).
Q3. Segment merging and UG
Thank you for noting the attribution issue. Beyond disabling merging, UG uses MC-dropout prediction-loss variance, non-monotonic weighting, filtering, and weighted averaging (§3.2, p. 5). In six matched SR@20 streams, disabling merging alone changes success from 32.6% to 34.0%, a +1.39-point difference with 95% interval [−3.47, +6.94]. The mechanism subset does not establish independent UG or restoration benefits. These results are separate from SR@30. See p. 8 and Appendix E.2 (pp. 19–20).
Q4. Shuffled weights
Thank you for noting this distinction. Shuffling permutes sample-to-weight assignments, not model parameters. Online calibration achieves 51.5% versus 53.5% for shuffled weights, a −2.00-point difference with 95% interval [−4.53, +0.53]. This does not establish a control benefit from calibration or systematic superiority of shuffling. Prediction-error risk may not identify useful control updates, but that explanation remains unverified. See pp. 8–9 and Fig. 4(b).
Q5. Restoration thresholds
Thank you for noting the unclear correspondence. No restoration achieves 56.1%, above P95/P99/P99.5 at 49.7%/49.1%/53.0% (Fig. 4(c), p. 9); a restoration benefit is not established. Removing useful adaptation is a possible, unverified explanation. KL = 2000 matches none of these percentiles and exceeds the calibration maximum of 601.66. The primary rule also includes loss CV ≥ 1.0 as an OR condition and a 20-step cooldown, so it cannot equal a single KL percentile (§3.2, p. 6; Appendix B.1, pp. 13–14).
Q6. Ranking reversal
Our audit of 36 runs verified matched ordered task states; reversal persists when comparing seed 100 in both batches. “Non-overlap” applies to the original two-seed collection; two task-state pairs per shape overlap with the expanded five-seed collection. Interaction-driven visual timing can change exposure histories, but no clock-by-batch intervention has established causality. The cause remains unresolved, and the negative batch demonstrates task-dependent gains. See §4.1 (p. 6), p. 8, and §5.1 (p. 10).
Q7. Computational cost
On an RTX 5080 under SR@30, median planning-plus-adaptation time increases from 0.7950 s for persistent AdaJEPA to 1.0401 s for CUG-JEPA (+30.8%); adaptation alone increases from 0.0316 s to 0.2746 s (8.7 times). These trajectory-dependent measurements exclude full simulation, loading, and probes; they are not shared-input microbenchmarks. See p. 9 and Appendix E.3 (p. 20, Table E5).
Q8. CoTTA comparison
The added CoTTA-style world-model adapter achieves 28.0% versus CUG-JEPA’s 42.9% across 15 matched streams and 450 episodes per method. The difference is +14.89 points, with 95% interval [+10.44, +19.56] and Holm-adjusted p=0.00037 across three comparisons. Its classification objective is adapted to transition-prediction and teacher-consistency MSE. This supports superiority over this specified adapter, not unchanged official CoTTA or TTA methods generally. See Table 1 (p. 8) and Appendix D (pp. 16–17).
Q9. Fig. 4(d)
Thank you for noting the unclear label. “Continual update” is naive persistent adaptation on LeWorldModel, not CUG-JEPA. Under combined visual/dynamics shifts, persistent adaptation and full fine-tuning obtain MSEs of 0.0142/0.0078 but both achieve only 4% control success. This motivates directly measuring control and retention; it neither establishes CUG-JEPA’s effectiveness nor substitutes for primary-protocol forgetting evidence. See Fig. 4(d), p. 9, and Appendix A.3 (p. 12).