CUG-JEPA: Continual Test-Time Adaptation of Latent World Models under Visual Condition Shifts

Latent world models can exploit interactive feedback to improve planning, yet single-episode adaptation does not amount to reliable learning under continual deployment: resetting the model after each episode discards previously acquired corrections, whereas unconstrained continual updates may accumulate negative transfer and catastrophic forgetting. To address this challenge, we propose CUG-JEPA, a continual test-time adaptation framework for sequential shifts in visual conditions. Within the closed loop of model predictive control, CUG-JEPA jointly combines non-monotonic uncertainty weighting, online parameter-importance protection, and conditional local restoration, preserving model state across episodes while balancing adaptation to the current environment against long-term knowledge retention. At deployment time, the model uses real transitions naturally observed after action execution as self-supervision, without requiring additional expert demonstrations. Under the primary continual visual-stream protocol on PushObj, spanning three unseen shapes and five random seeds, CUG-JEPA performs one gradient update per feedback step and improves AdaJEPA's overall success rate from 37.3% to 42.9%, an absolute gain of 5.6 percentage points. Results on disjoint task batches further reveal that the gains depend on the task set, providing practical evidence for moving world models beyond short-term error correction toward controlled self-updating under continual deployment.