Recover and Protect: Managing Interference in Continual Test-Time Adaptation

Continual Test-Time Adaptation (CTTA) requires learning from changing, unlabeled domains while retaining knowledge for recurring Continual Test-Time Adaptation (CTTA) requires learning from changing, unlabeled domains while retaining knowledge for recurring conditions. However, adaptation updates can both transfer useful knowledge and interfere with previous learning. We propose Interference Recovery and Protection (IRP), a geometric framework that proposes a novel decomposition of shared and domain-specific knowledge. These components allow precise identification and mitigation of harmful interference for each domain while preserving plasticity. IRP unifies two complementary mechanisms: *recovery* adaptively corrects accumulated cross-domain interference to rapidly restore a model suited to the returning domain, while *protection* minimally adjusts optimizer updates that oppose the direction previously learned for that domain, limiting self-training error accumulation. Domain shifts and recurring domains are identified online through reconstruction losses and gradient signatures, without domain labels or annotated boundaries. Results on CTTA benchmarks show that IRP reaches state of the art results over multiple adaptation rounds, demonstrating the benefit of targeted interference management for knowledge transfer and retention.