acceptodds
Withdrawn from ICLR 2027

Recover and Protect: Managing Interference in Continual Test-Time Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual Test-Time Adaptation (CTTA) requires learning from changing, unlabeled domains while retaining knowledge for recurring Continual Test-Time Adaptation (CTTA) requires learning from changing, unlabeled domains while retaining knowledge for recurring conditions. However, adaptation updates can both transfer useful knowledge and interfere with previous learning. We propose Interference Recovery and Protection (IRP), a geometric framework that proposes a novel decomposition of shared and domain-specific knowledge. These components allow precise identification and mitigation of harmful interference for each domain while preserving plasticity. IRP unifies two complementary mechanisms: *recovery* adaptively corrects accumulated cross-domain interference to rapidly restore a model suited to the returning domain, while *protection* minimally adjusts optimizer updates that oppose the direction previously learned for that domain, limiting self-training error accumulation. Domain shifts and recurring domains are identified online through reconstruction losses and gradient signatures, without domain labels or annotated boundaries. Results on CTTA benchmarks show that IRP reaches state of the art results over multiple adaptation rounds, demonstrating the benefit of targeted interference management for knowledge transfer and retention.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Test-Time Adaptation, Continual Learning, Test-Time Adaptation, Domain Shift, Optimization Interference, Parameter-Space Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_rNRfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies knowledge transfer, forgetting, and error accumulation in continual test-time adaptation. Without target-domain annotations, source training data, or true domain boundaries, the authors propose Interference Recovery and Protection. The method treats the average parameters of historical domain model checkpoints as a shared component and the difference between each checkpoint and the average as a domain-specific direction. When a previous domain reappears, Recovery adaptively corrects the current adapter along the historical direction. During subsequent learning, Protection minimally adjusts optimizer updates that conflict with this direction. In addition, the method detects domain changes through reconstruction loss and identifies revisited domains through gradient signatures.

Critical strengths

The paper designs an online mechanism for identifying revisited domains based on gradient signatures. By comparing parameter adjustment directions for different data on a fixed source model, it matches the current domain to historical domains without requiring true domain labels. This design identifies the environment based on the needs of model adaptation and combines the identification results with historical knowledge recovery and update protection, providing a relatively complete approach to knowledge reuse when domain identities are unknown.

Critical weaknesses

1.Pseudo-labels may be incorrect, and image reconstruction does not guarantee correct semantic recognition. Historical checkpoints are also learned from these unlabeled objectives and may therefore already contain errors.

2.Equations (4)–(7) in Section 4.2 rely on the assumption that historical parameter residuals are approximately aligned with the negative gradient of the current domain loss. However, a historical checkpoint direction describes a displacement relative to a shared reference point and is not necessarily still a descent direction at the current parameter position. Therefore, an update opposite to the historical direction may also be correcting an excessive shift caused by cross-domain adaptation. The existing performance ablations are insufficient to validate this judgment. I suggest that the authors provide an alignment analysis between historical directions and the current true negative gradients, and directly test whether the updates modified by Protection actually harm performance on the current domain.

Questions for authors

1.Section 4.2 treats historical parameter residuals as approximate descent directions for the current domain and accordingly considers updates in the opposite direction to be harmful interference. However, after the model adapts to other domains, historical directions may no longer align with the negative gradient at the current position. How do the authors validate this assumption? Could an offline analysis compare the effects of updates before and after modification by Protection on the true task loss, demonstrating that Protection mainly blocks harmful updates rather than necessary corrections?

2.If the historical domain checkpoints themselves are of poor quality, can the method correct previous errors?

3.Could the authors provide some theoretical derivations that lead to an important conclusion?

Reviewer_F2YsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Interference Recovery and Protection (IRP) for continual test-time adaptation. It decomposes model parameters into domain-general and domain-specific components, using recovery to mitigate interference when domains recur and protection to constrain conflicting optimizer updates. Domain changes and recurrence are detected using reconstruction losses and gradient signatures. Experiments on semantic segmentation and image classification show improvements over existing methods across repeated adaptation rounds.

Critical strengths
  1. The paper addresses both cross-domain interference and error accumulation, with separate recovery and protection mechanisms that target different failure modes in continual adaptation.

  2. The experiments cover semantic segmentation and three corruption-classification benchmarks over repeated adaptation rounds. IRP achieves an overall mIoU of 62.7 on Cityscapes-to-ACDC and final-round error rates of 9.5%, 16.1%, and 32.7% on CIFAR-10-C, CIFAR-100-C, and ImageNet-C, respectively.

  3. The ablation studies examine recovery, protection, learned correction coefficients, and the tolerance parameter. The paper also analyzes domain detection failures rather than presenting the online assignment mechanism as universally reliable.

Critical weaknesses
  1. The writing is clearly rushed and lacks clarity, with significant portions showing signs of AI generation; it requires substantial revision. Furthermore, due to the excessive use of AI in the drafting process, it is currently unclear what the actual contribution of this paper is, and the underlying motivation for the work appears somewhat questionable. The decomposition assumes that the checkpoint difference between a domain-specific model and the averaged model approximates a useful descent direction for that domain. This assumption is not sufficiently validated, and parameter averaging does not guarantee that the resulting direction corresponds to a domain-specific gradient.

  2. The domain-general component is computed by averaging domain-adapted checkpoints. The paper does not establish how reliably this average separates shared and domain-specific knowledge when adaptation trajectories are nonlinear or when different domains produce incompatible parameter updates.

  3. The recovery mechanism optimizes correction coefficients using the current adaptation loss on a returning domain. It remains unclear whether this procedure consistently recovers useful domain-specific knowledge or sometimes overfits noisy pseudo-labels.

  4. The reconstruction-based detector misses some transitions, including Night-to-Rain in ACDC and several transitions between similar corruption types in CIFAR-10-C. Such errors directly affect the subsequent recovery and protection operations.

  5. Domain assignment uses benchmark-specific cosine-similarity thresholds, ranging from 0.35 to 0.90. The sensitivity of the results to these thresholds and to the number of previously encountered domains is not sufficiently established.

Questions for authors
  1. What empirical evidence supports treating the checkpoint difference τj=θj−θG\tau_j=\theta_j-\theta_G as an approximation to a useful descent direction for domain jj? Can the authors measure its alignment with actual domain gradients?

  2. How sensitive are the results to the domain-assignment thresholds? Can a single threshold-selection procedure work across benchmarks without using domain labels?

  3. How does IRP perform when domain shifts are gradual, domains recur at irregular intervals, or previously encountered domains appear in combinations not observed during earlier adaptation?

  4. Can the authors quantify memory and computational costs as the number of stored domain checkpoints and gradient signatures increases?

  5. Can the authors evaluate whether recovery remains beneficial when pseudo-labels are unreliable or when the reconstruction loss fails to detect a domain transition?

Reviewer_Z72QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses continual test-time adaptation (CTTA) when domains recur. It averages the checkpoints obtained after adapting to each domain to get a “domain-general” model, and treats each checkpoint’s offset from this average as a domain-specific direction. On top of a mean-teacher framework with masked-image-modeling loss and AdaptFormer adapters, it proposes two mechanisms. Recovery: when a domain returns, the part of the current adapter displacement that lies along that domain’s direction is subtracted, with per-block scaling coefficients learned on the current batch. Protection: optimizer updates that oppose the stored domain direction are minimally corrected through a closed-form half-space projection. Domain shifts and recurrences are identified online from reconstruction-loss statistics and gradient signatures, without domain labels. The paper reports state-of-the-art results on Cityscapes-to-ACDC segmentation and on CIFAR10-C, CIFAR100-C and ImageNet-C classification over multiple rounds.

Critical strengths
  • The paper targets a practical setting, and the ideas are well motivated.
  • The evaluation covers segmentation and three classification benchmarks.
Critical weaknesses
  1. Limited novelty. The “novel geometric view” combines task-vector construction (checkpoint averaging and differencing; Ilharco et al., 2023; Wortsman et al., 2022) with a one-sided linear projection of the GEM/GPM type (Lopez-Paz & Ranzato, 2017; Saha et al., 2021), with the reference direction replaced by a checkpoint difference. Shared versus domain-specific decomposition (ViDA, Liu et al., 2024b; BECoTTA, Lee et al., 2024) and recurring-domain reuse (ReservoirTTA, Vray et al., 2025; DPCore, Zhang et al., 2025) are already established. The first-order justification also depends on an unverified alignment assumption.

  2. Weak validation of the central claim. The only component ablation (Table 4) is on ACDC. The full method gains just 0.50 mIoU over the variant without recovery and protection, individual components change results by about 0.1, showing weak contribution of each component. The main classification gains (Table 3) have no ablation or sequential-training baseline, so improvement over rounds may simply reflect continued adapter training. Several text claims are also contradicted by the tables:

  • IRP is called the “only method” that improves over rounds, although TENT-continual and CMAE also improve on CIFAR10-C.
  • Gains are said to grow over rounds, but IRP’s mIoU peaks at round 4 (63.26) and falls to 62.35 by round 10.
  • IRP is said to beat the runner-up in all four conditions at round 10, but it trails ViDA and BECoTTA on Fog.

Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
Lee, D., Yoon, J., and Hwang, S. J. BECoTTA: Input-dependent online blending of experts for continual test-time adaptation. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of PMLR, pp. 27072–27093, 2024.
Liu, J., Yang, S., Jia, P., Zhang, R., Lu, M., Guo, Y., Xue, W., and Zhang, S. ViDA: Homeostatic visual domain adapter for continual test time adaptation. In International Conference on Learning Representations, 2024b.
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. Advances in Neural Information Processing Systems, 30, 2017.
Saha, G., Garg, I., and Roy, K. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2021.
Vray, G., Tomar, D., Gao, X., Thiran, J.-P., Shelhamer, E., and Bozorgtabar, B. ReservoirTTA: Prolonged test-time adaptation for evolving and recurring domains. Advances in Neural Information Processing Systems, 38:63357–63403, 2025.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965–23998. PMLR, 2022.
Zhang, Y., Mehra, A., Niu, S., and Hamm, J. DPCore: Dynamic prompt coreset for continual test-time adaptation. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of PMLR, pp. 75757–75778, 2025.

Questions for authors
  1. Beyond averaging checkpoints and projecting onto a stored direction, what is the conceptual contribution of the proposed geometric view? Please compare directly against subtracting the stored domain direction with a single tuned scalar, and against a standard one-sided gradient projection that uses checkpoint directions.

  2. How is the average model computed online, and does the stored domain direction remain valid as the average changes? Can you test empirically that the domain direction aligns with the negative loss gradient, and explain why averaging checkpoints from a single sequential trajectory is justified?

  3. Please show the significance of ablation tests.

Reviewer_LwHHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Interference Recovery and Protection (IRP), a geometric framework for Continual Test-Time Adaptation (CTTA) that aims to mitigate knowledge interference while preserving adaptation capability across evolving unlabeled domains. The method decomposes model knowledge into shared and domain-specific components to identify and manage harmful interference. Specifically, IRP introduces two complementary mechanisms: recovery, which corrects accumulated cross-domain interference when previously encountered domains reappear, and protection, which adjusts conflicting optimizer updates to reduce knowledge degradation during adaptation. Additionally, the framework identifies domain shifts and recurring domains online using reconstruction losses and gradient signatures, without requiring domain labels or explicit boundaries. Experiments on CTTA benchmarks demonstrate competitive performance across multiple adaptation rounds, highlighting the effectiveness of interference-aware knowledge management.

Critical strengths

The paper presents a geometric decomposition of shared and domain-specific knowledge, providing an intuitive approach to mitigating interference while preserving knowledge transfer.

The proposed recovery and protection mechanisms effectively address cross-domain interference and error accumulation while maintaining model plasticity.

The framework identifies domain shifts and recurring domains online without requiring domain labels, annotated boundaries, or access to source data.

Extensive experiments demonstrate strong performance across multiple adaptation rounds, while ablation studies validate the effectiveness of the proposed mechanisms.

Critical weaknesses
  1. The recovery mechanism relies on unlabeled adaptation losses, which may lead to unreliable corrections when pseudo-label predictions are inaccurate.

  2. Domain shift detection and assignment rely on reconstruction losses and gradient signatures, which may be sensitive to noise and gradual distribution shifts.

  3. The experiments primarily focus on predefined recurring domain sequences, leaving the robustness of IRP under more complex and unpredictable domain shifts insufficiently evaluated.

  4. The baseline methods included in the experiments are relatively outdated, with insufficient comparisons against recent state-of-the-art CTTA approaches. This limits the comprehensiveness of the experimental evaluation and makes it difficult to fully assess the competitiveness and advantages of IRP over more advanced methods.

Questions for authors

Please see weaknesses.

Comments on the paper

1 comment

We thank the reviewer for their time and thoughtful feedback on our submission. The comments were constructive and helped us identify several areas where the paper can be strengthened. We have decided to withdraw the paper at this time. We will carefully incorporate this feedback and improve the paper substantially for a future submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.