acceptodds
Under review as a conference paper at ICLR 2027

Reliability-Gated Source Anchoring for Continual Test-Time Adaptation

Abstract

Continual test-time adaptation (CTTA) updates a pretrained model on an unlabeled, changing stream while using source information to limit drift. The challenge is to preserve useful source knowledge without letting an uncertain reference restrict adaptation. We propose RMemSafe, a reliability-gated extension of ROID that uses the frozen source's normalized predictive entropy to modulate a parameter anchor and source-expert agreement weighting. As the source posterior approaches uniformity, these explicit source-dependent loss contributions vanish while the base adaptation losses remain active. Matched controls separate this mechanism from the auxiliary stabilizers in the complete method. On CCC-Hard with ResNet-50, runtime gating reduces error by percentage points relative to the same objective with the gate fixed to one, and by points relative to removing the explicit source terms. On two long-horizon stream orderings, it reduces error by and points, with improvement in all evaluated seeds. Auxiliary stabilizers account for most of the average CCC improvement, and source-term removal remains stronger on the CCC mean. Source-degradation, small-batch, and class-permutation experiments further characterize the method and its limits. Together, these results motivate adjusting source influence to the current stream rather than assuming that retaining or removing it is always preferable.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Test Time Adaptations, CTTA, TTA, Test Time adaptations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LKaHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RMEMSAFE, a reliability-gated framework for continual test-time adaptation. It uses the frozen source model's normalized predictive entropy to dynamically modulate a parameter anchor and source-expert agreement weighting, while incorporating several auxiliary stabilizers. Experiments on CCC and long-horizon corruption streams demonstrate improvements under certain distribution shifts, particularly in challenging adaptation scenarios.

Critical strengths
  1. The paper addresses an important challenge in CTTA: balancing source knowledge preservation and adaptation under distribution shifts. The proposed reliability-gated mechanism is intuitive and easy to integrate into existing frameworks.

  2. The experiments cover multiple benchmarks, model architectures, and challenging adaptation scenarios. The matched controls and ablation studies provide insights into the behavior of the proposed method.

Critical weaknesses
  1. The core idea of this paper overlaps with prior studies. VCoTTA [1] has already explored using the source model's predictive entropy to adaptively adjust the mixing weights of the source and teacher priors. OSTTA [2] dynamically controls the adaptation process by comparing the prediction confidence of the original source model and the adapted model. Therefore, the authors should better clarify the novelty of their approach and provide experimental comparisons with these closely related methods.

  2. As shown in Tab. 1, adding decoupled flip inference reduces the mean error from 61.28% to 59.43%. However, flip inference is an established test-time augmentation technique and is not directly related to the proposed source reliability gating mechanism. In contrast, dynamic gating provides marginal improvements over the fixed gating. These results suggest that most of the performance gains of the full method come from auxiliary components, while the independent contribution of reliability gating is relatively limited. The authors should further justify the effectiveness of the proposed mechanism and provide more comprehensive ablation studies.

  3. The use of predictive entropy as a measure of source reliability is not sufficiently justified. When the source model has low accuracy, the proposed reliability metric may fail to accurately reflect its actual performance degradation. The authors should evaluate the effectiveness of the proposed mechanism when the source model makes confidently incorrect predictions.

[1] Variational Continual Test-Time Adaptation

[2] Towards Open-Set Test-Time Adaptation Utilizing the Wisdom of Crowds in Entropy Minimization

Questions for authors
  1. What are the key methodological differences between RMEMSAFE and prior works such as VCoTTA and OSTTA?

  2. Can the authors provide further ablations to isolate the contribution of reliability gating from auxiliary components?

  3. How does the proposed gating mechanism behave when the source model makes confidently incorrect predictions?

Reviewer_dYSNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RMEMSAFE, a reliability-gated extension of ROID for continual test-time adaptation. The core mechanism is a scalar gate R_src = max(0, 1 − H_src) derived from the frozen source model's normalized predictive entropy, which modulates an ℓ2 parameter anchor and a source–expert agreement weighting so that source-dependent loss terms vanish as the source becomes unreliable under severe shift. Validation on CCC and long-horizon streams reports small error reductions (0.30 pp on CCC-Hard, 0.11 pp on the CCC mean) relative to a fixed gate, with the authors noting that auxiliary stabilizers account for most of the average improvement.

Critical strengths

The paper identifies a real and consequential failure mode. Static source anchoring is genuinely harmful when the frozen source itself collapses under severe shift (e.g., 1.33% top-1 accuracy on CCC-Hard), and the paper articulates this clearly. This is a useful observation for the CTTA community. However, the identification of a real problem does not, by itself, carry the proposed solution, which is where the paper falls short.

Critical weaknesses
  1. The proposed mechanism is a trivial scalar gate whose gains are marginal and partly self-contradictory. R_src = max(0, 1 − H_src) is a standard source-entropy reliability signal multiplied onto two loss terms: the signal is not new, and the structure is a scalar multiplication. Empirically the gate yields only 0.11 pp on the CCC mean and 0.30 pp on CCC-Hard (Table 2). The authors' own Table 1 shows that simply removing the source terms is better on the CCC mean (59.36% vs. 60.23%), with the gate winning only on the Hard subset; and the abstract concedes that auxiliary stabilizers account for most of the average improvement. These results undercut the claim that reliability gating is the key contribution. The formalism compounds the issue — Proposition 1 reduces to substituting R_src = 0 into Eq. (3)–(4), and the emphatic language ("categorically proves", "obliterates") overstates a marginal result.

  2. The reliability premise of the gate is unvalidated. Entropy measures predictive confidence, not correctness. Under severe OOD shift, models frequently emit low-entropy yet incorrect predictions (overconfidence), in which case the gate would retain the source anchor precisely when it should be relaxed. The paper provides no evidence that R_src correlates with actual source accuracy, so the method's central premise rests on an unverified assumption.

Questions for authors
  1. Given that removing the source terms outperforms the runtime gate on the CCC mean (59.36 vs. 60.23), and that auxiliary stabilizers account for most of the average improvement, what is the empirical evidence that the reliability gate — rather than the auxiliary stabilizers — is responsible for the reported gains? (Clarification that would materially affect my assessment.)

  2. R_src uses entropy as a proxy for source reliability, but entropy measures confidence rather than correctness. Have you validated that R_src correlates with actual source accuracy, and how does the method behave under overconfidence in severe OOD? (Clarification that would materially affect my assessment.)

  3. Please report comparisons against standard CTTA baselines (TENT, CoTTA, SAR, DeYO, ViDA) under matched inputs, since the current comparisons are confined to the ROID/ASR family. (Clarification that would materially affect my assessment.)

1 comment
Reviewer_dYSNOpenReview ↗

Official Review of Submission45384 to Reviewer dYSN

Thanks to the author for the detailed rebuttal. While I appreciate the clarifications, my main concerns remain unresolved. Therefore, I will keep my original rating.

Reviewer_s5cWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual test-time adaptation (CTTA), where regularizing the adapting model toward a frozen source can become harmful once the source degrades under severe shifts, while discarding source information entirely gives up useful stability. The authors propose RMEMSAFE, an extension of ROID that computes a reliability scalar from the frozen source’s normalized predictive entropy. This scalar scales two source-dependent components, an ℓ2 parameter anchor toward the source weights and a cosine-based source–expert agreement weight, so that when the source posterior approaches uniformity both terms vanish and only the base adaptation losses and marginal calibration remain. The full method additionally includes ungated auxiliary stabilizers: marginal calibration against an EMA class prior, a confidence-scaled learning rate, and flip-based logit interpolation at inference. The paper evaluates the method on CCC, CIFAR-10-C, and ImageNet-C streams, showing improvements

Critical strengths
  1. The proposed method is simple and interpretable.
  2. The method achieves positive results.
Critical weaknesses
  1. The benefit of dynamic gating is not isolated. Without a weaker constant-gate control, the gains could come from simply reducing source coupling rather than adapting to reliability. Source-term removal also achieves better CCC mean error than runtime gating (59.36% versus 60.23%), and Table 2 provides no gate-specific confidence intervals.

  2. Reliability and safety claims exceed the evidence. Table 16's high-/low-reliability outcome means match the i.i.d./correlated groups, so its whole-run AUC = 1.000 may reflect stream type and does not establish early warning. The permutation test also worsens ViT mean error by 1.14 percentage points (Table 18), undermining the claimed robustness to a confidently incorrect source.

Questions for authors
  1. Please resolve to weaknesses.

  2. Is dynamic gating better than weaker constant coupling? Does runtime gating outperform a constant gate in (0, 1) selected on the same held-out split and applied to both Eqs. (3) and (4)?

  3. What is frozen, and which stream is plotted? Please specify the source normalization/EMA behavior and explain Table 15's ordering-dependent mean entropy.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their responses to your comments. Please review their rebuttals and engage in further discussion, particularly regarding any remaining concerns, unresolved issues, or points requiring additional clarification.

Thank you for your time and continued contributions to the review process.

Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.