acceptodds
Withdrawn from ICLR 2027

Two-Stage Isotropic Whitening for Source-Free Test-Time Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When deploying machine learning models in real-world scenarios, distribution shift poses a critical challenge. Test data often deviates from training distributions and severely degrades model performance. This issue becomes especially acute in source-free test-time adaptation (SF-TTA) because models must adapt to unlabeled target data without source data or annotations. To address this challenge, we introduce Two-Stage Isotropic Whitening (TSIW) to facilitate target feature learning. TSIW integrates Whitening Batch Normalization (WBN) with Whitening Contrastive Learning (WCL). At the feature level, WBN enforces isotropic feature distributions via ZCA whitening. This suppresses domain-specific covariance structures and improves stability under distribution shifts. At the representation level, WCL extends standard contrastive learning through global feature whitening. This eliminates redundant feature correlations and optimizes representation geometry on a hypersphere to preserve semantic relationships. Through this two-stage process, WBN handles low-level feature standardization while WCL refines global representation geometry. Together, they yield highly generalizable features. Extensive experiments demonstrate that TSIW achieves state-of-the-art performance on major benchmarks, including VisDA-C, DomainNet-126, ImageNet-C, and CIFAR-100-C.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Source-freee test-time adaptation, Two-stage whitening, Whitening batch normalization, Whitening contrastive learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_2kvsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper considers Two-Stage Isotropic Whitening (TSIW) for source-free test-time adaptation. TSIW adds two components to AdaContrast: 1) Whitening Batch Normalization (WBN) which aims to reduce domain-specific correlations in feature-level, and 2) Whitening Contrastive Learning (WCL) which aims to reduce redundant dependencies and to form a well-structure hyper-spherical space while preserving semantic relationship. The proposed method is evaluated on VisDA-C (87.8% vs. 86.8% for AdaContrast) and DomainNet-126 (69.2% vs. 67.8%). The paper also reports that adding whitening improves Tent, CoTTA, MEMO and RoTTA on ImageNet-C and CIFAR-100-C.

Critical strengths

A simple add-on approach: WBN and WCL can be added to AdaContrast with few changes. They improve accuracy on VisDA-C and DomainNet-126, and the ablation (Table 3) shows the two components are complementary.

Critical weaknesses
  1. Limited novelty and unclear points:
  • WBN follows the Cholesky-based feature whitening of Roy et al. (2019), which was already proposed for domain adaptation.
  • WCL is the W-MSE loss of Ermolov et al. (2021), but lines 249–253 call it “our proposed W-MSE loss.”
  1. The results do not support the claims:
  • Tabl4 contradicts the text. The averages computed from the table differ from those stated in the text (Sec. 3.5). For example, the MEMO+whitening actually decreases the accuracy from 47.4 to 43.8 (big drop for Snow) while authors said that it improves from 43.2 to 44.2.
  • The objective is undefined in Eq. 12.
Questions for authors
  1. What is technically new in WBN and WCL compared to Roy et al. (2019) and Ermolov et al. (2021)?
  2. Please recalculate the averages in Table4 with those stated in the text.
  3. Please write out the full TSIW objective, eq12.
Reviewer_RunPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Two-Stage Isotropic Whitening (TSIW) for source-free test-time adaptation (SF-TTA), motivated by the hypothesis that domain-dependent feature correlations and covariance structures contribute to performance degradation under distribution shifts. The main contribution is the integration of two whitening mechanisms into an AdaContrast-based framework: (1) Whitening Batch Normalization (WBN), which decorrelates intermediate features to encourage isotropic feature distributions, and (2) Whitening Contrastive Learning (WCL), which applies representation-level whitening to reduce feature redundancy and improve embedding geometry while preserving semantic consistency. To validate the proposed approach, the authors conduct experiments on VisDA-C, DomainNet-126, ImageNet-C, and CIFAR-100-C, comparing against existing domain adaptation and test-time adaptation methods. They also provide ablation studies to examine the contributions of WBN and WCL individually. Based on these experiments, the authors claim that combining the two whitening stages improves classification accuracy and robustness across various distribution shifts.

Critical strengths

The paper demonstrates empirical improvements over the AdaContrast baseline on VisDA-C and DomainNet-126. In particular, the results on DomainNet-126 show improvements across all evaluated domain shifts, and the ablation study suggests that both WBN and WCL contribute to the overall performance gain. These results provide some evidence that combining feature-level and representation-level whitening can be beneficial for domain adaptation.

Critical weaknesses
  1. Insufficient justification of the core hypothesis and unclear methodological novelty.

The paper’s central motivation is not convincingly established. Although domain-dependent covariance structures may contribute to performance degradation under distribution shifts, the authors provide neither theoretical justification nor direct empirical evidence supporting this hypothesis. More importantly, the paper assumes that whitening removes undesirable domain-specific correlations while preserving semantic information, without addressing the possibility that feature correlations may also contain class-discriminative information. The experiments focus almost exclusively on classification accuracy; no analyses of covariance structures, feature anisotropy, or representation geometry (e.g., covariance discrepancy between domains before and after whitening, or the eigenvalue spectrum of the learned embeddings) are provided to verify the claimed mechanisms. Furthermore, the proposed framework primarily combines existing techniques within AdaContrast: WBN largely follows the batch whitening of DWT (Roy et al., 2019), and WCL closely mirrors the W-MSE loss of Ermolov et al. (2021). The distinctions from these methods are insufficiently discussed. In addition, the motivation and Figure 1 describe ZCA whitening, whereas the implementation adopts Cholesky-based whitening, and the effect of this choice is not examined. Consequently, the paper does not convincingly establish either the necessity or the distinctive contribution of the proposed two-stage approach.

  1. Unclear evaluation protocol and limited comparisons.

Although the paper defines TTA as adaptation during inference and explicitly targets online adaptation, the use of 15 epochs of target training raises concerns about whether the evaluation protocol aligns with this setting, especially when comparing methods designed for different adaptation protocols. The comparisons with recent SF-TTA methods are also limited, as the most recent baseline in Tables 1 and 2 is AdaContrast (2022), which weakens the claimed state-of-the-art performance. Moreover, given that the method targets test-time adaptation, the computational and memory overhead of whitening is not reported.

  1. Questionable reliability and reproducibility of the reported results.

(a) Inconsistencies between claims and results. Despite claims of consistent improvements, Table 4 shows substantial performance degradation under several corruptions, particularly for CoTTA (e.g., Defocus 46.8 → 37.9, Glass 51.5 → 39.3) and MEMO (e.g., Snow 54.7 → 34.5). The text states that MEMO improves on Motion and Brightness, whereas Table 4 shows decreases on both (41.2 → 36.5 and 66.7 → 56.4). The reported averages are also not clearly reconciled with the presented results, and only 12 of the 15 corruption types are shown. In addition, the “WCL” results in Figure 2 match the full TSIW results in Table 3 rather than the WCL-only ablation results.

(b) Lack of statistical evidence. The improvements are relatively modest (e.g., +1.0 percentage points on VisDA-C and +1.4 percentage points on DomainNet-126) and are reported without repeated runs, standard deviations, or confidence intervals.

(c) Missing implementation details. The final training objective is not explicitly defined (Eq. 12 does not specify how the WCL loss is combined with the AdaContrast losses or how they are weighted), and it is unclear how whitening is integrated into the different TTA baselines in Table 4.

These limitations substantially undermine confidence in the reliability, reproducibility, and generalizability of the reported improvements.

Questions for authors

Q1. Justification of the core hypothesis (Supporting evidence)

The paper argues that domain-dependent covariance structures are a major source of performance degradation under distribution shifts and that whitening mitigates this issue. However, the experiments primarily report classification accuracy without directly examining this hypothesis. Can the authors provide empirical evidence demonstrating that (1) covariance structures differ meaningfully across domains, (2) the proposed method reduces the relevant domain-dependent correlations, and (3) these changes are associated with improved target-domain performance? Such evidence is important for establishing whether the proposed method actually addresses the problem identified in the Introduction.

Q2. Necessity of the two-stage whitening framework and methodological novelty (Clarification and supporting evidence)

The proposed WBN and WCL components are based on existing whitening techniques. What is the specific methodological contribution beyond combining these components within AdaContrast, particularly in relation to prior whitening-based domain adaptation approaches such as DWT (Roy et al., 2019)? Furthermore, the paper argues that feature-level whitening alone is insufficient to regulate global representation geometry. What evidence supports this claim, and why is an additional whitening-based contrastive objective necessary? Although Table 3 demonstrates modest improvements from combining the components, it does not directly establish their claimed complementary mechanisms.

Q3. Preservation of semantic information and limitations of whitening (Supporting evidence)

The paper assumes that whitening suppresses undesirable domain-specific correlations while preserving semantic relationships. However, feature covariance can also reflect class-discriminative structure. How do the authors justify applying whitening without distinguishing between domain-specific and semantically useful correlations? Is there evidence that the proposed whitening operations preserve class separability and improve representation geometry rather than merely changing the feature statistics?

Q4. Consistency between the TTA setting and the evaluation protocol (Clarification)

The paper defines TTA as adaptation during inference and describes an online setting in which target data arrive as a stream. However, Section 3.4 states that the model is trained on target data for 15 epochs. Does this involve repeatedly accessing the same target samples, and when are predictions evaluated relative to adaptation? How is this protocol consistent with online TTA, and are all compared methods evaluated under equivalent target-data access and adaptation budgets? Additionally, does replacing the original BN layers with WBN require retraining the source model, or can the method be directly applied to an existing pretrained source model?

Q5. Inconsistencies and missing details in the implementation (Clarification)

The paper describes ZCA whitening in the Introduction and Conclusion, whereas Section 2.2.2 states that the whitening transformation is computed using Cholesky decomposition following Roy et al. (2019). Which whitening transformation is actually implemented? Since these transformations are not identical, this distinction is important for understanding and reproducing the proposed method. Furthermore, Eq. (12) does not clearly specify how the WCL objective is incorporated into AdaContrast. Does WCL replace the original contrastive loss or introduce an additional objective, and what are the corresponding optimization settings?

Q6. Inconsistencies between reported results and performance claims (Clarification)

The paper claims consistent performance improvements, but Table 4 shows noticeable degradation under several corruption settings, particularly for CoTTA and MEMO. Moreover, the average accuracies described in Section 3.5 do not appear to match the values presented in Table 4. For example, the arithmetic means of the 12 reported MEMO results are approximately 47.4% without whitening and 43.8% with whitening, whereas the text reports an improvement from 43.2% to 44.2%. Can the authors clarify how these averages were calculated and explain these discrepancies? Additionally, what exactly does "+ Whitening" represent for each baseline, and why are only 12 of the 15 corruption types reported?

Q7. Fairness and completeness of baseline comparisons (Clarification)

The main comparisons on VisDA-C and DomainNet-126 largely rely on relatively older methods, despite the availability of more recent SFDA and TTA approaches, including methods discussed in the Introduction. How were the baseline results obtained, and were all methods evaluated under comparable architectures, adaptation protocols, and computational budgets? Can the authors clarify why more recent and directly relevant methods were excluded? This is particularly important given the paper's claims of state-of-the-art performance.

Q8. Statistical reliability and robustness of the reported improvements (Supporting evidence)

The reported improvements over AdaContrast are relatively modest (+1.0 percentage point on VisDA-C and +1.4 percentage points on DomainNet-126), yet the paper provides no information about repeated runs, random seeds, or performance variance. Were the experiments repeated with different seeds, and if so, what are the standard deviations or confidence intervals? Additionally, how sensitive is the method to batch size and covariance estimation settings, particularly given the instability of high-dimensional whitening with limited target samples? These details are important for determining whether the reported gains are reliable and practically meaningful.

Reviewer_bhY8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Two-Stage Isotropic Whitening for source-free test-time adaptation, integrating Whitening Batch Normalization and Contrastive Learning into the AdaContrast framework. The method aims to mitigate domain-specific feature correlations and improve representation geometry, reporting performance improvements across several benchmarks.

Critical strengths

(1) The paper addresses the practically important problem of robust source-free TTA under distribution shifts.

(2) The paper evaluates whitening across multiple benchmarks to examine the contributions of WBN and WCL.

Critical weaknesses

(1) Insufficient justification of the core motivation. The central hypothesis that domain-dependent feature correlations introduce redundant information and compromise robustness under distribution shifts is not adequately substantiated. The paper lacks empirical evidence demonstrating how feature covariance structures vary across domains or whether such correlations are responsible for performance degradation. Further analysis is needed to establish a convincing connection between the proposed motivation and the effectiveness of whitening.

(2) Limited methodological novelty and unclear formulation. Both WBN and WCL are largely derived from existing techniques, and the methodological contribution beyond their integration appears relatively incremental. Furthermore, Eq.12 does not clearly specify how WCL interacts with the original AdaContrast objectives, particularly whether it replaces or supplements the existing contrastive loss.

(3) Insufficient clarity in the experimental methodology. Several important implementation details remain unclear. In particular, the authors should clarify whether WBN is introduced during source training or exclusively during target adaptation, and which model parameters are updated. Moreover, although the method is described as supporting online TTA, the reported 15-epoch target training raises questions about the actual adaptation protocol and whether target samples are revisited. In addition, Table 4 omits three of the 15 corruption types without explanation and does not specify the backbone architectures or report the average performance discussed in the text.

(4) Limited experimental validation. The improvements over AdaContrast are relatively modest, while important analyses of computational overhead, sensitivity to batch size, and statistical variability are missing. These evaluations would help establish the robustness and practical value of the proposed approach.

Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.