acceptodds
Under review as a conference paper at ICLR 2027

Do Not Cross the Boundary: Within-Class Whitening for Test-Time Adaptation

Abstract

Test-time adaptation (TTA) recalibrates a deployed model under domain shift using only unlabeled test data, and recent methods have substantially improved the robustness of such models. However, most existing TTA methods adjust only class-agnostic statistics and overlook the distortion of the feature distribution within each class. Concretely, domain shift elongates the covariance of each class, but existing TTA methods leave it elongated even after adaptation, causing some samples to cross the decision boundary and thus remain misclassified. To mitigate this, we propose Within-Class Whitening (WCW), a plug-in method that whitens the penultimate features with the class-averaged within-class covariance, without backpropagation or pre-computed source statistics. We analyze the family of affine feature transformations and show that WCW makes the within-class covariance isotropic, which lowers an upper bound on the probability of crossing the decision boundary under domain shift. We also identify that restricting the whitening scope to the within-class component avoids the implicit injection of a uniform class prior caused by the rearrangement of the class means. Extensive experiments on various datasets, backbones, and adaptation scenarios show that WCW consistently improves representative parameter-updating TTA methods, outperforms existing TTA plug-ins in most cases, and substantially stabilizes the adaptation procedure.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-time Adaptation, Feature Adjustment, Within-class Whitening

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_K4fKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem that existing test-time adaptation (TTA) methods adjust only class-agnostic statistics and leave the within-class covariance distorted under domain shift. The contributions consist of three parts. (1) A spectral-flatness measurement reveals that the within-class covariance remains far from isotropic even after adaptation by existing methods, which causes samples to cross decision boundaries and remain misclassified. (2) The paper proves that, among affine feature transformations, within-class whitening is the unique optimal choice, and that whitening only the within-class component avoids the implicit injection of a uniform class prior caused by total-covariance whitening. (3) It proposes WCW, a plug-in method that estimates the within-class covariance online in closed form from the test stream, requiring neither backpropagation nor source statistics. Experiments show that WCW consistently improves existing TTA methods on three ImageNet benchmarks, outperforms existing plug-ins in most cases, and substantially stabilizes the adaptation process.

Critical strengths
  1. The problem identification is precise, and the spectral-flatness measurement directly shows that existing methods leave the within-class distortion unaddressed, providing a solid motivation for the work.
  2. Every proposition is verified by a dedicated experiment, and the uniqueness result, the uniform-prior analysis, and the oracle comparison together form a complete chain of evidence.
Critical weaknesses
  1. The evaluation is limited to three ImageNet variants, and validation on other benchmarks or domains is missing, so the generalization of the conclusions remains unclear.
  2. The within-class covariance is estimated from pseudo-labels and can therefore be contaminated when pseudo-label quality is poor, but this robustness is verified only in the mild scenario, leaving the behavior under severe shifts unstudied.
Questions for authors
  1. When the pseudo-label error rate is high, how much do the contaminated moment estimates affect the whitening operator, and are the warm-up and periodic refresh sufficient protection against such contamination?
  2. The throughput drops by about 7–10%. How does this overhead compare with that of other plug-ins?
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

W1. Evaluation on other benchmarks

We are grateful for your insightful suggestion, and evaluated WCW on DomainNet-126 over all 12 ordered transfers among its four domains (Clipart, Painting, Real, and Sketch).

Table R1. Top-1 accuracy (%) on DomainNet-126, averaged over the 12 transfers.

BackboneBaselineNo Adapt.+TCA+NEO+MixTTA+WCW
ResNet-50 GNSource60.52 ± 0.0061.89 ± 0.0061.99 ± 0.01—63.64 ± 0.02
SAR60.68 ± 0.0162.10 ± 0.0062.24 ± 0.0360.70 ± 0.0163.85 ± 0.03
DeYO61.68 ± 0.0462.96 ± 0.0463.04 ± 0.0261.80 ± 0.0564.38 ± 0.05
ReCAP60.91 ± 0.0762.25 ± 0.0862.34 ± 0.0660.84 ± 0.0564.08 ± 0.06
ViT-B/16Source72.50 ± 0.0074.13 ± 0.0073.50 ± 0.01—74.37 ± 0.01
SAR70.89 ± 0.0272.90 ± 0.0173.42 ± 0.0271.35 ± 0.0174.38 ± 0.02
DeYO73.45 ± 0.1174.30 ± 0.0974.41 ± 0.0372.27 ± 0.0474.86 ± 0.03
ReCAP73.32 ± 0.0474.13 ± 0.0374.44 ± 0.0771.95 ± 0.0674.75 ± 0.05

As Table R1 shows, WCW improves every baseline on both backbones and achieves the highest average accuracy among the plug-ins in every row.

Furthermore, the results in measurement R2 of the response to Reviewer aWPS show that WCW can consistently improve accuracy on CLIP as well, across eight datasets.

W2, Q1. Effect of noisy pseudo-labels

Thank you for raising this point. We repeated the comparison of Appendix F under the three wild scenarios.

Table R2. Retained gain ratio.

BackboneMethodMildLabel shiftsBatch size 1Mixed shifts
ViT-B/16Source93%94%94%95%
ViT-B/16SAR93%94%91%87%
ViT-B/16DeYO89%85%91%88%
ViT-B/16ReCAP81%81%90%85%
ResNet-50 GNSource91%90%91%90%
ResNet-50 GNSAR96%97%98%96%
ResNet-50 GNDeYO96%97%99%96%
ResNet-50 GNReCAP96%97%99%91%

As Table R2 shows, pseudo-label WCW retains 81–99% of the gain obtained with ground-truth labels even under the wild scenarios. Notably, although the pseudo-labels of the frozen source model are poor (31.6% / 31.4% on ViT / ResNet, reported in the main paper), it still retains 90–95% of that gain. These results indicate that WCW is robust to pseudo-label contamination regardless of the adaptation scenario.

In addition, we compare pseudo-label and ground-truth WCW with varying warm-up lengths and refresh periods.

Table R3. Comparison with varying warm-up lengths and refresh periods.

Hyper-parameterValueViT pseudoViT GTViT RetainedRN pseudoRN GTRN Retained
Warm-up24 (default)65.9566.2186%51.6351.9396%
4865.8065.9889%51.6351.9296%
9665.5565.7288%51.0251.2796%
19265.2965.4189%49.5249.7295%
Refresh period165.9667.0659%51.7953.8576%
565.9666.2983%51.6552.1793%
10 (default)65.9566.2186%51.6351.9396%
5065.8365.8996%51.3651.5098%
  • The warm-up length does not change the effect of contamination. Extending it from 24 to 192 batches lowers the accuracy of the pseudo-label and the ground-truth runs by the same amount, and the retained gain stays flat at 86–89% (ViT-B/16) and 95–96% (ResNet-50 GN).
  • The effect of label noise depends on the refresh period. With ground-truth labels, refreshing the statistics every batch (period = 1) gives the best accuracy, whereas with pseudo-labels the accuracy is nearly identical between periods 1 and 10.

In summary, the periodic refresh is the safeguard that matters. With a period of 10 batches, pseudo-label contamination costs only 0.26–0.30 points.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q2. Computational overhead compared with other plug-ins

We appreciate your practical question and measured the throughput of all plug-ins under the protocol of Appendix D. We re-measured all configurations in a single session, so the numbers differ slightly from those in Table 5 of Appendix D.

Table R4. Throughput (images/s) of the plug-ins.

BackboneMethodBaseline+WCW+TCA+NEO+MixTTA
ViT-B/16SAR97.196.9 (−0.2%)96.3 (−0.8%)97.2 (+0.1%)94.8 (−2.4%)
ViT-B/16DeYO130.1129.4 (−0.5%)129.9 (−0.2%)129.9 (−0.2%)127.0 (−2.4%)
ViT-B/16ReCAP129.1128.3 (−0.6%)128.7 (−0.3%)129.0 (−0.1%)125.9 (−2.5%)
ResNet-50 GNSAR238.3220.7 (−7.4%)236.8 (−0.6%)236.8 (−0.6%)134.9 (−43.4%)
ResNet-50 GNDeYO330.8297.2 (−10.2%)327.2 (−1.1%)326.0 (−1.5%)193.3 (−41.6%)
ResNet-50 GNReCAP323.9292.4 (−9.7%)322.3 (−0.5%)320.7 (−1.0%)191.4 (−40.9%)

On ViT-B/16, WCW reduces the throughput by less than 1%, comparable to TCA and NEO and less than MixTTA (2.4–2.5%). On ResNet-50 GN, WCW reduces the throughput by 7.4–10.2%. This is more than TCA and NEO (0.5–1.5%), but about four times less than MixTTA (40.9–43.4%).

The main reason for the larger overhead on ResNet-50 GN is its higher feature dimension. Accumulating the second-moment sum and applying T^\hat{T} cost O(BD2)O(BD^2) per batch, and refreshing T^\hat{T} requires a symmetric eigendecomposition of a D×DD \times D matrix, which costs O(D3)O(D^3). Since the penultimate dimension is 2048 for ResNet-50 GN and 768 for ViT-B/16, these costs are about 7.11× and 18.96× larger, respectively.

Table R5. Throughput (images/s) and accuracy of WCW with varying refresh periods.

Backbone1510 (default)50200
ViT-B/16Accuracy (%)65.96 ± 0.0565.96 ± 0.0365.95 ± 0.0365.83 ± 0.0565.14 ± 0.07
Throughput (change vs DeYO)122.7 (−6.3%)128.9 (−1.5%)130.3 (−0.5%)131.1 (+0.1%)131.2 (+0.2%)
ResNet-50 GNAccuracy (%)51.79 ± 0.0051.65 ± 0.0251.63 ± 0.0251.36 ± 0.0748.53 ± 0.10
Throughput (change vs DeYO)167.7 (−49.5%)275.6 (−17.1%)299.8 (−9.8%)322.5 (−2.9%)328.6 (−1.1%)

To reduce this cost, we measured throughput and accuracy with longer refresh periods (Table R5). With a refresh period of 50 batches, the accuracy of DeYO+WCW stays within 0.27 points of the default, while the throughput reduction on ResNet-50 GN decreases from 9.8% to 2.9%. Although an excessively long refresh period causes a non-negligible accuracy drop (to 48.53 on ResNet-50 GN with a period of 200 batches), these observations suggest that the computational overhead of WCW can be traded off against a small loss in accuracy.

Thanks to the author for the reply. My concerns have basically been resolved, and I will maintain my original positive score.

Thank you very much for your thoughtful reviews and for taking the time! We believe that the discussion and the additional experiments helped us clarify the contributions of our work. We greatly appreciate your recognition of our work.

Best regards, The Authors

Reviewer_ivwrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper considers test-time adaptation (TTA) and addresses the problem that existing TTA methods leave the within-class covariance largely uncontrolled, so that a portion of the target samples may cross the decision boundary and be misclassified. The authors propose Within-Class Whitening (WCW), a plug-in module that whitens the penultimate features using the class-averaged within-class covariance estimated online from the predictions of the model. WCW makes the class-averaged variance equal to one in every direction and uses symmetric whitening to minimize changes to the within-class residuals. The method does not require backpropagation, offline access to source data, or source statistics. Experimental results across ImageNet-C, ImageNet-R, and ImageNet-Sketch show that WCW improves the performance and stability of multiple TTA methods across different architectures and test-time scenarios.

Critical strengths
  1. The paper provides a clear theoretical motivation for WCW. Proposition 1 bounds the probability of boundary crossing using the variance along the decision-boundary direction and the distance to the boundary. Propositions 2 and 3 further show that within-class whitening makes the class-averaged variance equal to one in every direction and that symmetric whitening minimizes changes to the within-class residuals.

  2. The paper provides broad and strong empirical support for WCW as a plug-in module. WCW is evaluated with multiple TTA methods and architectures across ImageNet-C, ImageNet-R, and ImageNet-Sketch. The results generally show improved performance and adaptation stability, while WCW does not require backpropagation, offline access to source data, or source statistics.

Critical weaknesses

The empirical validation of the proposed mechanism is limited. Figure 2 shows that spectral flatness decreases for many classes under domain shift, and Figure 4 demonstrates that WCW reduces boundary crossing for ResNet-50-GN with DeYO under zoom blur. However, low spectral flatness does not necessarily imply boundary crossing, because the elongated directions may not align with the relevant decision-boundary directions. Analyzing the relationship among spectral flatness, measured boundary crossing, and accuracy improvement across different settings, such as severity levels, corruption types, architectures, and TTA methods, would provide stronger support for the proposed mechanism.

Questions for authors
  1. Could the authors provide a broader analysis of boundary crossing? Analyzing the relationship among spectral flatness, measured boundary crossing, and accuracy improvement across different severity levels, corruption types, architectures, and TTA methods would provide stronger support for the proposed mechanism.

  2. Could the authors analyze whether covariance elongation occurs along the relevant decision-boundary directions? Such an analysis would help clarify when reduced spectral flatness actually leads to boundary crossing.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

1. Per-corruption analysis

We appreciate your constructive suggestion. Following the comments, we measured the changes in class-averaged spectral flatness (Δf\Delta f), class-averaged boundary crossing probability (Δp\Delta p), and top-1 accuracy (Δacc\Delta\text{acc}) before and after applying WCW, and computed the Pearson correlation coefficient between Δf\Delta f and Δp\Delta p (rfpr_{fp}) and between Δp\Delta p and Δacc\Delta\text{acc} (rpar_{pa}) across the 15 corruption types. All runs use a single seed, so a few Δacc\Delta\text{acc} cells differ from the 3-seed means reported in the main paper.

Table R1. Comparison of Δf\Delta f / Δp\Delta p / Δacc\Delta\text{acc} per corruption.

BackboneMethodGauss.ShotImpul.Defoc.GlassMotionZoomSnowFrostFogBrit.Contr.ElasticPixel.JPEGrfpr_{fp}rpar_{pa}
ViT-B/16SAR+0.160 / −0.053 / +3.9+0.163 / −0.043 / +4.2+0.160 / −0.051 / +4.0+0.116 / −0.037 / +2.8+0.126 / −0.069 / +4.4+0.128 / −0.043 / +3.5+0.140 / −0.070 / +5.4+0.117 / −0.030 / +4.9+0.135 / −0.095 / +6.4+0.113 / −0.045 / +2.6+0.083 / −0.017 / +1.4+0.108 / −0.028 / +1.8+0.124 / −0.082 / +6.7+0.107 / −0.027 / +2.2+0.114 / −0.036 / +2.5−0.48−0.87
ViT-B/16DeYO+0.135 / −0.017 / +2.0+0.135 / −0.013 / +2.1+0.134 / −0.013 / +1.9+0.116 / −0.011 / +1.3+0.116 / −0.019 / +1.6+0.113 / −0.013 / +1.7+0.148 / −0.082 / +10.7+0.104 / −0.006 / +1.4+0.108 / −0.018 / +1.2+0.097 / −0.007 / +0.5+0.077 / −0.005 / +0.2+0.110 / −0.010 / +0.8+0.103 / −0.013 / +1.4+0.092 / −0.009 / +0.7+0.099 / −0.008 / +0.8−0.63−0.98
ViT-B/16ReCAP+0.135 / −0.017 / +2.0+0.135 / −0.014 / +2.0+0.133 / −0.013 / +1.7+0.114 / −0.008 / +1.0+0.115 / −0.019 / +1.5+0.113 / −0.013 / +1.4+0.123 / −0.039 / +4.1+0.105 / −0.007 / +1.2+0.110 / −0.019 / +1.1+0.099 / −0.009 / +0.5+0.078 / −0.005 / +0.1+0.108 / −0.009 / +0.7+0.104 / −0.015 / +1.4+0.092 / −0.010 / +0.5+0.100 / −0.009 / +0.7−0.47−0.90
ResNet-50 GNSAR+0.136 / −0.109 / +6.8+0.131 / −0.104 / +6.6+0.133 / −0.096 / +6.2+0.143 / −0.236 / +13.4+0.141 / −0.261 / +13.4+0.146 / −0.165 / +8.5+0.126 / −0.189 / +10.5+0.108 / −0.153 / +8.9+0.126 / −0.094 / +7.9+0.110 / −0.138 / +9.4+0.095 / −0.017 / +1.5+0.118 / −0.085 / +6.6+0.189 / −0.469 / +27.6+0.103 / −0.063 / +6.9+0.114 / −0.044 / +3.3−0.86−0.98
ResNet-50 GNDeYO+0.110 / −0.038 / +3.3+0.107 / −0.031 / +3.5+0.109 / −0.037 / +3.4+0.130 / −0.214 / +14.6+0.139 / −0.097 / +12.0+0.107 / −0.084 / +6.7+0.101 / −0.135 / +10.7+0.097 / −0.069 / +4.7+0.113 / −0.072 / +5.0+0.100 / −0.044 / +4.4+0.093 / −0.008 / +0.1+0.109 / −0.063 / +4.6+0.096 / −0.138 / +11.4+0.105 / −0.052 / +5.0+0.110 / −0.052 / +3.2−0.43−0.94
ResNet-50 GNReCAP+0.109 / −0.031 / +2.5+0.106 / −0.024 / +2.7+0.107 / −0.025 / +2.5+0.126 / −0.187 / +11.6+0.132 / −0.098 / +10.6+0.103 / −0.064 / +5.1+0.099 / −0.091 / +7.7+0.098 / −0.053 / +3.3+0.110 / −0.059 / +3.8+0.100 / −0.031 / +3.5+0.093 / −0.007 / −0.1+0.108 / −0.055 / +3.8+0.132 / −0.301 / +29.6+0.103 / −0.047 / +4.3+0.109 / −0.048 / +2.7−0.77−0.97

Table R1 reports the per-corruption comparison at severity 5, and three observations follow.

  • WCW increases the flatness and reduces the crossing probability in all 90 cells, and improves the accuracy in 89 of them (the only exception is ReCAP on brightness with ResNet-50 GN, −0.1).
  • Δf\Delta f and Δp\Delta p are negatively correlated in every row. rfpr_{fp} lies between −0.43 and −0.86 in all rows.
  • Δp\Delta p and Δacc\Delta\text{acc} are tightly coupled. rpar_{pa} lies between −0.87 and −0.98 in all rows.

Taken together, the corruption-level correlation analysis supports the described mechanism of Sec. 5.4: WCW flattens the within-class covariance, the flatter covariance crosses the boundaries less often, and the reduced crossing translates into the accuracy gain.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

2. Per-severity analysis

We repeated the correlation analysis across severity levels 1 to 5. The Pearson correlation coefficients rfpr_{fp} and rpar_{pa} are computed across the 5 severity levels.

Table R2. Comparison of Δf\Delta f / Δp\Delta p / Δacc\Delta\text{acc} per severity level, averaged over 15 corruptions.

BackboneMethodSev. 1Sev. 2Sev. 3Sev. 4Sev. 5rfpr_{fp}rpar_{pa}
ViT-B/16SAR+0.088 / −0.018 / +1.3+0.097 / −0.024 / +1.7+0.103 / −0.027 / +2.1+0.113 / −0.034 / +2.6+0.126 / −0.048 / +3.8−0.99−1.00
ViT-B/16DeYO+0.080 / −0.005 / +0.1+0.087 / −0.007 / +0.4+0.091 / −0.007 / +0.5+0.099 / −0.010 / +0.8+0.113 / −0.016 / +1.9−0.98−1.00
ViT-B/16ReCAP+0.080 / −0.005 / +0.1+0.087 / −0.007 / +0.3+0.091 / −0.008 / +0.5+0.100 / −0.011 / +0.7+0.111 / −0.014 / +1.3−1.00−1.00
ResNet-50 GNSAR+0.101 / −0.018 / +1.3+0.106 / −0.032 / +2.5+0.110 / −0.050 / +4.2+0.116 / −0.079 / +6.2+0.128 / −0.148 / +9.2−0.99−0.98
ResNet-50 GNDeYO+0.096 / −0.010 / +0.4+0.098 / −0.020 / +1.4+0.101 / −0.029 / +2.3+0.104 / −0.046 / +4.0+0.108 / −0.076 / +6.2−0.99−1.00
ResNet-50 GNReCAP+0.096 / −0.009 / +0.3+0.099 / −0.017 / +1.0+0.100 / −0.025 / +1.9+0.103 / −0.037 / +3.1+0.109 / −0.075 / +6.2−0.99−1.00

Table R2 reports the per-severity comparison, with every cell averaged over the 15 corruptions and the correlations computed across the five severity levels.

  • The tendency of Table R1 holds at every severity: Δf>0\Delta f > 0, Δp<0\Delta p < 0 and Δacc>0\Delta\text{acc} > 0 in all 30 cells.
  • In all six rows Δf\Delta f, Δp\Delta p and Δacc\Delta\text{acc} grow monotonically from severity 1 to 5.
  • Pearson correlations are rfpr_{fp} = −0.98 to −1.00 and rpar_{pa} = −0.98 to −1.00.

These observations indicate that the effect of whitening on crossing and accuracy scales with the severity of the shift: the stronger the shift, the more the within-class flattening translates into fewer boundary crossings and a larger accuracy gain, consistently for both architectures and all three TTA methods.

3. Elongation along the relevant decision-boundary directions

We appreciate your insightful question. To investigate the elongation along the relevant decision boundary, we measured the following standard deviation ratio kik_i for each class ii:

ki:=σnσavg,k_i := \frac{\sigma_n}{\sigma_{\text{avg}}},

where σn\sigma_n denotes the feature standard deviation along the normal of its relevant decision boundary (the boundary with the class it is most often confused with), and σavg=tr(Σi)/D\sigma_{\text{avg}}=\sqrt{\text{tr}(\Sigma_i)/D} denotes the axis-averaged standard deviation where DD is the feature dimension. A ratio larger than 1 means that the class is elongated toward the relevant decision boundary.

Table R3. Comparison of the class-average of standard-deviation ratio kik_i.

BackboneSourceSource+WCWSARSAR+WCWDeYODeYO+WCWReCAPReCAP+WCW
ViT-B/163.863.37 (−0.49)4.433.74 (−0.69)4.473.79 (−0.68)4.433.78 (−0.65)
ResNet-50 GN3.953.47 (−0.48)4.093.58 (−0.51)3.993.50 (−0.49)3.943.49 (−0.45)

Table R3 reports kik_i averaged over the 1,000 classes and the 15 corruptions of ImageNet-C at severity 5, and suggests the following:

  • Without adaptation, the ratio is far above 1 (3.86 for ViT-B/16 and 3.95 for ResNet-50 GN). That is, the features of each class spread toward the relevant decision boundary about four times more than along an average direction.
  • Parameter-updating TTA does not reduce this elongation. The ratio rises to 4.43–4.47 on ViT-B/16 and stays at 3.94–4.09 on ResNet-50 GN.
  • Adding WCW lowers the ratio for all three methods on both backbones to below the source level.

These observations are consistent with Figures 1 and 2 of the main paper: the elongation that WCW reduces lies along the directions that cross the relevant decision boundary.

Thank you for the additional experiments. The results substantially address my concerns, and I will maintain my positive rating.

We greatly appreciate your thoughtful review. Your insightful feedback has helped us clarify our contributions. Again, we are grateful for your careful evaluation and recognition of our work.

Best regards,

The authors

Reviewer_aWPSOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WCW estimates a pooled within-class covariance from online features and predicted classes, then applies regularized whitening, global centering, and feature-norm restoration before the classifier. It needs neither source statistics nor backpropagation for its own transform, and can accompany parameter-updating adaptation. The paper motivates this choice through directional variance, boundary-crossing bounds, and class-prior effects. Experiments report consistent matched improvements across two backbones and several online scenarios. The empirical contribution is stronger than the general theoretical interpretation of reduced crossing risk and preserved prediction priors.

Critical strengths
  • The matched practical improvements are broad and often substantial. Table 6, improve every displayed SAR, DeYO, and ReCAP base-method mean across both backbones and six settings. For example, ResNet SAR under label shift rises from 36.0 to 49.7. WCW leads the ViT plug-in comparison in 22 of 24 cells. The reported three-seed variability is small after WCW. These results support useful stabilization in the tested settings; they do not mean every plug-in or individual corruption comparison is won.

  • Ablations address the main empirical design choices. Table 2, separates centering from whitening, showing that whitening adds value. Tables 3 and 7 compare source recoloring and true-label versus pseudo-label statistics. Pseudo-labels retain about 81-96% of the true-label accuracy gain in the tested mild setting. This supports practical tolerance of noisy assignments. It does not establish exact recovery of true covariance or the same relation in every scenario.

Critical weaknesses
  • The argument for reduced boundary crossing needs to account for changes in class margins. Proposition 1 (p. 4) bounds crossing using both directional variance and distance from the class mean to the boundary. Proposition 2 (p. 5) establishes isotropy of the pooled within-class covariance, but does not establish a decrease in each class's variance-to-margin ratio. Whitening also moves the class means relative to the fixed classifier. Figure 4 (p. 8) supports the proposed effect in the tested zoom-blur setting. The broader implication in the abstract and the discussion after Proposition 2 needs additional conditions or narrower wording.

  • Proposition 4 does not establish the claimed effect on predicted class frequencies. The proposition and its proof (pp. 5 and 14-15) give an upper bound on each transformed class mean's squared norm for positive alpha. The following paragraph goes further, arguing that frequent classes move closer to decision boundaries and become underpredicted. That step requires a connection to the classifier's weights and biases; the norm bound alone does not provide it. The authors should distinguish that observed effect from what Proposition 4 proves, including the claim that within-class whitening avoids uniform-prior injection.

Questions for authors
  1. What conditions control transformed class margins and individual directional variances so that the crossing bound decreases? Can the general claim be separated from the measured Figure 4 example?
  2. Which head assumptions connect Proposition 4's norm bound to predicted class frequencies? Please distinguish the proved result from the observed ImageNet-R prior effect.
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

W1, Q1. Conditions for a lower crossing bound; separating the claim from Fig. 4

We thank the reviewer for the insightful diagnosis and agree with it. Proposition 2 shows that a transformation with TΣwithinT⊤=IT\Sigma_{\text{within}}T^\top = I makes the class-averaged variance Ey∼π[σy2(n)]\mathbb{E}_{y\sim\pi}[\sigma_y^2(n)] equal in every direction; it does not establish a decrease of each pair's σij2/dij2\sigma_{ij}^2/d_{ij}^2, and, as the reviewer points out, the transformation also moves the class means.

The connection to individual directional variances. The elongation of each class covariance can be viewed as a distortion shared by all classes, combined with a class-specific shape. Since WCW applies a single transformation to all classes, it removes the shared distortion, which is what Proposition 2 guarantees, but it cannot completely equalize the class-specific anisotropy. No single transformation in Eq. (3) can whiten all class covariances simultaneously, because a transformation that makes one class isotropic generally distorts another. In other words, this residual anisotropy reflects a limitation of the affine transformation family. Combined with the discussion below, this separates what WCW controls (the shared distortion) from what no affine map can control (the class-specific anisotropy).

However, WCW can still increase the margin-to-variance ratio. Although the class-specific variances are beyond the reach of a single affine transformation, the margin measured in units of the class-averaged variance can be maximized under one condition: self-duality of neural collapse (NC3; Papyan et al., 2020), i.e., each class's classifier weight aligns with its globally centered feature mean, wc∥μc−μw_c \parallel \mu_c - \mu.

How WCW increases the margin-to-variance ratio under self-duality. Let dijd_{ij} denote the distance from the feature mean of class ii to the decision boundary between classes ii and jj, and djid_{ji} the distance from the mean of class jj to the same boundary. Since the boundary is shared, moving it toward one class enlarges one distance at the expense of the other, so we consider the two ratios together.

Following the notation in Appendix A, let nij=wi−wj∥wi−wj∥n_{ij} = \frac{w_i - w_j}{\lVert w_i - w_j \rVert}. For a transformation z′=γT(z−m)z' = \gamma T(z - m) of Eq. (3), the two distances are measured along nijn_{ij}, so their sum is γnij⊤T(μi−μj)\gamma n_{ij}^\top T(\mu_i - \mu_j), and the class-averaged variance along nijn_{ij} is σˉij2=γ2nij⊤TΣwithinT⊤nij\bar\sigma_{ij}^2 = \gamma^2 n_{ij}^\top T\Sigma_{\text{within}} T^\top n_{ij} (Eq. (4)). Hence the sum of margin-to-variance ratio is given as

dijσˉij+djiσˉij=nij⊤T(μi−μj)nij⊤TΣwithinT⊤nij=cos⁡θ∣∣Σwithin−1/2(μi−μj)∣∣,\frac{d_{ij}}{\bar\sigma_{ij}} + \frac{d_{ji}}{\bar\sigma_{ij}} = \frac{n_{ij}^\top T(\mu_i - \mu_j)}{\sqrt{n_{ij}^\top T\Sigma_{\text{within}}T^\top n_{ij}}} = \cos\theta || \Sigma_{\text{within}}^{-1/2}(\mu_i - \mu_j) ||,

where θ\theta is the angle between Σwithin1/2T⊤nij\Sigma_{\text{within}}^{1/2}T^\top n_{ij} and Σwithin−1/2(μi−μj)\Sigma_{\text{within}}^{-1/2}(\mu_i - \mu_j). The norm on the right-hand side does not depend on the transformation TT, so a transformation changes the sum only through cos⁡θ\cos\theta.

With WCW (T=Σwithin−1/2T = \Sigma_{\text{within}}^{-1/2}), θ\theta becomes the angle between wi−wjw_i - w_j and the displacement of the transformed class means, μi′−μj′=γΣwithin−1/2(μi−μj)\mu'_i - \mu'_j = \gamma \Sigma_{\text{within}}^{-1/2}(\mu_i - \mu_j). Under the self-duality, cos⁡θ\cos\theta equals 1 and the sum attains its largest value for every pair of classes. In contrast, without WCW, cos⁡θ=1\cos\theta = 1 requires nij∥Σwithin−1(μi−μj)n_{ij} \parallel \Sigma_{\text{within}}^{-1}(\mu_i - \mu_j). A classification head satisfying self-duality points along the difference of the class means, so it meets this requirement only when μi−μj\mu_i - \mu_j is an eigenvector of Σwithin\Sigma_{\text{within}}. Since this does not hold in general, the sum stays below its largest value.

Empirically, self-duality holds approximately in classifiers trained with the cross-entropy loss (Papyan et al., 2020; Measurement R1), so WCW naturally increases the sum of the two margin-to-variance ratios toward its largest value, which no other transformation of the form of Eq. (3) can exceed. When self-duality fails, cos⁡θ\cos\theta can be small or even negative, and WCW carries no accuracy gain. Measurement R2 examines this case.

  • Reference: Papyan, Vardan, Xiao Y. Han, and David L. Donoho. "Prevalence of neural collapse during the terminal phase of deep learning training." Proceedings of the National Academy of Sciences 117.40 (2020): 24652-24663.
AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

Measurement R1 — self-duality under TTA and WCW. To verify that self-duality indeed holds after adaptation and after applying WCW, we measure the cosine similarity between each class mean and the corresponding weight vector of the classification head, averaged over classes, for ViT-B/16 on ImageNet-C (severity 5, averaged over 15 corruptions). For the ++WCW rows, the class means are computed on the transformed features z′z'.

Table R1. Class-averaged cosine similarity between class means and head weight vectors (ViT-B/16, ImageNet-C, severity 5).

Methodcosine
Source model, source data0.784
SAR0.736
SAR + WCW0.737
DeYO0.777
DeYO + WCW0.748
ReCAP0.774
ReCAP + WCW0.746

As Table R1 reports, the source model attains a cosine similarity of 0.784 on the source data, and the models adapted with SAR, DeYO, or ReCAP remain close to this level on the target data (0.736–0.777). Adding WCW changes the cosine similarity by only +0.001+0.001 to −0.029-0.029. These observations suggest that (1) models adapted with existing TTA methods approximately satisfy self-duality, and (2) WCW largely preserves it.

Measurement R2 — when self-duality does not hold (CLIP). Neural collapse is observed in classifiers trained with the cross-entropy loss, and it need not hold in vision–language models such as CLIP. In zero-shot CLIP, the head (text embeddings) and the image features come from different encoders, so the image class means are not aligned with the head. To restore the alignment, as pre-processing, we apply an orthogonal Procrustes rotation R⋆=arg⁡min⁡R⊤R=I∑c∥Rμc−wc∥2R^\star = \arg\min_{R^\top R = I} \sum_c \lVert R\mu_c - w_c \rVert^2, which aligns the class means with the text embeddings as closely as a single rotation allows.

Table R2. Top-1 accuracy (%) of zero-shot CLIP, WCW, and Rotation + WCW.

DatasetZero-shotWCWRotation + WCW
ImageNet-A47.7946.6147.92
ImageNet-R73.9874.9376.95
ImageNet-Sketch46.1045.9847.56
Flowers10267.3663.0570.36
Food10185.2482.6685.73
DTD44.3941.9045.80
EuroSAT42.2241.6950.06
UCF10165.1964.7667.59
Mean (8)59.0357.7061.50

As reported in Table R2, without the rotation, WCW brings no improvement and even decreases the mean accuracy. However, once the rotation restores self-duality, WCW improves over zero-shot CLIP on all eight datasets (61.50, +2.47 on average). These results indicate that the benefit of WCW depends on self-duality: it vanishes when self-duality does not hold and reappears once self-duality is restored. In other words, the benefit of WCW is not specific to classifiers trained with the cross-entropy loss, such as ViT and ResNet; it generalizes to a vision–language model once self-duality is restored. These observations also suggest that WCW, combined with a pre-processing that restores self-duality, is a promising direction for test-time adaptation of vision–language models.

Revision. Following the reviewer’s comments, we have also revised the abstract and the discussion after Proposition 2 (Sec. 4.1) with narrower wording. All changes are listed below and highlighted in blue in the updated manuscript.

  • Abstract, Lines 18–19: We softened the causal claim, stating that the elongated within-class covariance can cause boundary crossing.
  • Abstract, Lines 22–25: We clarified that WCW is designed to make the class-averaged within-class covariance isotropic, and that we verify its reduction of both the upper bound and the measured crossing probability.
  • Sec. 4.1, Lines 222–224: We added a sentence that clarifies how making the class-averaged covariance isotropic relates to the individual directional variances of each class.
AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

W2, Q2. What Proposition 4 proves, and the head assumption behind the prior effect

Thank your for your feedback. Proposition 4 bounds only the norm of each transformed class mean and does not involve the geometry of the decision boundary, which is directly determined by the classification head.

What Proposition 4 proves. For α>0\alpha > 0, each transformed class mean satisfies ∥Tα(μc−μ)∥2<1/(απc)\lVert T_\alpha(\mu_c - \mu) \rVert^2 < 1/(\alpha\pi_c). As α\alpha grows, every class mean is confined closer to the global mean, and the confinement is tighter for a frequent class (large πc\pi_c).

The connection to the head. Let wiw_i and bib_i denote the weight vector and the bias of the head for class ii. For a transformed feature z′=Tα(z−μ)z' = T_\alpha(z - \mu), the head prefers class ii over class jj when

(wi−wj)⊤z′+bi−bj>0.(w_i - w_j)^\top z' + b_i - b_j > 0.

Combined with Eq. (3) and averaged over the features zz of class ii, it splits into two terms:

(wi−wj)⊤Tα(μi−μ)+(bi−bj).(w_i - w_j)^\top T_\alpha(\mu_i - \mu) + (b_i - b_j).

For any weights, the Cauchy–Schwarz inequality and Proposition 4 yield

∣(wi−wj)⊤Tα(μi−μ)∣≤∥wi−wj∥∥Tα(μi−μ)∥<∥wi−wj∥απi,\left| (w_i - w_j)^\top T_\alpha(\mu_i - \mu) \right| \le \lVert w_i - w_j \rVert \lVert T_\alpha(\mu_i - \mu) \rVert < \frac{\lVert w_i - w_j \rVert}{\sqrt{\alpha\pi_i}},

so the first term of class ii is capped by (απi)−1/2(\alpha\pi_i)^{-1/2}, whatever the weights are. As α\alpha grows, the averaged decision falls back on the biases, and it does so first for frequent classes. Hence, frequent classes are the first to lose the evidence that discriminates them, and their predictions are the first to be confused, which tends to flatten the predicted class frequencies. This is what we refer to as the “uniform-prior injection” in Sec. 4.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.