- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem that existing test-time adaptation (TTA) methods adjust only class-agnostic statistics and leave the within-class covariance distorted under domain shift. The contributions consist of three parts. (1) A spectral-flatness measurement reveals that the within-class covariance remains far from isotropic even after adaptation by existing methods, which causes samples to cross decision boundaries and remain misclassified. (2) The paper proves that, among affine feature transformations, within-class whitening is the unique optimal choice, and that whitening only the within-class component avoids the implicit injection of a uniform class prior caused by total-covariance whitening. (3) It proposes WCW, a plug-in method that estimates the within-class covariance online in closed form from the test stream, requiring neither backpropagation nor source statistics. Experiments show that WCW consistently improves existing TTA methods on three ImageNet benchmarks, outperforms existing plug-ins in most cases, and substantially stabilizes the adaptation process.
Critical strengths
- The problem identification is precise, and the spectral-flatness measurement directly shows that existing methods leave the within-class distortion unaddressed, providing a solid motivation for the work.
- Every proposition is verified by a dedicated experiment, and the uniqueness result, the uniform-prior analysis, and the oracle comparison together form a complete chain of evidence.
Critical weaknesses
- The evaluation is limited to three ImageNet variants, and validation on other benchmarks or domains is missing, so the generalization of the conclusions remains unclear.
- The within-class covariance is estimated from pseudo-labels and can therefore be contaminated when pseudo-label quality is poor, but this robustness is verified only in the mild scenario, leaving the behavior under severe shifts unstudied.
Questions for authors
- When the pseudo-label error rate is high, how much do the contaminated moment estimates affect the whitening operator, and are the warm-up and periodic refresh sufficient protection against such contamination?
- The throughput drops by about 7–10%. How does this overhead compare with that of other plug-ins?
4 comments
Rebuttal by Authors (2/2)
Q2. Computational overhead compared with other plug-ins
We appreciate your practical question and measured the throughput of all plug-ins under the protocol of Appendix D. We re-measured all configurations in a single session, so the numbers differ slightly from those in Table 5 of Appendix D.
Table R4. Throughput (images/s) of the plug-ins.
| Backbone | Method | Baseline | +WCW | +TCA | +NEO | +MixTTA |
|---|---|---|---|---|---|---|
| ViT-B/16 | SAR | 97.1 | 96.9 (−0.2%) | 96.3 (−0.8%) | 97.2 (+0.1%) | 94.8 (−2.4%) |
| ViT-B/16 | DeYO | 130.1 | 129.4 (−0.5%) | 129.9 (−0.2%) | 129.9 (−0.2%) | 127.0 (−2.4%) |
| ViT-B/16 | ReCAP | 129.1 | 128.3 (−0.6%) | 128.7 (−0.3%) | 129.0 (−0.1%) | 125.9 (−2.5%) |
| ResNet-50 GN | SAR | 238.3 | 220.7 (−7.4%) | 236.8 (−0.6%) | 236.8 (−0.6%) | 134.9 (−43.4%) |
| ResNet-50 GN | DeYO | 330.8 | 297.2 (−10.2%) | 327.2 (−1.1%) | 326.0 (−1.5%) | 193.3 (−41.6%) |
| ResNet-50 GN | ReCAP | 323.9 | 292.4 (−9.7%) | 322.3 (−0.5%) | 320.7 (−1.0%) | 191.4 (−40.9%) |
On ViT-B/16, WCW reduces the throughput by less than 1%, comparable to TCA and NEO and less than MixTTA (2.4–2.5%). On ResNet-50 GN, WCW reduces the throughput by 7.4–10.2%. This is more than TCA and NEO (0.5–1.5%), but about four times less than MixTTA (40.9–43.4%).
The main reason for the larger overhead on ResNet-50 GN is its higher feature dimension. Accumulating the second-moment sum and applying cost per batch, and refreshing requires a symmetric eigendecomposition of a matrix, which costs . Since the penultimate dimension is 2048 for ResNet-50 GN and 768 for ViT-B/16, these costs are about 7.11× and 18.96× larger, respectively.
Table R5. Throughput (images/s) and accuracy of WCW with varying refresh periods.
| Backbone | 1 | 5 | 10 (default) | 50 | 200 | |
|---|---|---|---|---|---|---|
| ViT-B/16 | Accuracy (%) | 65.96 ± 0.05 | 65.96 ± 0.03 | 65.95 ± 0.03 | 65.83 ± 0.05 | 65.14 ± 0.07 |
| Throughput (change vs DeYO) | 122.7 (−6.3%) | 128.9 (−1.5%) | 130.3 (−0.5%) | 131.1 (+0.1%) | 131.2 (+0.2%) | |
| ResNet-50 GN | Accuracy (%) | 51.79 ± 0.00 | 51.65 ± 0.02 | 51.63 ± 0.02 | 51.36 ± 0.07 | 48.53 ± 0.10 |
| Throughput (change vs DeYO) | 167.7 (−49.5%) | 275.6 (−17.1%) | 299.8 (−9.8%) | 322.5 (−2.9%) | 328.6 (−1.1%) |
To reduce this cost, we measured throughput and accuracy with longer refresh periods (Table R5). With a refresh period of 50 batches, the accuracy of DeYO+WCW stays within 0.27 points of the default, while the throughput reduction on ResNet-50 GN decreases from 9.8% to 2.9%. Although an excessively long refresh period causes a non-negligible accuracy drop (to 48.53 on ResNet-50 GN with a period of 200 batches), these observations suggest that the computational overhead of WCW can be traded off against a small loss in accuracy.
Thanks to the author for the reply. My concerns have basically been resolved, and I will maintain my original positive score.
Thank you very much for your thoughtful reviews and for taking the time! We believe that the discussion and the additional experiments helped us clarify the contributions of our work. We greatly appreciate your recognition of our work.
Best regards, The Authors
Rebuttal by Authors (1/2)
W1. Evaluation on other benchmarks
We are grateful for your insightful suggestion, and evaluated WCW on DomainNet-126 over all 12 ordered transfers among its four domains (Clipart, Painting, Real, and Sketch).
Table R1. Top-1 accuracy (%) on DomainNet-126, averaged over the 12 transfers.
As Table R1 shows, WCW improves every baseline on both backbones and achieves the highest average accuracy among the plug-ins in every row.
Furthermore, the results in measurement R2 of the response to Reviewer aWPS show that WCW can consistently improve accuracy on CLIP as well, across eight datasets.
W2, Q1. Effect of noisy pseudo-labels
Thank you for raising this point. We repeated the comparison of Appendix F under the three wild scenarios.
Table R2. Retained gain ratio.
As Table R2 shows, pseudo-label WCW retains 81–99% of the gain obtained with ground-truth labels even under the wild scenarios. Notably, although the pseudo-labels of the frozen source model are poor (31.6% / 31.4% on ViT / ResNet, reported in the main paper), it still retains 90–95% of that gain. These results indicate that WCW is robust to pseudo-label contamination regardless of the adaptation scenario.
In addition, we compare pseudo-label and ground-truth WCW with varying warm-up lengths and refresh periods.
Table R3. Comparison with varying warm-up lengths and refresh periods.
In summary, the periodic refresh is the safeguard that matters. With a period of 10 batches, pseudo-label contamination costs only 0.26–0.30 points.