- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates statistics-based, backpropagation-free test-time adaptation (TTA) for vision-language models (VLMs). The authors identify and empirically characterize two core vulnerabilities in online prototype-accumulation TTA methods: (1) prediction-biased accumulation, where systematic zero-shot class preferences repeatedly misallocate test features into over-responsive class prototypes; and (2) classifier homogenization, where additive aggregation of features causes shared, stream-wide feature coordinates (the global stream mean) to dominate class prototypes, collapsing inter-class angular separation even under oracle ground-truth assignment. To resolve these issues, the authors propose Unbiased Test-Time Adaptation (UTA), featuring two parameter-free mechanisms: History-Guided Allocation Debiasing (HAD), which tracks cumulative class responses and applies a standardized -compressed log-penalty to debias allocation logits; and Activity-Conditioned Classifier Construction (ACC), which tracks stream-wide squared coordinate activity and rescales class prototypes via prior to normalization. Evaluated across 15 image benchmarks (cross-dataset and ImageNet OOD) and 4 corrupted 3D point-cloud benchmarks (ULIP), UTA achieves state-of-the-art backpropagation-free accuracy while retaining 95% of zero-shot CLIP inference speed.
Critical strengths
- Insightful Problem Diagnosis and Elegant, Parameter-Free Formulation: The identification of classifier homogenization under oracle allocation (Eq. 8 and Fig. 3b) is a genuinely illuminating contribution to the VLM TTA literature. Showing that additive feature accumulation drives inter-class cosine similarity up to 0.81 even with 100% ground-truth labels cleanly explains why existing prototype-caching methods plateau. The resulting ACC solution-maximizing prototype alignment under an activity-weighted Mahalanobis norm , which reduces to element-wise division by accumulated second moments and reduces the pre-normalization common-energy fraction via —is mathematically principled, completely free of tuned hyperparameters, and incurs virtually zero compute/memory overhead.
- Exceptional Empirical Rigor, Efficiency, and Plug-in Versatility: The experimental validation is comprehensive. Beyond beating both backpropagation-based (e.g., TPT, DLAE) and backpropagation-free (e.g., TDA, PTA, SCA) baselines on cross-dataset, OOD, and 3D point-cloud benchmarks at 181.1 FPS (+2.0 MiB over zero-shot CLIP), the authors provide exhaustive mechanism diagnostics (Appendix C.2), prompt sensitivity tests (Appendix C.3), plug-in transferability across 7 baselines (Table 10), and non-IID Dirichlet stream stress tests (Table 13).
Critical weaknesses
- Tension Between History-Guided Debiasing and True Label Imbalance / Non-IID Streams: HAD penalizes classes with high cumulative historical probability , implicitly nudging the marginal allocation distribution toward uniformity across classes. While the authors show that zero-shot bias exists on uniform datasets (Fig. 4) and provide non-IID Dirichlet stream experiments in Table 13, there is a noticeable performance drop as the stream becomes temporally correlated (UTA drops from 70.66% on IID to 67.22% on Separate, where classes arrive sequentially). In a sequential or heavily long-tailed target stream, a surge of predictions for class is the ground truth, yet HAD will actively penalize class 's logit via , violating the sufficient non-overcorrection condition (Eq. 6) and forcing misallocations to inactive classes.
- Unweighted Logit Fusion and Unbounded State Accumulation: In Eq. (11), both and accumulate linearly with without exponential discounting, and Eq. (13) fuses logits via a static unweighted sum despite and both being unscaled cosine similarities. Because and , for large , , whereas for a rare class , grows much slower than . Furthermore, because is row-normalized to unit length, has the exact same norm scale as from step , meaning a single noisy visual allocation immediately carries 50% of the weight of the text classifier.
Questions for authors
- Clarification on Unweighted Logit Fusion (Eq. 13): Since both and are unit -normalized vectors, and have identical magnitude scales. Why is a 1:1 unweighted sum optimal, especially early in the stream when is dominated by (effectively doubling the text logit modulated by )?
- Clarification on Long-Tailed / Class-Imbalanced Target Distributions: While Table 13 evaluates temporal non-IID ordering (Dirichlet ordering over the same dataset), the overall marginal class distribution of most evaluated benchmarks remains relatively balanced (as shown by Data Imbalance in Table 7, except Caltech101 and Flowers102). Noticeably, on Caltech101 (where Data Imbalance TV is highest at 27.85%), adding HAD to zero-shot CLIP decreases assignment accuracy from 93.91% to 92.58% (Table 7), and adding +UTA to BCA, SCA, PTA, TDA, ADAPT, and DPE consistently hurts Caltech101 accuracy (Table 10). Can you discuss this failure mode and whether an EMA window on or a gentler saturation than could prevent HAD from over-penalizing genuinely frequent head classes?
6 comments
Response to Reviewer LQVH (2/3)
W2/Q1: Unweighted Logit Fusion and Unbounded State Accumulation.
Response: We thank the reviewer for raising this point. With and , row normalization gives , and therefore the initial visual logits are approximately equal to the textual logits. At , the allocation is still determined by the textual logits, so the visual-state update is text-routed; because this update is added to the text-initialized state , remains a text-anchored perturbation rather than an independent visual classifier. Thus, in the early stages, the visual doesn't actually introduce much noise that differs from the text.
We compare fixed visual-fusion weights and two adaptive fusion controllers, where . The fixed-fusion variants use constant , the confidence controller uses when text confidence is at least and otherwise, and the temporal controller uses .
| Method | Flowers | DTD | Pets | Cars | UCF | Cal. | Food | SUN | Air. | Euro. | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Text-only | 71.46 | 46.04 | 90.87 | 67.83 | 72.48 | 92.58 | 86.38 | 68.80 | 26.82 | 60.69 | 68.39 |
| Fixed fusion () | 73.61 | 48.64 | 91.99 | 69.72 | 74.91 | 94.16 | 87.25 | 70.66 | 26.91 | 65.04 | 70.29 |
| Fixed fusion () | 73.97 | 49.11 | 91.93 | 70.25 | 75.26 | 94.24 | 87.19 | 71.02 | 27.09 | 65.42 | 70.55 |
| Fixed fusion () | 74.14 | 49.00 | 91.93 | 70.44 | 75.55 | 94.20 | 87.01 | 71.00 | 27.09 | 65.37 | 70.57 |
| Fixed fusion () | 74.42 | 49.23 | 92.07 | 70.51 | 75.68 | 94.24 | 86.79 | 70.93 | 27.03 | 65.64 | 70.66 |
| Fixed fusion () | 74.67 | 48.88 | 91.88 | 70.46 | 75.50 | 94.32 | 86.55 | 70.86 | 27.06 | 65.62 | 70.58 |
| Fixed fusion () | 74.75 | 48.94 | 91.77 | 70.49 | 75.50 | 94.28 | 86.33 | 70.81 | 27.15 | 65.53 | 70.55 |
| Confidence threshold () | 74.26 | 49.41 | 92.07 | 70.60 | 75.50 | 94.20 | 87.11 | 71.01 | 27.06 | 65.83 | 70.70 |
| Temporal gate ( | 74.42 | 49.23 | 92.07 | 70.54 | 75.63 | 94.16 | 86.80 | 70.93 | 27.09 | 65.64 | 70.65 |
The results show that different fusion weights may benefit different datasets, while we retain 1:1 fusion to let UTA serve as a parameter-free method, even though it may not be the best for an individual dataset.
Thank you again for this insightful comment. We have added this discussion in lines 305--307, as well as the ablation in Appendix C.10, Tab.18.
Response to Reviewer LQVH (3/3)
Q2: A boundary case of HAD.
Response: We sincerely appreciate this insightful suggestion! It helps us to better analyse the sources of benefits for HAD. As discussed in W1, class imbalance is not the only the root cause of HAD performance degradation. Flowers102 (20.11% TV) and the long-tailed stream (10:1) are also imbalanced, but HAD can still be effective.
Since Caltech101 is a relatively easy dataset for CLIP, with high zero-shot accuracy (93.91%) and limited systematic bias, its history-future correlation is only , suggesting that its historical responses provide limited information. This special case may limit the scope of methods for TTA design, especially for statistics-based TTA. Nevertheless, the complete UTA still improves over CLIP on Caltech101 (93.91% 94.24%), suggesting that ACC can partially mitigate this problem. Moreover, HAD tends to be more effective on challenging datasets, consistent with TTA's original goal.
We also evaluate the similar existing prediction calibration method DLAE on this failed mode, with both DLAE and HAD using the same original single-view setting for a fair comparison (64-view/augmentation in the officially released code). The results are reported below.
| Method | Flowers | DTD | Pets | Cars | UCF | Cal. | Food | SUN | Air. | Euro. | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| CLIP$^\dagger$ | 70.73 | 44.09 | 89.07 | 66.21 | 67.54 | 93.91 | 85.90 | 66.31 | 24.72 | 48.21 | 65.67 |
| DLAE logit adjustment$^\dagger$ (1-view) | 70.73 | 45.98 | 90.32 | 67.23 | 69.36 | 91.24 | 86.22 | 66.85 | 26.13 | 56.96 | 67.10 |
| HAD | 71.46 | 46.04 | 90.87 | 67.83 | 72.48 | 92.58 | 86.38 | 68.80 | 26.82 | 60.69 | 68.39 |
We find that DLAE causes a larger decrease on Caltech101 (93.91% 91.24%). This result further validates our analysis: Statistics-based correction is limited if there is little inter-class bias information to exploit. We have added this result in Appendix C.13, Tab.21. We have also updated the code repository linked in the abstract for reproducing this comparison, including DLAE.
Following your insightful suggestion, we evaluate two lightweight strategies for this case. First, we replace the cumulative historical update with EMA: $M_t = (1-\alpha) M_{t-1} + \alpha q_t $, with = 0.05. Second, we keep the cumulative historical state unchanged and vary the correction strength in the corrected logit . is computed as final fused accuracy minus the zero-shot CLIP accuracy.
| History update | Allocation acc. (%) | Final fused acc. (%) | |
|---|---|---|---|
| ZS allocation (CLIP) | 93.91 | 94.40 | +0.49 |
| Cumulative HAD | 92.58 | 94.24 | +0.32 |
| EMA-HAD () | 93.06 | 94.28 | +0.37 |
| Correction strength | Allocation acc. (%) | Final fused acc. (%) | |
|---|---|---|---|
| 0 (no correction) | 93.91 | 94.40 | +0.49 |
| 0.20 | 93.75 | 94.44 | +0.53 |
| 0.50 | 93.51 | 94.36 | +0.45 |
| 0.75 | 93.35 | 94.60 | +0.69 |
| 1.00 (defult) | 92.58 | 94.24 | +0.32 |
The results show that both EMA and correction-strength attenuation can alleviate this issue. We have added this discussion in Limitations in Sec. 5, as well as the results in Appendix C.9, Tab.16-17. Thank you again for this insightful suggestion!
Response to Reviewer LQVH (1/3)
Thank you for the insightful feedback and positive comments! We summarize our responses to each point below. We hope our responses adequately address your concerns.
W1: Tension Between HAD and True Label Imbalance / Non-IID Streams.
Response: Thank you for this insightful comment! We first emphasize that strongly non-IID streams expose a fundamental limitation of statistics-based test-time adaptation. These methods summarize the stream into fixed-size statistics without retaining individual instances or using multi-view augmentation, and therefore require sufficient cross-class observations to estimate useful class-wise information. For example, when a stream contains only one class, they can not obtain additional useful information. This also explains the substantial degradation observed for statistics-based methods.
Nevertheless, HAD can still retain part of its benefit as additional classes arrive and the historical state becomes more informative (Table below). ACC remains useful because it is class-independent. However, the practical HAD correction may be poorly estimated during the early class-concentrated stage in non-IID. Accordingly, Eq.(6) serves as an idealized motivation for HAD rather than a guarantee for the practical adaptive correction for each sample.
Moreover, this limitation is distinct from class imbalance itself. In a randomly interleaved long-tailed stream, class frequencies are imbalanced, but different classes still appear throughout the stream, allowing HAD to accumulate cross-class responses and effectively improve performance (please see the Table 4 in revised manuscript or Q2 in reviewer fLAu). This suggests that HAD depends more on the availability of measurable cross-class response bias. That is also why we formulate the bias as a cross-class quantity and use the two-class case as the minimal illustrative example.
To characterize this limitation more systematically, we further evaluate BCA, SCA, and PTA, together with the isolated CLIP+HAD and CLIP+DLAE logit-adjustment variants[1], under the same non-IID protocols. For a fair comparison, all methods use the same prompt templates and data augmentation. The CLIP+DLAE variant is a statistics-based mechanism for logit adjustment (similar to HAD), which serves as an additional baseline.
In the Separate setting, BCA, SCA, and CLIP+DLAE all hurt the zero-shot CLIP performance, whereas PTA and our designs remain effective. This consistent degradation across multiple statistics-based methods provides evidence of a shared limitation, rather than a failure specific to HAD. We have added a disscussion of this issue (Sec.5 Limitations), as well as the full results in Appendix C.14, Tab.22.
Thank you again for this insightful comment. This discussion has identified an important limitation of statistics-based methods and highlighted a valuable direction for future work!
[1] Dynamic Logits Adjustment and Exploration for Test-Time Adaptation in Vision Language Models. CVPR 2026.