acceptodds
Under review as a conference paper at ICLR 2027

Towards Unbiased Test-Time Adaptation for Vision–Language Models

Abstract

Statistics-based test-time adaptation (TTA) provides an efficient way to adapt vision-language models (VLMs) by summarizing test streams into fixed-size class statistics without backpropagation or instance storage. However, there are two limitations in this paradigm. First, persistent class-wise prediction preferences can repeatedly misallocate incoming features, leading to prediction-biased accumulation in the class states. Second, additive feature aggregation can retain class-shared components, leading to classifier homogenization even under correct class allocation. To address these issues, we propose _**U**nbiased **T**est-Time **A**daptation_ (**UTA**), which jointly regulates class allocation and classifier construction. Specifically, to mitigate prediction-biased accumulation, a *History-Guided Allocation Debiasing* mechanism is designed to regulate class allocation using historical class responses, thereby reducing allocation errors in the accumulated class states. Furthermore, to alleviate classifier homogenization, we introduce an *Activity-Conditioned Classifier Construction* strategy to reweight accumulated class states according to stream-wide coordinate activity, which limits shared-component dominance and preserves class-specific distinctions. Extensive experiments demonstrate strong adaptation performance across multiple settings while preserving 95% of zero-shot CLIP throughput on large-scale ImageNet-1K. Moreover, the proposed components can serve as lightweight plug-ins for existing statistics-based TTA methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-Time Adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LQVHOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates statistics-based, backpropagation-free test-time adaptation (TTA) for vision-language models (VLMs). The authors identify and empirically characterize two core vulnerabilities in online prototype-accumulation TTA methods: (1) prediction-biased accumulation, where systematic zero-shot class preferences repeatedly misallocate test features into over-responsive class prototypes; and (2) classifier homogenization, where additive aggregation of features causes shared, stream-wide feature coordinates (the global stream mean) to dominate class prototypes, collapsing inter-class angular separation even under oracle ground-truth assignment. To resolve these issues, the authors propose Unbiased Test-Time Adaptation (UTA), featuring two parameter-free mechanisms: History-Guided Allocation Debiasing (HAD), which tracks cumulative class responses and applies a standardized arsinh⁡\operatorname{arsinh}-compressed log-penalty to debias allocation logits; and Activity-Conditioned Classifier Construction (ACC), which tracks stream-wide squared coordinate activity Gt=1+∑i=1tfi⊙fiG_t = \mathbf{1} + \sum_{i=1}^t f_i \odot f_i and rescales class prototypes via Pt,c⊘GtP_{t,c} \oslash G_t prior to ℓ2\ell_2 normalization. Evaluated across 15 image benchmarks (cross-dataset and ImageNet OOD) and 4 corrupted 3D point-cloud benchmarks (ULIP), UTA achieves state-of-the-art backpropagation-free accuracy while retaining 95% of zero-shot CLIP inference speed.

Critical strengths
  • Insightful Problem Diagnosis and Elegant, Parameter-Free Formulation: The identification of classifier homogenization under oracle allocation (Eq. 8 and Fig. 3b) is a genuinely illuminating contribution to the VLM TTA literature. Showing that additive feature accumulation drives inter-class cosine similarity up to 0.81 even with 100% ground-truth labels cleanly explains why existing prototype-caching methods plateau. The resulting ACC solution-maximizing prototype alignment arg max⁡∥w∥2=1s⊤w∥w∥Gt\operatorname*{arg\,max}_{\|w\|_2=1} \frac{s^\top w}{\|w\|_{G_t}} under an activity-weighted Mahalanobis norm ∥w∥Gt2=∑d=1DGt,dwd2\|w\|_{G_t}^2 = \sum_{d=1}^D G_{t,d} w_d^2, which reduces to element-wise division by accumulated second moments Pt,c⊘GtP_{t,c} \oslash G_t and reduces the pre-normalization common-energy fraction via ρcom′−ρcom=Cov⁡ν(h,r)Eν[h]\rho_{\text{com}}' - \rho_{\text{com}} = \frac{\operatorname{Cov}_\nu(h, r)}{\mathbb{E}_\nu[h]}—is mathematically principled, completely free of tuned hyperparameters, and incurs virtually zero compute/memory overhead.
  • Exceptional Empirical Rigor, Efficiency, and Plug-in Versatility: The experimental validation is comprehensive. Beyond beating both backpropagation-based (e.g., TPT, DLAE) and backpropagation-free (e.g., TDA, PTA, SCA) baselines on cross-dataset, OOD, and 3D point-cloud benchmarks at 181.1 FPS (+2.0 MiB over zero-shot CLIP), the authors provide exhaustive mechanism diagnostics (Appendix C.2), prompt sensitivity tests (Appendix C.3), plug-in transferability across 7 baselines (Table 10), and non-IID Dirichlet stream stress tests (Table 13).
Critical weaknesses
  • Tension Between History-Guided Debiasing and True Label Imbalance / Non-IID Streams: HAD penalizes classes with high cumulative historical probability Mt−1,cM_{t-1,c}, implicitly nudging the marginal allocation distribution toward uniformity across classes. While the authors show that zero-shot bias exists on uniform datasets (Fig. 4) and provide non-IID Dirichlet stream experiments in Table 13, there is a noticeable performance drop as the stream becomes temporally correlated (UTA drops from 70.66% on IID to 67.22% on Separate, where classes arrive sequentially). In a sequential or heavily long-tailed target stream, a surge of predictions for class cc is the ground truth, yet HAD will actively penalize class cc's logit via γt,c\gamma_{t,c}, violating the sufficient non-overcorrection condition 0≤γc−γj≤βc−βj0 \le \gamma_c - \gamma_j \le \beta_c - \beta_j (Eq. 6) and forcing misallocations to inactive classes.
  • Unweighted Logit Fusion and Unbounded State Accumulation: In Eq. (11), both Pt=Pt−1+AtftP_t = P_{t-1} + A_t f_t and Gt=Gt−1+ft⊙ftG_t = G_{t-1} + f_t \odot f_t accumulate linearly with tt without exponential discounting, and Eq. (13) fuses logits via a static unweighted sum ztUTA=zttext+ztvisualz_t^{\text{UTA}} = z_t^{\text{text}} + z_t^{\text{visual}} despite zttext=fte⊤z_t^{\text{text}} = f_t e^\top and ztvisual=ftWt⊤z_t^{\text{visual}} = f_t W_t^\top both being unscaled cosine similarities. Because P0=eP_0 = e and G0=1G_0 = \mathbf{1}, for large tt, Gt=O(t)G_t = \mathcal{O}(t), whereas for a rare class cc, Pt,cP_{t,c} grows much slower than tt. Furthermore, because WtW_t is row-normalized to unit ℓ2\ell_2 length, ztvisualz_t^{\text{visual}} has the exact same norm scale as zttextz_t^{\text{text}} from step t=1t=1, meaning a single noisy visual allocation immediately carries 50% of the weight of the text classifier.
Questions for authors
  1. Clarification on Unweighted Logit Fusion (Eq. 13): Since both ece_c and wt,cw_{t,c} are unit ℓ2\ell_2-normalized vectors, zttext=fte⊤z_t^{\text{text}} = f_t e^\top and ztvisual=ftWt⊤z_t^{\text{visual}} = f_t W_t^\top have identical magnitude scales. Why is a 1:1 unweighted sum zttext+ztvisualz_t^{\text{text}} + z_t^{\text{visual}} optimal, especially early in the stream when PtP_t is dominated by P0=eP_0 = e (effectively doubling the text logit modulated by Gt−1G_t^{-1})?
  2. Clarification on Long-Tailed / Class-Imbalanced Target Distributions: While Table 13 evaluates temporal non-IID ordering (Dirichlet ordering over the same dataset), the overall marginal class distribution of most evaluated benchmarks remains relatively balanced (as shown by Data Imbalance TV⁡(π,u)=12∥π−u∥1\operatorname{TV}(\pi, u) = \frac{1}{2}\|\pi - u\|_1 in Table 7, except Caltech101 and Flowers102). Noticeably, on Caltech101 (where Data Imbalance TV is highest at 27.85%), adding HAD to zero-shot CLIP decreases assignment accuracy from 93.91% to 92.58% (Table 7), and adding +UTA to BCA, SCA, PTA, TDA, ADAPT, and DPE consistently hurts Caltech101 accuracy (Table 10). Can you discuss this failure mode and whether an EMA window on MtM_t or a gentler saturation than arsinh⁡\operatorname{arsinh} could prevent HAD from over-penalizing genuinely frequent head classes?
6 comments
AuthorsOpenReview ↗

Response to Reviewer LQVH (1/3)

Thank you for the insightful feedback and positive comments! We summarize our responses to each point below. We hope our responses adequately address your concerns.

W1: Tension Between HAD and True Label Imbalance / Non-IID Streams.

Response: Thank you for this insightful comment! We first emphasize that strongly non-IID streams expose a fundamental limitation of statistics-based test-time adaptation. These methods summarize the stream into fixed-size statistics without retaining individual instances or using multi-view augmentation, and therefore require sufficient cross-class observations to estimate useful class-wise information. For example, when a stream contains only one class, they can not obtain additional useful information. This also explains the substantial degradation observed for statistics-based methods.

Nevertheless, HAD can still retain part of its benefit as additional classes arrive and the historical state becomes more informative (Table below). ACC remains useful because it is class-independent. However, the practical HAD correction may be poorly estimated during the early class-concentrated stage in non-IID. Accordingly, Eq.(6) serves as an idealized motivation for HAD rather than a guarantee for the practical adaptive correction for each sample.

Moreover, this limitation is distinct from class imbalance itself. In a randomly interleaved long-tailed stream, class frequencies are imbalanced, but different classes still appear throughout the stream, allowing HAD to accumulate cross-class responses and effectively improve performance (please see the Table 4 in revised manuscript or Q2 in reviewer fLAu). This suggests that HAD depends more on the availability of measurable cross-class response bias. That is also why we formulate the bias as a cross-class quantity and use the two-class case as the minimal illustrative example.

To characterize this limitation more systematically, we further evaluate BCA, SCA, and PTA, together with the isolated CLIP+HAD and CLIP+DLAE logit-adjustment variants[1], under the same non-IID protocols. For a fair comparison, all methods use the same prompt templates and data augmentation. The CLIP+DLAE variant is a statistics-based mechanism for logit adjustment (similar to HAD), which serves as an additional baseline.

ScenarioMethodImageNetFlowersDTDPetsCarsUCFCal.FoodSUNAir.Euro.Avg.
SeparateCLIP$^\dagger$66.7570.7343.4489.1065.5867.4993.3185.9162.5524.7848.3865.27
BCA$^\dagger$66.4770.4643.9088.4864.4367.3193.0785.7162.9124.4448.1665.03
SCA$^\dagger$66.0870.2944.4487.1964.1066.9287.8184.8763.6024.3047.2864.26
CLIP+DLAE$^\dagger$66.6868.9041.4687.4363.7666.5492.9583.8262.3823.7640.9963.51
CLIP+HAD67.1071.1144.0988.7365.6368.2693.0285.2263.7325.2649.3165.59
PTA$^\dagger$68.2372.6645.0589.3766.9770.0493.5184.9665.4922.5050.8966.33
UTA68.6074.3746.2290.6168.2470.8293.3083.4266.3125.4450.7267.22

In the Separate setting, BCA, SCA, and CLIP+DLAE all hurt the zero-shot CLIP performance, whereas PTA and our designs remain effective. This consistent degradation across multiple statistics-based methods provides evidence of a shared limitation, rather than a failure specific to HAD. We have added a disscussion of this issue (Sec.5 Limitations), as well as the full results in Appendix C.14, Tab.22.

Thank you again for this insightful comment. This discussion has identified an important limitation of statistics-based methods and highlighted a valuable direction for future work!

[1] Dynamic Logits Adjustment and Exploration for Test-Time Adaptation in Vision Language Models. CVPR 2026.

AuthorsOpenReview ↗

Response to Reviewer LQVH (2/3)

W2/Q1: Unweighted Logit Fusion and Unbounded State Accumulation.

Response: We thank the reviewer for raising this point. With P0=eP_0=e and G0=1G_0=1, row normalization gives W0≈eW_0\approx e, and therefore the initial visual logits are approximately equal to the textual logits. At t=1t=1, the allocation A1A_1 is still determined by the textual logits, so the visual-state update is text-routed; because this update is added to the text-initialized state P0=eP_0=e, W1W_1 remains a text-anchored perturbation rather than an independent visual classifier. Thus, in the early stages, the visual doesn't actually introduce much noise that differs from the text.

We compare fixed visual-fusion weights and two adaptive fusion controllers, where ztUTA=zttext+λtvztvisualz_t^{UTA} = z_t^{text} + \lambda_t^v z_t^{visual}. The fixed-fusion variants use constant λtv\lambda_t^v, the confidence controller uses λtv=0.5\lambda_t^v = 0.5 when text confidence τ=max⁡pt\tau =\max p_t is at least τ=0.6\tau = 0.6 and 11 otherwise, and the temporal controller uses λtv=t/(t+C)\lambda_t^v = t/(t+C).

MethodFlowersDTDPetsCarsUCFCal.FoodSUNAir.Euro.Avg.
Text-only71.4646.0490.8767.8372.4892.5886.3868.8026.8260.6968.39
Fixed fusion (λtv=0.25\lambda_t^v = 0.25)73.6148.6491.9969.7274.9194.1687.2570.6626.9165.0470.29
Fixed fusion (λtv=0.5\lambda_t^v = 0.5)73.9749.1191.9370.2575.2694.2487.1971.0227.0965.4270.55
Fixed fusion (λtv=0.75\lambda_t^v = 0.75)74.1449.0091.9370.4475.5594.2087.0171.0027.0965.3770.57
Fixed fusion (λtv=1\lambda_t^v = 1)74.4249.2392.0770.5175.6894.2486.7970.9327.0365.6470.66
Fixed fusion (λtv=1.5\lambda_t^v = 1.5)74.6748.8891.8870.4675.5094.3286.5570.8627.0665.6270.58
Fixed fusion (λtv=2\lambda_t^v = 2)74.7548.9491.7770.4975.5094.2886.3370.8127.1565.5370.55
Confidence threshold (τ=0.6\tau = 0.6)74.2649.4192.0770.6075.5094.2087.1171.0127.0665.8370.70
Temporal gate (t/(t+C))t/(t+C))74.4249.2392.0770.5475.6394.1686.8070.9327.0965.6470.65

The results show that different fusion weights may benefit different datasets, while we retain 1:1 fusion to let UTA serve as a parameter-free method, even though it may not be the best for an individual dataset.

Thank you again for this insightful comment. We have added this discussion in lines 305--307, as well as the ablation in Appendix C.10, Tab.18.

AuthorsOpenReview ↗

Response to Reviewer LQVH (3/3)

Q2: A boundary case of HAD.

Response: We sincerely appreciate this insightful suggestion! It helps us to better analyse the sources of benefits for HAD. As discussed in W1, class imbalance is not the only the root cause of HAD performance degradation. Flowers102 (20.11% TV) and the long-tailed stream (10:1) are also imbalanced, but HAD can still be effective.

Since Caltech101 is a relatively easy dataset for CLIP, with high zero-shot accuracy (93.91%) and limited systematic bias, its history-future correlation is only ρ=0.05\rho=0.05, suggesting that its historical responses provide limited information. This special case may limit the scope of methods for TTA design, especially for statistics-based TTA. Nevertheless, the complete UTA still improves over CLIP on Caltech101 (93.91% →\rightarrow 94.24%), suggesting that ACC can partially mitigate this problem. Moreover, HAD tends to be more effective on challenging datasets, consistent with TTA's original goal.

We also evaluate the similar existing prediction calibration method DLAE on this failed mode, with both DLAE and HAD using the same original single-view setting for a fair comparison (64-view/augmentation in the officially released code). The results are reported below.

MethodFlowersDTDPetsCarsUCFCal.FoodSUNAir.Euro.Avg.
CLIP$^\dagger$70.7344.0989.0766.2167.5493.9185.9066.3124.7248.2165.67
DLAE logit adjustment$^\dagger$ (1-view)70.7345.9890.3267.2369.3691.2486.2266.8526.1356.9667.10
HAD71.4646.0490.8767.8372.4892.5886.3868.8026.8260.6968.39

We find that DLAE causes a larger decrease on Caltech101 (93.91% →\rightarrow 91.24%). This result further validates our analysis: Statistics-based correction is limited if there is little inter-class bias information to exploit. We have added this result in Appendix C.13, Tab.21. We have also updated the code repository linked in the abstract for reproducing this comparison, including DLAE.

Following your insightful suggestion, we evaluate two lightweight strategies for this case. First, we replace the cumulative historical update with EMA: $M_t = (1-\alpha) M_{t-1} + \alpha q_t $, with α\alpha = 0.05. Second, we keep the cumulative historical state unchanged and vary the correction strength λ\lambda in the corrected logit zttext−λγtz_t^{text} - \lambda \gamma_t. ΔZS\Delta_{ZS} is computed as final fused accuracy minus the zero-shot CLIP accuracy.

History updateAllocation acc. (%)Final fused acc. (%)ΔZS\Delta_{ZS}
ZS allocation (CLIP)93.9194.40+0.49
Cumulative HAD92.5894.24+0.32
EMA-HAD (α=0.05\alpha = 0.05)93.0694.28+0.37
Correction strength λ\lambdaAllocation acc. (%)Final fused acc. (%)ΔZS\Delta_{ZS}
0 (no correction)93.9194.40+0.49
0.2093.7594.44+0.53
0.5093.5194.36+0.45
0.7593.3594.60+0.69
1.00 (defult)92.5894.24+0.32

The results show that both EMA and correction-strength attenuation can alleviate this issue. We have added this discussion in Limitations in Sec. 5, as well as the results in Appendix C.9, Tab.16-17. Thank you again for this insightful suggestion!

Reviewer_KejeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Unbiased Test-Time Adaptation (UTA), a lightweight framework for adapting vision-language models using unlabeled test data. The method aims to address two problems in statistics-based test-time adaptation: prediction-biased accumulation and classifier homogenization. Specifically, the authors introduce History-Guided Allocation Debiasing (HAD) to correct class allocation based on historical prediction responses, and Activity-Conditioned Classifier Construction (ACC) to improve the discrimination of the visual classifier by reweighting accumulated class features. Experimental results on multiple benchmarks demonstrate competitive accuracy with low computational overhead.

Critical strengths
  1. The paper addresses a meaningful problem by investigating prediction bias and classifier homogenization in statistics-based test-time adaptation.
  2. The proposed framework is simple and computationally efficient, requiring no backpropagation or storage of individual test samples.
Critical weaknesses
  1. The technical novelty of the proposed method is somewhat limited. HAD mainly uses historical prediction statistics to adjust class logits, while ACC reweights classifier features according to their accumulated activity. These operations are closely related to existing prediction calibration and feature normalization techniques. The authors should better explain the differences and advantages compared with existing methods.
  2. The theoretical analysis of HAD is not sufficient to fully support the proposed adaptive correction strategy. The theoretical result in Eq. (7) relies on an ideal correction that follows the ordering of the true class-wise bias. However, the actual correction is estimated from historical predictions, and this condition is not guaranteed. The authors should further analyze when the proposed adaptive correction can reduce prediction errors.
  3. The effectiveness of HAD under changing class distributions needs further investigation. HAD suppresses classes with high historical prediction responses. However, these responses may also reflect the actual class distribution rather than prediction bias. Although the paper evaluates non-IID test streams, more experiments under long-tailed class distributions and changing class priors would help demonstrate the robustness of HAD.
  4. The experimental comparisons for ACC could be improved. The paper mainly compares ACC with standard normalization of accumulated class features. However, the proposed activity-based reweighting is related to feature normalization and whitening methods. Comparisons with mean-centering, variance normalization, or other feature calibration strategies would better demonstrate the advantages of ACC.
  5. The performance improvements over strong baselines are relatively modest in some settings. For example, UTA improves the average accuracy over PTA from 69.26% to 70.66% on the cross-dataset benchmark, while some existing methods still achieve better results on individual datasets. More analysis of when and why the proposed method performs better would strengthen the experimental evaluation.
Questions for authors

Please see the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Keje (1/2)

Thank you for the insightful and valuable comments! Below, we present our point-by-point responses, which we hope will address your concerns.

W1/W4: Methodological novelty and component comparisons.

Response: We thank the reviewer for raising this important point. We would like to clarify that UTA's technical novelty lies in addressing two key limitations of statistics-based TTA at negligible additional computational cost. It provides an efficient, backpropagation-free, hyperparameter-free, and readily extensible method.

For HAD, we have added a detailed discussion in Lines 211–214. Specifically, we compare HAD with the recent dynamic logit-adjustment method DLAE[1] (CVPR 26). DLAE uses raw top-1 pseudo-labels to update its per-class statistics and directly adjusts the prediction logits. In contrast, HAD estimates persistent class bias from the full soft prediction vector and uses the resulting correction to regulate the allocation rather than directly calibrate the final prediction logits. It also avoids using noisy raw top-1 pseudo-labels to construct the historical bias state. For a fair comparison, we evaluate both DLAE and HAD using the same single-view setting (64-view/augmentation in the officially released code).

MethodFlowersDTDPetsCarsUCFCal.FoodSUNAir.Euro.Avg.
CLIP$^\dagger$70.7344.0989.0766.2167.5493.9185.9066.3124.7248.2165.67
DLAE logit adjustment$^\dagger$ (1-view)70.7345.9890.3267.2369.3691.2486.2266.8526.1356.9667.10
HAD71.4646.0490.8767.8372.4892.5886.3868.8026.8260.6968.39

We have provided these results in Appendix C.13, Tab.21. Notably, HAD outperforms DLAE logit adjustment on all ten datasets, improving the average accuracy from 67.10% to 68.39%. We have also updated the code repository linked in the abstract for reproducing this comparison, including DLAE.

For ACC, we have added ablations against alternative feature-normalization strategies under the same accumulated class states and class assignments. Below are the comparison results. We have added this comparison in Sec.4.2, Tab.7; full per-dataset results are also provided in Appendix C.11 Tab.19.

MethodVisual-onlyFinal prediction
Standard row normalization72.8975.11
Coordinate RMS74.1576.22
Mean-centering28.3534.68
Variance normalization45.7952.54
ACC75.8477.60

Thank you again for your suggestion! These additional discussions and ablations further help us to clarify both the methodological differences and the advantages of HAD and ACC over existing prediction-calibration and feature-normalization approaches.

[1] Dynamic Logits Adjustment and Exploration for Test-Time Adaptation in Vision Language Models. CVPR 2026.

AuthorsOpenReview ↗

Response to Reviewer Keje (2/2)

W2: Theoretical justification.

Response: We thank the reviewer for pointing out this limitation. We agree that Eq.(7) should be interpreted as an idealized sufficient condition rather than a per-sample guarantee for the implemented adaptive correction. Specifically, Eq. (7) assumes that the correction satisfies the offset-orderingcondition in Eq. (6). In the actual method, the correction is computed from historical responses as γt=ωtδt,\boldsymbol{\gamma}_t = \omega_t \boldsymbol{\delta}_t, and the ordering condition is not necessarily satisfied for every sample or every dataset. We therefore do not claim that HAD is guaranteed to reduce the prediction error of every individual sample.

Together with the empirical improvements, this theoretical result explains why regulation is beneficial when historical response imbalance aligns with persistent class-wise bias. This is supported by Fig.3(a), which measures whether historical class responses predict future class-wise prediction excess (ρ=0.76\rho=0.76 across the ten datasets) . Consistent with this alignment, HAD improves the zero-shot-only allocation accuracy from 65.67% to 68.39% on average. Thus, Eq.(7) provides the theoretical motivation for HAD, while the effectiveness of the adaptive correction over complete test streams is evaluated empirically.

Thank you again for this insightful suggestion. It helped us clarify why and under what conditions HAD can improve allocation. We have added a discussion of HAD in Section 5 (Limitations).

W3: Robustness to long-tailed class distributions.

Response: Thank you for the insightful comment. To examine the robustness of HAD, we follow the long-tailed stream protocol of L-TTA[2] (ICLR 26) and evaluate CLIP+HAD with head-to-tail ratios of 10:1, 5:1, and 2:1. We find that HAD still improves the ten-dataset average over zero-shot CLIP. Moreover, even without a long-tail-specific design, UTA outperforms L-TTA in both long-tail and normal settings. The Table below compares the two methods.

ScenarioCLIP$^\dagger$L-TTA$^\dagger$CLIP+HADUTA
10:110:165.3367.2765.8967.46
5:15:165.0467.3466.8768.62
2:12:165.0267.8568.0170.01
Normal65.6767.5068.3970.66
MethodBack propagationLong tailed-specificData AugmentationsHyperparametersEfficiency (ImageNet-1K)
L-TTA$^\dagger$✓✓✓ (8-views)✓ (per-dataset)9.8% of CLIP
UTA✗✗✗✗(parameters-free)95% of CLIP

These results demonstrate the robustness and efficiency of UTA under long-tailed distributions. Notably, UTA consistently outperforms the long-tail-specific L-TTA while achieving approximately 9.7× higher throughput, without backpropagation, multi-view augmentation, or any hyperparameters. We have added these long-tailed stream results to the revised manuscript in Sec.4.3. We have also updated the code repository linked in the abstract for reproducing this comparison, including L-TTA.

[2] Long-tailed Test-Time Adaptation for Vision-Language Models. ICLR 2026.

W5: Performance gains and more analysis.

Response: Thank you for pointing this out. On the cross-dataset benchmark, UTA improves the average accuracy over the strongest statistics-based method (PTA) from 69.26% to 70.66% (+1.40%). Moreover, among the statistics-based TTA methods compared in Table 1, UTA achieves the best performance on 8 out of 10 datasets and the second-best on the remaining two.

The overall gains depend on the extent to which prediction-biased allocation and classifier homogenization affect the adaptation process. Moreover, we observe that our UTA framework tends to provide greater benefits on more challenging datasets. This is consistent with the goal of TTA, which aims to improve model performance under distribution shifts. We have added a discussion in Sec.5.

Thank you again for these insightful suggestions!

AuthorsOpenReview ↗

Official Comment by Authors

Dear reviewer Keje,

We sincerely appreciate your thoughtful feedback throughout the review process. Following the initial comments, we have submitted detailed responses to all raised points.

It would be great if the reviewer could kindly check our responses and provide feedback with further questions/concerns (if any). We would be more than happy to address them. Thank you again for your time and support!

Best regards, Paper 12886 Authors

AuthorsOpenReview ↗

Official Comment by Authors

Dear reviewer Keje,

Thank you once again for your thoughtful review and valuable comments! With only 1 day remaining before the end of the Discussion Phase, we would greatly appreciate your feedback on our revised responses. Please let us know whether our clarifications adequately address your concerns. If any issues remain unresolved or if further explanation is needed, we would be grateful for your additional guidance.

Best regards, Paper 12886 Authors

Reviewer_fLAuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes Unbiased Test-Time Adaptation (UTA) for statistics-based test-time adaptation of vision-language models. UTA addresses two issues: prediction-biased accumulation and classifier homogenization. It introduces History-Guided Allocation Debiasing (HAD) to regulate class allocation using historical class responses, and Activity-Conditioned Classifier Construction (ACC) to reduce the influence of highly active feature coordinates when constructing visual classifiers. Experiments on 15 image classification datasets and 4 corrupted point cloud benchmarks show that UTA generally outperforms existing statistics-based TTA methods while maintaining low computational overhead.

Critical strengths

1.The paper identifies two concrete failure modes of statistics-based TTA and provides a clear motivation for the proposed method. 2.UTA is lightweight and does not require backpropagation, instance-level memory, or target annotations.

Critical weaknesses

1.The interpretation of HAD as debiasing requires further clarification. Historical class responses may reflect model bias or true class frequencies. Although Appendix C.8 evaluates temporally class-correlated streams, it does not directly establish whether HAD can distinguish prediction bias from genuine class-prior shifts. 2.The ablations support the contribution of ACC, but do not establish the advantage of its specific activity-based scaling over simpler normalization alternatives. It remains unclear whether comparable improvements could be obtained with a more general coordinate-wise feature normalization.

Questions for authors

1.How does HAD distinguish model prediction bias from genuine class-prior shifts or temporally concentrated test streams? 2.Can HAD hurt performance when the test stream is class-imbalanced but the model itself has no systematic bias? 3.How does ACC compare with simpler alternatives such as RMS normalization? 4.Are the main results stable across multiple random seeds and test-stream permutations? 5. The efficiency results are encouraging. Could the authors also report the FLOPs or briefly break down the computational cost of HAD and ACC?

6 comments
AuthorsOpenReview ↗

Response to Reviewer fLAu (1/3)

Thank you for your insightful comments! In the following, we provide our point-by-point response and hope our response helps address your concerns.

W1/Q1: Can HAD distinguish bias from genuine class-prior shifts?

Response: We thank the reviewer for raising this important concern. Since the HAD operates on observable soft response distributions rather than hard top-1 labels, it does not require an explicit decomposition of response imbalance into intrinsic model preference and genuine target-prior shift. The ρ=0.76\rho=0.76 correlation in Fig.3(a) shows that historical class-wise response preferences are associated with later prediction excess, supporting their use for allocation regulation.

This design is motivated by the fact that, in a label-free setting, top-1 predictions alone cannot reveal whether a class-wise response imbalance is caused by an intrinsic model preference or a genuine target class prior. This ambiguity is particularly relevant in TTA, where zero-shot CLIP achieves only 65.67% accuracy on the cross-dataset benchmark and 57.20% on the OOD average. Moreover, a class that achieves 100% accuracy may still yeild low historical response if its predicted probabilities remain low. Conversely, a persistent class-wise preference represented by βc\beta_c may also produce a high response across samples even when class cc is not always the top-1 prediction. Thus, we choose to maintain a prediction-wise response, rather than the labels.

To analyze the advantages of such design, we compare HAD with the recent logit-adjustment method DLAE[1] (CVPR 26). Unlike HAD, DLAE uses raw top-1 pseudo-labels to update its per-class statistics, assuming these labels can distinguish prediction bias from genuine class-prior shifts. To isolate the effect of the logit-adjustment strategy from the benefit of multi-view augmentation (64-view/augmentation in the officially released code), we evaluate both DLAE and HAD using the same original single-view setting. Below shows the comparison results.

MethodFlowersDTDPetsCarsUCFCal.FoodSUNAir.Euro.Avg.
CLIP$^\dagger$70.7344.0989.0766.2167.5493.9185.9066.3124.7248.2165.67
DLAE logit adjustment$^\dagger$ (1-view)70.7345.9890.3267.2369.3691.2486.2266.8526.1356.9667.10
HAD71.4646.0490.8767.8372.4892.5886.3868.8026.8260.6968.39

Notably, our HAD outperforms DLAE logit adjustment on all ten datasets, improving the average accuracy from 67.10% to 68.39% (1.29 percentage points). This further confirms our design. We also added a disscusion in lines 211-214, and provied these result in Appendix C.13, Tab.21. We have also updated the code repository linked in the abstract for reproducing this comparison, including DLAE.

[1] Dynamic Logits Adjustment and Exploration for Test-Time Adaptation in Vision Language Models. CVPR 2026.

AuthorsOpenReview ↗

Response to Reviewer fLAu (2/3)

Q2: Can HAD hurt performance when the test stream is class-imbalanced but the model itself has no systematic bias?

Response: We thank the reviewer for raising this important concern. The key factor is not class imbalance itself, but whether the model exhibits a persistent bias. If the zero-shot is already highly accurate and there is no systematic bias to correct, correction may indeed hurt performance. Caltech101 in Tab.9 illustrates this boundary case, its CLIP zero-shot already achieves 93.91% accuracy, while the history--future correlation is only ρ\rho = 0.05. It is the only one of the ten datasets on which HAD reduces the allocation accuracy. However, HAD tends to be more effective on challenging datasets, consistent with TTA's original goal. These results also demonstrate a fundamental limitation: statistics-based correction is limited when cross-class bias information is weak. For example, 4/8 existing bp-free methods hurt CLIP zero-shot performance on Caltech101, while others showed limited improvement (Table 1). Importantly, the table in A1 further shows that hard label statistic (DLAE) causes a larger decrease on Caltech101 than HAD (93.91% →\rightarrow 91.24%).

Nevertheless, the complete UTA still improves over CLIP on Caltech101 (93.91% →\rightarrow 94.24%), suggesting that ACC can partially mitigate this problem. In addition, this issue can be simply alleviated by performing an EMA update of the historical state MtM_t and an attenuated correction. We have added this analysis in Appendix C9, Tab.16-17.

To separate the effect of class imbalance from the systematic bias, we follow the long-tailed stream protocol of L-TTA[2] (ICLR 26) and evaluate CLIP+HAD with head-to-tail ratios of 10:1, 5:1, and 2:1. The Tables below compares the two methods.

ScenarioCLIP$^\dagger$L-TTA$^\dagger$CLIP+HADUTA
10:110:165.3367.2765.8967.46
5:15:165.0467.3466.8768.62
2:12:165.0267.8568.0170.01
Normal65.6767.5068.3970.66
MethodBack propagationLong tailed-specificData AugmentationsHyperparametersEfficiency (ImageNet-1K)
L-TTA$^\dagger$✓✓✓ (8-views)✓ (per-dataset)9.8% of CLIP
UTA✗✗✗✗(parameters-free)95% of CLIP

We find that even in the imbalanced test streams, HAD still improves zero-shot CLIP. Moreover, even without a long-tail-specific design, UTA outperforms L-TTA in both long-tail and normal settings. We have added these long-tailed stream results to the revised manuscript in Sec.4.3 Tab.4. We have also updated the code repository linked in the abstract for reproducing this comparison, including L-TTA.

Thank you again for this insightful comment, which helped us clarify the boundary case of HAD and strengthen the analysis of its robustness. We also added a discussion of this failure mode in Section 5 (limitations).

[2] Long-tailed Test-Time Adaptation for Vision-Language Models. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer fLAu (3/3)

W2/Q3: Compared with simpler normalization alternatives

Response: We sincerely appreciate for this insightful suggestion! It motivated us to add ablation comparing ACC with simpler coordinate-wise normalization alternatives. To isolate classifier construction from allocation errors, we added an ACC alternative ablation under matched oracle allocation in Sec.4.2 Tab.7, below shows the results. Full results are also provided in Appendix C.11, Tab.19.

MethodVisual-onlyFinal prediction
Standard row normalization72.8975.11
Coordinate RMS74.1576.22
Mean-centering28.3534.68
Variance normalization45.7952.54
ACC75.8477.60

Among the alternatives, Coordinate RMS is the only one that improves over standard row normalization, providing a substantial gain! ACC achieves the best performance in both settings. These results support our classifier-homogenization analysis and suggest that coordinate activity provides a promising direction for constructing more discriminative classifiers from accumulated class states.

Thank you again for this insightful suggestion!

Q4: Are the results stable across multiple random seeds?

Response: We evaluated UTA on three independently shuffled test streams generated with random seeds 1, 2, and 3 (Appendix C.7, Tab.14). The standard deviation is only 0.04 percentage points, indicating that UTA is robust to random stream permutations.

Q5: Can you report FLOPs or break down the computational cost of HAD and ACC?

Response: Of course! With C=1000C=1000 classes and feature dimension D=512D=512, HAD introduces only O(C)O(C) overhead, whereas ACC requires O(CD)O(CD) operations. Throughput is measured with batch size 1 on a single NVIDIA RTX 5090 GPU.

ComponentState or operationComplexityFLOPs/imageThroughput (FPS)
CLIP ViT-B/16 encoderFrozen image encoder--≈17.6G\approx 17.6\mathrm{G}190.6 (100%)
HADMt∈RCM_t \in \mathbb{R}^{C}O(C)O(C)<0.02M<0.02\mathrm{M}186.4 (98%) (CLIP + HAD)
ACC state updatePt∈RC×D, Gt∈RDP_t \in \mathbb{R}^{C\times D},\ G_t \in \mathbb{R}^{D}O(CD)O(CD)≈1.5\approx 1.5--1.6M1.6\mathrm{M}--
ACC visual readoutWtftW_t f_tO(CD)O(CD)1.024M1.024\mathrm{M}184.2 (97%)(CLIP + ACC)
UTA (all)Mt, Pt, GtM_t,\ P_t,\ G_tO(CD)O(CD)≈2.5\approx 2.5--2.6M2.6\mathrm{M}181.1 (95%)

Theoretically, UTA introduces only about 2.5–2.6 MFLOPs per image, compared with 17.6 GFLOPs for the CLIP encoder, corresponding to only approximately 0.014–0.015% of CLIP’s computation. The reported 95% throughput is an end-to-end measurement. With a more optimized implementation, the throughput could exceed 95% and approach the CLIP baseline.

Thank you for this comment. We have added this result in Appendix C.12, Tab.20.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript and summary of changes

We sincerely thank all reviewers for their time and constructive, encouraging feedback. In response to their comments, we have revised the manuscript and uploaded the updated version. We have also updated the repository linked in the abstract with the code needed to reproduce these additional experiments. For clarity, all updates are highlighted in blue\color{blue}\text{blue}.

Specifically, the main revisions fall into the following five categories:

  • (1) We have added analyses under additional stream settings, including the long-tailed results in Table 4 and the extended non-IID comparison in Appendix C.14. These additions address W3 from Reviewer Keje and W1 from Reviewer LQVH.

  • (2) We have added further analyses of HAD, including a comparison with DLAE (Lines 210--214; Appendix C.13) and a boundary-case analysis (Appendix C.9). We have also discussed the scope and limitations in Section 5, addressing Q1/Q2 from Reviewer fLAu and W1/W2/W5 from Reviewer Keje.

  • (3) We have added ACC calibration ablations (Table 7; Appendix C.11), addressing Q3 from Reviewer fLAu and W4/W5 from Reviewer Keje.

  • (4) We have added an analysis of final prediction fusion (Lines 305--307; Appendix C.10), addressing W2 from Reviewer LQVH.

  • (5) We have added a detailed computational-cost and FLOPs breakdown (Appendix C.12), addressing Q5 from Reviewer fLAu.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.