acceptodds
Under review as a conference paper at ICLR 2027

Active Test-Time Adaptation via Shift-Aware Querying and Class-Conditional Alignment

Abstract

Representations trained on one distribution can encode both predictive information that remains stable across distributions and distribution-specific associations that become unreliable under distribution shift. Test-time adaptation should reduce reliance on these associations while preserving predictive information, but unlabeled target data provides limited supervision for this objective. We propose Active Contrastive Test-Time Adaptation (ACTTA), which combines selective label acquisition with class-conditional representation alignment. ACTTA queries diverse target samples whose representations deviate from class-conditional source statistics. A supervised contrastive objective encourages representation alignment to reduce distribution-specific associations, while cross-entropy maintains class discrimination. Under an idealized data-generating process, our analysis characterizes conditions under which class-conditional invariance excludes distribution-varying information while preserving label information. Experiments on four vision and language benchmarks spanning temporal, geographic, and synthetic shifts show that ACTTA achieves the highest performance among the evaluated methods. Further evaluations demonstrate improved generalization to unseen distributions and preservation of performance in the training distribution.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, distribution shift, causality, representation, active learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ynAxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes ACTTA, an active test-time adaptation method that aims to reduce the distribution-specific associations in the representations while retaining the predictive information that is stable across distributions. The method consists of two parts: (1) It uses pre-computed class-conditional Gaussian statistics to calculate the normalized Mahalanobis distance, and selects the samples whose representations deviate the farthest from the source distribution and differ the most from each other for label querying. (2) It samples synthetic pseudo-source representations from the same set of statistics, applies a low-rank residual adapter to impose a shared transformation on both the source and the target representations, and achieves class-conditional alignment while maintaining class discriminability through a supervised contrastive loss. Experiments show that ACTTA outperforms the compared active and non-active TTA baselines in terms of test-time accuracy, generalization to unseen distributions, and preservation of performance on the original training distribution.

Critical strengths
  1. The active sample selection in Module 1 is well motivated for the active TTA setting, and it significantly outperforms non-active TTA methods in practice.
  2. The method does not rely on causal-structure priors and does not require manually designed data augmentations, making it naturally suitable for text tasks.
Critical weaknesses
  1. It is unclear where the training representations come from. A TTA setting should be source-free and should not use any data other than the test data, even if the source data is publicly available.
  2. The annotation budget is a crucial variable for active TTA, but no corresponding ablation results are provided. The test sets also differ in size across benchmarks, so a fixed annotation budget may not be reasonable.
Questions for authors
  1. The motivation starts from feature deviation (involving both causal and non-causal features), but why does the method perform the screening at the data-sample level? Is the sample level equivalent to the feature level?
  2. During testing, are the samples whose ground-truth labels were obtained using the annotation budget excluded from the evaluation?
3 comments
AuthorsOpenReview ↗

Reply to reviewer ynAx

We are grateful for the valuable feedback, and we thank the reviewer for acknowledging our motivation and our methods. The concerns and questions raised in the review are discussed in detail below.

Weakness 1. training representation and source-free

We acknowledge that ACTTA assumes access to source statistics computed from training representations before adaptation, but we do not need access to the original training representations during adaptation. We generate synthetic training representations using precomputed source statistics (equation 3) for our contrastive alignment.

We want to highlight that there is an existing TTA method with this assumption. One of the SOTA methods we compared to, CAFA, leverages the precomputed source statistics as well for better performance. We will make this assumption clear and characterize ACTTA as source-data-free but dependent on precomputed source statistics.

Weakness 2. annotation budget and ablation

We fixed the annotation budget to 1 sample per batch for all active TTA methods, following the practice of previous work. The total number of samples being annotated will be higher for larger datasets with more batches, and lower for smaller datasets with fewer batches.

We test the performance when the annotation budget per batch is 3. The table below shows the results on FMoW and iWildCam. The performance of active TTA methods improves as the annotation budget increases, and ACTTA performs the best among active TTA methods.

MethodFMoW (1 per batch)FMoW (3 per batch)iWildCam (1 per batch)iWildCam (3 per batch)
BiTTA32.46 ±\pm 1.2331.48 ±\pm 2.2735.26 ±\pm 0.0637.19 ±\pm 1.19
EATTA34.12 ±\pm 1.0137.67 ±\pm 1.9234.57 ±\pm 0.1236.67 ±\pm 1.12
SimATTA38.28 ±\pm 0.4840.01 ±\pm 1.4837.58 ±\pm 0.4839.96 ±\pm 0.25
ACTTA39.49 ±\pm 0.3842.28 ±\pm 1.0540.31 ±\pm 1.1541.57 ±\pm 0.37

Question 1. screening at sample level

The method performs the screening at the sample level. We consider causal and non-causal features to be what the sample is generated from (figure 2), but we do not assume that they are observable. Therefore, we cannot perform the screening at the feature level. For example, in the Waterbirds dataset, the bird type and background are the causal and non-causal features, but we do not have the information about bird type and background for every sample at test time.

We are motivated by previous work's understanding that the learned representation contains both causal information, which has a stable relationship with the label across distributions, and non-causal information, which does not have a stable relationship with the label across distributions. We want to update the learned representation to reduce the latter. To achieve this, we propose to query labels of some samples by screening at the sample level.

Question 2. evaluation of annotated sample

Annotated samples were not excluded from the evaluation, but the predictions are made before the labels were obtained. Following the practice of other TTA methods, the model makes predictions first, then performs a batch update where the label is queried. The performance is evaluated using the predictions before labels are known.

Thank you for the response and the additional experiments. I consider W2 and Q2 resolved.

On W1, I maintain my concern. Even if source data is used only before adaptation to obtain prior statistics, this still violates the setting of SFDA and TTA. I do not see practical meaning in the distinction between "using source data during adaptation" and "using it anywhere in the method". Access to source data is precisely the core bottleneck of SFDA and TTA. Once it is available, the problem is essentially domain adaptation. The CAFA precedent does not change this. A shared assumption does not make it compatible with the setting. Note that the pretrained weights all TTA methods rely on are part of the given problem interface, whereas statistics computed from labeled source data go beyond it.

On Q1, the response explains why screening must be done at the sample level, but it does not show that sample level deviation actually carries the feature level noncausal shift signal. No new evidence is provided. A correlation analysis between Mahalanobis distance and the degree of spurious attribute shift would be an example. As stated, this is a restatement of the design rationale rather than a verification.

Given these reasons, I have decided to maintain my original score.

AuthorsOpenReview ↗

Reply to reviewer ynAx

Thank you for clarifying your remaining concerns. In this response, we clarify the additional resources our setting assumes and provide a new Waterbirds analysis to test whether the Mahalanobis distance captures feature-level shift.

Weakness 1. training representation and source-free

We acknowledge that ACTTA requires information beyond pretrained weights. Our intended setting is deployment with precomputed source statistics available. These statistics can be computed by the model provider before the target distribution is known and supplied alongside the model without sharing the source dataset. Thus, ACTTA requires access to labeled source data to prepare these statistics, but does not require access to source examples during adaptation. This setting is practically relevant when a provider can distribute a model and summary statistics but cannot distribute the source dataset to deployment users.

Question 1. screening at sample level

We evaluate whether ACTTA’s sample-level deviation score is associated with background shift on Waterbirds, where bird type is the prediction target and background is a noncausal attribute. In the training set, 95.00% of landbirds are with a land background, and 94.97% of waterbirds are with a water background. In the test set, half of the landbirds are with a water background, and half of the waterbirds are with a land background.

For each test sample, we compute the Mahalanobis distance following equation 1. Within each bird class, we compare samples with the typical training background against those with the atypical background. The result is presented below. Samples with atypical backgrounds have higher distance at every reported percentile for both bird classes. This provides evidence that the sample-level deviation measured by Mahalanobis distance captures a signal associated with background shift.

Group5th-percentile25th-percentileMedian75th-percentile95th-percentile
Landbird/land background0.80430.94161.07351.20521.4288
Landbird/water background0.98371.13571.26581.44711.8553
Waterbird/water background0.82970.97781.13911.33411.6892
Waterbird/land background0.96091.15341.30761.46191.7338

We hope these clarifications and the new experiment address your remaining concerns. If these responses sufficiently address your concerns, we would be grateful if you would consider raising your recommendation.

Reviewer_Dk2MOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies active test-time adaptation (TTA), where a model can request a small number of labels during deployment. The proposed ACTTA method selects target samples that are most shifted relative to class-conditional source representations and adapts the model using queried labels, pseudo-labels, and synthetic source features. A contrastive alignment objective is used to encourage class-conditional invariance across distributions while preserving predictive information. Experiments on four benchmarks demonstrate the effectiveness of the proposed method over prior active and non-active TTA methods, with additional analyses of robustness and generalization.

Critical strengths

Exhaustive Evaluation - The paper evaluates not only adaptation accuracy, but also unseen-distribution generalization, in-distribution retention, noisy annotations, delayed feedback, and multiple ablation settings, providing strong evidence for the proposed method's effectiveness. This exhaustive experiments clearly demonstrate the effectiveness of proposed method ACTTA.

Well-Validated Query Selection Strategy - The proposed query selection criterion is carefully justified and validated through ablations and comparisons against entropy- and density-based alternatives, showing that the design choices are both effective and well-supported.

Theoretical Analysis - The theoretical analysis motivates the proposed adaptation objective, and the experimental results consistently validate the intended behavior, resulting in a coherent and convincing overall contribution.

Critical weaknesses

Limited Impact of the Contrastive Alignment Objective - One of the paper's central contribution is class-conditional contrastive alignment, yet the ablation in Table 6 of Appendix shows almost no performance change on iWildCam (40.29 -> 40.31), HuffPost (71.08 -> 71.14), and Waterbirds (76.86 -> 76.86) when the contrastive objective is added. I observe that it is effective for FMoW, but the other datasets do not show performance gain. This makes it unclear whether the main gains come from the proposed alignment mechanism or from active querying and supervised adaptation.

Reliance on Oracle Hyperparameter Selection - The paper selects hyperparameters using oracle selection and tunes several method-specific hyperparameters in addition to learning rate. The hyperparameter analysis further shows that performance can vary noticeably across settings, raising concerns about whether the reported gains would persist under a realistic model-selection scenario. While the use of oracle hyperparameter selection is consistent with prior work, I would recommend additionally evaluating a realistic model-selection setting using a separate validation set. Comparing against oracle selection would clarify how sensitive the method is to hyperparameter tuning and improve the practical relevance of the results.

Questions for authors

Please refer to the weaknesses

3 comments
AuthorsOpenReview ↗

Reply to reviewer Dk2M

We appreciate the helpful feedback provided by the reviewer and thank the reviewer for acknowledging the breadth of our evaluation, the selection strategy, and the theoretical analysis. We present our discussions below:

Weakness 1. limited Impact of the Contrastive Alignment Objective

The results in Table 8 in the updated manuscript (Table 6 in the original manuscript) of Appendix ablates line 269's loss function by removing the contrastive term. We perform an additional study that ablates line 269's loss function by removing the cross-entropy term. The results are presented below. The main gain is from supervised adaptation with active querying.

MethodFMoWiWildCamHuffPostWaterbirds
No TTA31.1632.2970.3560.28
ACTTA w/o contrastive objective36.05 ±\pm 1.8540.29 ±\pm 1.5171.08 ±\pm 0.2176.86 ±\pm 0.29
ACTTA w/o cross-entropy objective31.12 ±\pm 0.0831.29 ±\pm 0.1070.35 ±\pm 0.2359.97 ±\pm 0.00
ACTTA39.49 ±\pm 0.3840.31 ±\pm 1.1571.14 ±\pm 0.0776.86 ±\pm 0.24

However, we found that ACTTA w/o contrastive objective performs worse than ACTTA on the post-adaptation performance on the held-out validation distribution. These results suggest that the contrastive alignment objective helps the model generalize better to unseen distributions. Nevertheless, alignment alone is not sufficient to improve generalization to unseen distributions, as shown by the bad performance of ACTTA w/o cross-entropy objective. The results are presented below.

MethodFMoWiWildCamHuffPost
No TTA50.1936.4471.41
ACTTA w/o contrastive objective51.24 ±\pm 1.2534.26 ±\pm 0.7971.61 ±\pm 0.07
ACTTA w/o cross-entropy objective50.15 ±\pm 0.0636.26 ±\pm 0.6271.42 ±\pm 0.01
ACTTA52.10 ±\pm 0.6836.60 ±\pm 0.7071.63 ±\pm 0.04

Weakness 2. more realistic selection method

We use oracle selection to benchmark the upper-bound performance of the methods [1]. We present the results of test-time adaptation performance using the hyperparameters selected by in-distribution data. The best hyperparameter is selected by the model's test-time adaptation performance on the in-distribution test split for FMoW, iWildCam and HuffPost, and training distribution for Waterbirds, due to lacking a test split of in-distribution data. ACTTA still performs the best in FMoW, iWildCam, and HuffPost, and the second best in Waterbirds. The results suggest that ACTTA can maintain a performance advantage over other active TTA methods when the hyperparameters are selected by the in-distribution data. Compared to oracle selection in Table 1, we see a drop in methods' performance.

MethodFMoWiWildCamHuffPostWaterbirds
no TTA31.1632.2970.3560.28
BiTTA31.29 ±\pm 0.6434.80 ±\pm 0.3967.85 ±\pm 0.1573.31 ±\pm 1.36
EATTA33.96 ±\pm 0.8034.25 ±\pm 0.4869.99 ±\pm 0.2559.40 ±\pm 0.53
SimATTA32.50 ±\pm 1.4937.58 ±\pm 0.4870.14 ±\pm 0.4961.58 ±\pm 2.72
ACTTA34.08 ±\pm 1.3339.50 ±\pm 1.0270.73 ±\pm 0.2567.55 ±\pm 0.83

[1] Zhao et al., On Pitfalls of Test-Time Adaptation, ICML 2023

The authors have resolved my concerns and I would like to main my positive assessment.

Thank you again for your careful review and constructive feedback! We are pleased that our responses have resolved your concerns, and we appreciate your continued positive assessment of our work.

Reviewer_QfLQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ACTTA combines active label acquisition with test-time feature adaptation. It queries diverse samples far from predicted-class source statistics, then trains a residual adapter and classifier using queried labels, pseudo-labels, and Gaussian source-feature replay. A contrastive term encourages class-conditional alignment. The paper reports the highest main-table means on four image and text benchmarks, plus held-out-distribution and source-retention results. Its information-theoretic analysis describes conditions under an explicitly idealized process; it does not establish that finite ACTTA training attains exact invariance.

Critical strengths
  • The evaluation supports utility across several kinds of shift. Table 1 covers FMoW, iWildCam, HuffPost, and Waterbirds, using their appropriate metrics and three-run deviations. ACTTA has the largest listed mean in each column. Gains over the best other active mean are 1.21, 2.73, 0.37, and 1.78 percentage points. This breadth is useful, although small differences are not established as statistically significant and the results use oracle tuning.

  • The combined design differs from the inspected active and class-aware methods. The paper uses source-relative querying and a shared transformation of real target and synthetic source features. SimATTA uses clustering , EATTA uses perturbation-sensitive queries, and CAFA uses fixed class-statistic distances. The contribution lies in the tested combination.

Critical weaknesses
  • The results establish an oracle-selected comparison rather than an available budget-limited deployment rule.
  • Paper page 7, hyperparameter paragraph, discloses oracle selection over learning rates, quantiles, and adapter dimensions. The oracle protocol also applies to baselines, so the comparisons retain value. This is a deployment limitation, with no demonstrated leakage or extra annotation cost.
  • The idealized process has limited reach to the motivating spurious-correlation example. Eq. 5 and following paragraph, explicitly makes labels independent of environment and designated non-causal features. The Waterbirds description on p. 7 includes training bird-background association. This is a minor scope limit because the paper labels the analysis idealized.
Questions for authors
  • Which split and labels select each configuration? Can one fixed or budget-valid selection rule retain the advantage on the existing benchmarks, and are held-out results independent of that selection?
  • What part of the idealized argument applies when non-causal features correlate with labels in training?
  • Please specify empty-memory querying, covariance conditioning, and whether predictions precede or follow the batch update. These are unresolved implementation details .
3 comments
AuthorsOpenReview ↗

Reply to reviewer QfLQ (1/2)

We thank the reviewer for the constructive feedback and for acknowledging the novelty of our method's design. We present our replies below:

Strength 1. statistical significance

We use the t-test to compute the statistical significance between the results of ACTTA and the second best performed method. The p-value shows how likely the mean performance difference between ACTTA and the second best performed method is zero. The table below shows the results.

FMoWiWildCamHuffPostWaterbirds
p-value0.12070.11920.01540.1602

Weakness 1 and 2. alternative selection strategy to oracle selection

We set the same annotation budget for all active TTA methods to 1 sample queried per batch. There was no tuning on the annotation budget. We use oracle selection as suggested in TTAB [1] to benchmark the upper-bound performance of the methods. The full labels are only used to select the best hyperparameters, are not leaked during adaptation, and do not incur extra annotation costs.

We present the results of test-time adaptation performance using the hyperparameters selected by in-distribution data. The best hyperparameter is selected by the model's test-time adaptation performance on the in-distribution test split for FMoW, iWildCam and HuffPost, and training distribution for Waterbirds, due to lacking a test split of in-distribution data. ACTTA still performs the best in FMoW, iWildCam, and HuffPost, and the second best in Waterbirds. The results suggest that ACTTA can maintain a performance advantage over other active TTA methods when the hyperparameters are selected by the in-distribution data.

MethodFMoWiWildCamHuffPostWaterbirds
no TTA31.1632.2970.3560.28
BiTTA31.29 ±\pm 0.6434.80 ±\pm 0.3967.85 ±\pm 0.1573.31 ±\pm 1.36
EATTA33.96 ±\pm 0.8034.25 ±\pm 0.4869.99 ±\pm 0.2559.40 ±\pm 0.53
SimATTA32.50 ±\pm 1.4937.58 ±\pm 0.4870.14 ±\pm 0.4961.58 ±\pm 2.72
ACTTA34.08 ±\pm 1.3339.50 ±\pm 1.0270.73 ±\pm 0.2567.55 ±\pm 0.83

Weakness 3. assumption of theoretical analysis

The analysis suggests that the test-time adaptation objectives that ACTTA proposes encourage reducing distribution-specific associations while maintaining predictive performance. The assumption is on the data observed during test-time, under the idealized condition that we have infinitely many independent samples from the data-generating process during adaptation.

Question 1.split and labels used for selection, performance using other selection methods, and held-out results

In the paper, we select the hyperparameters using the oracle, i.e. the out-of-distribution test distribution. The best hyperparameter is selected by the model's test-time adaptation performance on the OOD test split. Labels of the OOD test distribution are used for the selection.
Our results of selecting using in-distribution data (reply to weakness 1 and 2) suggest that ACTTA can maintain performance advantage over other active TTA methods.

The performance on the held-out validation split was evaluated on the model after test-time adaptation finishes. We use this result to show the adapted model's generalization performance to an unseen distribution. The result is not used for hyperparameter selection. We present the result of the post-adaptation performance on a held-out validation distribution when hyperparameters are selected by in-distribution data below. ACTTA achieves the best accuracy on FMoW and HuffPost, suggesting that models adapted by ACTTA generalize better to unseen distributions compared to other active TTA methods on the two datasets.

MethodFMoWiWildCamHuffPost
no TTA50.1936.4471.41
BiTTA48.36 ±\pm 1.0435.45 ±\pm 1.2667.52 ±\pm 0.33
EATTA49.32 ±\pm 0.7033.65 ±\pm 0.5070.73 ±\pm 0.24
SimATTA48.07 ±\pm 1.0836.32 ±\pm 0.2871.31 ±\pm 0.09
ACTTA50.48 ±\pm 0.8233.81 ±\pm 0.1071.35 ±\pm 0.19

[1] Zhao et al., On Pitfalls of Test-Time Adaptation, ICML 2023

AuthorsOpenReview ↗

Reply to reviewer QfLQ (2/2)

Question 2. theoretical analysis when non-causal features correlate with labels in training

The analysis focuses on test-time adaptation. Regardless of what the training data is like, under the idealized scenario where we have infinitely many independent data samples drawn from the data-generating process during adaptation, ACTTA's objective encourages reducing distribution-specific associations while maintaining predictive performance.

Question 3. implementation details

When the memory bank is empty, we query the label of the sample that has the largest normalized Malanobis distance; when the memory bank is not empty, we retain samples whose normalized Mahalanobis distance isat or above the τd\tau_d-quantile as candidates for annotation, and select samples that differ the most from previously selected samples stored in the memory bank.

We use torch.linalg.pinv to obtain an inverse of the covariance matrix. Small singular values are discarded according to PyTorch’s default numerical. For synthetic source representations, we use torch.linalg.svd to compute the eigenvalues and eigenvectors.

Following the practice of other TTA methods, the model make predictions first, then performs a batch update where the label is queried. The performance is evaluated using the prediction before labels are known.

Thank you again for your constructive feedback! We would appreciate your thoughts on whether our reply addressed your questions and concerns. If any points remain unresolved, we would be happy to provide further clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of author-reviewer discussion

We thank the reviewers for their valuable time and constructive feedback. We have updated our manuscript to include the points discussed. We summarize the questions and our responses below.

Reviewer QfLQ

  • statistical significance: we compute the statistical significance between the results of ACTTA and the second best performed method.
  • alternative strategies to oracle selection: we present the results using in-distribution data for hyperparameter selection. ACTTA still performs the best on 3 datasets and the second best on 1 dataset.
  • assumption of theoretical analysis: the analysis concerns an idealized test-time data-generating process.
  • split and label used for selection, and held-out results: we clarify the split and label used for selection, and mention that the held-out result was performed on the model after being adapted using the selected hyperparameters. We additionally include the held-out results when hyperparameters are selected by in-distribution data, showing ACTTA generalizes better to held-out unseen distributions.
  • implementation details: we clarify the empty-memory querying, covariance, and prediction order.

Reviewer Dk2M

  • impact of contrastive alignment objective: we perform an ablation study to show that supervised adaptation and active querying account for the main adaptation gains. However, contrastive alignment helps the model generalize better to unseen distributions.
  • more realistic selection method than oracle selection: we present the results using in-distribution data for hyperparameter selection. ACTTA still performs the best on 3 datasets and the second best on 1 dataset.

Reviewer ynAx

  • access to source information: we clarify that ACTTA requires access to precomputed source statistics, but not the original training data during adaptation. Our intended setting is deployment with precomputed source statistics available. Our setting is relevant when a provider can distribute a model and source statistics but cannot distribute the source dataset to deployment users.
  • annotation budget and ablation: we clarify that we fixed the annotation budget to 1 sample per batch for all active TTA methods, and total annotations scale with dataset size. We present the result when the budget is set to 3 samples per batch on FMoW and iWildCam, showing that ACTTA still performs the best among active TTA methods.
  • screening at sample level: we state the rationle to perform screening at sample level rather than feature level. We supply an analysis on Waterbirds to show that sample level deviation measured by our proposed Mahalanobis distance can capture feature level shift.
  • evaluation of annotated samples: we clarify that queried samples remain in evaluation, but their predictions are recorded before their labels are acquired.

In conclusion, we present an active test-time adaptation method ACTTA, that learns a new representation space where distribution-specific associations are reduced. Our theoretical analysis explains that ACTTA's objective encourages reducing distribution-specific associations while maintaining predictive performance. Empirical analysis on four vision and language datasets shows superior test-time adaptation performance, better generalization to unseen domains, and less performance degradation on the training distribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.