- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes ACTTA, an active test-time adaptation method that aims to reduce the distribution-specific associations in the representations while retaining the predictive information that is stable across distributions. The method consists of two parts: (1) It uses pre-computed class-conditional Gaussian statistics to calculate the normalized Mahalanobis distance, and selects the samples whose representations deviate the farthest from the source distribution and differ the most from each other for label querying. (2) It samples synthetic pseudo-source representations from the same set of statistics, applies a low-rank residual adapter to impose a shared transformation on both the source and the target representations, and achieves class-conditional alignment while maintaining class discriminability through a supervised contrastive loss. Experiments show that ACTTA outperforms the compared active and non-active TTA baselines in terms of test-time accuracy, generalization to unseen distributions, and preservation of performance on the original training distribution.
Critical strengths
- The active sample selection in Module 1 is well motivated for the active TTA setting, and it significantly outperforms non-active TTA methods in practice.
- The method does not rely on causal-structure priors and does not require manually designed data augmentations, making it naturally suitable for text tasks.
Critical weaknesses
- It is unclear where the training representations come from. A TTA setting should be source-free and should not use any data other than the test data, even if the source data is publicly available.
- The annotation budget is a crucial variable for active TTA, but no corresponding ablation results are provided. The test sets also differ in size across benchmarks, so a fixed annotation budget may not be reasonable.
Questions for authors
- The motivation starts from feature deviation (involving both causal and non-causal features), but why does the method perform the screening at the data-sample level? Is the sample level equivalent to the feature level?
- During testing, are the samples whose ground-truth labels were obtained using the annotation budget excluded from the evaluation?
3 comments
Thank you for the response and the additional experiments. I consider W2 and Q2 resolved.
On W1, I maintain my concern. Even if source data is used only before adaptation to obtain prior statistics, this still violates the setting of SFDA and TTA. I do not see practical meaning in the distinction between "using source data during adaptation" and "using it anywhere in the method". Access to source data is precisely the core bottleneck of SFDA and TTA. Once it is available, the problem is essentially domain adaptation. The CAFA precedent does not change this. A shared assumption does not make it compatible with the setting. Note that the pretrained weights all TTA methods rely on are part of the given problem interface, whereas statistics computed from labeled source data go beyond it.
On Q1, the response explains why screening must be done at the sample level, but it does not show that sample level deviation actually carries the feature level noncausal shift signal. No new evidence is provided. A correlation analysis between Mahalanobis distance and the degree of spurious attribute shift would be an example. As stated, this is a restatement of the design rationale rather than a verification.
Given these reasons, I have decided to maintain my original score.
Reply to reviewer ynAx
Thank you for clarifying your remaining concerns. In this response, we clarify the additional resources our setting assumes and provide a new Waterbirds analysis to test whether the Mahalanobis distance captures feature-level shift.
Weakness 1. training representation and source-free
We acknowledge that ACTTA requires information beyond pretrained weights. Our intended setting is deployment with precomputed source statistics available. These statistics can be computed by the model provider before the target distribution is known and supplied alongside the model without sharing the source dataset. Thus, ACTTA requires access to labeled source data to prepare these statistics, but does not require access to source examples during adaptation. This setting is practically relevant when a provider can distribute a model and summary statistics but cannot distribute the source dataset to deployment users.
Question 1. screening at sample level
We evaluate whether ACTTA’s sample-level deviation score is associated with background shift on Waterbirds, where bird type is the prediction target and background is a noncausal attribute. In the training set, 95.00% of landbirds are with a land background, and 94.97% of waterbirds are with a water background. In the test set, half of the landbirds are with a water background, and half of the waterbirds are with a land background.
For each test sample, we compute the Mahalanobis distance following equation 1. Within each bird class, we compare samples with the typical training background against those with the atypical background. The result is presented below. Samples with atypical backgrounds have higher distance at every reported percentile for both bird classes. This provides evidence that the sample-level deviation measured by Mahalanobis distance captures a signal associated with background shift.
| Group | 5th-percentile | 25th-percentile | Median | 75th-percentile | 95th-percentile |
|---|---|---|---|---|---|
| Landbird/land background | 0.8043 | 0.9416 | 1.0735 | 1.2052 | 1.4288 |
| Landbird/water background | 0.9837 | 1.1357 | 1.2658 | 1.4471 | 1.8553 |
| Waterbird/water background | 0.8297 | 0.9778 | 1.1391 | 1.3341 | 1.6892 |
| Waterbird/land background | 0.9609 | 1.1534 | 1.3076 | 1.4619 | 1.7338 |
We hope these clarifications and the new experiment address your remaining concerns. If these responses sufficiently address your concerns, we would be grateful if you would consider raising your recommendation.
Reply to reviewer ynAx
We are grateful for the valuable feedback, and we thank the reviewer for acknowledging our motivation and our methods. The concerns and questions raised in the review are discussed in detail below.
Weakness 1. training representation and source-free
We acknowledge that ACTTA assumes access to source statistics computed from training representations before adaptation, but we do not need access to the original training representations during adaptation. We generate synthetic training representations using precomputed source statistics (equation 3) for our contrastive alignment.
We want to highlight that there is an existing TTA method with this assumption. One of the SOTA methods we compared to, CAFA, leverages the precomputed source statistics as well for better performance. We will make this assumption clear and characterize ACTTA as source-data-free but dependent on precomputed source statistics.
Weakness 2. annotation budget and ablation
We fixed the annotation budget to 1 sample per batch for all active TTA methods, following the practice of previous work. The total number of samples being annotated will be higher for larger datasets with more batches, and lower for smaller datasets with fewer batches.
We test the performance when the annotation budget per batch is 3. The table below shows the results on FMoW and iWildCam. The performance of active TTA methods improves as the annotation budget increases, and ACTTA performs the best among active TTA methods.
Question 1. screening at sample level
The method performs the screening at the sample level. We consider causal and non-causal features to be what the sample is generated from (figure 2), but we do not assume that they are observable. Therefore, we cannot perform the screening at the feature level. For example, in the Waterbirds dataset, the bird type and background are the causal and non-causal features, but we do not have the information about bird type and background for every sample at test time.
We are motivated by previous work's understanding that the learned representation contains both causal information, which has a stable relationship with the label across distributions, and non-causal information, which does not have a stable relationship with the label across distributions. We want to update the learned representation to reduce the latter. To achieve this, we propose to query labels of some samples by screening at the sample level.
Question 2. evaluation of annotated sample
Annotated samples were not excluded from the evaluation, but the predictions are made before the labels were obtained. Following the practice of other TTA methods, the model makes predictions first, then performs a batch update where the label is queried. The performance is evaluated using the predictions before labels are known.