acceptodds
Under review as a conference paper at ICLR 2027

Beyond Holistic Representations: Dual Distribution Modeling for Test-Time Adaptation of Vision–Language Models

Abstract

Although Vision-Language Models (VLMs) exhibit strong zero-shot generalization, their performance degrades substantially under domain shifts and out-of-distribution (OOD) scenarios. Test-Time Adaptation (TTA) addresses this issue using only unlabeled test samples, among which training-free TTA is particularly appealing for its efficiency. Existing training-free methods refine predictions using cached samples or representation distributions, but their reliance on holistic image representations limits the extraction of more semantically meaningful features. To this end, we decompose visual features into in-distribution features and class-related OOD features based on their alignment with textual semantics, and empirically show that both exhibit clear class-wise clustering in the embedding space. Based on this finding, we propose Dual Distribution Modeling for Adaptation (DDMA), a novel training-free TTA framework that separates these two feature types to better exploit semantic information while reducing interference from semantically irrelevant features. DDMA independently models their class-wise distributions during test time and jointly refines the original predictions by their distribution statistics. Extensive experiments across cross-domain and OOD benchmarks show that DDMA outperforms state-of-the-art TTA methods, achieving average accuracy gains of 3.37% and 4.02% over the pre-adaptation performance. Moreover, DDMA maintains competitive efficiency with an average throughput of 29.37 items/s.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-Language Models, Test-time Adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WaPFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets training-free, online test-time adaptation of CLIP. It argues that existing cache- and distribution-based methods rely on the holistic image embedding, in which poorly aligned features interfere with class semantics. Based on a token-masking analysis on ImageNet, the authors split high-attention patch tokens into "ID features" (tokens whose similarities to the class prompts have low entropy) and "class-related OOD features" (high entropy), and report that both groups cluster by class. The proposed method, DDMA, extracts the two token groups from confidently predicted test samples and projects their representations into the text-embedding subspace. It keeps a running class mean for each group and adds the similarities to these means to the zero-shot logits. With ViT-B/16 on 10 cross-domain datasets and on ImageNet with four variants, DDMA reports average gains of 3.37 and 4.02 points over CLIP and the best average accuracy among the 10 compared baselines, at a throughput comparable to other training-free methods. Ablations cover the feature decomposition, token selection, the projection and the hyperparameters, and ViT-B/32 results are given in the appendix.

Critical strengths

Training-free and efficient. DDMA needs no backpropagation and has no learnable parameters. Prior training-free methods use the holistic image embedding; DDMA instead models the class-wise distributions of two token-level feature types (ID and class-related OOD features) and uses them to correct the zero-shot logits. It runs at 29.37 items/s with 15.46 GFLOPs, comparable to TT-RAA and faster than the cache-based TDA and DMN on the same hardware.

Consistent gains over CLIP across 15 benchmarks. With ViT-B/16, DDMA improves on zero-shot CLIP on cross-domain datasets and ImageNet sets. The gains also hold with ViT-B/32. Against 10 baselines covering prompt-tuning, cache-based and distribution-based methods, it reaches the highest average accuracy on both benchmark groups.

Critical weaknesses

W1. The gain seems to come from the projection rather than from the dual decomposition. The text-subspace projection gets one sentence in Sec. 3.3, is described only in App. C. Without it, DDMA falls below zero-shot CLIP on 3 of 4 datasets (Tab. 7: Cars 61.69 vs 66.11, Pets 80.31 vs 86.92, ImageNet-V2 59.98 vs 61.88). The dual decomposition, by contrast, adds only +0.22 / +0.86 / +3.01 / +0.81 over a holistic prototype (Tab. 6). The paper has no ablation crossing holistic vs. dual with/without projection, and does not apply the projection to TDA or TT-RAA. Without these, I cannot tell how much of the improvement is due to the paper's main idea.

W2. The "interference" motivation is never tested. The split is motivated by the claim that OOD features interfere with class semantics in holistic representations, but no experiment measures this. In Tab. 6, neither branch alone beats the holistic prototype everywhere: ID-only is lower on ImageNet-A, and C-OOD-only is lower on Cars and Pets. Only the combination wins. The combined setting also has one more correction term with its own tuned weight. A capacity-matched control (e.g., holistic + ID, or two holistic branches) is needed before the gain can be attributed to the separation.

W3. Observations I and II lack proper controls. Keeping the top-80% attention tokens lowers accuracy by 3.3 points and changes entropy by only 0.015. I would read that as information loss rather than the "reduced interference" claimed in Observation I. The M̄1 collapse compares 20% of the tokens against 80%, and no mask is compared with a random mask of the same size. For Observation II, nothing tests whether M̄2 tokens are "insufficiently learned during pre-training", and 23.45% accuracy is just as consistent with weakly informative background tokens.

W4. Several decomposition and modelling choices are not justified or ablated. Max-over-heads for M1 is motivated by head variance, while the analysis that defines the token groups uses mean-over-heads, and Fig. 4 compares the two on only four datasets, with differences of about 0.5 points or less on the cross-domain ones. The entropy criterion for M2 is not compared with simpler splits such as max cosine, attention rank or random, and the masking layer (L−1L-1) is not ablated. Finally, each "distribution" is just a class mean. Although the method is framed as distribution modelling, it is not compared with the Gaussian models of DOTA or ADAPT.

W5. No comparison with recent TTA methods. Although the abstract and contributions claim that DDMA outperforms state-of-the-art TTA methods, the paper does not compare with recent existing TTA methods, such as DOTA (NeurIPS 2025), ADAPT (NeurIPS 2025), MCP (ICCV 2025), ADTE (ICLR 2026) and MuRA (ACM MM 2026).

Questions for authors

Q1. Could you report Tabs. 3–4 with a single configuration chosen without target labels (e.g., on a held-out split or a separate source dataset), averaged over at least three stream orders?

Q2. On ImageNet, each of the 1000 class centres is initialised from a single unfiltered sample. How does accuracy evolve along the stream there? Is there an early phase in which DDMA is below zero-shot CLIP?

Q3. Does the GFLOPs estimate for DDMA (15.46 vs. 15.05 for CLIP) include the two extra passes through the last block and the per-token similarity to all CC class prompts?

Q4. Could you add the ablation crossing holistic vs. dual with/without projection on all 15 datasets, together with TDA and TT-RAA using the same projection?

6 comments
AuthorsOpenReview ↗

Responses to the Feedback of Reviewer WaPF

Thank you very much for your valuable suggestions on our work! You can find the responses to your questions as follows:

W1&Q1&Q4: First, we would like to clarify that the projection module is specifically designed to mitigate the modality gap between visual and textual embeddings, enabling the distribution statistics of visual embeddings to be effectively compared with textual semantics for refining the original predictions.

However, this module is not directly applicable to other baseline methods due to differences in their adaptation mechanisms. Specifically, TDA performs prediction refinement based on similarities between visual embeddings, without requiring cross-modal similarity computation, and thus does not suffer from the same modality gap. Similarly, TT-RAA incorporates a specifically designed Multimodal Space Retrieval Augmentation mechanism to mitigate the modality gap. Therefore, applying our projection module to these methods would not necessarily provide additional benefits, as their respective adaptation mechanisms already avoid or explicitly address this issue.

Following the reviewer's suggestion, we have conducted additional ablation experiments without the projection module to more rigorously demonstrate the contribution of our dual feature decomposition strategy. The results are reported in Tab. R1, covering eight representative datasets due to the limited time during the rebuttal period. We will incorporate the complete results into the revised manuscript later.

Table R1: Ablation study on our feature decomposition without the projection module.

CarsPetsFlower102Food101ImageNet-AImageNet-V2ImageNet-SImageNet-R
Holistic58.4480.0866.1282.4346.954.7242.8274.08
ID Feat.59.1681.5466.7981.8946.8154.6540.7775.10
C-OOD Feat.53.8378.2662.3578.2048.5256.4344.3975.96
ID + C-OOD Feat.61.6983.3168.5383.7450.2759.9845.5678.61

The additional results show that, without the projection module, our feature decomposition strategy achieves more substantial improvements over holistic representations, with a particularly notable gain on OOD benchmarks. This can demonstrate the effectiveness of the proposed feature decomposition.

W2: The results in Tab. 6 are intended to demonstrate the effectiveness of feature decomposition, rather than directly quantify the interference between the two feature groups. To provide a more direct investigation of the interference between ID and class-related OOD features, we conduct an additional experiment in which we combine half of the tokens with lower entropy from M2M_2 and M‾2\overline{M}_2 respectively during the feature selection process. We also remove the projection module. The results, comparing this combined setting with the ID-only and class-related OOD-only settings, are reported in Tab. R2.

Table R2: Comparison of combined setting with the ID-only and C-OOD-only settings.

CarsPetsImageNet-AImageNet-V2
ID Feat.59.1681.5446.8154.65
C-OOD Feat.53.8378.2648.5256.43
Combined54.0778.3143.4953.18

The results show that combining ID and C-OOD features leads to a substantial degradation in adaptation performance, particularly on OOD benchmarks. This indicates that mixing the two types of features leads to mutual interference, which motivates us to separate two types of features to avoid this interference.

AuthorsOpenReview ↗

Responses to the Feedback of Reviewer WaPF (2)

W3: First, we argue that the classification performance of CLIP can directly reflect how well visual features of inputs have been learned and aligned with textual semantics during pre-training. In CLIP-based classification task, the predictions rely on the alignment and similarity between image representations and their corresponding textual semantics. Indeed, the pre-training objective of CLIP is to align image and text embeddings in a shared embedding space. Therefore, features that are insufficiently learned and poorly aligned with textual semantics are expected to result in less reliable predictions, characterized by lower classification accuracy and higher prediction entropy. This idea can also be supported by many current works, e.g. [c] and [d].

Regarding Observation I, we have additionally conducted experiments using a random mask that retains 20% tokens. The randomly selected tokens achieve a classification accuracy of 14.65% and a prediction entropy of 1.132. In comparison, the tokens selected by M‾1\overline{M}_1 yield even lower accuracy (12.76%) and comparable prediction entropy, indicating that these low-attention tokens contain almost no meaningful class-related semantic information, rather than simply suffering from information loss due to a reduced number of tokens.

Furthermore, the performance gap between retaining high-attention tokens and using the random mask demonstrates that selecting tokens based on attention provides a more effective way to preserve semantically informative features. These results also suggest that incorporating low-attention tokens into the image representation may introduce noise and interfere with the contribution of more informative features, thereby motivating our attention-based token selection strategy.

As for Observation II, our experimental results reveal that token embeddings selected by M2{M}_2 consistently exhibit lower classification accuracy and higher prediction entropy in classification, but their classification accuracy is substantially higher than that of the M‾1\overline{M}_1 tokens. This indicates that the M‾2\overline{M}_2 tokens still preserve meaningful class-related semantic information. In addition, as discussed in L202 of the original manuscript, the tokens selected by M2M_2 and M‾2\overline{M}_2 have comparable attention weights, yet the latter exhibit considerably higher prediction entropy. This further suggests that these tokens cannot simply be attributed to low attention or a lack of semantic information. Instead, these tokens contain potentially useful class-related features that are insufficiently learned and stably aligned with textual semantics during CLIP pre-training.

W4&W5:

  1. Mean-over-heads is a commonly adopted practice in existing works. Therefore, we use it only in the exploratory experiments in Sec. 3, as it provides a more intuitive characterization. In contrast, in our proposed method, we adopt max-over-heads to preserve as many potentially informative tokens as possible, including those that may receive strong attention from only a subset of heads. The corresponding ablation results comparing the two aggregation strategies are presented in Fig. 4. Consistent with findings in existing studies in TTA [a, c, d], a modest improvement of about 0.5% can demonstrate the effectiveness of this design.

  2. When distinguishing ID and class-related OOD features, our primary concern is whether the visual features can be consistently and reliably aligned with textual semantics. Therefore, we adopt prediction entropy as the criterion, as it directly reflects the stability of vision–language alignment. Furthermore, we have conducted additional experiments comparing entropy with two alternative criteria, Max Cosine and Attention Rank. The results are reported in Tab. R3.

    Table R3: Comparison of different criterions for token selection.

    CarsPetsImageNet-AImageNet-V2
    Max Cos.66.5488.9356.8562.38
    Atten. Rank66.2989.1652.4160.57
    DDMA (Ours)66.3289.5761.4664.50

    According to the results, although the performance differences among the three criteria are marginal on CD datasets such as Cars and Pets, Max Cosine and Attention Rank perform obviously worse than our method on OOD benchmarks, which demonstrates the superiority of our criterion. This is because Attention Rank cannot reflect vision–language alignment. Although Max Cosine can identify tokens with relatively high semantic alignment, it fails to characterize the stability of such alignment. Consequently, it remains limited when adapting to datasets containing a larger proportion of unseen and OOD features.

AuthorsOpenReview ↗

Responses to the Feedback of Reviewer WaPF (3)

W4&W5:

  1. Our feature decomposition strategy follows the feature cropping techniques in prior works [b, c]. Accordingly, we operate on the features from the (L−1)(L-1)-th layer, which preserve richer and more comprehensive information from the input image than other layers. This design choice is recognized as a common and effective practice in existing studies.

  2. DOTA and ADAPT models the distribution of embeddings based on conditional probabilities and refines the prediction probabilities accordingly. However, it relies on an important assumption that the test-time class distribution is uniform, i.e., P(y=yi)=1/KP(y=y_i)=1/K, where KK denotes the number of classes. This assumption may not hold in practical online TTA scenarios, where the class distribution of incoming test samples can be arbitrary and dynamically changing. Consequently, the applicability of ADAPT may be limited under such settings.

    Following the reviewer's suggestion, we have additionally implemented and evaluated two recent methods, DOTA (NeurIPS 2025) and ADAPT (NeurIPS 2025). The corresponding experimental results are reported in Tab. R4.

    Table R4: Comparison results among DOTA, ADAPT and our method on several benchmarks.

    AircraftCaltech101CarsFlowerPetsImageNet-AImageNet-V2
    DOTA24.1894.5367.8471.6287.5854.9361.28
    ADAPT24.4794.7667.0970.2487.4556.6861.92
    DDMA (Ours)25.2994.2566.3272.5389.5761.4664.50

    Compared with DOTA and ADAPT, our method achieves slightly lower accuracy on Caltech101 and Stanford Cars, but outperforms them on the remaining benchmarks, with particularly notable gains on ImageNet-A and ImageNet-V2. We will include the complete experimental results in the revised version.

Q2: We have evaluated the prediction accuracy of DDMA and zero-shot CLIP on ImageNet as the number of processed test samples increases. The results are presented in Tab. R5. During the early stage of the test stream, DDMA may achieve slightly lower accuracy than zero-shot CLIP because only a limited number of high-quality samples have been collected, and the class-wise feature distributions have not yet been sufficiently established. However, after 1000 test samples, the accumulated distribution information becomes more representative, leading to a noticeable improvement in prediction accuracy. Subsequently, DDMA's performance gradually stabilizes and maintains a clear advantage over zero-shot CLIP.

Table R5: The accuracy results during a test stream.

10030050010002000500010000
CLIP68.0067.6767.6068.3068.2567.9468.05
DDMA61.0065.3366.8068.9069.4071.4870.23

Q3: When estimating the GFLOPs, the first (L−1)(L-1) attention blocks are executed only once, whereas the final attention block and the projection layer are executed twice to process the two separately selected token groups. Regarding the text embeddings of all classes, they are precomputed before the adaptation process begins and are therefore excluded from the GFLOPs calculation.

[a] Is less more? exploring token condensation as training-free test-time adaptation. ICCV 2025.

[b] From local details to global context: advancing vision-language models with attention-based selection. ICML 2025

[c] Test-time retrieval-augmented adaptation for vision-language models. ICCV 2025.

[d] BaFTA: Backprop-free test-time adaptation for zero-shot vision-language models. arxiv 2024.

Reviewer_23YWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DDMA partitions high-attention visual tokens according to their text-alignment entropy and maintains two sets of class prototypes to refine CLIP predictions. The related experiments show promising performance with a clear inference burden.

Critical strengths
  1. A clear motivation. The method attempts to exploit visual features that align weakly with text but still contain class information, instead of simply discarding them.

  2. Broad benchmark coverage. Evaluation includes 10 cross-domain datasets, ImageNet and four variants, with additional ViT-B/32 results.

  3. A relatively simple adaptation mechanism. DDMA requires no backpropagation or learnable parameters and maintains only two sets of class prototypes.

Critical weaknesses
  1. The average gains over the strongest baselines are only 0.75 and 0.64 percentage points on cross-domain and OOD benchmarks, respectively. Without mean/std across different test stream orders, it is unclear how consistent these gains are.

  2. The method combines existing ideas, including attention-based token selection, prototype updates, and text-based projection. The novelty mainly lies in modeling two feature groups separately, but the theoretical explanation for why this split works is limited.

  3. Section 4.1 states that hyperparameters are tuned separately for each dataset, including dataset-dependent choices of different hyperparameters. How are these values selected, and do the baselines receive comparable tuning? This needs clarification to ensure a fair comparison.

  4. The ID/OOD split is mainly based on token–text prediction entropy. However, high entropy does not necessarily mean that a feature is outside the pretraining distribution or poorly learned. The terminology therefore needs stronger evidence.

  5. DDMA uses text prototypes for feature splitting and projection, but how much of the improvement specifically depends on vision–language alignment? Could replacing text prototypes with classifier weights also improve TTA for non-VLM models?

  6. Lack of some discussions with related work, such as SSG-NeurIPS2025, BoostAdapter-NeurIPS2024, DPE-NeurIPS2024, and more.

Questions for authors

Please refer to "Critical Weaknesses".

4 comments
AuthorsOpenReview ↗

Responses to the Feedback of Reviewer 23YW

Thank you very much for your valuable suggestions on our work! You can find the responses to your questions as follows:

W1: We conduct experiments using four different random seeds. On the cross-domain and OOD benchmarks, the strongest baseline TT-RAA, achieves average accuracies of 67.45±0.5167.45 \pm 0.51 and 64.36±0.8364.36 \pm 0.83, while DDMA achieves 68.17±0.3568.17 \pm 0.35 and 65.57±0.4965.57 \pm 0.49. These results show that DDMA consistently achieves higher average accuracy than TT-RAA across both benchmark settings. Moreover, DDMA exhibits lower deviations, indicating more stable performance under varying test stream orders.

W2: The core of DDMA is its feature decomposition strategy, which splits the input representations into ID features and class-related OOD features based on the attention weights and their alignment with textual semantics. The rationale behind this decomposition is also supported by prior studies [a, b], which have investigated the crop and merging of visual features from similar perspectives and demonstrated the effectiveness of distinguishing features based on their semantic relevance and alignment.

Furthermore, we provide additional analyses in our manuscript (Tab. 1 and Tab. 2, and Fig. 2), demonstrating that both feature groups contain meaningful semantic information and exhibit clear class-wise clustering patterns in the embedding space. These observations provide the motivation for modeling the two feature groups separately: doing so helps mitigate interference between them while allowing the semantic information carried by each group to be fully exploited.

W3: The hyperparameters tuned separately for each dataset are limited to four parameters: α\alpha and β\beta in Eq. (11), and the weighting factors w1w_1 and w2w_2 used in computing pfinalp_{\text{final}} (L339).

As for the hyperparameter selection, we follow the configurations and parameter choices reported in existing TTA methods, such as TDA, DMN, and TT-RAA, and make minor adjustments based on their recommended values to accommodate differences in dataset distributions. For a fair comparison, we follow the original experimental settings and recommended configurations in their official code of the baseline methods.

AuthorsOpenReview ↗

Responses to the Feedback of Reviewer 23YW (2)

W4: We argue that the classification performance of CLIP can directly reflect how well visual features of inputs have been learned and aligned with textual semantics during pre-training. In CLIP-based classification task, the predictions rely on the alignment and similarity between image representations and their corresponding textual semantics. Indeed, the pre-training objective of CLIP is to align image and text embeddings in a shared embedding space. Therefore, features that are insufficiently learned and poorly aligned with textual semantics are expected to result in less reliable predictions, characterized by lower classification accuracy and higher prediction entropy. This idea can also be supported by many current works, e.g. [c] and [d].

Our experimental results in Sec. 3 (Tab. 1) reveal that token embeddings with higher token–text prediction entropy (selected by M2{M}_2) consistently exhibit lower classification accuracy and higher prediction entropy in classification. These results directly indicate that the features in these tokens are insufficiently learned and poorly aligned with textual semantics during CLIP pre-training, which are identified as class-related OOD features in our work.

W5: DDMA is specifically designed for VLM adaptation, and its feature decomposition primarily relies on the degree of vision–language alignment. Specifically, DDMA leverages the similarity between visual and textual embeddings, together with the corresponding prediction entropy, to distinguish ID features from class-related OOD features. Therefore, the current DDMA framework cannot be directly applied to unimodal vision classifier.

Nevertheless, the underlying idea of token-level feature decomposition is not limited to VLMs and could potentially be extended to unimodal models. For example, alternative criteria, such as attention weights and interaction patterns among visual tokens, could be explored to distinguish features with different semantic relevance or learning characteristics. We believe this is a promising direction and would be happy to investigate its potential in future work.

W6: The baselines selected in our experiments primarily include representative and SOTA methods for VLM TTA. Following the reviewer's suggestion, we have additionally implemented and evaluated two recent methods, DOTA (NeurIPS 2025) and ADAPT (NeurIPS 2025). The corresponding experimental results are reported in Tab. R1.

Table R1: Comparison results among DOTA, ADAPT and our method on several benchmarks.

AircraftCaltech101CarsFlowerPetsImageNet-AImageNet-V2
DOTA24.1894.5367.8471.6287.5854.9361.28
ADAPT24.4794.7667.0970.2487.4556.6861.92
DDMA (Ours)25.2994.2566.3272.5389.5761.4664.50

Compared with DOTA and ADAPT, our method achieves slightly lower accuracy on Caltech101 and Stanford Cars, but outperforms them on the remaining benchmarks, with particularly notable gains on ImageNet-A and ImageNet-V2. We will include the complete experimental results in the revised version.

[a] Is less more? exploring token condensation as training-free test-time adaptation. ICCV 2025.

[b] From local details to global context: advancing vision-language models with attention-based selection. ICML 2025

[c] Test-time retrieval-augmented adaptation for vision-language models. ICCV 2025.

[d] BaFTA: Backprop-free test-time adaptation for zero-shot vision-language models. arxiv 2024.

AuthorsOpenReview ↗

Kindly refer to our rebuttal and revised manuscript

Dear Reviewer 23YW,

Considering the deadline of discussion period is approaching, we would like to kindly draw your attention to our rebuttal and revised manuscript, in which we have provided detailed responses to the concerns raised in your review.

Thank you for your time and consideration.

Dear Reviewer 23YW

Thank you for your thoughtful feedback and time. As the discussion period is drawing to a close, we would appreciate hearing your thoughts on our responses. If you feel that they have addressed your concerns, we would be grateful if you could consider to update your rating. We are happy to clarify any remaining points. Thank you again for your consideration.

Best regards

The Authors

Reviewer_JzHJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that existing training-free TTA methods rely on holistic image representations and thus struggle to extract semantically richer features. To address this, the paper decomposes visual features into ID features and class-related OOD features according to their degree of alignment with textual semantics, and proposes DDMA: at test time, it separately models the class distributions of the two types of features and uses the two sets of distribution statistics to jointly correct predictions. On cross-domain and OOD benchmarks, the proposed method shows certain advantages.

Critical strengths
  1. The paper is logically and clearly written.

  2. The figures and tables are clear and easy to understand.

Critical weaknesses
  1. There is a lack of in-depth analysis of the source of the method’s gains. The authors provide Table 6 to verify the effectiveness of their feature decomposition; however, their method contains many design elements, not just feature decomposition. We cannot know how much other factors beyond “feature decomposition” affect performance. On the other hand, the results in Table 6 are not that strong; it can be seen that on the four datasets selectively reported by the authors, the effects are not very significant.

  2. How do the authors view the relationship between the current method and the core problem of TTA? The feature decomposition proposed in this paper seems to be a common and general technique rather than something specific to the TTA task.

  3. At L368, the authors mention that “The remaining hyper-parameters are tuned separately for each dataset.” I am concerned about how many hyperparameters here have been tuned specifically. Also, considering the “test-time” condition, such tuning seems unreasonable under the TTA task setting.

  4. The core motivation of this paper may not hold. At L71, the authors point out what they regard as a common limitation of TTA methods: “existing training-free TTA methods share a common limitation.” It can be seen that this is their common “practice,” but there is no clear evidence that this is a “limitation.”

  5. There are highly related methods that are not compared. The authors cite ADAPT at L131 but do not compare with it later; I hope the authors can provide an explanation.

  6. The authors’ statements lack rigor. The authors mention (L360): “All baselines are implemented using their official open-source code and evaluated with a batch size of 1.” However, many of their results are identical digit by digit to those in the original papers, which should have been copied.

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Responses to the Feedback of Reviewer JzHJ

Thank you very much for your valuable suggestions on our work! You can find the responses to your questions as follows:

W1: DDMA separately model the distributions of ID and class-related OOD features after feature decomposition, and leverage the corresponding distribution statistics to refine the original predictions. In addition, DDMA incorporates a projection module to mitigate the modality gap between image and text representations.

To more clearly demonstrate the contribution of feature decomposition from that of the projection module, we conduct an additional ablation study by removing the projection module while keeping the other components unchanged. This allows us to evaluate the effectiveness of feature decomposition independently of the projection module. The results in four selected datasets are reported in Tab. R1.

Table R1: Ablation study on our feature decomposition without the projection module.

CarsPetsImageNet-AImageNet-V2
Holistic58.4480.0846.9054.72
ID + C-OOD Feat.61.6983.3150.2759.98

The results show that, without the projection module, our feature decomposition strategy achieves more substantial improvements over holistic representations on the four selected benchmarks, with a particularly notable gain of approximately 5% on ImageNet-V2. This can demonstrate the effectiveness of the proposed feature decomposition. We will update these results in the revised version.

W2: The core of DDMA is token-level feature decomposition, which offers two key advantages that make it particularly suitable for TTA. First, the proposed decomposition introduces only limited computational overhead and does not require backpropagation or parameter updates. This makes it well suited to online TTA under real-time constraints.

Second, it enables fine-grained exploitation of token embeddings by separating class-related OOD features that are insufficiently learned by the VLM from well-aligned ID features. This allows DDMA to mitigate the interference of insufficiently learned features with ID features while exploiting the valuable semantic information in these class-related OOD features. Consequently, DDMA can make more effective use of test-time information for prediction refinement.

Therefore, its integration into DDMA is specifically motivated by the efficiency and adaptation requirements of TTA. In particular, DDMA leverages the feature decomposition to address the limitations of existing TTA methods that rely on holistic image representations, enabling more effective and efficient test-time adaptation.

W3: The hyperparameters tuned separately for each dataset are limited to four parameters: α\alpha and β\beta in Eq. (11), and the weighting factors w1w_1 and w2w_2 used in computing pfinalp_{\text{final}} (L339).

We would also like to clarify that dataset-specific hyperparameter tuning is a common practice in existing TTA methods. For example, prior works such as TDA, DMN, and TT-RAA also adjust their hyperparameters across different datasets to account for variations in data distributions. This is because the optimal parameter configurations may depend on the characteristics of each test distribution. Importantly, our hyperparameter selection does not rely on test labels, thus preserving the label-free nature of the TTA setting. Therefore, our experimental protocol is consistent with common practices in existing TTA studies.

AuthorsOpenReview ↗

Responses to the Feedback of Reviewer JzHJ (2)

W4: We would like to clarify that our motivation is not merely based on observing a "common practice" in existing methods. In fact, prior works [a, b] have also pointed out that existing TTA methods typically rely on holistic image embeddings, which limits their ability to explore fine-grained visual representations and perform more detailed token-level analysis.

Although TCA [a] further explores token-level adaptation by selecting the most semantically informative tokens, it handles semantically ambiguous tokens (i.e., token embeddings that are insufficiently learned and poorly aligned) by directly merging or removing. Such a strategy may fail to fully exploit the potentially valuable semantic information contained in these tokens, particularly in OOD scenarios where such features are more prevalent, as reflected by its relatively weaker performance on OOD benchmarks such as ImageNet-A.

As for this work, based on the token-level perspective, our work takes a step further by explicitly decomposing visual features into well-aligned ID features and insufficiently aligned yet semantically meaningful class-related OOD features. Then DDMA separately models the distributions of these two feature types, allowing both to contribute to test-time prediction refinement while mitigating their mutual interference.

Therefore, our motivation is to address the limitations of holistic representation-based TTA methods and the insufficient exploitation of potentially informative token-level features, rather than simply proposing token-level processing as a new technique.

W5: ADAPT models the distribution of embeddings based on conditional probabilities and refines the prediction probabilities accordingly. However, it relies on an important assumption that the test-time class distribution is uniform, i.e., P(y=yi)=1/KP(y=y_i)=1/K, where KK denotes the number of classes. This assumption may not hold in practical online TTA scenarios, where the class distribution of incoming test samples can be arbitrary and dynamically changing. Consequently, the applicability of ADAPT may be limited under such settings.

Following the reviewer's suggestion, we have additionally evaluated ADAPT on several benchmarks. The corresponding results are reported in Tab. R2.

Table R2: Comparison results between ADAPT and our method on several benchmarks.

AircraftCaltech101CarsFlowerPetsImageNet-AImageNet-V2
ADAPT24.4794.7667.0970.2487.4556.6861.92
DDMA (Ours)25.2994.2566.3272.5389.5761.4664.50

Compared with ADAPT, our method achieves slightly lower accuracy on Caltech101 and Stanford Cars, but outperforms ADAPT on the remaining benchmarks, with particularly notable gains on ImageNet-A and ImageNet-V2. We will include the complete experimental results in the revised manuscript.

W6: Thank the review for pointing out this. We would like to clarify that, for prompt-tuning-based methods, we cite the results from current works [a, b, c], as their performance is relatively insensitive to random seed variations and deployment configuration. In contrast, cache-based and distribution-based methods are more sensitive to experimental environments and seed settings. Therefore, we reproduce these methods using their official implementations, which leads to the difference between our reproduced results and those reported in the original papers. We will further correct our statement in the revised manuscript to avoid any potential ambiguity.

[a] Is less more? exploring token condensation as training-free test-time adaptation. ICCV 2025.

[b] Test-time retrieval-augmented adaptation for vision-language models. ICCV 2025.

[c] Backpropagation-free test-time adaptation via probabilistic gaussian alignment. NeurIPS 2025.

AuthorsOpenReview ↗

Kindly refer to our rebuttal and revised manuscript

Dear Reviewer JzHJ,

Considering the deadline of discussion period is approaching, we would like to kindly draw your attention to our rebuttal and revised manuscript, in which we have provided detailed responses to the concerns raised in your review.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.