- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets training-free, online test-time adaptation of CLIP. It argues that existing cache- and distribution-based methods rely on the holistic image embedding, in which poorly aligned features interfere with class semantics. Based on a token-masking analysis on ImageNet, the authors split high-attention patch tokens into "ID features" (tokens whose similarities to the class prompts have low entropy) and "class-related OOD features" (high entropy), and report that both groups cluster by class. The proposed method, DDMA, extracts the two token groups from confidently predicted test samples and projects their representations into the text-embedding subspace. It keeps a running class mean for each group and adds the similarities to these means to the zero-shot logits. With ViT-B/16 on 10 cross-domain datasets and on ImageNet with four variants, DDMA reports average gains of 3.37 and 4.02 points over CLIP and the best average accuracy among the 10 compared baselines, at a throughput comparable to other training-free methods. Ablations cover the feature decomposition, token selection, the projection and the hyperparameters, and ViT-B/32 results are given in the appendix.
Critical strengths
Training-free and efficient. DDMA needs no backpropagation and has no learnable parameters. Prior training-free methods use the holistic image embedding; DDMA instead models the class-wise distributions of two token-level feature types (ID and class-related OOD features) and uses them to correct the zero-shot logits. It runs at 29.37 items/s with 15.46 GFLOPs, comparable to TT-RAA and faster than the cache-based TDA and DMN on the same hardware.
Consistent gains over CLIP across 15 benchmarks. With ViT-B/16, DDMA improves on zero-shot CLIP on cross-domain datasets and ImageNet sets. The gains also hold with ViT-B/32. Against 10 baselines covering prompt-tuning, cache-based and distribution-based methods, it reaches the highest average accuracy on both benchmark groups.
Critical weaknesses
W1. The gain seems to come from the projection rather than from the dual decomposition. The text-subspace projection gets one sentence in Sec. 3.3, is described only in App. C. Without it, DDMA falls below zero-shot CLIP on 3 of 4 datasets (Tab. 7: Cars 61.69 vs 66.11, Pets 80.31 vs 86.92, ImageNet-V2 59.98 vs 61.88). The dual decomposition, by contrast, adds only +0.22 / +0.86 / +3.01 / +0.81 over a holistic prototype (Tab. 6). The paper has no ablation crossing holistic vs. dual with/without projection, and does not apply the projection to TDA or TT-RAA. Without these, I cannot tell how much of the improvement is due to the paper's main idea.
W2. The "interference" motivation is never tested. The split is motivated by the claim that OOD features interfere with class semantics in holistic representations, but no experiment measures this. In Tab. 6, neither branch alone beats the holistic prototype everywhere: ID-only is lower on ImageNet-A, and C-OOD-only is lower on Cars and Pets. Only the combination wins. The combined setting also has one more correction term with its own tuned weight. A capacity-matched control (e.g., holistic + ID, or two holistic branches) is needed before the gain can be attributed to the separation.
W3. Observations I and II lack proper controls. Keeping the top-80% attention tokens lowers accuracy by 3.3 points and changes entropy by only 0.015. I would read that as information loss rather than the "reduced interference" claimed in Observation I. The M̄1 collapse compares 20% of the tokens against 80%, and no mask is compared with a random mask of the same size. For Observation II, nothing tests whether M̄2 tokens are "insufficiently learned during pre-training", and 23.45% accuracy is just as consistent with weakly informative background tokens.
W4. Several decomposition and modelling choices are not justified or ablated. Max-over-heads for M1 is motivated by head variance, while the analysis that defines the token groups uses mean-over-heads, and Fig. 4 compares the two on only four datasets, with differences of about 0.5 points or less on the cross-domain ones. The entropy criterion for M2 is not compared with simpler splits such as max cosine, attention rank or random, and the masking layer () is not ablated. Finally, each "distribution" is just a class mean. Although the method is framed as distribution modelling, it is not compared with the Gaussian models of DOTA or ADAPT.
W5. No comparison with recent TTA methods. Although the abstract and contributions claim that DDMA outperforms state-of-the-art TTA methods, the paper does not compare with recent existing TTA methods, such as DOTA (NeurIPS 2025), ADAPT (NeurIPS 2025), MCP (ICCV 2025), ADTE (ICLR 2026) and MuRA (ACM MM 2026).
Questions for authors
Q1. Could you report Tabs. 3–4 with a single configuration chosen without target labels (e.g., on a held-out split or a separate source dataset), averaged over at least three stream orders?
Q2. On ImageNet, each of the 1000 class centres is initialised from a single unfiltered sample. How does accuracy evolve along the stream there? Is there an early phase in which DDMA is below zero-shot CLIP?
Q3. Does the GFLOPs estimate for DDMA (15.46 vs. 15.05 for CLIP) include the two extra passes through the last block and the per-token similarity to all class prompts?
Q4. Could you add the ablation crossing holistic vs. dual with/without projection on all 15 datasets, together with TDA and TT-RAA using the same projection?
6 comments
Responses to the Feedback of Reviewer WaPF (2)
W3: First, we argue that the classification performance of CLIP can directly reflect how well visual features of inputs have been learned and aligned with textual semantics during pre-training. In CLIP-based classification task, the predictions rely on the alignment and similarity between image representations and their corresponding textual semantics. Indeed, the pre-training objective of CLIP is to align image and text embeddings in a shared embedding space. Therefore, features that are insufficiently learned and poorly aligned with textual semantics are expected to result in less reliable predictions, characterized by lower classification accuracy and higher prediction entropy. This idea can also be supported by many current works, e.g. [c] and [d].
Regarding Observation I, we have additionally conducted experiments using a random mask that retains 20% tokens. The randomly selected tokens achieve a classification accuracy of 14.65% and a prediction entropy of 1.132. In comparison, the tokens selected by yield even lower accuracy (12.76%) and comparable prediction entropy, indicating that these low-attention tokens contain almost no meaningful class-related semantic information, rather than simply suffering from information loss due to a reduced number of tokens.
Furthermore, the performance gap between retaining high-attention tokens and using the random mask demonstrates that selecting tokens based on attention provides a more effective way to preserve semantically informative features. These results also suggest that incorporating low-attention tokens into the image representation may introduce noise and interfere with the contribution of more informative features, thereby motivating our attention-based token selection strategy.
As for Observation II, our experimental results reveal that token embeddings selected by consistently exhibit lower classification accuracy and higher prediction entropy in classification, but their classification accuracy is substantially higher than that of the tokens. This indicates that the tokens still preserve meaningful class-related semantic information. In addition, as discussed in L202 of the original manuscript, the tokens selected by and have comparable attention weights, yet the latter exhibit considerably higher prediction entropy. This further suggests that these tokens cannot simply be attributed to low attention or a lack of semantic information. Instead, these tokens contain potentially useful class-related features that are insufficiently learned and stably aligned with textual semantics during CLIP pre-training.
W4&W5:
-
Mean-over-heads is a commonly adopted practice in existing works. Therefore, we use it only in the exploratory experiments in Sec. 3, as it provides a more intuitive characterization. In contrast, in our proposed method, we adopt max-over-heads to preserve as many potentially informative tokens as possible, including those that may receive strong attention from only a subset of heads. The corresponding ablation results comparing the two aggregation strategies are presented in Fig. 4. Consistent with findings in existing studies in TTA [a, c, d], a modest improvement of about 0.5% can demonstrate the effectiveness of this design.
-
When distinguishing ID and class-related OOD features, our primary concern is whether the visual features can be consistently and reliably aligned with textual semantics. Therefore, we adopt prediction entropy as the criterion, as it directly reflects the stability of vision–language alignment. Furthermore, we have conducted additional experiments comparing entropy with two alternative criteria, Max Cosine and Attention Rank. The results are reported in Tab. R3.
Table R3: Comparison of different criterions for token selection.
Cars Pets ImageNet-A ImageNet-V2 Max Cos. 66.54 88.93 56.85 62.38 Atten. Rank 66.29 89.16 52.41 60.57 DDMA (Ours) 66.32 89.57 61.46 64.50 According to the results, although the performance differences among the three criteria are marginal on CD datasets such as Cars and Pets, Max Cosine and Attention Rank perform obviously worse than our method on OOD benchmarks, which demonstrates the superiority of our criterion. This is because Attention Rank cannot reflect vision–language alignment. Although Max Cosine can identify tokens with relatively high semantic alignment, it fails to characterize the stability of such alignment. Consequently, it remains limited when adapting to datasets containing a larger proportion of unseen and OOD features.
Responses to the Feedback of Reviewer WaPF (3)
W4&W5:
-
Our feature decomposition strategy follows the feature cropping techniques in prior works [b, c]. Accordingly, we operate on the features from the -th layer, which preserve richer and more comprehensive information from the input image than other layers. This design choice is recognized as a common and effective practice in existing studies.
-
DOTA and ADAPT models the distribution of embeddings based on conditional probabilities and refines the prediction probabilities accordingly. However, it relies on an important assumption that the test-time class distribution is uniform, i.e., , where denotes the number of classes. This assumption may not hold in practical online TTA scenarios, where the class distribution of incoming test samples can be arbitrary and dynamically changing. Consequently, the applicability of ADAPT may be limited under such settings.
Following the reviewer's suggestion, we have additionally implemented and evaluated two recent methods, DOTA (NeurIPS 2025) and ADAPT (NeurIPS 2025). The corresponding experimental results are reported in Tab. R4.
Table R4: Comparison results among DOTA, ADAPT and our method on several benchmarks.
Aircraft Caltech101 Cars Flower Pets ImageNet-A ImageNet-V2 DOTA 24.18 94.53 67.84 71.62 87.58 54.93 61.28 ADAPT 24.47 94.76 67.09 70.24 87.45 56.68 61.92 DDMA (Ours) 25.29 94.25 66.32 72.53 89.57 61.46 64.50 Compared with DOTA and ADAPT, our method achieves slightly lower accuracy on Caltech101 and Stanford Cars, but outperforms them on the remaining benchmarks, with particularly notable gains on ImageNet-A and ImageNet-V2. We will include the complete experimental results in the revised version.
Q2: We have evaluated the prediction accuracy of DDMA and zero-shot CLIP on ImageNet as the number of processed test samples increases. The results are presented in Tab. R5. During the early stage of the test stream, DDMA may achieve slightly lower accuracy than zero-shot CLIP because only a limited number of high-quality samples have been collected, and the class-wise feature distributions have not yet been sufficiently established. However, after 1000 test samples, the accumulated distribution information becomes more representative, leading to a noticeable improvement in prediction accuracy. Subsequently, DDMA's performance gradually stabilizes and maintains a clear advantage over zero-shot CLIP.
Table R5: The accuracy results during a test stream.
| 100 | 300 | 500 | 1000 | 2000 | 5000 | 10000 | |
|---|---|---|---|---|---|---|---|
| CLIP | 68.00 | 67.67 | 67.60 | 68.30 | 68.25 | 67.94 | 68.05 |
| DDMA | 61.00 | 65.33 | 66.80 | 68.90 | 69.40 | 71.48 | 70.23 |
Q3: When estimating the GFLOPs, the first attention blocks are executed only once, whereas the final attention block and the projection layer are executed twice to process the two separately selected token groups. Regarding the text embeddings of all classes, they are precomputed before the adaptation process begins and are therefore excluded from the GFLOPs calculation.
[a] Is less more? exploring token condensation as training-free test-time adaptation. ICCV 2025.
[b] From local details to global context: advancing vision-language models with attention-based selection. ICML 2025
[c] Test-time retrieval-augmented adaptation for vision-language models. ICCV 2025.
[d] BaFTA: Backprop-free test-time adaptation for zero-shot vision-language models. arxiv 2024.
Responses to the Feedback of Reviewer WaPF
Thank you very much for your valuable suggestions on our work! You can find the responses to your questions as follows:
W1&Q1&Q4: First, we would like to clarify that the projection module is specifically designed to mitigate the modality gap between visual and textual embeddings, enabling the distribution statistics of visual embeddings to be effectively compared with textual semantics for refining the original predictions.
However, this module is not directly applicable to other baseline methods due to differences in their adaptation mechanisms. Specifically, TDA performs prediction refinement based on similarities between visual embeddings, without requiring cross-modal similarity computation, and thus does not suffer from the same modality gap. Similarly, TT-RAA incorporates a specifically designed Multimodal Space Retrieval Augmentation mechanism to mitigate the modality gap. Therefore, applying our projection module to these methods would not necessarily provide additional benefits, as their respective adaptation mechanisms already avoid or explicitly address this issue.
Following the reviewer's suggestion, we have conducted additional ablation experiments without the projection module to more rigorously demonstrate the contribution of our dual feature decomposition strategy. The results are reported in Tab. R1, covering eight representative datasets due to the limited time during the rebuttal period. We will incorporate the complete results into the revised manuscript later.
Table R1: Ablation study on our feature decomposition without the projection module.
The additional results show that, without the projection module, our feature decomposition strategy achieves more substantial improvements over holistic representations, with a particularly notable gain on OOD benchmarks. This can demonstrate the effectiveness of the proposed feature decomposition.
W2: The results in Tab. 6 are intended to demonstrate the effectiveness of feature decomposition, rather than directly quantify the interference between the two feature groups. To provide a more direct investigation of the interference between ID and class-related OOD features, we conduct an additional experiment in which we combine half of the tokens with lower entropy from M2 and M2 respectively during the feature selection process. We also remove the projection module. The results, comparing this combined setting with the ID-only and class-related OOD-only settings, are reported in Tab. R2.
Table R2: Comparison of combined setting with the ID-only and C-OOD-only settings.
The results show that combining ID and C-OOD features leads to a substantial degradation in adaptation performance, particularly on OOD benchmarks. This indicates that mixing the two types of features leads to mutual interference, which motivates us to separate two types of features to avoid this interference.