- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ACME, a training-free test-time adaptation method for vision-language models. It estimates the online cross-covariance between frozen CLIP predictions and visual features, using the resulting class-specific correlations to correct the original predictions. Experiments demonstrate promising accuracy improvements while maintaining low inference overhead and memory consumption.
Critical strengths
- ACME requires no backpropagation or historical sample storage, introducing minimal additional inference time and memory overhead.
- Experiments cover multiple cross-domain and out-of-distribution benchmarks, different visual backbones, and various test stream orders.
Critical weaknesses
-
Limited novelty in motivation. The motivation of ACME substantially overlaps with that of PTA, which already accumulates visual features using soft class-wise prediction probabilities without storing historical samples. Although ACME replaces prototype updates with posterior–feature cross-covariance estimation, the paper does not sufficiently explain how this reformulation addresses the limitations of prototype-based methods, particularly the influence of incorrect predictions. Furthermore, the mathematical relationship between cross-covariance estimation and centered soft prototypes raises additional concerns about the methodological novelty.
-
Insufficient justification of the correction mechanism. ACME derives its correction scores from the correlation between frozen CLIP predictions and visual features. However, such correlations do not necessarily provide reliable signals for correcting classification errors, particularly when the frozen model exhibits systematic biases.
-
Limited adaptation to Non-IID streams. ACME accumulates statistics over the entire test stream without an explicit forgetting mechanism. Consequently, outdated observations may continue to influence predictions when the target distribution changes over time. Its robustness to temporally evolving distribution shifts therefore remains unclear.
-
Limited improvements over existing methods. ACME achieves only a modest accuracy improvement over PTA (69.94% vs. 69.27%). Its efficiency advantage over PTA is also marginal, with testing times of 6.8 and 7.0 minutes, respectively. These results raise questions about the practical significance of the improvements given the methodological changes.
Questions for authors
-
How does ACME fundamentally differ from PTA, which already accumulates visual features using soft class-wise prediction probabilities? Can the authors provide a controlled comparison to demonstrate the unique contribution of cross-covariance estimation?
-
How does ACME perform when the target distribution changes over time? Can the authors provide experiments demonstrating its robustness to temporal distribution shifts?
-
Since ACME relies on frozen CLIP predictions to estimate cross-covariance, how does it prevent systematic prediction errors from biasing the adaptation statistics?
10 comments
Response to Reviewer H8Wr (2/4)
W2/Q3: How does ACME correct frozen CLIP errors without reinforcing them?
Response: In test-time adaptation, using the baseline model's predictions to guide adaptation is a standard practice. ACME's correction additionally uses posterior-feature associations accumulated across the target stream. It uses the complete soft posterior rather than hard pseudo-labels, so a wrong top-1 prediction does not become a hard assignment of the sample to the wrong class, as can happen in cache-based methods such as TDA. This allows smooth accumulation of class-wise evidence across the stream. Crucially, ACME's corrected predictions are never fed back into its statistics, avoiding recursive reinforcement of the adapter's own errors. Bounded corrections and the maturity weight further reduce the risk of over-correction. These safeguards mitigate error amplification; they do not imply that the statistics are free of frozen-model bias.
In our main experiments, ACME improves over frozen CLIP on all cross-domain and OOD datasets. ACME also remains effective on tasks with very low zero-shot performance. For example, on EuroSAT and DTD, where zero-shot accuracy is below 50%, ACME improves accuracy from 48.46% to 62.46% (+14.00%) and 44.21% to 48.53% (+4.32%), respectively.
W1: Centered soft-prototype equivalence.
Response: Thank you for pointing out this connection. We agree that ACME's cross-covariance readout has an exact centered soft-prototype interpretation. Let and denote the frozen feature and posterior probability for class , and let and denote their empirical means. If we define the posterior-weighted prototype as
=\frac{\sum_{i=1}^t p_{i,c}\mathbf{x}_i}{\sum_{i=1}^t p_{i,c}},$$ then, for $\bar p_{t,c}>0$, the empirical cross-covariance satisfies $$\mathbf{C}^{px}_{t,c} =\frac{1}{t}\sum_{i=1}^t (p_{i,c}-\bar p_{t,c})(\mathbf{x}_i-\boldsymbol{\mu}_t^x)^\top =\bar p_{t,c}\left(\boldsymbol{\mu}^{\mathrm{soft}}_{t,c}-\boldsymbol{\mu}_t^x\right)^\top.$$ Since the positive scaling factor $\bar p_{t,c}$ cancels under cosine normalization, the resulting readout is mathematically equivalent to a **centered, variance-standardized soft prototype**. However, cross-covariance provides a principled, correlation-based interpretation of the readout: it measures how deviations in class evidence co-vary with deviations in visual features, with centering following directly from the covariance definition. This motivates ACME's formulation through standard statistical moments, while the prototype identity provides an equivalent geometric interpretation. To our knowledge, prior online VLM TTA methods have not explored this posterior-feature cross-covariance readout, equivalently expressed through centered, variance-standardized soft prototypes. This equivalent interpretation therefore does not make ACME the same adaptation method as existing approaches. PTA uses confidence-dependent EMA prototype updates and text interpolation, controlled by an update temperature and an interpolation weight. These tuned hyperparameter settings may not transfer equally well to unseen datasets. Its EMA prototypes generally differ from the empirical $\boldsymbol{\mu}^{\mathrm{soft}}_{t,c}$ above, and it does not center and variance-standardize the query and class directions as ACME does. DPE constructs visual prototypes from pseudo-labeled, entropy-selected feature queues and evolves textual prototypes with optimized residuals. ProtoMM[1] (ICCV'25) updates multimodal prototype distributions through optimal transport, while MCP and MCP++ [2] (ICCV'25) construct and refine prototypes using multiple feature caches. DOTA estimates posterior-weighted class means and class-specific feature-covariance matrices for Gaussian classification. These methods use different estimators or prediction rules from ACME's standardized cross-covariance correction. ACME maintains **exact empirical means, coordinate-wise variance, and posterior-feature cross-covariance** through streaming recurrences over the complete frozen posterior, with no selected update rate, decay parameter, or confidence threshold. The resulting standardized scores provide a bounded, maturity-weighted correction to frozen logits without target-specific hyperparameter tuning. We have added the exact soft-prototype equivalence and clarified the distinctions from PTA in **Section 3.4 (lines 263–277)**, with an expanded related-work discussion in **Section 2.2 (lines 173–182)** of the revised manuscript. [1] Dynamic Multimodal Prototype Learning in Vision-Language Models. ICCV 2025. [2] Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Models. ICCV 2025.Response to Reviewer H8Wr (3/4)
W3/Q2: Robustness to non-IID test streams.
Response: We sincerely appreciate this insightful suggestion. It motivated us to add experiments under non-IID test streams, where consecutive samples are class-correlated and class proportions vary over time.
We evaluate TDA, PTA under both its published default (, ) and conservative non-IID (, ) configurations, and ACME on ImageNet and ten cross-domain datasets using CLIP ViT-B/16 and batch size one. Following StatA's Dirichlet-based stream construction [1], we evaluate Low (), Medium (), and High () temporal class correlation; smaller produces stronger correlation. We also evaluate Separate, where classes arrive sequentially. ACME retains cumulative statistics without explicit forgetting and uses its unchanged submitted configuration throughout.
Accuracy (%); Avg. is the unweighted mean across the 11 datasets. PTA-D and PTA-C denote the default and conservative configurations, respectively. Bold marks the highest adaptation accuracy in each column. Frozen CLIP is shown only in Low because its predictions and 66.00% average are unchanged by stream order.
Low (0.1)
| Method | ImageNet | Flowers | DTD | Pets | Cars | UCF | Caltech | Food | SUN | Aircraft | EuroSAT | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Frozen CLIP | 68.78 | 70.77 | 44.21 | 89.13 | 66.22 | 67.57 | 93.87 | 85.88 | 66.26 | 24.96 | 48.35 | 66.00 |
| TDA | 69.60 | 71.50 | 45.74 | 89.48 | 67.28 | 71.37 | 94.08 | 86.31 | 67.61 | 25.59 | 61.86 | 68.22 |
| PTA-D | 70.16 | 74.58 | 47.75 | 90.19 | 68.72 | 73.06 | 94.65 | 85.90 | 69.08 | 25.38 | 54.48 | 68.54 |
| PTA-C | 69.37 | 72.92 | 44.74 | 90.52 | 67.85 | 71.66 | 94.52 | 86.25 | 68.18 | 21.90 | 59.17 | 67.92 |
| ACME | 70.84 | 74.62 | 47.99 | 90.00 | 69.46 | 73.54 | 94.00 | 86.06 | 69.77 | 27.36 | 59.27 | 69.36 |
Medium (0.01)
| Method | ImageNet | Flowers | DTD | Pets | Cars | UCF | Caltech | Food | SUN | Aircraft | EuroSAT | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TDA | 69.47 | 72.23 | 45.80 | 89.70 | 67.28 | 70.71 | 93.83 | 86.08 | 67.65 | 26.22 | 50.32 | 67.21 |
| PTA-D | 69.86 | 73.16 | 46.93 | 88.63 | 67.48 | 70.98 | 93.59 | 84.24 | 68.34 | 25.17 | 46.77 | 66.83 |
| PTA-C | 69.47 | 72.43 | 44.80 | 89.37 | 68.01 | 70.34 | 94.24 | 85.72 | 68.35 | 22.44 | 59.86 | 67.73 |
| ACME | 70.19 | 73.12 | 47.40 | 87.84 | 67.57 | 71.79 | 92.41 | 84.86 | 68.54 | 26.07 | 60.07 | 68.17 |
High (0.001)
| Method | ImageNet | Flowers | DTD | Pets | Cars | UCF | Caltech | Food | SUN | Aircraft | EuroSAT | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TDA | 69.59 | 72.47 | 45.33 | 89.04 | 66.99 | 69.55 | 94.12 | 85.80 | 67.47 | 26.04 | 48.06 | 66.77 |
| PTA-D | 68.63 | 72.31 | 46.16 | 87.44 | 65.84 | 68.65 | 93.43 | 83.25 | 66.58 | 25.11 | 48.88 | 66.02 |
| PTA-C | 69.64 | 72.35 | 45.74 | 88.91 | 66.98 | 69.34 | 94.48 | 85.52 | 68.24 | 21.87 | 52.12 | 66.84 |
| ACME | 69.57 | 71.99 | 47.12 | 87.28 | 67.11 | 69.35 | 91.81 | 84.84 | 67.51 | 27.09 | 50.23 | 66.72 |
Separate
| Method | ImageNet | Flowers | DTD | Pets | Cars | UCF | Caltech | Food | SUN | Aircraft | EuroSAT | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TDA | 69.10 | 71.25 | 46.04 | 89.45 | 67.47 | 70.66 | 93.91 | 86.02 | 67.62 | 26.31 | 50.95 | 67.16 |
| PTA-D | 67.62 | 72.80 | 45.86 | 87.35 | 64.73 | 68.75 | 93.31 | 82.82 | 65.83 | 25.02 | 44.63 | 65.34 |
| PTA-C | 69.58 | 73.45 | 44.50 | 89.53 | 67.12 | 70.31 | 94.69 | 84.83 | 68.25 | 22.62 | 61.68 | 67.87 |
| ACME | 68.89 | 73.11 | 46.43 | 87.74 | 66.32 | 69.76 | 90.74 | 84.92 | 67.53 | 27.25 | 61.01 | 67.61 |
ACME improves average accuracy over frozen CLIP in all four settings. In Separate, PTA's default configuration falls below frozen CLIP (65.34% versus 66.00%), whereas ACME reaches 67.61%. With its non-IID-specific conservative configuration, PTA recovers to 67.87% in Separate and slightly exceeds ACME in High and Separate. These results demonstrate strong empirical robustness to the evaluated non-IID streams, with consistent average improvements over frozen CLIP and no change to ACME's configuration. They support the usefulness of cumulative moments under temporal changes in class composition.
We have incorporated these experiments into Appendix A.2 of the revised manuscript, with results for all four stream settings reported in Table 9.
[1] Realistic Test-Time Adaptation of Vision-Language Models. CVPR 2025.
Response to Reviewer H8Wr (1/4)
Thank you for your insightful comments and for recognizing ACME's lightweight design and the breadth of our evaluation. We provide our point-by-point responses below and hope they address your concerns.
W1/Q1: Motivation and methodological differences from PTA.
Response: ACME is a statistics-based adapter, with SCA [NeurIPS’25] and DOTA [NeurIPS’25] as key methodological comparators within this family. Our central motivation is to retain useful target-domain structure through compact streaming statistics, without full feature-covariance matrices, matrix inversion, or target-specific hyperparameter tuning. PTA remains an important lightweight baseline. Our contribution concerns the statistics retained, their exact streaming updates, and the resulting accuracy-efficiency trade-off.
PTA uses heuristic confidence-weighted EMA updates and text interpolation, which depend on an update temperature and an interpolation weight. ACME instead maintains exact empirical feature and posterior means, coordinate-wise feature variance, and posterior-feature cross-covariance through exact streaming recurrences. It uses the complete frozen posterior without confidence filtering, and its moment updates require no selected learning rate or decay parameter.
PTA's hyperparameter choices are assessed on ImageNet and the standard cross-domain benchmarks, but settings effective there may not transfer equally well to other target distributions. To examine this transfer, we additionally evaluate CIFAR-100-C, CIFAR-10-C, and SVHN using CLIP ViT-B/16 and identical stream order across methods. PTA retains its published configuration, h=20 and w=0.01; ACME is unchanged. Neither method is retuned on these targets. Both CIFAR benchmarks use all 15 standard corruptions at severity 5; SVHN uses the complete official test split.
CIFAR-100-C: all 15 corruptions at severity 5, accuracy (%).
CIFAR-10-C: all 15 corruptions at severity 5, accuracy (%).
Additional benchmarks: SVHN and ImageNet-C.
ImageNet-C reports the submitted single Vanilla prompt results, averaged over all 15 corruptions at severity 5 (see Appendix A.8, Table 14 of the revised manuscript).
On CIFAR-100-C, CIFAR-10-C, and SVHN, PTA provides only marginal improvements over frozen CLIP, whereas ACME improves accuracy by 2.61%, 5.66%, and 4.63%, respectively. ACME exceeds PTA by 1.93%, 3.57%, and 3.68%, and outperforms PTA on all 15 corruptions in both CIFAR benchmarks. These results demonstrate stronger transfer gains on the additional benchmarks without target-specific tuning.
In the revised manuscript, we have added these transfer experiments and discussion to Section 4.2 (lines 415–429), Table 3, with detailed per-corruption results in Appendix A.7, Table 13.
On ImageNet-C with a single Vanilla prompt, ACME achieves 28.08%, compared with 26.28% for PTA and 25.50% for frozen CLIP (see Appendix A.8, Table 14 of the revised manuscript), further demonstrating its effectiveness under severe corruption shifts.