acceptodds
Under review as a conference paper at ICLR 2027

ACME: Online Adaptation via Cross-Moment Estimation for Vision-Language Models

Abstract

Vision-language models (VLMs) provide strong zero-shot predictions, but their performance can deteriorate under test-time distribution shift. Test-time adaptation (TTA) aims to mitigate this degradation using unlabeled test samples during inference. Existing cache-based TTA methods retain confidence-selected examples or hard pseudo-labels, which can discard useful target evidence and propagate unreliable predictions. Distributional methods instead estimate target-domain feature distributions, but often require costly covariance state or task-sensitive hyperparameters whose improper configuration can cause severe performance degradation. To address these issues, we introduce Adaptive Cross-Moment Estimation (ACME), a tuning-free and backpropagation-free online adapter that uses the complete frozen posterior of every test sample. ACME maintains exact streaming feature and posterior statistics together with their cross-covariance, from which it derives class-specific directions for a bounded residual correction to the frozen logits. ACME requires no source data, stored test samples, or full class-conditional covariance matrices. Extensive experiments across distribution-shift and cross-dataset benchmarks demonstrate that ACME achieves strong adaptation performance with a negligible additional GPU memory overhead of 3 MiB and the highest inference throughput among the evaluated TTA methods, retaining 88% of frozen CLIP's throughput on ImageNet-1K.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-Time Adaptation, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_H8WrOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ACME, a training-free test-time adaptation method for vision-language models. It estimates the online cross-covariance between frozen CLIP predictions and visual features, using the resulting class-specific correlations to correct the original predictions. Experiments demonstrate promising accuracy improvements while maintaining low inference overhead and memory consumption.

Critical strengths
  • ACME requires no backpropagation or historical sample storage, introducing minimal additional inference time and memory overhead.
  • Experiments cover multiple cross-domain and out-of-distribution benchmarks, different visual backbones, and various test stream orders.
Critical weaknesses
  • Limited novelty in motivation. The motivation of ACME substantially overlaps with that of PTA, which already accumulates visual features using soft class-wise prediction probabilities without storing historical samples. Although ACME replaces prototype updates with posterior–feature cross-covariance estimation, the paper does not sufficiently explain how this reformulation addresses the limitations of prototype-based methods, particularly the influence of incorrect predictions. Furthermore, the mathematical relationship between cross-covariance estimation and centered soft prototypes raises additional concerns about the methodological novelty.

  • Insufficient justification of the correction mechanism. ACME derives its correction scores from the correlation between frozen CLIP predictions and visual features. However, such correlations do not necessarily provide reliable signals for correcting classification errors, particularly when the frozen model exhibits systematic biases.

  • Limited adaptation to Non-IID streams. ACME accumulates statistics over the entire test stream without an explicit forgetting mechanism. Consequently, outdated observations may continue to influence predictions when the target distribution changes over time. Its robustness to temporally evolving distribution shifts therefore remains unclear.

  • Limited improvements over existing methods. ACME achieves only a modest accuracy improvement over PTA (69.94% vs. 69.27%). Its efficiency advantage over PTA is also marginal, with testing times of 6.8 and 7.0 minutes, respectively. These results raise questions about the practical significance of the improvements given the methodological changes.

Questions for authors
  • How does ACME fundamentally differ from PTA, which already accumulates visual features using soft class-wise prediction probabilities? Can the authors provide a controlled comparison to demonstrate the unique contribution of cross-covariance estimation?

  • How does ACME perform when the target distribution changes over time? Can the authors provide experiments demonstrating its robustness to temporal distribution shifts?

  • Since ACME relies on frozen CLIP predictions to estimate cross-covariance, how does it prevent systematic prediction errors from biasing the adaptation statistics?

10 comments
AuthorsOpenReview ↗

Response to Reviewer H8Wr (1/4)

Thank you for your insightful comments and for recognizing ACME's lightweight design and the breadth of our evaluation. We provide our point-by-point responses below and hope they address your concerns.

W1/Q1: Motivation and methodological differences from PTA.

Response: ACME is a statistics-based adapter, with SCA [NeurIPS’25] and DOTA [NeurIPS’25] as key methodological comparators within this family. Our central motivation is to retain useful target-domain structure through compact streaming statistics, without full feature-covariance matrices, matrix inversion, or target-specific hyperparameter tuning. PTA remains an important lightweight baseline. Our contribution concerns the statistics retained, their exact streaming updates, and the resulting accuracy-efficiency trade-off.

PTA uses heuristic confidence-weighted EMA updates and text interpolation, which depend on an update temperature and an interpolation weight. ACME instead maintains exact empirical feature and posterior means, coordinate-wise feature variance, and posterior-feature cross-covariance through exact streaming recurrences. It uses the complete frozen posterior without confidence filtering, and its moment updates require no selected learning rate or decay parameter.

PTA's hyperparameter choices are assessed on ImageNet and the standard cross-domain benchmarks, but settings effective there may not transfer equally well to other target distributions. To examine this transfer, we additionally evaluate CIFAR-100-C, CIFAR-10-C, and SVHN using CLIP ViT-B/16 and identical stream order across methods. PTA retains its published configuration, h=20h=20 and w=0.01w=0.01; ACME is unchanged. Neither method is retuned on these targets. Both CIFAR benchmarks use all 15 standard corruptions at severity 5; SVHN uses the complete official test split.

CIFAR-100-C: all 15 corruptions at severity 5, accuracy (%).

MethodGauss.ShotImpulseDefocusGlassMotionZoomSnowFrostFogBright.ContrastElasticPixel.JPEGAvg.
Frozen CLIP21.9323.0627.5142.4219.9742.3947.6348.4049.8141.4157.1434.1928.9724.5232.3236.11
PTA21.9223.4029.2444.0120.4842.8649.1948.8050.4642.0658.6734.0729.4524.8432.4136.79
ACME24.0326.0632.9646.3821.8645.1051.8050.1252.3743.4761.1334.5830.3626.7733.8938.73

CIFAR-10-C: all 15 corruptions at severity 5, accuracy (%).

MethodGauss.ShotImpulseDefocusGlassMotionZoomSnowFrostFogBright.ContrastElasticPixel.JPEGAvg.
Frozen CLIP40.5644.7856.6072.5144.0868.8974.6475.4778.9670.9385.5562.6355.9049.0859.4062.67
PTA42.2245.8860.0774.4846.0971.3277.3977.6479.5872.5287.2666.9857.9050.9660.9364.75
ACME49.9552.1564.1677.0348.5674.6980.5780.3681.9975.0789.1270.6260.2155.8464.4968.32

Additional benchmarks: SVHN and ImageNet-C.

ImageNet-C reports the submitted single Vanilla prompt results, averaged over all 15 corruptions at severity 5 (see Appendix A.8, Table 14 of the revised manuscript).

MethodSVHN (%)ImageNet-C, Vanilla (%)
Frozen CLIP31.3725.50
PTA32.3226.28
ACME36.0028.08

On CIFAR-100-C, CIFAR-10-C, and SVHN, PTA provides only marginal improvements over frozen CLIP, whereas ACME improves accuracy by 2.61%, 5.66%, and 4.63%, respectively. ACME exceeds PTA by 1.93%, 3.57%, and 3.68%, and outperforms PTA on all 15 corruptions in both CIFAR benchmarks. These results demonstrate stronger transfer gains on the additional benchmarks without target-specific tuning.

In the revised manuscript, we have added these transfer experiments and discussion to Section 4.2 (lines 415–429), Table 3, with detailed per-corruption results in Appendix A.7, Table 13.

On ImageNet-C with a single Vanilla prompt, ACME achieves 28.08%, compared with 26.28% for PTA and 25.50% for frozen CLIP (see Appendix A.8, Table 14 of the revised manuscript), further demonstrating its effectiveness under severe corruption shifts.

AuthorsOpenReview ↗

Response to Reviewer H8Wr (2/4)

W2/Q3: How does ACME correct frozen CLIP errors without reinforcing them?

Response: In test-time adaptation, using the baseline model's predictions to guide adaptation is a standard practice. ACME's correction additionally uses posterior-feature associations accumulated across the target stream. It uses the complete soft posterior rather than hard pseudo-labels, so a wrong top-1 prediction does not become a hard assignment of the sample to the wrong class, as can happen in cache-based methods such as TDA. This allows smooth accumulation of class-wise evidence across the stream. Crucially, ACME's corrected predictions are never fed back into its statistics, avoiding recursive reinforcement of the adapter's own errors. Bounded corrections and the maturity weight further reduce the risk of over-correction. These safeguards mitigate error amplification; they do not imply that the statistics are free of frozen-model bias.

In our main experiments, ACME improves over frozen CLIP on all cross-domain and OOD datasets. ACME also remains effective on tasks with very low zero-shot performance. For example, on EuroSAT and DTD, where zero-shot accuracy is below 50%, ACME improves accuracy from 48.46% to 62.46% (+14.00%) and 44.21% to 48.53% (+4.32%), respectively.

W1: Centered soft-prototype equivalence.

Response: Thank you for pointing out this connection. We agree that ACME's cross-covariance readout has an exact centered soft-prototype interpretation. Let xi\mathbf{x}_i and pi,cp_{i,c} denote the frozen feature and posterior probability for class cc, and let μtx\boldsymbol{\mu}_t^x and pˉt,c\bar p_{t,c} denote their empirical means. If we define the posterior-weighted prototype as

=\frac{\sum_{i=1}^t p_{i,c}\mathbf{x}_i}{\sum_{i=1}^t p_{i,c}},$$ then, for $\bar p_{t,c}>0$, the empirical cross-covariance satisfies $$\mathbf{C}^{px}_{t,c} =\frac{1}{t}\sum_{i=1}^t (p_{i,c}-\bar p_{t,c})(\mathbf{x}_i-\boldsymbol{\mu}_t^x)^\top =\bar p_{t,c}\left(\boldsymbol{\mu}^{\mathrm{soft}}_{t,c}-\boldsymbol{\mu}_t^x\right)^\top.$$ Since the positive scaling factor $\bar p_{t,c}$ cancels under cosine normalization, the resulting readout is mathematically equivalent to a **centered, variance-standardized soft prototype**. However, cross-covariance provides a principled, correlation-based interpretation of the readout: it measures how deviations in class evidence co-vary with deviations in visual features, with centering following directly from the covariance definition. This motivates ACME's formulation through standard statistical moments, while the prototype identity provides an equivalent geometric interpretation. To our knowledge, prior online VLM TTA methods have not explored this posterior-feature cross-covariance readout, equivalently expressed through centered, variance-standardized soft prototypes. This equivalent interpretation therefore does not make ACME the same adaptation method as existing approaches. PTA uses confidence-dependent EMA prototype updates and text interpolation, controlled by an update temperature and an interpolation weight. These tuned hyperparameter settings may not transfer equally well to unseen datasets. Its EMA prototypes generally differ from the empirical $\boldsymbol{\mu}^{\mathrm{soft}}_{t,c}$ above, and it does not center and variance-standardize the query and class directions as ACME does. DPE constructs visual prototypes from pseudo-labeled, entropy-selected feature queues and evolves textual prototypes with optimized residuals. ProtoMM[1] (ICCV'25) updates multimodal prototype distributions through optimal transport, while MCP and MCP++ [2] (ICCV'25) construct and refine prototypes using multiple feature caches. DOTA estimates posterior-weighted class means and class-specific feature-covariance matrices for Gaussian classification. These methods use different estimators or prediction rules from ACME's standardized cross-covariance correction. ACME maintains **exact empirical means, coordinate-wise variance, and posterior-feature cross-covariance** through streaming recurrences over the complete frozen posterior, with no selected update rate, decay parameter, or confidence threshold. The resulting standardized scores provide a bounded, maturity-weighted correction to frozen logits without target-specific hyperparameter tuning. We have added the exact soft-prototype equivalence and clarified the distinctions from PTA in **Section 3.4 (lines 263–277)**, with an expanded related-work discussion in **Section 2.2 (lines 173–182)** of the revised manuscript. [1] Dynamic Multimodal Prototype Learning in Vision-Language Models. ICCV 2025. [2] Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Models. ICCV 2025.
AuthorsOpenReview ↗

Response to Reviewer H8Wr (3/4)

W3/Q2: Robustness to non-IID test streams.

Response: We sincerely appreciate this insightful suggestion. It motivated us to add experiments under non-IID test streams, where consecutive samples are class-correlated and class proportions vary over time.

We evaluate TDA, PTA under both its published default (h=20h=20, w=0.01w=0.01) and conservative non-IID (h=1000h=1000, w=0.001w=0.001) configurations, and ACME on ImageNet and ten cross-domain datasets using CLIP ViT-B/16 and batch size one. Following StatA's Dirichlet-based stream construction [1], we evaluate Low (γ=0.1\gamma=0.1), Medium (γ=0.01\gamma=0.01), and High (γ=0.001\gamma=0.001) temporal class correlation; smaller γ\gamma produces stronger correlation. We also evaluate Separate, where classes arrive sequentially. ACME retains cumulative statistics without explicit forgetting and uses its unchanged submitted configuration throughout.

Accuracy (%); Avg. is the unweighted mean across the 11 datasets. PTA-D and PTA-C denote the default and conservative configurations, respectively. Bold marks the highest adaptation accuracy in each column. Frozen CLIP is shown only in Low because its predictions and 66.00% average are unchanged by stream order.

Low (0.1)

MethodImageNetFlowersDTDPetsCarsUCFCaltechFoodSUNAircraftEuroSATAvg.
Frozen CLIP68.7870.7744.2189.1366.2267.5793.8785.8866.2624.9648.3566.00
TDA69.6071.5045.7489.4867.2871.3794.0886.3167.6125.5961.8668.22
PTA-D70.1674.5847.7590.1968.7273.0694.6585.9069.0825.3854.4868.54
PTA-C69.3772.9244.7490.5267.8571.6694.5286.2568.1821.9059.1767.92
ACME70.8474.6247.9990.0069.4673.5494.0086.0669.7727.3659.2769.36

Medium (0.01)

MethodImageNetFlowersDTDPetsCarsUCFCaltechFoodSUNAircraftEuroSATAvg.
TDA69.4772.2345.8089.7067.2870.7193.8386.0867.6526.2250.3267.21
PTA-D69.8673.1646.9388.6367.4870.9893.5984.2468.3425.1746.7766.83
PTA-C69.4772.4344.8089.3768.0170.3494.2485.7268.3522.4459.8667.73
ACME70.1973.1247.4087.8467.5771.7992.4184.8668.5426.0760.0768.17

High (0.001)

MethodImageNetFlowersDTDPetsCarsUCFCaltechFoodSUNAircraftEuroSATAvg.
TDA69.5972.4745.3389.0466.9969.5594.1285.8067.4726.0448.0666.77
PTA-D68.6372.3146.1687.4465.8468.6593.4383.2566.5825.1148.8866.02
PTA-C69.6472.3545.7488.9166.9869.3494.4885.5268.2421.8752.1266.84
ACME69.5771.9947.1287.2867.1169.3591.8184.8467.5127.0950.2366.72

Separate

MethodImageNetFlowersDTDPetsCarsUCFCaltechFoodSUNAircraftEuroSATAvg.
TDA69.1071.2546.0489.4567.4770.6693.9186.0267.6226.3150.9567.16
PTA-D67.6272.8045.8687.3564.7368.7593.3182.8265.8325.0244.6365.34
PTA-C69.5873.4544.5089.5367.1270.3194.6984.8368.2522.6261.6867.87
ACME68.8973.1146.4387.7466.3269.7690.7484.9267.5327.2561.0167.61

ACME improves average accuracy over frozen CLIP in all four settings. In Separate, PTA's default configuration falls below frozen CLIP (65.34% versus 66.00%), whereas ACME reaches 67.61%. With its non-IID-specific conservative configuration, PTA recovers to 67.87% in Separate and slightly exceeds ACME in High and Separate. These results demonstrate strong empirical robustness to the evaluated non-IID streams, with consistent average improvements over frozen CLIP and no change to ACME's configuration. They support the usefulness of cumulative moments under temporal changes in class composition.

We have incorporated these experiments into Appendix A.2 of the revised manuscript, with results for all four stream settings reported in Table 9.

[1] Realistic Test-Time Adaptation of Vision-Language Models. CVPR 2025.

Reviewer_PktWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies online test-time adaptation for frozen vision-language models through Adaptive Cross-Moment Estimation (ACME), a backpropagation-free method that uses unlabeled test samples to improve classification performance. ACME incrementally maintains the cross-covariance between visual features and predicted class probabilities, along with the corresponding feature statistics. These statistics are used to compute class-specific correction scores, which are incorporated into the original CLIP logits through maturity-weighted fusion. The method requires neither stored test samples nor dataset-specific hyperparameter tuning. Experiments on cross-domain and ImageNet distribution-shift benchmarks demonstrate consistent improvements over frozen CLIP, with competitive accuracy against recent TTA methods. ACME also maintains near-frozen inference speed with minimal additional memory overhead.

Critical strengths

1.Computational efficiency. ACME provides a lightweight adaptation mechanism without backpropagation or storing historical test samples. Its compact statistical state introduces minimal GPU memory overhead while maintaining inference speed close to frozen CLIP. This makes the method practical for resource-constrained online adaptation. 2.Consistent empirical performance. The experiments cover a range of cross-domain and distribution-shift benchmarks, showing consistent improvements over frozen CLIP and competitive accuracy compared with recent TTA methods. The method also avoids dataset-specific hyperparameter tuning, which is a useful property for deployment across different target domains.

Critical weaknesses
  1. Limited methodological novelty. The proposed cross-moment formulation appears closely related to existing prototype-based adaptation methods. In particular, the posterior-feature cross-covariance can be rewritten as
\right)^\top$$ where $$\boldsymbol{\mu}_{t,c}^{\mathrm{soft}} = \frac{\sum_{i=1}^{t} p_{i,c}\mathbf{x}_i}{\sum_{i=1}^{t} p_{i,c}}$$ denotes the posterior-weighted class prototype. Since the positive scaling factor $\bar{p}_{t,c}$ cancels under cosine normalization, the resulting readout is mathematically equivalent to a centered, variance-standardized soft prototype similarity. Although the normalization and fusion strategies may offer practical benefits, the paper does not sufficiently clarify the methodological distinction from existing prototype-based approaches, particularly PTA. This raises concerns about the novelty of the central contribution. 2. Limited improvements over strong baselines. While ACME achieves competitive accuracy with low computational overhead, its improvements over recent methods are relatively modest. For example, the cross-domain average accuracy exceeds DOTA and PTA by only 0.26 and 0.67 percentage points, respectively. Moreover, its efficiency advantage over PTA is small, while its OOD accuracy remains below several competing methods. Although Appendix A.2 reports low variability across random test orders for ACME, corresponding variability or paired statistical comparisons with strong baselines are not provided, leaving the significance of the small performance gains unclear.Consequently, the experimental evidence does not fully establish a substantial advantage over the closest existing approaches.
Questions for authors

Given the mathematical similarities between ACME and existing prototype-based approaches, particularly PTA, what is the key methodological distinction of the proposed method?

6 comments
AuthorsOpenReview ↗

Response to Reviewer PktW (1/2)

Thank you for your constructive comments and for recognizing ACME's efficiency, consistent empirical performance, and deployment benefits. We provide our point-by-point responses below and hope they address your concerns.

W1/Q1: Centered soft-prototype interpretation and distinctions from prototype-based methods.

Response: Thank you for pointing out this connection. We agree that ACME's cross-covariance readout has an exact centered soft-prototype interpretation. Let xi\mathbf{x}_i and pi,cp_{i,c} denote the frozen feature and posterior probability for class cc, and let μtx\boldsymbol{\mu}_t^x and pˉt,c\bar p_{t,c} denote their empirical means. If we define the posterior-weighted prototype as

=\frac{\sum_{i=1}^t p_{i,c}\mathbf{x}_i}{\sum_{i=1}^t p_{i,c}},$$ then, for $\bar p_{t,c}>0$, the empirical cross-covariance satisfies $$\mathbf{C}^{px}_{t,c} =\frac{1}{t}\sum_{i=1}^t (p_{i,c}-\bar p_{t,c})(\mathbf{x}_i-\boldsymbol{\mu}_t^x)^\top =\bar p_{t,c}\left(\boldsymbol{\mu}^{\mathrm{soft}}_{t,c}-\boldsymbol{\mu}_t^x\right)^\top.$$ Since the positive scaling factor $\bar p_{t,c}$ cancels under cosine normalization, the resulting readout is mathematically equivalent to a **centered, variance-standardized soft prototype**. However, cross-covariance provides a principled, correlation-based interpretation of the readout: it measures how deviations in class evidence co-vary with deviations in visual features, with centering following directly from the covariance definition. This motivates ACME's formulation through standard statistical moments, while the prototype identity provides an equivalent geometric interpretation. To our knowledge, prior online VLM TTA methods have not explored this posterior-feature cross-covariance readout, equivalently expressed through centered, variance-standardized soft prototypes. This equivalent interpretation therefore does not make ACME the same adaptation method as existing approaches. PTA uses confidence-dependent EMA prototype updates and text interpolation, controlled by an update temperature and an interpolation weight. These tuned hyperparameter settings may not transfer equally well to unseen datasets. Its EMA prototypes generally differ from the empirical $\boldsymbol{\mu}^{\mathrm{soft}}_{t,c}$ above, and it does not center and variance-standardize the query and class directions as ACME does. DPE constructs visual prototypes from pseudo-labeled, entropy-selected feature queues and evolves textual prototypes with optimized residuals. ProtoMM[1] (ICCV'25) updates multimodal prototype distributions through optimal transport, while MCP and MCP++ [2] (ICCV'25) construct and refine prototypes using multiple feature caches. DOTA estimates posterior-weighted class means and class-specific feature-covariance matrices for Gaussian classification. These methods use different estimators or prediction rules from ACME's standardized cross-covariance correction. ACME maintains **exact empirical means, coordinate-wise variance, and posterior-feature cross-covariance** through streaming recurrences over the complete frozen posterior, with no selected update rate, decay parameter, or confidence threshold. The resulting standardized scores provide a bounded, maturity-weighted correction to frozen logits without target-specific hyperparameter tuning. We have added the exact soft-prototype equivalence and clarified the distinctions from PTA in **Section 3.4 (lines 263–277)**, with an expanded related-work discussion in **Section 2.2 (lines 173–182)** of the revised manuscript. **Transfer without target-specific tuning.** Existing prototype-based methods such as PTA depend on tuned hyperparameters whose settings may not transfer equally well to unseen datasets. To evaluate this transfer, we additionally test CIFAR-100-C, CIFAR-10-C, and SVHN using PTA's published configuration, without retuning either method. ACME exceeds PTA by 1.93%, 3.57%, and 3.68%, respectively, and outperforms PTA on all 15 corruptions in each CIFAR benchmark. In the revised manuscript, we have added these transfer experiments and discussion to **Section 4.2 (lines 415–429), Table 3**, with detailed per-corruption results in **Appendix A.7, Table 13**. The revised manuscript also reports the highest cross-domain average under all three prompt settings (**Appendix A.4, Table 11**), including 68.83% versus PTA's 67.60% with a single Vanilla prompt. On ImageNet-C with a Vanilla prompt, ACME reaches 28.08% versus PTA's 26.28% (**Appendix A.8, Table 14**; please see our response to Reviewer H8Wr for the detailed results). [1] Dynamic Multimodal Prototype Learning in Vision-Language Models. ICCV 2025. [2] Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Models. ICCV 2025.
AuthorsOpenReview ↗

Response to Reviewer PktW (2/2)

W2: Accuracy gains, efficiency, and variability across test-stream orders.

Response: We thank the reviewer for this comment. We would like to clarify that our baseline comparison covers a broad range of existing state-of-the-art methods.

Performance on the Task 1 and Task 2 benchmarks is approaching saturation, where the gap between top-tier methods is typically less than 0.5%.

ACME achieves the highest cross-domain average and ranks third in the OOD comparison. DOTA [NeurIPS’25] and BoostAdapter [NeurIPS’24] slightly exceed its OOD accuracy; however, as noted in the paper, these results follow their officially released implementations and per-dataset hyperparameter configurations. For a fixed-configuration evaluation, we now evaluate DOTA and BoostAdapter on the OOD datasets using their unchanged ImageNet configurations, without any tuning.

Accuracy (%) using CLIP ViT-B/16.

MethodImageNetImageNet-AImageNet-V2ImageNet-RImageNet-SketchAvg.
Frozen CLIP68.7950.6862.3677.6848.2661.55
DOTA70.6960.6364.8681.1150.9765.65
BoostAdapter69.3863.7965.2680.7650.5765.95
ACME70.7960.6764.8480.6251.6965.72

Under this fixed-configuration comparison, ACME slightly exceeds DOTA on the five-dataset average (65.72% versus 65.65%) and achieves comparable performance to BoostAdapter (65.95%). BoostAdapter's average advantage is primarily driven by ImageNet-A, where its lead is 3.12%. Across the four datasets excluding ImageNet-A, ACME averages 66.99%, compared with 66.49% for BoostAdapter.

ACME therefore provides comparable performance to state-of-the-art methods at substantially lower memory cost (only 3 MiB of additional GPU memory, compared with 78 MiB for BoostAdapter and 4,068 MiB for DOTA) and near-frozen inference speed (approximately 2× faster than BoostAdapter and 4.3× faster than DOTA; please see Table 4 in the revised manuscript).

Our results also show that full per-class feature-covariance matrices are not required for competitive adaptation. ACME uses global feature means and coordinate-wise variances for normalization, together with a compact C×DC\times D posterior-feature cross-covariance for class-specific associations. This reduces the statistical state from DOTA's O(CD2)O(CD^2) to O(CD)O(CD), without matrix inversion. Our contribution is this tuning-free online adaptation rule built from compact empirical moments and a standardized readout.

Consistency across test-stream orders. Below, we provide mean ± standard deviation for ACME and the baselines over ten random test-stream orders.

MethodCross-domain avg. (%), mean ± SDImageNet (%), mean ± SD
TDA67.92 ± 0.1169.48 ± 0.06
PTA69.30 ± 0.1370.11 ± 0.08
DOTA69.71 ± 0.0770.64 ± 0.07
BoostAdapter68.64 ± 0.1269.41 ± 0.07
ADAPT68.86 ± 0.1070.58 ± 0.07
ACME69.97 ± 0.0870.73 ± 0.07

ACME achieves the highest accuracy on both the cross-domain benchmark and ImageNet on all ten random test-stream orders, with substantially lower additional memory and inference cost than all the above baselines (Table 4 of the revised manuscript).

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer PktW,

Thank you again for your thoughtful review and valuable comments. As the discussion deadline approaches, we would appreciate your feedback on our responses and revised manuscript. Please let us know whether our clarifications address your concerns or whether any points require further explanation.

Best regards, Paper 5150 Authors

Reviewer_dyNaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Adaptive Cross-Moment Estimation (ACME), a tuning-free and backpropagation-free online adaptation method for frozen vision-language models. ACME uses the complete frozen posterior of each test sample to incrementally estimate posterior-feature cross-covariance and derives class-wise corrections through standardized cosine similarity, with a maturity weight to reduce unreliable early-stage adaptation. Moreover, ACME requires neither test-sample storage nor model updates and maintains only (O(CD)) memory. Extensive experiments demonstrate competitive performance and a strong accuracy-efficiency trade-off.

Critical strengths
  1. The paper targets an important deployment regime: tuning-free, backpropagation-free online adaptation of frozen VLMs, with explicit attention to computational and memory constraints.
  2. ACME maintains only streaming first-order statistics, coordinate-wise variance, and a (C\times D) posterior-feature cross-covariance. The resulting method is conceptually simple, easy to implement, does not store test examples, and introduces negligible memory overhead.
Critical weaknesses
  1. The same frozen CLIP prediction simultaneously generates the posterior being accumulated and provides the base logits being corrected. It is therefore unclear how ACME corrects systematic model errors rather than reinforcing them.
  2. The maturity weight has an intuitive sample-to-dimension interpretation, but there is no statistical derivation connecting it to the uncertainty of the cross-covariance estimator.
Questions for authors
  1. Since the cross-moment is constructed entirely from the frozen CLIP posterior, how does ACME correct systematic errors made by the frozen model?
  2. What is the fundamental difference between ACME and the "soft-posterior prototypical method with centering and variance normalization"?
  3. How sensitive is ACME to the initial variance estimate?
7 comments
AuthorsOpenReview ↗

Response to Reviewer dyNa (1/3)

Thank you for your insightful comments and for recognizing ACME's practical value and efficient streaming formulation. Below, we provide our point-by-point responses and hope these address your concerns.

W1/Q1: How does ACME correct frozen CLIP errors without reinforcing them?

Response: In test-time adaptation, using the baseline model's predictions to guide adaptation is a standard practice. We clarify that ACME uses the complete soft posterior rather than hard pseudo-labels, so a wrong top-1 prediction does not assign the entire sample to the wrong class, as can happen in cache-based methods such as TDA. This allows smooth accumulation of class-wise evidence across the stream. Crucially, ACME's corrected predictions are never fed back into its statistics, avoiding recursive reinforcement of the adapter's own errors. Bounded corrections and the maturity weight further reduce the risk of over-correction. In our main experiments, ACME improves over frozen CLIP on all cross-domain and OOD datasets. ACME also remains effective on tasks with very low zero-shot performance. For example, on EuroSAT and DTD, where zero-shot accuracy is below 50%, ACME improves accuracy from 48.46% to 62.46% (+14.00%) and 44.21% to 48.53% (+4.32%), respectively.

AuthorsOpenReview ↗

Response to Reviewer dyNa (2/3)

W2/Q3: Statistical role of the maturity weight and sensitivity to the initial variance estimate.

Response: We address these concerns together, as both relate to the reliability of early-stream statistics.

Maturity weight. We employ maturity as a heuristic warm-up to suppress corrections early in the stream. It is not derived from the uncertainty of the cross-covariance estimator. Our ablations show that maturity slightly improves the cross-domain average (Fig. 3b), provides considerable gains on Caltech101, and has a negligible effect on the longer ImageNet and Food101 streams:

DatasetWithout maturity (%)With maturity (%)
ImageNet70.7870.79
Food10186.3086.37
Caltech10193.7594.52
Cross-Domain Avg.69.8169.94

Sensitivity to the initial variance estimate. ACME computes the empirical variance directly from the observed stream. At the first update, the feature mean equals the first feature, so the variance recurrence gives

μ1x=x1,v1x=v0x+(0−v0x)=0.\boldsymbol{\mu}_1^x=\mathbf{x}_1,\qquad \mathbf{v}_1^x=\mathbf{v}_0^x+(0-\mathbf{v}_0^x)=\mathbf{0}.

Therefore, ACME is independent of the variance initialization under this recurrence, and subsequent estimates depend only on observed samples. For t < 2, ACME reduces to the frozen prediction. For numerical stability, we lower-bound each feature variance by a small constant, ε = 10⁻⁸.

Since the standardized readout is a cosine similarity, its absolute value is at most 1. Early in the stream, when the sample count is much smaller than the feature dimension, the logit correction is bounded by

∣ℓt,cfinal−ℓt,cCLIP∣≤τt2D,t≪D.\left|\ell_{t,c}^{\mathrm{final}}-\ell_{t,c}^{\mathrm{CLIP}}\right|\leq\frac{\tau t}{2D},\qquad t\ll D.

This suppresses the correction relative to its full scale and limits the influence of uncertain variance estimates early in the stream.

To evaluate early-stream sensitivity, we additionally perturb the stored variance immediately after the first observation, keeping the other statistics fixed. We set every coordinate of this first variance estimate to 0.1/D0.1/D, 1/D1/D, 10/D10/D, 100/D100/D, or 11. These settings cover values below and above the reference scale 1/D1/D, up to the coordinate-wise upper bound of 1 for unit-normalized features. The variance then follows the original streaming recurrence, so the added offset decreases as a/t\mathbf{a}/t:

v~tx=vtx+at.\widetilde{\mathbf{v}}_t^x=\mathbf{v}_t^x+\frac{\mathbf{a}}{t}.

Here, a\mathbf{a} is the vector inserted after the first observation. The following table reports the accuracy range (%) across all five settings for the first 50 and 100 samples of a fixed stream order:

DatasetFirst 50, no maturityFirst 50, maturityFirst 100, no maturityFirst 100, maturity
ImageNet74.00–76.0076.0070.0072.00
DTD40.0040.0045.0046.00
Food10178.00–80.0088.0085.00–86.0092.00
EuroSAT38.00–40.0036.0053.00–54.0050.00

With maturity, accuracy remains unchanged across all five initial variance settings for both the first 50 and 100 samples on all four datasets. Even without maturity, accuracy over the first 100 samples differs by only one correct prediction on Food101 and EuroSAT, and remains unchanged on ImageNet and DTD. These results show very low sensitivity to the tested early variance estimates, with maturity further stabilizing accuracy across the initializations.

We have incorporated this sensitivity analysis into Section 4.3 (lines 506–521) of the revised manuscript, with the results reported in Table 6.

AuthorsOpenReview ↗

Response to Reviewer dyNa (3/3)

Q2: Centered soft-prototype interpretation and distinctions from prototype-based methods.

Response: Thank you for pointing out this connection. We agree that ACME's cross-covariance readout has an exact centered soft-prototype interpretation. Let xi\mathbf{x}_i and pi,cp_{i,c} denote the frozen feature and posterior probability for class cc, and let μtx\boldsymbol{\mu}_t^x and pˉt,c\bar p_{t,c} denote their empirical means. If we define the posterior-weighted prototype as

=\frac{\sum_{i=1}^t p_{i,c}\mathbf{x}_i}{\sum_{i=1}^t p_{i,c}},$$ then, for $\bar p_{t,c}>0$, the empirical cross-covariance satisfies $$\mathbf{C}^{px}_{t,c} =\frac{1}{t}\sum_{i=1}^t (p_{i,c}-\bar p_{t,c})(\mathbf{x}_i-\boldsymbol{\mu}_t^x)^\top =\bar p_{t,c}\left(\boldsymbol{\mu}^{\mathrm{soft}}_{t,c}-\boldsymbol{\mu}_t^x\right)^\top.$$ Since the positive scaling factor $\bar p_{t,c}$ cancels under cosine normalization, the resulting readout is mathematically equivalent to a **centered, variance-standardized soft prototype**. However, cross-covariance provides a principled, correlation-based interpretation of the readout: it measures how deviations in class evidence co-vary with deviations in visual features, with centering following directly from the covariance definition. This motivates ACME's formulation through standard statistical moments, while the prototype identity provides an equivalent geometric interpretation. To our knowledge, prior online VLM TTA methods have not explored this posterior-feature cross-covariance readout, equivalently expressed through centered, variance-standardized soft prototypes. This equivalent interpretation therefore does not make ACME the same adaptation method as existing approaches. PTA uses confidence-dependent EMA prototype updates and text interpolation, controlled by an update temperature and an interpolation weight. These tuned hyperparameter settings may not transfer equally well to unseen datasets. Its EMA prototypes generally differ from the empirical $\boldsymbol{\mu}^{\mathrm{soft}}_{t,c}$ above, and it does not center and variance-standardize the query and class directions as ACME does. DPE constructs visual prototypes from pseudo-labeled, entropy-selected feature queues and evolves textual prototypes with optimized residuals. ProtoMM[1] (ICCV'25) updates multimodal prototype distributions through optimal transport, while MCP and MCP++ [2] (ICCV'25) construct and refine prototypes using multiple feature caches. DOTA estimates posterior-weighted class means and class-specific feature-covariance matrices for Gaussian classification. These methods use different estimators or prediction rules from ACME's standardized cross-covariance correction. ACME maintains **exact empirical means, coordinate-wise variance, and posterior-feature cross-covariance** through streaming recurrences over the complete frozen posterior, with no selected update rate, decay parameter, or confidence threshold. The resulting standardized scores provide a bounded, maturity-weighted correction to frozen logits without target-specific hyperparameter tuning. Our results also show that full per-class feature-covariance matrices are not required for competitive adaptation. ACME uses global feature means and coordinate-wise variances for normalization, together with a compact $C\times D$ posterior-feature cross-covariance for class-specific associations. This reduces the statistical state from DOTA's $O(CD^2)$ to **$O(CD)$**, without matrix inversion. Our contribution is this tuning-free online adaptation rule built from compact empirical moments and a standardized readout. We have added the exact soft-prototype equivalence and clarified the distinctions from PTA in **Section 3.4 (lines 263–277)**, with an expanded related-work discussion in **Section 2.2 (lines 173–182)** of the revised manuscript. [1] Dynamic Multimodal Prototype Learning in Vision-Language Models. ICCV 2025. [2] Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Models. ICCV 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript and Responses to Reviewers

Revised Manuscript and Responses to Reviewers

We sincerely thank all reviewers for their time and constructive feedback. In response to their comments, we have revised the manuscript and uploaded the updated version.

The main revisions and additional responses are:

1. Methodological clarifications. We have added the exact soft-prototype equivalence and clarified ACME's estimator and readout differences from PTA in Section 3.4 (lines 263–277), with an expanded related-work discussion in Section 2.2. These updates address Q2 from Reviewer dyNa, W1/Q1 from Reviewer PktW, and W1/Q1 from Reviewer H8Wr.

2. Adaptation reliability. We clarify the error-correction mechanism and safeguards against error amplification in our responses, and add variance-sensitivity experiments in Section 4.3 (lines 506–521), Table 6, addressing W1/W2 and Q1/Q3 from Reviewer dyNa and W2/Q3 from Reviewer H8Wr. Our follow-up to Reviewer H8Wr also reports readout and maturity ablations on all four OOD datasets.

3. Additional benchmarks and larger VLMs. We have added CIFAR-100-C, CIFAR-10-C, and SVHN comparisons without retuning (Section 4.2, Table 3; Appendix A.7, Table 13), addressing W1/W2 from Reviewer PktW and W1/W4 from Reviewer H8Wr. In response to Reviewer H8Wr's follow-up, we also evaluate SigLIP 2 ViT-g-opt-384 and CoCa ViT-L/14 on ImageNet and all ten cross-domain datasets (Appendix A.6, Table 12).

4. Non-IID robustness. We have added comparisons with TDA and both the default and conservative PTA configurations under three temporal-correlation levels and class-separated streams (Appendix A.2, Table 9). These experiments use ACME's unchanged configuration and address W3/Q2 from Reviewer H8Wr.

5. Baseline comparisons and variability. In our response, we provide fixed-configuration OOD comparisons and mean ± standard deviation over ten random stream orders for ACME and strong baselines, addressing W2 from Reviewer PktW. These additional results clarify the accuracy-efficiency trade-off and sensitivity to test-stream order.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.