acceptodds
Under review as a conference paper at ICLR 2027

SHARP: Spherical-Harmonic-Inspired Distribution Modeling and Prototype Recalibration for Test-Time Adaptation

Abstract

Online test-time adaptation (TTA) adapts pre-trained vision-language models (VLMs) to distribution shifts using only an unlabeled test stream. Cache-based methods retain past test samples as instance-level evidence, but a finite cache covers only a limited portion of the target distribution. Class-conditional distributions provide a complementary global view, yet their statistics are hard to estimate from few observations in the high-dimensional CLIP embedding space, especially covariance estimation. We propose SHARP, which estimates class-wise distributions in a compact angular feature space. Motivated by the hyperspherical geometry of normalized CLIP embeddings, SHARP builds a spherical-harmonic-inspired representation from learnable directional projections and low-order angular functions, reducing the number of covariance parameters and improving estimation reliability. The resulting class statistics drive both distribution-aware scoring and variance-adaptive prototype recalibration. SHARP further combines these statistics with instance-level evidence from multiple caches and applies lightweight residual adaptation to the classifier and cache features while keeping the CLIP encoders frozen. Across 10 cross-dataset benchmarks, SHARP improves average accuracy over the strongest baseline by 0.97% and 1.16% with ResNet-50 and ViT-B/16, respectively. On ImageNet and its four distribution-shift variants, the corresponding gains are 0.94% and 0.95%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Model, Test-Time Adaptation, Zero-Shot Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yvVpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SHARP studies the online temporal adaptation of CLIP through the compact class-conditional distribution modeling. It operates in the angular feature space, where the embeddings are normalized, and it maintains the class-wise means and covariances for distribution scoring and prototype recalibration. The latter is combined with multiple instance caches and residual updates, while the CLIP encoders are frozen.

Critical strengths
  1. The reported gains are broad: over all tasks, datasets, and types of visual concepts (fine grained recognition, texture, scene, action). The 0.97 and 1.16 points improvement in average cross-dataset accuracy for the two backbones used (vs. MCP) is considerable. It suggests the practical value of the overall method.
  2. The probabilistic formulation has nice properties: the covariance update is mean-shift corrected, the Gaussian score uses the same shrunk covariance for the quadratic and log-determinant terms. These are solid theoretical grounding for most of the paper and help it address the numerical instabilities of the early streams.
Critical weaknesses
  1. The effect of either of the two design decisions is limited to some extent. SH introduce an alteration to the feature mapping as well as the dimension. But, the participants mentioned in Table 3 are not able to differentiate their angle shape from typical dimension reduction. A linear projection learning with the same output dimension as text-fitting could serve as an interesting control. For VAPR variance and correlation gates, the numbers are summed: SH and VAPR respectively account for 0.33 and 0.32 points. However, since no spread over runs is shown, these are less likely to be trusted entirely on their own and together than the full pipeline.

  2. The cost is not specified (correctly limits this claim of 'compactness' to the output space). Equation (1) remains expressed in terms of L dense d x d matrices. For the given dimensions, these are 67 million projection parameters for ViT-B/16 (268 million for Res Net-50, not counting class covariances).Only the runtime of ViT-B/16 for Image Net-A is given; no information is provided about peak memory. All other metrics are only given as observations for calculating accuracy; the ratio of online to initial time is not provided (regardless of the terminology used in the text), although it is also relevant for the accuracy-cost relation.

Questions for authors
  1. It would be interesting to compare this to a learned 256-dimensional linear projection, fit to the same text-similarity objective. This could help reveal whether the angular construction is a key determinant.

  2. Need clarification regarding the process followed for choosing the final set of hyperparameters and some investigation whether the improvement of SH and VAPR varies across stream orders and augmentation seeds.

  3. Need to determine if the reported runtime also includes the 200 step projection fitting as specified in Appendix A.1. Need to explain the maximum GPU memory needed and whether or not refactor each class covariance after each sample.

4 comments
AuthorsOpenReview ↗

Response to Reviewer yvVp Critical Weaknesses

W1: Linear-projection control. Table R3 provides the requested comparison with a learned 256-dimensional linear projection, fitted using the same text-similarity objective as SH and evaluated under the same adaptation pipeline (ViT-B/16, ten benchmarks, and shrunk covariance). The learned linear projection achieves 72.60%, compared with 72.82% for SH, yielding a modest improvement of 0.22 percentage points. Without matched multi-seed comparisons, we cannot establish the statistical reliability of this difference or attribute the improvement uniquely to the angular construction. We will therefore narrow the corresponding claim in the revised manuscript. Variability of component gains. We agree that the reliability of individual component gains should be evaluated separately from that of the full pipeline. Table R3 reports a three-seed standard deviation of 0.14 percentage points for the full SHARP pipeline on the ViT-B/16 cross-dataset evaluation. However, this does not characterize the variability of individual component-level improvements. In the cumulative ablation reported in Table 3 of the submission, SH improves accuracy from 72.49% to 72.82% (+0.33 points), while VAPR further improves accuracy to 73.14% (+0.32 points). These gains are conditional on the preceding components, and their statistical reliability cannot be established from the full-pipeline standard deviation alone. Furthermore, the variance and correlation gates of VAPR have not been independently ablated in the submission. We will extend the ablation study using matched random seeds, independently evaluate the two VAPR gates, and report the mean and standard deviation of the component-level gains in the revised manuscript.


W2: We agree that the reviewer's parameter counts are correct. We clarify the mapping size, covariance storage, initialization overhead, and measured GPU memory below. Projection parameters. For ViT-B/16 (d=512d=512), the mapping contains 67.1M parameters, corresponding to approximately 268 MB in FP32. For ResNet-50 (d=1024d=1024), it contains 256×10242≈268M256\times 1024^2 \approx 268\text{M} parameters, corresponding to approximately 1.07 GB in FP32. Class covariance storage. Each set of class-conditional covariance matrices contains K×256×256K\times256\times256 elements. For ImageNet (K=1000K=1000), this requires approximately 262 MB in FP32 per covariance tensor set, excluding other model states and intermediate buffers. Initialization versus online runtime. As detailed in our response to Question 3, the one-time 200-step projection fitting on DTD with ViT-B/16 takes 1.77 s, compared with 850.79 s for online adaptation. Thus, projection fitting accounts for approximately 0.21% of the online adaptation time in this setting. These timings are indicative because the GPU was shared with other jobs during some measurements. The corresponding projection fitting time on ImageNet-A has not yet been measured separately. Peak GPU memory. The measured peak GPU memory on Caltech101 is 2.63 GB (Table R5), covering the entire run, including projection fitting. During fitting, AdamW maintains the mapping parameters, their gradients, and two optimizer moment estimates, requiring approximately 4×268 MB≈1.07 GB4\times268\,\text{MB}\approx1.07\,\text{GB} of parameter-related storage. On DTD, the measured peak memory during the online stage is 1130.2 MB, excluding projection fitting. These measurements involve different datasets and profiling scopes and are therefore not directly comparable. Peak GPU memory on ImageNet-A with ViT-B/16, corresponding to the setting used for the runtime comparison in Table 5 of the submission, has not yet been measured. We will report this measurement in the revision. Overall, we will revise the efficiency section to explicitly distinguish mapping parameters, class-covariance storage, one-time initialization overhead, and online peak-memory consumption, providing a more transparent account of the computational and memory costs.

AuthorsOpenReview ↗

Response to Reviewer yvVp Question 1-3

Q1: Table R3 provides the comparison under the same pipeline (ViT-B/16, ten benchmarks, L=256L=256, and covariance shrinkage). The learned linear projection is fitted using the same text-similarity objective as SH. It achieves 72.60% average accuracy, compared with 72.82% for SH, corresponding to a 0.22 improvement. This result shows that SH achieves a modest advantage over the learned linear projection under the evaluated setting. However, without matched multi-seed comparisons, we cannot establish the statistical reliability of this difference or conclude that the angular construction itself is the determining factor. We will clarify this distinction in the revised manuscript.

Q2: Hyperparameter selection. The distribution-weight schedule, cache capacities, and ϵ\epsilon were selected using aggregate accuracy on the evaluation benchmarks, without a separate held-out validation set. A single configuration was shared across all datasets and both backbones. The selection protocol and sensitivity of the reported gains to these choices are discussed in our responses to Reviewer RWNA, Weakness 1 and Question 1. Stream order and augmentation seeds. Table R1 reports three runs using different global random seeds. Each seed jointly controls stream order, augmentation sampling, and projection initialization. The reported standard deviations therefore reflect their combined variability. Table R4 evaluates projection initialization stability on EuroSAT using three paired runs, while keeping stream order, augmentations, and other settings fixed. The fitted mapping achieves 67.91±0.15%67.91\pm0.15\% accuracy, with a paired improvement of 1.13±0.131.13\pm0.13% over the unfitted mapping. This provides additional evidence of stability across projection initializations, although it does not isolate stream-order or augmentation effects. Variation of SH and VAPR gains. Table R1 reports a 3-seed standard deviation of 0.14% for the full SHARP pipeline on the ViT-B/16 cross-dataset evaluation. In the cumulative ablation of Table 3, SH and VAPR provide incremental gains of 0.33 and 0.32%, respectively. However, we have not run this ablation over multiple seeds, so the full-pipeline standard deviation does not establish the variability of these individual component gains. We will extend the +DA, +SH, and +VAPR ablations to matched random seeds, keeping the evaluation protocol consistent, and report the mean and standard deviation of the paired component-level improvements.

Q3: The reported runtime excludes projection fitting. The submitted implementation recomputes the inverse of every class covariance after each sample. The largest peak GPU memory we measured is 2.63 GB on Caltech101 (ViT-B/16), reached during the one-time projection fitting. Projection fitting. The 35-minute ImageNet-A runtime in Table 5 excludes the one-time 200-step projection fitting. On DTD (ViT-B/16, 1,692 samples), fitting takes 1.77 s, compared with 850.79 s for online adaptation, accounting for approximately 0.2% of the online adaptation time. We have not separately measured fitting time on ImageNet-A. The projection is fitted on the class-text embeddings before the test stream and remains frozen afterwards. Covariance computation. Class statistics are updated incrementally using Eqs. 8–9, but the inverse of each class covariance is recomputed explicitly after every sample for distribution scoring. This involves KK dense 256×256256\times256 matrix inversions per sample (e.g., 47 on DTD). Appendix A.3 describes a Cholesky-based computation, whereas the submitted implementation uses explicit inversion. The two methods yield the same score mathematically, and we will correct the appendix to accurately reflect the implementation. Under Eq.10, each covariance update consists of a scalar rescaling, an isotropic shift, and a rank-one correction. Therefore, the inverse could potentially be updated incrementally. However, all results reported in this submission use explicit inversion. Runtime and memory. All timings in this rebuttal and Table 5 of the submission were obtained on a single NVIDIA RTX 3060, which was shared with other jobs during some runs. They should therefore be regarded as indicative. We will re-time the experiments on a dedicated GPU in the revision.In particular, we have not yet reconciled the DTD per-image runtime (approximately 0.50 s) with the Caltech101 result in Table R5 (approximately 0.28 s). Peak GPU memory is 2.63 GB on Caltech101 and 1.13 GB on DTD. Caltech101 measurement is the peak over the entire run, including projection fitting, during which AdamW maintains approximately 4×2684\times268 MB of parameter-related storage for the mapping. The DTD measurement covers online adaptation only. These measurements involve different datasets and profiling scopes and are therefore not directly comparable. Peak memory on ImageNet-A has not yet been measured. We will report this measurement in the revised manuscript.

Thanks for your thorough response and the effort you've put in. The extra comparisons and clarifications are appreciated. The linear projection baseline was helpful, and the cost breakdown improved transparency.

That said, one concern persists. As noted in W1, VAPR's variance and correlation gating were assessed together. Although the 0.32-point gain validates the combined module, it doesn't confirm if both gates are effective or if one dominates the improvement. Would the total performance boost still be convincing if we decoupled these two gating mechanisms?

AuthorsOpenReview ↗

Response to Reviewer yvVp

We thank the reviewer for suggesting that we disentangle the contributions of the two gating mechanisms in VAPR. We conducted an additional ablation evaluating the variance gate (V-Gate) and the correlation gate (C-Gate) separately and jointly, while keeping all other components and experimental settings unchanged. As shown in Table R7, the model without VAPR achieves 72.82% average accuracy across ten cross-dataset benchmarks. Enabling only the V-Gate increases accuracy to 72.93% (+0.11 points), while enabling only the C-Gate yields 73.06% (+0.24 points). Enabling both gates achieves the highest average accuracy of 73.14% (+0.32 points). The decoupled results show that the C-Gate provides the larger observed contribution. The V-Gate provides a smaller positive increment, improving accuracy by 0.11 points on its own and by a further 0.08 points when added to the C-Gate. Conversely, adding the C-Gate to the V-Gate improves accuracy by 0.21 points. These comparisons support using both gates based on the reported average accuracies. However, the reliability of the smaller V-Gate increment remains to be assessed through paired comparisons across multiple seeds. On Caltech101, the V-Gate and C-Gate individually add 1.4 and 1.8 ms/image, respectively. Together, they increase online time from 276.2 to 279.1 ms/image, an overhead of approximately 1.05%. All configurations report a whole-run peak GPU memory of 2.63 GB at the displayed precision. These timings were measured on a shared GPU and are indicative. We will include this decoupled ablation in the revised manuscript and, as stated in our earlier response, extend it to matched random seeds. We will also revise the description of VAPR to state that the correlation gate is the main contributor and the variance gate a smaller one.

Table R7. Decoupled ablation of the variance and correlation gates in VAPR with ViT-B/16. All configurations are built upon the same SHARP model with all other components and experimental settings unchanged. Accuracy is averaged over ten cross-dataset benchmarks. Online time and peak GPU memory are measured on Caltech101; online time excludes the one-time projection fitting, and peak memory covers the entire run.

ConfigurationV-GateC-GateAvg. Acc. (%)Avg. Δ (pp)Online Time (ms/img)Peak Mem (GB)
+SH (No VAPR)✗✗72.82--276.22.63
+SH + Variance✓✗72.93+0.11277.62.63
+SH + Correlation✗✓73.06+0.24278.02.63
+SH + VAPR (Both)✓✓73.14+0.32279.12.63
Reviewer_g3y1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SHARP, an online test-time adaptation method for CLIP that builds on the MCP multi-cache framework. Its main contribution is a compact spherical-harmonic-inspired representation, built from learnable directional projections and low-order angular functions, for online class-conditional distribution modeling with an isotropic prior and covariance shrinkage, reducing the number of covariance parameters and improving estimation reliability. The resulting class statistics drive both distribution-aware scoring, which adds the log-density of each class distribution to the logits, and variance-adaptive prototype recalibration (VAPR), which downweights less reliable prototype coordinates. SHARP further adapts MCP's entropy and negative-cache construction into a confidence cache and a hard-negative cache, and applies lightweight residual test-time optimization to the classifier and cache features while keeping the CLIP encoders frozen.

Across 10 cross-dataset benchmarks and the ImageNet shift suite with ResNet-50 and ViT-B/16, SHARP consistently improves over MCP at similar runtime.

Critical strengths

1. The paper clearly articulates the limitations of cache-based and distribution-based TTA and makes a coherent attempt to combine them. As the authors point out, a finite cache covers only a limited portion of the target distribution, while class statistics provide a global view but are hard to estimate from few observations in the high-dimensional CLIP space, especially the covariance. SHARP targets exactly this gap, with its compact angular representation making the estimate tractable and the multi-cache framework retaining the instance-level evidence. The experiments show consistent gains over MCP with both backbones, at a cost of only 5 extra minutes of runtime on ImageNet-A.

2. The compact-space distribution estimation offers clear advantages in estimating class distributions online. A full covariance contains d(d+1)/2 free entries, yet only a small number of pseudo-labeled observations are available for each class early in the stream, which makes covariance estimation sensitive to these limited and noisy samples. The compact representation with an isotropic prior and covariance shrinkage reduces the number of covariance parameters and improves estimation reliability, and the authors' analysis of why the log-determinant can be dominated by the diluted-prior directions further shows the design handles a genuine pitfall of online estimation.

Critical weaknesses

1. Assembled components and marginal additions. SHARP inherits the multi-cache framework and the residual adaptation from MCP/MCP++, and takes the idea of class-distribution scoring from DOTA, which are components that already existed. What the paper adds on its own, such as the compact angular representation and the VAPR gates, reads as incremental rather than a new or consequential one.

2. Too many hyperparameters to tune, which limits practical use. The method requires choosing at least fifteen hyperparameters, while sensitivity analysis is provided for only two of them. It is hard to know which transfer to a new task, and each new deployment becomes a fresh tuning effort, so the practical value beyond the reported benchmarks is unclear.

Questions for authors
  1. Per-component time, memory, and accuracy breakdown. The paper reports only an overall runtime comparison against baselines (Table 5). It would be useful to see, for each component, the added time and memory usage alongside its accuracy gain, so that readers can judge whether each part is worth its cost.
  2. Accuracy over the online process. Since this is an online TTA method whose statistics and caches accumulate over the stream, a curve of running accuracy versus the number of processed test samples, comparing SHARP with the baselines, would better reveal how performance evolves during adaptation and how the methods compare at different stream stages.
  3. More comprehensive hyperparameter sensitivity analysis. Sensitivity is currently reported for only two of the many hyperparameters (the cache capacities and the schedule parameters in Tables 4 and 6). It would be helpful to see more sensitivity results covering the other hyperparameters, so that readers can tell which choices the method actually depends on.
7 comments
AuthorsOpenReview ↗

Response to Reviewer g3y1 Critical Weaknesses

W1: SHARP builds upon existing components: multi-cache from MCP, residual adaptation from MCP++, and the general idea of class-distribution scoring from DOTA. We do not claim these inherited components as our contributions. Our contributions focus on two complementary designs: 1) SH estimates class-conditional statistics in a 256-dimensional feature space. Table R3 compares SH with simpler dimensionality-reduction alternatives under the same adaptation pipeline. SH achieves 72.82%, compared with 72.25% for random projection and 72.60% for a learned linear projection fitted using the same text-similarity objective. The 0.22-point advantage over the learned linear projection is modest, and its statistical reliability has not been established through matched multi-seed comparisons. Therefore, we do not attribute the overall performance improvement solely to the angular mapping. 2) Beyond distribution-aware scoring, VAPR uses class-wise variance and correlation statistics to recalibrate class prototypes by attenuating coordinates with high estimated variance or strong average absolute correlation with other coordinates. Thus, the estimated class distributions contribute to both prediction scoring and prototype recalibration.In the cumulative ablation in Table 3 of the submission, SH improves accuracy from 72.49% to 72.82% (+0.33%), and VAPR further improves it to 73.14% (+0.32). Together, these two additions provide a 0.65 improvement over distribution-aware modeling in the original embedding space. We will revise the contribution statement to clearly distinguish inherited components from our proposed designs. We will also clarify that distribution-aware modeling produces the largest individual increment (+0.57) in the cumulative ablation, while SH and VAPR provide additional gains of 0.33 and 0.32, respectively.

W2: SHARP uses a single hyperparameter configuration across all datasets and both CLIP backbones, without dataset-specific tuning. However, this configuration was selected based on accuracy on the evaluation benchmarks (see our response to Reviewer RWNA, Weakness 1). We acknowledge the resulting benchmark-level selection bias and do not claim independently validated hyperparameter transfer to unseen tasks. Expanded sensitivity analysis. Table R6 extends the analysis to 16 hyperparameters on a DTD using ViT-B/16. Each parameter is evaluated at three values while all other parameters remain fixed at their defaults. Across the tested configurations, accuracy ranges from 57.82% to 58.46%, corresponding to a maximum spread of 0.64%. The default configuration achieves the highest or tied-highest accuracy in all 16 parameter sweeps. This pattern is consistent with the defaults having been selected using the evaluation benchmarks, so we do not interpret the table as evidence of hyperparameter insensitivity. Rather, it quantifies the effect of individual parameter changes within the examined ranges. It does not establish robustness to joint parameter variations or generalization of the selected settings to unseen datasets. Distribution-weight schedule. The original submission additionally evaluates the schedule parameters (η,ρ)(\eta,\rho) across 25 configurations (Table 6), with accuracy ranging from 71.81% to 73.14%, corresponding to a spread of 1.33%. This larger variation indicates greater observed sensitivity to the distribution-weight schedule than to any individually varied parameter in Table R6, although the two analyses use different evaluation sets. We will explicitly document the hyperparameter selection protocol, include the expanded sensitivity analysis with these caveats, and remove claims of hyperparameter insensitivity.

Table R6: One-factor-at-a-time hyperparameter sensitivity analysis on DTD using ViT-B/16. Each parameter is varied individually while all other parameters remain fixed at their default values.

HyperparameterTested valuesDefaultDTD Acc. (%)
LL128, 256, 51225658.33, 58.46, 58.33
ϵ\epsilon0.001, 0.01, 0.050.0157.91, 58.46, 58.39
n0n_01, 5, 10158.46, 58.19, 58.32
σ02\sigma_0^20.001, 0.01, 0.10.0158.39, 58.46, 58.46
τc\tau_c0.5, 1, 2158.35, 58.46, 58.12
βr\beta_r3, 5, 10558.14, 58.46, 58.26
λcache\lambda_{\mathrm{cache}}0.3, 0.5, 1.00.557.82, 58.46, 58.24
α\alpha0.1, 0.3, 0.50.158.46, 58.37, 58.32
γ\gamma0.1, 0.2, 0.50.258.19, 58.46, 57.98
λ\lambda0.1, 0.2, 0.50.558.42, 58.31, 58.46
ζ\zeta0.1, 0.2, 0.50.258.16, 58.46, 57.87
msm_s0.05, 0.1, 0.20.158.17, 58.46, 58.46
τs\tau_s0.05, 0.1, 0.20.157.94, 58.46, 58.33
λ1\lambda_10.2, 0.5, 1.00.558.46, 58.46, 58.30
λ2\lambda_20.1, 0.2, 0.50.258.26, 58.46, 58.40
At\mathcal{A}_t2, 4, 8458.19, 58.46, 58.32
AuthorsOpenReview ↗

Response to Reviewer g3y1 Question 1-3

Q1: Table R5 reports the component-wise accuracy, online runtime, and peak GPU memory of cumulative configurations using ViT-B/16. Accuracy. On the average, CA, HNC, DA, SH, and VAPR yield improvements of 0.22, 0.13, 0.57, 0.33, and 0.32%, respectively. Runtime. The runtime increases from 244.0 ms/img for the baseline to 292.2 ms/img after adding DA. Adding SH reduces the measured latency to 276.2 ms/img, while VAPR increases it slightly to 279.1 ms/img. These measurements were collected on an NVIDIA RTX 3060, which was shared with other jobs during some runs. Therefore, the timings should be regarded as indicative rather than definitive. In particular, the 16.0 ms reduction after adding SH requires further verification on a dedicated GPU. Memory. Peak GPU memory on Caltech101 remains at 1.32–1.33 GB up to HNC, increases to 1.43 GB with DA, and reaches 2.63 GB after adding SH. VAPR introduces no additional measured peak memory overhead. The 1.20 GB increase from DA to SH primarily comes from the one-time projection fitting stage. The mapping contains 67.1M parameters. During fitting, AdamW additionally maintains gradients and two optimizer moment estimates, resulting in approximately 4×2684 \times 268 MB ≈1.07\approx 1.07 GB of parameter-related storage. Thus, the reported peak memory includes this initialization overhead and should not be interpreted as the incremental online memory cost of SH. Cost–Benefit Analysis. DA provides the largest accuracy gain (+0.57), with an additional 31.9 ms/img and 0.10 GB of peak memory. SH contributes another 0.33 and incurs 1.20 GB of additional peak memory during one-time fitting; its measured online latency is 16.0 ms/img lower, although this reduction requires confirmation. VAPR further improves accuracy by 0.32, with only 2.9 ms/img additional runtime and no measurable increase in peak memory. We will include this analysis in the revised manuscript to provide a more transparent assessment of the accuracy–efficiency trade-off.

Table R5: Component-wise accuracy, online runtime, and peak GPU memory with ViT-B/16. Configurations are cumulative. Accuracy is averaged over ten datasets, while runtime and memory are measured on Caltech101. Online runtime excludes projection fitting; peak memory includes the entire run.

ConfigurationAvg. Acc. (%)Δ (points)Online Time (ms/img)Peak Mem (GB)
Baseline (re-implemented MCP)71.57--244.01.32
+CA71.79+0.22246.71.32
+HNC71.92+0.13260.31.33
+DA (Shrunk)72.49+0.57292.21.43
+SH72.82+0.33276.22.63
+VAPR73.14+0.32279.12.63

Q2: Fig. R1 (supplementary) plots the running accuracy against the number of processed test samples on Oxford-Pets for SHARP and MCP++, with one run per method.Both methods exhibit substantial fluctuations during the first 100 samples, and their curves cross several times up to approximately 700 samples. Between roughly 700 and 2,900 samples, MCP++ achieves higher running accuracy than SHARP, with the largest gap being approximately 1.5 percentage points around 1,200 samples. After approximately 3,000 samples, the two curves nearly overlap and converge to similar final accuracy levels. These observations show that SHARP does not consistently outperform MCP++ in terms of running accuracy on Oxford-Pets, although both methods achieve comparable final performance. We will extend this analysis to additional datasets and random seeds in the revised manuscript.

Q3: Table R6 extends the sensitivity analysis to 16 hyperparameters on DTD using ViT-B/16. Each hyperparameter is individually varied across three values while all other parameters are kept fixed at their default settings. The default configuration achieves the highest or tied-highest accuracy in all 16 sweeps. As acknowledged in our response to Weakness 2, the default settings were selected based on the evaluation benchmarks. Therefore, we do not interpret these results as evidence of general hyperparameter insensitivity. The largest accuracy drops are observed for λcache=0.3\lambda_{\mathrm{cache}}=0.3 (0.64), ζ=0.5\zeta=0.5 (0.59), ϵ=0.001\epsilon=0.001 (0.55), τs=0.05\tau_s=0.05 (0.52), and γ=0.5\gamma=0.5 (0.48). All remaining tested configurations exhibit accuracy drops of at most 0.34%. These results suggest that the cache weight (λcache\lambda_{\mathrm{cache}}), covariance shrinkage coefficient (ϵ\epsilon), and hard-negative-related parameters (γ\gamma, ζ,τs\zeta, \tau_s) are relatively more sensitive within the tested ranges, whereas the other parameters exhibit smaller variations. We emphasize that this is a single-dataset, one-factor-at-a-time analysis. It does not establish robustness to joint parameter variations or the transferability of the selected hyperparameters to unseen datasets. We will include these results and explicitly clarify the scope and limitations of the sensitivity analysis in the revised manuscript.

Thank you for your detailed and thorough response.

Regarding the component-wise accuracy experiment you presented, I noticed that while accuracy improved by 1.57, this came at the cost of increased latency and a memory usage increase of over 1 GB, which doesn’t seem like a very cost-effective trade-off.

Additionally, regarding the “Accuracy over the online process” on the currently presented datasets, I noticed that the method described in this paper still performs below the baseline in many test phases, which makes it somewhat unconvincing. The authors could also include “Accuracy over the online process” results for additional datasets.

Reviewer_rWNAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies online test-time adaptation (TTA) of CLIP-style vision-language models, where an unlabeled test stream must be classified sample by sample without source data or labels. The authors note that cache-based methods keep only a limited set of past samples, while distribution-based methods are hard to estimate reliably in the high-dimensional CLIP space, especially their covariances. To address this, they propose SHARP, which maps CLIP embeddings into a lower-dimensional, spherical-harmonic-inspired angular representation built from learnable projections and low-order angular functions, and estimates per-class Gaussian statistics in that space with an isotropic prior and covariance shrinkage. These class statistics are used in two ways: a distribution-aware score added to the prediction, and a variance-adaptive prototype recalibration (VAPR) that down-weights unreliable prototype coordinates. SHARP builds on the multi-cache framework of MCP, with a confidence cache, an alignment cache and a hard-negative cache, and updates lightweight residuals on the text classifier and cached features while keeping both CLIP encoders frozen. On 10 cross-dataset benchmarks, SHARP improves average accuracy over MCP by 0.97% (ResNet-50) and 1.16% (ViT-B/16), and on ImageNet and four distribution-shift variants by 0.94% and 0.95%. The paper also includes a cumulative ablation, sensitivity analyses of cache capacity and the distribution-weight schedule, and a runtime comparison.

Critical strengths
  1. SHARP achieves the best average accuracy on both backbones and both benchmark suites (Tables 1–2). It ranks first on 8/10 (ResNet-50) and 9/10 (ViT-B/16) cross-dataset benchmarks and improves over MCP on all four ImageNet shifts with ResNet-50. The runtime overhead over MCP is small: 35 vs. 30 minutes on ImageNet-A (Table 5). This consistency across 20 settings is the main reason I see value in the paper.

  2. The online statistics update is exact and derived in the appendix (Eq. 8–9). The reason for using one shrunk covariance in both terms of the score is explained (Eq. 10–11). The full iteration is given in Algorithm 1, with parameters in Table 7. Using the same covariance estimate for both scoring and prototype recalibration is a neat idea, and the limitations section is honest.

Critical weaknesses
  1. The gains are small, and the tuning protocol makes them hard to trust.

    • The improvement over MCP is about 1% (Tables 1–2).
    • The distribution-weight schedule (η,ρ)(\eta, \rho) and the cache capacities appear to be selected on the evaluation benchmarks themselves (Tables 4 and 6), and the best settings are the ones reported as main results.
    • In Table 6, accuracy across schedule settings ranges from 71.81% to 73.14%. That 1.33-point spread is larger than the 1.16-point gain over MCP. In Table 4, a different cache setting (∣MC∣=5|\mathcal{M}_C|=5, ∣MH∣=3|\mathcal{M}_H|=3) gives 65.42% OOD, only 0.33 above MCP (65.09%).
    • SHARP adopts the residual adaptation of MCP++ (Sec. 1, Sec. 3.6), but the main tables compare only against published MCP. The re-implemented baseline used in the ablation (71.57%) is also below the published MCP (71.98%).
  2. The main contribution, the spherical-harmonic-inspired representation, is not well supported.

    • The functions used are Legendre polynomials, which the paper describes as zonal harmonics on S2S^2 (Sec. 3.2), while CLIP embeddings lie on a much higher-dimensional sphere. Also, qiq_i depends on xx, so the coordinates are not angles to fixed learned directions.
    • The representation is "compact" only in its output. The mapping uses L=256L=256 full d×dd \times d matrices, about 67M parameters for ViT-B/16 assuming its standard 512-dimensional embedding. These are fitted only on the KK class-text embeddings by matching their Gram matrix (Sec. 3.2), which gives very few constraints, e.g., K=10K=10 for EuroSAT. It is not clear why a map fitted on text alone should give better class statistics for image features.
    • In the cumulative ablation (Table 3), SH adds only +0.33. There is no comparison with simpler ways to reduce covariance parameters, such as PCA or a random projection to LL dimensions, or a diagonal or shrunk covariance in the original CLIP space. So it is unclear whether the gain comes from the spherical-harmonic design or from dimensionality reduction and regularization in general.
Questions for authors
  1. How were (η,ρ)(\eta, \rho), the cache capacities, LL and ϵ\epsilon selected? Were Tables 4 and 6 computed on the same benchmarks used for the main results, and was any held-out data used for selection? (Clarification; this directly affects how I read the ~1% gains.)
  2. Have you compared the SH representation with simple alternatives under the same pipeline: PCA or a random projection to L=256L=256, or a diagonal or shrunk covariance in the original CLIP space? Does the "+DA" row in Table 3 already use covariance shrinkage?
  3. How many parameters does the mapping {Bi}\{B_i\} have, and how stable is the fit when KK is small (e.g., EuroSAT)? Has the map been checked on image features, given that it is fitted only on text embeddings?
5 comments
AuthorsOpenReview ↗

Response to Reviewer rWNA Critical Weaknesses 1(1-2)

W1-1: We agree that the gains are modest. The additional three-seed experiments and MCP++ comparisons provide a more comprehensive assessment of SHARP. Against the published results, SHARP improves over MCP by 0.97 / 1.16 percentage points on cross-dataset evaluation (RN50 / ViT-B/16) and 0.94 / 1.06 points on domain generalization. For domain generalization with RN50, we use 47.95, the mean of MCP's four published per-dataset results; the MCP paper reports 48.02, against which the gain is 0.87 points. Compared with published MCP++, SHARP achieves gains of 0.56 / 0.53 points on cross-dataset evaluation and 0.72 / 0.73 points on domain generalization. Against our reproduced baselines, whose results are lower than the published results in all four settings, SHARP achieves gains of 1.10 / 1.57 points over MCP and 0.72 / 0.80 points over MCP++ on cross-dataset evaluation, as well as 1.11 / 1.09 points over MCP and 0.92 / 0.80 points over MCP++ on domain generalization. Importantly, on a seed-by-seed basis, SHARP outperforms reproduced MCP++ in all 12 matched seed-setting pairs, with improvements ranging from 0.41 to 1.07 percentage points. Moreover, the lowest SHARP result exceeds the highest MCP++ result in every evaluation setting. We emphasize that the three-seed experiments address run-to-run variability only; they do not resolve potential hyperparameter selection bias associated with the evaluation benchmarks. We address this separate concern in the following responses. We will include these additional results and explicitly report the improvements in percentage points in the revised manuscript.

Table R1: The following table reports the mean accuracy and standard deviation (%) over three random seeds. A dash (–) indicates that the original paper did not report results for individual seeds. We reproduced MCP and MCP++ using their official code. An asterisk (*) denotes results reproduced by us using the official implementations.

Cross-dataset (RN50)

MethodSeed 1Seed 2Seed 3Mean ± Std
MCP (published)–––65.08
MCP* (reproduced)64.8464.9065.1064.95 ± 0.14
MCP++ (published)–––65.49
MCP++* (reproduced)65.4865.3565.1765.33 ± 0.16
SHARP65.8966.0766.2066.05 ± 0.16

Cross-dataset (ViT-B/16)

MethodSeed 1Seed 2Seed 3Mean ± Std
MCP (published)–––71.98
MCP* (reproduced)71.5671.8071.3471.57 ± 0.23
MCP++ (published)–––72.61
MCP++* (reproduced)72.5872.4472.0072.34 ± 0.30
SHARP73.2873.1373.0173.14 ± 0.14

Domain Generalization (RN50)

MethodSeed 1Seed 2Seed 3Mean ± Std
MCP (published)–––48.02
MCP* (reproduced)48.0347.9147.4047.78 ± 0.33
MCP++ (published)–––48.17
MCP++* (reproduced)47.7348.0948.1047.97 ± 0.21
SHARP48.7248.9149.0448.89 ± 0.16

Domain Generalization (ViT-B/16)

MethodSeed 1Seed 2Seed 3Mean ± Std
MCP (published)–––65.09
MCP* (reproduced)65.4064.9264.8765.06 ± 0.29
MCP++ (published)–––65.42
MCP++* (reproduced)65.4465.1365.4865.35 ± 0.19
SHARP65.9066.2066.3566.15 ± 0.23

W1-2: We agree with the reviewer's observation. The distribution-weight schedule (η,ρ)(\eta,\rho) and cache capacities were selected based on aggregate accuracy across the evaluation benchmarks, without using a separate held-out validation set. The reported configuration corresponds to the best-performing setting among the configurations evaluated in Tables 4 and 6 of the submission. We used a single shared configuration across all datasets and both backbones. However, we acknowledge that sharing the configuration does not eliminate potential benchmark-level selection bias. In the next response, we further examine how the reported improvements vary across the tested parameter configurations. We will explicitly clarify the hyperparameter selection protocol in the revision and will no longer characterize the results in Tables 4 and 6 as evidence of hyperparameter insensitivity.

AuthorsOpenReview ↗

Response to Reviewer rWNA Critical Weaknesses 1(3-4)

W1-3: We agree that the 1.33-point spread in Table 6 of the submission exceeds the 1.16-point gain over MCP, indicating that the improvement depends on the parameter setting. We further examine this sensitivity across the tested parameter grids. (i) Distribution-weight schedule. Across the 25 settings in Table 6 of the submission, the mean accuracy is 72.70%. This is 0.72 percentage points higher than published MCP (71.98%), with 24 out of 25 settings outperforming it. Compared with published MCP++ (72.61%), the grid mean is only 0.09 points higher, with 16 out of 25 settings outperforming it. Relative to our reproduced MCP++ result (72.34%, reported in the additional three-seed comparison above), the grid mean is 0.36 points higher, with 22 out of 25 settings outperforming it. (ii) Cache capacity. Across the six settings in Table 4 of the submission, the mean OOD accuracy is 65.77%. All six settings outperform published MCP (65.09%). Compared with published MCP++ (65.42%), five out of six settings perform better, while the remaining setting matches its accuracy. All six settings also exceed our reproduced MCP++ result (65.35%). Notably, the configuration highlighted by the reviewer (65.42%) matches the published MCP++ result and exceeds our reproduced result. Overall, these results show that SHARP generally outperforms MCP across the tested parameter ranges, whereas its advantage over MCP++ is more sensitive to parameter selection, particularly for the distribution-weight schedule. We emphasize that these comparisons are descriptive and do not eliminate potential benchmark-level selection bias.

W1-4: Table R1 reports three-seed results for MCP++, which shares the residual adaptation strategy inherited by SHARP. Compared with published MCP++, SHARP improves accuracy by 0.56 and 0.53 percentage points on cross-dataset evaluation (RN50 / ViT-B/16), and by 0.72 and 0.73 points on domain generalization, respectively. Compared with our reproduced MCP++ results, the corresponding gains are 0.72 / 0.80 and 0.92 / 0.80 points, respectively. The MCP results in the additional three-seed comparison were obtained by running the official code. For ViT-B/16 cross-dataset evaluation, our reproduced mean is 71.57%, numerically matching the reimplemented MCP baseline reported in Table 3 of the submission and 0.41 points below the published result of 71.98%. This discrepancy is primarily driven by EuroSAT, where our reimplementation obtains 63.74% compared with the published 68.42%, a difference of 4.68 points. On the remaining nine datasets, our reimplementation averages 0.06 points above the published results. We have not yet identified the cause of the EuroSAT discrepancy and will explicitly acknowledge this limitation in the revised manuscript. Across the four evaluation settings, our reproduced MCP results are 0.13, 0.41, 0.17, and 0.03 points below the corresponding published results, respectively. Importantly, for ViT-B/16 cross-dataset evaluation, our main comparison uses the higher published MCP result (71.98%), rather than our lower reproduced result (71.57%). Consequently, the reported improvement of SHARP over MCP is 1.16 points, compared with 1.57 points when using our reproduced baseline. We will clarify the provenance of the baseline results and explicitly discuss the EuroSAT reproduction discrepancy alongside Table 3 in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer rWNA Critical Weaknesses 2(1-2)

W2-1: Legendre polynomials characterize zonal spherical harmonics on S2\mathbb{S}^{2} when evaluated on cosine similarities with a fixed reference direction, whereas zonal harmonics on the higher-dimensional sphere Sd−1\mathbb{S}^{d-1} involve Gegenbauer polynomials. Moreover, since qi\mathbf{q}_i depends on the input x\mathbf{x}, ai=x⊤qia_i=\mathbf{x}^{\top}\mathbf{q}_i represents an input-dependent cosine similarity rather than the cosine of an angle to a fixed learned direction. Our construction is therefore not a formal spherical-harmonic expansion. Instead, it uses low-order nonlinear angular responses, including an absolute-value function and Legendre polynomial terms, to construct a lower-dimensional (L=256L=256) feature space for class-conditional distribution modeling. The term "spherical-harmonic-inspired" refers to the use of Legendre-based angular responses, not to harmonic orthogonality or a complete harmonic basis. We acknowledge that the theoretical framing in Sections 1 and 3.2 of the submission was insufficiently precise. We will revise these sections to clarify the actual construction and remove language suggesting a formal spherical-harmonic expansion or an orthogonal harmonic basis.

W2-2: Compactness refers to the output representation and class statistics, rather than the mapping itself. At L=256L=256, a symmetric class covariance matrix contains 32,896 distinct entries, compared with 131,328 at d=512d=512. The mapping has 67.1M parameters on ViT-B/16, is fitted once using class-text embeddings, and remains frozen during online adaptation. Cross-modal transfer. The text-only fitting objective encourages preservation of pairwise cosine similarities among class-text anchors. Since CLIP embeds images and text in a shared space, preserving the text-anchor geometry motivates applying the same mapping to image features, although image representations are not directly constrained by the fitting objective. The average cosine similarity to the correct-class anchor increases from 0.31 to 0.89 on EuroSAT, from 0.30 to 0.87 on DTD, and from 0.33 to 0.78 on Caltech101. These similarities alone do not establish improved class discrimination, since similarities to incorrect-class anchors may also increase. Direct validation of text-only fitting. To isolate the contribution of text-only fitting, we compare the same SH mapping with and without the 200-step fitting procedure. Both variants use matched initializations, feature dimensions, covariance estimation, stream orders, augmentation seeds, and online adaptation settings. With ViT-B/16, text-only fitting improves accuracy on all three reported datasets, by 0.33 to 1.13 percentage points, and raises the ten-dataset average from 72.77% to 73.14% (Table R2). These controlled comparisons provide evidence that text-only fitting improves downstream adaptation beyond the unfitted angular mapping under the evaluated settings. Limitations. We acknowledge that fitting a high-capacity mapping from a small number of text anchors remains underdetermined, particularly on EuroSAT (K=10K=10, with only 45 distinct off-diagonal class pairs). The observed downstream improvements do not constitute a theoretical guarantee of cross-modal generalization. We will clarify this limitation and the computational cost of the mapping in the revision.

Table R2: Text-only fitting ablation on ViT-B/16. EuroSAT results use three paired projection-initialization runs, with stream order and augmentations fixed. The other nine datasets use three paired global seeds. The ten-dataset average is the unweighted mean of dataset-level mean accuracies.

DatasetUnfitted SHText-fitted SHΔ (points)
EuroSAT66.7867.91+1.13
DTD57.9358.46+0.53
Caltech10195.5695.89+0.33
10-dataset Avg.72.7773.14+0.37

Comments on the paper

1 comment
AuthorsOpenReview ↗

Acknowledgment to All Reviewers

We sincerely thank all the reviewers for their valuable comments, insightful suggestions, and constructive questions, which have helped us improve the quality and clarity of our manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.