acceptodds
Under review as a conference paper at ICLR 2027

Covariance-Aware Scoring: A Geometric Correction for Vision-Language Test-Time Adaptation

Abstract

Test-time adaptation (TTA) enables vision–language models such as CLIP to adapt to distribution shifts without labels or access to source data. Existing methods adapt through feature caching, prototype estimation, or the online accumulation of feature statistics, but most ultimately score a test feature against a class representative in the ambient embedding space, typically through cosine similarity. We identify this shared decision rule as a limitation under anisotropic feature distributions, which means when a small number of directions account for a large fraction of feature variance, the score can be dominated by high-variance directions that are not necessarily discriminative. In light of this, we propose CARe, a covariance- aware scoring for TTA that replaces the ambient geometry with a Mahalanobis geometry induced by the target feature second moment. Theoretically, we show that under a shared-covariance Gaussian model the covariance-aware decision direction coincides with that of the Bayes-optimal linear discriminant, with ambient-space scoring as the isotropic special case, and establish that the potential benefit of covariance-aware scoring grows with feature anisotropy. Empirically, we show that CARe improves adaptation accuracy over its base cosine scoring on both standard and fine-grained distribution-shift benchmarks across two backbones. Consistent with our analysis, the gains are largest on datasets whose features are most anisotropic and vanish as the features approach isotropy.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
test-time adaptation, feature covariance, vision-language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5trFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CARe modifies SCA’s scoring geometry using a regularized inverse-square-root transform of the online stream’s uncentered feature second moment. It compares transformed image features and adapted class representatives, then combines adapted and zero-shot scores through entropy-based fusion. Evaluation covers ImageNet, four natural shifts, and ten cross-domain datasets using RN50 and ViT-B/16.

Critical strengths
  • Simple and practical intervention: CARe reuses statistics maintained by SCA and requires no encoder updates or additional encoder passes. This makes the scoring correction straightforward to integrate into online adaptation.

  • Broad empirical support: CARe improves over SCA in 27 of 30 dataset–backbone comparisons, with reported ImageNet-family average gains of 1.41–1.97 percentage points. Two-sided transform ablations support the scoring design, while evaluation across five stream orders provides evidence of robustness to ordering.

Critical weaknesses
  • Incomplete theoretical justification: The Bayes analysis uses within-class covariance (S), whereas CARe transforms the total second moment (G), which also contains shared-mean and between-class terms. Additional conditions are needed to justify the claimed Bayes-direction correspondence. The synthetic eigenvalue sweep also increases total noise variance, so it does not isolate anisotropy. Limited gains on Caltech101 warrant clearer discussion of when the correction helps.

  • Reporting and efficiency gaps: RN50 CARe entries in Table 2 average 64.73%, rather than 64.66%. The prose reverses the backbone-specific cross-domain gains, and default-setting results differ across ablation tables. Table 8 reports 9.3 times the base readout cost at the default refresh interval. Absolute end-to-end measurements are needed to support negligible-overhead claims.

Questions for authors
  • Under what conditions does the total-moment metric preserve the Bayes discriminant direction? How do mean removal and centered-covariance scoring compare under matched settings?

  • Does the synthetic anisotropy–gain trend persist when covariance trace and the mean-to-noise ratio are held fixed?

  • Can you reconcile the averages, backbone-specific gains, and default configurations across Tables 2, 4, and 8 using seed-level results?

  • What are the absolute end-to-end latency and memory costs relative to SCA at refresh intervals 1 and 100 for both backbones? Are measurements available on low-power edge hardware?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5trF: fixed-trace study, latency measured, tables reconciled

We thank Reviewer 5trF for the thorough review and for noting CARe’s simplicity, practicality, and broad empirical support. We address each point below.

W1:

  • S vs G and the Bayes-direction correspondence: The correspondence holds under a stated condition that is met in CARe’s target regime. Writing G=S+μˉμˉ⊤+SB\boldsymbol{G} = \boldsymbol{S} + \bar\mu\bar\mu^\top + \boldsymbol{S}_B, the metric preserves the Bayes direction when the shared mean and high within-class variance dominate the spectrum and the between-class term SB\boldsymbol{S}_B does not. This matches CLIP’s geometry, where the dominant directions are the modality-gap mean and high-variance non-discriminative structure (Fig. 1), which is what CARe down-weights. We will state the correspondence as directional and conditional.

  • Synthetic sweep and anisotropy: Please see response to Q2 and full results from reviewer STGv (Q2) response.

  • Small Caltech101 gain: Caltech is among the least anisotropic datasets, so there is little high-variance non-discriminative structure for CARe to correct, and the covariance-aware readout stays close to the base rule. This is exactly the regime where our analysis predicts negligible gain, and it matches the near-zero gain at κ=1 in the synthetic study.

W2 and Q3: On reconciling averages, backbone gains, and default configurations. The reviewer is right, and the issue is an isolated transcription error. The RN-50 CARe entries in Table 2 average to 64.73, matching the seed-level mean in Table 7 exactly, but the Average cell was mistyped as 64.66; we will correct it. We re-checked every other reported average against its per-dataset entries and found them consistent, so this was the only discrepancy.

A note that: Tables 1, 2, and 7 are five-seed averages (mean ± std in Table 7); Table 8 is a single-seed amortization study, so its η=1 mean (64.74) differs from the five-seed mean by 0.01. These are reporting corrections and do not change the method or conclusions.

Q1: Conditions for preserving the Bayes direction, and mean removal vs centered-covariance. The Bayes direction uses S−1Δ\boldsymbol{S}^{-1}\Delta, while CARe uses (G+γI)−1(\boldsymbol{G}+\gamma\boldsymbol{I})^{-1} with G=S+μˉμˉ⊤+SB\boldsymbol{G} = \boldsymbol{S} + \bar\mu\bar\mu^\top + \boldsymbol{S}_B. The total-moment metric preserves the Bayes direction when the extra terms do not rotate S−1Δ\boldsymbol{S}^{-1}\Delta, i.e. when μˉμˉ⊤+SB\bar\mu\bar\mu^\top + \boldsymbol{S}_B is small relative to S\boldsymbol{S} along the discriminative directions or lies (near-)orthogonal to the Δ\Delta. This is favorable when the shared mean and high within-class variance dominate the spectrum (as in CLIP) and the between-class term SB\boldsymbol{S}_B does not.

On the comparison: in our synthetic study under matched settings, removing the global mean before estimating the metric leaves the anisotropy–gain trend essentially unchanged (−2.4 at κ=1 to +31 at κ=1000), so the gain is driven by covariance anisotropy, not the mean offset. Full results are in our response to Reviewer STGv (Q2).

Q2: Does the trend persist with covariance trace and mean-to-noise ratio held fixed? Yes. We reran the synthetic study with the total trace held fixed, so both the covariance trace and the mean-to-noise ratio are constant and anisotropy is the only varying factor. Cosine accuracy stays flat (~68%) while CARe’s gain still grows monotonically with anisotropy (−2.4 at κ=1 to +31 at κ=1000), confirming the trend is not a signal-to-noise artifact. We also verified that centering the features first leaves the trend unchanged. Full results are in our response to Reviewer STGv (Q2).

Q4 :Absolute end-to-end latency and memory, η ∈ {1, 100}, both backbones We measured end-to-end latency (encoder plus readout) and peak memory on UCF101 over 2000 samples:

BackboneConfigEnd-to-end (ms/sample)vs SCAPeak mem (MB)
ViT-B/16SCA14.38—372
CARe, η=134.96+143%375
CARe, η=10014.66+2.0%375
RN-50SCA16.98—284
CARe, η=131.61+86%301
CARe, η=10017.18+1.2%301

At η=1 the per-step eigendecomposition roughly doubles end-to-end latency (the 9.3× readout figure); this is the exact-reference setting, not the deployment one. At η=100 it is amortized and the overhead drops to +2.0% (ViT-B/16) and +1.2% (RN-50), with peak memory essentially unchanged (+3 and +17 MB). Since Table 8 shows η=100 matches η=1 accuracy, deployment gets the full accuracy of CARe at 1–2% latency and negligible memory.

Edge hardware. We have not measured on edge devices. CARe adds no trainable parameters, no extra forward pass, and only a fixed D×D accumulator, so its cost scales with the readout, not the encoder. The one device-sensitive part is the eigendecomposition, controlled by η; at η=100 it stays at ~1–2% of end-to-end latency even on the faster RN-50 encoder.

Thank you for the detailed replies. The authors fixed the Table 2 average, added the fixed-trace synthetic sweep in Figure 4b, ran the source-of-gains ablation in Table 4, measured end-to-end latency in Table 5, and checked the ridge collinearity ratio in Table 9.

Even with these updates, I am keeping my rating at 2 (Weak rejection) because the theory-to-algorithm gap and incremental novelty remain unresolved. Scoping the Bayes analysis as directional and conditional leaves a wide disconnect between the idealized within-class covariance model and the actual algorithm, which uses the uncentered total second moment, ridge-regression representatives, and class-dependent normalization that can compress discriminative between-class scatter without theoretical guarantees (Reviewer STGv and Reviewer RbkE). CARe also retains SCA's pseudo-labeling, statistic accumulation, and ridge-regression updates, making the regularized Mahalanobis cosine similarity an incremental adjustment over SCA and classical whitening (Reviewer RbkE). In addition, reporting errors still appear in the revised draft: page 7 (line 372) lists the RN-50 gain over SCA as +1.91% instead of +2.19% (64.73 vs. 62.54), page 8 (line 424) swaps the RN-50 and ViT-B/16 gains (+2.12 and +1.91), and Table 11 omits that it reports a single-seed run unlike the five-seed tables. Because the theoretical motivation does not explain the implemented rule and the methodological contribution over SCA remains limited, the paper stays below the acceptance bar for me.

About reporting errors: We have carefully rechecked the revised manuscript and corrected the remaining reporting inconsistencies. We apologize for the oversight and confirm that these corrections do not affect the experimental results.

About Theory. We acknowledge that the shared-covariance analysis serves as a motivating idealization rather than a formal derivation of the implemented correction. We have therefore explicitly scoped the theoretical discussion as directional and conditional. However, this distinction does not undermine the underlying motivation: similar to many practical whitening and discriminant methods, our approach is motivated by a simplified Gaussian model without requiring the implemented method to strictly follow all its assumptions. Importantly, our central claim is not that the theory fully characterizes the method, but that anisotropy is the key mechanism underlying the observed improvements. We validate this mechanism empirically and independently of the theoretical analysis through the fixed-trace study in Fig. 4b.

About Novelty. We believe the results in Table 4 provide concrete evidence of CARe's methodological contribution. Within the same experimental pipeline, CARe consistently outperforms three established correction strategies: diagonal whitening, global mean removal, and top-k component removal, across both backbones. This demonstrates that its gains cannot be attributed simply to conventional whitening or mean removal, but instead arise from leveraging the full off-diagonal covariance structure. Furthermore, CARe applies this correction in a label-free, online setting on top of a frozen adapter, distinguishing it from standard correction approaches. We therefore argue that CARe represents a meaningful methodological contribution rather than a minor variation of SCA.

Reviewer_RbkEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies feature anisotropy in CLIP-based test-time adaptation (TTA). The authors observe that much of the spectral energy of CLIP features is concentrated in a few dominant directions. Since existing methods typically use standard cosine similarity for classification, directions with high variance but limited discriminative value may have a disproportionate influence on predictions.

To address this issue, the authors propose CARe, a covariance-aware scoring method. CARe accumulates the uncentered second-moment matrix GG of test features online, constructs a regularized transformation M=(G+γI)−1/2M=(G+\gamma I)^{-1/2}, and computes cosine similarity after transforming both test features and class representatives. The method inherits soft pseudo-label generation and ridge-regression-based representative updates from SCA, and adaptively combines the original CLIP and CARe scores using predictive entropy. The CLIP encoders remain frozen throughout adaptation. Experiments cover two backbones, ImageNet distribution-shift benchmarks, and ten cross-domain classification datasets, with improvements over the base method in most settings.

Critical strengths

1. The paper addresses a meaningful problem from the perspective of scoring geometry.

Existing CLIP-based TTA methods primarily focus on pseudo-labels, caching strategies, class representative estimation, or model updates. This paper instead examines the geometry used for the final similarity computation. Studying how feature anisotropy affects scoring provides a reasonable direction for improving existing adaptation methods.

2. The method is simple and has a clear mathematical interpretation.

CARe constructs a regularized inverse-square-root transformation from existing second-order statistics without introducing trainable parameters or requiring backpropagation. Its score can be interpreted as cosine similarity in a Mahalanobis inner-product space, and it reduces to standard cosine similarity when the second moment is isotropic. The design is straightforward to understand and integrate into a statistics-based adaptation framework.

3. The experiments provide broad coverage and reasonably consistent improvements.

The evaluation uses ResNet-50 and ViT-B/16 across natural distribution shifts and several cross-domain classification tasks. CARe improves over SCA on most datasets. Ablations on regularization strength, one-sided versus two-sided transformations, fusion weights, refresh intervals, and stream ordering provide additional evidence of its empirical effectiveness.

Critical weaknesses

1. Distinction from Existing Methods

CARe largely inherits its class representative updates and soft pseudo-label mechanism from SCA. Its main addition is an inverse-square-root transformation constructed from the second moment, which is closely related to classical regularized whitening and Mahalanobis metrics. The specific contribution of this operation in online TTA could be articulated more clearly. Moreover, SCA already incorporates second-order statistics through ridge regression, so the relationship between the additional scoring transformation and the existing statistical correction remains somewhat unclear.

2. Connection Between Theory and Algorithm

The theoretical analysis mainly relies on the shared within-class covariance SS, whereas the algorithm uses the uncentered total second moment GG, which also includes the global mean and between-class variation. These components may contain discriminative information. Consequently, suppressing high-energy directions in the total second moment does not necessarily have the same effect as an ideal within-class covariance correction.

The theory also assumes that class representatives point in the same directions as the true class centers, but representatives obtained through ridge regression need not satisfy this assumption. In addition, the algorithm normalizes the transformed class representatives, which can affect classification decisions. An analysis of directional angles alone therefore does not fully explain the behavior of the actual algorithm.

3. Source of the Performance Gains

The ablations demonstrate the effectiveness of transforming both features and representatives, but they do not fully separate the contributions of mean correction, variance balancing, and full covariance modeling. The concentration of CLIP features along dominant directions may partly arise from the global mean. Simple comparisons with mean centering, per-dimension normalization, or dominant-component removal would help clarify the role of the full transformation. Without these comparisons, the main source of the observed gains remains uncertain.

4. Computational Efficiency

The default setting refreshes the transformation at every step, while the appendix reports that scoring under this setting takes 9.3 times as long as standard cosine scoring. This does not imply an equivalent increase in end-to-end inference time, but the absence of overall latency measurements makes the practical cost difficult to assess. Although less frequent refreshes reduce the overhead while largely preserving accuracy, the efficiency of the default configuration still requires clarification.

5. Consistency of Experimental Results

Some reported results require verification. For example, the arithmetic mean of the ten RN-50 CARe results in Tables 2 and 3 is approximately 64.73, rather than 64.66. The RN-50 DTD results in Tables 4 and 5 also differ, and several entries in Table 8 do not match the results associated with the stated default regularization setting. These discrepancies make it harder to verify the experimental configurations and reported comparisons.

4. Overall Assessment

The paper investigates CLIP-based TTA from the perspective of feature-space geometry and proposes a simple method that produces reasonably consistent improvements across multiple datasets and two backbones. However, its distinction from classical whitening methods, the connection between the theory and the implemented algorithm, and the source of the performance gains remain insufficiently clear. The computational cost and several inconsistencies in the reported results also require clarification.

Questions for authors

Please kindly refer to the weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer RbkE: distinction clarified + new ablation done for source of gains

We thank Reviewer RbkE for the detailed and constructive review, and for recognizing that CARe addresses a meaningful problem from the perspective of scoring geometry, is simple with a clear mathematical interpretation, and is supported by broad experiments with consistent improvements. We address each concern below.

W1: Distinction from Existing Methods: CARe is not a minor add-on to SCA. While SCA and other backprop-free CLIP-TTA methods improve what is compared (representatives, caches, pseudo-labels), they retain ambient cosine similarity. CARe instead changes the decision geometry. This is not redundant with SCA’s ridge solve: SCA uses G\boldsymbol{G} to construct wkw_k but retains Euclidean comparisons, whereas CARe uses G\boldsymbol{G} to induce a Mahalanobis comparison. Our factor ablation confirms this distinction: even with fixed representatives, the transform alone improves accuracy.

W2:

Connection Between Theory and Algorithm: We agree and use the analysis to motivate the mechanism, not to predict the exact algorithm. On S\boldsymbol{S} vs G\boldsymbol{G}: the total moment is G=S+μˉμˉ⊤+SB\boldsymbol{G} = \boldsymbol{S} + \bar\mu\bar\mu^\top + \boldsymbol{S}_B, so inverting it is a trade-off, attenuating the non-discriminative shared mean and dominant within-class variance but also partially compressing the between-class term SB\boldsymbol{S}_B. It is favorable when the shared mean and high within-class variance dominate the spectrum (as in CLIP) and SB\boldsymbol{S}_B does not; we no longer claim G≈S\boldsymbol{G}\approx\boldsymbol{S}.

On the representatives: wk=(G+λridgeI)−1pkw_k = (\boldsymbol{G}+\lambda_{\text{ridge}}\boldsymbol{I})^{-1}p_k is approximately collinear with μk\mu_k only when the ridge dominates. We measured this (the ratio λridge/λmax⁡(Gt)\lambda_{\text{ridge}}/\lambda_{\max}(\boldsymbol{G}_t)) and it exceeds 1 on all datasets across both architectures (except for Food101 for ViT). Please see response to Q1 from Reviewer STGv.

On the normalization: the class-dependent ∣Mwk∣|\boldsymbol{M}w_k| changes decision boundaries. Thus, CARe shares only the decision direction with LDA, not the full decision rule. We clarify that all three points are directional and conditional. And none affect the reported accuracies, which are empirically measured.

W3: Source of the Performance Gains: We ran the suggested comparison, replacing CARe’s full transform with (i) mean removal, (ii) per-dimension standardization (diagonal of G\boldsymbol{G}), and (iii) top-kk component removal, in the same pipeline (ViT-B/16, γ=10):

Datasetcosinemean rem.per-dimtop-kCARe
Flowers75.7264.5675.3176.0977.87
DTD55.3253.0754.7356.0956.97
Pets91.0389.9791.1491.4192.48
Cars67.5062.4968.2468.6269.67
UCF10172.5669.6372.4072.7274.02
Caltech94.6993.7594.5287.8794.16
Food10185.5583.9186.0786.3786.80
SUN39768.1965.3470.1270.6171.33
Aircraft28.8026.3728.5328.5929.22
EuroSAT55.9656.6259.4759.4162.00

From this result, we can confirm that mean removal hurts performance, while per-dimension standardization recovers only part of the gain, highlighting the importance of off-diagonal covariance. Top-k removal is brittle, whereas CARe remains stable and achieves the best results on 9 of 10 datasets, matching cosine on near-isotropic Caltech. We will add this ablation to the revision.

W4: Computational efficiency.

We measured absolute end-to-end latency and memory for both backbones at η=1 and η=100. In short: at the amortized setting (η=100) the end-to-end overhead is +2.0% (ViT-B/16) and +1.2% (RN-50) with negligible extra memory, and Table 8 (from the paper) shows η=100 matches η=1 accuracy. Full measurements and the latency table are in our response to Reviewer 5trF (efficiency / end-to-end latency).

W5: Consistency of Experimental Results: The Table 2 RN-50 CARe average is a transcription error: the entries average to 64.73 (matching Table 7), not the printed 64.66; we will correct it. The RN-50 DTD differences arise from mixed sourcing: the main, fusion, and seed tables (1, 2, 5, 7) are five-seed averages (DTD = 55.14), while the γ\gamma-sensitivity (Table 4) and amortization (Table 8) tables are single-seed trend studies (since those study trends (γ\gamma near-optimality, η\eta amortization) where the absolute value is secondary), so on DTD (the smallest, highest-variance dataset) their single-seed values differ.

We will state the sourcing in every caption so the configuration behind each table is unambiguous. These are labeling corrections and do not affect the conclusions.

Dear Reviewer RbkE,

As the discussion period nears its close, we wanted to follow up. To address your concern on the source of the gains, we added a controlled ablation (Table 4) comparing CARe’s transform against mean removal, per-dimension whitening, and top-k component removal in the same pipeline; CARe outperforms all three, indicating the gain comes from the off-diagonal covariance structure rather than a simpler correction. We also clarified the distinction from SCA and classical whitening. We would welcome any further feedback.

Thank you.

Reviewer_STGvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The draft proposes a covariance-aware scoring method for CLIP based test-time adaptation. It uses the accumulated uncentered second moment of test features to compute the cos similarity in a Mahalanobis geometry, then combines these scores with zero-shot predictions. Experiments across two backbones and multiple distribution-shift benchmarks show improvements over the contenders.

Critical strengths
  1. The propsoed method is quite simple, requires no backpropagation, and reuses existing feature statistics. Improvements across two backbones and multiple datasets suggest its practical value.
  2. Ablations covering many aspects provide useful evidence supporting the empirical gains.
Critical weaknesses
  1. The proposed theory analyzes the within-class covariance, whereas CARe uses the second moment and ridge-regression representatives. Class-dependent cos normalization further changes the decision boundaries. Although these limitations are acknowledged, the Bayes-optimal direction claims remain stronger to me than the analysis supports for the proposed algorithm.

  2. I also have some concerns on the synthetic experiments. Increasing covariance eigenvalues over [1,\kappa] also increases total noise and changes the signal-to-noise ratio. Consequently, this experiment and the real-data correlations do not establish as the mechanism driving the gains.

Questions for authors
  1. The analysis assumes class representatives align with class means. Does this hold for the ridge-regression representatives used by CARe?
  2. Increasing \kappa in the synthetic experiment also increases total noise. Do the gains still grow with \kappa if total variance is fixed? Does centering the features before estimating the metric change the results?
2 comments
AuthorsOpenReview ↗

Response to Reviewer STGv: 2 added experiments, theory scope tightened

We thank Reviewer STGv for recognizing the method’s simplicity and the breadth of our ablations, and for the constructive concerns on the theory and synthetic study, which we address below.

W1: About theory: We do not claim CARe implements the Bayes-optimal rule; we claim only that, under an idealized shared-covariance Gaussian model, the covariance-aware score shares the decision direction of the Bayes-optimal discriminant. We make the three gaps the reviewer identifies explicit:

  • Within-class S\boldsymbol{S} vs. total moment G\boldsymbol{G}: The ideal metric is S−1\boldsymbol{S}^{-1}, but S\boldsymbol{S} is unidentifiable without labels, so CARe uses (G+γI)−1(\boldsymbol{G}+\gamma\boldsymbol{I})^{-1}; the two differ by the rank-KK mean term. We will phrase inverting G\boldsymbol{G} as a trade-off rather than a recovery of S−1\boldsymbol{S}^{-1}, and drop any wording implying G≈S\boldsymbol{G}\approx\boldsymbol{S}.

  • Ridge representatives: wk=(G+λridgeI)−1pkw_k=(\boldsymbol{G}+\lambda_{\text{ridge}}\boldsymbol{I})^{-1}p_k is approximately collinear with μk\mu_k when the ridge dominates the accumulated spectrum. We verified this holds across datasets and both backbones (see our response to the ridge question (Q1) below), and will state the assumption and its regime explicitly.

  • Class-dependent normalization: The CARe score divides by ∥Mwk∥2\|\boldsymbol{M}w_k\|_2 while the Bayes rule carries an additive bias, so the two share only the decision direction, not the full rule. We already note this and will state it slightly more prominently.

In short, the theory identifies the mechanism CARe corrects, not the gains, which do not depend on the Bayes correspondence. These are wording fixes that affect no result.

W2/Q2 About kappa and associated signal to noise: We thank the reviewer for this precise observation. We reran the study with the total variance held fixed: we rescale the eigenvalues so Tr(Σ) is constant across the entire κ sweep, leaving anisotropy as the only varying factor.

κCosineCAReGain
167.565.1−2.40
267.766.7−1.01
567.472.1+4.71
1067.379.1+11.81
2067.987.2+19.35
5067.995.2+27.27
10067.598.3+30.82
20068.499.6+31.17
50068.099.9+31.98
100068.9100.0+31.07

Under this control, cosine accuracy stays flat (≈67–69%) across all κ, while CARe’s gain still grows monotonically with anisotropy, from −2.4 points at κ=1 to +31 points at κ=1000. Because the total variance is now constant, the gain cannot be attributed to a change in signal-to-noise; it tracks anisotropy alone. As predicted, CARe is marginally worse at κ=1 (−2.4): with no anisotropy to correct, the transform only adds estimation noise.

On centering: We also ran the fixed-trace sweep with features centered by the global mean before estimating the metric. The trend is essentially unchanged (gain −2.4 at κ=1 to +31 at κ=1000), indicating the effect is driven by covariance anisotropy rather than mean removal.

Q1: Does wk∝μkw_k \propto \mu_k hold for the ridge-regression representatives? Yes, and we verified it through additional experiment. The adapter computes wk=(Gt+λridgeI)−1pkw_k = (\boldsymbol{G}_t + \lambda_{\mathrm{ridge}}\boldsymbol{I})^{-1} p_k with pk≈Nkμkp_k \approx N_k \mu_k, so wkw_k is approximately collinear with the centroid whenever the ridge dominates the accumulated spectrum: if λridge≫λmax⁡(Gt)\lambda_{\mathrm{ridge}} \gg \lambda_{\max}(\boldsymbol{G}_t), then (Gt+λridgeI)−1≈λridge−1I(\boldsymbol{G}_t + \lambda_{\mathrm{ridge}}\boldsymbol{I})^{-1} \approx \lambda_{\mathrm{ridge}}^{-1}\boldsymbol{I} and wk≈(Nk/λridge)μkw_k \approx (N_k / \lambda_{\mathrm{ridge}}) \mu_k. The condition is therefore captured by a single ratio λridge/λmax⁡(Gt)\lambda_{\mathrm{ridge}} / \lambda_{\max}(\boldsymbol{G}_t), with values above 11 meaning the ridge dominates and the assumption holds. To measure it, we accumulated Gt=∑ififi⊤\boldsymbol{G}_t = \sum_i f_i f_i^{\top} over the test stream exactly as in the method and recorded its largest eigenvalue λmax⁡(Gt)\lambda_{\max}(\boldsymbol{G}_t) at the end of each stream. The resulting ratio, on every cross-domain benchmark across both backbones, is as follows:

DatasetNViT-B/16RN-50
DTD16928.996.1
Caltech10124657.385.3
Flowers24636.853.5
Aircraft33334.343.4
Pets36693.738.5
UCF10137834.548.6
Cars80411.921.4
EuroSAT81001.514.9
Food101303000.515.39
SUN397198501.28.1

On RN-50 the ridge dominates throughout for every dataset (ratios 5–96), so the assumption holds cleanly. On ViT-B/16 it holds on 9 of the 10 datasets (ratios 1.2–8.9), with a single exception: Food101, by far the longest stream ($\sim$30k samples).

Dear Reviewer STGv,

As the discussion period nears its close, we wanted to follow up. In response to your comments, we reran the synthetic study with the covariance trace held fixed (Fig. 4b), which isolates anisotropy from signal-to-noise and confirms the gain is driven by anisotropy alone, and we verified the ridge-collinearity assumption directly across all datasets and both backbones (Table 9). We also scoped the theory as directional and conditional, as you suggested. We would be grateful for any further feedback and are happy to clarify or run additional experiments.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.