- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
CARe modifies SCA’s scoring geometry using a regularized inverse-square-root transform of the online stream’s uncentered feature second moment. It compares transformed image features and adapted class representatives, then combines adapted and zero-shot scores through entropy-based fusion. Evaluation covers ImageNet, four natural shifts, and ten cross-domain datasets using RN50 and ViT-B/16.
Critical strengths
-
Simple and practical intervention: CARe reuses statistics maintained by SCA and requires no encoder updates or additional encoder passes. This makes the scoring correction straightforward to integrate into online adaptation.
-
Broad empirical support: CARe improves over SCA in 27 of 30 dataset–backbone comparisons, with reported ImageNet-family average gains of 1.41–1.97 percentage points. Two-sided transform ablations support the scoring design, while evaluation across five stream orders provides evidence of robustness to ordering.
Critical weaknesses
-
Incomplete theoretical justification: The Bayes analysis uses within-class covariance (S), whereas CARe transforms the total second moment (G), which also contains shared-mean and between-class terms. Additional conditions are needed to justify the claimed Bayes-direction correspondence. The synthetic eigenvalue sweep also increases total noise variance, so it does not isolate anisotropy. Limited gains on Caltech101 warrant clearer discussion of when the correction helps.
-
Reporting and efficiency gaps: RN50 CARe entries in Table 2 average 64.73%, rather than 64.66%. The prose reverses the backbone-specific cross-domain gains, and default-setting results differ across ablation tables. Table 8 reports 9.3 times the base readout cost at the default refresh interval. Absolute end-to-end measurements are needed to support negligible-overhead claims.
Questions for authors
-
Under what conditions does the total-moment metric preserve the Bayes discriminant direction? How do mean removal and centered-covariance scoring compare under matched settings?
-
Does the synthetic anisotropy–gain trend persist when covariance trace and the mean-to-noise ratio are held fixed?
-
Can you reconcile the averages, backbone-specific gains, and default configurations across Tables 2, 4, and 8 using seed-level results?
-
What are the absolute end-to-end latency and memory costs relative to SCA at refresh intervals 1 and 100 for both backbones? Are measurements available on low-power edge hardware?
3 comments
Thank you for the detailed replies. The authors fixed the Table 2 average, added the fixed-trace synthetic sweep in Figure 4b, ran the source-of-gains ablation in Table 4, measured end-to-end latency in Table 5, and checked the ridge collinearity ratio in Table 9.
Even with these updates, I am keeping my rating at 2 (Weak rejection) because the theory-to-algorithm gap and incremental novelty remain unresolved. Scoping the Bayes analysis as directional and conditional leaves a wide disconnect between the idealized within-class covariance model and the actual algorithm, which uses the uncentered total second moment, ridge-regression representatives, and class-dependent normalization that can compress discriminative between-class scatter without theoretical guarantees (Reviewer STGv and Reviewer RbkE). CARe also retains SCA's pseudo-labeling, statistic accumulation, and ridge-regression updates, making the regularized Mahalanobis cosine similarity an incremental adjustment over SCA and classical whitening (Reviewer RbkE). In addition, reporting errors still appear in the revised draft: page 7 (line 372) lists the RN-50 gain over SCA as +1.91% instead of +2.19% (64.73 vs. 62.54), page 8 (line 424) swaps the RN-50 and ViT-B/16 gains (+2.12 and +1.91), and Table 11 omits that it reports a single-seed run unlike the five-seed tables. Because the theoretical motivation does not explain the implemented rule and the methodological contribution over SCA remains limited, the paper stays below the acceptance bar for me.
About reporting errors: We have carefully rechecked the revised manuscript and corrected the remaining reporting inconsistencies. We apologize for the oversight and confirm that these corrections do not affect the experimental results.
About Theory. We acknowledge that the shared-covariance analysis serves as a motivating idealization rather than a formal derivation of the implemented correction. We have therefore explicitly scoped the theoretical discussion as directional and conditional. However, this distinction does not undermine the underlying motivation: similar to many practical whitening and discriminant methods, our approach is motivated by a simplified Gaussian model without requiring the implemented method to strictly follow all its assumptions. Importantly, our central claim is not that the theory fully characterizes the method, but that anisotropy is the key mechanism underlying the observed improvements. We validate this mechanism empirically and independently of the theoretical analysis through the fixed-trace study in Fig. 4b.
About Novelty. We believe the results in Table 4 provide concrete evidence of CARe's methodological contribution. Within the same experimental pipeline, CARe consistently outperforms three established correction strategies: diagonal whitening, global mean removal, and top-k component removal, across both backbones. This demonstrates that its gains cannot be attributed simply to conventional whitening or mean removal, but instead arise from leveraging the full off-diagonal covariance structure. Furthermore, CARe applies this correction in a label-free, online setting on top of a frozen adapter, distinguishing it from standard correction approaches. We therefore argue that CARe represents a meaningful methodological contribution rather than a minor variation of SCA.
Response to Reviewer 5trF: fixed-trace study, latency measured, tables reconciled
We thank Reviewer 5trF for the thorough review and for noting CARe’s simplicity, practicality, and broad empirical support. We address each point below.
W1:
S vs G and the Bayes-direction correspondence: The correspondence holds under a stated condition that is met in CARe’s target regime. Writing G=S+μˉμˉ⊤+SB, the metric preserves the Bayes direction when the shared mean and high within-class variance dominate the spectrum and the between-class term SB does not. This matches CLIP’s geometry, where the dominant directions are the modality-gap mean and high-variance non-discriminative structure (Fig. 1), which is what CARe down-weights. We will state the correspondence as directional and conditional.
Synthetic sweep and anisotropy: Please see response to Q2 and full results from reviewer STGv (Q2) response.
Small Caltech101 gain: Caltech is among the least anisotropic datasets, so there is little high-variance non-discriminative structure for CARe to correct, and the covariance-aware readout stays close to the base rule. This is exactly the regime where our analysis predicts negligible gain, and it matches the near-zero gain at κ=1 in the synthetic study.
W2 and Q3: On reconciling averages, backbone gains, and default configurations. The reviewer is right, and the issue is an isolated transcription error. The RN-50 CARe entries in Table 2 average to 64.73, matching the seed-level mean in Table 7 exactly, but the Average cell was mistyped as 64.66; we will correct it. We re-checked every other reported average against its per-dataset entries and found them consistent, so this was the only discrepancy.
A note that: Tables 1, 2, and 7 are five-seed averages (mean ± std in Table 7); Table 8 is a single-seed amortization study, so its η=1 mean (64.74) differs from the five-seed mean by 0.01. These are reporting corrections and do not change the method or conclusions.
Q1: Conditions for preserving the Bayes direction, and mean removal vs centered-covariance. The Bayes direction uses S−1Δ, while CARe uses (G+γI)−1 with G=S+μˉμˉ⊤+SB. The total-moment metric preserves the Bayes direction when the extra terms do not rotate S−1Δ, i.e. when μˉμˉ⊤+SB is small relative to S along the discriminative directions or lies (near-)orthogonal to the Δ. This is favorable when the shared mean and high within-class variance dominate the spectrum (as in CLIP) and the between-class term SB does not.
On the comparison: in our synthetic study under matched settings, removing the global mean before estimating the metric leaves the anisotropy–gain trend essentially unchanged (−2.4 at κ=1 to +31 at κ=1000), so the gain is driven by covariance anisotropy, not the mean offset. Full results are in our response to Reviewer STGv (Q2).
Q2: Does the trend persist with covariance trace and mean-to-noise ratio held fixed? Yes. We reran the synthetic study with the total trace held fixed, so both the covariance trace and the mean-to-noise ratio are constant and anisotropy is the only varying factor. Cosine accuracy stays flat (~68%) while CARe’s gain still grows monotonically with anisotropy (−2.4 at κ=1 to +31 at κ=1000), confirming the trend is not a signal-to-noise artifact. We also verified that centering the features first leaves the trend unchanged. Full results are in our response to Reviewer STGv (Q2).
Q4 :Absolute end-to-end latency and memory, η ∈ {1, 100}, both backbones We measured end-to-end latency (encoder plus readout) and peak memory on UCF101 over 2000 samples:
At η=1 the per-step eigendecomposition roughly doubles end-to-end latency (the 9.3× readout figure); this is the exact-reference setting, not the deployment one. At η=100 it is amortized and the overhead drops to +2.0% (ViT-B/16) and +1.2% (RN-50), with peak memory essentially unchanged (+3 and +17 MB). Since Table 8 shows η=100 matches η=1 accuracy, deployment gets the full accuracy of CARe at 1–2% latency and negligible memory.
Edge hardware. We have not measured on edge devices. CARe adds no trainable parameters, no extra forward pass, and only a fixed D×D accumulator, so its cost scales with the readout, not the encoder. The one device-sensitive part is the eigendecomposition, controlled by η; at η=100 it stays at ~1–2% of end-to-end latency even on the faster RN-50 encoder.