- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SHARP studies the online temporal adaptation of CLIP through the compact class-conditional distribution modeling. It operates in the angular feature space, where the embeddings are normalized, and it maintains the class-wise means and covariances for distribution scoring and prototype recalibration. The latter is combined with multiple instance caches and residual updates, while the CLIP encoders are frozen.
Critical strengths
- The reported gains are broad: over all tasks, datasets, and types of visual concepts (fine grained recognition, texture, scene, action). The 0.97 and 1.16 points improvement in average cross-dataset accuracy for the two backbones used (vs. MCP) is considerable. It suggests the practical value of the overall method.
- The probabilistic formulation has nice properties: the covariance update is mean-shift corrected, the Gaussian score uses the same shrunk covariance for the quadratic and log-determinant terms. These are solid theoretical grounding for most of the paper and help it address the numerical instabilities of the early streams.
Critical weaknesses
-
The effect of either of the two design decisions is limited to some extent. SH introduce an alteration to the feature mapping as well as the dimension. But, the participants mentioned in Table 3 are not able to differentiate their angle shape from typical dimension reduction. A linear projection learning with the same output dimension as text-fitting could serve as an interesting control. For VAPR variance and correlation gates, the numbers are summed: SH and VAPR respectively account for 0.33 and 0.32 points. However, since no spread over runs is shown, these are less likely to be trusted entirely on their own and together than the full pipeline.
-
The cost is not specified (correctly limits this claim of 'compactness' to the output space). Equation (1) remains expressed in terms of L dense d x d matrices. For the given dimensions, these are 67 million projection parameters for ViT-B/16 (268 million for Res Net-50, not counting class covariances).Only the runtime of ViT-B/16 for Image Net-A is given; no information is provided about peak memory. All other metrics are only given as observations for calculating accuracy; the ratio of online to initial time is not provided (regardless of the terminology used in the text), although it is also relevant for the accuracy-cost relation.
Questions for authors
-
It would be interesting to compare this to a learned 256-dimensional linear projection, fit to the same text-similarity objective. This could help reveal whether the angular construction is a key determinant.
-
Need clarification regarding the process followed for choosing the final set of hyperparameters and some investigation whether the improvement of SH and VAPR varies across stream orders and augmentation seeds.
-
Need to determine if the reported runtime also includes the 200 step projection fitting as specified in Appendix A.1. Need to explain the maximum GPU memory needed and whether or not refactor each class covariance after each sample.
4 comments
Response to Reviewer yvVp Question 1-3
Q1: Table R3 provides the comparison under the same pipeline (ViT-B/16, ten benchmarks, , and covariance shrinkage). The learned linear projection is fitted using the same text-similarity objective as SH. It achieves 72.60% average accuracy, compared with 72.82% for SH, corresponding to a 0.22 improvement. This result shows that SH achieves a modest advantage over the learned linear projection under the evaluated setting. However, without matched multi-seed comparisons, we cannot establish the statistical reliability of this difference or conclude that the angular construction itself is the determining factor. We will clarify this distinction in the revised manuscript.
Q2: Hyperparameter selection. The distribution-weight schedule, cache capacities, and were selected using aggregate accuracy on the evaluation benchmarks, without a separate held-out validation set. A single configuration was shared across all datasets and both backbones. The selection protocol and sensitivity of the reported gains to these choices are discussed in our responses to Reviewer RWNA, Weakness 1 and Question 1. Stream order and augmentation seeds. Table R1 reports three runs using different global random seeds. Each seed jointly controls stream order, augmentation sampling, and projection initialization. The reported standard deviations therefore reflect their combined variability. Table R4 evaluates projection initialization stability on EuroSAT using three paired runs, while keeping stream order, augmentations, and other settings fixed. The fitted mapping achieves accuracy, with a paired improvement of % over the unfitted mapping. This provides additional evidence of stability across projection initializations, although it does not isolate stream-order or augmentation effects. Variation of SH and VAPR gains. Table R1 reports a 3-seed standard deviation of 0.14% for the full SHARP pipeline on the ViT-B/16 cross-dataset evaluation. In the cumulative ablation of Table 3, SH and VAPR provide incremental gains of 0.33 and 0.32%, respectively. However, we have not run this ablation over multiple seeds, so the full-pipeline standard deviation does not establish the variability of these individual component gains. We will extend the +DA, +SH, and +VAPR ablations to matched random seeds, keeping the evaluation protocol consistent, and report the mean and standard deviation of the paired component-level improvements.
Q3: The reported runtime excludes projection fitting. The submitted implementation recomputes the inverse of every class covariance after each sample. The largest peak GPU memory we measured is 2.63 GB on Caltech101 (ViT-B/16), reached during the one-time projection fitting. Projection fitting. The 35-minute ImageNet-A runtime in Table 5 excludes the one-time 200-step projection fitting. On DTD (ViT-B/16, 1,692 samples), fitting takes 1.77 s, compared with 850.79 s for online adaptation, accounting for approximately 0.2% of the online adaptation time. We have not separately measured fitting time on ImageNet-A. The projection is fitted on the class-text embeddings before the test stream and remains frozen afterwards. Covariance computation. Class statistics are updated incrementally using Eqs. 8–9, but the inverse of each class covariance is recomputed explicitly after every sample for distribution scoring. This involves dense matrix inversions per sample (e.g., 47 on DTD). Appendix A.3 describes a Cholesky-based computation, whereas the submitted implementation uses explicit inversion. The two methods yield the same score mathematically, and we will correct the appendix to accurately reflect the implementation. Under Eq.10, each covariance update consists of a scalar rescaling, an isotropic shift, and a rank-one correction. Therefore, the inverse could potentially be updated incrementally. However, all results reported in this submission use explicit inversion. Runtime and memory. All timings in this rebuttal and Table 5 of the submission were obtained on a single NVIDIA RTX 3060, which was shared with other jobs during some runs. They should therefore be regarded as indicative. We will re-time the experiments on a dedicated GPU in the revision.In particular, we have not yet reconciled the DTD per-image runtime (approximately 0.50 s) with the Caltech101 result in Table R5 (approximately 0.28 s). Peak GPU memory is 2.63 GB on Caltech101 and 1.13 GB on DTD. Caltech101 measurement is the peak over the entire run, including projection fitting, during which AdamW maintains approximately MB of parameter-related storage for the mapping. The DTD measurement covers online adaptation only. These measurements involve different datasets and profiling scopes and are therefore not directly comparable. Peak memory on ImageNet-A has not yet been measured. We will report this measurement in the revised manuscript.
Thanks for your thorough response and the effort you've put in. The extra comparisons and clarifications are appreciated. The linear projection baseline was helpful, and the cost breakdown improved transparency.
That said, one concern persists. As noted in W1, VAPR's variance and correlation gating were assessed together. Although the 0.32-point gain validates the combined module, it doesn't confirm if both gates are effective or if one dominates the improvement. Would the total performance boost still be convincing if we decoupled these two gating mechanisms?
Response to Reviewer yvVp
We thank the reviewer for suggesting that we disentangle the contributions of the two gating mechanisms in VAPR. We conducted an additional ablation evaluating the variance gate (V-Gate) and the correlation gate (C-Gate) separately and jointly, while keeping all other components and experimental settings unchanged. As shown in Table R7, the model without VAPR achieves 72.82% average accuracy across ten cross-dataset benchmarks. Enabling only the V-Gate increases accuracy to 72.93% (+0.11 points), while enabling only the C-Gate yields 73.06% (+0.24 points). Enabling both gates achieves the highest average accuracy of 73.14% (+0.32 points). The decoupled results show that the C-Gate provides the larger observed contribution. The V-Gate provides a smaller positive increment, improving accuracy by 0.11 points on its own and by a further 0.08 points when added to the C-Gate. Conversely, adding the C-Gate to the V-Gate improves accuracy by 0.21 points. These comparisons support using both gates based on the reported average accuracies. However, the reliability of the smaller V-Gate increment remains to be assessed through paired comparisons across multiple seeds. On Caltech101, the V-Gate and C-Gate individually add 1.4 and 1.8 ms/image, respectively. Together, they increase online time from 276.2 to 279.1 ms/image, an overhead of approximately 1.05%. All configurations report a whole-run peak GPU memory of 2.63 GB at the displayed precision. These timings were measured on a shared GPU and are indicative. We will include this decoupled ablation in the revised manuscript and, as stated in our earlier response, extend it to matched random seeds. We will also revise the description of VAPR to state that the correlation gate is the main contributor and the variance gate a smaller one.
Table R7. Decoupled ablation of the variance and correlation gates in VAPR with ViT-B/16. All configurations are built upon the same SHARP model with all other components and experimental settings unchanged. Accuracy is averaged over ten cross-dataset benchmarks. Online time and peak GPU memory are measured on Caltech101; online time excludes the one-time projection fitting, and peak memory covers the entire run.
| Configuration | V-Gate | C-Gate | Avg. Acc. (%) | Avg. Δ (pp) | Online Time (ms/img) | Peak Mem (GB) |
|---|---|---|---|---|---|---|
| +SH (No VAPR) | ✗ | ✗ | 72.82 | -- | 276.2 | 2.63 |
| +SH + Variance | ✓ | ✗ | 72.93 | +0.11 | 277.6 | 2.63 |
| +SH + Correlation | ✗ | ✓ | 73.06 | +0.24 | 278.0 | 2.63 |
| +SH + VAPR (Both) | ✓ | ✓ | 73.14 | +0.32 | 279.1 | 2.63 |
Response to Reviewer yvVp Critical Weaknesses
W1: Linear-projection control. Table R3 provides the requested comparison with a learned 256-dimensional linear projection, fitted using the same text-similarity objective as SH and evaluated under the same adaptation pipeline (ViT-B/16, ten benchmarks, and shrunk covariance). The learned linear projection achieves 72.60%, compared with 72.82% for SH, yielding a modest improvement of 0.22 percentage points. Without matched multi-seed comparisons, we cannot establish the statistical reliability of this difference or attribute the improvement uniquely to the angular construction. We will therefore narrow the corresponding claim in the revised manuscript. Variability of component gains. We agree that the reliability of individual component gains should be evaluated separately from that of the full pipeline. Table R3 reports a three-seed standard deviation of 0.14 percentage points for the full SHARP pipeline on the ViT-B/16 cross-dataset evaluation. However, this does not characterize the variability of individual component-level improvements. In the cumulative ablation reported in Table 3 of the submission, SH improves accuracy from 72.49% to 72.82% (+0.33 points), while VAPR further improves accuracy to 73.14% (+0.32 points). These gains are conditional on the preceding components, and their statistical reliability cannot be established from the full-pipeline standard deviation alone. Furthermore, the variance and correlation gates of VAPR have not been independently ablated in the submission. We will extend the ablation study using matched random seeds, independently evaluate the two VAPR gates, and report the mean and standard deviation of the component-level gains in the revised manuscript.
W2: We agree that the reviewer's parameter counts are correct. We clarify the mapping size, covariance storage, initialization overhead, and measured GPU memory below. Projection parameters. For ViT-B/16 (d=512), the mapping contains 67.1M parameters, corresponding to approximately 268 MB in FP32. For ResNet-50 (d=1024), it contains 256×10242≈268M parameters, corresponding to approximately 1.07 GB in FP32. Class covariance storage. Each set of class-conditional covariance matrices contains K×256×256 elements. For ImageNet (K=1000), this requires approximately 262 MB in FP32 per covariance tensor set, excluding other model states and intermediate buffers. Initialization versus online runtime. As detailed in our response to Question 3, the one-time 200-step projection fitting on DTD with ViT-B/16 takes 1.77 s, compared with 850.79 s for online adaptation. Thus, projection fitting accounts for approximately 0.21% of the online adaptation time in this setting. These timings are indicative because the GPU was shared with other jobs during some measurements. The corresponding projection fitting time on ImageNet-A has not yet been measured separately. Peak GPU memory. The measured peak GPU memory on Caltech101 is 2.63 GB (Table R5), covering the entire run, including projection fitting. During fitting, AdamW maintains the mapping parameters, their gradients, and two optimizer moment estimates, requiring approximately 4×268MB≈1.07GB of parameter-related storage. On DTD, the measured peak memory during the online stage is 1130.2 MB, excluding projection fitting. These measurements involve different datasets and profiling scopes and are therefore not directly comparable. Peak GPU memory on ImageNet-A with ViT-B/16, corresponding to the setting used for the runtime comparison in Table 5 of the submission, has not yet been measured. We will report this measurement in the revision. Overall, we will revise the efficiency section to explicitly distinguish mapping parameters, class-covariance storage, one-time initialization overhead, and online peak-memory consumption, providing a more transparent account of the computational and memory costs.