- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies cache retrieval in training-free, cache-based test-time adaptation of CLIP, observing that caches are organized by image–text pseudo-labels but retrieved by full-space image–image similarity, a mismatch that can misrank references and dilute localized class evidence. The proposed DCM is a training-free plug-in that preserves the base adapter's pseudo-labeling and cache update rules while improving retrieval along two axes, globally, by computing cache affinities in a fixed text-induced subspace built from the principal directions of a broad reference vocabulary, and locally, by building class-wise patch caches from the retained images' own ViT patch tokens, with patches selected by contrasting their support in the assigned pseudo-class against confusing classes, and query patches matched competitively across all classes' patch memories. A conditional margin analysis characterizes when projection and re-normalization can correct misleading reference rankings.
Across 10 cross-dataset benchmarks and the ImageNet shift suite with ResNet-50 and ViT-B/16, DCM integrated with both TDA and MCP yields consistent improvements over the base adapters.
Critical strengths
1. The paper clearly separates cache retrieval from cache construction and makes the improvements attributable by changing only the retrieval rule. As the authors point out, cache membership is determined by CLIP's image–text predictions while retrieval relies on image–image similarity, and these relations need not preserve the same class ordering. DCM targets exactly this mismatch while keeping the base adapter untouched, so the gains can be attributed to the retrieval change, and the experiments show consistent improvements over both base adapters on both backbones.
Critical weaknesses
1. The components are assembled from prior work. The text-induced subspace and the discriminative patch selection each come from cited prior work, so what the paper contributes is mainly their application to TTA cache retrieval, which is incremental.
Questions for authors
- Efficiency. It would be useful to see wall-clock time and memory usage of DCM. It would also be helpful to report the time and memory cost of each component separately, alongside their accuracy contributions.
- Accuracy over the online process. Since cache-based TTA accumulates evidence over the test stream, a curve of running accuracy versus the number of processed test samples, comparing DCM with the base adapters, would better reveal how performance evolves during adaptation and how the methods compare at different stream stages.
- The fusion weights and . The paper states that and control the global and local cache contributions but does not explain how their values were chosen. Are the two weights shared across all datasets or tuned per dataset? It would be helpful to see sensitivity results for them.
8 comments
Response to Reviewer hjYG [2/2]
Q1: Wall-clock time, GPU memory, and component-wise contributions.
We evaluate the cost of each component on datasets spanning 47–1000 classes, using ViT-B/16 on a single RTX 3090 (batch size 1; 64 views). Table 3-2 reports these measurements alongside average accuracy on 10 cross domain benchmarks and 5 OOD benchmarks.
Global matching provides accuracy gains at little additional cost. It improves average accuracy by 1.34 pts. on CD10 and 0.53 pts. on OOD5, adding at most 1.60 ms/img and 9.9 MiB across the evaluated datasets. Local retrieval provides a further 1.58 and 0.71 pts., respectively, over global matching alone, showing that the additional computation delivers complementary gains from finer-grained cache evidence. The full method remains computationally practical even with 1000 classes. On IN-V2, DCM requires 99.74 ms/img and 1.36 GiB of peak GPU memory. Its overhead is smaller on datasets with fewer classes: on DTD and Caltech101, it adds 17.64–19.39 ms/img and 32.1–67.8 MiB over TDA†. These measurements support a practical accuracy–cost trade-off across the evaluated class scales.
Table 3-2. Component-wise accuracy and computational cost (latency (ms/image)/ peak GPU memory (MiB)).
| Configuration | CD10 acc. (%) | OOD5 acc. (%) | DTD (C=47) | Caltech101 (C=101) | SUN397 (C=397) | IN-V2 (C=1000) |
|---|---|---|---|---|---|---|
| TDA† | 69.63 | 66.74 | 58.16 / 701.7 | 58.10 / 702.0 | 58.44 / 707.5 | 60.45 / 719.6 |
| TDA† + Global | 70.97 | 67.27 | 58.37 / 702.4 | 58.61 / 702.8 | 59.11 / 711.4 | 62.05 / 729.5 |
| TDA† + Local | 71.69 | 67.33 | 75.16 / 733.2 | 78.64 / 769.1 | 76.05 / 973.2 | 97.54 / 1384.6 |
| TDA† + Global + Local (DCM) | 72.55 | 67.98 | 75.80 / 733.8 | 77.49 / 769.8 | 76.77 / 976.7 | 99.74 / 1393.2 |
Q2: Accuracy throughout the online test stream.
DCM's advantage emerges early and generally widens as adaptation proceeds. Table 3-3 reports average cumulative accuracy across the ten cross-domain benchmarks at normalized stream checkpoints. DCM outperforms TDA† at every reported checkpoint, with the gain increasing from 0.84 points at 10% of the stream to 2.92 points at completion. These results show that improving cache retrieval benefits prediction across the evaluated stages of online adaptation. We will include the corresponding running-accuracy curve in the revised appendix.
Table 3-3. Average cumulative accuracy at different stages of the test stream.
| Stream progress (%) | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100 |
|---|---|---|---|---|---|---|---|---|---|---|
| TDA† (%) | 69.36 | 69.07 | 69.30 | 69.69 | 69.56 | 69.46 | 69.52 | 69.53 | 69.63 | 69.63 |
| TDA†+DCM (%) | 70.20 | 70.93 | 71.52 | 72.14 | 71.99 | 72.09 | 72.28 | 72.32 | 72.47 | 72.55 |
Q3: Selection and sensitivity of the fusion weights.
We select the global/local weights and the matching parameter on an additional small validation set for each dataset, consistent with the dataset-specific selection in prior cache based TTA methods, such as Tip-Adapter/TDA/MCP. The selected parameters fixed during test-time adaptation. We search and . We will report the exact selected values for each dataset in the appendix of the revised manuscript.
To assess sensitivity, we evaluate Pets and ImageNet-A, covering cross-domain recognition and ImageNet OOD shift. The validation-selected global weight and local weight are (0.75; 50) for Pets and (0.75; 20) for IN-A. We scale either the global or local contribution by factors ranging from 0 to 2, keeping all other parameters fixed. For TDA† + DCM, we scale the positive and negative cache weights by the same factor, preserving their relative balance and varying the overall global cache contribution.
Performance is not tied to a precise fusion-weight setting: increasing or decreasing either contribution by 25% changes accuracy by at most 0.25 percentage points on both datasets. Removing either contribution lowers accuracy, supporting the complementary roles of text-induced global matching and local patch retrieval.
Table 3-4. Accuracy (%) under fusion-weight scaling on Pets and ImageNet-A. A multiplier of 1.0 corresponds to the validation-selected configuration.
| Dataset | Scaled contribution | 0× | 0.25× | 0.5× | 0.75× | 1.0× | 1.25× | 1.5× | 2.0× |
|---|---|---|---|---|---|---|---|---|---|
| Pets | Global | 91.98 | 92.31 | 92.67 | 92.78 | 92.72 | 92.64 | 92.34 | 92.01 |
| Pets | Local | 91.33 | 91.74 | 91.96 | 92.50 | 92.72 | 92.48 | 92.31 | 91.82 |
| ImageNet-A | Global | 64.48 | 64.86 | 65.04 | 65.22 | 65.27 | 65.32 | 65.20 | 65.12 |
| ImageNet-A | Local | 64.26 | 64.82 | 64.98 | 65.19 | 65.27 | 65.18 | 65.26 | 64.96 |
Thank you for your detailed and thorough response.
Could you report the performance of the unified fusion weight parameters across all datasets? I believe that would better demonstrate the practicality of the method.
I also noticed that when testing accuracy throughout the online test stream, there was a dip in accuracy at the 50%. Could you analyze the reason for this? Also, on which datasets did the method fail to achieve a stable, sustained improvement?
Response to Reviewer hjYG [1/2]
We thank the reviewer for their thoughtful and constructive feedback. We appreciate their recognition of our motivation to address the mismatch between cache construction and retrieval, our design that isolates the effect of retrieval changes, and the consistent gains across both adapters and backbones. We address their concerns below.
DCM separates cache retrieval from cache construction and improves both the comparison geometry and spatial granularity of matching. It combines text-induced global matching with class-discriminative patch memory and class-conditioned, competitive patch matching, and provides a conditional analysis of when projection and re-normalization can correct misleading cache-reference rankings.
Global retrieval: separating cache construction from cache retrieval. [1] investigate the intra-modal misalignment hypothesis in CLIP. Our question concerns two different relations within cache-based TTA: the cache is organized using image–text pseudo-labels, whereas cached images contribute to subsequent predictions through image–image retrieval. Even correctly classified queries and references can produce misleading cache rankings. Appendix A explains how residual similarity and unequal projected feature magnitudes can reverse cache-reference rankings, and derives sufficient conditions under which projection and re-normalization recover the same-class ordering. Text-induced global matching alone improves TDA† by 1.34 pts. while preserving its cache construction rules(Table 3 in the manuscript), demonstrating that retrieval quality provides a separate source of improvement.
Local retrieval: class-discriminative memory and class-conditioned, competitive matching. Improving the global comparison geometry still leaves each image represented by a single pooled feature, which can dilute localized discriminative evidence. DCM therefore extends retrieval to patch-level correspondences using the existing patch tokens. CDPS [2] constructs patch sets from a predefined few-shot support set and aggregates the strongest within-class patch matches. DCM constructs its patch memory from the evolving test-time cache and uses cross-class competitive assignments to weight local visual correspondences. Thus, the contribution of a local match depends on both its within-class similarity and the matching evidence from competing visual memories.
Evidence for the contribution of DCM's retrieval design. Table 3-1 shows that directly combining text-PCA with CDPS-style matching does not recover DCM's full improvement. Under the same cached images, prompts, and tuning budget, DCM achieves 72.55 versus 71.02, with gains on nine of the ten datasets. Relative to global matching alone at 70.97%, these correspond to gains of 1.58 and 0.05 pts. Targeted ablations further identify the value of its retrieval choices. Cross-class competitive matching improves over within-class soft matching by 1.21 pts., demonstrating the benefit of weighting local matches using competing-class evidence. Class-conditioned query-patch selection improves over using all query patches by 0.51 pts., showing that selecting relevant query regions makes the patch memory more effective for prediction. These results show that DCM's local scoring and selection choices improve retrieval beyond the tested text-PCA + CDPS-style combination.
[1] Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP. CVPR 2026.
[2] Training-free Discriminative Patch Mining for Robust Few-Shot Recognition with CLIP. ECCV 2026.
Table 3-1. Retrieval comparisons on CD10 with TDA†.