acceptodds
Under review as a conference paper at ICLR 2027

Let Classes Compete: Discriminative Cache Matching for CLIP Test-Time Adaptation

Abstract

Cache-based test-time adaptation (TTA) improves CLIP under distribution shifts by retaining previous test images and reusing them as visual references. Orthogonal to how such caches are constructed, we study a complementary question: once an observation has been stored, how should a newly arriving test image retrieve useful evidence from it? We identify two limitations of conventional cache matching. First, cache membership is determined by CLIP's image–text predictions, whereas subsequent retrieval relies on full-space image–image similarity, so even correctly retained references can be misranked. Second, correcting the global comparison geometry still represents each image with a single pooled feature, which can overlook localized class evidence. We propose Discriminative Cache Matching (DCM), a training-free plug-in that preserves the pseudo-labeling and image-cache update rules of the underlying adapter while improving retrieval at both levels. Globally, DCM uses a fixed text-induced subspace to guide image–image cache matching. Locally, it reuses ViT patch tokens from the same retained images to perform fine-grained, class-competitive patch–patch retrieval. We further provide a conditional margin analysis characterizing when projection and re-normalization can correct misleading cache-reference rankings. Experiments on cross-dataset and out-of-distribution benchmarks demonstrate the effectiveness of DCM.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test-Time Adaptation;Vision-language Models;Distribution Shift

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hjYGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies cache retrieval in training-free, cache-based test-time adaptation of CLIP, observing that caches are organized by image–text pseudo-labels but retrieved by full-space image–image similarity, a mismatch that can misrank references and dilute localized class evidence. The proposed DCM is a training-free plug-in that preserves the base adapter's pseudo-labeling and cache update rules while improving retrieval along two axes, globally, by computing cache affinities in a fixed text-induced subspace built from the principal directions of a broad reference vocabulary, and locally, by building class-wise patch caches from the retained images' own ViT patch tokens, with patches selected by contrasting their support in the assigned pseudo-class against confusing classes, and query patches matched competitively across all classes' patch memories. A conditional margin analysis characterizes when projection and re-normalization can correct misleading reference rankings.

Across 10 cross-dataset benchmarks and the ImageNet shift suite with ResNet-50 and ViT-B/16, DCM integrated with both TDA and MCP yields consistent improvements over the base adapters.

Critical strengths

1. The paper clearly separates cache retrieval from cache construction and makes the improvements attributable by changing only the retrieval rule. As the authors point out, cache membership is determined by CLIP's image–text predictions while retrieval relies on image–image similarity, and these relations need not preserve the same class ordering. DCM targets exactly this mismatch while keeping the base adapter untouched, so the gains can be attributed to the retrieval change, and the experiments show consistent improvements over both base adapters on both backbones.

Critical weaknesses

1. The components are assembled from prior work. The text-induced subspace and the discriminative patch selection each come from cited prior work, so what the paper contributes is mainly their application to TTA cache retrieval, which is incremental.

Questions for authors
  1. Efficiency. It would be useful to see wall-clock time and memory usage of DCM. It would also be helpful to report the time and memory cost of each component separately, alongside their accuracy contributions.
  2. Accuracy over the online process. Since cache-based TTA accumulates evidence over the test stream, a curve of running accuracy versus the number of processed test samples, comparing DCM with the base adapters, would better reveal how performance evolves during adaptation and how the methods compare at different stream stages.
  3. The fusion weights αg\alpha _g and αloc\alpha _{loc}. The paper states that αg≥0\alpha_g \geq 0 and αloc≥0\alpha_{loc} \geq 0 control the global and local cache contributions but does not explain how their values were chosen. Are the two weights shared across all datasets or tuned per dataset? It would be helpful to see sensitivity results for them.
8 comments
AuthorsOpenReview ↗

Response to Reviewer hjYG [1/2]

We thank the reviewer for their thoughtful and constructive feedback. We appreciate their recognition of our motivation to address the mismatch between cache construction and retrieval, our design that isolates the effect of retrieval changes, and the consistent gains across both adapters and backbones. We address their concerns below.

W1: Technical contribution beyond existing work.

DCM separates cache retrieval from cache construction and improves both the comparison geometry and spatial granularity of matching. It combines text-induced global matching with class-discriminative patch memory and class-conditioned, competitive patch matching, and provides a conditional analysis of when projection and re-normalization can correct misleading cache-reference rankings.

Global retrieval: separating cache construction from cache retrieval. [1] investigate the intra-modal misalignment hypothesis in CLIP. Our question concerns two different relations within cache-based TTA: the cache is organized using image–text pseudo-labels, whereas cached images contribute to subsequent predictions through image–image retrieval. Even correctly classified queries and references can produce misleading cache rankings. Appendix A explains how residual similarity and unequal projected feature magnitudes can reverse cache-reference rankings, and derives sufficient conditions under which projection and re-normalization recover the same-class ordering. Text-induced global matching alone improves TDA† by 1.34 pts. while preserving its cache construction rules(Table 3 in the manuscript), demonstrating that retrieval quality provides a separate source of improvement.

Local retrieval: class-discriminative memory and class-conditioned, competitive matching. Improving the global comparison geometry still leaves each image represented by a single pooled feature, which can dilute localized discriminative evidence. DCM therefore extends retrieval to patch-level correspondences using the existing patch tokens. CDPS [2] constructs patch sets from a predefined few-shot support set and aggregates the strongest within-class patch matches. DCM constructs its patch memory from the evolving test-time cache and uses cross-class competitive assignments to weight local visual correspondences. Thus, the contribution of a local match depends on both its within-class similarity and the matching evidence from competing visual memories.

Evidence for the contribution of DCM's retrieval design. Table 3-1 shows that directly combining text-PCA with CDPS-style matching does not recover DCM's full improvement. Under the same cached images, prompts, and tuning budget, DCM achieves 72.55 versus 71.02, with gains on nine of the ten datasets. Relative to global matching alone at 70.97%, these correspond to gains of 1.58 and 0.05 pts. Targeted ablations further identify the value of its retrieval choices. Cross-class competitive matching improves over within-class soft matching by 1.21 pts., demonstrating the benefit of weighting local matches using competing-class evidence. Class-conditioned query-patch selection improves over using all query patches by 0.51 pts., showing that selecting relevant query regions makes the patch memory more effective for prediction. These results show that DCM's local scoring and selection choices improve retrieval beyond the tested text-PCA + CDPS-style combination.

[1] Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP. CVPR 2026.

[2] Training-free Discriminative Patch Mining for Robust Few-Shot Recognition with CLIP. ECCV 2026.

Table 3-1. Retrieval comparisons on CD10 with TDA†.

ConfigurationAvg. accuracy (%)
TDA†69.63
TDA† + Global matching only70.97
TDA† + Text-PCA + CDPS-style matching71.02
(TDA† + DCM) with within-class soft matching71.34
(TDA† + DCM) with mean-of-max matching71.38
(TDA† + DCM) with all query patches72.04
TDA† + DCM72.55
AuthorsOpenReview ↗

Response to Reviewer hjYG [2/2]

Q1: Wall-clock time, GPU memory, and component-wise contributions.

We evaluate the cost of each component on datasets spanning 47–1000 classes, using ViT-B/16 on a single RTX 3090 (batch size 1; 64 views). Table 3-2 reports these measurements alongside average accuracy on 10 cross domain benchmarks and 5 OOD benchmarks.

Global matching provides accuracy gains at little additional cost. It improves average accuracy by 1.34 pts. on CD10 and 0.53 pts. on OOD5, adding at most 1.60 ms/img and 9.9 MiB across the evaluated datasets. Local retrieval provides a further 1.58 and 0.71 pts., respectively, over global matching alone, showing that the additional computation delivers complementary gains from finer-grained cache evidence. The full method remains computationally practical even with 1000 classes. On IN-V2, DCM requires 99.74 ms/img and 1.36 GiB of peak GPU memory. Its overhead is smaller on datasets with fewer classes: on DTD and Caltech101, it adds 17.64–19.39 ms/img and 32.1–67.8 MiB over TDA†. These measurements support a practical accuracy–cost trade-off across the evaluated class scales.

Table 3-2. Component-wise accuracy and computational cost (latency (ms/image)/ peak GPU memory (MiB)).

ConfigurationCD10 acc. (%)OOD5 acc. (%)DTD (C=47)Caltech101 (C=101)SUN397 (C=397)IN-V2 (C=1000)
TDA†69.6366.7458.16 / 701.758.10 / 702.058.44 / 707.560.45 / 719.6
TDA† + Global70.9767.2758.37 / 702.458.61 / 702.859.11 / 711.462.05 / 729.5
TDA† + Local71.6967.3375.16 / 733.278.64 / 769.176.05 / 973.297.54 / 1384.6
TDA† + Global + Local (DCM)72.5567.9875.80 / 733.877.49 / 769.876.77 / 976.799.74 / 1393.2

Q2: Accuracy throughout the online test stream.

DCM's advantage emerges early and generally widens as adaptation proceeds. Table 3-3 reports average cumulative accuracy across the ten cross-domain benchmarks at normalized stream checkpoints. DCM outperforms TDA† at every reported checkpoint, with the gain increasing from 0.84 points at 10% of the stream to 2.92 points at completion. These results show that improving cache retrieval benefits prediction across the evaluated stages of online adaptation. We will include the corresponding running-accuracy curve in the revised appendix.

Table 3-3. Average cumulative accuracy at different stages of the test stream.

Stream progress (%)102030405060708090100
TDA† (%)69.3669.0769.3069.6969.5669.4669.5269.5369.6369.63
TDA†+DCM (%)70.2070.9371.5272.1471.9972.0972.2872.3272.4772.55

Q3: Selection and sensitivity of the fusion weights.

We select the global/local weights and the matching parameter β\beta on an additional small validation set for each dataset, consistent with the dataset-specific selection in prior cache based TTA methods, such as Tip-Adapter/TDA/MCP. The selected parameters fixed during test-time adaptation. We search αg∈0,0.05,0.25,0.5,0.75,1,5,10,αloc∈0,0.5,1,3,5,10,20,30,50\alpha_g\in\\{0,0.05,0.25,0.5,0.75,1,5,10\\},\alpha_{\mathrm{loc}}\in\\{0,0.5,1,3,5,10,20,30,50\\} and β∈75,100,150,300,500,750,1000\beta\in\\{75,100,150,300,500,750,1000\\}. We will report the exact selected values for each dataset in the appendix of the revised manuscript.

To assess sensitivity, we evaluate Pets and ImageNet-A, covering cross-domain recognition and ImageNet OOD shift. The validation-selected global weight and local weight are (0.75; 50) for Pets and (0.75; 20) for IN-A. We scale either the global or local contribution by factors ranging from 0 to 2, keeping all other parameters fixed. For TDA† + DCM, we scale the positive and negative cache weights by the same factor, preserving their relative balance and varying the overall global cache contribution.

Performance is not tied to a precise fusion-weight setting: increasing or decreasing either contribution by 25% changes accuracy by at most 0.25 percentage points on both datasets. Removing either contribution lowers accuracy, supporting the complementary roles of text-induced global matching and local patch retrieval.

Table 3-4. Accuracy (%) under fusion-weight scaling on Pets and ImageNet-A. A multiplier of 1.0 corresponds to the validation-selected configuration.

DatasetScaled contribution0×0.25×0.5×0.75×1.0×1.25×1.5×2.0×
PetsGlobal αg\alpha_{g}91.9892.3192.6792.7892.7292.6492.3492.01
PetsLocal αloc\alpha_{loc}91.3391.7491.9692.5092.7292.4892.3191.82
ImageNet-AGlobal αg\alpha_{g}64.4864.8665.0465.2265.2765.3265.2065.12
ImageNet-ALocal αloc\alpha_{loc}64.2664.8264.9865.1965.2765.1865.2664.96

Thank you for your detailed and thorough response.

Could you report the performance of the unified fusion weight parameters across all datasets? I believe that would better demonstrate the practicality of the method.

I also noticed that when testing accuracy throughout the online test stream, there was a dip in accuracy at the 50%. Could you analyze the reason for this? Also, on which datasets did the method fail to achieve a stable, sustained improvement?

Reviewer_5WVTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper looks at how cached test samples should be matched with new queries in CLIP-based test-time adaptation. The proposed DCM has two parts: a global branch that uses a fixed text-derived PCA subspace for image-level matching, and a local branch that builds a memory of discriminative patches. For the local branch, patches are selected based on how well they are supported by samples from the same class and how much they overlap with potentially confusing classes. At inference time, the method selects query patches for each class and uses soft assignments across the patch memory to compute local matching scores.

The authors evaluate DCM on cross-dataset classification and ImageNet distribution shifts, using both ViT-B/16 and RN50, with TDA and MCP as the underlying adapters.

Critical strengths
  1. I like the idea of looking at cache construction and cache retrieval separately. Most cache-based adaptation methods seem to focus more on what gets stored, so I think it makes sense to ask whether the matching process itself can be improved. Another nice aspect is that DCM does not require changing the original cache update rules, which could make it easier to use with other adapters.

  2. The experimental coverage is reasonably broad. The authors test the method across several datasets, two adapters, and two backbones. I appreciate that they do not limit the evaluation to a single adapter or backbone.

Critical weaknesses
  1. I am not fully convinced that the paper clearly establishes what is new about DCM. The global branch is closely related to the text-PCA projection in [1], the patch selection criteria in Eqs. (8)–(9) resemble those in CDPS [2], and the soft matching mechanism draws on ABS [3]. While I appreciate that the authors acknowledge these connections, I would like to see clearer evidence of what DCM adds beyond combining existing ideas. Table 3 only ablates the entire local branch, and Table 7 focuses on patch selection strategies. Neither isolates the contribution of cross-class competition or class-conditioned query patch selection. I think comparisons with a simpler text-PCA + CDPS-style baseline and an ablation without competitive soft assignment would help clarify where the improvements actually come from.

  2. I am also concerned about the stability of the local branch, particularly with a small patch budget. In Figure 4, retaining only one patch per image reduces accuracy to 62.14%, compared with 70.97% for the global-only variant and 63.58% for the original CLIP. This is a substantial drop, and I am not sure that insufficient patch information fully explains it. In Eqs. (11)–(12), the local scores depend on a softmax over the full patch memory followed by class-wise averaging, so their scale may change with memory size and patch budget. This makes the local matching temperature and fusion weight potentially important, yet their values and selection procedures are not clearly reported. I would like to understand whether the degradation comes from unreliable local evidence or could be partly addressed through better score calibration.

[1] Herzog, J., & Wang, Y. (2026). Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP. CVPR 2026.

[2] Ye, Z., Danier, D., Zhao, B., & Bilen, H. (2026). Training-free discriminative patch mining for robust few-shot recognition with CLIP. ECCV 2026.

[3] Cai, L., Kang, J., Li, S., Ma, W., Xie, B., Qin, Z., & Liang, J. (2025). From local details to global context: Advancing vision-language models with attention-based selection. ICML 2025.

Questions for authors

Regarding the actual contribution of the proposed matching mechanism:

  1. Could the authors compare DCM with a simpler combination of text-PCA projection and CDPS-style patch matching, using the same cached images, prompts, and tuning budget?
  2. I am particularly interested in what happens when the competitive soft assignment is removed. How much accuracy is actually gained from allowing patches to compete across classes, rather than matching them independently?

Regarding the sensitivity of the local branch:

  1. Could the authors provide the exact values of the local matching temperature and the global-local fusion weight, along with how they were selected?
  2. For Figure 4, were these parameters fixed across all patch budgets? I would be interested to see whether the large accuracy drop at a patch budget of one can be reduced by adjusting the relative contribution of the local branch.

Regarding the patch memory construction and update process:

  1. I have two questions about how the patch memory is constructed and updated. First, in Eq. (8), patches from the source image are excluded when computing within-class support. What happens if a class has only one cached image? Where does the positive support come from in this case?
  2. Second, Eq. (9) considers information from competing classes when selecting patches, but Algorithm 1 seems to update a class's patch memory only when its own image cache changes. If another class's cache changes, wouldn't this also affect which patches should be selected? I'm wondering whether this is an intentional approximation to save computation or if I have misunderstood the update process. It would also be helpful to clarify which cache update rules were used in the experiments.

Regarding the dependence on the CLIP backbone:

  1. The RN50 and ViT-B/16 results are helpful, but I am curious whether the improvements would still hold with a stronger CLIP backbone. My concern is that some of the gains might come from correcting limitations in the retrieval behavior of these relatively older backbones. If the visual representations are already stronger, would the proposed matching rules still provide a similar benefit?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 5WVT [1/3]

We thank the reviewer for their thoughtful and constructive feedback. We appreciate their recognition of the value of studying cache retrieval separately from cache construction, the potential ease of integrating DCM with other adapters while preserving their cache update rules, and our evaluation across multiple datasets, two adapters, and two backbones. We address their concerns below.

W1/Q1: Contributions beyond existing work.

DCM separates cache retrieval from cache construction and improves both the comparison geometry and spatial granularity of matching. It combines text-induced global matching with class-discriminative patch memory and class-conditioned, competitive patch matching, and provides a conditional analysis of when projection and re-normalization can correct misleading cache-reference rankings. The simple combination baseline and targeted ablations provide direct evidence of the value of our design.

Global retrieval: separating cache construction from cache retrieval. [1] investigate the intra-modal misalignment hypothesis in CLIP. We analyze two different relations within cache-based TTA: the cache is organized using image–text pseudo-labels, whereas cached images contribute to subsequent predictions through image–image retrieval. Even correctly classified queries and references can produce misleading cache rankings. Appendix A analyzes how residual similarity and unequal projected feature magnitudes affect these rankings and derives sufficient conditions for their correction through projection and re-normalization. Global matching alone improves TDA† by 1.34 pts. while preserving its cache construction rules (Table 3 in the manuscript), demonstrating the value of studying retrieval separately from construction.

Local retrieval: class-discriminative memory and class-conditioned, competitive matching. Global projection still represents each image through a single pooled feature, which can dilute localized discriminative evidence. DCM therefore extends retrieval to patch-level correspondences using the existing patch tokens. ABS [3] uses extra DINO-guided crops and corresponding contextual features to match image regions with relevant textual descriptions through soft matching. CDPS [2] constructs patch sets from a predefined few-shot support set and aggregates the strongest within-class patch matches. DCM constructs its patch memory from the evolving test-time cache and uses cross-class competitive assignments to weight local visual correspondences. Thus, the contribution of a local match depends on both its within-class similarity and the matching evidence from competing visual memories.

The simpler combination does not account for DCM’s full improvement. With the same cached images, prompts, and tuning budget, adding CDPS-style matching to global matching yields only 0.05 pts. over global matching alone (71.02% vs. 70.97%), whereas DCM’s local retrieval adds 1.58 pts. on the same global branch. DCM therefore exceeds the simpler combination by 1.53 pts. on average, with improvements on 9/10 datasets.

The targeted controls identify the benefits of the retrieval mechanisms. Replacing the cross-class assignment mass πc(q)\pi_c(q) with 1/C1/C preserves within-class soft matching but reduces accuracy by 1.21 pts. Mean-of-max matching is also 1.17 pts. below DCM. These comparisons demonstrate the benefit of weighting local matches using competing-class evidence. Using all query patches reduces accuracy by 0.51 pts., showing that selecting relevant query regions makes the patch memory more effective for prediction.

Table 2-1. Retrieval comparisons on CD10 using TDA† as the base adapter.

ConfigurationAvg. accuracy (%)
TDA†69.63
TDA† + global matching only70.97
TDA† + Text-PCA + CDPS-style matching71.02
(TDA† + DCM) with within-class soft matching71.34
(TDA† + DCM) with mean-of-max matching71.38
(TDA† + DCM) with all query patches72.04
TDA† + DCM72.55
AuthorsOpenReview ↗

Response to Reviewer 5WVT [2/3]

Q2: Contribution of cross-class competition.

The competitive weight in Eq. (11) decomposes into cross-class assignment and within-class soft matching:

\underbrace{ \frac{\sum_{u\in P_c}\exp(\beta q^\top u)} {\sum_{j=1}^{C}\sum_{u\in P_j}\exp(\beta q^\top u)} }_{\text{cross-class assignment }\pi_c(q)} \cdot \underbrace{ \frac{\exp(\beta q^\top p)} {\sum_{u\in P_c}\exp(\beta q^\top u)} }_{\text{within-class soft matching}}, \qquad p\in P_c.$$ We replace $\pi_c(q)$ with $1/C$, assigning equal mass to each class while preserving within-class soft matching. This removes query-dependent cross-class competition. We also compare against mean-of-max matching, which averages each query patch’s strongest match within the candidate class. **Cross-class competition contributes beyond matching within each class.** As shown in Table 2-1, DCM outperforms within-class soft matching and mean-of-max matching by 1.21 and 1.17 pts., respectively. These comparisons show that accounting for competing-class evidence is more effective than evaluating each class independently. The gains demonstrate the value of assessing a local match’s discriminative strength relative to competing classes, beyond its within-class similarity. > **Q3: Local matching temperature and weights.** We select the global/local weights and the matching parameter $\beta$ on an additional small validation set for each dataset, consistent with the dataset-specific selection in prior cache based TTA method, such as Tip-Adapter/TDA/MCP. For TDA† + DCM, $\alpha_g$ scales the positive and negative cache contributions by the same factor, preserving their relative balance. The selected parameters remain fixed during test-time adaptation. We search $\alpha_g\in\\{0,0.05,0.25,0.5,0.75,1,5,10\\},\alpha_{\mathrm{loc}}\in\\{0,0.5,1,3,5,10,20,30,50\\}$ and $\beta\in\\{75,100,150,300,500,750,1000\\}$. We will report the exact selected values for each dataset in the appendix of the revised manuscript. We also evaluate a shared configuration across all 15 datasets, fixing $\alpha_g=0.75$, $\alpha_{\mathrm{loc}}=5$, and $\beta_{\mathrm{loc}}=300$. As shown in Table 2-2, DCM improves over TDA† on all 15 datasets, yielding average gains of 2.42 pts. on CD10 and 0.95 pts. on OOD5. Average accuracy is 0.50 and 0.29 pts. below the dataset-specific configuration, respectively. This shared configuration therefore offers a practical trade-off, retaining most of DCM’s gains while simplifying parameter setup across datasets. Table 2-2. Performance with dataset-specific and shared parameters. | Configuration | Caltech101 | DTD | EuroSAT | Flowers | Pets | UCF101 | Food101 | SUN397 | Aircraft | Cars | CD10 Avg. | │ | IN-A | IN-R | IN-S | IN-V2 | IN | OOD5 Avg. | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|:---:|---:|---:|---:|---:|---:|---:| | TDA† | 94.16 | 53.61 | 61.79 | 75.36 | 90.71 | 70.90 | 86.19 | 68.91 | 28.32 | 66.40 | 69.63 | │ | 63.56 | 80.94 | 52.20 | 65.23 | 71.77 | 66.74 | | + DCM (per-dataset) | 94.81 | 56.21 | 67.33 | 76.78 | 92.72 | 76.02 | 87.12 | 70.64 | 32.28 | 71.63 | 72.55 | │ | 65.27 | 81.78 | 54.16 | 66.00 | 72.69 | 67.98 | | + DCM (shared) | 94.73 | 55.44 | 66.48 | 76.13 | 92.20 | 75.07 | 87.08 | 70.36 | 31.89 | 71.14 | **72.05** | │ | 64.76 | 81.68 | 53.77 | 65.68 | 72.56 | **67.69** | > **W2/Q4: Small patch budgets and local-score calibration.** In Figure 4, the fusion weights and local matching parameter $\beta$ are held fixed at their validation-selected values across patch budgets. Reselecting the params on an independent validation set at $K_{\mathrm{mem}}=1$ raises accuracy from 62.14% to 70.99%, essentially matching global-only performance (70.97%). The selected local weight is zero on 8/10 cross-domain datasets, with nonzero values only on DTD and EuroSAT. Recalibration therefore primarily suppresses the local contribution, indicating that one retained patch per view offers limited complementary value on most datasets even after adjusting local weight. This underscores our design objective of retaining sufficient class-discriminative patch information in the local memory to support subsequent query matching and complement global retrieval.
AuthorsOpenReview ↗

Response to Reviewer 5WVT [3/3]

Q5: Patch selection with a single cached image.

For a newly admitted image with no other cached image of the same pseudo-class, we select the top-KmemK_{\mathrm{mem}} patches in each view by CLS-to-patch attention. If another same-class image is available at admission, we apply the discriminative selection in Eqs. (8)–(9), excluding the source image when computing within-class support. Previously retained patches follow the admission-time selection schedule described in Q6.

We also compare with delayed activation, which postpones the class’s participation in local matching until a second image is cached, while retaining its candidate features for support calculations. With identical inputs and fixed parameters, the two strategies differ by only 0.10 pts.(Table 2-3), indicating that the final average accuracy is not strongly dependent on this initialization choice.

Table 2-3. Handling classes with a single cached image.

InitializationDelayed activationAttention fallback (ours)
CD10 Avg. accuracy (%)72.4572.55

Q6: Local patch-memory updates.

In our experiments, patch selection is performed when an image enters the cache, retaining 15 patches per view. The selection uses the patches already retained for other cached images to compute same-class support and competing-class penalties. These selected patches remain fixed while the image stays in the cache and are deleted when it is evicted. Whenever membership of a class cache changes, we pool the retained candidates of its current images and apply greedy coverage to recompress its matching memory to at most M=12M=12 representatives. Changes in competing classes can alter the scores in Eq. (9), but are incorporated into selections for newly admitted images rather than triggering reselection of existing patch sets. This incremental schedule avoids repeated patch selection over previously cached images. We will correct Algorithm 1 to distinguish admission-time candidate selection from class-memory recompression, and clarify the retained reference sets used in Eq. (8). The underlying image-cache admission and eviction rules remain those of the base adapter.

Q7: Generalization to a stronger CLIP backbone.

DCM remains effective with the stronger ViT-L/14 backbone. We evaluate zero-shot CLIP, TDA†, and TDA†+DCM on the ten cross-domain benchmarks. As shown in Table 2-4, DCM improves over TDA† on all ten datasets, yielding an average gain of 2.29 points. This consistent improvement shows that stronger visual representations do not eliminate the benefit of combining text-induced global matching with class-conditioned, competitive retrieval from class-discriminative patch memories, extending DCM’s effectiveness beyond the RN50 and ViT-B/16 backbones.

Table 2-4. Cross-domain accuracy (%) with CLIP ViT-L/14.

MethodCaltech101DTDEuroSATFlowersPetsUCF101Food101SUN397AircraftCarsAvg.
Zero-shot CLIP96.6361.9460.3279.3793.4976.0891.2872.3435.7378.5874.58
TDA†96.8462.4767.1081.6193.5478.4391.2672.7237.1179.1776.02
TDA†+DCM97.5765.6068.8384.6594.7482.1891.9874.7940.6882.1378.31
Reviewer_m5svOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies cache retrieval in training-free test-time adaptation for CLIP and proposes Discriminative Cache Matching (DCM). DCM improves retrieval through text-induced global matching, which projects image features into a semantic subspace derived from text embeddings, and class-competitive local matching using discriminative patch memories. The method preserves the underlying cache construction and update rules and can be integrated with existing adapters such as TDA and MCP. Experiments across cross-domain and ImageNet distribution-shift benchmarks demonstrate consistent improvements with both ViT-B/16 and ResNet-50 backbones.

Critical strengths

The paper presents a clearly motivated method and demonstrates consistent improvements across multiple datasets, cache-based adapters, and CLIP backbones.

Critical weaknesses
  1. The analysis of the local branch is incomplete. Although the paper separates the global and local branches, the local module still combines discriminative patch selection, class-conditioned query-patch selection, and cross-class competitive matching. The individual contribution of these components, particularly the class competition emphasized in the title, is not isolated. Sensitivity analyses for KqryK_{\mathrm{qry}}, NN, and η\eta, as well as robustness to erroneous cache pseudo-labels, are also missing.

  2. The computational trade-off is not evaluated. DCM rebuilds the relevant patch memories whenever the underlying image cache changes and performs class-competitive patch matching for every query. These operations may introduce substantial computation and memory overhead, particularly for large-class datasets. The paper should report runtime, throughput, GPU memory usage, and scaling with the number of classes and cached patches to clarify the cost associated with the reported performance improvements.

Questions for authors

Please refer to the concerns raised in the Critical Weaknesses section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer m5sv [1/2]

We thank the reviewer for their thoughtful and constructive feedback. We appreciate their recognition of the clear motivation of our approach and its consistent improvements across datasets, adapters, and backbones. We address their questions below.

W1(a): Contributions of the local components.

We evaluate the three local components with TDA† and CLIP ViT-B/16 on the ten cross-domain benchmarks (CD10).

  • Cross-class competitive matching.

We compare DCM with two alternatives. The weight in Eq. (11) decomposes as wq(p)=πc(q),[softmax⁡u∈Pc(βq⊤u)]pw_q(p)=\pi_c(q)\\,[\operatorname{softmax}_{u\in P_c}(\beta q^\top u)]_p, where πc(q)\pi_c(q) is the total weight assigned to class cc. Within-class soft matching fixes πc(q)=1/C\pi_c(q)=1/C, removing query-dependent cross-class competition while preserving within-class soft weighting. Mean-of-max matching instead averages the strongest match for each selected query patch: scmax⁡=∣Qc∣−1∑q∈Qcmax⁡p∈Pcq⊤ps_c^{\max}=|Q_c|^{-1}\sum_{q\in Q_c}\max_{p\in P_c}q^\top p. As shown in Table 1-1(a), DCM outperforms these alternatives by 1.21 and 1.17 pts., respectively. Accounting for cross-class competition is therefore more effective than evaluating each class independently, demonstrating the value of competing-class evidence in determining how strongly a local match should support a prediction.

Table 1-1(a). Matching strategies.

MethodAccuracy (%)
Within-class soft matching71.34
Mean-of-max matching71.38
Cross-class competitive matching (ours)72.55
  • Class-conditioned query-patch selection.

We replace the query selection in Eq. (10) with all available query patches (Qc=FQ_c=F for every class), retaining the patch memory and matching rule. As shown in Table 1-1(b), class-conditioned selection improves accuracy by 0.51 points despite using fewer query patches. This demonstrates that retaining more patches does not necessarily provide more effective evidence, supporting our design of focusing retrieval on regions relevant to each candidate class.

Table 1-1(b). Query-patch selection.

MethodAccuracy (%)
All query patches72.04
Class-conditioned selection (ours)72.55
  • Discriminative memory-patch selection.

We compare our selection criterion with random, low-entropy, and attention-based selection. As shown in Table 1-1(c), our criterion outperforms all three alternatives, improving over low-entropy and attention-based selection by 1.54 and 0.73 points, respectively. These results demonstrate the effectiveness of selecting local patterns that recur across images assigned to the same pseudo-class but are less prevalent among confusing classes.

Table 1-1(c). Memory-patch selection.

MethodAccuracy (%)
Random72.17
Low Entropy71.01
Attention71.82
Ours72.55

W1(b): Sensitivity to KqryK_{qry}, N, and η\eta.

We evaluate the sensitivity of the local branch to KqryK_{qry}, N, and η\eta on the 10 cross-domain benchmarks, varying one parameter at a time while keeping the others fixed. Table 1-2 summarizes the results.

Table 1-2(a). Query-patch budget.

KqryK_{qry}510152025
Avg. (%)71.6472.2172.5572.5272.43

The improvement over the smallest query budget favors retaining a broader set of local correspondences for each candidate class. The plateau at larger budgets shows that this benefit does not require a precise query-patch count.

Table 1-2(b). Number of confusing classes.

N13510
Avg. (%)72.5272.5372.5572.52

The stability also holds at the dataset level: the range across N is at most 0.18 percentage points on every dataset. Thus, the memory-selection rule does not depend on precisely choosing the number of competing classes within the tested range.

Table 1-2(c). Inter-class penalty in patch selection.

η\eta00.050.10.30.51.0
Avg. (%)72.3872.4072.5572.2972.1871.53

A modest inter-class penalty is useful, but stronger suppression of cross-class overlap is not uniformly better. This supports using competing-class evidence to refine the within-class support criterion when selecting local representatives.

AuthorsOpenReview ↗

Response to Reviewer m5sv [2/2]

W1(c): Robustness to erroneous cache pseudo-labels.

We evaluate TDA† and TDA†+DCM on the ten cross-domain benchmarks using ViT-B/16, introducing noise into positive-cache labels at three levels: 0%, 20%, and 40%. For randomly selected cache images, we replace the label with zero-shot CLIP’s highest-probability class excluding the ground-truth class, and apply the same label to their patches in DCM’s local memory. Both methods use identical cached images and label perturbations, with the cache trajectory and hyperparameters held fixed. The reported actual error rate includes both existing pseudo-label errors and the injected errors.

DCM’s retrieval gains persist despite substantial cache-label errors. As shown in Table 1-3, its advantage remains 2.60 points when the mean error rate rises to 33.69%, and even at 50.16% error, DCM still improves accuracy by 1.33 points. These results show that DCM continues to extract useful predictive evidence from the cache under substantially corrupted class assignments, extending its benefits beyond the original pseudo-label conditions.

Table 1-3. Accuracy with erroneous cache pseudo-labels.

Noise level (%)Actual cache-label error rate (%)TDA† acc. (%)TDA†+DCM acc. (%)
016.6769.6372.55
2033.6965.3167.91
4050.1661.3762.70

W2: Runtime, throughput, and GPU memory.

To assess how DCM’s computational cost scales with the number of classes, we evaluate DTD (47 classes), Caltech101 (101), SUN397 (397), and ImageNet-V2 (1000), using ViT-B/16 on a single RTX 3090 (batch size 1; 64 views for the adaptation methods).

Runtime: DCM reuses patch tokens from the base adapter’s existing ViT forwards, avoiding separate region encoding. Its additional latency remains manageable: only 17.64–19.39 ms/image over TDA† for 47–397 classes, and even at 1,000 classes, the increase is limited to 39.29 ms/image. At this largest class scale, DCM requires 64.7% less inference time than BoostAdapter.

Memory and scaling: With at most M=12M=12 compressed patch representatives per class, patch-matching computation scales as O(QCMD)O(QCMD) and representative storage as O(CMD)O(CMD), where CC, QQ, and DD denote the class count, candidate query-patch count, and feature dimension. For fixed QQ and DD, these bounds grow linearly with the class count and representative budget, while the per-class capacity prevents representative storage from growing indefinitely with the test stream. The practical overhead is small at lower class counts (32.1–67.8 MiB for 47–101 classes) and even at 1000 classes, the additional 673.6 MiB brings the total footprint to just 1.36 GiB.

Accuracy–cost trade-off: These costs accompany average accuracy gains of 2.92 pts. on Cross Domain benchmarks and 1.24 pts. on OOD5 benchmarks over TDA†. Together, the results establish that DCM’s richer cache retrieval provides accuracy gains with practical runtime and GPU memory requirements, including at larger class scales.

Table 1-4. Computational cost across class scales. We reports latency (ms/image) / peak GPU memory (MiB).

Dataset (classes)TDA†TDA†+DCMBoostAdapter
DTD (47)58.16 / 701.775.80 / 733.871.49 / 715.2
Caltech101 (101)58.10 / 702.077.49 / 769.884.08 / 723.0
SUN397 (397)58.44 / 707.576.77 / 976.7171.41 / 824.3
ImageNet-V2 (1000)60.45 / 719.699.74 / 1393.2282.36 / 709.7

Dear Reviewer m5sv,

We sincerely thank you for your valuable feedback, which has helped us improve the quality of our work.As the discussion deadline approaches,we kindly request that you review our rebuttal.We’d appreciate your thoughts on whether this addresses your concern, and we’d be happy to clarify anything that remains unclear.

Thank you again for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers and Area Chair for their careful evaluation and discussion. We appreciate their recognition of the value of separating cache retrieval from cache construction, DCM’s compatibility with existing adapters while preserving their image-cache rules, and consistent improvements across datasets, adapters and backbones. We summarize the additional evidence and manuscript revisions below.

1. Contribution and distinction from prior work.

Motivation. Existing cache-based TTA methods typically assign retained images to classes using image–text predictions, but retrieve them using full-space image–image similarity. Correct cache assignments do not necessarily ensure correct cache-reference rankings. Moreover, pooled image features can dilute localized discriminative evidence. These limitations motivate us to study cache retrieval separately from cache construction, asking: how should retained visual evidence be matched to a newly arriving test image? Herzog and Wang use text-PCA projection to investigate task ambiguity in image–image similarity and reassess the intra-modal misalignment hypothesis. We focus on how comparison geometry and spatial granularity affect retrieval from an evolving test-time cache.

Mechanism. Globally, DCM computes query–cache similarities after text-induced projection and re-normalization. Our conditional margin analysis characterizes how residual similarity and unequal projected magnitudes affect reference rankings and derives sample-wise sufficient conditions for recovering the correct same-class ordering. Locally, DCM combines class-discriminative patch memory with class-conditioned, competitive matching. CDPS constructs class-discriminative patch sets from a predefined labeled few-shot support set and aggregates the strongest within-class patch matches. ABS uses cross-category soft matching to emphasize textual descriptions relevant to each image region and suppress irrelevant region–description matches. DCM operates on an unlabeled test stream and constructs its patch memories from the evolving pseudo-labeled cache. For each candidate class, DCM selects relevant query patches and weights their matches across all class-wise visual memories before aggregating class evidence. Thus, the contribution of a local match depends on both its within-class similarity and the matching evidence from competing visual memories.

Experimental evidence. On the ten cross-domain benchmarks with TDA† and ViT-B/16, DCM achieves 72.55%, compared with 71.02% for the tested text-PCA + CDPS-style matching under the same cached images, prompts, and tuning budget (response to 5WVT, Table 2-1). Targeted ablations show a 1.21-point benefit of cross-class competitive matching over within-class soft matching and a 0.51-point benefit of the proposed query-patch selection over using all query patches (revised Table 5).

2. Computational efficiency.

DCM offers a practical accuracy–cost trade-off. By reusing patch tokens from the base adapter’s existing encoder passes, DCM improves recognition accuracy without additional region encoding. Component-wise measurements show that global matching adds little computational cost, while local retrieval provides further accuracy gains at additional cost. On ImageNet-V2 with 1,000 classes, DCM requires 64.7% less inference time than BoostAdapter. Detailed runtime, memory, and class-scale comparisons are provided in revised Table 6 and our responses to m5sv and hjYG.

3. Generality and robustness.

With ViT-L/14, DCM improves over TDA† on all ten cross-domain datasets, yielding an average gain of 2.29 points (response to 5WVT, Q7). In the ViT-B/16 online evaluations, DCM maintains positive average cumulative gains at all evaluated stream checkpoints, with full-stream improvements preserved under the tested targeted reordering and random permutations (follow-up responses to hjYG). DCM also retains gains under matched cache-label corruption (response to m5sv, W1(c)). Shared fusion weights and a shared local matching parameter preserve improvements on all 15 datasets (Appendix C). The small-budget analysis further shows that recalibration recovers near-global-only performance primarily by suppressing the local contribution (response to 5WVT, W2/Q4).

4. Manuscript revisions.

We clarified patch-memory construction and incremental updates in Section 4.2 and Algorithm 1 (Appendix D). The revised main text includes local-component ablations (Table 5), parameter sensitivity analyses (Figure 3), and computational-cost comparisons (Table 6). Appendix C reports hyperparameter settings and shared-configuration results, while Figure 4 presents memory-budget sensitivity and online cumulative accuracy. The main text revisions are highlighted in blue.

We hope these clarifications and additional results address the main concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.