acceptodds
Under review as a conference paper at ICLR 2027

Retrieval Geometry Shapes Cache-Based Clip Adaptation

11 views

Abstract

Cache-based test-time adaptation improves CLIP predictions by storing and retrieving examples from the target stream while keeping the model frozen. However, existing methods largely treat the feature space used for image–image retrieval as fixed, leaving open how much adaptation depends on the retrieval space itself. We study this question by fixing the memory and changing only the retrieval encoder, finding that the same memory can yield very different gains: across sixteen retrieval spaces, ImageNet-A cache gain ranges from at most +0.44 points for CLIP and MAE to +19.7 ± 0.4 for DINOv2-L, while label-free retrieval-space selection retains 98% of oracle gain on ImageNet-V2. These results show that memory quality depends not only on which examples are stored, but also on how they are retrieved. Motivated by this finding, we propose MARC (Memory Augmented Retrieval for CLIP), a training-free system that uses frozen CLIP for prediction and DINOv2-B for retrieval with a single fusion weight. A single-view cache repairs 1,074 ± 21 baseline errors, compared with 878 ± 4 for a 64-view ensemble, at roughly one seventh of the cost. Across four ImageNet distribution shifts, MARC reaches a 67.91% OOD average and, at matched DINOv2-B scale and eight views, achieves 64.17 ± 0.31% versus 62.75 ± 0.15% for a graph-based cache system while running 2.6 times faster. Overall, our results establish retrieval space as a first-order design choice for robust cache-based adaptation in remote sensing, scientific imaging, and changing visual environments.

Primary area:
optimization
Keywords:
Test-Time Adaptation, Distribution Shift, Retrieval Geometry, Training-Free Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xvDBOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how much of the benefit of cache-based test-time adaptation for CLIP comes from the feature space used for image–image retrieval, as opposed to the cache mechanism itself. The setup separates the two roles. Frozen CLIP produces the base logits btb_t, the pseudo-labels y^t\hat y_t and the admission priorities πt\pi_t, and a second frozen encoder gsg_s only provides query and key embeddings. Because admission never depends on gsg_s, the retained index sets are identical across retrieval spaces (Prop. 1). Swapping gsg_s therefore changes only the matched query–key similarities, and all comparisons can be paired per image.

With sixteen retrieval encoders on ImageNet-A/V2, the authors report three main findings:

  • The oracle-weight cache gain on ImageNet-A ranges from roughly zero (CLIP ViT-B/16, MAE, BEiT) to about +20 points (DINOv2-L).
  • Labeled neighborhood purity@10 ranks this gain well (ρ=0.959\rho = 0.959 / 0.9430.943). Anti-hubness tracks the useful capacity K⋆K^\star better than it tracks gain.
  • A pseudo-label version of purity selects the best space on 5 of 8 streams, with 1.48 points of mean regret.

The cache gain also shrinks as more augmented views are added: +9.30 at M=1M=1 versus +6.56 at M=64M=64 for DINOv2-B on ImageNet-A.

The resulting reference system, MARC, combines CLIP prediction, DINOv2-B retrieval, and a single fusion weight. It reaches a 67.91 OOD average on the four ImageNet shifts at M=64M=64. At matched M=8M=8 on ImageNet-A, it obtains 64.17 versus 62.75 for COSMIC at about 2.6x lower latency.

Validation includes:

  • three seeds for most rows, with paired CIs and McNemar tests;
  • a second predictor (CLIP RN50);
  • five admission policies;
  • six cross-domain datasets, plus Aircraft and Food101 reported as failure cases.
Critical strengths

1. The intervention is clean, and the authors exploit it properly. Prior work that brings DINOv2 into the cache, COSMIC in particular, changes the encoder together with cache construction and aggregation. In those systems it is impossible to tell whether DINOv2 helps because of its neighborhoods or because of everything around it. Here the retained-population term in Eq. (5) is removed by construction. Prop. 1 itself is almost definitional, and I don't count that against the paper, because the value lies in what it enables: every space is compared against the same no-cache predictor on the same images, and the reported paired intervals and exact tests make sense. I also appreciate that the negative results stay in the paper, including EuroSAT, the Aircraft/Food101 losses at w=10w=10, and the composite score discarded in App. F. This is the main reason I take the empirical claims seriously.

2. The practical message is clear and robust. The retrieval encoder matters more than the knobs people usually tune. A single-view DINOv2-B cache beats the 64-view no-cache ensemble (60.14 vs. 58.69) at roughly 7x lower cost, and single-view DINOv2-L beats 64-view DINOv2-B. The ordering survives random and FIFO admission (the DINOv2-L vs. DINOv2-B gap stays around 9.6 points), a different predictor backbone, and several seeds. Anyone building a TDA-style system can act on this directly.

Critical weaknesses

1. The central finding is close to what one would expect, and "geometry" is not separated from prior exposure to the target label space.

The cache logit in Eq. (3) is a kernel-weighted kNN vote over pseudo-labeled keys, and purity@10 is essentially kNN label agreement measured on the same test split. A strong rank correlation between the two is therefore not surprising. The kNN advantage of DINOv2 over CLIP is also already known from the DINOv2 paper and from COSMIC. Labeled purity is computed with ground truth on the full evaluation split (Sec. 3.4), so it works as an offline diagnostic rather than a selection rule. On the method side, MARC is TDA's positive cache with keys moved to DINOv2 space and the negative cache removed. That is fine for a reference system, but it is not a methodological contribution in itself.

The confound worries me more. On ImageNet-V2, the two best spaces are DeiT-III (+5.06) and the supervised ViT (+3.48), both trained with ImageNet-1k labels, which are exactly the target classes. As far as I know, the LVD-142M curation behind DINOv2 retrieved images close to ImageNet and to several of the cross-domain benchmarks used here. Part of what the paper calls retrieval geometry may therefore be prior exposure to the target label space or data distribution. This matters for the repeated framing around remote sensing and scientific imaging, where no such encoder exists. The one remote-sensing stream in the paper, EuroSAT, is exactly where the purity ranking breaks down.

2. The deployable and benchmark claims are weaker than the headline numbers suggest.

(a) Most retrieval-space results are oracle envelopes over ww and KK on the evaluation stream. The ImageNet-V2 to ImageNet-A weight transfer works, but these two streams share a label space and a predictor regime. In the cross-domain setting, the fixed w=10w=10 costs 8.46 points on Aircraft and 4.79 on Food101, and the eight-stream average cache contribution drops from +3.32 to +0.84. Cache-logit scale differs by encoder, so choosing ww without labels is a real open problem rather than a detail.

(b) The comparison with COSMIC does not support "better," only "competitive, simpler, and faster":

  • In Table 2, MARC's OOD advantage (67.91 vs. 67.66) uses M=64M=64 against COSMIC's M=8M=8. At M=4M=4, MARC is at 66.79.
  • COSMIC is ahead on ImageNet, V2 and Sketch.
  • At DINOv2-L scale, COSMIC leads by about two points (72.83 vs. 70.86, Table A18).
  • The matched M=8M=8 comparison covers only ImageNet-A, and its seed interval over stream orders includes zero (+1.12 [−0.62, +2.86], App. R).

The large gaps over CLIP-only baselines in Table 2 (e.g., 53.00 vs. 48.20 with RN50) mostly reflect the added encoder.

Presentation. The paper is very dense, and the same quantity appears with different values in different places. DINOv2-L's ImageNet-A gain is given as 19.36, 19.74 ± 0.40, 20.06 ± 0.34, 19.16 and 18.19, and CLIP's as 0.44, 0.56 and 0.62. Each value is explained somewhere, but the reader has to reconstruct which number is which. The Table A9 caption says two seeds while the text says three. Some appendix text reads like a rebuttal log (e.g., "DINOv2-B, previously omitted for lacking a third seed, is now measured") and should be rewritten for a standalone paper.

Questions for authors
  1. Can you separate geometry from label/data exposure? For example, could you report the intervention restricted to encoders never trained on ImageNet labels or ImageNet-curated data? For the cross-domain streams, could you check whether those datasets were among DINOv2's curation sources? If label-supervised ImageNet encoders such as DeiT-III are allowed as retrieval spaces, I would also like to know whether the authors still consider "training-free" the right framing.

  2. Pseudo-purity definition. Is only yiy_i replaced by the CLIP prediction, or both yiy_i and yjy_j? If both, pseudo-purity rewards neighborhoods that agree with CLIP's decisions, including its mistakes. An encoder that clusters CLIP's errors together would then score well while being useless as a cache. On the three streams where pseudo-purity misses the best space, what does it pick instead, and why?

  3. Label-free choice of ww. Have you tried a label-free way to set ww per encoder, for instance rescaling LtsL^s_t to match the spread of btb_t on the unlabeled stream? Does such a scheme remove the Aircraft and Food101 losses? An answer here would make the deployability claims much more convincing.

  4. COSMIC comparison. Are matched-MM results for MARC and COSMIC available on V2, R, Sketch and the cross-domain suite? Is COSMIC's 14x14 fusion grid, tuned on the evaluation stream, also behind the other COSMIC entries in Table 2? I would also find TDA with DINOv2 keys and its negative cache kept to be the most direct baseline for MARC. Have you run it?

Smaller clarifications

  1. Please state which value is canonical for each headline number (DINOv2-L and CLIP gains on ImageNet-A), and fix the seed count in Table A9.
7 comments
AuthorsOpenReview ↗

Response to Reviewer xvDB (Part: 1/5)

We thank the reviewer for a close reading of the identification argument. You note that prior systems such as COSMIC [Huang2025] change the encoder together with cache construction and aggregation, so "it is impossible to tell whether DINOv2 helps because of its neighborhoods or because of everything around it," and that removing the retained-population term in Eq. (5) is what makes the paired intervals and exact tests meaningful. We also appreciate the attention to the negative results we retained (EuroSAT, Aircraft/Food101, and the discarded composite score) and to the practical importance of the retrieval encoder. We address each question with an additional controlled evaluation below. All corresponding revisions are marked in blue in the paper.

Below we address the weaknesses and questions separately:


Weaknesses

  • W1. The central finding is close to expected; purity is near-tautological with kNN voting; MARC is TDA with keys moved; geometry is not separated from exposure to the target label space

  • Response: We agree that DINOv2's kNN strength is known and that MARC is deliberately close to TDA [Karmanov2024]. The contribution is the controlled quantification of how retrieval space affects cache adaptation; we address the four parts in turn.

(a) Purity is not the cache's accuracy restated. Purity and cache gain are computed on different objects. Purity is static, offline, symmetric over all n images, and uses true labels. Cache gain is causal (only earlier images), uses CLIP pseudo-labels (with CLIP's errors), keeps only the top-K per pseudo-class by priority, and counts only where the fused logit flips a CLIP error (Eq. 9). Four results show that the link between them is an empirical finding that could have failed:

  1. Not every kNN statistic predicts gain. Anti-hubness is computed from the same kNN graph, yet ranks gain at Spearman 0.54 over 16 spaces, against 0.96 for purity (Table 1; calibration-subsample intervals [0.47, 0.68] vs. [0.956, 0.962] are disjoint).
  2. The relation has a clear boundary. On BloodMNIST (new), CLIP is at 16.0% accuracy (chance 12.5%) and purity ranks gain at only 0.24; 12 of 16 encoders give exactly 0. The sufficient condition in the purity-threshold corollary (margin appendix) requires local purity minus pseudo-label error to exceed about 1/2, which is not satisfied when about 84% of stored labels are wrong. This boundary behavior shows that the empirical association is not mechanically guaranteed.
  3. Purity measured on different data predicts gain on the test stream. Purity computed on a class-matched ImageNet-val split (no test images) ranks the held-out ImageNet-A gains at Spearman 0.83 over the 11 encoders trained without ImageNet labels, and purity on ImageNet-val ranks ImageNet-V2 gains at 0.92.
  4. It predicts a reversal not implied by generic kNN performance. On EuroSAT, the older DINO v1 outperforms DINOv2-B by 7.29 points (+9.31 vs. +2.02), and purity predicts this ordering (90.9 vs. 85.1). Over six cross-domain datasets, the sign of the purity difference matches the sign of the gain difference in 6 of 6 cases (Pearson 0.96).

(b) What is new beyond "DINOv2 has good neighbors." The DINOv2 paper [Oquab2024] reports kNN classification accuracy; COSMIC adopts DINOv2 as one part of a larger system. Neither measures the share of cache-based adaptation attributable to retrieval space. With all else fixed, our intervention quantifies that share (+0.44 to +19.74 on ImageNet-A), tests it in other released implementations, and shows that additional views have diminishing value once retrieval is strong (+9.30 at M = 1 vs. +6.56 at M = 64). Related controlled analyses have clarified which components account for gains in few-shot classification [Chen2019], domain generalization [Gulrajani2021], and network pruning [Liu2019]; here we isolate the retrieval key space in cache-based TTA.

(c) "MARC is TDA's positive cache with keys in DINOv2 space." We agree, and this is by design: MARC is a reference system that changes one variable. Following the reviewer's request, we evaluated TDA with DINOv2 keys while retaining its negative cache, using the identical 6,862 images evaluated by TDA's released loader (Q4 explains the subset).

New analysis (Table R1): MARC vs. TDA on the same 6,862 ImageNet-A images, 64 views, 3 seeds each, image-clustered 95% intervals.

ConfigurationAccuracyMARC minus configuration
MARC (paper defaults: K = 16, w = 10)64.50
TDA, CLIP base only59.66+4.84 [+3.97, +5.71]
TDA as published (CLIP keys, negative cache)60.18+4.31 [+3.45, +5.17]
TDA, DINOv2 keys, TDA's weight (w = 1)60.43+4.07 [+3.22, +4.92]
TDA, DINOv2 keys, w = 2065.24-0.74 [-1.49, +0.00]
TDA, DINOv2 keys, w = 30 (best on the test stream)65.58-1.08 [-1.82, -0.35]

Once TDA retrieves with DINOv2 keys and a weight suited to their scale, it reaches MARC's accuracy; CLIP keys provide no gain over the tested weight range in TDA's implementation. On ImageNet-R, TDA with DINOv2 keys at the weight chosen on ImageNet-V2 gains +2.51 [+2.27, +2.78] over TDA as published. The component ablation (Sec. 4, Final Design) is consistent with this result: adding pseudo-label refinement, a negative memory, and adaptive fusion lowers ImageNet-A accuracy from 65.25 to 64.12, 64.14, and 62.12. These components provide no measurable improvement in this ablation; the released-code intervention supports the same retrieval-space conclusion.

(d) Geometry vs. exposure, and the remote-sensing framing. We agree that exposure to the target label space or data is one way an encoder acquires high purity, and we reworded the claim accordingly: purity mediates cache gain, whatever its source. The evidence below shows that exposure alone does not predict gain while purity does.

New analysis (Table R2): contrasts that separate data from purity.

ContrastEncoder A (gain, purity)Encoder B (gain, purity)Exposure predictsPurity predictsObserved
Same architecture, different data (IN-A)DINOv3-L web (+17.86, 65.8)DINOv3-L satellite (+0.08, 4.8)AAA
Large uncurated data vs. curated data (IN-A)SEER 1B Instagram (+0.05, 11.6)DINOv2-B (+10.32, 49.0)unclearBB
Same data, registers removed (IN-A)DINOv2-B (+10.32, 49.0)DINOv2-B no registers (+8.11, 47.4)tieAA
Curated vs. uncurated, stream outside curation (EuroSAT)DINOv2-B (+2.02, 85.1)DINO v1 (+9.31, 90.9)ABB
No ImageNet labels or curation (IN-A)CLIP-L/14, WIT data (+5.52, 42.8)CLIP-B/16 (+0.44, 29.7)tieAA

Across 14 control encoders, purity ranks gain at Spearman 0.96 (ImageNet-A) and 0.94 (ImageNet-V2). On remote sensing specifically: EuroSAT is where the purity ranking among the five DINOv2-family spaces was unresolved, because their purities lie within 14.4 points and their gains overlap across seeds (App. "EuroSAT failure case"). Once DINO v1 is included, purity identifies the winner on EuroSAT, and on the new remote-sensing stream RESISC45 purity ranks 17 encoders at Spearman 0.84 (best: DINOv3-L web, +6.22). The cache gain itself on EuroSAT is well resolved: +7.32 [+6.21, +8.43] at M = 4.

Revised text (Sec. 6):

"Purity is acquired through pretraining data, recipe, and resolution, and on ImageNet-A the high-purity encoders were all trained on web-scale curated data or ImageNet labels. We therefore do not claim that retrieval geometry is independent of data exposure. We claim that purity mediates cache gain: it ranks 14 control encoders at Spearman 0.96 and predicts the EuroSAT case in which an uncurated encoder outperforms a curated one."

How this resolves the concern: the purity-gain association is specific to purity, has a documented boundary, transfers across datasets, and predicts a reversal. The requested TDA baseline attributes the larger improvement to replacing the retrieval keys, while exposure is treated as one source of retrieval purity. These changes are stated in blue in the revised analysis and discussion.

  • W2. Deployable and benchmark claims are weaker than the headline numbers: (a) oracle envelopes and fixed-w losses; (b) the COSMIC comparison supports "competitive, simpler, faster," not "better"

(a) Oracle envelopes and the fixed weight.

  • Response: We agree that the oracle envelope should be an upper bound only. We added a held-out protocol for every encoder: (K, w) are chosen on ImageNet-val restricted to ImageNet-A's 200 classes (50 images per class, no ImageNet-A images) and applied unchanged. This also handles the per-encoder scale of the cache logit you point out, because each encoder gets its own validated weight.

New analysis (Table R3): oracle vs. validation-selected gain on ImageNet-A, 3 replay seeds; both columns use the same seeds (selected rows; all 16 in our response to Reviewer 83v1).

EncoderOracleSelected on validationKept
DINOv2-L+19.76+19.06 (online evaluation: [+18.08, +20.02])96%
DINOv2-B+9.96+8.96 (online evaluation: +8.99 [+8.19, +9.80])90%
DeiT-III+8.33+7.2587%
CLIP-L/14+5.88+4.9985%
CLIP-B/16+0.72+0.4968%
Spearman over 16 encoders0.97

On the cross-domain streams, the losses at fixed w = 10 on Aircraft and Food101 come from the weight, not from the cache (per-stream weight: +1.39 and +0.50). Q3 below gives the label-free rule.

We also disclose a correction found while re-running the ImageNet-V2 to ImageNet-A transfer: at the kernel stated in the paper, (alpha, beta) = (2, 5), the transferred gain for DINOv2-L is +14.52, not the value in the held-out transfer table, which used stream-specific kernel constants. Table R3 replaces it as the deployable headline.

(b) The COSMIC comparison.

  • Response: We agree with your reading, and the revised paper uses "competitive with COSMIC at 2.6x lower latency." We extended the matched comparison to every stream (Q4, Table R5). With 10 matched orders on ImageNet-A, the seed interval no longer includes zero: +1.11 [+0.76, +1.46] across orders (+1.11 [+0.67, +1.55] image-clustered). Elsewhere COSMIC as released is ahead on several streams, but its released protocol tunes two betas from a 14x14 grid on each test stream; with betas fixed from another stream, the two systems are within 1.8 points on every ImageNet shift, and the mean over 8 cross-domain datasets is +0.30 to +0.78 for MARC.

The gaps over CLIP-only baselines in Table 2 reflect the additional frozen encoder. The revised Table 2 caption states this explicitly, and Table R3 of our response to Reviewer s4gy shows that replacing SCA's retrieval keys with the same encoder yields +4.45 (ImageNet-A) and +3.12 (ImageNet-V2). Thus, the intervention is not specific to MARC.

Revised text (Table 2 caption, added sentence):

"Methods in the CLIP + DINOv2-B block use a second frozen encoder for retrieval; the gap to the CLIP-only block is the key-space effect that this paper isolates, and the new appendix section on plug-in experiments shows that it transfers to SCA and TDA."

How this resolves the concern: every headline now has a version selected without evaluation-stream labels, the COSMIC claim is stated as competitive, and the comparison is matched across all streams.

  • W3. Presentation: inconsistent values for the same quantity, A9 seed count, rebuttal-log wording

  • Response: We agree; see Q5 for the canonical value of each headline number and the corrections.


Questions

  • Q1. Separate geometry from label/data exposure; DINOv2 curation sources; is "training-free" the right framing?

  • Response: Encoders without ImageNet labels. Over the 11 encoders trained without ImageNet labels (dropping supervised ViT, DeiT-III, ConvNeXt, ResNet-50, and BEiT), selection on the class-matched validation set picks DINOv2-L, and validation purity ranks held-out ImageNet-A gain at Spearman 0.83 (validation gain: 0.87). Among encoders trained on neither ImageNet labels nor ImageNet-curated data, CLIP-L/14 (WIT-400M) reaches +5.52 with purity 42.8, while SEER (1B uncurated Instagram images) and DINOv3-L satellite reach +0.05 and +0.08 with purity 11.6 and 4.8. Each of these encoders sits where its purity predicts.

Curation sources. Per the DINOv2 data curation, LVD-142M retrieval used Caltech101, DTD, Flowers102, and Pets, and did not use UCF101 or EuroSAT. On the two streams outside curation, DINOv2-B leads DINO v1 by +0.70 on UCF101 and trails it by 7.29 on EuroSAT; purity predicts both signs (83.8 vs. 74.9; 85.1 vs. 90.9).

Training-free. We use "training-free" as TDA and COSMIC do: no parameter is updated at test time and no target label is used. A DeiT-III retrieval encoder trained on ImageNet-1k labels has the same status as CLIP pretrained on captions, but we agree that on ImageNet-variant streams it shares the target label space. The revised Table 1 marks the five label-supervised rows, and MARC itself uses DINOv2, which uses no labels.

Reviewer_s4gyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work targets a critical limitation of retrieval-cache test-time adaptation (TTA): CLIP’s image-text-aligned representations fail to form high-quality image-image neighborhoods under distribution shifts. The authors propose a dual-cache framework, decoupling sample quality screening (CLIP admission cache) and nearest-neighbour retrieval for prediction (retrieval cache), leveraging neighborhood purity and anti-hubness to govern sample admission and cache capacity allocation. Extensive analytical and ablation experiments are conducted across distribution-shift benchmarks; results verify the importance of tailored retrieval space and demonstrate the superiority of the dual-cache design over prior single-cache TTA baselines.

Critical strengths
  1. The paper investigates an interesting perspective that is orthogonal to existing cache-based TTA approaches.
  2. The authors provide comprehensive analytical experiments to validate the importance of the retrieval space.
Critical weaknesses
  1. The work requires a substantially more rigorous and well-grounded motivation. Figure 1 alone is insufficient to support the core argument: representations optimized for image-text matching may not yield the most informative image-image neighborhoods under distribution shifts. Moreover, a number of recent works have improved cache-based TTA performance within the CLIP embedding space. The manuscript omits analysis and discussion of their limitations, which raises concerns regarding the sufficiency of the motivation.
  2. The experimental comparisons are not sufficiently fair, for three reasons. First, the paper lacks comparisons against recent state-of-the-art cache-based TTA methods. Second, evaluations are only conducted on vanilla CLIP memory, without validating the proposed design atop stronger cache-based TTA baselines. This makes it impossible to verify the necessity of the proposed design. Third, the set of evaluation datasets is incomplete.
  3. The readability of this manuscript needs substantial improvement. Several terms (e.g., anti-hubness) are introduced without prior definition. Some claims lack natural logical flow and require more rigorous justification, such as “Neighborhood purity predicts gain, anti-hubness captures useful capacity”. Key equations are not accompanied by explicit physical interpretation. Collectively, these issues raise the cognitive burden for readers and hurt overall readability.
Questions for authors
  1. The schematic in Figure 1 cannot adequately substantiate the motivation. Recent state-of-the-art cache-based methods including DLAE, MCP and ProtoDCS demonstrate that the CLIP space can be effective from alternative viewpoints, yet related analysis and discussion are absent in the paper.
  2. Certain claims in the introduction are logically opaque, for instance, “Neighborhood purity predicts the best space” and “anti-hubness captures useful capacity”. These statements are presented without rigorous logical justification, making them hard to follow at the introductory stage.
  3. While the method section covers the overall pipeline, its readability remains poor. I have re-read this section multiple times and still find multiple points ambiguous. Equations such as Eq.(5) and Eq.(9) lack explicit physical interpretation, which hinders understanding of subsequent conclusions. Many claims are stated in this section without adequate supporting material or sufficient details. The dynamic dual-cache pipeline is described only by text and equations without algorithm pseudocode, making the sample admission and eviction rules ambiguous.
  4. The proof of Proposition 1 in Appendix Q remains incomplete. This obscures the analysis immediately following the proposition, e.g., the statement “only samples whose fused decision margin is smaller than the induced … matrix at a fixed cache population”.
  5. The experimental benchmark lacks fair comparisons against recent advanced VLM-based approaches, including SCA, DLAE, ProtoDCS, MCP, ADAPT, and A*Memory.
  6. The current experiments are insufficient to fairly validate the necessity of the dual-memory design. I understand that the proposed retrieval space can be plugged into any other cache-based TTA methods. Therefore, to fairly validate its contributions and demonstrate the necessity of this dual-memory design, it is necessary to evaluate the performance gains when integrated with other state-of-the-art cache-based TTA approaches, rather than only comparing against vanilla CLIP memory construction baselines. Unfortunately, such evaluations are absent from the current submission.
  7. Full experiments on all ten cross-domain datasets should be provided for a fair comparison.
8 comments
AuthorsOpenReview ↗

Response to Reviewer s4gy (Part: 1/5)

We thank the reviewer for describing our perspective as "orthogonal to existing cache-based TTA approaches" and for recognizing that the analytical experiments validate the importance of the retrieval space. In response to the requests for released-code interventions and measured motivation, we replaced only the key encoder inside SCA's released implementation (NeurIPS 2025), with no added hyperparameter, increasing accuracy by +4.45 points on ImageNet-A and +3.12 on ImageNet-V2. All corresponding revisions are marked in blue in the paper.

Before the point-by-point answers, one clarification shapes several of them. The paper's contribution is the controlled intervention on the retrieval (key) space and the purity diagnostic, not a dual-cache architecture. The CLIP memory in MARC has zero output weight; it exists only so that admission depends on CLIP alone, which is what makes the retained samples identical across encoders (Prop. 1). Admission computed from the CLIP logits without that memory (K_C = 0) satisfies Prop. 1 equally, and we show below that it gives the same accuracy. We have therefore simplified MARC to a single memory, which makes the key encoder the only difference from a standard TDA-style positive cache.

Below we address the weaknesses and questions. Related questions are answered together with the weakness they expand on.


Weaknesses

  • W1 / Q1. Motivation rests on Figure 1; recent methods (DLAE, MCP, ProtoDCS) show CLIP space can be effective

"The schematic in Figure 1 cannot adequately substantiate the motivation. Recent state-of-the-art cache-based methods including DLAE, MCP and ProtoDCS demonstrate that the CLIP space can be effective from alternative viewpoints, yet related analysis and discussion are absent in the paper."

  • Response: We agree that a schematic should not carry the motivation, and that the paper should not suggest that CLIP space cannot work. Our claim is narrower and measured: with everything else fixed, retrieving in CLIP space leaves most of the attainable cache gain unrealized. The revised Introduction motivates this with four measurements already in the paper and one new one.

Measured motivation (Table R1), ImageNet-A, M = 1, identical retained samples.

MeasurementCLIP ViT-B/16 keysDINOv2-L keysSource
Neighborhood purity@1029.767.3Table 1
Cache gain, oracle weight+0.44+19.36Table 1
Cache gain, weight selected on held-out validation+0.49+19.06new (Reviewer 83v1, Table R1)
Cache scored alone, CLIP logits discarded27.0064.20App. Table "cache-only prediction"
CLIP keys after mean subtraction and whitening+1.07n/aSec. 5.1

The cache-only row provides a direct controlled comparison: with the same stored CLIP pseudo-labels, a CLIP-space cache predicts 27.00% of ImageNet-A correctly and a DINOv2-L-space cache predicts 64.20%. Because the stored labels are fixed, this comparison isolates the neighborhood geometry.

How the recent methods relate. Each of them improves a component that we hold fixed, while retrieving in CLIP space:

MethodWhat it changesRetrieval spaceTest-time gradient updates
MCP / MCP++ [Chen2025] (ICCV 2025)prototypes from entropy, align, and negative caches to make classes compactCLIPMCP: no; MCP++: residual fine-tuning
DLAE [Wu2026] (CVPR 2026)logit recalibration and an exploratory cache that admits boundary samplesCLIPyes (loss on textual representations)
ProtoDCS [Luo2026] (arXiv 2026)open-set separation (GMM double check) and prototype updatesCLIPyes (prototype level)
A²Memory [A2Memory2026] (ICML 2026)soft attribute-based assignment and associative memoryCLIPyes (delta-rule updates)
ADAPT [Zhang2025] (NeurIPS 2025)class-conditional Gaussian model with closed-form inferenceCLIPno
SCA [Guan2025] (NeurIPS 2025)incrementally updated feature statistics instead of stored featuresCLIPno
COSMIC [Huang2025] (CVPR 2025)dual semantic graphs and clique hyper-classesCLIP + DINOv2no

MCP is the closest in spirit to our motivation: its authors report a positive correlation between cache-based performance and intra-class compactness, and then improve compactness inside CLIP space with prototypes [Chen2025]. Our purity measure is the neighborhood-level counterpart of that observation. Changing the retrieval space moves purity from 29.7 to 67.3, while the SCA intervention below shows that this change can complement improvements made within CLIP space: SCA gains +4.45 / +3.12 points when only its key encoder is replaced. Thus, CLIP-space cache refinement and retrieval-space selection are complementary directions.

Revised text (Introduction, second paragraph):

"Recent methods improve cache-based adaptation within CLIP space by refining prototypes [MCP], statistics [SCA], admission [DLAE], or memory assignment [A²Memory], and MCP links cache gains to intra-class compactness. We ask a complementary question: how much does the space itself matter? With predictions, pseudo-labels, and retained samples held fixed, CLIP-space keys reach a neighborhood purity of 29.7 on ImageNet-A and a cache gain below 1 point, whereas DINOv2-L keys reach 67.3 and +19.06 points with hyperparameters selected on held-out data."

How this resolves the concern: the motivation is now a set of measured quantities rather than a schematic, the recent CLIP-space methods are discussed by what they change, and the plug-in result shows that our axis adds to theirs.

  • W2. Fairness of experimental comparisons (W2a: recent baselines; W2b: plug-in into stronger methods and necessity of the dual memory; W2c: all ten datasets) / Q5, Q6, Q7

W2a / Q5. Comparison with recent methods (SCA, DLAE, ProtoDCS, MCP, ADAPT, A²Memory)

  • Response: We agree that the benchmark table was missing recent methods. We re-ran four methods in their released code (TDA, SCA, BoostAdapter, COSMIC) and added published numbers for the others under the shared ImageNet protocol.

New analysis (Table R2): published results, CLIP ViT-B/16, top-1 (%). Methods marked † are our runs.

MethodVisual encodersIN-AIN-V2IN-RIN-SketchOOD avg
TDA [Karmanov2024]CLIP60.1164.6780.2450.5463.89
MCP [Chen2025]CLIP59.7165.1681.5953.8965.09
MCP++ [Chen2025]CLIP59.7865.7781.7354.3965.42
ADAPT, online [Zhang2025]CLIP63.3264.6480.6653.1365.44
A²Memory [A2Memory2026]CLIP64.3865.3880.9853.2165.99
COSMIC† (M = 8) [Huang2025]CLIP + DINOv2-B62.7567.5482.3957.9567.66
MARC† (M = 4)CLIP + DINOv2-B63.0065.7382.1856.2666.79
MARC† (M = 64)CLIP + DINOv2-B65.2566.5083.1556.7667.91

DLAE and ProtoDCS are not in this table because their protocols differ: DLAE optimizes textual representations with a loss at test time and reports cross-dataset results [Wu2026], and ProtoDCS targets open-set TTA on CIFAR-10/100-C and Tiny-ImageNet-C [Luo2026]. We discuss both in the revised related work.

Table R2 is not intended to establish a uniform ranking. MARC uses an additional frozen encoder, and the table therefore groups methods by encoder resources, as Table 2 does. Methods in the first block improve the cache within CLIP space, while the intervention below shows that changing the key space provides a complementary gain. For a matched-resource comparison with COSMIC, which also uses DINOv2, see our response to Reviewer 83v1 (W2): when neither system tunes on the test stream, the two are within 1.8 points on every ImageNet shift.

W2b / Q6. Plug the retrieval space into stronger cache methods; is the dual memory necessary?

"it is necessary to evaluate the performance gains when integrated with other state-of-the-art cache-based TTA approaches, rather than only comparing against vanilla CLIP memory construction baselines."

  • Response: We performed the requested intervention in the released implementations of three methods. In each case, the method's own CLIP pass produces its base logits, pseudo-labels, and admission; only the vectors used for retrieval are replaced by DINOv2-B features of the unaugmented view.

New analysis (Table R3): key-space plug-in into released code. Paired per image; 95% intervals.

Method (released code)StreamAs released (CLIP keys)+ DINOv2-B keysPaired differenceWeight for DINOv2 keys
SCA [Guan2025] (3 seeds)ImageNet-A64.2668.71+4.45 [+3.83, +5.06]SCA's own entropy fusion (no additional tuning)
SCA (3 seeds)ImageNet-V265.5368.65+3.12 [+2.72, +3.52]SCA's own entropy fusion (no additional tuning)
TDA [Karmanov2024] (seed 0)ImageNet-R80.7483.25+2.51 [+2.27, +2.78]selected on ImageNet-V2
TDA (3 seeds)ImageNet-A60.1865.03+4.86 (per-image mean; columns rounded separately)selected on ImageNet-V2
BoostAdapter [Zhang2024] (3 seeds)ImageNet-V265.2365.50+0.27 (+0.20 to +0.36 per seed)fixed w = 10
BoostAdapter (3 seeds)ImageNet-A64.11no gain at any tested weightw in {0.5, ..., 30}

Notes on fidelity. Each released implementation was first checked against its published number: SCA ImageNet-V2 65.53 vs. 65.38, TDA ImageNet-A 60.18 vs. 60.11, BoostAdapter ImageNet-A 64.11 vs. 64.53. The source of SCA's ImageNet-A no-cache offset (64.26 vs. 60.33) remains unresolved; however, +4.45 is a paired difference within the same run and is unaffected by that offset. We also found that TDA's released data loader skips any ImageNet-A file whose name contains "sh" (e.g., "jellyfish", "shovel"), so TDA, SCA, and BoostAdapter evaluate 6,862 of 7,500 ImageNet-A images. We report their results on that subset and compare MARC on the identical subset (Reviewer xvDB, Q4).

What this shows.

  1. The key-space effect is not specific to MARC. It appears in a feature cache (TDA) and in a statistics cache (SCA, a NeurIPS 2025 method), with no added hyperparameter in SCA's case.
  2. It adds to a state-of-the-art CLIP-space method rather than competing with it: SCA + DINOv2-B (68.65 on ImageNet-V2) is above every CLIP-only method in Table R2 on that stream.
  3. BoostAdapter marks a boundary that is consistent with our mechanism. BoostAdapter fills its memory largely with augmented crops of the current image, labeled by CLIP. DINOv2 is trained to match crops of one image, so a DINOv2 query plausibly retrieves its own crops, and the memory then repeats CLIP's prediction for that image. TDA, which uses the same memory without these crops, gains +4.86 with DINOv2 keys. The key space helps when the memory holds other test images; we state this boundary in the revised paper.

The auxiliary CLIP memory is not required and was removed. Paired online evaluations:

ConfigurationWith CLIP memory (K_C = 8)Without (K_C = 0)Difference
ImageNet-A, M = 1, K = 860.1160.29+0.18 [-0.03, +0.39]
ImageNet-V2, M = 1, K = 865.7665.77+0.00 [-0.04, +0.05]
ImageNet-A, M = 8, K = 1663.8364.12+0.29 [+0.11, +0.47]

With K_C = 0, admission priority is the negative entropy of the CLIP base prediction and remains independent of the retrieval encoder, so Prop. 1 holds unchanged. MARC now uses one memory with CLIP admission and DINOv2 keys. This removes an unnecessary component without changing the controlled intervention.

W2c / Q7. Full experiments on all ten cross-domain datasets

  • Response: We agree. The paper reported eight (six in Table A, Aircraft and Food101 in the extra table). We have now run Cars and SUN397 as well.

New analysis (Table R4): MARC (DINOv2-B) on the ten cross-domain datasets, M = 4, 3 seeds, with the same pipeline without the cache.

DatasetMARCNo cacheCache contribution
Caltech10194.7394.65+0.08
DTD53.7652.36+1.40
EuroSAT55.6050.37+5.23
Flowers10281.2772.80+8.47
Pets92.0389.89+2.14
UCF10172.3669.73+2.63
Aircraft19.2527.71-8.46
Food10178.9883.77-4.79
Stanford Cars65.9465.78+0.16
SUN39765.1667.99-2.82
Mean of ten67.9167.51+0.40

At the shared fixed weight w = 10, the cache contribution is negative on Aircraft, Food101, and SUN397. With the weight selected per stream, the contributions are +1.39, +0.50, and +2.22, respectively, and +1.53 on Stanford Cars. A label-free weight rule reduces the worst loss over 12 streams from -7.92 to -2.80 (Reviewer xvDB, Q3). The cross-domain retrieval-space rankings are reported in App. "Cross-domain rankings," with purity-gain correlations of at least 0.80 on five of six streams.

Reviewer_83v1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how caching improves test-time adaptation of CLIP models. It focuses on study the role of the feature space for retrieval. By evaluating sixteen different feature spaces while keeping other things fixed, the authors show that caching gains strongly from the retrieval feature space. They identify some metrics to measure caching effectiveness. They propose MARC, a training-free method that uses CLIP for prediction and DINOv2-B for retrieval. Experiments across ImageNet distribution shifts and cross-domain datasets show that MARC outperforms several existing cache-based adaptation methods.

Critical strengths

(1) The experiment is clean and simple. Only the choice of encoder for finding similar images differs, while predictions, labels, and stored images are identical (Prop.1). Yet, the benefit from including past test images ranges from negligible to ~20 on ImageNet-A across 16 encoders. This ranking is consistent across different random seeds, a different CLIP model, and different storing rules.

(2) The idea is simple but no one has tried this before. Finding similar images using one view of each image is better than averaging 64 views at ~1/7 cost.

Critical weaknesses
  1. The main results are tuned on the test set. In Tab. 1, the fusion weight of each encoder is selected on the test stream (Eq. 10), and purity is computed with the test labels. With a fixed weight, the cache instead decreases accuracy by 10.28% for CLIP and 43.7% for MAE (Fig. 3), showing that this choice is sensitive. Could the authors select K and the fusion weight on a separate validation set and report all 16 encoders?

  2. The comparison with COSMIC favors MARC. In Tab. 2, COSMIC is re-run with DINOv2-B and 8 views instead of DINOv2-L and 16 views, whereas MARC uses 64 views, yet COSMIC still outperforms MARC on ImageNet, ImageNet-V2, and ImageNet-Sketch. At DINOv2-L scale, COSMIC is better, but this is only reported in the appendix, and the matched comparison in the main text covers only ImageNet-A. Could the authors move this comparison to the main paper and adjust the claims accordingly?

  3. DINOv2 is known to provide better nearest neighbors than CLIP, and COSMIC already uses it, so the correlation between purity and cache gain is not surprising. Moreover, DINO v1 improves accuracy by only 0.17%, compared with 9.88% for DINOv2-B (Tab. A24), suggesting that the gain may come from the pretraining data. Since most baselines use CLIP alone, without the extra model budget, could the authors investigate this and compare with a larger CLIP predictor?

  4. The abstract states that the results "establish" retrieval space as a key design choice for scientific imaging, but no such experiments are provided. Moreover, the cache brings almost no improvement on Caltech101 and DTD, and only 0.84% on average on the two additional datasets. Could the authors tone down these claims or add experiments?

Questions for authors

See above weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 83v1 (Part: 1/5)

We thank the reviewer for recognizing that the experiment is "clean and simple": only the encoder used for finding similar images changes, while predictions, labels, and stored images are identical (Prop. 1), and the resulting cache benefit spans from negligible to about 20 points on ImageNet-A across 16 encoders, with a ranking that holds across seeds, a second CLIP predictor, and different storing rules. We also appreciate the observation that single-view retrieval with DINOv2 keys outperforms a 64-view ensemble at approximately one seventh of the cost. The four concerns below address the scope of the evidence: test-set tuning, the COSMIC [Huang2025] comparison, the source of DINOv2's advantage, and the abstract's application claim. We answer each with additional evaluations that use no evaluation-stream labels for selection. All corresponding revisions are marked in blue in the paper.

Below we address the weaknesses separately:


Weaknesses

  • W1. Test-set tuning of K and the fusion weight; please select them on a separate validation set and report all 16 encoders

    "Could the authors select K and the fusion weight on a separate validation set and report all 16 encoders?"

    • Response: We agree that the oracle envelope (Eq. 10) is an upper bound and should not be the primary reported result. We implemented the requested protocol.

      Protocol. The validation set is ImageNet-val restricted to ImageNet-A's 200 classes, 50 clean images per class (10,000 images, none from ImageNet-A). Restricting to the target label space uses no target information beyond what zero-shot CLIP already requires (the class names). For each encoder, (K, w) is chosen on this set by the same grid as Eq. 10 and then applied unchanged to ImageNet-A. No ImageNet-A label enters selection.

      New analysis (Table R1): all 16 encoders, (K, w) selected on held-out validation, applied to ImageNet-A (3 seeds). Both columns are computed on the same three replay seeds, so the oracle values differ slightly from the single-seed cells of Table 1.

      Retrieval encoderOracle on test (upper bound)Selected on validationFraction kept
      DINOv2-L+19.76+19.0696%
      DINOv2-B+9.96+8.9690%
      DeiT-III+8.33+7.2587%
      DINOv3-B+6.81+6.6598%
      CLIP ViT-L/14+5.88+4.9985%
      Supervised ViT+4.76+4.4694%
      EVA-02 CLIP+4.38+4.2998%
      ConvNeXt-B+3.24+2.6080%
      DINOv2-S+1.36+1.0476%
      SigLIP+1.18+0.5244%
      OpenCLIP+0.79+0.7392%
      CLIP ViT-B/16+0.72+0.4968%
      DINO v1+0.11-0.11n/a
      MAE+0.07-0.00n/a
      BEiT+0.02-0.08n/a
      ResNet-50 (sup.)+0.13-4.50n/a
      Spearman (validation-selected gain, oracle gain)0.97

      Online evaluation. We re-ran the two DINOv2 rows as online GPU evaluations at the validation-chosen (K, w), using 3 stream orders and image-level 95% intervals: DINOv2-L +19.06 [+18.08, +20.02], DINOv2-B +8.99 [+8.19, +9.80]. These agree with the replay values (+19.06, +8.96).

      What this shows.

      1. The ranking of encoders does not depend on test tuning: Spearman 0.97 between validation-selected and oracle gains over all 16 encoders.
      2. The held-out result retains most of the oracle gain: 96% for DINOv2-L and 90% for DINOv2-B.
      3. The fixed-weight losses quoted from Fig. 3 (-10.28 for CLIP, -43.7 for MAE) do not occur for these encoders under this validation protocol: validation assigns them w between 0 and 2, giving +0.49 and +0.00. Fig. 3 uses w = 10 for every encoder to diagnose the effect of strongly weighting low-purity spaces; it is not a deployment setting.
      4. One encoder is harmed: ResNet-50 (-4.50). Validation predicts that it helps (+3.09 on clean ImageNet), but ImageNet-A was built by retaining images that ResNet-50 classifiers misclassify [Hendrycks2021]. Clean ImageNet validation is therefore expected to transfer poorly for this encoder. We report the exception and its likely cause in the revised table.

      Sensitivity to validation size. With DINOv2-L, keeping 50, 25, and 10 images per class retains 96%, 95%, and 78% of the oracle gain. ImageNet-A has 37.5 images per class and prefers K = 8 to 16; a validation set with only 10 images per class cannot reveal this and selects K = 2. Even then, 78% of the gain remains.

Purity without test labels. Labeled purity in Table 1 is an offline diagnostic, as Sec. 3.4 states. For deployment, encoder choice uses pseudo-purity, which requires no target labels and matches the labeled ranking at Spearman 0.98 (ImageNet-A) and 1.00 (ImageNet-V2). On ImageNet-A it selects DINOv2-L, so the target-stream-label-free pipeline (pseudo-purity for the encoder and held-out source validation for (K, w)) gives the +19.06 above. Purity measured on the validation set instead of the test set still ranks the held-out gains at Spearman 0.83 over the 11 encoders trained without ImageNet labels.

Correction we disclose. While re-running the held-out ImageNet-V2 to ImageNet-A transfer at the kernel stated in the paper, (alpha, beta) = (2, 5), we found that the transfer value for DINOv2-L is +14.52 (oracle +19.36), not the value in the held-out transfer table, which was computed with stream-specific kernel constants. Table R1 replaces that transfer as the deployable headline, and the revised paper states the kernel used for every row.

Revised text (Sec. 3.4, Controlled Evaluation):

"We report two quantities for every encoder: the oracle envelope (Eq. 10), an upper bound, and the deployable gain with (K, w) selected on a disjoint, class-matched ImageNet-val split. The deployable gain retains 96% of the oracle gain for DINOv2-L (+19.06 [+18.08, +20.02]) and ranks the 16 encoders at Spearman 0.97 against the oracle."

How this resolves the concern: every headline number now has a version selected without evaluation labels, all 16 encoders are reported under it, and the conclusions (DINOv2-L leads, CLIP-space keys give under 1 point, the ranking tracks purity) are unchanged.

  • W2. Fairness of the COSMIC comparison; move the matched comparison to the main paper and adjust the claims

"Could the authors move this comparison to the main paper and adjust the claims accordingly?"

  • Response: We agree, and we extended the matched comparison beyond ImageNet-A, as requested. Both systems now use DINOv2-B at M = 8 (or DINOv2-L at M = 16), the same stream orders, and per-image paired intervals. A key protocol difference was not visible in Table 2: COSMIC's released protocol selects two fusion weights (betas) from a 14x14 grid on each test stream, whereas MARC uses one prespecified w = 10 everywhere. We therefore report COSMIC both as released and with its betas fixed from a different stream, which is the setting MARC is evaluated in.

New analysis (Table R2): MARC minus COSMIC, matched encoder and views, image-clustered 95% intervals.

Stream (encoder, M, orders)MARCCOSMIC (betas tuned on test)Difference, COSMIC tuned on testDifference, COSMIC betas fixed from another stream
ImageNet-A (DINOv2-B, 8, 10 orders)63.9462.83+1.11 [+0.67, +1.55]+1.73 [+1.25, +2.21] (V2 betas)
ImageNet-V2 (DINOv2-B, 8, 3 orders)65.7167.15-1.44 [-1.81, -1.07]-1.13 [-1.46, -0.80] (A betas)
ImageNet-R (DINOv2-B, 8, seed 0)82.6082.37+0.23 [-0.07, +0.53]+0.35 [+0.06, +0.64] (A betas)
ImageNet-Sketch (DINOv2-B, 8, 3 orders)56.3157.93-1.62 [-1.83, -1.42]+0.43 [+0.20, +0.65] (V2 betas)
ImageNet-A (DINOv2-L, 16, seed 0)71.9772.92-0.95 [-1.69, -0.21]-0.83 [-1.58, -0.07] (V2 betas)
ImageNet-V2 (DINOv2-L, 16, seed 0)66.7468.71-1.97 [-2.54, -1.40]-1.46 [-2.01, -0.91] (A betas)
Cross-domain, mean of 8 datasets (DINOv2-B, 8, 3 orders)-4.15+0.30 (A betas), +0.78 (V2 betas)

With 10 matched orders, the ImageNet-A advantage now also holds across orders: +1.11 [+0.76, +1.46], which addresses the earlier 3-order interval that included zero.

What this shows.

  1. When neither method tunes on the test stream, the two systems are within 1.8 points of each other on every ImageNet shift, and on the 8 cross-domain datasets the mean difference is +0.30 to +0.78 in MARC's favor (COSMIC ahead on DTD, Flowers102, and UCF101; MARC ahead on EuroSAT and Food101; the remaining differences are not statistically resolved).
  2. COSMIC's per-stream beta tuning is worth up to 12 points on a single dataset (Aircraft: 30.48 with tuned betas vs. about 19 with betas from ImageNet-A), so test-stream tuning is a property of the field's protocol, not only of our oracle analysis.
  3. COSMIC also retrieves with DINOv2. Its cliques, dual graphs, and hyper-class queries change accuracy by about 1 to 2 points relative to MARC, whereas replacing the key encoder changes it by about 5 to 10 points (released TDA [Karmanov2024] implementation: 60.18 with CLIP keys vs. 65.58 with DINOv2 keys on the same 6,862 ImageNet-A images; controlled sweep: +0.44 to +19.74). Under these controlled comparisons, retrieval space accounts for the larger observed difference.

Revised text (Abstract):

"At matched DINOv2-B scale and eight views, MARC is within 1.8 points of COSMIC on every ImageNet shift when neither system tunes on the test stream, leads on ImageNet-A by +1.11 [+0.67, +1.55] points over ten stream orders, and runs 2.6 times faster."

Revised text (Sec. 5.3): Table R2 replaces the single-stream comparison in the main paper, and the DINOv2-L rows (including Table A18's 70.86 vs. 72.83) move from the appendix into it. We removed every sentence that states or implies that MARC is more accurate than COSMIC in general, and describe MARC as the minimal reference system that isolates the key encoder.

How this resolves the concern: the comparison is now matched in encoder, views, and stream order on all four ImageNet shifts and eight cross-domain datasets, appears in blue in Sec. 5.3, and is stated at the level supported by the evidence. The table supports the retrieval-space conclusion without relying on a uniform head-to-head advantage.

  • W3. DINOv2's advantage is expected and may come from pretraining data; compare with a larger CLIP predictor

"DINO v1 improves accuracy by only 0.17%, compared with 9.88% for DINOv2-B (Tab. A24), suggesting that the gain may come from the pretraining data. Since most baselines use CLIP alone, without the extra model budget, could the authors investigate this and compare with a larger CLIP predictor?"

  • Response: We agree that DINOv2's k-NN quality is known; Sec. 2 cites it. Our controlled intervention quantifies (i) how much cache gain changes with the key space while all other components are fixed (+0.44 to +19.74 points), (ii) how this difference compares with changes to the surrounding cache machinery (W2), and (iii) whether a pre-adaptation statistic predicts it. We address the data and larger-predictor questions with additional evaluations.

(a) Pretraining data moves gain, and purity tracks every such move. We added 14 control encoders chosen to separate architecture, data, and training details.

New analysis (Table R3): selected control encoders, ImageNet-A, M = 1, gain over CLIP ViT-B/16 alone (50.84).

ContrastEncoderGain (oracle w)Purity@10
Same architecture and recipe, different dataDINOv3-L, web data+17.8665.8
DINOv3-L, satellite data+0.084.8
Same data, smaller changesDINOv2-B (native normalization)+10.3249.0
DINOv2-B without registers+8.1147.4
DINOv2-B at 196 px+6.8742.5
Large data, no curationSEER RegNet-32GF (1B Instagram images)+0.0511.6
iBOT-B, ImageNet-22k+0.2720.5
Older recipeDINO v1 at 256 px+0.0517.2
Spearman (purity, gain), all 14 controls0.96 (ImageNet-A), 0.94 (ImageNet-V2)

Data matters (DINOv3 web vs. satellite: +17.86 vs. +0.08 with identical architecture), but data scale alone does not produce the effect (SEER with 1B images: +0.05), and with the data held fixed, registers and input resolution move gain by 2.2 and 3.5 points. In every contrast, purity moves with gain. The four RN50 and DINO v1 B/8 controls that finished since (torchvision RN50, SwAV, MoCo v3, DINO v1 B/8) all sit at 0.0 to +0.57 gain with purity 10 to 21, consistent with the same relation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Request for Review of Rebuttal Responses Before the Deadline

Dear Reviewers,

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript. We are grateful for your constructive feedback and insightful comments, which have helped us substantially improve our work. We have carefully addressed the concerns raised in the reviews through detailed explanations, additional experiments, and corresponding revisions to the manuscript.

As we have submitted our rebuttal responses to all three reviewers but have not yet received further feedback, we wanted to kindly follow up. With only 48 hours remaining before the discussion period concludes, we are concerned about the limited time available to clarify any remaining questions or address further concerns you may have.

We would greatly appreciate it if you could spare some time to review our responses. We would be more than happy to provide further clarification or address any outstanding concerns. If our responses have adequately addressed your initial concerns, we sincerely hope you might consider updating your initial assessment accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.