- Rating
- 2
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks how much of the benefit of cache-based test-time adaptation for CLIP comes from the feature space used for image–image retrieval, as opposed to the cache mechanism itself. The setup separates the two roles. Frozen CLIP produces the base logits , the pseudo-labels and the admission priorities , and a second frozen encoder only provides query and key embeddings. Because admission never depends on , the retained index sets are identical across retrieval spaces (Prop. 1). Swapping therefore changes only the matched query–key similarities, and all comparisons can be paired per image.
With sixteen retrieval encoders on ImageNet-A/V2, the authors report three main findings:
- The oracle-weight cache gain on ImageNet-A ranges from roughly zero (CLIP ViT-B/16, MAE, BEiT) to about +20 points (DINOv2-L).
- Labeled neighborhood purity@10 ranks this gain well ( / ). Anti-hubness tracks the useful capacity better than it tracks gain.
- A pseudo-label version of purity selects the best space on 5 of 8 streams, with 1.48 points of mean regret.
The cache gain also shrinks as more augmented views are added: +9.30 at versus +6.56 at for DINOv2-B on ImageNet-A.
The resulting reference system, MARC, combines CLIP prediction, DINOv2-B retrieval, and a single fusion weight. It reaches a 67.91 OOD average on the four ImageNet shifts at . At matched on ImageNet-A, it obtains 64.17 versus 62.75 for COSMIC at about 2.6x lower latency.
Validation includes:
- three seeds for most rows, with paired CIs and McNemar tests;
- a second predictor (CLIP RN50);
- five admission policies;
- six cross-domain datasets, plus Aircraft and Food101 reported as failure cases.
Critical strengths
1. The intervention is clean, and the authors exploit it properly. Prior work that brings DINOv2 into the cache, COSMIC in particular, changes the encoder together with cache construction and aggregation. In those systems it is impossible to tell whether DINOv2 helps because of its neighborhoods or because of everything around it. Here the retained-population term in Eq. (5) is removed by construction. Prop. 1 itself is almost definitional, and I don't count that against the paper, because the value lies in what it enables: every space is compared against the same no-cache predictor on the same images, and the reported paired intervals and exact tests make sense. I also appreciate that the negative results stay in the paper, including EuroSAT, the Aircraft/Food101 losses at , and the composite score discarded in App. F. This is the main reason I take the empirical claims seriously.
2. The practical message is clear and robust. The retrieval encoder matters more than the knobs people usually tune. A single-view DINOv2-B cache beats the 64-view no-cache ensemble (60.14 vs. 58.69) at roughly 7x lower cost, and single-view DINOv2-L beats 64-view DINOv2-B. The ordering survives random and FIFO admission (the DINOv2-L vs. DINOv2-B gap stays around 9.6 points), a different predictor backbone, and several seeds. Anyone building a TDA-style system can act on this directly.
Critical weaknesses
1. The central finding is close to what one would expect, and "geometry" is not separated from prior exposure to the target label space.
The cache logit in Eq. (3) is a kernel-weighted kNN vote over pseudo-labeled keys, and purity@10 is essentially kNN label agreement measured on the same test split. A strong rank correlation between the two is therefore not surprising. The kNN advantage of DINOv2 over CLIP is also already known from the DINOv2 paper and from COSMIC. Labeled purity is computed with ground truth on the full evaluation split (Sec. 3.4), so it works as an offline diagnostic rather than a selection rule. On the method side, MARC is TDA's positive cache with keys moved to DINOv2 space and the negative cache removed. That is fine for a reference system, but it is not a methodological contribution in itself.
The confound worries me more. On ImageNet-V2, the two best spaces are DeiT-III (+5.06) and the supervised ViT (+3.48), both trained with ImageNet-1k labels, which are exactly the target classes. As far as I know, the LVD-142M curation behind DINOv2 retrieved images close to ImageNet and to several of the cross-domain benchmarks used here. Part of what the paper calls retrieval geometry may therefore be prior exposure to the target label space or data distribution. This matters for the repeated framing around remote sensing and scientific imaging, where no such encoder exists. The one remote-sensing stream in the paper, EuroSAT, is exactly where the purity ranking breaks down.
2. The deployable and benchmark claims are weaker than the headline numbers suggest.
(a) Most retrieval-space results are oracle envelopes over and on the evaluation stream. The ImageNet-V2 to ImageNet-A weight transfer works, but these two streams share a label space and a predictor regime. In the cross-domain setting, the fixed costs 8.46 points on Aircraft and 4.79 on Food101, and the eight-stream average cache contribution drops from +3.32 to +0.84. Cache-logit scale differs by encoder, so choosing without labels is a real open problem rather than a detail.
(b) The comparison with COSMIC does not support "better," only "competitive, simpler, and faster":
- In Table 2, MARC's OOD advantage (67.91 vs. 67.66) uses against COSMIC's . At , MARC is at 66.79.
- COSMIC is ahead on ImageNet, V2 and Sketch.
- At DINOv2-L scale, COSMIC leads by about two points (72.83 vs. 70.86, Table A18).
- The matched comparison covers only ImageNet-A, and its seed interval over stream orders includes zero (+1.12 [−0.62, +2.86], App. R).
The large gaps over CLIP-only baselines in Table 2 (e.g., 53.00 vs. 48.20 with RN50) mostly reflect the added encoder.
Presentation. The paper is very dense, and the same quantity appears with different values in different places. DINOv2-L's ImageNet-A gain is given as 19.36, 19.74 ± 0.40, 20.06 ± 0.34, 19.16 and 18.19, and CLIP's as 0.44, 0.56 and 0.62. Each value is explained somewhere, but the reader has to reconstruct which number is which. The Table A9 caption says two seeds while the text says three. Some appendix text reads like a rebuttal log (e.g., "DINOv2-B, previously omitted for lacking a third seed, is now measured") and should be rewritten for a standalone paper.
Questions for authors
-
Can you separate geometry from label/data exposure? For example, could you report the intervention restricted to encoders never trained on ImageNet labels or ImageNet-curated data? For the cross-domain streams, could you check whether those datasets were among DINOv2's curation sources? If label-supervised ImageNet encoders such as DeiT-III are allowed as retrieval spaces, I would also like to know whether the authors still consider "training-free" the right framing.
-
Pseudo-purity definition. Is only replaced by the CLIP prediction, or both and ? If both, pseudo-purity rewards neighborhoods that agree with CLIP's decisions, including its mistakes. An encoder that clusters CLIP's errors together would then score well while being useless as a cache. On the three streams where pseudo-purity misses the best space, what does it pick instead, and why?
-
Label-free choice of . Have you tried a label-free way to set per encoder, for instance rescaling to match the spread of on the unlabeled stream? Does such a scheme remove the Aircraft and Food101 losses? An answer here would make the deployability claims much more convincing.
-
COSMIC comparison. Are matched- results for MARC and COSMIC available on V2, R, Sketch and the cross-domain suite? Is COSMIC's 14x14 fusion grid, tuned on the evaluation stream, also behind the other COSMIC entries in Table 2? I would also find TDA with DINOv2 keys and its negative cache kept to be the most direct baseline for MARC. Have you run it?
Smaller clarifications
- Please state which value is canonical for each headline number (DINOv2-L and CLIP gains on ImageNet-A), and fix the seed count in Table A9.
7 comments
Part: 2/5
New analysis (Table R1): MARC vs. TDA on the same 6,862 ImageNet-A images, 64 views, 3 seeds each, image-clustered 95% intervals.
| Configuration | Accuracy | MARC minus configuration |
|---|---|---|
| MARC (paper defaults: K = 16, w = 10) | 64.50 | |
| TDA, CLIP base only | 59.66 | +4.84 [+3.97, +5.71] |
| TDA as published (CLIP keys, negative cache) | 60.18 | +4.31 [+3.45, +5.17] |
| TDA, DINOv2 keys, TDA's weight (w = 1) | 60.43 | +4.07 [+3.22, +4.92] |
| TDA, DINOv2 keys, w = 20 | 65.24 | -0.74 [-1.49, +0.00] |
| TDA, DINOv2 keys, w = 30 (best on the test stream) | 65.58 | -1.08 [-1.82, -0.35] |
Once TDA retrieves with DINOv2 keys and a weight suited to their scale, it reaches MARC's accuracy; CLIP keys provide no gain over the tested weight range in TDA's implementation. On ImageNet-R, TDA with DINOv2 keys at the weight chosen on ImageNet-V2 gains +2.51 [+2.27, +2.78] over TDA as published. The component ablation (Sec. 4, Final Design) is consistent with this result: adding pseudo-label refinement, a negative memory, and adaptive fusion lowers ImageNet-A accuracy from 65.25 to 64.12, 64.14, and 62.12. These components provide no measurable improvement in this ablation; the released-code intervention supports the same retrieval-space conclusion.
(d) Geometry vs. exposure, and the remote-sensing framing. We agree that exposure to the target label space or data is one way an encoder acquires high purity, and we reworded the claim accordingly: purity mediates cache gain, whatever its source. The evidence below shows that exposure alone does not predict gain while purity does.
New analysis (Table R2): contrasts that separate data from purity.
| Contrast | Encoder A (gain, purity) | Encoder B (gain, purity) | Exposure predicts | Purity predicts | Observed |
|---|---|---|---|---|---|
| Same architecture, different data (IN-A) | DINOv3-L web (+17.86, 65.8) | DINOv3-L satellite (+0.08, 4.8) | A | A | A |
| Large uncurated data vs. curated data (IN-A) | SEER 1B Instagram (+0.05, 11.6) | DINOv2-B (+10.32, 49.0) | unclear | B | B |
| Same data, registers removed (IN-A) | DINOv2-B (+10.32, 49.0) | DINOv2-B no registers (+8.11, 47.4) | tie | A | A |
| Curated vs. uncurated, stream outside curation (EuroSAT) | DINOv2-B (+2.02, 85.1) | DINO v1 (+9.31, 90.9) | A | B | B |
| No ImageNet labels or curation (IN-A) | CLIP-L/14, WIT data (+5.52, 42.8) | CLIP-B/16 (+0.44, 29.7) | tie | A | A |
Across 14 control encoders, purity ranks gain at Spearman 0.96 (ImageNet-A) and 0.94 (ImageNet-V2). On remote sensing specifically: EuroSAT is where the purity ranking among the five DINOv2-family spaces was unresolved, because their purities lie within 14.4 points and their gains overlap across seeds (App. "EuroSAT failure case"). Once DINO v1 is included, purity identifies the winner on EuroSAT, and on the new remote-sensing stream RESISC45 purity ranks 17 encoders at Spearman 0.84 (best: DINOv3-L web, +6.22). The cache gain itself on EuroSAT is well resolved: +7.32 [+6.21, +8.43] at M = 4.
Revised text (Sec. 6):
"Purity is acquired through pretraining data, recipe, and resolution, and on ImageNet-A the high-purity encoders were all trained on web-scale curated data or ImageNet labels. We therefore do not claim that retrieval geometry is independent of data exposure. We claim that purity mediates cache gain: it ranks 14 control encoders at Spearman 0.96 and predicts the EuroSAT case in which an uncurated encoder outperforms a curated one."
How this resolves the concern: the purity-gain association is specific to purity, has a documented boundary, transfers across datasets, and predicts a reversal. The requested TDA baseline attributes the larger improvement to replacing the retrieval keys, while exposure is treated as one source of retrieval purity. These changes are stated in blue in the revised analysis and discussion.
- W2. Deployable and benchmark claims are weaker than the headline numbers: (a) oracle envelopes and fixed-w losses; (b) the COSMIC comparison supports "competitive, simpler, faster," not "better"
(a) Oracle envelopes and the fixed weight.
- Response: We agree that the oracle envelope should be an upper bound only. We added a held-out protocol for every encoder: (K, w) are chosen on ImageNet-val restricted to ImageNet-A's 200 classes (50 images per class, no ImageNet-A images) and applied unchanged. This also handles the per-encoder scale of the cache logit you point out, because each encoder gets its own validated weight.
Part: 3/5
New analysis (Table R3): oracle vs. validation-selected gain on ImageNet-A, 3 replay seeds; both columns use the same seeds (selected rows; all 16 in our response to Reviewer 83v1).
| Encoder | Oracle | Selected on validation | Kept |
|---|---|---|---|
| DINOv2-L | +19.76 | +19.06 (online evaluation: [+18.08, +20.02]) | 96% |
| DINOv2-B | +9.96 | +8.96 (online evaluation: +8.99 [+8.19, +9.80]) | 90% |
| DeiT-III | +8.33 | +7.25 | 87% |
| CLIP-L/14 | +5.88 | +4.99 | 85% |
| CLIP-B/16 | +0.72 | +0.49 | 68% |
| Spearman over 16 encoders | 0.97 |
On the cross-domain streams, the losses at fixed w = 10 on Aircraft and Food101 come from the weight, not from the cache (per-stream weight: +1.39 and +0.50). Q3 below gives the label-free rule.
We also disclose a correction found while re-running the ImageNet-V2 to ImageNet-A transfer: at the kernel stated in the paper, (alpha, beta) = (2, 5), the transferred gain for DINOv2-L is +14.52, not the value in the held-out transfer table, which used stream-specific kernel constants. Table R3 replaces it as the deployable headline.
(b) The COSMIC comparison.
- Response: We agree with your reading, and the revised paper uses "competitive with COSMIC at 2.6x lower latency." We extended the matched comparison to every stream (Q4, Table R5). With 10 matched orders on ImageNet-A, the seed interval no longer includes zero: +1.11 [+0.76, +1.46] across orders (+1.11 [+0.67, +1.55] image-clustered). Elsewhere COSMIC as released is ahead on several streams, but its released protocol tunes two betas from a 14x14 grid on each test stream; with betas fixed from another stream, the two systems are within 1.8 points on every ImageNet shift, and the mean over 8 cross-domain datasets is +0.30 to +0.78 for MARC.
The gaps over CLIP-only baselines in Table 2 reflect the additional frozen encoder. The revised Table 2 caption states this explicitly, and Table R3 of our response to Reviewer s4gy shows that replacing SCA's retrieval keys with the same encoder yields +4.45 (ImageNet-A) and +3.12 (ImageNet-V2). Thus, the intervention is not specific to MARC.
Revised text (Table 2 caption, added sentence):
"Methods in the CLIP + DINOv2-B block use a second frozen encoder for retrieval; the gap to the CLIP-only block is the key-space effect that this paper isolates, and the new appendix section on plug-in experiments shows that it transfers to SCA and TDA."
How this resolves the concern: every headline now has a version selected without evaluation-stream labels, the COSMIC claim is stated as competitive, and the comparison is matched across all streams.
-
W3. Presentation: inconsistent values for the same quantity, A9 seed count, rebuttal-log wording
-
Response: We agree; see Q5 for the canonical value of each headline number and the corrections.
Questions
-
Q1. Separate geometry from label/data exposure; DINOv2 curation sources; is "training-free" the right framing?
-
Response: Encoders without ImageNet labels. Over the 11 encoders trained without ImageNet labels (dropping supervised ViT, DeiT-III, ConvNeXt, ResNet-50, and BEiT), selection on the class-matched validation set picks DINOv2-L, and validation purity ranks held-out ImageNet-A gain at Spearman 0.83 (validation gain: 0.87). Among encoders trained on neither ImageNet labels nor ImageNet-curated data, CLIP-L/14 (WIT-400M) reaches +5.52 with purity 42.8, while SEER (1B uncurated Instagram images) and DINOv3-L satellite reach +0.05 and +0.08 with purity 11.6 and 4.8. Each of these encoders sits where its purity predicts.
Curation sources. Per the DINOv2 data curation, LVD-142M retrieval used Caltech101, DTD, Flowers102, and Pets, and did not use UCF101 or EuroSAT. On the two streams outside curation, DINOv2-B leads DINO v1 by +0.70 on UCF101 and trails it by 7.29 on EuroSAT; purity predicts both signs (83.8 vs. 74.9; 85.1 vs. 90.9).
Training-free. We use "training-free" as TDA and COSMIC do: no parameter is updated at test time and no target label is used. A DeiT-III retrieval encoder trained on ImageNet-1k labels has the same status as CLIP pretrained on captions, but we agree that on ImageNet-variant streams it shares the target label space. The revised Table 1 marks the five label-supervised rows, and MARC itself uses DINOv2, which uses no labels.
Response to Reviewer xvDB (Part: 1/5)
We thank the reviewer for a close reading of the identification argument. You note that prior systems such as COSMIC [Huang2025] change the encoder together with cache construction and aggregation, so "it is impossible to tell whether DINOv2 helps because of its neighborhoods or because of everything around it," and that removing the retained-population term in Eq. (5) is what makes the paired intervals and exact tests meaningful. We also appreciate the attention to the negative results we retained (EuroSAT, Aircraft/Food101, and the discarded composite score) and to the practical importance of the retrieval encoder. We address each question with an additional controlled evaluation below. All corresponding revisions are marked in blue in the paper.
Below we address the weaknesses and questions separately:
Weaknesses
W1. The central finding is close to expected; purity is near-tautological with kNN voting; MARC is TDA with keys moved; geometry is not separated from exposure to the target label space
Response: We agree that DINOv2's kNN strength is known and that MARC is deliberately close to TDA [Karmanov2024]. The contribution is the controlled quantification of how retrieval space affects cache adaptation; we address the four parts in turn.
(a) Purity is not the cache's accuracy restated. Purity and cache gain are computed on different objects. Purity is static, offline, symmetric over all n images, and uses true labels. Cache gain is causal (only earlier images), uses CLIP pseudo-labels (with CLIP's errors), keeps only the top-K per pseudo-class by priority, and counts only where the fused logit flips a CLIP error (Eq. 9). Four results show that the link between them is an empirical finding that could have failed:
(b) What is new beyond "DINOv2 has good neighbors." The DINOv2 paper [Oquab2024] reports kNN classification accuracy; COSMIC adopts DINOv2 as one part of a larger system. Neither measures the share of cache-based adaptation attributable to retrieval space. With all else fixed, our intervention quantifies that share (+0.44 to +19.74 on ImageNet-A), tests it in other released implementations, and shows that additional views have diminishing value once retrieval is strong (+9.30 at M = 1 vs. +6.56 at M = 64). Related controlled analyses have clarified which components account for gains in few-shot classification [Chen2019], domain generalization [Gulrajani2021], and network pruning [Liu2019]; here we isolate the retrieval key space in cache-based TTA.
(c) "MARC is TDA's positive cache with keys in DINOv2 space." We agree, and this is by design: MARC is a reference system that changes one variable. Following the reviewer's request, we evaluated TDA with DINOv2 keys while retaining its negative cache, using the identical 6,862 images evaluated by TDA's released loader (Q4 explains the subset).