- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper probes the within-class clustering agreement across 21 vision encoders spanning model sizes, pre-training objective, and architecture. Clustering is performed by running 20-cluster k-means 8 times, and using the results to produce a co-association matrix between samples. The matrix is then used to produce a binary partition with average-linkage clustering.
After deriving the clusters, the authors assign samples from a held out pool to the nearest centroid, and compare the ARI of the assignments across encoders.
The authors split encoders into cross-view self-supervised, label- or language-supervised and models trained to predict hidden content. They find that cross-view and label- or language-supervised have medium to high within-family and cross family agreement, and that hidden-content based models have low agreement between themselves and across families. The main results are on UrbanCars, but the effect extends, albeit with varying intensity, across other datasets, with some exceptions.
The authors train ViT-S and ResNet-18 from scratch, and, with several caveats, find that cross-view methods have low to medium agreement, and that MAE has almost zero agreement with cross-view methods. They also investigate the impact of augmentation on Colored-MNIST, where they find that partition selection is augmentation dependent.
Finally, the authors show that partitions are transferable across encoders with similar agreement, and they marginally help in mitigating spurious correlations.
Critical strengths
- The paper analyzes agreement across 15 encoders spanning architecture, training data, and objective.
- Cross-encoder agreements measured via cluster assignment are more inspectable than e.g., CKA.
Critical weaknesses
- I am not convinced of the soundness of the self-consistency measurement, as the discovery and held-out sets partially overlap, by up to 60% on average on UrbanCars.
- The paper is hard to read, dense , and often confusing, e.g., in "This threshold was fixed before EuroSAT’s self-consistency was computed, although its contrast was already known". I also can't clearly tell from the main text what are the 8 encoders evaluated for the main experiments.
- The sentence "[...] analyses of the development draws are exploratory with respect to that taxonomy." makes me doubt the soundness of the experiments.
- In light of the results of section 4.5, I am not convinced that the cross-encoder agreements are mainly due to the choice of objective, but could be rather aligned with the data augmentation policy and invariances it induces. For example, MoCo and DINOv1 have very similar aggressive augmentation policies, while MAE only uses minimal augmentation.
- Some of the results seem related to the findings of [1], which, for example, found that cross-view and supervised models cluster objects by shape, while MAE clusters by color. Discussing this would strengthen the paper.
- In the discussion, the authors mention that iBOT has high agreement with DINOv1, but DINOv1 has poor correlation with DINOv2, whose self-consistency is also low. This might suggest a training data or recipe effect.
[1] No Representation Rules Them All in Category Discovery, NeurIPS 2023
Questions for authors
- Why only the top-8 principal components of encoder features were used for clustering?
- It would be interesting to see if DINOv3 behaves similarly as DINOv2, and the degree of agreement between them.
3 comments
Response to Reviewer xDZM (2/2)
W6: iBOT, DINO and DINOv2.
We agree that data and recipe differences remain possible explanations. iBOT includes a masked-token term but agrees with DINO at 0.70 on UrbanCars; its agreement is 0.46 with whole-image models and 0.06 with hidden-content models. A masked-token term alone therefore does not explain DINOv2's result. Section 5.1 states that its data, scale and regularization remain confounded. DINOv2 is grouped by its objective rather than selected by the observed partition, and is the hidden-content model closest to the core whole-image encoders (≤0.25).
W2: Readability, EuroSAT and the original roster.
The eight pre-registered encoders, which the revision will name in §3.3, are CLIP-L/14, CLIP-B/32, CLIP-L/14 (LAION), SigLIP-B/16, SigLIP-L/16, supervised ViT-L/16, DINOv2-L and MAE-L. Their original grouping is in Table 4 and the full roster is in Table 3. These differ from the eight self-supervised cross-view encoders in the abstract (MSN, SwAV, SeLa-v2, MoCo-v3, DINO, VICReg, Barlow Twins, SimSiam), which belong to the expanded fifteen-encoder core roster.
The EuroSAT sentence concerns when the stability criterion was fixed. The null contrast (−0.005) was known on Sep 1. The regime threshold was fixed on Sep 16 using the other four datasets. EuroSAT's self-consistency was measured later and passed at 0.48. The revised A.13 will state this sequence explicitly; we do not present the threshold as a predictor of EuroSAT's contrast.
W3: Exploratory grouping and later checks.
Our wording was imprecise, and we should have stated its scope. “Exploratory” applies only to the expanded fifteen-encoder grouping, which was developed on draws 180 to 219. It does not apply to the original eight-encoder contrast, which was pre-registered, or to the later checks. We then fixed five numerical predictions before opening draws 700 to 739. All five passed, including contrasts of +0.428 [0.395, 0.458] on UrbanCars and +0.152 [0.134, 0.172] on CelebA (A.14). The revised §4.3 will state this scope directly.
Q1: Why eight principal components?
Dimension 8 was inherited from an earlier group-discovery pipeline whose constants were chosen on CelebA using group labels. It was not tuned on an encoder comparison. Table 12 reports the dimension sweep, where 8 does not give the largest contrast. The added matched-variance control also stays positive at 30% and 50% retained variance; our reply to Reviewer TZv3 gives the values.
Thank you for your response. My concerns and suggestions about related work, pool overlap and the number of principal components have been addressed.
I find the comparison between DINOv2 and DINOv3 interesting, even if the difference between pooled patch tokens and the [CLS] token is not examined in detail. On top of this, the same effect may apply to other ViT-based encoders. Hence, some of the effects measured in the paper may be specific to the [CLS] token rather than to the encoder as a whole, and a patch-level analysis might reveal different associations. The paper would benefit from further experiments in this direction.
As for augmentation, I am not sure that adding augmentations to MAE is a fair test, as a reconstruction objective does not enforce view invariance as, e.g., BYOL does.
Furthermore, one of my core concerns remains presentation. The rebuttal uses the same dense and cryptic vocabulary as the paper and requires considerable effort to follow.
Overall, I plan to keep my score.
Response to Reviewer xDZM (1/2)
W5: Relation to Vaze et al.
We should have discussed Vaze, Vedaldi and Zisserman (NeurIPS 2023); we will add it to Related Work. Their Clevr-4 study shows that frozen representations can favor different valid groupings: the tested supervised and cross-view models favor shape while MAE favors color.
Our analysis asks whether independently trained encoders share a partition on natural images. It reports stability alongside agreement and examines the intermediate supervised tier and the isolation of data2vec and I-JEPA. Matched training, input edits and transfer provide further evidence about the partition. The tier is dataset-dependent, and the joint isolation of the latent-prediction models is established only on UrbanCars. These results extend the factor-dependence finding rather than claim the MAE contrast as a first observation.
Q2: DINOv3 and pooling.
We evaluated DINOv3 ViT-L/16 with the frozen pipeline and the roster's CLS convention, as an extension check, not a roster change. The table uses draws 180–219 and five seeds.
Our pre-registered prediction of DINOv3–DINOv2 agreement ≥0.30 failed on UrbanCars. The information is nevertheless present: a linear probe on the same DINOv3 CLS features recovers the within-class background label at 0.95 accuracy (DINOv2 CLS: 0.96). What is absent is a stable first split on it: the label-free partition is unstable (SC 0.18) and does not align with background (balanced accuracy 0.51), nor does DINOv2's (0.60; SC 0.24). Elsewhere the nearest model differs by dataset (post-hoc), so we claim no settled placement.
Pooling changes this interpretation. On UrbanCars the patch means of DINOv2 and DINOv3 are stable (SC 0.74 and 0.50), agree with each other at 0.52 and with the cross-view encoders (0.63 and 0.48), unlike their CLS partitions. At matched representation DINOv3 therefore behaves like DINOv2; we have not tested what causes the CLS/patch difference. Our statements about DINOv2/DINOv3 concern the CLS representation.
W1: Overlap and self-consistency.
Self-consistency compares partitions fitted on different discovery samples and applied to the same reference pool. We checked both discovery–reference overlap and overlap between discovery samples.
The last check uses 20 attempted pairs; MAE has 15 valid disjoint pairs and 12 controls. Each fit sees its own 250 evaluation images, so this is not a fully held-out test. It nevertheless shows that shared discovery images are not required for the observed whole-image stability.
W4: Augmentation and objective.
The pretrained roster does not separate these effects. Section 4.4 and contribution 2 describe the matched comparison as objective-plus-recipe. We added two ResNet-18/COCO arms with changed augmentation, each trained for 200 epochs with three seeds; the original reference arms have one seed each. Crop ranges still follow the arm recipes.
Added augmentation does not make conv-MAE share the cross-view partition: two seeds are stable (SC 0.44/0.45), the third is below threshold (0.24), and all nine comparisons are ≤0.07. Reduced augmentation lowers DINO's stability below the regime threshold in all three seeds, so those partitions support no placement. These results argue against augmentation alone explaining the pattern in this setting, but they do not isolate the loss from its recipe. The testbed provides related controls: removing brightness augmentation moves SimCLR/BYOL onto the illumination factor, whereas adding full augmentation to MAE-lite does not produce the cross-view partition (§4.5, P4).