acceptodds
Under review as a conference paper at ICLR 2027

OISE: Objective-Induced Splits in Encoders

Abstract

What a frozen vision encoder splits images on first depends on whether its training objective matches a whole-image target or predicts hidden content. We examine this relationship in 21 frozen vision encoders, comprising 15 core models spanning objectives and architectures and six language-supervised variants spanning scale, pretraining data, a convolutional backbone, and a VLM tower. A fixed clustering procedure induces a binary partition within each class without group labels. We compare assignments on the same images using the Adjusted Rand Index (ARI), whose chance expectation is zero. On UrbanCars, 19 of the 21 encoders have self-consistency of 0.56–0.93 across discovery draws. Eight self-supervised cross-view encoders produce similar partitions (ARI 0.56–0.81). Label- and language-supervised models occupy an intermediate position, agreeing with the cross-view models at 0.37 on average. MAE, data2vec, I-JEPA, and the hybrid DINOv2 have low agreement with whole-image models (0.07) and with one another (0.06). The exception is MAE–data2vec (0.32 on UrbanCars and at most 0.08 elsewhere). Within the original four-dataset analysis, the three latent-prediction models (data2vec, I-JEPA, and DINOv2) are jointly isolated only on UrbanCars, where partitions are most stable. Matched training reproduces the ResNet-18 ordering with the same images, epochs, and batch size. Controlled interventions link partition selection to cross-view augmentation and show that relative variance can shift the partition among stably expressed factors. Finally, label-free partition agreement predicts cross-encoder group-transfer gains, although the gains are limited to a few worst-group accuracy points.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
frozen vision encoders, representation comparison, self-supervised learning, masked image modeling, training objectives, label-free group discovery, partition agreement, Adjusted Rand Index

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_xDZMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper probes the within-class clustering agreement across 21 vision encoders spanning model sizes, pre-training objective, and architecture. Clustering is performed by running 20-cluster k-means 8 times, and using the results to produce a co-association matrix CC between samples. The 1−C1 - C matrix is then used to produce a binary partition with average-linkage clustering.

After deriving the clusters, the authors assign samples from a held out pool to the nearest centroid, and compare the ARI of the assignments across encoders.

The authors split encoders into cross-view self-supervised, label- or language-supervised and models trained to predict hidden content. They find that cross-view and label- or language-supervised have medium to high within-family and cross family agreement, and that hidden-content based models have low agreement between themselves and across families. The main results are on UrbanCars, but the effect extends, albeit with varying intensity, across other datasets, with some exceptions.

The authors train ViT-S and ResNet-18 from scratch, and, with several caveats, find that cross-view methods have low to medium agreement, and that MAE has almost zero agreement with cross-view methods. They also investigate the impact of augmentation on Colored-MNIST, where they find that partition selection is augmentation dependent.

Finally, the authors show that partitions are transferable across encoders with similar agreement, and they marginally help in mitigating spurious correlations.

Critical strengths
  1. The paper analyzes agreement across 15 encoders spanning architecture, training data, and objective.
  2. Cross-encoder agreements measured via cluster assignment are more inspectable than e.g., CKA.
Critical weaknesses
  1. I am not convinced of the soundness of the self-consistency measurement, as the discovery and held-out sets partially overlap, by up to 60% on average on UrbanCars.
  2. The paper is hard to read, dense , and often confusing, e.g., in "This threshold was fixed before EuroSAT’s self-consistency was computed, although its contrast was already known". I also can't clearly tell from the main text what are the 8 encoders evaluated for the main experiments.
  3. The sentence "[...] analyses of the development draws are exploratory with respect to that taxonomy." makes me doubt the soundness of the experiments.
  4. In light of the results of section 4.5, I am not convinced that the cross-encoder agreements are mainly due to the choice of objective, but could be rather aligned with the data augmentation policy and invariances it induces. For example, MoCo and DINOv1 have very similar aggressive augmentation policies, while MAE only uses minimal augmentation.
  5. Some of the results seem related to the findings of [1], which, for example, found that cross-view and supervised models cluster objects by shape, while MAE clusters by color. Discussing this would strengthen the paper.
  6. In the discussion, the authors mention that iBOT has high agreement with DINOv1, but DINOv1 has poor correlation with DINOv2, whose self-consistency is also low. This might suggest a training data or recipe effect.

[1] No Representation Rules Them All in Category Discovery, NeurIPS 2023

Questions for authors
  1. Why only the top-8 principal components of encoder features were used for clustering?
  2. It would be interesting to see if DINOv3 behaves similarly as DINOv2, and the degree of agreement between them.
3 comments
AuthorsOpenReview ↗

Response to Reviewer xDZM (1/2)

W5: Relation to Vaze et al.

We should have discussed Vaze, Vedaldi and Zisserman (NeurIPS 2023); we will add it to Related Work. Their Clevr-4 study shows that frozen representations can favor different valid groupings: the tested supervised and cross-view models favor shape while MAE favors color.

Our analysis asks whether independently trained encoders share a partition on natural images. It reports stability alongside agreement and examines the intermediate supervised tier and the isolation of data2vec and I-JEPA. Matched training, input edits and transfer provide further evidence about the partition. The tier is dataset-dependent, and the joint isolation of the latent-prediction models is established only on UrbanCars. These results extend the factor-dependence finding rather than claim the MAE contrast as a first observation.

Q2: DINOv3 and pooling.

We evaluated DINOv3 ViT-L/16 with the frozen pipeline and the roster's CLS convention, as an extension check, not a roster change. The table uses draws 180–219 and five seeds.

DatasetSCvs DINOv2vs whole-image (n)vs ext.-supervised 3vs MAENearest
UrbanCars0.180.000.00 (11)0.00−0.02None reliably
CelebA0.470.500.18 (11)0.100.01DINOv2
ISIC0.500.140.19 (7)0.290.01I-JEPA 0.54
MetaShift0.210.320.09 (8)0.10−0.02DINOv2

Our pre-registered prediction of DINOv3–DINOv2 agreement ≥0.30 failed on UrbanCars. The information is nevertheless present: a linear probe on the same DINOv3 CLS features recovers the within-class background label at 0.95 accuracy (DINOv2 CLS: 0.96). What is absent is a stable first split on it: the label-free partition is unstable (SC 0.18) and does not align with background (balanced accuracy 0.51), nor does DINOv2's (0.60; SC 0.24). Elsewhere the nearest model differs by dataset (post-hoc), so we claim no settled placement.

Pooling changes this interpretation. On UrbanCars the patch means of DINOv2 and DINOv3 are stable (SC 0.74 and 0.50), agree with each other at 0.52 and with the cross-view encoders (0.63 and 0.48), unlike their CLS partitions. At matched representation DINOv3 therefore behaves like DINOv2; we have not tested what causes the CLS/patch difference. Our statements about DINOv2/DINOv3 concern the CLS representation.

W1: Overlap and self-consistency.

Self-consistency compares partitions fitted on different discovery samples and applied to the same reference pool. We checked both discovery–reference overlap and overlap between discovery samples.

CheckResult
Remove shared discovery–reference images on UrbanCarsAgreement summaries change by at most 0.02; the revision will state the values in §3.2.
Waterbirds/SpuCo with 7–30% overlapSame ordering (A.11).
Fit on disjoint discovery samples of 250 images, then apply both partitions to all 500 class-1 imagesWhole-image SC is 0.34–0.85; group mean 0.60 versus 0.63 for overlapping controls. Individual decreases reach 0.17. MAE is 0.24 and DINOv2 0.12.

The last check uses 20 attempted pairs; MAE has 15 valid disjoint pairs and 12 controls. Each fit sees its own 250 evaluation images, so this is not a fully held-out test. It nevertheless shows that shared discovery images are not required for the observed whole-image stability.

W4: Augmentation and objective.

The pretrained roster does not separate these effects. Section 4.4 and contribution 2 describe the matched comparison as objective-plus-recipe. We added two ResNet-18/COCO arms with changed augmentation, each trained for 200 epochs with three seeds; the original reference arms have one seed each. Crop ranges still follow the arm recipes.

Arm (seeds 0 / 1 / 2)SCvs SwAVvs BYOLvs DINOvs conv-MAE (original)
Conv-MAE + full cross-view augmentation0.44 / 0.45 / 0.240.02 / 0.04 / 0.060.02 / 0.06 / 0.050.03 / 0.03 / 0.070.28 / 0.17 / 0.24
DINO with crop + flip only0.26 / 0.12 / 0.170.07 / 0.04 / 0.060.06 / 0.06 / 0.04—0.07 / 0.06 / 0.07
DINO, full recipe (paper)0.470.350.23—0.11

Added augmentation does not make conv-MAE share the cross-view partition: two seeds are stable (SC 0.44/0.45), the third is below threshold (0.24), and all nine comparisons are ≤0.07. Reduced augmentation lowers DINO's stability below the regime threshold in all three seeds, so those partitions support no placement. These results argue against augmentation alone explaining the pattern in this setting, but they do not isolate the loss from its recipe. The testbed provides related controls: removing brightness augmentation moves SimCLR/BYOL onto the illumination factor, whereas adding full augmentation to MAE-lite does not produce the cross-view partition (§4.5, P4).

AuthorsOpenReview ↗

Response to Reviewer xDZM (2/2)

W6: iBOT, DINO and DINOv2.

We agree that data and recipe differences remain possible explanations. iBOT includes a masked-token term but agrees with DINO at 0.70 on UrbanCars; its agreement is 0.46 with whole-image models and 0.06 with hidden-content models. A masked-token term alone therefore does not explain DINOv2's result. Section 5.1 states that its data, scale and regularization remain confounded. DINOv2 is grouped by its objective rather than selected by the observed partition, and is the hidden-content model closest to the core whole-image encoders (≤0.25).

W2: Readability, EuroSAT and the original roster.

The eight pre-registered encoders, which the revision will name in §3.3, are CLIP-L/14, CLIP-B/32, CLIP-L/14 (LAION), SigLIP-B/16, SigLIP-L/16, supervised ViT-L/16, DINOv2-L and MAE-L. Their original grouping is in Table 4 and the full roster is in Table 3. These differ from the eight self-supervised cross-view encoders in the abstract (MSN, SwAV, SeLa-v2, MoCo-v3, DINO, VICReg, Barlow Twins, SimSiam), which belong to the expanded fifteen-encoder core roster.

The EuroSAT sentence concerns when the stability criterion was fixed. The null contrast (−0.005) was known on Sep 1. The regime threshold was fixed on Sep 16 using the other four datasets. EuroSAT's self-consistency was measured later and passed at 0.48. The revised A.13 will state this sequence explicitly; we do not present the threshold as a predictor of EuroSAT's contrast.

W3: Exploratory grouping and later checks.

Our wording was imprecise, and we should have stated its scope. “Exploratory” applies only to the expanded fifteen-encoder grouping, which was developed on draws 180 to 219. It does not apply to the original eight-encoder contrast, which was pre-registered, or to the later checks. We then fixed five numerical predictions before opening draws 700 to 739. All five passed, including contrasts of +0.428 [0.395, 0.458] on UrbanCars and +0.152 [0.134, 0.172] on CelebA (A.14). The revised §4.3 will state this scope directly.

Q1: Why eight principal components?

Dimension 8 was inherited from an earlier group-discovery pipeline whose constants were chosen on CelebA using group labels. It was not tuned on an encoder comparison. Table 12 reports the dimension sweep, where 8 does not give the largest contrast. The added matched-variance control also stays positive at 30% and 50% retained variance; our reply to Reviewer TZv3 gives the values.

Thank you for your response. My concerns and suggestions about related work, pool overlap and the number of principal components have been addressed.

I find the comparison between DINOv2 and DINOv3 interesting, even if the difference between pooled patch tokens and the [CLS] token is not examined in detail. On top of this, the same effect may apply to other ViT-based encoders. Hence, some of the effects measured in the paper may be specific to the [CLS] token rather than to the encoder as a whole, and a patch-level analysis might reveal different associations. The paper would benefit from further experiments in this direction.

As for augmentation, I am not sure that adding augmentations to MAE is a fair test, as a reconstruction objective does not enforce view invariance as, e.g., BYOL does.

Furthermore, one of my core concerns remains presentation. The rebuttal uses the same dense and cryptic vocabulary as the paper and requires considerable effort to follow.

Overall, I plan to keep my score.

Reviewer_TZv3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper aims to investigate how the kind of objective a vision encoder is trained on shapes the learned representation. The authors propose a binary partitioning approach, in which embeddings per class are first reduced by PCA, then repeatedly clustered, and then subdivided into two partitions based on another clustering in the average linkage matrix of the initial clusters. The ARI of the partition assignments is then used to quantify the agreement between encoders trained using different pretraining objectives.

Critical strengths

The paper addresses an overall interesting and relevant question: how the choice of a pretraining objective shapes the structure of the learned representation. Understanding such effects would be valuable for selecting and interpreting vision encoders. The idea of dominant partitions per class as a discrete, interpretable description of latent space organization and quantifying encoder agreement via ARI is in principle reasonable, and might offer more interpretable insights into what distinguishes encoders rather than only how much they differ.

Critical weaknesses
  • The core methodology is not sufficiently justified. The central assumption that the dominant organizational axes of an encoder’s latent space can be captured by a binary partitioning of a clustered, low-dimensional PCA projection is neither clearly theoretically motivated nor empirically validated. The approach is not compared against strong representation-similarity measures, nor against directly comparing principal components, leaving the added benefit of multi-step clustering unclear.
  • Critical hyperparameters are selected without adequate justification, undermining comparability across encoders. A fixed PCA dimensionality (d=8) and a fixed number of clusters are applied to all encoders without reporting explained variance or any clustering quality metrics. Parameters derived in a prior study on a different dataset are used without convincing evidence of their suitability of the transfer. Since explained variance at fixed d will differ between encoders and datasets, the drawn comparisons may not reflect genuine differences in representation structure.
  • the reliability of the reported results is unclear. The paper reports that the analysis was expanded iteratively with changing parameters, and Appendix A6 documents a bug in the analysis code. It is not clear which of the presented results were produced with the corrected code and a consistent protocol. A rerun under a predefined, uniform protocol appears necessary.
Questions for authors

I have no questions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer TZv3

W3: Implementation corrections and reliability.

The correction in Appendix A.6 affects only the depth-2 analysis. Root-level partitions do not use that code path. The revision will state this distinction in §5.3.

CheckResult
Regenerate tables and macros from stored outputs, at submission and on Nov 3Byte-identical, except linear-CKA partial correlations, which need extracted features.
Recompute UrbanCars partitions from stored features on Nov 5: 15 core encoders, draws 180 to 219ARI 1.000 against the cache.

Roster changes were separate from these corrections. The original eight-encoder contrast was pre-registered. The expanded grouping came later and passed all five predictions fixed before opening draws 700 to 739. We distinguish these stages in §4.3 and A.14.

W2: PCA dimension, cluster count and explained variance.

The submitted sensitivity analysis uses fresh draws 900 to 919 (Table 12). All intervals exclude zero, and dimension 8 does not give the largest contrast.

SettingContrast
PCA dimension 4 / 8 / 16 / 320.28 / 0.48 / 0.56 / 0.53
Cluster count 10 / 20 / 400.52 / 0.48 / 0.41
NMI instead of ARI0.40

The top eight components retain different amounts of variance: 23–34% for the UrbanCars whole-image encoders and 48–58% for MAE, data2vec and I-JEPA. DINOv2 retains 32%. We therefore repeated the comparison with encoder-specific dimensions.

PCA choiceContrast on draws 1040 to 1059
Fixed dimension 8+0.43 [0.37, 0.48]
Retain 30% variance; dimensions 2 to 14+0.44 [0.39, 0.49]
Retain 50% variance; dimensions 6 to 39+0.35 [0.30, 0.41]

On all four datasets, explained variance at dimension 8 is higher on average for hidden-content encoders; the revision will report the values. The silhouette of the root split in PCA-8 space is 0.15–0.30 and does not separate the families. The constants were inherited from an earlier pipeline tuned on CelebA, not on an encoder comparison.

W1: What the partition measures and how it is checked.

The root partition is the first binary split produced by the stated procedure. We do not assume that it identifies a unique dominant factor.

CheckEvidence
Alignment with annotated factorsBackground balanced accuracy is 0.73–0.91 on UrbanCars and 0.85–0.98 on Waterbirds/SpuCo. MAE aligns more with brightness than with background (0.60–0.84 versus 0.51–0.59).
Fixed-partition input edits (new)Background swaps change 40–86% of assignments for all eleven whole-image encoders, versus 13–25% for same-category swaps; brightness edits change at most 3.1%. Darkening changes 70% for MAE.
Graded background removal, partition refitted (new; predictions fixed first)Background balanced accuracy falls from 0.85 to 0.51 for whole-image encoders. At full removal the partition moves to the co-occurring object for four of them (0.86–0.92; we predicted six), and a second split after projecting out the first axis was mostly unstable. In 41 of 42 encoder × removal-level cells with SC ≥0.3 and some annotated factor at balanced accuracy ≥0.70, the partition and PC1 followed the same factor.
Relation to model behaviorAgreement predicts group-transfer ordering (ρ 0.71/0.82). For whole-image encoders, removing the encoder's own partition axis improves worst-group accuracy more than removing a random direction or PC1 (A.8).

A direct PC1 threshold retains the agreement pattern: within whole-image 0.49, whole-image versus hidden-content 0.10, within hidden-content 0.01 (root partition 0.52 / 0.07 / 0.05). The two partitions differ for SigLIP-L, DINOv2, MAE and data2vec (ARI 0.03–0.22) and agree elsewhere (0.40–0.83). The contrast therefore does not depend on the multi-step procedure; we claim no advantage for it at the root level. Its tree form lets the second split differ between root branches (A.6, CLIP-L/14: age in one CelebA branch, gender in the other), which a single global component cannot express; we have not compared the two directly.

We also compared the statistics directly, on each dataset's full roster, not the 15 core encoders used for PC1. The AUC for ranking within-whole-image pairs above whole-image versus hidden-content pairs (draws 180 to 219) is 0.90 for partition ARI and 0.87 for linear CKA on UrbanCars, and 0.56 / 0.54 / 0.63 versus 0.75 / 0.79 / 0.94 on CelebA, ISIC and MetaShift. CKA also separates these families and has the higher AUC on three of the four datasets; we do not claim partition agreement is a better similarity measure. The partition instead gives a per-image assignment that can be named, edited and transferred. Its association with transfer remains after controlling for CKA (partial ρ +0.57/+0.45; Table 14).

Is the concern a missing theoretical account of what the root partition captures, or insufficient empirical checks? If the latter, what further check would be decisive?

I thank the authors for the detailed clarifications and the additional analyses.

Unfortunately, these do not address what I consider the most critical gap: the approach itself is still not satisfactorily motivated. The question is posed whether the concern is a missing theoretical account or insufficient empirical checks. Either kind of argument could be convincing, but what is missing in both cases is a justification of this particular procedure relative to other possible approaches. The PCA-based multi-step pipeline is only one of many ways to analyze the embeddings, and it needs to be compared against the alternatives.

The new results reinforce this concern rather than resolve it. A direct PC1 threshold reproduces the agreement pattern, and no advantage is claimed for the multi-step procedure at the root level. Linear CKA separates the encoder families better on three of four datasets. The remaining argument that the tree form allows branch-specific second splits has not been explicitly compared, and the second split is reported as mostly unstable. The benefit of a per-image assignment that can be named and edited applies equally to a simple PC1 threshold.

The specific parameters also remain unmotivated. The constants are inherited from an earlier pipeline and the contrast varies considerably with the PCA dimension and cluster count.

My concern regarding soundness has therefore not really been addressed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We have uploaded a revised PDF. Changes are marked in blue. The main text remains within 10 pages, and the results, roster and figures are unchanged. The revisions implement the changes promised in our replies and correct one misstated bound.

LocationRevision
Related WorkDiscusses Vaze et al. (NeurIPS 2023) and how our analysis relates to it.
End of §3.1 (just before §3.2)Reports the agreement values after removing images shared between discovery and reference pools (within cross-view and the eight-encoder contrast). Our reply to Reviewer xDZM referred to this as §3.2.
§3.3Names the eight encoders of the original preregistered comparison.
§4 (opening paragraph) and A.13States when the stability threshold was fixed relative to the EuroSAT analysis: the contrast was already known, the threshold was fixed on the four original datasets, and EuroSAT's self-consistency (0.48) was computed afterwards.
§4.2Corrects the bound "at most 0.09" to "at most 0.15 (0.09 with DINOv2 and MAE)".
§4.3States the scope of the exploratory grouping: the expanded fifteen-encoder grouping was revised on draws 180–219, so its results on those draws do not test it, whereas the original eight-encoder contrast was preregistered and opened once per dataset (Table 2).
§5.3States that the implementation correction in A.6 affects only the depth-2 analysis and that root partitions do not use that code path.
A.5 and new Table 13Reports the explained variance at PCA dimension 8 for each core encoder and the mean per family on all four datasets.
A.8Defines the linear CKA used as a control.

Table numbers after Table 12 increase by one because of the new Table 13. The CKA partial correlations cited as Table 14 in our replies are Table 15 in the revised PDF.

The additional analyses reported in our replies (fixed-partition input edits, graded background removal, matched explained variance, disjoint discovery samples, the rerun from stored features, the matched ResNet-18 runs with changed augmentation, and DINOv3) are not yet in the PDF and will be added to the appendix in a later version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.