acceptodds
Under review as a conference paper at ICLR 2027

What matters for perceiving object-parts in vision encoders?

Abstract

Recent works provide evidence that large-scale pretraining makes vision and language models organize data similarly, however this similarity is exclusively studied on scene- and object-level. This leaves the unanswered question: are object-parts organized in vision models similar to language models? Here, we focus on parts, as user-defined components of 3D objects. Our work tries to answer this question through a series of analyses: **(i) Part-similarity.** We measure part-level representation similarity between frozen vision and language models using a modality-agnostic protocol. To this end, we conduct the first large-scale comparison of 39 vision encoders across 3 different vision modalities. **(ii) Part-alignment.** We further propose to test the ease of transforming vision models to language in the absence of parallel part annotations. To this end, we propose a simple probe based on differentiable optimal transport, particularly suited for data-scarce settings like 3D where large-scale paired part annotations are not available. **(iii) Part-transferability.** Finally, we find that our probe can transfer to unseen part hierarchies when trained on bags of discovered part masks and cheaply obtained VLM-provided part names. Our analyses demonstrate how visual pretraining strategy, architectural inductive biases, and modality influence both similarity and alignment with language on a part level with implications towards real-world applications.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Alignment, Platonic Hypothesis, Fine-grained Vision, Vision Language Alignment, Self-Supervised, Unsupervised Part Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_56cmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the questions of how much part-level information has been encoded in the embedding space of the vision models. The paper studies three sub-questions: part-similarity, part-alignment, and part-transferability, to cover the similarity of part-level visual features to language model features in both the labeled and unlabeled scenarios, along with the capability of generalizing to unseen part hierarchies. The study broadly covers many models spanning 2D, 3D, and video models to find out the rankings of each vision model on part-level information in their embeddings, and some general observations on the relationship (e.g., weak correlation) between properties like part-similarity and part-alignment.

Critical strengths
  • The problem of part-level features from vision models is largely underexplored in the literature, and the paper conducts a very comprehensive study across various models including 2D, 3D, and video models to demonstrate their capability of encoding part-level information in their embeddings. It can be useful for future work to choose which models to use for downstream vision and robotics tasks that require part-level understanding.

  • The results in the paper are decently presented with figures which clearly show the rankings of each model in different settings. Also, the findings of the each part is clearly presented (with the important takeaways highlighted), which makes audience easy to digest the takeaway from the paper.

Critical weaknesses
  • It seems not very natural to choose the isomorphic similarity metrics presented in Section 3.1 (Centered Kernel Alignment (CKA), Singular Vector Canonical Correlation Analysis (SVCCA), or local metrics such as Mutual k-Nearest Neighbors (Mutual-kNN)) as metrics to capture the similarity of embeddings. In many related work (e.g., in customized image generation), the DINO/CLIP similarity used in DreamBooth [1] is used to measure the similarity of two DINO features or two CLIP features from an image and text, directly by cosine similarity. The authors state that recent works also employ the metrics (CKA, SVCCA, Mutual-kNN) shown above, but the authors did not give specific references in this section about which works adopt these metrics. I think they are rarely adopted metrics so this is why I raise this question, because the whole paper is based on these metrics instead of simple (and much more broadly adopted cosine similarity), which may undermine the soundness of the paper.

  • For part-similarity, I feel the experimental design over-complicates the question. Basically, part-similarity focuses on studying whether part-level similar features have strong similarity. The paper adopts an unnecessary intermediate text modalities of labels as proxy to let features from different parts compare with the text embedding to calculate the similarity of the part-level patches. However, I think it can be done by directly calculating the similarity of the visual embeddings between the part-level patches. If the focus of the paper is really to study how the object-part features are similar to the features from language models, then the title of the paper should be changed to something like "object-parts feature alignment with language model features". The current title indicates that the study can be largely irrelevant with language models, which is very misleading and difficult to understand what is the focus of the paper.

[1] Ruiz et al. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. CVPR 2023.

Questions for authors
  • What is the motivation of choosing metrics like CKA, SVCCA, Mutual-kNN which are relatively uncommon, compared to the more broadly used (and more concise) cosine similarity? Basically, what are the aspects that these metrics can better measure in the evaluated tasks?

  • What is the design principle of the part-similarity experiment? Basically, why language models always need to be as an intermediate proxy to measure similarity for part-level visual embeddings? Also, is the current title well reflect which is the focus of the paper, as it does not mention language models, but all the experiments are actually studying the feature similarity between part-level visual embeddings and language embeddings?

Reviewer_1DVdOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether vision encoders organize representations similarly to language at the object-part level. The authors establish a modality-agnostic protocol on PartNet to compare the part-similarity of 39 image, point-cloud, and video encoders, and propose an optimal transport probe that does not require part-level correspondences to evaluate part-alignment. The main results suggest that high part-similarity can emerge without explicit vision-language pretraining and that part-similarity does not reliably predict part-alignment. The authors further evaluate whether the probe transfers to PartNet part hierarchies.

Critical strengths
  1. The problem is important and practically relevant. As discussed in the Introduction, existing work on vision-language representation similarity has primarily focused on complete scenes or objects. Fine-grained object-parts are important for applications such as robotic manipulation, 3D understanding, and content editing.

  2. The comparison between Figures 2 and 4 is convincing. Figure 2 measures part-similarity between frozen representations and language, whereas Figure 4 measures part-alignment after training a probe. Section 6.2 reports an overall Spearman correlation of 0.42 between the two rankings and a correlation of only 0.05 for 2D encoders. These results provide direct evidence that high part-similarity does not necessarily imply high part-alignment.

  3. The core idea of the paper is clear and easy to understand. The paper distinguishes between whether visual representations already have a structure similar to language and whether they can be readily transformed into the language space. This is a substantive and memorable insight rather than a complicated recipe that merely combines multiple models or metrics.

Critical weaknesses
  1. The part-similarity analysis does not sufficiently rule out the influence of object category and part-name co-occurrence. In Sections 3 and 4.2, visual part representations are obtained by pooling features within the ground-truth part masks from PartNet, while the language representations are produced from the corresponding part names. Consequently, CKA may partly capture object/category structure rather than exclusively measuring part-specific information. I suggest adding a category-preserving part-name permutation test, within-category CKA, and an object-level feature baseline.

  2. The interpretation of the model rankings in Section 5.1 should consider a broader set of factors. The models in Figure 2 differ simultaneously in model size, pretraining data, input resolution, architecture, and training objective. The current experiments can support associations between these factors and the observed rankings, but they are insufficient to establish that any single factor caused the differences. The corresponding conclusions should therefore be stated more cautiously.

Questions for authors
  1. After randomly permuting part names while preserving the object category and part-frequency distribution, how do the CKA/SVCCA scores and model rankings in Figure 2 change? Could the authors also provide within-category CKA and an object-level feature baseline?

  2. I suggest that the authors consider a broader set of factors when interpreting the model rankings, including model size, pretraining data, input resolution, architecture, and training objective. This would also make the paper a more reliable reference for future research.

  3. I suggest including visualizations for representative object-part cases, such as Grad-CAM or modality-appropriate feature-attribution/activation visualizations for transformer, point-cloud, and video encoders. These examples would help readers assess whether models with high part-similarity actually attend to the corresponding object-parts.

2 comments
AuthorsOpenReview ↗

Summary so far

Dear Reviewer,

Thank you again for your review. As the discussion period is closing soon, here is a brief summary of the new experiments and revisions we made in response to your questions:

  • Category confound (W1/Q1): Your suggested controls were critical in discovering important confounds. The original Fig. 2 ranking barely changes under a category-preserving part-name permutation (ρ = 0.94) or an object-level baseline (ρ = 0.86). We therefore revised the computation of Part-Similarity (while still using CKA as the backbone for computation and interpretation). Under the revised metric, agreement with the permutation control drops to ρ = 0.15, so the category signal is largely removed.
  • Similarity vs. alignment under the revised metric (W1): Importantly, under the revised part-similarity the main finding of §6.2 still holds. We hope this improves the scientific rigour of our work
  • Updated §5 findings (W1) The broader claims largely hold, with some further qualifications. Vision–language pretraining is neither necessary nor sufficient: CLIP and OpenShape stay near the top, while SigLIP, Uni3D and ULIP2 drop. Segmentation objectives help some models (SAM2, PartField) but not all of them (SAM, SuperFlex).
  • Broader factors (W2/Q2) We added an explicit taxonomy of families and sub-families by training objective, architecture and scale. With these we expand our findings: Scale alone does not explain the ranks: DINOv3 is last among lifted-2D models, and Sonata and Concerto fall below the 3D mean. The same factors predict the two metrics differently. For example, SigLIP is first on 2D alignment but only mid-ranked on similarity, and ULIP2 is near the top on 3D alignment but last on similarity.
  • Visualizations (Q3) Grad-CAM-style maps are not well defined for Part-Similarity, because it compares two independently trained spaces with no direct map between them. We will instead add qualitative figures that are well defined: text-conditioned activations after Part-Alignment, PCA feature maps comparing high- and low-scoring models, and the transport plans from our alignment probe.

The reviewers diagnostic suggestions led to a more rigorous Part-Similarity metric, and the paper's central conclusions hold under it. If anything remains unclear or there is anything more we can provide, please let us know.

Best regards, The Authors

Dear reviewer, As the discussion period ends soon we would appreciate updating of scores if we have addressed your concerns.

Best, The authors

Reviewer_KD2zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates part-level vision-language representations across 39 vision encoders. It evaluates representational similarity, alignment using optimal transport, and transferability to unseen part hierarchies. Results show that high part-similarity does not necessarily imply easy alignment.

Critical strengths
  1. Good research question. The paper extends vision-language representational similarity analysis to the object-part level, offering a valuable perspective.

  2. Comprehensive empirical analysis. The evaluation covers 39 vision encoders and distinguishes representational similarity from alignment, providing useful empirical findings.

Critical weaknesses
  1. Insufficient robustness analysis of part-alignment. Section 6.1 uses a fixed four-layer MLP trained for only one epoch to compare the alignment capabilities of different encoders. Without sensitivity analyses on training duration and probe capacity, it is unclear whether the observed rankings reflect intrinsic representation properties or differences in probe optimization.

  2. Limited validation of part-transferability. Section 7 evaluates transferability using ground-truth part segments and part names at test time, while Unpaired+ additionally assumes access to the downstream part-label vocabulary during training. Moreover, Unpaired achieves only 42.2% MA on unseen categories, compared to a chance baseline of 38.1%. These results provide limited evidence for generalization to practical automatic part-labeling scenarios.

Questions for authors
  1. How stable are the model rankings in Figure 4 and the correlation between part-similarity and part-alignment when varying the training duration or MLP capacity?

  2. Can the authors evaluate end-to-end performance without ground-truth part segments and predefined part-name candidates at test time? Additionally, what is the individual contribution of each modification introduced in Unpaired+?

2 comments
AuthorsOpenReview ↗

Summar So Far

Dear Reviewer KD2z,

Thank you again for your reviews. As the discussion period comes to an end, here is a brief summary of the new experiments we ran at your request:

  • Probe sensitivity (W1/Q1): On a five-encoder subset, we varied MLP depth (1, 2, 4) and training length (1, 3, 5 epochs). In 6 of the 8 non-default settings, the ranking agrees strongly with Fig. 4 (ρ ≥ 0.6). So the rankings reflect the representations themselves, not how the probe was optimized. Similarity and alignment also stay weakly related across this grid (ρ = 0.10).
  • Without ground-truth segments at test time (W2/Q2 - first part): We replaced ground-truth segments with segments discovered by SuperFlex. Unpaired+ (0.177 IoU) stays clearly above Unpaired (0.139) and chance (0.124), and close to the Combined oracle (0.180). This matches the pattern in Table 1.
  • Unpaired+ ablation (Q2 - second part): Each modification adds to the score, from 0.4605 to 0.5074 Matching Accuracy (MA)

What will be added to the paper: We will add the full-encoder sensitivity analysis, the SuperFlex-segment evaluation and the Unpaired+ ablation table to the revision. We refer the reviewers to the full discussion above for details. If anything remains unclear or their remains anything more we can provide, please let us know.

Best, Authors

Thank you for the additional experiments. The Unpaired+ ablation is informative. However, my concerns are only partially resolved. (1) With five encoders, ρ ≥ 0.6 is not statistically significant, and at the default depth (L4) the ranking changes notably with more epochs (ρ = 0.5/0.6), suggesting the 1-epoch probe may be under-trained; absolute MA with variance across seeds would help. (2) The SuperFlex evaluation still relies on GT part names at test time, and Unpaired (0.139) is close to chance (0.124). (3) Given the substantial revision of the part-similarity metric in response to Reviewer 1DVd (ρ = 0.22 with the original Fig. 2), the main findings would effectively need re-review. I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.