- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the questions of how much part-level information has been encoded in the embedding space of the vision models. The paper studies three sub-questions: part-similarity, part-alignment, and part-transferability, to cover the similarity of part-level visual features to language model features in both the labeled and unlabeled scenarios, along with the capability of generalizing to unseen part hierarchies. The study broadly covers many models spanning 2D, 3D, and video models to find out the rankings of each vision model on part-level information in their embeddings, and some general observations on the relationship (e.g., weak correlation) between properties like part-similarity and part-alignment.
Critical strengths
-
The problem of part-level features from vision models is largely underexplored in the literature, and the paper conducts a very comprehensive study across various models including 2D, 3D, and video models to demonstrate their capability of encoding part-level information in their embeddings. It can be useful for future work to choose which models to use for downstream vision and robotics tasks that require part-level understanding.
-
The results in the paper are decently presented with figures which clearly show the rankings of each model in different settings. Also, the findings of the each part is clearly presented (with the important takeaways highlighted), which makes audience easy to digest the takeaway from the paper.
Critical weaknesses
-
It seems not very natural to choose the isomorphic similarity metrics presented in Section 3.1 (Centered Kernel Alignment (CKA), Singular Vector Canonical Correlation Analysis (SVCCA), or local metrics such as Mutual k-Nearest Neighbors (Mutual-kNN)) as metrics to capture the similarity of embeddings. In many related work (e.g., in customized image generation), the DINO/CLIP similarity used in DreamBooth [1] is used to measure the similarity of two DINO features or two CLIP features from an image and text, directly by cosine similarity. The authors state that recent works also employ the metrics (CKA, SVCCA, Mutual-kNN) shown above, but the authors did not give specific references in this section about which works adopt these metrics. I think they are rarely adopted metrics so this is why I raise this question, because the whole paper is based on these metrics instead of simple (and much more broadly adopted cosine similarity), which may undermine the soundness of the paper.
-
For part-similarity, I feel the experimental design over-complicates the question. Basically, part-similarity focuses on studying whether part-level similar features have strong similarity. The paper adopts an unnecessary intermediate text modalities of labels as proxy to let features from different parts compare with the text embedding to calculate the similarity of the part-level patches. However, I think it can be done by directly calculating the similarity of the visual embeddings between the part-level patches. If the focus of the paper is really to study how the object-part features are similar to the features from language models, then the title of the paper should be changed to something like "object-parts feature alignment with language model features". The current title indicates that the study can be largely irrelevant with language models, which is very misleading and difficult to understand what is the focus of the paper.
[1] Ruiz et al. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. CVPR 2023.
Questions for authors
-
What is the motivation of choosing metrics like CKA, SVCCA, Mutual-kNN which are relatively uncommon, compared to the more broadly used (and more concise) cosine similarity? Basically, what are the aspects that these metrics can better measure in the evaluated tasks?
-
What is the design principle of the part-similarity experiment? Basically, why language models always need to be as an intermediate proxy to measure similarity for part-level visual embeddings? Also, is the current title well reflect which is the focus of the paper, as it does not mention language models, but all the experiments are actually studying the feature similarity between part-level visual embeddings and language embeddings?
Summary so far
Dear Reviewer,
Thank you again for your review. As the discussion period is closing soon, here is a brief summary of the new experiments and revisions we made in response to your questions:
The reviewers diagnostic suggestions led to a more rigorous Part-Similarity metric, and the paper's central conclusions hold under it. If anything remains unclear or there is anything more we can provide, please let us know.
Best regards, The Authors