- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work re-explores the relation between long-tailed generalization and the pre-training frequency vs semanticity importance; for accurate representational learning, classes require significant training data to enable the model to distinguish features of a class from another; however, it is extremely challenging to ensure equal object representation across large-scale datasets, as some objects are scarce. The learning mechanism can also confuse a low-frequency class with another high-frequency class that shares its features, provided the two are semantically close. The paper thus argues that semantic proximity among target classes is a second determinant of zero-shot accuracy alongside frequency, shows that the two are complementary across subset sizes, pre-training scales, and model sizes, and proposes ZAGT, a graph transformer predicting per-class accuracy from frequencies and pairwise class similarities.
Critical strengths
-
This work has shown that the commonly cited log-linear frequency-accuracy trend holds only for binned averages and is inflated by dataset pooling; the unbinned per-class relation is weaker.
-
The paper was quite enjoyable to read, especially in contrast with mostly generated papers with limited polishing (the only issue regarding presentation is that the figures could have been made bigger (as they are quite small) with better design (legends on top, not leaving white space, etc.)
Critical weaknesses
-
Semantic similarity may not matter as much as the paper claims. Even combined with frequency, it explains little of the per-class variance, and the gain is significant only on a few datasets. The text-embedding measure is also somewhat circular, as CLIP already classifies by nearest text embedding, and visual similarity is never ruled out as a potential cause.
-
ZAGT does not show a clear advantage over a linear model using the same two factors, while the ablation study suggests that its predictive signal comes primarily from class-name embeddings rather than the similarity graph. The graph model contribution appears thin with a little more than half a page of dedication. Furthermore, the paper overlooks prior CLIP studies that have already linked class confusability to per-class accuracy, such as Shao et al. [1] and Ren et al. [2].
[1] J.-J. Shao et al., “Investigating the Limitation of CLIP Models: The Worst-Performing Categories”, 2023 [2] S. Ren et al., “Delving into the Openness of CLIP” Findings of ACL, 2023.
Questions for authors
-
Does the semantic signal remain predictive after controlling for visual similarity between classes?
-
Is ZAGT’s improvement over the linear baseline statistically significant across different splits, and does it outperform a linear model that also incorporates class-name embeddings?
-
Could the authors concentrate the contributions into one idea so that any reader could clearly identify how this work contributes to significant new knowledge in DL?
Since the scale has no middle option, I selected weak reject, but my assessment is closer to borderline reject. The paper is generally sound and well written but might not meet the significance/originality threshold for this venue.
4 comments
ZAGT - linear models paired test results: We have added a new table (Table 8) under Appendix J. As a summary, when there is a significant difference, ZAGT is mostly the leading one: in 14 cases ZAGT results are better, 6 cases are non-significant, in 1 case (SUN397 with subset size 8) linear is better.
Adding text embeddings to linear model: We have now added a new section in Appendix K for this experiment (Table 9). Basically, we first directly concatenated SBERT embeddings to the inputs of the linear model and it did not help and decreased the performance. We also pushed it one step further and applied PCA to the embedding to reduce the dimension. It boosted the performance of the linear model on ImageNet and Places365, decreased CUB results and we don’t see a change in SUN397. We also tested how a linear model with class text embeddings fares when only a single model is trained over 4 datasets. This setting is one of the important results of ZAGT: the learned predictive abilities are not dataset specific but they generalize to similar downstream tasks as well. This property is important for the use case of estimating class accuracies on different settings. However, different from ZAGT, when the linear model is trained with multiple datasets, it has a negative impact on its overall performance.
Additional related-work: We would like to thank the reviewer for their suggestions on the related work on class-level semantic similarity usage in CLIP literature. We incorporated both into our related work section under Class Confusability and Semantic Similarity in CLIP paragraph. Shao et al. detailed the descriptions of the worst-performing classes by unveiling them from the image-text similarity matrix and assessing how close the most similar classes score. Ren et al. analyze how CLIP behaves when new open classes are introduced to the task and they argue that when they group classes under super classes, intra-cluster textual similarities are higher which makes them more likely to be confused with each other. Different from their setting, our similarity scores come from class names and we do not use images. Our aim is mainly the prediction of accuracy. We additionally explore text encoders other than CLIP’s, such as SBERT.
Figure presentation in the paper: We have improved the legibility of the figures in the paper.
[3] Zohar et al., LOVM: Language-Only Vision Model Selection, 2023.
Post-Rebuttal update
I thank the authors for their efforts on the rebuttal; my concerns have been addressed, and the score adjusted.
Minor comment:
- I would still recommend adding a few sentences in the abstract, introduction, or conclusion on how these contributions can be exploited in practice, e.g., to anticipate which classes of a new label set are likely to fail, or to guide data curation beyond frequency balancing. I trust that the authors will find the best use cases.
We thank the reviewer for their response. We have updated the first paragraph of Sec. 6 (Conclusion, Discussion & Limitations) to address their comment on how this work can be used in practice. We added the following:
-
Our work is a step toward class-level zero-shot performance prediction for CLIP models. ZAGT's generalization over multiple downstream datasets may indicate that what has been learned is not dataset specific.
-
Class-level performance prediction may help identify the classes most at risk and adjust class selection accordingly, even before data collection.
-
Semantic similarity between concepts, on top of their individual frequencies, may help flag possible weak spots during the pretraining data curation phase.
The revised pdf has been uploaded.
We thank the reviewer 3dwo for their feedback. We have made some updates in the paper and conducted additional experiments to answer the reviewer’s questions.
Main contribution of our paper and scope: We consider the analysis part of our paper as our main contribution: (i) frequency and accuracy correlations are not a straight line, (ii) semantic similarity and frequency may have correlations because how dataset granularity and difficulty interacts, (iii) however, despite that, on a single downstream-dataset level, frequency and semantic similarity together estimate accuracy better than just frequency or just semantic similarity; and semantic similarity can be considered as one of the missing factors for predicting accuracy.
Our secondary contribution is the proposal of a transformer-based architecture, which basically aims 2 things: (1) drawing attention to the information loss during the summarization of semantic distances with MIN- and MEAN-K and pointing out a simple solution to that, and (2) through mixing multiple downstream datasets and obtaining better results, supporting the possibility that the learned information is not specific to a single dataset but transferable to similar datasets as well.
On the importance of semantic similarity: As the reviewer noted, the R^2 values are rather weak, which, however, is also the case for frequency-based factors. Although weak, for the three largest datasets in our test suite in terms of number of classes (ImageNet, Places365, SUN397), semantic similarity significantly lowers the prediction MSE (Table 5). For the small datasets with fewer than 100 classes, the results are rather noisy after halving the class numbers to create a split and neither adding semantic similarity, nor frequency gives a statistically significant improvement.
Semantic similarity graph and ZAGT: The reviewer’s observation on Table 6 led us to re-examine this ablation, and we found that the frequency input was being lost whenever text embeddings were ablated. Class frequencies entered as scalars through a bias-free Linear layer followed directly by LayerNorm, so the normalization erased them. The full ZAGT is unaffected, as our control experiments confirm. With this correction, the graph + frequency model on ImageNet improves from 0.033 to 0.143, compared to 0.213 for frequency + text embeddings and 0.259 for all three. The corrected Table 6 shows that text embeddings remain the strongest component, and that the semantic graph adds a clear gain on top of them. Appendix L further shows that the trained model’s predictions respond strongly to the graph.
Circularity: During development, we had a similar concern that TDE would introduce circularity. To avoid this, we mostly chose to use its SBERT counterpart SDE. For example, throughout all ZAGT experiments, we strictly used SDE instead of TDE.
Semantic vs visual similarity: The scope of our paper is the semantic similarity among class names. Our setting assumes only a list of class names, with no images of the target classes at prediction time: the abstract describes an accuracy-estimation framework “for a given list of class names”, and our limitations section states that the approach “relies solely on semantic similarity and concept frequencies” (Sec. 6, second paragraph). We therefore do not claim that semantic similarity is a factor independent of visual similarity. We use it as a text-only proxy for the visual similarity that this setting cannot measure, similar to LOVM [3].
Nevertheless, upon the question of the reviewer, we were also curious to see what would happen if semantic information is added on top of the visual information. To perform this experiment, we took the average embedding of class images using CLIP (MIN-IDE) and DINOv2 ViT-L/14 (MIN-DDE). The class images are used from the same images on which accuracy is calculated (so this was somewhat an oracle-style experiment). The results are as follows (on ImageNet, ViT-B/32 LAION-400M, 50 class splits):
Statistical tests are performed using the methodology described in Appendix E (over Spearman instead of MSE). As expected, none of the semantic similarity factors was able to improve over visual similarity factors.
(continued in the next comment)