acceptodds
Under review as a conference paper at ICLR 2027

Effects of Pretraining Concept Frequency and Semantic Similarity among Classes on CLIP Generalized Zero-Shot Accuracy

Abstract

Foundation vision-language models (VLMs), such as CLIP, exhibit impressive generalized zero-shot recognition capabilities. However, their performance is shown to be affected by imbalances in their pretraining data. Understanding the factors that affect downstream zero-shot recognition accuracy under such pretraining imbalance is essential. Recent work has shown that across many downstream datasets, the zero-shot accuracy of individual concepts follows a log-linear trend with concept frequency in the pretraining dataset, and that achieving linear improvements in accuracy requires exponentially more training examples. In this paper, we (i) demonstrate that semantic similarity among target classes is another strong predictor of zero‑shot accuracy, (ii) analyze how this semantic effect interacts with concept‑frequency trends across different dataset sizes, model capacities and pre‑training scales, showing that the frequency‑accuracy link diminishes for larger pretraining corpora while the semantic‑similarity signal does not collapse, and (iii) propose a graph‑based zero-shot accuracy‑estimation framework for a given list of class names that jointly takes (a) concept frequency estimates and (b) a pair‑wise semantic‑similarity matrix as input; allowing for a more flexible and realistic setup than the fixed 1000-way classification. Overall, our results provide a more comprehensive understanding of how both semantic complexity and class cardinality influence generalized zero-shot accuracy of vision-language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
zero-shot accuracy, vision-language models, CLIP, pretraining concept frequency, semantic similarity

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_3dwoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work re-explores the relation between long-tailed generalization and the pre-training frequency vs semanticity importance; for accurate representational learning, classes require significant training data to enable the model to distinguish features of a class from another; however, it is extremely challenging to ensure equal object representation across large-scale datasets, as some objects are scarce. The learning mechanism can also confuse a low-frequency class with another high-frequency class that shares its features, provided the two are semantically close. The paper thus argues that semantic proximity among target classes is a second determinant of zero-shot accuracy alongside frequency, shows that the two are complementary across subset sizes, pre-training scales, and model sizes, and proposes ZAGT, a graph transformer predicting per-class accuracy from frequencies and pairwise class similarities.

Critical strengths
  • This work has shown that the commonly cited log-linear frequency-accuracy trend holds only for binned averages and is inflated by dataset pooling; the unbinned per-class relation is weaker.

  • The paper was quite enjoyable to read, especially in contrast with mostly generated papers with limited polishing (the only issue regarding presentation is that the figures could have been made bigger (as they are quite small) with better design (legends on top, not leaving white space, etc.)

Critical weaknesses
  • Semantic similarity may not matter as much as the paper claims. Even combined with frequency, it explains little of the per-class variance, and the gain is significant only on a few datasets. The text-embedding measure is also somewhat circular, as CLIP already classifies by nearest text embedding, and visual similarity is never ruled out as a potential cause.

  • ZAGT does not show a clear advantage over a linear model using the same two factors, while the ablation study suggests that its predictive signal comes primarily from class-name embeddings rather than the similarity graph. The graph model contribution appears thin with a little more than half a page of dedication. Furthermore, the paper overlooks prior CLIP studies that have already linked class confusability to per-class accuracy, such as Shao et al. [1] and Ren et al. [2].

[1] J.-J. Shao et al., “Investigating the Limitation of CLIP Models: The Worst-Performing Categories”, 2023 [2] S. Ren et al., “Delving into the Openness of CLIP” Findings of ACL, 2023.

Questions for authors
  • Does the semantic signal remain predictive after controlling for visual similarity between classes?

  • Is ZAGT’s improvement over the linear baseline statistically significant across different splits, and does it outperform a linear model that also incorporates class-name embeddings?

  • Could the authors concentrate the contributions into one idea so that any reader could clearly identify how this work contributes to significant new knowledge in DL?

Since the scale has no middle option, I selected weak reject, but my assessment is closer to borderline reject. The paper is generally sound and well written but might not meet the significance/originality threshold for this venue.

4 comments

We thank the reviewer 3dwo for their feedback. We have made some updates in the paper and conducted additional experiments to answer the reviewer’s questions.

Main contribution of our paper and scope: We consider the analysis part of our paper as our main contribution: (i) frequency and accuracy correlations are not a straight line, (ii) semantic similarity and frequency may have correlations because how dataset granularity and difficulty interacts, (iii) however, despite that, on a single downstream-dataset level, frequency and semantic similarity together estimate accuracy better than just frequency or just semantic similarity; and semantic similarity can be considered as one of the missing factors for predicting accuracy.

Our secondary contribution is the proposal of a transformer-based architecture, which basically aims 2 things: (1) drawing attention to the information loss during the summarization of semantic distances with MIN- and MEAN-K and pointing out a simple solution to that, and (2) through mixing multiple downstream datasets and obtaining better results, supporting the possibility that the learned information is not specific to a single dataset but transferable to similar datasets as well.

On the importance of semantic similarity: As the reviewer noted, the R^2 values are rather weak, which, however, is also the case for frequency-based factors. Although weak, for the three largest datasets in our test suite in terms of number of classes (ImageNet, Places365, SUN397), semantic similarity significantly lowers the prediction MSE (Table 5). For the small datasets with fewer than 100 classes, the results are rather noisy after halving the class numbers to create a split and neither adding semantic similarity, nor frequency gives a statistically significant improvement.

Semantic similarity graph and ZAGT: The reviewer’s observation on Table 6 led us to re-examine this ablation, and we found that the frequency input was being lost whenever text embeddings were ablated. Class frequencies entered as scalars through a bias-free Linear layer followed directly by LayerNorm, so the normalization erased them. The full ZAGT is unaffected, as our control experiments confirm. With this correction, the graph + frequency model on ImageNet improves from 0.033 to 0.143, compared to 0.213 for frequency + text embeddings and 0.259 for all three. The corrected Table 6 shows that text embeddings remain the strongest component, and that the semantic graph adds a clear gain on top of them. Appendix L further shows that the trained model’s predictions respond strongly to the graph.

Circularity: During development, we had a similar concern that TDE would introduce circularity. To avoid this, we mostly chose to use its SBERT counterpart SDE. For example, throughout all ZAGT experiments, we strictly used SDE instead of TDE.

Semantic vs visual similarity: The scope of our paper is the semantic similarity among class names. Our setting assumes only a list of class names, with no images of the target classes at prediction time: the abstract describes an accuracy-estimation framework “for a given list of class names”, and our limitations section states that the approach “relies solely on semantic similarity and concept frequencies” (Sec. 6, second paragraph). We therefore do not claim that semantic similarity is a factor independent of visual similarity. We use it as a text-only proxy for the visual similarity that this setting cannot measure, similar to LOVM [3].

Nevertheless, upon the question of the reviewer, we were also curious to see what would happen if semantic information is added on top of the visual information. To perform this experiment, we took the average embedding of class images using CLIP (MIN-IDE) and DINOv2 ViT-L/14 (MIN-DDE). The class images are used from the same images on which accuracy is calculated (so this was somewhat an oracle-style experiment). The results are as follows (on ImageNet, ViT-B/32 LAION-400M, 50 class splits):

Added factorBase factorHeld-out Spearman (base -> base + added)Spearman gainpp (Holm, 6 tests)Adds?
MIN-SYNDMIN-IDE0.426 -> 0.423−0.0020.6831.000no
MIN-TDEMIN-IDE0.426 -> 0.422−0.0030.9791.000no
MIN-SDEMIN-IDE0.426 -> 0.422−0.0040.4811.000no
MIN-SYNDMIN-DDE0.464 -> 0.457−0.0070.5281.000no
MIN-TDEMIN-DDE0.464 -> 0.462−0.0020.0210.123no
MIN-SDEMIN-DDE0.464 -> 0.457−0.0070.6491.000no

Statistical tests are performed using the methodology described in Appendix E (over Spearman instead of MSE). As expected, none of the semantic similarity factors was able to improve over visual similarity factors.

(continued in the next comment)

ZAGT - linear models paired test results: We have added a new table (Table 8) under Appendix J. As a summary, when there is a significant difference, ZAGT is mostly the leading one: in 14 cases ZAGT results are better, 6 cases are non-significant, in 1 case (SUN397 with subset size 8) linear is better.

Adding text embeddings to linear model: We have now added a new section in Appendix K for this experiment (Table 9). Basically, we first directly concatenated SBERT embeddings to the inputs of the linear model and it did not help and decreased the performance. We also pushed it one step further and applied PCA to the embedding to reduce the dimension. It boosted the performance of the linear model on ImageNet and Places365, decreased CUB results and we don’t see a change in SUN397. We also tested how a linear model with class text embeddings fares when only a single model is trained over 4 datasets. This setting is one of the important results of ZAGT: the learned predictive abilities are not dataset specific but they generalize to similar downstream tasks as well. This property is important for the use case of estimating class accuracies on different settings. However, different from ZAGT, when the linear model is trained with multiple datasets, it has a negative impact on its overall performance.

Additional related-work: We would like to thank the reviewer for their suggestions on the related work on class-level semantic similarity usage in CLIP literature. We incorporated both into our related work section under Class Confusability and Semantic Similarity in CLIP paragraph. Shao et al. detailed the descriptions of the worst-performing classes by unveiling them from the image-text similarity matrix and assessing how close the most similar classes score. Ren et al. analyze how CLIP behaves when new open classes are introduced to the task and they argue that when they group classes under super classes, intra-cluster textual similarities are higher which makes them more likely to be confused with each other. Different from their setting, our similarity scores come from class names and we do not use images. Our aim is mainly the prediction of accuracy. We additionally explore text encoders other than CLIP’s, such as SBERT.

Figure presentation in the paper: We have improved the legibility of the figures in the paper.

[3] Zohar et al., LOVM: Language-Only Vision Model Selection, 2023.

Reviewer_3dwoOpenReview ↗

Post-Rebuttal update

I thank the authors for their efforts on the rebuttal; my concerns have been addressed, and the score adjusted.

Minor comment:

  • I would still recommend adding a few sentences in the abstract, introduction, or conclusion on how these contributions can be exploited in practice, e.g., to anticipate which classes of a new label set are likely to fail, or to guide data curation beyond frequency balancing. I trust that the authors will find the best use cases.

We thank the reviewer for their response. We have updated the first paragraph of Sec. 6 (Conclusion, Discussion & Limitations) to address their comment on how this work can be used in practice. We added the following:

  • Our work is a step toward class-level zero-shot performance prediction for CLIP models. ZAGT's generalization over multiple downstream datasets may indicate that what has been learned is not dataset specific.

  • Class-level performance prediction may help identify the classes most at risk and adjust class selection accordingly, even before data collection.

  • Semantic similarity between concepts, on top of their individual frequencies, may help flag possible weak spots during the pretraining data curation phase.

The revised pdf has been uploaded.

Reviewer_KfSvOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the determinants of downstream generalized zero-shot recognition performance in foundation vision-language models (VLMs) like CLIP. While prior work primarily attributes zero-shot accuracy variations to a log-linear trend with pretraining concept frequencies, this paper demonstrates that concept frequency alone is insufficient and heavily confounded by semantic similarity (i.e., semantic competition/opposition among target classes).

Critical strengths
  1. Important Nuance on a Hot Topic: Challenging the dominant narrative that "pretraining frequency is all you need" for VLM zero-shot performance is timely, highly valuable, and backed by comprehensive empirical analysis.

  2. Methodological Rigor: The use of permutation tests (with 20,000 permutations) and Holm corrections to validate whether frequency and semantics genuinely add complementary information prevents over-claiming from noisy correlations.

  3. Practical Modeling Solution (ZAGT): Formulating accuracy estimation as a graph-transformer problem that takes arbitrary lists of class names, frequencies, and pairwise distances is creative and steps beyond rigid 1000-way classification setups.

  4. Thorough Evaluation: The study evaluates multiple backbone sizes (ViT-B/32, ViT-B/16, ViT-L/14), pretraining datasets (LAION-400M, LAION-2B, MetaCLIP), and an extensive suite of 11 downstream datasets.

Critical weaknesses
  1. Vulnerability to Text-Encoder Circularity and Vocabulary Mismatches: The text-distance metrics (TDE) rely on the VLM's own text encoder, which the authors rightly note introduces circularity. While SBERT (SDE) bypasses this, text-based semantic distances fail to capture visual confusion caused by out-of-vocabulary terms, multi-word synonyms, or fine-grained visual ambiguities that are textually distant (e.g., visually similar objects with completely different etymologies). A brief discussion or limitation on how text space diverges from visual feature space would strengthen the text.

  2. Interpretability of ZAGT vs. Linear Models: While ZAGT achieves strong generalization across datasets, transformer black-box models obscure the exact feature weights. Given that the linear model analysis successfully uncovers intuitive interactions (e.g., how semantic crowding buffers the loss of frequency correlation in fine-grained datasets like CUB), a quick feature-attribution or attention-weight analysis inside ZAGT would help verify whether it learns similar intuitive rules.

Questions for authors

none

1 comment

We thank the reviewer KfSv for their feedback. We added a section that inspects ZAGT in detail to Appendix L upon their suggestions. Below we address their questions and the issues raised in the Weaknesses.

Limitation of text similarity, compared to visual similarity: We had a small discussion of this for SYND (synset distance on WordNet hierarchy) in the paper but upon the reviewer’s request, we extended it to SDE (SBERT Euclidean distance) in Sec. 4.1 (Methods of Quantification). Here is the version that is sectioned according to the reviewer’s discussion:

  • Out of vocabulary terms: This is where our system would start to produce unpredictable results. A visual-similarity-based score would probably be more robust in this situation. To make the argument more solid, consider an extreme case: Imagine that there are 3 people: person A, B, and C. Probably, the similarity between the text embeddings of their names would not correspond to their visual similarities (assuming that they are not famous people and do not have many images on the internet). However, if we take their photos and provide them to CLIP’s image encoder, then, the distances have more chances to reflect their similarities. Here is the awkward part: although image classification is all about similarities and the visual similarity would reflect itself on the performance, we are still trying to estimate the zero-shot performance of CLIP. So, when its text encoder fails unpredictably (they produce some embeddings, but they are not grounded to the visual concepts), the accuracies would not follow our usual notion of similar concepts. Therefore, even if there were cases where our text-based system predicts the results better, it would still be by chance, not that it predicts the accuracy better in this out-of-vocabulary case. However, there is also a second factor to consider in this case: the frequency-based factors would be able to indicate that there would actually be a problem here by predicting low accuracy. Our work currently does not inspect these extreme cases where text encoders do not provide meaningful results, however, it would be a nice future work to inspect them.

  • Multi-word synonyms: In this case, we are mostly dependent on the capability of the text model to encode synonyms into similar embeddings. Visual similarities in this situation could be less noisy but similar to the argument above, the zero-shot CLIP itself is dependent on those varying texts. Since we mostly use a different text encoder (SBERT) than CLIP’s, their results may not coincide.

  • Fine-grained visual ambiguities that are textually distant (e.g., visually similar objects with completely different etymologies): Actually we think that this fits to the example we provided for SYND in the paper: bakery and baguette classes are distant in the wordnet hierarchy but when we inspect the images visually, they are actually very similar. We think that this would be less of an issue for SBERT compared to the WordNet-based similarity calculations; however, the discussions for the synonym part are also valid here.

Feature attribution and attention analysis: This was a great idea. We inspected how ZAGT responds to some perturbations in its input space and how semantic distance affects the attention bias. We added the experimental setup and results to Appendix L (Fig. 13 and 14). To summarize:

  • As we increase the distance between a class pair, the learned attention bias in ZAGT decreases.

  • As we increase the log frequency term in the input, ZAGT assigns higher accuracy to the class on average.

  • As we make a class more distant to the others by increasing its SDE to its nearest neighbor in the SDE matrix, ZAGT assigns higher accuracy to that class.

  • When we change the log frequency and distance of a class together, we observe that when it is distant from other classes, even if its log frequency values get lower, its predicted accuracy changes little. This is not the case when it has similar opponent classes where we observe a decrease in ZAGT’s output. This result is actually similar to what had been shown in our toy example figure in the paper. (Fig. 1)

Reviewer_kjB9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper (1) demonstrates that semantic similarity between categories is another key predictor of classification accuracy in VLMs, particularly as the scale of pre-training corpora increases; (2) reveals that while the correlation between "frequency" and "accuracy" weakens as the training set grows, the signal derived from semantic similarity remains robust; and (3) proposes a graph-based framework for estimating zero-shot accuracy.

Critical strengths

S1: A highly insightful finding.

Existing research on vision-language models (VLMs) often merely asserts that CLIP possesses zero-shot classification capabilities without delving into the underlying nature of this ability—in some cases, the authors of such studies have not even read the original CLIP paper. This research offers a compelling explanation for CLIP’s zero-shot classification performance: the semantics of the categories present in the pre-training dataset exert a significant influence on this capability. This serves as a powerful complement to previous studies that focused primarily on category frequency.

S2: Clear presentation.

This paper is well-written, both in terms of specific word choices and overall organization. I think many readers will be able to easily grasp what this paper is trying to convey.

S3: Solid experiments.

Critical weaknesses

W1: Some relevant studies have been omitted from the citations, and certain technical terms—such as "zero-shot accuracy" and "generalized zero-shot accuracy"—remain undefined. It is worth noting that the "zero-shot classification" as applied to CLIP refers to the model's ability to classify samples from a dataset it has not encountered before; however, the test classes themselves may have been included during the model's pre-training phase. In contrast, another line of research focuses on "strict" zero-shot classification [1][2][3], where the test classes are entirely unseen by the model during pre-training.

I suggest that the authors provide clear definitions for these technical terms to prevent the potential misuse of their research findings. Furthermore, the discussion should be broadened to include references to the literature on strict zero-shot classification.

W2: Some specific experimental details still need to be disclosed—for instance, the duration of pre-training and the GPU resources used (i.e., whether the model was trained in-house or if a pre-trained model from another study was utilized).

[1] Zerodiff: Solidified visual-semantic correlation in zero-shot learning, ICLR25.

[2] Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning, CVPR24.

[3] ZeroDiff++: Generative Test-Time Adaptation for Zero-shot Learning, TPAMI26.

Questions for authors

Model training time, GPU usage?

1 comment

We thank the reviewer kjB9 for their feedback. Below we address their questions and the issues raised in the Weaknesses.

Semantic similarity in the “strict” zero-shot literature: The “zero-shot” in our paper is the CLIP-style “zero-shot” and we are aware that this is different than the Conventional Zero-Shot Learning and Generalized Zero-Shot Learning (Xian et al. [4]). In the conventional setting, the class labels are the ones that are strictly not-seen during the training. CLIP may see the downstream class captions of its test classes during training (that’s why we can calculate the occurrence frequencies for the downstream classes in the first place). Our intention for putting “Generalized” in the title is to signify that it is not the conventional “strict” zero-shot setting. However, we did not specifically discuss this in the paper as the reviewer mentioned and we now made this separation clear and cited the suggested and relevant related work as semantic relation between seen and unseen classes is the basis for conventional ZSL. The changes are done under the newly added “Conventional Zero-Shot Learning” paragraph under Related Work.

Missing computation details: For the CLIP models, we directly used the pretrained weights of Ilharco et al. [5] (OpenCLIP) but this was not specified. We corrected this issue (Sec. 5, Experimental Setting). For our models, we additionally added ZAGT (our transformer-based model) training time information to the paper. Since our models are only predicting accuracies, they are rather small (~1.3M parameters) and are trained quickly. Using a consumer-level GPU (e.g. RTX 3090), it takes about an hour to train the model. More detailed training hyperparameters and newly added compute details reside in Appendix G.

[4] Xian et al., Zero-Shot Learning -- The Good, the Bad and the Ugly, 2017 [5] Ilharco et al., OpenCLIP, 2021

Comments on the paper

1 comment

We sincerely thank all reviewers for their time and constructive feedback. We appreciate their recognition of the paper's finding that is "a powerful complement to previous studies that focused primarily on category frequency" (kjB9), its "important nuance on a hot topic" backed by "methodological rigor" and a "thorough evaluation" (KfSv), and its demonstration that "the commonly cited log-linear frequency-accuracy trend holds only for binned averages and is inflated by dataset pooling" (3dwo), as well as its clear presentation (kjB9, KfSv, 3dwo).

The central contribution of the paper is its analysis. We show that the log-linear frequency-accuracy trend reported by prior work appears stronger than it is because of binning and dataset pooling: fine-grained datasets have both rarer and semantically closer classes, so frequency and semantic similarity are entangled at the dataset level. Despite this entanglement, within a single dataset the two factors are still complementary. Semantic similarity among the target classes predicts CLIP's per-class zero-shot accuracy as frequency does, and using the two together predicts better than either alone. This holds as the class subset size, the pretraining scale, and the model size vary. ZAGT is a secondary contribution: a graph Transformer that reads the whole similarity structure instead of a scalar summary and transfers across datasets, which is what a tool that predicts per-class accuracy from a list of class names alone needs: generalization.

The summary of our changes:

  • Conventional vs. CLIP-style zero-shot learning (kjB9). We added a new first paragraph to Sec. 2 (Related Work). It defines the conventional and generalized zero-shot settings following Xian et al. (2017), cites the compatibility-based ZSL line (DeViSE, ALE, SynC) and the recent works suggested by the reviewer, and explains why we borrow the term "generalized" for the CLIP setting.
  • Compute and checkpoint details (kjB9). Sec. 5 (Experimental Setting) now states that we use the OpenCLIP checkpoints of Ilharco et al. (2021). Appendix G now reports the size of ZAGT (1.32M parameters) and its training time (about an hour on an RTX 3090).
  • Text similarity as a proxy for visual similarity (KfSv, 3dwo). We added a closing paragraph to Sec. 4.1 (Methods of Quantification) that states that we use text similarity as a proxy for visual similarity and describes where this proxy fails: out-of-vocabulary names, synonyms, and visually similar classes with unrelated names. Our reply to KfSv discusses the three cases raised in the review one by one.
  • Corrected input ablation (3dwo). We corrected Table 6 (Appendix I). In the submitted version, the graph-only row had lost its frequency input, because the scalar frequency passed through a bias-free linear layer and a LayerNorm. With a sinusoidal frequency encoding, this row rises from 0.033 to 0.143 on ImageNet, and a new control row shows that the full model is unaffected by the change. Text embeddings remain the strongest input and the semantic graph adds a clear gain on top of them.
  • Paired tests of ZAGT against the linear model (3dwo). We added Appendix J with Table 8 which reports paired tests between ZAGT and the linear model for every dataset and subset size. ZAGT is significantly better in 14 of the 21 cells, the difference is not significant in 6 cells, and the linear model is significantly better in 1 cell.
  • Linear models with class text embeddings (3dwo). We added Appendix K with Table 9, which gives the class text embedding to the linear model. Concatenating the raw embedding hurts, a PCA-reduced embedding helps the per-dataset linear model, and a single linear model trained on the four datasets loses what ZAGT gains from the mixture.
  • What ZAGT learned (KfSv, 3dwo). We added Appendix L with Figs. 13 and 14, which inspect the trained model. The learned attention bias decreases with semantic distance. The predicted accuracy rises with frequency and with the distance to the closest class. More importantly, frequency matters mostly when a close opponent exists, which supports the idea of Fig. 1.
  • Semantic vs. visual similarity (3dwo). Our setting assumes a list of class names and no images of the target classes, so visual similarity is outside the scope of the paper. However, in our reply to 3dwo we report a control experiment that adds the text-based factors on top of visual similarity factors computed from CLIP and DINOv2 image features. None of the text-based factors adds to them, which is consistent with their role as a text-only proxy.
  • Related work on class confusability in CLIP (3dwo). We added a paragraph to Sec. 2 that discusses Shao et al. (2023) and Ren et al. (2023) and stated how our setting differs from theirs.
  • Figures (3dwo). The legibility of the figures is increased without changing their contents.

Detailed responses are provided in our comments to each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.