- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In the paper, the authors study whether benchmarks evaluating CLIP-style models are contaminated because the models being evaluated have seen the evaluation bindings during training. They first collect binding pairs from large-scale image datasets such as CC12M, finding that most current benchmarks contain a significant number of the tested pairs in the training data used for large-scale pretraining. They construct a benchmark with pairings not contained in the large-scale datasets. They then train models on existing backbones, showing that models perform better on seen pairs than on unseen ones and that current benchmarks overstate how good models are at generalising.
Critical strengths
The paper poses an important question. To evaluate whether models are truly generalising, we need to assess what they saw during training, which is easy when trained on small synthetic tasks such as PUG but hard for large-scale models. Properly understanding whether these models can generalise helps to understand generalisation capabilities, understanding of training dynamics, and can aid in developing new architectures.
The paper reveals systematic contamination in current benchmarks for evaluating CLIP-style models, which is important to know.
Critical weaknesses
Evaluating whether images contain a concept based on the caption relies heavily on whether every image has all its concepts labeled. In a complex image containing multiple entities, relations, and attributes, it might not be super common to label every colour; for example, if the scene contains a dog chasing a cat in front of a car, then not all entities will be labeled with their colour with some likelihood. Given the size of the pretraining dataset, this could mean that a substantial number of combinations are only unseen for their text label and not the image.
The trained models perform around chance or sometimes even below chance even for bindings considered to be seen; to me, this looks like a systematic issue either in the original model or the training. In order to speak of a systematic failure, the model should perform above chance before the distribution shift is introduced. We already know from other benchmarks that models perform worse on certain combinations that are more often seen, and given the high correlation with colorswap, the question becomes how much additional knowledge we gain compared to other benchmarks.
Questions for authors
Do you have an explanation for why the correlation decreases for the models whose exposure you checked? How do you know that the drop in unseen quality is not confounded by unseen being harder, given that a drop persists when only evaluating the first binding? This seems to mean that the unseen bindings are harder.
7 comments
Thank you for providing the additional EmbeddingGemma 2 experiment. It is good to see that it scores above chance on the benchmark. Thank you as well for running the recognizability experiment.
My two remaining concerns are:
1) Where the gap is coming from. From my understanding, the models in the PUG setting are still fine-tuned. Hence, fine-tuning on PUG does not establish that the held-out bindings were not seen during pretraining. Furthermore, the widened check found 300 of 644 unseen samples contained a seen binding. However, the gap did not widen when excluding them from the benchmark. This could mean that seeing a binding very rarely over hundreds of millions of images does not provide a sufficient training signal. It could mean that the remaining ones might similarly be partly present in the dataset but not labeled through text. Or it could mean that one part of the split is inherently more difficult. Or something else. Do you have an idea what the reason is?
2) I find the improvements via additional training hard to judge. I didn’t find any mean values for Table 3 in the appendix. And when comparing some of the text values myself, the improvement on seen concepts for NegCLIP, LABCLIP, CS-CLIP, and CLIC seems small. For your table in the response comparing against previous benchmarks, could you also provide the increase on seen-split performance, not only unseen performance?
I also apologise. I meant that we know that they perform worse on less frequently seen ones, not more frequently seen ones.
Addressing two remaining concerns
Thank you for your discussion, and for acknowledging our new experiments and results!
We address your two remaining concerns here.
C1. Where is the seen-unseen gap from?
C1.1. PUG:SPARE held-out bindings
"From my understanding, the models in the PUG setting are still fine-tuned. Hence, fine-tuning on PUG does not establish that the held-out bindings were not seen during pretraining."
We agree. Our new analysis confirms that some held-out bindings appear in pretraining captions. Our PUG:SPARE results show that holding bindings out of fine-tuning contributes to the seen-unseen gap. Seeing these bindings in the pretraining captions does not close this gap.
In one of our three random held-out blocks (setup discussed in Sec 5.1), the held-out bindings appear about as often in Re-LAION-400M captions as the seen bindings (median: in 152 vs. 196 captions). The LAION-400M models fine-tuned on PUG:SPARE still score lower on the fully unseen split (with the held-out bindings) than on the seen bindings split (R@1: 84.6% vs. 100%). The gap also remains in the other two held-out blocks and for the CC12M and YFCC-15M models.
C1.2. Why does the gap not widen after removing samples found by widened reference check?
"Furthermore, the widened check found 300 of 644 unseen samples contained a seen binding. However, the gap did not widen when excluding them from the benchmark. [...] Do you have an idea what the reason is?"
Thank you for this thoughtful question!
The 300 removed samples contain bindings that are very rare in pretraining, so their unseen accuracy stays the same as that of the other unseen samples (i.e., the gap does not widen). We think that the first interpretation you mentioned is the most likely reason: "seeing a binding very rarely over hundreds of millions of images does not provide a sufficient training signal."
In each of the 300 removed samples, the expanded check found at least one seen binding. In Re-LAION-400M (361M captions), these found bindings appear in a median of 1 caption (max 5). In contrast, a seen-split binding appears in a median of 196 captions. We show the binding frequency statistics below:
| Group | Samples | Median captions per binding | Max captions per binding |
|---|---|---|---|
| Seen split | 675 | 196 | 46,646 |
| Unseen, removed | 300* | 1 | 5 |
| Unseen, kept | 344 | 0 | 0 |
* 9 of the 300 samples were found in YFCC-15M only. This row counts the bindings found in Re-LAION-400M for the other 291 samples.
C2. Improvements via additional training
"I find the improvements via additional training hard to judge. I didn’t find any mean values for Table 3 in the appendix. And when comparing some of the text values myself, the improvement on seen concepts for NegCLIP, LABCLIP, CS-CLIP, and CLIC seems small. For your table in the response comparing against previous benchmarks, could you also provide the increase on seen-split performance, not only unseen performance?"
Thank you for raising this. We added the seen-split row in the table below. We also added a LABCLIP (a linear alignment method) column to the table.
Key finding: The four compositional methods do not improve our seen split either. All four methods improve ARO, and the three fine-tuning methods (NegCLIP, CS-CLIP, CLIC) also improve the other existing benchmarks.
| Benchmark | CLIC | LABCLIP | CS-CLIP | NegCLIP |
|---|---|---|---|---|
| ColorSwap | 8/12 (+0.2) | 0/16 (−5.8) | 11/12 (+2.9) | 8/14 (+1.0) |
| ARO-VGA | 10/12 (+3.3) | 16/16 (+8.2) | 12/12 (+8.1) | 14/14 (+9.0) |
| BiVLC | 10/12 (+1.1) | 7/16 (−0.4) | 12/12 (+2.9) | 11/14 (+0.9) |
| VisMin | 9/12 (+1.1) | 5/16 (−1.3) | 12/12 (+3.0) | 11/14 (+0.9) |
| BindSplit seen | 5/12 (−0.2) | 1/16 (−4.4) | 3/12 (−1.7) | 5/14 (−0.8) |
| BindSplit unseen | 3/12 (−0.6) | 1/16 (−3.1) | 4/12 (−1.0) | 5/14 (−1.0) |
Each cell: backbones improved / total backbones (mean text accuracy change in percentage points vs. zero-shot).
Why does seen accuracy not improve? All four methods train on COCO. Our seen bindings appear in pretraining data (CC12M and LAION-400M), but few appear in COCO. The table below shows the percentage of test samples with at least one / at least two bindings (in the positive or negative caption) that appear in COCO captions:
| ARO-VGA | VisMin | BiVLC | ColorSwap | BindSplit seen | |
|---|---|---|---|---|---|
| ≥ 1 binding | 98.4% | 89.3% | 81.9% | 72.4% | 35.3% |
| ≥ 2 bindings | 88.2% | 78.6% | 60.6% | 61.0% | 6.8% |
Note: BiVLC and VisMin also test objects and relations; we count only their samples with an attribute–object binding (536 of 2,933 and 159 of 2,084).
Rebuttal by Authors
Thank you for your thoughtful and constructive feedback.
1. Images can contain unlabelled concepts
We agree that some combinations may appear in images without being mentioned in captions. However, in our controlled PUG:SPARE setting (Sec. 5.4), all objects and attributes are explicitly labeled (e.g., "red lion and blue zebra"). Unseen bindings on PUG:SPARE are absent from both images and captions. Our findings still hold in this setting.
2. Models have low performance
Low accuracy on our benchmark does not necessarily mean there is an issue with the models or training. The same models achieve higher compositional accuracy on existing benchmarks (Table 23).
Our evaluated models that perform above chance on text accuracy for seen bindings show a drop on unseen bindings (Table 3). We also evaluated a stronger model, EmbeddingGemma 2, which performs well above chance on seen bindings. It still shows a drop across all three accuracy measures (table below).
3. What we know from other benchmarks
Could you please clarify this point, and possibly point us to benchmarks showing that models perform worse on more frequently seen combinations?
Our finding is the opposite: models perform worse on unseen bindings than on seen bindings.
4. Additional knowledge we gain compared to other benchmarks
Our benchmark provides three additional insights compared to existing benchmarks:
Each cell:
backbones improved/total backbones(mean text accuracy changein percentage points vs. zero-shot).5. Correlation explanations
Models with unchecked pretraining data may have seen some of our unseen bindings. This could explain the higher seen–unseen correlation for these models. Those models also show a smaller accuracy drop on unseen bindings (−5.78 points for unchecked exposure vs. −8.13 for checked; Table 5).
6. Is the unseen drop because of unseen is harder?
On seen and unseen samples with matched object and attribute recognizability, the accuracy gap remains: text accuracy drops by 9.2 points across 20 zero-shot models (95% CI [6.1, 12.3]).
In our controlled PUG:SPARE setting (Sec. 5.4), the drop also persists, even though all objects and attributes in unseen bindings are observed during training.