acceptodds
Under review as a conference paper at ICLR 2027

BindSplit: Benchmarking Seen and Unseen Compositional Generalization in VLMs

Abstract

Compositional generalization requires vision-language models (VLMs) to understand novel combinations of familiar concepts, and it is measured with compositional benchmarks. However, if most evaluation samples contain combinations that models already saw during training, high benchmark scores may not reflect generalization to new compositions. In this work, we ask whether compositional benchmark scores reflect generalization to unseen compositions. We focus on attribute–object bindings and introduce the BindSplit protocol, which splits the samples of existing benchmarks into seen and unseen by whether their bindings appear in a reference corpus that approximates the training data. We find that existing benchmarks contain very few samples with unseen bindings: at most 88 per benchmark against CC12M. To address this gap, we introduce BindSplit Bench, a benchmark built to compare performance on seen and unseen bindings directly. Its seen and unseen splits have similar sizes, share their attributes, and are generated and human-reviewed through the same pipeline. Across 75 VLMs, accuracy drops from seen to unseen bindings for 72 models, and seen accuracy, although correlated with unseen accuracy, overstates how much models improve on unseen bindings. Our results suggest that progress on existing compositional benchmarks may not translate directly to unseen compositions. By separating seen and unseen bindings, BindSplit Bench makes this generalization gap measurable.

Primary area:
datasets and benchmarks
Keywords:
compositional generalization, benchmark, bindings, CLIP, vision language model

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_hxVcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In the paper, the authors study whether benchmarks evaluating CLIP-style models are contaminated because the models being evaluated have seen the evaluation bindings during training. They first collect binding pairs from large-scale image datasets such as CC12M, finding that most current benchmarks contain a significant number of the tested pairs in the training data used for large-scale pretraining. They construct a benchmark with pairings not contained in the large-scale datasets. They then train models on existing backbones, showing that models perform better on seen pairs than on unseen ones and that current benchmarks overstate how good models are at generalising.

Critical strengths

The paper poses an important question. To evaluate whether models are truly generalising, we need to assess what they saw during training, which is easy when trained on small synthetic tasks such as PUG but hard for large-scale models. Properly understanding whether these models can generalise helps to understand generalisation capabilities, understanding of training dynamics, and can aid in developing new architectures.

The paper reveals systematic contamination in current benchmarks for evaluating CLIP-style models, which is important to know.

Critical weaknesses

Evaluating whether images contain a concept based on the caption relies heavily on whether every image has all its concepts labeled. In a complex image containing multiple entities, relations, and attributes, it might not be super common to label every colour; for example, if the scene contains a dog chasing a cat in front of a car, then not all entities will be labeled with their colour with some likelihood. Given the size of the pretraining dataset, this could mean that a substantial number of combinations are only unseen for their text label and not the image.

The trained models perform around chance or sometimes even below chance even for bindings considered to be seen; to me, this looks like a systematic issue either in the original model or the training. In order to speak of a systematic failure, the model should perform above chance before the distribution shift is introduced. We already know from other benchmarks that models perform worse on certain combinations that are more often seen, and given the high correlation with colorswap, the question becomes how much additional knowledge we gain compared to other benchmarks.

Questions for authors

Do you have an explanation for why the correlation decreases for the models whose exposure you checked? How do you know that the drop in unseen quality is not confounded by unseen being harder, given that a drop persists when only evaluating the first binding? This seems to mean that the unseen bindings are harder.

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your thoughtful and constructive feedback.

1. Images can contain unlabelled concepts

W1: "a substantial number of combinations [in pretraining dataset] are only unseen for their text label and not the image"

We agree that some combinations may appear in images without being mentioned in captions. However, in our controlled PUG:SPARE setting (Sec. 5.4), all objects and attributes are explicitly labeled (e.g., "red lion and blue zebra"). Unseen bindings on PUG:SPARE are absent from both images and captions. Our findings still hold in this setting.


2. Models have low performance

W2a: "trained models perform around chance or sometimes even below chance even for bindings considered to be seen; to me, this looks like a systematic issue either in the original model or the training"

Low accuracy on our benchmark does not necessarily mean there is an issue with the models or training. The same models achieve higher compositional accuracy on existing benchmarks (Table 23).


W2b: "In order to speak of a systematic failure, the model should perform above chance before the distribution shift is introduced."

Our evaluated models that perform above chance on text accuracy for seen bindings show a drop on unseen bindings (Table 3). We also evaluated a stronger model, EmbeddingGemma 2, which performs well above chance on seen bindings. It still shows a drop across all three accuracy measures (table below).

ModelText seen / unseenImage seen / unseenGroup seen / unseen
EmbeddingGemma 268.9 / 50.544.7 / 27.338.7 / 21.3
Chance252516.7

3. What we know from other benchmarks

W2c: "We already know from other benchmarks that models perform worse on certain combinations that are more often seen"

Could you please clarify this point, and possibly point us to benchmarks showing that models perform worse on more frequently seen combinations?

Our finding is the opposite: models perform worse on unseen bindings than on seen bindings.


4. Additional knowledge we gain compared to other benchmarks

W2d: "given the high correlation with colorswap, the question becomes how much additional knowledge we gain compared to other benchmarks."

Our benchmark provides three additional insights compared to existing benchmarks:

  1. Seen vs unseen bindings: We show that model accuracy depends on whether attribute-object bindings were seen during training.
  2. Limits of compositional fine-tuning: Methods that improve accuracy on existing benchmarks rarely improve accuracy on our unseen split (table below).
  3. Separate seen/unseen evaluation: Existing benchmarks, including ColorSwap, have too few unseen samples to evaluate seen and unseen bindings separately. Our benchmark supports this evaluation for both pretrained and fine-tuned models.
BenchmarkCLICCS-CLIPNegCLIP
ColorSwap8/12 (+0.2)11/12 (+2.9)8/14 (+1.0)
ARO-VGA10/12 (+3.3)12/12 (+8.1)14/14 (+9.0)
BiVLC10/12 (+1.1)12/12 (+2.9)11/14 (+0.9)
VisMin9/12 (+1.1)12/12 (+3.0)11/14 (+0.9)
BindSplit unseen3/12 (−0.6)4/12 (−1.0)5/14 (−1.0)

Each cell: backbones improved / total backbones (mean text accuracy change in percentage points vs. zero-shot).


5. Correlation explanations

Q1: "Do you have an explanation for why the correlation decreases for the models whose exposure you checked?"

Models with unchecked pretraining data may have seen some of our unseen bindings. This could explain the higher seen–unseen correlation for these models. Those models also show a smaller accuracy drop on unseen bindings (−5.78 points for unchecked exposure vs. −8.13 for checked; Table 5).


6. Is the unseen drop because of unseen is harder?

Q2: "How do you know that the drop in unseen quality is not confounded by unseen being harder, given that a drop persists when only evaluating the first binding? This seems to mean that the unseen bindings are harder."

On seen and unseen samples with matched object and attribute recognizability, the accuracy gap remains: text accuracy drops by 9.2 points across 20 zero-shot models (95% CI [6.1, 12.3]).

In our controlled PUG:SPARE setting (Sec. 5.4), the drop also persists, even though all objects and attributes in unseen bindings are observed during training.

Thank you for providing the additional EmbeddingGemma 2 experiment. It is good to see that it scores above chance on the benchmark. Thank you as well for running the recognizability experiment.
My two remaining concerns are: 
1) Where the gap is coming from. From my understanding, the models in the PUG setting are still fine-tuned. Hence, fine-tuning on PUG does not establish that the held-out bindings were not seen during pretraining. Furthermore, the widened check found 300 of 644 unseen samples contained a seen binding. However, the gap did not widen when excluding them from the benchmark. This could mean that seeing a binding very rarely over hundreds of millions of images does not provide a sufficient training signal. It could mean that the remaining ones might similarly be partly present in the dataset but not labeled through text. Or it could mean that one part of the split is inherently more difficult. Or something else. Do you have an idea what the reason is?
2) I find the improvements via additional training hard to judge. I didn’t find any mean values for Table 3 in the appendix. And when comparing some of the text values myself, the improvement on seen concepts for NegCLIP, LABCLIP, CS-CLIP, and CLIC seems small. For your table in the response comparing against previous benchmarks, could you also provide the increase on seen-split performance, not only unseen performance?


I also apologise. I meant that we know that they perform worse on less frequently seen ones, not more frequently seen ones.

AuthorsOpenReview ↗

Addressing two remaining concerns

Thank you for your discussion, and for acknowledging our new experiments and results!

We address your two remaining concerns here.

C1. Where is the seen-unseen gap from?

C1.1. PUG:SPARE held-out bindings

"From my understanding, the models in the PUG setting are still fine-tuned. Hence, fine-tuning on PUG does not establish that the held-out bindings were not seen during pretraining."

We agree. Our new analysis confirms that some held-out bindings appear in pretraining captions. Our PUG:SPARE results show that holding bindings out of fine-tuning contributes to the seen-unseen gap. Seeing these bindings in the pretraining captions does not close this gap.

In one of our three random held-out blocks (setup discussed in Sec 5.1), the held-out bindings appear about as often in Re-LAION-400M captions as the seen bindings (median: in 152 vs. 196 captions). The LAION-400M models fine-tuned on PUG:SPARE still score lower on the fully unseen split (with the held-out bindings) than on the seen bindings split (R@1: 84.6% vs. 100%). The gap also remains in the other two held-out blocks and for the CC12M and YFCC-15M models.


C1.2. Why does the gap not widen after removing samples found by widened reference check?

"Furthermore, the widened check found 300 of 644 unseen samples contained a seen binding. However, the gap did not widen when excluding them from the benchmark. [...] Do you have an idea what the reason is?"

Thank you for this thoughtful question!

The 300 removed samples contain bindings that are very rare in pretraining, so their unseen accuracy stays the same as that of the other unseen samples (i.e., the gap does not widen). We think that the first interpretation you mentioned is the most likely reason: "seeing a binding very rarely over hundreds of millions of images does not provide a sufficient training signal."

In each of the 300 removed samples, the expanded check found at least one seen binding. In Re-LAION-400M (361M captions), these found bindings appear in a median of 1 caption (max 5). In contrast, a seen-split binding appears in a median of 196 captions. We show the binding frequency statistics below:

GroupSamplesMedian captions per bindingMax captions per binding
Seen split67519646,646
Unseen, removed300*15
Unseen, kept34400

* 9 of the 300 samples were found in YFCC-15M only. This row counts the bindings found in Re-LAION-400M for the other 291 samples.


C2. Improvements via additional training

"I find the improvements via additional training hard to judge. I didn’t find any mean values for Table 3 in the appendix. And when comparing some of the text values myself, the improvement on seen concepts for NegCLIP, LABCLIP, CS-CLIP, and CLIC seems small. For your table in the response comparing against previous benchmarks, could you also provide the increase on seen-split performance, not only unseen performance?"

Thank you for raising this. We added the seen-split row in the table below. We also added a LABCLIP (a linear alignment method) column to the table.

Key finding: The four compositional methods do not improve our seen split either. All four methods improve ARO, and the three fine-tuning methods (NegCLIP, CS-CLIP, CLIC) also improve the other existing benchmarks.

BenchmarkCLICLABCLIPCS-CLIPNegCLIP
ColorSwap8/12 (+0.2)0/16 (−5.8)11/12 (+2.9)8/14 (+1.0)
ARO-VGA10/12 (+3.3)16/16 (+8.2)12/12 (+8.1)14/14 (+9.0)
BiVLC10/12 (+1.1)7/16 (−0.4)12/12 (+2.9)11/14 (+0.9)
VisMin9/12 (+1.1)5/16 (−1.3)12/12 (+3.0)11/14 (+0.9)
BindSplit seen5/12 (−0.2)1/16 (−4.4)3/12 (−1.7)5/14 (−0.8)
BindSplit unseen3/12 (−0.6)1/16 (−3.1)4/12 (−1.0)5/14 (−1.0)

Each cell: backbones improved / total backbones (mean text accuracy change in percentage points vs. zero-shot).

Why does seen accuracy not improve? All four methods train on COCO. Our seen bindings appear in pretraining data (CC12M and LAION-400M), but few appear in COCO. The table below shows the percentage of test samples with at least one / at least two bindings (in the positive or negative caption) that appear in COCO captions:

ARO-VGAVisMinBiVLCColorSwapBindSplit seen
≥ 1 binding98.4%89.3%81.9%72.4%35.3%
≥ 2 bindings88.2%78.6%60.6%61.0%6.8%

Note: BiVLC and VisMin also test objects and relations; we count only their samples with an attribute–object binding (536 of 2,933 and 159 of 2,084).

Reviewer_HS36OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether compositional benchmark scores reflect generalization to attribute-object bindings that were not present in training. The authors introduce BindSplit, a protocol that labels benchmark samples as seen, unseen, or mixed by checking the bindings in both correct and incorrect captions against reference caption corpora. An audit of four benchmarks finds few samples whose bindings are all unseen under this definition. The authors then construct BindSplit Bench, with 675 seen and 644 unseen human-reviewed samples in a two-image, two-caption attribute-swap setting. Across 75 model configurations, text accuracy is lower on unseen bindings for 72 configurations, with an average drop of 6.02 percentage points.

Critical strengths
  • The paper is generally well-written and easy to follow. The main research question is clear and well motivated: distinguishing recognition of familiar bindings from generalization to new ones is important for interpreting progress on compositional benchmarks.

  • The benchmark design addresses several relevant evaluation problems. Both splits use the same generation and review pipeline, and checking all four bindings prevents a difference in familiarity between the correct and wrong caption from directly revealing the answer. The two-image, two-caption format also tests both retrieval directions.

  • The evaluation across 20 pretrained backbones and multiple adaptation methods is a strong point. Reporting the exposure-checked subset separately and resampling backbones rather than treating all 75 configurations as independent are useful methodological choices.

Critical weaknesses
  1. My main concern is whether the new benchmark sufficiently isolates unseen binding from other distribution changes. The two splits share only 9 object names, with 113 seen-only and 121 unseen-only names (Section 4.2). The recognizability check is based on all-pairs binary classification on ImageNet images and VAW crops, rather than recognition of the objects and attributes in the generated benchmark scenes. Similar, near-ceiling object scores in that setting do not establish equal difficulty in cluttered two-object scenes or under unusual appearances. The paper acknowledges this limitation but it remains unclear how much of the gap on BindSplit Bench itself comes from binding novelty rather than differences in objects and scene difficulty between the splits.

  2. The meaning of “checked” training exposure is stronger than the procedure clearly establishes. Section 4.2 reports searching a 173M-sample subset of LAION-400M, while Sections 5.1-5.2 describe absence from the pretraining data as checked for LAION-400M backbones. The paper also inherits filtering from ImageNet-AO, but it is unclear whether this guarantees coverage of the remaining data for the specific checkpoints. More generally, absence under caption matching does not establish absence of the visual binding e.g. captions can omit attributes or use alternative names. The reference-corpus limitation for other models is already clearly acknowledged, but the stronger interpretation for the checked subset needs better justification. This matters because training exposure is central to the paper's interpretation of the benchmark.

  3. The claim that seen accuracy overstates progress on unseen bindings is not fully established by the cross-model regression. A slope below one in Figure 3 shows an association across configurations that differ in architecture, pretraining data, and adaptation method. It does not directly show that e.g. an intervention improving a given model by 10 points on seen bindings improves it by only 6 points on unseen bindings. The backbone bootstrap and probit analysis are useful, but do not resolve this distinction.

  4. The benchmark-validity checks leave some uncertainty about what the very low scores measure. All models are below the stated random-chance levels in seen image and group accuracy in Table 3. Reducing captions to one binding improves these scores, but the pooled means are only 27.5% image accuracy and 17.4% group accuracy, close to the stated chance levels of 25% and 16.7% in Table 13. Improvement relative to the two-binding setting alone is therefore limited evidence that models reliably perceive the attributes in these images. Each retained sample also has only one human review, without reported agreement on a shared subset. These observations do not demonstrate that the benchmark is flawed, but stronger checks of readability, annotation reliability, and systematic image/caption preferences would increase confidence that the gap measures the intended capability.

  5. The audit supports a narrower conclusion than the general statement that existing benchmarks barely test unseen compositions. “Unseen” requires all bindings in both captions to be absent, and mixed samples are excluded. These samples can nevertheless contain an unseen target binding and may test some compositional generalization, even if familiarity could provide a shortcut in some cases. The extraction also covers only 536 of 791 attribute-change BiVLC samples and 159 of 294 VisMin samples (Table 21). I understand the motivation for a clean all-unseen evaluation, but the paper should distinguish the scarcity of that particular setting from the broader absence of tests of unseen composition.

Questions for authors

My questions are mainly about the weaknesses mentioned above. I would appreciate it if the authors could elaborate further on these concerns. In addition:

  1. For the adapted models, are unseen bindings also audited in generated negative captions, or only in positive COCO captions?
7 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your thoughtful and constructive feedback.

1. Seen / unseen object overlap

W1a: “The two splits share only 9 object names, with 113 seen-only and 121 unseen-only names (Section 4.2)”

Yes, this is by design to satisfy our key seen/unseen criteria (discussed in Section 4.2).

In our controlled PUG:SPARE experiment (Section 5.4), the all objects and attributes in the unseen split are also in the seen split. Accuracy still drops on unseen.


2. Isolating unseen binding performance from recognizability

W1b: “... how much of the gap on BindSplit Bench itself comes from the binding novelty rather than from differences in objects and scene difficulty between the splits”

Thank you for raising this. Even when the seen and unseen samples have matched object and attribute recognizability, the seen and unseen accuracy gap still remains.

We measured recognizability with 8 CLIP-family models on our generated two-object scenes. After filtering for matched recognizability, 282 samples remain in each split. Across 20 zero-shot CLIP-family models, unseen text accuracy is 9.2 percentage points lower than seen (95% CI [6.1, 12.3]). Image and group accuracy also drop.


3. Meaning of "checked" training exposure

W2a: "Section 4.2 reports searching a 173M-sample subset of LAION-400M, while Sections 5.1-5.2 describe absence from the pretraining data as checked for LAION-400M backbones"

Thank you for raising this. You are right that "checked" was too strong for LAION-400M. We originally only had access to a 173M-sample subset with LLM-extracted noun phrases to check for binding exposure.

We are now checking the unseen bindings against captions beyond this subset. We will share these results in a follow-up post in the next few days.

W2b: "The paper also inherits filtering from ImageNet-AO, but it is unclear whether this guarantees coverage of the remaining data for the specific checkpoints."

We agree. The filtering inherited from ImageNet-AO does not guarantee coverage of checkpoints trained on YFCC-15M. Table 5 therefore labels checkpoints covered only by this filtering as "partly checked".

We are now checking our unseen bindings against YFCC-15M. We will share the results in a follow-up post this week.

W2c: "absence under caption matching does not establish absence of the visual binding e.g. captions can omit attributes or use alternative names."

Our controlled experiment (Section 5.4), where captions fully label all visual bindings, also shows the seen-unseen performance gap observed in our benchmark.

We agree that caption matching cannot establish the absence of visual bindings from training images. We will state this in the limitations.

We are also testing sensitivity to alernate names in caption matching. These analyses are running, and we will share the results in a follow-up post in the next few days.


4. Accuracy overstates progress on unseen bindings claim

W3: "A slope below one in Figure 3 ... does not directly show that e.g. an intervention improving a given model by 10 points on seen bindings improves it by only 6 points on unseen bindings"

Yes, we agree. Our "model that gains 10 pp on seen bindings gains only about 6 pp on unseen ones" sentence suggests an intervention effect that we didn't intend to establish.

We removed this sentence and similar wording in the re-loaded paper (edits in blue).


5. Benchmark validity checks: low benchmark scores

W4a: "All models are below the stated random-chance levels in seen image and group accuracy in Table 3."

The below-chance scores suggest limitations of the evaluated models. A stronger model, EmbeddingGemma 2 (Google), exceeds chance on all seen accuracies and still shows a seen-unseen gap.

ModelText seen / unseenImage seen / unseenGroup seen / unseen
EmbeddingGemma 268.9 / 50.544.7 / 27.338.7 / 21.3
Chance252516.7

6. Benchmark validity checks: concept readability

W4b: "[One caption binding] improvement relative to the two-binding setting alone is therefore limited evidence that models reliably perceive the attributes in these images."

Direct recognition tests show high pairwise accuracy for objects (97%) and attributes (88%). These results are averaged across 8 CLIP-family models on all benchmark images. Each test compares a correct label with an incorrect label and checks whether the correct label scores higher.


7. Benchmark validity checks: annotation reliability

W4c: "Each retained sample also has only one human review"

The accuracy gap remains after additional annotation checks. We added two external VLM judges: Gemini 3.1 Pro and Qwen3.5-9B. Of the 1,319 human-approved samples, 937 also passed at least one VLM judge.

Across 75 models, the text accuracy drop is 5.6 points (95% CI [3.4, 7.8]), compared with 6.0 on the full set.

AuthorsOpenReview ↗

Rebuttal by Authors #2

8. Unseen split excludes mixed samples

W5a: "“Unseen” requires all bindings in both captions to be absent, and mixed samples are excluded. These samples can nevertheless contain an unseen target binding and may test some compositional generalization, even if familiarity could provide a shortcut in some cases."

We agree that mixed samples can test some compositional generalization. On ARO VG-A, however, mixed accuracy matches or exceeds seen accuracy in all five settings, while unseen accuracy is consistently lower (table below).

Mixed samples retain familiar bindings that may help models answer correctly. Our unseen split excludes this shortcut by requiring all bindings in both captions to be absent from the checked training captions.

ARO VG-A, split against COCO captionsFullMixedSeenUnseen
Zero-shot60.1 ± 1.360.6 ± 1.860.3 ± 0.757.1 ± 2.7
NegCLIP71.1 ± 2.273.1 ± 2.667.1 ± 1.461.6 ± 3.9
LABCLIP69.4 ± 1.571.5 ± 1.964.4 ± 1.061.3 ± 2.3
CS-CLIP68.9 ± 2.170.3 ± 2.366.6 ± 2.259.8 ± 3.2
CLIC64.2 ± 2.165.3 ± 2.563.1 ± 1.558.0 ± 3.5

Image-to-text accuracy (%, chance 50). Mean ± SD over the 9 backbones shared by all five settings.


9. Distinguish scarcity of the all bindings unseen setting

W5b: "the paper should distinguish the scarcity of that particular setting (all bindings in both captions to be unseen) from the broader absence of tests of unseen composition"

We will clarify that our scarcity claim concerns tests where all bindings in both captions are unseen. This requirement removes familiar bindings as a possible shortcut. As shown in #8 (W5a) above, mixed accuracy (not all bindings are unseen) can match or exceed seen accuracy, while fully unseen accuracy is lower.


10. Generated negative captions audit

Q1: "For the adapted models, are unseen bindings also audited in generated negative captions, or only in positive COCO captions?"

We audited generated negative captions for NegCLIP and LABCLIP and found no unseen bindings. We did not audit the generated hard negatives for CLIC and CS-CLIP. Both methods still show a seen–unseen accuracy gap (Table 5). We will clarify the audit coverage.

AuthorsOpenReview ↗

Follow up: analyses on LAION-400M and YFCC-15M

We're following up on two of the analyses we had mentioned that we were running in our first rebuttal response (§3).

Follow up on W2a ("§3. Meaning of "checked" training exposure")

W2a: "Section 4.2 reports searching a 173M-sample subset of LAION-400M, while Sections 5.1-5.2 describe absence from the pretraining data as checked for LAION-400M backbones"

We expanded our LAION-400M caption check beyond the 173M-sample subset (see note below). We removed 291 of 644 unseen samples because they no longer met our requirement that all four bindings are unseen. 224 of these 291 (77%) have only one of four bindings seen in the full LAION-400M.

The seen-unseen gap remains stable on the remaining 353 unseen samples, compared to the original 644 unseen samples (table below). Across all 75 models, the gap is 6.05 pp [4.04, 8.04], compared to 6.02 pp in the paper.

The table below compares the seen-unseen text accuracy gap before and after the expanded caption check. Accuracy gaps are averaged over the 17 models with a LAION-400M pre-trained backbone from Table 3 of our paper.

Evaluation splitSeen / unseen sample countsSeen − unseen text accuracy (pp) [95% CI]
Original (submitted)675 / 6449.20 [6.53, 11.67]
After expanded LAION-400M check675 / 3539.28 [6.35, 12.05]

Note on LAION-400M: We checked Re-LAION-400M (361M captions). This is LAION's re-release and the only version still available for us to download.

Follow up on W2b ("§3. Meaning of "checked" training exposure")

W2b: "The paper also inherits filtering from ImageNet-AO, but it is unclear whether this guarantees coverage of the remaining data for the specific checkpoints."

We have now checked our unseen bindings directly against YFCC-15M captions. Previously, we inherited ImageNet-AO's filtering.

This check changes the label of 20 of 644 unseen samples (3%). In each of these 20 samples, we found exactly one of the four bindings in YFCC-15M captions. We found none of the four bindings in the remaining 624 samples.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your thoughtful and constructive feedback.

1. Reliability of the unseen split

W1a: "the proposed benchmark remains subject to the same dependence on corpus coverage: bindings absent from the selected corpora may be found in additional data."

We agree, our unseen labels are relative to the checked corpora. Models with unchecked pretraining data may have seen some of our unseen bindings, which could raise their unseen accuracy and reduce the measured gap.

Even with this possible unseen exposure, 40/41 models with uncheckable corpora perform worse on the unseen split (Table 5). EmbeddingGemma 2 (new model), whose training data we cannot check, also drops from 68.9% (seen) to 50.5% (unseen) in text accuracy.


W1b: "[the paper] does not establish how the new benchmark's labels and resulting model comparisons change as reference coverage expands"

We are running analyses to examine how our unseen labels and the resulting model accuracies might change as we expand our reference corpus coverage. We will share these results in a follow-up post over the next few days.


2. Additional evaluation value over existing benchmarks

2.1 Findings consistent with prior work

W2a: "The reported difficulty with unseen compositions and the association between seen and unseen are broadly consistent with prior findings discussed in the paper"

We agree our findings are consistent with prior work. Our benchmark adds separate evaluation of seen and unseen bindings within the same benchmark, so we can directly measure the accuracy gap for pretrained and compositionally fine-tuned models.


2.2 Practical consequence our proposed evaluation/benchmark

W2b: "Section 5.3 mainly compares existing benchmark scores with BindSplit's seen scores, leaving unclear what different assessment of current models or compositional fine-tuning methods is obtained from the new unseen split. The PUG:SPARE experiment usefully distinguishes models whose seen performance is saturated, but a corresponding demonstration on BindSplit itself would better establish its added value. For example, identifying methods that improve on existing benchmarks yet fail to improve on the new unseen split would make the practical consequence of the proposed evaluation clearer."

Thank you for this suggestion! We find that compositional fine-tuning methods that improve existing benchmark scores rarely improve accuracy on BindSplit's unseen split (table below).

BenchmarkCLICCS-CLIPNegCLIP FT
ColorSwap8/12 (+0.2)11/12 (+2.9)8/14 (+1.0)
ARO-VGA10/12 (+3.3)12/12 (+8.1)14/14 (+9.0)
BiVLC10/12 (+1.1)12/12 (+2.9)11/14 (+0.9)
VisMin9/12 (+1.1)12/12 (+3.0)11/14 (+0.9)
BindSplit unseen3/12 (−0.6)4/12 (−1.0)5/14 (−1.0)

Each method is compared with the zero-shot model of the same backbone. Cells report backbones that improve / all backbones (mean change in text accuracy in percentage points).

These results show that higher scores on existing benchmarks do not necessarily translate into better performance on unseen bindings. We added this finding to the re-uploaded paper (Section 5.2, Appendix A.12), with changes in blue.

AuthorsOpenReview ↗

Follow up with new results: reference coverage expansion

Follow up on W1b: reference coverage expansion (§1 Reliability of the unseen split)

W1b: "[the paper] does not establish how the new benchmark's labels and resulting model comparisons change as reference coverage expands"

Our main finding holds on the 344 unseen samples that remain after the expanded reference coverage check. 73 of 75 models still perform worse on unseen attribute-object bindings, and the mean seen-unseen gap in text accuracy remains the same as originally reported in our paper: 6.02 percentage points [3.95, 7.98] across 75 models (table below).

Expanded reference coverage: Our paper checked COCO, CC12M, and a 173M-sample subset of LAION-400M. We expanded the LAION-400M check to 361M captions (see note below), and added YFCC-15M.

Benchmark label changes: After our expanded check, at least one binding is seen in 300 of the original 644 unseen samples. In 78% of these 300 samples, only one of the four bindings is seen. We removed all 300 samples, because our unseen split requires all four bindings to be unseen, and evaluated on the remaining 344.

The table below shows the mean seen-unseen text accuracy gap across 75 models at each step of the check. The 675 seen samples stay the same; only the unseen samples change.

Reference captionsNN unseen samplesSeen − unseen text acc (pp) [95% CI]Models with lower unseen acc
Submitted paper coverage6446.02 [4.10, 7.83]72 / 75
+ LAION-400M (361M captions)3536.05 [4.04, 8.04]73 / 75
+ LAION-400M + YFCC-15M3446.02 [3.95, 7.98]73 / 75

Note on LAION-400M: We checked Re-LAION-400M (361M captions). This is LAION's re-release and the only version still available for us to download.

Dear Reviewer Eb5Z,

As the discussion period is coming to a close, we would appreciate hearing whether our responses have addressed your concerns. If they have, could you please consider increasing your score to reflect your updated assessment?

If you have any remaining concerns, please let us know. We would be happy to address them before the discussion period ends.

Thank you for your time!

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.