acceptodds
Withdrawn from ICLR 2027

Aligned Is Not Supported: Attributing Image–Text Alignment to Caption Phrases for Data Selection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image–text filters for CLIP-style pretraining score each pair with one number. We show that this number saturates: on CC3M, once the least-aligned 30% of pairs are removed, stricter alignment thresholds stop improving compositional benchmarks and begin to reduce zero-shot transfer. The reason is structural. A global score measures whether a pair matches, not whether the caption’s phrases carry that match; a caption can be well aligned yet hollow, sustained by one salient object, an OCR fragment, or a text prior. We introduce Counterfactual Phrase Intervention (CPI), which attributes the alignment score back to the caption: each nominal head is replaced by a nonce token matched in CLIP-BPE length and surface form, and the resulting similarity drop measures how much of the match that phrase carries. In other words, CPI asks not whether a caption matches its image, but which words in it do the matching. Mean phrase sensitivity correlates only weakly with alignment (r = 0.084), and rankings induced by noun-phrase and relational contexts are nearly independent (ρ = 0.059), so the signal adds information that no global score can recover. Ranking the aligned pool by this signal yields a 50%-data subset that improves VL-CheckList Relation by +1.73 over full data and +0.98 over alignment-only filtering at matched budget (five paired seeds, p = 0.002), is within seed variance of the 70% alignment pool on compositional metrics with 28.7% fewer samples, and preserves zero-shot and retrieval transfer. Used without any change to the objective, the same subset improves NegCLIP by +3.84 and CE-CLIP by +4.92 on VL-CheckList Relation, and the effect replicates on CC12M, on ViT-S/32 and ViT-L/14, and under a SigLIP scorer (80.6% selection overlap against 71.3% at chance). We present these results as controlled evidence at CC3M–CC12M scale; curation costs 1.70 GPU-hours with cached embeddings, and code is provided.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
data curation, vision-language pretraining, CLIP, compositionality, counterfactual intervention, data attribution, data filtering, contrastive learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dQKfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to propose a new metric to curate image-text pair datasets from noisy-webly collected image-text pairs. If I understand correctly, they attempt to incorporate the sensitivity to the caption-perturbation to filter a well-aligned image-text pairs. Their empirical results suggest that the proposed approach sometimes outperforms the baseline model in vision-language alignment benchmark dataset. However, overall, they did not present their approach well, and it was hard to understand the reason why their proposed approach can outperform the baseline methods.

Critical strengths
  1. The idea of using the sensitivity value to get well-aligned image-text pairs might be novel. However, the intuition behind why the method can outperform the existing approach was not very clear.
Critical weaknesses
  1. The empirical effectiveness of the proposed method is limited. Their proposed approach does not much improve the performance in some cases.
  2. The baseline is limited. They list a lot of related work, but only a few naive baselines are used for comparison.
  3. Other than the quality of image-text pair alignment can affect the pre-training of CLIP-like model. As shown in [1, 2], ensuring the diversity of the data is critical in image-text pair pre-training. They are missing this kind of baseline and miss the detailed discussion.
  4. Poor presentation quality. Especially their writing is poor. It was hard to understand their main claim throughout the paper. Also, the figure needs improvement to better help readers to understand the paper.
  5. There can be some bug or mistakes in their experiments. In Table 1 and 2, imagenet zero-shot performance is around 7%. This value is too low.

[1] SemDeDup: Data-efficient learning at web-scale through semantic deduplication [2] MoDE: CLIP Data Experts via Clustering

Questions for authors

It would be great if the authors answer whether my understanding of their methodology is correct.

Reviewer_BK5qOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Counterfactual Phrase Intervention (CPI) for selecting image–text training pairs. After retaining the top 70% of pairs by global alignment, it replaces nominal heads with strings described as lexically empty nonces matched in CLIP-BPE length and surface form, measures the resulting similarity drops, and ranks captions by their mean drop to retain 50% of the original corpus. The central hypothesis is that substitution sensitivity adds useful information beyond global alignment. The main experiments continue training a common CLIP checkpoint under matched optimization budgets, with comparisons against alignment filtering, random selection, and T-MARS. Additional experiments vary the corpus, backbone, scorer, aggregation rule, and training objective. The paper also relates singleton substitution scores to a Taylor expansion and to Shapley values plus interaction corrections.

Critical strengths

The question is relevant and the selection rule is simple and cheap. Distinguishing captions whose alignment depends on their content words from captions that are merely globally similar is a useful curation target, and the scoring cost is small (Table 79). The main five-seed comparison (Table 1, with subset breakdowns in Tables 14–15 and the paired analysis in Table 17) is internally consistent, and Table 18's random selection from the same Align70 pool isolates the second-stage ranking.

The paper reports unfavorable results. Weaker or negative results on swap tasks and SugarCrepe++ are shown rather than omitted.

Critical weaknesses

The "nonce" substitutions do not appear to be meaningless. The whole method rests on replacing a head noun with a string that carries no meaning of its own, so that the similarity drop reflects how much the caption relied on that word. When I looked at how the released code generates these strings, I could not find any check that the result is not a real word. Because the replacement must also have the same number of CLIP tokens as the original, and most common nouns are a single token, the replacement will often be another ordinary word: in my attempt to reproduce the procedure, "tree" became "rock", "bicycle" became "clock", and "wine" became "room". If this is right, the score frequently measures what happens when one concrete object is swapped for another, which is a different quantity from the one motivated in Section 4 and assumed in Proposition 2. It is telling that Table 6, which is meant to show representative perturbations, only displays a "[nonce]" placeholder. The theoretical reading has a separate gap as well: the paper itself notes that the interaction terms in Proposition 3 are not bounded, so mean PAS cannot simply be read as each phrase's share of the support.

I could not reconcile several reported numbers with how they are said to have been produced. Some entries in Tables 4 and 56, which are described as single runs, are not values that a single evaluation could give (e.g., an ImageNet accuracy of 7.413% on 50,000 images); these rows carry the saturation argument in Section 3. The same CPI50 run reports different CIFAR numbers in different appendix tables. In the linear-probe results (Table 13), three different models obtain exactly the same EuroSAT accuracy and standard deviation, which looks like cached features being reused across conditions. The CE-CLIP SugarCrepe++ numbers in Table 3 appear to use a different averaging than the other rows; computed the same way, the gain roughly halves. The normalized and raw PAS statistics reported in Tables 52–54 and 73 are also mutually inconsistent. Finally, from my reading of the released training script, gradients do not seem to be synchronized across GPUs, which would not match the reported batch size of 2048. Any one of these could be a clerical slip, but together they make it hard to take the appendix analyses, and to some extent the training setup, at face value.

Even taking the numbers as reported, the evidence for CPI is thin. Against the main baseline, alignment filtering to 50%, the gains are around 0.4 points on SugarCrepe and 1 point on VL-CheckList Relation, with a small loss on SugarCrepe++. On Relation, CPI is only 0.2 points better than simply sampling half of the data at random, and about half of the large NegCLIP and CE-CLIP improvements are likewise already achieved by random halving. The filtering threshold, aggregation rule, and drop ratio seem to have been chosen by looking at the same benchmarks used for the final comparison. All runs also start from a checkpoint that has already seen the full dataset many times, so the results speak to continued training rather than to data efficiency. Several of the claimed experiments, including NegCLIP, T-MARS, and CC12M, have no corresponding code in the supplement.

For these reasons I recommend rejection: the intervention that was actually run seems to differ from the one the paper describes, several results do not hold together, and the remaining gains are small and were tuned on the evaluation benchmarks.

Questions for authors

Could you share the actual replacement strings used for scoring and report how many of them are real English words? Does the selected subset change much if real words are filtered out?

Could you provide per-seed results for the main and appendix tables, and explain the single-run entries, the identical linear-probe results, and the averaging used for CE-CLIP on SugarCrepe++?

Were gradients synchronized across GPUs in the reported runs, and what was the effective batch size?

Which data were used to choose the 70% threshold, the mean aggregation, and the drop ratio, and were any of the reported benchmarks held out from these choices?

Reviewer_fVRmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that global image-text similarity saturates as a filter for CLIP training data, and proposes replacing each caption's noun heads with matched nonce tokens; the drop in similarity becomes a phrase-level score used to rank captions inside an alignment-filtered pool. Training CLIP ViT-B/32 on the selected half of CC3M improves VL-CheckList Relation from 37.9 to 39.6 and SugarCrepe by a similar margin, with replications on CC12M, other backbones, and SigLIP.

Critical strengths
  1. Cheap and reusable diagnostic: one cached image pass plus a few text passes, and it captures something the alignment filter does not already capture.

  2. Broad and honest evaluation: several seeds, datasets, backbones and scorers, with negative results reported rather than hidden.

Critical weaknesses
  1. The gain over a random 50% subset is negligible. On VL-CheckList Relation the selected subset scores 39.6 against 39.4 for a random half of the same aligned pool, well inside seed noise, and no paired test is reported. Since every run takes the same number of steps, a smaller subset also sees each caption more often, which is exactly what the random control should absorb. The abstract instead leads with gains over the full dataset and over alignment-only filtering. As it stands, the evidence does not separate phrase attribution from simply training on half the data.

  2. The saturation claim rests on a single-seed sweep whose peak is within the noise the paper reports elsewhere, and on a zero-shot curve that is not monotone. The appendix calls the sweep exploratory, but the abstract states saturation as a finding, and every later experiment uses that operating point.

Questions for authors
  1. Please report the paired difference between the selected subset and a random 50% subset of the same aligned pool on VL-CheckList Relation, over the five matched seeds, with an interval, and the same comparison on SugarCrepe. If the interval includes zero, I would read the gain as a data-budget effect rather than attribution.

  2. Please repeat the retention sweep with three to five seeds and say whether the 70% peak is separable from 60% and 80%. If it is not, does the second stage still help at other retention rates, and is 70% still the right operating point?

  3. The appendix reports three different averages for the same attribution quantity, with different signs and magnitudes, and I could not reconcile them. Please state for each which captions were sampled, over which pool, how the values were normalized, and which sample the reported correlation with alignment is computed on.

  4. Selected captions are longer and contain more phrases than dropped ones, and the length and phrase-count controls are single-seed. Please report those controls with multiple seeds. A blinded human check of, say, 200 kept and 200 dropped captions, judging whether the content words are actually supported by the image, would test the mechanism directly; please say whether such a check exists or will be added.

  5. The CE-CLIP gain is smaller than its seed spread over only two seeds. Please give per-seed values and a third seed if available. For NegCLIP, please state which control the plug-in claim should be read against, since the gain over the full dataset and the gain over the strongest data control differ substantially.

  6. All runs start from a checkpoint trained on the full corpus, so pairs that are later dropped were seen for far more steps than the second phase. How much of the relation gain remains if the second phase starts from a much shorter pretraining run, or if dropped pairs are also excluded from pretraining?

  7. The SigLIP replication compares the selected subset with alignment-only filtering but not with a random subset. Please add that control, since the CLIP results show this is the comparison that decides the claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.