Aligned Is Not Supported: Attributing Image–Text Alignment to Caption Phrases for Data Selection

Image–text filters for CLIP-style pretraining score each pair with one number. We show that this number saturates: on CC3M, once the least-aligned 30% of pairs are removed, stricter alignment thresholds stop improving compositional benchmarks and begin to reduce zero-shot transfer. The reason is structural. A global score measures whether a pair matches, not whether the caption’s phrases carry that match; a caption can be well aligned yet hollow, sustained by one salient object, an OCR fragment, or a text prior. We introduce Counterfactual Phrase Intervention (CPI), which attributes the alignment score back to the caption: each nominal head is replaced by a nonce token matched in CLIP-BPE length and surface form, and the resulting similarity drop measures how much of the match that phrase carries. In other words, CPI asks not whether a caption matches its image, but which words in it do the matching. Mean phrase sensitivity correlates only weakly with alignment (r = 0.084), and rankings induced by noun-phrase and relational contexts are nearly independent (ρ = 0.059), so the signal adds information that no global score can recover. Ranking the aligned pool by this signal yields a 50%-data subset that improves VL-CheckList Relation by +1.73 over full data and +0.98 over alignment-only filtering at matched budget (five paired seeds, p = 0.002), is within seed variance of the 70% alignment pool on compositional metrics with 28.7% fewer samples, and preserves zero-shot and retrieval transfer. Used without any change to the objective, the same subset improves NegCLIP by +3.84 and CE-CLIP by +4.92 on VL-CheckList Relation, and the effect replicates on CC12M, on ViT-S/32 and ViT-L/14, and under a SigLIP scorer (80.6% selection overlap against 71.3% at chance). We present these results as controlled evidence at CC3M–CC12M scale; curation costs 1.70 GPU-hours with cached embeddings, and code is provided.