SimFlow: Learning Similarity Flows for Noisy Relations in Contrastive Learning

Cross-modal contrastive learning on web-collected data suffers from noisy relations: mismatched image-text pairs are often treated as positives, whereas unpaired but semantically related pairs are treated as negatives. Existing methods reduce the influence of noisy relations, potentially discarding useful information about image-text relationships. We introduce SimFlow, which provides the required similarity for noisy relations by reformulating noisy correspondence correction as a trajectory learning problem. We first identify clean and noisy relations and train an auxiliary model on the clean ones to obtain matching trajectories from clean samples resembling those in the noisy relations. To derive target similarities for noisy relations, we introduce a velocity field supervised by these trajectories to predict similarity changes based on image-text representations and detected noise types. As the standard contrastive objective attracts false positives and repels false negatives, we reduce their contributions to avoid conflicts with similarity correction. We further preserve the similarity structure of semantically matched relations across modalities through structural regularization. Extensive experiments show that SimFlow outperforms existing methods on image-text retrieval tasks under synthetic and real-world noise on Flickr30K, MS-COCO, and CC140K, and achieves strong cross-domain transferability across diverse classification datasets.