acceptodds
Under review as a conference paper at ICLR 2027

SimFlow: Learning Similarity Flows for Noisy Relations in Contrastive Learning

Abstract

Cross-modal contrastive learning on web-collected data suffers from noisy relations: mismatched image-text pairs are often treated as positives, whereas unpaired but semantically related pairs are treated as negatives. Existing methods reduce the influence of noisy relations, potentially discarding useful information about image-text relationships. We introduce SimFlow, which provides the required similarity for noisy relations by reformulating noisy correspondence correction as a trajectory learning problem. We first identify clean and noisy relations and train an auxiliary model on the clean ones to obtain matching trajectories from clean samples resembling those in the noisy relations. To derive target similarities for noisy relations, we introduce a velocity field supervised by these trajectories to predict similarity changes based on image-text representations and detected noise types. As the standard contrastive objective attracts false positives and repels false negatives, we reduce their contributions to avoid conflicts with similarity correction. We further preserve the similarity structure of semantically matched relations across modalities through structural regularization. Extensive experiments show that SimFlow outperforms existing methods on image-text retrieval tasks under synthetic and real-world noise on Flickr30K, MS-COCO, and CC140K, and achieves strong cross-domain transferability across diverse classification datasets.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Cross-modal matching, Contrastive learning, Complementary learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eg5NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SimFlow to correct similarity for mismatched image-text pairs. In particular, SimFlow identifies noisy and clean relations via a GMM model and obtains target trajectories. It further introduces a flow matching mechanism to predict the similarity of noisy pairs, which is used to guide the learning of image and text encoders. Experimental results show that the proposed method outperforms existing methods on image-text retrieval tasks.

Critical strengths
  1. This paper presents an interesting perspective, which formulates noisy correspondence correction as a trajectory learning problem.

  2. The experiments are comprehensive.

  3. This paper is easy to follow with clear writing.

Critical weaknesses
  1. It is not clear why learning a velocity field. The similarities of reference trajectories are already available during noisy relation discovery. Why not directly use them to adapt the noisy pairs?

  2. Fig. 4(b) shows that the performances between two points and ten points are similar. Does this suggest that the intermediate trajectory may contribute little?

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer eg5N - Comment#1

We thank the reviewer for the thoughtful feedback and for recognizing our approach as an "interesting perspective" that formulates noisy correspondence correction as a trajectory learning problem. We also appreciate the acknowledgement that our "experiments are comprehensive" and that the paper is "easy to follow with clear writing". We appreciate these encouraging comments and will address all remaining concerns in the response below. All clarifications and supplementary analyses from our response are included in the revised manuscript and its Appendix.

1. Directly Using Reference Trajectories.

  • We evaluated the mentioned alternative as the Direct Trajectory variant in Figure 4(b) of Appendix B. This variant uses the reference trajectory of each noisy relation as its correction target: at the current adaptation progress, the target is obtained by linearly interpolating the two nearest recorded similarities of the reference relation. On Flickr30K at 20% noise, this variant is lower than SimFlow by 3.4 to 3.9 RSum for K∈{2,5,10,20}K \in \{2, 5, 10, 20\}.

  • The performance drops because the reference relations cannot fully represent the noisy relations. Although each reference relation resembles its noisy relation, the reference relations are constructed from a limited set of clean samples and cannot cover all the noisy relations newly detected during adaptation. As a result, the reference similarities start from a different value and do not reflect the content of each noisy relation. In contrast, the velocity field starts from the initial similarity of the noisy relation and is conditioned on its own image-text representations and noise type, which provides a target specific to that relation.

  • For additional comparisons with other simpler alternatives, i.e., target similarities directly computed by the auxiliary encoders and direct similarity regression with the same MLP as the velocity field, please refer to our response to Reviewer 2f4y (2-1. Comparison with Simpler Options for Obtaining Target Similarities). The results show that the proposed method consistently outperforms both alternatives, indicating the benefit of deriving target similarities along a coherent correction trajectory, rather than from fixed or independently predicted targets.

AuthorsOpenReview ↗

Official Response to Reviewer eg5N - Comment#2

2. Number of Reference Trajectory Points.

  • The reference trajectory specifies the direction and magnitude of the similarity adjustment required for each noisy relation as adaptation proceeds. As shown in Figure 1 in the main paper, the direction and magnitude of the change differ across relations, yet the matching trajectory of each relation progresses steadily toward its final similarity without severe fluctuations. Therefore, a small number of recorded points is sufficient to supervise the velocity field, leading to similar retrieval performance at K=2K=2 and K=10K=10.

  • We expect intermediate points to become more important when similarities fluctuate more during training, for example, when adapting to distant domains with high visual overlap (such as medical image-report retrieval), or during continual adaptation across shifting data distributions. We consider the evaluation in such settings a promising direction for future work.

Thanks for the authors’ responses. I think the authors have solved most of my concerns, so I will maintain my rating.

Reviewer_Yo53OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

To address noisy relations in cross-modal contrastive learning, this paper proposes SimFlow, a framework that reformulates noisy correspondence correction as a matching-trajectory learning problem to preserve semantic information while accommodating relation-specific similarity adjustments. These adjustments are jointly optimized with the backbone encoders alongside a noise-aware contrastive loss that down-weights noisy updates and a structural regularization loss that preserves intra-modal semantic structures.

Critical strengths

The paper presents an inspiring perspective by concurrently addressing both false-positive and false-negative relations. Based on the insightful observation that the magnitude of similarity evolution varies considerably across relations with differing degrees of semantic correspondence during adaptation (Figure 1), the authors reasonably and solidly motivate the continuous trajectory learning framework to derive target similarity supervision tailored to individual noisy relations.

Critical weaknesses

W1: Applying flow matching to one-dimensional scalar linear interpolation appears to be severe over-engineering that lacks core ablation justification. Moreover, given that the correction path is constructed via a deterministic linear combination zm(tk)z_m(t_k) with piecewise constant velocity um(t)u_m(t) derived from simple linear interpolation, it remains conceptually unjustified why an auxiliary velocity field vϕv_\phi must be trained to predict derivatives and integrated via Euler approximation (Eq. 4) at every epoch to recover a scalar, rather than directly regressing the target similarity s^m(tn)\hat{s}_m(t_n) using the exact same three-layer MLP. Finally, the ablation in Figure 4(b) merely compares SimFlow against Direct Trajectory, thereby conflating the incorporation of relation context cmc_m with the necessity of the velocity field itself, rendering the claimed 3.4–3.9 RSum gain uninformative regarding the true benefit of flow matching.

W2: There exists a severe representation covariate shift between the velocity field training and the flow-guided adaptation phases. According to Appendix A.1, the image and text representations in the relation context cmc_m are extracted by the auxiliary encoders during velocity field training. However, during the subsequent adaptation epochs, the auxiliary encoders are discarded and vϕv_\phi remains frozen, while the context cmc_m is dynamically constructed at each epoch using representations produced by the continuously updated backbone encoders Fv\mathcal{F}_v and Ft\mathcal{F}_t. The authors merely assume compatibility based on identical architecture and initialization. Consequently, the evolving feature space of the backbone encoders inevitably diverges from that of the auxiliary encoders, introducing unmanaged distribution drift at the input of the frozen velocity field.

W3: Section 3.2.1 states that reference relations are constructed by retrieving clean counterparts via the auxiliary encoders, while simultaneously asserting that their similarity trajectories {smref(tk)}k=0K−1\{s_m^{\text{ref}}(t_k)\}_{k=0}^{K-1} are tracked and recorded during the training of these very auxiliary encoders. This creates an unaddressed causal dilemma regarding how the reference relations could be retrieved and tracked from t=0t=0 before the auxiliary encoders are even trained. Second, while Appendix A.1 specifies that clean and noisy relations are dynamically re-identified at the beginning of each of the eight adaptation epochs, the integral for deriving the target similarity in Eq. (4) strictly relies on the initial similarity sm(0)s_m(0), leaving it entirely ambiguous how sm(0)s_m(0) is defined or retrieved if an image-text pair is newly detected as noisy only at an intermediate epoch (e.g., epoch 5). Third, candidate FN relations are ephemeral, non-diagonal image-text combinations (Xi,Yj)(X_i, Y_j) (i≠ji \neq j) mined exclusively via mutual nearest neighbors within individual mini-batches of size B=512B=512. Consequently, due to standard epoch-wise data shuffling, an off-diagonal pair formed in one epoch will almost certainly not co-occur in the same batch in subsequent epochs, making it practically impossible to maintain a persistent initial similarity sm(0)s_m(0) or track a continuous evolution trajectory across adaptation epochs for these fleeting relations.

W4: The greedy low-similarity batch construction during auxiliary encoder training distorts the reference trajectory distribution and overlooks computational feasibility. According to Section 3.2.1, in order to reduce false negatives during the training of the auxiliary encoders Fvaux\mathcal{F}_v^{\text{aux}} and Ftaux\mathcal{F}_t^{\text{aux}}, each batch is constructed greedily by selecting relations whose images and texts exhibit low similarity to those already chosen. While this heuristic minimizes the incidence of false negatives, it artificially eliminates informative hard negative pairs within each batch. In cross-modal contrastive learning, the absence of hard negatives depresses the repulsive force between unmatched samples, which inevitably inflates cosine similarities across unpaired instances and introduces a systematic upward bias into the collected reference trajectories {smref(tk)}\{s_m^{\text{ref}}(t_k)\}. Furthermore, the authors provide no explanation or empirical analysis regarding the computational complexity and scalability of executing such greedy similarity-based batch selection on large-scale datasets.

W5: The total objective in Eq. (8) exhibits a severe loss weighting imbalance with λflow=5\lambda_{\text{flow}}=5 and λstruct=25\lambda_{\text{struct}}=25. Yet, neither the main text nor the appendix provides sensitivity analyses for either λflow\lambda_{\text{flow}} or λstruct\lambda_{\text{struct}}. Furthermore, the manuscript only reports an isolated adaptation speed on the smallest dataset (Flickr30K) that omits the prerequisite overhead of auxiliary encoder training, nearest-neighbor retrieval, and velocity field optimization. This fundamentally masks the true end-to-end wall-clock training time required for full datasets such as MS-COCO and CC140K, where repeated GMM re-fitting and Euler integrations are executed.

Questions for authors

Please provide detailed clarifications on the temporal execution and state tracking of the framework. First, clarify whether the reference relation retrieval is conducted before training the auxiliary encoders or after their three-epoch training is completed. If the latter is true, explain how the reference trajectory {smref(tk)}k=0K−1\{s_m^{\text{ref}}(t_k)\}_{k=0}^{K-1} across normalized training progress tkt_k within the unit interval can be tracked and recorded during the auxiliary training process without retroactively computing historical representations. Second, given that clean and noisy relations are re-identified at the beginning of each of the eight adaptation epochs under randomly shuffled batches, explain how the initial similarity sm(0)s_m(0) in Equation 4 is formally defined, retrieved, and stored for a relation that is identified as noisy for the first time at an intermediate epoch e>1e > 1.

6 comments
AuthorsOpenReview ↗

Official Response to Reviewer Yo53 - Comment#1

We thank the reviewer for the detailed feedback and for recognizing that our paper presents an “inspiring perspective” based on an “insightful observation.” We carefully address all identified weaknesses and questions below. All clarifications and supplementary analyses from our response are included in the revised manuscript and its Appendix.

1. Ablation of Flow Matching.

  • Following the reviewer’s suggestion, we compared SimFlow with the direct-regression baseline, a regressor with the same MLP architecture as our velocity field, trained to directly predict the target similarity at each progress point. The table below reports RSum of the two methods on Flickr30K and MS-COCO at 20% and 80% noise ratios. SimFlow outperforms this baseline by 3.2 and 4.4 RSum on Flickr30K, and by 3.8 and 4.7 RSum on MS-COCO, respectively. For a comparison with another baseline that uses the auxiliary encoder similarities directly as targets, please refer to our response to Reviewer 2f4y (2-1. Comparison with Simpler Options for Obtaining Target Similarities).
MethodFlickr30KMS-COCO
20% noise80% noise20% noise80% noise
Direct regression549.9545.4534.0529.1
SimFlow (Ours)553.1549.8537.8533.8
  • We attribute these gains to the continuous trajectory along which SimFlow obtains the target similarity. Direct regression predicts the target value at each progress point in a single step, and its output at one point does not depend on the other points. In contrast, SimFlow learns a velocity field defined on the current similarity state and the progress, conditioned on the image-text representations and the noise type, to predict the instantaneous direction and rate of similarity change. The target at the current progress tn is then obtained by numerically integrating this velocity up to tn, so the target evolves continuously along the progress rather than being predicted separately at each point. Since each change depends on the current similarity state, the target accumulates the predicted changes over all earlier points and forms a continuous trajectory by construction.

  • Moreover, this improvement comes at negligible computational cost. As detailed in Appendix B (Computational Overhead), the velocity field comprises only 0.09M parameters, requires 0.8 minutes of training time, and is discarded at inference. The detailed wall-clock time is also minimal, as reported in 5. Hyperparameter Sensitivity and Wall-clock Time below. These consistent empirical gains at this cost demonstrate that the velocity field is a lightweight and practical component rather than unnecessary complexity.

AuthorsOpenReview ↗

Official Response to Reviewer Yo53 - Comment#2

2. Representation Alignment.

  • Both the auxiliary and main encoders adopt an identical architecture, start from the same pretrained CLIP weights, and learn from samples of the same dataset. Prior studies show that models fine-tuned from the same initialization tend to stay in similar loss basins and learn similar features [B-1, B-2]. We therefore consider their feature spaces to remain closely aligned during adaptation. To verify this, we examine the representation drift on Flickr30K and its impact on the final retrieval performance.

  • We quantify the similarity between the representations produced by the auxiliary and main encoders for both images and texts, focusing on detected false-positive and false-negative relations. We report two metrics: (i) the average cosine similarity between sample representations (measuring per-sample alignment), and (ii) linear CKA (measuring the structural similarity of the feature spaces).

  • As a reference, we compare the feature alignment between the auxiliary and main encoders against that between the pretrained CLIP and the main encoders. The table below reports the results at the final adaptation checkpoint, where the representation drift is expected to be largest. The auxiliary and main encoders remain closer to each other than the main encoders are to the pretrained CLIP, for both modalities and noise types. This observation is consistent with prior findings that models fine-tuned from the same initialization remain close in feature space [B-1, B-2].

MetricCompared with main encoders20% noise80% noise
FPFNFPFN
Linear CKA (image)Auxiliary0.8560.8750.8360.849
Pretrained0.6560.7140.6390.681
Linear CKA (text)Auxiliary0.8480.8810.8700.881
Pretrained0.6600.7170.6580.687
Cosine similarity (image)Auxiliary0.8510.8690.8140.822
Pretrained0.8220.8310.8110.820
Cosine similarity (text)Auxiliary0.8850.8950.8830.888
Pretrained0.8610.8690.8490.857
  • Impact on retrieval performance: We further evaluated a variant of SimFlow in which cmc_m is constructed with the auxiliary encoders throughout adaptation. This variant yields 552.8 and 548.6 RSum at 20% and 80% noise, compared to 553.1 and 549.8 RSum for SimFlow. This marginal difference is consistent with the close feature alignment between the auxiliary and main encoders shown above. We therefore construct cmc_m with the main encoders, which preserves the retrieval performance while avoiding the memory overhead of retaining an additional 151.3M parameters for extracting cmc_m from the auxiliary encoders during adaptation.

References.

[B-1] Neyshabur et al., What is being transferred in transfer learning?, NeurIPS, 2020.

[B-2] Wortsman et al., Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, ICML, 2022.

AuthorsOpenReview ↗

Official Response to Reviewer Yo53 - Comment#3

3. Obtaining the Initial Similarity (and Question 1).

  • The reference relations are retrieved prior to training the auxiliary encoders, using their initial pretrained CLIP weights. Once retrieved, these relations remain fixed, and their pairwise similarities are recorded as the initial state at t=0t=0. During the subsequent training of the auxiliary encoders, we record the similarities of these fixed references at the remaining K−1K-1 steps.

  • Initial similarity of relations detected at intermediate epochs (and Question 2):

    • The initial similarity sm(0)s_m(0) of a relation mm is defined as the cosine similarity between its image and text embeddings, computed using the pretrained CLIP encoders before training. To make this value accessible across intermediate epochs without extra forward passes, we pre-compute and store the normalized image and text embeddings of all training samples prior to training. Whenever any relation (paired or unpaired) is newly detected as noisy during adaptation, sm(0)s_m(0) is obtained immediately via the dot product of the cached embeddings.

    • This caching strategy removes the need to maintain or execute an additional pretrained CLIP model during training. Furthermore, the storage overhead for these cached embeddings is small: 6.5% of the original data on average for Flickr30K, MS-COCO, and CC140K.

  • Ephemeral false-negative relations:

    • We acknowledge that false-negative relations depend on the composition of each batch, and that the same false-negative relation is unlikely to reappear across consecutive epochs. However, SimFlow does not require tracking individual relation histories across epochs.

    • Initial Similarity is Persistent: As discussed above, the initial similarity sm(0)s_m(0) of a given noisy relation mm is simply computed by using the embeddings stored before training. When a false-negative (or false-positive) relation is newly detected, we obtain its initial similarity, even if the pair first appears in an intermediate epoch.

    • Target Acquisition: To generate the target for correcting a given noisy relation, SimFlow does not track how the similarity of that relation has changed over the preceding epochs. Instead, starting from the initial similarity sm(0)s_m(0) of a detected noisy relation, the velocity field iteratively predicts the change in its similarity, and these changes are accumulated up to the current progress tnt_n to yield the target s^m(tn)\hat{s}_m(t_n). Since this accumulation is carried out entirely at the current step, SimFlow computes the target on the fly without any previously recorded similarity of the same relation. As a result, whether a pair co-occurred in earlier epochs is irrelevant to the target of a relation detected in the current epoch.

Reviewer_2f4yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SimFlow targets noisy correspondence in CLIP-style fine-tuning, covering both mismatched pairs (false positives) and semantically matched non-pairs (false negatives). It first detects these relations with GMMs on contrastive losses and trains an auxiliary CLIP on the detected clean set. For each noisy relation, it builds a reference pair from the nearest clean image and nearest clean text and records that pair's similarity trajectory during auxiliary training. A small MLP velocity field is then fit with flow matching on paths from the noisy relation's initial similarity toward this reference trajectory. During fine-tuning, the main encoders are trained with an MSE that pulls noisy-relation similarities toward the integrated target, together with a contrastive loss down-weighted by noise probability and an intra-modal structure regularizer. The paper reports the best RSum on Flickr30K and MS-COCO under 20 to 80% synthetic noise and on CC140K, as well as gains in zero-shot and linear-probe transfer.

Critical strengths
  • The idea of giving each noisy relation a graded target similarity, instead of discarding or relabeling it, is reasonable. Fig. 3 shows targets that differ within the same noise type in a sensible way.

  • The empirical coverage is wide. The paper uses three retrieval datasets with four noise levels, four extra backbones (Tab. 5), transfer to 10 classification datasets, and a cost analysis. SimFlow is also cheaper to train than the co-teaching style baselines (Fig. 5).

Critical weaknesses
  • The basic reference points are missing, so the size of the real gain is unclear. The tables never report zero-shot CLIP, plain CLIP fine-tuning on the noisy data (CLIP-FT appears only in Fig. 3), or fine-tuning on the detected clean subset only. This matters most for transfer. From memory, published zero-shot CLIP ViT-B/32 accuracy on these 10 datasets averages roughly 60. SimFlow gets 60.9 and every baseline is below it. If that holds, the "strong cross-domain transferability" result says SimFlow forgets less of the pretrained model, not that it learns better representations from noisy data. The retrieval results look similar: performance at 80% noise is nearly the same as at 20% (549.8 vs 553.1). That suggests the pretrained backbone does most of the work and synthetic shuffled captions are easy for CLIP to detect.

  • The flow matching component is not shown to be needed. The velocity field fits a deterministic 1D path for a scalar similarity, so no distribution is being transported. The target is in effect "what a model trained on clean data would assign to a similar pair." Two simpler options are not tested: (a) Use the auxiliary encoders' own similarity on the noisy pair as the target (plain distillation from a clean-trained model). (b) Regress s_ref(t) directly from (s(0), c_m, t) with the same MLP. The only comparison is against raw interpolation of the reference trajectory (−3.4 to −3.9 RSum), which is too weak a baseline. There are also mismatches built into the design: the field is trained on auxiliary-encoder features but queried with main-encoder features, and the 3-epoch auxiliary timeline is mapped onto 8 adaptation epochs.

Questions for authors
  • Please add these rows to Tab. 1–3: zero-shot CLIP, CLIP-FT on the noisy data, and CLIP-FT on the detected clean subset. If zero-shot CLIP matches SimFlow on Tab. 3, the transfer claim should be dropped.

  • Please compare against the two simpler targets above: auxiliary-model similarity on the noisy pair, and direct MLP regression without flow integration. A small gap would remove the main technical contribution.

  • In the ablation, removing any single loss drops SimFlow below MSCN or SPS (e.g. 521.8 without L_flow at 20%). How were λ_struct = 25 and λ_flow = 5 chosen? Was a validation split used, or the test folds?

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer 2f4y - Comment#1

We thank the reviewer for the thoughtful feedback and for highlighting that “the idea is reasonable.” We also appreciate the recognition that our “empirical coverage is wide,” that our method is “cheaper to train than the co-teaching style baselines,” and that our presentation is “clear and easy to follow.” We are grateful for the reviewer’s constructive comments and questions, and we carefully address all identified weaknesses and concerns below. All clarifications and supplementary analyses from our response are included in the revised manuscript and its Appendix.

1. Basic Baselines (and Question 1).

  • We have added the three baselines suggested by the reviewer to Tables 1-3 in the revised manuscript: pretrained CLIP, CLIP fine-tuned on the full noisy training set (noisy), and CLIP fine- tuned on the detected clean subset (clean). The results are summarized below.
SettingCLIP (zero-shot)CLIP-FT (noisy)CLIP-FT (clean)SimFlow
Flickr30K noise ratio 20%504.2433.4456.5553.1
Flickr30K noise ratio 80%504.2393.3428.8549.8
MS-COCO noise ratio 20%473.8436.6456.5537.8
MS-COCO noise ratio 80%473.8401.2421.1533.8
CC140K456.4440.4442.4513.7
Cross-domain transfer60.626.727.560.9
  • Overall, pretrained CLIP and the two fine-tuned variants show unsatisfactory retrieval performance on Flickr30K, MS-COCO, and CC140K. In particular, CLIP-FT (noisy) and CLIP-FT (clean) perform worse than pretrained CLIP on all three datasets. Notably, CLIP-FT (clean) falls below zero-shot CLIP even though it is fine-tuned only on the detected clean relations. This indicates that fine-tuning with the detected clean relations is insufficient: the detection is not always reliable, and the information in the noisy relations is discarded.

  • Cross-domain Transfer: We would like to note that the cross-domain transfer results are intended to show that the representations learned from noisy data can generalize to unseen domains, rather than to show that SimFlow preserves the pretrained knowledge. Although SimFlow is trained on the noisy, web-collected dataset, CC140K, it achieves the highest average accuracy across ten unseen classification datasets (60.9%), slightly exceeding that of the pretrained CLIP (60.6%).

    • In contrast, both fine-tuning methods show unsatisfactory cross-domain transferability: CLIP-FT (noisy) and CLIP-FT (clean) yield 26.7% and 27.5%, respectively, and existing noisy corre-spondence baselines also show limited performance (46.5%-56.1%). These results show that the transferability of SimFlow results from how it learns from the noisy data, without any component designed to protect the pretrained knowledge.

    • To the best of our knowledge, this work is the first in image-text noisy correspondence learning to investigate cross-domain transferability to unseen domains and classes after adaptation on noisy data. We also appreciate the reviewer's constructive feedback on the wording of our contribution, and we will revise the manuscript accordingly.

  • Similar performance under different noise ratios: SimFlow maintains stable retrieval performance at both 20% and 80% noise, which highlights the robustness of the proposed method against severe noise. The pretrained backbone alone clearly cannot explain this stability. At 80% noise, zero-shot CLIP underperforms SimFlow by a large margin, 45.6 and 60.0 RSum on Flickr30K and MS-COCO, respectively, and the CLIP-FT variants perform even worse than the zero-shot baseline. These substantial gaps demonstrate that the correction of noisy relations, rather than the backbone, is the main contributor to the retrieval performance under severe noise.

AuthorsOpenReview ↗

Official Response to Reviewer 2f4y - Comment#2

2-1. Comparison with Simpler Options for Obtaining Target Similarities (and Question 2).

  • The key motivation behind our flow matching is to obtain each target along a coherent correction trajectory. Instead of moving directly to a static target or predicting independent targets at each epoch, SimFlow uses a representation-conditioned velocity field to predict how the similarity of each relation should evolve over time, and integrates this velocity from the initial similarity up to the current progress to obtain the target.

  • Following the reviewer’s suggestion, we compared SimFlow on Flickr30K and MS-COCO with the two simpler alternatives: (a) target similarities directly computed by the auxiliary encoders and (b) direct similarity regression using the same MLP architecture as the proposed velocity field. The table below summarizes the retrieval results at 20% and 80% noise. On MS-COCO, SimFlow outperforms (a) with performance gaps of 3.7 and 4.2 in terms of RSum, and (b) with gaps of 3.8 and 4.7, at 20% and 80% noise, respectively. These gains demonstrate that continuous, state-aware trajectories given by a velocity field provide more effective and coherent supervision than static or independently regressed targets.

MethodFlickr30K (20% noise)Flickr30K (80% noise)MS-COCO (20% noise)MS-COCO (80% noise)
Distillation (a)549.6544.2534.1529.6
Direct regression (b)549.9545.4534.0529.1
SimFlow (Ours)553.1549.8537.8533.8
  • Conceptually, SimFlow offers distinct advantages over these alternatives:

    • (a) Fixed final target via auxiliary encoders: The auxiliary encoders yield a single, static target similarity for each relation, regardless of the adaptation progress. Consequently, the main encoders are supervised with this terminal value from the beginning of adaptation. In contrast, SimFlow provides a dynamic target that aligns with the current adaptation phase.

    • (b) Direct regression of the similarity value: This approach predicts the similarity at each progress point as an independent point-wise target. While it can fit to sampled values, it lacks a mechanism that constrains how the similarity changes between steps, which can result in temporally inconsistent targets across progress points. However, SimFlow learns the required rate of change at each step and integrates it, yielding a target that remains coherent with the changes integrated up to the current progress.

2-2. Alignment Between Encoders.

  • We expect the auxiliary and main encoders to remain closely aligned in feature space because they share identical architectures, are both initialized from the same pretrained CLIP weights, and are trained on samples from the same dataset [A-1, A-2]. To empirically examine the effect of a potential feature mismatch, we constructed cm with the auxiliary encoders during adaptation. This variant yields 552.8 and 548.6 RSum at 20% and 80% noise on Flickr30K, compared to 553.1 and 549.8 RSum for SimFlow. Given this marginal difference, constructing cm with the main encoders preserves the performance while avoiding the memory burden of maintaining an additional 151.3M parameters. We also provide a detailed analysis of feature similarity in our response to Reviewer Yo53 (2. Representation Alignment).

  • Training length: The auxiliary encoders are trained on 5,000 relations sampled from the detected clean set. Because this subset is considerably smaller than the full training set used for the main encoders, the auxiliary encoders require a shorter training schedule. To align the two schedules, we use the normalized adaptation progress, which increases from 0 at the start to 1 at the end of each training phase. To verify that this alignment does not compromise the performance, we fix the adaptation epochs at 8 and vary the auxiliary epochs on Flickr30K:

Aux. epochs20% noise80% noise
3 (default)553.1549.8
5552.6549.2
8552.8549.7
  • The results show that with the adaptation epochs fixed, varying the auxiliary epochs from 3 to 8 changes RSum by at most 0.6. This indicates that the auxiliary schedule length has minimal impact on the final performance within the tested range, allowing the shorter schedule to effectively reduce the training overhead without compromising the results.

References.

[A-1] Neyshabur et al., What is being transferred in transfer learning?, NeurIPS, 2020.

[A-2] Wortsman et al., Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, ICML, 2022.

AuthorsOpenReview ↗

Official Response to Reviewer 2f4y - Comment#3

3. Response to Question: Hyperparameter Selection.

  • We selected the values of λflow and λstruct on the Flickr30K validation set and kept them fixed across all datasets without additional tuning. We further conducted a hyperparameter sensitivity experiment on these two weights. As shown in the newly added Figure 7 in Appendix B, SimFlow is robust to the choice of these weights, yielding consistent retrieval performance when either weight is varied over {1,5,10,20,30,40,50}\{1,5,10,20,30,40,50\} while the other is fixed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank all the reviewers for their thoughtful and constructive feedback. We appreciate that the reviewers recognize our approach as an “inspiring perspective” and an “interesting perspective” with a “reasonable and solid motivation”. We are also grateful for the positive remarks that our “empirical coverage is wide” and that our method is “cheaper to train than other baselines.” Finally, we thank Reviewers 2f4y and eg5N for highlighting the clarity of our presentation. We have carefully addressed all reviewer concerns in our response. The revised manuscript includes all clarifications, additional baselines, and supplementary analyses, with further details provided in the Appendix. We will make the code publicly available upon acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.