acceptodds
Under review as a conference paper at ICLR 2027

SMOOTHALIGN: PROBABILISTIC SMOOTHING FOR MULTIMODAL REPRESENTATION LEARNING AND ALIGNMENT

Abstract

Multimodal representation learning and alignment are fundamental to cross-modal understanding and retrieval. However, common multimodal alignment objectives represent samples as deterministic points and assign every unpaired observation the same negative status: semantically related but identity-different samples are repelled as strongly as unrelated ones. We identify this structural failure as the semantic overlap bias. To correct this bias, we propose a probabilistic method, SmoothAlign, which uses a confidence-calibrated distributional prior derived from Gaussian semantic clouds of intermediate encoder representations to assign different repulsion strengths to unpaired samples according to their semantic overlap. We further prove that SmoothAlign admits a conservative mutual-information lower bound. SmoothAlign consistently outperforms the corresponding baselines in both video retrieval and image–text retrieval. A controlled IEMOCAP diagnostic confirms that the correction assigns stronger softening to semantically overlapping non-pairs than to ordinary negatives. The code is available at https://anonymous.4open.science/r/SmoothAlign-4EF0/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal representation learning, multimodal alignment, contrastive learning, semantic overlap bias, cross-modal retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kfmxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes SmoothAlign, a term that regulates contrastive loss functions to take care of the semantic similarity of hard negatives.

Critical strengths

The paper obtains promising results in the retrieval task.

The method to compute the weight is simple and interesting.

Critical weaknesses

Weaknesses 1&2:

The paper lacks a discussion on works strictly related to the method itself. Indeed, the problem faced by this paper is a well-known problem in the literature, and there already exist several works that address it. Examples are (among many Others):

Jiang et al., “Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation,” ACL 2023

Gao et al., “SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger,” AAAI 2024

Wang et al., “Sample-Specific Debiasing for Better Image-Text Models,” Machine Learning for Healthcare Conference 2023

Byun et al., “MAFA: Managing False Negatives for Vision-Language Pre-training,” CVPR 2024

Nevertheless, the paper does not compare the proposed method with the relevant literature and this leads to two major concerns:

  1. The paper novelty is limited, as the problem they name already exists in the literature. The contribution is then reframed as another, among the existing ones, method for better modeling the negatives in contrastive learning.

  2. it is hard to understand whether the proposed methods is more effective than the existing ones, since authors do not run appropriate comparisons.

Weakness 3:

The notation should be lighter, and \hat{r}_ij appears eq.4 but it is not defined and never appears again in the paper. I imagine that \hat{r}_ij becomes \tilde{P}_ij later in the paper, but this is not stated. Authors should carefully check the definitions and the equations and considerably make it lighter.

Weakness 4:

The paper lacks more detailed analysis. While the method seems effective for retrieval, very few clues are provided on the learned latent space. How similar objects are mapped in the learned space? Are similar instances placed closer than in InfoNCE or GRAM latent spaces? Also, it is not clear why retrieval should improve. Intuitively, smoothalign should place similar object closer than contrastive methods based on infoNCE or GRAM, but this may make retrieval harder: retrieval is based on instance rankings, why placing all the similar instances close improve retrieval?

Weakness 5:

Why applying the smoothing to L_dam too? L_dam is computed from a multimodal encoder that reranks the pairs trhough attention mechanisms, why the authors apply smoothaling to it? I don't get the reason behind it.

Questions for authors

N/A

Reviewer_tFuCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multimodal representation learning, and focuses on solving the semanic overlap bias: all unpaired samples are treated as negatives, causing excessive repulsion between semantically related samples. It proposes the SmoothAlign, using a diagonal Gaussian semantic cloud to represent the sample and measures the cross-modal affinity with the KL divergence between the the Gaussian semantic clouds. The proposed method is compared to GRAM and other retrieval methods on multimodal text-to-video (T2V) and video-to-text (V2T) retrieval, and compared to InfoNCE, PCME++ on ECCV Caption, CxC, and COCO Caption, with slight improvement.

Critical strengths
  1. The paper addresses a valid problem in multimodal retrieval: treating all unpaired samples as negatives can cause excessive repulsion between semantically related samples.

  2. The paper covers comparison on different cross-modal retrieval tasks, including multimodal text-to-video (T2V) and video-to-text (V2T) retrieval, image–text matching methods.

Critical weaknesses
  1. The retrieval improvements are marginal, and their statistical support is limited. The gains over GRAM, InfoNCE, and PCME++ in Tables 1 and 2 are modest, with most improvements in the individual retrieval metrics below one percentage point. Although the paper reports matched comparison settings, seed variation is provided only for SmoothAlign on MSR-VTT, without corresponding baseline variation. It is therefore difficult to determine whether the gains exceed run-to-run variation or represent a substantial improvement in multimodal retrieval.

  2. The computational overhead and practical benefit are unclear. Constructing Gaussian semantic clouds and computing their pairwise affinities introduce additional computation compared with simpler baseline methods. However, the paper does not analyze computational complexity or report computational costs relative to the baselines. Without these comparisons, it is difficult to assess whether the modest retrieval gains justify the additional cost in practice.

  3. The novelty and improvements over existing false-negative correction methods are insufficiently established and discussed. SmoothAlign reduces the repulsion of semantically related unpaired samples using affinities between Gaussian semantic clouds. However, related approaches have already addressed false negatives and relaxed one-to-one alignment, and the following works are absent from the references:

a. Debiased Contrastive Learning, NeurIPS 2020

b. CrossCLR: Cross-modal Contrastive Learning for Multi-modal Video Representations, ICCV 2021 

c. SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger, AAAI 2024 

It is suggested to explain how the proposed method differs from prior works that solve the similar issue.

  1. The interpretation and numerical stability of the Gaussian semantic clouds need further justification. Their mean and variance are computed across the modality embeddings of each individual sample. For image–text matching, this uses only two embeddings, so the variance directly reflects their disagreement. The paper should explain why these statistics reliably estimate semantic overlap between different samples and how near-zero variances are handled in the KL computation.
Questions for authors
  1. Why are different baseline methods used for different tasks—for example, GRAM for text-to-video (T2V) and video-to-text (V2T) retrieval, and InfoNCE and PCME++ for image–text matching? Could the authors explain this choice and evaluate the same set of baseline methods across tasks where applicable?

  2. Are the improvements over the baseline methods statistically significant? Can you report mean ± std over ≥3 seeds for both baseline and SmoothAlign on all datasets, together with a paired significance test?

  3. What is the computational complexity of SmoothAlign, and what additional computational cost does it introduce? Could the authors provide comparisons with the corresponding baselines, including training time and memory usage?

  4. Figure 5 shows increased affinity between semantically related samples compared with GRAM. Could the authors also provide concrete retrieval examples affected by semantic overlap bias where SmoothAlign succeeds and the corresponding baseline fails? Showing the queries and retrieved results would help explain how the affinity changes translate into improved retrieval, including for the InfoNCE and PCME++ comparisons.

  5. What value is used for the minimum negative weight wminw_{min} in Eq. (15)? How sensitive are the results to the hyperparameters wminw_{min}, the prior temperature τprior\tau_{prior} in Eq. (10), the confidence temperature τconf\tau_{conf} in Eq. (11), and the confidence exponent γ\gamma in Eq. (13)? How were these hyperparameters selected, and was a held-out validation split used?

  6. Which intermediate layer is used for each encoder to construct the Gaussian semantic clouds in Eqs. (7)–(8), and how are the features projected into the shared dd-dimensional space? Are the projection heads reused from the baseline, and are they updated during training? Please clarify how this implementation supports the claim of "no additional trainable parameters."

Reviewer_mogxOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper claims that contrastive objectives treat all unpaired samples as equal negatives and so over-repel semantically related ones, which it calls the “semantic overlap bias.” To this end, this work introduces SmoothAlign, which builds Gaussian semantic clouds from intermediate encoder representations and down-weights the repulsion of negatives according to how much their clouds overlap. SmoothAlign also proves a conservative mutual-information lower bound. Exstenstive experiments demonstrate that SmoothAlign could be applied to different methods, such as GRAM, InfoNCE and PCME++, on both video and image–text retrieval tasks.

Critical strengths

This work aims to address false-negative problem in multimodal representation learning, the problem is interesting and wellmotivated.

The proposed method SmoothAlign exhibits great generalization capacity, it can be applied to GRAM, InfoNCE and PCME++ with minimal changes. This makes it easy to adopt and to combine with future alignment objectives.

The authors conducted extensive experimental evaluation. The paper covers four video retrieval benchmarks (MSR-VTT, DiDeMo, ActivityNet, VATEX) under three modality configurations (T-V, T-VA, T-VAS), plus three image–text benchmarks (ECCV Caption, CxC, COCO) on two base losses. Across more than 20 comparable settings, SmoothAlign matches or improves its base method in nearly every case with almost no regressions, which suggests the effect is real even if small.

Critical weaknesses

Although this work is quite interesting, I have few concerns:

(1) I found limited novelty in this work and there are several missing key related work. The “semantic overlap bias” is essentially the well-known false-negative (class-collision) problem in contrastive learning, yet the authors do not discuss or compare against prior work that directly addresses it, including debiased contrastive learning [1], false-negative cancellation [2], momentum-distillation soft targets in ALBEF [3], SoftCLIP [4] and the analysis of temperature as tolerance to similar samples [5]; a tuned-temperature baseline is also missing.

(2) The gains are small, and it’s unclear whether they’re real. As shown in Table 1, most R@1 improvements over GRAM are under one point; on MSR-VTT, 0.1 point is a single video. Only one run is reported, and the GRAM numbers seem to be copied from the original paper rather than re-run. Please re-run the baselines in the same setup and report the mean and standard deviation over several seeds.

(3) The ablations don’t show that the probabilistic part is needed. In Table 3, on T-VAS, simply comparing the means with cosine similarity gives the same result as the full Gaussian method (65.5/65.7 vs. 65.4/65.7), so what does the distributional modeling add? The results are also fragile: on T-V V2T, five of the six alternative variants fall below the GRAM baseline. Smoothing only the GRAM loss works about as well as the full method, so the DAM part adds little. These choices also appear to have been tuned on the MSR-VTT test set.

[1] C.-Y. Chuang, J. Robinson, L. Yen-Chen, A. Torralba, S. Jegelka. Debiased Contrastive Learning. NeurIPS, 2020. [2] T. Huynh, S. Kornblith, M. R. Walter, M. Maire, M. Khademi. Boosting Contrastive Self-Supervised Learning with False Negative Cancellation. WACV, 2022. [3] J. Li, R. R. Selvaraju, A. D. Gotmare, S. Joty, C. Xiong, S. C. H. Hoi. Align before Fuse: Vision and Language Representation Learning with Momentum Distillation. NeurIPS, 2021. [4] Y. Gao, J. Liu, Z. Xu, T. Wu, E. Zhang, K. Li, J. Yang, W. Liu, X. Sun. SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger. AAAI, 2024. [5] F. Wang, H. Liu. Understanding the Behaviour of Contrastive Loss. CVPR, 2021.

Questions for authors

The prior comes from the model’s own intermediate features and may be unreliable and self-reinforcing early in training. Is there a warmup? Why intermediate layers rather than the final ones?

Please report the computational overhead of building the Gaussian clouds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.