acceptodds
Under review as a conference paper at ICLR 2027

Theoretical Analysis of Multimodal Contrastive Learning: When and How Hard Negatives Help

Abstract

Multimodal contrastive learning has emerged as a powerful framework for learning visual and semantic representations, yet it often struggles to capture rare features. Hard negative sampling has been empirically shown to improve fine-grained recognition, but a rigorous understanding of when and how it helps remains limited. In this work, we develop a theoretical framework to analyze the training dynamics of multimodal contrastive learning with Transformer-based encoders. Our analysis reveals an implicit bias in vanilla contrastive learning toward common features, while rare features can remain weak or even be overlooked. Motivated by this finding, we show that hard negative samples can alleviate this imbalance by suppressing common features shared across negative pairs, thereby strengthening the learning of rare features. We further characterize when hard negative sampling becomes harmful, showing that excessive use of hard negatives can overly suppress common features and disrupt the balance between common and rare feature learning. Finally, we show that a simple selection strategy based on thresholding can mitigate this effect and enable more effective use of hard negative samples. Our main theoretical findings are verified by experiments on both synthetic data and standard benchmark datasets.

Primary area:
learning theory
Keywords:
Multimodal Contrastive Learning, Feature learning, Training dynamics, Theoretical analysis, Hard Negative Samples

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_x6GjOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper provides a theoretical analysis of multimodal contrastive learning with hard negative samples. The authors consider a sparse-coding multimodal data model containing common features with occurrence frequency ϵ1\epsilon_1 and rarer discriminative features with frequency ϵ2<ϵ1\epsilon_2<\epsilon_1, together with a simplified Transformer–MLP encoder and a sigmoid-based contrastive objective. The theoretical analysis develops four regimes. First, under random negative sampling, the model is shown to preferentially learn common features while rare features remain weak. Second, hard negatives that share common features with the anchor while differing in rare features induce a negative gradient along shared common-feature directions. The paper argues that this reduces the effective learning rate of common features from ϵ1\epsilon_1to$\left(1-\rho\right)\epsilon_1$, and that common and rare features can be learned at comparable strength when ρ=1−Θ(ϵ2/ϵ1)\rho=1-\Theta\left(\epsilon_2/\epsilon_1\right). Third, when the hard-negative effect is too strong, (1−ρ)ϵ1≪ϵ2\left(1-\rho\right)\epsilon_1\ll\epsilon_2, common features become over-suppressed and rare features dominate. Finally, the authors propose a similarity-gap thresholding rule intended to filter excessively influential hard negatives and restore balanced feature learning. The paper supports the theory with synthetic experiments examining feature-direction alignment, feature-subspace recovery, and downstream prediction, and with SigLIP2 fine-tuning results on ARO, VALSE, VSR, and FOIL-COCO.

Critical strengths
  1. The paper addresses a meaningful question that is insufficiently understood theoretically: why hard negatives can improve fine-grained multimodal representation learning and why overly aggressive hard-negative sampling may become harmful. The analysis is more ambitious than a static representation result, as it attempts to characterize feature-learning dynamics and separates common, rare, and noise directions.
  2. The synthetic experiments visualize the intended feature-learning phenomenon at a relatively fine level, including neuron-to-feature alignment, rare-feature subspace recovery, and downstream classification/regression. The real-data evaluation also reports consistent improvements from random negatives to hard negatives and then to thresholded hard negatives across the tested fine-grained benchmarks.
Critical weaknesses
  1. The central theoretical control variable ρ\rho is not defined consistently across the manuscript, and this ambiguity affects the interpretation of Theorems 3.2–3.4. In the main-text statement of Theorem 3.2, ρ\rho is introduced as the fraction of negative samples that are hard negatives. Elsewhere, including the Key Insights section and Appendix D, ρ\rho is described as the proportion of shared common features in hard-negative pairs. These are different quantities and generally need not coincide. This becomes particularly problematic under Assumption 3.5 / D.1. The paper assumes that for every common-feature coordinate p∈S1p\in S_1, the image and text members of a hard-negative pair share exactly the same latent signal, zi,p=zj,pz_{i,p}=z_{j,p}, while rare-feature coordinates remain independent. Under this formulation it is unclear what a continuously varying “proportion of shared common features” ρ\rho represents, since S1S_1 has already been defined as the shared common-feature subset. A precise probabilistic definition of ρ\rho and its relation to the mixture probability of hard versus random negatives is necessary.
  2. Although the paper repeatedly frames the contribution as a theoretical analysis of Transformer-based multimodal encoders, the actual optimization analysis appears to concern MLP neuron weights acting on unspecified self-attention representations. In the model definition, Attnr(X)\mathrm{Attn}_r(X) is introduced only as “the self-attention output at the rr-th token position.” In the appendix it is renamed hX(r)h_X^{(r)}, but I could not find a definition of the query/key/value parameterization, softmax attention, or a training-dynamics analysis of attention parameters. The gradients and feature-learning theorems are instead written with respect to the MLP neuron weights wkw_k and vkv_k. If the attention map is fixed, the theoretical result is more accurately a result for an MLP trained on Transformer-derived representations rather than a result about Transformer feature-learning dynamics. If the attention parameters are intended to be trainable, their optimization needs to be included in the analysis or the assumptions under which they can be treated as fixed should be stated explicitly.
Questions for authors

Q1. In Theorem 3.2, ρ\rho is described as the fraction of negative samples that are hard negatives, whereas the Key Insights section and Appendix D define ρ\rho as the proportion of shared common features in hard-negative pairs. Which definition is intended? If both quantities are required, please introduce separate symbols and rederive the common-feature gradient with the hard-negative mixture probability and within-hard-negative feature-sharing probability explicitly separated. Q2. Appendix Theorem E.2 assumes that after thresholding, ρ=1−Θ(ϵ2/ϵ1)\rho=1-\Theta(\epsilon_2/\epsilon_1), whereas the main-text Theorem 3.4 appears to state that applying the thresholding rule itself restores balanced learning. Can the authors derive a condition on ThT_h and the similarity distribution under which thresholding provably yields the required effective ρ\rho? Otherwise, I believe Theorem 3.4 should be restated as a conditional result. Q3. Please provide the explicit definition of Attnr(X)\mathrm{Attn}_r(X) and clarify which attention parameters are trainable.The current derivation appears to analyze gradients only with respect to the MLP neuron weights wkw_k and vkv_k. If attention is fixed, why should the result be interpreted as Transformer training dynamics rather than MLP feature learning on fixed attention representations? Q4. Please sweep the hard-negative fraction over a sufficiently wide range and report common-feature alignment, rare-feature alignment, and downstream performance. Does performance peak near the theoretically predicted balancing regime and decrease when (1−ρ)ϵ1≪ϵ2(1-\rho)\epsilon_1\ll\epsilon_2? Please additionally show whether thresholding recovers this degradation.

Reviewer_eNVUOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Hard negative sampling is widely used in CLIP/SigLIP system, but it really lacks theoretical explanation and guidance. In this paper, it is of clear value to study related theories from the specific and descriptive angle of "unbalanced learning of common/rare features". Through theoretical deduction, the author draws four main conclusions: ordinary comparative learning has implicit deviation, mainly learning common features, and the coefficient of rare features is only noise level; After adding hard negative samples, hard negative samples and positive samples share common features and adjust the effective learning rate of common features, so rare features are better learned; After adding a certain number of hard negative samples, common features will be over-suppressed, while rare features will dominate the representation; The proposed threshold strategy of screening by similarity interval can alleviate excessive inhibition and restore balance in theory. Around the relevant theories, the author has conducted many experiments on synthetic data to verify that ordinary contrast learning tends to common features and hard negative samples can improve the learning of rare features in the model. The author has also done fine-tuning experiments of SigLIP2 on real data such as ARO, VALSE, VSR and FOIL-COCO, and given the experimental results of random negative examples, hard negative examples and hard negative examples with thresholds.

Critical strengths
  1. In view of the lack of theoretical support, the author can study related theories from the specific and descriptive angle of "unbalanced learning of common/rare features", and provide research contributions for the theoretical research of hard negative sampling. Mechanism insight is correct and explanatory. Hard negative samples are useful because they share common features with positive samples, thus introducing negative gradients in the direction of common features, lowering their effective learning rate and relatively raising the learning space of rare features. This chain is self-consistent under the framework of sparse coding and contrast gradient, and can also explain the empirical benefits of hard negative sampling in fine-grained visual language tasks, which I think is the most valuable contribution of the whole paper.
  2. Previously, the analysis of learning dynamics of comparative learning features mostly stayed in a layer of feedforward network plus random negative examples. This paper advances to the Transformer-MLP and SigLIP-type sigmoid loss with single-headed self-attention, and explicitly distinguishes common and rare features, and tells a complete story of "moderate beneficial, excessive inversion, and the threshold can be alleviated", with a lot of theoretical proofs.
Critical weaknesses
  1. From the experimental verification, the experiment in this paper is not enough to support the author's starting point and the whole process of theoretical derivation, and more importantly, the core prediction has not been verified by experiments. The sigmoid coefficient is taken as 1/2 in the whole derivation, which is equivalent to assuming that the similarity is always close to 0, but the theorem is about decomposition after convergence to ω (1). The optimal hard negative ratio ρ*≈1-ε2/ε1 calculated from this requires that almost all negative cases are hard negative cases. More importantly, the headline "When"-namely, the excessive harm of Theorem 3.3 and the threshold mitigation of Theorem 3.4-has not been directly tested in the synthetic experiment and the real experiment: there is no experiment in which ρ changes from 0 to 1, and there is no ablation of Th. The real table has only three rows and most of the gains are within the noise.
  2. The author didn't give the experimental settings of appendix details promised in the reproducibility statement: hyperparameter of real experiment, construction mode of hard negative sample, values of ρ and Th, and multi-seed variance.
Questions for authors
  1. Formulas (64), (102) and (147) all take the sigmoid coefficient as 1/2. What is the basis? Can it be demonstrated that the weight ratio σ(sim_HN)/σ(-sim_pos) of the positive and negative gradient is still approximately constant in the whole second stage when the feature component reaches ω (1) and the similarity has increased?
  2. Is ρ the proportion of hard negative samples or the proportion of sharing common features? Equation (63) supports the former, but the words Assumption D.1 and K2 are written as the latter, while Assumption 3.5 keeps the latter as 1. Please define it uniformly. Please make up the experiment that ρ sweeps from 0 to 1, and at the same time give the ablation curve and multi-seed variance of Th to directly test the inversion of Theorem 3.3 and the threshold effect of Theorem 3.4.
  3. Please explain why there will be a jump of +6.4 on VSR after adding filtering when the gain of other tasks is only one decimal point.
Reviewer_ysQxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper first constructs a sparse-coding data model—utilizing a simplified Transformer encoder and a sigmoid-based contrastive loss—to theoretically analyze how random and hard negative samples influence the learning dynamics of common versus rare features. It further examines how an excess of hard negative samples suppresses common features and proposes a filtering strategy based on a threshold for the similarity difference between positive and negative samples. This approach aims to mitigate the over-suppression issue and foster a more balanced learning process for both common and rare features.

Critical strengths
  1. The paper focuses on the impact of hard negative samples on fine-grained vision-language understanding and attempts to explain their underlying mechanism from the perspective of feature learning dynamics. This provides a theoretical analysis of the imbalance in feature learning often encountered in multimodal contrastive learning.

  2. The proposed method is simple and practical. The filtering strategy is computationally inexpensive to implement and does not require the addition of complex network architectures.

Critical weaknesses
  1. The core method relies on thresholding the similarity gap between positive and negative pairs, which is relatively simple and closely related to existing hard-negative mining and selection strategies. The paper does not sufficiently establish the methodological novelty of this approach over prior work. Moreover, it lacks comparisons with relevant state-of-the-art methods from 2026, such as FALCON and FG-CLIP 2, making it difficult to assess the proposed method's competitiveness.

  2. The paper assumes that hard negative pairs share common features while remaining independent in rare features, which is an idealized data assumption. In addition, the gradient analysis involves a dependence between the sigmoid weighting term and sample similarity, but the use of independence assumptions and the validity of the corresponding derivations are not sufficiently explained. Furthermore, Theorem 3.4 relies on the effective parameter after thresholding satisfying ρ=1−Θ(ϵ2/ϵ1)\rho = 1-\Theta(\epsilon_2/\epsilon_1), yet the paper does not sufficiently explain why the proposed thresholding rule guarantees this condition. This leaves a gap between the theoretical analysis and the actual selection mechanism.

  3. Although the paper evaluates the method on multiple benchmarks, the real-world experiments mainly compare random negatives, hard negatives, and thresholded hard negatives. Systematic analyses of threshold values, hard-negative ratios, and alternative selection strategies are lacking, as is sufficient validation across multiple random seeds. Moreover, the theoretical analysis uses a simplified Transformer encoder, whereas the empirical evaluation is primarily conducted with SigLIP2. The effectiveness of the proposed strategy across different vision-language architectures remains unverified.

  4. The paper analyzes feature-learning balance under random negatives, appropriately selected hard negatives, excessive hard negatives, and thresholded hard negatives. However, the conclusions of several theorems appear substantially overlapping. Some derivations rely on prespecified conditions, and the connection between the theoretical results and the practical algorithm remains insufficiently clear. As a result, the theoretical analysis is lengthy relative to the substantive methodological contribution.

Questions for authors

All issues listed in the "Critical Weaknesses" section must be explained.

Reviewer_Kd4dOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks why multimodal contrastive learning under-learns rare (low-frequency) features, and when adding hard negatives helps or hurts. The setting is a two-tower sparse-coding model of image-text data: inputs are token tuples generated from shared sparse latents over column-orthonormal dictionaries, with common features active far more often than rare ones. Positives share latents. The encoder is a self-attention layer followed by a one-hidden-layer dead-zone ReLU head, trained with a SigLIP-style pairwise sigmoid loss with weight decay. Hard negatives are constructed to share the anchor's common features while differing in its rare ones, and a two-sided similarity-threshold rule retains a hard negative only if its similarity to the anchor stays within a fixed threshold of the positive's.

The analysis tracks the per-feature coefficients learned through a two-phase training dynamic, and the four main results form a clean arc. With random negatives only, common features are learned to order-one coefficients while rare features remain negligible. Mixing in hard negatives slows the common-feature learning rate in proportion to their share while leaving rare features untouched; at the right mix the two rates match and both groups are learned, and pushing the share further suppresses the common features instead, so hard negatives become harmful. Finally, the authors show that thresholding the hard negatives restores the balance. A suite of nine synthetic experiments qualitatively matches the predicted dynamics, and fine-tuning SigLIP2 with hard negatives plus thresholding improves compositionality benchmarks (ARO, VALSE, VSR, FOIL-COCO).

Critical strengths

The paper reduces hard-negative sampling to a single scalar intervention on the common-feature learning rate, with rare features untouched and a sharp balance condition separating benefit from harm. One formula explains both why hard negatives help and why too many hurt, and it makes coefficient-level predictions that controlled experiments could falsify. I re-derived the central gradient-cancellation computation and verified the mechanism at the heuristic level.

Critical weaknesses
  1. The abstract advertises "Transformer-based encoders," but attention is never parameterized or trained: the analyzed object is an L-view average of a per-token MLP (Definition F.1). Theorem 3.4 is stated unconditionally while Theorem E.2 supposes the balance condition, so the main-text statement fails for degenerate thresholds, and its E.3 "proof" redefines ρ\rho post hoc. Theorem 3.2's hypothesis (a sample fraction) describes a different model from the one analyzed (per-pair sharing, Assumption D.1), with the equivalence resting on an unstated loss weighting.
  2. Lemma F.1's gradients hold for a linearized surrogate with no error bounds; the accumulated SGD error is never shown negligible against a concrete batch size that appears only mid-proof; and no experiment tests the harmful regime (Theorem 3.3) or the thresholding mechanism (Theorem 3.4) — no ρ\rho or ThT_h sweep exists in the paper — while Table 1 lacks the zero-shot baseline, variance estimates, method comparisons, and the implementation details promised in the Reproducibility Statement.
Questions for authors
  • Where are the "experimental settings, implementation details, and hyperparameter configurations" promised in the Reproducibility Statement? What values of dd, d1d_1, LL, mm, learning rates, trial counts, hard-negative construction, and ThT_h were used?
  • Lemma F.6 as printed sets its threshold at σ0⋅d\sigma_0 \cdot d, which with initialization standard deviation σ0\sigma_0 appears to make the stated condition hold trivially for all pp. What was the intended threshold? And is any attention parameter ever trained in the analysis — if not, will the abstract's "Transformer-based encoders" be revised?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.