- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper provides a theoretical analysis of multimodal contrastive learning with hard negative samples. The authors consider a sparse-coding multimodal data model containing common features with occurrence frequency and rarer discriminative features with frequency , together with a simplified Transformer–MLP encoder and a sigmoid-based contrastive objective. The theoretical analysis develops four regimes. First, under random negative sampling, the model is shown to preferentially learn common features while rare features remain weak. Second, hard negatives that share common features with the anchor while differing in rare features induce a negative gradient along shared common-feature directions. The paper argues that this reduces the effective learning rate of common features from to$\left(1-\rho\right)\epsilon_1$, and that common and rare features can be learned at comparable strength when . Third, when the hard-negative effect is too strong, , common features become over-suppressed and rare features dominate. Finally, the authors propose a similarity-gap thresholding rule intended to filter excessively influential hard negatives and restore balanced feature learning. The paper supports the theory with synthetic experiments examining feature-direction alignment, feature-subspace recovery, and downstream prediction, and with SigLIP2 fine-tuning results on ARO, VALSE, VSR, and FOIL-COCO.
Critical strengths
- The paper addresses a meaningful question that is insufficiently understood theoretically: why hard negatives can improve fine-grained multimodal representation learning and why overly aggressive hard-negative sampling may become harmful. The analysis is more ambitious than a static representation result, as it attempts to characterize feature-learning dynamics and separates common, rare, and noise directions.
- The synthetic experiments visualize the intended feature-learning phenomenon at a relatively fine level, including neuron-to-feature alignment, rare-feature subspace recovery, and downstream classification/regression. The real-data evaluation also reports consistent improvements from random negatives to hard negatives and then to thresholded hard negatives across the tested fine-grained benchmarks.
Critical weaknesses
- The central theoretical control variable is not defined consistently across the manuscript, and this ambiguity affects the interpretation of Theorems 3.2–3.4. In the main-text statement of Theorem 3.2, is introduced as the fraction of negative samples that are hard negatives. Elsewhere, including the Key Insights section and Appendix D, is described as the proportion of shared common features in hard-negative pairs. These are different quantities and generally need not coincide. This becomes particularly problematic under Assumption 3.5 / D.1. The paper assumes that for every common-feature coordinate , the image and text members of a hard-negative pair share exactly the same latent signal, , while rare-feature coordinates remain independent. Under this formulation it is unclear what a continuously varying “proportion of shared common features” represents, since has already been defined as the shared common-feature subset. A precise probabilistic definition of and its relation to the mixture probability of hard versus random negatives is necessary.
- Although the paper repeatedly frames the contribution as a theoretical analysis of Transformer-based multimodal encoders, the actual optimization analysis appears to concern MLP neuron weights acting on unspecified self-attention representations. In the model definition, is introduced only as “the self-attention output at the -th token position.” In the appendix it is renamed , but I could not find a definition of the query/key/value parameterization, softmax attention, or a training-dynamics analysis of attention parameters. The gradients and feature-learning theorems are instead written with respect to the MLP neuron weights and . If the attention map is fixed, the theoretical result is more accurately a result for an MLP trained on Transformer-derived representations rather than a result about Transformer feature-learning dynamics. If the attention parameters are intended to be trainable, their optimization needs to be included in the analysis or the assumptions under which they can be treated as fixed should be stated explicitly.
Questions for authors
Q1. In Theorem 3.2, is described as the fraction of negative samples that are hard negatives, whereas the Key Insights section and Appendix D define as the proportion of shared common features in hard-negative pairs. Which definition is intended? If both quantities are required, please introduce separate symbols and rederive the common-feature gradient with the hard-negative mixture probability and within-hard-negative feature-sharing probability explicitly separated. Q2. Appendix Theorem E.2 assumes that after thresholding, , whereas the main-text Theorem 3.4 appears to state that applying the thresholding rule itself restores balanced learning. Can the authors derive a condition on and the similarity distribution under which thresholding provably yields the required effective ? Otherwise, I believe Theorem 3.4 should be restated as a conditional result. Q3. Please provide the explicit definition of and clarify which attention parameters are trainable.The current derivation appears to analyze gradients only with respect to the MLP neuron weights and . If attention is fixed, why should the result be interpreted as Transformer training dynamics rather than MLP feature learning on fixed attention representations? Q4. Please sweep the hard-negative fraction over a sufficiently wide range and report common-feature alignment, rare-feature alignment, and downstream performance. Does performance peak near the theoretically predicted balancing regime and decrease when ? Please additionally show whether thresholding recovers this degradation.