acceptodds
Withdrawn from ICLR 2027

What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contrastive learning has become the standard paradigm for multimodal representation learning, aligning semantically matched pairs while separating mismatched ones in a shared embedding space. Although various contrastive training strategies have been proposed across negative selection, batch construction, and gradient modulation, their actual effectiveness under controlled, reproducible conditions remains poorly understood. This motivates a fundamental question: Do sophisticated contrastive strategies consistently bring meaningful downstream gains in multimodal embedding learning? To address this, we systematically evaluate representative strategies under a unified benchmarking framework on MMEB. Surprisingly, direct evaluations reveal that most existing strategies offer marginal gains over a standard aligned baseline, with several even degrading overall performance. To uncover the underlying failure modes, we introduce an MLLM-as-a-Judge diagnostic framework that dissects top-1 retrieval behaviors into labeled positives, false negatives (unlabeled valid matches), and hard/easy negative errors. Our systematic analysis yields three key insights: (1) Gradient distortion: Upweighting gradients from high-similarity negatives consistently hurts downstream generalization; (2) Distribution illusion: Wider separation between positive and negative similarity distributions does not necessarily translate to better retrieval performance; and (3) Synergistic training: While false-negative mitigation or hard-batch construction yields limited gains in isolation, integrating hard-negative mining, false-negative mitigation, and hard-batch construction achieves optimal performance, delivering an average improvement of 6.0% on MMEB with Qwen2-VL-2B. Overall, our findings demystify popular contrastive strategies and provide actionable recipes for training universal multimodal embedding models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Embeddings, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pSYYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares eight contrastive training strategies under a unified MMEB setting, with supplementary experiments on two additional backbones. It uses a multimodal large language model (MLLM) as a judge to classify top-1 predictions into labeled positives, unlabeled but valid matches, and hard- or easy-negative errors. Aligning the baseline settings substantially reduces several previously reported gains. The existing B3 strategy improves average Precision@1 on Qwen2-VL-2B from 65.1 to 69.0. The paper argues that further upweighting gradients can hurt performance, aggregate similarity separation is an unreliable performance indicator, and negative filtering, mining, and batch construction are complementary.

Critical strengths
  1. The paper is well written and addresses a practical issue: using appropriately aligned baselines when comparing training strategies.
  2. The evaluation covers 36 MMEB datasets and three backbones. Ablations of individual B3 components provide useful evidence beyond comparisons between different methods.
Critical weaknesses
  1. Insufficient validation of MLLM judgments. Appendix B.2 compares aggregate category proportions across different judges, but does not validate the correctness of individual false-negative (FN), hard-negative (HN), or easy-negative (EN) judgments. Similar category proportions can conceal different errors. The paper reports neither human verification nor agreement on individual cases, and this check covers only the baseline checkpoint. This limits confidence in the semantic interpretations of Table 4, especially for differences below one percentage point.

  2. Insufficient evidence on the effects of gradient upweighting. Section 4.2 shows that specific LLaVE/EGA configurations, including B3+EGA, degrade performance. However, it provides neither a sweep over upweighting strength nor an intervention on false-negative contamination. The results support findings within the tested settings, but do not establish the general claim that stronger upweighting causes greater harm. The proposed gradient-distortion mechanism remains insufficiently tested. Small performance differences also lack uncertainty estimates from repeated runs.

  3. No direct analysis of per-query rankings. Figure 2 provides evidence that aggregate distributional overlap does not consistently track performance. However, Section 4.3 does not directly analyze per-query ranking margins or rank changes. Its HN/EN distributions use mean similarities over the five highest- or lowest-ranked non-positive candidates, respectively, rather than the semantic categories in Table 4. These aggregate statistics do not measure the strongest competing candidate that determines top-1 accuracy. Direct measurements would make the instance-level explanation more convincing.

  4. Limited additional insight beyond prior work. The main training recipe already appears in B3 [1], while MLLM-based semantic scoring and negative mining appear in UniME-V2 [2]. MAFA also combines informative batch construction with false-negative management [3]. The paper already cites these works; my concern is that the current analysis establishes only limited additional insight.

[1] Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining. NeurIPS 2025.

[2] UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning. AAAI 2026.

[3] MAFA: Managing False Negatives for Vision-Language Pre-training. CVPR 2024.

Questions for authors

See the weaknesses.

Reviewer_FS7vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares several multimodal contrastive learning strategies under a unified setup and uses an MLLM-as-a-Judge framework to analyze retrieval errors. The problem is relevant, but the main conclusions rely on a limited set of methods and an automatic evaluation procedure whose reliability is not sufficiently established. Moreover, a shared training configuration may not reflect the best performance of each strategy.

Critical strengths
  • The paper addresses a practical question: whether contrastive training strategies deliver reliable gains when evaluated under controlled conditions.
  • It compares methods targeting several aspects of training, including gradient modulation, false-negative mitigation, hard-negative mining, and batch construction.
  • Evaluation on MMEB covers 36 datasets across classification, visual question answering, retrieval, and visual grounding.
  • The error analysis distinguishes unlabeled but semantically valid matches from genuinely incorrect retrievals, offering more detail than aggregate benchmark scores alone.
Critical weaknesses
  • Limited methodological novelty. The paper’s main contribution is an empirical comparison of existing strategies. B3 combines several previously proposed components, and its technical novelty appears limited. The paper should clarify what it contributes beyond evaluating this combination in a controlled setting.

  • The shared training configuration may not provide a fair comparison. Different strategies may require different learning rates, sampling procedures, or loss weights. If each method was not adequately tuned, the results may reflect how well it fits the chosen configuration rather than its effectiveness.

  • The MLLM-as-a-Judge analysis is insufficiently validated. A central part of the paper’s explanation depends on the judge distinguishing false negatives, hard-negative errors, and easy-negative errors. The paper provides limited evidence about the accuracy and stability of these judgments. Without human-annotated comparisons, agreement across judges, or prompt-sensitivity analyses, the resulting error analysis remains uncertain.

  • The source of B3’s gains is not fully established. B3 combines false-negative mitigation, hard-negative mining, and batch construction. Although the ablations offer some evidence about individual components, it remains unclear whether their contributions are stable, how strongly they interact, and whether the results depend on specific implementation choices.

Questions for authors
  • Were hyperparameters tuned separately for each contrastive strategy? If not, how do the authors rule out that the shared configuration disadvantages some methods?
  • What is the agreement between MLLM judgments and human annotations? How sensitive are the results to the judge model and prompt?
  • Are the ablation results for B3’s components consistent across random seeds and model backbones?
  • Can the authors report confidence intervals or significance tests for comparisons with small score differences?
  • Have the proposed mechanistic findings been reproduced on other datasets or training settings?
Reviewer_31shOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper contributes a controlled negative result on contrastive training for MLLM-based universal multimodal embedding. It reevaluates eight strategies across gradient modulation, false-negative mitigation, batch construction, and combined designs; seven are published methods and FN Risk is the authors' baseline. They first align the VLM2Vec baseline's learning rate, visual token count, LoRA rank, and pooling, reaching 65.1 on MMEB. Under this setting they train eight strategies and a B3 + EGA variant on Qwen2-VL-2B. Five strategies plus B3 + EGA are repeated on Qwen3.5-2B and Qwen2-VL-7B. Only B3 rises clearly, to 69.0; the other seven stay between 63.9 and 65.3. An MLLM judge splits 7,200 top-1 predictions into labeled positives, false negatives, and hard or easy negatives. Three insights follow: similarity-based gradient weighting hurts, distribution separation does not predict performance, and combining false-negative mitigation, hard-negative mining, and hard-batch construction works best.

Critical strengths
  • The authors align the learning rate, visual token count, LoRA rank, and pooling before adding any strategy. The aligned baseline's 65.1 matches LLaVE-2B's reported 65.2 and exceeds UniME-V2-2B's reported 63.6. This control is why the negative result matters.
  • B3's advantage repeats on all three backbones, by 3.9 points on Qwen2-VL-2B, 1.5 on Qwen3.5-2B, and 3.3 on Qwen2-VL-7B. The one clearly positive result in the paper is therefore not a single-backbone artifact.
Critical weaknesses
  • Main concern is that no table reports seed variance or significance. LLaVE (-1.2) and GloFND (-0.8) may exceed noise; EGA (-0.3), FN Risk (-0.1), and FALCON (-0.3) could be ties. In Table 4, 17 of the 24 LP/FN/HN/EN differences for the six single-dimension strategies lie within 0.5 point.
  • Second critical concern is the evidence for the Section 4 explanations. B3's gain is not separated from its teacher or its compute. It needs a pretrained teacher whose identity is unstated and trains about 1.5 times longer than the baseline (29:44 vs 19:12). The reported timing also excludes its one-time preprocessing.
Questions for authors
  • Please report mean and standard deviation over at least three seeds for the baseline, LLaVE, EGA, HOBIT, B3, and B3 + EGA. If the differences fall inside seed variance, "degrade" must become "no gain"; if B3's +3.9 is well outside, the core claim stands.
  • How were each strategy's own hyperparameters set (LLaVE's weighting strength, EGA's margin settings, GloFND's quantile, and so on)? Please test whether a small tuning grid changes the ranking.
  • What is B3's teacher, and is its preparation cost included in the reported training times? Is the paper's B3 the original or B3++ with explicit hard negatives? Please add a baseline trained with the same total compute.
  • Does exact_top1_match in Figure C.1 compare candidate indices or content, and how are designated positives judged false_positive reflected in LP?
  • Please report Cohen's kappa between the judge and human labels on a few hundred cases, and case-level agreement between the two judges, Qwen3.8-Flash and GPT-5.4.
  • What was the InfoNCE temperature, and was it fixed or learned? If lowering the baseline temperature reproduces the LLaVE or EGA effect, insight (1) should be restated as "no gain beyond temperature tuning".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.