- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper compares eight contrastive training strategies under a unified MMEB setting, with supplementary experiments on two additional backbones. It uses a multimodal large language model (MLLM) as a judge to classify top-1 predictions into labeled positives, unlabeled but valid matches, and hard- or easy-negative errors. Aligning the baseline settings substantially reduces several previously reported gains. The existing B3 strategy improves average Precision@1 on Qwen2-VL-2B from 65.1 to 69.0. The paper argues that further upweighting gradients can hurt performance, aggregate similarity separation is an unreliable performance indicator, and negative filtering, mining, and batch construction are complementary.
Critical strengths
- The paper is well written and addresses a practical issue: using appropriately aligned baselines when comparing training strategies.
- The evaluation covers 36 MMEB datasets and three backbones. Ablations of individual B3 components provide useful evidence beyond comparisons between different methods.
Critical weaknesses
-
Insufficient validation of MLLM judgments. Appendix B.2 compares aggregate category proportions across different judges, but does not validate the correctness of individual false-negative (FN), hard-negative (HN), or easy-negative (EN) judgments. Similar category proportions can conceal different errors. The paper reports neither human verification nor agreement on individual cases, and this check covers only the baseline checkpoint. This limits confidence in the semantic interpretations of Table 4, especially for differences below one percentage point.
-
Insufficient evidence on the effects of gradient upweighting. Section 4.2 shows that specific LLaVE/EGA configurations, including B3+EGA, degrade performance. However, it provides neither a sweep over upweighting strength nor an intervention on false-negative contamination. The results support findings within the tested settings, but do not establish the general claim that stronger upweighting causes greater harm. The proposed gradient-distortion mechanism remains insufficiently tested. Small performance differences also lack uncertainty estimates from repeated runs.
-
No direct analysis of per-query rankings. Figure 2 provides evidence that aggregate distributional overlap does not consistently track performance. However, Section 4.3 does not directly analyze per-query ranking margins or rank changes. Its HN/EN distributions use mean similarities over the five highest- or lowest-ranked non-positive candidates, respectively, rather than the semantic categories in Table 4. These aggregate statistics do not measure the strongest competing candidate that determines top-1 accuracy. Direct measurements would make the instance-level explanation more convincing.
-
Limited additional insight beyond prior work. The main training recipe already appears in B3 [1], while MLLM-based semantic scoring and negative mining appear in UniME-V2 [2]. MAFA also combines informative batch construction with false-negative management [3]. The paper already cites these works; my concern is that the current analysis establishes only limited additional insight.
[1] Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining. NeurIPS 2025.
[2] UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning. AAAI 2026.
[3] MAFA: Managing False Negatives for Vision-Language Pre-training. CVPR 2024.
Questions for authors
See the weaknesses.