- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies how to use relevance scores from a multimodal large language model to train universal multimodal embeddings across heterogeneous tasks. The authors argue that teacher score levels and positive–negative score gaps vary by task, making their direct use as supervision potentially inconsistent. Their method, SCALE, computes task-specific score references and scales. It uses calibrated teacher gaps and student similarity to adjust the training logits of mined negatives, and uses within-task teacher rankings to align positive-pair similarities. On MMEB, the authors report Overall Precision@1 gains of 2.3 and 2.8 percentage points for Qwen3-VL-2B and 8B students over their same-backbone, same-teacher-score KL-Distillation reproductions. They also present component ablations, parameter and teacher-sensitivity analyses, and qualitative examples.
Critical strengths
- The method considers how informative and potentially ambiguous hard negatives should influence contrastive learning. It uses teacher score gaps to determine the direction of each adjustment and student similarity to control its magnitude.
- On MMEB, SCALE improves Overall Precision@1 by 2.3 and 2.8 points over the authors’ same-backbone, same-teacher-score distillation baselines for the 2B and 8B students, while retaining single-vector inference without teacher calls.
Critical weaknesses
- The paper does not establish that task-conditioned calibration, its central contribution, is responsible for the reported gains. The main same-backbone comparison in Table 1 is against JS-based distillation, which does not explicitly route or reweight individual negative logits. Thus, the improvements from 68.4 to 70.7 for the 2B model and from 72.1 to 74.9 for the 8B model reflect the complete SCALE training recipe. Several additional elements, including query-level positive weights, the forward-KL anchor, EMA-centered ranking embeddings, and the release and decay schedules, are not separately ablated. More importantly, no experiment replaces the task-specific score references with global ones while keeping the rest of SCALE fixed. The published QQMM, UniME-V2, and ReMatch results use different backbones or training pipelines and therefore do not provide controlled method comparisons. Useful same-backbone controls would include InfoNCE with the same mined negatives, a single global teacher-score threshold, or hardness-weighted contrastive learning. The paper should also position its negative-handling mechanism against related work such as BACL (Ye et al., NeurIPS 2025), which explicitly addresses ambiguous negatives. Since BACL reports results on different benchmarks, its published scores cannot be compared directly; a matched adaptation where feasible, or a clear methodological comparison, would be informative. Without these controls, the results show that the full training recipe outperforms one distillation baseline, but do not isolate the contribution of task-conditioned calibration.
- The experimental evidence is insufficient to support the component-level and generalization conclusions. Each reported configuration was trained once, while removing an individual component changes the Overall score by only 0.5–0.6 points in Table 2. Some ablated variants also outperform the full model on grounding or IND. Table 2 reports 66.9 for its baseline and 68.2 for its full model, whereas the corresponding 2B figures in Table 1 are 68.4 and 70.7. Tables 3 and 4 also report a SCALE score of 68.2, but the backbone and exact configuration underlying these analyses are not clearly identified. Table 3 varies the base boundary and a “tightness” setting without defining the latter or explaining how the settings were selected. The gains are concentrated in-distribution: OOD improves by only 0.2 points for 2B and 1.2 points for 8B. Finally, Figure 3 depicts ImageNet-A within the calibration module even though Appendix C lists it as an OOD evaluation dataset. The authors should clarify whether this is only a schematic label and confirm that evaluation data did not enter the calibration statistics. Together, these issues make it difficult to assess the stability of the smaller effects or connect the ablations to the headline results.
- The boundary and novelty of the contribution are not sufficiently clear. Related work has already used MLLM judgments for hard-negative mining and soft-score supervision (UniME-V2), explicitly adjusted hard-negative gradients (QQMM), and developed boundary-aware training for ambiguous negatives (BACL). SCALE appears to distinguish itself through task-specific score references combined with negative routing and positive-pair ranking. However, the paper does not clearly explain which aspects of this combination go beyond these earlier mechanisms, and it does not discuss BACL. More importantly, without isolating the task-specific references from simpler global or hardness-based alternatives, it is difficult to judge whether the claimed contribution is a substantive new insight or an incremental combination of existing techniques. A sharper comparison with prior work and a targeted task-specific-versus-global ablation would clarify the contribution.
Questions for authors
-
With all other parts of SCALE held fixed, how does performance change when the task-specific references G_t, S_t, and P_t are replaced by statistics computed globally over the training data? Can the effects of the negative-gap references and positive-score reference be reported separately?
-
The routing boundary increases when a task has weaker lower-tail teacher separation. What evidence supports interpreting this as greater negative-sample ambiguity rather than a task-dependent shift in the teacher's scoring scale? How do alternative boundary rules and the suppression rates for each task compare?
-
On the same Qwen3-VL backbones and with the same mined negatives, how does SCALE compare with an InfoNCE baseline and a task-agnostic hard-negative weighting or teacher-score threshold? What are the separate effects of the positive-pair weights and the forward-KL anchor?
-
What exact backbone, teacher, training budget, and settings produced Tables 2–4? Why is the full-model score of 68.2 in these analyses different from the 2B main result of 70.7 in Table 1? Can the authors report variation across multiple seeds for the main comparison and component ablations?
-
What is the granularity of a “task” when computing calibration statistics, and which data are used to estimate them and select hyperparameters? In particular, is ImageNet-A in Figure 3 illustrative only? Please confirm that no OOD evaluation examples contributed to calibration or hyperparameter selection.
6 comments
Response to Reviewer kKfz (Part 2)
Weak2.1: Task Trade-offs in Component Ablations
Thanks for raising this point. Table 2 shows that removing any component reduces Overall P@1 by 0.5–0.6 percentage points, supporting each component's contribution to aggregate performance. Full SCALE improves over the baseline by 1.3 points and achieves the highest Overall, VQA, Retrieval, and OOD scores. Since UME aims to learn general-purpose representations across diverse tasks, we assess component value through aggregate performance. Some ablated variants achieve higher Grounding or IND scores, but these gains come with lower scores on other tasks and Overall. Similar trade-offs appear in the MMEB results of ReMatch [2] and UniME-V2 [3], whose final models improve overall performance without leading every individual task. Full SCALE achieves the best aggregate performance across all 36 datasets in our ablations.
Weak2.2 & Question4: Configuration Differences
Thanks for pointing out that the training settings were unclear. Due to compute constraints, we used different training budgets. Table 1 uses 2,000 steps for both SCALE and the baseline, following UniME-V2 [3], yielding Overall scores of 70.7 and 68.4, respectively. Tables 2–4 use 0.5 epochs, by which point loss and performance had stabilized, yielding 68.2 for SCALE and 66.9 for the baseline in Table 2. These budget differences account for the different reported scores. Within each comparison, training settings are matched except for the factor under study. We will clarify these configurations in the revision.
Weak2.3: OOD Performance Gains
Thanks for raising this concern. Our per-dataset analysis shows that the 2B model improves on most OOD datasets, while ObjectNet declines by 14.1 points and offsets much of the gain elsewhere. The other 15 OOD datasets improve by 1.2 points on average. The aggregate OOD gains are 1.2 points for Qwen3-VL-8B and 0.9 points in the added LLaVA-OV experiment, showing positive aggregate OOD gains across backbones. We will clarify these results in the revision.
Weak2.4: Figure 3 ImageNet-A Confusion
Thanks for flagging this. ImageNet-A appears in Figure 3 as an illustrative label. We estimate all calibration statistics (, , and ) from training data only, and no OOD evaluation data is used for calibration or hyperparameter selection. We will revise Figure 3 to avoid this ambiguity.
Weak3.1: Distinction from Related Work
Thanks for pointing out these related works. Lines 108–114 discuss related baselines and their differences from SCALE. UniME-V2, QQMM, and BACL improve multimodal representations through teacher-score supervision, hard-negative gradient adjustment, and boundary-aware curriculum learning, respectively. SCALE focuses on a different issue. In unified multitask representation learning, the same teacher score or score gap can carry different meanings across tasks. Using raw values directly can therefore introduce task-dependent scoring differences into the supervision.
SCALE combines fine-grained supervision from an MLLM teacher with task-conditioned calibration. Teacher scores provide relevance information beyond binary labels, and task statistics provide a reference for interpreting that information. This focus informs how SCALE uses teacher supervision to adjust negative competition and learn differences in positive match quality. We will clarify this distinction in the revision.
Response to Reviewer kKfz (Part 3)
Question2: Justification of Task-Specific Boundaries
To mitigate task-dependent scoring shifts, SCALE first computes positive–negative log-odds differences, which cancel shared additive offsets in that space. As discussed in Section 3 and Figure 2, teacher-score and score-gap distributions differ across tasks, motivating task-specific references. SCALE therefore uses task-specific boundaries and scales to account for these differences. Within each task, routing varies smoothly with the normalized gap, treating negatives with weaker teacher separation more conservatively. A smaller calibrated gap thus serves as a signal of potential negative-sample ambiguity.
We compared task-conditioned and global calibration to examine suppression behavior across tasks. The standard deviation of task-level suppression rates is approximately 10 percentage points under task-conditioned calibration and 8 under global calibration, indicating more differentiated suppression across tasks. Under the same training settings, task-conditioned calibration also improves Overall P@1 from 67.5 to 68.0. Together, these results support the value of adapting calibration to task-specific score distributions.
Question5: Calibration Granularity and Data Sources
Thanks for the question. Calibration uses each dataset in the training data as the task unit. We estimate the quantiles and scales using teacher scores and positive–negative score gaps from training examples only. OOD evaluation data is used for neither calibration nor hyperparameter selection. ImageNet-A is an illustrative label in Figure 3, and we will correct the figure in the revision.
References
[1] Xue, Youze, Dian Li, and Gang Liu. “Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying.” arXiv preprint arXiv:2506.02020, 2025.
[2] Liu, Qianying, et al. “ReMatch: Boosting Representation through Matching for Multimodal Retrieval.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026.
[3] Gu, Tiancheng, et al. “UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning.” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 26, 2026.
[4] Ye, Hua, et al. “Aligning by Misaligning: Boundary-Aware Curriculum Learning for Multimodal Alignment.” Advances in Neural Information Processing Systems, vol. 38, 2025.
Response to Reviewer kKfz (Part 1)
Weak1.1 & Weak3.3 & Question1: Task-Conditioned Versus Global Calibration
Thanks for the suggestion. Calibration interprets teacher scores within task context before they guide negative routing and positive-pair ranking. MMEB tasks differ in difficulty, input modality, label granularity, and teacher-score distributions. A score gap that is large for one task may be ordinary for another, so global references can over-suppress negatives in some tasks and under-suppress them in others.
We therefore added a controlled experiment that replaces Gt, St, and Pt with shared global statistics computed over all training tasks. Task-conditioned calibration improves Overall P@1 from 67.5 to 68.0, supporting the value of adapting teacher supervision to each task's score distribution. This experiment evaluates the joint effect of the three references. We will include this comparison in the revision.
Weak1.2: Controlled Comparisons Across Backbones
Thanks for raising this point. We chose the more recent Qwen3-VL backbones to assess whether SCALE provides additional gains on strong multimodal backbones. In Table 1, SCALE and KL-Distillation use the same backbone. SCALE improves Overall P@1 by 2.3 and 2.8 percentage points on the 2B and 8B models, respectively. We report the published results of QQMM [1], UniME-V2 [3], and ReMatch [2] for performance context, following the reporting practice in prior work. We will clarify the scope of these comparisons in the revision.
Weak1.3 & Question3: Comparisons Under Matched Training Settings
We compare InfoNCE, KL-Distillation (UniME-V2), and SCALE on Qwen3-VL-2B under identical settings, including the backbone, MMEB split, mined negatives, 2,000 training steps, and 8 A100 GPUs. Only the objective differs. SCALE adds task-conditioned calibration, routing, and ranking to teacher-scored candidates. It improves over InfoNCE by 4.2 IND, 0.9 OOD, and 2.7 Overall points, and over KL-Distillation by 3.9, 0.2, and 2.3 points, respectively.
Weak1.4 & Weak3.2: Negative Handling Compared with BACL
Thanks for pointing out this related work. BACL [4] addresses multimodal alignment through an easy-to-hard sampling curriculum and a local attention loss to learn fine-grained mismatches in hard negatives. SCALE addresses how to interpret MLLM teacher scores and score gaps across heterogeneous tasks in universal multimodal embedding. After task-conditioned calibration, the teacher score gap determines whether to increase or decrease negative competition, while student difficulty controls the adjustment magnitude. Within-task positive-pair ranking transfers differences in match quality. We will clarify how BACL's focus on a hard-negative curriculum differs from SCALE's focus on transferring teacher supervision across tasks in the revision.