acceptodds
Under review as a conference paper at ICLR 2027

SCALE: Task-Conditioned Calibration of Heterogeneous MLLM Scores for Universal Multimodal Embedding Learning

Abstract

Existing approaches to universal multimodal embedding learning adopt an MLLM-as-teacher paradigm but often overlook how the meanings of teacher scores and score gaps vary across tasks and instructions, leading to misaligned negative-sample and positive-pair supervision. To address this issue, we propose SCALE, which uses task-conditioned score calibration to coordinate negative routing and positive-pair ranking for reliable cross-task supervision. Specifically, Task-Conditioned Score Calibration establishes task-specific score references using task quantiles and scales, yielding positive reliability and a negative boundary to guide the two learning objectives. Guided by the negative boundary, Collaboration-Based Negative-Sample Routing assigns negatives to logit enhancement or suppression based on calibrated teacher score gaps, thereby matching their training influence to task-relative relevance. Complementing this candidate-level supervision, Ranking-Based Positive-Pair Alignment uses positive reliability to weight the alignment of student similarities with the teacher's ordering of positive pairs within each task, preserving differences in match quality across queries. Experiments on MMEB demonstrate the effectiveness of SCALE, with overall Precision@1 gains of 2.3 and 2.8 percentage points for 2B and 8B students, respectively.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
universal multimodal retrieval, embedding learning, contrastive learning, multimodal large language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kKfzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies how to use relevance scores from a multimodal large language model to train universal multimodal embeddings across heterogeneous tasks. The authors argue that teacher score levels and positive–negative score gaps vary by task, making their direct use as supervision potentially inconsistent. Their method, SCALE, computes task-specific score references and scales. It uses calibrated teacher gaps and student similarity to adjust the training logits of mined negatives, and uses within-task teacher rankings to align positive-pair similarities. On MMEB, the authors report Overall Precision@1 gains of 2.3 and 2.8 percentage points for Qwen3-VL-2B and 8B students over their same-backbone, same-teacher-score KL-Distillation reproductions. They also present component ablations, parameter and teacher-sensitivity analyses, and qualitative examples.

Critical strengths
  1. The method considers how informative and potentially ambiguous hard negatives should influence contrastive learning. It uses teacher score gaps to determine the direction of each adjustment and student similarity to control its magnitude.
  2. On MMEB, SCALE improves Overall Precision@1 by 2.3 and 2.8 points over the authors’ same-backbone, same-teacher-score distillation baselines for the 2B and 8B students, while retaining single-vector inference without teacher calls.
Critical weaknesses
  1. The paper does not establish that task-conditioned calibration, its central contribution, is responsible for the reported gains. The main same-backbone comparison in Table 1 is against JS-based distillation, which does not explicitly route or reweight individual negative logits. Thus, the improvements from 68.4 to 70.7 for the 2B model and from 72.1 to 74.9 for the 8B model reflect the complete SCALE training recipe. Several additional elements, including query-level positive weights, the forward-KL anchor, EMA-centered ranking embeddings, and the release and decay schedules, are not separately ablated. More importantly, no experiment replaces the task-specific score references with global ones while keeping the rest of SCALE fixed. The published QQMM, UniME-V2, and ReMatch results use different backbones or training pipelines and therefore do not provide controlled method comparisons. Useful same-backbone controls would include InfoNCE with the same mined negatives, a single global teacher-score threshold, or hardness-weighted contrastive learning. The paper should also position its negative-handling mechanism against related work such as BACL (Ye et al., NeurIPS 2025), which explicitly addresses ambiguous negatives. Since BACL reports results on different benchmarks, its published scores cannot be compared directly; a matched adaptation where feasible, or a clear methodological comparison, would be informative. Without these controls, the results show that the full training recipe outperforms one distillation baseline, but do not isolate the contribution of task-conditioned calibration.
  2. The experimental evidence is insufficient to support the component-level and generalization conclusions. Each reported configuration was trained once, while removing an individual component changes the Overall score by only 0.5–0.6 points in Table 2. Some ablated variants also outperform the full model on grounding or IND. Table 2 reports 66.9 for its baseline and 68.2 for its full model, whereas the corresponding 2B figures in Table 1 are 68.4 and 70.7. Tables 3 and 4 also report a SCALE score of 68.2, but the backbone and exact configuration underlying these analyses are not clearly identified. Table 3 varies the base boundary and a “tightness” setting without defining the latter or explaining how the settings were selected. The gains are concentrated in-distribution: OOD improves by only 0.2 points for 2B and 1.2 points for 8B. Finally, Figure 3 depicts ImageNet-A within the calibration module even though Appendix C lists it as an OOD evaluation dataset. The authors should clarify whether this is only a schematic label and confirm that evaluation data did not enter the calibration statistics. Together, these issues make it difficult to assess the stability of the smaller effects or connect the ablations to the headline results.
  3. The boundary and novelty of the contribution are not sufficiently clear. Related work has already used MLLM judgments for hard-negative mining and soft-score supervision (UniME-V2), explicitly adjusted hard-negative gradients (QQMM), and developed boundary-aware training for ambiguous negatives (BACL). SCALE appears to distinguish itself through task-specific score references combined with negative routing and positive-pair ranking. However, the paper does not clearly explain which aspects of this combination go beyond these earlier mechanisms, and it does not discuss BACL. More importantly, without isolating the task-specific references from simpler global or hardness-based alternatives, it is difficult to judge whether the claimed contribution is a substantive new insight or an incremental combination of existing techniques. A sharper comparison with prior work and a targeted task-specific-versus-global ablation would clarify the contribution.
Questions for authors
  1. With all other parts of SCALE held fixed, how does performance change when the task-specific references G_t, S_t, and P_t are replaced by statistics computed globally over the training data? Can the effects of the negative-gap references and positive-score reference be reported separately?

  2. The routing boundary increases when a task has weaker lower-tail teacher separation. What evidence supports interpreting this as greater negative-sample ambiguity rather than a task-dependent shift in the teacher's scoring scale? How do alternative boundary rules and the suppression rates for each task compare?

  3. On the same Qwen3-VL backbones and with the same mined negatives, how does SCALE compare with an InfoNCE baseline and a task-agnostic hard-negative weighting or teacher-score threshold? What are the separate effects of the positive-pair weights and the forward-KL anchor?

  4. What exact backbone, teacher, training budget, and settings produced Tables 2–4? Why is the full-model score of 68.2 in these analyses different from the 2B main result of 70.7 in Table 1? Can the authors report variation across multiple seeds for the main comparison and component ablations?

  5. What is the granularity of a “task” when computing calibration statistics, and which data are used to estimate them and select hyperparameters? In particular, is ImageNet-A in Figure 3 illustrative only? Please confirm that no OOD evaluation examples contributed to calibration or hyperparameter selection.

6 comments
AuthorsOpenReview ↗

Response to Reviewer kKfz (Part 1)

Weak1.1 & Weak3.3 & Question1: Task-Conditioned Versus Global Calibration

Thanks for the suggestion. Calibration interprets teacher scores within task context before they guide negative routing and positive-pair ranking. MMEB tasks differ in difficulty, input modality, label granularity, and teacher-score distributions. A score gap that is large for one task may be ordinary for another, so global references can over-suppress negatives in some tasks and under-suppress them in others.

We therefore added a controlled experiment that replaces GtG_t, StS_t, and PtP_t with shared global statistics computed over all training tasks. Task-conditioned calibration improves Overall P@1 from 67.5 to 68.0, supporting the value of adapting teacher supervision to each task's score distribution. This experiment evaluates the joint effect of the three references. We will include this comparison in the revision.

MetricGlobalSCALE
IND71.471.5
OOD62.863.6
Overall67.568.0

Weak1.2: Controlled Comparisons Across Backbones

Thanks for raising this point. We chose the more recent Qwen3-VL backbones to assess whether SCALE provides additional gains on strong multimodal backbones. In Table 1, SCALE and KL-Distillation use the same backbone. SCALE improves Overall P@1 by 2.3 and 2.8 percentage points on the 2B and 8B models, respectively. We report the published results of QQMM [1], UniME-V2 [3], and ReMatch [2] for performance context, following the reporting practice in prior work. We will clarify the scope of these comparisons in the revision.

Weak1.3 & Question3: Comparisons Under Matched Training Settings

We compare InfoNCE, KL-Distillation (UniME-V2), and SCALE on Qwen3-VL-2B under identical settings, including the backbone, MMEB split, mined negatives, 2,000 training steps, and 8 A100 GPUs. Only the objective differs. SCALE adds task-conditioned calibration, routing, and ranking to teacher-scored candidates. It improves over InfoNCE by 4.2 IND, 0.9 OOD, and 2.7 Overall points, and over KL-Distillation by 3.9, 0.2, and 2.3 points, respectively.

MetricInfoNCEKL-DistillationSCALE
IND71.171.475.3
OOD64.164.865.0
Overall68.068.470.7

Weak1.4 & Weak3.2: Negative Handling Compared with BACL

Thanks for pointing out this related work. BACL [4] addresses multimodal alignment through an easy-to-hard sampling curriculum and a local attention loss to learn fine-grained mismatches in hard negatives. SCALE addresses how to interpret MLLM teacher scores and score gaps across heterogeneous tasks in universal multimodal embedding. After task-conditioned calibration, the teacher score gap determines whether to increase or decrease negative competition, while student difficulty controls the adjustment magnitude. Within-task positive-pair ranking transfers differences in match quality. We will clarify how BACL's focus on a hard-negative curriculum differs from SCALE's focus on transferring teacher supervision across tasks in the revision.

AuthorsOpenReview ↗

Response to Reviewer kKfz (Part 2)

Weak2.1: Task Trade-offs in Component Ablations

Thanks for raising this point. Table 2 shows that removing any component reduces Overall P@1 by 0.5–0.6 percentage points, supporting each component's contribution to aggregate performance. Full SCALE improves over the baseline by 1.3 points and achieves the highest Overall, VQA, Retrieval, and OOD scores. Since UME aims to learn general-purpose representations across diverse tasks, we assess component value through aggregate performance. Some ablated variants achieve higher Grounding or IND scores, but these gains come with lower scores on other tasks and Overall. Similar trade-offs appear in the MMEB results of ReMatch [2] and UniME-V2 [3], whose final models improve overall performance without leading every individual task. Full SCALE achieves the best aggregate performance across all 36 datasets in our ablations.

Weak2.2 & Question4: Configuration Differences

Thanks for pointing out that the training settings were unclear. Due to compute constraints, we used different training budgets. Table 1 uses 2,000 steps for both SCALE and the baseline, following UniME-V2 [3], yielding Overall scores of 70.7 and 68.4, respectively. Tables 2–4 use 0.5 epochs, by which point loss and performance had stabilized, yielding 68.2 for SCALE and 66.9 for the baseline in Table 2. These budget differences account for the different reported scores. Within each comparison, training settings are matched except for the factor under study. We will clarify these configurations in the revision.

Weak2.3: OOD Performance Gains

Thanks for raising this concern. Our per-dataset analysis shows that the 2B model improves on most OOD datasets, while ObjectNet declines by 14.1 points and offsets much of the gain elsewhere. The other 15 OOD datasets improve by 1.2 points on average. The aggregate OOD gains are 1.2 points for Qwen3-VL-8B and 0.9 points in the added LLaVA-OV experiment, showing positive aggregate OOD gains across backbones. We will clarify these results in the revision.

Weak2.4: Figure 3 ImageNet-A Confusion

Thanks for flagging this. ImageNet-A appears in Figure 3 as an illustrative label. We estimate all calibration statistics (GtG_t, StS_t, and PtP_t) from training data only, and no OOD evaluation data is used for calibration or hyperparameter selection. We will revise Figure 3 to avoid this ambiguity.

Weak3.1: Distinction from Related Work

Thanks for pointing out these related works. Lines 108–114 discuss related baselines and their differences from SCALE. UniME-V2, QQMM, and BACL improve multimodal representations through teacher-score supervision, hard-negative gradient adjustment, and boundary-aware curriculum learning, respectively. SCALE focuses on a different issue. In unified multitask representation learning, the same teacher score or score gap can carry different meanings across tasks. Using raw values directly can therefore introduce task-dependent scoring differences into the supervision.

SCALE combines fine-grained supervision from an MLLM teacher with task-conditioned calibration. Teacher scores provide relevance information beyond binary labels, and task statistics provide a reference for interpreting that information. This focus informs how SCALE uses teacher supervision to adjust negative competition and learn differences in positive match quality. We will clarify this distinction in the revision.

AuthorsOpenReview ↗

Response to Reviewer kKfz (Part 3)

Question2: Justification of Task-Specific Boundaries

To mitigate task-dependent scoring shifts, SCALE first computes positive–negative log-odds differences, which cancel shared additive offsets in that space. As discussed in Section 3 and Figure 2, teacher-score and score-gap distributions differ across tasks, motivating task-specific references. SCALE therefore uses task-specific boundaries and scales StS_t to account for these differences. Within each task, routing varies smoothly with the normalized gap, treating negatives with weaker teacher separation more conservatively. A smaller calibrated gap thus serves as a signal of potential negative-sample ambiguity.

We compared task-conditioned and global calibration to examine suppression behavior across tasks. The standard deviation of task-level suppression rates is approximately 10 percentage points under task-conditioned calibration and 8 under global calibration, indicating more differentiated suppression across tasks. Under the same training settings, task-conditioned calibration also improves Overall P@1 from 67.5 to 68.0. Together, these results support the value of adapting calibration to task-specific score distributions.

Question5: Calibration Granularity and Data Sources

Thanks for the question. Calibration uses each dataset in the training data as the task unit. We estimate the quantiles and scales using teacher scores and positive–negative score gaps from training examples only. OOD evaluation data is used for neither calibration nor hyperparameter selection. ImageNet-A is an illustrative label in Figure 3, and we will correct the figure in the revision.

References

[1] Xue, Youze, Dian Li, and Gang Liu. “Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying.” arXiv preprint arXiv:2506.02020, 2025.

[2] Liu, Qianying, et al. “ReMatch: Boosting Representation through Matching for Multimodal Retrieval.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026.

[3] Gu, Tiancheng, et al. “UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning.” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 26, 2026.

[4] Ye, Hua, et al. “Aligning by Misaligning: Boundary-Aware Curriculum Learning for Multimodal Alignment.” Advances in Neural Information Processing Systems, vol. 38, 2025.

Reviewer_qV4KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SCALE, a framework designed to address the issue of inconsistent scoring among different tasks when distilling Multimodal Large Language Models (MLLMs). SCALE uses task-conditioned score calibration to coordinate negative routing and positive-pair ranking for reliable cross-task supervision. Experimental results on the MMEB benchmark demonstrate that, compared to baseline fine-tuning/distillation methods, SCALE improves the overall Precision@1 metric for Qwen3-VL-2B and Qwen3-VL-8B by 2.3 and 2.8 points, respectively.

Critical strengths
  1. This paper is well-motivated. This paper comprehensively discusses the challenges that existing multimodal embedding approaches often overlook the meanings of teacher scores and score gaps across different tasks.

  2. The paper is well-structured and highly readable.

Critical weaknesses
  1. The OOD performance in Table 1 seems to yield only marginal improvements, which may undermine the utility in real-world applications (as practical scenarios are typically OOD).

  2. This paper aims to facilitate more reasonable multimodal distillation via task conditions and the student's own states (e.g., similarity). This view is somewhat similar to recent on-policy distillation (OPD); I would like to see some discussion or comparisons with this technique, which would help to better position the proposed method.

Questions for authors
  1. The experiments are conducted mainly on 2B and 8B models. Can the authors provide results on larger models to see whether the proposed method can scale effectively?

  2. This paper mainly evaluates on MMEB. Can the authors provide results on more benchmarks?

3 comments
AuthorsOpenReview ↗

Response to Reviewer qV4K

Weak1: Limited OOD Performance Gains

Thanks for raising this point. We believe that both IND and OOD performance are crucial for practical applications. As detailed in our response to Reviewer Y892 (Weak3), the 2B model improves on most OOD datasets in Table 1, while a pronounced fluctuation on ObjectNet offsets much of the gain elsewhere. We did not observe a comparable single-dataset drop in our Qwen3-VL-8B or LLaVA-OV experiments, which show aggregate OOD gains of 1.2 and 0.9 percentage points, respectively. Moreover, the substantial IND gains across all three backbones further support the practical value of our method. We will clarify this pattern in the revision.

Weak2: Relation to On-Policy Distillation

Thanks for the suggestion. Both SCALE and on-policy distillation (OPD) use teacher supervision to improve a student model and account for the student's current behavior [1]. OPD typically provides token-level teacher supervision on student-generated sequences to reduce the mismatch between the training and student generation distributions. In SCALE, the student independently encodes queries and candidates to learn retrieval representations, with a training objective that does not involve autoregressive token prediction.

SCALE addresses how to interpret MLLM teacher scores and score gaps across heterogeneous multimodal tasks. It calibrates offline teacher scores using task-level statistics, uses current student similarities to adjust negative competition, and transfers match quality through within-task positive-pair ranking. We will discuss this distinction in the revision.

Question1: Scalability to Larger Models

Thanks for the suggestion. Larger-student experiments would help assess SCALE's scalability. While larger models can benefit generation and multi-step reasoning in QA and agent tasks, retrieval requires balancing accuracy with encoding throughput and query latency at scale.

Models from both academia and industry on the MMEB leaderboard are predominantly in the 2B–7B parameter range, consistent with the model sizes evaluated in prior UME work such as UniME-V2 [2] and ReMatch [3]. In our matched-backbone comparisons, SCALE improves Overall P@1 by 2.3 and 2.8 percentage points on the 2B and 8B models, respectively, showing gains at both evaluated sizes.

We also added a LLaVA-OV-1.5-4B experiment to assess whether the gains extend to a different backbone family.

MetricKL-DistillationSCALE
IND69.472.1
OOD60.261.1
Overall65.367.2

Question2: Evaluation Beyond MMEB

Thanks for the suggestion. We chose MMEB because it offers a broad, unified evaluation for universal multimodal embedding. Its 36 datasets span four task categories, including classification, visual question answering, retrieval, and visual grounding. They cover diverse domains and include image, text, and image–text inputs.

MMEB is widely used in universal multimodal embedding research and has been adopted by published work including UniME-V2 [2], ReMatch [3] and MetaEmbed [4].

References

[1] Agarwal, Rishabh, et al. “On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes.” International Conference on Learning Representations, 2024.

[2] Gu, Tiancheng, et al. “UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning.” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 26, 2026.

[3] Liu, Qianying, et al. “ReMatch: Boosting Representation through Matching for Multimodal Retrieval.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026.

[4] Xiao, Zilin, et al. “MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction.” International Conference on Learning Representations, 2026.

Thanks for the authors' rebuttal, and I keep my positive score.

Thank you for your thoughtful review and for reaffirming your positive assessment after the rebuttal. Your questions prompted us to examine why the OOD gains vary across datasets and to clarify how SCALE differs from on-policy distillation. We appreciate the time you took to read our responses and follow up during the discussion.

Reviewer_Y892OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper proposes SCALE for universal multimodal embedding learning. It uses task-specific score statistics to calibrate MLLM teacher supervision, combines teacher score gaps with student difficulty to adjust negative-sample logits, and introduces within-task positive-pair ranking to transfer differences in match quality.

Critical strengths
  1. The paper addresses the reliability of MLLM supervision for universal multimodal embedding learning, which is a critical and practical research area
  2. The empirical results, within the context of Qwen3 models, appear good.
Critical weaknesses
  1. Although the comparisons with KL-Distillation provide validation of improvement, comparisons with other published methods involve different backbones and training pipelines, which makes it difficult to attribute the performance differences to SCALE itself. Moreover, the main student experiments are limited to Qwen3-VL-2B/8B, leaving generalization beyond the Qwen family unverified.

  2. The grouping in Table 1 might be confusing. Are all models in this third group (SCALE) also fine-tuned on MMEB?

  3. The paper reports aggregate performance but does not provide per-dataset results, even in the appendix, making it difficult to assess the consistency of the improvements across datasets. The OOD gain is modest at 0.2 points, and no statistical significance results are reported.

  4. The hyperparameter sensitivity analysis is incomplete.

Questions for authors

Please see the weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Y892 (Part 1)

Weak1: Backbone Generalization and Controlled Comparisons

Thank you for raising this point. We agree that published methods such as QQMM and UniME-V2 use different backbones and training pipelines. We report their results for context, following the reporting practice in prior work [1, 2]. Our primary comparison in Table 1 is with KL-Distillation under the same backbone and training settings. SCALE improves Overall P@1 by 2.3 and 2.8 percentage points on Qwen3-VL-2B and Qwen3-VL-8B, respectively. These controlled comparisons support the effectiveness of the full SCALE training recipe on both backbones.

To further assess generalization across backbones, we added an experiment on the open-sourced LLaVA-OV-1.5-4B, where SCALE improves Overall P@1 from 65.3 to 67.2. We will include these results and the corresponding experimental settings in the revision.

LLaVA-OVKL-DistillationSCALE
IND69.472.1
OOD60.261.1
Overall65.367.2

Weak2: Ambiguous Table 1 Grouping

All models in the third group of Table 1, including the KL-Distillation baselines and SCALE, were fine-tuned on the MMEB training set. We will label this group “With MMEB fine-tuning” and use subheadings to identify the matched-backbone comparisons between KL-Distillation and SCALE.

AuthorsOpenReview ↗

Response to Reviewer Y892 (Part 2)

Weak3: Dataset Consistency and Statistical Significance

Thank you for the suggestion. Here, we report per-dataset scores for the Qwen3-VL-2B comparison in Table 1 across all 36 MMEB datasets to assess how broadly the gains hold and explain the modest aggregate OOD gain.

SCALE matches or exceeds the baseline on 30 datasets. The gains span 20 of 20 IND datasets and 10 of 16 OOD datasets. On the OOD split, GQA improves by 10.8 percentage points, but ObjectNet declines by 14.1 points. The other 15 OOD datasets improve by an average of 1.2 percentage points. The pronounced ObjectNet decline was limited to the Qwen3-VL-2B configuration. Qwen3-VL-8B and LLaVA-OV-4B showed no comparable drop and achieved OOD gains of 1.2 and 0.9 percentage points, respectively. We will include the per-dataset results and the statistical analysis in the revision.

Scores are P@1 (%). Δ denotes SCALE minus KL-Distillation, in percentage points.

TaskSplitDatasetKL-DistillationSCALEΔ
ClassificationINDImageNet-1K78.079.9+1.9
ClassificationINDN24News64.278.8+14.6
ClassificationINDHatefulMemes64.368.6+4.3
ClassificationINDVOC200784.887.4+2.6
ClassificationINDSUN39778.680.2+1.6
ClassificationOODPlace36541.845.5+3.7
ClassificationOODImageNet-A63.061.3-1.7
ClassificationOODImageNet-R89.591.4+1.9
ClassificationOODObjectNet64.850.7-14.1
ClassificationOODCountry21117.617.2-0.4
VQAINDOK-VQA60.866.4+5.6
VQAINDA-OKVQA54.457.2+2.8
VQAINDDocVQA94.195.4+1.3
VQAINDInfographicsVQA66.871.7+4.9
VQAINDChartQA57.963.3+5.4
VQAINDVisual7W61.965.1+3.2
VQAOODScienceQA50.049.5-0.5
VQAOODVizWiz54.054.2+0.2
VQAOODGQA59.069.8+10.8
VQAOODTextVQA85.785.4-0.3
RetrievalINDVisDial80.484.1+3.7
RetrievalINDCIRR61.664.2+2.6
RetrievalINDVisualNews-t2i71.974.7+2.8
RetrievalINDVisualNews-i2t72.075.5+3.5
RetrievalINDMSCOCO-t2i78.179.6+1.5
RetrievalINDMSCOCO-i2t68.374.2+5.9
RetrievalINDNIGHTS68.268.9+0.7
RetrievalINDWebQA90.490.40.0
RetrievalOODOVEN65.366.6+1.3
RetrievalOODFashionIQ26.026.3+0.3
RetrievalOODEDIS86.187.3+1.2
RetrievalOODWiki-SS-NQ66.967.2+0.3
Visual GroundingINDMSCOCO70.979.8+8.9
Visual GroundingOODVisual7W-Pointing81.882.3+0.5
Visual GroundingOODRefCOCO91.693.3+1.7
Visual GroundingOODRefCOCO-Matching93.792.1-1.6

Weak4: Hyperparameter Sensitivity Analysis

Thanks for the suggestion. Table 3 already examines different boundary and tightness settings. We agree that the original analysis could cover more parameters. We therefore added a sensitivity analysis for two task-conditioned calibration parameters.

For ρneg\rho_{\mathrm{neg}}, the negative-gap reference is computed at quantile 1−ρneg1-\rho_{\mathrm{neg}}. Increasing it from 0.90 to 0.975 shifts the reference from the 10th to the 2.5th percentile, focusing on weaker teacher separation when setting task-specific routing boundaries. For ρpos\rho_{\mathrm{pos}}, increasing it from 0.10 to 0.50 raises the positive-score reference from the 10th percentile to the median, making positive-pair reliability weighting more selective.

Across these settings, Overall scores range from 67.71 to 68.02, a spread of 0.31 percentage points, suggesting that aggregate performance is relatively stable over the tested parameter ranges.

Metric0.95 / 0.25 (SCALE)0.9 / 0.250.95 / 0.10.975 / 0.250.95 / 0.5
IND71.2071.2671.4571.3371.39
OOD63.4963.2763.7363.7263.53
Overall68.0067.7168.0267.9567.90
Δ Overall0.00-0.29+0.02-0.05-0.10

Column headings specify ρneg/ρpos\rho_{\mathrm{neg}} / \rho_{\mathrm{pos}}. Δ Overall is measured relative to the SCALE reference configuration, in percentage points.

References

[1] Liu, Qianying, et al. “ReMatch: Boosting Representation through Matching for Multimodal Retrieval.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026.

[2] Gu, Tiancheng, et al. “UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning.” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 26, 2026.

We would appreciate hearing whether our responses have addressed your concerns or if any questions remain. If your concerns are resolved, we hope you could consider raising your rating, which would mean a great deal to us.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.