acceptodds
Under review as a conference paper at ICLR 2027

Modality-Assignment Sensitivity in Universal Multimodal Retrieval

Abstract

In mixed-modality retrieval, fixing candidate identities and modality counts still leaves open which entities are represented as images or text. We study sensitivity to this modality assignment across 16 representative universal multimodal embedding models across three image–text datasets. For each pool, the query, instruction, positive, and negative identities are fixed. First, we increase the number of hard negatives represented in the modality opposite to the positive while decreasing that number among controls, preserving the total image and text counts. Second, we keep each negative group half image and half text and enumerate all assignments of which entities appear in each modality. Retrieval shows assignment-sensitivity under both settings. We conduct analyses and find that rank sensitivity is concentrated among a few competitors. A mean-shift approximation built on the original scores in the positive modality reproduces a substantial part of both coverage-path endpoint changes and fixed-group-count sensitivity despite candidate-level score errors. An audit further reveals attainable reversals in model comparisons that could be missed by a specific assignment. These findings identify modality assignment as an evaluation variable, alongside candidate identities and modality counts, in mixed-modality retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal embedding model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9wU4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates an overlooked evaluation issue in universal multimodal retrieval: the sensitivity of retrieval performance to different modality assignment settings. The authors conduct a systematic empirical study across multiple multimodal retrieval models and benchmarks, showing that retrieval results can vary substantially under different modality assignment configurations even when the same models and datasets are used.

Critical strengths
  1. The paper reveals an interesting evaluation issue in universal multimodal retrieval that is largely overlooked.
  2. The analysis is carefully conducted across multiple models and datasets.
Critical weaknesses
  1. The main contribution is primarily an empirical observation rather than a new methodology, algorithm, or theoretical framework. The novelty and broader impact should be further strengthened beyond reporting the existence of modality-assignment sensitivity.
  2. Although the phenomenon is clearly demonstrated, the underlying mechanism causing modality-assignment sensitivity is not sufficiently investigated. More analysis is needed to explain whether the issue originates from model architecture, training objectives, modality imbalance, or benchmark design.
  3. The practical implications of this finding remain unclear. The authors should further discuss how this phenomenon should influence future model training strategies, benchmark construction, or evaluation protocols.
  4. The study mainly focuses on demonstrating sensitivity under existing settings. Additional experiments exploring whether training-time interventions, modality balancing strategies, or alternative evaluation designs can alleviate this issue would improve the significance of the work.
Questions for authors
  1. What are the fundamental factors responsible for modality-assignment sensitivity in multimodal retrieval models?
  2. How should future multimodal retrieval benchmarks be redesigned to reduce the impact of modality assignment bias?
  3. Can the observed sensitivity be mitigated through model training strategies or architectural modifications?
3 comments

Thank you for your valuable feedback. We appreciate your suggestions on understanding the underlying factors and making the findings more useful. We have added further analysis and practical evaluations, with the corresponding manuscript changes highlighted in blue.

[W1] Strengthening the contribution beyond empirical observations

We agree that identifying modality-assignment sensitivity is only a starting point. Our contribution is to understand when and why retrieval outcomes change under controlled modality reassignment, and how these changes affect model comparisons.

Beyond the original sensitivity measurements, we now provide an evaluation protocol, paired uncertainty estimates, and a score correction experiment. Together, these results connect our observations to practical decisions in multimodal retrieval evaluation.

[W2, Q1] What factors drive modality-assignment sensitivity?

Our analysis points to two closely related factors: (1) Modality-dependent scores. The same negative entity can receive different similarity scores depending on its representation modality. Reassigning its modality can therefore change its position relative to the positive candidate. (2) The difficulty of competing candidates. These score changes matter most when a negative is close to the positive-score boundary. Our experiments show that hard negatives account for most of the observed rank influence.

We agree that understanding the roles of model architecture and training is important for explaining the origins of this phenomenon. While our analysis clarifies how modality-dependent scores lead to assignment-sensitive rankings, it does not fully disentangle the architectural and training choices underlying these score differences. The evaluated models differ in architecture, training objectives, and training data simultaneously, and controlled comparisons would be needed to separate their effects. We view such an investigation as an important direction for future work.

[W3, Q2] How should future benchmarks account for modality assignment?

Future benchmarks should specify both modality proportions and how candidate entities are assigned to modalities, and release shared assignments for model comparison. We suggest a simple protocol: use the same random modality assignments for all models, average the results over several draws, and compare the models on the same queries.

We add two experiments to examine this approach.

  1. Reduce the effect of a single random assignment. We sample from all possible assignments (4900 in our setting) per pool, sharing each draw across all 16 models. For each model, we use its Hit@1 averaged over all possible assignments as the reference, then measure the error when only KK random draws are used. Across 2000 repeated trials, increasing KK from 1 to 16 reduced this error substantially:

    DatasetHit@1 RMSE, K=1K=16
    Flickr30K0.21720.0535
    OmniSET0.80230.1994
    noCaps0.13190.0329

    Values are in percentage points and averaged over the 16 models. Using a pairwise-gap error criterion (95-percentile gap error of at most 0.5 points), the smallest tested KK was 4/32/1 for Hit@1 and 1/16/1 for MRR (Flickr30K/OmniSET/noCaps), respectively.

  2. Show which model differences are supported by the queries. We perform 5000 paired bootstrap resamples at the query level, keeping each query's two directions and three control replicates together. Among the 120 model pairs per dataset, the 95% confidence interval contains zero for 8/11/1 pairs in Hit@1 and 6/9/1 pairs in MRR (Flickr30K/OmniSET/noCaps). Reporting these paired intervals makes it clear where the available queries support a model's advantage and where the comparison remains uncertain.

[W4, Q3] Can modality-assignment sensitivity be mitigated?

Our results suggest cross-modal score calibration as a concrete direction to investigate. We investigate an inference-time score correction for mitigating the sensitivity motivated by our analysis. We estimate the mean difference between target and non-target scores with five-fold cross-fitting in the paper, then now subtract it from the observed non-target scores. The candidate pools remain unchanged. Averaged across 16 models, Hit@1 improve by 16.00/11.32/8.25 points on the three datasets. In 45 of 48 model-dataset cases, both Hit@1 and MRR improve, and 34 of 48 futher reduce the sensitivity. We also note that MM-Embed is an exception, with lower retrieval quality after correction, consistent with its different response to modality reassignment. A direct evaluation of training-time and architectural interventions remains an important direction for future work.

Area_Chair_xyAWOpenReview ↗

please examine rebuttal response

Hello Reviewer 9wU4,

As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal and comment and decide if it changes your thoughts regarding your current evaluation.

Thanks.

Although the author responded, the issues of novelty and the limitations of the experimental scope still remain unresolved. Therefore, I maintain the previous score.

Reviewer_YkEbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the sensitivity of modality assignment in text-image retrieval. For this purpose, this paper reveals and analyzes some phenomena by the experiments across 16 representative universal multimodal embedding models and across three image–text datasets, and presents some findings. The experiments include the influence of negative example changes and modal assignments methods respectively on the retrieval results. Based on these, this paper further discusses related issues, and provides significant conclusions.

Critical strengths
  1. This paper studies an interesting issue and, through a series of comprehensive experiments, reveals some phenomena regarding the sensitivity of modality assignment in text-image retrieval. These findings are conducive to in-depth research in this field.
  2. The experiments in this paper across 16 representative universal multimodal embedding models and across three image–text datasets and can be regarded as having universality.
Critical weaknesses
  1. In the current multimodal and omni-modal situation, the work presented in this paper only covers text and images, which is insufficient for a multimodality approach.
  2. This paper is an empirical study. Therefore, before conducting the experiment, it is necessary to dedicate a section to describing the research plan and related theories, rather than directly moving on to the experimental setup. Currently, these contents are mixed with the experimental setup in this paper, which affects readability and the contribution of the paper.
  3. For an academic conference of such high level as ICLR, merely presenting some findings is not sufficient. If this paper can further explore how to apply these findings in multimodal retrieval based on this, it would be more comprehensive.
Questions for authors

This paper places the contribution summary in the "Related Work" section. Although there is no fixed pattern for this, it seems more appropriate to place it in the "Introduction" section, as this would make the survey of related work more targeted.

3 comments

Thank you for the suggestion that we broaden the modality coverage and clarify the study design. We address these points and their practical implications below. The corresponding changes in the revised manuscript are highlighted in blue.

[W1] Extending beyond image-text retrieval

We extend our evaluation to text-video and video-text retrieval using the released OmniSET video data. We keep the same 101 queries, 8 hard negatives and 8 controls per pool, and 4900 possible modality assignments. Three video-capable models process the videos using their native eight-frame inputs. All three show clear assignment sensitivity:

ModelHit@1-sensitiveRank-sensitive
Qwen3-VL-Embedding-2B16.5%24.1%
Qwen3-VL-Embedding-8B21.0%25.4%
VLM2Vec-V231.8%44.2%

Hard negatives account for 96.5%/92.4%/88.1% of the rank influence in these models, respectively. These results show that modality assignment also matters for video retrieval under the same setting.

[W2] Clarifying the research design

We add a separate Study Design section before the experimental setup. It explains the two questions behind our experiments: (1) how retrieval changes when modality counts are redistributed between hard negatives and controls, and (2) whether the identities assigned to each modality still matter when these counts are fixed.

We also explain the connection between score differences and retrieval rankings. A negative candidate changes the positive rank when its score crosses that of the positive candidate. This provides the motivation for our experiments before introducing their implementation details.

[W3] Applying the findings to retrieval evaluation

We use the findings in two ways: (1) to make model comparisons more reliable and (2) to improve retrieval through a training-free score correction.

We suggest a simple protocol: use the same random modality assignments for all models, average the results over several draws, and compare the models on the same queries.

We add three experiments to examine this approach.

  1. Reduce the effect of a single random assignment. We sample from all possible assignments (4900 in our setting) per pool, sharing each draw across all 16 models. For each model, we use its Hit@1 averaged over all possible assignments as the reference, then measure the error when only KK random draws are used. Across 2000 repeated trials, increasing KK from 1 to 16 reduced this error substantially:

    DatasetHit@1 RMSE, K=1K=16
    Flickr30K0.21720.0535
    OmniSET0.80230.1994
    noCaps0.13190.0329

    Values are in percentage points and averaged over the 16 models. Using a pairwise-gap error criterion (95-percentile gap error of at most 0.5 points), the smallest tested KK was 4/32/1 for Hit@1 and 1/16/1 for MRR (Flickr30K/OmniSET/noCaps), respectively.

  2. Show which model differences are supported by the queries. We perform 5000 paired bootstrap resamples at the query level, keeping each query's two directions and three control replicates together. Among the 120 model pairs per dataset, the 95% confidence interval contains zero for 8/11/1 pairs in Hit@1 and 6/9/1 pairs in MRR (Flickr30K/OmniSET/noCaps). Reporting these paired intervals makes it clear where the available queries support a model's advantage and where the comparison remains uncertain.

  3. Improve retrieval through score correction. We investigate an inference-time score correction for mitigating the sensitivity motivated by our analysis. We estimate the mean difference between target and non-target scores with five-fold cross-fitting in the paper, then now subtract it from the observed non-target scores. The candidate pools remain unchanged. Averaged across 16 models, Hit@1 improve by 16.00/11.32/8.25 points on the three datasets. In 45 of 48 model-dataset cases, both Hit@1 and MRR improve, and 34 of 48 futher reduce the sensitivity. We also note that MM-Embed is an exception, with lower retrieval quality after correction, consistent with its different response to modality reassignment.

[Q1] Moving the contributions to the Introduction

We originally placed the contribution summary at the end of related work section to help readers see how our work differs from prior studies. We agree with the suggestion that presenting the contributions earlier would make the paper easier to follow. We have therefore moved the summary to the end of the introduction, while keeping the discussion of our differences from prior work in related work.

Thanks for your rebuttal. I have carefully read it.

Thank you for taking time to read our rebuttal. We would appreciate your feedback on whether the additional experiments and revisions address your main concerns. If any issues remain, we would be happy to discuss them and provide further clarification.

Reviewer_ENFzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper showed modality assignment sensitivity in mixed modality retrieval with through evaluations on 16 embedding models and three datasets. They found the even well controlled cases, modality assignment matters, depending on which modalities are given to hard negative samples, the performance varies, to the point that the rank of an embedding model changes.

Critical strengths

Thorough experiments on all the combinations of modality assignment.

Critical weaknesses

It is already known that embedding methods have dependency on modality assignment. Also when models performances' are getting better, hard samples have more impact on the results. It would be great if the detailed evaluations could reach to a good solution that makes benchmarks more robust to those modality assignment.

Questions for authors

I see that algorithm comparison can be affected by small number of hard negative samples and their modality assignment. Could you suggest an approach that can mitigates this issue and can make those comparisons more meaningful?

2 comments

Thank you for suggesting that we consider how to make model comparisons more meaningful. We clarify our contribution and present three additional evaluations below. The corresponding changes in the revised manuscript are highlighted in blue.

[W1] Contribution beyond known modality differences

We agree that modality-dependent embedding scores are well documented. Our focus is narrower: even with fixed candidate identities, hard negatives, and modality counts, which entities are assigned to each modality can change retrieval outcomes and model comparisons. We now connect this observation to an explicit evaluation protocol described below.

[Q1] How can we make model comparisons more meaningful?

We suggest a simple protocol: use the same random modality assignments for all models, average the results over several draws, and compare the models on the same queries.

We add two experiments to examine this approach.

  1. Reduce the effect of a single random assignment. We sample from all possible assignments (4900 in our setting) per pool, sharing each draw across all 16 models. For each model, we use its Hit@1 averaged over all possible assignments as the reference, then measure the error when only KK random draws are used. Across 2000 repeated trials, increasing KK from 1 to 16 reduced this error substantially:

    DatasetHit@1 RMSE, K=1K=16
    Flickr30K0.21720.0535
    OmniSET0.80230.1994
    noCaps0.13190.0329

    Values are in percentage points and averaged over the 16 models. Using a pairwise-gap error criterion (95-percentile gap error of at most 0.5 points), the smallest tested KK was 4/32/1 for Hit@1 and 1/16/1 for MRR (Flickr30K/OmniSET/noCaps), respectively.

  2. Show which model differences are supported by the queries. We perform 5000 paired bootstrap resamples at the query level, keeping each query's two directions and three control replicates together. Among the 120 model pairs per dataset, the 95% confidence interval contains zero for 8/11/1 pairs in Hit@1 and 6/9/1 pairs in MRR (Flickr30K/OmniSET/noCaps). Reporting these paired intervals makes it clear where the available queries support a model's advantage and where the comparison remains uncertain.

We also investigate an inference-time score correction for mitigating the sensitivity motivated by our analysis. We estimate the mean difference between target and non-target scores with five-fold cross-fitting in the paper, then now subtract it from the observed non-target scores. The candidate pools remain unchanged. Averaged across 16 models, Hit@1 improve by 16.00/11.32/8.25 points on the three datasets. In 45 of 48 model-dataset cases, both Hit@1 and MRR improve, and 34 of 48 futher reduce the sensitivity. We also note that MM-Embed is an exception, with lower retrieval quality after correction, consistent with its different response to modality reassignment.

Area_Chair_xyAWOpenReview ↗

please examine rebuttal response

Hello Reviewer ENFz,

As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal and comment and decide if it changes your thoughts regarding your current evaluation.

Thanks.

Comments on the paper

1 comment

We thank all reviewers for your thoughtful and constructive feedback. Your comments help us clarify the study design, broaden the evaluation, and strengthen the practical value of our findings. We have revised the manuscript accordingly, with changes highlighted in blue. We summarize the major revisions below.

  1. Video retrieval experiments. We extend the controlled reassignment experiments to text-video and video-text retrieval, observing clear assignment sensitivity across all three evaluated video-capable models (Sec. 5.1).
  2. Training-free score correction. Building on our score-level analysis, we evaluate a cross-fitted correction that improves both Hit@1 and MRR in 45 of 48 model–dataset cases, connecting the analysis to a practical inference-time intervention (Sec. 5.4).
  3. More reliable model comparisons. We evaluate shared-assignment averaging and paired query bootstrap intervals, showing how to reduce dependence on a single assignment and quantify uncertainty in model comparisons (Appendix C.1).
  4. Improved readability. We add a dedicated study design section (Sec. 3) and move the contribution summary to the introduction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.