- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates an overlooked evaluation issue in universal multimodal retrieval: the sensitivity of retrieval performance to different modality assignment settings. The authors conduct a systematic empirical study across multiple multimodal retrieval models and benchmarks, showing that retrieval results can vary substantially under different modality assignment configurations even when the same models and datasets are used.
Critical strengths
- The paper reveals an interesting evaluation issue in universal multimodal retrieval that is largely overlooked.
- The analysis is carefully conducted across multiple models and datasets.
Critical weaknesses
- The main contribution is primarily an empirical observation rather than a new methodology, algorithm, or theoretical framework. The novelty and broader impact should be further strengthened beyond reporting the existence of modality-assignment sensitivity.
- Although the phenomenon is clearly demonstrated, the underlying mechanism causing modality-assignment sensitivity is not sufficiently investigated. More analysis is needed to explain whether the issue originates from model architecture, training objectives, modality imbalance, or benchmark design.
- The practical implications of this finding remain unclear. The authors should further discuss how this phenomenon should influence future model training strategies, benchmark construction, or evaluation protocols.
- The study mainly focuses on demonstrating sensitivity under existing settings. Additional experiments exploring whether training-time interventions, modality balancing strategies, or alternative evaluation designs can alleviate this issue would improve the significance of the work.
Questions for authors
- What are the fundamental factors responsible for modality-assignment sensitivity in multimodal retrieval models?
- How should future multimodal retrieval benchmarks be redesigned to reduce the impact of modality assignment bias?
- Can the observed sensitivity be mitigated through model training strategies or architectural modifications?
3 comments
please examine rebuttal response
Hello Reviewer 9wU4,
As you are likely aware, the rebuttal engagement period is coming to a close. Thus, at your earliest convenience, please take a look at the rebuttal and comment and decide if it changes your thoughts regarding your current evaluation.
Thanks.
Although the author responded, the issues of novelty and the limitations of the experimental scope still remain unresolved. Therefore, I maintain the previous score.
Thank you for your valuable feedback. We appreciate your suggestions on understanding the underlying factors and making the findings more useful. We have added further analysis and practical evaluations, with the corresponding manuscript changes highlighted in blue.
[W1] Strengthening the contribution beyond empirical observations
We agree that identifying modality-assignment sensitivity is only a starting point. Our contribution is to understand when and why retrieval outcomes change under controlled modality reassignment, and how these changes affect model comparisons.
Beyond the original sensitivity measurements, we now provide an evaluation protocol, paired uncertainty estimates, and a score correction experiment. Together, these results connect our observations to practical decisions in multimodal retrieval evaluation.
[W2, Q1] What factors drive modality-assignment sensitivity?
Our analysis points to two closely related factors: (1) Modality-dependent scores. The same negative entity can receive different similarity scores depending on its representation modality. Reassigning its modality can therefore change its position relative to the positive candidate. (2) The difficulty of competing candidates. These score changes matter most when a negative is close to the positive-score boundary. Our experiments show that hard negatives account for most of the observed rank influence.
We agree that understanding the roles of model architecture and training is important for explaining the origins of this phenomenon. While our analysis clarifies how modality-dependent scores lead to assignment-sensitive rankings, it does not fully disentangle the architectural and training choices underlying these score differences. The evaluated models differ in architecture, training objectives, and training data simultaneously, and controlled comparisons would be needed to separate their effects. We view such an investigation as an important direction for future work.
[W3, Q2] How should future benchmarks account for modality assignment?
Future benchmarks should specify both modality proportions and how candidate entities are assigned to modalities, and release shared assignments for model comparison. We suggest a simple protocol: use the same random modality assignments for all models, average the results over several draws, and compare the models on the same queries.
We add two experiments to examine this approach.
Reduce the effect of a single random assignment. We sample from all possible assignments (4900 in our setting) per pool, sharing each draw across all 16 models. For each model, we use its Hit@1 averaged over all possible assignments as the reference, then measure the error when only K random draws are used. Across 2000 repeated trials, increasing K from 1 to 16 reduced this error substantially:
Values are in percentage points and averaged over the 16 models. Using a pairwise-gap error criterion (95-percentile gap error of at most 0.5 points), the smallest tested K was 4/32/1 for Hit@1 and 1/16/1 for MRR (Flickr30K/OmniSET/noCaps), respectively.
Show which model differences are supported by the queries. We perform 5000 paired bootstrap resamples at the query level, keeping each query's two directions and three control replicates together. Among the 120 model pairs per dataset, the 95% confidence interval contains zero for 8/11/1 pairs in Hit@1 and 6/9/1 pairs in MRR (Flickr30K/OmniSET/noCaps). Reporting these paired intervals makes it clear where the available queries support a model's advantage and where the comparison remains uncertain.
[W4, Q3] Can modality-assignment sensitivity be mitigated?
Our results suggest cross-modal score calibration as a concrete direction to investigate. We investigate an inference-time score correction for mitigating the sensitivity motivated by our analysis. We estimate the mean difference between target and non-target scores with five-fold cross-fitting in the paper, then now subtract it from the observed non-target scores. The candidate pools remain unchanged. Averaged across 16 models, Hit@1 improve by 16.00/11.32/8.25 points on the three datasets. In 45 of 48 model-dataset cases, both Hit@1 and MRR improve, and 34 of 48 futher reduce the sensitivity. We also note that MM-Embed is an exception, with lower retrieval quality after correction, consistent with its different response to modality reassignment. A direct evaluation of training-time and architectural interventions remains an important direction for future work.