Multimodal Representation Alignment based on Spherical Concentration

Multimodal retrieval often requires more flexibility than standard full-modality inference: the same representation space should support pairwise retrieval when only two modalities are available, and joint retrieval when richer multimodal evidence is present. Recent multi-vector similarities effectively address joint retrieval, but variable-modality inference remains challenging, as the learned embeddings must remain useful across different modality subsets. We propose SCORE, a multi-vector similarity that measures the spherical concentration of normalized modality embeddings through a weighted resultant vector. This additive formulation provides a simple score for any available subset of modalities and encourages embeddings to concentrate around a shared consensus direction. To improve flexibility when multiple modalities are combined, we introduce instance-pair-adaptive weights that modulate modality contributions while ensuring that no modality is ignored. When involved in contrastive training, SCORE improves joint retrieval while preserving strong pairwise performance under variable-modality inference. For instance, on MSR-VTT, it reaches 55.8 R@1 for text-to-video/audio/subtitles retrieval and 54.0 R@1 for text-to-video retrieval. Code available at: https://anonymous.4open.science/r/SCORE-A20F.