acceptodds
Under review as a conference paper at ICLR 2027

Multimodal Representation Alignment based on Spherical Concentration

Abstract

Multimodal retrieval often requires more flexibility than standard full-modality inference: the same representation space should support pairwise retrieval when only two modalities are available, and joint retrieval when richer multimodal evidence is present. Recent multi-vector similarities effectively address joint retrieval, but variable-modality inference remains challenging, as the learned embeddings must remain useful across different modality subsets. We propose SCORE, a multi-vector similarity that measures the spherical concentration of normalized modality embeddings through a weighted resultant vector. This additive formulation provides a simple score for any available subset of modalities and encourages embeddings to concentrate around a shared consensus direction. To improve flexibility when multiple modalities are combined, we introduce instance-pair-adaptive weights that modulate modality contributions while ensuring that no modality is ignored. When involved in contrastive training, SCORE improves joint retrieval while preserving strong pairwise performance under variable-modality inference. For instance, on MSR-VTT, it reaches 55.8 R@1 for text-to-video/audio/subtitles retrieval and 54.0 R@1 for text-to-video retrieval. Code available at: https://anonymous.4open.science/r/SCORE-A20F.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal alignment, multi-vector similarity, modality gap

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XmKDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces SCORE (Spherical Concentration of Representations), a multi-modal representation alignment method based on spherical concentration. Unlike geometric metrics used in GRAM or TRIANGLE, SCORE measures cross-modal consistency by computing the length of a weighted sum of normalized modal vectors. The authors also propose Instance-pair Adaptive Weighting to dynamically adjust modal contributions. Experiments show that SCORE narrows the modality gap, improves clustering and classification, maintains competitive retrieval performance, and features low computational complexity.

Critical strengths
  1. SCORE eliminates the need for extra fusion modules, supporting both pairwise and joint retrieval with unified embeddings. It also avoids the high computational costs of calculating Gram determinants or SVD.

  2. Evaluated across video-text retrieval benchmarks, modality gap analyses, clustering, classification, and efficiency tests, with ablation studies validating the adaptive weighting mechanism.

Critical weaknesses
  1. According to Equation (9), SCORE's square equals a weighted sum of pairwise cosine similarities among all modalities. Under uniform weighting, it maps monotonically to the average pairwise cosine similarity. Consequently, SCORE is mathematically close to simple all-pair cosine aggregation. Without a direct all-pair cosine similarity baseline, the incremental value of the new similarity function remains unclear.

  2. Validation of representation organization is restricted to limited datasets, improvements in shared space organization focus primarily on the MSR-VTT dataset. Furthermore, due to missing checkpoints, PMRL is excluded from clustering and classification comparisons, leaving SCORE's generalizability across diverse data distributions unconfirmed.

Questions for authors
  1. Given its mathematical link to pairwise cosine similarities (Eq. 9), can the authors include a baseline that directly optimizes all-pair cosine similarity to isolate SCORE's exact advantage?

  2. Can the authors evaluate clustering, classification, and modality gaps beyond MSR-VTT, and include PMRL or other strong baselines for comparison?

  3. Why does adaptive weighting harm specific pairwise retrieval tasks while aiding joint retrieval? Can the authors analyze the learned modal weights?

5 comments

On weakness 1 and question 1.

We thank the reviewer for giving us the opportunity to clarify this point. Under uniform weights, Eq. (9) implies that ℓSCORE-U2\ell^2_{\text{SCORE-U}} is an affine (hence monotone) function of the average pairwise cosine similarity. Nevertheless, SCORE differs from a plain pairwise-cosine sum in two respects.

  • Learning dynamics: Relative to a plain sum of pairwise cosines, the SCORE gradient carries an additional instance-dependent factor 1/∥r∥21/\|\mathbf{r}\|_2: This relatively amplifies the gradient for tuples exhibiting weaker multimodal agreement, whereas a plain pairwise-cosine sum lacks such adaptive scaling. Importantly, this nonlinear transformation also changes the relative contributions of positive and negative logits within InfoNCE, leading to different contrastive optimization dynamics.

  • Interface for adaptive weighting: SCORE directly assigns weights to individual modalities through r=∑kwkzk\mathbf{r}=\sum_k w_k\mathbf{z}_k, naturally defining a weighted consensus direction. Expanding the squared resultant reveals that these modality-level weights induce structured pairwise interactions, with coefficients wpwqw_pw_q. Thus, rather than independently assigning weights to every modality pair, SCORE uses a compact, factorized parameterization in which adapting a single modality weight consistently modulates all its pairwise interactions. This provides a natural interface for instance-pair-adaptive weighting while preserving the interpretation of a shared multimodal consensus.

To directly address the Reviewer's request, we conducted additional experiments on MSR-VTT, including both VAST, which employs pairwise cosine-based alignment, and the explicitly requested all-pair cosine similarity baseline, which directly optimizes the aggregation of cosine similarities across all modality pairs. We evaluated these methods on shared-space organization tasks, including clustering, k-NN classification, and linear probing. The results are reported below:

MethodV-Measure↑K-NN↑Acc↑
VAST(cosine TV-TA)16.8935.0235.59
All-pair-cosine18.6736.2236.88
SCORE20.7338.9838.41

The results show that SCORE consistently outperforms both cosine-based baselines across all three evaluation metrics. In particular, the improvements over the all-pair cosine baseline provide empirical evidence that SCORE's benefits extend beyond simply aggregating pairwise cosine similarities, supporting the effectiveness of the proposed spherical concentration formulation in learning a better-organized shared representation space.

Furthermore, to investigate the generalizability of SCORE across different modalities and varying numbers of inputs, we conducted an additional experiment on REASSEMBLE: A Multimodal Dataset for Contact-rich Robotic Assembly and Disassembly [1], whose results are reported in Appendix I of the revised manuscript. The REASSEMBLE dataset contains multi-modal sensor data, including event cameras, force-torque sensors, microphones, and multi-view RGB cameras, designed for contact-rich manipulation tasks. We select the same encoders and hyperparameters for Cosine and SCORE, and we show the results for classification and retrieval tasks below:

KMethodTop-1Bal AccMacro-F1D2A R@1
3Cosine86.8965.4368.551.42
3SCORE88.9470.0373.821.36
4Cosine94.4781.3784.859.00
4SCORE94.7982.7685.7315.64
5Cosine92.7381.2183.7713.11
5SCORE93.2180.0883.9017.85

SCORE outperforms the cosine baseline in almost all scenarios and metrics, confirming its robustness once again in retrieval as well as in other downstream tasks, different input modalities and varying the number of input modalities.

[1] D. Sliwowski et al., Reassemble: A multimodal dataset for contact-rich robotic assembly and disassembly, RSS International Conference 2025

On weakness 2 and question 2. We thank the Reviewer for the valuable feedback and for the opportunity to further strengthen the experimental evaluation of our manuscript. In the revised manuscript, we have expanded our experimental analysis by introducing an additional evaluation dataset, VGGSound+, to further assess the improvements in shared-space organization. Furthermore, we have extended the comparison by including two additional baseline methods, VAST and SCORE-Uniform. The updated results are presented in the revised Table 1, providing a more comprehensive evaluation of the proposed approach.

MethodV-Measure↑K-NN↑Acc↑
MSR-VTT
VAST16.8935.0235.59
GRAM16.7629.9437.28
TRIANGLE17.8930.5035.59
SCORE-Uniform20.4437.8138.32
SCORE20.7338.9838.41
VGGSound+
VAST65.0223.5038.50
GRAM65.2522.8340.50
TRIANGLE65.0224.3339.50
SCORE-Uniform65.4922.3340.00
SCORE66.6326.6742.00

Answer to Question 3. We thank the Reviewer for the suggestion. To better understand the behavior of the adaptive weighting mechanism, we analyzed the learned modality weights during training on the MSR-VTT dataset. We observed that the model consistently assigns higher weights to the video modality than to the audio modality, with average weights of approximately 1.20 and 0.84, respectively (within the predefined range of [0.8,1.2]). To interpret this behavior, we manually inspected a subset of retrieval examples where the audio modality received a low weight. Many of these samples exhibited sparse or noisy audio signals, or audio content that was only weakly aligned with the corresponding caption. This observation is consistent with the nature of MSR-VTT, which was originally designed for video-text retrieval and does not guarantee that the audio stream is always informative for the retrieval task. These findings suggest that the adaptive weighting mechanism effectively identifies less informative modalities and reduces their contribution during similarity computation, while emphasizing more reliable modalities. In particular, the largest benefit over SCORE-Uniform is observed in examples where one modality provides limited or noisy semantic information, allowing SCORE to rely more heavily on the modality that better matches the query. Therefore, SCORE optimizes the weights for the joint retrieval and may slightly harm pairwise retrieval for modalities containing low-informative examples. Appendix G reports this interesting discussion. Thanks for the valuable suggestion once again.

Thanks for running the extra experiments. A couple of my original concerns still remain, though.

Specifically, it's still unclear whether SCORE's improvement over all-pair cosine comes from the nonlinear transformation or the adaptive weighting, rather than the similarity formulation itself. Also, simply reporting the average modality weights doesn't explain why adaptive weighting hurts pairwise retrieval—could you provide the actual weight distributions alongside a comparison against fixed weights to demonstrate genuine instance-level adaptation?

Reviewer_QKsSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SCORE, a multi-vector similarity that measures the spherical concentration of normalized modality embeddings through a weighted resultant vector. For a given subset of available modalities, SCORE forms the corresponding (weighted) resultant and returns its length. SCORE aims to learn a better-organized shared multimodal space, reduce the modality gap across all modality pairs, and improve downstream clustering and classification. The authors conduct experiments to evaluate modality gaps, clustering, k-nearest-neighbor classification, linear probing, and multimodal retrieval. The results show that SCORE reduces the modality gap across all pairs relative to the other multi-vector models in the reported MSR-VTT evaluation. SCORE outperforms GRAM and TRIANGLE on V-Measure, K-NN accuracy, and linear-probe accuracy. SCORE is also strong in joint retrieval.

Critical strengths
  1. The scoring design is clearly motivated by geometric property analysis. The paper applies the mean resultant length from directional statistics to multimodal contrastive learning. The authors identify that multiplicative dispersion logits can be satisfied by partial agreement and show that SCORE reaches its maximum only when all modality embeddings with positive weight are codirectional. This provides an interpretable rationale for using SCORE to encourage joint multimodal agreement.
Critical weaknesses
  1. The settings of shared-space evaluation are not sufficiently specified. For Table 1, it's unclear whether the evaluations use the embeddings of each modality separately, pool embeddings from different modalities, or use one fused representation per video. The clustering algorithm used for calculating V-Measure is not specified; the K-NN neighborhood size, distance metric, and construction of the labeled reference and test sets are also unspecified; the training and test split, optimizer, learning rate, the number of training epochs, and batch size for the linear probe are also unclear. These omissions make it difficult to reproduce the reported shared-space improvements and weaken the evidence for the paper's claim of improved semantic organization.
  2. The sources of the shared-space improvements are not sufficiently isolated. Although the paper includes uniform-weight retrieval comparisons, Figure 2 and Table 1 lack corresponding SCORE-Uniform and matched cosine-based baselines. Furthermore, the comparison with equal weights doesn't establish whether the improvement comes from input-dependent adaptation or simply learning a global preference for certain modalities, e.g., assigning greater weight to video than audio. Consequently, the contributions of the scoring and dynamic weighting remain inadequately separated.
Questions for authors
  1. For Table 1, could the authors explain what representations are used for clustering or classification: modality embeddings pooled as separate samples, a single concatenated or fused representation per video, or another construction? What clustering algorithm is used for V-Measure? Could the authors specify the K-NN neighborhood size, distance metric, and which samples constitute the labeled reference set and the test set? For the linear probe, please provide the training and test split, optimizer, learning rate, number of training epochs, and batch size?
  2. To assess whether adaptive weighting contributes to shared-space improvements, could the authors report the performance of SCORE-Uniform model on the evaluations in Figure 2 and Table 1 under matched training and evaluation conditions? For SCORE, GRAM, and TRIANGLE in Figure 2 and Table 1, is DTM enabled, and what is its coefficient λ\lambda? Are these auxiliary-loss settings identical across methods?
6 comments

On Weakness 1 and Question 1. We thank the Reviewer for highlighting this point. We have revised the experimental settings in Section 5.1 and Appendix D.4 to clarify the setting used in Table 1. We independently extract L2-normalized, d-dimensional text, video, and audio embeddings from examined models (VAST, GRAM, TRIANGLE and SCORE) and pool them as separate samples, without fusion or concatenation. In all examined models, the dimension of the latent space is d=512d=512. Each sample contributes three embeddings (one text embedding for the caption, one video embedding for the frames and one audio embedding) that inherit its semantic label. Our objective is to predict semantic class independently of modality.

Clustering. On the MSR-VTT dataset, we apply K-means to the pooled embeddings using 20 clusters, corresponding to the 20 semantic categories which MSR-VTT includes. The same applies to the VGGSound+ dataset, but the semantic classes here are 294. V-measure is computed between the resulting cluster assignments and the ground-truth semantic labels.

k-NN classification. We randomly split the pooled embeddings into 80% labeled reference samples and 20% validation samples, with all three modalities represented in both partitions. We use k=5k=5, cosine distance, and uniform majority voting, and report validation classification accuracy.

Linear probing. On MSR-VTT and VGGSound+, we train a single d-to-20 and d-to-294 linear layer, respectively, on frozen embeddings using an 80/20 random training/validation split, Adam with learning rate 10−310^{-3}, cross-entropy loss, 200 epochs, and batch size 64. Training samples are shuffled, and no learning-rate scheduler is used. Validation accuracy is evaluated every epoch, and the maximum is reported.

Thanks again for raising these concerns. This question has been very useful for improving clarity and reproducibility of our manuscript.

On weakness 2 and Question 2. We would like to thank the Reviewer for this suggestion.

Following the Reviewer's suggestion, we ran additional experiments and updated Table 1 with the cosine-based method (VAST) and SCORE-Uniform in the revised paper. We updated Figure 2 too. However, since SCORE and SCORE-Uniform have pretty similar distributions that mostly overlap, the plots are unclear and unreadable. Thus, we included VAST in Figure 2, and we report here the numerical results for Figure 2, joined with Table 1 too:

MethodV-Measure ↑K-NN ↑Acc ↑Gap T-VGap T-AGap A-V
VAST16.8935.0235.590.6540.5950.759
GRAM16.7629.9437.280.6720.6570.731
TRIANGLE17.8930.5035.590.4760.6991.098
SCORE-Uniform20.4437.8138.320.3810.4410.559
SCORE20.7338.9838.410.3610.4490.547

As expected, VAST has very low performance in clustering and linear probing due to a large modality gap. Conversely, SCORE-Uniform has a modality gap comparable to SCORE and thus obtains comparable results in clustering and linear probing. Nevertheless, thanks to the adaptive weight mechanism, SCORE still consistently outperforms SCORE-Uniform in the largest part of retrieval experiments by 2-4 points, providing the best tradeoff. We have updated the paper accordingly. Thanks for the suggestion!

On Question 2: DTM Setting across models. SCORE, as well as GRAM and TRIANGLE, inherit standard hyperparameter configuration from the VAST architecture. DTM Loss function is therefore enabled and set to λ=0.1\lambda=0.1.

Thank you for the clarifications and additional experiments. For the k-NN and linear-probe results in Table 1, were the 80/20 splits in Appendix D.4 made by source video, so that all captions and modality embeddings from each video stayed in the same partition? Moreover, I suggest including the MSR-VTT all-pairs cosine results and the DTM settings in the manuscript.

We thank the Reviewer for the answer and valuable suggestions.

Regarding the 80/20 splits, yes, they are performed by source video, ensuring that all associated captions and modality embeddings remain within the same partition, thus preventing information leakage between training/reference and validation sets. We have explicitly clarified this in Section 5.1 and Appendix D.4.

Following the Reviewer's suggestion, we have also included the MSR-VTT all-pair cosine results in Table 1 and explicitly specified the DTM settings (λ=0.1\lambda = 0.1) in the manuscript.

We thank the Reviewer again for helping us improve the clarity and reproducibility of our work.

Reviewer_rmMdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on multimodal representation alignment and considers how well of the organization shared embedding space. For this purpose, this paper proposes SCORE, a multi-vector similarity for measuring the spherical concentration of normalized modality embeddings through a weighted resultant vector. In SCORE, an instance-pair-adaptive weighting mechanism is introduced for increasing the expressivity of the joint score by using bounded weights to prevent collapse and preserving multi-way alignment. This paper proves the advantage of SCORE in representation and downstream tasks such as multimodal clustering and retrieval comparing with related works.

Critical strengths
  1. The SCORE multimodal representation alignment method proposed in this paper is innovative. The paper provides a clear description of this method through text and illustrations.
  2. This paper theoretically analyzed the rationality of the SCORE method and experimentally verified that SCORE has advantages over related works in terms of learning the organization of shared spaces and the implementation of downstream tasks.
Critical weaknesses
  1. In the introduction section, this paper may be able to demonstrate the limitations of other multimodal representation alignment methods through illustrations, thereby leading to the research motivation of this paper. If the differences between SCORE and them can be shown in the illustrations, it would be even more vivid.
  2. The purpose of multimodal representation alignment is to perform downstream tasks. Therefore, merely demonstrating the advantages of SCORE from the perspectives of clustering and retrieval seems insufficient. More tasks should be considered.
Questions for authors

In Table 1, GRAM, TRIANGLE and SCORE are multimodal representation alignment methods, while V-Measure, K-NN and Acc are metrics. So, what clustering algorithm is adopted?

3 comments

On weakness 1. We would like to thank the Reviewer for the valuable suggestion. In the revised paper, we have included a new figure highlighting the limitations of multiplicative multi-vector comparison methods and the advantages of SCORE. Thanks again for the suggestion that helped us improve the clarity of the paper!

On weakness 2. Thanks to the Reviewer for this valuable suggestion. In addition to retrieval and clustering, we have also performed linear probing and reported the accuracy in Table 1 for MSR-VTT (already in the paper) and VGG-Sound+ (added in the revised version).

Additionally, we perform a brand new set of experiments on REASSEMBLE: A Multimodal Dataset for Contact-rich Robotic Assembly and Disassembly [1] that contains 5 modalities related to sensor data. The REASSEMBLE dataset [1] contains multi-modal sensor data, including event cameras, force-torque sensors, microphones, and multi-view RGB cameras, designed for contact-rich manipulation tasks. This dataset comprises 781 minutes of recording, four high-level actions, 17 objects, and 68 action-object combinations. We select the same encoders and hyperparameters for Cosine and SCORE, and we show the results for classification and retrieval tasks below:

KMethodTop-1Bal AccMacro-F1D2A R@1
3Cosine86.8965.4368.551.42
3SCORE88.9470.0373.821.36
4Cosine94.4781.3784.859.00
4SCORE94.7982.7685.7315.64
5Cosine92.7381.2183.7713.11
5SCORE93.2180.0883.9017.85

SCORE outperforms the cosine baseline in almost all scenarios and metrics, confirming its robustness once again in retrieval as well as in other downstream tasks, different input modalities and varying the number of input modalities.

We included this additional experiment in the revised paper. Thanks again for the suggestion!

[1] D. Sliwowski et al., Reassemble: A multimodal dataset for contact-rich robotic assembly and disassembly, RSS International Conference 2025

On Question 1. We thank the reviewer for highlighting this lack of details. We would like to exploit this space to further expand the experimental settings in Section 5.1 and Appendix D.4 to clarify the protocol used in Table 1.

We independently extract L2-normalized, d-dimensional text, video, and audio embeddings from examined models (VAST, GRAM, TRIANGLE and SCORE) and pool them as separate samples, without fusion or concatenation. In all examined models, the dimension of the latent space is d=512d=512. Each sample contributes three embeddings (one text embedding for the caption, one video embedding for the frames and one audio embedding) that inherit its semantic label. Our objective is to predict semantic class independently of modality.

Clustering. On the MSR-VTT dataset, we apply K-means to the pooled embeddings using 20 clusters, corresponding to the 20 semantic categories which MSR-VTT includes. The same applies to the VGGSound+ dataset, but the semantic classes here are 294. V-measure is computed between the resulting cluster assignments and the ground-truth semantic labels.

k-NN classification. We randomly split the pooled embeddings into 80% labeled reference samples and 20% validation samples, with all three modalities represented in both partitions. We use k=5k=5, cosine distance, and uniform majority voting, and report validation classification accuracy.

Linear probing. On MSR-VTT and VGGSound+, we train a single d-to-20 and d-to-294 linear layer, respectively, on frozen embeddings using an 80/20 random training/validation split, Adam with learning rate 10−310^{-3}, cross-entropy loss, 200 epochs, and batch size 64. Training samples are shuffled, and no learning-rate scheduler is used. Validation accuracy is evaluated every epoch, and the maximum is reported.

Thanks again for raising this concern. It has been very useful for improving clarity and reproducibility of our manuscript.

Thank you for your rebuttal. I have carefully read it.

We would like to thank the Reviewer for taking the time to read our responses. We are pleased to know that we have addressed all the Reviewer's concerns, and we particularly appreciate the positive assessment expressed regarding our manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.