- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces SCORE (Spherical Concentration of Representations), a multi-modal representation alignment method based on spherical concentration. Unlike geometric metrics used in GRAM or TRIANGLE, SCORE measures cross-modal consistency by computing the length of a weighted sum of normalized modal vectors. The authors also propose Instance-pair Adaptive Weighting to dynamically adjust modal contributions. Experiments show that SCORE narrows the modality gap, improves clustering and classification, maintains competitive retrieval performance, and features low computational complexity.
Critical strengths
-
SCORE eliminates the need for extra fusion modules, supporting both pairwise and joint retrieval with unified embeddings. It also avoids the high computational costs of calculating Gram determinants or SVD.
-
Evaluated across video-text retrieval benchmarks, modality gap analyses, clustering, classification, and efficiency tests, with ablation studies validating the adaptive weighting mechanism.
Critical weaknesses
-
According to Equation (9), SCORE's square equals a weighted sum of pairwise cosine similarities among all modalities. Under uniform weighting, it maps monotonically to the average pairwise cosine similarity. Consequently, SCORE is mathematically close to simple all-pair cosine aggregation. Without a direct all-pair cosine similarity baseline, the incremental value of the new similarity function remains unclear.
-
Validation of representation organization is restricted to limited datasets, improvements in shared space organization focus primarily on the MSR-VTT dataset. Furthermore, due to missing checkpoints, PMRL is excluded from clustering and classification comparisons, leaving SCORE's generalizability across diverse data distributions unconfirmed.
Questions for authors
-
Given its mathematical link to pairwise cosine similarities (Eq. 9), can the authors include a baseline that directly optimizes all-pair cosine similarity to isolate SCORE's exact advantage?
-
Can the authors evaluate clustering, classification, and modality gaps beyond MSR-VTT, and include PMRL or other strong baselines for comparison?
-
Why does adaptive weighting harm specific pairwise retrieval tasks while aiding joint retrieval? Can the authors analyze the learned modal weights?
5 comments
Answer to Question 3. We thank the Reviewer for the suggestion. To better understand the behavior of the adaptive weighting mechanism, we analyzed the learned modality weights during training on the MSR-VTT dataset. We observed that the model consistently assigns higher weights to the video modality than to the audio modality, with average weights of approximately 1.20 and 0.84, respectively (within the predefined range of [0.8,1.2]). To interpret this behavior, we manually inspected a subset of retrieval examples where the audio modality received a low weight. Many of these samples exhibited sparse or noisy audio signals, or audio content that was only weakly aligned with the corresponding caption. This observation is consistent with the nature of MSR-VTT, which was originally designed for video-text retrieval and does not guarantee that the audio stream is always informative for the retrieval task. These findings suggest that the adaptive weighting mechanism effectively identifies less informative modalities and reduces their contribution during similarity computation, while emphasizing more reliable modalities. In particular, the largest benefit over SCORE-Uniform is observed in examples where one modality provides limited or noisy semantic information, allowing SCORE to rely more heavily on the modality that better matches the query. Therefore, SCORE optimizes the weights for the joint retrieval and may slightly harm pairwise retrieval for modalities containing low-informative examples. Appendix G reports this interesting discussion. Thanks for the valuable suggestion once again.
Thanks for running the extra experiments. A couple of my original concerns still remain, though.
Specifically, it's still unclear whether SCORE's improvement over all-pair cosine comes from the nonlinear transformation or the adaptive weighting, rather than the similarity formulation itself. Also, simply reporting the average modality weights doesn't explain why adaptive weighting hurts pairwise retrieval—could you provide the actual weight distributions alongside a comparison against fixed weights to demonstrate genuine instance-level adaptation?
On weakness 1 and question 1.
We thank the reviewer for giving us the opportunity to clarify this point. Under uniform weights, Eq. (9) implies that ℓSCORE-U2 is an affine (hence monotone) function of the average pairwise cosine similarity. Nevertheless, SCORE differs from a plain pairwise-cosine sum in two respects.
Learning dynamics: Relative to a plain sum of pairwise cosines, the SCORE gradient carries an additional instance-dependent factor 1/∥r∥2: This relatively amplifies the gradient for tuples exhibiting weaker multimodal agreement, whereas a plain pairwise-cosine sum lacks such adaptive scaling. Importantly, this nonlinear transformation also changes the relative contributions of positive and negative logits within InfoNCE, leading to different contrastive optimization dynamics.
Interface for adaptive weighting: SCORE directly assigns weights to individual modalities through r=∑kwkzk, naturally defining a weighted consensus direction. Expanding the squared resultant reveals that these modality-level weights induce structured pairwise interactions, with coefficients wpwq. Thus, rather than independently assigning weights to every modality pair, SCORE uses a compact, factorized parameterization in which adapting a single modality weight consistently modulates all its pairwise interactions. This provides a natural interface for instance-pair-adaptive weighting while preserving the interpretation of a shared multimodal consensus.
To directly address the Reviewer's request, we conducted additional experiments on MSR-VTT, including both VAST, which employs pairwise cosine-based alignment, and the explicitly requested all-pair cosine similarity baseline, which directly optimizes the aggregation of cosine similarities across all modality pairs. We evaluated these methods on shared-space organization tasks, including clustering, k-NN classification, and linear probing. The results are reported below:
The results show that SCORE consistently outperforms both cosine-based baselines across all three evaluation metrics. In particular, the improvements over the all-pair cosine baseline provide empirical evidence that SCORE's benefits extend beyond simply aggregating pairwise cosine similarities, supporting the effectiveness of the proposed spherical concentration formulation in learning a better-organized shared representation space.
Furthermore, to investigate the generalizability of SCORE across different modalities and varying numbers of inputs, we conducted an additional experiment on REASSEMBLE: A Multimodal Dataset for Contact-rich Robotic Assembly and Disassembly [1], whose results are reported in Appendix I of the revised manuscript. The REASSEMBLE dataset contains multi-modal sensor data, including event cameras, force-torque sensors, microphones, and multi-view RGB cameras, designed for contact-rich manipulation tasks. We select the same encoders and hyperparameters for Cosine and SCORE, and we show the results for classification and retrieval tasks below:
SCORE outperforms the cosine baseline in almost all scenarios and metrics, confirming its robustness once again in retrieval as well as in other downstream tasks, different input modalities and varying the number of input modalities.
[1] D. Sliwowski et al., Reassemble: A multimodal dataset for contact-rich robotic assembly and disassembly, RSS International Conference 2025
On weakness 2 and question 2. We thank the Reviewer for the valuable feedback and for the opportunity to further strengthen the experimental evaluation of our manuscript. In the revised manuscript, we have expanded our experimental analysis by introducing an additional evaluation dataset, VGGSound+, to further assess the improvements in shared-space organization. Furthermore, we have extended the comparison by including two additional baseline methods, VAST and SCORE-Uniform. The updated results are presented in the revised Table 1, providing a more comprehensive evaluation of the proposed approach.