- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Method
The authors present MUSA (Multilingual Semantic Aligner) to improve semantic alignment among multilingual documents. The same query can perform differently across documents represented in different languages (also shown in Observations 1 and 2).
The proposed loss function is:
I introduce here to align with their proposed linear approximation of the original paper. The linear approximation helps reduce the complexity from to . Here, the authors note that a high positive distance prevents other positives from learning better in the original loss function, but the same is still true here.
The term is the original Barlow Twins loss function, which minimizes cross-correlation between languages and maximizes correlation within languages. Again, this is not convincing and needs more analysis of the contribution of the cross-correlation term, especially since there are languages where strong correlation helps. Utilizing distances between languages would have been a useful analysis to better understand the choice of (the weight of the cross-correlation term).
Results
The authors show improvements over baseline models on XQuAD and BELEBELE. However, improvements over the baselines are expected. The comparisons with other similar models are too close, and in my opinion, highlighting this method misrepresents the gains in performance (no confidence intervals across different seeds, or even intervals over a single seed, are shown).
Important analyses are also missing, including ablations on the choice of , understanding how each loss term progresses during training, and correlation analysis between languages.
What would make this a good paper
- Rigorous analysis on what worked, what did not work, the contribution and evolution of the individual loss terms across languages.
- Rigorous reporting and comparison with other baseline methods.
- Understanding the role of the language and it's relation with other languages on what cross-languages can be learnt.
Critical strengths
- The linear approximation is suitable
Critical weaknesses
- Poor motivations of choices which are worsened with no real analysis or ablations of why their method works.
- Poor presentation of comparison with other methods where the numbers of the proposed method and the best one are very close.
- Limited novelty apart from the linear approximation
- A lot of related work is missing (contrastive training in multilingual setups have been done a lot)
Questions for authors
N/A
11 comments
Response to Reviewer 8VFN (2)
W2 (continued)
Additional comparison with CLEAR
| Model | Method | XQuAD Comp@20 ↑ | XQuAD Max@R ↓ | BELEBELE Comp@20 ↑ | BELEBELE Max@R ↓ |
|---|---|---|---|---|---|
| BGE-M3 | Baseline | 64.15 | 42.50 | 50.07 | 103.85 |
| BGE-M3 | XLCO | 68.24 | 34.60 | 59.66 | 78.69 |
| BGE-M3 | SupCon | 72.66 | 30.50 | 60.80 | 73.93 |
| BGE-M3 | CLEAR | 63.54 | 41.46 | 59.10 | 87.47 |
| BGE-M3 | MuSA | 73.33 | 30.10 | 63.11 | 72.58 |
| BGE-M3 | w/o BT | 70.68 | 33.20 | 57.43 | 81.43 |
| Qwen3-Embed-0.6B | Baseline | 43.06 | 85.20 | 31.95 | 188.25 |
| Qwen3-Embed-0.6B | XLCO | 46.79 | 73.20 | 39.20 | 156.97 |
| Qwen3-Embed-0.6B | SupCon | 45.13 | 84.40 | 33.84 | 178.32 |
| Qwen3-Embed-0.6B | CLEAR | 48.49 | 68.94 | 42.80 | 150.50 |
| Qwen3-Embed-0.6B | MuSA | 50.47 | 67.00 | 41.45 | 146.25 |
| Qwen3-Embed-0.6B | w/o BT | 49.92 | 69.80 | 39.63 | 153.19 |
Comparison including SHIFT
We apply SHIFT to the original models using public language vectors for en, hi, vi, and zh, while retaining the full multilingual document corpus. Because this covers fewer query languages than the comparison above, we report all methods separately for these same four query languages.
Comp@20 ↑
| Model | Method | en | hi | vi | zh |
|---|---|---|---|---|---|
| BGE-M3 | Baseline | 65.63 | 60.50 | 62.77 | 66.81 |
| BGE-M3 | XLCO | 69.41 | 64.03 | 68.07 | 70.25 |
| BGE-M3 | SupCon | 75.21 | 67.31 | 73.19 | 74.20 |
| BGE-M3 | CLEAR | 64.87 | 59.16 | 64.03 | 64.20 |
| BGE-M3 | SHIFT | 65.97 | 61.18 | 63.45 | 67.39 |
| BGE-M3 | MuSA | 75.80 | 67.82 | 73.45 | 76.30 |
| BGE-M3 | w/o BT | 73.45 | 66.22 | 70.17 | 72.02 |
| Qwen3-Embed-0.6B | Baseline | 54.12 | 39.75 | 45.88 | 48.24 |
| Qwen3-Embed-0.6B | XLCO | 55.21 | 42.77 | 48.40 | 52.10 |
| Qwen3-Embed-0.6B | SupCon | 54.29 | 39.50 | 47.82 | 51.09 |
| Qwen3-Embed-0.6B | CLEAR | 57.14 | 45.38 | 50.84 | 55.04 |
| Qwen3-Embed-0.6B | SHIFT | 49.58 | 37.98 | 43.61 | 45.46 |
| Qwen3-Embed-0.6B | MuSA | 59.66 | 44.45 | 52.44 | 56.22 |
| Qwen3-Embed-0.6B | w/o BT | 58.57 | 45.13 | 52.18 | 55.97 |
Max@R ↓
| Model | Method | en | hi | vi | zh |
|---|---|---|---|---|---|
| BGE-M3 | Baseline | 35.30 | 54.60 | 46.60 | 37.80 |
| BGE-M3 | XLCO | 29.40 | 43.90 | 36.30 | 31.50 |
| BGE-M3 | SupCon | 25.20 | 37.50 | 30.70 | 26.80 |
| BGE-M3 | CLEAR | 35.59 | 58.97 | 46.13 | 38.67 |
| BGE-M3 | SHIFT | 35.26 | 56.82 | 46.45 | 39.44 |
| BGE-M3 | MuSA | 24.80 | 37.30 | 31.00 | 26.70 |
| BGE-M3 | w/o BT | 28.00 | 40.90 | 34.70 | 29.80 |
| Qwen3-Embed-0.6B | Baseline | 39.70 | 108.30 | 64.80 | 47.70 |
| Qwen3-Embed-0.6B | XLCO | 42.00 | 95.70 | 58.90 | 45.10 |
| Qwen3-Embed-0.6B | SupCon | 44.80 | 117.30 | 68.60 | 48.10 |
| Qwen3-Embed-0.6B | CLEAR | 42.75 | 99.61 | 59.43 | 46.32 |
| Qwen3-Embed-0.6B | SHIFT | 50.94 | 130.01 | 73.57 | 56.25 |
| Qwen3-Embed-0.6B | MuSA | 38.00 | 91.20 | 55.30 | 40.30 |
| Qwen3-Embed-0.6B | w/o BT | 39.60 | 94.70 | 57.90 | 42.40 |
W3
We appreciate your positive assessment of the efficient loss computation. The contribution of this work extends beyond computational efficiency. We identify the omission of relevant documents in some languages even by strong multilingual models and validate the distinct roles of query-based ranking supervision and direct alignment among relevant documents. The efficient LSEPair computation is an exact reformulation that preserves the original objective and its gradients. MuSA uses it to learn the ranking of multilingual positives, while BT directly aligns semantically equivalent documents.
This reformulation retains the original LSEP weighting that gives stronger signals to difficult positives. MuSA additionally addresses a different issue: relevant documents can receive high scores for the same query while their representations remain far apart. BT explicitly constrains this relationship, which ranking supervision alone does not determine. Compared with w/o BT, which retains the same ranking objective, MuSA achieves higher Comp@20 and lower Max@R across all five backbones and both benchmarks. These results support a retrieval benefit from direct alignment that is distinct from computational efficiency.
Response to Reviewer 8VFN (3)
W4
The submitted Related Work distinguishes cross-lingual pretraining and sentence representation learning[3][4], multilingual retrieval with contrastive learning[5], alignment for retrieval[6], and reverse-training[1]. Our comparisons with the XLCO and SupCon baselines, supplemented by CLEAR and SHIFT[1][2], cover established contrastive objectives and recent alignment approaches. We do not claim multilingual contrastive learning as a new concept. We address retrieval failures that persist after applying it.
The distinction between multi-monolingual IR and multilingual IR is central to our scope.[7] The former performs monolingual retrieval separately in several languages, whereas the latter ranks documents from different languages within a shared candidate pool. The issue is not how many languages a model supports, but whether relevance takes precedence over language in a shared ranking. Success in separate monolingual retrieval settings does not establish this ability. MuSA targets relevant documents being ranked too low or omitted in multilingual retrieval through multi-positive ranking and direct document alignment. Its contribution is to improve retrieval of the complete relevant set regardless of language, rather than to reaffirm the general utility of contrastive learning.
[1] Lee et al. (2026). CLEAR: Cross-Lingual Enhancement in Alignment via Reverse-training.
[2] Jang et al. (2026). SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval.
[3] Chi et al. (2021). InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training. NAACL.
[4] Feng et al. (2022). Language-agnostic BERT Sentence Embedding. ACL.
[5] Hu et al. (2023). Language Agnostic Multilingual Information Retrieval with Contrastive Learning. Findings of ACL.
[6] Hong et al. (2026). Improving Semantic Proximity in Information Retrieval through Cross-Lingual Alignment. ICLR.
[7] Lawrie et al. (2025). NeuCLIRBench: A Modern Evaluation Collection for Monolingual, Cross-Language, and Multilingual Information Retrieval. arXiv:2511.14758.
W5 - Additional analysis of loss contributions and relationships between representations across languages
We examine BT's incremental contribution through retrieval performance and multilingual representation alignment. We compare w/o BT and MuSA under the same LSEPair ranking objective, measuring mean squared L2 distance between normalized embeddings of corresponding passages for each document language and all other languages. On BELEBELE, adding BT reduces alignment distance from 0.2317→0.1278 for BGE-M3 and 0.3626→0.3245 for Qwen, while Comp@20 rises from 57.43→63.11 and 39.63→41.45, respectively. Both models also improve average alignment distance and Comp@20 on XQuAD. Thus, smaller differences between representations across languages accompany more queries retrieving the entire relevant set within the top-20.
These alignment improvements are not confined to particular languages or an overall average. Both models have lower alignment distances than w/o BT in all 12 XQuAD and 18 BELEBELE document languages, and higher Comp@20 in all 18 BELEBELE query languages. These results show the benefit of directly aligning multilingual documents that express the same meaning. Whereas Figure 4 shows the overall change from Baseline to MuSA, the comparison between w/o BT and MuSA isolates the incremental contribution of direct alignment.
XQuAD: Comp@20 ↑
| Model | Method | Ar | De | El | En | Es | Hi | Ro | Ru | Th | Tr | Vi | Zh | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BGE-M3 | Baseline | 64.20 | 64.29 | 64.45 | 65.63 | 63.70 | 60.50 | 63.19 | 67.65 | 64.54 | 62.10 | 62.77 | 66.81 | 64.15 |
| BGE-M3 | MuSA | 70.50 | 74.71 | 73.36 | 75.80 | 75.21 | 67.82 | 72.61 | 75.13 | 72.52 | 72.52 | 73.45 | 76.30 | 73.33 |
| BGE-M3 | w/o BT | 69.08 | 71.26 | 70.50 | 73.45 | 71.85 | 66.22 | 70.59 | 73.03 | 70.34 | 69.66 | 70.17 | 72.02 | 70.68 |
| Qwen3-Embed-0.6B | Baseline | 40.42 | 48.66 | 28.15 | 54.12 | 48.74 | 39.75 | 39.33 | 49.50 | 39.75 | 34.12 | 45.88 | 48.24 | 43.06 |
| Qwen3-Embed-0.6B | MuSA | 46.64 | 54.96 | 38.49 | 59.66 | 56.97 | 44.45 | 50.76 | 55.97 | 47.48 | 41.60 | 52.44 | 56.22 | 50.47 |
| Qwen3-Embed-0.6B | w/o BT | 44.45 | 54.20 | 38.24 | 58.57 | 56.30 | 45.13 | 49.41 | 54.96 | 48.91 | 40.67 | 52.18 | 55.97 | 49.92 |
Response to Reviewer 8VFN (1)
We appreciate your positive assessment of the efficient loss computation and the paper's clarity. Below, we explain the respective roles of ranking supervision and direct alignment through the ablations, comparisons with recent methods, and additional alignment analysis. We hope this provides a clearer account of how MuSA addresses the omission of relevant documents in multilingual retrieval.
W1
The interpretation that BT “minimizes correlations between languages and maximizes correlations within languages” differs from the actual objective. In the paper's C∈RD×D, i,j index embedding dimensions, not languages. Driving Cii→1 aligns corresponding dimensions of semantically equivalent documents across languages, whereas Cij→0;(i=j) discourages redundancy between distinct dimensions across the two views. The loss therefore does not remove useful relationships across languages. The coefficient μ controls regularization of redundancy across dimensions, not distances between language pairs. This design is needed because high retrieval scores for the same query do not determine agreement among document representations. MuSA uses LSEPair to learn the ranking of multilingual positives and BT to provide direct alignment supervision between semantically equivalent relevant documents.
Table 3 isolates the contribution of direct alignment: MuSA achieves higher Comp@20 and lower Max@R than w/o BT across all five backbones and both benchmarks. On BELEBELE, adding BT improves Comp@20 by 5.68pp for BGE-M3 and 1.82pp for Qwen, with gains in each of the 18 query languages for both models. Figure 4 complements these ablations by showing changes in alignment distance and Comp@20 from Baseline to MuSA. The submitted paper thus provides evidence for both retrieval benefits and changes in representation space. Section 5 below extends this analysis by comparing alignment with and without BT.
W2
The key comparison is not whether a base model benefits from additional training, but whether MuSA retrieves multilingual relevant sets more completely than trained alternative objectives. In submitted Table 3, mMPNet improves Comp@20 over SupCon by +3.49pp on XQuAD and +3.58pp on BELEBELE. BGE-M3 improves by +2.31pp on BELEBELE, and Qwen improves over XLCO by +3.68pp and +2.25pp, respectively. MuSA also has lower Max@R than XLCO, SupCon, and w/o BT for all five backbones on both benchmarks. Comp@20 is the proportion of queries whose relevant documents are all retrieved within the top-20, and Max@R is the rank of the last relevant document. Improvements therefore represent concrete retrieval benefits: complete retrieval of the relevant set and an earlier last relevant document. In the ablations for individual languages, Comp@20 improves over w/o BT in 139 of 150 settings defined by the model, benchmark, and query language and Max@R in 146, showing that the effect is not confined to a particular language or an aggregate average.
Against CLEAR[1] trained with the same source mMARCO data and backbones, MuSA improves BGE-M3's Comp@20 by 9.79pp on XQuAD and 4.01pp on BELEBELE, and Qwen's by 1.98pp on XQuAD. Max@R is also lower for both backbones on both benchmarks. Against SHIFT[2] on XQuAD, MuSA achieves higher Comp@20 and lower Max@R in each of the four query languages with both backbones.