acceptodds
Under review as a conference paper at ICLR 2027

Improving Multilingual Semantic Alignment with Multi-Positive Contrastive Learning for Information Retrieval

Abstract

Multilingual embedding models do not always retrieve all semantically equivalent relevant documents within the top-ranked results. For a given query, relevant documents in some languages are ranked substantially lower than parallel documents written in other languages, and retrieval performance also varies across query languages. In this paper, we introduce Multilingual Semantic Alignment (MuSA), a training method that aims to enhance semantic similarity among parallel document embeddings while jointly optimizing retrieval over multilingual positives associated with each query. Beyond query–document relevance, MuSA directly aligns the embeddings of multilingual parallel documents through a dimension-wise cross-correlation alignment objective. Experiments with 13 multilingual embedding models show that MuSA improves multilingual retrieval performance in settings with multiple references and reduces performance disparities across query languages observed in the baselines. Across the evaluated settings, MuSA reduces the depth required to retrieve all relevant documents by up to 84.62% relative to the corresponding baselines. Analysis shows that improvements in multilingual retrieval are associated with better alignment of parallel document embeddings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multilingual Alignment, Information Retrieval, Multilingual Embedding, Multilingual Information Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_8VFNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Method

The authors present MUSA (Multilingual Semantic Aligner) to improve semantic alignment among multilingual documents. The same query can perform differently across documents represented in different languages (also shown in Observations 1 and 2).

The proposed loss function is:

L=LLSEP(1)+λLBTL = L_{LSEP}^{(1)} + \lambda L_{BT}

I introduce LLSEP(1)L_{LSEP}^{(1)} here to align with their proposed linear approximation of the original LLSEPL_{LSEP} paper. The linear approximation helps reduce the complexity from O(pn)O(pn) to O(p+n)O(p+n). Here, the authors note that a high positive distance prevents other positives from learning better in the original loss function, but the same is still true here.

The LBTL_{BT} term is the original Barlow Twins loss function, which minimizes cross-correlation between languages and maximizes correlation within languages. Again, this is not convincing and needs more analysis of the contribution of the cross-correlation term, especially since there are languages where strong correlation helps. Utilizing distances between languages would have been a useful analysis to better understand the choice of μ\mu (the weight of the cross-correlation term).

Results

The authors show improvements over baseline models on XQuAD and BELEBELE. However, improvements over the baselines are expected. The comparisons with other similar models are too close, and in my opinion, highlighting this method misrepresents the gains in performance (no confidence intervals across different seeds, or even intervals over a single seed, are shown).

Important analyses are also missing, including ablations on the choice of μ\mu, understanding how each loss term progresses during training, and correlation analysis between languages.

What would make this a good paper

  1. Rigorous analysis on what worked, what did not work, the contribution and evolution of the individual loss terms across languages.
  2. Rigorous reporting and comparison with other baseline methods.
  3. Understanding the role of the language and it's relation with other languages on what cross-languages can be learnt.
Critical strengths
  1. The linear approximation is suitable
Critical weaknesses
  1. Poor motivations of choices which are worsened with no real analysis or ablations of why their method works.
  2. Poor presentation of comparison with other methods where the numbers of the proposed method and the best one are very close.
  3. Limited novelty apart from the linear approximation
  4. A lot of related work is missing (contrastive training in multilingual setups have been done a lot)
Questions for authors

N/A

11 comments
AuthorsOpenReview ↗

Response to Reviewer 8VFN (1)

We appreciate your positive assessment of the efficient loss computation and the paper's clarity. Below, we explain the respective roles of ranking supervision and direct alignment through the ablations, comparisons with recent methods, and additional alignment analysis. We hope this provides a clearer account of how MuSA addresses the omission of relevant documents in multilingual retrieval.

 

W1

The interpretation that BT “minimizes correlations between languages and maximizes correlations within languages” differs from the actual objective. In the paper's C∈RD×D\mathcal C\in\mathbb R^{D\times D}, i,ji,j index embedding dimensions, not languages. Driving Cii→1\mathcal C_{ii}\to1 aligns corresponding dimensions of semantically equivalent documents across languages, whereas Cij→0;(i≠j)\mathcal C_{ij}\to0\\;(i\ne j) discourages redundancy between distinct dimensions across the two views. The loss therefore does not remove useful relationships across languages. The coefficient μ\mu controls regularization of redundancy across dimensions, not distances between language pairs. This design is needed because high retrieval scores for the same query do not determine agreement among document representations. MuSA uses LSEPair to learn the ranking of multilingual positives and BT to provide direct alignment supervision between semantically equivalent relevant documents.

Table 3 isolates the contribution of direct alignment: MuSA achieves higher Comp@20 and lower Max@R than w/o BT across all five backbones and both benchmarks. On BELEBELE, adding BT improves Comp@20 by 5.68pp for BGE-M3 and 1.82pp for Qwen, with gains in each of the 18 query languages for both models. Figure 4 complements these ablations by showing changes in alignment distance and Comp@20 from Baseline to MuSA. The submitted paper thus provides evidence for both retrieval benefits and changes in representation space. Section 5 below extends this analysis by comparing alignment with and without BT.

   

W2

The key comparison is not whether a base model benefits from additional training, but whether MuSA retrieves multilingual relevant sets more completely than trained alternative objectives. In submitted Table 3, mMPNet improves Comp@20 over SupCon by +3.49pp on XQuAD and +3.58pp on BELEBELE. BGE-M3 improves by +2.31pp on BELEBELE, and Qwen improves over XLCO by +3.68pp and +2.25pp, respectively. MuSA also has lower Max@R than XLCO, SupCon, and w/o BT for all five backbones on both benchmarks. Comp@20 is the proportion of queries whose relevant documents are all retrieved within the top-20, and Max@R is the rank of the last relevant document. Improvements therefore represent concrete retrieval benefits: complete retrieval of the relevant set and an earlier last relevant document. In the ablations for individual languages, Comp@20 improves over w/o BT in 139 of 150 settings defined by the model, benchmark, and query language and Max@R in 146, showing that the effect is not confined to a particular language or an aggregate average.

Against CLEAR[1] trained with the same source mMARCO data and backbones, MuSA improves BGE-M3's Comp@20 by 9.79pp on XQuAD and 4.01pp on BELEBELE, and Qwen's by 1.98pp on XQuAD. Max@R is also lower for both backbones on both benchmarks. Against SHIFT[2] on XQuAD, MuSA achieves higher Comp@20 and lower Max@R in each of the four query languages with both backbones.

[1] Lee et al. (2026). CLEAR: Cross-Lingual Enhancement in Alignment via Reverse-training.
[2] Jang et al. (2026). SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval.

AuthorsOpenReview ↗

Response to Reviewer 8VFN (2)

W2 (continued)

Additional comparison with CLEAR

ModelMethodXQuAD Comp@20 ↑XQuAD Max@R ↓BELEBELE Comp@20 ↑BELEBELE Max@R ↓
BGE-M3Baseline64.1542.5050.07103.85
BGE-M3XLCO68.2434.6059.6678.69
BGE-M3SupCon72.6630.5060.8073.93
BGE-M3CLEAR63.5441.4659.1087.47
BGE-M3MuSA73.3330.1063.1172.58
BGE-M3w/o BT70.6833.2057.4381.43
Qwen3-Embed-0.6BBaseline43.0685.2031.95188.25
Qwen3-Embed-0.6BXLCO46.7973.2039.20156.97
Qwen3-Embed-0.6BSupCon45.1384.4033.84178.32
Qwen3-Embed-0.6BCLEAR48.4968.9442.80150.50
Qwen3-Embed-0.6BMuSA50.4767.0041.45146.25
Qwen3-Embed-0.6Bw/o BT49.9269.8039.63153.19

 

Comparison including SHIFT

We apply SHIFT to the original models using public language vectors for en, hi, vi, and zh, while retaining the full multilingual document corpus. Because this covers fewer query languages than the comparison above, we report all methods separately for these same four query languages.

Comp@20 ↑

ModelMethodenhivizh
BGE-M3Baseline65.6360.5062.7766.81
BGE-M3XLCO69.4164.0368.0770.25
BGE-M3SupCon75.2167.3173.1974.20
BGE-M3CLEAR64.8759.1664.0364.20
BGE-M3SHIFT65.9761.1863.4567.39
BGE-M3MuSA75.8067.8273.4576.30
BGE-M3w/o BT73.4566.2270.1772.02
Qwen3-Embed-0.6BBaseline54.1239.7545.8848.24
Qwen3-Embed-0.6BXLCO55.2142.7748.4052.10
Qwen3-Embed-0.6BSupCon54.2939.5047.8251.09
Qwen3-Embed-0.6BCLEAR57.1445.3850.8455.04
Qwen3-Embed-0.6BSHIFT49.5837.9843.6145.46
Qwen3-Embed-0.6BMuSA59.6644.4552.4456.22
Qwen3-Embed-0.6Bw/o BT58.5745.1352.1855.97

   

Max@R ↓

ModelMethodenhivizh
BGE-M3Baseline35.3054.6046.6037.80
BGE-M3XLCO29.4043.9036.3031.50
BGE-M3SupCon25.2037.5030.7026.80
BGE-M3CLEAR35.5958.9746.1338.67
BGE-M3SHIFT35.2656.8246.4539.44
BGE-M3MuSA24.8037.3031.0026.70
BGE-M3w/o BT28.0040.9034.7029.80
Qwen3-Embed-0.6BBaseline39.70108.3064.8047.70
Qwen3-Embed-0.6BXLCO42.0095.7058.9045.10
Qwen3-Embed-0.6BSupCon44.80117.3068.6048.10
Qwen3-Embed-0.6BCLEAR42.7599.6159.4346.32
Qwen3-Embed-0.6BSHIFT50.94130.0173.5756.25
Qwen3-Embed-0.6BMuSA38.0091.2055.3040.30
Qwen3-Embed-0.6Bw/o BT39.6094.7057.9042.40

   

W3

We appreciate your positive assessment of the efficient loss computation. The contribution of this work extends beyond computational efficiency. We identify the omission of relevant documents in some languages even by strong multilingual models and validate the distinct roles of query-based ranking supervision and direct alignment among relevant documents. The efficient LSEPair computation is an exact reformulation that preserves the original objective and its gradients. MuSA uses it to learn the ranking of multilingual positives, while BT directly aligns semantically equivalent documents.

This reformulation retains the original LSEP weighting that gives stronger signals to difficult positives. MuSA additionally addresses a different issue: relevant documents can receive high scores for the same query while their representations remain far apart. BT explicitly constrains this relationship, which ranking supervision alone does not determine. Compared with w/o BT, which retains the same ranking objective, MuSA achieves higher Comp@20 and lower Max@R across all five backbones and both benchmarks. These results support a retrieval benefit from direct alignment that is distinct from computational efficiency.

AuthorsOpenReview ↗

Response to Reviewer 8VFN (3)

W4

The submitted Related Work distinguishes cross-lingual pretraining and sentence representation learning[3][4], multilingual retrieval with contrastive learning[5], alignment for retrieval[6], and reverse-training[1]. Our comparisons with the XLCO and SupCon baselines, supplemented by CLEAR and SHIFT[1][2], cover established contrastive objectives and recent alignment approaches. We do not claim multilingual contrastive learning as a new concept. We address retrieval failures that persist after applying it.

The distinction between multi-monolingual IR and multilingual IR is central to our scope.[7] The former performs monolingual retrieval separately in several languages, whereas the latter ranks documents from different languages within a shared candidate pool. The issue is not how many languages a model supports, but whether relevance takes precedence over language in a shared ranking. Success in separate monolingual retrieval settings does not establish this ability. MuSA targets relevant documents being ranked too low or omitted in multilingual retrieval through multi-positive ranking and direct document alignment. Its contribution is to improve retrieval of the complete relevant set regardless of language, rather than to reaffirm the general utility of contrastive learning.

[1] Lee et al. (2026). CLEAR: Cross-Lingual Enhancement in Alignment via Reverse-training.
[2] Jang et al. (2026). SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval.
[3] Chi et al. (2021). InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training. NAACL.
[4] Feng et al. (2022). Language-agnostic BERT Sentence Embedding. ACL.
[5] Hu et al. (2023). Language Agnostic Multilingual Information Retrieval with Contrastive Learning. Findings of ACL.
[6] Hong et al. (2026). Improving Semantic Proximity in Information Retrieval through Cross-Lingual Alignment. ICLR.
[7] Lawrie et al. (2025). NeuCLIRBench: A Modern Evaluation Collection for Monolingual, Cross-Language, and Multilingual Information Retrieval. arXiv:2511.14758.

     

W5 - Additional analysis of loss contributions and relationships between representations across languages

We examine BT's incremental contribution through retrieval performance and multilingual representation alignment. We compare w/o BT and MuSA under the same LSEPair ranking objective, measuring mean squared L2 distance between normalized embeddings of corresponding passages for each document language and all other languages. On BELEBELE, adding BT reduces alignment distance from 0.2317→0.1278 for BGE-M3 and 0.3626→0.3245 for Qwen, while Comp@20 rises from 57.43→63.11 and 39.63→41.45, respectively. Both models also improve average alignment distance and Comp@20 on XQuAD. Thus, smaller differences between representations across languages accompany more queries retrieving the entire relevant set within the top-20.

These alignment improvements are not confined to particular languages or an overall average. Both models have lower alignment distances than w/o BT in all 12 XQuAD and 18 BELEBELE document languages, and higher Comp@20 in all 18 BELEBELE query languages. These results show the benefit of directly aligning multilingual documents that express the same meaning. Whereas Figure 4 shows the overall change from Baseline to MuSA, the comparison between w/o BT and MuSA isolates the incremental contribution of direct alignment.  

XQuAD: Comp@20 ↑

ModelMethodArDeElEnEsHiRoRuThTrViZhAvg.
BGE-M3Baseline64.2064.2964.4565.6363.7060.5063.1967.6564.5462.1062.7766.8164.15
BGE-M3MuSA70.5074.7173.3675.8075.2167.8272.6175.1372.5272.5273.4576.3073.33
BGE-M3w/o BT69.0871.2670.5073.4571.8566.2270.5973.0370.3469.6670.1772.0270.68
Qwen3-Embed-0.6BBaseline40.4248.6628.1554.1248.7439.7539.3349.5039.7534.1245.8848.2443.06
Qwen3-Embed-0.6BMuSA46.6454.9638.4959.6656.9744.4550.7655.9747.4841.6052.4456.2250.47
Qwen3-Embed-0.6Bw/o BT44.4554.2038.2458.5756.3045.1349.4154.9648.9140.6752.1855.9749.92
Reviewer_4Hc5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a problem in multilingual information retrieval in which an embedding model may return the English page near the top but rank a semantically equivalent page when the same information is available in other languages.

It introduces MUSA, a training method that aims to enhance semantic similarity among document embeddings while jointly optimizing retrieval over multilingual positives associated with each query. It claims better semantic alignment among multilingual relevant documents. Experiments across 13 multilingual embedding models spanning different scales and architectures show that MUSA improves multilingual retrieval performance across all evaluated models on both benchmarks.

Critical strengths

This paper is very well organized and presented, integrating clear and logical descriptions of motivations and pre-analysis.

The problem statement is timely, and the method is sophisticated.

The paper shows that, rather than treating a single relevant document as the sole positive target, the method explicitly accounts for multiple semantically equivalent documents across languages.

Critical weaknesses

The work has very limited novelty; it executes the problem more like a sophisticated engineering rather than a scientific contribution in the field. It combines two objectives, which outputs a better result overall but cannot be claimed as a novel work.

Another major concern about this work is in the experimental part. The authors do report how it improves over the base model, but they don’t exactly show if the method improves over the SotA methods, and thus its verify difficult to judge where the method stands in terms of performance.

Though the paper shows a lot of experiments on different datasets, it is quite difficult to assess or understand why this method works. What additional signals does the pretrained model get so that it is more confident about predicting better semantic alignment?

There are a lot of hyperparameters involved in the loss function, and controlling them for optimal results might be a challenge for anyone. How difficult is it to tune these parameters from model to model, and sensitivity as well?

Questions for authors

Comparison with SotA methods is needed.

Intuition of why the method works is not clear.

How to set the hyperparameters?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 4Hc5 (1)

We are grateful for your positive assessment of the timely problem and our explicit treatment of multilingual positives. Below, we clarify the scientific contribution and mechanism, and present comparisons with recent related methods.

   

W1

Our experiments show that even strong multilingual retrieval models do not adequately handle settings where semantically equivalent relevant documents coexist in multiple languages. Relevant documents in some languages are omitted from multilingual candidate pools, so multilingual support alone does not guarantee retrieval regardless of language. MuSA examines whether bringing multilingual positives closer to a query is sufficient, or whether alignment among the relevant documents must also be learned directly. LSEPair trains positives to score above negatives, while BT directly aligns semantically equivalent multilingual positives. Characterizing this as merely combining losses overlooks the identification of a remaining retrieval failure and the validation of learning constraints designed to address it. With the ranking objective held fixed, adding direct alignment improves Comp@20 and reduces Max@R relative to w/o BT across all 5 backbones × 2 benchmarks, providing evidence for this design choice.

   

W2·Q1

We compare MuSA with base models, trained baselines, and recent alignment methods. The experiments on 13 public models assess applicability across backbones, while XLCO, SupCon, and w/o BT in Table 3 assess the contributions of different learning signals. We additionally compare with CLEAR[1], which uses reverse-training, and SHIFT[2], which corrects document representations according to the query language, evaluating how recent, distinct alignment approaches perform in the same multilingual retrieval setting.

Against CLEAR trained with the same source mMARCO data and backbones, MuSA improves Comp@20 by 9.79pp and 4.01pp on XQuAD and BELEBELE with BGE-M3, respectively, and by 1.98pp on XQuAD with Qwen. Max@R is also lower for both backbones on both benchmarks. Against SHIFT on XQuAD, MuSA achieves higher Comp@20 and lower Max@R in all four query languages with both backbones.

[1] Lee et al. (2026). CLEAR: Cross-Lingual Enhancement in Alignment via Reverse-training.
[2] Jang et al. (2026). SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval.

AuthorsOpenReview ↗

Response to Reviewer 4Hc5 (2)

W2·Q1 - Additional Results

 

Additional comparison with CLEAR

ModelMethodXQuAD Comp@20 ↑XQuAD Max@R ↓BELEBELE Comp@20 ↑BELEBELE Max@R ↓
BGE-M3Baseline64.1542.5050.07103.85
BGE-M3XLCO68.2434.6059.6678.69
BGE-M3SupCon72.6630.5060.8073.93
BGE-M3CLEAR63.5441.4659.1087.47
BGE-M3MuSA73.3330.1063.1172.58
BGE-M3w/o BT70.6833.2057.4381.43
Qwen3-Embed-0.6BBaseline43.0685.2031.95188.25
Qwen3-Embed-0.6BXLCO46.7973.2039.20156.97
Qwen3-Embed-0.6BSupCon45.1384.4033.84178.32
Qwen3-Embed-0.6BCLEAR48.4968.9442.80150.50
Qwen3-Embed-0.6BMuSA50.4767.0041.45146.25
Qwen3-Embed-0.6Bw/o BT49.9269.8039.63153.19

   

Comparison including SHIFT

We apply SHIFT to the original models using public language vectors and evaluate en, hi, vi, and zh queries against the full multilingual corpus. The tables below compare all methods on these same four query languages.

Comp@20 ↑

ModelMethodenhivizh
BGE-M3Baseline65.6360.5062.7766.81
BGE-M3XLCO69.4164.0368.0770.25
BGE-M3SupCon75.2167.3173.1974.20
BGE-M3CLEAR64.8759.1664.0364.20
BGE-M3SHIFT65.9761.1863.4567.39
BGE-M3MuSA75.8067.8273.4576.30
BGE-M3w/o BT73.4566.2270.1772.02
Qwen3-Embed-0.6BBaseline54.1239.7545.8848.24
Qwen3-Embed-0.6BXLCO55.2142.7748.4052.10
Qwen3-Embed-0.6BSupCon54.2939.5047.8251.09
Qwen3-Embed-0.6BCLEAR57.1445.3850.8455.04
Qwen3-Embed-0.6BSHIFT49.5837.9843.6145.46
Qwen3-Embed-0.6BMuSA59.6644.4552.4456.22
Qwen3-Embed-0.6Bw/o BT58.5745.1352.1855.97

 

Max@R ↓

ModelMethodenhivizh
BGE-M3Baseline35.3054.6046.6037.80
BGE-M3XLCO29.4043.9036.3031.50
BGE-M3SupCon25.2037.5030.7026.80
BGE-M3CLEAR35.5958.9746.1338.67
BGE-M3SHIFT35.2656.8246.4539.44
BGE-M3MuSA24.8037.3031.0026.70
BGE-M3w/o BT28.0040.9034.7029.80
Qwen3-Embed-0.6BBaseline39.70108.3064.8047.70
Qwen3-Embed-0.6BXLCO42.0095.7058.9045.10
Qwen3-Embed-0.6BSupCon44.80117.3068.6048.10
Qwen3-Embed-0.6BCLEAR42.7599.6159.4346.32
Qwen3-Embed-0.6BSHIFT50.94130.0173.5756.25
Qwen3-Embed-0.6BMuSA38.0091.2055.3040.30
Qwen3-Embed-0.6Bw/o BT39.6094.7057.9042.40

   

W3·Q2

The additional signal is not only that these documents are relevant to the same query, but also that these documents express the same meaning in different languages. High query similarity does not guarantee that two document representations are close to each other. Write normalized document vectors as d1=aq+ud_1=a q+u, d2=aq+vd_2=a q+v, where u,vu,v are orthogonal to the unit query vector qq. Both documents have query similarity aa, but their mutual similarity, d1⊤d2=a2+u⊤vd_1^\top d_2=a^2+u^\top v, can differ. BT directly constrains the representational differences left unresolved by supervision based only on query scores. In the cross-correlation matrix computed from corresponding representations across two linguistic views, BT drives corresponding dimensions toward 1 and non-corresponding dimensions toward 0, encouraging semantic consistency and reducing redundancy across dimensions.

LSEPair supervises the ranking of relevant documents, while BT supervises the consistency of document representations across languages. Figure 4 supports this account directly in the representation space. On parallel documents in 18 BELEBELE languages, MuSA reduces average squared L2 distance and improves Comp@20 for most models. In ablations that retain ranking supervision and remove only BT, MuSA improves XQuAD/BELEBELE Comp@20 over w/o BT by +2.15/+2.40pp for mMPNet, +2.35/+2.72pp for mGTE, and +2.65/+5.68pp for BGE-M3. Thus, the changes in representation space and the incremental gains from direct alignment provide complementary evidence.

AuthorsOpenReview ↗

Response to Reviewer 4Hc5 (3)

W4·Q3

The experiments show that MuSA's core loss configuration is not narrowly tailored to a single backbone. We use the same similarity scale and alignment coefficients across 13 models with different architectures and scales, yet all improve on both retrieval benchmarks. This supports using a shared configuration across diverse backbones, rather than requiring careful tuning for every model to obtain improvements.

The ablations on 5 backbones using the same alignment coefficients also show higher Comp@20 and lower Max@R with BT on both benchmarks. This indicates that the alignment term's benefit does not depend solely on selecting coefficients separately for each model, and that the configuration transfers across models. For practical use, the settings in our study provide a common starting point: similarity scale 50 (τ=0.02\tau=0.02), alignment weight λ=0.02\lambda=0.02, and BT off-diagonal weight μ=5×10−4\mu=5\times10^{-4}.

     

We hope these explanations and comparisons help resolve your questions about the scientific contribution and how MuSA works. If they address your main concerns, we would appreciate your reassessment of our work. Please let us know if you have any further questions. We would be happy to address them.

Reviewer_PgzAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper try to improve the performance of an LAReQA-style multilingual info retrieval problem. The authors propose MUSA, which combines the LSEPair multi-positive loss with a Barlow Twins loss applied to parallel positive pairs (Eq 1). The training data are sampled from mMARCO. Evaluation is conducted on parallel multilingual corpora constructed from XQuAD and BELEBELE, using metrics Comp@20, MAP@20, and Max@R. Additional evaluations are performed on MLQA (for cross-lingual information retrieval) and Wikipedia (for multi-to-monolingual retrieval). The paper claims performance improvements across 13 different embedding models.

Critical strengths
  • The research question is well defined in multilingual retrieval: semantically equivalent documents in different languages may exhibit significant ranking disparities in a mixed-language corpus. Language bias in multilingual search can affect the visibility of non-English documents.
  • The 13 embedding models span the XLM-R family, the decoder family, and various scales. The appendix provides a complete table broken down by language. Good transparency.
  • The ablation design is well-thought-out, with 3 control groups XLCO, SupCon, and w/o BT. Their abalation also includes a representation alignment analysis. Robustness checks for t-SNE were conducted using various perplexities and seeds, reflecting a serious and thorough approach.
Critical weaknesses
  • The proposed evaluation protocol requires retrieving all parallel translations of a relevant document. It may not reflect the objectives of multilingual retrieval systems in reality, which in turn limits the pragmatic virtue of this study.. It remains unclear whether improvements in complete-set retrieval translate into better knowledge accessibility or downstream task performance.
  • For your main contribution, aka Eq 1, I am afraid the novelty is insufficient. Two terms, LSEPair and Barlow Twins (BT), are already well-established, while the author didnt explain their motivation to combine the both onto multilingual document alignment problems. From Tab 3, what I can tell is that the majority of performance gains come from LSEPair, rather than BT. With or Without BT, there is merely ~1pp difference for several models.
  • The main results (Tab 1) compare non-ft base models with yours, which confound the performance gain from ft on mMARCO data or the method per se. Plus, methods like MiMo, SHIFT and CLEAR were cited in your work, but none served as a comparison. The only actual comparison (Tab 3) was between XLCO and SupCon (2021 work), which is not persuasive enough to me.
Questions for authors

See Critical Weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer PgzA (1)

Thank you for recognizing the clear problem formulation and the breadth and transparency of our experiments and ablations. We clarify the evaluation objective and BT's contribution, and provide additional comparisons with recent methods.  
 

W1

Our evaluation assesses whether language limits access to relevant documents. Its purpose is not to present users with redundant translations. For a Chinese query, the desired ranking relation is s(qzh,den+)>s(qzh,dzh−)s(q_{zh},d^{+}_{en}) > s(q_{zh},d^{-}_{zh}), or more generally, s(qℓ,dm+)>s(qℓ,dn−)s(q_{\ell},d^{+}_{m}) > s(q_{\ell},d^{-}_{n}). Here, ss denotes retrieval similarity, and ++ and −- denote relevant and non-relevant documents. Parallel documents reveal how rankings vary with language while controlling for content. Counting retrieval as successful after finding a relevant document in only one language can conceal omissions in other languages. Comp@20 and Max@R therefore measure incomplete retrieval of the relevant set and the rank of the last relevant document. Prior work has likewise used evaluations that control for content to assess whether relevant documents are retrieved regardless of language.[1][2][3]

The practical significance is not the number of translations retrieved, but whether relevant information remains accessible regardless of the language in which it is available. When relevant information is available only in another language, ranking that document below non-relevant documents in the query language can prevent a downstream system from accessing it. Parallel evaluation isolates this failure from content differences, while retrieval of the complete relevant set and retrieval depth directly characterize access across languages. Output language selection and deduplication are subsequent decisions. We therefore assess whether relevant documents can be retrieved regardless of language before those decisions are made.

[1] Roy et al. (2020). LAReQA: Language-Agnostic Answer Retrieval from a Multilingual Pool. EMNLP.
[2] Hong et al. (2026). Improving Semantic Proximity in Information Retrieval through Cross-Lingual Alignment. ICLR.
[3] Jang et al. (2026). MLAIRE: Multilingual Language-Aware Information Retrieval Evaluation Protocol. NeurIPS, Evaluations & Datasets Track.

 
 

W2

The combination is motivated by query–document relevance and multilingual document alignment providing distinct learning signals. LSEPair trains relevant documents to score above non-relevant documents, but does not directly constrain alignment among the relevant documents. Documents can have high similarity to the same query while their representations point in different directions. BT complements this objective by directly constraining dimension-wise cross-correlations between semantically equivalent multilingual document representations. MuSA is therefore not an arbitrary sum of two losses, but a method designed to jointly optimize multi-positive ranking and multilingual representation alignment.

Describing BT's contribution as approximately 1pp does not capture the scope of the ablation results. Relative to w/o BT, Comp@20 improves on XQuAD/BELEBELE by +2.15/+2.40pp for mMPNet, +2.35/+2.72pp for mGTE, and +2.65/+5.68pp for BGE-M3. Moreover, Comp@20 increases and Max@R decreases across all 5 models × 2 benchmarks in the ablations. More queries retrieve the full relevant set, and the last relevant document is ranked earlier. Because w/o BT retains ranking supervision, this comparison isolates the contribution of representation alignment from fine-tuning the base model.

AuthorsOpenReview ↗

Response to Reviewer PgzA (2)

W3

Our initial comparison set prioritized methods that isolate the effect of the training objective using the same backbone and mMARCO training data. XLCO, SupCon, and w/o BT compare contrastive learning, multi-positive learning, and direct alignment. CLEAR's reverse-training focuses on alignment for cross-lingual retrieval, whereas our objective is to retrieve relevant documents in multiple languages together from a multilingual corpus.[1] SHIFT introduces a correction to document embeddings at inference time that depends on the query language,[2] while MIMO incorporates additional supervision through distillation from a large teacher. We therefore initially prioritized separating the contributions of training objectives over comparisons that also introduce these differences between systems.

 
Additional comparison with CLEAR

ModelMethodXQuAD Comp@20 ↑XQuAD Max@R ↓BELEBELE Comp@20 ↑BELEBELE Max@R ↓
BGE-M3Baseline64.1542.5050.07103.85
BGE-M3XLCO68.2434.6059.6678.69
BGE-M3SupCon72.6630.5060.8073.93
BGE-M3CLEAR63.5441.4659.1087.47
BGE-M3MuSA73.3330.1063.1172.58
BGE-M3w/o BT70.6833.2057.4381.43
Qwen3-Embed-0.6BBaseline43.0685.2031.95188.25
Qwen3-Embed-0.6BXLCO46.7973.2039.20156.97
Qwen3-Embed-0.6BSupCon45.1384.4033.84178.32
Qwen3-Embed-0.6BCLEAR48.4968.9442.80150.50
Qwen3-Embed-0.6BMuSA50.4767.0041.45146.25
Qwen3-Embed-0.6Bw/o BT49.9269.8039.63153.19

   

Comparison including SHIFT

SHIFT corrects document embeddings using the language vector for the query language, so this comparison uses en, hi, vi, and zh queries for which public vectors are available. To distinguish this scope from the averages across all query languages reported above, we report all methods, including SHIFT, separately for the same four query languages. The full multilingual document corpus is retained. Only the query languages are restricted. SHIFT is applied to the original models. The tables present results from the submitted paper alongside additional evaluations of CLEAR and SHIFT.

Comp@20 ↑

ModelMethodenhivizh
BGE-M3Baseline65.6360.5062.7766.81
BGE-M3XLCO69.4164.0368.0770.25
BGE-M3SupCon75.2167.3173.1974.20
BGE-M3CLEAR64.8759.1664.0364.20
BGE-M3SHIFT65.9761.1863.4567.39
BGE-M3MuSA75.8067.8273.4576.30
BGE-M3w/o BT73.4566.2270.1772.02
Qwen3-Embed-0.6BBaseline54.1239.7545.8848.24
Qwen3-Embed-0.6BXLCO55.2142.7748.4052.10
Qwen3-Embed-0.6BSupCon54.2939.5047.8251.09
Qwen3-Embed-0.6BCLEAR57.1445.3850.8455.04
Qwen3-Embed-0.6BSHIFT49.5837.9843.6145.46
Qwen3-Embed-0.6BMuSA59.6644.4552.4456.22
Qwen3-Embed-0.6Bw/o BT58.5745.1352.1855.97

 

Max@R ↓

ModelMethodenhivizh
BGE-M3Baseline35.3054.6046.6037.80
BGE-M3XLCO29.4043.9036.3031.50
BGE-M3SupCon25.2037.5030.7026.80
BGE-M3CLEAR35.5958.9746.1338.67
BGE-M3SHIFT35.2656.8246.4539.44
BGE-M3MuSA24.8037.3031.0026.70
BGE-M3w/o BT28.0040.9034.7029.80
Qwen3-Embed-0.6BBaseline39.70108.3064.8047.70
Qwen3-Embed-0.6BXLCO42.0095.7058.9045.10
Qwen3-Embed-0.6BSupCon44.80117.3068.6048.10
Qwen3-Embed-0.6BCLEAR42.7599.6159.4346.32
Qwen3-Embed-0.6BSHIFT50.94130.0173.5756.25
Qwen3-Embed-0.6BMuSA38.0091.2055.3040.30
Qwen3-Embed-0.6Bw/o BT39.6094.7057.9042.40

[1] Jang et al. (2026). SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval.
[2] Lee et al. (2026). CLEAR: Cross-Lingual Enhancement in Alignment via Reverse-training.

   

Our evaluation assesses the ability to retrieve relevant information regardless of language before subsequent selection, rather than the redundant presentation of translations. Alongside the ablation design and analysis by language you positively assessed, we hope the rationale and additional comparisons clarify MuSA's practical relevance and its contribution beyond fine-tuning alone.

AuthorsOpenReview ↗

Dear Reviewer PgzA

As the discussion period draws to a close, we would appreciate your thoughts on our response. We clarified the evaluation’s practical motivation and BT’s contribution beyond ranking supervision, and added comparisons with CLEAR and SHIFT. If these explanations and results address your main concerns, we would appreciate your reassessment of our work. Please let us know if any point needs further clarification.

Reviewer_UZqbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MUSA, a method that trains models on related documents across multiple languages, bringing documents with identical content closer together within the model’s representation space. Experiments conducted across 13 different models demonstrate that this approach reduces the omission of relevant documents and narrows performance gaps between certain languages.

Critical strengths

The experiments cover a diverse range of models, showing consistent overall improvements, which suggests that the method possesses a degree of generalizability. The paper clearly addresses the issue of relevant documents being overlooked across different languages, and the comparative experiments validate the effectiveness of the proposed training objective.

Critical weaknesses

The primary experiments rely on relatively small-scale, perfectly aligned multilingual document pairs; this differs significantly from real-world retrieval scenarios, leaving the method’s practical efficacy somewhat uncertain. Furthermore, as the approach primarily integrates existing techniques, the performance gains over other training methods are occasionally marginal. The lack of reported results from multiple experimental runs also makes it difficult to fully substantiate the specific contribution of the proposed additions.

Questions for authors

Given the modest performance gains over other methods, does the approach remain stable across multiple training runs? Does the method remain effective if the documents are no longer direct translations of one another, or if the retrieval corpus is scaled up significantly?

3 comments
AuthorsOpenReview ↗

Response to Reviewer UZqb (1)

We appreciate your recognition of the consistent gains across diverse models and the clearly defined problem. Your remaining concerns center on practical relevance and the strength of the evidence, which we address below.  
 

W1

Relevant documents should receive higher scores than non-relevant documents, regardless of language. For a Chinese query, the desired relation is s(qzh,den+)>s(qzh,dzh−)s(q_{zh},d^{+}_{en}) > s(q_{zh},d^{-}_{zh}). The reverse relation, s(qzh,dzh−)>s(qzh,den+)s(q_{zh},d^{-}_{zh}) > s(q_{zh},d^{+}_{en}), is a failure in which language matching takes precedence over relevance. Here, ss denotes retrieval similarity, ++ a relevant document, and −- a non-relevant document. More generally, the desired relation is s(qℓ,dm+)>s(qℓ,dn−)s(q_{\ell},d^{+}_{m}) > s(q_{\ell},d^{-}_{n}), without restrictions on the languages m,nm,n of relevant and non-relevant documents. Parallel documents allow us to measure this relation while controlling for content differences. Prior work has likewise used controlled evaluations to examine whether relevant documents are retrieved regardless of language.[1][2][3] Parallelism is therefore not a convenience disconnected from practical retrieval. It separates language bias from content differences to reveal whether relevance takes precedence over language.

[1] Roy et al. (2020). LAReQA: Language-Agnostic Answer Retrieval from a Multilingual Pool. EMNLP.
[2] Hong et al. (2026). Improving Semantic Proximity in Information Retrieval through Cross-Lingual Alignment. ICLR.
[3] Jang et al. (2026). MLAIRE: Multilingual Language-Aware Information Retrieval Evaluation Protocol. NeurIPS, Evaluations & Datasets Track.

   

W2

Even strong publicly available multilingual embedding models fail to retrieve the full set of relevant documents in our setting, where relevant documents in multiple languages must be retrieved together, and their performance varies considerably across query languages. For example, the standard deviation of Max@R across query languages for the public Qwen model is 34.85 on XQuAD and 68.30 on BELEBELE. Strong performance on general retrieval tasks therefore does not guarantee that ranking is independent of language in a multilingual candidate pool. Because query–document ranking alone does not directly constrain alignment among multilingual relevant documents, MuSA combines the complementary supervision provided by the two objectives. Our contribution lies in a training design that addresses this unresolved problem and its validation across backbones and languages.

The gains cannot be attributed solely to fine-tuning a base model. Against trained comparison methods, BGE-M3 exceeds SupCon by 2.31pp in BELEBELE Comp@20, while Qwen exceeds XLCO by 3.68pp and 2.25pp on XQuAD and BELEBELE, respectively. Comp@20 measures the proportion of queries for which all relevant documents are retrieved within the top 20. These gains therefore represent more queries with complete retrieval of the relevant set. These gains reflect improved retrieval completeness beyond strong trained baselines.

   

W3

We isolate BT's incremental contribution by comparing MuSA directly with w/o BT, which retains ranking supervision and removes only the representation alignment term. This ablation covers 5 backbones and both benchmarks. Across mMPNet, mGTE, BGE-M3, Qwen, and PPLX, MuSA has higher Comp@20 and lower Max@R in all 10 combinations of models and benchmarks. This comparison captures the benefit of adding direct alignment to the same ranking objective, rather than simply fine-tuning a base model. More queries retrieve the complete relevant set at high ranks, and the last relevant document is ranked earlier. BT thus complements ranking supervision to improve retrieval completeness.

AuthorsOpenReview ↗

Response to Reviewer UZqb (2)

Q1

Our experiments provide broad evidence that the effect is not confined to a particular trained model or a few languages. We trained and evaluated 13 multilingual embedding models spanning different architectures and scales. Every model improved Comp@20 and reduced Max@R relative to its corresponding base model on both XQuAD and BELEBELE.

The ablations on 5 models further cover 150 settings defined by the model, benchmark, and query language. Comp@20 improves over w/o BT in 139 settings and Max@R in 146. We do not report repeated training runs with different seeds under an identical configuration, so these results should not be interpreted as a measurement of variability across repeated training runs. Nevertheless, the main experiments across 13 models and ablations across 5 models show that the average gains do not depend solely on a particular model or a few languages with large improvements, and that the effect is consistent across diverse trained models and languages.

   

Q2

We chose a parallel setting because holding semantic content constant while varying language makes language bias particularly clear. Parallelism is a diagnostic evaluation condition, not a requirement at retrieval time. Retrieval evaluates query–document relevance rather than translation relationships, so documents that are not translations but provide the requested information can also be relevant. The principle of learning to prioritize relevance over language continues to apply as the corpus grows. MuSA learns encoder representations and query–document ranking relations rather than a correction tied to a particular corpus size. Retrieval does not require constructing translation pairs or realigning the entire corpus. A larger corpus can therefore use the same encoder and retrieval procedure, retaining the objective of ranking relevant documents highly regardless of language.

   

We hope these explanations help address your concerns about practical relevance and the contribution of the additional component. If these points are now clearer, we would appreciate your considering them in your final assessment alongside the experimental strengths you recognized.

Area_Chair_exzaOpenReview ↗

Please respond

Hello,

The discussion period ends soon. Please respond to the authors' last comment and note whether it changed your evaluation.

Thanks,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.