acceptodds
Under review as a conference paper at ICLR 2027

There is No Theoretical Curse of Multilinguality For Embedding Space Structure

11 views

Abstract

A central goal of multilingual NLP is to achieve high monolingual performance per language and cross-lingual alignment for large-scale language coverage with a multilingual model. The curse of multilinguality describes the phenomenon of degradation in multilingual model performance as we increase language coverage, posing a threat to the above goal. This paper asks whether multilingual embedding spaces are inherently incapable of achieving perfect multilinguality without a prohibitive increase in required capacity. We first formalize the goal of “perfect multilinguality”, embodied in two multilinguality conditions. We then prove that the minimum dimensionality required for perfect multilinguality grows only logarithmically, as , in the number of languages . That is, we show that there is no theoretical curse of multilinguality for embedding space structure. This suggests that the empirical curse of multilinguality is a result of real world data and training conditions. We back this understanding with a small-scale empirical study. Our paper provides the first theoretical and intrinsic perspective on the curse of multilinguality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
curse of multilinguality, theoretical, dimensionality, embeddings, language scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ewPPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the theoretical bounds on embedding dimensionality for the well-known curse of multilinguality (COM). The authors argue that scaling the embedding dimensionality logarithmically with the number of languages is sufficient to overcome the COM, concluding that there is no theoretical COM for multilingual embedding spaces.

Critical strengths

The study is intriguing. The COM has been studied empirically over the past few years. While it is well known that increasing capacity or dimensionality is an effective way to solve the COM, the theoretical boundaries are rarely studied.

Critical weaknesses

The math and theory are not clear. Many notations are undefined or overloaded, making it hard to assess the framework even though the key idea is valid. Here are some examples:

  1. Definition 1 (lines 134–138) is not clear. What are the specific definitions of c,c′,d,c, c', d, and d′d' in Definition 1? Do (c,c′)(c, c') and (d,d′)(d, d') represent two arbitrary pairs of concepts whose similarity scores are being compared? Furthermore, must c≠c′c \neq c' and d≠d′d \neq d'? Allowing arbitrary pairs (c,c′)(c, c') and (d,d′)(d, d') enforces a global similarity ranking across the entire space, whereas Figure 1 and the empirical metric MS-NNO (Section 5) only evaluate local kk-nearest-neighbor rankings around a single anchor concept (c=dc = d).

  2. Notations are undefined and overloaded in the Similarity Decomposition (Equation 2). What is the exact definition of the relationship on the right-hand side of Equation? The inner product notation ⟨⋅,⋅⟩\langle \cdot, \cdot \rangle is never defined in the Preliminaries, and in Equation (2) it is overloaded across three different spaces of different dimensions in a single line: RDZ+DL\mathbb{R}^{D_Z + D_L} on the left, RDZ\mathbb{R}^{D_Z} for ⟨zc,zd⟩\langle z_c, z_d \rangle, and RDL\mathbb{R}^{D_L} for ⟨uℓ,um⟩\langle u_\ell, u_m \rangle. In addition, the claim that "zcz_c and uℓu_\ell are orthogonal when embedded in XLX^L" (line 241) is not formally defined via zero-padded embeddings.

  3. I suppose the definition xc,ℓ:=11+r2zc⊕uℓx_{c,\ell} := \frac{1}{\sqrt{1+r^2}} z_c \oplus u_\ell is missing parentheses around (zc⊕uℓ)(z_c \oplus u_\ell). As written, only zcz_c is scaled by 11+r2\frac{1}{\sqrt{1+r^2}}, which contradicts Equation (2) and does not yield a unit-norm vector.

  4. In Section 3 (line 122), γ\gamma is defined as the maximum cosine similarity between distinct concepts in ZZ (with γ<1\gamma < 1). However, the proof of Lemma 5 (Appendix A.1, line 765) silently assumes γ>0\gamma > 0, which is never stated in the main text. Later, in Lemma 7 (Appendix A.3, line 918), γ=0.9\gamma = 0.9 is suddenly used to denote the minimum pairwise cosine similarity between language offsets.

  5. Throughout the paper, the symbols ZZ, XLX^L, XℓLX_\ell^L, and UU are used interchangeably to mean both finite sets of embedded points and ambient vector spaces with dimensionality DZD_Z, DXLD_{X^L}, and DLD_L.

Questions for authors

Please see Critical Weaknesses.

7 comments

Thanks for reading! We really appreciate the readability feedback. We have addressed these concerns and changes are highlighted in blue in the updated PDF.

W1 Definition 1 says that the ranking of all pairwise similarity scores should be the same for each language in the multilingual space as in the monolingual concept space. It’s a strong condition for global monolingual structure. This should be true for any arbitrary concepts c,c’,d,d’c, c’, d, d’. It should hold even when $ c = c’$ or $ d = d’ $. This is a theoretical ideal which we use to show that good scaling exists even when we want strict conditions. MS-NNO and other metrics are inspired from the strong theoretical conditions rather than exactly measuring them (399), used to observe the underlying behaviour in empirical settings. Often, what we care about in practice for monolingual structure is the structure of local neighbourhoods, which is why MS-NNO looks at that (1080).

W2 We’ve added the inner product definition to the Preliminaries. Inner product can be used for any two points in the same space; its usage in Equation 2 is mathematically valid to the best of our understanding. We have added the formalization of the orthogonality with 0 padding. We have added visual aids to Equation 2 that may help with understanding it. The idea here is that the inner product in XLX^L can be decomposed into the sum of inner products in ZZ and UU since xc,lx_{c,l} is a direct sum. We would be happy to incorporate a more broken down explanation in the appendix if you feel it would be helpful to the reader.

W3 You’re right - the parenthesis is missing. Added in, thank you!

W4 $ \gamma > 0 $ is justified in A.1 (819). Since it is the maximum permissible cosine similarity for non-degeneracy to hold, it can always be set to a positive number less than 1 with the non-degeneracy condition still holding, even if the observed maximum cosine similarity is negative. The reason $ \gamma $ is the minimum cosine similarity for language offsets in Lemma 7 is because in our construction of language offsets, we set the minimum cosine similarity allowed for language offsets to be 1−1−γr21-\frac{1-\gamma}{r^2}. When r=1r=1 as set in Appendix A.1 (WLOG), it simplifies to γ\gamma. In Appendix A.3 we are now making arguments about this language offset space. We understand that this was hard to understand earlier, and we’ve added a clarification to A.3 making it clearer!

W5 We have clarified our (ab)use of notation for simplicity regarding this in Preliminaries.

Thank you again for spending time with the proofs and the notation. If there are any further concerns with the presentation of the paper, we would be more than happy to fix them.

Reviewer_ewPPOpenReview ↗

Thank you for your feedback

Thank you for your feedback. Unfortunately, I lean toward keeping the score unchanged. The arbitrary assumption is too strong and is hard to prove. Considering a well-defined scope is a better idea. Hopefully, my assessment can help you revise your paper.

Thanks for engaging! Could you perhaps clarify which assumption or scoping issue you mean? This would help us with addressing the concern and improving the work. If the assumption under question is that monolingual structure in terms of global rather than local similarity ranking, we note that our results for both upper and lower bounds of scaling would hold even for the weaker local monolingual structure condition. We are not sure which scoping problem you mean, and would be happy to integrate your feedback on it given clarification.

Reviewer_bTDmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper analyzes the dimensionality required to maintain the structure of a multilingual embedding space as the number of languages increases. It requires preservation of monolingual similarity rankings, strong cross-lingual alignment, and fixed separation between distinct representations, and presents a construction that adds language offsets to a shared concept space.

Critical strengths

It seems to be a strength that the paper states the conditions for preserving monolingual structure, cross-lingual alignment, and separation between representations, and provides a construction and dimensionality bounds for a space satisfying them simultaneously. By comparing eight settings that combine compute budget, sampling, and evaluation languages, it shows the need to distinguish experimental conditions when discussing multilingual performance degradation.

Critical weaknesses

I acknowledge the usefulness of the formalization, but I am not convinced that this theoretical paper establishes a previously unknown fact.

  1. The novelty of the main conclusion seems very limited. The idea of separating shared representations from language-specific components has already been explored in many studies, and logarithmic lower bounds on dimensionality in terms of the number of items to distinguish with fixed separation also appear to be known geometric results. Weller et al. (2026) also use this relationship in a retrieval problem requiring a fixed margin. The previous work and this theorem do not give the same result, and there is a difference in the formalization that simultaneously guarantees monolingual structure and cross-lingual alignment; however, the construction of attaching language tags to a shared concept space is largely predictable from existing perspectives, and I see it as closer to organizing existing intuitions than to discovering an important new fact. Regardless of the difficulty of the proof, I think the novelty of the current contribution falls well short of what is needed to justify acceptance at ICLR.

References:

  • The Geometry of Multilingual Language Model Representations (Chang et al., EMNLP 2022)
  • Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations (Xie et al., EMNLP 2022)
  • On the theoretical limitations of embedding-based retrieval. (Weller, Orion, et al. ICLR 2026)
  1. I think the result that a good representation space exists should be distinguished from the claim that an actual model can learn that space. The theorem starts from a given concept space and does not address the parameter count or learning difficulty of an encoder that maps actual text into that space. It is therefore difficult to use this result alone to attribute actual multilingual performance degradation to data and training conditions or to rule out the possibility that the model lacks sufficient parameter capacity. There is no evidence to support the causal interpretation regarding data and training conditions presented in the Abstract.

  2. The authors fixed embedding dimensionality, which is the key variable in the theory, but the dependence of multilingual interference on data size, sampling, and model size has already been reported by Shaham et al. (2023), and the difference in this study is essentially limited to analyzing structural metrics of the embedding space. It is not even clear what new explanation this difference adds to prior work. Since the number of languages and dimensionality were not varied jointly, it is impossible to determine whether the small increase in dimensionality allowed by the theory alleviates actual performance degradation.

  • Causes and Cures for Interference in Multilingual Translation (Shaham et al., ACL 2023)
Questions for authors

See Critical Weaknesses.

4 comments

We thank the reviewer for noting the theoretical and empirical contributions of our work.

W1: Novelty: We agree with the reviewer that the sphere packing results we use previously existed, and we do not claim novelty for them. Our contribution lies in formalizing the theoretical curse of multilinguality and in the statement we are able to make about it using these bounds. The COM frequently is attributed to a fundamental issue of model capacity; however, our work demonstrates that this is not the case given the small additional dimensionality required to accommodate language information. Our work (a) formalizes a theoretical model of perfect multilinguality, and (b) interprets the COM formally as a statement about dimensionality growth within this context, constructs a perfectly multilingual space, and uses spherical sphere packing bounds to make claims about both its minimum sufficient and necessary dimensionality. The paper also (c) unifies empirical frameworks within which the COM has previously been studied and shows trends for the COM in terms of intrinsic structure. While the theoretical model, space construction, and dimensionality bounds are drawn from existing intuition in multilinguality and results from geometry, this paper formalizes and combines these to provide a novel theoretical lens on the COM.

W2: Learnability of an ideal space: Certainly these two claims should be separated. The focused contribution of this work is to discuss the theoretical existence of a perfectly multilingual space and the theoretical cost of multilinguality. This is why our theoretical argument starts from a given monolingual concept space, which could be any monolingual space of choice. We do not suggest that the way to achieving perfect multilinguality is by explicitly mapping languages onto a certain concept space: this construction is for the sake of our theoretical argument. We also do not comment on whether such a multilingual space is learnable or not. We simply state that if there is an empirical curse of multilinguality but no theoretical curse in terms of capacity, then current paradigms of learning are not achieving the theoretical potential (071). The learning difficulty of an encoder can be understood as a problem with training paradigm, optimization, objectives, architectures, or data: this is what we mean by “data and training conditions”. The gap between theory and practice does exist, and our empirical section shows that the empirical curse should be understood in terms of data and training choices. Understanding this gap is important future work. We hope that we have clarified any misunderstandings regarding our claims.

W3: Fixed dimensionality Regarding fixing dimensionality: The primary goal of our empirical section is to show that the empirical COM is sensitive to configuration, which is observable even without the extra permissible dimensions. We discuss our choice to maintain fixed dimensionality for all models in footnote 6. Briefly: the number of minimally sufficient dimensions in theory is small (A.3), and computing it exactly in practice requires knowing the characteristics of the space such as separation margin. Changing the model size would also introduce confounds for fixed data configurations in terms of training convergence and choice of expanded architecture. Future work can investigate the empirical relationship between required extra dimensions for a given model and added languages, and study its correspondence to our theoretical results. Regarding difference from previous work: Previous works are (a) interested in extrinsic performance metrics and (b) define the COM in various different ways corresponding to various training configurations (386, “Note on previous work”). Our work is (a) interested in embedding space structure and looks at COM for intrinsic metrics as corresponding to a theoretical model of multilinguality, and (b) looks at 8 configurations comprising a superset of several previous works including Shaham et al 2023, and recommends this framing in the future (481). Regarding specific comparison with Shaham et al: this paper shows the effect of proportional presence of target language token count as driving performance preservation for a generative MT task for 15 languages, while we observe quality preservation for intrinsic metrics when absolute token count is maintained even as its proportional presence in the data decreases with the addition of more languages, across 100 languages. This indicates that embedding space structure and extrinsic task performance may not correspond straightforwardly with each other, and may have different behaviour from the perspective of a curse of multilinguality. We leave the exploration of this tension at comparable model and language scales to future work. Shaham et al also show empirically that increasing the model size is beneficial; our work provides a theoretical perspective on this.

Thank you for the responses, and I apologize for my very late reply, as I was re-checking the paper and all the other reviewers' reviews. From now on, I will respond to your replies as quickly as possible.

  1. From what the author mention, the author seem to acknowledge that the mathematical results you use are existing ones regarding novelty. Although I did misunderstand some of your claims, when I re-examined the paper from this perspective, my judgment remains that it is difficult to say that it reveals new facts.

  2. The author said that you do not address learnability, but as far as I checked, I understood the Abstract as attributing the actual performance degradation to data and training conditions. In addition, in the further discussion reply, you said that these conditions include optimization, training objectives, model architecture, and data. If so, wouldn't this refer to almost every factor other than the embedding dimension, making it a claim that cannot be verified? Since this could be interpreted as a claim entirely different from the paper's claim, I ask that you please respond to this point.

  3. The possibility that the model is not large enough to learn such a space still does not seem to be ruled out. Therefore, I strongly argue that this statement should be toned down. I understand why you fixed the dimension, but the core prediction of the theory, that increasing the dimension only slightly is sufficient, has still not been confirmed experimentally. Can you confirm this for me? In addition, the explanation that structural metrics of the embedding space and actual task performance can move differently actually increases my doubts about how much this paper explains the actual performance degradation.

Given this, I will maintain my score and lower the Presentation score. Although there is not much time left in the period, I request a sufficient discussion on these points. I will respond as quickly as possible.

Thanks for responding!

  1. Our paper is about providing a new perspective on the curse of multilinguality rather than providing new mathematical results. In the multilinguality community, the question of inherent bottlenecks on language scaling is relevant and an open question. Our paper is discussing this issue.
  2. The claim is that such a space does exist, and that if we observe empirical degradation with language scaling, that is an issue with the way we are trying to learn it. The point here is simply to situate our work in the literature and reconcile the new theoretical argument showing that it is possible for the curse not to exist, with existing observations from the community that the curse does exist. Why would that be? Probably because current methods (data, models, etc. included) are not getting us there. It is a simple point seeking to state the implications of the theoretical result on empirical understandings of the COM. We apologise if the current wording does not convey this idea and would be very happy to rephrase it as we have done in the updated PDF. We would be very happy to iterate on this.
  3. The point of the empirical section is just to back the understanding that the COM is sensitive to data, training, and evaluation choices. Again, this is contributes to the discussion in the multilinguality community regarding the inevitability of the COM as a property of language scaling. Our result about sufficiency is a theoretical result saying that building a perfect space should be possible in theory - this does not need empirical confirmation, it's just a mathematical fact. In the empirical section, we certainly do not claim that we have actually managed to learn this space in practice (that would be incredible). The idea of the paper is to challenge the idea that the COM is inherent: the empirical section shows its sensitivity to real-world conditions. We leave it to future work to explore and close this gap between theory and practice.

We are happy to talk further!

Thank you for the quick response and the revisions to the paper. Given the discussion timeline, this will probably be my last reply.

  1. Novelty: I understand that the contribution of this paper is not a new mathematical result but a new perspective on COM, and I also agree that whether there is a fundamental bottleneck to language expansion is an open question. However, since the bound used is an existing result and the construction is also within what could be expected from existing intuitions, my assessment that the novelty is still limited remains unchanged.

  2. Causal interpretation: I see that the revised Abstract reflects my point, and I thank you for that. However, the Introduction still contains a sentence with the same intent as the previous Abstract ("any empirical curse of multilinguality for embedding space structure is a consequence of real-world conditions such as data and compute constraints"), so please revise it as well. In addition, in your first response you said that you showed COM is not a problem of model capacity, but the paper treats capacity as the embedding dimension, whereas the COM described in prior work refers to performance degradation in models with a fixed number of parameters. Since you also included the model among the causes in this response, please state explicitly in the main text that this result does not rule out the possibility of insufficient parameter capacity in the encoder. What do you think about this point?

  3. Experiments: I now agree that sufficiency is a mathematical fact. I did not mean to ask you to verify the theorem itself through experiments. My concern was that, because the current experiments vary only the data and compute settings while keeping the dimension fixed, their results do not distinguish between interpretations of the theory. Considering the remaining time, I will not request new experiments, but please limit the interpretation of the experimental results to the observation that COM varies depending on the setting.

In summary, the causal wording in the Abstract has been resolved, but my concerns about the novelty, the same wording remaining in the Introduction and the capacity interpretation, and the connection between the theory and the experiments have not yet been resolved. Therefore, I will increase the presentation and soundness scores, and maintain the current scores for the rest. I thank the authors for the quick response and the revisions to the paper, and I wish you the best of luck.

Reviewer_x5chOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors explore the curse of multilinguality in the embedding space by showing that the embedding must only change by Θ(log⁡L)\Theta(\log L) where LL is the number of languages to maintain concept embeddings satisfying several theoretical constraints. The mathematical results are primarily an application of the standard spherical cap bound.

The authors also provided a small scale empirical study showing that when with certain training and metrics decisions, the curse of multilinguality can be mitigated.

Critical strengths

The authors formalized a difficult to analyze problem and provide empirical results to go along with their theory.

Critical weaknesses

I am not a linguist, but the multi-lingual shared embedding requirements seem very strong. In particular, the monolingual structure requires the similarity rankings to be the same across all languages. Conceptual similarity rankings will almost surely have some differences as language is strongly influenced by culture.

The non-degeneracy condition is somewhat arbitrary and significantly simplified the problem to the standard spherical cap bound.

I'm not sure that the models trained from scratch were able to sufficiently learn embeddings of concepts on a small scale.

Questions for authors

Does anything prevent different language conceptual embeddings from being the same across languages except for the "non-degeneracy" condition?

MS-NNO is between 10 and 20 percent, which seems low. Why is this? Was the model given a sufficient amount of compute/data to learn embeddings well?

3 comments

We thank the reviewer for their time and noting our formalization of a difficult problem. We address the stated weaknesses and questions below:

W1: [Effect of culture]: The idea that word or sentence translation equivalents should be embedded with isomorphic structure is common in prior work regarding cross-lingual representations [1][2][3] and our work builds upon these. Our goal is to introduce a theoretical model of multilinguality based on these existing notions in the field. We agree that this conception of multilinguality is scoped to concepts that have language-agnostic semantics (earlier discussed in Limitations in Appendix C) and leaves out language- or culture-specific relationships. This scoping is now further clarified in the abstract and introduction, and the Limitations section discussing this has been moved up to the main text in the updated PDF (highlighted blue). As stated there, future work can incorporate the cost of language- and culture-specific differences into more nuanced multilinguality conditions and dimensionality cost analyses.

We also note that “concepts” in this work refers to sentence-level concepts as in our experiments rather than words. According to prior linguistic work [4], while it is the case that meaning can vary greatly at the word level, whole utterances, like sentences, tend to stabilize (but not always match exactly) in terms of shared semantic content across languages. This phenomenon still does not fully account for culturally different interpretations of the same sentence which are not covered by the current scope of the model.

[1]Are All Good Word Vector Spaces Isomorphic? (https://aclanthology.org/2020.emnlp-main.257.pdf)

[2] A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings (https://aclanthology.org/P18-1073.pdf)

[3] Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation https://aclanthology.org/2020.emnlp-main.365.pdf

[4] “On linguistic aspects of translation” R. Jakobson, 1959.

W2 and Q1: [Utility of non-degeneracy] It is an important property of a multilingual embedding space to retain distinguishability between two equivalent texts in different languages. Downstream applications often need language information, e.g. if we are decoding from this space or doing cross-lingual or monolingual retrieval, where the language identity of the encoded text is important. This is why we need a non-degeneracy condition - the space should not collapse two inputs that differ either in meaning or in language to the same embedding.

Additionally, in practice, finite machine precision of embeddings means that any two distinct points must be separated by some fixed margin. This assumption and argument is taken from established previous work including Weller et al. (2026) (078, footnote 5). This gives us the non-degeneracy condition. The above motivation is more explicitly provided in the updated PDF (Section 3, “Non-degeneracy”).

W3 and Q2: [Do our models learn concepts; MS-NNO is low] At random, MS-NNO would be 3.983.98%, showing that our models learn some monolingual structure (~10-18%). Our experiments look at 100 languages in a data-controlled manner, forcing us to work at a small data and model scale when training from scratch (discussed in Limitations). We are interested in observed trends rather than absolute performance.

Thank you for your constructive feedback, which has helped strengthen our work. Please let us know if any points need further clarification.

To retain the identity of the language why is it more reasonable to use this minimum distance requirement rather than using a single dimension to hold the language index e.g. i=1,2,3,...? This could hold the same information but in a single dimension. In your theoretical analysis, the only growth in embedding dimension is from identifying the language. However these assumptions seem like they are simplifying the problem so much that the results no longer provide any insight to the original problem. It is intuitive that a significant portion of the dimension growth should be from the interaction of language specific relationships and quirks.

[Why need minimum distance?] Suppose you did have a single dimension to represent language identity somehow and the remaining part of the embedding was identical. So xc,l1x_{c,l_1} and xc,l2x_{c,l_2} only differ in one dimension. Crucially, they still differ from each other: they cannot collapse onto the same point i.e. there is some distance between any two points that differ either in concept or language. Since we usually think in terms of cosine distances for embedding spaces, we enforce a fixed resolution scale in terms of a minimum cosine distance. That is all that non-degeneracy says.

[Why isn't a single dimension enough?] Your proposal is reasonable, but there are some problems coming from the fact that you've now imposed an ordering on languages, so some languages are closer to each other than others, and you've changed the norm of the embedding, so we're no longer in a unit norm space. Firstly, you have to show that this maintains cross-lingual alignment: a word in language 1 should be closer to its translation equivalent in language 7000 than any different concept other word in language 1. Translation equivalents in language 1 and 2 seem close to each other, but you have to prove that translation equivalents in language 1 and 7000 are close enough. Secondly, suppose you norm this space, you will find that as you keep adding points to the single dimension per language, the squashed normalised points no longer maintain the resolution scale you wanted. Some intuition from combinatorics to help with understanding: if you have a bounded embedding space with finite precision, there is a finite number of points you can fit. That number is exponential in the number of dimensions. If you want to fit LL points, you will need logLlog L dimensions. The value in our theoretical argument lies in formalising the theoretical COM as a question of asymptotic dimensionality growth, providing a construction that exhibits perfect multilinguality as per the intuition of the field, and showing its scaling behaviour based on existing results from geometry.

[Oversimplified the problem] Many or most studies in the field in cross-lingual alignment and multilingual representation spaces care about shared concepts and language-neutral representations [1][2]. Another body of work is interested in the idea of low-rank language-specific components to any embedding [e.g. 3], in simply using linear steering vectors to go between languages [4] and other such ideas which deal in the ideal premise of universal representations for shared concepts. Studies of real embedding spaces on the other hand show that they often fail to share concept representations for shared concepts, degrade in monolingual quality for low-resource languages [5], and cluster by token frequency or typology as shown [6]. These are some of many works that try to understand the mechanics and flaws of multilingual representation spaces on shared concepts.

The above body indicates that there is value in formalising what perfect multilinguality looks like for shared concepts, in making a theoretical argument about what it costs, and in measuring it empirically as we do. While it may be intuitive to believe that in reality, the cost of more languages only comes from language-specific quirks, this is only the case if we have mastered language-universal representations for shared concepts, which we have not. Language specific phenomena may certainly contribute to the cost. This does not contradict our model, simply expands its scope in future work that can build upon these ideas.

[1] Understanding Cross-Lingual Alignment—A Survey (https://aclanthology.org/2024.findings-acl.649/)

[2] On the Language Neutrality of Pre-trained Multilingual Representations (https://aclanthology.org/2020.findings-emnlp.150/)

[3] Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations (https://aclanthology.org/2022.emnlp-main.379/)

[4] Language Steering for Multilingual In-Context Learning (https://arxiv.org/pdf/2602.02326v1)

[5] On the Limitations of Unsupervised Bilingual Dictionary Induction (https://aclanthology.org/P18-1072/)

[6] The Geometry of Multilingual Language Models: An Equality Lens (https://arxiv.org/pdf/2305.07839)

We really appreciate that you are taking the time to engage further with the work. We are very happy to help with any further clarifications about the work or its contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.