- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores the theoretical bounds on embedding dimensionality for the well-known curse of multilinguality (COM). The authors argue that scaling the embedding dimensionality logarithmically with the number of languages is sufficient to overcome the COM, concluding that there is no theoretical COM for multilingual embedding spaces.
Critical strengths
The study is intriguing. The COM has been studied empirically over the past few years. While it is well known that increasing capacity or dimensionality is an effective way to solve the COM, the theoretical boundaries are rarely studied.
Critical weaknesses
The math and theory are not clear. Many notations are undefined or overloaded, making it hard to assess the framework even though the key idea is valid. Here are some examples:
-
Definition 1 (lines 134–138) is not clear. What are the specific definitions of and in Definition 1? Do and represent two arbitrary pairs of concepts whose similarity scores are being compared? Furthermore, must and ? Allowing arbitrary pairs and enforces a global similarity ranking across the entire space, whereas Figure 1 and the empirical metric
MS-NNO(Section 5) only evaluate local -nearest-neighbor rankings around a single anchor concept (). -
Notations are undefined and overloaded in the Similarity Decomposition (Equation 2). What is the exact definition of the relationship on the right-hand side of Equation? The inner product notation is never defined in the Preliminaries, and in Equation (2) it is overloaded across three different spaces of different dimensions in a single line: on the left, for , and for . In addition, the claim that " and are orthogonal when embedded in " (line 241) is not formally defined via zero-padded embeddings.
-
I suppose the definition is missing parentheses around . As written, only is scaled by , which contradicts Equation (2) and does not yield a unit-norm vector.
-
In Section 3 (line 122), is defined as the maximum cosine similarity between distinct concepts in (with ). However, the proof of Lemma 5 (Appendix A.1, line 765) silently assumes , which is never stated in the main text. Later, in Lemma 7 (Appendix A.3, line 918), is suddenly used to denote the minimum pairwise cosine similarity between language offsets.
-
Throughout the paper, the symbols , , , and are used interchangeably to mean both finite sets of embedded points and ambient vector spaces with dimensionality , , and .
Questions for authors
Please see Critical Weaknesses.
7 comments
Thank you for your feedback
Thank you for your feedback. Unfortunately, I lean toward keeping the score unchanged. The arbitrary assumption is too strong and is hard to prove. Considering a well-defined scope is a better idea. Hopefully, my assessment can help you revise your paper.
Thanks for engaging! Could you perhaps clarify which assumption or scoping issue you mean? This would help us with addressing the concern and improving the work. If the assumption under question is that monolingual structure in terms of global rather than local similarity ranking, we note that our results for both upper and lower bounds of scaling would hold even for the weaker local monolingual structure condition. We are not sure which scoping problem you mean, and would be happy to integrate your feedback on it given clarification.
Thanks for reading! We really appreciate the readability feedback. We have addressed these concerns and changes are highlighted in blue in the updated PDF.
W1 Definition 1 says that the ranking of all pairwise similarity scores should be the same for each language in the multilingual space as in the monolingual concept space. It’s a strong condition for global monolingual structure. This should be true for any arbitrary concepts c,c’,d,d’. It should hold even when $ c = c’$ or $ d = d’ $. This is a theoretical ideal which we use to show that good scaling exists even when we want strict conditions.
MS-NNOand other metrics are inspired from the strong theoretical conditions rather than exactly measuring them (399), used to observe the underlying behaviour in empirical settings. Often, what we care about in practice for monolingual structure is the structure of local neighbourhoods, which is whyMS-NNOlooks at that (1080).W2 We’ve added the inner product definition to the Preliminaries. Inner product can be used for any two points in the same space; its usage in Equation 2 is mathematically valid to the best of our understanding. We have added the formalization of the orthogonality with 0 padding. We have added visual aids to Equation 2 that may help with understanding it. The idea here is that the inner product in XL can be decomposed into the sum of inner products in Z and U since xc,l is a direct sum. We would be happy to incorporate a more broken down explanation in the appendix if you feel it would be helpful to the reader.
W3 You’re right - the parenthesis is missing. Added in, thank you!
W4 $ \gamma > 0 $ is justified in A.1 (819). Since it is the maximum permissible cosine similarity for non-degeneracy to hold, it can always be set to a positive number less than 1 with the non-degeneracy condition still holding, even if the observed maximum cosine similarity is negative. The reason $ \gamma $ is the minimum cosine similarity for language offsets in Lemma 7 is because in our construction of language offsets, we set the minimum cosine similarity allowed for language offsets to be 1−r21−γ. When r=1 as set in Appendix A.1 (WLOG), it simplifies to γ. In Appendix A.3 we are now making arguments about this language offset space. We understand that this was hard to understand earlier, and we’ve added a clarification to A.3 making it clearer!
W5 We have clarified our (ab)use of notation for simplicity regarding this in Preliminaries.
Thank you again for spending time with the proofs and the notation. If there are any further concerns with the presentation of the paper, we would be more than happy to fix them.