acceptodds
Under review as a conference paper at ICLR 2027

Transfer Runs Through the Lexicon: Sentence Embeddings for Languages Without Bitext

Abstract

Multilingual sentence encoders are trained on parallel text, which most of the languages of the world do not have. Nevertheless, many of these languages do have dictionaries. We study a sentence encoder whose only language-specific component is a lexicon, which can be built by hand with native speakers or collected directly from dictionary entries. A student Transformer is distilled from a pretrained sentence embedding model (SONAR) on English text alone. It represents words using their frequency plus a static word embedding derived from a multilingual concept space. We build that space by propagating ConceptNet vectors over the PanLex translation graph, extended with dictionary entries harvested by an LLM. A new language is added by swapping these static word embeddings via lookup tables, with no training. For languages whose lexicon covers at least 85% of running words it reaches 99.1% xsim and 53.7% xSIM++. On all 216 FLORES+ devtest languages, many of them with small lexicons, the model still reaches 60.8% xsim and 23.9% xSIM++, against 21.9% and 10.1% for a training-free baseline that averages the same concept vectors. A key result of this work is an empirical coverage law: two statistics available before any training explain of the cross-language variance in xSIM++. Deleting known words at random, an intervention that holds the language, the sentences and the encoder fixed, reproduces about four fifths of the slope of the law. Coverage also keeps a partial correlation of 0.90 with quality after controlling for relatedness to high-resource languages (0.30 for relatedness). Finally, we confirm that xsim is saturated for this scenario: it overstates xSIM++ by 36.9 points on average, and it barely registers a 12.8-point xSIM++ difference between two encoders built to differ.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
zeroshot, sentence embedding, multilingual, low-resource languages, sonar

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_rbPPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether multilingual sentence embeddings can be extended to languages without parallel corpora using lexical resources alone. The authors propose a lexicon-swappable sentence encoder distilled from a pretrained SONAR teacher using English text only. Its input representations are derived from a multilingual concept space built by propagating ConceptNet vectors through the PanLex translation graph, augmented with dictionary entries harvested by LLMs. For a new language, the model replaces its static word-embedding lookup table without updating the encoder parameters.

Critical strengths

I think the motivation of this paper is good: The lack of parallel corpora remains a major obstacle to developing NLP technologies for low-resource languages. Dictionaries and word lists are often more accessible than parallel sentences, so investigating whether they can support cross-lingual sentence representations is worthwhile. The proposed method is clean: the lexicon-swappable architecture is soild, the encoder remains fixed, and a new language can be introduced through lookup tables without language-specific model training. This could reduce the computational burden of supporting additional languages.

Critical weaknesses

1.My main concern of this paper is the novelty: The core architecture replaces a conventional token-embedding layer with static multilingual concept vectors and trains a Transformer to reproduce SONAR embeddings. The main technical components: knowledge distillation, graph-based lexical propagation, set encoders, and static cross-lingual embeddings are individually established techniques. The potentially novel element is their combination into a model whose lexical input can be swapped without further training. However, the paper does not sufficiently isolate how much of the improvement comes from this particular architecture versus the quality and alignment of the underlying concept space.

  1. Another possible miscliam is the language-independent transfer. The paper emphasizes languages without bitext and languages unseen by the teacher. However, the authors acknowledge that only 13 of the 216 FLORES+ devtest languages are genuinely outside SONAR's language list. This is a substantial limitation because the intended application is to languages that lack both parallel data and strong overlap with high-resource languages.
Questions for authors

see weakness part

Reviewer_Piq1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method for expanding the set of languages supported by multilingual sentence embeddings using a lexicon without bitext. It distills the SONAR teacher's sentence representations into a student encoder using only English sentences and applies the student to other languages without additional training by replacing the lookup table that maps each language's words to shared concept vectors. It evaluates the method on 216 languages in FLORES+ and analyzes the relationship between lexical coverage and cross-lingual retrieval performance.

Critical strengths

The problem of transfer to low-resource and zero-resource languages addressed by the authors remains largely unresolved and challenging. Also, extending the languages supported by sentence embeddings through a lexicon without bitext seems practical, and the analysis of the relationship between coverage and performance across languages is useful.

Critical weaknesses
  1. The principle of transferring an English model to other languages through a shared word embedding space has already been explored in many prior studies (e.g., Glavaš et al. (2019)), and the approach of distilling a teacher's sentence embedding space was also presented by Reimers & Gurevych (2020). Therefore, in my view, the contribution is closer to a combination of existing methods and lexicon construction than to a new transfer principle. I find that the paper does not demonstrate novelty.
  • How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions (Glavaš et al., ACL 2019)
  • Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation (Reimers & Gurevych, EMNLP 2020)
  1. A strong baseline using the same lexicon is needed. It is difficult to assess the benefits of the new encoder using word vector averaging alone.

  2. Although I have reviewed the paper several times, I still do not know how much it helps with actual unsupported languages. Of the 216 languages, 13 are unsupported, but the average student performance over this entire set is not provided. It is difficult to substantiate usefulness for low-resource, unsupported languages, particularly the 6 languages with the least lexical overlap with existing training languages, based solely on results from high-coverage languages. I recommend that the authors adjust their claims or conduct additional experiments.

  3. It is necessary to disentangle what coverage actually measures. Since identical word forms from other languages are also used, lexical knowledge is mixed with the effects of shared proper names, loanwords, and so on. Also, experiments that remove words from high-coverage languages are not equivalent to adding a lexicon to low-resource languages. The current results are too limited in scope to be interpreted as a general coverage law.

Questions for authors
  • Could you specify the source of the sentences used to compute frequency ranks and the FLORES+ split used to construct the lexicon, and clarify whether devtest sentences or vocabulary were included? If devtest data were used, it is necessary to check whether the results hold on separate text that was not used to construct the resources.
1 comment

Because I re-examined the paper and reviewed all of the reviewers' responses, my score adjustment is somewhat late. I ask for your understanding on this. From now on, we can have an active discussion. I will respond to the authors' replies as quickly as possible.

After reviewing the other reviewers' reviews together, I decided that I should lower my overall score to Clear rejection because of concerns about how the evaluation resources are built and because my own concerns have grown. According to Appendix C, the word vectors for evaluation are selected to match the FLORES+ word list, and words that are still missing are filled in through LLM search. Since both coverage and performance are measured on FLORES+ devtest, this reads as matching the words that appear in the test sentences in advance. Could you provide a response to this? Also, Table 3 shows that this final LLM step alone noticeably improves performance. In addition, the source of the sentences used to compute the word frequency ranking is not disclosed, and their amount is almost the same as the combined size of FLORES+ dev and devtest. If so, the main results and the coverage law may reflect performance under a condition where the words of the test sentences are known in advance, rather than performance on unseen sentences.

In addition, only a small number of languages are not supported by SONAR, and the coverage law has not been separately validated on language families that were not used for fitting.

  1. Please disclose the source of the word lists and of the sentences used for the frequency computation, and tell us whether devtest sentences or words were used.
  2. Please show the main results re-measured after building the lexicon and frequency ranking without using devtest at all.
  3. Please report the average performance on languages not supported by SONAR under the same conditions. Fourth, please fit the coverage law with one language family held out and show how well it predicts the performance of that family.

If you show the above, I will consider adjusting my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.