acceptodds
Under review as a conference paper at ICLR 2027

Back to the Roots? Rethinking Multilingual LLMs through Foundational Learner Vocabulary

43 views

Abstract

Multilingual large language models increasingly serve users across languages, yet their tokenizers fragment the lexical core: the words learners encounter first and rely on most. We introduce a new perspective that connects LLM vocabulary to language learner vocabulary, operationalized through LexCore, a 105-language foundational learner-vocabulary lexicon covering approximately 86.2% of the world's non-English-speaking population. Using LexCore, we study multilingual LLMs from diagnosis to intervention. Across 53 languages and 10 tokenizers, we show that contemporary tokenizers fragment most foundational learner vocabulary, with systematic disparities across resource levels, scripts, and language families. We further show that this fragmentation has a representation-level association: in bilingual retrieval, lexical grounding degrades as target words require more subword pieces, especially for lower-resource languages, while SUM emerges as the most stable composition operator. Building on these findings, we introduce LIFT, a learner-grounded, tokenizer-native vocabulary-injection method that reuses existing tokenizers, selects fragmented learner words under a fairness-aware maximin budget, and provides a worst-language relative-burden guarantee. LIFT improves several fairness-sensitive tokenization, fragmentation, and adoption metrics on FLORES+ and FineWeb2 without retraining tokenizers from scratch. Finally, in a controlled Gemma-3 4B continued-pretraining study, sum-initialized LIFT-3K achieves the best observed quality–parity–efficiency tradeoff among the controlled variants across multilingual QA, bidirectional translation, Information Parity, and decoding efficiency. These results show that foundational learner vocabulary is both a compact diagnostic lens for multilingual tokenizer inequity and a practical basis for targeted, budget-controlled adaptation of existing multilingual LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multilingual LLMs, tokenization, tokenizer fairness, vocabulary fragmentation, learner vocabulary, lexical grounding, multilingual evaluation, vocabulary adaptation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_ETQ9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines multilingual tokenizer fragmentation and cross-language disparities through foundational learner vocabulary, and proposes LIFT as a targeted vocabulary-expansion method. It evaluates tokenization and embedding-based bilingual retrieval using basic vocabulary across multiple languages. Under a fixed vocabulary budget, LIFT selects words to maximize the minimum relative reduction in fragmentation burden across languages. The paper evaluates tokenization efficiency and injected-token utilization on natural text, followed by continued pretraining to assess multilingual QA and translation performance.

Critical strengths
  • Using foundational learner vocabulary as a common reference for multilingual tokenizer coverage is an interesting perspective. The type-level single-token retention analysis reveals coverage disparities without weighting words by corpus frequency, and examines these disparities across resource levels, scripts, and language families. This provides an interpretable diagnostic that complements aggregate corpus-level tokenization metrics.

  • The paper explicitly formulates how a limited vocabulary-expansion budget should be allocated across languages. Rather than maximizing total token savings alone, LIFT maximizes the minimum relative reduction in each language’s initial fragmentation burden. This formulation accounts for differing starting conditions and guarantees a minimum improvement across languages within the defined lexical objective.

Critical weaknesses

Major

  • The paper lacks comparisons between selection methods under the same vocabulary budget. The current Greedy baseline adds all candidates, so its vocabulary expansion differs in size from LIFT. Comparisons with greedy, frequency-based, or random selection from the same candidate pool under matched budgets are needed to establish the added value of the maximin objective.

  • The contribution of the LexCore candidate pool is not isolated. Both LIFT and Greedy draw candidates from LexCore. Without comparisons against candidates extracted from general corpora under the same budget and selection method, it is difficult to determine whether the gains come specifically from learner vocabulary or simply from adding useful words.

  • The authors suggest that LIFT-5K underperforms LIFT-3K because the fixed CPT budget provides insufficient adaptation, but this explanation remains untested. Including smaller budgets, such as 1.5K or 2K, would clarify the vocabulary-budget–performance relationship. Increasing the CPT budget for LIFT-5K would more directly test the proposed explanation.

  • Although training tokenizers from scratch falls outside the paper’s main focus on adapting existing tokenizers, a comparison under the same data and final vocabulary size would strengthen the broader argument for learner-based vocabulary allocation. For example, a 30K tokenizer trained from scratch could be compared with a 25K tokenizer augmented with 5K LexCore entries, measuring compression and cross-language fairness.

  • Downstream validation is limited to Gemma-3 4B. Tokenization outcomes vary across tokenizers, but QA, translation, and efficiency after CPT are evaluated on only one model. The evidence for downstream benefits therefore remains specific to this setting, and validation on another model would strengthen the generalization claim.

Minor

  • The choice of Sum initialization is not validated through downstream comparisons. The embedding-retrieval analysis motivates this choice, but comparisons with Mean or random initialization under the same selected vocabulary and CPT conditions are needed to establish whether Sum is preferable for model adaptation.
Questions for authors
  • Were any matched-budget comparisons conducted against greedy or frequency-based selection from LexCore, or against candidates drawn from general corpora? If available, these results would help distinguish the contribution of the learner vocabulary from that of the maximin selection objective.

  • Under the same vocabulary and CPT budgets, could you compare LIFT with a simple selection baseline and a corpus-derived candidate pool? This would directly address the two main concerns raised in the Weaknesses section.

Reviewer_D2AUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LexCore, a foundational learner vocabulary covering 105 languages and around 86.2% of the world’s non-English-speaking population. The authors show that existing tokenizers fragment most foundational learner vocabulary, with systematic disparities across resource levels, scripts, and language families. They introduce LIFT, a vocabulary-injection method for expanding existing tokenizers using a fairness-aware vocabulary-selection strategy. LIFT improves several fairness-sensitive tokenization, fragmentation, and adoption metrics on FLORES+ and FIneWeb2 without retraining the tokenizers from scratch. They also show that a model further trained with LIFT achieves the best observed results.

Critical strengths
  • The paper introduces LexCore that covers 105 languages and 86.2% of the world’s non-English-speaking population, while even the strongest tokenizer (among those evaluated) preserves only about one-fifth of the foundational learner vocabulary as single tokens.
  • The proposed LIFT for expanding existing tokenizers is interesting.
  • The experiments are thorough, covering multiple languages and tokenizers.
Critical weaknesses
  • The experiments mainly compare LIFT against the original tokenizers. Given that the improvement is modest, it would be better to compare LIFT with stronger baselines and several existing vocabulary adaptation approaches. It would be also interesting to see if the improvement on downstream tasks still holds on larger models.
  • LIFT-3K performs better than LIFT-5K on downstream tasks. Though the authors already discuss this in the results, but it is still unclear how much vocabulary size and training budget affect the performance. How frequently do 3K and 5K injected tokens occur in the 2B CPT corpus? Does the difference in token frequency explain why LIFT-3K performs better than LIFT-5K?
  • Minor: some prior studies [1][2] explore strategies for improving multilingual adaptation through tokenizer design and modification that are worth discussing in the related work.

[1] Abagyan et al. (2026). One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers. ACL 2026.

[2] Minixhofer et al. (2024). Zero-Shot Tokenizer Transfer. NeurIPS 2024

Questions for authors

Refer to the weaknesses

Comments on the paper

1 comment

Thank you for the additional analyses and clarifications. After considering the rebuttal, I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.