- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines multilingual tokenizer fragmentation and cross-language disparities through foundational learner vocabulary, and proposes LIFT as a targeted vocabulary-expansion method. It evaluates tokenization and embedding-based bilingual retrieval using basic vocabulary across multiple languages. Under a fixed vocabulary budget, LIFT selects words to maximize the minimum relative reduction in fragmentation burden across languages. The paper evaluates tokenization efficiency and injected-token utilization on natural text, followed by continued pretraining to assess multilingual QA and translation performance.
Critical strengths
-
Using foundational learner vocabulary as a common reference for multilingual tokenizer coverage is an interesting perspective. The type-level single-token retention analysis reveals coverage disparities without weighting words by corpus frequency, and examines these disparities across resource levels, scripts, and language families. This provides an interpretable diagnostic that complements aggregate corpus-level tokenization metrics.
-
The paper explicitly formulates how a limited vocabulary-expansion budget should be allocated across languages. Rather than maximizing total token savings alone, LIFT maximizes the minimum relative reduction in each language’s initial fragmentation burden. This formulation accounts for differing starting conditions and guarantees a minimum improvement across languages within the defined lexical objective.
Critical weaknesses
Major
-
The paper lacks comparisons between selection methods under the same vocabulary budget. The current Greedy baseline adds all candidates, so its vocabulary expansion differs in size from LIFT. Comparisons with greedy, frequency-based, or random selection from the same candidate pool under matched budgets are needed to establish the added value of the maximin objective.
-
The contribution of the LexCore candidate pool is not isolated. Both LIFT and Greedy draw candidates from LexCore. Without comparisons against candidates extracted from general corpora under the same budget and selection method, it is difficult to determine whether the gains come specifically from learner vocabulary or simply from adding useful words.
-
The authors suggest that LIFT-5K underperforms LIFT-3K because the fixed CPT budget provides insufficient adaptation, but this explanation remains untested. Including smaller budgets, such as 1.5K or 2K, would clarify the vocabulary-budget–performance relationship. Increasing the CPT budget for LIFT-5K would more directly test the proposed explanation.
-
Although training tokenizers from scratch falls outside the paper’s main focus on adapting existing tokenizers, a comparison under the same data and final vocabulary size would strengthen the broader argument for learner-based vocabulary allocation. For example, a 30K tokenizer trained from scratch could be compared with a 25K tokenizer augmented with 5K LexCore entries, measuring compression and cross-language fairness.
-
Downstream validation is limited to Gemma-3 4B. Tokenization outcomes vary across tokenizers, but QA, translation, and efficiency after CPT are evaluated on only one model. The evidence for downstream benefits therefore remains specific to this setting, and validation on another model would strengthen the generalization claim.
Minor
- The choice of Sum initialization is not validated through downstream comparisons. The embedding-retrieval analysis motivates this choice, but comparisons with Mean or random initialization under the same selected vocabulary and CPT conditions are needed to establish whether Sum is preferable for model adaptation.
Questions for authors
-
Were any matched-budget comparisons conducted against greedy or frequency-based selection from LexCore, or against candidates drawn from general corpora? If available, these results would help distinguish the contribution of the learner vocabulary from that of the maximin selection objective.
-
Under the same vocabulary and CPT budgets, could you compare LIFT with a simple selection baseline and a corpus-derived candidate pool? This would directly address the two main concerns raised in the Weaknesses section.
Thank you for the additional analyses and clarifications. After considering the rebuttal, I will maintain my original score.