- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This study presents an empirical investigation of the language-specific effects of the tokenizers in multilingual LLMs. Through controlled experiments of 123 LLMs and 54 tokenizers, they show that the sensitivity varies across languages. This paper also reveals that tokenizer quality can be predicted without training full language models. The study is organized as four connected research questions to offer some insights to the community.
Critical strengths
- Empirical investigation of training 123 LMs and 54 tokenizers across multiple dimensions.
- The paper is well structured around four RQs, making it easier to follow.
Critical weaknesses
-
Although I liked the experimental depth of this work, however, the distinction from existing work is not substantial, like Ali et al. (2024) already investigate tokenizer algorithms, vocabulary sizes, preprocessing choices, and multilingual training configurations, and Goldman et al. (2024) and the recent work Altıntaş et al. (2026) investigate relationships between tokenizer properties and model performance. I see the primary contribution appears to be the language-specific grouping of these effects, I encourage the authors to clarify the which findings are substantial new insights for the community.
-
The paper claims that lower-resource languages are more sensitive to tokenizer choice, and the authors acknowledge that language resource levels and writing systems are confounded. I would say the conclusion can be presented as an association, rather than a general property as the experiments are primarily conducted at a fixed model scale and training-data budget. I wonder these conclusions may change with the parameter and/or training data scale. Based on what I know and experienced personally the more data can help reducing this effect. Can the authors clarify this?
Questions for authors
See "Critical Weaknesses"
2 comments
I thank the authors for their responses and additional analyses.
I still remain hesitant to recommend for acceptance as the language-specific findings offer insufficient insights beyond Ali et al. (2024), Goldman et al. (2024), and Altıntaş et al. (2026) and the 3B experiments are still incomplete. This paper may need another round of reviews after the experiments are fully complete and incorporated in the manuscript.
I will maintain my original rating.
We thank the reviewer for their constructive feedback and recognition of the study's experimental breadth.
We agree that the per-language analyses and results are the central contribution, and have made this explicit in the introduction. As discussed in our general response, the novelty lies in providing systematic evidence in a setting that closely resembles contemporary multilingual LM pretraining. The cited studies are important prior work, but this study differs in important respects:
While a subset of our results are similar, this work establishes them in a highly relevant domain, where they have not yet been tested.
We agree. The revision now explicitly presents the relationship between LM training-data share and tokenizer sensitivity as an association at the evaluated model scale and training budget, rather than a universal or causal property. We have also made the writing-system confound more prominent and added explicit caveats to each research question. As detailed in our response to YF8K, neither resource level nor writing system can be isolated in our setup as a possible explanation for the observed cross-language differences. Our results partly support the hypothesis that some of the observed negative impacts from tokenizer design can be mitigated with more data. The submission already includes a direct test of data repetition (Appendix E.4). Under repeated-data training, we see that the BPB penalty from reallocating tokenizer-training data toward lower-resource languages is larger for languages with smaller LM training-data shares (Spearman ρ = −0.59). Yet with single-pass training, this association is no longer significant (albeit per-language BPB still increases on average). Whether scale can likewise mitigate some of the observed effects remains open. As described in our general response, we are training six 3B-parameter models to test whether tokenizer rankings and per-language effect directions persist beyond 1.27B.