acceptodds
Under review as a conference paper at ICLR 2027

Whose Tokenizer? When and How Much Tokenizer Choice Matters for Multilingual Language Models

Abstract

Tokenizer choice affects multilingual language modeling, but vocabulary capacity is finite and vocabulary size is often constrained: improving representation for some languages often comes at the expense of others. We therefore ask whether tokenizer choice matters equally across languages. To this end, we train 123 language models across 54 tokenizers. Our main comparison spans 54 tokenizers; architecture, training corpus, training-token budget, and optimization are held fixed across the corresponding models, allowing us to isolate differences due to the tokenizer. We measure every outcome per language. Tokenizer choice matters more for languages with less language-model training data: across the 54 tokenizers, the standard deviation of a language's bits-per-byte (BPB) increases as its model training-data share decreases (Spearman rho=-0.54 over the 31 trained languages and -0.69 over the 28 written with word boundaries). Leaving a language out of tokenizer training raises its BPB in every language we study, and the penalty tends to be larger for languages with less language-model training data. Giving lower-resource languages a larger share of tokenizer-training data, however, does not unconditionally help those languages: both equal weighting and an allocation inverted against their language model training data share increase their BPB, particularly when language-model training repeats data. Finally, which intrinsic tokenizer properties are associated with better BPB differs across languages, providing further evidence that what makes a good tokenizer depends on the language. Over the 31 languages, we achieve 0.89 accuracy on pairwise rankings of trained models' mean BPB using just tokenizer intrinsic metrics, suggesting a practical strategy for screening tokenizer candidates before training language models. All resources will be released upon de-anonymization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tokenization, Multilingual Language Modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wRgyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This study presents an empirical investigation of the language-specific effects of the tokenizers in multilingual LLMs. Through controlled experiments of 123 LLMs and 54 tokenizers, they show that the sensitivity varies across languages. This paper also reveals that tokenizer quality can be predicted without training full language models. The study is organized as four connected research questions to offer some insights to the community.

Critical strengths
  1. Empirical investigation of training 123 LMs and 54 tokenizers across multiple dimensions.
  2. The paper is well structured around four RQs, making it easier to follow.
Critical weaknesses
  1. Although I liked the experimental depth of this work, however, the distinction from existing work is not substantial, like Ali et al. (2024) already investigate tokenizer algorithms, vocabulary sizes, preprocessing choices, and multilingual training configurations, and Goldman et al. (2024) and the recent work Altıntaş et al. (2026) investigate relationships between tokenizer properties and model performance. I see the primary contribution appears to be the language-specific grouping of these effects, I encourage the authors to clarify the which findings are substantial new insights for the community.

  2. The paper claims that lower-resource languages are more sensitive to tokenizer choice, and the authors acknowledge that language resource levels and writing systems are confounded. I would say the conclusion can be presented as an association, rather than a general property as the experiments are primarily conducted at a fixed model scale and training-data budget. I wonder these conclusions may change with the parameter and/or training data scale. Based on what I know and experienced personally the more data can help reducing this effect. Can the authors clarify this?

Questions for authors

See "Critical Weaknesses"

2 comments

We thank the reviewer for their constructive feedback and recognition of the study's experimental breadth.

W1. "Although I liked the experimental depth of this work … distinction from existing work is not substantial (Ali 2024; Goldman 2024; Altintas 2026) ... clarify which findings are substantial new insights"

We agree that the per-language analyses and results are the central contribution, and have made this explicit in the introduction. As discussed in our general response, the novelty lies in providing systematic evidence in a setting that closely resembles contemporary multilingual LM pretraining. The cited studies are important prior work, but this study differs in important respects:

  • Experimental setting and recent developments. The studies discussed above consider at most four non-English languages, whereas our study covers 30 languages in addition to English. Furthermore, Altintas et al. study off-the-shelf tokenizers, targeting a different setting where practitioners reuse existing tokenizers, whereas we investigate the effects of tokenizer design and training choices. While Ali et al. and Goldman et al. train their own tokenizers, our work additionally considers more recent algorithms specifically designed for multilingual settings, including Script BPE, Parity-Aware BPE, and SuperBPE.
  • Language-dependent effects in a shared multilingual model. We study language-specific effects specifically in the multilingual setting that is particularly relevant to contemporary multilingual: a setting with highly imbalanced representation in the training data (every language except English and Russian accounts for less than 2% of the training bytes). This is a substantial difference from prior work, which evaluates aggregate outcomes or dedicated per-language tokenizers.
  • Tokenizer training composition and controlled multilingual interventions. The effects of tokenizer-training data composition in realistic multilingual settings remain relatively underexplored. We address this by systematically intervening on the tokenizer-training data while keeping the model-training mixture fixed. This allows us to isolate how the representation of different languages during tokenizer training affects downstream performance, particularly at the small language shares characteristic of multilingual models.
  • Intrinsic metrics and tokenizer screening. Finally, we investigate the relationship between per-language intrinsic tokenization metrics and downstream performance. Building on this, we introduce a screening protocol that ranks held-out tokenizers using per-language intrinsic metrics, evaluated through family-held-out cross-validation. This provides a practical way to assess tokenizer quality without requiring full model-training experiments for every candidate.

While a subset of our results are similar, this work establishes them in a highly relevant domain, where they have not yet been tested.

W2. "Present the conclusion as an association, not a general property ... may change with scale ... more data can help reduce this effect"

We agree. The revision now explicitly presents the relationship between LM training-data share and tokenizer sensitivity as an association at the evaluated model scale and training budget, rather than a universal or causal property. We have also made the writing-system confound more prominent and added explicit caveats to each research question. As detailed in our response to YF8K, neither resource level nor writing system can be isolated in our setup as a possible explanation for the observed cross-language differences. Our results partly support the hypothesis that some of the observed negative impacts from tokenizer design can be mitigated with more data. The submission already includes a direct test of data repetition (Appendix E.4). Under repeated-data training, we see that the BPB penalty from reallocating tokenizer-training data toward lower-resource languages is larger for languages with smaller LM training-data shares (Spearman ρ = −0.59). Yet with single-pass training, this association is no longer significant (albeit per-language BPB still increases on average). Whether scale can likewise mitigate some of the observed effects remains open. As described in our general response, we are training six 3B-parameter models to test whether tokenizer rankings and per-language effect directions persist beyond 1.27B.

I thank the authors for their responses and additional analyses.

I still remain hesitant to recommend for acceptance as the language-specific findings offer insufficient insights beyond Ali et al. (2024), Goldman et al. (2024), and Altıntaş et al. (2026) and the 3B experiments are still incomplete. This paper may need another round of reviews after the experiments are fully complete and incorporated in the manuscript.

I will maintain my original rating.

Reviewer_YF8KOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how tokenizer choice affects different languages in multilingual language models. The authors train 123 models with 54 tokenizers while keeping other training settings largely fixed. They find that lower-resource languages are generally more sensitive to tokenizer choice, and that including a language in tokenizer training usually improves its results. They also show that simple tokenizer metrics can help predict model performance, though the findings depend on the language and training setup.

Critical strengths
  • The paper provides a useful controlled study of tokenizer choice across many languages. Keeping the model architecture and training setup fixed makes the comparisons more informative.

  • The authors also report results for individual languages rather than relying only on overall averages. This helps reveal differences that multilingual evaluations can hide.

  • The study explores whether simple tokenizer metrics can predict model performance, which could help reduce the cost of testing tokenizer designs.

Critical weaknesses
  • The experiments use relatively small models and a limited training budget. It is unclear whether the same trends would hold for larger models and longer training runs.

  • The relationship between language resource levels and tokenizer sensitivity may be affected by differences in writing systems and text characteristics. These factors make it difficult to attribute all observed differences to training-data availability alone.

  • The study mainly evaluates language-modeling performance using bits per byte. More evidence is needed to show how these findings translate to downstream tasks and practical multilingual model quality.

Questions for authors
  • Do the main findings hold for larger models and longer training budgets?

  • How much of the observed relationship between language resources and tokenizer sensitivity can be explained by differences in writing systems and text characteristics?

  • The paper reports that giving lower-resource languages more weight in tokenizer training can worsen performance. What factors explain this result, and under what settings might reweighting help?

3 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for their thoughtful review.

Scale of the study

"Relatively small models and a limited training budget" / "Do the main findings hold for larger models and longer budgets?" For larger models. We agree that our inability to test generalization to much larger scales is a main limitation of the work. We are in the process of training six additional models at 3B model scale to address thisperform some generalization assessments of the results.

For longer budgets. The submission includes an analysis showing that aggregate tokenizer rankings remain stable when extending training by approximately 10B tokens (Appendix C: the single-pass variants continued with +10B of new text over 11 runs of 10 tokenizers; tokenizer ranking between these and the main panel is very consistent with Spearman ρ = 0.94). We also have an analysis at three smaller model scales: validation-BPB rankings match the 1.27B ordering with 0.83–0.88 pairwise accuracy (Table 16), which suggests that assessments at our studied scale should be relatively stable. Beyond this, the training data budget for the 3B models currently training is naturally larger than for the main 1.27B models, which should give us even further insights into whether findings hold at larger budgets.

"May be affected by differences in writing systems and text characteristics" / "How much is explained by writing systems?"

This is an excellent point and is a focal point of Appendix C. In summary, we agree that writing system and resource level are confounded. Several of our analyses try to disentangle this to the extent that we can in our setup. First, we see that the correlation between resource level and BPB spread remains significant when restricting the analysis to the 28 languages written with word boundaries. Second, the tokenizer-training coverage experiment compares each language directly to itself, i.e., holding its writing system and text characteristics fixed. Here, we see that the exclusion penalty (difference in model performance with and without tokenizer coverage) is positive for all 31 languages. Appendix C provides some additional analyses beyond this. Nonetheless, we still present the resource-level relationship as an association, not an isolated causal effect and have made this limitation more prominent in the revision.

"Mainly evaluates ... bits per byte. More evidence needed on downstream tasks"

We agree that BPB alone cannot establish downstream task effects. The submission already includes MultiBLiMP results foracross 24 languages. Here, eight of nine intrinsic metrics show significant associations in the same directions as for BPB (Table 17, Appendix G). The revision additionally includes XNLI across 13 languages, for which four intrinsic metrics retain significant associations. Other task benchmarks provide limited corroboration at this model scale: models perform at near chance on Belebele, XCOPA and XWinograd (latest arelatter two are new results since submission). We therefore limit our main conclusions to language-modeling performance rather than claiming broad downstream generalization.

AuthorsOpenReview ↗

Response (2/2)

(Q3). "Giving lower-resource languages more weight in tokenizer training can worsen performance. What factors explain this, and when might reweighting help?"

This is (in our opinion) one of the more interesting results of the paper and we have extended our analyses to try to identify possible explanations. We have come to the following conclusions:

  • Data repetition amplifies the effect. Under the inverted tokenizer-training data configuration, when the models are trained with data repeating, Tamil has a BPB increase of +0.27 (in comparison to the baseline tokenizer); in contrast, under single-pass training, we observe only a +0.14 increase in Tamil’s BPB over the baseline tokenizer.
  • Improved compression on a language does not mean improved performance on that language. Giving low-resource languages a higher share of the tokenizer-training data reduces their token counts without necessarily improving BPB. The penalty cannot be explained by a resulting decrease in the amount of underlying training data to Tamil; rather, Tamil's number of training bytes increases under the inverted configuration. One possible mechanism is that longer, more language-specific tokens receive insufficiently diverse training contexts, particularly when data is repeated. We do not claim to have isolated this mechanism.
  • Tokenizer coverage alone does not help a language’s performance. When tokenizer-training data is expanded to include an additional 183 languages absent from LM training, mean BPB increases by +0.20 to +0.42 across those languages. This is consistent across four matched comparisons. Conversely, excluding languages that are present in LM training consistently worsens their BPB.

Together, these results suggest that tokenizer-training allocation should be considered jointly with the LM training-data distribution. We have added these points to the revision, but determining when reweighting becomes beneficial remains an open question.

I thank the authors for their detailed response. Their rebuttal has adequately addressed my previous concerns, hence i raise my score.

Reviewer_e1NSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper describes a tokenizer specificity for multi-lingual language models. The authors find that more training data doesn't necessarily improve BPB for low-represented languages and languages with low shares in training data show greater variation across tokenizers. This paper highlights tokenizer metrics that help identify the right tokenizer for a specific language. The authors train 123 language models using decoder models spanning 54 tokenizers and find that for over 31 languages, mean BPB has around 90% accuracy on pairwise rankings of trained models. To quantify correlation between tokenizer metrics and downstream outcomes, the authors train a mixed effects model and fit logistic ranker for pairwise accuracy.

Critical strengths

The paper’s strongest contribution is comparison of 54 tokenizers across 123 models across 31 languages. This reveals that aggregate multilingual scores hide language-specific differences in tokenizer rankings, an important finding for multilingual model design.

Critical weaknesses

In the current form, the paper reads more of an empirical study of tokenizers across multiple languages rather than a solution to the problem highlighted by the authors. The use of mixed effects model and logistic rankers is not justified anywhere. The run-to-run variance in table 17 is a big concern. Table 15 shows the linear relationship is not significant given p-value > 0.05. Details around logistic regression are limited apart from sharing the implementation details.

Questions for authors
  1. Do the main results change if you remove the three tokenizers that do not reproduce the original text exactly after encoding and decoding?
  2. Since each tokenizer causes the model to see a different amount and share of text, do the findings still hold when the training data or compute is matched?
  3. The fixed bigram-entropy baseline scores 0.909, above the full ranker’s 0.890. What does the learned ranker add?
  4. What is the new finding beyond running a larger comparison, especially relative to How Robust is NMT to Language Imbalance in Multilingual Tokenizer Training? and Beyond Text Compression: Evaluating Tokenizers Across Scales?
3 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for thoroughly engaging with our work.

Framing and Significance

E1. "Reads more of an empirical study ... rather than a solution to the problem" E9 (Q4). "What is the new finding ... relative to [Zhang et al. 2022] and [Lotz et al. 2025]?"

Regarding novelty and contribution, please see our general response. We thank the reviewer for highlighting related work. The revision now discusses Zhang et al. (2022) explicitly: they study tokenizer-training imbalance in encoder–decoder translation models over eight languages, reporting aggregate translation quality. In contrast, we study contemporary multilingual LM pretraining, hold the LM training mixture fixed, isolate tokenizer design choices, and measure outcomes per language, which we find an arguably more relevant analysis for model development foci today. Lotz et al. (2025) was already cited. We have also added an explicit contributions statement to the introduction.

Statistical Protocol

"The use of mixed effects model and logistic rankers is not justified anywhere" "Details around logistic regression are limited apart from sharing the implementation details"

We respectfully point out that both justifications are in the submission: the mixed-effects model in Section 2.3 (Analysis) and the ranker in Appendix F.4. We have clarified their justifications in the revision and moved them closer to where they’re used to make it easier for the reader (Section 6.1). The justifications can be summarized as follows:

  • Mixed-effects model: Language identity accounts for 99.4% of the variance in per-(model, language) BPB scores. Language-specific random intercepts allow us to account for this variation and isolate any within-language associations. Ordinary least squares with a variable per language (i.e., fixed effects) yields the same coefficients to within one unit of the last digit printed (Table 11). The main advantage of the random effects specification is that it gives us more accurate standard error estimates.
  • Dependence between tokenizers: The 54 tokenizers belong to 42 families, with some differing only in minor settings. We therefore use family-clustered standard errors rather than treating all tokenizers as independent data points.
  • Pairwise ranker: Logistic regression predicts relative BPB from differences between intrinsic-metric vectors. The model uses L2 regularization, with strength selected through inner family-wise cross-validation and evaluation through outer leave-one-family-out folds. The complete specification is in Appendix F.4.
AuthorsOpenReview ↗

Response (2/2)

Statistical Significance and results

E3. "The run-to-run variance in table 17 is a big concern"

Table 17 does not report run-to-run variance; it reports coefficients relating intrinsic tokenizer metrics to MultiBLiMP accuracy, together with their standard errors. Eight of nine associations remain significant after accounting for dependence between tokenizer families and correcting for multiple testing (adjusted p ≤ 0.004). Thus, uncertainty in the estimated coefficients is explicitly accounted for in the reported significance tests. Separately, we quantify actual run-to-run variation using dedicated seed replicates. The across-tokenizer spread in MultiBLiMP accuracy exceeds estimated seed variation for 21 of 24 languages. However, for 16 languages, benchmark sampling uncertainty is at least as large as seed variation, limiting reliable per-language comparisons. We explicitly acknowledge this limitation and base our principal per-language conclusions on FLORES+ BPB, for which the across-tokenizer spread exceeds seed variation for all 31 languages.

E4. "Table 15 shows the linear relationship is not significant given p-value > 0.05"

Table 15 reports within-language associations separately across three resource tiers, rather than one overall relationship. The pooled relationships appear in Table 11, where eight of nine metrics are significant. We have clarified this distinction in the revision.

(Q1). "Do the main results change if you remove the three tokenizers that do not reproduce the original text exactly?"

This is a fair concern, and we tested it. No main conclusion changes. In more detail, we repeated the principal analyses after excluding these three tokenizers:

  • Intrinsic-metric associations: The significance pattern is unchanged (8eight of 9nine metrics); the largest coefficient change is 0.0026.
  • Resource sensitivity: The correlation between LM training-data share and BPB spread changes from −0.52 to −0.51.
  • Cross-language rankings: Mean pairwise Kendall τ changes from 0.418 to 0.420.
  • Predictive ranking: Accuracy changes from 0.890 to 0.879 for the full ranker and from 0.909 to 0.885 for the bigram-entropy block. The missing bytes are rare in training text (they make up 0.13% of the validation sets’ bytes) and absent from the FLORES+ evaluation text. Nevertheless, their absence could affect model training, so the exclusion analysis is the stronger robustness check. We have added these results and clarified the defect in the revision.

(Q2). "Do the findings still hold when the training data or compute is matched?"

The submission explicitly examines this issue (§6, Appendix F.3, Table 14); the revision makes the reference more prominent. Compute is approximately controlled by design: models share the same architecture and training-token budget. However, different tokenizers expose models to different amounts of underlying text, so training bytes are not matched. To assess this confound, we measured per-language training bytes for 51 tokenizers and adjusted the within-language analyses using four empirically estimated relationships between training bytes and BPB(§F.3). No coefficient changes sign. Fertility, UTF-8 character splitting, bigram and trigram entropy, and boundary crossing remain significant under all four adjustments. Compression rate, the metric most directly related to training-byte exposure, loses significance under three of four. We also conducted two direct byte-matched comparisons, one of which reverses direction after matching. Thus, not every tokenizer effect is invariant to training-byte exposure. We report this limitation explicitly while identifying the associations that remain robust to the adjustments.

(Q3). "The fixed bigram-entropy baseline scores 0.909, above the full ranker's 0.890. What does the learned ranker add?"

The comparison is correct. However, selecting bigram entropy as the strongest fixed baseline after inspecting the results benefits from hindsight. When the best single metric block is instead selected within each cross-validation training fold, accuracy is 0.864 on FLORES+ and 0.896 on validation BPB, compared with 0.890 and 0.907 for the full ranker. We added this comparison to Table 16. More importantly, our claim concerns the predictive value of per-language intrinsic measurements, not the superiority of a particular classifier. Using all nine metrics, per-language measurements achieve 0.890 accuracy versus 0.735 using one aggregate value per metric and tokenizer. This is the central practical finding. We therefore recommend the prespecified nine-metric feature set for screening rather than selecting a single metric post hoc.

Thanks for the detailed response. I had misread Tables 15 and 17. Table 17 reports coefficients and standard errors, not run-to-run variance, and Table 15 reports results by resource tier rather than the pooled relationship. The explanation clarifies the reasons for using the mixed-effects model and logistic ranker.

The results after removing the three non-round-tripping tokenizers address my first question. The fold-wise comparison with bigram entropy also gives a fairer assessment of the full ranker. I still have some concern about the fixed-token setup because different tokenizers cause the models to see different amounts and mixtures of text, and one byte-matched comparison reverses direction. This limitation should remain clear in the paper.

The comparison with Zhang et al. and Lotz et al. also makes the contribution clearer. I still see this primarily as an empirical paper, but the scale of the study and the per-language analysis provide enough new evidence to support acceptance. Overall, the responses addressed enough of my concerns to change my rating to weak accept and my soundness score to 2.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Update on 3B model training

All 3B models are most of the way through training but some have not yet finished. We show here the results at a comparable checkpoint (81% through training), i.e., one that all models have made it past so far. Notably, at this point, we see only a single pair whose BPB ranking flips in comparison to the 1.27B ranking: the reference tokenizer and the tokenizer with NFC normalization. Under the 1.27B models, the val BPB of these two tokenizers differs by 0.0003, which is smaller than seed noise (the standard deviation of val BPB across 1.27B runs that differ only in random seed; 0.0004). This pair therefore had no measurable order at the 1.27B scale. The 3B order is thus far consistent with the 1.27B results, which gives some initial evidence that our results would extrapolate to larger scales.

Tokenizer1.27B val BPB3B val BPB @ step 21,000
reference (GPT-4o balanced BPE)0.71420.6469
NFC normalization0.71450.6458
Punctuation regex0.71960.6530
SuperBPE0.72480.6545
UnigramLM0.73410.6661
English-only data (fullbyte rebuild)0.74750.6683

As the discussion period closes, we are giving the last update on the progress of the 3B models (including evaluations on the held out FLORES dataset), for which we see the consistency appears to still hold:

Tokenizer1.27B FLORES mean3B FLORES meanΔ vs reference @ 1.27BΔ vs reference @ 3B
Reference: GPT-4o balanced BPE1.13801.0163——
NFC normalization1.13561.0133−0.0024−0.0030
Punctuation regex1.14621.0178+0.0082+0.0016
SuperBPE1.15441.0372+0.0164+0.0209
UnigramLM1.1696pending+0.0316pending
English-only data1.18001.0473+0.0420+0.0310

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.