acceptodds
Under review as a conference paper at ICLR 2027

SEPARATING SCRIPT AND VOCABULARY EFFECTS IN MULTILINGUAL TOKENIZATION COST

Abstract

Cross-lingual token cost depends both on a language's writing system and on how its tokenizer encodes that writing system. We show that token-count ratios on parallel text factor exactly into these two components, and measure them across seven production tokenizers, fifteen languages, and five script families on FLORES. This decomposition distinguishes languages that are expensive because their writing system requires more characters from those with inefficient tokenization. Devanagari and Arabic-script languages require two to three times as many tokens as English despite similar writing-system cost, with most of the excess attributable to the tokenizer (convention-dependent for Devanagari: 99.9% under code points, 76.9% under grapheme clusters, though the cost ratio is not). We then measure, in simulation, how much of this tokenizer penalty can be recovered by replacing unused entries with high-impact fragments at fixed vocabulary size. At 5,000 slots, mean characters-per-token improvement reaches 58.8% for Devanagari and 51.1% for Arabic-script languages (per-language means over the seven tokenizers, evaluated on held-out devtest), with a maximum of 159.9%. CJK languages have comparable tokenizer penalties, but a segmentation-free generator with supply well in excess of the budget recovers almost nothing more. These results show that multilingual tokenization disparities should first be diagnosed by cause: substantial inefficiency can often be recovered by reallocating existing vocabulary capacity, but fixed-vocabulary interventions have clear limits.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
tokenization, multilingual large language models, subword vocabulary, cross-lingual, byte-pair encoding, vocabulary allocation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PvRZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates multilingual LLM tokenizer costs using cross-lingual token-count ratios into writing system and components of the tokenizers, across 15 languages and seven tokenizers. Authors analyze the tokenization disparities and simulates fixed-size vocab reallocations.

Critical strengths
  1. The problem is important as the cross-lingual tokenization disparities have implications for multilingual LLM costs and context window utilization and the experiments cover the seven tokenizers and 15 languages covering five script families.
Critical weaknesses
  1. This paper doesn’t establish a sufficiently novel contributions beyond applying established tokenization statistics to a selected collection of languages. Prior studies have already examined multilingual tokenization disparities, compression, and vocab allocation, and adaptation. The incremental contribution appears limited in the context of literature, weakening the chance for ICLR.

  2. The paper interprets relative character counts as a writing system component and relative chars per token as a tokenizer component. Char counts in translated sentences may also reflect lexical, morphological, and grammatical differences and chars per token is influenced by the information density and linguistic properties of the underlying text, not directly by tokenizer design. These raise concerns about the validity and interpretability of the proposed attribution and affects the claims of this paper. I would encourage authors to defend these cases if possible or revise the paper to make it better for future revisions.

Questions for authors

See "Critical Weaknesses"

Reviewer_wgQoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper presents an interesting study of different tokenisers. The aspects related to the writing system, tokenisation, and relocation are valuable.

  • It works on 15 different languages with vocabulary in the FLORES dataset.

  • The overall study is interesting but lacks strong methodological novelty.

Critical strengths
  • Seprates writing system from tokenizer which is interesting insight.

  • Rellocation experiments form another interesting part.

  • Ranking languages by compression and token cost identifies problems in different languages.

Critical weaknesses
  • No strong method novelty.

  • FLORES webstie says that dataset is outdated and refers to OLDI FLORES dataset.

  • In Fig. 1, caption mentions that diamond is product of blue and orange bars, but the same does not appear to be true for Chinese,

  • Japanese and Korean langauges. E.g. for chinese, the blue bar is 0.34, orange is around 1.2, so product should be below 1! Please clarify.

Questions for authors
  • Any improvements in existing tokeniser algorithms you can suggest based on your work?

  • Please reason why the latest OLDI FLORES dataset is not used?

  • Please provide response based on other weaknesses covered above.

Reviewer_Q55BOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors try to answer how much headroom does each tokenizer have to improve efficiency across languages given script and linguistic nature of a language. The authors decompose this headroom into two components: 1) language-dependent component, which accounts for the information to character content given a language, and 2) a tokenizer-dependent  component, which could be improved with tokenizer-level interventions.

The authors evaluate seven off-the-shelf tokenizers  with vocabulary sizes ranging from 33K to 262K across 15 languages using the parallel FLORES corpus.

To estimate the headroom, they propose a simple reallocation experiment, where they remove one unused token from the vocabulary to add a frequent token from a target language, though this is done in isolation for each language.

Critical strengths

I believe the main research question studied in the paper to be an important one that could navigate tokenizer researchers.

Critical weaknesses

I find the problem studied in the paper to be an important one that should be disentangled but I don’t find the experiments and analysis in the paper to be substantial enough. The background is lacking, experiments in the paper are very shallow, and they don’t improve our understanding of multilingual tokenization and the script-level differences. There is also no specific hypothesis tested and hence neither an actionable insight.

The paper fails to situate itself in the broad tokenization literature that touch on this very aspect. The paper in my personal opinion is reinventing the notions from Arnett et al 2025 without a proper account of the differences. I mainly see this paper as a combination of notions from token premiums in Petrov et al and byte premiums in Arnett et al. The authors do not cite the byte premiums work at all.

The paper also does not provide an account or hypothesis into where these disparities come from. Hence, there is also no mention and testing of different design factors that are central to tokenization: pre-tokenization, normalization, training algorithm. This effectively makes the choice of the seven tokenizers a random one as well. Reporting the Gini coefficient and some other parity/compression metrics (see Meister, et al. 2026 for some alternatives) is also necessary.

The authors also fail to cite and evaluate Script-Tok work, which is directly relevant here to the claims about writing system.

I found Section 4.2 to be very hard to follow. I think for the claims regarding CJK and different scripts, it would be useful to know what is the coverage of these scripts so far. Also having a pareto curve of when these additions start hurting those scripts and other languages.

I find it unrealistic to study this headroom in isolation. At the end of the day, tokenizer vocabulary is a zero-sum game, where each added token impacts the tokenizer’s performance on every other language. Hence, I think these interactions and the relative opportunity cost of each added token is a more impactful one for the broader literature.

Overall I think this paper is far from acceptance.

Minor:

I think there is some terminology in the paper that is not fully explained and sounds weird at times

Questions for authors

In the reallocation experiment, the authors report that they remove a token for every token they add. Are these terminal tokens, i.e. they don’t make up a merge? Could the authors please elaborate on the frequency with respect to the full vocabulary and a short qualitative analysis on the nature of such tokens?

How should one read Figure 2? If the tokenizer has no coverage of a language, e.g. Farsi, the tokenizer would mostly be reverting to byte-fallback. Then a 157% improvement would effectively mean that now the tokenizer packs 1.5x more bytes per token, i.e. it now has some 2-3 byte tokens for this language. Is my read correct? What kind of impact does this have for language modeling? Does this really improve fairness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.