- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies the compression cost of pre-tokenisation boundaries and their effect on language-model prediction. It introduces an occurrence-price certificate that lower-bounds the minimum achievable token count through shortest-path computations and vocabulary selection. The authors prove that optimising these certificates recovers the LP relaxation and provide an independent integer-arithmetic checker.
Experiments across 12 languages show substantial compression costs from pre-tokenisation, including a certified 28.3-36.8% increase in optimal token count for English. However, vocabularies fitted without boundaries generally yield worse language-model prediction, despite achieving better compression. The authors validate this through multilingual experiments.
They introduce boundary licences, which allow a limited number of vocabulary entries to cross boundaries. Their experiments show that licensing 10% of the vocabulary recovers 85.2% and 100% of the achieved compression gains in English and Chinese, respectively.
Critical strengths
The paper provides certified bounds on how much compression is lost due to pre-tokenisation, rather than relying on comparisons between specific tokenizers. This is a meaningful contribution, given the substantial costs observed across languages.
Strong empirical evaluation - the multilingual experiments demonstrate that better compression does not necessarily improve prediction.
Critical weaknesses
The proposed intermediate boundary policies recover substantial compression gains, but their effect on language-model prediction is not tested. It remains unclear whether they offer a useful compression–prediction tradeoff.
Questions for authors
How does language-model prediction change with the boundary-licence budget? A comparison at 0%, 10%, and 100% would help establish whether intermediate policies recover compression gains while preserving prediction quality.