- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors investigate whether morpheme-based tokenization can improve Transformer performance. Drawing on the Windows identifier system, they construct a dataset with standardized and objective morpheme segmentations and build tokenizers accordingly. Specifically, they construct tokenizers whose segmentation boundaries agree with the morpheme-based segmentation to varying degrees, and their experiments show that morpheme-aligned tokenization performs better. However, the authors point out that this improvement is not entirely attributable to the segmentation producing tokens with complete meanings; part of it also arises because such segmentation leads to a higher rate of fragment reuse.
Critical strengths
- This paper introduces a rule-based and reproducible morpheme-segmentation dataset. WINMORPH derives reference boundaries from published Windows naming conventions rather than subjective human judgments. This makes morphological segmentation sufficiently well-defined to be manipulated as an experimental variable.
- It replaces correlational comparisons of existing tokenizers with a controlled intervention. The authors construct tokenizers that vary in how often their boundaries coincide with reference morpheme boundaries while keeping the number of tokens for each identifier fixed. This design directly addresses the usual confounding between morphological alignment and compression.
- It uses two complementary matching designs to strengthen the comparison. The segmentation-matched design controls tokenization length and vocabulary capacity, whereas the symbol-matched design further accounts for character fallback by controlling the symbols actually consumed during training and the resulting training exposure. Agreement across these designs makes the observed benefit harder to attribute solely to shorter effective inputs or additional training passes.
- It evaluates compositional generalization and tests the finding on another corpus. The emphasis on identifiers withheld from training examines whether aligned pieces can be reused to model novel combinations rather than merely memorize observed forms. Replicating the experiment on Hungarian morpheme segmentation further suggests that the result is not solely an artifact of Windows naming conventions.
Critical weaknesses
The constructed tokenizers rely on reference morpheme boundaries, whereas practical LLM tokenizers are learned from raw text and must process unseen identifiers without access to such boundaries. It therefore remains unclear how the findings could be translated into concrete improvements to real-world tokenizer training.
Questions for authors
I have no major questions about the paper overall. I found it clearly written and the experimental evidence compelling. I have only a few minor questions and comments:
- Even at a speculative level, how do the authors think this work could guide improvements to tokenizer training for real-world language models?
- I found the arguments and derivations surrounding the two propositions in Section 3 somewhat lengthy and tangential to the paper’s main line of argument. Would the authors consider retaining only the statements and their implications in the main text and moving the remaining derivations to an appendix?
- Minor notation issue: On page 3, line 145, the range in the definition of should be enclosed in set braces, i.e.,
B_T(w) \subseteq \{1,\dots, |w|-1\}
2 comments
I appreciate the authors for their responses, which have clarified all my questions. I will retain my score. Best of luck with the submission!
Clarification and Response to Reviewer ixZ6
We thank the reviewer for the careful reading and the positive assessment. We answer the weakness together with Question 1, and then Questions 2 and 3.
We agree that our arms use reference boundaries and that the paper does not yet show a practical tokenizer. Appendix A states this limit. With that said, we see four ways in which the results can guide tokenizer training. They are directions and not results.
a) For code, the boundaries can be computed without labels. The reference cuts in WinMorph come from written naming rules (Microsoft), such as underscores in constants, capital letters in function names and type prefixes in declarations. A program applies these rules without human labels. A tokenizer can therefore apply the same rules to an identifier it has never seen, as a first splitting step, much as current tokenizers first split text at spaces and punctuation.
b) The aligned arm is already a simple recipe. It starts from the rule-based pieces and merges the most frequent neighboring pieces until it reaches the target token count (Please refer to Section 3.2). This is the usual pair merging, but it starts from morphemes and not from characters, so every remaining cut is a reference boundary. The same procedure could build the vocabulary of a code model. Shi et al. (2022) tested rule-based splitting for a code model and found mixed results, but the token count changed at the same time. Our results suggest keeping the token budget fixed and changing only where the cuts are.
c) WinMorph can serve as a check that needs no model training. Any tokenizer can be scored for boundary precision on the 83,900 identifiers, and for consistency on the A/W pairs, without training a language model. Between two tokenizers with the same token count, our results favor the one whose cuts agree more with the reference.
d) For natural language, a morphological analyzer can take the place of the naming rules. The Hungarian result (41.2% and 30.5% lower held-out bits per character) suggests that the same idea is worth testing with such an analyzer.
We will add a short paragraph with these directions to the paper and mark it clearly as an outlook.
Yes. We will keep the two statements and their meaning in the main text and move the proofs and derivations to an appendix. We will use the space to explain more clearly what each of the two designs holds fixed.
Thank you for catching this. The definition should read B_T(w) ⊆ {1, …, |w| − 1}. We will correct it.
Other changes in the revised paper
In reply to the other reviews we will make three more changes. First, we will state the scope of the claim more exactly, in line with your own summary of it. The effect is one of cut positions at a fixed token count, and much of it comes through piece reuse. Second, we will rename the "Reuse-matched" arm to "reuse control", because its reuse (2.26) is below that of the aligned arm (5.55). Third, we will move the limitations from Appendix A into the main text. None of these changes alters the results you reviewed.