acceptodds
Under review as a conference paper at ICLR 2027

WINMORPH: DOES TOKENIZING CODE IDENTIFIERS AT MORPHEME BOUNDARIES IMPROVE LANGUAGE MODEL PERFORMANCE ?

Abstract

Code identifiers such as AddAccessAllowedAce combine meaningful sub-parts (morphemes), such as Add, Access, Allowed, and Ace. However, a language model observes only tokenizer outputs, and a tokenizer may place boundaries that do not match these morphemes. Although better boundary placement is often assumed to help, it is difficult to isolate because standard tokenizers confound alignment with token count; making more correct splits typically also changes the number of tokens. We therefore ask: when the token count is held fixed, does aligning token boundaries with reference morpheme boundaries improve language modeling? We address this question with W IN M ORPH , a corpus of 83,900 Windows identifiers with gold cut points derived from published naming rules, and a benchmark that varies the fraction of correct cuts from none to all while keeping tokens-per-identifier constant. We study two controls: (i) segmentation matching under a shared vocabulary cap, and (ii) matching the average number of model-input symbols on the training set by adjusting vocabulary coverage. On held-out identifiers, full alignment reduces bits per character by 39.8% under segmentation matching and by 18.9% under training-symbol matching, compared to no reference-aligned cuts. These results show that, when we control for token count, splitting identifiers at reference morpheme boundaries improves language model performance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Morphological alignment, Tokenization, Compression, Code language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ixZ6OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors investigate whether morpheme-based tokenization can improve Transformer performance. Drawing on the Windows identifier system, they construct a dataset with standardized and objective morpheme segmentations and build tokenizers accordingly. Specifically, they construct tokenizers whose segmentation boundaries agree with the morpheme-based segmentation to varying degrees, and their experiments show that morpheme-aligned tokenization performs better. However, the authors point out that this improvement is not entirely attributable to the segmentation producing tokens with complete meanings; part of it also arises because such segmentation leads to a higher rate of fragment reuse.

Critical strengths
  1. This paper introduces a rule-based and reproducible morpheme-segmentation dataset. WINMORPH derives reference boundaries from published Windows naming conventions rather than subjective human judgments. This makes morphological segmentation sufficiently well-defined to be manipulated as an experimental variable.
  2. It replaces correlational comparisons of existing tokenizers with a controlled intervention. The authors construct tokenizers that vary in how often their boundaries coincide with reference morpheme boundaries while keeping the number of tokens for each identifier fixed. This design directly addresses the usual confounding between morphological alignment and compression.
  3. It uses two complementary matching designs to strengthen the comparison. The segmentation-matched design controls tokenization length and vocabulary capacity, whereas the symbol-matched design further accounts for character fallback by controlling the symbols actually consumed during training and the resulting training exposure. Agreement across these designs makes the observed benefit harder to attribute solely to shorter effective inputs or additional training passes.
  4. It evaluates compositional generalization and tests the finding on another corpus. The emphasis on identifiers withheld from training examines whether aligned pieces can be reused to model novel combinations rather than merely memorize observed forms. Replicating the experiment on Hungarian morpheme segmentation further suggests that the result is not solely an artifact of Windows naming conventions.
Critical weaknesses

The constructed tokenizers rely on reference morpheme boundaries, whereas practical LLM tokenizers are learned from raw text and must process unseen identifiers without access to such boundaries. It therefore remains unclear how the findings could be translated into concrete improvements to real-world tokenizer training.

Questions for authors

I have no major questions about the paper overall. I found it clearly written and the experimental evidence compelling. I have only a few minor questions and comments:

  1. Even at a speculative level, how do the authors think this work could guide improvements to tokenizer training for real-world language models?
  2. I found the arguments and derivations surrounding the two propositions in Section 3 somewhat lengthy and tangential to the paper’s main line of argument. Would the authors consider retaining only the statements and their implications in the main text and moving the remaining derivations to an appendix?
  3. Minor notation issue: On page 3, line 145, the range 1,…,∣w∣−11,\dots, |w|-1 in the definition of BT(w)B_T(w) should be enclosed in set braces, i.e., B_T(w) \subseteq \{1,\dots, |w|-1\}
2 comments
AuthorsOpenReview ↗

Clarification and Response to Reviewer ixZ6

We thank the reviewer for the careful reading and the positive assessment. We answer the weakness together with Question 1, and then Questions 2 and 3.

  1. (Question 1)

We agree that our arms use reference boundaries and that the paper does not yet show a practical tokenizer. Appendix A states this limit. With that said, we see four ways in which the results can guide tokenizer training. They are directions and not results.

a) For code, the boundaries can be computed without labels. The reference cuts in WinMorph come from written naming rules (Microsoft), such as underscores in constants, capital letters in function names and type prefixes in declarations. A program applies these rules without human labels. A tokenizer can therefore apply the same rules to an identifier it has never seen, as a first splitting step, much as current tokenizers first split text at spaces and punctuation.

b) The aligned arm is already a simple recipe. It starts from the rule-based pieces and merges the most frequent neighboring pieces until it reaches the target token count (Please refer to Section 3.2). This is the usual pair merging, but it starts from morphemes and not from characters, so every remaining cut is a reference boundary. The same procedure could build the vocabulary of a code model. Shi et al. (2022) tested rule-based splitting for a code model and found mixed results, but the token count changed at the same time. Our results suggest keeping the token budget fixed and changing only where the cuts are.

c) WinMorph can serve as a check that needs no model training. Any tokenizer can be scored for boundary precision on the 83,900 identifiers, and for consistency on the A/W pairs, without training a language model. Between two tokenizers with the same token count, our results favor the one whose cuts agree more with the reference.

d) For natural language, a morphological analyzer can take the place of the naming rules. The Hungarian result (41.2% and 30.5% lower held-out bits per character) suggests that the same idea is worth testing with such an analyzer.

We will add a short paragraph with these directions to the paper and mark it clearly as an outlook.

  1. (Question 2)

Yes. We will keep the two statements and their meaning in the main text and move the proofs and derivations to an appendix. We will use the space to explain more clearly what each of the two designs holds fixed.

  1. (Question 3)

Thank you for catching this. The definition should read B_T(w) ⊆ {1, …, |w| − 1}. We will correct it.

Other changes in the revised paper

In reply to the other reviews we will make three more changes. First, we will state the scope of the claim more exactly, in line with your own summary of it. The effect is one of cut positions at a fixed token count, and much of it comes through piece reuse. Second, we will rename the "Reuse-matched" arm to "reuse control", because its reuse (2.26) is below that of the aligned arm (5.55). Third, we will move the limitations from Appendix A into the main text. None of these changes alters the results you reviewed.

I appreciate the authors for their responses, which have clarified all my questions. I will retain my score. Best of luck with the submission!

Reviewer_v2z3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces WinMorph, a benchmark to study whether aligning tokenizer boundaries with morpheme boundaries improves language modeling when the number of tokens per identifier is kept constant. The authors evaluate tokenizers with different levels of alignment on Windows API identifiers and Hungarian words, using both seen and held-out examples. Their results show that full alignment improves language modeling performance on both datasets, even when the number of training symbols is controlled. The authors also introduce a reuse-matched control to examine whether the gains are partly explained by greater reuse of token pieces across examples. I think the paper asks an interesting question and makes a reasonable effort to study it through controlled experiments. However, I have concerns about whether the experiments fully isolate the effect of morphological alignment, how well the evaluation reflects broader language modeling tasks, and how far the results support the paper's conclusions about tokenizer design.

Critical strengths
  • The research question is generally well-motivated. The paper identifies a confound in evaluating morphology-aware tokenization: tokenizers that better align with meaningful morpheme boundaries may also produce different numbers of tokens, making it difficult to attribute performance differences to alignment itself. By constructing tokenizers that vary in boundary alignment while holding the number of segmentation tokens per identifier fixed, the authors provide a more controlled way to study this relationship.

  • WinMorph provides reference segmentations derived from published Windows naming conventions and enables systematic variation of alignment precision like they describe. The additional Hungarian experiments provide some evidence that the observed pattern is not restricted to Windows identifiers.

  • Their experiments seem well scoped, where they distinguish between segmentation-matched and symbol-matched settings is useful and could potentially be applied outside of their domain. As their symbol-matched results suggest that the observed gains cannot be explained entirely by the additional training exposure available to the aligned tokenizer in the segmentation-matched setting.

Critical weaknesses
  • The experiments do not really isolate the effects of the morphological alignment from piece reuse. In the paper, they acknowledge that alignment changes the frequency with which token pieces recur across identifiers. The reuse-matched control is a useful attempt to address this issue, but it does not match the reuse level of the aligned arm that the authors describe. The reported reuse indices are 2.26 for the control and 5.55 for the aligned arm on WinMorph. The control also captures a substantial fraction of the improvement, including 54% under symbol matching. Also their results does establish that the constructed aligned tokenizer performs better, but they do not establish how much of the gain is attributed to meaningful morphological boundaries rather than increased reuse of pieces.

  • The symbol-matched experiment introduces differences in model capacity and vocabulary coverage. I get that this setting matches the average number of training symbols and the number of training passes, it seems to do so by changing the embeddings. For WinMorph, the aligned arm covers 2,825 token types and has approximately 2.3M trainable parameters, but the zero-alignment arm covers 32,266 types and has approximately 13.6M parameters. The remaining performance difference is therefore observed under matched training exposure, but not under matched model capacity or vocabulary coverage. It makes it sort of difficult to interpret the result as isolating alignment alone.

  • The paper motivation is sound, but their evaluation is too narrow with respect to broader implications of tokenizer design. Just evaluating language modeling performance in bits per character on held-out Windows identifiers and Hungarian words doesn’t mean that your results will translate to practical coding tasks. The evaluation is limited to a single family of code identifiers, and it remains unclear whether the results would hold across other programming languages, identifier naming conventions, or identifiers encountered in the context of actual source code. The experiments also use a small four-layer transformer with approximately 14M parameters in the main setting. It is very possible that the findings here will also not translate to model scales that are currently used in both academia and industry alike. I am just generally curious if these findings will generalize across model scales, to practical tokenizers and downstream tasks.

  • The paper claims to evaluate generalization to identify or words excluded from training. However there might still be similarities across both sets. Since the proposed benefit depends on reusing pieces across examples, it would be useful to understand how much of the improvement persists when test examples are structurally less similar to the training data. At least this might help us separate compositional generalization from generalization within familiar naming patterns.

Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Clarification and Response to Reviewer v2z3 (PART 1)

We thank the reviewer for a detailed review. We are glad that the review finds the question well motivated and the two matching designs useful. The review raises four concerns. We agree with several points in them and say so where we do.

  1. (Weakness 1)

The numbers you cite are correct. However, we do not see “reuse” as a second cause separate from alignment. It is simply how correct cuts help. A correct cut keeps a real part of a name together, and that same part shows up in many other names. A wrong cut splits a part into two, and those pieces show up less often. So when the number of tokens is fixed, parts show up more often as the cuts get closer to the correct ones (Table 1). This is what we set out to test: that a model can make a new name from parts it has already seen (Section 4). We claim that cutting at the right boundaries helps when the token count is fixed, and we see this in both setups and both datasets (Table 3). This claim does not rely on splitting the gain into “meaning” versus reuse, and the paper does not make that split (Sections 5 and 6).

This control supports our view. We built it to see how much of the improvement is left when parts show up often, but every cut is in the wrong place (Section 3.2). It keeps some of the improvement, so reuse matters. It also does about as well as the settings where parts show up at about the same rate (Tables 1 and 3), which suggests that the improvement mainly comes from reuse. At the same time, only the fully aligned setting is clearly better than the control on every test set (Table 3). The control could not match the aligned setting’s reuse, even after we tried many settings (Section 3.2), because cuts made inside morphemes create parts that show up less often (Table 1). So far, correct cuts are the best way we found to get high reuse with a fixed token count. In the revised paper, we will rename this setting “reuse control”, report the search, and state this limit in the main text.

  1. (Weakness 2)

The “symbol-matched” setup was not meant to test only whether we cut in the right places, and the paper does not claim that (Section 5). It has a smaller goal. In the other setup, the aligned setting ends up seeing the training names more times, and the symbol-matched setup removes that extra practice. As you point out, this shows the improvement is not only because the model got more training examples.

Two clarifications help. First, “coverage” and “capacity” are really the same issue here. The model is the same in every setting (Section 4.3); only the size of its embedding table changes. Second, we also compare settings where capacity is exactly the same. In the “segmentation-matched” setup, every WinMorph setting has the same table size and the same number of parameters (Table 2), and the improvement is larger there (Table 3). So we see the improvement both when capacity is held constant but training exposure differs, and when exposure is held constant but capacity differs.

We cannot make both values equal in one run when we use one shared table of this size. The different settings create very different numbers of token types (Table 1). So if we keep the same number of table rows, the input sequences end up with very different lengths. And if we want equal input length, we would need a different number of table rows (Section 3.1). This is why we report two setups (Table 2).

The table size also seems to matter much less than where we place the cuts. For every setting except the aligned one, we use a full table in one setup and a smaller table in the other. Its results on held-out data change only a little, much less than the gap between the aligned setting and the “zero-alignment” setting (Tables 2 and 3). This is only a rough check, because the number of training passes also changes, and for Hungarian some of the gain may come from the smaller table. The aligned setting also uses the smaller model, so it is not winning because it has more parameters. We will add this point in Section 5 and say clearly that neither number is an exact estimate, which is why we report both.

AuthorsOpenReview ↗

Clarification and Response to Reviewer v2z3 (PART 2)

  1. (Weakness 3)

We agree the assessment is rather narrow, that's absolutely intentional and stems from the research question itself: Does setting token boundaries at the level of morphemes lead to any improvement in "language modeling when the number of tokens is kept constant?"

Testing this requires training models from scratch with controlled tokenisation. The existing models (Llama, Qwen etc.) all use different tokenisers, different training data and different architectures so it's very hard to attribute any performance differences to the boundary placement itself. We therefore use the same 4-layer Transformer architecture, three random seeds and two experimental designs: one where we match the segmentation count and one where we additionally match the training symbol exposure. This allows us to look at boundary alignment in isolation from all the other confounds that would arise if we were comparing independently pretrained models.

We clarify three points about the scope:

1- Programming languages and naming conventions: WinMorph contains 83,900 Windows identifiers covering three naming conventions: underscore-delimited constants (FILE_ATTRIBUTE_READONLY), PascalCase functions (RegSetValueExW) and Hungarian style type prefixes (lpszFileName). The conventions are taken from Microsoft’s published naming rules rather than from the syntax of any one programming language. We do not claim that our findings generalize to identifiers from all programming languages and all naming conventions.

2- Cross domain evidence: We do the same experiments on 84k Hungarian words (a dataset completely unrelated to Windows APIs). Under symbol matching we get a full alignment reduction in held-out bits per character of 18.9% on WinMorph and 30.5% on Hungarian (Table 3). This demonstrates that the benefit we see is not restricted to Windows naming conventions, though it doesn't prove generalization to other programming languages.

3- Model scale and practical applications: we evaluate isolated identifiers using a small Transformer, not identifiers embedded in the source code. We therefore do not make any claims as to improvements for larger pretrained models, practical BPE or Unigram tokenizers, or for any downstream tasks such as code completion or vulnerability detection. Those would have to be the subject of separate experiments.

Our contribution is a controlled demonstration that morpheme-aligned boundaries lead to better modeling of novel identifiers at a fixed segmentation count (under matched training exposure). Whether this advantage carries over to larger model sizes, with real-world tokenizers or in the case of source code is an open question (as noted in Appendix A).

  1. (Weakness 4)

This is a valid question and the paper gives an initial answer. None of the held-out identifiers are present in the training data (Section 4). They do however share constituent parts with the training identifiers. This is at the heart of our hypothesis: the model should be able to generalize to new identifiers by combining parts that it has already seen.Our evaluation is therefore one of new combinations of familiar parts rather than entirely novel identifier structures.

Two further findings provide some context. Firstly, under segmentation matching, the held-out identifiers require more input symbols than the training identifiers in all arms (Table 2), which again points to poorer embedding coverage of the held-out names. That does of course not prove that the held-out set is structurally different. Secondly the improvement is also seen on Hungarian words where Windows naming conventions don't really come into play (Section 5.1, Table 3). That suggests the benefit isn't restricted to Windows naming convention.

This question also matters for cybersecurity. Malware analysis often involves previously unseen programs that import or call familiar Windows APIs. Understanding whether models can represent unfamiliar identifier combinations using learned constituents is therefore relevant. We do not claim to demonstrate improved malware detection, since no downstream security task is evaluated.

The reviewer’s suggestion motivates a more targeted analysis. Our hypothesis predicts that the benefit should be stronger when held-out identifiers contain familiar morphemes and weaker as the proportion of unseen morphemes increases. In the revised paper, we will group held-out identifiers by the proportion of reference morphemes observed during training and report performance across alignment settings within each group. This will quantify how much the gain depends on familiar constituents, while leaving broader structural generalization as a separate question.

Thank you for the clarifications. The proposed analysis of held-out identifiers based on morpheme overlap is useful, although its addition also makes it difficult to fully assess the completeness of the current version of the paper. I am still not entirely convinced by the broader contribution of the paper. As I mentioned in my review, the evaluation is limited to a relatively narrow set of domains and small-scale models, making it difficult to see how these findings would translate to practical settings or inform the design and adoption of tokenizers in language models. I appreciate the authors responses, but my overall concerns remain, and I will maintain my original rating.

Reviewer_vAfNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WINMORPH, a corpus of 83,900 Windows identifiers with reference segmentations derived from naming conventions, and constructs segmentation variants that vary boundary alignment while controlling segmentation-piece counts. It evaluates segmentation consistency without training and uses a small Transformer to model seen and held-out strings, with an additional experiment on Hungarian natural-language words. Across segmentation-matched and training-symbol-matched settings, fully aligned variants achieve lower held-out bits per character (BPC). The corpus and experiments provide a testbed for studying morphological alignment in tokenization.

Critical strengths
  • A useful reference-segmented testbed. WINMORPH enables deliberate variation of boundary alignment rather than relying solely on correlations across existing tokenizers. The Hungarian experiment extends the evaluation beyond Windows naming conventions.
  • Transparent reporting of experimental differences. The paper distinguishes segmentation-piece counts from actual model-input lengths and reports coverage, training passes, and parameter counts. Its symbol-matched and reuse controls help assess alternative explanations for the headline improvements.
Critical weaknesses
  • Limited practical design impact. Morphological alignment is academically interesting, but is only a proxy for objectives such as BPC, downstream quality, and computational cost. I would expect data-driven selection against those objectives to produce better practical choices than prioritizing morphology. The paper does not establish alignment's added design value: its small-model experiments concern isolated Windows identifiers and Hungarian words, while practical tokenizers and contextual evaluations remain future work (Appendix A).
  • Limited isolation of the source of improvement. Segmentation matching leaves input length and training exposure unequal; symbol matching still changes coverage, parameter count, and held-out length (Table 2). The "Reuse-matched" control also has substantially less reuse than the aligned variant (2.26 versus 5.55), yet recovers approximately 54% of its held-out BPC improvement under symbol matching (Section 5). The remaining gain therefore does not isolate morphological meaning from fragment reuse and other accompanying changes, limiting the study's explanatory contribution.
Questions for authors
  • Practical significance: What design value does morphological alignment add beyond data-driven selection against BPC, downstream quality, and computational cost? What evidence supports prioritizing it on realistic code or mixed-text inputs?
  • Explanatory isolation: Can a control matching the aligned variant's reuse level separate alignment from fragment reuse? How do the remaining coverage, parameter-count, and held-out-length differences affect the symbol-matched comparison, and which conclusions remain supported without isolating these effects?
2 comments
AuthorsOpenReview ↗

Clarification and Response to Reviewer vAfN

We thank the reviewer for a careful and fair review.

  1. (Weakness 1, Question 1)

The paper does not propose morphological alignment as a replacement for empirical evaluation. We evaluate one of the reviewer’s suggested metrics—bits per character (BPC) on held-out identifiers—and show that greater alignment improves performance at a fixed segmentation token count, across both experimental designs and datasets (Table 3).

However data-driven selection can only select from among the tokenizers under comparison. The 15 traditional tokenizers we evaluated all fall short of full alignment and their alignment and token count are co-varying (Figure 1a). Selection among them therefore cannot in any way isolate the benefit of correct boundary placement. Our results do suggest that morphological alignment is a useful additional criterion in the design of candidate tokenizers, whose performance can then be assessed in a data-driven manner.

Alignment also has potential computational benefits. Under segmentation matching the aligned WinMorph arm requires fewer input symbols (4.79 vs 14.23 per training identifier) whilst keeping the same parameter count (13.8M). Under symbol matching it achieves better BPC with a vastly reduced number of parameters (2.3M vs 13.6M; Tables 2 and 3). That suggests greater representation efficiency though we don't directly measure inference latency or computational cost.

For Windows identifiers the approach is equally viable to implement. Our reference boundaries are in accordance with the published Microsoft naming conventions, enabling rule-based splitting without the need for manual annotation. The same rules are applied to both the training and held-out identifiers (Sections 1 and 3.2). The question of course is whether they will work well when incorporated into standard BPE or Unigram tokenisers.

The setting is also pertinent to cybersecurity applications (malware analysis in particular, where you may have a sequence of imported/called Windows APIs, possibly with no source code at all and where compact models can be of use). Our experiments look at the representation of such identifiers, not at malware detection performance.

We agree that our controlled benchmark does not establish gains on full source files, mixed text inputs or downstream cybersecurity tasks. We'll make this limitation explicit in the main text (Appendix A). Our contribution is to establish morphological alignment as a promising tokenizer design criterion, not to suggest that it should in any way replace empirical selection based on downstream performance and computational cost.

  1. (Weakness 2, Question 2)

Your reading of Table 2 and the control is right, but we draw a different conclusion. We claim that cutting names at the right boundaries helps even when the total number of tokens is kept the same, and we see this in both setups and both datasets (Table 3). This claim does not rely on splitting the gain into “meaning” versus reuse, and the paper does not make that split (Sections 5 and 6).

Reuse is simply how correct cuts help. As cuts get closer to the correct ones, the same pieces show up more often (Table 1). The control uses only wrong cuts, and it does about as well as the settings where pieces show up at about the same rate (Tables 1 and 3). So the paper does give an explanation: much of the gain comes from reuse (Section 5). We tried many settings, but we could not make the control reach the aligned setting’s level of reuse (Section 3.2). So far, correct cuts are the best way we found to get high reuse.

On WinMorph, we also see the gain in two cases: (1) when the table size is the same but the model sees the training data a different number of times, and (2) when the model sees the training data the same number of times but the table size is different (Tables 2 and 3). We cannot make both table size and training repeats equal in one run with one shared table, because the settings create very different kinds of tokens (Table 1, Section 3.1).

In cybersecurity, malware analysis models read Windows API names and keep meeting new ones. For a new name, a model can still use pieces it has seen before. The paper does not include a malware task, so this is only the motivation, not a result. We will rename the setting “reuse control” and state this scope clearly in the main text.

Reviewer_vAfNOpenReview ↗

Response to Rebuttal

The rebuttal clarifies that morphological alignment is proposed as a way to design candidate tokenizers, not as a substitute for empirical evaluation. I agree that fragment reuse can explain part of the alignment benefit rather than invalidate it: the aligned variants achieve lower held-out BPC in this testbed even when training-symbol exposure is matched. My main concern remains the practical significance of that result. The reuse control does not match the aligned variant's reuse rate, so it remains unclear whether similarly reusable pieces need to respect morpheme boundaries. More importantly, the experiments do not show whether morphology adds value over a competitive data-driven tokenizer on code in context. I therefore retain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.