acceptodds
Under review as a conference paper at ICLR 2027

MRT: Regularizing Tokenization with Morphology for Low-Resource Indic Languages

10 views

Abstract

Standard subword tokenizers Byte-Pair Encoding (BPE), Unigram, and WordPiece choose merges purely from corpus frequency statistics, so the resulting tokens routinely cut across morpheme boundaries. In morphologically rich languages with limited data, this fragmentation spreads probability mass over redundant subword forms and hurts downstream accuracy. Current alternatives are unsatisfying: hard-constraint approaches forbid cross-boundary merges outright, but depend on supervised morphological annotation and expose no knob for trading compression against morphological fidelity, while unsupervised segmentation learners become unreliable precisely when data is scarce. We propose Morphology-Regularized Tokenization (MRT), which injects morphological priors into BPE using nothing more than small, curated suffix inventories. MRT comes in two variants. MRT-Base pre-splits words at morpheme boundaries, enforcing a hard prohibition on merges that span them. MRT-Penalized instead attaches a normalised, scale-invariant penalty to each candidate merge, proportional to how often that merge crosses a morpheme boundary; a single strength parameter then tunes the compression-fidelity trade-off continuously. This makes MRT-Penalized strictly more general than prior hard-constraint methods: at zero strength it recovers the original BPE objective exactly, and as the strength increases it interpolates smoothly toward a morpheme-only merge regime. Our evaluation spans nine Indic languages, two corpora, an intrinsic analysis of suffix consistency, and five extrinsic tasks. MRT-Penalized cuts perplexity by roughly 32–65% relative to the strongest baseline, gives consistent gains on downstream tasks, and collapses the surface realisations of morphological suffixes into a markedly smaller set of canonical subwords all using one default penalty strength that transfers across languages with no per-language tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Tokenization, Vocabulary, Morphology, Indic Language

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_n2BYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Morphology-Regularized Tokenization (MRT), a way of
bringing morphological knowledge into BPE for low-resource Indic languages. Morpheme boundaries come from a cheap rule-based oracle that strips suffixes using a short, hand-curated suffix list per language (11 to 44 entries), so no full morphological annotation is needed. Two variants build on this
oracle. MRT-B pre-segments the corpus at the oracle boundaries and trains
standard BPE on the result, so no learned token can cross a boundary. MRT-P, which I see as the main contribution, leaves the corpus intact and downweights each candidate merge by the fraction of its occurrences that cross a boundary, controlled by the hyperparameter λ\lambda. At λ=0\lambda = 0 this is standard BPE, and for large λ\lambda it moves towards purely
morpheme-respecting merges. The authors evaluate on nine Indic languages and two corpora. Language models are trained from scratch and compared in bits
per character, alongside intrinsic morphological metrics and several
downstream tasks. With an 8-layer GPT-2, MRT-P reaches the lowest BPC in 7
of 9
languages, although the margins are small.

Critical strengths
  • Interesting idea, especially the aspect of the tunable pareto between standard BPE and morpheme-only merging. This enables flexibility linguistic rigorousness vs statistical methods. Nice property: Segmentation happens automatically.

  • Broad evaluation (within one language family) of 9 languages and 3 model scales and also shared vocabulary experiment

Critical weaknesses
  • Performance results are not good, only 2.8%

  • MRT-P has many differences apart from morphological informedness. Most of all it has the smallest vocabulary of all compared methods. Also it has higher fragmentation. This could be the reason for improvement (instead of morphology). Also: only one seed.

  • The authors interpret the results in appendix F5 as robustness. But they could also be interpreted as suffix information having little impact: swapping half of the suffix list barely affects NER.

  • Proposition 2.2 seems wrong to me. It should be the other way around. The conclusion would then also be wrong.

  • I also have doubts about the proof of proposition 2.3. Induction does not hold because \lambda and \lambda' diverge after the first step. (This point is based on input from AI.)

  • Missing citations of simillar ideas: o Hofmann et al. (2022, ACL), FLOTA [1] o Jabbar (2024), MorphPiece [2] o Arnett et al. (2025), MorphScore [3]

  • Results in the multilingual setting are weak. So even if one believes the strength of the method in the monolingual setting, the paper does not provide good evidence that it works in the multilingual setting.

[1] Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. 2022. An
Embarrassingly Simple Method to Mitigate Undesirable Properties of
Pretrained Language Model Tokenizers https://aclanthology.org/2022.acl-
short.43/. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 385–393, Dublin,
Ireland. Association for Computational Linguistics.

[2] Jabbar, Haris. "MorphPiece: A linguistic tokenizer for large language
models. arXiv." arXiv preprint arXiv:2307.07262 (2024)

[3] Arnett, C., Hudspeth, M., & O'Connor, B. (2025). Evaluating
morphological alignment of tokenizers in 70 languages. arXiv preprint
arXiv:2507.06378.

Questions for authors
  1. Do you have controls that would rule out the alternative explanations in Weakness 1? For example, how does MRT-P behave when the penalty is computed from random boundary positions at the same rate as the oracle boundaries,
    and do the BPC improvements survive this control? A comparison with BPE and Unigram trained at MRT-P’s vocabulary size would also help.

  2. Regarding Weakness 3, could the authors either weaken the statement of
    Proposition 2.3 accordingly, or report the number of cross-boundary merges
    as a function of λ\lambda from the existing sweep, to show that
    monotonicity holds at least empirically?

2 comments

Thank you for a very thorough and constructive review. You are right on both propositions, and we ran every control you asked for, with 3 seeds for the main comparisons. We hope this addresses your concerns, and we would be grateful if you would consider raising your score.

W1 / Q1. Is the morphological signal what drives the gains?

To address this, we isolated each factor. BPC is averaged over 9 languages and 3 seeds. Seed std is 0.13%, so the original margins (BPE +3.5%, Unigram +1.2%) are 9-27× the noise.

TokenizerBPC (± seed std)Boundary recall
BPE @ MRT-P's |V|1.959 ± .0030.11
Unigram @ MRT-P's |V|1.915 ± .0030.34
MRT-P, λ = 0 (same pipeline)1.922 ± .0030.40
MRT-P, random boundaries (3 draws)1.926 ± .0020.44-0.54
MRT-P, curated1.925 ± .0020.71
MRT-P, expanded list1.918 ± .003*0.69*

*7 languages; recall vs. external gold, where curated MRT-P scores 0.58.

  • Fertility does not explain the gain. Fertility-matched baselines are worse (BPE by 2.4%), and Unigram at matched fertility only ties MRT-P.
  • Vocabulary explains the gap to Unigram (0.5% lower at matched |V|), but MRT-P is still 1.7% ahead of BPE in 8 of 9 languages.
  • The λ = 0 and random controls show that the training/export pipeline accounts for much of the BPC gain over BPE. What the real boundaries add is morphological alignment at negligible LM cost. Boundary recall rises from 0.40 (λ = 0) and 0.44-0.54 (random, at similar fertility) to 0.71, while BPC stays within 0.2%.
  • Under the hard constraint, real boundaries beat random ones in BPC: MRT-B wins in 6 of 9 languages.
  • Better lists help further: on the same 7 languages, the expanded list lowers BPC by 0.2% vs. curated.

We will report this decomposition in the main text.

Tamil. Your diagnosis is correct: at λ = 6.5 only 31 Tamil merges cross a boundary. With the corrected list (oracle recall 0.13 → 0.59), Tamil BPC improves monotonically with the amount of real morphological information: random 1.797 → λ = 0 1.793 → drop-50% 1.791 → curated 1.789 → expanded 1.783.

Perturbation study. We agree that a flat response is equally consistent with the list having little influence. We now judge changes against the seed noise floor and report the dose-response above rather than claiming robustness. Bengali and Hindi vary by under 0.7%, so the alignment gains cost little BPC.

W2. Proposition 2.2

To address this: you are right. We will restate it as B(w) ⊆ P_τB(w), i.e. RecB = 1 on the segmented corpus, not bBVR = 0. No empirical result used the wrong direction.

W3 / Q2. Proposition 2.3

To address this, we will do both things you suggested:

  • Weaken the statement to per-step monotonicity, which holds up to the first divergence.
  • Report the counts. The number of crossing merges CB(λ), measured over a 9-point λ grid in all 9 languages and a 20-point grid for Bengali, is non-increasing in 8 of 9 languages. The single exception is +6 merges in Bengali (λ = 3 → 3.5, 163 → 169). CB falls by over 95% from λ = 0 to λ = 6.5 (e.g. bn 3,431 → 132).

We will also correct Proposition 2.4: the λ → ∞ limit is BPE restricted to pairs that never cross a boundary, not MRT-B.

W4. Related work

To address this, we added all three works and ran two of them as baselines:

  • FLOTA. It modifies inference only, so it is complementary to MRT. We will correct our "requires supervision" sentence.
  • MorphPiece-style (built from UniMorph tables, 5 languages). This is the closest analog to MRT-B.
FLOTA-BPEFLOTA-UniMorphPiece (5 langs)MRT-BMRT-P
Mean BPC1.9761.9601.9801.9651.925

MRT-P is lowest in 8 of 9 languages (tied with FLOTA-Unigram in Bengali). Downstream, it wins 31 of 37 comparisons against FLOTA and 16 of 22 against MorphPiece.

Arnett et al. Their gold construction matches ours, and their finding agrees with our Table 28. We also ran their MorphScore. On Tamil, MRT-P scores 0.51, against 0.30 for Unigram and 0.09 for BPE. Our claim is about an intervention, which is what the W1 controls test.

W5. Multilingual setting

To address this, we tuned λ ∈ {2, 3, 4, 5, 6.5} with a shared vocabulary. λ = 5 is best (1.9085). MRT-P beats BPE in 8 of 9 languages and also beats MRT-B, but Unigram remains stronger in the joint setting. We will state this as a limitation in the main text and scope the LM claim to monolingual models.

W6. Figure 1

To address this: thank you, we will fix both (11-44 suffixes; λ = 6.5, range [5.5, 7.0]).

Your review has made the paper considerably more careful. We believe the controls now show clearly what MRT contributes, namely morphological alignment at negligible LM cost and consistent gains over BPE. If our response addresses your concerns, we would sincerely appreciate it if you would consider increasing your score.

I thank the authors for engaging with the review and for their additional experiments!

Some of my concerns have been addressed. However, others have not been: robustness, doubts about the math (I'm not a mathematician, so the things I found are just the most obvious problems), weak multilinguality, Tamil (very small differences), the fact that gains may be consistent, but they are quite small.

I will therefore keep my rating.

Thank you!

Reviewer_YwNXOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The authors propose two new additions to subword tokenizers to make them tokenize Indic languages better. First, they propose splitting words at the boundaries of suffixes before running BPE (which require some manual effort in the form of a hand-written suffix list per language). They also propose, instead of splitting, instead down-weighting a candidate merge by how often it crosses a boundary on one of the suffix. They train some small language models from scratch to evaluate this, with their second proposal getting the lowest loss in 7 of 9 languages. They also do favorably in comparisons on downstream tasks, such as named entity recognition.

Critical strengths
  • the method is simple, cheap, and a drop in replacement! the paper also contains broad evaluations to support some broad claims, which is very good.
Critical weaknesses
  • One thing i noticed is that vocabulary size may explain the language-modelling gains. So looking at Table 11, MRT-P’s vocabulary is much less than the size of most of the baselines’: roughly 16K against mostly 32K. If you match Table 1 to Table 11, MRT-P seems to do really well when the strongest baseline has a vocabulary of 21–32K. But, in the four languages where the strongest baseline has 16K, MRT-P only ties. similarly: MRT-B, which uses the same suffix boundaries but has the full 32K vocabulary, wins in only three languages and loses on the rest. My impression is that this pattern is suggestive of vocabulary size accounting for much of the advantage in bits per character, it seems like you should try to retrain Unigram and BPE at about 16K in all nine languages and rerun Table 1.
Questions for authors
  • How much does vocabulary size affect the difference in performance in language modelling?
2 comments

Thank you for your careful reading and for the kind words about the method. Your vocabulary-size observation was spot on, and we ran exactly the comparison you suggested. We hope the results below address your concern, and we would be grateful if you would consider raising your score.

Weakness: vocabulary size may explain the LM gains

To address this concern, we first explain where the ~16K vocabulary comes from. MRT-P's merges are exported as a SentencePiece model, and that export caps the vocabulary at 55% of the distinct pieces. The size was therefore not tuned. We will state this in Section 3.

We then retrained BPE, Unigram, WordPiece and MRT-B at exactly MRT-P's |V| in all nine languages, and trained the GPT-2 models with 3 seeds each. The table shows BPC (mean ± std over 3 seeds).

Lang.|V|BPEUnigramWordPieceMRT-BMRT-P
as16,2821.956 ± .0041.899 ± .0051.956 ± .0031.915 ± .0011.901 ± .002
bn16,0331.881 ± .0031.833 ± .0021.881 ± .0031.862 ± .0011.840 ± .001
gu15,9962.094 ± .0042.040 ± .0032.084 ± .0042.082 ± .0032.058 ± .002
hi15,4741.994 ± .0031.966 ± .0011.990 ± .0041.980 ± .0031.976 ± .001
mr16,8412.106 ± .0082.054 ± .0052.096 ± .0012.095 ± .0012.063 ± .004
ne16,5301.942 ± .0021.878 ± .0021.928 ± .0011.897 ± .0011.886 ± .002
pa15,1142.007 ± .0051.982 ± .0052.003 ± .0031.995 ± .0021.986 ± .003
ta17,2621.832 ± .0011.789 ± .0011.822 ± .0041.808 ± .0011.789 ± .002
te8,3351.813 ± .0021.792 ± .0011.810 ± .0041.806 ± .0061.820 ± .002

What this shows:

  • Against BPE, the algorithm MRT-P modifies, the advantage holds at matched size. MRT-P is 1.7% lower on average, in 8 of 9 languages, which is more than 10× the seed noise. It also beats WordPiece in 8 of 9.
  • Suffix-boundary segmentation helps even under a hard constraint. At matched size, MRT-B beats BPE in all 9 languages, and real boundaries beat random ones in 6 of 9 (see our response to Reviewer n2BY).
  • Unigram benefits most from the smaller vocabulary. As you anticipated, it moves ahead by 0.5% on average. MRT-P ties it in Assamese and Tamil, within seed variation.
  • On downstream tasks, MRT-P keeps its lead at matched size. It beats Unigram @|V| in 28 of 38 comparisons and BPE @|V| in 30 of 37.

Question: how much does vocabulary size affect the LM difference?

To answer your question directly, we trained BPE, Unigram and MRT-P at K = 8K, 16K and 32K in all nine languages (4K, 8K and 16K for Telugu). Mean BPC:

8K16K32KGain 32K → 16K
BPE1.9571.9561.9921.8%
Unigram1.9111.9161.9582.2%
MRT-P1.9211.9241.9511.4%

So vocabulary size matters substantially at this data scale: every tokenizer gains roughly 1.4-2.2% going from 32K to 16K. MRT-P is the least sensitive to K and is lowest in 8 of 9 languages at the standard 32K. On average, its advantage over BPE persists at every size we tested.

In short, your intuition was right: vocabulary size accounts for part of the Table 1 gap, and the matched comparison now makes its contribution explicit. MRT-P still gives a consistent gain over BPE, and its morphological prior adds much better alignment at no LM cost: boundary recall rises from 0.11 for BPE to 0.71 for MRT-P at the same |V|. We will:

  • replace Table 1 with the matched 3-seed version,
  • add the K-sweep as a figure,
  • calibrate the margins in the abstract to match.

On MRT-B

To address your second observation: MRT-B was at 32K in Table 1. At matched size it improves by 1.4% and beats BPE everywhere. Its weaker showing in Table 1 was therefore largely a vocabulary effect, as you suspected. We will report this interaction explicitly.

Thank you again. Your comment led to the most informative control in the revision, and we believe the paper is clearly stronger for it. We hope this fully answers your question, and if so, we would truly appreciate it if you would consider increasing your score.

Thanks! I appreciate the work put into this. This, for me, makes the results somewhat mixed. I will retain my score and increase confidence.

Reviewer_83bBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary
  • The paper presents MRT regularization for tokenizer.

  • It is using suffix vocabulary list to improve tokenization.

  • It involves working on 9 different Indian Languages.

  • The languages are from two families.

  • Suffix dictionary size is too low (44)

Critical strengths
  • Morphological penalty for BPE is introduced.

  • Work is on low resource languages which is valuable.

  • Base and penalty based approaches are important and relavant aspects.

Critical weaknesses
  • Suffix dictionary size is too low (11-44), that might be the reason for less improvements is some langauges. Refer compound piece and unimorph for increasing suffix dictionaries.

  • Morphilogical aspects for tokenization is not new. E.g. Morphtok paper.

  • Generative down stream tasks are not conducted, which are important for tokenizers.

Questions for authors

Please answer based on weaknesses mentioned above.

1 comment

Thank you for your review and for recognising the value of working on low-resource languages. We ran new experiments for all three weaknesses. We hope they address your concerns, and if so, we would be grateful if you would consider raising your score.

W1. The suffix dictionaries are small (11-44 entries)

Following your suggestion, we built expanded suffix lists from UniMorph (as, bn, gu, hi, te) and the UD training splits (mr, ta). Recall is on gold the lists never saw (UD test sets, held-out UniMorph lemmas). BPC: mean ± std over 3 seeds.

Lang.|S|Oracle recallMRT-P BPC
as11 → 710.25 → 0.771.900 ± .002 → 1.898 ± .002
bn44 → 910.49 → 0.841.843 ± .001 → 1.849 ± .001
gu15 → 380.62 → 0.722.063 ± .003 → 2.056 ± .002
hi26 → 360.41 → 0.581.978 ± .002 → 1.979 ± .004
mr17 → 340.15 → 0.492.057 ± .002 → 2.045 ± .005
ta16 → 700.13 → 0.591.789 ± .002 → 1.783 ± .003
te15 → 330.04 → 0.761.824 ± .003 → 1.814 ± .002

You were right that list size was a limiting factor:

  • Oracle recall roughly doubles on average.
  • MRT-P's boundary recall against external gold rises from 0.58 to 0.69.
  • BPC improves in 5 of 7 languages, most where the curated lists were weakest (mr −0.5%, te −0.5%, ta −0.3%).
  • Downstream, the expanded-list MRT-P beats the curated one in 26 of 32 comparisons.

Since MRT only needs a boundary set B(w), we also added CompoundPiece boundaries to the oracle for the six languages it covers. Recall against CompoundPiece's own splits rises from 0.57 to 0.82, and recall against external gold rises in Tamil (0.17 → 0.34), Telugu (0.18 → 0.28) and Gujarati, at a small BPC cost (+0.37%). We will release the expanded lists.

W2. Morphology-aware tokenization is not new (MorphTok)

To address this: we agree, and the paper cites MorphTok, MorphBPE and García et al. as the closest prior work. We differ in three ways:

  1. A soft penalty, not a hard filter. MorphTok pre-segments the corpus with a hard accept/reject rule. MRT-P instead reweights each merge by how often it crosses a boundary, f(a,b)·max(0, 1 − λr(a,b)), so one λ trades compression against morphological fidelity. MRT-B, in every table, is the hard-constraint counterpart.
  2. Almost no supervision. An 11-44-entry suffix list per language replaces morpheme-segmented data, which lets us cover nine languages.
  3. New baselines. Against FLOTA and MorphPiece, MRT-P has lower BPC in 8 of 9 languages (tied in the ninth) and wins 31 of 37 downstream comparisons against FLOTA.

W3. No generative downstream tasks

To address this, we added two generative tasks. A Transformer is trained from scratch per tokenizer, and outputs are scored on detokenized text.

(a) MT, English → 9 languages (OPUS-100 train, FLORES-200 devtest, chrF++). @V: retrained at MRT-P's vocabulary size. Best in bold, second-best marked ².

BPEUnigramBPE @VUnigram @VMRT-BMRT-P
bn17.9116.4318.6619.47²18.2319.65
hi16.5114.3714.0116.75²15.3018.39
ta12.1813.1713.0113.72²13.8012.94
te9.827.0110.35²9.5210.2710.93
gu12.45²11.0112.3411.8611.7014.52
mr5.105.144.885.64²4.696.41
ne14.7917.02²14.3411.699.9017.08
pa7.74²6.847.777.016.487.70
as5.824.938.6110.20²11.535.27

MRT-P has the best chrF++ in 6 of 9 languages, and an MRT variant is best in 8 of 9. MRT-P's gain over the best baseline is significant under paired bootstrap (p < 0.001) in four languages: hi +1.65, gu +2.07, mr +0.77 and te +0.58. This suggests that respecting suffix boundaries helps when generating inflected words.

(b) XL-Sum summarization (8 languages, ROUGE-L; same marking):

BPEUnigramBPE @VUnigram @VMRT-BMRT-P
bn6.167.95²6.207.937.198.20
hi19.7819.3619.4619.1319.64²17.83
ta8.509.598.109.948.459.83²
te4.734.37²4.353.974.353.55
gu9.789.058.619.79²8.8910.15
mr10.228.9210.1310.4910.7710.56²
ne10.9111.7310.6412.2811.4512.23²
pa15.1514.2714.7414.98²14.6914.71

An MRT variant is best in 3 of 8 languages, and MRT-P is within 0.5 of the best system in 6 of 8. On chrF, MRT-P is best in Gujarati (+1.7) and Punjabi.

Both tasks will be added to the paper.

We believe these additions directly address each of your concerns and substantially strengthen the paper. If you agree, we would sincerely appreciate it if you would reconsider your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.