MRT: Regularizing Tokenization with Morphology for Low-Resource Indic Languages

Standard subword tokenizers Byte-Pair Encoding (BPE), Unigram, and WordPiece choose merges purely from corpus frequency statistics, so the resulting tokens routinely cut across morpheme boundaries. In morphologically rich languages with limited data, this fragmentation spreads probability mass over redundant subword forms and hurts downstream accuracy. Current alternatives are unsatisfying: hard-constraint approaches forbid cross-boundary merges outright, but depend on supervised morphological annotation and expose no knob for trading compression against morphological fidelity, while unsupervised segmentation learners become unreliable precisely when data is scarce. We propose Morphology-Regularized Tokenization (MRT), which injects morphological priors into BPE using nothing more than small, curated suffix inventories. MRT comes in two variants. MRT-Base pre-splits words at morpheme boundaries, enforcing a hard prohibition on merges that span them. MRT-Penalized instead attaches a normalised, scale-invariant penalty to each candidate merge, proportional to how often that merge crosses a morpheme boundary; a single strength parameter then tunes the compression-fidelity trade-off continuously. This makes MRT-Penalized strictly more general than prior hard-constraint methods: at zero strength it recovers the original BPE objective exactly, and as the strength increases it interpolates smoothly toward a morpheme-only merge regime. Our evaluation spans nine Indic languages, two corpora, an intrinsic analysis of suffix consistency, and five extrinsic tasks. MRT-Penalized cuts perplexity by roughly 32–65% relative to the strongest baseline, gives consistent gains on downstream tasks, and collapses the surface realisations of morphological suffixes into a markedly smaller set of canonical subwords all using one default penalty strength that transfers across languages with no per-language tuning.