- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Morphology-Regularized Tokenization (MRT), a way of
bringing morphological knowledge into BPE for low-resource Indic languages.
Morpheme boundaries come from a cheap rule-based oracle that strips suffixes
using a short, hand-curated suffix list per language (11 to 44 entries), so
no full morphological annotation is needed. Two variants build on this
oracle. MRT-B pre-segments the corpus at the oracle boundaries and trains
standard BPE on the result, so no learned token can cross a boundary. MRT-P,
which I see as the main contribution, leaves the corpus intact and downweights each candidate merge by the fraction of its occurrences that cross a
boundary, controlled by the hyperparameter . At
this is standard BPE, and for large it moves towards purely
morpheme-respecting merges. The authors evaluate on nine Indic languages and
two corpora. Language models are trained from scratch and compared in bits
per character, alongside intrinsic morphological metrics and several
downstream tasks. With an 8-layer GPT-2, MRT-P reaches the lowest BPC in 7
of 9
languages, although the margins are small.
Critical strengths
-
Interesting idea, especially the aspect of the tunable pareto between standard BPE and morpheme-only merging. This enables flexibility linguistic rigorousness vs statistical methods. Nice property: Segmentation happens automatically.
-
Broad evaluation (within one language family) of 9 languages and 3 model scales and also shared vocabulary experiment
Critical weaknesses
-
Performance results are not good, only 2.8%
-
MRT-P has many differences apart from morphological informedness. Most of all it has the smallest vocabulary of all compared methods. Also it has higher fragmentation. This could be the reason for improvement (instead of morphology). Also: only one seed.
-
The authors interpret the results in appendix F5 as robustness. But they could also be interpreted as suffix information having little impact: swapping half of the suffix list barely affects NER.
-
Proposition 2.2 seems wrong to me. It should be the other way around. The conclusion would then also be wrong.
-
I also have doubts about the proof of proposition 2.3. Induction does not hold because \lambda and \lambda' diverge after the first step. (This point is based on input from AI.)
-
Missing citations of simillar ideas: o Hofmann et al. (2022, ACL), FLOTA [1] o Jabbar (2024), MorphPiece [2] o Arnett et al. (2025), MorphScore [3]
-
Results in the multilingual setting are weak. So even if one believes the strength of the method in the monolingual setting, the paper does not provide good evidence that it works in the multilingual setting.
[1] Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. 2022. An
Embarrassingly Simple Method to Mitigate Undesirable Properties of
Pretrained Language Model Tokenizers https://aclanthology.org/2022.acl-
short.43/. In Proceedings of the 60th Annual Meeting of the Association for
Computational Linguistics (Volume 2: Short Papers), pages 385–393, Dublin,
Ireland. Association for Computational Linguistics.
[2] Jabbar, Haris. "MorphPiece: A linguistic tokenizer for large language
models. arXiv." arXiv preprint arXiv:2307.07262 (2024)
[3] Arnett, C., Hudspeth, M., & O'Connor, B. (2025). Evaluating
morphological alignment of tokenizers in 70 languages. arXiv preprint
arXiv:2507.06378.
Questions for authors
-
Do you have controls that would rule out the alternative explanations in Weakness 1? For example, how does MRT-P behave when the penalty is computed from random boundary positions at the same rate as the oracle boundaries,
and do the BPC improvements survive this control? A comparison with BPE and Unigram trained at MRT-P’s vocabulary size would also help. -
Regarding Weakness 3, could the authors either weaken the statement of
Proposition 2.3 accordingly, or report the number of cross-boundary merges
as a function of from the existing sweep, to show that
monotonicity holds at least empirically?
2 comments
I thank the authors for engaging with the review and for their additional experiments!
Some of my concerns have been addressed. However, others have not been: robustness, doubts about the math (I'm not a mathematician, so the things I found are just the most obvious problems), weak multilinguality, Tamil (very small differences), the fact that gains may be consistent, but they are quite small.
I will therefore keep my rating.
Thank you!
Thank you for a very thorough and constructive review. You are right on both propositions, and we ran every control you asked for, with 3 seeds for the main comparisons. We hope this addresses your concerns, and we would be grateful if you would consider raising your score.
W1 / Q1. Is the morphological signal what drives the gains?
To address this, we isolated each factor. BPC is averaged over 9 languages and 3 seeds. Seed std is 0.13%, so the original margins (BPE +3.5%, Unigram +1.2%) are 9-27× the noise.
*7 languages; recall vs. external gold, where curated MRT-P scores 0.58.
We will report this decomposition in the main text.
Tamil. Your diagnosis is correct: at λ = 6.5 only 31 Tamil merges cross a boundary. With the corrected list (oracle recall 0.13 → 0.59), Tamil BPC improves monotonically with the amount of real morphological information: random 1.797 → λ = 0 1.793 → drop-50% 1.791 → curated 1.789 → expanded 1.783.
Perturbation study. We agree that a flat response is equally consistent with the list having little influence. We now judge changes against the seed noise floor and report the dose-response above rather than claiming robustness. Bengali and Hindi vary by under 0.7%, so the alignment gains cost little BPC.
W2. Proposition 2.2
To address this: you are right. We will restate it as B(w) ⊆ P_τB(w), i.e. RecB = 1 on the segmented corpus, not bBVR = 0. No empirical result used the wrong direction.
W3 / Q2. Proposition 2.3
To address this, we will do both things you suggested:
We will also correct Proposition 2.4: the λ → ∞ limit is BPE restricted to pairs that never cross a boundary, not MRT-B.
W4. Related work
To address this, we added all three works and ran two of them as baselines:
MRT-P is lowest in 8 of 9 languages (tied with FLOTA-Unigram in Bengali). Downstream, it wins 31 of 37 comparisons against FLOTA and 16 of 22 against MorphPiece.
Arnett et al. Their gold construction matches ours, and their finding agrees with our Table 28. We also ran their MorphScore. On Tamil, MRT-P scores 0.51, against 0.30 for Unigram and 0.09 for BPE. Our claim is about an intervention, which is what the W1 controls test.
W5. Multilingual setting
To address this, we tuned λ ∈ {2, 3, 4, 5, 6.5} with a shared vocabulary. λ = 5 is best (1.9085). MRT-P beats BPE in 8 of 9 languages and also beats MRT-B, but Unigram remains stronger in the joint setting. We will state this as a limitation in the main text and scope the LM claim to monolingual models.
W6. Figure 1
To address this: thank you, we will fix both (11-44 suffixes; λ = 6.5, range [5.5, 7.0]).
Your review has made the paper considerably more careful. We believe the controls now show clearly what MRT contributes, namely morphological alignment at negligible LM cost and consistent gains over BPE. If our response addresses your concerns, we would sincerely appreciate it if you would consider increasing your score.