- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper compares five Myanmar word segmentation systems on 1,100 annotated sentences across Government, News, and Novel domains. It evaluates intrinsic segmentation accuracy, subword expansion under three tokenizers, and similarity between embeddings of predicted and gold segmentations under three sentence encoders. The grammar-informed mmdt-tokenizer achieves the highest Word F1 on News and Novels, while Gemini leads on Government text. Gemma exhibits a substantial Boundary F1 decline on Novels. The paper reports a strong correlation between segmentation accuracy and embedding similarity and argues for the value of linguistic knowledge under domain variation.
Critical strengths
- A practically relevant problem. The paper studies segmentation in an underserved language where word boundaries are difficult to identify.
- A useful comparison across domains. Evaluating five systems on the same sentences provides a consistent view of their behavior across formal, journalistic, and literary text.
- Complementary diagnostics. Word F1, Boundary F1, token counts, and multiple tokenizers and encoders help expose differences in segmentation granularity and model sensitivity.
Critical weaknesses
- The incremental contribution is not sufficiently established. The relationship between segmentation and downstream performance has been studied for many years in Chinese NLP, including task-oriented segmentation for machine translation (Chang et al., 2008). Tokenizer effects on multilingual representations and downstream performance are also well established (Rust et al., 2021), as the paper acknowledges. The potential novelty therefore lies in the Myanmar-specific comparison and cross-domain findings. The paper needs to explain more clearly what these experiments establish beyond the known importance of tokenization and how the findings inform practical segmentation choices.
- The representation metric does not independently establish semantic quality. Sections 3.4 and 4.3.2 measure cosine similarity between embeddings of predicted and gold segmentations of the same sentence. This measures proximity to a reference preprocessing convention, but does not demonstrate better semantic discrimination or downstream performance. For mE5, predicted-segmentation similarities occupy a narrow range of 0.988–0.998, while raw text already reaches 0.990–0.994 (Table 4). The reported correlation therefore does not establish a practically meaningful semantic improvement, and the statement that it confirms segmentation “directly improves” semantic fidelity is too strong.
- The robustness conclusion is broader than the evidence supports. The cross-domain Boundary F1 range is 0.044 for mmdt-tokenizer and 0.049 for Gemini. Their News-to-Novel Word F1 decreases are also relatively close: 0.110 and 0.130. The large 0.168 Boundary F1 decline applies specifically to Gemma. These results support observations about the evaluated implementations more strongly than a general rules-versus-neural conclusion. Confidence intervals and source-level analyses are needed to assess the reliability of the differences.
- The reliability and compatibility of the reference segmentation need clarification. The supplied main paper does not sufficiently describe independent annotation agreement, adjudication, or the conventions governing compounds and grammatical particles. These details matter because the systems produce markedly different segmentation granularities, and the qualitative examples include disagreements about precisely these boundaries. The paper states that annotation guidelines are provided in supplementary materials; those resources may resolve part of this concern.
Questions for authors
- Which Myanmar-specific or cross-domain findings constitute the main new empirical contribution beyond prior work on tokenization? What practical preprocessing decisions do they support?
- How are predicted, gold, and raw texts serialized and normalized before encoding? Are character preservation, unknown-token rates, and truncation checked? What independent evidence supports interpreting cosine-to-gold similarity as semantic quality?
- What observations and sample size are used for the reported correlations? The coefficients can approximately be reproduced from the 15 system–domain averages in Tables 3 and 4. Can the authors clarify the dependence assumptions and provide sentence-level analyses and confidence intervals for the main comparisons?
- Can the authors provide the annotation guidelines, annotator qualifications, independent agreement and adjudication results, corpus sampling procedure, and confirmation that prompt demonstrations are separate from evaluation sentences?