- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MORPHOT is a reference-based translation evaluation metric that leverages frozen multilingual language representations and optimal transport to compare words and their morphological features, such as prefixes and suffixes. It also accounts for missing or unmatched information and checks whether the translation preserves the meaning of the original sentence. Its inner transport component compares candidate affixes within word pairs while its outer transport can align words across sentences and use unmatched mass as an indicator of missing or extra content. A source-relative term compares the discrepancy of the hypothesis from the source with that of the reference. The metric combines eight penalty components using a constrained model trained on human preferences and artificially generated translation errors. MORPHOT demonstrates superior correlation with human judgements across 13 Indic languages and some ablations demonstrate how its individual components can be effective to identify different translation error types.
Critical strengths
-
Detailed evaluation and Transparent reporting.
The paper provides comprehensive details around human-correlation, contrastive, per-language, and per-component analysis, and acknowledges cases where simpler variants outperform the full model. -
Strong gains over established simpler baselines.
The paper reports a pooled Pearson of 0.611 versus 0.558 against the next best metric which is BLEURT-20, with statistical comparisons also being provided by the authors in Appendix G. Number-hallucination level-3 accuracy is 55.9% significantly high in comparison to 5.2% for BLEURT-20 (Table 12).
Critical weaknesses
-
Directly relevant metric comparisons and novelty context are incomplete.
The evaluation omits the COMET-based metrics trained on COMTAIL - the work whose training dataset the authors consume. Indic-COMET which is a metric introduced by Sai b. et al.'s work which the authors cite is also not compared against. Indic-COMET and COMTAIL metrics are the most critical baselines here as these are relevant language-adapted scoring baselines well calibrated for these morphologically rich languages. Further, the authors do not compare against xCOMET which is a comparable baseline as it provides both quality scores and error-span localization. These are critical baselines which are absolutely necessary to be compared against are omitted by the authors which makes it difficult to determine practical benefits of this work in light of the well established literature. -
The incremental benefits of the complete method remains insufficiently demonstrated
Table 1, the lexical-only and balanced variants achieve higher Pearson than the full model: 0.630 and 0.628 versus 0.611. Lower correlation need not invalidate a diagnostic metric; the authors explicitly justify the full model through error attribution (Section 3.1). The reduced-budget ablations do not establish the contribution of the inner affix UOT, where we observe that removing inner affix UOT gives 0.377 versus 0.375 Pearson (Table 16) with MorphACES sensitivity unchanged. On the other hand, we observe that removing the broader morphological transport component lowers Pearson to 0.341. These mixed results I believe leave the incremental benefit of the full method unresolved. Matched full-budget, multi-seed comparisons on both quality correlation and diagnostic tasks are needed to establish the proposed trade-off. -
The diagnostic claims need stronger evidence beyond error classification.
The reported 74.3% macro-F1 and component ablations (Table 3) provide some diagnostic indications of the usefulness of the eight-component vector for predicting error categories. However, a classifier can combine and remap these features; accurate classification does not by itself establish that the named components reliably identify their intended error types. Comparing eight features against one scalar cannot sufficiently isolate the value of the proposed decomposition from the benefit of retaining multiple quality signals. I think this is important because Section 3.1 justifies the full model's lower correlation through superior error attribution. Some useful evidence that could strengthen the claim would include a capacity-matched multi-feature baseline, or providing some class-wise results, and quantitative tests of the components' intended error-type associations. For localization, I think it is unclear and the authors should specify how transport quantities are converted by them into predicted spans. Further, I think the authors should compare their error diagnostics with other baselines like xCOMET [4] and an error-annotating LLM judge following AutoMQM [6] or GEMBA-MQM [7]. These experiments would test whether their proposed morphological decomposition superior type-specific and localized diagnostics over and above these baselines.
Questions for authors
- What are the exact MQM training/test splits, seven-category mapping, class counts, and token-localization algorithm I think these details are a bit underspecified ? I did not understand what does "the same segments" refer to in Appendix C? Can you please clarify this.
Thank you for the detailed review and concrete suggestions. We ran the requested baselines, ablations and diagnostic tests, and hope you will consider raising your score in light of them.
W1. Addressing your concern on COMTAIL COMET, Indic-COMET and xCOMET As the closest control to COMTAIL COMET, we fine-tuned COMET-22 on ComTaIL, once on DA scores and once on our preference pairs (r on ComTaIL, 9,887 segments):
Notation. r: Pearson with ComTaIL human scores. Preference pairs: the better/worse translation pairs MORPHOT is trained on. DA: direct-assessment human scores. ACES omit / add: accuracy (%) on ACES omission / addition pairs.
MORPHOT is well ahead of the 3.5B xCOMET-XL (0.628 vs. 0.530) with about 8,000 times fewer parameters. With the same training signal, MORPHOT beats COMET-22 on correlation (0.628 vs. 0.573) and ACES addition (51.9 vs. 47.2). The DA-fine-tuned models win in-domain (0.777 and 0.809) but fall below MORPHOT on ACES addition (35.6 and 42.6 vs. 51.9). COMTAIL COMET, Indic-COMET and xCOMET-XXL could not be completed within the time limits of the rebuttal period. We will report these numbers in the final version.
W2. Addressing your concern on the incremental benefit of the full method Retrained at the production budget:
Notation. MQM F1: error-type macro-F1 (see W3). MorphACES sens.: mean score drop under morphological perturbation, in score standard deviations.
The full model ties on correlation (0.628 vs. 0.628 and 0.630) and is better on diagnostics. In line with your observation, inner affix UOT has a negligible effect (21.2 vs. 21.1 MQM F1), so we now call it optional. Morphological transport clearly matters (MQM F1 21.1 vs. 16.9, ACES add 51.9 vs. 37.9). Additional seeds and the lexical-gate, role and root-vector ablations could not be completed within the time limits of the rebuttal period. We will report these numbers, with mean ± std, in the final version.
W3. Addressing your concern on whether the components identify their intended error types Notation. MORPHOT Φ: MORPHOT's 8 penalty components used as features. 8 metric scores: BLEU, chrF, chrF++, TER, COMET-22, BERTScore P/R/F. XLM-R PCA: top 8 principal components of XLM-R-large embeddings. O_{r→h}: reference-to-hypothesis omission penalty. R^role / R^morph: role / morphology mismatch penalties. Capacity-matched typing (macro-F1, held-out test): 8 metric scores 23.2, XLM-R PCA 22.8, GEMBA-MQM judge (Qwen2.5-32B) 22.3, MORPHOT Φ 21.1, xCOMET-XL score + span features 19.6. MORPHOT is on par with these and best on grammar/inflection (34.7 F1 vs. 13.3 for xCOMET and 7.9 for the LLM judge). Classifier-free association (AUROC of each raw component): the intended component ranks first for omission (O_{r→h}, 0.704) and mistranslation (R^role, 0.583). R^morph is above chance on grammar errors (0.548) but less specific. Localization (token F1): xCOMET-XL, trained on spans, leads (51.9). Without span supervision, MORPHOT beats the LLM judge on inflection (15.7 vs. 4.3), terminology (15.2 vs. 0.0) and untranslated text (20.6 vs. 9.3). Span algorithm: each component splits exactly into per-token contributions read off the solved transport plans, weighted by the component's learned coefficient. A token is flagged when its penalty in the channel matching the gold class exceeds the sentence mean + 1 std.
Q. Answering your question on splits, mapping, class counts, localization and "the same segments" Data: IndicMT Eval Hindi+Tamil, official splits. We fit on train+val (1,980 labelled segments) and evaluate once on test (460). Label: the class with the largest severity-weighted span mass. Mapping (test counts): Mistranslation (150), Grammar/inflection (137), Terminology (47), Omission (40), Untranslated/transliteration incl. character encoding (40), Style/other = Awkward + Register + Addition + Other (35), Spelling (11). "The same segments": every feature set is fit and evaluated on an identical set of segments, so all comparisons are paired. This stricter protocol gives lower absolute F1 for every system, and these numbers replace the earlier ones.
Thank you again. We hope the added baselines and tests resolve your concerns, and we would be grateful if you would raise your score.