acceptodds
Under review as a conference paper at ICLR 2027

MORPHOT: Evaluating Morphologically Rich Indic Languages with Hierarchical Unbalanced Optimal Transport

Abstract

Embedding-based translation metrics correlate strongly with human judgments on English benchmarks, yet the same metrics fail systematically on morphologically rich languages (MRLs) for three structural reasons. Such metrics stay blind to fine-grained inflection errors, conserve mass between hypothesis and reference and therefore conflate omissions with hallucinations, and reward reference mimicry without verifying fidelity to the source. We introduce MORPHOT, a hierarchical unbalanced optimal transport (UOT) metric that answers each failure mode with one architectural component. MORPHOT represents a sentence as a discrete measure over morphological units. An inner UOT over affix lattices produces a morphology cost sensitive to inflectional differences, an outer UOT decomposes into a matched discrepancy together with separate omission and hallucination residuals, and a source-anchored adequacy term guards against reference-mimicking drift. A monotone softplus-parameterized head then aggregates the components under a formal guarantee that no training run inverts the sign of a penalty. With  M trainable parameters atop a frozen multilingual encoder, MORPHOT attains the highest pooled Pearson, Spearman, and Kendall correlation with human quality ratings on Indic MRLs among all baselines evaluated, improving Pearson by over BERTScore and by over BLEURT-20, which carries more trainable parameters. The component decomposition additionally classifies Multidimensional Quality Metrics (MQM) error categories at macro-F1 against for the strongest scalar baseline.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Evaluation Metric, Morphology, Indic Language, Machine Translation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Mk5YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MORPHOT is a reference-based translation evaluation metric that leverages frozen multilingual language representations and optimal transport to compare words and their morphological features, such as prefixes and suffixes. It also accounts for missing or unmatched information and checks whether the translation preserves the meaning of the original sentence. Its inner transport component compares candidate affixes within word pairs while its outer transport can align words across sentences and use unmatched mass as an indicator of missing or extra content. A source-relative term compares the discrepancy of the hypothesis from the source with that of the reference. The metric combines eight penalty components using a constrained model trained on human preferences and artificially generated translation errors. MORPHOT demonstrates superior correlation with human judgements across 13 Indic languages and some ablations demonstrate how its individual components can be effective to identify different translation error types.

Critical strengths
  • Detailed evaluation and Transparent reporting.
    The paper provides comprehensive details around human-correlation, contrastive, per-language, and per-component analysis, and acknowledges cases where simpler variants outperform the full model.

  • Strong gains over established simpler baselines.
    The paper reports a pooled Pearson of 0.611 versus 0.558 against the next best metric which is BLEURT-20, with statistical comparisons also being provided by the authors in Appendix G. Number-hallucination level-3 accuracy is 55.9% significantly high in comparison to 5.2% for BLEURT-20 (Table 12).

Critical weaknesses
  • Directly relevant metric comparisons and novelty context are incomplete.
    The evaluation omits the COMET-based metrics trained on COMTAIL - the work whose training dataset the authors consume. Indic-COMET which is a metric introduced by Sai b. et al.'s work which the authors cite is also not compared against. Indic-COMET and COMTAIL metrics are the most critical baselines here as these are relevant language-adapted scoring baselines well calibrated for these morphologically rich languages. Further, the authors do not compare against xCOMET which is a comparable baseline as it provides both quality scores and error-span localization. These are critical baselines which are absolutely necessary to be compared against are omitted by the authors which makes it difficult to determine practical benefits of this work in light of the well established literature.

  • The incremental benefits of the complete method remains insufficiently demonstrated
    Table 1, the lexical-only and balanced variants achieve higher Pearson than the full model: 0.630 and 0.628 versus 0.611. Lower correlation need not invalidate a diagnostic metric; the authors explicitly justify the full model through error attribution (Section 3.1). The reduced-budget ablations do not establish the contribution of the inner affix UOT, where we observe that removing inner affix UOT gives 0.377 versus 0.375 Pearson (Table 16) with MorphACES sensitivity unchanged. On the other hand, we observe that removing the broader morphological transport component lowers Pearson to 0.341. These mixed results I believe leave the incremental benefit of the full method unresolved. Matched full-budget, multi-seed comparisons on both quality correlation and diagnostic tasks are needed to establish the proposed trade-off.

  • The diagnostic claims need stronger evidence beyond error classification.
    The reported 74.3% macro-F1 and component ablations (Table 3) provide some diagnostic indications of the usefulness of the eight-component vector for predicting error categories. However, a classifier can combine and remap these features; accurate classification does not by itself establish that the named components reliably identify their intended error types. Comparing eight features against one scalar cannot sufficiently isolate the value of the proposed decomposition from the benefit of retaining multiple quality signals. I think this is important because Section 3.1 justifies the full model's lower correlation through superior error attribution. Some useful evidence that could strengthen the claim would include a capacity-matched multi-feature baseline, or providing some class-wise results, and quantitative tests of the components' intended error-type associations. For localization, I think it is unclear and the authors should specify how transport quantities are converted by them into predicted spans. Further, I think the authors should compare their error diagnostics with other baselines like xCOMET [4] and an error-annotating LLM judge following AutoMQM [6] or GEMBA-MQM [7]. These experiments would test whether their proposed morphological decomposition superior type-specific and localized diagnostics over and above these baselines.

Questions for authors
  • What are the exact MQM training/test splits, seven-category mapping, class counts, and token-localization algorithm I think these details are a bit underspecified ? I did not understand what does "the same segments" refer to in Appendix C? Can you please clarify this.
1 comment

Thank you for the detailed review and concrete suggestions. We ran the requested baselines, ablations and diagnostic tests, and hope you will consider raising your score in light of them.

W1. Addressing your concern on COMTAIL COMET, Indic-COMET and xCOMET As the closest control to COMTAIL COMET, we fine-tuned COMET-22 on ComTaIL, once on DA scores and once on our preference pairs (r on ComTaIL, 9,887 segments):

MetricParamsTrained on ComTaILrACES omit / add
xCOMET-XL3.5Bno0.53071.5 / 62.9
COMET-22 on our preference pairs580Myes0.57387.7 / 47.2
COMET-22 fine-tuned on ComTaIL DA580Myes0.77779.9 / 35.6
BLEURT-20 fine-tuned on ComTaIL DA579Myes0.80992.2 / 42.6
MORPHOT0.44Myes0.62881.1 / 51.9

Notation. r: Pearson with ComTaIL human scores. Preference pairs: the better/worse translation pairs MORPHOT is trained on. DA: direct-assessment human scores. ACES omit / add: accuracy (%) on ACES omission / addition pairs.

MORPHOT is well ahead of the 3.5B xCOMET-XL (0.628 vs. 0.530) with about 8,000 times fewer parameters. With the same training signal, MORPHOT beats COMET-22 on correlation (0.628 vs. 0.573) and ACES addition (51.9 vs. 47.2). The DA-fine-tuned models win in-domain (0.777 and 0.809) but fall below MORPHOT on ACES addition (35.6 and 42.6 vs. 51.9). COMTAIL COMET, Indic-COMET and xCOMET-XXL could not be completed within the time limits of the rebuttal period. We will report these numbers in the final version.

W2. Addressing your concern on the incremental benefit of the full method Retrained at the production budget:

VariantrMQM F1MorphACES sens.ACES add
Full0.62821.11.3451.9
− inner affix UOT0.62821.21.3451.8
− unbalanced OT0.62818.21.4750.8
− morph. transport0.63016.91.1137.9

Notation. MQM F1: error-type macro-F1 (see W3). MorphACES sens.: mean score drop under morphological perturbation, in score standard deviations.

The full model ties on correlation (0.628 vs. 0.628 and 0.630) and is better on diagnostics. In line with your observation, inner affix UOT has a negligible effect (21.2 vs. 21.1 MQM F1), so we now call it optional. Morphological transport clearly matters (MQM F1 21.1 vs. 16.9, ACES add 51.9 vs. 37.9). Additional seeds and the lexical-gate, role and root-vector ablations could not be completed within the time limits of the rebuttal period. We will report these numbers, with mean ± std, in the final version.

W3. Addressing your concern on whether the components identify their intended error types Notation. MORPHOT Φ: MORPHOT's 8 penalty components used as features. 8 metric scores: BLEU, chrF, chrF++, TER, COMET-22, BERTScore P/R/F. XLM-R PCA: top 8 principal components of XLM-R-large embeddings. O_{r→h}: reference-to-hypothesis omission penalty. R^role / R^morph: role / morphology mismatch penalties. Capacity-matched typing (macro-F1, held-out test): 8 metric scores 23.2, XLM-R PCA 22.8, GEMBA-MQM judge (Qwen2.5-32B) 22.3, MORPHOT Φ 21.1, xCOMET-XL score + span features 19.6. MORPHOT is on par with these and best on grammar/inflection (34.7 F1 vs. 13.3 for xCOMET and 7.9 for the LLM judge). Classifier-free association (AUROC of each raw component): the intended component ranks first for omission (O_{r→h}, 0.704) and mistranslation (R^role, 0.583). R^morph is above chance on grammar errors (0.548) but less specific. Localization (token F1): xCOMET-XL, trained on spans, leads (51.9). Without span supervision, MORPHOT beats the LLM judge on inflection (15.7 vs. 4.3), terminology (15.2 vs. 0.0) and untranslated text (20.6 vs. 9.3). Span algorithm: each component splits exactly into per-token contributions read off the solved transport plans, weighted by the component's learned coefficient. A token is flagged when its penalty in the channel matching the gold class exceeds the sentence mean + 1 std.

Q. Answering your question on splits, mapping, class counts, localization and "the same segments" Data: IndicMT Eval Hindi+Tamil, official splits. We fit on train+val (1,980 labelled segments) and evaluate once on test (460). Label: the class with the largest severity-weighted span mass. Mapping (test counts): Mistranslation (150), Grammar/inflection (137), Terminology (47), Omission (40), Untranslated/transliteration incl. character encoding (40), Style/other = Awkward + Register + Addition + Other (35), Spelling (11). "The same segments": every feature set is fit and evaluated on an identical set of segments, so all comparisons are paired. This stricter protocol gives lower absolute F1 for every system, and these numbers replace the earlier ones.

Thank you again. We hope the added baselines and tests resolve your concerns, and we would be grateful if you would raise your score.

Reviewer_CbNbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MORPHOT, a machine translation evaluation metric for morphologically rich Indic languages. It combines a frozen XLM-R encoder with morphological features and latent root and role representations, comparing hypotheses and references through inner affix-level and outer sentence-level unbalanced optimal transport (UOT). The score decomposes into matched discrepancy, omission and hallucination residuals, morphology and role penalties, and source-anchored adequacy. Softplus-constrained weights guarantee score monotonicity with respect to each defined penalty. On 9,887 ComTaIL test segments across 13 Indic languages, MORPHOT achieves a pooled Pearson correlation of 0.611, exceeding BERTScore’s 0.519 and BLEURT-20’s 0.558. The authors also evaluate phenomenon-specific robustness on 36,476 ACES examples, while an MQM error classifier using MORPHOT’s eight components achieves 74.3% macro-F1. However, the lexical-only and balanced-OT variants obtain higher correlations than the full model, and BLEURT-20 performs better on overall ACES accuracy.

Critical strengths

1. A design that decomposes translation quality into diagnostic components.

  • The architecture explicitly attempts to distinguish morphological differences, semantic discrepancies, omissions, and hallucinations rather than relying on a single embedding similarity. Exposing transport plans and residuals enables analysis of why a translation receives a low score. Sign-constrained weights also prevent the score from increasing when a defined penalty increases. This guarantee should, however, be distinguished from a guarantee that actual translation errors always increase the corresponding penalties.

2. Multifaceted evaluation on under-resourced languages.

  • The paper combines human judgments across 13 Indic languages with an independent ACES challenge set and reports per-language results, phenomenon-specific performance, ablations, statistical significance, and inference costs. These analyses help identify where the metric succeeds and fails beyond its pooled correlation.
Critical weaknesses

1. The additional value of the core morphological and UOT components is insufficiently established.

  • The full model’s Pearson correlation of 0.611 is lower than MORPHOT-lex’s 0.630 and MORPHOT-bal’s 0.628. Reduced-budget ablations also show that removing inner affix UOT, role realization, or unbalanced OT preserves or improves correlation. Although the authors justify the full model through error attribution, the current evidence does not clearly establish the necessity of the complex hierarchical design. Moreover, the morphology and role representations are latent clusters without validation against gold linguistic categories, so their linguistic interpretation requires further evidence. Ablations under the same sufficiently large training budget and validation against gold morphological annotations are needed.

2. The comparisons provide limited support for claims of superior performance and interpretability.

  • The evaluation omits COMET-family metrics, xCOMET, and MetricX, despite discussing them in related work, making it difficult to assess the additional value over existing source-aware or error-detection metrics. In particular, MQM classification gives MORPHOT eight features but each baseline only a scalar score. This demonstrates the usefulness of multidimensional outputs, but provides limited evidence that MORPHOT’s specific decomposition is more accurate or interpretable. Comparisons with competing metrics trained on the same data and with multidimensional feature baselines are needed.
Questions for authors
  1. How does MORPHOT compare with strong source-aware and error-detection baselines?
  • Could you include COMET-family metrics, xCOMET, and MetricX under comparable training and evaluation conditions? For MQM attribution, comparisons with multidimensional or span-predicting baselines would help establish the value of MORPHOT’s decomposition beyond the eight-feature versus single-scalar comparison.
  1. Does the full hierarchical design provide consistent benefits at the production training budget?
  • Since MORPHOT-lex and MORPHOT-bal outperform the full model on correlation, could you provide full-budget ablations evaluating both correlation and error attribution to clarify the trade-off?

Potential impact on my assessment: Including these key baselines and demonstrating competitive correlation together with stronger error attribution would materially strengthen the paper. I would be open to raising my overall recommendation if these comparisons substantiate the proposed method’s advantages.

1 comment

Thank you for the constructive review and for telling us what would change your assessment. In short, at a matched training budget the full model is competitive on correlation and gives better error attribution than its simpler variants. We hope this lets you raise your score.

W1. Addressing your concern on the added value of the morphological and UOT components We retrained the ablation variants with the production protocol:

VariantrρMQM F1ACES omit / addMorphACES sens.O/H sep.
Full MORPHOT0.6280.55021.181.1 / 51.91.340.29
− inner affix UOT0.6280.55021.281.1 / 51.81.340.29
− unbalanced OT (bal)0.6280.57218.278.7 / 50.81.470.30
− morph. transport (lex)0.6300.54916.981.7 / 37.91.110.15

Notation. r / ρ: Pearson / Spearman with ComTaIL human scores. MQM F1: error-type macro-F1 (see W2). ACES omit / add: accuracy (%) on ACES omission / addition pairs. MorphACES sens.: mean score drop under morphological perturbation, in score standard deviations (σ). O/H sep.: specificity of the omission vs. hallucination penalties on ACES omissions vs. additions (σ). bal: balanced OT. lex: lexical transport only.

  • The full model ties with bal and lex on Pearson (0.628 vs. 0.628 and 0.630). The lower value in Table 1 came from an earlier training run.
  • Over lex, morphological transport improves MQM F1 (21.1 vs. 16.9), ACES addition (51.9 vs. 37.9), MorphACES sens. (1.34 vs. 1.11) and O/H sep. (0.29 vs. 0.15).
  • Over bal, unbalanced OT improves MQM F1 (21.1 vs. 18.2) and ACES omission (81.1 vs. 78.7), while bal keeps a higher ρ (0.572 vs. 0.550).
  • Inner affix UOT has no measurable effect, so we will describe it as optional.
  • The lexical-gate, role and root-vector ablations and additional seeds could not be completed within the time limits of the rebuttal period. We will report these numbers in the final version.

We probed MORPHOT's morphosyntactic posterior z and role posterior ψ against gold Universal Dependencies features in 6 languages. z aligns weakly with gold values (V-measure, a 0–1 cluster-agreement score, ≤ 0.06), so the preference objective shapes z around quality rather than grammar. We will call z and ψ learned latent clusters and drop the case/number/role labels. The diagnostics above do not rely on them.

W2. Addressing your concern on the missing baselines and multi-feature comparisons We added baselines, including models trained on MORPHOT's data (r on ComTaIL, 9,887 segments):

MetricParamsTrained on ComTaILrACES omit / add
xCOMET-XL3.5Bno0.53071.5 / 62.9
COMET-22 on our preference pairs580Myes0.57387.7 / 47.2
COMET-22 fine-tuned on ComTaIL DA580Myes0.77779.9 / 35.6
BLEURT-20 fine-tuned on ComTaIL DA579Myes0.80992.2 / 42.6
MORPHOT0.44Myes0.62881.1 / 51.9

Notation. Preference pairs: the better/worse translation pairs MORPHOT is trained on. DA: direct-assessment human scores.

MORPHOT is well ahead of the 3.5B xCOMET-XL (0.628 vs. 0.530) with about 8,000 times fewer parameters. xCOMET-XXL could not be completed within the time limits of the rebuttal period. We will report these numbers in the final version.

MQM typing (macro-F1 over 7 error classes, held-out test split of 460 segments): 8 metric scores 23.2, 8-dim XLM-R PCA 22.8, GEMBA-MQM LLM judge 22.3, MORPHOT Φ 21.1, xCOMET-XL score + span features 19.6. (MORPHOT Φ: its 8 penalty components as features. 8 metric scores: BLEU, chrF, chrF++, TER, COMET-22, BERTScore P/R/F. PCA: top 8 principal components of XLM-R-large embeddings.) MORPHOT is on par with these and best on grammar/inflection errors (34.7 F1 vs. 13.3 for xCOMET).

Span localization (token-level F1 against human MQM spans): xCOMET-XL, trained with span supervision, is best overall (51.9). MORPHOT has no span supervision, yet beats the LLM judge on inflection (15.7 vs. 4.3), terminology (15.2 vs. 0.0), untranslated text (20.6 vs. 9.3) and mistranslation (23.8 vs. 21.3).

Q1. Answering your question: How does MORPHOT compare under comparable training conditions? With the same training signal (our preference pairs), MORPHOT is higher than COMET-22 on correlation (0.628 vs. 0.573) and ACES addition (51.9 vs. 47.2). COMET-22 fine-tuned on ComTaIL DA reaches 0.777 in-domain, but MORPHOT is more robust out of domain, with higher ACES omission (81.1 vs. 79.9) and addition (51.9 vs. 35.6).

Q2. Answering your question: Does the full design pay off at the production budget? Yes, for attribution, at no cost in Pearson (see W1).

Thank you again. Since the new results provide the baselines, the competitive correlation and the stronger attribution you asked for, we would be very grateful if you would consider raising your score.

Reviewer_XvM6OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MORPHOT, a learned machine translation (MT) evaluation metric designed for morphologically rich Indic languages ​​(Indic MRLs), based on the framework of Hierarchical Unbalanced Optimal Transport (UOT). Sentences are represented as a collection of morphological units—extracted from a frozen multilingual encoder (XLM-R-large) and augmented with a character/grapheme branch and an affix lattice distribution. The architecture addresses three key limitations of existing metrics: (1) an "inner" UOT problem operating on the affix lattice to mitigate "morphology blindness"; (2) three "outer" UOT problems, employing KL-divergence-based relaxed marginal constraints to disentangle omission residuals from hallucination residuals; and (3) a source-anchored adequacy component designed to counteract "reference mimicry." Finally, a linear predictor—with weights parameterized via a softplus function to ensure monotonicity (Proposition 1), that aggregates eight penalty components.

Critical strengths

Parameter efficiency and high inference speed: By freezing the XLM-R-large encoder and training only the linear heads and the aggregation layer (~0.44 million parameters), combining with top-kk (k=16k=16) transport plan pruning—the metric achieves a processing speed of approximately 68 segments per second on an A100 GPU, while performing on par with BERTScore.

Interpretability via an 8-component penalty vector: Decomposing the score into 8 channels with clear geometric significance allows users to diagnose the root cause of errors (such as omissions or additions, as seen in the ACES dataset), rather than relying on a single scalar value.

Critical weaknesses

The ablation study (Tables 3 & 16) was conducted under severely degraded conditions (a Pearson correlation of 0.375 versus 0.611). The authors briefly explain that the ablation grid was trained with a reduced budget. This resulted in the "full model" in Table 3 achieving a Pearson correlation of only 0.375 (i.e., a figure 0.236 points lower than the 0.611 reported in Table 1). Given that the model contains only 0.44 million trainable parameters and requires less than two hours of training time on a single A100 GPU (as stated in Appendix D, line 1110), conducting an ablation study in a non-converged state is entirely unacceptable.

The study lacks standard baselines (such as the COMET family) and suffers from an asymmetry in training data: MORPHOT is trained directly on 199,737 samples from the ComTaIL training set.

The design of the MQM error classification task (Table 3, right) is flawed due to an architectural imbalance. From the perspective of linear algebra, it is entirely expected that an 8-dimensional vector would outperform a single scalar value in classifying 7 distinct labels (yielding an F1 score of 74.3% versus 26.1%).

Questions for authors

In Table 1, why do both MORPHOT-lex (0.630 Pearson) and MORPHOT-bal (0.628 Pearson, 0.572 Spearman, 0.402 Kendall) outperform the full MORPHOT model (0.611 Pearson, 0.556 Spearman, 0.392 Kendall)?

Why do the ablation grids in Tables 3 and 16 require up to 90 GPU hours while achieving only a 0.375 Pearson correlation for the Full Model (lower than the 0.411 achieved by chrF)?

What are the results for COMET-22, XCOMET, and the COMET model fine-tuned on the ComTaIL training set, or for BLEURT-20 when fine-tuned on the same set of preference pairs Dpref\mathcal{D}_{\text{pref}} from ComTaIL?

2 comments

Thank you for the careful review and for pointing out MORPHOT's efficiency and interpretability. We ran the experiments you asked for and believe they address your main concerns. We would be grateful if you would consider raising your score.

W1. Addressing your concern on the ablation grid (0.375 vs 0.611) We agree, and retrained the variants with the exact production protocol:

VariantrρMQM F1MorphACES sens. (σ)ACES addO/H sep. (σ)
Full MORPHOT0.6280.55021.11.3451.90.29
− inner affix UOT0.6280.55021.21.3451.80.29
− unbalanced OT (bal)0.6280.57218.21.4750.80.30
− morph. transport (lex)0.6300.54916.91.1137.90.15

Notation. r / ρ: Pearson / Spearman with ComTaIL human scores. MQM F1: error-type macro-F1 (see W3). MorphACES sens.: mean score drop under morphological perturbation, in score standard deviations (σ). ACES add: accuracy (%) on ACES addition pairs. O/H sep.: specificity of the omission vs. hallucination penalties on ACES omissions vs. additions (σ). bal: balanced OT. lex: lexical transport only.

At a matched budget, the full model correlates as well as the simpler variants (r 0.628 vs. 0.628 and 0.630) and gives clearly better diagnostics: MQM F1 21.1 vs. 16.9 and ACES addition 51.9 vs. 37.9 over lex, and MQM F1 21.1 vs. 18.2 over bal. Inner affix UOT has no measurable effect, so we now present it as optional. The remaining ablations (lexical gate, role realization, root vectors) and additional seeds could not be completed within the time limits of the rebuttal period. We will report these numbers in the final version.

W2. Addressing your concern on COMET baselines and the training-data asymmetry We added the controls you suggested (r on ComTaIL, 9,887 segments):

MetricParamsTrained on ComTaILrACES omit / add
xCOMET-XL3.5Bno0.53071.5 / 62.9
COMET-22 on our preference pairs580Myes0.57387.7 / 47.2
COMET-22 fine-tuned on ComTaIL DA580Myes0.77779.9 / 35.6
BLEURT-20 fine-tuned on ComTaIL DA579Myes0.80992.2 / 42.6
MORPHOT0.44Myes0.62881.1 / 51.9

Notation. Preference pairs: the better/worse translation pairs MORPHOT is trained on. DA: direct-assessment human scores. ACES omit / add: accuracy (%) on ACES omission / addition pairs.

  • Same training signal. Against COMET-22 trained on our preference pairs, MORPHOT is higher on correlation (0.628 vs. 0.573) and ACES addition (51.9 vs. 47.2).
  • Off-the-shelf metric. MORPHOT is well ahead of the 3.5B xCOMET-XL (0.628 vs. 0.530) with about 8,000 times fewer parameters.
  • Full DA fine-tuning. It wins in-domain (0.777 and 0.809) but hurts robustness: ACES addition drops to 35.6 and 42.6, below MORPHOT's 51.9, and MORPHOT also beats fine-tuned COMET-22 on ACES omission (81.1 vs. 79.9).

W3. Addressing your concern on 8 features vs. 1 scalar for MQM We redid MQM typing on the official held-out test split (460 segments, 7 classes) with capacity-matched baselines:

Features#featmacro-F1
8 metric scores823.2
8-dim PCA of XLM-R822.8
GEMBA-MQM LLM judge (Qwen2.5-32B)–22.3
MORPHOT Φ821.1
xCOMET-XL score + span features619.6
best single scalar (xCOMET-XL)113.4

Notation. MORPHOT Φ: MORPHOT's 8 penalty components used as features. 8 metric scores: BLEU, chrF, chrF++, TER, COMET-22 and BERTScore P/R/F. 8-dim PCA of XLM-R: top 8 principal components of XLM-R-large embeddings. Span features: 5 statistics of xCOMET's predicted error spans.

MORPHOT Φ is on par with the 8-dim baselines and a 32B LLM judge, and better than xCOMET's span features. They are the strongest feature set on grammar/inflection errors (34.7 F1 vs. 13.3 for xCOMET and 7.9 for the LLM judge).

Q1. Answering your question: Why do lex and bal outperform the full model? As W1 shows, the gap came from the training run behind Table 1, not the design. Retrained, the full model ties with bal and lex (0.628) and keeps its diagnostic advantage.

Q2. Answering your question: Why did the grid need 90 GPU-hours? Nearly all of it went into scoring 17 variants on ComTaIL and MorphACES, not training. We will correct Appendix D: a production run takes one to three GPU-days, not "under two hours".

Q3. Answering your question: Results for COMET-22, xCOMET and fine-tuned COMET/BLEURT See W2. BLEURT-20 on our preference pairs could not be completed within the time limits of the rebuttal period, so we report its standard DA fine-tuning. We will report these numbers in the final version.

At a matched budget the full model is competitive on correlation and adds diagnostic value that the simpler variants lack. If you agree, we would sincerely appreciate it if you would raise your score.

Reviewer_XvM6OpenReview ↗

Response to the rebuttal

Thank you for the detailed response and the additional experiments. They answer my questions, but they also weaken the main claims. With the corrected ablation, the full model is no better than the simpler variants (r = 0.628 vs. 0.628 and 0.630), and the inner affix UOT, presented as a core contribution, has no measurable effect. Standard metrics fine-tuned on ComTaIL clearly outperform MORPHOT on correlation (0.777 and 0.809 vs. 0.628). For MQM error typing, MORPHOT's features are now below the capacity-matched baselines (21.1 vs. 23.2 and 22.8), and the drop from the 74.3% reported in the paper is not explained. The efficiency statement in Appendix D was also incorrect (one to three GPU-days rather than under two hours), and several ablations and seeds are still missing. The paper's claims would need substantial revision, so I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.