acceptodds
Under review as a conference paper at ICLR 2027

BeyondAdeEval: Machine Translation Evaluation Beyond Semantic Adequacy

17 views

Abstract

Machine translation metrics and LLM-based evaluators have made substantial progress in translation evaluation. However, existing methods such as COMET primarily focus on semantic adequacy and offer limited interpretability. Evaluating overall translation quality beyond semantic adequacy remains challenging. We present BeyondAdeEval (BAE), a reference-free and interpretable method for evaluating translation quality from semantic adequacy to finer-grained aspects of translation quality. We design , a principled, human-aligned hierarchical rubric that formalizes translation quality, guides data construction, and enables our models to learn structured, rubric-grounded rationales. Building on this rubric, we introduce (i) , a preference-graded translation benchmark constructed through controlled modification of high-quality translations, with 6,960 training samples and a human-annotated test set supporting pointwise scoring and pairwise ranking; and (ii) , a family of evaluation models ranging from 4B to 9B parameters that produce human-aligned, multidimensional, and explainable translation evaluations. On BAE-BENCH, our models outperform all evaluated baselines, achieving relative improvements of 28% and 24% in Spearman and Pearson correlation, respectively, over the strongest learned metrics. We also achieve state-of-the-art performance on T-index (Human) and demonstrate zero-shot cross-lingual generalization on MQM24. We will release our code and data upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
Machine Translation, Translation Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LmWJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BeyondAdeEval (BAE), a reference-free translation evaluation framework targeting quality distinctions beyond semantic adequacy. Its training data and primary benchmark focus on English-to-Chinese literary translation. The authors propose a seven-dimensional rubric and construct five quality tiers through controlled degradation of professionally translated anchors, then train 4B and 9B evaluators using supervised fine-tuning and GRPO. The primary evaluation uses a source-disjoint test set with human scoring blinded to the construction tiers. Additional evaluations on T-index (Human) and MQM24 English-to-German assess transfer to naturally occurring translations and an additional language pair, respectively. The results support the usefulness of rubric-guided supervision relative to the same backbone with rubric prompting alone. My main reservation concerns whether the synthetic construction reliably isolates the intended quality dimensions, particularly whether Tier-4 candidates preserve adequacy while differing from Tier 5 primarily in higher-order quality.

Critical strengths
  1. A relevant evaluation problem with evidence that targeted supervision helps. Distinguishing translations that are broadly correct but differ in naturalness, style, and communicative effectiveness is important. The same-backbone comparison provides useful evidence: supervised training improves Qwen3.5-9B over rubric prompting, including an increase in Tier-4-versus-Tier-5 accuracy from 49.5% to 61.4%.

  2. Human evaluation and external benchmarks provide meaningful safeguards against a purely construction-dependent result. The test annotator and reviewer are blinded to construction tiers and intended ordering, and the final human scores are used as evaluation labels. The external T-index evaluation additionally tests naturally occurring system outputs. These choices strengthen the empirical contribution beyond agreement with synthetic construction labels alone.

Critical weaknesses
  1. The reliability and dimensional specificity of the synthetic quality control are insufficiently quantified. Section 2.3 describes human inspection of a subset of constructed groups, but does not report the sample size, sampling procedure, agreement with intended tiers, or correction/rejection rates. This is particularly consequential for Tier 4: the generation prompt requires preservation of semantic accuracy, language quality, and functional equivalence while degrading only higher-order qualities. Without an independent, aspect-specific audit, it remains unclear how consistently this constraint is satisfied. If Tier-4 candidates also contain adequacy errors, correctly ranking them below Tier 5 would not uniquely demonstrate beyond-adequacy discrimination. The blind overall test scores are valuable, but the absence of per-aspect judgments leaves this central interpretation under-validated. A related concern is whether prompting alone provides sufficiently reliable control over the intended quality tiers. The construction procedure assumes that the model can realize fine-grained quality distinctions through rubric-guided editing. However, Table 3 shows substantial disagreement between prompted LLM judges and human grades, even with BAE-RUBRIC (50% acc.). These capabilities need not be equivalent. However, the effectiveness of generating training data through prompting needs reconsideration.

  2. The relationship between construction labels, human verification, and teacher judgments needs clarification. The rationale-generation prompt describes the quality bands as already established through human annotation, whereas the training-data description documents constructed tiers and subset inspection. It is therefore unclear whether all training labels were independently assigned or corrected by humans. Moreover, label-conditioned rationales explain a supplied tier and cannot independently establish that tier’s correctness.

Questions for authors
  1. Validity of the high-quality diagnostic: What evidence establishes that Tier-4 and Tier-5 candidates preserve comparable semantic adequacy and differ primarily along the intended higher-order dimensions? An independently annotated audit separating adequacy from stylistic quality would materially strengthen this claim. For the reported pairwise diagnostic, is the preferred candidate determined by construction order or final human scores, and how are human ties or reversals handled?

  2. Human-label reliability: Can the authors report the relationship between construction tiers and blind human grades across the test set, including a confusion matrix and adjacent-tier ties/reversals? Were any judgments collected independently from multiple annotators before review? Please report available pre-adjudication agreement and the frequency of review changes or adjudication; final consensus alone does not establish independent reproducibility.

  3. Training-label provenance and verification: Were the 6,960 training labels retained from the requested construction tiers, independently assigned by humans, or revised through a combination of these procedures? Please clarify the statement in the rationale prompt that the bands were established by human annotation. How many training groups were inspected, how were they sampled, and how frequently were candidates relabeled, corrected, regenerated, or discarded?

6 comments
AuthorsOpenReview ↗

Response to reviewer LmWJ [1/4]

We sincerely appreciate your insightful feedback and constructive suggestions. We address your questions and concerns below:

W1.1: The reliability and dimensional specificity of the synthetic quality control are insufficiently quantified. Section 2.3 describes human inspection of a subset of constructed groups, but does not report the sample size, sampling procedure, agreement with intended tiers, or correction/rejection rates.

Thank you for pointing this out. We report the requested statistics below:

ItemValue
Sample size501 of 1,412 groups (2,505 candidates, 35.5%)
Sampling procedureRandom sampling
Correction rate74 / 2,505 (2.95%)
Rejection rate20 / 501 (3.99%)
Agreement with intended tiers2,331 / 2,505 (93.05%)

After removing the 20 rejected groups, the final training set contains 1,392 groups, as reported in the paper. Regarding dimensional specificity, we conducted a per-dimension human audit of all test candidates; please see our response to W1.2.

W1.2: This is particularly consequential for Tier 4: the generation prompt requires preservation of semantic accuracy, language quality, and functional equivalence while degrading only higher-order qualities. Without an independent, aspect-specific audit, it remains unclear how consistently this constraint is satisfied. If Tier-4 candidates also contain adequacy errors, correctly ranking them below Tier 5 would not uniquely demonstrate beyond-adequacy discrimination. The blind overall test scores are valuable, but the absence of per-aspect judgments leaves this central interpretation under-validated.

To directly examine whether Tier-4 and Tier-5 candidates differ primarily in dimensions beyond adequacy, we conducted an additional aspect-specific human audit of all test candidates across the seven BAE-RUBRIC dimensions, blinded to their construction tiers and intended ordering. The Tier-4/Tier-5 results are summarized below.

DimensionTier 4Tier 5Δ (T5−T4)
S1 Accuracy4.794.84+0.05
S2 Language Quality4.544.74+0.20
S3 Functional Equivalence4.734.82+0.09
Core Average (S1–S3)4.694.80+0.11
SA1 Native / Natural Expression3.814.46+0.65
SA2 Stylistic Refinement3.934.52+0.59
SA3 Register / Genre Fit4.424.67+0.25
SA4 Tone / Formality / Attitude4.334.66+0.33
Beyond-Adequacy Average (SA1–SA4)4.124.58+0.46

The audit shows that Tier-4/Tier-5 differences are concentrated in beyond-adequacy dimensions, with an average gap of 0.46 points across SA1–SA4, compared with only 0.11 across S1–S3. At a threshold of 4.5 on the respective dimension-group averages, Tier 4 and Tier 5 exhibit similar pass rates in core quality (78.2% vs. 82.2%), but a substantially larger gap in beyond-adequacy quality (31.7% vs. 81.2%). These findings support that Tier 4 largely preserves core translation quality while differing from Tier 5 primarily in dimensions beyond adequacy.

AuthorsOpenReview ↗

Response to reviewer LmWJ [2/4]

W1.3: A related concern is whether prompting alone provides sufficiently reliable control over the intended quality tiers. The construction procedure assumes that the model can realize fine-grained quality distinctions through rubric-guided editing. However, Table 3 shows substantial disagreement between prompted LLM judges and human grades, even with BAE-RUBRIC (50% acc.). These capabilities need not be equivalent. However, the effectiveness of generating training data through prompting needs reconsideration.

Thank you for raising this concern. We would like to clarify that rubric-guided data construction and reference-free quality evaluation are distinct tasks. Unlike evaluation, which requires assigning an absolute quality score to a translation without a reference, our construction procedure starts from a verified Tier-5 human translation and applies targeted, rubric-guided modifications under explicit quality constraints (Section 2.3.1). Thus, the exact-score accuracy of prompted LLM judges in Table 3 does not directly measure the reliability of our controlled construction process. To empirically validate the construction quality, we compared the intended construction tiers with assigned human grades on all 505 test candidates. The test candidates were produced by exactly the same construction pipeline as the training data. The agreement below therefore serves as a direct estimate of how reliably the pipeline realizes the intended tiers in the training data. Annotators were blinded to the construction tiers and intended ordering. The results are summarized below:

Construction Tier \ Human Grade12345
1919001
2980912
318651215
402147015
502131868

The intended construction tiers demonstrate substantial agreement with independent human judgments (Krippendorff's ordinal α = 0.864), providing empirical support for the reliability of rubric-guided quality construction.

W2.1: The relationship between construction labels, human verification, and teacher judgments needs clarification. The rationale-generation prompt describes the quality bands as already established through human annotation, whereas the training-data description documents constructed tiers and subset inspection. It is therefore unclear whether all training labels were independently assigned or corrected by humans.

Thank you for pointing out this ambiguity. We apologize for the misleading wording in the rationale-generation prompt and would like to clarify the actual procedure. The training quality tiers were established through rubric-guided construction, rather than independently assigned by human annotators. We first curated and verified high-quality Tier-5 translations, then generated Tier-1–4 variants through controlled degradation under BAE-RUBRIC. Human annotators inspected a subset of the constructed groups to assess the intended quality distinctions, rather than independently labeling every training example. The wording in the rationale-generation prompt originated from an earlier prompt-development stage, where we used human-annotated examples to calibrate the teacher's explanations. Because the teacher is conditioned on the supplied tier regardless of this phrasing, the statement does not change which labels the rationales explain. We would like to clarify that this wording does not describe how the full training dataset was labeled. We further discuss the role of label-conditioned rationales and the validity of the constructed tiers in W2.2 below.

W2.2: Moreover, label-conditioned rationales explain a supplied tier and cannot independently establish that tier’s correctness.

Thank you for this important point. We agree that label-conditioned rationales cannot independently establish the correctness of the supplied tier, and we do not use them for that purpose. Their role is to serve as distillation targets during SFT, teaching the student model to express its judgments in terms of the BAE-RUBRIC dimensions. As described in Section 2.3.1, we condition the teacher on the assigned tier so that the rationale explains the supervised tier rather than independently re-estimating the score. Label reliability is assessed separately through independent evidence: the agreement between construction tiers and blind human grades (see our response to Q2.1) and the aspect-specific human audit (see our response to W1.2).

AuthorsOpenReview ↗

Response to reviewer LmWJ [3/4]

Q1.1: Validity of the high-quality diagnostic: What evidence establishes that Tier-4 and Tier-5 candidates preserve comparable semantic adequacy and differ primarily along the intended higher-order dimensions? An independently annotated audit separating adequacy from stylistic quality would materially strengthen this claim.

Please see our response to W1.2.

Q1.2: For the reported pairwise diagnostic, is the preferred candidate determined by construction order or final human scores, and how are human ties or reversals handled?

Thank you for pointing this out. We acknowledge that this was not sufficiently clarified in the original manuscript. We would like to clarify that the high-quality pairwise diagnostic directly compares candidates assigned final human grades 4 and 5 within each source group. Both pair selection and ground-truth preferences are determined by the final human-annotated scores, rather than the construction order.

During annotation, annotators could view all five candidates within each source group but were blinded to their construction tiers and intended ordering. Under our group-level annotation protocol, annotators assigned distinct scores to the five candidates, resulting in no human ties. When human judgments reverse the intended construction order, the human scores take precedence. Accordingly, pairwise accuracy is computed against the preferences induced by the final human scores.

Q2.1: Human-label reliability: Can the authors report the relationship between construction tiers and blind human grades across the test set, including a confusion matrix and adjacent-tier ties/reversals?

Thank you for the question. The table below compares construction tiers with blind human grades on all 505 test candidates (Krippendorff's ordinal α = 0.864). 74.1% of candidates receive exactly their construction tier, and 92.7% fall within one tier. Under our group-level annotation protocol, the human labels contain no ties(see our response to Q1.2). Adjacent-tier reversals are reported in the second table.

Overall, 90.2% of within-group pairwise orderings agree with the construction order. Reversals concentrate in the upper tiers, which by construction are intended to differ mainly in dimensions beyond semantic adequacy; our additional aspect-specific audit confirms this, with an average Tier-4/Tier-5 gap of only 0.11 on the core correctness dimensions (S1–S3) versus 0.46 on the dimensions beyond semantic adequacy (SA1–SA4) (see our response to W1.2). Precisely because separating these tiers is harder, we evaluate tier-4-vs-tier-5 discrimination as a dedicated diagnostic, and all evaluations use the human judgments rather than construction tiers as ground truth.

Construction Tier \ Human Grade12345
1919001
2980912
318651215
402147015
502131868
Construction Tier PairReversalsTotal PairsAgreement
1 vs 2910191.1%
2 vs 3910191.1%
3 vs 42410176.2%
4 vs 52610174.3%
Non-adjacent3160694.9%
All99101090.2%
Reviewer_BhE8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BeyondAdeEval (BAE), a reference-free evaluation framework designed to assess machine translation (MT) quality beyond standard semantic adequacy. To address the limitations of existing learned metrics (e.g., COMET), the authors propose:

BAE-RUBRIC: A hierarchical rubric formalizing fine-grained quality dimensions such as rhythm, register, and idiomaticity.

BAE-BENCH: An English-to-Chinese (EN-ZH) paragraph-level benchmark featuring 6,960 training samples derived via controlled modifications of human translations, alongside a 505-candidate human-annotated test set.

BAE-SERIES: A family of evaluator models (4B to 9B parameters) that provide multi-dimensional, explainable evaluations.

The authors claim state-of-the-art performance, including a 24–28% relative improvement in correlation scores over existing baselines and zero-shot cross-lingual generalization on MQM24.

Critical strengths

Significant Target Domain Gap Addressed:

Moving MT evaluation beyond raw semantic correctness and adequacy is a crucial frontier. The paper tackles highly neglected linguistic properties—such as rhythm, register, and idiomaticity—which are vital for creative and human-aligned translation workflows.

Strong External Validation:

The framework demonstrates genuine cross-lingual generalization capability by achieving state-of-the-art performance on T-index (Human) and showing zero-shot capabilities on MQM24. This external validation alleviates fears that the system is entirely constrained to its own internal data pipeline.

High Empirical & Artifact Output:

The authors contribute a valuable open-source suite, including an entire family of trained evaluator models (BAE-SERIES from 4B to 9B parameters), a newly structured evaluation dataset, and a reusable hierarchical rubric.

Critical weaknesses

Methodological Vulnerabilities in Human Annotation Protocol:

As detailed in Table 8, the entire foundational "gold standard" test set relies on a single annotator and a single reviewer (N=2). While their linguistic qualification is solid, a pool of just two annotators from a single vendor is statistically insufficient to validate highly subjective, multidimensional rubric exploring rhythm, register, and idiomaticity.

Absence of Inter-Annotator Agreement (IAA):

Authors indicate that the annotator and reviewer discussed where they did not agree to each other, but they fail to report any formal statistical agreement metrics (e.g., Cohen’s Kappa or Krippendorff’s Alpha). Without reporting the baseline agreement, the reader cannot evaluate if the test set labels are reliable.

Benchmark Labels:

As mentioned in paragraph - "Effect of model scale.", if scaling up the model size degrades the performance on author's own benchmark, it strongly implies that the ground-truth labels in BAE-BENCH contain high variance or systematic noise.

Domain & Dimensional Sparsity of the Test Set:

A pool of just 101 unique source paragraphs is highly restrictive for an evaluation benchmark. It raises critical question regarding the domain coverage. If these 101 examples do not represent a broad, and balanced mix of text genre, the reported improvements may be artificial and fail to generalize.

Also, given that the BAE-RUBRIC evaluates fine-grained multi-dimensional parameters (such as rhythm, register, and idiomaticity), a total of 505 candidates means that individual stylistic dimensions will have an incredibly small footprint in the test data. The authors must provide a clear distribution table showing exactly how many candidates explicitly trigger or isolate failures in rhythm or register. Without this, it is impossible to verify if the 24–28% correlation improvement is uniform across all dimensions or heavily skewed by a single, dominant category like semantic adequacy.

Questions for authors

Inter-Annotator Agreement (IAA):

As shown in table 8, the annotator and reviewer possess excellent linguistic credentials, what was the quantitative inter-annotator agreement between them. Reporting this is vital to prove the objective reproducibility of the BAE-RUBRIC on highly subjective dimensions like rhythm and register.

Dimensional Distribution:

Please provide a table detailing the distribution of the 505 test candidates across the specific sub-dimensions of the BAE-RUBRIC. How many examples isolate rhythm, register, and idiomaticity respectively? Rationale Faithfulness:

As you claimed in the paper that the model learns "rubric-grounded rationales." How did you quantitatively evaluate the faithfulness of these rationales to prove they are linked to the final score rather than post-hoc hallucinations?

5 comments
AuthorsOpenReview ↗

Response to reviewer BhE8 [1/3]

We are truly grateful for your recognition of our work and your encouraging feedback. We address your questions and concerns below:

W1: As detailed in Table 8, the entire foundational "gold standard" test set relies on a single annotator and a single reviewer (N=2). While their linguistic qualification is solid, a pool of just two annotators from a single vendor is statistically insufficient to validate highly subjective, multidimensional rubric exploring rhythm, register, and idiomaticity.

Thank you for raising this concern. To assess label reliability, we conducted an additional round of independent quality verification through blinded re-annotation by a separate professional annotation company. We randomly sampled 50 of the 101 source groups (250 candidates, approximately 50% of the test set). The company independently assigned quality scores under BAE-RUBRIC without access to the original human labels or construction tiers.

The independently assigned scores achieved Krippendorff's ordinal α = 0.884 against the original human annotations, with 94.8% of candidates agreeing within one quality tier. These results provide additional evidence for the reproducibility of our human labels across independent annotation teams.

W2: Authors indicate that the annotator and reviewer discussed where they did not agree to each other, but they fail to report any formal statistical agreement metrics (e.g., Cohen’s Kappa or Krippendorff’s Alpha). Without reporting the baseline agreement, the reader cannot evaluate if the test set labels are reliable.

Thank you for this comment. In our original procedure, the reviewer examined the annotator's existing scores rather than scoring independently, and agreed with 495 of the 505 initial scores (98.0%); the remaining 10 were resolved through discussion. Because the reviewer's judgments were not made independently, agreement metrics such as Cohen's κ or Krippendorff's α cannot be meaningfully computed for this pair. However, we conducted the independent blinded re-annotation by a separate professional annotation company (see our response to W1), which yields Krippendorff's ordinal α = 0.884 against the original human labels, supporting the reliability of the test-set labels.

W3: As mentioned in paragraph - "Effect of model scale.", if scaling up the model size degrades the performance on author's own benchmark, it strongly implies that the ground-truth labels in BAE-BENCH contain high variance or systematic noise.

Thank you for raising this concern. In addition to the explanations discussed in "Effect of model scale," a paired cluster-level bootstrap test resampling the 101 source groups shows that the 4B–9B Spearman gap is within sampling uncertainty (ΔSpearman = +0.027, 95% CI [−0.001, 0.056], p = 0.058), suggesting performance saturation at 4B rather than a statistically significant degradation with scale. Moreover, the same test set clearly distinguishes models with larger performance gaps (BAE-9B vs. MetricX-24-XXL: ΔSpearman = +0.148, 95% CI [0.098, 0.197]), indicating that BAE-Bench retains meaningful discriminative signal rather than being dominated by label noise. Regarding the reliability of human annotations, please see our responses to W1 and W2.

W4.1: A pool of just 101 unique source paragraphs is highly restrictive for an evaluation benchmark. It raises critical question regarding the domain coverage. If these 101 examples do not represent a broad, and balanced mix of text genre, the reported improvements may be artificial and fail to generalize.

Thank you for bringing up this important point. We agree that a more diverse test set would further strengthen the benchmark. However, human annotation is costly, as each source paragraph requires expert grading of five candidate translations. Despite this limitation, our models generalize well beyond BAE-Bench, to different domains, language pairs, and evaluation protocols (pairwise ranking rather than pointwise scoring). Specifically, BAE-9B achieves state-of-the-art performance on T-index (Human) with 77.3% pairwise accuracy, and demonstrates zero-shot cross-lingual generalization on MQM24 En–De with 72.2%, outperforming specialized evaluators such as FiRE and MetricX-24-XXL. Moreover, the literary domain covered by our test set is a particularly challenging setting for evaluating translation quality beyond semantic adequacy. Therefore, although the test set is not yet fully diverse, BAE-Bench remains a meaningful benchmark for this task.

AuthorsOpenReview ↗

Response to reviewer BhE8 [2/3]

W4.2: Also, given that the BAE-RUBRIC evaluates fine-grained multi-dimensional parameters (such as rhythm, register, and idiomaticity), a total of 505 candidates means that individual stylistic dimensions will have an incredibly small footprint in the test data. The authors must provide a clear distribution table showing exactly how many candidates explicitly trigger or isolate failures in rhythm or register. Without this, it is impossible to verify if the 24–28% correlation improvement is uniform across all dimensions or heavily skewed by a single, dominant category like semantic adequacy.

Thank you for raising this concern. In our paper, rhythm, register, and idiomaticity are illustrative higher-order translation qualities encompassed by the four beyond-adequacy dimensions (SA1–SA4). To assess the dimensional coverage of BAE-BENCH, we conducted an additional human audit of all 505 test candidates across the seven BAE-RUBRIC dimensions. We define a dimension as triggered when its human score is ≤ 3, and isolated when its score is ≤ 3 while all six remaining dimensions score ≥ 4. The distribution is summarized below:

DimensionTriggeredIsolated
S1 Accuracy2211
S2 Language Quality2512
S3 Functional Equivalence2320
SA1 Natural Expression32022
SA2 Stylistic Refinement31313
SA3 Register / Genre Fit2823
SA4 Tone / Formality2802

To further examine whether the reported performance gains (28% Spearman and 24% Pearson improvements) are driven primarily by semantic adequacy, we compared BAE-4B with MetricX-24-XXL, the strongest learned-metric baseline, on the 229 candidates whose core dimensions (S1–S3) all scored ≥ 4. On this subset, BAE-4B achieves a Spearman correlation of 0.374, compared with −0.018 for MetricX-24-XXL (Δ = +0.393; 95% source-group bootstrap CI: [+0.244, +0.549]).

Furthermore, on the same 229-candidate subset, using only the models' overall predictions, BAE-4B shows stronger correlations with each of the four human-annotated beyond-adequacy dimensions:

DimensionBAE-4BMetricX-24-XXLΔ
SA1 Natural Expression0.304−0.003+0.307
SA2 Stylistic Refinement0.400−0.002+0.402
SA3 Register / Genre Fit0.3690.122+0.247
SA4 Tone / Formality0.309−0.080+0.389

These findings provide additional evidence that BAE's correlation improvements are not driven solely by semantic adequacy, but also reflect sensitivity to beyond-adequacy quality differences.

Q1: Inter-Annotator Agreement (IAA): As shown in table 8, the annotator and reviewer possess excellent linguistic credentials, what was the quantitative inter-annotator agreement between them. Reporting this is vital to prove the objective reproducibility of the BAE-RUBRIC on highly subjective dimensions like rhythm and register.

Please see our response to W2.

Q2: Dimensional Distribution: Please provide a table detailing the distribution of the 505 test candidates across the specific sub-dimensions of the BAE-RUBRIC. How many examples isolate rhythm, register, and idiomaticity respectively?

Please see our response to W4.2.

AuthorsOpenReview ↗

Response to reviewer BhE8 [3/3]

Q3: Rationale Faithfulness: As you claimed in the paper that the model learns "rubric-grounded rationales." How did you quantitatively evaluate the faithfulness of these rationales to prove they are linked to the final score rather than post-hoc hallucinations?

Thank you for this important question. We address its two parts in turn: whether the rationales are grounded in the translation rather than hallucinated, and whether rationale supervision improves scoring.

First, we conducted a human audit of all 505 rationales produced by BAE-9B during its original evaluation on the BAE-Bench test set, assessing whether each dimension-level explanation accurately identifies observable translation properties and is faithful to the translation evidence. Among them, 464 out of 505 rationales (91.9%) contained no issues and identified the observable translation properties, while the remaining 41 contained 51 dimension-level errors in total. Across all seven rubric dimensions, 3,484 out of 3,535 explanations (98.6%) were judged correct. The audit results are summarized below:

DimensionIncorrect / AuditedCorrectness
S1 Accuracy25 / 50595.0%
S2 Language Quality6 / 50598.8%
S3 Functional Equivalence15 / 50597.0%
SA1 Natural Expression2 / 50599.6%
SA2 Stylistic Refinement1 / 50599.8%
SA3 Register / Genre Fit2 / 50599.6%
SA4 Tone / Formality0 / 505100.0%
Overall51 / 353598.6%

Second, regarding the contribution of rationale supervision to scoring, we trained a score-only variant of BAE-9B using exactly the same training data, recipe, and evaluation protocol, but with rationales removed from both the supervision targets and the prompts. Compared with this variant, BAE-9B achieves a 4.5% relative improvement on T-index (Human) and 3.6% on MQM24 En–De.

Together, these results show that the rationales cite properties actually present in the translation rather than post-hoc hallucinations, and that rationale supervision improves scoring performance.

Reviewer_sWkkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BeyondAdeEval (BAE), a framework for reference-free machine translation evaluation that explicitly distinguishes semantic adequacy from finer-grained aspects of translation quality. Its contributions are a hierarchical evaluation “rubric”, a preference-graded English–Chinese benchmark constructed through controlled degradation of professional translations, and a family of fine-tuned evaluators trained using supervised fine-tuning and GRPO. The models produce both an overall quality score and a structured rationale. The paper reports promising results on its in-domain benchmark and two external pairwise evaluation benchmarks.

Critical strengths
  • Constructing multiple quality levels for the same source provides controlled comparisons and makes it possible to evaluate discrimination between adjacent quality tiers. The independently annotated test set is preferable to evaluating exclusively against synthetic construction labels.
  • Experiments on T-index Human and MQM24 En–De provide useful evidence that the approach is not restricted to its own benchmark. The latter also probes transfer to a language pair absent from training.
  • The paper compares rubric prompting, SFT, and GRPO, and provides relevant training hyperparameters and computational costs.

I really like the idea of structured data construction; the other two points reflect the authors’ scientific rigour.

Critical weaknesses
  • The training data are generated by degrading high-quality translations according to the proposed rubric, and the rationale teacher receives the target quality tier explicitly. Consequently, the evaluator may learn to recognize the artefacts of the synthetic degradation process and reproduce the rubric's assumptions rather than learn a general notion of translation quality. Human annotation of the test set mitigates direct label leakage but does not establish robustness to naturally occurring quality differences. Evaluation on naturally generated translations, stratified by error type and source system, would substantially strengthen the paper.
  • The models produce structured rationales, but the paper does not sufficiently evaluate whether these explanations faithfully identify actual translation properties or merely rationalize the predicted score. Human assessment of rationale correctness and agreement at the individual rubric-dimension level would help support this claim.
  • there is very little discussion of how the evaluation rubric itself was designed and justified, and the paper offers few insights into the actual evaluation of translations. More broadly, the related work on machine translation evaluation is rather limited, with no clear positioning relative to established frameworks such as MQM or approaches such as GEMBA and its variants.
Questions for authors
  • How well does BAE perform on naturally occurring translation outputs that do not exhibit the characteristic artefacts of the synthetic degradation process?
  • Can the authors demonstrate that the generated rationales faithfully reflect identifiable translation properties rather than merely justify the predicted score?
  • How does BAE compare with strong prompted LLM baselines under a fully matched evaluation protocol across all external benchmarks?
3 comments
AuthorsOpenReview ↗

Response to reviewer sWkk [1/3]

We sincerely thank you for your careful review and valuable feedback on our work. We address your questions and concerns below:

W1: The training data are generated by degrading high-quality translations according to the proposed rubric, and the rationale teacher receives the target quality tier explicitly. Consequently, the evaluator may learn to recognize the artefacts of the synthetic degradation process and reproduce the rubric's assumptions rather than learn a general notion of translation quality. Human annotation of the test set mitigates direct label leakage but does not establish robustness to naturally occurring quality differences. Evaluation on naturally generated translations, stratified by error type and source system, would substantially strengthen the paper.

Thank you for pointing this out. We would like to clarify that we have already evaluated BAE on two independent benchmarks containing naturally generated translations (Section 3.5). Specifically, T-index (Human) consists of translations generated by seven MT systems/LLMs without controlled degradation, while MQM24 En–De contains outputs from multiple MT systems and LLMs in the WMT24 General MT Shared Task, covering four domains with expert MQM annotations. BAE-9B achieves 77.3% pairwise accuracy on T-index (Human) and 72.2% on MQM24 En–De. These results provide direct evidence of generalization beyond our construction pipeline.

While T-index (Human) does not provide per-candidate source-system identities or human-annotated MQM error types, MQM24 provides both. We therefore further stratified the MQM24 results by MQM error type and source system, including comparisons with established baselines, and additionally examined subsets without annotated Accuracy errors, as shown below:

(1) Stratification by Error Type

MQM Error Type# PairsCOMET-QEFiREQwen+RBAE-9B
Accuracy75273.6773.5471.9473.67
Fluency75669.5869.0567.7270.50
Style70869.3570.7669.4973.31
Terminology12071.6779.1775.8380.83
Overall1,20069.5070.6769.7572.17

(2) Stratification by Source System

Source System# PairsCOMET-QEFiREQwen+RBAE-9B
MSLC23986.6187.4589.1287.45
IKUN-C21467.7671.9671.9672.90
Llama3-70B18666.6768.8264.5274.73
ONLINE-A18164.6470.7271.8269.61
IOL_Research17967.0467.0462.5764.80
IKUN17877.5376.9771.9175.28
ONLINE-B16467.6871.3468.9070.12
Dubformer16063.1271.8870.6273.12
Gemini-1.5-Pro15569.0365.8163.2370.97
Aya2315362.0963.4067.3264.71
Mistral-Large14366.4359.4462.2466.43
CommandR-plus14172.3469.5067.3873.76
Claude-3.514168.7968.0965.9668.79
GPT-413366.9266.1769.9268.42
ONLINE-W3360.6166.6760.6172.73
Overall1,20069.5070.6769.7572.17

(3) Beyond-Adequacy Subset Analysis

MQM Condition# PairsCOMET-QEFiREQwen+RBAE-9B
No Accuracy errors44862.5065.8566.0769.64
No Accuracy errors; Style present27161.2567.1666.0572.32
Style-only category15261.1867.7665.7971.05

All results are pairwise accuracy (%). Qwen+R denotes Qwen3.5-9B + BAE-RUBRIC. Error-type and source-system groups may overlap because a pair can exhibit multiple MQM error types and involves two source systems. The 271-pair subset contains an annotated Style error and no annotated Accuracy error in either candidate.

Notably, on 271 naturally generated MT pairs with Style errors but no annotated Accuracy errors, BAE-9B achieves 72.32% accuracy, outperforming Qwen3.5-9B + BAE-RUBRIC (66.05%), FiRE (67.16%), and COMET-QE (61.25%), providing further evidence that its quality judgments generalize beyond artifacts specific to our synthetic construction process.

AuthorsOpenReview ↗

Response to reviewer sWkk [2/3]

W2: The models produce structured rationales, but the paper does not sufficiently evaluate whether these explanations faithfully identify actual translation properties or merely rationalize the predicted score. Human assessment of rationale correctness and agreement at the individual rubric-dimension level would help support this claim.

Thank you for raising this important concern. We agree that evaluating the correctness and faithfulness of generated rationales is essential. To address this, we conducted a human audit of the rationales produced by BAE-9B during its original evaluation on the BAE-Bench test set, assessing whether each dimension-level explanation accurately identifies observable translation properties and is faithful to the translation evidence.

We reviewed all 505 rationales. Among them, 464 out of 505 rationales (91.9%) contained no identified issues, while the remaining 41 contained 51 dimension-level errors in total. Across all seven rubric dimensions, 3,484 out of 3,535 explanations (98.6%) were judged correct. The audit results are summarized below:

DimensionIncorrect / AuditedCorrectness
S1 Accuracy25 / 50595.0%
S2 Language Quality6 / 50598.8%
S3 Functional Equivalence15 / 50597.0%
SA1 Natural Expression2 / 50599.6%
SA2 Stylistic Refinement1 / 50599.8%
SA3 Register / Genre Fit2 / 50599.6%
SA4 Tone / Formality0 / 505100.0%
Overall51 / 353598.6%

To further examine the contribution of rationale supervision, we conducted an additional ablation by training a score-only variant of BAE-9B using the same training data, training recipe, and evaluation protocol, but removing rationales from both the supervision targets and prompts. Compared with this variant, BAE-9B achieves 4.5% and 3.6% relative improvements on T-index (Human) and MQM24 En–De, respectively.

Together, the human audit provides direct evidence that the generated rationales are faithful to the translation evidence and accurately reflect observable translation properties at both the overall and individual rubric-dimension levels, rather than citing unsupported properties merely to rationalize the predicted score, while the ablation results further support the effectiveness of rationale supervision in improving evaluation performance.

W3: there is very little discussion of how the evaluation rubric itself was designed and justified, and the paper offers few insights into the actual evaluation of translations. More broadly, the related work on machine translation evaluation is rather limited, with no clear positioning relative to established frameworks such as MQM or approaches such as GEMBA and its variants.

Thank you for this suggestion. BAE-RUBRIC draws on established criteria concerning source fidelity, target-language quality, and stylistic appropriateness (Sec. 2.2). Its organization into three core dimensions and four finer-grained dimensions makes core translation requirements explicit while providing additional criteria for distinguishing otherwise adequate translations. Expert pilot annotation identified ambiguous criteria and unclear dimension boundaries, informing revisions to the guidelines. Stylistic refinement is assessed relative to the source and context.

GEMBA-MQM is explicitly discussed in our Related Work (Sec. 4.2) and included as a baseline (Sec. 3.1). MQM already covers style and register; our contribution lies in organizing overlapping criteria into explicit quality tiers and using them for evaluator training. Whereas GEMBA-MQM identifies and aggregates translation errors through prompting, BAE combines graded assessment with dimension-specific rationales and supervision across quality tiers.

To assess inference-time utility, the following comparison uses the same Qwen3.5-9B backbone without parameter updates.

Evaluation approachBAE-Bench Spearman ↑T-index (Human) Acc. ↑MQM24 En–De Acc. ↑
GEMBA-MQM0.52749.354.6
Rubric-MQM0.49253.357.8
EAPrompt0.40240.753.8
AutoMQM0.44548.048.8
BAE-RUBRIC prompting0.68375.369.8

Results from Table 3 and Figure 3. Accuracy is reported in %; Methods retain their respective prompting and scoring procedures.

BAE-RUBRIC prompting outperforms the evaluated MQM-based prompting baselines across all three benchmarks, supporting its practical effectiveness beyond our own annotation scheme. Our high-quality pairwise diagnostic (Sec. 3.4) further highlights the need to evaluate fine-grained quality discrimination separately from aggregate score correlation. We will clarify these design choices, empirical insights, and comparisons in Secs. 2.2 and 4.2 and the experimental discussion.

AuthorsOpenReview ↗

Response to reviewer sWkk [3/3]

Q1: How well does BAE perform on naturally occurring translation outputs that do not exhibit the characteristic artefacts of the synthetic degradation process?

Please see our response to W1.

Q2: Can the authors demonstrate that the generated rationales faithfully reflect identifiable translation properties rather than merely justify the predicted score?

Please see our response to W2.

Q3: How does BAE compare with strong prompted LLM baselines under a fully matched evaluation protocol across all external benchmarks?

Thank you for raising this concern. In our experiments, BAE-SERIES and BAE-RUBRIC-augmented LLM baselines are evaluated under exactly the same protocol. Specifically, they use the same evaluation prompt (Prompt 6 in Appendix F.2), with identical inputs, tasks, and evaluation criteria. Under this matched protocol, BAE-9B outperforms its Qwen3.5-9B + BAE-RUBRIC backbone on both benchmarks (77.3% vs. 75.3% on T-index; 72.2% vs. 69.8% on MQM24 En–De, Section 3.5). On T-index, it also surpasses both frontier LLMs with the rubric, GPT-6 Astra (71.3%) and DeepSeek-V4.1-Flash (70.7%). On MQM24 En–De, it approaches them (78.3% and 76.5%, respectively), despite never seeing German or pairwise data in training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.