- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces BeyondAdeEval (BAE), a reference-free translation evaluation framework targeting quality distinctions beyond semantic adequacy. Its training data and primary benchmark focus on English-to-Chinese literary translation. The authors propose a seven-dimensional rubric and construct five quality tiers through controlled degradation of professionally translated anchors, then train 4B and 9B evaluators using supervised fine-tuning and GRPO. The primary evaluation uses a source-disjoint test set with human scoring blinded to the construction tiers. Additional evaluations on T-index (Human) and MQM24 English-to-German assess transfer to naturally occurring translations and an additional language pair, respectively. The results support the usefulness of rubric-guided supervision relative to the same backbone with rubric prompting alone. My main reservation concerns whether the synthetic construction reliably isolates the intended quality dimensions, particularly whether Tier-4 candidates preserve adequacy while differing from Tier 5 primarily in higher-order quality.
Critical strengths
-
A relevant evaluation problem with evidence that targeted supervision helps. Distinguishing translations that are broadly correct but differ in naturalness, style, and communicative effectiveness is important. The same-backbone comparison provides useful evidence: supervised training improves Qwen3.5-9B over rubric prompting, including an increase in Tier-4-versus-Tier-5 accuracy from 49.5% to 61.4%.
-
Human evaluation and external benchmarks provide meaningful safeguards against a purely construction-dependent result. The test annotator and reviewer are blinded to construction tiers and intended ordering, and the final human scores are used as evaluation labels. The external T-index evaluation additionally tests naturally occurring system outputs. These choices strengthen the empirical contribution beyond agreement with synthetic construction labels alone.
Critical weaknesses
-
The reliability and dimensional specificity of the synthetic quality control are insufficiently quantified. Section 2.3 describes human inspection of a subset of constructed groups, but does not report the sample size, sampling procedure, agreement with intended tiers, or correction/rejection rates. This is particularly consequential for Tier 4: the generation prompt requires preservation of semantic accuracy, language quality, and functional equivalence while degrading only higher-order qualities. Without an independent, aspect-specific audit, it remains unclear how consistently this constraint is satisfied. If Tier-4 candidates also contain adequacy errors, correctly ranking them below Tier 5 would not uniquely demonstrate beyond-adequacy discrimination. The blind overall test scores are valuable, but the absence of per-aspect judgments leaves this central interpretation under-validated. A related concern is whether prompting alone provides sufficiently reliable control over the intended quality tiers. The construction procedure assumes that the model can realize fine-grained quality distinctions through rubric-guided editing. However, Table 3 shows substantial disagreement between prompted LLM judges and human grades, even with BAE-RUBRIC (50% acc.). These capabilities need not be equivalent. However, the effectiveness of generating training data through prompting needs reconsideration.
-
The relationship between construction labels, human verification, and teacher judgments needs clarification. The rationale-generation prompt describes the quality bands as already established through human annotation, whereas the training-data description documents constructed tiers and subset inspection. It is therefore unclear whether all training labels were independently assigned or corrected by humans. Moreover, label-conditioned rationales explain a supplied tier and cannot independently establish that tier’s correctness.
Questions for authors
-
Validity of the high-quality diagnostic: What evidence establishes that Tier-4 and Tier-5 candidates preserve comparable semantic adequacy and differ primarily along the intended higher-order dimensions? An independently annotated audit separating adequacy from stylistic quality would materially strengthen this claim. For the reported pairwise diagnostic, is the preferred candidate determined by construction order or final human scores, and how are human ties or reversals handled?
-
Human-label reliability: Can the authors report the relationship between construction tiers and blind human grades across the test set, including a confusion matrix and adjacent-tier ties/reversals? Were any judgments collected independently from multiple annotators before review? Please report available pre-adjudication agreement and the frequency of review changes or adjudication; final consensus alone does not establish independent reproducibility.
-
Training-label provenance and verification: Were the 6,960 training labels retained from the requested construction tiers, independently assigned by humans, or revised through a combination of these procedures? Please clarify the statement in the rationale prompt that the bands were established by human annotation. How many training groups were inspected, how were they sampled, and how frequently were candidates relabeled, corrected, regenerated, or discarded?
6 comments
Response to reviewer LmWJ [2/4]
W1.3: A related concern is whether prompting alone provides sufficiently reliable control over the intended quality tiers. The construction procedure assumes that the model can realize fine-grained quality distinctions through rubric-guided editing. However, Table 3 shows substantial disagreement between prompted LLM judges and human grades, even with BAE-RUBRIC (50% acc.). These capabilities need not be equivalent. However, the effectiveness of generating training data through prompting needs reconsideration.
Thank you for raising this concern. We would like to clarify that rubric-guided data construction and reference-free quality evaluation are distinct tasks. Unlike evaluation, which requires assigning an absolute quality score to a translation without a reference, our construction procedure starts from a verified Tier-5 human translation and applies targeted, rubric-guided modifications under explicit quality constraints (Section 2.3.1). Thus, the exact-score accuracy of prompted LLM judges in Table 3 does not directly measure the reliability of our controlled construction process. To empirically validate the construction quality, we compared the intended construction tiers with assigned human grades on all 505 test candidates. The test candidates were produced by exactly the same construction pipeline as the training data. The agreement below therefore serves as a direct estimate of how reliably the pipeline realizes the intended tiers in the training data. Annotators were blinded to the construction tiers and intended ordering. The results are summarized below:
| Construction Tier \ Human Grade | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| 1 | 91 | 9 | 0 | 0 | 1 |
| 2 | 9 | 80 | 9 | 1 | 2 |
| 3 | 1 | 8 | 65 | 12 | 15 |
| 4 | 0 | 2 | 14 | 70 | 15 |
| 5 | 0 | 2 | 13 | 18 | 68 |
The intended construction tiers demonstrate substantial agreement with independent human judgments (Krippendorff's ordinal α = 0.864), providing empirical support for the reliability of rubric-guided quality construction.
W2.1: The relationship between construction labels, human verification, and teacher judgments needs clarification. The rationale-generation prompt describes the quality bands as already established through human annotation, whereas the training-data description documents constructed tiers and subset inspection. It is therefore unclear whether all training labels were independently assigned or corrected by humans.
Thank you for pointing out this ambiguity. We apologize for the misleading wording in the rationale-generation prompt and would like to clarify the actual procedure. The training quality tiers were established through rubric-guided construction, rather than independently assigned by human annotators. We first curated and verified high-quality Tier-5 translations, then generated Tier-1–4 variants through controlled degradation under BAE-RUBRIC. Human annotators inspected a subset of the constructed groups to assess the intended quality distinctions, rather than independently labeling every training example. The wording in the rationale-generation prompt originated from an earlier prompt-development stage, where we used human-annotated examples to calibrate the teacher's explanations. Because the teacher is conditioned on the supplied tier regardless of this phrasing, the statement does not change which labels the rationales explain. We would like to clarify that this wording does not describe how the full training dataset was labeled. We further discuss the role of label-conditioned rationales and the validity of the constructed tiers in W2.2 below.
W2.2: Moreover, label-conditioned rationales explain a supplied tier and cannot independently establish that tier’s correctness.
Thank you for this important point. We agree that label-conditioned rationales cannot independently establish the correctness of the supplied tier, and we do not use them for that purpose. Their role is to serve as distillation targets during SFT, teaching the student model to express its judgments in terms of the BAE-RUBRIC dimensions. As described in Section 2.3.1, we condition the teacher on the assigned tier so that the rationale explains the supervised tier rather than independently re-estimating the score. Label reliability is assessed separately through independent evidence: the agreement between construction tiers and blind human grades (see our response to Q2.1) and the aspect-specific human audit (see our response to W1.2).
Response to reviewer LmWJ [3/4]
Q1.1: Validity of the high-quality diagnostic: What evidence establishes that Tier-4 and Tier-5 candidates preserve comparable semantic adequacy and differ primarily along the intended higher-order dimensions? An independently annotated audit separating adequacy from stylistic quality would materially strengthen this claim.
Please see our response to W1.2.
Q1.2: For the reported pairwise diagnostic, is the preferred candidate determined by construction order or final human scores, and how are human ties or reversals handled?
Thank you for pointing this out. We acknowledge that this was not sufficiently clarified in the original manuscript. We would like to clarify that the high-quality pairwise diagnostic directly compares candidates assigned final human grades 4 and 5 within each source group. Both pair selection and ground-truth preferences are determined by the final human-annotated scores, rather than the construction order.
During annotation, annotators could view all five candidates within each source group but were blinded to their construction tiers and intended ordering. Under our group-level annotation protocol, annotators assigned distinct scores to the five candidates, resulting in no human ties. When human judgments reverse the intended construction order, the human scores take precedence. Accordingly, pairwise accuracy is computed against the preferences induced by the final human scores.
Q2.1: Human-label reliability: Can the authors report the relationship between construction tiers and blind human grades across the test set, including a confusion matrix and adjacent-tier ties/reversals?
Thank you for the question. The table below compares construction tiers with blind human grades on all 505 test candidates (Krippendorff's ordinal α = 0.864). 74.1% of candidates receive exactly their construction tier, and 92.7% fall within one tier. Under our group-level annotation protocol, the human labels contain no ties(see our response to Q1.2). Adjacent-tier reversals are reported in the second table.
Overall, 90.2% of within-group pairwise orderings agree with the construction order. Reversals concentrate in the upper tiers, which by construction are intended to differ mainly in dimensions beyond semantic adequacy; our additional aspect-specific audit confirms this, with an average Tier-4/Tier-5 gap of only 0.11 on the core correctness dimensions (S1–S3) versus 0.46 on the dimensions beyond semantic adequacy (SA1–SA4) (see our response to W1.2). Precisely because separating these tiers is harder, we evaluate tier-4-vs-tier-5 discrimination as a dedicated diagnostic, and all evaluations use the human judgments rather than construction tiers as ground truth.
| Construction Tier \ Human Grade | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| 1 | 91 | 9 | 0 | 0 | 1 |
| 2 | 9 | 80 | 9 | 1 | 2 |
| 3 | 1 | 8 | 65 | 12 | 15 |
| 4 | 0 | 2 | 14 | 70 | 15 |
| 5 | 0 | 2 | 13 | 18 | 68 |
| Construction Tier Pair | Reversals | Total Pairs | Agreement |
|---|---|---|---|
| 1 vs 2 | 9 | 101 | 91.1% |
| 2 vs 3 | 9 | 101 | 91.1% |
| 3 vs 4 | 24 | 101 | 76.2% |
| 4 vs 5 | 26 | 101 | 74.3% |
| Non-adjacent | 31 | 606 | 94.9% |
| All | 99 | 1010 | 90.2% |
Response to reviewer LmWJ [1/4]
We sincerely appreciate your insightful feedback and constructive suggestions. We address your questions and concerns below:
Thank you for pointing this out. We report the requested statistics below:
After removing the 20 rejected groups, the final training set contains 1,392 groups, as reported in the paper. Regarding dimensional specificity, we conducted a per-dimension human audit of all test candidates; please see our response to W1.2.
To directly examine whether Tier-4 and Tier-5 candidates differ primarily in dimensions beyond adequacy, we conducted an additional aspect-specific human audit of all test candidates across the seven BAE-RUBRIC dimensions, blinded to their construction tiers and intended ordering. The Tier-4/Tier-5 results are summarized below.
The audit shows that Tier-4/Tier-5 differences are concentrated in beyond-adequacy dimensions, with an average gap of 0.46 points across SA1–SA4, compared with only 0.11 across S1–S3. At a threshold of 4.5 on the respective dimension-group averages, Tier 4 and Tier 5 exhibit similar pass rates in core quality (78.2% vs. 82.2%), but a substantially larger gap in beyond-adequacy quality (31.7% vs. 81.2%). These findings support that Tier 4 largely preserves core translation quality while differing from Tier 5 primarily in dimensions beyond adequacy.