- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Paper provides an view on how corrections often provides no positive effects and sometimes reduces the quality which is an well thought and aligns with real world over correction issue. Paper also provides an theortical view with bounds in which translation can be corrected without overall degradation of population. Paper introduces RCAS which takes MT and generates multiple revision using LLM, then uses an trained predictor to estimate candidates benefit or degradation relative to current translation.
Critical strengths
- Paper provides an theortical analysis that gains are subject to available opportunites in its candidate pool, and break down excess risk at a fixed edit rate into two sources: editing the wrong inputs and applying the wrong repair, which seems intuitive and clean.
- Paper connect the theortical analysis to proposed approach RCAS which formulates repair as maximizing bounded improvement subject to a population degradation budget.
Critical weaknesses
- Even though formulation is intuitive, but its mostly standard, the main new idea is decomposition of errors, but paper never measures their effects so it is unclear whether this decomposition actually explains why RCAS works. Optimization framework is also not explored experimentally. Finally the way ties are handeld seems to favour editing the method counts any positive gain as a reason to edit but ignores small losses when calculating another part of the decision rule.
- Experimental gains of the methods are very marginal. The experiments do not clearly show that RCAS makes better decisions than simpler methods. The simple QE-best baseline performs better than RCAS on COMET-22 across all five datasets. Method only performs better on the hybrid metrics it was trained on. The ablation results also suggest that a simpler signed-gain scoring method can perform as well as or better than the proposed net-value rule.
Questions for authors
- Can authors estimate how much of RCAS's error comes from editing the wrong inputs versus choosing the wrong repair?
- Does RCAS still perform well on metrics that were not used for training, such as MetricX-24, xCOMET, and BLEURT?
- The current rule rewards small positive gains but ignores small negative gains. Could this cause the model to edit too often, even when the changes are just noise?
- Can authors compare RCAS with methods that use a similar number of model calls? For example, compare it with generating nine candidate translations and selecting the best using QE while allowing KEEP, and with a version of TEaR that performs multiple rounds.
5 comments
W1 & Q3. Small gains, small losses, and the risk of over-editing
We thank the reviewer for asking whether our treatment of small gains and losses encourages RCAS to edit too often. A small positive gain contributes to expected benefit, but the decision to edit depends on the revision’s net score. RCAS scores each eligible revision as b̂ − λℓ̂ + c, where b̂ is expected benefit, ℓ̂ is expected excess loss, and c is a calibrated offset. It edits only when the highest net score exceeds the score of KEEP, which is zero. During training, decreases within the tolerance m retain their negative values in the gain predictor’s target (Eq. 21).
The loss penalty reduces editing when all revisions score about as well as the initial translation. To examine the reviewer’s concern, we used reference translations to identify inputs for which every eligible revision changed the utility of the initial translation by at most m in either direction (Eq. 18). This criterion identified 43 of 1,468 WMT23 En→Zh inputs and 130 of 2,521 WMT23 Zh→En inputs. We compared decisions on the same inputs with the default RCAS penalty (λ = 1) and with the penalty removed (λ = 0). Both settings used the same trained selector, candidates, and offset c, so the comparison isolates the effect of the loss penalty.
| WMT23 test set | Inputs evaluated | No. of inputs | Edit rate without loss penalty (λ = 0) | Edit rate with loss penalty (λ = 1) | Edit → KEEP |
|---|---|---|---|---|---|
| En→Zh | Full test set | 1,468 | 99.6% | 72.7% | 26.9% |
| En→Zh | All revisions within ±m | 43 | 100% | 46.5% | 53.5% |
| Zh→En | Full test set | 2,521 | 99.2% | 85.6% | 13.6% |
| Zh→En | All revisions within ±m | 130 | 100% | 67.5% | 32.5% |
Table 3. Edit rates with and without the loss penalty.
Without the loss penalty, the edit rate was 100% in both subsets. With the penalty, it fell to 46.5% for En→Zh and 67.5% for Zh→En. In both directions, the drop was larger when all revisions scored close to the initial translation. For En→Zh, the edit rate fell by 53.5 percentage points on this subset and by 26.9 points on the full test set. For Zh→En, the corresponding drops were 32.5 and 13.6 points.
Case study. When RCAS does edit, expected excess loss can also change which revision it selects. In Table 4, the rewrite ranks first by expected benefit and fusion ranks fourth. The rewrite offers 0.00377 more expected benefit than fusion but carries 0.00596 more expected excess loss. At λ = 1, fusion therefore has the highest net score, and RCAS selects it.
| Action | Expected benefit | Expected excess loss | Net score | Selected |
|---|---|---|---|---|
| KEEP | 0 | 0 | 0 | ✗ |
| Rewrite | 0.17270 | 0.00811 | 0.16459 | ✗ |
| Retranslation 1 | 0.16997 | 0.03699 | 0.13298 | ✗ |
| Retranslation 2 | 0.16997 | 0.04298 | 0.12699 | ✗ |
| Fusion | 0.16893 | 0.00215 | 0.16679 | ✓ |
| Composite | 0.09608 | 0.03439 | 0.06168 | ✗ |
Table 4. Scores for all eligible actions on one WMT23 En→Zh test sentence. Candidate generation and clause composition are described in Appendix C.3. The numbers distinguish the two retranslation candidates.
Choosing the loss penalty weight. The cost of a harmful edit in a given application can guide the choice of λ on development data. On the WMT23 En→Zh test set, increasing λ from 0.5 to 2 reduced mean realized excess loss from 11.81% to 2.53% of the loss cap across five seeds, while mean realized benefit fell from 0.0524 to 0.0393.
W2 & Q2. Comparison with QE-best
Thank you for raising this point. We agree that QE-best achieves higher mean COMET-22 than RCAS on all five datasets. To examine whether the comparison extends beyond the training objective, we compared two versions of RCAS, one trained on Qω and the other on COMET-22, with QE-best on the same candidate pool. We evaluated the final translations with BLEURT, XCOMET-XXL, and MetricX-24, none of which was used to train RCAS. Alongside mean quality, we report the degradation rate.
Additional metric scores and degradation rates for the two RCAS variants and QE-best. As Table 5 shows, the versions trained on Qω and COMET-22 scored 0.003574 and 0.000761 lower than QE-best on COMET-22, respectively. Their degradation rates were 11.93% and 14.79%, compared with 16.84% for QE-best. In Table 5, both versions of RCAS achieved higher mean BLEURT than QE-best. The version trained on COMET-22 also scored better than QE-best on MetricX-24 and XCOMET-XXL, although the XCOMET-XXL difference was small.
| Method | Qω ↑ | COMET-22 ↑ | BLEURT ↑ | XCOMET-XXL ↑ | MetricX-24 ↓ | Degradation rate ↓ |
|---|---|---|---|---|---|---|
| QE-best | 1.245109 | 0.865654 | 0.761903 | 0.960726 | 0.790192 | 16.84% |
| RCAS (trained on Qω) | 1.255789 | 0.862081 | 0.764082 | 0.960173 | 0.822842 | 11.93% |
| RCAS (trained on COMET-22) | 1.252224 | 0.864893 | 0.765035 | 0.960883 | 0.774487 | 14.79% |
Table 5. Translation quality and degradation rates for RCAS and QE-best using the same candidate pool on WMT22 En→De. Throughout this response, all quality metrics are reported on their original scales. COMET-22 and Qω scores and gains are therefore not multiplied by 100 as in the paper.
We also assessed human preferences directly, since automatic metrics can rank translation systems differently from human evaluators [1]. We sampled 100 WMT23 En→Zh pairs (seed 42) for which RCAS (trained on Qω) and QE-best produced different translations. Two experts judged each pair independently, without knowing which method produced either translation. Reported percentages are adjusted to account for differences in sampling probability.
| Expert judgment | Preference (%) |
|---|---|
| RCAS preferred | 28.6% |
| Tie | 47.8% |
| QE-best preferred | 23.6% |
Table 6. Blinded expert comparison of RCAS and QE-best on 100 WMT23 En→Zh pairs with different outputs.
When the outputs differed, the share of judgments favoring RCAS was 5.0 percentage points higher than the share favoring QE-best. Counting identical outputs as ties, the estimated overall net preference for RCAS was +3.3 percentage points (95% CI [−4.4, +11.2]). The point estimate favors RCAS, but the interval includes zero, so the data do not establish a clear overall preference.
[1] Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies. ACL’24.
We thank the reviewer for the careful reading and constructive feedback. We address the comments and questions below.
W1 & Q1: Errors in sentence selection and revision selection
Thank you for asking how much error comes from choosing which inputs to edit and how much comes from choosing a revision. At matched edit counts, RCAS reduced degradation relative to QE mainly by choosing better revisions for the inputs it edited.
Sentence selection and revision selection. On WMT23 En→Zh and Zh→En, RCAS and QE used the same candidate pool and edited the same number of inputs. For each method, we first held its edited inputs fixed and counted how many degradations could have been avoided by choosing another eligible revision for the same input. This gives the revision selection error. We then counted how many additional degradations could have been avoided by editing different inputs while keeping the edit count fixed. This gives the sentence selection error.
Table 1. Degradation rate and its two avoidable components at matched edit counts.
Compared with QE, RCAS reduced revision selection error by 3.11 percentage points on En→Zh and 1.71 points on Zh→En. Among RCAS’s own remaining avoidable degradations, sentence selection accounted for 63.3% on En→Zh and 56.9% on Zh→En. Revision selection accounted for the remaining 36.7% and 43.1%, respectively.
Revision selection under low and high predicted loss. For each input, we identified the eligible revision with the highest predicted benefit according to RCAS. We then compared how often RCAS and QE-best selected that revision. Both methods could choose from the same revisions or keep the initial translation. QE-best chose the option with the highest QE score.
Table 2. Selection of the highest benefit revision on WMT24. We defined the low loss groups and high loss groups using the lower and upper tertile cutoffs for predicted loss among these high benefit development examples.