acceptodds
Under review as a conference paper at ICLR 2027

Which Revisions Are Worth Applying? Risk-Aware Repair Decisions for Translation Self-Correction

Abstract

Large language models can refine machine translations, but a proposed revision may also damage content that is already correct. This matters more as initial translations improve, i.e., stronger translations contain fewer beneficial revisions and offer smaller attainable gains, motivating self-correction as a selective decision over whether and how to revise. We formulate translation repair as maximizing expected bounded improvement subject to a population degradation budget; the Lagrangian yields a net-value rule that compares each repair with retaining the current translation. We instantiate this rule in Reliable Correction via Action Selection (RCAS), which generates candidates along multiple repair directions, estimates their benefit and degradation relative to the current translation without test-time references, and applies the highest positive-net-value repair or retains the current translation. Across five WMT test sets, RCAS achieves the highest mean COMET among the evaluated repair pipelines on four datasets. Frozen-pool controls show that overriding its retain decisions substantially increases population regression, and a blinded audit of 250 executed edits finds that 23.2% repair a major error while only 2.0% introduce one. These results show that reliable repair depends as much on declining unjustified revisions as on generating good ones.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Machine Translation, LLM Self-Correction, Automatic Post-Editing, Risk-Aware Repair Decisions, Constrained Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_r1rsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper provides an view on how corrections often provides no positive effects and sometimes reduces the quality which is an well thought and aligns with real world over correction issue. Paper also provides an theortical view with bounds in which translation can be corrected without overall degradation of population. Paper introduces RCAS which takes MT and generates multiple revision using LLM, then uses an trained predictor to estimate candidates benefit or degradation relative to current translation.

Critical strengths
  1. Paper provides an theortical analysis that gains are subject to available opportunites in its candidate pool, and break down excess risk at a fixed edit rate into two sources: editing the wrong inputs and applying the wrong repair, which seems intuitive and clean.
  2. Paper connect the theortical analysis to proposed approach RCAS which formulates repair as maximizing bounded improvement subject to a population degradation budget.
Critical weaknesses
  1. Even though formulation is intuitive, but its mostly standard, the main new idea is decomposition of errors, but paper never measures their effects so it is unclear whether this decomposition actually explains why RCAS works. Optimization framework is also not explored experimentally. Finally the way ties are handeld seems to favour editing the method counts any positive gain as a reason to edit but ignores small losses when calculating another part of the decision rule.
  2. Experimental gains of the methods are very marginal. The experiments do not clearly show that RCAS makes better decisions than simpler methods. The simple QE-best baseline performs better than RCAS on COMET-22 across all five datasets. Method only performs better on the hybrid metrics it was trained on. The ablation results also suggest that a simpler signed-gain scoring method can perform as well as or better than the proposed net-value rule.
Questions for authors
  1. Can authors estimate how much of RCAS's error comes from editing the wrong inputs versus choosing the wrong repair?
  2. Does RCAS still perform well on metrics that were not used for training, such as MetricX-24, xCOMET, and BLEURT?
  3. The current rule rewards small positive gains but ignores small negative gains. Could this cause the model to edit too often, even when the changes are just noise?
  4. Can authors compare RCAS with methods that use a similar number of model calls? For example, compare it with generating nine candidate translations and selecting the best using QE while allowing KEEP, and with a version of TEaR that performs multiple rounds.
5 comments

We thank the reviewer for the careful reading and constructive feedback. We address the comments and questions below.

W1 & Q1: Errors in sentence selection and revision selection

Thank you for asking how much error comes from choosing which inputs to edit and how much comes from choosing a revision. At matched edit counts, RCAS reduced degradation relative to QE mainly by choosing better revisions for the inputs it edited.

Sentence selection and revision selection. On WMT23 En→Zh and Zh→En, RCAS and QE used the same candidate pool and edited the same number of inputs. For each method, we first held its edited inputs fixed and counted how many degradations could have been avoided by choosing another eligible revision for the same input. This gives the revision selection error. We then counted how many additional degradations could have been avoided by editing different inputs while keeping the edit count fixed. This gives the sentence selection error.

Test setMethodDegradation rateSentence selection error rateRevision selection error rate
WMT23 En→ZhQE16.58%8.39%8.19%
WMT23 En→ZhRCAS13.83%8.75%5.08%
WMT23 Zh→EnQE16.70%8.16%7.81%
WMT23 Zh→EnRCAS14.90%8.07%6.11%

Table 1. Degradation rate and its two avoidable components at matched edit counts. 

Compared with QE, RCAS reduced revision selection error by 3.11 percentage points on En→Zh and 1.71 points on Zh→En. Among RCAS’s own remaining avoidable degradations, sentence selection accounted for 63.3% on En→Zh and 56.9% on Zh→En. Revision selection accounted for the remaining 36.7% and 43.1%, respectively.

Revision selection under low and high predicted loss. For each input, we identified the eligible revision with the highest predicted benefit according to RCAS. We then compared how often RCAS and QE-best selected that revision. Both methods could choose from the same revisions or keep the initial translation. QE-best chose the option with the highest QE score.

Test setExpected lossSelected by RCASSelected by QE-best
WMT24 En→ZhLow92.6%69.3%
WMT24 En→ZhHigh43.3%86.4%
WMT24 En→DeLow97.3%67.3%
WMT24 En→DeHigh16.3%65.4%

Table 2. Selection of the highest benefit revision on WMT24. We defined the low loss groups and high loss groups using the lower and upper tertile cutoffs for predicted loss among these high benefit development examples.

W1 & Q3. Small gains, small losses, and the risk of over-editing

We thank the reviewer for asking whether our treatment of small gains and losses encourages RCAS to edit too often. A small positive gain contributes to expected benefit, but the decision to edit depends on the revision’s net score. RCAS scores each eligible revision as b̂ − λℓ̂ + c, where b̂ is expected benefit, ℓ̂ is expected excess loss, and c is a calibrated offset. It edits only when the highest net score exceeds the score of KEEP, which is zero. During training, decreases within the tolerance m retain their negative values in the gain predictor’s target (Eq. 21).

The loss penalty reduces editing when all revisions score about as well as the initial translation. To examine the reviewer’s concern, we used reference translations to identify inputs for which every eligible revision changed the utility of the initial translation by at most m in either direction (Eq. 18). This criterion identified 43 of 1,468 WMT23 En→Zh inputs and 130 of 2,521 WMT23 Zh→En inputs. We compared decisions on the same inputs with the default RCAS penalty (λ = 1) and with the penalty removed (λ = 0). Both settings used the same trained selector, candidates, and offset c, so the comparison isolates the effect of the loss penalty.

WMT23 test setInputs evaluatedNo. of inputsEdit rate without loss penalty (λ = 0)Edit rate with loss penalty (λ = 1)Edit → KEEP
En→ZhFull test set1,46899.6%72.7%26.9%
En→ZhAll revisions within ±m43100%46.5%53.5%
Zh→EnFull test set2,52199.2%85.6%13.6%
Zh→EnAll revisions within ±m130100%67.5%32.5%

Table 3. Edit rates with and without the loss penalty. 

Without the loss penalty, the edit rate was 100% in both subsets. With the penalty, it fell to 46.5% for En→Zh and 67.5% for Zh→En. In both directions, the drop was larger when all revisions scored close to the initial translation. For En→Zh, the edit rate fell by 53.5 percentage points on this subset and by 26.9 points on the full test set. For Zh→En, the corresponding drops were 32.5 and 13.6 points.

Case study. When RCAS does edit, expected excess loss can also change which revision it selects. In Table 4, the rewrite ranks first by expected benefit and fusion ranks fourth. The rewrite offers 0.00377 more expected benefit than fusion but carries 0.00596 more expected excess loss. At λ = 1, fusion therefore has the highest net score, and RCAS selects it.

ActionExpected benefitExpected excess lossNet scoreSelected
KEEP000✗
Rewrite0.172700.008110.16459✗
Retranslation 10.169970.036990.13298✗
Retranslation 20.169970.042980.12699✗
Fusion0.168930.002150.16679✓
Composite0.096080.034390.06168✗

Table 4. Scores for all eligible actions on one WMT23 En→Zh test sentence. Candidate generation and clause composition are described in Appendix C.3. The numbers distinguish the two retranslation candidates.

Choosing the loss penalty weight. The cost of a harmful edit in a given application can guide the choice of λ on development data. On the WMT23 En→Zh test set, increasing λ from 0.5 to 2 reduced mean realized excess loss from 11.81% to 2.53% of the loss cap across five seeds, while mean realized benefit fell from 0.0524 to 0.0393.

W2 & Q2. Comparison with QE-best

Thank you for raising this point. We agree that QE-best achieves higher mean COMET-22 than RCAS on all five datasets. To examine whether the comparison extends beyond the training objective, we compared two versions of RCAS, one trained on Qω and the other on COMET-22, with QE-best on the same candidate pool. We evaluated the final translations with BLEURT, XCOMET-XXL, and MetricX-24, none of which was used to train RCAS. Alongside mean quality, we report the degradation rate.

Additional metric scores and degradation rates for the two RCAS variants and QE-best. As Table 5 shows, the versions trained on Qω and COMET-22 scored 0.003574 and 0.000761 lower than QE-best on COMET-22, respectively. Their degradation rates were 11.93% and 14.79%, compared with 16.84% for QE-best. In Table 5, both versions of RCAS achieved higher mean BLEURT than QE-best. The version trained on COMET-22 also scored better than QE-best on MetricX-24 and XCOMET-XXL, although the XCOMET-XXL difference was small.

MethodQω ↑COMET-22 ↑BLEURT ↑XCOMET-XXL ↑MetricX-24 ↓Degradation rate ↓
QE-best1.2451090.8656540.7619030.9607260.79019216.84%
RCAS (trained on Qω)1.2557890.8620810.7640820.9601730.82284211.93%
RCAS (trained on COMET-22)1.2522240.8648930.7650350.9608830.77448714.79%

Table 5. Translation quality and degradation rates for RCAS and QE-best using the same candidate pool on WMT22 En→De. Throughout this response, all quality metrics are reported on their original scales. COMET-22 and Qω scores and gains are therefore not multiplied by 100 as in the paper.

We also assessed human preferences directly, since automatic metrics can rank translation systems differently from human evaluators [1]. We sampled 100 WMT23 En→Zh pairs (seed 42) for which RCAS (trained on Qω) and QE-best produced different translations. Two experts judged each pair independently, without knowing which method produced either translation. Reported percentages are adjusted to account for differences in sampling probability.

Expert judgmentPreference (%)
RCAS preferred28.6%
Tie47.8%
QE-best preferred23.6%

Table 6. Blinded expert comparison of RCAS and QE-best on 100 WMT23 En→Zh pairs with different outputs.

When the outputs differed, the share of judgments favoring RCAS was 5.0 percentage points higher than the share favoring QE-best. Counting identical outputs as ties, the estimated overall net preference for RCAS was +3.3 percentage points (95% CI [−4.4, +11.2]). The point estimate favors RCAS, but the interval includes zero, so the data do not establish a clear overall preference.

[1] Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies. ACL’24.

Reviewer_14VuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates selective self-correction in machine translation (MT): given an initial translation and several revision candidates, the task is to decide whether to apply a revision and, if so, which candidate to choose. The authors propose RCAS, a method that trains and calibrates a predictor to estimate the expected gains and degradation of each candidate relative to the original translation. It then selects the candidate that yields the highest positive net gain; otherwise, the original translation is retained. The approach is evaluated on five WMT datasets, with its effectiveness further examined through ablation studies and human evaluation.

Critical strengths

Critical Strengths:

  1. The problem is clear defined. Improving an already strong translation requires accounting for the possibility of damaging correct content. Treating the current translation as a zero-value action, defining gains relative to it, and separating candidate-pool opportunity (θ, O) from decision regret (Rw) is a clean way to think about LLM refinement of already-strong MT.
  2. The human evaluation provides useful evidence beyond automatic metrics. Among executed edits, the major-error repair rate is 23.2%, compared with a 2.0% introduction rate (Table 3). This provides evidence that the selected edits deliver meaningful semantic improvements.
  3. The paper reports its own negative aspect. It states that QE-best wins on COMET-22, that signed gain is not dominated by net-value scoring, and that the regression is a screening signal rather than a human harm rate. Such honest statement enhances the credibility to the reported results.
Critical weaknesses
  1. The added value of RCAS over QE selection is not demonstrated convincingly on the primary metric. In Table 1(b), QE-best, which needs no training achieves higher COMET-22 than RCAS on all five datasets. RCAS leads on Qω on four datasets, but Qω is the selector’s training objective, so this does not show transfer to any metric outside the optimization loop.

  2. The ablations do not show a benefit of net-value scoring over signed gain. Table 2 shows the signed-gain variant is better than RCAS on Qω, COMET, and Regression in both directions (en–zh and zh–en). As it stands, the benefit/degradation factorization adds complexity without demonstrated decision-quality gains.

  3. The budget-control formulation lacks direct empirical validation. Section 2.3 formulates the problem as maximizing bounded benefit subject to an expected-degradation budget κ and derives the Lagrangian decision rule (Eqs. (11)–(13)). However, RCAS fixes λ = 1 without specifying a target budget κ. The paper neither reports a sensitivity analysis over λ nor demonstrates how λ can be selected to satisfy a chosen budget. Consequently, the experiments do not establish whether the resulting policy satisfies L(w) ≤ κ for a prespecified κ. In other words, the experiments validate a fixed-weight scoring rule, not the constrained-optimization framework the paper presents as its contribution.

  4. Evaluation metrics are the same as the method’s training surpervision, and comparisons with published methods are incomplete. (a) Nearly all reported results use Qω or COMET-22, both of which appear in the selector’s supervision (Eq. (16)). Appendix says BLEU and chrF are also reported, but no such table appears in the paper, and chrF is also a component of Qω. Additional evaluation metrics not used for training, such as MetricX-24 and XCOMET-XXL, are needed to assess whether the reported gains extend beyond the metrics used for supervision. Evaluating both RCAS and QE-best with such metrics would also help clarify the quality trade-off raised in point 1. (b) The human evaluation only compares RCAS edits against the original translation; it cannot support a claim of superiority over any competing method. A blinded comparison of RCAS vs. QE-best vs. signed-gain would be helpful. (c) Comparisons with published related methods are missing: MT-Ladder [1] and sMBR [2].

[1] Feng, Zhaopeng, et al. "Ladder: A model-agnostic framework boosting LLM-based machine translation to the next level." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

[2] Lyu, Boxuan, et al. "Unveiling the power of source: Source-based minimum Bayes risk decoding for neural machine translation." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Questions for authors

Please address the above weakness points: E.g.,

  1. demonstrate in suitable metrics that RCAS is better than QE selection;
  2. compare with existing methods on more evaluation metrics.
6 comments

We thank the reviewer for the careful reading and constructive feedback. We address the comments and questions below.

W1 & W4 & Q1. Comparison with QE-best

Thank you for suggesting that we evaluate RCAS against QE-best using metrics outside the training objective. We compared two versions of RCAS, one trained on Qω and the other on COMET-22, with QE-best on the same candidate pool. We evaluated the final translations with BLEURT, XCOMET-XXL, and MetricX-24, none of which was used to train RCAS. Alongside mean quality, we report the degradation rate.

Additional metric scores and regression rates for the two RCAS variants and QE-best. As Table 1 shows, the versions trained on Qω and COMET-22 scored 0.003574 and 0.000761 lower than QE-best on COMET-22, respectively. Their regression rates were 11.93% and 14.79%, compared with 16.84% for QE-best. In Table 1, both versions of RCAS achieved higher mean BLEURT than QE-best. The version trained on COMET-22 also scored better on both metrics suggested by the reviewer, MetricX-24 and XCOMET-XXL, although the XCOMET-XXL difference was small.

MethodQω ↑COMET-22 ↑BLEURT ↑XCOMET-XXL ↑MetricX-24 ↓Degradation rate ↓
QE-best1.2451090.8656540.7619030.9607260.79019216.84%
RCAS (trained on Qω)1.2557890.8620810.7640820.9601730.82284211.93%
RCAS (trained on COMET-22)1.2522240.8648930.7650350.9608830.77448714.79%

Table 1. Translation quality and degradation rates for RCAS and QE-best using the same candidate pool on WMT22 En→De. Throughout this response, all quality metrics are reported on their original scales. COMET-22 and Qω scores and gains are therefore not multiplied by 100 as in the paper.

We also assessed human preferences directly, since automatic metrics can rank translation systems differently from human evaluators [1]. To assess human preferences directly, we sampled 100 WMT23 En→Zh pairs for which RCAS (trained on Qω) and QE-best produced different translations. Two experts judged each pair independently, without knowing which method produced either translation. Reported percentages are adjusted to account for differences in sampling probability.

Expert judgmentPreference (%)
RCAS preferred28.6%
Tie47.8%
QE-best preferred23.6%

Table 2. Blinded expert comparison of RCAS and QE-best on 100 WMT23 En→Zh pairs with different outputs.

When the outputs differed, the share of judgments favoring RCAS was 5.0 percentage points higher than the share favoring QE-best. Counting identical outputs as ties, the estimated overall net preference for RCAS was +3.3 percentage points (95% CI [−4.4, +11.2]). The point estimate favors RCAS, but the interval includes zero, so the data do not establish a clear overall preference.

BLEU and chrF in Appendix E. Thank you for pointing out that we mentioned BLEU and chrF in Appendix E but omitted the corresponding table. We report the results in Table 3 and will include the table in the revised appendix. RCAS achieved the highest BLEU and chrF scores in both translation directions. BLEU was not used to train RCAS, whereas chrF is a component of Qω.

MethodEn→Zh BLEU ↑En→Zh chrF ↑Zh→En BLEU ↑Zh→En chrF ↑
Initial translation0.47490.43200.29050.5514
QE-best0.44810.40570.28630.5602
RCAS0.50150.45220.29500.5654

Table 3. Corpus BLEU and chrF on WMT23. QE-best and RCAS used the same candidate pool. BLEU and chrF are reported on a 0–1 scale.

[1] Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies. ACL’24.

W2 & W4. Comparison with signed gain

We agree that signed gain is a competitive baseline. In the ablation reported in the paper, it achieved slightly higher mean Qω and COMET-22 than RCAS. The two methods share the same gain predictor. Signed gain ranks revisions by their predicted change in Qω relative to the initial translation. Since COMET-22 is a component of Qω, this helps explain why signed gain is competitive on both metrics. RCAS estimates the expected benefit and potential loss of each revision separately, then weighs them when deciding whether to keep the initial translation and, if editing, which revision to adopt.

We also compared the two methods on three metrics that were not used in training. In Table 4, RCAS scored better on MetricX-24 and had a small edge on BLEURT. Signed gain scored higher on XCOMET-XXL.

MethodMetricX-24 ↓BLEURT ↑XCOMET-XXL ↑COMET-22 ↑Qω ↑
RCAS1.9969670.7187010.8839130.8636001.104674
Signed gain2.0038910.7186770.8858300.8638011.103814

Table 4. Comparison of RCAS and signed gain on WMT23 En→Zh.

To assess human preferences directly, we sampled 62 WMT23 En→Zh pairs for which RCAS and signed gain produced different outputs. The experts assessed each pair independently, without knowing which method produced either translation. Reported percentages are adjusted to account for differences in sampling probability.

Expert judgmentPreference (%)
RCAS preferred25.7%
Tie57.8%
Signed gain preferred16.5%

Table 5. Blinded comparison of RCAS and signed gain on 62 WMT23 En→Zh pairs with different outputs.

The estimated net preference for RCAS remained positive when identical outputs were counted as ties. For pairs with different outputs, the weighted net preference for RCAS was +9.2 percentage points (25.7% − 16.5%). Across the full test set, counting identical outputs as ties, the estimated net preference was +2.7 percentage points (95% CI [−2.0, +7.4]).

W3. Sensitivity to λ and validation of the degradation budget

We thank the reviewer for pointing out that the original experiments did not directly test budget attainment. The paper fixes λ = 1, giving one unit of bounded loss the same weight as one unit of bounded benefit. We examined how realized benefit and loss change with λ, and whether a policy selected on development data meets a specified budget on the test set.

Sensitivity of benefit and loss to λ. We varied λ while holding the model, calibrator, candidate pool, and edit offset fixed. In both WMT23 directions, increasing λ reduced realized loss at the cost of some benefit. On En→Zh, for example, raising λ from 0.5 to 2 reduced mean loss from 11.81% to 2.53% of the loss cap and mean bounded benefit from 0.0524 to 0.0393.

λEn→Zh mean lossEn→Zh mean benefitZh→En mean lossZh→En mean benefit
0.511.81%0.05249.83%0.0484
18.17%0.04856.39%0.0441
22.53%0.03933.58%0.0357

Table 6. Sensitivity of realized benefit and loss to λ on WMT23.

Choice of λ for a loss budget κ. We evaluated nine values of λ on the development set and also considered a policy that keeps every initial translation. For each budget κ, we selected the policy with the highest mean bounded benefit among those whose mean development loss was at or below κ. We then applied the selected policy unchanged to the test set and checked whether its mean test loss met the same budget. Both losses are averaged over all inputs in the corresponding set, with zero loss when the initial translation is kept.

Budget κSelected λMean development lossMean development benefitMean test loss
1%40.57%0.03270.49%
3%21.51%0.03831.56%
6%15.40%0.05055.87%

Table 7. Selecting λ on development data and checking the budget on WMT23 En→Zh test data. Loss is expressed as a percentage of the loss cap.

At the 6% budget, λ = 1 gave the highest development benefit among all policies that met the constraint. Every smaller λ exceeded the budget: even λ = 0.5 incurred a development loss of 10.50%. Increasing λ to 2 reduced development loss to 1.51%, but also reduced benefit from 0.0505 to 0.0383. We therefore selected λ = 1. Its test loss was 5.87%, within the 6% budget.

Reviewer_ravHOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper formulates machine translation post-editing as a selective decision problem: whether to retain the current translation or apply one of several proposed revisions. It develops a theoretical framework that separates candidate-pool opportunity from decision regret and derives a selection rule balancing expected benefit against expected degradation. The proposed RCAS framework estimates these quantities without test-time references. Experiments, particularly fixed-candidate-pool controls and a blinded human audit, support the value of rejecting unjustified revisions to reduce regression, while revealing trade-offs between average translation quality and degradation risk.

Critical strengths
  1. A completed framework for selective repair on post-editing gives new insight into considering the risk when introducing new candidates
  2. The paper highlights the importance of candidate-pool quality by distinguishing the improvement opportunities available in the pool from the selection policy’s ability to recover them.
  3. The experiments demonstrate competitive translation quality against repair baselines, while controlled comparisons support the value of selective repair in reducing regression.
Critical weaknesses
  1. Effectiveness with stronger repair models remains unclear. The main experiments use Qwen3-4B-Instruct-2507 to generate revisions. Although the initial translations come from multiple MT systems, it remains unclear whether selective repair offers comparable benefits when stronger, state-of-the-art LLMs generate the initial translations and revisions.
  2. Cross-model generalization needs broader validation. The cross-generator experiment with Qwen2.5-7B provides useful evidence, but both generators belong to the Qwen family. Evaluations across different model families and sizes would strengthen the framework’s generality claims.
  3. Dependence on automatic evaluation metrics requires further examination. Training and calibration rely on a COMET-22–chrF composite utility, while automatic regression is measured using COMET-22. Although the human audit provides complementary evidence, would the framework retain its effectiveness when using LLM-as-a-judge evaluation, and would its selected revisions remain preferable under such evaluators?
Questions for authors

NA

3 comments

We thank the reviewer for the careful reading and constructive feedback. We address the comments and questions below.

W1. Selective repair when a stronger model revises its own translations

Thank you for raising this question. We compared three settings on the same 314 WMT23 En→Zh source segments. In the original setting, Qwen3-4B generated revisions for translations produced by the WMT systems. We then kept those initial translations fixed and replaced only the revision generator with DeepSeek-V4-Pro-0813 [1]. Finally, we kept DeepSeek as the revision generator and asked it to translate the source segments itself, so that it revised its own initial translations. We used the original RCAS selectors in all three settings without retraining or recalibration.

Initial translation sourceRevision modelInitial COMET-22RCAS gain in COMET-22 [95% CI]RCAS gain in Qω [95% CI]Best available gain in Qω
Existing translationsQwen3-4B82.21+4.41 [3.76, 5.21]+6.13 [5.17, 7.37]+11.71
Existing translationsDeepSeek82.21+5.37 [4.58, 6.34]+7.83 [6.64, 9.34]+13.68
DeepSeekDeepSeek86.52+0.21 [−0.11, 0.61]+0.25 [−0.14, 0.74]+2.33

Table 1. Effect of the initial translation and revision model on RCAS gains. All gains are relative to keeping the initial translation. The last column gives the mean gain available if a reference translation is used to select the highest scoring option under the composite metric Qω, including KEEP. Throughout this response, COMET-22 and Qω scores and gains are reported in points (×100), consistent with the paper.

RCAS achieved larger mean gains with DeepSeek revisions than with Qwen3-4B revisions on the same initial translations: 5.37 versus 4.41 COMET-22 points and 7.83 versus 6.13 Qω points.

DeepSeek’s own initial translations were stronger and left less room for improvement. They scored 86.52 on COMET-22 compared with 82.21 for the existing translations. With DeepSeek generating revisions in both cases, the best available Qω gain was 13.68 points on the original outputs and 2.33 points on its own translations. This helps explain the modest gains over keeping DeepSeek’s translations: 0.21 COMET-22 points and 0.25 Qω points, with both confidence intervals including zero.

[1] DeepSeek. DeepSeek-V4-Pro Update. DeepSeek API Docs, August 13, 2026.

W2. Generalization across revision models of different families and sizes

We thank the reviewer for pointing out that our original transfer experiment used two generators from the same family, Qwen3-4B and Qwen2.5-7B. We extended the evaluation to both directions of WMT23 and added two generators. Gemma 3 4B IT tests transfer to another family at roughly the same model size as Qwen3-4B. DeepSeek-V4-Pro-0813 tests transfer to a substantially larger model from a third family.

Within each translation direction, we kept the initial translations, revision prompts, and candidate generation budget fixed. Only the revision generator changed. We applied the three submitted RCAS selectors for that direction, trained on Qwen3-4B candidates, directly to each new candidate pool without retraining or recalibration.

Revision modelen→zh COMET-22 gainen→zh gain in Qωzh→en COMET-22 gainzh→en gain in Qω
Qwen3-4B-Instruct-2507+4.38 [3.79, 5.12]+5.94 [5.13, 7.00]+2.04 [1.40, 3.38]+3.61 [2.95, 4.89]
Qwen2.5-7B-Instruct+3.71 [3.12, 4.46]+5.21 [4.46, 6.12]+1.55 [0.99, 2.95]+2.55 [1.84, 4.09]
Gemma 3 4B IT+3.57 [3.00, 4.32]+2.79 [1.94, 3.77]+1.64 [1.03, 3.10]+1.63 [0.84, 3.30]
DeepSeek-V4-Pro-0813+5.29 [4.55, 6.18]+7.48 [6.38, 8.81]+2.77 [1.95, 4.16]+5.09 [4.26, 6.68]

Table 2. Mean quality gains over keeping the initial translation (KEEP) on WMT23. 

With revisions generated by Gemma or DeepSeek, RCAS improved mean COMET-22 and Qω over keeping the initial translations in both directions. With Gemma revisions, RCAS gained 3.57 COMET-22 points on En→Zh and 1.64 on Zh→En over KEEP. With DeepSeek revisions, the corresponding gains were 5.29 and 2.77 points. All reported confidence intervals for the COMET-22 and Qω gains in these four cross-family settings were above zero.

W3. LLM judgments of RCAS outputs and initial translations

Thank you for suggesting an evaluation from a different perspective. We asked DeepSeek-V4-Pro to compare the translations selected by the submitted RCAS system with the initial translations. On WMT23 En→Zh, the judge preferred the revision in 54.2% of cases where RCAS made an edit and preferred the initial translation in 5.1%. For Zh→En, the corresponding figures were 53.1% and 6.0%.

We used the frozen outputs from all five seeds in the original submission, covering all 1,468 En→Zh and 2,521 Zh→En test records. The judge saw only the source and two anonymous translations. We asked it to prioritize accuracy and completeness, consider fluency when both translations conveyed the source equally well, and allow ties. We presented each pair of distinct translations in both orders in separate calls. A translation counted as preferred only if the judge chose it in both orders.

Decisions evaluatedDirectionRCAS preferredTie or inconsistent judgmentsInitial translation preferredNet preference for RCAS (pp), 95% CI
Edits onlyEn→Zh54.2%40.7%5.1%+49.1 [43.9, 55.2]
Edits onlyZh→En53.1%40.9%6.0%+47.1 [34.4, 59.9]
All decisions, including KEEPEn→Zh39.4%56.9%3.7%+35.7 [31.9, 40.4]
All decisions, including KEEPZh→En45.4%49.5%5.1%+40.3 [28.2, 54.6]

Table 3. LLM preference between RCAS outputs and initial translations on WMT23. The middle category includes ties, identical outputs, and judgments that differ between the two presentation orders. Net preference is the percentage favoring RCAS minus that favoring the initial translation.

The net preference for RCAS remained positive when KEEP cases were included. Across the full test sets, the judge favored the RCAS output over the initial translation by a net margin of 35.7 percentage points for En→Zh and 40.3 points for Zh→En.

Comments on the paper

1 comment

Thanks for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.