- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes DocGRPO, a reinforcement learning framework designed to optimize document-level machine translation by dynamically balancing source-content adequacy and discourse fluency. The authors identify a non-monotonic trend where coverage-based adequacy and fluency often conflict, particularly as document length increases. To address this, paper introduce a fused reward with weighting mechanism and a dual relative advantage strategy that utilizes both policy-generated candidates and human references as anchors. Experiments demonstrate that DocGRPO outperforms other baselines in several machine translation metrics, with human evaluation confirming higher translation quality and a significant reduction in over- and under-translation errors.
Critical strengths
-
The paper introduces a novel stage- and length-aware adaptive weighting mechanism that effectively balances the inherent tension between adequacy and fluency in document-level machine translation.
-
The proposed reward formulation demonstrates a high correlation with human direct assessment scores, confirming that the model’s optimization objectives are effectively aligned with perceived translation quality and discourse naturalness
Critical weaknesses
-
The manuscript lacks technical details regarding the word alignment process used to compute the coverage score. Please clarify whether off-the-shelf tools (e.g., FastAlign, GIZA++) or other mapping skills were employed.
-
The evaluation is restricted to a single domain (News Commentary), which limits the claims regarding the model's robustness. Providing additional experiments on other evaluation datasets is necessary to demonstrate the generalizability of the proposed framework.
-
The description of the human evaluation process is insufficient. Please specify the number of annotators per sample, the inter-annotator agreement metrics, and the aggregation strategy used to derive final quality scores from multiple human judgments.
-
Utilizing large-scale models like Qwen3.6-27B and mBERT during the training to calcualte reward incurs significant computational and memory overhead. The authors should provide a throughput analysis or an ablation study on the reward model size to justify this design, especially given the potential memory bottlenecks during the training.
-
While the paper combines group-relative and reference-relative advantages, it is unclear how much each component contributes to the overall policy improvement. The authors should provide an ablation study that evaluates the model performance when using only one of these advantages, to justify the necessity of the dual-advantage formulation.
Questions for authors
-
Are there any existing studies that have applied the GRPO framework specifically to machine translation tasks? It would be better to include among baselines.
-
To better establish the efficacy of the proposed rewards, could you compare DocGRPO against baselines optimized directly with automatic metrics like BLEU or COMET? It would be insightful to see if the performance gains are uniquely tied to your dual-reward design.
-
Could you provide an analysis of the training dynamics, such as reward convergence or policy stability throughout the training process? Insights into how the adaptive weighting schedule impacts these dynamics would greatly strengthen the paper's contribution.
5 comments
Response to Reviewer Wkpq: Part 2/3
W3:Clarification of Human Evaluation.
We thank the reviewer for highlighting the insufficient description of our human evaluation procedure.
Each translated document was independently evaluated by two bilingual annotators with professional translation experience. Both annotators evaluated all document translations and assigned overall Direct Assessment (DA) scores on a 0–100 scale.
We assessed inter-annotator agreement using ICC(2,1) (two-way random-effects, absolute-agreement, single-measure ICC), obtaining values of 0.89 for Zh→En and 0.92 for De→En, indicating good and excellent agreement, respectively.
For score aggregation, the final DA score of each document was computed as the arithmetic mean of the two independent ratings. System-level DA scores were then obtained by averaging the document-level scores across the evaluation set. For under-translation and over-translation errors, disagreements were resolved through adjudication, and the final error counts were averaged across documents.
We have incorporated these details into Appendix C of the revised manuscript to improve transparency and reproducibility.
W4: Computational Overhead and Flexibility of Fluency Reward Estimation.
We thank the reviewer for highlighting the computational and memory overhead associated with reward estimation. We agree that using a 27B model introduces additional training costs, but would like to emphasize that the 27B scorer is an optional auxiliary component rather than an indispensable part of DocGRPO.
Our fluency reward primarily focuses on sentence-level discourse coherence, estimated by mBERT with a weight of 0.7. The 27B model provides a complementary token-level fluency signal with a smaller weight of 0.3. Given the importance of cross-sentence coherence in DocMT, we intentionally assign greater emphasis to the former.
Importantly, the 27B scorer can be entirely removed when computational efficiency is prioritized, without modifying the core optimization framework. To verify this, we conducted an additional ablation using only the mBERT-based fluency reward:
| Fluency Reward | Zh→En | De→En | Es→En | Average |
|---|---|---|---|---|
| mBERT Only | 32.1/85.4/50.1 | 45.2/88.0/58.8 | 51.4/87.7/63.0 | 42.9/87.0/57.3 |
| mBERT + 27B (Full) | 32.4/85.7/51.2 | 45.1/88.3/60.1 | 51.7/87.9/64.1 | 43.1/87.3/58.5 |
Results are reported as BLEU / COMET / BlonDe (↑).
Removing the 27B scorer results in only modest average decreases of 0.2 BLEU and 0.3 COMET, although BlonDe decreases by 1.2 points. These results demonstrate that the proposed optimization framework remains effective without the large fluency scorer, offering a practical option to trade some discourse-sensitive performance for lower training resource requirements.
Furthermore, both mBERT and the 27B model are frozen reward estimators used exclusively during offline training. Neither is required during inference, meaning that their computational and memory overhead does not affect the deployed translation model.
W5: Contribution of Group-relative and Reference-relative Advantages.
We thank the reviewer for this insightful comment. To address this concern, we conducted additional controlled ablations on Qwen3-4B across three translation directions (Zh→En, De→En, and Es→En), comparing the standard group-relative advantage, its scale-matched variant, and the full dual-advantage formulation:
| Method | Zh→En | De→En | Es→En | Average |
|---|---|---|---|---|
| Group Only() | 30.3/82.1/48.8 | 42.8/86.8/58.3 | 47.1/85.2/61.3 | 40.1/84.7/56.1 |
| GRPO (2× Scale, ) | 31.0/82.9/49.5 | 43.7/87.2/58.8 | 47.9/85.8/62.4 | 40.9/85.3/56.9 |
| DocGRPO () | 32.4/85.7/51.2 | 45.1/88.3/60.1 | 51.7/87.9/64.1 | 43.1/87.3/58.5 |
Results are reported as BLEU / COMET / BlonDe (↑).
Compared with Group Only, the full dual-advantage formulation improves average BLEU, COMET, and BlonDe by 3.0, 2.6, and 2.4 points, respectively. Furthermore, even after matching the advantage scale, DocGRPO achieves additional improvements of 2.2 BLEU, 2.0 COMET, and 1.6 BlonDe.
These results indicate that introducing reference-dependent calibration provides empirical benefits beyond simply increasing the magnitude of group-relative advantages. The group-relative component provides within-group discrimination, while the reference-relative component introduces a reference-dependent group offset into policy optimization.
Response to Reviewer Wkpq: Part 3/3
Q1&2: GRPO-based MT Methods and Metric-based Reward Baselines.
We thank the reviewer for these valuable suggestions. Recent studies have explored applying GRPO to machine translation, such as MT-R1-Zero (Feng et al., 2025), which employs mixed reward signals for translation optimization, and GRRM (Yang et al., 2026), which investigates group-relative reward modeling for MT.
However, directly optimizing automatic quality metrics may not always align with human-perceived translation quality. For example, MO-GRPO (Ichihara et al., 2026) identifies reward hacking risks in multi-objective GRPO, while GRRM (Yang et al., 2026) highlights the limitations of pointwise quality metrics for group-relative optimization.
Motivated by these concerns, DocGRPO does not directly optimize BLEU or COMET. Instead, it explicitly coordinates source-content coverage and discourse fluency to improve document-level translation quality.
To validate this design, we conducted additional experiments using Qwen3-8B on Zh→En, comparing GRPO variants directly optimized with BLEU and COMET against DocGRPO. Given the potential evaluation bias associated with optimizing and evaluating using the same metric, we additionally report independent human Direct Assessment (DA) scores.
| Method | BLEU ↑ | COMET ↑ | Human DA ↑ |
|---|---|---|---|
| GRPO-BLEU | 37.8 | 86.1 | 80.2 |
| GRPO-COMET | 34.8 | 90.5 | 81.4 |
| DocGRPO | 35.2 | 89.1 | 85.1 |
As shown above, GRPO-BLEU and GRPO-COMET achieve the highest scores on their respective optimization metrics. However, these improvements do not necessarily translate into better human-perceived translation quality. DocGRPO achieves the highest human DA score (85.1), outperforming GRPO-BLEU and GRPO-COMET by 4.9 and 3.7 points, respectively.
These results reveal a discrepancy between optimizing individual automatic metrics and improving human-perceived translation quality, consistent with concerns about metric overoptimization. They further support our motivation for explicitly coordinating source-content preservation and discourse fluency rather than maximizing a single automatic quality metric.
*Q3: Training Dynamics Analysis
We thank the reviewer for this insightful suggestion. To investigate this process, we analyze two aspects of training dynamics: (1) the evolution of coverage and fluency rewards, (2) policy stability measured by KL divergence and clipping statistics. These analyses help clarify whether the proposed weighting strategy effectively coordinates the two optimization objectives while maintaining stable policy updates. We have included the training dynamics and further discussion in Appendix F of the revised manuscript.
References
- Feng, Z., et al. (2025). MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning. arXiv:2504.10160.
- Ichihara, Y., et al. (2026). MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems. Transactions of the Association for Computational Linguistics, 14, 1867–1882.
- Yang, S., et al. (2026). GRRM: Group Relative Reward Modeling for Machine Translation. Proceedings of EMNLP 2026, 51934–51956.
Response to Reviewer Wkpq: Part 1/3
We really appreciate you for your precious time in reviewing our paper and providing helpful comments. We have carefully studied all the comments and tried our best to address every one of them. Below, we address each concern and question point by point:
W1:Clarification of Word Alignment Implementation.
We thank the reviewer for pointing out the missing implementation details. We employ a two-level alignment procedure. Specifically, (1) Segment-level Alignment: We first adopt the dynamic programming algorithm from Align-then-Slide (Guo et al., 2025) to establish source–target segment correspondences, accommodating mismatched sentence boundaries in document-level translation. (2) Word-level Alignment: We then employ the publicly available awesome-align toolkit (Dou and Neubig, 2021) to extract word-level alignment pairs within each aligned segment pair. The resulting alignments are aggregated into the document-level alignment set Mi. We have add these implementation details and the corresponding references to Section 3.2 in the revised manuscript.
W2: Cross-domain Generalization.
We thank the reviewer for this valuable comment. To address this concern, we conducted additional experiments on two out-of-domain datasets, TED Talks (spoken presentations) and OpenSubtitles (movie subtitles), using both Qwen3-4B and Qwen3-8B. Importantly, all models were trained exclusively on News Commentary and directly evaluated on these datasets without additional domain-specific fine-tuning:
Results are reported as d-BLEU / COMET / BlonDe (↑).
The results demonstrate that DocGRPO consistently outperforms all baselines across both out-of-domain datasets and model scales, supporting its effectiveness beyond the News Commentary domain.
More interestingly, DocGRPO achieves substantially larger improvements in discourse-sensitive translation quality on TED than on News Commentary. For example, with Qwen3-8B, the BlonDe improvements over DeMPT increase from +0.9/+0.7 on News Commentary to +3.2/+2.8 on TED for Zh→En and De→En, respectively.
We hypothesize that this stronger benefit is related to the discourse characteristics of spoken-language translation. Compared with formal written news, TED talks and conversational subtitles can involve more context-dependent expressions, implicit references, and ellipsis, where optimizing explicit word-level coverage alone may be insufficient. Our joint coverage–fluency optimization is designed to better accommodate such phenomena.
These findings not only demonstrate the cross-domain transferability of DocGRPO but also provide additional empirical support for the motivation behind coordinating adequacy and discourse fluency.