- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes L-GRPO, a critic-free method for using span-level LLM-judge feedback in GRPO. Standard GRPO collapses token rewards into a single rollout return and broadcasts one group-relative advantage to all tokens. L-GRPO keeps this rollout-level advantage but adds a mean-zero token-level redistribution term derived from localized judge rewards, so local errors receive different credit without changing the rollout’s average advantage. Experiments on document-level machine translation and long-form factuality show consistent improvements over GRPO, PPO, SDPO, and a token-only estimator.
Critical strengths
- The paper provides strong empirical evidence across two different generation tasks. L-GRPO improves over GRPO on ten-language document-level MT under both an LLM-based error metric and METRICX-24, and also improves factuality across four benchmarks.
- The token-only analysis is useful and empirically supported. The paper shows analytically that sparse token-only rewards produce vanishing advantages for most tokens and very large advantages for the few scored tokens.
- The implementation pays attention to practical reward-hacking issues.The character-length scaling for MT omissions and the auxiliary relevance reward for factuality address concrete failure modes rather than assuming that localized rewards are automatically well behaved.
Critical weaknesses
- The paper does not directly isolate whether correct span localization is the source of the improvement. L-GRPO outperforms GRPO when the judge-derived localized term is added, but there is no control where the same localized reward magnitudes are assigned to shuffled or randomly selected token spans.
- The localized judge signal is considerably noisier than the framing may suggest, especially for MT. On the human-annotated WMT error-span benchmark, the training judge Gemma 4-31B achieves only 0.239 span F1 and 0.295 segment-level Spearman correlation with human judgments; even GPT-5.5 reaches only 0.221 span F1.
- The method performs localized output supervision rather than full causal credit assignment. The judge marks erroneous or supported spans in the final answer, but earlier reasoning tokens that caused those spans are not localized.
Questions for authors
- Can the authors add a control that preserves the same number and magnitude of localized rewards but randomly shuffles their token positions within each response?
- How sensitive are RL gains to judge localization quality? Can the authors correlate judge localization accuracy with downstream improvements across languages, or train with different judges of varying localization quality?