acceptodds
Under review as a conference paper at ICLR 2027

Localized GRPO: Utilizing Dense Reward Signal From LLM-Judge Error Localization

Abstract

Reinforcement learning for LLMs is shifting from PPO toward critic-free, group-relative methods such as GRPO, which remove the value network that PPO uses to form token-level advantages. Vanilla GRPO instead forms one group-relative advantage per rollout and broadcasts it to every token, so the algorithm has no explicit mechanism for assigning reward credit within a response. On verifiable reasoning tasks, recent methods address this limitation using intrinsic proxies such as token entropy, likelihoods, or Monte-Carlo rollout estimates. We study a complementary setting: generation tasks where external LLM judges can identify localized quality signals in a response, such as error spans in machine translation (MT) and supported or hallucinated spans in long-form factuality. We convert these explicit fine-grained span judgments into per-token rewards, and introduce L-GRPO, a localized GRPO advantage estimator that uses those rewards while preserving GRPO's sequence-level group-relative signal. On document-level MT across ten language pairs, L-GRPO reduces LLM-judged error density by 7.2% relative to GRPO. On long-form factuality, it improves factual precision across four benchmarks by 9.6% relative. Both gains are statistically significant. L-GRPO also outperforms two strong dense credit baselines on both tasks: PPO+GAE, which uses a learned critic, and SDPO, which adds feedback-conditioned self-distillation.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, GRPO, Token-level rewards, Dense Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LEZWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes L-GRPO, a critic-free method for using span-level LLM-judge feedback in GRPO. Standard GRPO collapses token rewards into a single rollout return and broadcasts one group-relative advantage to all tokens. L-GRPO keeps this rollout-level advantage but adds a mean-zero token-level redistribution term derived from localized judge rewards, so local errors receive different credit without changing the rollout’s average advantage. Experiments on document-level machine translation and long-form factuality show consistent improvements over GRPO, PPO, SDPO, and a token-only estimator.

Critical strengths
  1. The paper provides strong empirical evidence across two different generation tasks. L-GRPO improves over GRPO on ten-language document-level MT under both an LLM-based error metric and METRICX-24, and also improves factuality across four benchmarks.
  2. The token-only analysis is useful and empirically supported. The paper shows analytically that sparse token-only rewards produce vanishing advantages for most tokens and very large advantages for the few scored tokens.
  3. The implementation pays attention to practical reward-hacking issues.The character-length scaling for MT omissions and the auxiliary relevance reward for factuality address concrete failure modes rather than assuming that localized rewards are automatically well behaved.
Critical weaknesses
  1. The paper does not directly isolate whether correct span localization is the source of the improvement. L-GRPO outperforms GRPO when the judge-derived localized term is added, but there is no control where the same localized reward magnitudes are assigned to shuffled or randomly selected token spans.
  2. The localized judge signal is considerably noisier than the framing may suggest, especially for MT. On the human-annotated WMT error-span benchmark, the training judge Gemma 4-31B achieves only 0.239 span F1 and 0.295 segment-level Spearman correlation with human judgments; even GPT-5.5 reaches only 0.221 span F1.
  3. The method performs localized output supervision rather than full causal credit assignment. The judge marks erroneous or supported spans in the final answer, but earlier reasoning tokens that caused those spans are not localized.
Questions for authors
  1. Can the authors add a control that preserves the same number and magnitude of localized rewards but randomly shuffles their token positions within each response?
  2. How sensitive are RL gains to judge localization quality? Can the authors correlate judge localization accuracy with downstream improvements across languages, or train with different judges of varying localization quality?
Reviewer_Qq3uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how a critic-free group-relative policy optimizer can use judges that identify particular erroneous or supported spans, rather than reducing all feedback to a single response score. Its L-GRPO estimator adds a normalized, within-response centered token-reward term to standard GRPO's broadcast advantage. The intended construction preserves each response's mean numerical advantage while changing the relative weights of its graded tokens; Appendix C.2 specifies the task-dependent graded-token mask. Rewards come from character-weighted MQM error spans for translation and search-supported, four-way claim judgments for long-form factuality. Unmatched spans receive a response-wide fallback reward, and the factuality objective also includes an answer-level relevance term.

The experiments fine-tune Qwen3-8B on document translation into ten target languages and long-form factuality, comparing against GRPO, PPO, SDPO, and a token-only estimator. Training and evaluation use different judges. Relative to GRPO, reported translation error density decreases from 0.323 to 0.299, with a complementary MetricX improvement from 6.18 to 5.88 (Table 2). Macro-averaged factual precision increases from 66.2% to 72.5%, alongside a modest increase in supported claims (Table 3). The appendices provide paired significance tests, checkpoint trajectories, localization-weight sensitivity, and baseline diagnostics. The main contribution is a simple practical way to exploit structured judge feedback in GRPO; the analysis establishes an advantage-mean identity and a sparse-reward pathology for a particular token-only estimator, rather than a general policy-improvement guarantee.

Critical strengths
  1. A simple, useful method with a well-controlled primary comparison. L-GRPO reuses the judge feedback already needed to score a rollout and introduces neither a learned critic nor extra prefix-conditioned generations. Its broadcast-plus-redistribution construction is easy to integrate with GRPO, and the graded-set definition in Appendix C.2 preserves the intended mean advantage. GRPO and L-GRPO share the main experimental settings apart from the localization weight (Appendix C.3, Table 10). The improvements therefore concern the proposed use of feedback rather than changing the judge, policy initialization, or training data.

  2. Substantial empirical corroboration across tasks and complementary analyses. Table 2 shows favorable point estimates in every MT direction under both the independent judge and MetricX, with significant aggregate gains. Table 3 reports factuality gains accompanied by more supported claims, which helps address a simple precision-by-abstention explanation. Appendices A.4 and B.5 use paired tests and appropriate source clustering for the multi-parallel MT data; Appendices A.5 and B.6 examine metric coverage and the localization-weight trade-off. The paper also explicitly reports that Biography accounts for roughly 75% of the macro-averaged factuality gain and gives the smaller, still positive micro-averaged improvement. This transparency makes the main empirical result more convincing.

Critical weaknesses
  1. The contribution of correct localization is not isolated from the contribution of adding a normalized token-reweighting term. Section 5 concludes that the judge's span locations provide additional optimization signal, but the comparisons are scalar GRPO, L-GRPO, a token-only estimator, PPO, and SDPO. None preserves L-GRPO's reward values, graded-token mask, local-term variance, and response-level advantage while disrupting the association between each reward and its actual span. Thus the results establish that the complete L-GRPO recipe helps, but less directly establish how much depends on the semantic correctness of localization. The failure of the specific token-only estimator also does not establish that retaining the broadcast is necessary for every critic-free token estimator. A within-response permutation or matched-length random-location control would test the central mechanism with minimal changes. The text should also distinguish preservation of the arithmetic mean in Eq. 8 from preservation of the policy-gradient update: token gradients and clipping differ across positions, so the former does not imply the latter. This is an attribution and claim-scope concern, not evidence that the practical estimator is invalid.

  2. The judge studies leave uncertainty about the end-to-end factuality improvement, and the reported uncertainty does not characterize independent training variation. Different training/evaluation models are a sensible safeguard, and MetricX independently corroborates MT. Nevertheless, MT judge agreement is modest (Appendix A.3, Table 4: span F1 approximately 0.22–0.24 for the selected judges and segment correlations around 0.29). More importantly, the factuality agreement study supplies gold human-edited claims, so it explicitly does not validate unconstrained claim extraction (Appendix B.3, p. 19). The main factuality metrics depend on that extractor and its retrieved evidence, and there is no reported human comparison of the actual GRPO and L-GRPO final responses. A blinded audit of paired responses, including missed claims and answer coverage, would help determine whether the measured delta reflects factual quality rather than changes in what is extractable or searchable. Independent training-seed counts and between-run variation are not clearly specified; the paired bootstrap/randomization intervals characterize held-out evaluation uncertainty for the produced checkpoints, not training reproducibility. Reporting this distinction and, where feasible, repeated GRPO/L-GRPO runs would strengthen confidence, particularly in the smaller benchmark-level improvements and the sample-efficiency claim.

Questions for authors
  1. With the broadcast, graded-token set, reward magnitudes, and localization weight held fixed, how does L-GRPO compare with within-response shuffled rewards or matched-length random span assignments? This would directly affect my assessment of the claim that the judge's actual locations drive the improvement.

  2. Is there any blinded human assessment, or an end-to-end extraction-and-verification audit, of paired final GRPO/L-GRPO factuality outputs? In particular, does the reported gain persist when omitted claims and response coverage are checked rather than assuming the automatic extractor identifies all relevant factual content?

  3. How many independent training seeds underlie each main comparison and Figure 2? If only one, can the authors distinguish evaluation-set confidence intervals from variability across training runs, and report any existing repeated-run evidence for the main GRPO/L-GRPO difference?

Reviewer_pg3cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Localized GRPO (L-GRPO), a critic-free method that incorporates span-level feedback from LLM judges into GRPO. It converts translation errors or factuality judgments into token-level rewards and augments the sequence-level advantage with a mean-zero local term, preserving each response’s mean advantage. The paper also presents task-specific reward construction procedures and analyzes the instability of a token-only estimator. Experiments with Qwen3-8B on document-level translation across ten language pairs and long-form factuality across four benchmarks report a 7.2% relative reduction in translation error density and a 9.6% relative improvement in factual precision over GRPO, while outperforming the evaluated PPO and SDPO-based baselines. Additional analyses examine training stability, sensitivity to the localization weight, and rollout sample efficiency.

Critical strengths
  1. L-GRPO augments GRPO's sequence-level advantage with a token-level redistribution term derived from span-level judgments, while preserving the overall rollout-level signal. This is a meaningful contribution because it enables finer-grained credit assignment without requiring a learned critic or additional prefix-conditioned rollouts, providing a straightforward extension to existing GRPO pipelines when structured judge feedback is available.

  2. L-GRPO improves over GRPO on both document-level translation and long-form factuality, with statistically significant aggregate gains. Evaluation uses judges different from the training judge, and translation improvements are also supported by MetricX. These complementary evaluations provide stronger evidence that L-GRPO's improvements are not merely artifacts of optimizing against the training judge, supporting the effectiveness of the proposed method across different generation tasks.

Critical weaknesses
  1. The paper attributes the improvements of L-GRPO to more precise token-level credit assignment based on LLM-judge error localization. However, the experiments do not sufficiently isolate the contribution of accurate reward positions from the general effect of introducing non-uniform token-level advantages. A simple but important control would be to randomly shuffle the localized rewards within each rollout while preserving their values and overall distribution. Without such a comparison, it remains unclear whether the performance gains primarily arise from meaningful error localization or from the altered advantage distribution itself.

  2. All main experiments are conducted using Qwen3-8B, without evaluating the proposed method on models of different sizes or architectures. Although the paper considers two distinct generation tasks, this does not establish whether L-GRPO generalizes across different model capacities and families. Since the effectiveness of token-level advantage estimation may depend on model capacity, initial policy quality, and training dynamics, the observed improvements on a single backbone may not transfer to other settings. Additional experiments across model scales or families would be necessary to substantiate the broader applicability of L-GRPO as a general-purpose improvement to GRPO.

Questions for authors
  1. Could the authors provide a controlled experiment in which the localized rewards are randomly shuffled among eligible tokens within each rollout, while preserving their values and overall distribution? Such a comparison would help clarify whether the improvements primarily stem from accurate error localization or from introducing non-uniform token-level advantages.

  2. Have the authors evaluated L-GRPO on models of different sizes or families beyond Qwen3-8B? If not, could the authors provide additional evidence or justification for why the proposed advantage estimator is expected to generalize across different model capacities and architectures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.