acceptodds
Under review as a conference paper at ICLR 2027

Improving the Credit Assignment of Group Relative Policy Optimization via λ-Return Advantage Estimation for LLM Reasoning

21 views

Abstract

reinforcement learning from verifiable rewards (RLVR) in large language model (LLM) reasoning. Yet GRPO assigns one outcome-level, group normalized advantage to every token of a response: it cannot tell which tokens actually helped the model reach a correct answer, it pays no price for intermediate verbosity, and its Monte-Carlo estimator carries high variance when responses are long. We introduce λ-GRPO, a practical recipe that repairs GRPO’s credit assignment while keeping its group-relative stabil ity. A lightweight critic network, attached to the policy trunk, is trained on λ-return targets; token-level λ-return advantages are then computed from temporal-difference errors, standardized across the token batch, and com bined with an asymmetric clip-higher surrogate, an explicit per-token length penalty, a KL regularizer to the reference policy, and a λ-annealing schedule that shifts from outcome-level credit early in training to fine-grained token level credit as the critic matures. On GSM8K and MATH with Qwen2.5- 1.5B/7B, λ-GRPO improves accuracy over GRPO, PPO, RLOO, VAPO and DAPO at equal compute, reaching 91.2% GSM8K / 65.8% MATH on the 7B model while shortening the average response from 502 to 327 to kens, and 78.8% / 48.4% on the 1.5B model. Component ablations and bias–variance analysis show that each ingredient contributes consistently, and that value-based token-level credit, not verbosity, is what drives the gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning from Verifiable Rewards, Group Relative Policy Optimization, Credit Assignment, Large Language Models, Mathematical Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_D3tZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes λ\lambda-GRPO, an actor-critic variant of GRPO for RLVR that combines group-normalized terminal rewards with a per-token length penalty and a lightweight critic trained on λ\lambda-return targets. Token-level GAE advantages are then standardized across the token batch and used in a clipped policy objective with asymmetric upper clipping and reference-policy KL regularization. Experiments on GSM8K and MATH with Qwen2.5-1.5B and 7B report improved greedy accuracy and shorter responses relative to PPO, RLOO, GRPO, VAPO, and DAPO.

Critical strengths

The paper addresses an important limitation of outcome-level RLVR: whether finer-grained credit assignment can improve accuracy without encouraging unnecessarily long responses. The proposed method achieves the highest accuracy among the compared methods on both GSM8K and MATH at both model sizes, while generating shorter GSM8K responses. These results suggest that the method could be practically useful.

The proposed method combines a lightweight critic, GAE, token-level advantage standardization, asymmetric clipping, and an annealed λ\lambda schedule. The paper provides a clear training procedure, component ablations, and additional analyses of the λ\lambda schedule and response length. These experiments help explain the behavior of the overall method, although they do not fully isolate the contribution of the learned critic.

Critical weaknesses
  1. The contribution of learned token-level credit assignment is not sufficiently isolated, and the incremental novelty remains unclear. Although the paper provides useful ablations, the contributions of individual components are not fully isolated. The full method adds at least four consequential modifications to GRPO: the learned critic and λ\lambda-returns, a per-token penalty, token-batch advantage normalization, and an asymmetric clip-higher objective. It lacks two important controls: λ\lambda-GRPO without the length penalty (η=0\eta=0), and a critic-free GRPO/DAPO variant with the same penalty and otherwise matched optimization settings. As a result, the specific contribution of the learned critic cannot be separated from reward shaping and other changes to the training recipe. Another concern is the limited novelty compared with closely related work. GRPO-λ\lambda [1] already explores critic-free λ\lambda-based credit assignment, while VAPO [2] combines a learned value model with GAE variants. Although λ\lambda-GRPO uses a different training recipe, its incremental contribution beyond these approaches is not yet sufficiently established.

  2. The theoretical analysis contains errors that weaken the justification for the proposed method. Proposition 3 contains an Rmax⁡/(1−γ)R_{\max}/(1-\gamma) term, which is undefined under the experimental setting γ=1\gamma=1. Its discussion of the λ=1\lambda=1 case is also inconsistent with the displayed bound. In addition, the variance recursion omits the covariance term 2γλCov⁡(δt,At+1λ)2\gamma\lambda\operatorname{Cov}(\delta_t,A^\lambda_{t+1}), so the claimed monotonic increase in variance with λ\lambda is not established without further assumptions. These issues weaken the bias-variance justification for the proposed λ\lambda-annealing schedule.

Questions for authors
  1. Could the authors report (i) λ\lambda-GRPO with η=0\eta=0 and (ii) a critic-free GRPO/DAPO control with the same length penalty and otherwise matched clipping, KL treatment, and token-level loss?

  2. What is the specific contribution beyond the critic-free λ\lambda-based method GRPO-λ\lambda [1] and the value-model/GAE design already explored in VAPO [2]? A budget-matched comparison with GRPO-λ\lambda would help determine whether the proposed method provides additional benefits under comparable computational resources.

  3. Can the authors provide a finite-horizon version of Proposition 3 valid for γ=1\gamma=1, clarify the λ=1\lambda=1 case, and account for the missing covariance term in the variance analysis?

  4. Can the authors provide wall-clock throughput, peak memory, retained rollout counts after group filtering, and total FLOPs or a comparable compute accounting? This would clarify the equal-compute claim.

[1] Parthasarathi, Prasanna et al. “GRPO-λ: Credit Assignment improves LLM Reasoning.”, ArXiv abs/2510.00194 (2025)

[2] Yue, Yu et al. “VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks.” ArXiv abs/2504.05118 (2025)

Reviewer_zCtDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes λ-GRPO, a modification of GRPO that adds a small critic head trained with λ-return targets, converts GRPO’s group-relative outcome advantage into a token-level λ-return advantage, standardizes advantages across the token batch, adds a per-token length penalty, uses asymmetric clip-higher and a KL regularizer, and anneals λ from 0.95 to 0.5. The authors evaluate on GSM8K and MATH with Qwen2.5-1.5B and Qwen2.5-7B against PPO, RLOO, GRPO, VAPO, and DAPO, and report higher accuracy together with shorter average responses. They also provide component ablations, a λ sweep, and a length-bucket analysis.

Critical strengths
  1. The paper attempts to combine GRPO’s group-relative stability with PPO/GAE-style token-level credit assignment. The problem motivation is reasonable, and the overall procedure is easy to follow.
  2. On Qwen2.5-7B, λ-GRPO is reported to reach 91.2%/65.8% on GSM8K/MATH, compared with 89.5%/63.5% for DAPO, while using shorter average responses. Similar trends are reported at the 1.5B scale. If the comparison is fair and robust, this is a valuable empirical result.
Critical weaknesses
  1. The verbosity claim is not isolated: there is no η=0 ablation, so shorter responses may come from the length penalty rather than token-level credit.
  2. Equal compute is not established: matching steps ignores the critic’s extra forward/backward cost.
  3. Results are single-seed, and the gains over DAPO are small.
  4. Task and model coverage is narrow: only GSM8K/MATH and Qwen2.5-1.5B/7B.
Questions for authors
  1. Please add an η=0 ablation. Without the length penalty, does λ-GRPO still produce substantially shorter responses than GRPO/DAPO? Please also report sensitivity to η.
  2. Please report FLOPs, wall-clock time, or critic overhead for all methods, and compare methods under matched total compute rather than only matched optimization steps. Does λ-GRPO remain better under matched GPU time?
  3. Please report multi-seed results and confidence intervals, at least for Qwen2.5-1.5B and ideally for the key 7B comparisons. Are the reported gains over DAPO/GRPO larger than evaluation and training variance?
  4. In the λ=1 ablation, is the critic used as a baseline? The current description “critic effectively unused” is inconsistent with the estimator when λ=1 and the critic is nonzero. Please clarify exactly what is removed in this ablation.
Reviewer_AF1rOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

GRPO-style post-training usually assigns one reward to a complete trajectory, so all tokens within the same response share the same advantage. It is hard to distinguish which tokens are more important than others within the same response. To deal with this, the authors propose λ-GRPO by adding a small critic head to the policy model. They use λ-return and TD errors to compute token-level advantages. Besides, the authors introduce λ annealing and use a larger λ at the beginning of training to rely more on the final reward. As the critic becomes more reliable, λ gradually gets smaller so that the model can make better use of token-level value estimates. The authors also use other techniques such as length penalty, token-level advantage normalization, clip-higher, and KL regularization.

Critical strengths

This paper provides a possible way to combine PPO and GRPO. It keeps the group-relative advantage from GRPO and uses the critic and GAE from PPO to estimate token-level advantages. Actually, token weighting has been studied by many GRPO related works such as DAPO and Dr.GRPO, these works try to improve token weighting by changing the loss normalization. This paper studies this problem from another direction by using token-level advantage estimation. Although most parts of the method are existing techniques, I think this combination is still worth exploring.

Critical weaknesses
  1. Many existing LLM post-training works have already tried to combine the advantages of PPO and GRPO, or improve the credit assignment in GRPO. For example, TEMPO(https://arxiv.org/abs/2509.18314) (Tran et al., 2025) combines the group-relative signal from GRPO with TD corrections. Le Critique(https://arxiv.org/abs/2608.16739) (Venkatraman et al., 2026) also combine group-relative and learned value baselines. Compared with these works, this paper mainly adds a small critic and uses λ-return with λ annealing to estimate token-level advantages. I think the authors should explain more clearly what are the main differences and new contributions compared with these existing works. It would also be better to have direct experimental comparisons with them.

  2. I think the experiments are quite limited today. The authors only evaluate performance on two relatively easy math reasoning benchmarks, GSM8K and MATH. However, there are many other commonly used math reasoning benchmarks in recent LLM post-training works, such as AIME 2024/2025, HMMT 2025, and OlympiadBench. A lot of 4B-8B model today already have very good performance on these two benchmarks. I think only using these two benchmarks is not enough to show the usefulness of the proposed method, especially for harder and longer reasoning tasks. More experiments on challenging math benchmarks would make the results more convincing. Or even OOD benchmarks are worth trying.

  3. GRPO assigns the same advantage to all tokens in one response, which is not reasonable, and λ-GRPO can better identify the contribution of different tokens. However, different token-level advantages do not necessarily mean more accurate credit assignment. The current experiments mainly show improvements in final accuracy, but do not directly analyze whether the critic can really identify important reasoning steps. For example, do correct reasoning steps, wrong steps, and useless repeated steps really receive reasonable advantages? I think the authors should provide more token-level analysis to support this claim, instead of only using final accuracy as evidence.

Questions for authors

See weakness.

1 comment

I will maintain my original score, as I have not received a response from the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.