acceptodds
Under review as a conference paper at ICLR 2027

Future-KL Regularized GRPO: Process-Level Credit Assignment from -Divergence Regularization

Abstract

GRPO is widely used for critic-free LLM post-training, but its KL regularization is usually implemented as a local loss-side token penalty. We show that this misses the policy-gradient signal induced by autoregressive KL regularization. Unlike standard KL-regularized RL objectives, GRPO's group normalization induces a non-linear prompt-level utility; for binary verifier rewards, this utility is . As a result, reward and KL cannot be fused before normalization without changing the implicit objective. We derive the on-policy gradient of GRPO-style objectives with token-wise -divergence regularization. The reward term recovers the standardized GRPO advantage, while the regularizer contains a causal future cost-to-go omitted by local KL losses. For reverse KL, this yields Future-KL Regularized Policy Optimization (FRPO), a critic-free correction implemented by a reverse cumulative sum of sampled token log-ratios after advantage construction. We further study a windowed variant that interpolates between local and full-horizon regularization credit. On mathematical-reasoning tasks, full-horizon FRPO improves pass@16 in the experiments, maintaining a better exploration ability compared to other regularization methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcment Learning, Large Language Models, GRPO, f-divergence, KL regularization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_QRhPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the consistency between GRPO’s training updates and the intended optimization objective, focusing on KL regularization. It proposes FRPO, which first computes GRPO advantages from the original outcome rewards and then applies a token-level correction based on the cumulative KL cost from the current position to the end of the response. Experiments across multiple mathematical reasoning benchmarks demonstrate the method’s effectiveness.

Critical strengths
  1. The paper addresses a well-defined problem and provides a clear and convincing motivation.
  2. The proposed method is simple to implement and adds little computational overhead to KL-regularized GRPO. Its default implementation reuses existing token log-probabilities to compute a reverse cumulative sum of log-probability ratios, requiring neither a value model nor additional forward passes.
Critical weaknesses
  1. Evaluation on additional GRPO-style algorithms, such as DAPO, is needed to assess the method’s generality.

  2. All experiments use Qwen backbones. Given the reported contamination concerns on mathematical benchmarks for this model family [1], evaluation on other backbone families would strengthen the empirical evidence.

  3. The authors show that existing methods are inconsistent with their chosen objective, �0��0�, but do not sufficiently justify why this objective should be preferred. Gradient inconsistency alone does not establish that the original method is flawed: a local KL penalty can serve as a constraint evaluated on fixed sampled contexts, even if it does not recover the gradient of a full trajectory-level objective.

  4. The paper does not directly explain why restoring the missing gradient terms should preserve exploration and improve reasoning accuracy. The connection between the theoretical correction and these benefits remains primarily empirical.

[1] Shao R, Li S S, Xin R, et al. Spurious rewards: Rethinking training signals in rlvr[J]. arXiv preprint arXiv:2506.10947, 2025.

Questions for authors

See Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer QRhP

We sincerely thank the reviewer for their careful reading and thoughtful questions.

1. Additional GRPO-style algorithms

We re-ran DAPO on Qwen3-1.7B-Base under exactly our configuration, using clip-higher and no KL. We report pass@16, mean@16, and mean response length for FRPO and DAPO with different values of ϵhi\epsilon_{\rm hi}.

MetricDAPO ϵhi=0.2\epsilon_{\rm hi}=0.2DAPO ϵhi=0.28\epsilon_{\rm hi}=0.28DAPO ϵhi=0.35\epsilon_{\rm hi}=0.35FRPO
pass@1685.7588.3386.8190.13
mean@1666.4465.2163.9467.09
mean response length1563376911542979

We tune the hyperparameters of the DAPO baseline, and the one with ϵhi=0.28\epsilon_{\rm hi}=0.28 reaches a mean response length of 3769 tokens, exceeding the response length of FRPO. Despite its longer responses, DAPO reaches 88.33, compared with FRPO's 90.13, still with a gap of 1.8 points.

2. Additional backbone model families other than Qwen

We use Gemma 4 E2B-it as an additional backbone model to verify the effectiveness of FRPO, and the table below summarizes the results across different configurations, with each cell showing the pass@16 performance as a percentage. FRPO achieves the best overall pass@n performance compared to other baselines, which further validates the generalizability of the proposed method.

SettingMATH500AIME24AIME25Minerva MathOlympiadBenchOverall
GRPO (no KL)96.4073.3360.0055.5173.6377.64
Loss-side KL97.8066.6753.3355.8873.6377.90
FRPO97.8073.3356.6755.8876.3079.30

3. Regularized RL objective

We agree that a local KL penalty is a valid way to regularize the policy on fixed sampled prefixes. Our objective asks how much the policy drifts from the reference over complete responses that it generates. We retain GRPO’s reward utility ℓ(p)\ell(p) and use KtrajK_{\mathrm{traj}} to measure that response-level drift.

The distinction matters because the model chooses its own prefixes. An early token can incur a small immediate KL cost and still lead the model into a line of reasoning where it repeatedly diverges from the reference. By the KL chain rule, those later costs contribute to the divergence between the two distributions over complete responses. A fixed-context penalty treats the distribution of prefixes as given. Future-KL accounts for how an early choice changes which prefixes the model visits and assigns the resulting downstream KL cost back to that choice. This is why we prefer the trajectory-level objective when the goal is to regularize the model’s full generated responses.

4. Connection between FRPO and exploration ability

With a binary verifier, every correct completion contributes the same reward. At a fixed success probability pp, the reward utility ℓ(p)\ell(p) is therefore indifferent to whether the policy spreads its successful responses across several solution paths or concentrates them on one. Completion-level KL distinguishes these cases. By the same chain rule, at fixed pp it is minimized by matching the reference’s conditional distribution over correct completions, and likewise over incorrect completions. Thus, at the unrestricted population optimum with β>0\beta>0, increasing the overall probability of success can preserve the relative probability of every reference-supported correct path. This gives a concrete reason to regularize the completion distribution. It protects existing ways of solving the problem even though the verifier gives no extra reward for keeping them available. Future-KL carries that regularization signal back to the earlier tokens that select a path.

In Appendix F.7, at the same β\beta, suffix credit reached 92.0% pass@16 and 68.1% mean@16 on MATH500, versus 86.0% and 61.2% for whole-response broadcast credit, even though the broadcast run had higher token entropy. This comparison supports the practical importance of appropriate credit assignment and regularization, not simply boosting the entropy.

AuthorsOpenReview ↗

Response to Reviewer QRhP

Dear Reviewer QRhP,

Thank you again for your thoughtful feedback. When you have a chance, we would appreciate hearing whether our rebuttal addresses your concerns. We added results on Gemma 4 E2B-it and a tuned DAPO-style clip-higher baseline, and clarified why we use trajectory-level KL and how it can help preserve reference-supported solution paths.

If the new results and clarifications address your concerns, we would be grateful if you would consider revisiting your rating. We are happy to clarify any points that remain open.

Thank you for your time, The authors

Reviewer_W7utOpenReview ↗
Rating
3
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

The paper revisits KL regularization in GRPO. It argues that GRPO’s group-normalized advantage induces a non-linear arcsin utility, so KL-in-reward changes reward/KL scaling, while loss-side token KL misses the future cost-to-go induced by early tokens. The authors derive an on-policy token-wise f-divergence gradient and propose FRPO: add a future-KL credit after GRPO advantage construction. Experiments on Qwen3-1.7B/30B-A3B math reasoning show FRPO improves pass@16/mean@16 over loss-side KL and KL-in-reward, with higher entropy and lower PPO-KL.

Critical strengths
  1. The paper gives a clear and useful diagnosis: under GRPO’s non-linear group-normalized objective, reward and KL regularization scale differently, and local loss-side KL omits future cost-to-go. This is conceptually consequential and leads to a simple algorithmic fix.
  2. FRPO is lightweight—one reverse cumulative sum—and the 30B-A3B pass@16 gains, together with horizon/estimator/KL-direction ablations, provide promising empirical support.
Critical weaknesses
  1. The theory-to-practice gap is not sufficiently quantified. The derivation relies on population group statistics, binary rewards, on-policy sampling, and full horizon, while experiments use G=5, mini-batch staleness, clipping, and finite groups. The claim that post-advantage placement is the “only” consistent placement is also too strong without discussing alternative normalization-aware corrections.
  2. The empirical case relies heavily on pass@16 and default KL coefficients. Matched-KL or matched-β comparisons, pass@1/greedy accuracy, and training-seed variance (especially for 30B) are needed to show gains are not driven by regularization strength, entropy, or coefficient choice.
Questions for authors
  1. Under matched actual KL or matched β, does FRPO still outperform loss-side KL and no-KL on pass@1, mean@16, and pass@16?
  2. Can you provide a process-level placebo or diagnostic isolating the effect of future-KL credit from overall KL strength and entropy changes?
  3. For the 30B-A3B main result, can you report training-seed variance or independent-run variance, rather than only evaluation-bootstrap std?
2 comments
AuthorsOpenReview ↗

Response to Reviewer W7ut

We sincerely thank the reviewer for their careful reading and thoughtful questions.

1. Theory-to-practice gap

The arcsine utility describes the population limit for binary rewards. FRPO retains the finite-group GRPO outcome advantage. Under Eq. (3), with I.I.D. responses, equal response weighting, and no clipping, the expected reward gradient at G=5G=5 is 57% of the population gradient for a prompt with success probability p=0.1p=0.1, and 86% at p=0.5p=0.5. This illustrates how finite groups can change the relative weighting of prompts before token-mean aggregation. The on-policy KL cost-to-go identity holds independently of the binary-reward assumption and group size. Full-horizon credit recovers that identity, while finite windows truncate it.

Appendix F.12 quantifies the implementation gap at 1.7B. With four actor updates per rollout batch, clipping zeroed 0.173% of token updates, and the clipped and unclipped token coefficient vectors had a cosine similarity of 0.9986. Recomputing Future-KL at each mini-batch changed MATH500 pass@16 by −0.49 points and mean@16 by +0.17 points in a single run, both within the observed spread of the three fixed-credit seeds. With other hyperparameters staying the same in both runs, strict on-policy training and the four-mini-batch schedule reached 89.37% and 89.39% pass@16, respectively, and performed similarly as well.

For the post-advantage placement claim, we will also narrow our claim. The requirement for our objective is to preserve verifier-only normalization and add the KL gradient to its specified weight. We will clarify this and distinguish it from recomputing group statistics on KL-shaped rewards more clearly.

2. Matched-β\beta comparison, pass@1 accuracy, and multi-seed experiments

In Table 11 in Appendix F.8, we show the results of matched-β\beta based on Qwen-1.7-Base for FRPO and loss-side KL regularization. Below are the pass@16 results evaluated on MATH500.

β =Loss-side KLFRPO
1e-184.785.6
1e-284.891.8
1e-386.385.6
1e-483.683.2

From the results, we can see that with the same hyperparameter sweep efforts, under the same regularization parameter $\beta=$1e-2, FRPO achieves 7 percent higher pass@16 performance compared to loss-side KL regularization, validating the effectiveness of the proposed method.

For greedy accuracy, we instead report mean@16 as an estimation for the sampled pass@1 accuracy under the same decoding protocol by averaging correctness over 16 samples per problem.

Due to the computation resource limitation, we first conduct multi-seed training on the Qwen3-1.7B-Base model. At 1.7B, Table 7 reports independent-run MATH500 pass@16 of 89.43±0.4889.43\pm 0.48 for FRPO, 87.11±0.3887.11\pm 0.38 for loss-side KL, and $ 85.93\pm 1.03$ for GRPO without KL, showing the stable improvement of FRPO versus the loss-side KL baseline.

3. Process-level placebo

We tested a process-level placebo that preserves the strength of the supplied future-KL signal while disrupting its alignment with individual tokens. Starting from the same Gemma checkpoint, we continued the training for both variants. The placebo permutes the strictly future-KL tail among valid tokens within four relative-position bins in each response, preserving the set of tail values in each bin and leaving the current-token KL term intact. Aligned FRPO achieved higher pass@8 and pass@16 on all five benchmarks. On OlympiadBench, aligned FRPO improved pass@1, pass@8, and pass@16 by 0.88, 2.15, and 2.96 percentage points, respectively. Mean rollout reference KL was nearly identical across the training steps, while entropy was higher under aligned FRPO. This supports a contribution from token-level credit alignment, with entropy and late-training policy drift remaining plausible mediators.

MetricMATH500AIME24AIME25Minerva MathOlympiadBench
FRPO pass@188.23%43.96%31.04%40.21%56.63%
Shuffled pass@188.96%40.63%31.67%39.45%55.75%
FRPO pass@896.58%63.89%51.60%52.96%72.01%
Shuffled pass@896.43%59.41%47.41%51.03%69.85%
FRPO pass@1697.80%73.33%56.67%55.88%76.30%
Shuffled pass@1697.00%63.33%50.00%54.41%73.33%

I will maintain my original score

Reviewer_6uEROpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper starts from a fundamental problem in the GRPO update: group normalization implicitly introduces non-linear utility, causing differences from standard maximum entropy RL. Based on this non-linearity, the added KL term also produces unexpected effects.

Starting from this theoretical analysis, the paper proposes a lightweight modification of GRPO that computes the group-normalized outcome advantage from the original verifier reward. Furthermore, it adds a token-level Future-KL correction. Intuitively, this Future-KL correction accounts for the accumulating KL divergence between the current policy and the reference policy.

The authors conduct experiments across many tasks and models. In general, for the RLVR task with binary rewards, it outperforms the baselines and shows strong results, especially on math tasks.

Critical strengths

Firstly, I think this paper starts from a fundamental issue in the GRPO update: the non-linear utility introduced by group normalization. The follow-up analysis of the KL divergence is solid, so in general, I think the analysis is strong.

The motivation and target problem of this paper are very important, since currently RLVR has become a major method to improve model performance.

The proposed Future-KL method is also meaningful, and the empirical results show that it can make the KL more reasonable.

Critical weaknesses

The weakness is that I can only find the pass@16 result in the main paper. How about other results, like pass@8 or pass@1?

Questions for authors

Will the correlation also be stable for different group sizes? Will the analysis and empirical designs be different with different group sizes, like 16, 32, or larger?

Can this method be used to some other tasks beyond math? Even consider some agentic tasks?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6uER

We sincerely thank the reviewer for their careful reading and thoughtful questions.

1. pass@n metrics

We mainly report the pass@16 metric in our paper, as our primary goal is to show that, with the appropriate KL-regularized objective after introducing the extra future KL term, FRPO maintains stronger exploration in the policy model and therefore achieves higher pass@n accuracy. In the table below, we report pass@1, pass@8, and pass@16, respectively, with each cell reporting pass@1 / pass@8 / pass@16 in percent.

ModelSettingMATH500AIME24AIME25Minerva MathOlympiadBench
Qwen3-1.7B-BaseBase56.74 / 67.95 / 69.652.08 / 11.21 / 16.670.83 / 5.89 / 10.009.79 / 32.85 / 41.9111.21 / 35.72 / 44.21
Qwen3-1.7B-BaseGRPO (no KL)65.99 / 84.60 / 88.108.33 / 23.83 / 30.004.17 / 14.78 / 20.0028.33 / 46.86 / 51.4733.25 / 55.28 / 60.53
Qwen3-1.7B-BaseLoss-side KL67.78 / 85.80 / 89.209.38 / 20.99 / 23.334.58 / 11.65 / 13.3327.23 / 44.55 / 48.1630.80 / 52.49 / 57.86
Qwen3-1.7B-BaseFRPO66.72 / 88.29 / 91.206.88 / 24.80 / 30.004.79 / 14.54 / 20.0025.60 / 45.27 / 49.6331.92 / 56.08 / 62.02
Qwen3-4B-BaseBase63.90 / 70.20 / 70.994.79 / 19.42 / 23.334.17 / 19.88 / 26.6717.10 / 44.97 / 50.0018.03 / 49.42 / 57.12
Qwen3-4B-BaseGRPO (no KL)78.64 / 81.94 / 82.5216.25 / — / 29.4312.29 / — / 34.1639.48 / — / 55.0744.95 / — / 64.28
Qwen3-4B-BaseLoss-side KL84.31 / 89.89 / 90.6522.50 / 37.43 / 43.3323.33 / 48.38 / 56.6740.65 / — / 58.6453.46 / — / 72.50
Qwen3-4B-BaseFRPO84.58 / 91.83 / 92.9321.67 / 38.44 / 46.6719.58 / 39.16 / 50.0039.22 / 57.09 / 61.0351.03 / 70.65 / 74.93
Qwen3-30B-A3B-BaseBase77.16 / 86.42 / 87.779.17 / 32.27 / 40.003.96 / 19.19 / 23.3318.45 / 49.25 / 55.1523.44 / 58.37 / 66.47
Qwen3-30B-A3B-BaseGRPO (no KL)85.34 / 89.50 / 90.4824.17 / 50.41 / 56.6717.71 / 26.33 / 30.0044.28 / 56.32 / 59.1951.00 / 65.08 / 68.55
Qwen3-30B-A3B-BaseLoss-side KL83.61 / 88.85 / 89.6422.29 / 46.63 / 60.0010.21 / 19.99 / 26.6742.69 / 54.42 / 57.3547.53 / 61.56 / 64.99
Qwen3-30B-A3B-BaseFRPO88.13 / 95.05 / 96.0732.08 / 60.44 / 70.0024.17 / 42.48 / 50.0043.18 / 57.50 / 60.6658.32 / 76.42 / 79.82

From the results, we can see that FRPO generally improves the pass@n performance across different settings, especially as n grows, showing that FRPO preserves better exploration ability for the models compared to other baselines.

2. Correlation to group size

The Future-KL correction follows from the on-policy gradient of the trajectory-level regularizer. It depends on the policy and reference log-probabilities along each rollout and is added after the outcome advantage is normalized. Both the derivation and the update therefore have the same form for different group sizes, like G=5,16,32G=5,16,32, or larger.

3. Tasks beyond math

The same derivation applies to tasks with a task-specific reward, including multi-turn agents evaluated by unit tests or task completion. In a tool-use episode, tool outputs enter the agent’s history, and the KL sum covers policy-generated actions across turns, including tool calls. Because an early action can change what the agent observes and does later, its correction includes the resulting future KL costs. The binary-reward analysis applies directly to success/failure feedback, while other reward settings can use a task-appropriate advantage estimator with the same Future-KL term. Similar derivations establish applicability to these settings.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author Responses and Additional Results

We thank the reviewers for their careful reading and constructive feedback. We appreciate their recognition of the paper’s analysis of how group normalization changes the GRPO objective, the simplicity of the Future-KL update, and the empirical evidence from the experiment results and ablations. FRPO implements the Future-KL with a single reverse cumulative sum using log-probabilities already computed during training.

Our main point is that an early token can change how much the rest of a response diverges from the reference policy. Local KL regularizes the model on fixed prefixes. Future-KL accounts for the cost of the later prefixes that the model’s own choices lead it to. This correction has the same form across group sizes, and we clarify how finite-group normalization and clipping affect the practical update.

In our replies, we report pass@1 and pass@8 alongside pass@16, a matched-β\beta sweep, independent runs at 1.7B, and a tuned clip-higher baseline. On Gemma, FRPO reaches 79.30% overall pass@16, compared with 77.90% for loss-side KL and 77.64% for GRPO without KL.

When we shuffle which tokens receive the future-KL costs, pass@8 and pass@16 drop on all five benchmarks. On OlympiadBench, aligned FRPO gains 2.96 percentage points in pass@16. Average reference KL is nearly identical across the two runs, while entropy differs. This supports a role for assigning future costs to the right tokens.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.