acceptodds
Under review as a conference paper at ICLR 2027

REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Normalization

Abstract

Critic-free reinforcement learning reduces the cost of post-training large language models, but its behavior depends strongly on how rewards are converted into advantages. In group-relative methods, normalizing rewards separately for each prompt couples the update scale to statistics estimated from a small set of responses. We introduce REINFORCE++, a critic-free policy optimization framework that instead normalizes advantages across the rollout batch. The basic variant supports one or more responses per prompt and incorporates a token-level KL penalty into Monte Carlo returns. A second variant, REINFORCE++-Baseline, combines a within-prompt reward baseline with global normalization and a separate KL regularizer. Our analysis characterizes finite-sample normalization and its convergence to population standardization under independent sampling. Experiments cover preference-based alignment, mathematical and logical reasoning, and multi-step tool use. With one response per prompt, REINFORCE++ obtains an Arena-Hard score of 46.7, compared with 46.8 for four-response GRPO. In tool-use experiments, REINFORCE++-Baseline achieves an average score of 24.10, compared with 22.58 for GRPO and 21.85 for PPO. These results support global normalization as a practical design choice for critic-free language-model training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Large Language Models, RLHF

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2wssOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes REINFORCE++, a critic-free policy optimization framework that replaces prompt-level advantage normalization with global batch normalization. The authors also provide a theoretical analysis of normalization and evaluate the methods on preference alignment, mathematical reasoning, logical reasoning, and multi-step tool use.

Critical strengths
  1. It provides a simple and practical alternative to prompt-level normalization, potentially improving training stability without requiring a learned critic.

  2. These experiments span multiple domains, including general reinforcement learning versus human performance (RLHF), mathematical reasoning, and tool use. The experimental results show that, across a variety of scenarios, the global normalization method can match the performance of GRPO and PPO.

Critical weaknesses
  1. The experiments primarily rely on Llama-3-8B-SFT and Qwen2.5, which are relatively outdated for an ICLR 2027 submission. Evaluation on more recent models (e.g., Qwen3) is needed to establish generalizability. Moreover, the general RLHF experiments lack comparisons with DPO or similar preference-optimization methods, while the reasoning experiments omit stronger recent RL baselines such as DAPO and VAPO. Also, Human evaluation is needed for human preference.

  2. This method primarily replaces local normalization with global batch normalization, offering limited methodological innovation beyond existing approaches. Furthermore, the paper lacks sufficient ablation experiments, making it impossible to distinguish the contribution of global normalization from that of other components (such as the reward baseline and KL regularization). Some of the improvements are negligible, and the exceptionally large generalization gap shown in Table 2 lacks sufficient statistical validation, raising doubts about the robustness of the experimental conclusions.

Questions for authors

How exactly are Pass@1 and Pass@16 computed? It looks very weird. GRPO achieves only 0.4% Pass@16 on AIME-25, which contains 30 problems. This value cannot result from a single evaluation using the standard binary Pass@16 metric with 16 samples per problem. Were multiple evaluation runs or a different estimator used? Please clarify the evaluation protocol and report variance across random seeds or temerpture.

Reviewer_UP3HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

This paper proposes REINFORCE++, a critic-free variant of PPO which main idea is to normalize advantages using rollout-batch-level statistics rather than the prompt-level statistics used in GRPO. The paper introduces two variants. The basic variant which supports (k≥1(k \geq 1), incorporates a token-level k1-style KL penalty into the reward before applying global normalization. The Baseline variant, designed for (k>1(k > 1), first subtracts the within-group mean and then scales the resulting advantages by the global standard deviation, with a separate k2-style KL regularization term.

Appendix B presents finite-sample bounds and asymptotic limits for self-normalized statistics, while Appendix C reviews the pointwise gradients of the k1, k2, and k3 KL estimators.

The experiments cover Arena-Hard (where REINFORCE++ with (k=1(k=1) performs comparably to GRPO with (k=4(k=4): 46.7 vs. 46.8), generalization to AIME under limited training data, Knights and Knaves (K&K) logic puzzles, RL-from-zero mathematical reasoning, and ZeroTIR tool-use tasks. On ZeroTIR, the Baseline variant achieves an average score of 24.10 across five benchmarks, compared with 22.58 for GRPO and 21.85 for PPO.

Critical strengths
  1. The paper usefully distinguishes the choice of which mean to subtract from the choice of which scale to divide by. The discussion of the pointwise gradients of k1, k2, and k3 in Appendix C is also technically correct and informative. In particular, the expected gradient of k2 matches the reverse-KL gradient, whereas the pointwise gradient of k3 actually points in the direction of forward KL. This analysis provides useful guidance for practitioners. However, it seems like a summary of results from Liu et al. (2025a), rather than constituting a new theoretical contribution.

  2. The paper evaluates the proposed method across preference alignment, RL with verifiable rewards (RLVR), and tool-use tasks. The improvements on 8-person K&K puzzles (approximately 37 vs. 20) and the positive results on tool-use benchmarks could be practically meaningful if they can be reproduced across multiple random seeds under controlled experimental settings. These observations are the main reason I do not assign the paper the lowest overall rating.

Critical weaknesses
  1. The central claim does not directly isolated or validated: The paper's main claim is that global advantage normalization is preferable to prompt-level normalization. However, none of the experiments isolates the normalization scale while holding the centering strategy, KL formulation, number of rollouts (k), and training budget fixed. In Section 4.1, both (k) (1 vs. 4) and the baseline structure change simultaneously. In Section 4.2, the basic REINFORCE++ variant does not appear to perform per-prompt centering when (k>1), meaning that its differences from GRPO extend beyond the normalization scale. Moreover, Figure 3 compares GRPO trained for 400 rounds with REINFORCE++ trained for only 100 rounds. Consequently, the generalization differences reported in Table 2 could potentially be explained entirely by differences in training budget. The reported Pass@1 of 0.0 for GRPO on AIME-25 is also more suggestive of training failure or collapse than an intrinsic property of the algorithm.

The only clearly result in Table 4 comes from the Baseline variant. However, Appendix D explicitly acknowledges that this result reflects the combined effect of the "complete training configuration," including group-mean centering, global standard-deviation scaling, k2 KL regularization, and response-wise aggregation. It therefore cannot be attributed specifically to global normalization.

Also, the advantage formulation of the Baseline variant differs from that of Dr. GRPO (Liu et al., 2025, Understanding R1-Zero-Like Training) only by a batch-level multiplicative constant, yet this closely related method is neither cited nor included as a baseline. Since a batch-level constant scaling of the policy-gradient term is approximately invariant under Adam, the more substantive change is the removal of the prompt-specific (1/σq1/\sigma_q) factor. This distinction is not adequately reflected in the paper's central narrative.

  1. The theoretical analysis is not aligned from the main conclusions and the empirical evidence is not sufficient. Appendix B shows classical properties of scalar self-normalized statistics, including the (N−1\sqrt{N-1}) bound, which dates back to Samuelson (1968) but is not cited, together with asymptotic results based on the law of large numbers. However, the analysis provides no results concerning policy-gradient bias, variance, or convergence. It also does not explain why the proposed finite-sample distortion should be harmful.

Furthermore, the Gaussian i.i.d. assumption does not adequately reflect the binary rewards used in Sections 4.2–4.3. For example, with (k=4) and binary rewards, the within-group standard deviation in GRPO can take only two distinct nonzero values. Despite the emphasis on "Stabilizing" in the title, the paper does not report any direct measures of training stability.

Finally, some essential experimental details are missing. The base model identities and model sizes for Section 4.2.1, the K&K experiments, and Table 3 are not reported, nor are the complete hyperparameter configurations.

Questions for authors
  1. Why are GRPO and REINFORCE++ trained for different numbers of rounds in Figure 3? What is the AIME-25 performance of REINFORCE++ when trained for the same number of rounds as GRPO?

  2. In Table 2, GRPO achieves a test Pass@1 of 0.0. Have the authors investigated whether this result is caused by output-format issues, training instability, or model collapse?

  3. What are the base model identities and parameter sizes used in Section 4.2.1, the K&K experiments, and Table 3? Additionally, does the "Pass@8" metric in Table 3 refer to the conventional pass@8 metric or to avg@8?

  4. Does the basic REINFORCE++ variant apply per-prompt centering when (k>1(k>1) in Section 4.2? If not, how can the observed differences relative to GRPO be attributed specifically to the normalization scale rather than to the change in centering?

  5. How does the proposed Baseline variant relate to Dr. GRPO? In particular, why should the implicit weighting proportional to (pq(1−pq)(p_q(1-p_q)) in the Baseline variant be preferable to the (1/σq(1/\sigma_q) weighting induced by GRPO? Can the authors provide a theoretical justification or a controlled empirical comparison?

Reviewer_BGb5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents REINFORCE++, a critic-free framework for RL that combines PPO clipping with advantage normalization across the full rollout batch. The basic variant supports one or more responses per prompt and incorporates token-level KL penalties into Monte Carlo returns. The paper analyzes finite sample normalization under independent Gaussian sampling and evaluates both variants across preference alignment, mathematical and logical reasoning, and multi-step tool use. Reported results on Arena-Hard benchmark seems to be promising.

Critical strengths
  • The method is simple and practical. The basic variant needs no learned critic and supports one response per prompt, while achieving the benchmark score close to GRPO's score with slightly shorter responses.
  • I appreciate the evaluation across preference alignment, mathematical and logical reasoning, and tool use. The baseline variant outperforms GRPO on four of five tool-use benchmarks. This breadth makes the results more reliable than an improvement on a single task.
Critical weaknesses
  • I think the contribution of global normalization is not isolated well because the comparisons also vary prompt coverage, reward centering, and KL treatment. Without controlled ablations, We can't tell which changes drive the gains or whether simpler alternatives would perform similarly.
  • The evidence lacks sufficient detail and uncertainty estimates. The paper does not report training-seed variability or confidence intervals for benchmark results, and important settings such as learning rates and KL coefficients are missing. Fig 3 also shows roughly 400 rounds for GRPO versus 100 for REINFORCE++, without explaining budget comparability. These gaps weaken the evidence for better generalization and the smaller reported gains.
Questions for authors
  • Please clarify, were training budgets and checkpoint-selection rules matched in Tab 2? Please explain Fig 3's different training horizons and specify the starting model, decoding settings, and pass@1 and pass@16 estimators.
  • Can you compare local, global, and no standard-deviation normalization while holding the reward baseline, group size, KL treatment, loss reduction, and training budget fixed?
  • What are the complete experimental configs and numbers of independent training runs? Do the reasoning and tool-use gains persist across seeds, with uncertainty reported?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.