acceptodds
Withdrawn from ICLR 2027

Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning (RL) has become a key driver of language model reasoning. Among RL algorithms, Group Relative Policy Optimization (GRPO) is the de facto standard, avoiding the need for a critic by using per-prompt baselines and variance normalization. Yet why and when this normalization helps remains unclear. In this work, we answer both questions through the lens of local curvature of the sequence-level policy gradient: standard deviation normalization implements an adaptive gradient. Theoretically, we prove that under mild conditions GRPO attains an improved convergence rate over unnormalized REINFORCE, with gains characterized by the average within-prompt reward standard deviation across prompts and iterations. We further introduce IS-GRPO, an importance-sampling variant of GRPO whose expected update remains aligned with the full gradient, and prove a convergence guarantee of the same form as REINFORCE's, which is tighter in practice during real training runs by a factor we measure. Empirically, on GSM8K and MATH we validate the curvature–variance link and measure the quantities that govern our bounds along real training runs. At 1.5B scale, per-prompt normalization outperforms both unnormalized and globally normalized baselines, with gains that emerge in a middle phase of training where per-prompt variances become heterogeneous. At 7B scale, the normalization schemes become statistically indistinguishable while IS-GRPO retains a small lead, in line with the theory's prediction that the gains require heterogeneity.

Primary area:
optimization
Keywords:
optimization algorithm, large language models, reinforcement learning, RLVR, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vWzuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyzes the impact of advantage normalization in GRPO on convergence rates and, based on this analysis, proposes IS-GRPO, which uses importance sampling over the question distribution to further relax the assumptions on the question distribution required for the convergence-rate benefits of normalization. The paper mainly conducts experiments on mathematical tasks, and the experimental results indicate that the proposed method performs well.

Critical strengths
  1. The paper attempts to directly study the convergence rates of RL for LLMs from a theoretical perspective. This is challenging. Such research would be highly valuable if appropriate theorems could be established.
  2. The theoretical framework of the paper is broadly reasonable, characterizing convergence rates through the rate of decrease in the gradient norm.
  3. The experimental analysis of gradient similarities provides important insights for related research.
Critical weaknesses
  1. The theoretical model is inconsistent with practical settings. Most of the theorems and conclusions in the paper rely on conditions or assumptions that do not align with RL for LLMs. For example: 1) The paper uses a softmax policy to approximate the LLM policy, which makes the analysis of $ X $ meaningless, especially considering that GRPO also applies length normalization; 2) It treats an entire response as an action, rather than tokens as used in algorithms such as GRPO, which makes the convergence-rate analysis less meaningful; 3) The parameter updates appear to be based on standard steepest descent, without considering optimizers such as Adam; 4) It assumes that each question has only one correct answer, but multiple chains of thought can lead to the same correct answer. These are only the issues I have identified so far, but I suggest that the authors carefully examine the entire paper for similar problems.
  2. The convergence-rate analysis is not sufficiently convincing. The paper analyzes the L2 norm of the gradient and the Hessian matrix, but: 1) It is based on a softmax policy, which is inconsistent with LLM policies; 2) It relies on standard steepest descent iterations and therefore does not apply to optimizers such as Adam. Even setting these issues aside, the paper focuses on the coefficient governing the decrease in the gradient norm. Consider a toy example: optimizing $ f(x) $ and optimizing $ 4*f(x) $ have the same convergence rate, but the coefficients governing the decrease in their gradient norms are clearly different. In fact, it is almost impossible to quantitatively analyze the convergence rates of RL for LLMs. Such analyses inevitably require unreasonable assumptions or simplifications, which is not inherently problematic. However, it is inappropriate to present conclusions derived from these simplifications as the main contributions and theorems of the paper.
  3. The importance sampling in IS-GRPO is incorrect. The paper ignores the prior distribution of $ s $. The values of $ s $ in the samples follow the dataset distribution, and simply normalizing them cannot yield the target distribution.
  4. The experimental results are limited. Moreover, several aspects of the experimental setup seem unusual, including the LoRA hyperparameters, learning rates, and the use of H200 GPUs with verl without full-parameter fine-tuning.
Questions for authors
  1. Which theorems do not depend on idealized assumptions such as the softmax policy and the uniqueness of the correct answer?
  2. What are the authors' thoughts on the bias introduced by GRPO's length normalization (compared with DAPO)? This makes the two algorithms different, yet both can be simplified into softmax policies using normalized advantages. Are there more rigorous theorems that can distinguish between them?
  3. Is the authors' theoretical framework applicable to algorithms such as MaxRL, which normalize advantages using accuracy?
1 comment

While we appreciate your recognition of our convergence theory and gradient-similarity findings, we respectfully disagree with your central critiques, which mischaracterize our setup and overlook guarantees already established in the paper. Below, we address each point directly.

W1, W2 & Q1: Importance of the log-linear sequence-level model

Log-linear (softmax) policies over fixed features are standard in policy-gradient [1,2] and GRPO theory [3, 4]. Rather than claiming constant transfer to Transformers with AdamW, this abstraction isolates the geometric mechanism of std normalization—Bernoulli variance governs local Hessian smoothness—yielding falsifiable predictions validated at 1.5B and 7B scales. We refer to Reviewer KoGJ (Q1)\color{#8EBA42}{\textsf{Reviewer KoGJ (Q1)}}, Assumption 1 is a common simplification: all proofs hold for multiple correct responses.

W1, W2: Scale-invariance and AdamW vs. SGD

Scale-invariance: Our comparison is strictly invariant to objective rescaling: all methods optimize the identical objective J(θ)=1n∑i=1nJi(θ)J(\theta) = \frac{1}{n}\sum_{i=1}^n J_i(\theta) under the same stationarity metric min⁡tE[∥∇J(θt)∥2]\min_t \mathbb{E}[\Vert\nabla J(\theta_t)\Vert^2] in all theorems. With objective and metric scales fixed, C(T)<1C(T) < 1 and DIS(T)<D(T)D_{\mathrm{IS}}(T) < D(T) reflect genuine variance-adaptive curvature conditioning.

Complementarity with AdamW: Normalization acts orthogonally to coordinate-wise adaptation: AdamW applies batch-aggregated coordinate-wise scaling, whereas GRPO applies prompt-wise reward-std scaling. Controlled 1.5B ablations confirm this non-redundancy, as per-prompt std consistently outperforms REINFORCE and global-std GRPO under the identical AdamW optimizer.

W3: Importance sampling in IS-GRPO

Algorithm 3 strictly preserves the uniform empirical dataset objective J(θ)J(\theta). Sampling prompts with probability pt,ip_{t,i} proportional to their reward standard deviation while scaling updates inversely (∝1/pt,i\propto 1/p_{t,i}) applies the exact importance-sampling weight relative to the uniform dataset distribution [5, 6]. Consequently, the expected update remains a positive scalar multiple of the full dataset gradient.

W4: Experimental protocol and held-out 7B evaluation

  • Compute protocol and parameterization: At the 1.5B scale, adopting LoRA (r=64,α=128r=64, \alpha=128) on H200 GPUs via verl enabled a rigorous, controlled 45-run hyperparameter sweep (5 LRs×3 methods×3 seeds5\text{ LRs} \times 3\text{ methods} \times 3\text{ seeds}) under strictly identical rollout budgets. At the 7B scale, experiments were executed on a single 24 GB RTX A5000 GPU, where LoRA with FSDP and gradient checkpointing follows standard memory-efficient practice.
  • Held-out 7B benchmark results: To substantiate out-of-distribution generalization at scale, we evaluated Qwen2.5-Math-7B trained on DAPO-Math-17k (600 steps) across three standard held-out benchmarks, where IS-GRPO consistently outperforms both REINFORCE and per-prompt GRPO:
MethodMATH-500AMC23AIME24
REINFORCE67.350.519.8
GRPO (per-prompt std)69.351.723.1
IS-GRPO73.954.725.0

Q2: Orthogonality of length and standard-deviation normalization

Recent analyses show that length normalization in GRPO (LjL_j), DAPO (∑jLj\sum_j L_j), and Dr. GRPO (MM) alters within-prompt response weights, inducing length-dependent gradient bias. Within our framework, length aggregation acts orthogonally to std normalization: length weights rescale individual trajectories within a prompt (modeled via length-weighted score functions), whereas std normalization scales the entire prompt-level update by 1/V[πθ(i)]1/\sqrt{\mathbb{V}[\pi_\theta(i)]}.

Q3: Extension to accuracy-normalized methods (MaxRL)

Our local-curvature perspective extends naturally to accuracy-normalized methods such as MaxRL. Viewed through our local smoothness lens (Lemma 1), this places greater relative weight on low-accuracy prompts than GRPO while remaining bounded on nearly solved ones, aligning with MaxRL's focus on hard prompts without over-amplifying easy ones.


References

[1] Yuan et al., Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policies, arXiv:2210.01400, 2022. [2] Lin et al., Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation, arXiv:2505.03155, 2025. [3] Mroueh et al., Revisiting Group Relative Policy Optimization, arXiv:2505.22257, 2025. [4] Jain et al., Towards Understanding the Optimization Landscape of GRPO and Its Variants, FRLM Workshop, 2025. [5] Needell et al., Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz Algorithm, NeurIPS, 2014. [6] Zhao & Zhang, Stochastic Optimization with Importance Sampling for Regularized Loss Minimization, ICML, 2015.

Reviewer_KoGJOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines why and when standard-deviation normalization helps in GRPO. The authors link within-prompt reward variance to a bound on local curvature and interpret normalization as an adaptive gradient mechanism. The authors derive convergence guarantees under a log-linear policy model. They propose Importance Sampling GRPO (IS-GRPO) that samples questions in proportion to their reward standard deviations to remove the orthogonality assumption from the convergence analysis. Experiments are conducted using Qwen2.5-Math-1.5B and Qwen2.5-Math-7B backbones on the GSM8K and MATH datasets. The authors compare methods including per-prompt standard-deviation normalization, global standard-deviation normalization, REINFORCE, and IS-GRPO. Results show that per-prompt normalization improves performance when active prompts have heterogeneous reward variances, while IS-GRPO shows a small gain when most training prompts are already solved.

Critical strengths
  • This paper connects reward variance to local curvature to provide a new perspective on standard-deviation normalization in GRPO. The convergence analysis identifies concrete conditions under which normalization improves the convergence bound.
  • The comparisons between different methods help clarify the benefit of per-prompt normalization. Experiments on training sets with different difficulty levels also show when normalization helps and when its benefits are limited.
Critical weaknesses
  • The empirical evidence for the curvature-adaptive mechanism is insufficient. Lemma 3.1 provides an upper bound for local curvature in terms of reward variance, but Table 1 measures a Fisher proxy rather than the reward-objective curvature considered in the theory. The correlation does not directly show that normalization can improve training by adapting to local curvature.
  • The claimed alignment property and tighter convergence bound of IS-GRPO are not verified in experiments. The theory uses the same current reward standard deviation for sampling and normalization, but the implementation uses lagged estimates for sampling and fresh estimates for normalization. Figure 4 uses estimated standard deviations that are not guaranteed upper bounds on the true values and computes both methods’ coefficients using only IS-GRPO’s training history. It is unclear whether the theoretical advantage holds in practice.
Questions for authors
  • Can the local-curvature and convergence results apply when a prompt has multiple correct output sequences, as in the experiments?
  • Do the experiments use token-level clipping? How many optimization passes are made over each rollout batch?
  • How much training overhead does IS-GRPO introduce? Are generated token budgets matched across different methods?
1 comment

Thank you for the constructive and insightful feedback. Below, we address each concern with additional clarifications.

W1: Empirical evidence for the curvature-adaptive mechanism

While the diagonal Fisher information serves as an approximation rather than the exact Hessian ∇2Ji(θ)\nabla^2 J_i(\theta) in Lemma 1, computing exact per-prompt Hessians for billion-parameter LLMs is computationally intractable. Following established practice in large-scale optimization [1, 2], we adopt the diagonal Fisher as the standard, tractable second-order curvature proxy. Moreover, Table 1 functions as a specificity control, demonstrating that curvature and reward variance exhibit a positive within-checkpoint correlation yet a negligible cross-iteration correlation, thereby excluding confounding from global training dynamics.

W2: Empirical verification of IS-GRPO alignment and trajectory coefficients

Lagged sampler and fresh normalizer. We refer to W1/Q1 section for Reviewer sgq4\color{#8EBA42}{\textsf{Reviewer sgq4}}.

Estimated standard deviations. In the revised manuscript, we will further recompute μk,t\mu_{k,t} via high-sample rollouts per prompt at saved checkpoints of both runs, and we will also compute D(T)D(T) directly on the REINFORCE algorithm's trajectory alongside C(T)C(T) and DIS(T)D_{\mathrm{IS}}(T) on IS-GRPO's.

Q1: Multiple correct output sequences per prompt

The unique-answer assumption is a common simplification in RL theory [3,4,5], and it is not essential. Mei et al. (2020) show that it can be lifted by replacing the probability of the optimal action with the total probability of all optimal actions. The same replacement works in our setting and we will generalize Assumption 1 in the revision.

Q2: Token-level clipping and optimization passes

Each rollout batch receives exactly one optimization pass (Table 4 in Appendix G.3), so updates are strictly on-policy. We use verl's standard token-level clipped loss and KL and entropy terms are disabled.

Q3: Training overhead and matched token budgets

All methods share the exact same rollout budget per step (16 prompts ×\times 8 rollouts = 128 rollouts at 7B; 64 prompts ×\times 8 rollouts = 512 rollouts at 1.5B). The IS-GRPO Beta-posterior tracker updates solely from these on-policy rollouts, introducing negligible overhead compared to 7B rollout generation and backpropagation. We will report exact wall-clock step times and generated token counts in the revision.


References

[1] Liu et al., Sophia: A Scalable Stochastic Second-Order Optimizer for Language Model Pre-Training, arXiv:2305.14342, 2023.

[2] Martens, New Insights and Perspectives on the Natural Gradient Method, JMLR, 2020.

[3] Mei et al., "On the Global Convergence Rates of Softmax Policy Gradient Methods," ICML 2020.

[4] Mei et al., "Stochastic Gradient Succeeds for Bandits," ICML 2023.

[5] Lin et al., "Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation," arXiv:2505.03155, 2025.

Reviewer_sgq4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper interprets GRPO normalization as adaptive gradient scaling linked to local curvature. It provides convergence bounds in an idealized log-linear setting and introduces IS-GRPO, an importance-sampling variant evaluated on GSM8K and MATH.

Critical strengths

The paper studies an important component of GRPO and distinguishes per-prompt normalization from global scaling. The 1.5B experiments include three seeds and a learning-rate sweep, and the authors also report settings where normalization provides little benefit.

Critical weaknesses
  • The theory does not directly cover the implemented IS-GRPO. The alignment argument relies on using the same true standard deviation for sampling and update normalization, whereas the implementation samples using stale EMA estimates and normalizes using current rollout statistics. These quantities do not cancel, so the claimed expected-gradient alignment remains unproven for the evaluated method.

  • The evidence for practical generalization is limited. The three-seed 7B comparison in Table 4 measures accuracy on training prompts, while the held-out 7B normalization experiments use only one seed. A multi-seed held-out evaluation of IS-GRPO is needed to establish whether its advantage extends beyond fitting the training set.

Questions for authors
  • Can the alignment argument be extended to the actual EMA-based sampler and finite-group normalization?
  • Can you provide multi-seed held-out results for IS-GRPO and move the main numerical comparisons into the main paper?
1 comment

Thank you for the constructive and insightful feedback. Below, we address each concern with additional experiments and clarifications.

W1/Q1: Theoretical alignment of the implemented IS-GRPO

While Algorithm 3 applies the oracle standard deviation Vi\sqrt{V_i} (Vi:=V[πθt(i)]V_i := \mathbb{V}[\pi_{\theta_t}(i)]) to both sampling and normalization, our theoretical framework extends directly to practical implementations that decouple these quantities.

Tracker-normalized alignment. Exact expected-gradient alignment requires only matching the sampling and normalization scalars rather than knowing Vi\sqrt{V_i}. For any positive tracker sis_i, sampling prompts proportionally to sis_i while scaling the mean-centered group gradient by 1/(2si)1/(2s_i) cancels sis_i identically, guaranteeing E[gt]∝∇J(θt)\mathbb{E}[g_t] \propto \nabla J(\theta_t) across arbitrary tracker lag and group size GG. Imposing the multiplicative bound κ−1Vi≤si≤κVi\kappa^{-1}\sqrt{V_i} \le s_i \le \kappa\sqrt{V_i} therefore preserves the non-asymptotic convergence guarantee of Theorem 3.5 without oracle variances.

Evaluated within-group normalization. Dividing by within-group estimator σ^i\hat{\sigma}_i induces effective prompt weights wi∝siE[σ^i]/Viw_i \propto s_i \mathbb{E}[\hat{\sigma}_i] / V_i. the resulting residual error from tracker lag and the finite-sample bias of σ^i\hat{\sigma}_i can likewise be controlled via analogous arguments. We will formalize this non-oracle analysis in the revised manuscript and include the tracker-normalized variant in our 7B comparison.

W2/Q2: Limited practical generalization

We agree that training-prompt accuracy alone does not establish generalization. To address this, we conducted a 7B experiment with held-out evaluation on mathematical reasoning benchmarks.

  • Setup: We train Qwen2.5-Math-7B (base) on 4,096 prompts from DAPO-Math-17k. Every method uses 16 prompts ×\times 8 rollouts per step for 600 steps, with AdamW (η=10−5\eta = 10^{-5}, constant schedule, weight decay 0.010.01, gradient clipping 1.01.0), 1 PPO epoch (mini-batch size 16, on-policy updates), and KL/entropy penalties disabled. The IS-GRPO tracker updates solely from these on-policy rollouts, keeping the rollout budget identical across all methods. Following an initial learning-rate sweep over {5×10−6,10−5,2×10−5}\{5\times 10^{-6}, 10^{-5}, 2\times 10^{-5}\}, all methods use η=10−5\eta = 10^{-5}. We evaluate on MATH-500 (avg@4), AMC23 (avg@16), and AIME24 (avg@16) at temperature 1.01.0.

  • Results at step 600 (%):

MethodMATH-500AMC23AIME24
REINFORCE67.350.519.8
GRPO (per-prompt std)69.351.723.1
IS-GRPO73.954.725.0

IS-GRPO outperforms both standard GRPO and REINFORCE across all three held-out benchmarks, and standard GRPO consistently outperforms REINFORCE. We will run additional training seeds to report multi-seed mean ±\pm std, and we will move the main 1.5B and 7B numerical tables into the main body of the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.