- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper analyzes the impact of advantage normalization in GRPO on convergence rates and, based on this analysis, proposes IS-GRPO, which uses importance sampling over the question distribution to further relax the assumptions on the question distribution required for the convergence-rate benefits of normalization. The paper mainly conducts experiments on mathematical tasks, and the experimental results indicate that the proposed method performs well.
Critical strengths
- The paper attempts to directly study the convergence rates of RL for LLMs from a theoretical perspective. This is challenging. Such research would be highly valuable if appropriate theorems could be established.
- The theoretical framework of the paper is broadly reasonable, characterizing convergence rates through the rate of decrease in the gradient norm.
- The experimental analysis of gradient similarities provides important insights for related research.
Critical weaknesses
- The theoretical model is inconsistent with practical settings. Most of the theorems and conclusions in the paper rely on conditions or assumptions that do not align with RL for LLMs. For example: 1) The paper uses a softmax policy to approximate the LLM policy, which makes the analysis of $ X $ meaningless, especially considering that GRPO also applies length normalization; 2) It treats an entire response as an action, rather than tokens as used in algorithms such as GRPO, which makes the convergence-rate analysis less meaningful; 3) The parameter updates appear to be based on standard steepest descent, without considering optimizers such as Adam; 4) It assumes that each question has only one correct answer, but multiple chains of thought can lead to the same correct answer. These are only the issues I have identified so far, but I suggest that the authors carefully examine the entire paper for similar problems.
- The convergence-rate analysis is not sufficiently convincing. The paper analyzes the L2 norm of the gradient and the Hessian matrix, but: 1) It is based on a softmax policy, which is inconsistent with LLM policies; 2) It relies on standard steepest descent iterations and therefore does not apply to optimizers such as Adam. Even setting these issues aside, the paper focuses on the coefficient governing the decrease in the gradient norm. Consider a toy example: optimizing $ f(x) $ and optimizing $ 4*f(x) $ have the same convergence rate, but the coefficients governing the decrease in their gradient norms are clearly different. In fact, it is almost impossible to quantitatively analyze the convergence rates of RL for LLMs. Such analyses inevitably require unreasonable assumptions or simplifications, which is not inherently problematic. However, it is inappropriate to present conclusions derived from these simplifications as the main contributions and theorems of the paper.
- The importance sampling in IS-GRPO is incorrect. The paper ignores the prior distribution of $ s $. The values of $ s $ in the samples follow the dataset distribution, and simply normalizing them cannot yield the target distribution.
- The experimental results are limited. Moreover, several aspects of the experimental setup seem unusual, including the LoRA hyperparameters, learning rates, and the use of H200 GPUs with verl without full-parameter fine-tuning.
Questions for authors
- Which theorems do not depend on idealized assumptions such as the softmax policy and the uniqueness of the correct answer?
- What are the authors' thoughts on the bias introduced by GRPO's length normalization (compared with DAPO)? This makes the two algorithms different, yet both can be simplified into softmax policies using normalized advantages. Are there more rigorous theorems that can distinguish between them?
- Is the authors' theoretical framework applicable to algorithms such as MaxRL, which normalize advantages using accuracy?
While we appreciate your recognition of our convergence theory and gradient-similarity findings, we respectfully disagree with your central critiques, which mischaracterize our setup and overlook guarantees already established in the paper. Below, we address each point directly.
W1, W2 & Q1: Importance of the log-linear sequence-level model
Log-linear (softmax) policies over fixed features are standard in policy-gradient [1,2] and GRPO theory [3, 4]. Rather than claiming constant transfer to Transformers with AdamW, this abstraction isolates the geometric mechanism of std normalization—Bernoulli variance governs local Hessian smoothness—yielding falsifiable predictions validated at 1.5B and 7B scales. We refer to Reviewer KoGJ (Q1), Assumption 1 is a common simplification: all proofs hold for multiple correct responses.
W1, W2: Scale-invariance and AdamW vs. SGD
Scale-invariance: Our comparison is strictly invariant to objective rescaling: all methods optimize the identical objective J(θ)=n1∑i=1nJi(θ) under the same stationarity metric mintE[∥∇J(θt)∥2] in all theorems. With objective and metric scales fixed, C(T)<1 and DIS(T)<D(T) reflect genuine variance-adaptive curvature conditioning.
Complementarity with AdamW: Normalization acts orthogonally to coordinate-wise adaptation: AdamW applies batch-aggregated coordinate-wise scaling, whereas GRPO applies prompt-wise reward-std scaling. Controlled 1.5B ablations confirm this non-redundancy, as per-prompt std consistently outperforms REINFORCE and global-std GRPO under the identical AdamW optimizer.
W3: Importance sampling in IS-GRPO
Algorithm 3 strictly preserves the uniform empirical dataset objective J(θ). Sampling prompts with probability pt,i proportional to their reward standard deviation while scaling updates inversely (∝1/pt,i) applies the exact importance-sampling weight relative to the uniform dataset distribution [5, 6]. Consequently, the expected update remains a positive scalar multiple of the full dataset gradient.
W4: Experimental protocol and held-out 7B evaluation
verlenabled a rigorous, controlled 45-run hyperparameter sweep (5 LRs×3 methods×3 seeds) under strictly identical rollout budgets. At the 7B scale, experiments were executed on a single 24 GB RTX A5000 GPU, where LoRA with FSDP and gradient checkpointing follows standard memory-efficient practice.Q2: Orthogonality of length and standard-deviation normalization
Recent analyses show that length normalization in GRPO (Lj), DAPO (∑jLj), and Dr. GRPO (M) alters within-prompt response weights, inducing length-dependent gradient bias. Within our framework, length aggregation acts orthogonally to std normalization: length weights rescale individual trajectories within a prompt (modeled via length-weighted score functions), whereas std normalization scales the entire prompt-level update by 1/V[πθ(i)].
Q3: Extension to accuracy-normalized methods (MaxRL)
Our local-curvature perspective extends naturally to accuracy-normalized methods such as MaxRL. Viewed through our local smoothness lens (Lemma 1), this places greater relative weight on low-accuracy prompts than GRPO while remaining bounded on nearly solved ones, aligning with MaxRL's focus on hard prompts without over-amplifying easy ones.
References
[1] Yuan et al., Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policies, arXiv:2210.01400, 2022. [2] Lin et al., Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation, arXiv:2505.03155, 2025. [3] Mroueh et al., Revisiting Group Relative Policy Optimization, arXiv:2505.22257, 2025. [4] Jain et al., Towards Understanding the Optimization Landscape of GRPO and Its Variants, FRLM Workshop, 2025. [5] Needell et al., Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz Algorithm, NeurIPS, 2014. [6] Zhao & Zhang, Stochastic Optimization with Importance Sampling for Regularized Loss Minimization, ICML, 2015.