- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates when exponential moving average (EMA) momentum benefits reinforcement learning with verifiable rewards (RLVR). Using a KL-regularized bandit formulation, the authors theoretically show that momentum accelerates optimization under exact gradients, while its expected benefit under sampled gradients can be obscured by stochastic fluctuations. The analysis further suggests adjusting the momentum coefficient with the learning rate to preserve optimization gains. Experiments on simulated bandits and RLVR training with Qwen3-4B and Qwen2.5-32B provide empirical support for using larger momentum coefficients.
While the theoretical analysis provides useful insights, its connection to practical RLVR optimization remains insufficiently established. In addition, the relatively limited empirical evaluation leaves questions about the generalizability of the proposed momentum scaling principle.
Critical strengths
-
Meaningful theoretical contribution. The paper provides a principled analysis of why conventional momentum settings may offer limited benefits in RLVR. The characterization of deterministic acceleration and stochastic fluctuations offers useful insights into momentum-based optimization.
-
Promising practical implications. The proposed momentum coefficient selection principle is simple and potentially useful. The experiments on Qwen3-4B and Qwen2.5-32B demonstrate encouraging performance improvements without modifying the underlying RLVR objective.
Critical weaknesses
-
Significant gap between theoretical analysis and practical optimization. The theoretical guarantees are derived under a simplified KL-regularized bandit formulation using natural policy gradients, whereas the practical experiments employ GRPO with SGD or AdamW. In particular, the extremely large momentum coefficient used in the 32B experiment is not clearly justified by the derived theoretical conditions. Therefore, it remains unclear to what extent the theoretical results explain the improvements observed in practical LLM training.
-
Insufficient experimental validation of the proposed scaling principle. Although the paper presents detailed theoretical analysis, the practical experiments are relatively limited. The LLM evaluations focus on mathematical reasoning using DAPO-17k, with validation data drawn from the same distribution. Moreover, only a small number of momentum coefficients are evaluated, making it difficult to determine whether the proposed scaling principle consistently identifies effective settings across different optimization regimes. Stronger empirical evidence is needed to support the broader practical conclusions.
Questions for authors
-
Theorem 5.4 provides a sufficient condition on η/(1−β), whereas the 32B experiment uses a substantially larger ratio. Could the authors clarify how the theoretical analysis justifies this practical configuration, particularly given the differences between natural policy gradients and AdamW?
-
Could the authors provide a more systematic analysis of momentum coefficients and learning rates to demonstrate whether the proposed scaling principle reliably predicts practical optimization performance, rather than simply indicating that larger coefficients can be beneficial?
-
Do the observed improvements generalize to independent mathematical reasoning benchmarks or other RLVR tasks beyond the held-out DAPO-17k validation set?
4 comments
Response to Reviewer 8huV (Part 2)
Generalization to external mathematics benchmarks and other RLVR tasks [W2&Q3]
Mathematics benchmarks. We evaluate DAPO-trained Qwen3-4B-Base on MATH-500, AIME24, AIME25, AIME26, and AMC23. The tables report mean accuracy (%) across 3 training seeds. Accuracy is averaged over 8 responses per problem for MATH-500 and 32 for other benchmarks. We evaluate step 150 for all settings except AdamW with , for which we use step 100 because its DAPO validation curve approximately plateaus by then (Figure 2(b)).
| β | MATH-500 | AIME | AMC23 | Avg. |
|---|---|---|---|---|
| 0 | 82.06 | 23.32 | 70.96 | 58.78 |
| 0.9 | 81.74 | 22.35 | 69.90 | 58.00 |
| 0.9992 | 83.45 | 24.86 | 73.12 | 60.48 |
(a) SGD and EMA.
| β₁ | MATH-500 | AIME | AMC23 | Avg. |
|---|---|---|---|---|
| 0 | 77.77 | 23.52 | 72.66 | 57.98 |
| 0.9 | 79.92 | 23.66 | 71.93 | 58.50 |
| 1−10⁻⁶ | 79.57 | 27.01 | 75.73 | 60.77 |
(b) AdamW.
AIME is the mean of AIME24, AIME25, and AIME26. Avg. is the mean of MATH-500, AIME, and AMC23.
Larger momentum improves all three benchmark groups with SGD–EMA. With AdamW, it outperforms the baseline without first-moment momentum on all benchmarks, while outperforming the baseline on AIME and AMC23 and maintaining comparable MATH-500 accuracy.
Coding. We train and evaluate Qwen3-4B-Base on the LeetCode subsets of Code-R1’s official training and validation dataset, respectively. Training uses GRPO with AdamW ().
| First-moment coefficient | Step 20 | Step 40 | Step 60 | Step 80 | Step 100 | Step 120 | Step 140 | Step 150 |
|---|---|---|---|---|---|---|---|---|
| 47.40 | 48.77 | 49.88 | 50.60 | 52.23 | 52.77 | 53.42 | 53.00 | |
| 47.83 | 48.73 | 49.23 | 50.02 | 50.96 | 51.81 | 52.29 | 52.58 | |
| 48.38 | 51.27 | 52.19 | 54.38 | 56.15 | 56.58 | 57.10 | 57.50 |
Conventional momentum () performs similarly to the baseline without first-moment momentum (), while the larger coefficient outperforms both. At step 150, the larger coefficient improves accuracy by nearly 5 percentage points over conventional momentum and 4.5 percentage points over the baseline without first-moment momentum, extending the empirical support to coding.
We have added these results to revised Section 6.2.
Systematic analysis of momentum coefficients and learning rates [Q2]
Although our current results already establish the benefits of larger momentum coefficients in theory and practice, we agree that a more systematic analysis of momentum coefficients and learning rates could provide more detailed predictions. We include this as future work in the revision.
Dear Reviewer, Thank you again for your thoughtful feedback. As the discussion period is coming to an end, please let us know if you have any remaining concerns. We would be happy to provide further clarification.
Thank you for addressing my concerns. I maintain my recommendation of Weak Accept (Rating: 3).
Response to Reviewer 8huV (Part 1)
We thank the reviewer for recognizing our theoretical contribution and its practical implications. We hope the clarifications and additional experimental evidence below address your concerns.
Connection between the theoretical analysis and practical optimization [W1&Q1]
We acknowledge the gap between our theoretical model and practical LLM training, a common challenge in RL and ML theory. Our bandit model nevertheless captures reward maximization and sampling from an evolving policy, which distinguish RLVR from pretraining and SFT. Such bandit formulations are widely used in theoretical studies of LLM post-training [e.g., 1–3], and NPG is a common analytical abstraction for GRPO-style updates [e.g., 3–6].
The central observation of our analysis is that η/(1−β) must remain non-negligible to preserve a nonvanishing benefit from momentum. At small learning rates, this motivates choosing a momentum coefficient substantially above the conventional fixed value (e.g., 0.9) in LLM RLVR settings. To validate this principle, we conduct comprehensive experiments. Our GRPO experiments across Qwen3-4B-Base and Qwen2.5-32B, including both SGD–EMA and AdamW, provide empirical evidence that the coefficient selection principle generalizes beyond the analytical model to practical LLM optimization.
For the choice of β1 in the 32B experiment, Theorem 5.4 provides a conservative sufficient condition specific to our theoretical model. It guides how momentum should scale with the learning rate, while the specific coefficient for practical LLM training still requires empirical tuning. Our LLM experiments show that larger momentum coefficients generally improve performance. These results provide empirical evidence that our principle can be applied flexibly in practical LLM training.
[1] Xiong et al., 2024, “Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint”.
[2] Dang et al., 2025, “Weight Ensembling Improves Reasoning in Language Models”.
[3] Gai et al., 2026, “Demystifying Entropy Control in LLM RL Training: Theoretical Analysis and Dynamic Scheduling”.
[4] Cui et al., 2025, “The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models”.
[5] Park et al., 2025, “Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models”.
[6] Chen et al., 2026, “Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward”.