acceptodds
Under review as a conference paper at ICLR 2027

When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?

Abstract

Momentum is a classical tool for accelerating convex optimization and a standard component of modern large language model training. Yet recent empirical work has questioned the usefulness of first-moment momentum in reinforcement learning with verifiable rewards (RLVR). We study this question through a bandit formulation of RLVR with bias-corrected exponential moving average (EMA) momentum, where the EMA coefficient controls how quickly the weights on past gradients decay. When policy gradients are evaluated exactly, we prove that increasing within a range that prevents overshoot strictly accelerates optimization. When gradients are instead estimated from sampled responses, a positive expected gain can be obscured by sampling fluctuations. We show that as the learning rate decreases, the expected gain with fixed vanishes faster than these fluctuations, whereas increasing to keep within the safe regime preserves a nonvanishing gain that eventually dominates them. Bandit simulations quantitatively validate these predictions. RLVR experiments on Qwen3-4B-Base using SGD with EMA and AdamW support our principle of choosing larger coefficients in practical LLM post-training. On Qwen2.5-32B with AdamW, simply increasing improves mean validation accuracy by nearly percentage points over the conventional choice, demonstrating the principle's effectiveness at a larger model scale.

Primary area:
reinforcement learning
Keywords:
Momentum, Reinforcement Learning, RLVR, Stochastic Optimization, Large Language Model Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8huVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates when exponential moving average (EMA) momentum benefits reinforcement learning with verifiable rewards (RLVR). Using a KL-regularized bandit formulation, the authors theoretically show that momentum accelerates optimization under exact gradients, while its expected benefit under sampled gradients can be obscured by stochastic fluctuations. The analysis further suggests adjusting the momentum coefficient with the learning rate to preserve optimization gains. Experiments on simulated bandits and RLVR training with Qwen3-4B and Qwen2.5-32B provide empirical support for using larger momentum coefficients.

While the theoretical analysis provides useful insights, its connection to practical RLVR optimization remains insufficiently established. In addition, the relatively limited empirical evaluation leaves questions about the generalizability of the proposed momentum scaling principle.

Critical strengths
  1. Meaningful theoretical contribution. The paper provides a principled analysis of why conventional momentum settings may offer limited benefits in RLVR. The characterization of deterministic acceleration and stochastic fluctuations offers useful insights into momentum-based optimization.

  2. Promising practical implications. The proposed momentum coefficient selection principle is simple and potentially useful. The experiments on Qwen3-4B and Qwen2.5-32B demonstrate encouraging performance improvements without modifying the underlying RLVR objective.

Critical weaknesses
  1. Significant gap between theoretical analysis and practical optimization. The theoretical guarantees are derived under a simplified KL-regularized bandit formulation using natural policy gradients, whereas the practical experiments employ GRPO with SGD or AdamW. In particular, the extremely large momentum coefficient used in the 32B experiment is not clearly justified by the derived theoretical conditions. Therefore, it remains unclear to what extent the theoretical results explain the improvements observed in practical LLM training.

  2. Insufficient experimental validation of the proposed scaling principle. Although the paper presents detailed theoretical analysis, the practical experiments are relatively limited. The LLM evaluations focus on mathematical reasoning using DAPO-17k, with validation data drawn from the same distribution. Moreover, only a small number of momentum coefficients are evaluated, making it difficult to determine whether the proposed scaling principle consistently identifies effective settings across different optimization regimes. Stronger empirical evidence is needed to support the broader practical conclusions.

Questions for authors
  1. Theorem 5.4 provides a sufficient condition on η/(1−β), whereas the 32B experiment uses a substantially larger ratio. Could the authors clarify how the theoretical analysis justifies this practical configuration, particularly given the differences between natural policy gradients and AdamW?

  2. Could the authors provide a more systematic analysis of momentum coefficients and learning rates to demonstrate whether the proposed scaling principle reliably predicts practical optimization performance, rather than simply indicating that larger coefficients can be beneficial?

  3. Do the observed improvements generalize to independent mathematical reasoning benchmarks or other RLVR tasks beyond the held-out DAPO-17k validation set?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8huV (Part 1)

We thank the reviewer for recognizing our theoretical contribution and its practical implications. We hope the clarifications and additional experimental evidence below address your concerns.

Connection between the theoretical analysis and practical optimization [W1&Q1]

We acknowledge the gap between our theoretical model and practical LLM training, a common challenge in RL and ML theory. Our bandit model nevertheless captures reward maximization and sampling from an evolving policy, which distinguish RLVR from pretraining and SFT. Such bandit formulations are widely used in theoretical studies of LLM post-training [e.g., 1–3], and NPG is a common analytical abstraction for GRPO-style updates [e.g., 3–6].

The central observation of our analysis is that η/(1−β)\eta/(1-\beta) must remain non-negligible to preserve a nonvanishing benefit from momentum. At small learning rates, this motivates choosing a momentum coefficient substantially above the conventional fixed value (e.g., 0.9) in LLM RLVR settings. To validate this principle, we conduct comprehensive experiments. Our GRPO experiments across Qwen3-4B-Base and Qwen2.5-32B, including both SGD–EMA and AdamW, provide empirical evidence that the coefficient selection principle generalizes beyond the analytical model to practical LLM optimization.

For the choice of β1\beta_1 in the 32B experiment, Theorem 5.4 provides a conservative sufficient condition specific to our theoretical model. It guides how momentum should scale with the learning rate, while the specific coefficient for practical LLM training still requires empirical tuning. Our LLM experiments show that larger momentum coefficients generally improve performance. These results provide empirical evidence that our principle can be applied flexibly in practical LLM training.

[1] Xiong et al., 2024, “Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint”.

[2] Dang et al., 2025, “Weight Ensembling Improves Reasoning in Language Models”.

[3] Gai et al., 2026, “Demystifying Entropy Control in LLM RL Training: Theoretical Analysis and Dynamic Scheduling”.

[4] Cui et al., 2025, “The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models”.

[5] Park et al., 2025, “Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models”.

[6] Chen et al., 2026, “Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward”.

AuthorsOpenReview ↗

Response to Reviewer 8huV (Part 2)

Generalization to external mathematics benchmarks and other RLVR tasks [W2&Q3]

Mathematics benchmarks. We evaluate DAPO-trained Qwen3-4B-Base on MATH-500, AIME24, AIME25, AIME26, and AMC23. The tables report mean accuracy (%) across 3 training seeds. Accuracy is averaged over 8 responses per problem for MATH-500 and 32 for other benchmarks. We evaluate step 150 for all settings except AdamW with β1=1−10−6\beta_1=1-10^{-6}, for which we use step 100 because its DAPO validation curve approximately plateaus by then (Figure 2(b)).

βMATH-500AIMEAMC23Avg.
082.0623.3270.9658.78
0.981.7422.3569.9058.00
0.999283.4524.8673.1260.48

(a) SGD and EMA.

β₁MATH-500AIMEAMC23Avg.
077.7723.5272.6657.98
0.979.9223.6671.9358.50
1−10⁻⁶79.5727.0175.7360.77

(b) AdamW.

AIME is the mean of AIME24, AIME25, and AIME26. Avg. is the mean of MATH-500, AIME, and AMC23.

Larger momentum improves all three benchmark groups with SGD–EMA. With AdamW, it outperforms the baseline without first-moment momentum on all benchmarks, while outperforming the β1=0.9\beta_1=0.9 baseline on AIME and AMC23 and maintaining comparable MATH-500 accuracy.

Coding. We train and evaluate Qwen3-4B-Base on the LeetCode subsets of Code-R1’s official training and validation dataset, respectively. Training uses GRPO with AdamW (η=10−6\eta=10^{-6}).

First-moment coefficientStep 20Step 40Step 60Step 80Step 100Step 120Step 140Step 150
β1=0\beta_1=047.4048.7749.8850.6052.2352.7753.4253.00
β1=0.9\beta_1=0.947.8348.7349.2350.0250.9651.8152.2952.58
β1=0.999999\beta_1=0.99999948.3851.2752.1954.3856.1556.5857.1057.50

Conventional momentum (β1=0.9\beta_1=0.9) performs similarly to the baseline without first-moment momentum (β1=0\beta_1=0), while the larger coefficient outperforms both. At step 150, the larger coefficient improves accuracy by nearly 5 percentage points over conventional momentum and 4.5 percentage points over the baseline without first-moment momentum, extending the empirical support to coding.

We have added these results to revised Section 6.2.

Systematic analysis of momentum coefficients and learning rates [Q2]

Although our current results already establish the benefits of larger momentum coefficients in theory and practice, we agree that a more systematic analysis of momentum coefficients and learning rates could provide more detailed predictions. We include this as future work in the revision.

Dear Reviewer, Thank you again for your thoughtful feedback. As the discussion period is coming to an end, please let us know if you have any remaining concerns. We would be happy to provide further clarification.

Thank you for addressing my concerns. I maintain my recommendation of Weak Accept (Rating: 3).

Reviewer_N79nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the optimizer design for LLM post training and investigates whether the classical momentum can help in reinforcement learning with verifiable rewards (RLVR). Specifically, this work studies this by taking a bandit formulation of RLVR and considers bias-corrected exponential moving average (EMA) momentum. The main contributions are: (1) This paper shows that with exact gradients, increasing the EMA coefficient within a safe regime can accelerate optimization. The authors further extend the results to the case of stochastic gradient. (2) Algorithmically, this work provides a principled way to select the momentum coefficients. This paper also empirically validates the predicted gain and fluctuation scales in numerical bandit experiments and on math problems by using GRPO on the Qwen3-4B-Base model and Qwen2.5-32B.

Critical strengths
  • This paper studies an important and fundamental problem in the optimizer design of LLM post training.

  • The theoretical results in Section 5 offer a useful and meaningful insight: If deterministic momentum can lead to a gain bounded away from zero, then sufficiently small sampling fluctuations preserve that gain in expectation. In the analysis, this is established by separating expected improvement from reliable improvement.

  • The toy experiments in Figure 1 are helpful in corroborating the benefit of the EWA momentum mechanism and the proposed scaling regime.

Critical weaknesses
  • My main concern is that the bandit model is too simplified for LLM post training and makes momentum unusually favorable. Specifically, from Lemma 4.1, we can see that all deterministic gradients are perfectly aligned, and optimization essentially reduces to a scalar residual recurrence. Therefore, momentum can benefit from averaging the earlier gradients without seeing changing gradient directions.

  • Moreover, there is a substantial gap between the theoretical analysis and the experiments. The NPG algorithm used in the analysis differs substantially from the algorithm (GRPO, which is closer to the standard policy gradient) used in the experiments.

  • While the analysis offers partial characterization of the effect of momentum, I found that it remains somewhat inconclusive whether momentum can generally help for RLVR in practice. Specifically: (1) In the analysis, this work considers tabular softmax NPG with a specific estimator, which does not really reflect the actual optimizer used in LLM post training. (2) In Lemma 5.1, the second term in RHS can possibly be negative, and hence it is not clear how small η\eta shall be to obtain positive gain under stochastic EWA momentum. (3) Similarly, in Theorem 5.4, the expected gain is established as η→0\eta\rightarrow 0. These results provide asymptotics for small learning rates, without an explicit practically useful threshold.

  • Another concern is that a similar analysis of the momentum mechanism has been studied in the literature (Wang et al., 2024). Specifically, (Wang et al., 2024) has studied the role of momentum in stochastic gradient settings where the learning rate is small and gradient noise is the dominant source of instability. In Sections 3 and 4 of (Wang et al., 2024), from the perspective of SDE, they characterized the first moment and second moment under weak approximation. This appears closely related to the analysis in this work.

  • Regarding the experiments, one issue is that this work uses large momentum coefficients (β\beta very close to 1), and this intuitively leads to something like cumulative gradient averaging. As a result, it is unclear whether the performance gain actually comes from the proposed scaling regime of learning-rate and momentum, or it comes primarily from long-memory gradient averaging?

  • The experiment scope is somewhat limited as the LLM experiments use only one mathematical RLVR task and relatively sparse momentum comparisons. To strengthen the results, it would be good to evaluate under joint sweeps of learning rate and momentum and other tasks or even model families (e.g., code generation).

Given the above concerns, I lean towards weak rejection for now, but I will be willing to adjust the score if the above concerns can be sufficiently addressed.

Reference:

Wang et al., "The marginal value of momentum for small learning rate SGD," ICLR 2024.

Questions for authors
  • Can the authors further justify or explain how the conclusion from a simplified bandit model can be carried over to the practical LLM post training setting?

  • The theoretical results are mainly about the asymptotics. Can the authors comment on the non-asymptotic regime? This can provide more useful guidance to LLM post training in practice.

  • Can the authors clarify the connection between this work and (Wang et al., 2024)?

  • Does the analysis and the proposed scaling regime still hold on tasks with dense rewards? As far as I understand, the analysis does not seem to rely on the RLVR assumption?

4 comments
AuthorsOpenReview ↗

Response to Reviewer N79n (Part 1)

We thank the reviewer for the constructive feedback. Below, we clarify our theoretical results and their practical relevance, and provide additional experiments. We would appreciate your reconsideration of our paper in light of these clarifications and new results.

Link between the theoretical model and practical LLM training [W1&W2&Q1]

We acknowledge the gap between our simplified theoretical model and practical LLM training. Such gaps are a common challenge in RL and ML theory, where tractable models are used to isolate key mechanisms. Nevertheless, our bandit formulation already captures two central features of RLVR that distinguish it from pretraining and SFT: a reward maximization objective and sampling from the evolving policy. This bandit model and NPG are widely used abstractions for RLVR in LLM post-training [e.g., 1-3] and GRPO-style policy updates [e.g., 3–6], respectively.

We agree with the reviewer that the aligned exact NPG dynamics make the deterministic momentum mechanism explicit. However, sampled gradients need not remain aligned, and sampling fluctuations can obscure a positive deterministic gain. Our analysis shows that keeping η/(1−β)\eta/(1-\beta) at a fixed positive value within the safe regime preserves a nonvanishing expected gain. This motivates choosing momentum coefficients substantially above the conventional fixed value (e.g., 0.9).

Given the gap between the theoretical model and practice, we conduct comprehensive GRPO experiments across model scales and optimizers, using Qwen3-4B-Base and Qwen2.5-32B with SGD–EMA and AdamW. These experiments use practical GRPO updates rather than the analyzed NPG updates and do not impose gradient alignment. Their results provide empirical evidence that the coefficient-selection principle generalizes beyond the analytical model to practical LLM optimization.

[1] Xiong et al., 2024, “Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint”.

[2] Dang et al., 2025, “Weight Ensembling Improves Reasoning in Language Models”.

[3] Gai et al., 2026, “Demystifying Entropy Control in LLM RL Training: Theoretical Analysis and Dynamic Scheduling”.

[4] Cui et al., 2025, “The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models”.

[5] Park et al., 2025, “Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models”.

[6] Chen et al., 2026, “Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward”.

Asymptotics of the theoretical results [W3&Q2]

Although the second term in Lemma 5.1 can be negative, Theorems 5.3 and 5.4 already guarantee positive expected gain under their respective assumptions for all sufficiently small learning rates 0<η<η00<\eta<\eta_0, for some problem-dependent η0>0\eta_0>0. Using the moment and approximation estimates in Appendices C, E, and G, together with explicit bounds on the derivatives of JJ, we can quantify the remainder terms and obtain an explicit threshold.

Our theoretical analysis aims to identify a coefficient-selection principle and explain why conventional momentum can have little observable benefit in practice. This principle guides how momentum should scale with the learning rate, while specific hyperparameter values for LLM training still require empirical tuning. Even an explicit non-asymptotic bound would offer limited additional guidance on these numerical choices, as its threshold would be conservative and specific to our analytical model. The asymptotic formulation already provides the key insight in a concise form: with fixed β\beta, the expected gain vanishes faster than sampling fluctuations, whereas keeping η/(1−β)\eta/(1-\beta) at a fixed positive value within the safe regime preserves a nonvanishing expected gain.

Novelty and connection with Wang et al. (2024) [W4&Q3]

Wang et al. (2024) study momentum in general stochastic optimization, with experiments on supervised learning, while our work focuses on the RLVR setting. Their main results establish equivalence between SGD and SGDM in the small-learning-rate limit when 1−β=Θ(ηα)1-\beta=\Theta(\eta^\alpha) with α<1\alpha<1. This corresponds to η/(1−β)→0\eta/(1-\beta)\to0 and is consistent with our vanishing-gain results. Our fixed-β\beta analysis further quantifies why sampling fluctuations obscure the positive expected gain.

For α=1\alpha=1, they identify a non-negligible effect of momentum on the limiting dynamics but do not quantify the objective gain. We address this question for bias-corrected EMA in our RLVR model. Within the safe range, we prove a strictly positive, nonvanishing expected objective gain and show that it dominates sampling fluctuations, thereby motivating larger momentum coefficients. We have clarified these distinctions alongside the corresponding results in revised Section 5.

AuthorsOpenReview ↗

Response to Reviewer N79n (Part 2)

Large momentum and long-memory gradient averaging [W5]

At update tt, bias-corrected EMA assigns the gradient from step t−kt-k a weight βk\beta^k times that of the current gradient. For any fixed tt, these normalized weights converge to 1/(t+1)1/(t+1) as β→1\beta\to1, yielding uniform averaging of all gradients observed so far. Long-memory averaging is thus the mechanism implemented by large-β\beta momentum. With exact gradients, increasing β\beta within the safe range improves the objective (Theorem 4.2). Our analysis of the joint scaling of η\eta and 1−β1-\beta explains when this gain is reliably observable under sampling noise.

Generalization [W6&Q4]

Code generation. We additionally conduct GRPO experiments using LeetCode programming problems from the Code-R1 dataset, removing train–validation overlap and moving some training problems into validation to reach 200 problems. We use Qwen3-4B-Base and AdamW. We fix η=10−6\eta=10^{-6}, β2=0.999\beta_2=0.999 and compare β1∈{0,0.9,1−10−6}\beta_1\in\{0,0.9,1-10^{-6}\}. The table reports mean validation accuracy (%).

First-moment coefficientStep 20Step 40Step 60Step 80Step 100Step 120Step 140Step 150
β1=0\beta_1=047.4048.7749.8850.6052.2352.7753.4253.00
β1=0.9\beta_1=0.947.8348.7349.2350.0250.9651.8152.2952.58
β1=0.999999\beta_1=0.99999948.3851.2752.1954.3856.1556.5857.1057.50

Conventional momentum (β1=0.9\beta_1=0.9) performs similarly to the baseline without first-moment momentum (β1=0\beta_1=0), while the larger coefficient outperforms both. After 150 updates, the larger coefficient improves mean accuracy by nearly 5 percentage points over conventional momentum and 4.5 points over the baseline without first-moment momentum. These additional results extend the empirical support for larger momentum coefficients beyond mathematical reasoning. We have added these results to revised Section 6.2.

Joint sweeps of learning rate and momentum coefficient. The practical learning rate range is narrow. For SGD, our preliminary experiments show that 10−110^{-1} leads to unstable training while 10−310^{-3} yields negligible improvement within the training budget (see the table below). For AdamW, the common choices are close to 10−610^{-6}. Thus, the learning rates reported in our paper (1×10−21\times 10^{-2} and 2×10−22\times10^{-2} for SGD, 1×10−61\times 10^{-6}, 2×10−62\times 10^{-6} and 5×10−75\times10^{-7} for AdamW) are representative. The table reports mean validation accuracy (%) from single-seed preliminary experiments.

Learning rate / Step020406080100120140160
10−110^{-1}48.1352.2353.4352.0852.1349.3836.7838.7836.18
10−310^{-3}49.0048.9048.7148.4348.8949.1149.3149.4148.84

For the momentum sweep, a denser grid would help characterize parameter sensitivity. Given the computational cost of LLM RL training, we prioritize evaluating theory-guided choices across learning rates, optimizers, and model scales. Our existing results across these settings support the practical benefit of choosing momentum substantially above the conventional value of 0.90.9. We leave more extensive joint sweeps to future work.

Dense rewards. When dense process rewards are aggregated into a trajectory-level return, they simply redefine the reward assigned to each arm in our bandit formulation. As our analysis does not rely on binary rewards, our main gain and fluctuation claims can generalize to dense process rewards RL settings. Although the use of process rewards in GRPO has been explored, to our knowledge, there is not yet a widely adopted, standardized recipe for dense rewards in RLVR. Thus, we focus our model and analysis on the standard RLVR setting with sparse reward.

Dear Reviewer, Thank you again for your thoughtful feedback. As the discussion period is coming to an end, please let us know if you have any remaining concerns. We would be happy to provide further clarification.

Thanks to the authors for the detailed response and for answering my questions. I appreciate the clarifications regarding (Wang et al. 2024) and dense rewards. The new code-generation results are encouraging and partly alleviate my concern about the scope of the empirical evaluation.

However, the following concerns still remain:

  1. Gap between theory and experiments: The theory analyzes tabular softmax NPG with a specific gradient estimator, whereas the experiments use GRPO with SGD–EMA or AdamW. It remains unclear to me which features of the theoretical dynamics carry over to these updates and why the proposed scaling should predict performance in this setting. The empirical gains are encouraging, but I believe it is also important to understand what drives them and under what conditions they can be expected.

  2. Perfect gradient alignment in the bandit model: All exact NPG gradients are perfectly aligned, reducing the optimization dynamics to a scalar recurrence. This can make momentum unusually favorable because past gradients point in the same direction throughout training. The rebuttal notes that bandit models have been used in the LLM literature. My concern is specifically about how much the gain depends on the perfect alignment in this model. The discussion of sampling noise is somewhat relevant, but still does not address what happens when the exact gradient direction changes.

  3. Regarding the asymptotics of theoretical results: The response clarifies that a problem-dependent threshold exists. I understand that the theoretical results only offer asymptotics, but some discussions about the non-asymptotic behavior would help show how small the learning rate must be for the expected gain to emerge.

  4. Joint sweep of learning rate and momentum: This appears unaddressed, though I understand that the RL post-training can require some time and computes. That said, even a small joint sweep with a cumulative-average baseline would help determine whether the proposed ratio predicts performance beyond the benefit of averaging over a long gradient history.

Given these remaining concerns, I am inclined to maintain my current score.

Reviewer_85CbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks when first-moment (EMA) momentum helps in RLVR. It analyzes natural policy gradient with bias-corrected EMA in a single-prompt, KL-regularized, response-level softmax bandit. With exact gradients, all NPG directions are collinear, and increasing β\beta within a safe range (β≤((1−ηγ)/(1+ηγ))2\beta \le ((1-\eta\gamma)/(1+\eta\gamma))^2) strictly improves the final objective. With sampled gradients, in the limit η→0\eta\to0 at fixed optimization time τ=ηT\tau=\eta T, the picture depends on how β\beta is chosen. A fixed β\beta yields an O(η)O(\eta) expected gain, which is swamped by O(η/B)O(\sqrt{\eta/B}) run-to-run noise, so the win probability tends to 1/2. Scaling β\beta so that η/(1−β)=ρ≤1/(4γ)\eta/(1-\beta)=\rho\le1/(4\gamma) yields a Θ(1)\Theta(1) gain that dominates the noise. Pairing a larger β\beta with a smaller η\eta can match expected performance with lower asymptotic variance. Bandit simulations (50k replicates) confirm the predicted scalings. GRPO runs on Qwen3-4B-Base (SGD+EMA and AdamW, 3 seeds, ≤200 steps) show that much larger coefficients beat β=0.9\beta=0.9 on held-out DAPO-17k accuracy. On Qwen2.5-32B, β1=1−10−9\beta_1=1-10^{-9} reaches 46.1% vs. 44.3% for β1=0.9\beta_1=0.9.

Critical strengths
  1. Momentum settings in RLVR are inherited from pretraining with little scrutiny, and recent work reports no consistent benefit from momentum. The paper offers a clean, testable explanation for this: with a fixed β\beta, the gain is O(η)O(\eta) while run-to-run noise is O(η/B)O(\sqrt{\eta/B}), so the benefit is hidden by noise. It also gives a constructive alternative: scaling β\beta with η\eta. This explanation is the main value of the paper.

  2. The stochastic estimator uses inverse-probability weighting, which produces heavy-tailed noise. The paper handles this with an auxiliary stopped process and Freedman's inequality. It obtains expansions that are uniform in β\beta, gives explicit constants, and proves a characteristic-function CLT for the win probability. This goes beyond a heuristic signal-to-noise argument.

  3. Across four reward/initialization settings with 50k replicates, the predicted η\sqrt{\eta} noise scaling and η\eta vs. Θ(1)\Theta(1) gain scaling are clearly visible. At η=1/4096\eta=1/4096, the SNR is 0.045 for fixed β\beta vs. 3.76 for scaled β\beta (3:7, uniform; Table 1).

Critical weaknesses
  1. Novelty relative to prior momentum theory is not clearly positioned. It is known that fixed-β\beta momentum behaves like SGD at small learning rates (Yuan et al., 2016; Wang et al., 2024, both cited). η/(1−β)\eta/(1-\beta) is a standard "effective learning rate", and scaling β\beta with η\eta appears in the SDE/scaling-rule literature. The new contribution is the specialization to a KL-regularized RLVR bandit with explicit constants. The paper should state this directly and compare against that literature.

  2. The theoretical model has a special structure that makes momentum trivially beneficial. The model has a single prompt, tabular logits over the full response set, and an exact KL term and Fisher matrix. Under these conditions, the NPG direction is exactly γ(θ⋆−θ)\gamma(\theta^\star-\theta) (Lemma 4.1). Every gradient therefore points the same way with shrinking magnitude, like a 1-D quadratic with condition number 1. In that geometry, averaging earlier and larger gradients must accelerate progress as long as it does not overshoot. Real RLVR has many prompts sharing parameters, so successive gradients point in very different directions. It also has a non-stationary policy, anisotropic curvature, PPO/GRPO clipping, group-normalized advantages, and Adam preconditioning. None of these are modeled, so it is unclear which conclusions transfer.

  3. The comparison at a fixed η\eta does not answer the practical question. In a linear contraction, the acceleration gained from a larger β\beta can also be obtained by increasing η\eta. Thms. 4.2–5.4 compare runs at the same η\eta; they do not show that the best-tuned (η,β)(\eta,\beta) beats the best-tuned (η,0)(\eta,0). Thm. 5.5 is the only result that addresses this tradeoff. Its variance ratio is (1+1−4γρ2)/(1+1−4γρ1)(1+\sqrt{1-4\gamma\rho_2})/(1+\sqrt{1-4\gamma\rho_1}), which is at least 1/2 for any admissible ρ2\rho_2. So the variance at best halves, and only in the double limit η→0\eta\to0, τ→∞\tau\to\infty.

  4. The asymptotic regime does not match practice. All stochastic results take η→0\eta\to0 with τ=ηT\tau=\eta T fixed, i.e., T→∞T\to\infty. The LLM runs use T=150T=150–200200 steps. Nothing indicates how large TT must be before the asymptotics apply.

  5. The LLM experiments test a degenerate optimizer, not the safe-regime EMA from the theory. With T≤200T\le200, every large coefficient used (0.99920.9992, 0.99960.9996, 1−10−61-10^{-6}, 1−10−91-10^{-9}) makes the bias-corrected EMA weights nearly uniform. For example, 0.9992150≈0.890.9992^{150}\approx0.89, and for 1−10−61-10^{-6} the weights differ by under 0.02% across the run. The update is therefore essentially the running average of all gradients since step 0, a different method from the finite-memory EMA analyzed. On Qwen2.5-32B, ρ=η/(1−β1)=2×10−6/10−9=2000\rho=\eta/(1-\beta_1)=2\times10^{-6}/10^{-9}=2000, far beyond the safe bound 1/(4γ)=251/(4\gamma)=25 (KL coefficient γ=0.01\gamma=0.01). This run is outside the regime the theory guarantees. The SGD rule η/(1−β)=1/(4γ)\eta/(1-\beta)=1/(4\gamma) also equates an NPG step size in logit space with an SGD step size on transformer parameters, which have different units.

  6. The empirical evidence is thin. The study uses only 3 seeds and at most 200 steps of 16 prompts each (≤3,200 prompts, under 20% of one epoch of DAPO-17k). Evaluation is only in-distribution held-out accuracy, with no downstream benchmarks such as AIME, MATH-500 or AMC. There is no learning-rate sweep for the β1=0.9\beta_1=0.9 or β=0\beta=0 baselines. The headline 32B gain (46.1 vs. 44.3) is comparable to the ±1 s.d. error bars in Fig. 3. The "more stable training" claim (Fig. 2c,d) rests on the spread of 3 seeds.

  7. Stale gradients are not examined. With near-uniform averaging, gradients from the step-1 policy keep the same weight as current gradients throughout training. The theory cannot reveal this risk, because in its model every past gradient points in the current descent direction. In real RLVR, stale gradients from a very different policy could hurt later training, and the short runs cannot show this.

Questions for authors
  1. Can you sweep the learning rate (e.g., 4–5 values) for each β\beta setting, including β1=0.9\beta_1=0.9 and β1=0\beta_1=0, and compare the best-tuned configurations? If the large-β\beta advantage disappears once η\eta is tuned, the practical recommendation changes substantially.

  2. Do the gains persist with longer training (e.g., ≥1,000 steps or a full epoch) and on downstream benchmarks (AIME, MATH-500, AMC)? With near-uniform averaging, early gradients from a very different policy keep full weight. Does performance eventually degrade?

  3. Your large coefficients make the EMA effectively a uniform running average over the whole run. Can you test intermediate values whose memory is shorter than TT (e.g., β1∈{0.95,0.99,0.995}\beta_1\in\{0.95, 0.99, 0.995\})? This would separate the effect of a larger finite-memory EMA, as in the theory, from cumulative averaging. Also, how do you reconcile the 32B run (ρ=2000≫1/(4γ)=25\rho=2000\gg1/(4\gamma)=25) with the safe-regime requirement?

  4. Can the analysis extend to multiple prompts sharing parameters (e.g., a log-linear policy), where gradient directions change across steps? Do strict acceleration and a meaningful safe range still hold there? This determines whether the theory explains the LLM results or only a special case.

  5. In Fig. 1, the mean gain at η=1/32\eta=1/32 appears negative for both schedules (where scaled β=0.875<0.9\beta=0.875<0.9). What causes this, and can you report the signed SNR in Table 1? Relatedly, does the variance analysis change qualitatively under GRPO-style group-normalized advantages instead of the importance-weighted estimator?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 85Cb (Part 1)

We thank the reviewer for the detailed review. Below, we provide clarifications and additional experimental evidence to address the concerns raised.

We aim to explain why momentum benefits can be difficult to observe in RLVR experiments and derive a principle for choosing the momentum coefficient. Our theoretical analysis provides explanations and insights, while the experiments are used to validate their practical implications.

Novelty relative to prior work [W1]

For sufficiently small learning rates, [1] assumes β\beta is bounded away from one, while [2] mainly discusses the regime 1−β=Θ(ηα)1-\beta=\Theta(\eta^\alpha) with α<1\alpha<1. Their results on the equivalence between momentum and SGD are consistent with our vanishing-gain claim. Our fixed-β\beta analysis further quantifies why sampling fluctuations obscure the positive expected gain.

For α=1\alpha=1, [2] identifies a non-negligible effect of momentum on the limiting dynamics but does not quantify the objective gain. We address this question for bias-corrected EMA in our RLVR model. Within the sufficient safe range, we prove a strictly positive, nonvanishing expected objective gain and show that it dominates sampling fluctuations. This provides a quantitative basis for choosing larger momentum coefficients at small learning rates. We have clarified these distinctions alongside the corresponding results in revised Section 5.

In particular, we use bias-corrected EMA to isolate the effect of temporal reallocation from changes in total gradient weight (Section 3.2). In our formulation, η/(1−β)\eta/(1-\beta) governs whether the effect of gradient reallocation vanishes, rather than representing an effective learning rate, as discussed in Section 5.

[1] Yuan et al., 2016, On the Influence of Momentum Acceleration on Online Learning.

[2] Wang et al., 2024, The Marginal Value of Momentum for Small Learning Rate SGD.

The theoretical model and its connection to practice [W2&Q4&Q5 on group normalization]

A reliably observable momentum benefit under sampling noise does not follow trivially from aligned exact gradients. Even in this model, sampling fluctuations can obscure a positive expected gain. Establishing when this happens and how to choose the momentum coefficient to preserve an observable benefit is the core of our analysis.

We acknowledge the gap between our simplified theoretical model and practical LLM training. Such gaps are a common challenge in RL and ML theory, where tractable models are used to isolate key mechanisms. Nevertheless, our model already captures reward maximization and sampling from an evolving policy, two central features of RLVR that distinguish it from pretraining and SFT. This bandit model and NPG are widely used abstractions for RLVR in LLM post-training [e.g., 3-5] and GRPO-style policy updates [e.g., 5–8], respectively.

To evaluate the coefficient selection principle beyond the theoretical model, we conduct LLM post-training experiments on Qwen3-4B-Base and Qwen2.5-32B using GRPO with SGD–EMA and AdamW. These experiments incorporate shared parameters, clipping, group-normalized advantages, and Adam preconditioning, without imposing gradient alignment. The results provide empirical evidence that the principle remains effective in practical LLM optimization when the modeling assumptions are relaxed.

[3] Xiong et al., 2024, “Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint”.

[4] Dang et al., 2025, Weight Ensembling Improves Reasoning in Language Models.

[5] Gai et al., 2026, Demystifying Entropy Control in LLM RL Training: Theoretical Analysis and Dynamic Scheduling.

[6] Cui et al., 2025, The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models.

[7] Park et al., 2025, Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models.

[8] Chen et al., 2026, Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward.

Fixed-learning-rate comparisons and learning-rate tuning [W3&Q1&W6 on learning-rate sweeps]

Our goal is to understand how EMA's temporal reallocation of gradient contributions affects RLVR optimization. To isolate this effect, we compare EMA settings with different coefficients at the same learning rate, initialization, batch size, and update count. Holding the learning rate fixed provides a fair, controlled comparison for isolating this effect. Comparing separately best-tuned (η\eta, β\beta) configurations is beyond the scope of our work.

Theorem 5.5 provides a complementary extension showing that momentum offers a stability benefit beyond acceleration. Increasing the learning rate can match the optimization progress achieved with larger momentum, but larger momentum paired with a smaller learning rate yields lower asymptotic variance. Figures 2(c,d) provide empirical support for this stability benefit.

AuthorsOpenReview ↗

Response to Reviewer 85Cb (Part 2)

Asymptotic guarantees and finite learning rates [W4&Q5 on negative mean gain]

Our main theorems (Theorems 5.3 and 5.4) explain why gains from fixed-coefficient momentum can be obscured by sampling fluctuations and how keeping η/(1−β)\eta/(1-\beta) at a fixed positive value within the safe range preserves observable gains. Under their respective assumptions, both theorems guarantee positive expected gain for all sufficiently small but finite learning rates. The asymptotic results already establish the distinction needed to motivate larger coefficients in practice. The bandit experiments validate the predicted scalings, and the LLM experiments validate the resulting coefficient selection principle in practical training.

Negative mean gain [Q5]. Our positive-gain guarantees apply to sufficiently small learning rates, so the negative mean gain at η=1/32\eta=1/32 does not contradict these results. The bandit simulations support the predicted gain and noise scalings as the learning rate decreases. We have updated Table 3 to report signed SNR values.

Large momentum, cumulative averaging, and the safe regime [W5&Q3]

EMA and cumulative averaging. At any fixed training horizon, cumulative gradient averaging is the limit of bias-corrected EMA as the momentum coefficient approaches one. There is therefore no need to distinguish their effects as those of two distinct algorithms.

Safe regime. Theorem 5.4 gives a sufficient condition under the theoretical model, rather than a strict requirement for practical LLM training. In the SGD experiments, we use the theoretical relation as a tuning heuristic. Our LLM experiments show that larger momentum coefficients generally help, providing empirical support for the coefficient selection principle beyond the theoretical model.

Additional experimental evidence [W6&Q2 on benchmarks]

External mathematics benchmarks. We evaluate DAPO-trained Qwen3-4B-Base on MATH-500, AIME24, AIME25, AIME26, and AMC23. The tables report mean accuracy (%) across 3 training seeds. Accuracy is averaged over 8 responses per problem for MATH-500 and 32 for other benchmarks. We evaluate step 150 for all settings except AdamW with β1=1−10−6\beta_1=1-10^{-6}, for which we use step 100 because its DAPO validation curve approximately plateaus by then (Figure 2(b)).

Larger momentum improves all three benchmark groups with SGD–EMA. With AdamW, it outperforms the baseline without first-moment momentum on all benchmarks, while outperforming the β1=0.9\beta_1=0.9 baseline on AIME and AMC23 and maintaining comparable MATH-500 accuracy.

βMATH-500AIMEAMC23Avg.
082.0623.3270.9658.78
0.981.7422.3569.9058.00
0.999283.4524.8673.1260.48

(a) SGD and EMA.

β₁MATH-500AIMEAMC23Avg.
077.7723.5272.6657.98
0.979.9223.6671.9358.50
1−10⁻⁶79.5727.0175.7360.77

(b) AdamW.

AIME is the mean of AIME24, AIME25, and AIME26. Avg. is the mean of MATH-500, AIME, and AMC23.

We have added these results to revised Section 6.2.

More seeds. We have run each configuration with three training seeds. Additional runs exceed our available compute budget during the rebuttal period, and we have acknowledged the limited precision of performance and variability estimates based on 3 seeds in revised Section 7, "Limitations and future work".

Longer training and stale gradients [W7&Q2]

Our experiments show stable training with larger β\beta up to the validation plateau, with no observed performance degradation despite preserving gradients from the early steps. If instability arises near convergence, resetting the momentum accumulator would be a simple option to investigate. Moreover, in practice, training stops at convergence because continuing to 1,000 steps or beyond is costly and offers no further benefit.

Dear Reviewer, Thank you again for your thoughtful feedback. As the discussion period is coming to an end, please let us know if you have any remaining concerns. We would be happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.