acceptodds
Under review as a conference paper at ICLR 2027

One Gradient, Yet More Robust Policies: Momentum-Guided Policy Optimization

Abstract

Robust reinforcement learning (Robust RL) aims to learn policies that remain effective under environmental uncertainty. Recent methods pursue such robustness by seeking flat reward maxima, often through sharpness-aware minimization (SAM), which encourages high returns under small parameter perturbations. However, vanilla SAM first constructs an adversarial parameter perturbation from a stochastic policy gradient and then computes a second gradient at the perturbed parameters to update the policy. This procedure relies on noisy gradient estimates and incurs additional computation, which can make it complex and unstable. In this paper, we introduce a policy optimization method, based on Momentum-SAM (mSAM), for Robust RL that utilizes accumulated momentum to construct adversarial perturbations. The resulting method is effective and computationally lightweight, requiring only one policy-gradient evaluation per update while reducing reliance on a single noisy estimate. Furthermore, we establish finite-time stationarity bounds for SAM and mSAM in policy optimization, providing a theoretical foundation for sharpness-aware policy learning. Our experiments show the effectiveness of mSAM in enhancing the robustness of RL compared with existing methods under diverse perturbations, including actions, transition dynamics, and reward functions.

Primary area:
reinforcement learning
Keywords:
Robust reinforcement learning, flatter reward landscapes, policy gradient optimization, sharpness-aware minimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qNe4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes applying Momentum-SAM (mSAM) to policy optimization in order to improve the robustness of RL policies while reducing the computational cost of sharpness-aware minimization (SAM). Instead of computing an extra gradient to construct the adversarial parameter perturbation, mSAM perturbs the parameters along the normalized accumulated momentum direction, requiring a single gradient evaluation per update. The authors integrate mSAM into PPO and provide finite-time stationarity bounds for both SAM and mSAM policy optimization. Experiments on three MuJoCo tasks (HalfCheetah, Hopper, Walker2d) evaluate robustness to action noise, friction and mass variations, and noisy training rewards, along with flatness measurements and a computational-efficiency comparison against PPO, PPO+SAM, PPO+NAG, RARL, and RNAC.

Critical strengths
  1. The method is simple and easy to integrate into existing policy-gradient pipelines, and it consistently reduces training time compared to PPO+SAM.
  2. The empirical evaluation is broad, covering action, dynamics, and reward perturbations.
  3. The theoretical analysis appears sound.
  4. The authors are candid about the limitations of their results, openly noting in the captions and the Limitations section where SAM outperforms mSAM and where the theory does not apply.
Critical weaknesses

1. Very limited contribution. The proposed method is Momentum-SAM applied to PPO, within the PPO+SAM framework of Lee & Yoon (2025), using the same environments, perturbation protocols, and experimental setup. The single-gradient property, which is the paper's main selling point, is inherited directly from Momentum-SAM, and evaluating both gradients on the same batch is standard SAM practice. I therefore see no conceptual contribution beyond combining two existing works. Since this concerns the core framing of the paper rather than a fixable detail, I doubt it can be addressed in the rebuttal phase.

2. The theory does not support the paper's claims. The stationarity results (Theorems 1–3) are essentially sanity checks: they show that SAM/mSAM still converge to an approximate first-order stationary point, with an additional error term due to the perturbation. The assumptions (smoothness, unbiased gradients, bounded variance) are generic and not specific to RL, and the analysis does not cover the PPO objective actually used (as acknowledged in the Limitations). Most importantly, the results say nothing about flatness, robustness, or why the momentum direction is a good perturbation direction; the perturbation only appears as a cost in the bound.

3. The experiments do not support the central claim. The paper claims that mSAM achieves SAM-level robustness at lower cost, but its own results show that PPO+SAM is more robust in several settings (e.g., stronger action noise and the Walker2d friction/mass sweeps, per the captions of Figs. 1–3) and flatter (Table 2), which undercuts the proposed mechanism; most other differences are within noise (5 seeds, large overlapping confidence bands, stds often exceeding 1000 on HalfCheetah in Table 1); and the reward-noise experiment adds zero-mean noise, which does not change the optimal policy and thus tests learning under higher gradient variance rather than robustness.

4. Presentation and rigor (minor). Theorem 1 relies on assumptions and notation defined only in the appendix; Theorem 2 is presented as a special case of Theorem 3, which appears only in the appendix; and the message of Section 3.3 is unclear. The paper would benefit from a careful revision for clarity and mathematical precision.

Questions for authors

Could the authors clarify what they view as the main contribution of the paper beyond combining Momentum-SAM with the PPO+SAM framework of Lee & Yoon (2025)?

Reviewer_ApAxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper applies Momentum-SAM to policy learning, which utilizes accumulated momentum to construct adversarial perturbations. It presents finite-time stationarity bounds for SAM and mSAM in policy optimization. Experiments compare PPO variants and robust reinforcement learning baselines on three continuous control tasks from MuJoCo. The results demonstrate the effectiveness of mSAM under action noise, transition perturbations, and noisy training rewards. However, their theoretical analysis is neither rigorous nor well-formulated.

Critical strengths

The implementation that performs only one policy-gradient evaluation per update has a clear computational motivation. The experimental evaluation is fairly comprehensive and demonstrates the effectiveness of the proposed algorithm in several respects.

Critical weaknesses

There are substantive correctness and applicability gaps. No smoothness assumption of every ξt\xi_t can be found in the paper, yet the theoretical analysis throughout makes use of the LL-Lipschitz continuity. In Theorem 1, the constant step size η\eta is chosen independently of c1c_1, whereas the subsequent proof requires c1Lη≤18c_1 L \eta \leq \frac{1}{8}. The negative normalized momentum perturbation and single-gradient update already appear in [1], while SAM+PPO and the robust-RL connection appear in [2]. Furthermore, there are numerous places throughout the paper where errors in proof details and writing occur.

[1] Marlon Becker, Frederick Altrock, and Benjamin Risse. Momentum-SAM: Sharpness aware minimization without computational overhead. In NeurIPS, 2025.

[2] Hyun Kyu Lee and Sung Whan Yoon. Flat reward in policy parameter space implies robust reinforcement learning. In ICLR, 2025.

Questions for authors
  1. How can you ensure c1Lη≤18c_1 L \eta \leq \frac{1}{8} for general c1c_1?
  2. Do Theorems 2 concern negative expected return, a fixed-behavior PPO surrogate, or a changing rollout-dependent objective?
  3. Do the gains over plain momentum and SAM persist with identical base optimizers, clipping, tuning budgets, and validation protocols? Please report β\beta, selection criteria, whether test perturbations informed tuning, and seed-level robustness statistics.
Reviewer_7mTbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a cheaper proxy of sharpness aware minimezation for policy optimization. Instead of spending an extra gradient computation to find the pessimistic perturbation direction, it uses the optimizer's accumulated momentum, so each update needs only one gradient. The authors plug this into PPO, give finite-time convergence bounds for both the SAM and PPO baselines and also thier method in policy optimization, and evaluate robustness on a few MuJoCo tasks under action noise, dynamics changes, and noisy training rewards.

Critical strengths
  1. A simple and cheap idea:
    1. Reusing the momentum that the optimizer already keeps is an easy way to remove SAM's extra gradient computation, and it can be dropped into existing policy-gradient code with little effort.
  2. Standard benchmakrs and several kinds of perturbation:
    1. The experiments use common MuJoCo tasks and test robustness to action noise, friction, mass, and reward noise, which makes the results easy to relate to prior work.
    2. That said, it would have been nicer to include more violant perturbations; these are pretty standard, and for a paper regarding sharpness-aware min work, I would like to set the standard slightly higher.
  3. Honest reporting:
    1. The captions acknowledge where the plain SAM baseline does beter, which I appreciate. The paper certainly isn't trying to paint a rosy picture of the method's performance.

P.S. I also really appreciate the table of notations in the appendix; it made my life so much easier! :)

Critical weaknesses

My over-simplified understanding of the flow is as follows; please correct me if I am woefully wrong in the rebuttals:

  • SAM looks for the "worst" nearby weights using one gradient, then updates the policy using a second gradient computed at those worst weights.
  • Your method replaces the first gradient with the momentum direction the optimizer already has, so only one gradient is needed per update.
  • The theory shows that training still reaches a point where the gradient is small, up to an extra error caused by the "perturbation".
  • The experiments compare against PPO, PPO with SAM, PPO with simple Nesterov momentum, and two robust RL methods I'm not.

Given that,

  1. Regarding what the perturbation actually does:
    1. The pertrbation moves the weights along the direction the optimizer is already heading. To me, this sounds closer to a look-ahead step (like Nesterov momentum) than to searching for the worst nearby weights.
    2. The paper itself notes that the resulting perturbation "need not maximize the local loss" in Line 200.
    3. If so, what makes the method sharpness-aware? rather than just a different momentum scheme? A clearer explanation (or a simple experiment showing that the perturbed point really has higher loss in the vast majority of times) would help.
    4. This is a significant concern for me. If we had a more specialized momentum for SAM's needs, I would have been much more at ease.
  2. Regarding the theory:
    1. The theorems show convergence to a stationary point, which is a generic optimization result; the paper's own remark says it applies to general nonconvex problems.
    2. You don't say anything about robustness, which is the main claim of the paper. The link between flatness and robustness is taken from prior work rather than shown here.
    3. It would help to clarify what, if anything, in the analysis is specific to RL, and how it supports the robustness claims. If not, maybe the introduction claims can be toned down slightly.
  3. Regarding novelty:
    1. If I understand correctly, momentum-guided SAM and SAM applied to PPO have each been proposed before, and the paper cites both as its starting points; I'm looking up NeurIPS/ICLR 2025, and there is clearly literature trying to do momentums and PPO separately in SAM. I won't cite them since I haven't had the time to read them, but the literature search could include such at least relevant-looking papers.
    2. It would help if the authors stated more explicitly what is new here beyond combining the two. For example, what had to change to make it practically work in RL? what would break if one applied the existing method directly?
    3. To be clear, I don't think a combination of existing ideas is a bad contribution in itself. You have some thin theory added at the very least. I just want the paper to make the delta clear.
  4. Regarding statistical reliability of the results:
    1. I don't fault the authurs for not showing massive improvements. If the evaluations are done rigorously, even a modest or negative result is worth reporting. My concern is with how reliable the reported comparisons are.
    2. All results use five training seeds (pls correct me if I'm wrong or missing an implementation detail or categorization). Deep RL is notorious for unreliable results across seeds and implementation details [1-5], and five seeds can't reliably estimate confidence intervals or error bars. I've used 100 seeds in my own papers, yet the error bars remained large. In particular, large failures (e.g., a policy collapsing in some runs) may happen less often than one in five runs, so five seeds can easily misscharacterize the error bars entirely.
    3. Multiple reported differences seem small compared to their spreads. For instance, in the noisy-reward table, many of the "best" entries overlap heavily with other methods once the standard deviations are taken into account.
    4. I'd encourage the authors to use more seeds and to report confidence intervals instead of just mean and standard deviation. Showing per-seed results would also help readers see whether gains are consistent or driven by a few rare runs.
    5. An unfortunate egregious practice in preceeding RL literature is to cherry-pick the seeds or show the top-5 out of 20 runs or things of that nature. It would help if the authors explicitly mention that the seeds were completley random and you did not bias the presentation by picking best outcomes in any shape or form.
  5. Regarding the comparison with plain SAM:
    1. By the paper's own account, plain SAM is often as robust or more robust than your method, especially at higher noise levels and on Walker2d.
    2. That makes the computational savings the main selling point. However, the main text claims lower cost without showing the numbers. I'd suggest bringing a clear cost comparison (e.g., wall-clock time and gradient evaluations per update) into the main text.

References

  1. Islam et al., "Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control", arXiv 2017 (but I think I saw this in ICML at some point).
  2. Henderson et al., "Deep Reinforcement Learning that Matters", AAAI 2018.
  3. Ilyas et al., "A Closer Look at Deep Policy Gradients", arXiv, 2018 (this and the next paper were originally one paper, but they got split so many times I'm not sure where they ended up).
  4. Engstrom et al., "Implementation Matters in Deep RL: A Case Study on PPO and TRPO", arXiv.
  5. Patterson et al., "Empirical Design in Reinforcement Learning", JMLR 2025.
Questions for authors

See the critical weaknesses. In addition:

  1. Does the perturbed point actually have a higher loss than the current point in practice? If not, how should we interpret it as a "worst-case" perturbation?
  2. Were the method-specific settings (e.g., the perturbation size) tuned with the same effort for all baselines?
  3. What hav

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.