- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes applying Momentum-SAM (mSAM) to policy optimization in order to improve the robustness of RL policies while reducing the computational cost of sharpness-aware minimization (SAM). Instead of computing an extra gradient to construct the adversarial parameter perturbation, mSAM perturbs the parameters along the normalized accumulated momentum direction, requiring a single gradient evaluation per update. The authors integrate mSAM into PPO and provide finite-time stationarity bounds for both SAM and mSAM policy optimization. Experiments on three MuJoCo tasks (HalfCheetah, Hopper, Walker2d) evaluate robustness to action noise, friction and mass variations, and noisy training rewards, along with flatness measurements and a computational-efficiency comparison against PPO, PPO+SAM, PPO+NAG, RARL, and RNAC.
Critical strengths
- The method is simple and easy to integrate into existing policy-gradient pipelines, and it consistently reduces training time compared to PPO+SAM.
- The empirical evaluation is broad, covering action, dynamics, and reward perturbations.
- The theoretical analysis appears sound.
- The authors are candid about the limitations of their results, openly noting in the captions and the Limitations section where SAM outperforms mSAM and where the theory does not apply.
Critical weaknesses
1. Very limited contribution. The proposed method is Momentum-SAM applied to PPO, within the PPO+SAM framework of Lee & Yoon (2025), using the same environments, perturbation protocols, and experimental setup. The single-gradient property, which is the paper's main selling point, is inherited directly from Momentum-SAM, and evaluating both gradients on the same batch is standard SAM practice. I therefore see no conceptual contribution beyond combining two existing works. Since this concerns the core framing of the paper rather than a fixable detail, I doubt it can be addressed in the rebuttal phase.
2. The theory does not support the paper's claims. The stationarity results (Theorems 1–3) are essentially sanity checks: they show that SAM/mSAM still converge to an approximate first-order stationary point, with an additional error term due to the perturbation. The assumptions (smoothness, unbiased gradients, bounded variance) are generic and not specific to RL, and the analysis does not cover the PPO objective actually used (as acknowledged in the Limitations). Most importantly, the results say nothing about flatness, robustness, or why the momentum direction is a good perturbation direction; the perturbation only appears as a cost in the bound.
3. The experiments do not support the central claim. The paper claims that mSAM achieves SAM-level robustness at lower cost, but its own results show that PPO+SAM is more robust in several settings (e.g., stronger action noise and the Walker2d friction/mass sweeps, per the captions of Figs. 1–3) and flatter (Table 2), which undercuts the proposed mechanism; most other differences are within noise (5 seeds, large overlapping confidence bands, stds often exceeding 1000 on HalfCheetah in Table 1); and the reward-noise experiment adds zero-mean noise, which does not change the optimal policy and thus tests learning under higher gradient variance rather than robustness.
4. Presentation and rigor (minor). Theorem 1 relies on assumptions and notation defined only in the appendix; Theorem 2 is presented as a special case of Theorem 3, which appears only in the appendix; and the message of Section 3.3 is unclear. The paper would benefit from a careful revision for clarity and mathematical precision.
Questions for authors
Could the authors clarify what they view as the main contribution of the paper beyond combining Momentum-SAM with the PPO+SAM framework of Lee & Yoon (2025)?