acceptodds
Withdrawn from ICLR 2027

Certified Diffusion Smoothing for Robust Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deep Reinforcement Learning (DRL) is known to be vulnerable to small adversarial perturbations in the input space, which limits its deployment in real-world applications. While many empirical studies have sought to improve its robustness against adversarial attacks, these methods typically lack formal robustness guarantees. Existing certified defenses, despite providing certification criteria, often suffer from low robust rewards and weak robustness guarantees. In this work, we introduce a certifiably robust reinforcement learning framework that leverages the diffusion process to incorporate the objective of randomized smoothing directly into policy training. We propose Certified Diffusion Smoothing (CDS), a novel method that trains a smoothed agent with stronger certified robustness guarantees and higher robust rewards. We evaluate CDS under multiple certifications and adversarial attacks in standard RL environments, including MuJoCo and Atari. Across these benchmarks, CDS consistently outperforms previous smoothed agents by an average factor of 1.5–3.2 and robustly trained agents by an average factor of 2.8. Our code is available at https://anonymous.4open.science/r/CDS_DRL.

Primary area:
reinforcement learning
Keywords:
Certified Robustness, Robust Reinforcement Learning, Randomized Smoothing, Adversarial Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4zVrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Certified Diffusion Smoothing (CDS) to improve the adversarial robustness of deep reinforcement learning. The method combines randomized smoothing with diffusion-based denoising during policy training. Two implementations, CDS-DQN and CDS-PPO, are developed for discrete and continuous action spaces. Experiments on Atari and MuJoCo show improvements in certified robustness and rewards under adversarial attacks compared with existing methods.

Critical strengths
  1. The paper addresses a problem in reinforcement learning by improving robustness with theoretical guarantees.
  2. The proposed framework supports both discrete and continuous action spaces through CDS-DQN and CDS-PPO.
  3. The experiments include multiple adversarial attacks, robustness certification metrics, and comparisons with existing methods. The DQN ablation studies also provide useful evidence for the effectiveness of diffusion denoising.
Critical weaknesses
  1. In Appendix A.1, Algorithm 3 only bounds the selected runner-up action, potentially overlooking other actions with higher probabilities. The authors should provide confidence bounds for all competing actions.
  2. The theoretical guarantees assume ideal smoothed policies, while CDS-DQN and CDS-PPO use only 10 and 50 samples, respectively (Appendix B). The authors should examine how finite sampling affects certification validity.
  3. The reward certification in Equation (15) only covers fixed, non-adaptive perturbations. The authors should clarify how this limitation affects the robustness claims under adaptive attacks.
  4. Section 5.2 lacks attacks specifically targeting the diffusion and smoothing mechanisms. The authors should evaluate EOT-based attacks to account for the stochastic defense.
  5. The CDS-PPO ablations are insufficient (Appendix D). The authors should separately remove conditional history and autoregressive loss weighting to demonstrate their contributions.
  6. Table 19 shows that CDS-DQN runs at only 14.5 FPS compared with 64 FPS for S-DQN. The authors should evaluate robustness under comparable computational budgets.
Questions for authors

see weeknesses

Reviewer_i8N6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes to combines randomized smoothing with diffusion denoising and trains DQN and PPO policies under Gaussian perturbations. The certification part is about local robustness of the smoothed policy to bounded observation perturbations. Novelty is limited: the certification machinery comes from prior randomized smoothing, where diffusion helps denoise perturbed states and strengthen certificates. It improves certified radii and adversarial rewards on three Atari tasks and two relatively simple MuJoCo tasks, but the benchmark scope is limited. Overall it does not yet establish a fundamentally new certification framework and the diffusion part feels more like an add-on following pretty standard recipes.

Critical strengths
  • The paper gives guarantees on worst-case robustness certificates.
  • The method covers both discrete and continuous control and shows consistent empirical gains in certified robustness and adversarial reward over several baselines.
Critical weaknesses
  • The conceptual novelty is limited: the certification still comes from standard randomized smoothing, while diffusion mainly serves as a denoising mechanism to improve the certificate.
  • It does not consider the effect of the policy outputs in closed-loops, which limits the importance in RL problems.
  • The evaluation is relatively weak for a broad robust-RL claim: only three Atari games and two simple MuJoCo tasks.
Questions for authors
  1. Have you tried more complex RL environments? I assume the sampling with diffusion improvement becomes challenging in high-dimensional state spaces.
  2. Have you compared against simpler alternatives such as MLP/autoencoder denoiser, or consistency regularization?
  3. The discussion on Kumar et al on policy smoothing is not fully convincing. Could you provide a more detailed theoretical and empirical comparison especially given that your cumulative-reward certificate is restricted to fixed trajectory perturbations?
  4. How would you approach closed-loop properties such as when the environment feedback is intrinsically sensitive?
Reviewer_hSTiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Certified Diffusion Smoothing (CDS) for robust reinforcement learning. It aligns Gaussian smoothing noise with the diffusion forward process, denoises perturbed observations, and trains the policy using the resulting representations. The authors instantiate CDS with DQN for discrete actions and PPO for continuous. Experiments are conducted on three Atari and two MuJoCo environments to compute robustness certificates and evaluate performance under adversarial attacks.

Critical strengths

The combination of diffusion denoising and smoothed policy training is well motivated: it addresses the mismatch when smoothing is added only at test time and allows the policy to adapt to denoised, perturbed observations during learning.

Critical weaknesses
  • In sec 4.1, eq. 6 estimates action probabilities using Monte Carlo samples, and eq. 7 uses these estimates to give a certified radius. I wanted to understand how authors address sampling uncertainty, so I followed appendix A.1: Theorem 2 selects a top and a runner-up actions from a preliminary sample, then uses fresh samples to bound their probabilities. However, the selected runner-up is not necessarily the true runner-up. Think about a state where actions A, B, and C have probabilities 60%, 23%, and 17%. The preliminary sample can mistakenly rank C second, and estimating A and C more accurately still leaves B unaccounted for. In the proof of theorem 2, the upper bound for the selected runner-up is used as though it bounds every competing action, which is incorrect.

  • I found it difficult to judge the attacks results in table 1. Looking at sec 5.2, the authors list PGD, PA-AD, and other attacks, but do not explain if and how the attack objectives account for CDS's denoising and random sampling. This matters because [1] already shows that ordinary PGD can substantially overestimate the robustness of a smoothed DQN (see their table 1). This needs clarification before I can assess the empirical robustness gains separately from the certification results.

References

[1] Sun, C.-E., Gao, S., and Weng, T.-W. Breaking the Barrier: Enhanced Utility and Robustness in Smoothed DRL Agents. ICML 2024.

Questions for authors
  1. How does Theorem 2 account for a stronger competing action missed by the preliminary sample? Does the implementation use the procedure in Theorem 2 and Algorithm 3? Please clarify the sample counts and confidence level used for fig. 3(a). Please also check if any necessary correction change the reported results.
  2. Does the DQN evaluation use the smoothed PGD and PA-AD attacks of Sun et al., 2024, or their original versions? What objective is attacked, how does it account for diffusion denoising and random sampling? It would help to know if a stronger optimization does not substantially change the conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.