- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Certified Diffusion Smoothing (CDS) to improve the adversarial robustness of deep reinforcement learning. The method combines randomized smoothing with diffusion-based denoising during policy training. Two implementations, CDS-DQN and CDS-PPO, are developed for discrete and continuous action spaces. Experiments on Atari and MuJoCo show improvements in certified robustness and rewards under adversarial attacks compared with existing methods.
Critical strengths
- The paper addresses a problem in reinforcement learning by improving robustness with theoretical guarantees.
- The proposed framework supports both discrete and continuous action spaces through CDS-DQN and CDS-PPO.
- The experiments include multiple adversarial attacks, robustness certification metrics, and comparisons with existing methods. The DQN ablation studies also provide useful evidence for the effectiveness of diffusion denoising.
Critical weaknesses
- In Appendix A.1, Algorithm 3 only bounds the selected runner-up action, potentially overlooking other actions with higher probabilities. The authors should provide confidence bounds for all competing actions.
- The theoretical guarantees assume ideal smoothed policies, while CDS-DQN and CDS-PPO use only 10 and 50 samples, respectively (Appendix B). The authors should examine how finite sampling affects certification validity.
- The reward certification in Equation (15) only covers fixed, non-adaptive perturbations. The authors should clarify how this limitation affects the robustness claims under adaptive attacks.
- Section 5.2 lacks attacks specifically targeting the diffusion and smoothing mechanisms. The authors should evaluate EOT-based attacks to account for the stochastic defense.
- The CDS-PPO ablations are insufficient (Appendix D). The authors should separately remove conditional history and autoregressive loss weighting to demonstrate their contributions.
- Table 19 shows that CDS-DQN runs at only 14.5 FPS compared with 64 FPS for S-DQN. The authors should evaluate robustness under comparable computational budgets.
Questions for authors
see weeknesses