acceptodds
Withdrawn from ICLR 2027

Belief-Based Risk Reduction for Robust Reinforcement Learning under Adversarial Observation Perturbations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deep reinforcement learning (DRL) policies are highly vulnerable to adversarial noise in observations, posing significant risks in safety-critical scenarios. The primary challenge of adversarial perturbations lies in their ability to distort the information observed by an agent. Recently, state purification defenses have emerged as a promising solution. However, existing methods suffer from a distribution mismatch between clean states and adversarial observations, as well as uncertainty in state restoration and risks in decision-making. To address these challenges, we propose Belief-Based Risk Reduction (BBRR), a risk-aware state purification framework. BBRR uses a diffusion bridge to mitigate the distribution gap between adversarial observations and clean samples, while employing belief state approximation and Conditional Value-at-Risk (CVaR) to reduce restoration uncertainty and decision risks, respectively. Theoretical analysis and comprehensive evaluations across diverse environments demonstrate that BBRR consistently achieves state-of-the-art performance against a range of strong attacks, validating its robustness and practical applicability. Our code is available at https://anonymous.4open.science/r/Robust_BBRR.

Primary area:
reinforcement learning
Keywords:
Robust Reinforcement Learning, Adversarial Observation Perturbations, Belief States, Risk-Sensitive Decision Making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iMC1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents BBRR by combining the adversarial diffusion bridge construction of ADBM with the belief-based decision-making framework of BEP. BBRR-DQN estimates lower-tail Conditional Value-at-Risk (CVaR) by averaging the lowest kk Q-values across purified belief states for each action. BBRR-PPO combines median policy smoothing with trajectory-level lower-tail CVaR optimization. The method is evaluated on Atari games and MuJoCo continuous-control tasks.

Critical strengths

The paper’s motivation is reasonable. The results reported in the main tables are reasonably good, but the performance claims require further validation once the concerns raised below have been addressed.

Critical weaknesses
  1. The novelty is limited, and the overall framework is closely related to [BEP, ICLR 2024]. The main changes are adopting the existing adversarial diffusion bridge construction of [ADBM, ICLR 2025] for purification, generalizing minimum aggregation across purified belief states to the average of the lowest kk Q-values, and extending the framework to PPO. The incremental contributions therefore lie primarily in risk-sensitive aggregation, history-conditioned purification, and adaptation to continuous control. However, the current experiments do not sufficiently establish the individual benefits of these additions.

  2. Several inconsistencies in the reported experimental results raise concerns about their reliability. Comparisons with recent competitive baselines are also missing, and the ablation studies do not adequately isolate the contributions of the added components. The current evidence therefore does not sufficiently support the main claims regarding the effectiveness of the proposed improvements and their performance advantages.

Sun, X., & Zheng, Z. (2024, May). Belief-enriched pessimistic q-learning against adversarial state perturbations. In International Conference on Learning Representations (Vol. 2024, pp. 23223-23246).

Li, X., Sun, W., Chen, H., Li, Q., He, Y., Shi, J., & Hu, X. (2025, May). Adbm: Adversarial diffusion bridge model for reliable adversarial purification. In International Conference on Learning Representations (Vol. 2025, pp. 78366-78388).

Questions for authors

1. Limited novelty The bridge coefficient in Eq. (2) and the composite noise prediction objective in Eq. (4) are algebraically equivalent to Eqs. (7) and (9) of [ADBM, ICLR 2025], respectively. Sections 3.4-3.5 of [BEP, ICLR 2024] already use repeated diffusion-based purification to generate a belief set and employ pessimistic action selection and value learning. Its PF-RNN variant also incorporates historical information into belief approximation.

In BBRR-DQN, the main change is to replace BEP's minimum aggregation, min⁡m∈MtQ(m,a)\min_{m\in M_t}Q(m,a), with the average of the lowest kk Q-values across the belief set. For a fixed belief set and Q-function, k=1k=1 recovers minimum aggregation. Thus, in terms of belief set construction and action-value aggregation, the proposed method is closely connected to prior work, and the methodological increment appears limited.

The authors need to clearly delineate BBRR's contributions beyond these prior works. In particular, they should explain which limitations of existing methods are addressed by replacing minimum aggregation with CVaR, introducing a history-conditioned diffusion bridge, and provide corresponding experimental evidence.

1.1 Diffusion bridge training and generalization across attacks. The manuscript does not adequately specify how the adversarial perturbations δt\delta_t are generated during diffusion bridge pretraining. Which attack methods and perturbation budgets are used? Is a separate diffusion bridge trained for each attack, or is a single frozen purifier shared across attacks? The evaluation should also report the performance of the same frozen purifier under different test attacks, especially adaptive attacks that jointly target the diffusion bridge, belief set construction, and the final policy. These results are needed to assess generalization across attacks and the robustness of the complete defense

2. Insufficient baseline comparisons competitive methods for defending against adversarial observation perturbations, including[BEP, ICLR 2024], [CAR-DQN, ICML 2024, CAR-PPO, 2025 preprint], [BAR-PPO, ICLR 2026], and the belief-based methods [A2B/A3B, ICLR 2025].

A2B/A3B are particularly relevant because they also address uncertainty about the underlying true state induced by observation attacks. The differences from these methods and the additional benefits of BBRR require clarification. Without comparisons to representative methods from this group, the claims of state-of-the-art performance are not sufficiently supported. The authors should provide such comparisons or explain why the relevant methods were excluded.

Sun, X., & Zheng, Z. (2024, May). Belief-enriched pessimistic q-learning against adversarial state perturbations. In International Conference on Learning Representations (Vol. 2024, pp. 23223-23246).

Li, H., Zhang, Z., Luo, W., Han, C., Lv, J., Guo, T., & Hu, Y. (2025). Towards Optimal Adversarial Robust Reinforcement Learning with Infinity Measurement Error. arXiv preprint arXiv:2502.16734.

Li, H., Lv, J., Han, C., Zhang, Z., Li, A., Liu, Y., ... & Jiang, N. (2026, April). On the Tension Between Optimality and Adversarial Robustness in Policy Optimization. In International Conference on Learning Representations (Vol. 2026, pp. 127436-127481).

Belaire, R., Sinha, A., & Varakantham, P. (2024). On Minimizing Adversarial Counterfactual Error in Adversarial RL. _arXiv preprint arXiv:2406.04724.

3. Issues with experimental reporting and validation

3.1 Consistency between Tables 1 and 6

For Radial-DQN on BankHeist, the nine corresponding attack entries in Tables 1 and 6 have identical means and standard deviations, although their perturbation budgets are labeled as 3/255,5/255,15/2553/255,5/255,15/255 and 1/255,10/255,20/2551/255,10/255,20/255, respectively. For example, the three corresponding PGD results in both tables are 1164.2±10.61164.2\pm10.6, 1031.0±22.41031.0\pm22.4, and 516.9±16.2516.9\pm16.2.

This exact correspondence across different perturbation budgets requires an explanation. The authors should verify the correspondence between budget labels, experimental configurations, and reported statistics, and provide the associated logs to check for possible labeling or result-compilation errors.

In addition, Table 6 reports 0±00\pm0 for BBRR on both Freeway and BankHeist under all three PA-AD budgets, whereas Table 1 reports high returns under another set of budgets. Although the budgets differ, this substantial change in performance still requires an explanation. The current description is insufficient to reconcile the two sets of results.

3.2 Inconsistent values in Figure 1(b) and Table 4 For Walker2d under PA-AD with ϵ=0.05\epsilon=0.05, the plotted mean episodic return of BBRR in Figure 1(b) is approximately 45004500, whereas Table 4 reports 3302±2833302\pm283. This is a substantial discrepancy. The authors should clarify it.

3.3 Unspecified belief set size in Table 5 Table 5 considers k=6,8,10k=6,8,10, whereas the default DQN configuration uses K=4K=4 (page 23, line 1194). The manuscript does not specify whether a larger KK is used for this sensitivity analysis. The authors need to report the corresponding KK and tail probability α=k/K\alpha=k/K.

3.4 Missing comparison with the fast purification variant Table 7 reports DP-DQN at 6.66.6 FPS without identifying the specific variant, and it omits results for the fast variant, DP-DQN-F. The authors should specify the variant, sampling configuration, and source of the timing measurements, and include the fast variant under the same hardware, input, and evaluation protocol. FPS values reported in different papers are not directly comparable. The current comparison does not sufficiently establish the efficiency advantage of BBRR over fast purification methods.

3.5 Ablations do not sufficiently isolate the contributions of the added components The ablations change both the procedure for generating purified belief states and the policy configuration, making it difficult to attribute the gains to individual components. For DQN, the diffusion bridge, belief set size, attack settings, and fine-tuning budget should be fixed when comparing mean, min, median, and CVaR aggregation on the same belief set. For PPO, the conditional diffusion bridge and median policy smoothing should be fixed while varying only the trajectory-level risk weight, comparing λ=0\lambda=0 with λ>0\lambda>0. The effect of history conditioning should also be evaluated separately.

In Table 3, diffusion Bridge Only already approaches the full method in average performance, while the magnitude and consistency of the additional gains from the risk-sensitive components have not been sufficiently established.

Table 3 uses ϵ=0.075\epsilon=0.075 for Walker2d, whereas Table 4 uses ϵ=0.05\epsilon=0.05. This prevents the ablation from directly explaining the performance advantage reported in the main comparison. The authors should provide ablations at the same perturbation budget as the main table and report uncertainty intervals for the additional gains.

Risk reduction is a central claim of the paper, yet the main experiments do not directly establish improvements in the corresponding lower tail. In particular, the PPO evaluation should report empirical lower-tail CVaR of trajectory returns, lower return quantiles, or a predefined failure rate, together with uncertainty intervals. CVaR across the Q-values of purified belief states in DQN and CVaR over trajectory returns in PPO concern different quantities and should also be evaluated separately.

The evaluation should additionally examine restoration error relative to the true state, the relationship between the diversity of purified belief states and restoration error, and how these quantities change with KK and the amount of diffusion noise. It should also assess whether lower-tail Q-values across the belief set predict low-return outcomes. These analyses are necessary to determine whether the gains reflect meaningful modeling of state uncertainty rather than a generic stochastic ensemble effect.

3.6 The theoretical bound does not explain the relative advantage of CVaR

The key step in the proof of Theorem 2 uses the fact that all candidate Q-values lie within Qn(s,a)±LdQ_n(s,a)\pm Ld. Under the same idealized assumptions, this part of the argument also applies to other aggregation rules whose outputs remain between the minimum and maximum candidate Q-values. Thus, the bound does not establish an advantage of CVaR over mean, min, or median aggregation, nor does it justify a particular choice of the tail parameter. The current theoretical analysis therefore provides limited insight into the added value of the proposed CVaR mechanism.

Reviewer_vdFTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a defense named BBRR against observation perturbation attacks in deep reinforcement learning. BBRR relies on diffusion-based purification to remove potential perturbations, and leverages belif approximation and Conditional Value-at-Risk to recover the best state observation after purification. BBRR is evaluated on Atari (based on DQN) and MuJoCo (based on PPO), under 3 and 5 attacks, respectively.

Critical strengths
  • The method uses multiple purified observations to select actions, reducing reliance on a single potentially inaccurately purified observation. Results across both Atari and MuJoCo provide useful evidence that this approach can improve returns under the tested attacks.
Critical weaknesses
  • The main text claims robustness against adaptive attacks but does not explain how the attacks optimize against the complete BBRR pipeline (strong adaptive attackers). Random diffusion sampling and median aggregation can make attacks ineffective unless their optimization accounts for these operations. Without this information, the reported returns do not establish robustness to an attacker that knows and targets the defense.

  • The ablation study does not fully support the benefit of CVaR. The diffusion-bridge-only baseline uses one purified state with a vanilla policy, while the standalone risk variants use random states from the perturbation region. These comparisons change both the state samples and the policy training. They therefore leave unclear whether the gains require risk-sensitive optimization or mainly come from multiple purified samples and additional training.

Questions for authors
  1. How are the attacks used in the experiments adapted to the BBRR pipeline? Providing some results from attack optimization that accounts for diffusion randomness and action aggregation would be good, so that the reported robustness can be assessed against an attacker targeting the full defense.

  2. How does BBRR-PPO perform when only CVaR training is removed?

  3. The state observation purification used in BBRR is somehow related to provably robust DRL in which random purturbations are added to state observations during training to augment a DRL policy for test-time policy smoothing [1,2]. Beyond empirical attack evaluations, how does BBRR compare with the baselines in terms of certified return bounds under the same policy smoothing and certification settings?

[1] A. Kumar et al. Policy Smoothing for Provably Robust Reinforcement Learning, ICLR 2022.

[2] D. wang et al. CAMP in the Odyssey: Provably Robust Reinforcement Learning with Certified Radius Maximization, USENIX Security 2025.

Reviewer_BudbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a diffusion modeling-based defense

Critical strengths
  • Diffusion-based purification is an interesting step forward in the literature for generating belief states in this problem setting.
  • empirical analysis is appropriately thorough
Critical weaknesses
  • The main critical weakness stems from a lack of engagement with recent and relevant literature via empirical comparisons. [1,2,3] are more recent defense works in this line of literature; while [2] is cited, it seems like it would be a prudent baseline to include in the experiments due to the shared identification of belief structures. More critically, the paper mentions prior diffusion or other purification methods in related works, but does not make the expected empirical comparisons.
  • Somewhat related, the paper would also benefit from a clear distinction between the proposed method and prior diffusion-based defenses.

[1] Liu 2024: Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies. ICLR 2024

[2] Belaire 2025: On Minimizing Adversarial Counterfactual Error in Adversarial Reinforcement Learning. ICLR 2025

[3] Chen 2026: Robust Reinforcement Learning via Leveraging Historically Optimal Policy With Regulation of Performance. IEEE Trans. Neural Networks Learn. Syst. 37(9): 4559-4573 (2026)

Questions for authors
  • Could the authors describe the contribution made by the paper? While it is inferrable that the framework fitting diffusion purification to belief states and selecting from belief states via CVaR is potentially novel, it is unclear in what specific ways it is novel or different from existing pieces of literature.
  • What is the rationale behind using CVaR (a measure of tail risk) versus something mean- or median-derived, or comparative e.g. regret?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.