- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents BBRR by combining the adversarial diffusion bridge construction of ADBM with the belief-based decision-making framework of BEP. BBRR-DQN estimates lower-tail Conditional Value-at-Risk (CVaR) by averaging the lowest Q-values across purified belief states for each action. BBRR-PPO combines median policy smoothing with trajectory-level lower-tail CVaR optimization. The method is evaluated on Atari games and MuJoCo continuous-control tasks.
Critical strengths
The paper’s motivation is reasonable. The results reported in the main tables are reasonably good, but the performance claims require further validation once the concerns raised below have been addressed.
Critical weaknesses
-
The novelty is limited, and the overall framework is closely related to [BEP, ICLR 2024]. The main changes are adopting the existing adversarial diffusion bridge construction of [ADBM, ICLR 2025] for purification, generalizing minimum aggregation across purified belief states to the average of the lowest Q-values, and extending the framework to PPO. The incremental contributions therefore lie primarily in risk-sensitive aggregation, history-conditioned purification, and adaptation to continuous control. However, the current experiments do not sufficiently establish the individual benefits of these additions.
-
Several inconsistencies in the reported experimental results raise concerns about their reliability. Comparisons with recent competitive baselines are also missing, and the ablation studies do not adequately isolate the contributions of the added components. The current evidence therefore does not sufficiently support the main claims regarding the effectiveness of the proposed improvements and their performance advantages.
Sun, X., & Zheng, Z. (2024, May). Belief-enriched pessimistic q-learning against adversarial state perturbations. In International Conference on Learning Representations (Vol. 2024, pp. 23223-23246).
Li, X., Sun, W., Chen, H., Li, Q., He, Y., Shi, J., & Hu, X. (2025, May). Adbm: Adversarial diffusion bridge model for reliable adversarial purification. In International Conference on Learning Representations (Vol. 2025, pp. 78366-78388).
Questions for authors
1. Limited novelty The bridge coefficient in Eq. (2) and the composite noise prediction objective in Eq. (4) are algebraically equivalent to Eqs. (7) and (9) of [ADBM, ICLR 2025], respectively. Sections 3.4-3.5 of [BEP, ICLR 2024] already use repeated diffusion-based purification to generate a belief set and employ pessimistic action selection and value learning. Its PF-RNN variant also incorporates historical information into belief approximation.
In BBRR-DQN, the main change is to replace BEP's minimum aggregation, , with the average of the lowest Q-values across the belief set. For a fixed belief set and Q-function, recovers minimum aggregation. Thus, in terms of belief set construction and action-value aggregation, the proposed method is closely connected to prior work, and the methodological increment appears limited.
The authors need to clearly delineate BBRR's contributions beyond these prior works. In particular, they should explain which limitations of existing methods are addressed by replacing minimum aggregation with CVaR, introducing a history-conditioned diffusion bridge, and provide corresponding experimental evidence.
1.1 Diffusion bridge training and generalization across attacks. The manuscript does not adequately specify how the adversarial perturbations are generated during diffusion bridge pretraining. Which attack methods and perturbation budgets are used? Is a separate diffusion bridge trained for each attack, or is a single frozen purifier shared across attacks? The evaluation should also report the performance of the same frozen purifier under different test attacks, especially adaptive attacks that jointly target the diffusion bridge, belief set construction, and the final policy. These results are needed to assess generalization across attacks and the robustness of the complete defense
2. Insufficient baseline comparisons competitive methods for defending against adversarial observation perturbations, including[BEP, ICLR 2024], [CAR-DQN, ICML 2024, CAR-PPO, 2025 preprint], [BAR-PPO, ICLR 2026], and the belief-based methods [A2B/A3B, ICLR 2025].
A2B/A3B are particularly relevant because they also address uncertainty about the underlying true state induced by observation attacks. The differences from these methods and the additional benefits of BBRR require clarification. Without comparisons to representative methods from this group, the claims of state-of-the-art performance are not sufficiently supported. The authors should provide such comparisons or explain why the relevant methods were excluded.
Sun, X., & Zheng, Z. (2024, May). Belief-enriched pessimistic q-learning against adversarial state perturbations. In International Conference on Learning Representations (Vol. 2024, pp. 23223-23246).
Li, H., Zhang, Z., Luo, W., Han, C., Lv, J., Guo, T., & Hu, Y. (2025). Towards Optimal Adversarial Robust Reinforcement Learning with Infinity Measurement Error. arXiv preprint arXiv:2502.16734.
Li, H., Lv, J., Han, C., Zhang, Z., Li, A., Liu, Y., ... & Jiang, N. (2026, April). On the Tension Between Optimality and Adversarial Robustness in Policy Optimization. In International Conference on Learning Representations (Vol. 2026, pp. 127436-127481).
Belaire, R., Sinha, A., & Varakantham, P. (2024). On Minimizing Adversarial Counterfactual Error in Adversarial RL. _arXiv preprint arXiv:2406.04724.
3. Issues with experimental reporting and validation
3.1 Consistency between Tables 1 and 6
For Radial-DQN on BankHeist, the nine corresponding attack entries in Tables 1 and 6 have identical means and standard deviations, although their perturbation budgets are labeled as and , respectively. For example, the three corresponding PGD results in both tables are , , and .
This exact correspondence across different perturbation budgets requires an explanation. The authors should verify the correspondence between budget labels, experimental configurations, and reported statistics, and provide the associated logs to check for possible labeling or result-compilation errors.
In addition, Table 6 reports for BBRR on both Freeway and BankHeist under all three PA-AD budgets, whereas Table 1 reports high returns under another set of budgets. Although the budgets differ, this substantial change in performance still requires an explanation. The current description is insufficient to reconcile the two sets of results.
3.2 Inconsistent values in Figure 1(b) and Table 4 For Walker2d under PA-AD with , the plotted mean episodic return of BBRR in Figure 1(b) is approximately , whereas Table 4 reports . This is a substantial discrepancy. The authors should clarify it.
3.3 Unspecified belief set size in Table 5 Table 5 considers , whereas the default DQN configuration uses (page 23, line 1194). The manuscript does not specify whether a larger is used for this sensitivity analysis. The authors need to report the corresponding and tail probability .
3.4 Missing comparison with the fast purification variant Table 7 reports DP-DQN at FPS without identifying the specific variant, and it omits results for the fast variant, DP-DQN-F. The authors should specify the variant, sampling configuration, and source of the timing measurements, and include the fast variant under the same hardware, input, and evaluation protocol. FPS values reported in different papers are not directly comparable. The current comparison does not sufficiently establish the efficiency advantage of BBRR over fast purification methods.
3.5 Ablations do not sufficiently isolate the contributions of the added components The ablations change both the procedure for generating purified belief states and the policy configuration, making it difficult to attribute the gains to individual components. For DQN, the diffusion bridge, belief set size, attack settings, and fine-tuning budget should be fixed when comparing mean, min, median, and CVaR aggregation on the same belief set. For PPO, the conditional diffusion bridge and median policy smoothing should be fixed while varying only the trajectory-level risk weight, comparing with . The effect of history conditioning should also be evaluated separately.
In Table 3, diffusion Bridge Only already approaches the full method in average performance, while the magnitude and consistency of the additional gains from the risk-sensitive components have not been sufficiently established.
Table 3 uses for Walker2d, whereas Table 4 uses . This prevents the ablation from directly explaining the performance advantage reported in the main comparison. The authors should provide ablations at the same perturbation budget as the main table and report uncertainty intervals for the additional gains.
Risk reduction is a central claim of the paper, yet the main experiments do not directly establish improvements in the corresponding lower tail. In particular, the PPO evaluation should report empirical lower-tail CVaR of trajectory returns, lower return quantiles, or a predefined failure rate, together with uncertainty intervals. CVaR across the Q-values of purified belief states in DQN and CVaR over trajectory returns in PPO concern different quantities and should also be evaluated separately.
The evaluation should additionally examine restoration error relative to the true state, the relationship between the diversity of purified belief states and restoration error, and how these quantities change with and the amount of diffusion noise. It should also assess whether lower-tail Q-values across the belief set predict low-return outcomes. These analyses are necessary to determine whether the gains reflect meaningful modeling of state uncertainty rather than a generic stochastic ensemble effect.
3.6 The theoretical bound does not explain the relative advantage of CVaR
The key step in the proof of Theorem 2 uses the fact that all candidate Q-values lie within . Under the same idealized assumptions, this part of the argument also applies to other aggregation rules whose outputs remain between the minimum and maximum candidate Q-values. Thus, the bound does not establish an advantage of CVaR over mean, min, or median aggregation, nor does it justify a particular choice of the tail parameter. The current theoretical analysis therefore provides limited insight into the added value of the proposed CVaR mechanism.