acceptodds
Under review as a conference paper at ICLR 2027

Reachability-Aware Diffusion Policy Optimization

Abstract

Diffusion policies provide expressive action distributions for continuous-control reinforcement learning. However, safety-aware online diffusion policy optimization remains underexplored, particularly methods that use predictive reachability information without an explicit dynamics model. We propose Reachability-Aware Diffusion Policy Optimization (RADPO), a model-free method that combines predictive first-hit safety estimation with cumulative-cost budget feedback. RADPO learns a discounted first-hit reachability value that captures the discounted risk of a cost event, assigns larger weight to events that occur sooner, and uses this signal to shape the reward. A separate dual-like multiplier adjusts the shaping strength according to realized episodic costs relative to a prescribed budget. The diffusion actor improves through weighted denoising regression on candidate actions scored by the reward critic. Our approach requires neither a learned dynamics model, action gradients through the critics, nor differentiation through the reverse diffusion sampler. We establish theoretical properties of the reachability value and show that accumulated reachability penalty provides a conservative surrogate for future discounted cumulative cost. Across ten continuous-control safety tasks, RADPO achieves competitive reward-cost trade-offs, with substantial reductions in constraint violations on several tasks relative to the compared baselines. Our theoretical and empirical analysis supports that combining reachability with cumulative budget feedback is a viable approach to safety-aware diffusion policies.

Primary area:
reinforcement learning
Keywords:
Safe Reinforcement Learning, Diffusion Policies, Policy Mirror Descent, Reachability Analysis, Continuous Control

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Pi9oOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies model-free safe online optimization of diffusion policies. The authors propose RADPO, which learns a discounted first-hit reachability value via binary cross-entropy regression and trains the critics on the resulting shaped reward. The paper also provides a series of theoretical derivations, and demonstrates a competitive reward–cost trade-off on ten continuous-control safety tasks in Safety Gymnasium, with substantial reductions in constraint violations on several tasks.

Critical strengths
  1. The study decouples predictive first-hit reachability from the episodic budget-feedback multiplier into two complementary components.
  2. The theoretical proofs are complete, and the authors candidly note in Sec 6 and A.1 that the results hold only at the level of summary statistics.
  3. The proposed safe RL method requires no dynamics model, no critic action-gradients, and no differentiation through the reverse sampler.
Critical weaknesses
  1. The paper formulates the safe RL problem, yet nowhere does it analyze, on a per-task basis, whether the trained policies actually satisfy the safety constraints.
  2. \lambda is not a standard Lagrange multiplier for the actor's objective.
  3. The main text lacks an overall algorithmic workflow, and too much space is devoted to defensive writing.
Questions for authors
  1. If RADPO's cost indeed exceeds d = 25 on certain tasks, how should the degree of attainment of the "constraint satisfaction" objective be characterized?
  2. Under the simplified setting with \lambda held fixed, can the feasibility or suboptimality of the optimal policy of the shaped objective, relative to the original CMDP, be guaranteed?
  3. If RADPO is run with a unified hyperparameter setting, how much does the performance degrade?
Reviewer_GknPOpenReview ↗
Rating
4(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RADPO for online safe reinforcement learning with diffusion policies. A critic estimates discounted first-hit risk, and its value at successor states is used to shape the reward. Feedback from realized episodic costs adjusts the penalty strength. The actor builds on DPMD, using a shaped reward critic to weight candidate actions for denoising regression, without critic action gradients or differentiation through the sampler. The highest-scoring candidate is selected for execution. The paper analyzes the reachability signal, candidate reweighting, and budget feedback, and evaluates the method on navigation and velocity-constrained locomotion tasks.

Critical strengths
  1. The method makes sense and addresses a useful problem. Reachability provides a predictive safety signal, while realized episodic costs provide feedback on the cumulative budget. Using the former to shape the reward lets the existing DPMD update work with a critic that already accounts for safety. This is a straightforward way to adapt diffusion policy optimization to constrained online learning.

  2. The empirical evidence is strongest on navigation. RADPO achieves the highest mean returns among the evaluated methods on both Goal tasks and favorable reward–cost trade-offs on the Circle tasks (Fig. 3). Figure 4 compares different safety signals and ways of applying the penalty. I also appreciate the evaluation curves alongside the training results, and the discussion of the trade-offs on locomotion tasks and the limits of the theoretical guarantees.

Critical weaknesses

Where does the benefit over prior uses of reachability come from?

Section 2 cites RCRL and RESPO and outlines a high-level distinction, but it is not yet clear which differences concern the safety estimate itself and which concern how it is used. RESPO uses a closely related discounted reachability backup. FCSRL [1], which is not cited, learns a maximum discounted cost signal that has the same first-hit interpretation for binary costs. These methods use reachability differently: RCRL for statewise constraints, RESPO for feasibility-dependent objectives, and FCSRL for representation learning. Can you clarify what is new in the estimate versus its use for reward shaping and episodic budget feedback, and explain when and why this use should be preferable?

[1] Cen, Yao, Liu and Zhao. Feasibility Consistent Representation Learning for Safe Reinforcement Learning. ICML 2024. https://arxiv.org/abs/2405.11718

Questions for authors
  1. To better contextualize RADPO, could the authors discuss its position within the taxonomy proposed in FlowRL [2]? Equation (13) appears to follow DPMD-style weighted matching with a previous-policy reference, while execution additionally uses best-of-K action selection. Is this characterization accurate, or are there aspects of the policy update that fall outside this classification?

  2. Please explicitly define the denoising loss used in Eqs. (2) and (13), including the prediction target and the timestep/noise sampling distributions. Section 3 introduces the notation, but I could not find its actual expression in the paper or appendix.

  3. Table 3 lists 30k uniform-random warm-up transitions, whereas the released CarGoal1 and PointCircle1 configurations set start_step to 120k. Which setting was used for the reported results?

  4. Figure 5 shows persistent budget violations on HalfCheetah when β is reduced to one quarter of its default value, while increasing β fourfold causes the PointGoal1 return to collapse. Does the multiplier reach its cap in these runs? It would be great to show the corresponding multiplier trajectories and explain how to choose β and the multiplier cap on a new task.

  5. Figure 5 already compares K from 8 to 64. FLAG [3, Sec. 5.1] separately controls policy-update samples and the best-of-P action-selection budget, including for DPMD. Could you report return and cost for the same trained RADPO policy when executing a single sampled action without critic-based selection? This would clarify the contribution of selection at deployment.

[2] Gao, Chen, Chen and Dai. FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies. 2026. https://arxiv.org/abs/2603.27450

[3] Kim et al. FLAG: Flow Policy MaxEnt-RL by Latent Augmented Guidance. 2026. https://arxiv.org/abs/2605.30749

1 comment

Thank you for the responses. The revision addresses my questions, and I have accordingly raised my score to 4.

Reviewer_pmy7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies online safe RL in a CMDP with an expected episodic-cost budget, using diffusion policies. To achieve this, the paper proposes Reachability-Aware Diffusion Policy Optimization (RADPO). RADPO combines discounted first-hit reachability prediction, successor-state reward shaping, and episodic-cost feedback for online diffusion-policy optimization. The actor uses value-weighted denoising regression. Experiments on ten safety tasks show promising reward–cost trade-offs, particularly in navigation.

Critical strengths
  • The method composes a safety critic, reward shaping and a scalar controller on top of a DPMD-style actor without critic action gradients or sampler backpropagation.
  • The paper performs extensive evaluation, ten tasks, six baselines including an online diffusion safe-RL method (ALGD) and a reachability method (RESPO), a reward-only reference, and both training-rollout and deterministic-evaluation curves.
Critical weaknesses
  • The reachability value is not new [1][2][3], and the related-work positioning overstates the distinction.
  • The theory does not distinguish reachability shaping from simpler alternatives, and most results are standard.
  • The experiments do not isolate what drives the improvements. The placement ablation is confounded. The "lag-style shaping" arm changes three things at once: the signal (a discounted cost critic instead of reachability), the placement (weights instead of reward), and the normalization (a separately clipped cost advantage in [−3, 3]). The conclusion that reward shaping is the right placement does not follow. For the "discounted critic" arm, γ_c is not reported; with γ_c = 0.99 the horizon and scale differ greatly from reachability at γ_c = 0.9. Gain matching is also inconsistent: β is matched on HalfCheetah but cost-only shaping uses the same β as RADPO on the navigation tasks, and the "matched run" procedure is not specified.
  • RADPO tunes five per-task hyperparameters over wide ranges (β from 0.025 to 2.4, about 100×; η_λ from 0.002 to 0.12, 60×; UTD; σ_0; α_0).
Questions for authors

Please refer to the weaknesses section.

1 comment

I appreciate the authors' feedback. It has addressed most of my previous concerns. After carefully considering the other reviewers’ comments, I lean toward raising my score to a weak accept.

Comments on the paper

1 comment

We thank the Area Chair and all three reviewers for their time and constructive feedback throughout the review process. We appreciate their suggestions, which have helped us substantially improve our manuscript. We have addressed each reviewer’s concerns in detail and revised the manuscript accordingly. Below, we summarize the main concerns and the corresponding revisions:

  • Novelty and positioning relative to prior work (Reviewers GknP and pmy7). We expanded the related work discussion and added a comparison table distinguishing RADPO from existing reachability-based methods. We clarified that our primary contribution is not the reachability estimator itself, but its integration into reward shaping for diffusion policy optimization, together with a separate cumulative-cost budget-feedback mechanism. We also positioned RADPO within the recently proposed FlowRL taxonomy, as suggested by Reviewer GknP.
  • Theoretical analysis and guarantees (Reviewers pmy7 and Pi9o). We substantially expanded our theoretical analysis in our revised manuscript. In particular, we strengthened Corollary 1 to characterize how first-hit reachability shaping weights repeated violations according to their temporal spacing, distinguishing it from conventional cumulative-cost shaping. Furthermore, we clarified the interpretation of our safety multiplier as a budget-feedback controller rather than a conventional Lagrange multiplier, and added theoretical analysis of its interaction with reachability-based shaping.
  • Ablation studies and additional empirical analysis (Reviewers pmy7 and GknP). We revised our ablation studies to make the comparisons, discount factors, normalization procedures, and coefficient calibration more explicit, and expanded the discussion of what conclusions can be drawn from each experiment. We also added experiments comparing critic-based best-of-(P) action selection against single-sample execution of the same trained diffusion policies, revealing task-dependent effects on safety and performance. In addition, we introduced multiplier trajectory analyses and targeted experiments investigating the effects of the multiplier cap, controller step size, and shaping strength.
  • Constraint satisfaction and hyperparameter sensitivity (Reviewers Pi9o and pmy7). We added a separate evaluation of the final trained policies over 200 fresh episodes per task, showing that their mean episodic costs remain below the prescribed budget on all ten tasks under the task-specific configurations. We also conducted new experiments using a shared hyperparameter configuration across all ten tasks to quantify performance degradation and identify remaining sensitivities. These experiments highlight the importance of controller tuning for reliable budget regulation. We further expanded our discussion of hyperparameter selection and the limitations of our safety guarantees.

Beyond these main revisions, we improved the presentation and reproducibility of our method by moving the algorithmic pseudocode to the main text, explicitly defining the diffusion denoising loss, correcting an inconsistency in the reported warm-up configurations, and clarifying several implementation details.

We believe that the revised manuscript provides a clearer account of RADPO’s contributions, stronger theoretical and empirical support, and a more transparent discussion of its limitations. We hope that our revisions and detailed responses have adequately addressed the concerns raised during the review process. We would be happy to address any remaining questions or concerns and welcome any further feedback. We thank the Area Chair and reviewers once again for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.