- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies model-free safe online optimization of diffusion policies. The authors propose RADPO, which learns a discounted first-hit reachability value via binary cross-entropy regression and trains the critics on the resulting shaped reward. The paper also provides a series of theoretical derivations, and demonstrates a competitive reward–cost trade-off on ten continuous-control safety tasks in Safety Gymnasium, with substantial reductions in constraint violations on several tasks.
Critical strengths
- The study decouples predictive first-hit reachability from the episodic budget-feedback multiplier into two complementary components.
- The theoretical proofs are complete, and the authors candidly note in Sec 6 and A.1 that the results hold only at the level of summary statistics.
- The proposed safe RL method requires no dynamics model, no critic action-gradients, and no differentiation through the reverse sampler.
Critical weaknesses
- The paper formulates the safe RL problem, yet nowhere does it analyze, on a per-task basis, whether the trained policies actually satisfy the safety constraints.
- \lambda is not a standard Lagrange multiplier for the actor's objective.
- The main text lacks an overall algorithmic workflow, and too much space is devoted to defensive writing.
Questions for authors
- If RADPO's cost indeed exceeds d = 25 on certain tasks, how should the degree of attainment of the "constraint satisfaction" objective be characterized?
- Under the simplified setting with \lambda held fixed, can the feasibility or suboptimality of the optimal policy of the shaped objective, relative to the original CMDP, be guaranteed?
- If RADPO is run with a unified hyperparameter setting, how much does the performance degrade?
Thank you for the responses. The revision addresses my questions, and I have accordingly raised my score to 4.