- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that existing constrained reinforcement learning (CRL) underutilizes the information contained in the sampled trajectories. To address this, the paper extends phasic training to constrained reinforcement learning. An auxiliary phase reuses trajectories to refine reward and cost critics and distill value targets into the actor under a KL penalty. The framework is evaluated with CPO, CPPO-PID, and TRPO-PID on four tasks.
Critical strengths
- The modular design applies across three distinct constrained-policy optimizers, making the proposed training extension practically relevant.
- The two Goal tasks show consistent improvements in median return across all three backbones; several comparisons also reduce median cost.
Critical weaknesses
- The auxiliary phase simultaneously updates the actor and both critics. The ablation removes actor-side cost distillation but retains reward distillation and critic replay; it therefore does not isolate representation transfer from additional critic fitting. Since dual-signal representation learning motivates the contribution, a critic-replay-only control is important.
- The KL results require reconciliation with the stated constraint. Equation 5 and Appendix A specify , whereas the final-epoch KL in Figure 5(b) appears mostly around 0.03–0.09. If these use the same definition, the plot does not establish enforcement of the stated budget; if they differ, the aggregation and conversion need explanation. The multiplier update, initialization, learning rate, and stopping or rollback rules are also insufficiently specified.
- The safety statistics do not fully establish performance at comparable feasibility. The problem constrains expected cost, while the main results report medians across seeds, with several proposed-method entries above the budget of 25. The deployment distributions are valuable but do not replace per-policy mean-cost estimates and uncertainty. This is a request to assess the stated expectation constraint, not to require every episode to remain below budget (Tables 1–2; Appendix E).
- Prior work on safe representation learning is insufficiently discussed, e.g., FCSRL [ICML 2024].
Questions for authors
Please refer to the weaknessese section.
Thank you for pointing out the need for a critic-only control. We conducted the ablation for our CPPO-PID based version CPPGPID and added the results to the revision of our paper (Appendix F). The results show that both, the critic-only as well as the distillation-only versions of our method achieve strong results and retain most of the gains of our full method. However, the full-method has shown to achieve higher results than the ablated versions, while only incurring higher costs in some of the environment. The best cost-adhering ablation, which is critic-only, lacks behind the full method and the distillation-only ablation when it comes to reward performance.
Thank you for catching this discrepancy. The original Figure 5(b) summed KL across HalfCheetahVelocity’s six action dimensions, while training applies the 0.01 target to the mean across minibatch states and action dimensions. We revised Eq. (3) and Eq. (5) accordingly and divided the plotted values by six. On the corrected scale, 8 of 31 phases in the plotted run still end above 0.01, due to the nature of the Lagrangian formulation which does not strictly enforce exceeding the target. Following your request, we also added information on the full update rule to Appendix A in our revised paper. We initialize beta_clone=1 and take one Adam step per auxiliary minibatch, immediately before the actor update, with a learning rate of 0.01. The multiplier and optimizer state carry over between phases. We hope that we could clarify your questions.
We assess safety using cumulative episode cost, following prior Safety-Gymnasium benchmarks such as OmniSafe. Our main tables report the median of the per-seed mean episode costs and a 95% bootstrap confidence interval across seeds. This shows how typical performance compares with the expected-cost budget of 25. Our environments are still constrained by an expected value target and judged by an expected value target. What changed is that we utilize robust statistics to aggregate across runs, as shown by Agarwal’s seminal work “Deep Reinforcement Learning at the Edge of the Statistical Precipice”, which demonstrates that simple averaging over independent runs might produce misleading results due to DeepRL’s high noise floor. For this reason, they motivate aggregating multiple seeds via robust statistics and using robust scale estimators like bootstrap CIs instead of mean and standard deviation across runs. This does not alter the individual runs, it just tightens the center and scale estimate during aggregation. We will clarify this in the main paper.
Thank you for pointing us to FCSRL. We added it to the Background section, where we acknowledge its methodological closeness to our approach in cost-aware representation learning while highlighting that its on-policy PPO-Lag variant follows the same collect, update, and discard cycle we study. Where our experiments showed that for CPO, CPPO-PID, and TRPO-PID, retaining rollouts after the standard updates improved returns on both goal-reaching tasks at the same interaction budget.
Thank you again for giving us a positive rating. We hope we could address your concerns adequately and would greatly appreciate if you could raise your score accordingly.