acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Constraint: Probing Constrained RL Bottlenecks with Phasic Policy Gradient

Abstract

Constrained Reinforcement Learning (CRL) extends policy optimization to Constrained Markov Decision Processes (CMDPs), where agents must maximize reward while satisfying explicit cost budgets. Although existing on-policy CRL methods differ in how they enforce constraints, they typically follow the same training paradigm where collected experience is used once and then discarded. We investigate whether this practice underutilizes information contained in sampled CRL trajectories, which provide supervision through both reward and cost signals. To this end, we introduce a phasic training framework for CRL that separates policy optimization from a dedicated auxiliary phase. During this phase, retained rollouts are reused to refine reward and cost representations that are then distilled into the policy network. The framework acts as a modular wrapper and can extend existing CRL algorithms without modifying their constraint-handling mechanisms. Experiments on Safety Gymnasium show that our extension to CPO, CPPO-PID and TRPO-PID yields consistent improvements in sample efficiency and cumulative reward in environments with rich observation structure. This suggests that representation learning and data utilization are important and underexplored factors in on-policy CRL performance.

Primary area:
reinforcement learning
Keywords:
Constrained Reinforcement Learning, Reinforcement Learning, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rg4dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that existing constrained reinforcement learning (CRL) underutilizes the information contained in the sampled trajectories. To address this, the paper extends phasic training to constrained reinforcement learning. An auxiliary phase reuses trajectories to refine reward and cost critics and distill value targets into the actor under a KL penalty. The framework is evaluated with CPO, CPPO-PID, and TRPO-PID on four tasks.

Critical strengths
  • The modular design applies across three distinct constrained-policy optimizers, making the proposed training extension practically relevant.
  • The two Goal tasks show consistent improvements in median return across all three backbones; several comparisons also reduce median cost.
Critical weaknesses
  • The auxiliary phase simultaneously updates the actor and both critics. The αcost=0\alpha_{\mathrm{cost}}=0 ablation removes actor-side cost distillation but retains reward distillation and critic replay; it therefore does not isolate representation transfer from additional critic fitting. Since dual-signal representation learning motivates the contribution, a critic-replay-only control is important.
  • The KL results require reconciliation with the stated constraint. Equation 5 and Appendix A specify (δclone=0.01)(\delta_{\mathrm{clone}}=0.01), whereas the final-epoch KL in Figure 5(b) appears mostly around 0.03–0.09. If these use the same definition, the plot does not establish enforcement of the stated budget; if they differ, the aggregation and conversion need explanation. The multiplier update, initialization, learning rate, and stopping or rollback rules are also insufficiently specified.
  • The safety statistics do not fully establish performance at comparable feasibility. The problem constrains expected cost, while the main results report medians across seeds, with several proposed-method entries above the budget of 25. The deployment distributions are valuable but do not replace per-policy mean-cost estimates and uncertainty. This is a request to assess the stated expectation constraint, not to require every episode to remain below budget (Tables 1–2; Appendix E).
  • Prior work on safe representation learning is insufficiently discussed, e.g., FCSRL [ICML 2024].
Questions for authors

Please refer to the weaknessese section.

1 comment
  1. Thank you for pointing out the need for a critic-only control. We conducted the ablation for our CPPO-PID based version CPPGPID and added the results to the revision of our paper (Appendix F). The results show that both, the critic-only as well as the distillation-only versions of our method achieve strong results and retain most of the gains of our full method. However, the full-method has shown to achieve higher results than the ablated versions, while only incurring higher costs in some of the environment. The best cost-adhering ablation, which is critic-only, lacks behind the full method and the distillation-only ablation when it comes to reward performance.

  2. Thank you for catching this discrepancy. The original Figure 5(b) summed KL across HalfCheetahVelocity’s six action dimensions, while training applies the 0.01 target to the mean across minibatch states and action dimensions. We revised Eq. (3) and Eq. (5) accordingly and divided the plotted values by six. On the corrected scale, 8 of 31 phases in the plotted run still end above 0.01, due to the nature of the Lagrangian formulation which does not strictly enforce exceeding the target. Following your request, we also added information on the full update rule to Appendix A in our revised paper. We initialize beta_clone=1 and take one Adam step per auxiliary minibatch, immediately before the actor update, with a learning rate of 0.01. The multiplier and optimizer state carry over between phases. We hope that we could clarify your questions.

  3. We assess safety using cumulative episode cost, following prior Safety-Gymnasium benchmarks such as OmniSafe. Our main tables report the median of the per-seed mean episode costs and a 95% bootstrap confidence interval across seeds. This shows how typical performance compares with the expected-cost budget of 25. Our environments are still constrained by an expected value target and judged by an expected value target. What changed is that we utilize robust statistics to aggregate across runs, as shown by Agarwal’s seminal work “Deep Reinforcement Learning at the Edge of the Statistical Precipice”, which demonstrates that simple averaging over independent runs might produce misleading results due to DeepRL’s high noise floor. For this reason, they motivate aggregating multiple seeds via robust statistics and using robust scale estimators like bootstrap CIs instead of mean and standard deviation across runs. This does not alter the individual runs, it just tightens the center and scale estimate during aggregation. We will clarify this in the main paper.

  4. Thank you for pointing us to FCSRL. We added it to the Background section, where we acknowledge its methodological closeness to our approach in cost-aware representation learning while highlighting that its on-policy PPO-Lag variant follows the same collect, update, and discard cycle we study. Where our experiments showed that for CPO, CPPO-PID, and TRPO-PID, retaining rollouts after the standard updates improved returns on both goal-reaching tasks at the same interaction budget.

Thank you again for giving us a positive rating. We hope we could address your concerns adequately and would greatly appreciate if you could raise your score accordingly.

Reviewer_Up47OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adds a PPG-style auxiliary phase to on-policy constrained RL algorithms, distilling reward and cost values into the policy network under a KL constraint. It improves return on the goal-reaching navigation tasks, while the results are mixed on the button task and worse on the locomotion task.

Critical strengths

The wrapper is simple and leaves each backbone's constraint handling untouched, and the gains on the goal-reaching tasks hold for every backbone. I also appreciate that the paper is candid about where the method fails.

Critical weaknesses
  1. The contribution is incremental. The method is essentially PPG placed on top of existing CRL backbones, and the adaptive KL constraint follows PPO's adaptive penalty. The only CMDP-specific part, cost distillation, does not seem to matter, since most of the gain remains when it is switched off.
  2. Some results do not fit together. The main CPPGPID results are better than every setting in the α_cost sweep, the default α_cost is not part of that sweep, and the stated training budget does not match the learning curves. The headline gain for CPPO-PID depends on these results.
  3. The source of the gain is not isolated. The phasic variants also give the critics extra training and add an extra policy update, but there is no component ablation, no compute-matched baseline and no wall-clock comparison, and the representation-learning claim in the title is never tested.
  4. Safety is not analyzed. The auxiliary update changes the policy without regard to cost, the KL budget appears to be exceeded on the locomotion task, and constraint satisfaction is judged by medians instead of expected cost or violation rates.
Questions for authors
  1. How were the main CPPGPID numbers obtained? Were they produced with the same code, seeds and training budget as the α_cost sweep, and how was α_cost = 0.3 chosen? Please add α_cost = 0.3 to Fig. 4 and Tables 3-6 under identical settings.
  2. Are Tables 1-2 measured after 10⁶ or 10⁷ environment steps? Do all baselines use the same network size (2×256), number of policy and value epochs, minibatch size and step budget, and were they tuned? The CPPO-PID baseline (10.5 and 10.2 on the Goal tasks) is much weaker than TRPO-PID and CPO, which inflates the largest reported gain.
  3. Could you report (a) the backbone with only the extra critic training of Alg. 1, (b) auxiliary heads without the extra critic training, (c) shuffled auxiliary targets under the same KL budget and (d) a compute-matched backbone, together with wall-clock times?
  4. How exactly is the KL multiplier updated (step size, update frequency, initialization, reset between phases), and why does the KL on HalfCheetahVelocity exceed δ_clone? Since α_cost = 0 also underperforms on that task (Table 5), do the results change with normalized value targets or a smaller δ_clone?
  5. Please report mean deployment costs, the fraction of episodes whose cost exceeds d, and the episodic cost right before and after each auxiliary phase.
2 comments

First and foremost we want to thank the reviewer for the time and efforts reviewing our paper.

Regarding your weaknesses:

  1. Thank you for pointing that out. Indeed, our work builds on top of known work and existing concepts but we believe that exactly this is the key benefit and insight of our work: that for a fixed methodological backbone, a different sample reuse strategy is able to achieve remarkably better results in some and generally better results for object-centric navigation environments. However, through your feedback, we added an evaluation to our work that sharpens the contribution of the CMDP-specific part (Table 9, Appendix F). It shows that in 11 out of 12 algorithm, environment combinations, the cost-aware augmented distillations achieve lower costs than the reward-only distilled methods. Your argument that most of the gain remains when the cost-aware distillation is switched off remains true – however we believe that our work is still an important contribution because the evidence we present in our revised work hints that features learnt from a dedicated cost-critic can be used to achieve better cost-adherence of the underlying actor. In general, the objective of this work is not to prove that the specific intervention of phasic policy gradients is universally “the right thing to do”, and more to show that existing CRL methods actually perform better than their benchmarking results suggest: The problem lies in the estimator quality, not the algorithm. Our contribution therefore should be seen less as a specific algorithm, and more as a way to inform future development: If the limiting factor is the quality of estimator, not the quality of algorithm, that changes where researchers should spend their time.

  2. Thank you for pointing this out. We corrected the training-budget discrepancy - the main table results were also obtained from 10^7 environment interactions. Furthermore, we clarified that alpha_cost=0.3 was selected in a preliminary sweep and thus chosen for the main comparison. In our revised work, we have combined the alpha_cost sensitivity results with the main results(Appendix C), which led to further valuable evaluation results. Does this resolve your concerns?

  3. We agree with the reviewer here. Following the feedback, we conducted a critic- and distillation-only ablation study for CPPGPID. (the last results are still pending) We added the ablation results to the revision of our work in Appendix F, Table 10.

We agree that a dedicated computationally matched comparison between our method and the backbone algorithms could benchmark the effect of our phasic extension with respect to the total computational requirements. In short, we operate under the assumption that compute is plentiful but environment interactions are very expensive, which is a common assumption in Reinforcement Learning. Our aim was to show that our phasic wrapper can achieve superior performance applied to multiple backbone algorithms compared to their standalone version under the same interaction budget. In fact, we are not entirely clear how you would try to compute match the baselines: The direct avenue would be to add additional training steps per datapoint, but this is already what our method does. This means that compute matching the baseline would inevitably build an experiment which is close or identical to our phasic experiments. We agree with your point, that the claims about representation learning were too strong. We believe that the CMDP that methods solve are essentially a downstream measure of how useful the learned actor features are – and that this follows the practice of downstream evaluation in self-supervised learning (e.g., SimCLR, Chen et al., 2020). However, we admit that we do not have a dedicated alternative experiment to prove the better representational quality and consequentially we have attenuated our wording in the conclusion of our revised paper. We hope that we could resolve this weakness with the undertaken measures.

  1. We assess safety using cumulative episode cost, following prior Safety-Gymnasium benchmarks such as OmniSafe. Our main tables report the median of the per-seed mean episode costs and a 95% bootstrap confidence interval across seeds. This shows how typical performance compares with the expected-cost budget of 25. We believe this is the same point we elaborated to W12Z: Our environments are still constrained by an expected value target and judged by an expected value target. What changed is that we utilize robust statistics to aggregate across runs, as shown by Agarwal’s seminal work “Deep Reinforcement Learning at the Edge of the Statistical Precipice”, which demonstrates that simple averaging over independent runs might produce misleading results due to DeepRL’s high noise floor.

For this reason, they motivate aggregating multiple seeds via robust statistics and using robust scale estimators like bootstrap CIs instead of mean and standard deviation across runs. This does not alter the individual runs, it just tightens the center and scale estimate during aggregation. We will clarify this in the main paper. Regarding the doubts on changing of the policy in the auxiliary phase without considering the costs. The auxiliary actor update underlies the adaptive KL penalty targeting 0.01 per action dimension, while the backbone applies its usual constraint-handling rule when the policy is updated. Whether a trained policy meets the expected-cost budget is an empirical question assessed through its cumulative episodic cost. In the main tables, phasic variants have lower median episodic costs than their corresponding backbones in nine of twelve comparisons – despite the unconstrained feature distillation during the auxiliary phase.

To address your questions:

1.Thank you for pointing this missing information out, we clarified this in the paper that we obtained alpha_cost=0.3 from a preliminary sweep. The results from the main table were obtained under exactly the same conditions (code, seeds & training budget) as we used for the alpha_cost sweep. In our revised work, we added your requested addition of the alpha_cost=0.3 results to Figure 4.

  1. We updated the caption of the tables, which were obtained - similarly as the training curves - after 10^7 steps. Thank you for pointining this out - this was a labeling error. For the hyperparameters, we used the published OmniSafe benchmark configurations for all the baseline methods, which are the following:
MethodsActor/ Critic MLP sizesMinibatch sizeActor update per rolloutPasses per criticTarget KL
CPPOPID, FOCOPS, PDO, PPOLag[64, 64]64Up to 40 minibatch passesUp to 400.02
CUP[64, 64]64Up to 40 minibatch passes, then up to 40 actor-only projection passesUp to 400.01
TRPOPID, OnCRPO[64, 64]128One update attempt with line search100.01
CPO, PCPO[64, 64]128One update attempt with line search100.01
RCPO[64, 64]128One natural policy-gradient step100.01

Indeed, the CPPOPID baseline performance corresponds to the weak performance reported in the Omnisafe benchmark compared to CPO and TRPOPID.

  1. Thank you for suggesting these controls. We conducted a critic-/ and distillation-ony ablation for CPPGPID and added the so far obtained results to our revised work in appendix F (Table 10). This control shows that either mechanism alone achieves nearly en–par performance to the full method. Even though individual ablations might perform well on a few specific tasks, when we look at the big picture across all tasks using a robust ranking metric (Table 11), the complete method consistently comes out on top overall. As we touched on in our response to weakness point (3), we hope those details help answer your question regarding a compute-matched comparison.

4.Thank you for flagging the KL discrepancy. We added Table 3 in Appendix A with the full multiplier update information. We initialize beta_clone at 1 and take one Adam step per auxiliary minibatch, with a learning rate of 0.01. The multiplier and optimizer state carry over between phases. The original Figure 5(b) summed KL over HalfCheetahVelocity’s six action dimensions. Training averages over those dimensions, so the target of 0.01 corresponds to 0.06 on the original scale. We corrected Equation (5), (3) and Figure 5(b) accordingly to express KL per action dimension. On the corrected scale, 8 of 31 phases in the plotted run still end above 0.01. The adaptive penalty steers KL toward the target through minibatch updates, so the final KL can exceed it. The effects of normalizing the value targets or lowering delta_clone remain open in the reported experiments.

  1. We believe that we addressed the cost related part of this question with our response to weakness point (4). While we fully acknowledge that the episodic cost before and after the auxiliary phase would be informative, we cannot provide data for this.

Thank you again for your thoughtful review. Do you have additional questions?

Reviewer_W12ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The ablations show that setting the cost-distillation weight to zero often produces comparable performance. For example, PTRPOPID achieves approximately the same reward on PointGoal1 without cost-value distillation. This raises the question of whether the improvements primarily arise from conventional reward-based phasic learning rather than the proposed cost-aware extension.

The framework is evaluated with CPO, CPPO-PID, and TRPO-PID across four Safety Gymnasium environments. Results show substantial improvements in navigation environments, although performance deteriorates on HalfCheetahVelocity (Tables 1–2, p. 7).

Critical strengths

The phasic extension consistently improves reward across all three backbone algorithms on CarGoal1 and PointGoal1 (Table 1, p. 7). Particularly notable is CPPGPID on CarGoal1, where the median reward increases from 10.5 to 29.0. The improvements across different policy optimization methods support the practical usefulness of the framework.

The experiments consider three constrained navigation tasks and a locomotion task (Section 4.1, p. 6). The consistent deterioration on HalfCheetahVelocity (Table 2, p. 7) is informative and suggests that the effectiveness of phasic learning depends on the observation and reward-cost structure. The authors appropriately acknowledge this limitation rather than claiming universal improvements.

Critical weaknesses

The proposed method closely follows the existing PPG framework described in Section 2.2 (p. 3, Equation 2). The principal modifications are the addition of a cost-value auxiliary objective (Equations 3–4, p. 4) and replacement of the fixed policy-cloning coefficient with an adaptive KL constraint (Equation 5, p. 4).

The evidence that cost-value distillation provides a substantial additional benefit is mixed. In particular, the ablations in Appendix C (Table 6, p. 14) show that PTRPOPID achieves a reward of 24.544 on PointGoal1 with the cost-distillation weight set to zero, compared with 24.5 for the main configuration reported in Table 1 (p. 7). Similarly, PhasicCPO achieves 24.353 with zero cost-distillation weight, exceeding the main configuration's reported reward of 23.7.

To be fair, cost distillation does improve some configurations, such as CPPGPID on CarGoal1 (Tables 1 and 4, pp. 7 and 14). However, the results do not demonstrate a consistent, substantial contribution beyond reward-only phasic training.

Furthermore, Algorithm 1 (p. 5) introduces additional auxiliary optimization epochs, with six epochs over the retained buffer in the default implementation (Appendix A, p. 12). Comparisons with computationally matched baselines are missing, making it difficult to distinguish improvements due to the proposed representation-learning mechanism from improvements due to additional training.

These concerns are consequential because the paper's primary contribution is presented as cost-aware representation learning rather than simply applying the established PPG training paradigm to constrained RL.

The CMDP formulation explicitly requires expected cumulative cost to remain below a prescribed budget (Equation 1, p. 2). The experiments use a cost budget of 25 (Section 4, p. 6).

However, several proposed configurations exceed this budget. In Table 1 (p. 7), CPPGPID on CarGoal1 increases median reward from 10.5 to 29.0, but also increases median cost from 22.6 to 26.0, moving from below to above the prescribed budget. Similarly, PhasicCPO reports a median cost of 33.9 on CarButton1 (Table 2, p. 7). Although this improves upon the corresponding CPO baseline's cost, it remains above the specified limit.

The main tables report medians across seeds rather than explicitly establishing expected-cost feasibility for each trained policy. Since the CMDP constraint concerns expected cost, a per-seed feasibility analysis would strengthen the evaluation.

A related concern arises with the adaptive KL constraint. Equation 5 (p. 4) specifies a target divergence of 0.01. However, Figure 5(b) (p. 13) appears to show KL divergences approximately between 0.04 and 0.09 for HalfCheetahVelocity under the proposed adaptive formulation. While the adaptive method substantially reduces KL divergence relative to the fixed-coefficient baseline, it does not appear to enforce the stated numerical threshold.

The authors acknowledge on p. 4 that the auxiliary update is not covered by CPO's original feasibility guarantee. This is an appropriate limitation, but it makes empirical validation of constraint satisfaction especially important.

Questions for authors
  1. Contribution of cost-aware distillation (Equations 3–4, p. 4; Tables 4 and 6, p. 14).

The cost-weight ablations show that setting the cost-distillation weight to zero often produces comparable performance. Can the authors quantify the incremental improvement attributable specifically to cost-value supervision, ideally with matched computational budgets and additional reward-only training controls? What evidence supports the claim that the cost signal improves the actor's learned representation rather than simply modifying auxiliary optimization?

  1. Adaptive KL constraint (Equation 5, p. 4; Figure 5(b), p. 13).

The proposed adaptive method targets a KL divergence of 0.01, yet Figure 5(b) appears to show substantially larger divergences on HalfCheetahVelocity. Is the reported divergence evaluated using precisely the same quantity and state distribution as Equation 5? If so, what explains the apparent constraint violations?

  1. Empirical constraint satisfaction (Equation 1, p. 2; Tables 1–2, p. 7).

Several proposed methods report costs above the budget of 25. Can the authors report the expected episodic cost for each seed, along with the fraction of trained policies satisfying the constraint? This would help establish whether the reward improvements represent better feasible solutions to the CMDP.

  1. Training budget and computational overhead (Figure 3, p. 6; Tables 1–2, p. 7; Appendix A, p. 12).

Can the authors clarify the discrepancy between the stated (10^6) training steps in the tables and the (10^7) steps shown in the learning curves? Additionally, how much computation and wall-clock time does the auxiliary phase introduce relative to each baseline?

3 comments

First and foremost thank you for your reviewing our paper.

Regarding the weaknesses:

  1. We acknowledge your point, that we could not show that cost-aware phasic training provides a clear and significant benefit over reward-based phasic feature distillation. However, we believe that the central point of our work is to show that on-policy CRL methods discard data before fully exploiting it. Following your feedback, to nevertheless sharpen the evidence that cost-aware distillation benefits the learning, we added Table 9 (Appendix F) to the revision of our paper. There we directly compare the performance of our method without cost-aware distillation with grouped evidence for cost-aware distillation. This comparison shows that across all backbone algorithms, for the object centric navigation tasks, cost-critic distillation achieved lower final costs compared to reward-only distillation runs. We hope that we could address your point - we position our work primarily as an investigation on how data reuse can benefit multiple backbone methods, while also sharpening the evidence for our second contribution, that using the CMDP’s cost signal as information source for distillation has a measurable influence on the algorithms performance.

  2. We agree that a dedicated computationally matched comparison between our method and the backbone algorithms could benchmark the effect of our phasic extension with respect to the total computational requirements. In short, we operate under the assumption that compute is plentiful but environment interactions are very expensive, which is a common assumption in Reinforcement Learning. Our aim was to show that our phasic wrapper can achieve superior performance applied to multiple backbone algorithms compared to their standalone version under the same interaction budget. Besides that, we compare against a highly tuned set of hyperparameters for the baseline from the Omnisafe publication, which we consider to be optimally tuned with respect to the amount of updates per samples. In fact, we are not entirely clear how you would try to compute match the baselines: The direct avenue would be to add additional training steps per datapoint, but this is already what our method does. This means that compute matching the baseline would inevitably build an experiment which is close or identical to our phasic experiments. Note that the objective of our representation learning mechanism is precisely to allow us to increase the update-to-data ratio without running into trouble by making the actor updates off-policy: If you want to enrich the training of an actor in an on-policy setup, you need prevent policy drift, which is what the KL constraint during the auxiliary epoch is designed to do. Does this clarify your question?

  3. We agree that the reported median costs for both configurations exceed the budget of 25. We still consider our results important, since they show that despite of a method’s inability to adhere to the cost budget, e.g. CPO on CarButton1, the phasic extension has shown to reduce its cost (from 35.1 to 33.9) while keeping comparable returns to the baseline. For CarGoal1, where the phasic extension increased the median cost from 22.6 to 26.0, we believe that from the perspective of the underlying return–cost trade-off, the concomitant increase in reward from 10.5 to 29.0 is still a clear evidence for the effectivity of our method. We do think it is important to highlight that CRL with hard guarantees (i.e. “every seed will definitely find the feasible set after 10 Mio steps”) is still an open problem. Our method does not claim to completely resolve existing CRL limitations, but rather that existing CRL methods appear to be less limited on the algorithmic side, and much more limited on how well they can practically absorb the information already present. Nevertheless, we have added our per-seed results in the appendix: This reveals that phasic augmentation consistently increases the likelihood of an algorithm reaching the feasible set (e.g. in PointGoal, CPO is never under the 25 point budget, while its phasic counterpart PhasicCPO is below the budget in 4/10 runs). This reveals that, while more training cannot overcome the fundamental limitations of the algorithms, it can noticeably boost performance without needing to change the algorithm itself. Does the additional data resolve your concerns?

  1. We see why there might be some confusion regarding our benchmarking: What we show in our tables are the expected returns with confidence intervals across multiple seeds. I.e. we (1) Run all configurations multiple times independently and compute the expected reward and expected costs. (i.e. the standard CMDP setup) (2) To also model the run-to-run variance, we then provide a confidence interval using bootstrap CI and the median performance of multiple runs. This has been shown necessary to make robust comparisons between RL algorithms, see “Deep Reinforcement Learning at the Edge of the Statistical Precipice” by Agarwal et al.

The actual constraint is still an expected cost constraint (and the reward an expected reward), but we use robust estimators to aggregate performance across runs. We consider this best-practice for DeepRL where run-to-run variance is a major concern when comparing different algorithms. We have improved the description of our evaluation setup to clarify our exact benchmarking scheme.

However, we still agree to your argument that a more detailed per-seed feasibility analysis strengthens the reporting and thus we added this to our revised work in Appenix I.

Regarding the questions:

  1. Thank you for pointing to this missing comparison in our work. As already discussed at point 1 of the weaknesses, we added an evaluation to the revision where we specifically compare enabled and disabled cost-distillation and their influence on the final performance of the method. We hope that we could clarify this point and answer your request to quantify the improvement attributable to cost-value supervision within the auxiliary phase (Appendix F, Table 9).

With respect to your second question, which evidence supports that the learned representations are the driver behind these gains, we agree that we don’t provide explicit evidence for that. Consequentially, we adapted the wording in the conclusion of the paper, which previously hinted at representational quality as lever to superior performance in constraint policy optimization. Still, we argue that similar to other unsupervised-/ representation learning publications, which test their effectivity with the performance of learnt representations on downstream tasks (e.g. BYOL), one can take the perspective that the underlying CMDP that we solve is essentially this downstream task.

  1. We thank the reviewer for the careful review and drawing our attention to the apparent mismatch between Eq. (5) and Figure 5(b). This observation led us to check the implementation and identify a discrepancy in the KL normalization between Eq. (3), Eq. (5) and the implementation. The implementation applies the KL target of 0.01 to the mean KL across action dimensions. We therefore revised Eq. (3) and Eq. (5) by adding a factor of 1/d_act before the expectation, where d_act is the number of action dimensions. We also revised Figure 5(b) by dividing the plotted KL values by d_act and adding a dotted horizontal line at the KL budget of 0.01. We also added a table to the appendix detailing how we initialize and update the auxiliary KL multiplier beta_clone. The still appearing constraint violation is explained by the Lagrangian formulation, which post-hoc punishes overshoots and steers the updated Lagrangian parameter only in the next iteration.

  2. We added a more detailed overview for the costs -per seed, giving a more detailed insight to the median value we reported in our main table. As mentioned above, we have improved the description of our benchmarking recipe to clarify that each individual run still is constrained by expectation: The bootstrap CI and median construction is used to follow best practices in reproducible RL experimentation (see Agarwal, 2021)

  3. We appreciate the reviewer catching the inconsistency in the reported training budget. The results in Tables 1–2 were measured after 10^7 environment steps, consistent with the learning curves in Figure 3. The 10^6 stated in the tables was a labeling error, which we corrected. As noted above, we deliberately designed our experiments to assess performance at a fixed environment-interaction budget, treating environment interactions as the scarce resource. We consider them as computational bottleneck in (C)-RL, in comparison to the lightweight MLP’s behind actors/critics and buffers to store the data over multiple sample epochs. We found that the computational overhead is generally small, since environment simulations make up the large majority of wall-clock time.

We sincerely thank you for your valuable feedback. We hope that our responses and additions to the paper have addressed the raised concerns and would greatly appreciate it if you would consider increasing your score accordingly.

Thank you for the thoughtful and detailed rebuttal and for the substantial effort invested in revising the manuscript. I appreciate the authors' constructive engagement with my concerns.

Several clarifications and additions are convincing. In particular, the correction of the KL normalization inconsistency satisfactorily addresses my concern regarding Equation 5 and Figure 5(b). The clarification of the training budget also resolves the reported discrepancy. Furthermore, I find the argument for evaluating algorithms under a fixed environment-interaction budget reasonable, given the importance of sample efficiency in reinforcement learning.

The additional experiments in Appendix F are valuable. Table 9 provides evidence that cost-aware distillation has a measurable effect, particularly on reducing costs. I acknowledge that this strengthens the empirical support for incorporating cost information into the auxiliary phase.

Nevertheless, I remain somewhat unconvinced that cost-aware distillation consistently provides a substantial advantage over reward-only phasic learning. While the additional results show improvements in cost, the corresponding reward improvements are mixed. I also appreciate the new per-seed feasibility analysis, which provides a clearer picture of constraint satisfaction. These results demonstrate improvements for some configurations, but also confirm that substantial reward gains can sometimes accompany reduced feasibility, as illustrated by CPPGPID on CarGoal1.

I recognize the practical value of extending phasic training to multiple constrained RL algorithms and the demonstrated improvements in sample efficiency on navigation tasks. However, my principal reservation remains that the methodological contribution is relatively incremental with respect to the established PPG framework, while the evidence for the distinct benefits of cost-aware supervision and improved constrained performance remains mixed.

Comments on the paper

2 comments

As the discussion period is approaching its end, please take a moment to review the authors’ responses and share any remaining thoughts or updates to your assessment. Thanks for your thoughtful reviews and discussion!

First and foremost we thank all reviewers for their thorough and detailed review.

As a quick overview, we provide a summary of reviewers concerns and our changes in the document (in no particular order).

Can the authors quantify the incremental improvements attributable to cost-value supervision:

We clarified that the objective of our paper is to showcase the general notion of data underutilization with phasic updates just acting as an example. Nevertheless, we have added an additional table into the appendix demonstrating the performance under different distillation recipes

Provide a computationally matched baseline:

We clarify that the paper’s argument is precisely that increasing the compute spent on CRL algorithms yields significant improvements. This means providing “computationally matched” variants of algorithms would be identical to the existing methods

Some algorithms exceed the budget:

We note that this is expected since not all CRL algorithms are safe on all environments. For those algorithms we nevertheless demonstrate a reduction in excess costs

What is the effect of the median aggregation:

We clarify that the median/bootstrap CI is across seeds not within environments, following established best-practices in RL benchmarkint. Individual environments retain their expected value constraints

Mismatch between KL barrier and reported results:

We repaired a presentation error that erroneously showed summed instead of averaged KL errors

Why alpha_cost=0.3:

We clarify that this is from a previous hyperparameter sweep and additionally provide results for different alpha in the appendix

Isolate the impact of cost and reward distillation:

We have added results for this experiment in Appendix C.

Prior work on safe representation learning is insufficiently discussed:

We have added FCSRL to the background section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.