- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies quantile-constrained RL, reformulated as a bound on violation probability. It uses a budget-augmented distributional critic to estimate the change in violation probability and uses it as a risk advantage in a Lagrangian PPO algorithm. Theoretically, the paper provides convergence guarantees with a three time-scale learning scheme and also a conditional estimation error bound. The experiment section is, however, limited to a synthetic downside risk and a SimpleButton navigation task, with two baselines: QCPO and TQPO.
Critical strengths
- The paper presents a coherent approach to chance-constrained RL, combining a budget-augmented distributional critic with per-step risk feedback.
- The theoretical analysis provides error bounds and convergence guarantees for a learning scheme under stated stability assumptions.
Critical weaknesses
-
Chow et al. (2018) already develop a chance-constrained actor–critic method with budget-based state augmentation and TD updates that propagate terminal violation penalties to earlier decisions. Although cited, it receives limited discussion. A more detailed comparison would clarify the methodological contribution of DCA.
-
Experiments cover synthetic downside-risk environments and a single navigation task, with only QCPO and TQPO as baselines. A comparison with an actor–critic method similar to Chow et al., which use a scalar violation-probability critic with the same augmented state, policy, and multiplier updates, would be beneficial. A risk-neutral baseline would also provide further context.
-
The number of tested environments is limited. Additional Safety-Gymnasium tasks would strengthen evidence of generality.
-
The paper does not examine sensitivity to the number of quantiles or whether small differences between estimated CDF values remain reliable relative to critic approximation and action-sampling errors. These analyses would help assess the robustness of the proposed risk-credit signal.
-
Minor: Table 2 in Appendix E.3 bolds DCA’s mean return although TQPO achieves a higher value.
Questions for authors
The main gap is a thorough theoretical and empirical comparison with Chow et al. (2018), which would clarify the proposed method’s novelty and practical benefits.
7 comments
W4: Quantile count and reliability of small risk differences.
In H20, varying the quantile count at or the number of auxiliary actions at during training yields similar mean returns and violation rates (Table D-1).
Table D-1: H20 DCA policy performance across and . Each final policy is evaluated on 100,000 episodes. Mean ± standard deviation over ten seeds; feasibility requires test violation rate ≤20%. Fixed learning rates: 0.001 for the actor, 0.002 for both critics.
| Setting | Mean return | 20th-percentile return | Violation rate (%) | Feasible seeds | ||
|---|---|---|---|---|---|---|
| Default | 64 | 4 | 27.57 ± 0.51 | 16.67 ± 0.82 | 18.17 ± 2.20 | 9/10 |
| Vary | 32 | 4 | 27.61 ± 0.37 | 16.76 ± 0.75 | 18.51 ± 1.77 | 10/10 |
| 128 | 4 | 27.49 ± 0.45 | 16.94 ± 0.80 | 18.08 ± 2.04 | 10/10 | |
| 256 | 4 | 27.50 ± 0.47 | 16.84 ± 0.80 | 18.34 ± 2.05 | 10/10 | |
| Vary | 64 | 2 | 27.58 ± 0.49 | 16.72 ± 0.90 | 18.07 ± 2.37 | 8/10 |
| 64 | 8 | 27.57 ± 0.50 | 16.74 ± 0.87 | 18.18 ± 2.21 | 8/10 | |
| 64 | 16 | 27.57 ± 0.55 | 16.65 ± 0.87 | 18.36 ± 2.16 | 9/10 | |
| 64 | 64 | 27.58 ± 0.52 | 16.64 ± 0.87 | 18.28 ± 2.16 | 7/10 | |
| 64 | 128 | 27.55 ± 0.53 | 16.71 ± 0.87 | 18.16 ± 2.26 | 9/10 |
We then fix and vary over 32, 64 (baseline), 128, and 256. To prevent the choice of critic from changing the policy and the data it collects, we train all four on the same trajectories in each of ten seeds. Only the baseline critic supplies the risk signal for the actor update. We then vary for each frozen critic on the same evaluation transitions. We compute the reference violation probabilities by backward numerical integration under each frozen policy. Table D-2 reports mean absolute errors (MAEs) of state-action and state violation probabilities; their difference determines the actor’s risk signal.
Table D-2: H20 violation probability and risk-credit errors across and . MAEs are in percentage points, averaged over ten seeds.
| Quantity | ||||
|---|---|---|---|---|
| State-action violation probability | 2.32 | 1.89 | 2.21 | 2.50 |
| State violation probability (integrated) | 2.31 | 1.84 | 2.20 | 2.48 |
| , | 0.93 | 0.73 | 0.69 | 0.70 |
| , | 0.83 | 0.62 | 0.60 | 0.62 |
| , | 0.80 | 0.60 | 0.58 | 0.60 |
| , | 0.79 | 0.59 | 0.58 | 0.60 |
| , integrated | 0.79 | 0.58 | 0.58 | 0.59 |
The risk-credit signal depends on and :
- Fixing , the MAE of at iteration 400 decreases as goes from 32 to 64 to 128, then rises slightly at 256. This holds for every tested . At , the errors are 0.83, 0.62, 0.60, and 0.62 percentage points, respectively.
- Fixing , increasing reduces the MAE of for every critic during training (supplementary figure). At and iteration 400, the MAE falls from 0.73 percentage points at to 0.62 at and 0.59 at . For all four , the MAE at is within 0.0021 percentage points of the MAE obtained by integrating the same critic over actions. Here, 64 sampled actions per state suffice to estimate the critic's violation probability averaged over actions under the policy.
Table D-3: H20 risk-credit sign accuracy, zero predictions, and MAE by reference magnitude . Groups use the reference change.
| (%) | Correct sign | Reversed sign | Zero | MAE (%) | ||
|---|---|---|---|---|---|---|
| 64 | 4 | 3.69% | 1.93% | 94.38% | 0.12 | |
| 128 | 4 | 4.36% | 1.26% | 94.38% | 0.11 | |
| 256 | 4 | 15.96% | 2.01% | 82.03% | 0.12 | |
| 256 | 32 | 44.33% | 1.64% | 54.03% | 0.10 | |
| 256 | 64 | 54.36% | 1.41% | 44.23% | 0.10 | |
| 64 | 4 | 31.56% | 2.54% | 65.90% | 0.31 | |
| 128 | 4 | 75.10% | 1.40% | 23.50% | 0.29 | |
| 256 | 4 | 96.44% | 1.24% | 2.32% | 0.29 | |
| 256 | 32 | 98.97% | 0.92% | 0.11% | 0.27 | |
| 256 | 64 | 99.02% | 0.94% | 0.04% | 0.27 | |
| 64 | 4 | 98.27% | 1.03% | 0.70% | 0.40 | |
| 64 | 4 | 98.90% | 1.10% | 0.00% | 1.05 |
In Table D-3, the default setting correctly identifies the risk-change direction in 98.27% of estimates for in [0.5%,1%) and 98.90% for in [1%,100%). For [0.1%,0.5%), the correct-sign rate drops to 31.56%, with 65.90% of estimates reporting zero change. Increasing to 256 at fixed raises the correct-sign rate to 96.44%, and increasing to 64 raises it further to 99.02%. For (0.001%,0.1%), the same increases in and raise the correct-sign rate from 3.69% to 54.36%.
W5: Minor issue
Thank you for pointing this out. TQPO had the highest mean return in Table 2 of the original submission. The revised SimpleButton comparison uses matched undiscounted targets across methods. Table 12 in Appendix E.2.2 now bolds the highest mean return at each violation limit.
Q1: Comparison with Chow et al. (2018)
Q1-1: Empirical comparisons.
As suggested, we add a comparison with the chance-constrained actor–critic method in Section 5.2 of Chow et al. (2018). We adapt it to our continuous-action H20 task. We use discounted negative rewards as costs, preserving the same violation event. The state includes the remaining budget. The critic estimates the value including the terminal violation penalty. Following Chow et al. (2018), we use a linear critic, diminishing step sizes, and projected multiplier updates. The actor and critic update after each episode using per-step TD errors.
We compare the methods at α = 20%, 10%, 5%. The return threshold is 16 and the discount factor is 0.99. Both methods use ten seeds and the same training interaction budget. Results are summarized in Table E. We will provide implementation details in the revised manuscript. Training curves are available in the anonymous supplementary figures.
Table E: Comparison with Chow-AC in H20. Values are mean ± standard deviation across ten seeds. is the return quantile at the corresponding risk limit. Feasible seeds count final policies whose empirical violation rate does not exceed that limit.
| Risk limit | Method | Mean return | Violation (%) | Feasible seeds | |
|---|---|---|---|---|---|
| 20% | DCA | 27.57 ± 0.51 | 16.67 ± 0.82 | 18.17 ± 2.20 | 9/10 |
| 20% | Chow-AC | 26.37 ± 0.98 | 16.43 ± 1.37 | 18.91 ± 3.06 | 7/10 |
| 10% | DCA | 25.09 ± 1.61 | 14.34 ± 2.92 | 10.54 ± 1.01 | 4/10 |
| 10% | Chow-AC | 24.19 ± 1.23 | 14.04 ± 1.52 | 11.96 ± 1.60 | 1/10 |
| 5% | DCA | 23.39 ± 1.86 | 0.83 ± 12.09 | 8.89 ± 1.22 | 0/10 |
| 5% | Chow-AC | 22.80 ± 1.12 | 2.70 ± 6.80 | 9.87 ± 1.25 | 0/10 |
The results show that: (1) At the 20% limit, DCA achieves a higher mean return with similar violation rates and more feasible seeds. (2) At the 10% limit, DCA achieves a higher mean return and a lower violation rate, although both methods exceed the limit on average. (3) At the 5% limit, DCA achieves a higher mean return and a lower violation rate than Chow-AC, although neither method satisfies the constraint in any seed.
Q1-2: Theoretical comparisons.
Please refer to our response to W1 for the methodological comparison. In addition, we clarify two theoretical differences below:
- The analysis of Chow-AC assumes a uniformly bounded time horizon to reach a target state (Assumption 2 in Chow et al., 2018). We study infinite-horizon discounted returns without this termination requirement. Our Assumption 1 controls the probability mass near the return threshold. This allows us to control the contribution of late decisions. We also bound the error from truncating the risk-gradient sum (Proposition 4).
- Our Theorem 1 separates trajectory-sampling error, critic-estimation error, and action-sampling error. Corollary 1 gives the corresponding result for transition-based risk feedback. Proposition 3 links the distributional Bellman residual to conditional-CDF error and risk-gradient error. These results explain how distributional approximation and finite sampling affect the accuracy of risk feedback.
We will clarify these theoretical distinctions in the revised manuscript.
Author Response to Reviewer WPpD
We sincerely thank Reviewer WPpD for the constructive and valuable comments. The concerns are addressed as follows.
W1: Relation to Chow et al. (2018).
We agree that Chow et al. (2018) already introduced budget-augmented states and TD updates that propagate terminal violation penalties to earlier decisions. We refer to their chance-constrained actor–critic method in Section 5.2 as Chow-AC. DCA shares this foundation and differs in the following aspects:
Our main methodological contribution is the distributional construction of risk feedback. We learn the conditional future-return distribution, evaluate its CDF at the remaining return requirement, and use the resulting violation probabilities to construct stepwise risk feedback. This feedback is then incorporated into PPO updates. We have also added an empirical comparison with Chow-AC. Please see our response to Q1 for the experimental setup and results. We will expand the discussion of these connections in the revised manuscript.
W2: Additional scalar and risk-neutral baselines.
Please refer to our response to Reviewer DeDA, W2-1, for the comparison against a scalar violation-probability critic. We also add a risk-neutral PPO baseline in the original Downside-risk H20 and H40 settings. It uses the same actor, reward critic, and PPO settings as DCA, with the risk multiplier fixed at zero. The results are reported in Table C. Risk-neutral PPO violates the risk constraint in both settings. Compared with DCA, it achieves higher mean returns but lower 20th-percentile returns. These results illustrate a trade-off between mean return and lower-tail quantile performance in these tasks.
Table C: Risk-neutral comparison. Values are mean ± standard deviation over ten seeds. Feasible seeds count runs whose final test violation rate is at most 20%. Risk-neutral PPO is newly trained; the other methods use the original experimental results.
W3: Additional tasks.
Thank you for the suggestion. We have added two synthetic environments, Gaussian and Orders, in the comparison with the scalar risk critic. Please see our response to Reviewer DeDA, W2-1 for details. We also report results for SimpleButton at the 10% violation limit and for Dynamic at the 20% limit. Please see our responses to Reviewer DeDA, W1-1 and W1-2, respectively. The corresponding results appear in Appendices E.2.2 and E.2.3.