acceptodds
Under review as a conference paper at ICLR 2027

Quantile-Constrained Reinforcement Learning via Distributional Credit Assignment

Abstract

In many sequential decision-making applications, such as healthcare and inven- tory control, the goal is not only to achieve high average returns but also to avoid poor lower-tail outcomes. We study expected-return maximization under a lower- quantile constraint and focus on the key challenge of assigning trajectory-level risk to individual decisions. Using an equivalent violation-probability formula- tion, we propose distributional credit assignment with two key components. (1) At each step, we use accumulated rewards to determine how much future return is needed to meet the constraint. We then define a transition-based risk advantage from the change in violation probability between consecutive augmented states. (2) We estimate these violation probabilities from conditional future-return dis- tributions learned through distributional Bellman updates. This uses intermedi- ate reward magnitudes to estimate risk, rather than relying solely on terminal binary outcomes. We combine the resulting risk advantages with reward advan- tages in proximal policy optimization (PPO). Theoretically, we derive a density- free risk-gradient representation, provide error bounds for risk-gradient estima- tion, and establish convergence under a three-time-scale stochastic-approximation formulation. Experiments in synthetic mean-risk trade-off and SimpleButton environments show that our method provides favorable performance in balanc- ing expected return and lower-tail constraints compared with existing quantile- constrained baselines.

Primary area:
reinforcement learning
Keywords:
Quantile-Constrained Reinforcement Learning, Distributional Reinforcement Learning, Distributional Credit Assignment, Chance-Constrained Reinforcement Learning, Risk-Aware Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WPpDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies quantile-constrained RL, reformulated as a bound on violation probability. It uses a budget-augmented distributional critic to estimate the change in violation probability and uses it as a risk advantage in a Lagrangian PPO algorithm. Theoretically, the paper provides convergence guarantees with a three time-scale learning scheme and also a conditional estimation error bound. The experiment section is, however, limited to a synthetic downside risk and a SimpleButton navigation task, with two baselines: QCPO and TQPO.

Critical strengths
  • The paper presents a coherent approach to chance-constrained RL, combining a budget-augmented distributional critic with per-step risk feedback.
  • The theoretical analysis provides error bounds and convergence guarantees for a learning scheme under stated stability assumptions.
Critical weaknesses
  • Chow et al. (2018) already develop a chance-constrained actor–critic method with budget-based state augmentation and TD updates that propagate terminal violation penalties to earlier decisions. Although cited, it receives limited discussion. A more detailed comparison would clarify the methodological contribution of DCA.

  • Experiments cover synthetic downside-risk environments and a single navigation task, with only QCPO and TQPO as baselines. A comparison with an actor–critic method similar to Chow et al., which use a scalar violation-probability critic with the same augmented state, policy, and multiplier updates, would be beneficial. A risk-neutral baseline would also provide further context.

  • The number of tested environments is limited. Additional Safety-Gymnasium tasks would strengthen evidence of generality.

  • The paper does not examine sensitivity to the number of quantiles or whether small differences between estimated CDF values remain reliable relative to critic approximation and action-sampling errors. These analyses would help assess the robustness of the proposed risk-credit signal.

  • Minor: Table 2 in Appendix E.3 bolds DCA’s mean return although TQPO achieves a higher value.

Questions for authors

The main gap is a thorough theoretical and empirical comparison with Chow et al. (2018), which would clarify the proposed method’s novelty and practical benefits.

7 comments

Author Response to Reviewer WPpD

We sincerely thank Reviewer WPpD for the constructive and valuable comments. The concerns are addressed as follows.

W1: Relation to Chow et al. (2018).

We agree that Chow et al. (2018) already introduced budget-augmented states and TD updates that propagate terminal violation penalties to earlier decisions. We refer to their chance-constrained actor–critic method in Section 5.2 as Chow-AC. DCA shares this foundation and differs in the following aspects:

  • Chow-AC learns a scalar value function for the Lagrangian objective, which combines the objective target and the terminal violation penalty. DCA learns the conditional future-return distribution through distributional Bellman updates.
  • Chow-AC uses TD errors from the Lagrangian critic to provide combined objective and constraint feedback. DCA evaluates the learned return CDF at the remaining return requirement to estimate conditional violation probabilities. These probabilities are used to construct the explicit risk signal Up,tU_{p,t}, which is combined with a separately estimated reward advantage.
  • Chow-AC applies projected policy-gradient updates after each episode. DCA incorporates the reward and risk advantages into a PPO clipped surrogate and performs multiple minibatch updates on each batch of trajectories.

Our main methodological contribution is the distributional construction of risk feedback. We learn the conditional future-return distribution, evaluate its CDF at the remaining return requirement, and use the resulting violation probabilities to construct stepwise risk feedback. This feedback is then incorporated into PPO updates. We have also added an empirical comparison with Chow-AC. Please see our response to Q1 for the experimental setup and results. We will expand the discussion of these connections in the revised manuscript.

W2: Additional scalar and risk-neutral baselines.

Please refer to our response to Reviewer DeDA, W2-1, for the comparison against a scalar violation-probability critic. We also add a risk-neutral PPO baseline in the original Downside-risk H20 and H40 settings. It uses the same actor, reward critic, and PPO settings as DCA, with the risk multiplier fixed at zero. The results are reported in Table C. Risk-neutral PPO violates the risk constraint in both settings. Compared with DCA, it achieves higher mean returns but lower 20th-percentile returns. These results illustrate a trade-off between mean return and lower-tail quantile performance in these tasks.

Table C: Risk-neutral comparison. Values are mean ± standard deviation over ten seeds. Feasible seeds count runs whose final test violation rate is at most 20%. Risk-neutral PPO is newly trained; the other methods use the original experimental results.

SettingMethodMean return20th-percentile returnViolation rate (%)Feasible seeds
H20DCA27.56 ± 0.5316.68 ± 0.8818.19 ± 2.228/10
H20TQPO26.26 ± 0.3318.61 ± 0.9818.29 ± 0.6210/10
H20QCPO26.18 ± 0.2211.66 ± 0.6128.28 ± 1.180/10
H20Risk-neutral PPO30.06 ± 0.398.76 ± 1.1539.20 ± 0.430/10
H40DCA51.19 ± 0.7332.62 ± 3.1818.60 ± 1.1310/10
H40TQPO52.95 ± 0.6927.19 ± 1.5420.72 ± 0.631/10
H40QCPO52.45 ± 0.7525.78 ± 1.4322.21 ± 1.130/10
H40Risk-neutral PPO53.93 ± 0.9920.51 ± 1.3526.21 ± 0.370/10

W3: Additional tasks.

Thank you for the suggestion. We have added two synthetic environments, Gaussian and Orders, in the comparison with the scalar risk critic. Please see our response to Reviewer DeDA, W2-1 for details. We also report results for SimpleButton at the 10% violation limit and for Dynamic at the 20% limit. Please see our responses to Reviewer DeDA, W1-1 and W1-2, respectively. The corresponding results appear in Appendices E.2.2 and E.2.3.

W4: Quantile count and reliability of small risk differences.

In H20, varying the quantile count NN at M=4M=4 or the number of auxiliary actions MM at N=64N=64 during training yields similar mean returns and violation rates (Table D-1).

Table D-1: H20 DCA policy performance across NN and MM. Each final policy is evaluated on 100,000 episodes. Mean ± standard deviation over ten seeds; feasibility requires test violation rate ≤20%. Fixed learning rates: 0.001 for the actor, 0.002 for both critics.

SettingNNMMMean return20th-percentile returnViolation rate (%)Feasible seeds
Default64427.57 ± 0.5116.67 ± 0.8218.17 ± 2.209/10
Vary NN32427.61 ± 0.3716.76 ± 0.7518.51 ± 1.7710/10
128427.49 ± 0.4516.94 ± 0.8018.08 ± 2.0410/10
256427.50 ± 0.4716.84 ± 0.8018.34 ± 2.0510/10
Vary MM64227.58 ± 0.4916.72 ± 0.9018.07 ± 2.378/10
64827.57 ± 0.5016.74 ± 0.8718.18 ± 2.218/10
641627.57 ± 0.5516.65 ± 0.8718.36 ± 2.169/10
646427.58 ± 0.5216.64 ± 0.8718.28 ± 2.167/10
6412827.55 ± 0.5316.71 ± 0.8718.16 ± 2.269/10

We then fix M=4M=4 and vary NN over 32, 64 (baseline), 128, and 256. To prevent the choice of critic from changing the policy and the data it collects, we train all four on the same trajectories in each of ten seeds. Only the baseline critic supplies the risk signal for the actor update. We then vary MM for each frozen critic on the same evaluation transitions. We compute the reference violation probabilities by backward numerical integration under each frozen policy. Table D-2 reports mean absolute errors (MAEs) of state-action and state violation probabilities; their difference determines the actor’s risk signal.

Table D-2: H20 violation probability and risk-credit errors across NN and MM. MAEs are in percentage points, averaged over ten seeds.

QuantityN=32N=32N=64N=64N=128N=128N=256N=256
State-action violation probability2.321.892.212.50
State violation probability (integrated)2.311.842.202.48
Up,tU_{p,t}, M=1M=10.930.730.690.70
Up,tU_{p,t}, M=4M=40.830.620.600.62
Up,tU_{p,t}, M=16M=160.800.600.580.60
Up,tU_{p,t}, M=64M=640.790.590.580.60
Up,tU_{p,t}, integrated0.790.580.580.59

The risk-credit signal depends on NN and MM:

  • Fixing MM, the MAE of Up,tU_{p,t} at iteration 400 decreases as NN goes from 32 to 64 to 128, then rises slightly at 256. This holds for every tested MM. At M=4M=4, the errors are 0.83, 0.62, 0.60, and 0.62 percentage points, respectively.
  • Fixing NN, increasing MM reduces the MAE of Up,tU_{p,t} for every critic during training (supplementary figure). At N=64N=64 and iteration 400, the MAE falls from 0.73 percentage points at M=1M=1 to 0.62 at M=4M=4 and 0.59 at M=64M=64. For all four NN, the MAE at M=64M=64 is within 0.0021 percentage points of the MAE obtained by integrating the same critic over actions. Here, 64 sampled actions per state suffice to estimate the critic's violation probability averaged over actions under the policy.

Table D-3: H20 risk-credit sign accuracy, zero predictions, and MAE by reference magnitude ∣Up,t∣\lvert U_{p,t}\rvert. Groups use the reference change.

∣Up,t∣\lvert U_{p,t}\rvert (%)NNMMCorrect signReversed signZeroMAE (%)
(0.001,0.1)(0.001,0.1)6443.69%1.93%94.38%0.12
(0.001,0.1)(0.001,0.1)12844.36%1.26%94.38%0.11
(0.001,0.1)(0.001,0.1)256415.96%2.01%82.03%0.12
(0.001,0.1)(0.001,0.1)2563244.33%1.64%54.03%0.10
(0.001,0.1)(0.001,0.1)2566454.36%1.41%44.23%0.10
[0.1,0.5)[0.1,0.5)64431.56%2.54%65.90%0.31
[0.1,0.5)[0.1,0.5)128475.10%1.40%23.50%0.29
[0.1,0.5)[0.1,0.5)256496.44%1.24%2.32%0.29
[0.1,0.5)[0.1,0.5)2563298.97%0.92%0.11%0.27
[0.1,0.5)[0.1,0.5)2566499.02%0.94%0.04%0.27
[0.5,1)[0.5,1)64498.27%1.03%0.70%0.40
[1,100)[1,100)64498.90%1.10%0.00%1.05

In Table D-3, the default setting correctly identifies the risk-change direction in 98.27% of estimates for ∣Up,t∣\lvert U_{p,t}\rvert in [0.5%,1%) and 98.90% for ∣Up,t∣\lvert U_{p,t}\rvert in [1%,100%). For [0.1%,0.5%), the correct-sign rate drops to 31.56%, with 65.90% of estimates reporting zero change. Increasing NN to 256 at fixed M=4M=4 raises the correct-sign rate to 96.44%, and increasing MM to 64 raises it further to 99.02%. For (0.001%,0.1%), the same increases in NN and MM raise the correct-sign rate from 3.69% to 54.36%.

W5: Minor issue

Thank you for pointing this out. TQPO had the highest mean return in Table 2 of the original submission. The revised SimpleButton comparison uses matched undiscounted targets across methods. Table 12 in Appendix E.2.2 now bolds the highest mean return at each violation limit.

Q1: Comparison with Chow et al. (2018)

Q1-1: Empirical comparisons.

As suggested, we add a comparison with the chance-constrained actor–critic method in Section 5.2 of Chow et al. (2018). We adapt it to our continuous-action H20 task. We use discounted negative rewards as costs, preserving the same violation event. The state includes the remaining budget. The critic estimates the value including the terminal violation penalty. Following Chow et al. (2018), we use a linear critic, diminishing step sizes, and projected multiplier updates. The actor and critic update after each episode using per-step TD errors.

We compare the methods at α = 20%, 10%, 5%. The return threshold is 16 and the discount factor is 0.99. Both methods use ten seeds and the same training interaction budget. Results are summarized in Table E. We will provide implementation details in the revised manuscript. Training curves are available in the anonymous supplementary figures.

Table E: Comparison with Chow-AC in H20. Values are mean ± standard deviation across ten seeds. QαQ_{\alpha} is the return quantile at the corresponding risk limit. Feasible seeds count final policies whose empirical violation rate does not exceed that limit.

Risk limitMethodMean returnQαQ_{\alpha}Violation (%)Feasible seeds
20%DCA27.57 ± 0.5116.67 ± 0.8218.17 ± 2.209/10
20%Chow-AC26.37 ± 0.9816.43 ± 1.3718.91 ± 3.067/10
10%DCA25.09 ± 1.6114.34 ± 2.9210.54 ± 1.014/10
10%Chow-AC24.19 ± 1.2314.04 ± 1.5211.96 ± 1.601/10
5%DCA23.39 ± 1.860.83 ± 12.098.89 ± 1.220/10
5%Chow-AC22.80 ± 1.122.70 ± 6.809.87 ± 1.250/10

The results show that: (1) At the 20% limit, DCA achieves a higher mean return with similar violation rates and more feasible seeds. (2) At the 10% limit, DCA achieves a higher mean return and a lower violation rate, although both methods exceed the limit on average. (3) At the 5% limit, DCA achieves a higher mean return and a lower violation rate than Chow-AC, although neither method satisfies the constraint in any seed.

Q1-2: Theoretical comparisons.

Please refer to our response to W1 for the methodological comparison. In addition, we clarify two theoretical differences below:

  • The analysis of Chow-AC assumes a uniformly bounded time horizon to reach a target state (Assumption 2 in Chow et al., 2018). We study infinite-horizon discounted returns without this termination requirement. Our Assumption 1 controls the probability mass near the return threshold. This allows us to control the contribution of late decisions. We also bound the error from truncating the risk-gradient sum (Proposition 4).
  • Our Theorem 1 separates trajectory-sampling error, critic-estimation error, and action-sampling error. Corollary 1 gives the corresponding result for transition-based risk feedback. Proposition 3 links the distributional Bellman residual to conditional-CDF error and risk-gradient error. These results explain how distributional approximation and finite sampling affect the accuracy of risk feedback.

We will clarify these theoretical distinctions in the revised manuscript.

Reviewer_DeDAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies expected-return maximization subject to a lower-quantile constraint, reformulated under regularity assumptions as a bound on the probability that total return falls below a prescribed threshold. Distributional Credit Assignment (DCA) augments the state with a remaining return requirement determined by accumulated rewards. A quantile-based critic learns the conditional future-return distribution through distributional Bellman updates; evaluating its CDF at the remaining requirement yields an estimate of trajectory-level violation probability. The change in policy-averaged violation probability across an observed transition provides a per-step risk advantage, with the observed violation indicator used at termination.

The practical algorithm combines reward and risk advantages through separately clipped PPO objectives and adjusts the Lagrange multiplier using a PI controller driven by empirical violation rates. The theoretical results cover risk-gradient identities, estimation-error bounds, and conditional convergence of idealized three-timescale critic–actor–multiplier updates, with a KKT conclusion when critic error vanishes. Experiments compare DCA with QCPO and TQPO on a synthetic downside-risk task at two horizons and on a SimpleButton extension with a separate cumulative-cost quantile constraint.

Critical strengths
  • The method connects accumulated rewards, distributional return estimation, and per-step risk feedback in a clear and well-motivated way. Numerical reward observations can inform risk estimation beyond terminal violation labels.
  • The analysis explicitly accounts for critic error and includes the transition-based estimator. Under the stated assumptions, the multiplier convergence argument and the resulting stationarity bound appear internally consistent; I did not identify a specific mathematical error in these derivations.
Critical weaknesses
  1. Limited evidence for generality across environments and risk levels. The evaluation covers one synthetic environment at two horizons and SimpleButton. QCPO [1] evaluates SimpleButton, Dynamic, and Gremlin, while TQPO [2] uses the corresponding Simple, Dynamic, and Gremlin tasks. The latter environments introduce random goal generation and moving obstacles, testing whether constraint control survives more complex dynamics. DCA includes only the simplest member of this family. Risk coverage is also narrower: QCPO evaluates 10% and 20% violation limits, TQPO evaluates 5% and 10%, and DCA evaluates only 20%. Because estimating conditional tail probabilities is central to the method, the stricter limits would be particularly informative about its empirical reliability.

    The reported results also reflect different return–risk trade-offs. At horizon 20, DCA improves mean return over TQPO by approximately 4.98%, but satisfies the empirical constraint in 8/10 runs versus 10/10 for TQPO. At horizon 40, DCA provides stronger constraint control while obtaining lower mean return. In SimpleButton, it improves both mean return and average violation rate over QCPO; TQPO obtains higher return but exceeds the violation limit on average. These are useful results, but they establish advantages in particular configurations. Return–risk curves across constraint levels, or comparisons of well-tuned feasible policies, would make the relative gains easier to assess.

  2. The comparisons do not establish which component provides the additional benefit over prior work. QCPO [1] already uses distributional RL to estimate cumulative-cost quantiles and tail probabilities, while TQPO [2] uses sampled quantile-gradient estimates and explicitly addresses multiplier overshoot through tilted updates. Chow et al. [3] develop augmented-state chance-constrained actor–critic methods with terminal violation penalties. Against this background, the important empirical question is how much DCA gains from learning the future-return distribution and evaluating risk at the remaining requirement, beyond the benefits of state augmentation and multiplier control.

    DCA changes the critic, risk-credit estimator, and multiplier controller together. Its PI controller differs from the baseline-specific multiplier rules, and optimization settings also differ. A scalar violation-probability critic using the same augmented state, actor, PPO procedure, and multiplier controller would directly test whether learning the full return distribution improves risk feedback. Comparing direct and transition-based risk advantages, and varying the controller while holding the risk estimator fixed, would help separate the contributions.

    Baseline implementation choices also limit the interpretation. The synthetic experiments use source-derived core reimplementations in a shared PPO scaffold rather than the original complete training pipelines (Appendix D). In SimpleButton, DCA uses undiscounted training targets, whereas TQPO retains discounted reward and cost targets with gamma = 0.99 despite evaluation under an undiscounted constraint (Appendix E.2). This mismatch could affect the observed feasibility gap. A matched-target comparison and a clear account of the tuning budget are needed before attributing baseline constraint failures to differences between the underlying methods.

  3. The convergence theorem has a narrower scope than the practical algorithm. Theorem 2 assumes unique, uniformly globally asymptotically stable critic and actor equilibria, a nonincreasing equilibrium violation probability as the multiplier grows, and an appropriate feasible equilibrium within the multiplier range (Assumption 3). These assumptions support the proof, but their validity for the proposed parameterization is not established. The proof also analyzes projected stochastic approximation with diminishing, separated step sizes, whereas the experiments use PPO, Adam, and a PI multiplier update with an additional controller state. No argument establishes that this implementation satisfies the theorem's conditions. This is a limitation in applicability and justification of assumptions, rather than an identified proof error. The exact-critic KKT conclusion also does not establish global optimality for the parameterized problem.

References

  1. Jung et al. Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probability NeurIPS, 2022.
  2. Li, Ruan, and Geng. Tilted Quantile Gradient Updates for Quantile-Constrained Reinforcement AAAI, 2025.
  3. Chow et al. Risk-Constrained Reinforcement Learning with Percentile Risk Criteria JMLR, 2018.
Questions for authors
  1. Can the authors provide a controlled comparison against a scalar violation-probability critic and separate the effect of the PI controller from that of distributional risk estimation?
  2. How sensitive are the conclusions to matching discounted versus undiscounted training targets in SimpleButton? Is there evidence that the advantages persist at a stricter violation limit or in one of the harder predecessor environments?
  3. Which conditions of Theorem 2 can be verified for the implemented algorithm? Can the authors clearly delimit the idealized scheme covered by the theorem, and provide a concrete setting satisfying Assumption 3 or evidence concerning multiplier saturation and equilibrium risk response?
6 comments

Author Response to Reviewer DeDA

We thank reviewer DeDA for the valuable and constructive comments.

W1: Broader evaluation across environments and violation limits

W1-1: Matched-targets and a stricter violation limit.

We add a matched-target comparison in SimpleButton. DCA, QCPO, and TQPO all use undiscounted reward and cost targets (γ=1\gamma=1) and a batch size of 25. We evaluate the constraint Pr⁡(C>15)≤ϵ\Pr(C>15)\leq\epsilon, where CC is the total undiscounted episode cost, at ε = 20% and 10%. Results are summarized in Table A-1. Training curves for both limits are available in the anonymous supplementary figures.

Table A-1: SimpleButton with matched undiscounted targets and batch size 25. Values are means ± sample standard deviations across 10 training seeds, with 2,048 evaluation episodes per seed. Q1−ϵ(C)Q_{1-\epsilon}(C) is the empirical cost quantile.

LimitMethodMean returnCost quantile Q1−ϵ(C)Q_{1-\epsilon}(C)Violation (%)Feasible seeds
20%DCA16.86 ± 2.778.90 ± 3.1113.75 ± 4.189/10
20%QCPO16.65 ± 2.9911.50 ± 13.2819.64 ± 15.579/10
20%TQPO10.21 ± 3.9420.56 ± 12.6926.81 ± 14.024/10
10%DCA17.01 ± 3.9213.13 ± 7.239.29 ± 3.506/10
10%QCPO13.84 ± 3.0214.46 ± 7.8210.29 ± 4.156/10
10%TQPO10.56 ± 5.7216.63 ± 9.4210.90 ± 5.224/10

At the 20% limit, DCA achieves a return comparable to QCPO with a lower average violation rate, and improves both metrics over TQPO. At 10%, DCA has the highest mean return and lowest average violation rate; DCA and QCPO have the same number of feasible seeds. We will include these results in the revised manuscript.

W1-2: Harder environment.

On Dynamic, we evaluated DCA, QCPO, and TQPO under Pr⁡(C>15)≤0.20\Pr(C>15)\leq 0.20, with CC the undiscounted episode cost. Each run used 10M on-policy steps, 25 episodes per batch, and 1,000 steps per episode. We evaluated each final policy on 2,048 held-out episodes. For each seed, we computed mean undiscounted return, the empirical 80th percentile of cost (Q^0.80C\widehat Q^C_{0.80}), and the percentage of episodes with C>15C>15. As in GRPO, we sample multiple continuations from a common starting point. On Dynamic, we draw 16 additional simulator continuations from each queried state under the current policy and use the fraction whose future cost exceeds the remaining budget to estimate the conditional violation probability. For each nonterminal transition, DCA uses the change in this probability from the current state to the next as the risk advantage in its PPO update.Tables A-2 and A-3 report means ± sample standard deviations across three seeds. Feasibility requires an unrounded held-out violation rate of at most 20%.

Table A-2: Final-policy evaluation on Dynamic after 10M on-policy steps. Feasible counts the seeds meeting the 20% violation limit.

MethodMean returnCost Q^0.80C\widehat Q^C_{0.80}Violation (%)Feasible
DCA (MC K=16K=16)17.77 ± 0.5917.00 ± 3.4621.47 ± 3.031/3
QCPO15.58 ± 0.7514.33 ± 5.5120.07 ± 3.172/3
TQPO (default)17.17 ± 0.5538.00 ± 5.0041.67 ± 4.710/3

Table A-2 shows that DCA achieved the highest mean return (17.77). Its violation rate was 21.47%, compared with 20.07% for QCPO and 41.67% for default TQPO (δ=0.10\delta=0.10). QCPO met the limit in two of three seeds, DCA in one, and default TQPO in none. The training curves for DCA, QCPO, and default TQPO show how these differences developed during training. At 2.5M steps, default TQPO already achieved a trailing mean return of 17.82, but 57.33% of its recent training episodes exceeded the cost threshold.

We varied δ\delta with other TQPO settings and all three seeds fixed. Table A-3 reports the final results, and the TQPO training curves show the training trajectories for all five values of δ. Increasing δ\delta from 0.10 to 0.50 reduces final-policy violation in all three paired seeds. Mean violation fell from 41.67% to 14.93%, while mean return fell from 17.17 to 11.65. The effect was not monotonic across the five settings. At δ=0.25\delta=0.25, the three violation rates were 7.32%, 20.65%, and 38.04%.

Table A-3: TQPO tilt-smoothing sweep on Dynamic. The evaluation protocol is the same as in Table A-2.

δ\deltaMean returnCost Q^0.80C\widehat Q^C_{0.80}Violation (%)Feasible
0.0514.18 ± 2.0540.00 ± 2.6537.34 ± 5.870/3
0.10 (default)17.17 ± 0.5538.00 ± 5.0041.67 ± 4.710/3
0.17512.94 ± 3.0813.67 ± 12.6619.71 ± 6.671/3
0.2511.42 ± 4.3017.00 ± 17.0022.01 ± 15.401/3
0.5011.65 ± 3.716.00 ± 8.7214.93 ± 5.942/3

W1-3: Comparisons across violation limits and return–risk trade-off.

We extend the comparison of DCA, QCPO, and TQPO to violation limits of 20%, 10%, and 5% in the original Downside-risk H20 setting. We report the original 400-round training budget and continue all runs to 800 rounds. Each configuration uses the same ten seeds. Hyperparameters remain unchanged during the extension. Table A-4 summarizes the results.

Table A-4: Downside-risk H20 at three violation limits and two training budgets. Returns and violation rates are mean ± standard deviation over ten paired seeds. Each policy is evaluated on 100,000 test episodes. Feasible counts seeds with an empirical violation rate at most the indicated limit.

LimitMethodReturn (400 rounds)Violation % (400)Feasible (400)Return (800 rounds)Violation % (800)Feasible (800)
20%DCA27.52 ± 0.5418.15 ± 2.2210/1028.15 ± 0.3818.48 ± 1.229/10
20%QCPO26.18 ± 0.2228.28 ± 1.180/1026.14 ± 0.3826.91 ± 1.690/10
20%TQPO26.26 ± 0.3318.29 ± 0.6210/1027.69 ± 0.2721.50 ± 0.800/10
10%DCA25.11 ± 1.7110.27 ± 0.955/1024.96 ± 2.229.32 ± 1.258/10
10%QCPO23.53 ± 0.1421.38 ± 0.500/1023.48 ± 0.2421.67 ± 0.810/10
10%TQPO23.54 ± 0.2014.23 ± 0.270/1025.57 ± 0.2617.21 ± 0.620/10
5%DCA23.38 ± 1.848.82 ± 1.200/1022.64 ± 2.357.42 ± 1.810/10
5%QCPO22.75 ± 0.1720.10 ± 0.390/1022.94 ± 0.1821.14 ± 0.560/10
5%TQPO19.73 ± 0.2315.34 ± 0.670/1022.11 ± 0.4812.60 ± 0.330/10

DCA achieves the lowest average violation rate across all three limits and both training budgets, while maintaining competitive mean returns. However, none of the methods satisfies the 5% constraint. Extending training does not consistently improve constraint satisfaction, highlighting the dependence of the return–risk trade-off on training budget.

QCPO with reference conditional quantiles. In Downside-risk H20, QCPO fails to satisfy the constraint at both the 10% and 5% violation limits. We therefore conduct an ablation to examine whether quantile estimation errors contribute to these violations. We replace the learned conditional quantiles with numerical reference quantiles computed from the known environment model, denoted as QCPO + ref. The Weibull model continues to be fitted to the network's learned quantiles. All other training settings are matched.

Table A-5 summarizes the performance of QCPO with learned versus reference conditional quantiles at the 10% and 5% violation limits in Downside-risk H20. Reference quantiles substantially reduce violation rates at both limits, at the cost of lower mean returns. At the 10% limit, 2/5 seeds are feasible with reference quantiles versus 0/5 with QCPO. At the 5% limit, neither variant achieves feasibility in any seed. These results suggest that quantile estimation errors may partly explain QCPO's difficulty in controlling violations in this environment.

Table A-5: QCPO ablation in Downside-risk H20. Both variants are trained for 400 rounds with 256 trajectories per round.

LimitMethodMean returnQαQ_\alphaViolation (%)Feasible seeds
10%QCPO23.75 ± 0.236.75 ± 0.2721.10 ± 0.480/5
10%QCPO + ref.19.02 ± 0.0615.94 ± 0.1710.23 ± 0.702/5
5%QCPO21.96 ± 0.501.82 ± 1.3419.38 ± 0.250/5
5%QCPO + ref.18.80 ± 0.0515.26 ± 0.207.38 ± 0.610/5

W2: Add comparisons.

W2-1: Add comparison with a scalar violation-probability critic.

As suggested, we add comparisons with a scalar risk critic that directly estimates the conditional violation probability. Our transition-based risk signal Up,tU_{p,t} only depends on these conditional probabilities. Distributional RL provides one way to estimate them. Alternatively, a scalar neural network can directly approximate the same probability:

pπ(s,b,a)=Pr⁡(Y=1∣St=s,Bt=b,At=a)=Fπ(b∣s,b,a),p^\pi(s,b,a)=\Pr(Y=1\mid S_t=s,B_t=b,A_t=a)=F^\pi(b\mid s,b,a),

where YY is the terminal violation indicator. We train a sigmoid-output network pϕp_\phi with binary cross-entropy on one-step probability Bellman targets. At termination, yt=Yy_t=Y. Otherwise, we sample the next action and compute the target as follows:

$ y_t = p_{\bar{\phi}}(S_{t+1}, B_{t+1}, A_{t+1}^{\prime}), \qquad A_{t+1}^{\prime} \sim \pi(\cdot \mid S_{t+1}, B_{t+1}). $

Here ϕˉ\bar\phi denotes the target-network parameters. The scalar critic uses probability Bellman updates to propagate terminal violation information to earlier states and actions. Intermediate rewards enter these updates through the remaining-requirement recursion Bt+1=(Bt−Rt)/γB_{t+1}=(B_t-R_t)/\gamma. In contrast, DCA additionally incorporates numerical rewards into distributional Bellman targets to learn the conditional future-return distribution, and evaluates its CDF at the remaining requirement to obtain violation probabilities. Both estimators supply these probabilities to the same transition-based Up,tU_{p,t} construction in Eq. (11).

We compare three 20-step environments: Downside-risk, the paper’s original continuous-action task; Gaussian, a binary-action task with Gaussian reward noise and action-dependent means and variances; and Orders, a synthetic order-acceptance task with random prices and a common uncertain cost settled at termination. Environment definitions, initial policies, and hyperparameters are in the revised manuscript. Risk limits are 20% for Downside-risk and 10% for Gaussian and Orders. Within each environment, we change only the risk estimator and keep other settings fixed. We conduct two evaluations over five seeds:

  • Freeze the initial policy and train both critics on identical batches of 512 trajectories per round. Evaluate mean absolute error (MAE) of U^p,t\widehat U_{p,t} against a model-based reference Up,tU_{p,t} on shared held-out transitions.
  • Compare DCA, Scalar and Oracle, which uses true conditional violation probabilities and has no risk-critic learning error. Oracle’s PI controller uses the same empirical violation rate as the other methods.

Table B-1: MAE of the estimated Up,tU_{p,t} for the fixed policy. Values are mean ± standard deviation over five paired seeds. MAEs and their standard deviations are multiplied by 100. The critic networks have approximately the same number of parameters.

EnvironmentDCA: UU MAEScalar: UU MAERelative reduction
Downside-risk0.488±0.0470.569±0.06714.3%
Gaussian0.290±0.0170.440±0.06134.1%
Orders0.209±0.0360.594±0.01264.9%

DCA achieves lower final MAE for the risk signal in all three fixed-policy tests (Table B-1). This shows an empirical benefit of distributional estimation under matched data and update budgets.

Table B-2: Controlled PPO comparison. First feasible round is the median across five seeds; it does not imply feasibility at all subsequent checkpoints. Final results are mean ± standard deviation. Violation rates and their standard deviations are expressed in percent. Feasible seeds counts runs satisfying the constraint at the final checkpoint.

EnvironmentMethodFirst feasible roundFinal returnFinal violation rate (%)Feasible seeds
Downside-risk (α = 20%)DCA16025.83±0.1317.39±0.335/5
Scalar16025.80±0.1817.22±0.365/5
Oracle16025.84±0.1417.54±0.375/5
Gaussian (α = 10%)DCA30020.64±0.107.39±0.815/5
Scalar38020.64±0.147.44±1.225/5
Oracle30020.65±0.097.50±0.735/5
Orders (α = 10%)DCA4034.98±0.1510.34±0.070/5
Scalar5034.07±0.199.59±0.414/5
Oracle4035.40±0.2110.10±0.161/5

DCA first reaches feasibility earlier in Gaussian and Orders (Table B-2). In Gaussian, the median first-feasible round is 300 for DCA and 380 for Scalar. In Orders, it is 40 for DCA and 50 for Scalar.

DCA and Scalar have similar final returns and violation rates in Downside-risk and Gaussian. In Orders, DCA has a higher return but exceeds the 10% limit on average; Scalar has a lower return but satisfies it on average. DCA and Oracle have similar final performance in all environments (Table B-2).

Training curves for all three environments are available in the anonymous supplementary figures.

In summary, DCA learns risk signals faster and more accurately. This can help PPO reach feasibility earlier. The resulting policy gains depend on the environment and other training components. We will add these ablation studies in the revised manuscript.

Reviewer_cXxGOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the problem of maximizing expected discounted reward, subject to a quantile constraint on the cumulative reward random variable. The paper introduces the random variable X_t=(S_t,B_t), which constructs a markov chain, and can be used to define the optimal policy. In particular, the quantile objective prevents us from using only state to specify the optimal policy. Next, the authors consider a regularized version of the their constrained optimization problem, and they show that policy gradient converges asymptotically to a stationary point of this Lagrangian.

Critical strengths

-The paper is well written. -The paper studies quantile constraint problem which is a difficult objective. In particular, the traditional RL where the state itself creates a Markov chain is not sufficient. To my knowledge, the introduction of the random variable X_t is novel.

Critical weaknesses
  • The paper only establishes asymptotic convergence, and only convergence to a stationary point, rather than finite time convergence to the global optimum.
  • The main critical weakness is that Theorem 2 assumes much of what it claims to establish. Theorem 2's conclusion that p(\theta^*)\leq\alpha follows largely from the assumption. Moreover, global stability and uniqueness of neural-network actor dynamics are not established.
Questions for authors

In your citations, your paper is missing the relevant paper "Optimistic Reinforcement Learning with Quantile Objectives" by Vaezi et al. Can you include it? Can you elaborate what Assumption 1 is saying, and could you motivate it?

3 comments

Author Response to Reviewer cXxG

We sincerely thank Reviewer cXxG for the recognition of our work and for providing constructive comments.

W1: Asymptotic stationarity and global optimality.

We thank the reviewer for this comment. We agree that our analysis does not establish finite-time convergence or global optimality guarantees.

Our theoretical focus is the accuracy of distributional risk credit and its effect on first-order stationarity in the nonconvex policy problem. We show that exact conditional risk credit preserves the original violation-probability gradient. Theorem 1 and Corollary 1 provide finite-sample error bounds for the direct and transition-based estimators. Under the stated assumptions, Theorem 2 bounds the limiting norm of the Lagrangian gradient of the idealized three-time-scale updates by λ∗CsHνδ∗\lambda^{\ast} C_{s} H_{\nu} \delta^{\ast}. This makes the effect of critic error explicit: when δ∗=0\delta^{\ast} = 0, the limit satisfies the KKT conditions. These results characterize gradient-estimation accuracy and conditional asymptotic convergence; they do not establish finite-time convergence to a global optimum.

We will clarify this scope in the paper and explicitly acknowledge these limitations.

W2: Stability and feasibility assumptions in Theorem 2.

We thank the reviewer for this comment. Assumption 3 assumes either a strictly feasible actor equilibrium at λ=0\lambda = 0 or an actor equilibrium satisfying p(θ∗(λ))=αp(\theta^{*}(\lambda)) = \alpha for some 0<λ<λmax⁡0 < \lambda < \lambda_{\max}. We agree that the feasibility conclusion relies on this assumption. We also agree that uniqueness and global asymptotic stability have not been established for our neural-network actor.

Given these assumptions, Theorem 2 establishes convergence of the idealized coupled stochastic updates. The analysis accounts for sampling noise and the interaction among the critic, actor, and multiplier. Under the stated conditions, it shows that the critic approaches the equilibrium associated with the current policy. It also shows that the actor approaches the equilibrium associated with the current multiplier. The proof then establishes convergence of the multiplier and shows that all three parameter sequences converge almost surely. It also gives an explicit bound on ∥∇θL(θ∗,λ∗)∥2\lVert \nabla_{\theta} L(\theta^{\ast}, \lambda^{\ast}) \rVert_2 in terms of critic approximation error.

We will state Assumption 3 in the revised paper and explain their role in the feasibility conclusion. We will also revise the conclusion to acknowledge that our convergence guarantee relies on these assumptions.

Q1: Add reference.

We thank the reviewer for bringing this interesting paper to our attention. Alipour-Vaezi et al. (2026) introduce UCB-QRL, which maximizes a fixed return quantile through optimistic model-based planning in finite-horizon tabular MDPs. They establish high-probability regret bounds under unknown transition dynamics. Our work maximizes expected return under a prescribed quantile constraint and learns conditional future-return distributions to estimate risk gradients. We discuss this work in Section 2, in the paragraph on “Quantile-based decision making.”

Q2: Meaning and motivation of Assumption 1.

We thank the reviewer for this question. Let Pt=∑j<tγjRjP_t = \sum_{j<t} \gamma^j R_j be the discounted return accumulated before time tt, and let qq be the required return threshold. Assumption 1 requires Pr⁡(∣Pt−q∣≤u)≤Cquν\Pr(\lvert P_t - q \rvert \le u) \le C_q u^{\nu} for all t≥t0t \ge t_0 and u>0u > 0. It limits how much probability mass can lie close to qq. As the interval shrinks, this upper bound approaches zero. The same constants must apply across all t≥t0t \ge t_0.

The assumption controls whether late actions can still affect the violation outcome. Bounded rewards imply that the remaining contribution to the total discounted return has magnitude at most γtR⋆\gamma^t R_{\star}. If PtP_t is farther than this distance from qq, the violation outcome is already determined, regardless of future actions. The exact risk advantage is therefore zero outside the event ∣Pt−q∣≤γtR⋆\lvert P_t - q \rvert \le \gamma^t R_{\star}. Assumption 1 bounds the probability of this event by CqR⋆νγνtC_q R_{\star}^{\nu} \gamma^{\nu t}, which decreases geometrically. Together with the stated moment bound on policy scores, this controls the infinite sum in the risk gradient.

For example, suppose the densities of PtP_t are bounded by the same constant DD for all t≥t0t \ge t_0. Then Pr⁡(∣Pt−q∣≤u)≤2Du\Pr(\lvert P_t - q \rvert \le u) \le 2Du, so the assumption holds with ν=1\nu = 1 and Cq=2DC_q = 2D. This is a sufficient condition for the analysis. We will add this explanation and example immediately after Assumption 1 in Section 4.1.

Thanks authors for their detailed response. I maintain my score.

Dear Reviewer cXxG,

Thank you for your positive assessment and support. We sincerely appreciate your time and effort in reviewing our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of responses

Dear Area Chair,

We sincerely thank reviewers and the AC for their time, effort, and constructive comments. We provide a summary of our responses for your convenience when making your decision. All three reviewers noted that our ablations substantially addressed their concerns.

1. What does distributional credit assignment (DCA) add?

  • DeDA (W2, Q1) and WPpD (W2) asked whether a scalar critic would suffice. We added two comparisons in the original Downside-risk setting and two additional settings, Gaussian and Orders. (1) We trained the distributional and scalar critics on the same fixed-policy data to compare their stepwise risk estimates (Table B-1). (2) We compared policies trained with each critic while holding the other components fixed (Table B-2). DCA estimated stepwise risk more accurately than the scalar critic on Downside-risk, Gaussian, and Orders.

  • DeDA (W2, Q1) asked what the risk signal and multiplier updates each contribute. We varied them separately on H20 (Table B-3). Transition-based feedback gave higher mean return. Direct feedback gave higher 20th-percentile return. With transition-based feedback fixed, both PI and projected dual updates gave similar mean returns.

  • WPpD (W1, Q1) asked how DCA differs from Chow’s method. We compared the two methods in terms of their risk modeling and feedback mechanisms (see our rebuttal to WPpD W1). We further evaluated both methods on the 20-step Downside-risk task (H20) under equal budgets (see our rebuttal to WPpD Q1, Table E).

  • WPpD (follow-up) asked us to isolate the effect of stepwise feedback. We replaced only the risk signal with the terminal violation indicator minus the current-state baseline. For each frozen DCA policy and critic, we compared both risk gradients on the same 1,024 trajectories, summing over steps (see our follow-up reply to WPpD).

2. What do the additional experiments show?

  • DeDA (W1, W2, Q2, follow-up) and WPpD (W3) asked for comparable targets and a harder task. We matched undiscounted reward and cost targets and batch sizes on SimpleButton (Table A-1). We also evaluated DCA on two additional tasks in the Safety-Gymnasium. We report the Dynamic results in Table A-2 and the Hopper results in our follow-up response to DeDA.

  • DeDA (W1–W2) and WPpD (W2) asked about tighter limits and additional baselines. We compared DCA, QCPO, and TQPO on H20 at 20%, 10%, and 5% (Table A-4). DCA had the lowest mean violation rate across all limits and budgets. We also added ablations for QCPO to compare with reference quantiles (Table A-5) and risk-neutral PPO (Table C).

  • WPpD (W4) asked whether the critic can resolve small risk changes. We varied the numbers of quantiles N and sampled actions M (Tables D-1 to D-3). Increasing numbers of N and M both raised step-wise risk sign accuracy.

3. What do the theoretical guarantees cover?

  • cXxG (W1–W2, Q2) and DeDA (W3) asked about the assumptions and scope of our guarantees. We motivate Assumption 1 in our response to cXxG (Q2), illustrate Assumption 3 in our response to DeDA (Q3), and clarify the scope of Theorem 2 in our response to DeDA (W3) and cXxG (W1–W2).

We added the quantile-maximization reference requested by cXxG (Q1, Section 2) and corrected the SimpleButton highlighting flagged by WPpD (W5).

Once again, we greatly appreciate the reviewers’ insightful comments and timely engagement during the rebuttal. Thank you for your time and for considering our submission.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.