acceptodds
Under review as a conference paper at ICLR 2027

Deep Reinforcement Learning with Buffered Quantile Objectives

Abstract

Quantile-based reinforcement learning provides an interpretable approach to risk-sensitive decision making by optimizing a prescribed quantile of the cumulative-return distribution. Despite this appeal, learning under a point quantile objective is challenging: quantiles can change abruptly under small perturbations of the return distribution, and exact quantile-sensitive planning requires computationally demanding distributional optimization. Lower-buffered quantiles alleviate the former difficulty by averaging neighboring quantiles immediately below the target level, providing a smoother surrogate while preserving the underlying point-quantile objective. Existing methods based on this principle, however, remain model-based and rely on explicit return-law planning, limiting their applicability beyond small tabular problems. We develop Deep-BQRL, a model-free distributional reinforcement-learning framework that extends buffered-quantile learning to neural function approximation. The method learns conditional return quantiles directly from sampled transitions, constructs buffered action scores from the relevant region of the learned quantile function, and uses ensemble disagreement to guide exploration. An augmented input representation allows the learned policy to respond to trajectory information without explicitly reproducing the quantile-state recursion required by exact planning. Experiments on an asset-selling optimal-stopping problem and slippery FrozenLake compare Deep-BQRL with model-based UCB-BQRL and tabular PPO and TRPO implementations. In asset selling, Deep-BQRL attains smaller mean cumulative point-quantile policy gaps than PPO and TRPO at the reported target levels, while UCB-BQRL retains the smallest gaps. The learned stopping decisions also vary with the target quantile, providing an interpretable illustration of the method's risk-sensitive behavior.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Quantile, Risk Sensitive

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 5 reviews, above 0% of the others.
Reviewer_fLFzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends lower-buffered quantile objectives (Q^beta_tau, the average of return quantiles over [tau-beta, tau], introduced in the authors' model-based UCB-BQRL) to model-free deep RL. DEEP-BQRL trains an ensemble of QR-DQN-style quantile critics on an augmented input (h, s, c) with c the cumulative reward, selects actions greedily by a "buffered score" — the interval-overlap-weighted average of predicted quantiles in [tau-beta, tau] — and adds ensemble disagreement on that score as an exploration bonus. Experiments on a 25-state asset-selling problem and 4x4 slippery FrozenLake show smaller point-quantile policy gaps than PPO/TRPO but gaps roughly 20-100x larger than the model-based UCB-BQRL reference.

Critical strengths

The paper is unusually honest: the criterion mismatch (training on Q^beta_tau, evaluating on point Q_tau), the non-isolation of components, the degeneracy of FrozenLake for quantile objectives, and the model-based reference's dominance are all stated explicitly. The component formulas are correct (the overlap weights in Eq. (1), the quantile Huber loss, terminal handling). The evaluation protocol has good hygiene (disjoint validation/test seeds, true-model quantile reference, exploration-free greedy-policy gap). The direction of the learned risk-sensitive stopping rules in Figure 2 is correct and interpretable.

Critical weaknesses

The policy class has a strict expressivity ceiling, and that the authors omit mentioning. Exact quantile-optimal planning is history-dependent through an evolving conditional quantile level (QMDP; Li et al. 2022): after a lucky start the optimal continuation locks in (low conditional level), after an unlucky one it gambles (high conditional level). Fixed-(tau, beta) buffered-greedy selection on (h, s, c) cannot express this level flip. Concretely, an H=2 counterexample (action A: deterministic 0.7; action B: first reward 0.6 or 0 w.p. 1/2, then a safe 0.5 or risky {0,1} choice) has median-optimal value 1.0 via hedging, while every fixed-(tau, beta) greedy policy scores at most 0.7 — even with perfect critics and infinite data. This is not estimation error; no policy-improvement lemma for buffered-greedy selection can exist, and the induced distributional optimality operator is not a contraction. Relatedly, the next action is selected and evaluated by the same target network (Eq. (3)), giving uncorrected maximization bias on the sorted quantiles.

The mechanism that motivates the architecture is never exercised: c is identically zero at every decision point in both environments (asset-selling rewards only on Stop, which terminates; FrozenLake rewards only at the terminal goal; Figure 2 is explicitly evaluated at C_h = 0), and asset-selling's quantile optimum is Markov in (h, s) anyway. So the claimed handling of history dependence has zero empirical evidence.

The novelty accounting needs major revision. The buffered score is exactly a distortion-risk greedy policy with a uniform window over [tau-beta, tau] — the mechanism IQN introduced in 2018 (Dabney et al., arXiv:1806.06923), which is not cited; lower-buffered quantile is Range VaR, and deep RL with RVaR/spectral objectives exists (Coache et al. 2023; Moghimi et al., AISTATS 2025). Model-free quantile-objective RL also exists (Hau et al., "Q-learning for Quantile MDPs," AISTATS 2025), as does (state, cumulative-reward) augmentation for percentile criteria (Chow et al., JMLR 2018) and ensemble-UCB exploration on quantile critics (Chen et al. 2017; SUNRISE 2021; Mavrin et al. 2019). Also worth noting for the AC: the submission appears to be arXiv:2609.21327, and the main baseline (arXiv:2608.22227) is the same group's unreviewed preprint — the framing "extends buffered-quantile learning to neural function approximation" is an incremental follow-up whose reference method has not been peer-reviewed.

The experiments cannot support the stated motivation. Both environments are tabular-exact, so the scalability claim — the paper's entire premise — is untested; and even there, UCB-BQRL's gaps are 20-100x smaller in every cell. FrozenLake is a zero-information quantile benchmark: with binary returns Q_tau = 1{p > 1-tau}, the tau=0.1 gap is identically zero for all methods (the reference quantile itself is 0), and the optimal policy is tau-invariant (the paper admits both). The baselines are not credible: TRPO scores ~0 on a 16-state problem where the optimal success rate is ~0.74, and each method was tuned on a single "dominant" hyperparameter ranked by the quantile-gap criterion that PPO/TRPO do not even optimize — which can actively misselect their configs. The tau=0.9 regret "win" is an artifact: a tau=0.9-optimal policy can have very low expected return, so low regret there indicates weak tau-differentiation, not virtue; and at tau=0.1 the regret of 1025 is genuine underlearning (the quantile-optimal policy there is nearly expectation-optimal). The learned policies also persist in strictly dominated actions (rejecting positive offers at the final period), which damages exactly the lower-tail paths a tau=0.1 agent exists to protect. Finally, reproducibility: the buffer width beta — the titular hyperparameter — is never given a value anywhere; K, M, architectures, evaluation period, and code are all unreported.

Minor: sorting the target quantiles has provably zero effect on the loss in Eq. (4) (the (i,j) double sum depends only on the target multiset), so the claim in Sec. 3 that sorted values are used for "learning targets" is misleading; the regret is defined with realized returns rather than pseudo-regret; and it is unclear whether the reference Q*_tau is the exact optimum over history-dependent policies (the redundant [·]+ clipping suggests not).

Questions for authors

Do you agree that fixed-(tau, beta) buffered-greedy policies form a strict subset of quantile-optimal policies (e.g., the hedging counterexample above), and can you bound the resulting gap in terms of the problem structure? If not, where does the argument fail? What values of beta, K, M, and the exploration parameters were used, and how does performance vary with beta (including the point-quantile limit)? This ablation is the only way to isolate whether buffering itself helps. Is Q*_tau computed as the exact optimum over history-dependent policies? Please specify. How does DEEP-BQRL compare to an IQN critic with a uniform-window distortion (which implements essentially the same action score) and to Hau et al.'s quantile Q-learning on these benchmarks? Why does TRPO score ~0 on 4x4 FrozenLake, and which hyperparameters were tuned for each baseline?

1 comment

Thank you for the suggestions. Labels follow your comment topics. We present new analysis and submitted evidence below. If the paper is accepted, the analysis, full proofs, and proposed additions can be included in the final manuscript.

W1/Q1: Quantifying local control. We analyze the buffered decision rule separately from the policy representation (h,s,c)(h,s,c). Fix a continuation policy π\pi and let επ(x,a)\varepsilon^\pi(x,a) be the maximum discrepancy between sorted critic outputs and true remaining-return quantiles over critics and assigned levels. For returns in [L,U][L,U], ∣μB(x,a)−Qτβ(Zπ(x,a))∣≤επ(x,a)+(U−L)/(2Kβ)|\mu_B(x,a)-Q_\tau^\beta(Z^\pi(x,a))|\le\varepsilon^\pi(x,a)+(U-L)/(2K\beta). Proof sketch: monotonicity bounds each cell's midpoint-integration error by its quantile increase divided by 2K2K; summing cancels internal endpoints. Normalized overlap weights bound the prediction contribution. Applying the score bound to the true best and estimated best actions gives local score loss at most 2ε+(U−L)/(Kβ)2\varepsilon+(U-L)/(K\beta) when prediction error is at most ε\varepsilon across actions. A unique best action is recovered when its advantage exceeds this bound.

W1/Q1: Conditions for global optimality. Construct πloc\pi^{\mathrm{loc}} backward using exact conditional buffered scores. Let αh\alpha_h be the largest positive distribution-function excess of its selected action over alternatives, across stage-hh inputs and thresholds under the same continuation. For finite state/action spaces, total returns in [L0,U0][L_0,U_0], and any policy π\pi, Qτβ(Rπ)−Qτβ(Rπloc)≤(U0−L0)min⁡{1,∑hαh/β}Q_\tau^\beta(R^\pi)-Q_\tau^\beta(R^{\pi^{\mathrm{loc}}})\le(U_0-L_0)\min\{1,\sum_h\alpha_h/\beta\}. Proof sketch: backward induction propagates one-sided distribution differences; integration converts their positive area into a quantile-area bound. If every αh=0\alpha_h=0, first-order stochastic dominance gives optimality for all point and buffered quantiles. This complements the score bound with a comparison for exact local control.

W1/W10: Target updates and sorting. Sorting determines the ordering used to construct buffered action scores. For a fixed next action, the double-sum regression loss is invariant to permutations of target values. We can clarify these two roles and specify that each target critic selects and evaluates its continuation action, while deployment uses the ensemble-mean score. A separate-selection/evaluation comparison can be added to assess the target-update choice.

W3/Q4: Relationship to prior methods. The submission connects the buffered criterion to Range VaR. Its uniform-window interpretation follows by changing variables in the buffered integral. We can expand the discussion of IQN, spectral-risk methods, quantile-MDP Q-learning, reward augmentation, and ensemble exploration, explaining how Deep--BQRL combines sampled-transition learning with buffered-score action selection. A matched-budget comparison with uniform-window IQN and quantile-MDP Q-learning can be added on these finite benchmarks.

W4/W5/Q4: Policy-level evidence already reported. Submitted Figure 1 evaluates learned policies under the true model. After 2,0002{,}000 episodes, Deep--BQRL's mean cumulative point-quantile gaps are 538.33538.33 and 92.7892.78 at τ=0.1,0.9\tau=0.1,0.9, versus PPO's 583.63583.63 and 173.51173.51 and TRPO's 855.22855.22 and 221.36221.36. Figure 4 provides a second comparison: FrozenLake median gaps are 400.0400.0, 3313.333313.33, and 5000.05000.0, respectively. The median distinguishes policies by whether success probability exceeds 0.50.5, while the reward curves track success throughout training. These results already supply policy-level quantile comparisons with both policy-gradient baselines.

W2/W4/W8: Additional diagnostics. We can add a controlled task with intermediate rewards to compare decisions after trajectories reach the same state and time with different accumulated rewards. Increasing problem sizes and reporting runtime would extend the computational assessment. Stage-wise comparisons with exact conditional actions, including the terminal stopping decision, can further explain the policy heatmaps.

W7/W10/Q3: Evaluation details. The two reported measures describe complementary aspects of learning: realized returns include exploration, while point-quantile gaps evaluate greedy policies. We can make this distinction more prominent and report expectation and quantile for the same final policy. The common numerical reference Qτ⋆Q_\tau^\star can be documented with its policy class, numerical procedure, and the role of positive-part clipping.

W6/W9/Q2/Q5: Reproducibility and sensitivity. A configuration table can report the selected β,K,M\beta,K,M, exploration and optimizer settings, baseline tuning ranges, evaluation frequency, and code instructions. Matched-budget single-quantile, buffer-width, and exploration ablations, together with baseline learning diagnostics, can complement the submitted search protocol and disjoint validation/test seeds.

Reviewer_SiPGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies static quantile optimization in reinforcement learning, with a slight modification to the objective by considering a range quantile, i.e., the average of quantiles over a small interval. Since the optimal policy for static quantile optimization is generally history-dependent, the authors propose a deep reinforcement learning method that aims to capture this dependence through three approximations: 1) augmenting the state with cumulative rewards to incorporate history information, 2) learning return distributions via neural quantile regression, and 3) performing greedy policy improvement based on estimated range quantiles. However, the paper does not provide sufficient theoretical justification for why the proposed method should optimize the intended objective.

Critical strengths

N/A

Critical weaknesses
  1. The current manuscript is not sufficiently self-contained. As indicated by the authors, the proposed method largely builds upon Alipour-Vaezi et al. (2026a) (Risk-Sensitive Reinforcement Learning with Smoothed Quantile Objectives), aiming to extend the existing approach to deep reinforcement learning. However, this closely related work is not adequately discussed or compared with the proposed method. Instead, a substantial portion of the manuscript is devoted to describing three approximations introduced to accommodate function approximation, without sufficiently explaining the original framework or motivating why these particular approximations are appropriate. Consequently, it is difficult for readers to understand the underlying problem formulation, the mathematical foundations of the proposed method, and the motivation behind its key design choices. More importantly, the lack of a detailed comparison with the work on which this paper is built makes it difficult to assess the novelty and technical contributions of the proposed method.

  2. Method and theoretical justification. The paper does not clearly establish the optimization objective corresponding to the proposed algorithm or explain its relationship to the original static quantile formulation, i.e., why the proposed method using distributional learning solves max⁡πQτ(Rπ)\max_\pi Q_\tau(R^\pi) (line 99); why using range quantile (so called buffered quantile in the paper) is better and how the policy obtained under range quantile connects to original max⁡πQτ(Rπ)\max_\pi Q_\tau(R^\pi)? As discussed in Li et al. (2022), optimizing a static quantile objective generally requires a history-dependent policy that dynamically adjusts the quantile level based on the realized trajectory. However, the proposed method does not explicitly implement such an adjustment. Instead, it augments the state with cumulative rewards while performing greedy policy improvement using a fixed quantile level. It remains unclear why this procedure should recover a policy that optimizes the original static quantile objective, even if the return distributions are estimated perfectly. This raises a fundamental concern regarding the theoretical justification of the proposed method. Furthermore, model-free learning for static quantile optimization has recently been investigated in Q-Learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis, which explicitly addresses how to learn optimal quantile values and construct an optimal policy in a model-free manner. The authors should discuss this closely related work and consider including it as an important baseline to better establish the contributions and empirical effectiveness of their proposed approach.

  3. Experiment. A key component of the proposed method is the incorporation of cumulative rewards into the inputs of the value and policy functions, which is intended to capture the history dependence required for static quantile optimization. However, in both experimental environments, all intermediate rewards are zero, implying that the cumulative reward remains zero at every nonterminal decision state. Consequently, the augmented state representation provides no additional information beyond the current state and time step. Therefore, the experiments do not demonstrate whether the proposed trajectory augmentation can effectively capture history dependence or improve policy optimization under a static quantile objective. Experiments involving nonzero intermediate rewards, where different trajectories may lead to the same state with different accumulated rewards, would be necessary to evaluate the effectiveness of this design choice.

Questions for authors

N/A

1 comment

Thank you for focusing on the relationship between static quantile optimization and our approximation. We address your three weaknesses through new analysis, existing evidence, and concrete additions. If the paper is accepted, we can incorporate these results, complete proofs, and additions into the final manuscript.

W1: Relationship to the earlier framework. We can add a self-contained comparison of the problem formulation, objective, and decision procedures. Deep--BQRL shares the lower-buffered criterion with UCB--BQRL and learns conditional remaining-return quantiles from sampled transitions. It compares actions through buffered scores and uses score disagreement during data collection. This replaces transition-confidence-set and return-distribution planning with learned conditional predictions. Stating the finite horizon, bounded rewards, quantile convention, and role of each approximation together would make the mathematical setting and contribution easier to follow.

W2: Accuracy of the computed scores. Fix a continuation policy π\pi, and let επ(x,a)\varepsilon^\pi(x,a) be the maximum error of sorted critic predictions against true remaining-return quantiles over critics and represented levels. For returns in [L,U][L,U], ∣μB(x,a)−Qτβ(Zπ(x,a))∣≤επ(x,a)+(U−L)/(2Kβ)|\mu_B(x,a)-Q_\tau^\beta(Z^\pi(x,a))|\le\varepsilon^\pi(x,a)+(U-L)/(2K\beta). Proof sketch: monotonicity bounds each cell's integrated midpoint error by its quantile increase divided by 2K2K. Summing cancels internal endpoints, giving (U−L)/(2K)(U-L)/(2K). Restricting to the buffer and normalizing gives the grid term; normalized overlap weights sum to one, bounding prediction error. Counting predictions around each true quantile proves that sorting preserves or reduces maximum error. Applying the score bound to the true best and selected actions, using their estimated-score ordering, gives conditional action loss at most 2ε+(U−L)/(Kβ)2\varepsilon+(U-L)/(K\beta) for uniform prediction error ε\varepsilon. This result controls action selection under the specified continuation policy.

W2: Buffered versus point objectives. Define bβ(π)=Qτ(Rπ)−Qτβ(Rπ)b_\beta(\pi)=Q_\tau(R^\pi)-Q_\tau^\beta(R^\pi). For policies attaining the two optima over the same class, Qτ(Rπτ⋆)−Qτ(Rπ^)≤bβ(πτ⋆)+Qτβ(Rπβ⋆)−Qτβ(Rπ^)Q_\tau(R^{\pi_\tau^\star})-Q_\tau(R^{\widehat\pi})\le b_\beta(\pi_\tau^\star)+Q_\tau^\beta(R^{\pi_\beta^\star})-Q_\tau^\beta(R^{\widehat\pi}). The bias is nonnegative and at most the quantile variation across the buffer. Proof sketch: express point values as buffered values plus biases, use −bβ(π^)≤0-b_\beta(\widehat\pi)\le0, and apply buffered optimality. Thus a globally buffered-optimal policy's point-quantile loss is controlled by bβ(πτ⋆)b_\beta(\pi_\tau^\star). This separates the effect of replacing the objective from the policy's buffered performance. We can include this explanation together with a buffer-width comparison, including single-quantile selection, to assess the empirical effect of buffering.

W2: A sufficient global condition. Construct an exact local policy backward by maximizing conditional buffered scores. Suppose, at every stage and augmented input, the selected action's remaining-return distribution first-order stochastically dominates every alternative's under that same later policy. Backward induction then establishes dominance of its total-return distribution over competing policies, yielding optimality for every point and buffered quantile. The proof conditions at the next stage and propagates distribution-function ordering backward. This provides an explicit sufficient condition for the exact local comparator, complementing the learned-score error bound. A comparison with Hau et al.'s quantile-MDP Q-learning can also be added on the finite benchmarks, with the same target levels, training budget, evaluation protocol, and held-out seeds.

W3: History conditioning and existing evidence. We can add a controlled diagnostic with nonzero intermediate rewards, where different trajectories reach the same state and stage with different accumulated rewards. Comparing inputs (h,s,c)(h,s,c) and (h,s)(h,s) would isolate the effect of reward-history information, with both policies evaluated under the known environment model. The submitted benchmarks already provide complementary policy-level evidence: Section 4.1 evaluates exploration-free greedy policies at the prescribed point quantile under the true model. After 2,0002{,}000 episodes, Figure 1 reports mean cumulative point-quantile gaps of 538.33538.33 and 92.7892.78 at τ=0.1,0.9\tau=0.1,0.9, compared with PPO's 583.63583.63 and 173.51173.51 and TRPO's 855.22855.22 and 221.36221.36. Figure 2 illustrates selected stopping decisions, while FrozenLake supplies a stochastic, sparse-reward test. We can make this interpretation of the existing results more explicit and present the proposed history diagnostic alongside it.

Reviewer_WSRoOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a novel framework that combines an existing quantile-based RL framework with nonlinear function approximation. The authors evaluate their proposed method on a number of tasks, and show that their proposed method can yield desirable results.

Critical strengths

The underlying research question is interesting, and the proposed approach appears to yield encouraging empirical results. In general, the writing is clear, and the figures are easy to understand.

Critical weaknesses

I have several concerns that would need to be addressed prior to publication:

  • My first concern relates to the underlying framework being used. In particular, the authors base their work on a recently-proposed and yet-to-be published work, Alipour-Vaezi et al., 2026a. While I am not implying that the underlying work is incorrect (I have not reviewed this work myself), I am nonetheless concerned that the underlying framework being used has yet to stand the test of peer review. This is a particularly important consideration to make, given that risk-sensitive methods are notoriously difficult to devise and analyze. In the context of this paper, I would encourage the authors to include a thorough discussion about the underlying framework, including its assumptions and limitations.
  • Another concern relates to the use of quantile-based distributional RL methods (see Approximation II in Section 3). In particular, one cannot explicitly set which quantile to learn when using such methods (one can only specify the number of quantiles to learn). This can potentially create challenges when a practitioner may want to learn a specific quantile. I would encourage the authors to discuss how they would go about addressing this limitation.
  • Some of the design choices are not well-motivated. For example, the choice to use multiple critics has no justification. Rather, the authors should show, either by citing a relevant and appropriate prior result, or conducting their own ablation studies, that the design choices are indeed needed. Moreover, the authors should aim to provide insights about how much the design choices affect performance.
  • Finally, although the authors provide a suite of empirical results, they only report how their method performs with respect to maximizing the expected return and the quality of the quantile estimates. Conversely, the authors do not show whether their proposed method actually optimizes the desired quantile of the return. Accordingly, the authors have not provided evidence that their methods accomplish its intended function. I strongly encourage the authors to report such results, as it would greatly strengthen their claims.
Questions for authors
  • What assumptions does the underlying framework from Alipour-Vaezi et al., 2026a utilize? Which of these assumptions would be violated by the proposed approximations?
  • Do the authors see any value in comparing their method to other (prior) quantile-based methods?
2 comments

Thank you for the positive assessment of the research question and presentation. We address your points through new analysis, the submitted evidence, and concrete additions. If the paper is accepted, we can incorporate the analysis, complete proofs, and additions described below into the final manuscript.

W1/Q1: Framework and assumptions. We can add a self-contained explanation of the finite-horizon MDP, bounded transition-dependent rewards, and quantile convention. The relationship to the earlier framework can also be made explicit: Deep--BQRL retains the lower-buffered criterion and learns conditional remaining-return quantiles directly from sampled transitions. These predictions replace transition-confidence-set and return-distribution planning in action selection. The analysis below directly characterizes the resulting scores and decisions.

W2: Specifying quantile levels. Approximation II explicitly assigns output jj to ηj=(j−12)/K\eta_j=(j-\tfrac12)/K, and this level appears in the quantile-regression loss. The network predicts the return value at each specified level. A target τ\tau between midpoints is handled by weighting grid cells according to their overlap with [τ−β,τ][\tau-\beta,\tau]. We can add a short numerical example to make this distinction clear.

W1/W2/Q1: Score and decision accuracy. Fix a continuation policy π\pi, and let επ(x,a)\varepsilon^\pi(x,a) be the maximum discrepancy between sorted predictions and true remaining-return quantiles over critics and levels. For remaining returns in [L,U][L,U], ∣μB(x,a)−Qτβ(Zπ(x,a))∣≤επ(x,a)+(U−L)/(2Kβ)|\mu_B(x,a)-Q_\tau^\beta(Z^\pi(x,a))|\le\varepsilon^\pi(x,a)+(U-L)/(2K\beta). Proof sketch: monotonicity bounds each cell's integrated midpoint error by its quantile increase divided by 2K2K. Summing cancels internal endpoints, giving (U−L)/(2K)(U-L)/(2K); buffer normalization gives the grid term. Normalized overlap weights sum to one, bounding the prediction contribution. Averaging preserves the bound. Counting predictions above and below each true quantile plus or minus the original maximum error proves that sorting preserves or reduces that error. If prediction error is at most ε\varepsilon across actions, applying the bound to the true best and estimated best actions gives conditional action loss at most 2ε+(U−L)/(Kβ)2\varepsilon+(U-L)/(K\beta). The estimated-score ordering supplies the intermediate inequality. This quantifies action accuracy under the specified continuation policy.

W1/Q1: Relating the objectives. Define bβ(π)=Qτ(Rπ)−Qτβ(Rπ)b_\beta(\pi)=Q_\tau(R^\pi)-Q_\tau^\beta(R^\pi). Monotonicity gives 0≤bβ(π)≤Qτ(Rπ)−Qτ−β(Rπ)0\le b_\beta(\pi)\le Q_\tau(R^\pi)-Q_{\tau-\beta}(R^\pi). For global maximizers over the same policy class, a buffered-optimal policy's point-quantile loss is at most bβ(πτ⋆)b_\beta(\pi_\tau^\star). Proof sketch: integrate differences between the target and neighboring quantiles; write point values as buffered values plus nonnegative biases and apply buffered optimality. Together, these results distinguish prediction error, finite-grid resolution, and the change in objective. Their assumptions and scope can be stated explicitly with the complete proofs.

W3: Multiple critics. The ensemble supplies an average score and an exploration bonus. We can add comparisons of a single critic with the configured ensemble and of ensemble action selection with and without the bonus, holding the quantile grid, architecture per critic, training budget, and evaluation seeds fixed. These comparisons would quantify the contribution of each design choice alongside the score guarantee.

W4: Existing policy-level evidence. Submitted Section 4.1, p. 6, evaluates Qτ(Rπˉt)Q_\tau(R^{\bar\pi_t}) under the true environment model with exploration removed. This directly measures the learned policy's point quantile. Figure 1 reports mean cumulative policy gaps of 538.33538.33 and 92.7892.78 after 2,0002{,}000 episodes at τ=0.1,0.9\tau=0.1,0.9, compared with PPO's 583.63583.63 and 173.51173.51 and TRPO's 855.22855.22 and 221.36221.36. Figure 2 illustrates selected policies accepting offers of at least 1616, 2121, and 2222 in the second period at τ=0.1,0.5,0.9\tau=0.1,0.5,0.9. FrozenLake's cumulative median gap is 400.0400.0, compared with 3313.333313.33 for PPO and 5000.05000.0 for TRPO. These results already provide the requested policy-level comparisons with PPO and TRPO. We can make the metric's interpretation and the role of the three-seed variability intervals more prominent.

Q2: Other quantile-based methods. A controlled comparison with QR-DQN or IQN using matched inputs, network capacity, training budget, evaluation protocol, and seeds can be added. Applying comparable quantile-sensitive action rules would help identify the contribution of the complete Deep--BQRL procedure.

I thank the authors for their response. While I agree that the proposed changes would improve the quality of the paper, I am not convinced that they can be made by the camera-ready deadline. Accordingly, I maintain my original score.

Reviewer_cEThOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Deep–BQRL, a model-free distributional RL method that combines quantile critics, lower-buffered action scores, cumulative-reward state augmentation, and ensemble-based exploration. Critics are trained using quantile Huber losses, while action selection averages estimated quantiles immediately below a target level. Experiments on asset selling and slippery FrozenLake report smaller cumulative point-quantile policy gaps than the tested PPO/TRPO implementations, although model-based UCB–BQRL remains stronger on this measure.

Critical strengths

The paper tries to address an important problem of implementing buffered-quantile control without explicit transition-model estimation or distributional planning. The action-score construction is transparent, and the algorithm clearly specifies how quantile learning and ensemble exploration interact. Separate validation and evaluation seeds, together with policy evaluation under the true environment model, are positive experimental choices. The discussion also appropriately acknowledges the distinction between the buffered learning criterion and point-quantile evaluation, as well as the limitations of the small benchmarks.

Critical weaknesses
  1. The conditional greedy update is not adequately connected to the episodic objective. Adding cumulative reward to the input does not establish that maximizing a fixed conditional buffered score improves the initial-state point quantile—or its buffered counterpart. The paper acknowledges this limitation but does not quantify the approximation gap, even using exact critics in small MDPs.

  2. The benchmarks do not test the central motivations. Both are small tabular problems, and no computational scaling results are provided. Moreover, their rewards are zero until termination, so cumulative reward is zero at every decision: the proposed history augmentation is effectively untested. FrozenLake’s binary returns also provide limited evidence of distinct risk-sensitive behavior.

  3. The comparison lacks closely matched distributional baselines, such as QR-DQN with comparable risk-sensitive action selection. Tabular PPO/TRPO do not isolate the contribution beyond existing quantile-based learning, while UCB–BQRL achieves substantially smaller cumulative quantile gaps. Thus, the reported comparisons provide limited evidence of a methodological advance.

  4. No ablations isolate buffering, output sorting, or ensemble exploration. Sensitivity to buffer width, quantile resolution, and ensemble size is also missing. Consequently, the experiments do not establish whether buffering improves upon single-quantile control or which components explain the gains. Three evaluation seeds further limit the precision of these comparisons.

  5. Essential implementation details are absent from the supplied manuscript: the actual buffer width, quantile count, ensemble size, critic architecture, optimizer settings, and evaluation frequency. The stated 16-point hyperparameter search neither identifies each method’s tuned parameter nor reports the selected values. These omissions substantially limit reproducibility.

  6. Relevant prior work is neither cited nor discussed. References [1] directly address non-crossing quantile representations and distribution-based exploration, warranting comparison with output sorting and buffered-score disagreement. Reference [2] is relevant to distributional critic estimation, although its primary objective is OPE. The authors should discuss these connections and distinguish their contribution.

[1] Zhou, F., Zhu, Z., Kuang, Q., and Zhang, L. (2021). Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning. Proceedings of IJCAI-21, 3455–3461. doi:10.24963/ijcai.2021/476.

[2] Kuang, Q., Wang, C., Jiao, Y., and Zhou, F. (2026). Distributional Off-Policy Evaluation with Deep Quantile Process Regression. Journal of the American Statistical Association. doi:10.1080/01621459.2026.2671449.

Questions for authors
  1. With exact return-distribution critics, how far can the fixed-level greedy update remain from the optimal episodic buffered or point-quantile policy? Can the authors quantify this separately from estimation error?

  2. Equation (3) uses critic-specific continuation actions, whereas Equation (2) deploys an ensemble-mean policy. Which policy’s return distribution does each critic estimate, and how to separate uncertainty from disagreement caused by different continuation policies?

  3. Can the authors evaluate environments with nonzero intermediate rewards and larger state spaces, reporting both runtime and sample efficiency, to test history conditioning and scalability?

  4. Can the authors provide the complete implementation settings and controlled comparisons against single-quantile selection, no exploration bonus, and non-crossing critics based on [1,2]?

1 comment

Thank you for recognizing the evaluation protocol. We provide new analysis below. If the paper is accepted, we will include these results, complete proofs, and clarifications in the final manuscript.

W1/Q1: Prediction and decision accuracy. Fix a continuation policy π\pi, and define επ(x,a)\varepsilon^\pi(x,a) as the maximum error between sorted critic predictions and true remaining-return quantiles at the represented levels. For returns in [L,U][L,U], ∣μB(x,a)−Qτβ(Zπ(x,a))∣≤επ(x,a)+(U−L)/(2Kβ)|\mu_B(x,a)-Q_\tau^\beta(Z^\pi(x,a))|\le\varepsilon^\pi(x,a)+(U-L)/(2K\beta). Proof sketch: monotonicity bounds each midpoint cell's integrated error by its quantile increase divided by 2K2K. Summing cancels internal endpoints; buffer normalization gives the grid term. Normalized overlap weights sum to one, bounding the prediction contribution. Counting predictions within the original maximum-error bounds shows that sorting preserves that bound. Applying the score bound to the true best and selected actions, using their estimated-score ordering, bounds local action loss by 2ε+(U−L)/(Kβ)2\varepsilon+(U-L)/(K\beta) for uniform error ε\varepsilon, with the same continuation policy.

Q2: Continuation policies and disagreement. Submitted Equation (3) constructs each target using that target critic's selected continuation, while Equation (2) deploys the ensemble-mean action. The error επ\varepsilon^\pi includes differences from the specified continuation π\pi. Ensemble disagreement reflects both prediction and continuation differences. Paired runs using a shared next-action selector can assess how continuation choice affects score disagreement.

W1/Q1: Structural comparison with exact critics. Construct the exact backward-greedy policy πloc\pi^{\mathrm{loc}}. Let αh\alpha_h be the greatest positive distribution-function excess of its selected action over alternatives, across stage-hh inputs and thresholds with the same continuation. For finite state and action spaces and total returns in [L0,U0][L_0,U_0], every competing policy π\pi satisfies Qτβ(Rπ)−Qτβ(Rπloc)≤(U0−L0)min⁡{1,∑hαh/β}Q_\tau^\beta(R^\pi)-Q_\tau^\beta(R^{\pi^{\mathrm{loc}}})\le(U_0-L_0)\min\{1,\sum_h\alpha_h/\beta\}. Proof sketch: propagate one-sided conditional distribution differences backward, then convert their integrated area into a quantile-area bound. When all αh=0\alpha_h=0, distributional dominance establishes global point- and buffered-quantile optimality of this exact comparator. The score bound above separately characterizes prediction-dependent action accuracy relative to a specified continuation.

W2/W3: Existing policy-quality findings. Submitted Figure 1 reports mean cumulative point-quantile gaps after 2,0002{,}000 asset-selling episodes of 538.33538.33 and 92.7892.78 for Deep--BQRL at τ=0.1,0.9\tau=0.1,0.9, versus 583.63583.63 and 173.51173.51 for PPO and 855.22855.22 and 221.36221.36 for TRPO. Policies are evaluated under the true model, so these measurements directly assess the learned decisions. Figure 4 gives FrozenLake median gaps of 400.0400.0, 3313.333313.33, and 5000.05000.0, respectively. The median identifies whether success probability exceeds 0.50.5; the reward curves provide complementary training-performance information. These findings establish the reported comparisons with the tested PPO/TRPO implementations.

W2/Q3: Targeted environment extensions. Nonzero intermediate rewards can create trajectories with different accumulated returns at the same state and stage, allowing a direct comparison with and without the accumulated-return input. Modest increases in grid size or horizon can support runtime and sample-efficiency comparisons under controlled budgets.

W3/W4/Q4: Matched comparisons and ablations. Same-budget QR-DQN/IQN comparisons can use matched architectures and seeds with the existing true-model policy evaluation. A compact study can vary β,K,M\beta,K,M, compare direct single-quantile selection with buffering, and switch sorting and the exploration bonus on and off. Non-crossing critics provide a further targeted comparison. The current three-seed means and intervals summarize variability, and additional seeds can strengthen these component comparisons.

W5/Q4: Implementation details. The submitted 1616-point logarithmic search, successive halving with factor 22, disjoint validation/test seeds, and training budgets establish independent selection. A complete settings table can add each method's tuned parameter, search range, selected value, β,K,M\beta,K,M, architecture, optimizer settings, and evaluation frequency, drawn from the experiment configurations.

W6/Q4: Related work. Zhou et al. (2021) provide relevant non-crossing quantile and exploration designs; Kuang et al. (2026) offer a complementary perspective on distributional critic estimation for off-policy evaluation. Their connections to output sorting, buffered-score disagreement, and critic estimation can be explained explicitly in the related-work discussion.

Reviewer_9PUFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Deep-BQRL, a model-free deep reinforcement learning framework for objectives based on quantiles of cumulative returns. It uses multiple distributional critics to estimate quantiles of conditional residual returns and aggregates quantiles within a specified range to construct a lower-buffered quantile action score. The network takes the timestep, current state, and accumulated return as inputs to account for history dependence. The method is evaluated on the asset-selling optimal stopping problem and Slippery FrozenLake. Deep-BQRL outperforms PPO and TRPO on some point-quantile metrics but generally underperforms the model-based UCB-BQRL. Overall, the experiments do not sufficiently support the paper's claims, and the contributions of the proposed components remain unclear.

Critical strengths
  • Model-free optimization of risk-sensitive quantile objectives is a focus problem. The proposed framework combines buffered quantile objectives with distributional deep reinforcement learning and trains directly from sampled transitions.
  • The method combines quantile weighting, critic updates, and ensemble-based exploration. The experiments report both expected-return metrics and point-quantile policy gaps. The authors also acknowledge that optimizing the buffered objective does not necessarily yield an optimal policy for the point-quantile objective.
Critical weaknesses
  • The method lacks a principled basis for policy improvement under the global quantile objective, which is generally time-inconsistent. In a Quantile MDP, the auxiliary quantile state needs to be updated according to the realized stochastic transitions. Although the authors describe their method as an approximation, they provide no theoretical conditions or error analysis to explain when this approximation is valid or how much performance it may lose.
  • In both experimental environments, all non-terminal rewards are zero. Consequently, the accumulated-return input remains unchanged during decision-making, and the proposed input augmentation does not demonstrate its ability to capture history dependence.
  • The paper lacks systematic ablations and parameter studies. It does not compare point quantiles with buffered quantiles or lower-tail averages, nor does it isolate the effects of buffer width, ensemble disagreement, trajectory augmentation, or the number of critics. It is therefore unclear which components contribute to the reported performance.
Questions for authors
  • Under what conditions does greedy action selection based on a local buffered quantile yield an optimal policy for the global cumulative-return quantile? If this cannot be guaranteed, can the resulting approximation error or performance loss be bounded?
  • Since the accumulated return remains zero at all non-terminal decision states in both Asset-Selling and FrozenLake, how does the proposed input augmentation address the history dependence of quantile-based control?
  • Can the authors compare point quantiles, buffered quantiles with different buffer widths, and lower-tail averages? How much does performance change when the buffering mechanism is removed?
  • How does Deep-BQRL's policy improvement mechanism differ from those in QR-DQN, IQN, and other risk-sensitive distributional RL methods? Can the authors provide controlled comparisons using the same critic architecture, training budget, and random seeds?
1 comment

Thank you for highlighting the importance of model-free quantile-sensitive learning. We present new analysis and clarify the experimental evidence below. If the paper is accepted, we will include these results, complete proofs, and clarifications in the final manuscript.

W1/Q1: Accuracy of the computed scores. Fix a continuation policy π\pi, and let επ(x,a)\varepsilon^\pi(x,a) be the maximum error of sorted critic predictions against the true remaining-return quantiles at the represented levels. For returns in [L,U][L,U], ∣μB(x,a)−Qτβ(Zπ(x,a))∣≤επ(x,a)+(U−L)/(2Kβ)|\mu_B(x,a)-Q_\tau^\beta(Z^\pi(x,a))|\le\varepsilon^\pi(x,a)+(U-L)/(2K\beta). Proof sketch: monotonicity bounds each cell's integrated midpoint error by its quantile increase divided by 2K2K. Summing cancels internal endpoints, giving (U−L)/(2K)(U-L)/(2K). Restricting to the buffer and dividing by β\beta gives the grid term. Normalized overlap weights sum to one, bounding the prediction contribution by επ(x,a)\varepsilon^\pi(x,a). Averaging critics preserves the bound. Counting predictions above and below each true quantile plus or minus the original maximum error shows that sorting preserves this error bound.

W1/Q1: Consequences for action selection. If prediction error is at most ε\varepsilon for every action, the selected action's loss in true conditional buffered score is at most 2ε+(U−L)/(Kβ)2\varepsilon+(U-L)/(K\beta). Apply the score bound to the true maximizing action and estimated maximizing action, using their estimated-score ordering between these steps. Both actions use the same continuation policy. A unique best action is recovered whenever its advantage exceeds this bound, giving a precise sufficient condition for accurate local decisions.

W1/Q1: A structural condition for global optimality. Construct πloc\pi^{\mathrm{loc}} backward using exact conditional scores. Let αh\alpha_h be the largest positive difference between its selected action's remaining-return distribution function and an alternative's, over stage-hh inputs and thresholds with the same local continuation. For finite state and action spaces and total returns in [L0,U0][L_0,U_0], every competing policy π\pi satisfies Qτβ(Rπ)−Qτβ(Rπloc)≤(U0−L0)min⁡{1,∑hαh/β}Q_\tau^\beta(R^\pi)-Q_\tau^\beta(R^{\pi^{\mathrm{loc}}})\le(U_0-L_0)\min\{1,\sum_h\alpha_h/\beta\}. Proof sketch: backward induction accumulates one-sided distribution-function differences; their integrated positive area equals the reversed positive quantile-difference area, yielding the bound. If every αh=0\alpha_h=0, distributional dominance establishes optimality of this exact local comparator for every point and buffered quantile. This separates the structural comparison from the prediction-dependent action bound above.

W1/Q1: Relating the objectives. Monotonicity also gives 0≤Qτ(Rπ)−Qτβ(Rπ)≤Qτ(Rπ)−Qτ−β(Rπ)0\le Q_\tau(R^\pi)-Q_\tau^\beta(R^\pi)\le Q_\tau(R^\pi)-Q_{\tau-\beta}(R^\pi). The proof integrates the differences between the target quantile and neighboring quantiles over the buffer. Thus the variation across the buffer explicitly bounds the change in the policy's objective value.

W3/Q3 and Q4: Policy-level evidence already provided. Submitted Section 4.1 evaluates exploration-free greedy policies under the true model. Figure 1 reports mean cumulative point-quantile gaps of 538.33538.33 and 92.7892.78 after 2,0002{,}000 episodes at τ=0.1,0.9\tau=0.1,0.9, versus PPO's 583.63583.63 and 173.51173.51 and TRPO's 855.22855.22 and 221.36221.36. These results show smaller mean policy gaps than the tested policy-gradient implementations. Figure 2 complements these measurements with selected stopping policies at different target levels. The three-seed means and intervals summarize variability.

W2/Q2: Evaluating history conditioning. A controlled extension with nonzero intermediate rewards can produce trajectories reaching the same state and stage with different accumulated returns. Comparing otherwise matched policies with and without the accumulated-return input would directly assess the contribution of this representation.

W3/Q3: Component comparisons. A compact ablation can compare a direct single-quantile action score, several buffer widths, and the lower-tail average obtained with β=τ\beta=\tau. Varying KK and MM and switching the disagreement bonus on and off can assess quantile resolution, ensemble size, and exploration while retaining the current evaluation protocol.

Q4: Matched distributional baselines. Deep--BQRL combines learned quantiles, buffered action scores, and ensemble-based exploration. Its buffered score admits a uniform-window weighting interpretation, providing a common basis for QR-DQN/IQN comparisons. Matched architecture, training-budget, and seed comparisons can be added to assess these methods under the same point-quantile policy metric.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.