- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper extends lower-buffered quantile objectives (Q^beta_tau, the average of return quantiles over [tau-beta, tau], introduced in the authors' model-based UCB-BQRL) to model-free deep RL. DEEP-BQRL trains an ensemble of QR-DQN-style quantile critics on an augmented input (h, s, c) with c the cumulative reward, selects actions greedily by a "buffered score" — the interval-overlap-weighted average of predicted quantiles in [tau-beta, tau] — and adds ensemble disagreement on that score as an exploration bonus. Experiments on a 25-state asset-selling problem and 4x4 slippery FrozenLake show smaller point-quantile policy gaps than PPO/TRPO but gaps roughly 20-100x larger than the model-based UCB-BQRL reference.
Critical strengths
The paper is unusually honest: the criterion mismatch (training on Q^beta_tau, evaluating on point Q_tau), the non-isolation of components, the degeneracy of FrozenLake for quantile objectives, and the model-based reference's dominance are all stated explicitly. The component formulas are correct (the overlap weights in Eq. (1), the quantile Huber loss, terminal handling). The evaluation protocol has good hygiene (disjoint validation/test seeds, true-model quantile reference, exploration-free greedy-policy gap). The direction of the learned risk-sensitive stopping rules in Figure 2 is correct and interpretable.
Critical weaknesses
The policy class has a strict expressivity ceiling, and that the authors omit mentioning. Exact quantile-optimal planning is history-dependent through an evolving conditional quantile level (QMDP; Li et al. 2022): after a lucky start the optimal continuation locks in (low conditional level), after an unlucky one it gambles (high conditional level). Fixed-(tau, beta) buffered-greedy selection on (h, s, c) cannot express this level flip. Concretely, an H=2 counterexample (action A: deterministic 0.7; action B: first reward 0.6 or 0 w.p. 1/2, then a safe 0.5 or risky {0,1} choice) has median-optimal value 1.0 via hedging, while every fixed-(tau, beta) greedy policy scores at most 0.7 — even with perfect critics and infinite data. This is not estimation error; no policy-improvement lemma for buffered-greedy selection can exist, and the induced distributional optimality operator is not a contraction. Relatedly, the next action is selected and evaluated by the same target network (Eq. (3)), giving uncorrected maximization bias on the sorted quantiles.
The mechanism that motivates the architecture is never exercised: c is identically zero at every decision point in both environments (asset-selling rewards only on Stop, which terminates; FrozenLake rewards only at the terminal goal; Figure 2 is explicitly evaluated at C_h = 0), and asset-selling's quantile optimum is Markov in (h, s) anyway. So the claimed handling of history dependence has zero empirical evidence.
The novelty accounting needs major revision. The buffered score is exactly a distortion-risk greedy policy with a uniform window over [tau-beta, tau] — the mechanism IQN introduced in 2018 (Dabney et al., arXiv:1806.06923), which is not cited; lower-buffered quantile is Range VaR, and deep RL with RVaR/spectral objectives exists (Coache et al. 2023; Moghimi et al., AISTATS 2025). Model-free quantile-objective RL also exists (Hau et al., "Q-learning for Quantile MDPs," AISTATS 2025), as does (state, cumulative-reward) augmentation for percentile criteria (Chow et al., JMLR 2018) and ensemble-UCB exploration on quantile critics (Chen et al. 2017; SUNRISE 2021; Mavrin et al. 2019). Also worth noting for the AC: the submission appears to be arXiv:2609.21327, and the main baseline (arXiv:2608.22227) is the same group's unreviewed preprint — the framing "extends buffered-quantile learning to neural function approximation" is an incremental follow-up whose reference method has not been peer-reviewed.
The experiments cannot support the stated motivation. Both environments are tabular-exact, so the scalability claim — the paper's entire premise — is untested; and even there, UCB-BQRL's gaps are 20-100x smaller in every cell. FrozenLake is a zero-information quantile benchmark: with binary returns Q_tau = 1{p > 1-tau}, the tau=0.1 gap is identically zero for all methods (the reference quantile itself is 0), and the optimal policy is tau-invariant (the paper admits both). The baselines are not credible: TRPO scores ~0 on a 16-state problem where the optimal success rate is ~0.74, and each method was tuned on a single "dominant" hyperparameter ranked by the quantile-gap criterion that PPO/TRPO do not even optimize — which can actively misselect their configs. The tau=0.9 regret "win" is an artifact: a tau=0.9-optimal policy can have very low expected return, so low regret there indicates weak tau-differentiation, not virtue; and at tau=0.1 the regret of 1025 is genuine underlearning (the quantile-optimal policy there is nearly expectation-optimal). The learned policies also persist in strictly dominated actions (rejecting positive offers at the final period), which damages exactly the lower-tail paths a tau=0.1 agent exists to protect. Finally, reproducibility: the buffer width beta — the titular hyperparameter — is never given a value anywhere; K, M, architectures, evaluation period, and code are all unreported.
Minor: sorting the target quantiles has provably zero effect on the loss in Eq. (4) (the (i,j) double sum depends only on the target multiset), so the claim in Sec. 3 that sorted values are used for "learning targets" is misleading; the regret is defined with realized returns rather than pseudo-regret; and it is unclear whether the reference Q*_tau is the exact optimum over history-dependent policies (the redundant [·]+ clipping suggests not).
Questions for authors
Do you agree that fixed-(tau, beta) buffered-greedy policies form a strict subset of quantile-optimal policies (e.g., the hedging counterexample above), and can you bound the resulting gap in terms of the problem structure? If not, where does the argument fail? What values of beta, K, M, and the exploration parameters were used, and how does performance vary with beta (including the point-quantile limit)? This ablation is the only way to isolate whether buffering itself helps. Is Q*_tau computed as the exact optimum over history-dependent policies? Please specify. How does DEEP-BQRL compare to an IQN critic with a uniform-window distortion (which implements essentially the same action score) and to Hau et al.'s quantile Q-learning on these benchmarks? Why does TRPO score ~0 on 4x4 FrozenLake, and which hyperparameters were tuned for each baseline?
Thank you for the suggestions. Labels follow your comment topics. We present new analysis and submitted evidence below. If the paper is accepted, the analysis, full proofs, and proposed additions can be included in the final manuscript.
W1/Q1: Quantifying local control. We analyze the buffered decision rule separately from the policy representation (h,s,c). Fix a continuation policy π and let επ(x,a) be the maximum discrepancy between sorted critic outputs and true remaining-return quantiles over critics and assigned levels. For returns in [L,U], ∣μB(x,a)−Qτβ(Zπ(x,a))∣≤επ(x,a)+(U−L)/(2Kβ). Proof sketch: monotonicity bounds each cell's midpoint-integration error by its quantile increase divided by 2K; summing cancels internal endpoints. Normalized overlap weights bound the prediction contribution. Applying the score bound to the true best and estimated best actions gives local score loss at most 2ε+(U−L)/(Kβ) when prediction error is at most ε across actions. A unique best action is recovered when its advantage exceeds this bound.
W1/Q1: Conditions for global optimality. Construct πloc backward using exact conditional buffered scores. Let αh be the largest positive distribution-function excess of its selected action over alternatives, across stage-h inputs and thresholds under the same continuation. For finite state/action spaces, total returns in [L0,U0], and any policy π, Qτβ(Rπ)−Qτβ(Rπloc)≤(U0−L0)min{1,∑hαh/β}. Proof sketch: backward induction propagates one-sided distribution differences; integration converts their positive area into a quantile-area bound. If every αh=0, first-order stochastic dominance gives optimality for all point and buffered quantiles. This complements the score bound with a comparison for exact local control.
W1/W10: Target updates and sorting. Sorting determines the ordering used to construct buffered action scores. For a fixed next action, the double-sum regression loss is invariant to permutations of target values. We can clarify these two roles and specify that each target critic selects and evaluates its continuation action, while deployment uses the ensemble-mean score. A separate-selection/evaluation comparison can be added to assess the target-update choice.
W3/Q4: Relationship to prior methods. The submission connects the buffered criterion to Range VaR. Its uniform-window interpretation follows by changing variables in the buffered integral. We can expand the discussion of IQN, spectral-risk methods, quantile-MDP Q-learning, reward augmentation, and ensemble exploration, explaining how Deep--BQRL combines sampled-transition learning with buffered-score action selection. A matched-budget comparison with uniform-window IQN and quantile-MDP Q-learning can be added on these finite benchmarks.
W4/W5/Q4: Policy-level evidence already reported. Submitted Figure 1 evaluates learned policies under the true model. After 2,000 episodes, Deep--BQRL's mean cumulative point-quantile gaps are 538.33 and 92.78 at τ=0.1,0.9, versus PPO's 583.63 and 173.51 and TRPO's 855.22 and 221.36. Figure 4 provides a second comparison: FrozenLake median gaps are 400.0, 3313.33, and 5000.0, respectively. The median distinguishes policies by whether success probability exceeds 0.5, while the reward curves track success throughout training. These results already supply policy-level quantile comparisons with both policy-gradient baselines.
W2/W4/W8: Additional diagnostics. We can add a controlled task with intermediate rewards to compare decisions after trajectories reach the same state and time with different accumulated rewards. Increasing problem sizes and reporting runtime would extend the computational assessment. Stage-wise comparisons with exact conditional actions, including the terminal stopping decision, can further explain the policy heatmaps.
W7/W10/Q3: Evaluation details. The two reported measures describe complementary aspects of learning: realized returns include exploration, while point-quantile gaps evaluate greedy policies. We can make this distinction more prominent and report expectation and quantile for the same final policy. The common numerical reference Qτ⋆ can be documented with its policy class, numerical procedure, and the role of positive-part clipping.
W6/W9/Q2/Q5: Reproducibility and sensitivity. A configuration table can report the selected β,K,M, exploration and optimizer settings, baseline tuning ranges, evaluation frequency, and code instructions. Matched-budget single-quantile, buffer-width, and exploration ablations, together with baseline learning diagnostics, can complement the submitted search protocol and disjoint validation/test seeds.