- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces GAN-DRL, a method that proposes the use of information gain from Bayesian inference to increase exploration. The authors claim that the resulting uncertainty-guided exploration leads to sample efficient distributional reinforcement learning with a quantile critic learning, seeking for improved risk-sensitivity in downstream applications. Experiments on Option Hedging and F1Tenth Driving demonstrate the applicability of the approach.
Critical strengths
- The paper aims to address the important and relevant problem of sample efficiency in risk-sensitive reinforcement learning.
Critical weaknesses
- There is a lack of appropriate baselines. The paper proposes a method for risk-sensitive reinforcement learning but compares only against risk-neutral baselines. Moreover, the proposed approach aims to improve sample efficiency via curiosity like exploration bonuses but similar exploration bonus baselines are not compared against. There are no ablations.
- The experiments are not very convincing. On both option hedging and driving, the risk-sensitive method performs similarly to risk-neutral baselines. While it does achieve better risk-sensitive performance more quickly, it is not clear from these experiments whether this is due to information gain bonus or since the proposed method is the only one optimizing for risk. Moreover, the risk-neutral baselines often surpass the risk-sensitive method in CVaR during the middle of training which places the appropriateness of the evaluation setting into question.
- The writing lacks clarity in some places, and notably in the methodology. Figure 1 contains variables and notations that are not consistent with itself and the text. For example, the Bellman target generator should have subscript but uses instead, the intrinsic rewards use superscripts but this is not used in text, the risk-sensitive control section uses but the remainder of the text uses . Several variables in the tex are left undefined, such as in (14), and are not formally defined (I assume they are Gaussian?), the discriminator , etc. There is also no description of implementation details, or hyperparameters used, which would make reproduction of this work difficult.
- Novelty is not clear. The proposed method is very similar to [R1] which the authors cite but do not discuss in detail nor compare against.
[R1] Li, L., & Faisal, A. A. (2021, May). Bayesian distributional policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 10, pp. 8429-8437).
Questions for authors
Please see weaknesses.
- Can the authors compare against or explain the lack of comparison against alternative exploration methods such as RND, and curiosity driven exploration? Similarly, can the authors baseline against alternative risk-sensitive approaches such as QR-DQN/IQN optimizing for CVaR/VaR, or more modern approaches?
- Can the authors position their method against the Bayesian distributional policy gradients proposed by Li, L., & Faisal, A. A. (2021, May).
- The authors argue for improved sample efficiency but evaluate on rather low-dimension and relatively simple environments for a limited number of episodes. Can the authors provide experiments on more complicated settings common in the risk-sensitive RL literature, such as stochastic mujoco, to demonstrate the efficiency gain of their approach?
Minor:
- L043: I believe this sentence should be clarified. Distributional RL, inherently, does not go beyond the the expectation objective and the cited papers still optimize for expectation. The authors can cite IQN, but even then the risk-sensitive optimization is heuristic. There are better works to cite for principled risk sensitive learning.
- Please fix the citations with \pcite instead of \cite.
- L165: Quantile Regression is how the quantile positions are learned, it is not the parameterization which is simply via quantile masses.
2 comments
Part 2 of the response
Stochastic MuJoCo evaluation. We agree that an additional continuous-control benchmark is useful, and we added a stochastic MuJoCo InvertedPendulum-v4 experiment with state-dependent reward uncertainty. We also note that state dimensionality alone does not capture the sources of risk in the original tasks: option hedging contains stochastic arrivals, overlapping positions, transaction costs, and volatility-dependent tail risk, while F1TENTH is a 12-dimensional continuous-control task with nonlinear vehicle dynamics and rollover failures.
The MuJoCo experiment provides a complementary source of risk: Gaussian reward perturbations are applied only when the cart enters a designated risky region. All methods are trained for 500,000 environment interactions across six seeds under the same evaluation protocol. Figure 4 shows that GAN-D4PG reaches strong lower-tail performance early and remains among the best methods during training; SAC is better at several intermediate checkpoints, but GAN-D4PG recovers and achieves the highest final CVaR. In the final evaluation, GAN-D4PG achieves the strongest mean return and $498.0\pm4.8$ and and the lowest fall rate %. The corresponding final CVaR values for QR-D4PG, IQN, QR-D4PG+RND, RACER, and SAC are , and , respectively.
Minor: distributional RL versus risk-sensitive optimization. We agree that distributional RL by itself does not imply a risk-sensitive objective. The revised text now distinguishes learning the return distribution from optimizing an explicit risk functional. GAN-D4PG, QR-D4PG, IQN, and RACER use CVaR-based policy objectives, whereas SAC optimizes expected return.
Minor: citation formatting. We reviewed the manuscript and corrected inconsistent citation commands so that parenthetical and textual citations use the appropriate (\citep/\citet) form consistently.
Minor: quantile representation. We agree that quantile regression is the learning procedure, not the parameterization. The revised background now describes the critic as a finite set of equally weighted Dirac masses located at learned quantile values, with those values learned using the quantile-regression loss.
References
[1] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning (ICML), pages 1096–1105, 2018.
[2] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations (ICLR), 2019.
[3] Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.08617, 2018.
[4] Luchen Li and A. Aldo Faisal. Bayesian distributional policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, 35(10):8429–8437, 2021.
Part 1 of the response
We thank the reviewer for the detailed feedback. We substantially revised the manuscript to strengthen the baseline coverage, ablation analysis, methodology presentation, and positioning relative to BDPG.
Baselines and ablations. We agree that the original evaluation did not sufficiently isolate the contribution of posterior-guided exploration. The revised paper now includes multiple risk-sensitive baselines: QR-D4PG, IQN [1], and RACER optimize CVaR-based objectives, while SAC remains the risk-neutral baseline. We also added QR-D4PG+RND, which uses the same quantile-based risk-sensitive D4PG backbone but replaces our posterior-shift signal with Random Network Distillation (RND) [2] prediction error. Appendix C.4 reports ablations in which GAN-D4PG keeps the same generator, encoder, discriminator, quantile critic, actor, and training setup but sets ηt=0. We additionally report sensitivity to η0∈0.075,0.15. Across hedging, F1TENTH, and stochastic MuJoCo, these results separate the effect of the exploration signal from the underlying risk-sensitive learner and from a simpler prediction-error bonus.
Experimental interpretation. We revised the discussion to avoid claiming pointwise or uniform dominance. In option hedging, GAN-D4PG reaches strong CVaR earlier than the matched risk-sensitive alternatives and achieves the strongest final CVaR: at σ=0.3, −7.34 versus −7.57, −7.98, and −8.00 for QR-D4PG, IQN, and QR-D4PG+RND; at σ=0.7, −16.44 versus −17.99, −103.12, and −17.44, respectively. For F1TENTH, the training-time advantage is more modest and nonuniform. On Flat, QR-D4PG and IQN outperform GAN-D4PG at some intermediate checkpoints, and IQN has slightly higher final CVaR 802.8 vs. 787.7. On Bumpy, GAN-D4PG becomes increasingly favorable later in training and achieves the strongest final CVaR 639.1. We now report these crossings explicitly. The broader conclusion is that GAN-D4PG reaches strong lower-tail performance with competitive sample efficiency while maintaining strong final CVaR across the evaluated domains.
Methodology clarity and reproducibility. We substantially revised Fig. 1 and Sec. 4 so that the generator, encoder, target encoder, discriminator, quantile critic, posterior-shift signal, and exploration-augmented reward use consistent notation throughout. We now explicitly define the latent prior, discriminator inputs, target networks, Bellman-target construction, and the distinction between the generative return model and the quantile critic. We also added the original D4PG citation [3] and expanded Appendix C with network architectures, learning rates, batch sizes, (n)-step returns, target-update schedules, exploration parameters, risk levels, and baseline-specific implementation details.
Novelty and relation to BDPG. We agree that BDPG [4] and GAN-DRL share important ingredients, including generative return modeling, and posterior-based curiosity. In BDPG, samples from the generative return model are used to construct the return and advantage estimates that drive the policy-gradient update. In GAN-DRL, the generative return model instead provides the posterior-guided exploration signal. The return distribution used directly for policy optimization, Zφ(s,a), is represented by an independently parameterized quantile critic trained with the exploration-augmented reward. The two return representations therefore serve distinct roles: generative modeling supports exploration, while the quantile representation supports explicit lower-tail control. Appendix C.5 evaluates this design choice directly. For CVaR0.95, we compare the quantile critic with an empirical CVaR estimate from 99 generator samples, matching the critic's 99 quantiles, against a reference computed from 5,000 Bellman-return samples. The quantile critic achieves lower absolute CVaR estimation error across the evaluated checkpoints, while repeated 99-sample generator estimates exhibit Monte Carlo variability. Thus, VaR/CVaR can be computed from learned lower-tail quantiles rather than estimated empirically from a finite set of generated returns.
Additional exploration/risk-sensitive baselines. As requested, we added QR-D4PG+RND [2] as a simpler curiosity-driven baseline and continuous-action IQN [1] with the same CVaR actor objective; RACER remains as a recent risk-sensitive actor-critic baseline. GAN-D4PG achieves stronger final CVaR than RND in both hedging settings, both F1TENTH tasks, and stochastic MuJoCo. Relative to IQN, GAN-D4PG is stronger in both hedging settings, on F1TENTH-Bumpy, and on MuJoCo, while IQN is slightly better on F1TENTH-Flat.