acceptodds
Under review as a conference paper at ICLR 2027

Bayesian Adversarial Distributional Reinforcement Learning for Sample-Efficient Risk-Sensitive Control

Abstract

Distributional reinforcement learning (DRL) enables risk-sensitive decision-making by learning the distribution of cumulative returns rather than only their expectation. However, accurately estimating return distributions can require substantial interaction with the environment, making DRL difficult to apply when data collection is risky, costly, or limited. With limited experience, uncertainty in the learned return distribution can remain substantial, yet many widely used DRL methods do not explicitly quantify epistemic uncertainty in the learned return model. We propose GAN-DRL, a Bayesian Generative Adversarial Network (GAN) framework for distributional reinforcement learning that combines Wasserstein-based return-distribution learning with Bayesian inference to quantify epistemic uncertainty in the learned return model. We use posterior information gain as an intrinsic signal to guide the agent toward informative experience, improving learning efficiency in stochastic risk-sensitive environments. We evaluate GAN-DRL on two continuous-control problems: European option hedging and F1TENTH autonomous driving. Across both domains, GAN-DRL reaches comparable or improved risk-sensitive performance using substantially fewer environment interactions than distributional and actor-critic baselines. In European option hedging, GAN-DRL improves tail-risk performance, as measured by conditional value-at-risk (CVaR), while maintaining competitive average performance. In autonomous driving, GAN-DRL improves lower-tail returns and reduces rollover rates on both the Offroad-Flat and Offroad-Bumpy tasks. Overall, these results show that uncertainty-guided distributional learning can substantially improve sample efficiency while maintaining or improving risk-sensitive performance.

Primary area:
reinforcement learning
Keywords:
Distributional Reinforcement Learning, generative adversarial networks, risk-sensitive control, sample-efficient reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wLkTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GAN-DRL, a method that proposes the use of information gain from Bayesian inference to increase exploration. The authors claim that the resulting uncertainty-guided exploration leads to sample efficient distributional reinforcement learning with a quantile critic learning, seeking for improved risk-sensitivity in downstream applications. Experiments on Option Hedging and F1Tenth Driving demonstrate the applicability of the approach.

Critical strengths
  1. The paper aims to address the important and relevant problem of sample efficiency in risk-sensitive reinforcement learning.
Critical weaknesses
  1. There is a lack of appropriate baselines. The paper proposes a method for risk-sensitive reinforcement learning but compares only against risk-neutral baselines. Moreover, the proposed approach aims to improve sample efficiency via curiosity like exploration bonuses but similar exploration bonus baselines are not compared against. There are no ablations.
  2. The experiments are not very convincing. On both option hedging and driving, the risk-sensitive method performs similarly to risk-neutral baselines. While it does achieve better risk-sensitive performance more quickly, it is not clear from these experiments whether this is due to information gain bonus or since the proposed method is the only one optimizing for risk. Moreover, the risk-neutral baselines often surpass the risk-sensitive method in CVaR during the middle of training which places the appropriateness of the evaluation setting into question.
  3. The writing lacks clarity in some places, and notably in the methodology. Figure 1 contains variables and notations that are not consistent with itself and the text. For example, the Bellman target generator should have subscript θˉ\bar{\theta} but uses ψ\psi instead, the intrinsic rewards use superscripts II but this is not used in text, the risk-sensitive control section uses r^\hat{r} but the remainder of the text uses r~\tilde{r}. Several variables in the tex are left undefined, such as PtargetP_{target} in (14), PXP_X and PZP_Z are not formally defined (I assume they are Gaussian?), the discriminator DϕD_\phi, etc. There is also no description of implementation details, or hyperparameters used, which would make reproduction of this work difficult.
  4. Novelty is not clear. The proposed method is very similar to [R1] which the authors cite but do not discuss in detail nor compare against.

[R1] Li, L., & Faisal, A. A. (2021, May). Bayesian distributional policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 10, pp. 8429-8437).

Questions for authors

Please see weaknesses.

  1. Can the authors compare against or explain the lack of comparison against alternative exploration methods such as RND, and curiosity driven exploration? Similarly, can the authors baseline against alternative risk-sensitive approaches such as QR-DQN/IQN optimizing for CVaR/VaR, or more modern approaches?
  2. Can the authors position their method against the Bayesian distributional policy gradients proposed by Li, L., & Faisal, A. A. (2021, May).
  3. The authors argue for improved sample efficiency but evaluate on rather low-dimension and relatively simple environments for a limited number of episodes. Can the authors provide experiments on more complicated settings common in the risk-sensitive RL literature, such as stochastic mujoco, to demonstrate the efficiency gain of their approach?

Minor:

  1. L043: I believe this sentence should be clarified. Distributional RL, inherently, does not go beyond the the expectation objective and the cited papers still optimize for expectation. The authors can cite IQN, but even then the risk-sensitive optimization is heuristic. There are better works to cite for principled risk sensitive learning.
  2. Please fix the citations with \pcite instead of \cite.
  3. L165: Quantile Regression is how the quantile positions are learned, it is not the parameterization which is simply via quantile masses.
2 comments
AuthorsOpenReview ↗

Part 1 of the response

We thank the reviewer for the detailed feedback. We substantially revised the manuscript to strengthen the baseline coverage, ablation analysis, methodology presentation, and positioning relative to BDPG.

Baselines and ablations. We agree that the original evaluation did not sufficiently isolate the contribution of posterior-guided exploration. The revised paper now includes multiple risk-sensitive baselines: QR-D4PG, IQN [1], and RACER optimize CVaR-based objectives, while SAC remains the risk-neutral baseline. We also added QR-D4PG+RND, which uses the same quantile-based risk-sensitive D4PG backbone but replaces our posterior-shift signal with Random Network Distillation (RND) [2] prediction error. Appendix C.4 reports ablations in which GAN-D4PG keeps the same generator, encoder, discriminator, quantile critic, actor, and training setup but sets ηt=0\eta_t=0. We additionally report sensitivity to η0∈0.075,0.15\eta_0\in{0.075,0.15}. Across hedging, F1TENTH, and stochastic MuJoCo, these results separate the effect of the exploration signal from the underlying risk-sensitive learner and from a simpler prediction-error bonus.

Experimental interpretation. We revised the discussion to avoid claiming pointwise or uniform dominance. In option hedging, GAN-D4PG reaches strong CVaR earlier than the matched risk-sensitive alternatives and achieves the strongest final CVaR: at σ=0.3\sigma=0.3, −7.34-7.34 versus −7.57-7.57, −7.98-7.98, and −8.00-8.00 for QR-D4PG, IQN, and QR-D4PG+RND; at σ=0.7\sigma=0.7, −16.44-16.44 versus −17.99-17.99, −103.12-103.12, and −17.44-17.44, respectively. For F1TENTH, the training-time advantage is more modest and nonuniform. On Flat, QR-D4PG and IQN outperform GAN-D4PG at some intermediate checkpoints, and IQN has slightly higher final CVaR 802.8802.8 vs. 787.7787.7. On Bumpy, GAN-D4PG becomes increasingly favorable later in training and achieves the strongest final CVaR 639.1639.1. We now report these crossings explicitly. The broader conclusion is that GAN-D4PG reaches strong lower-tail performance with competitive sample efficiency while maintaining strong final CVaR across the evaluated domains.

Methodology clarity and reproducibility. We substantially revised Fig. 1 and Sec. 4 so that the generator, encoder, target encoder, discriminator, quantile critic, posterior-shift signal, and exploration-augmented reward use consistent notation throughout. We now explicitly define the latent prior, discriminator inputs, target networks, Bellman-target construction, and the distinction between the generative return model and the quantile critic. We also added the original D4PG citation [3] and expanded Appendix C with network architectures, learning rates, batch sizes, (n)-step returns, target-update schedules, exploration parameters, risk levels, and baseline-specific implementation details.

Novelty and relation to BDPG. We agree that BDPG [4] and GAN-DRL share important ingredients, including generative return modeling, and posterior-based curiosity. In BDPG, samples from the generative return model are used to construct the return and advantage estimates that drive the policy-gradient update. In GAN-DRL, the generative return model instead provides the posterior-guided exploration signal. The return distribution used directly for policy optimization, Zφ(s,a)Z_\varphi(s,a), is represented by an independently parameterized quantile critic trained with the exploration-augmented reward. The two return representations therefore serve distinct roles: generative modeling supports exploration, while the quantile representation supports explicit lower-tail control. Appendix C.5 evaluates this design choice directly. For CVaR0.95\mathrm{CVaR}_{0.95}, we compare the quantile critic with an empirical CVaR estimate from 99 generator samples, matching the critic's 99 quantiles, against a reference computed from 5,000 Bellman-return samples. The quantile critic achieves lower absolute CVaR estimation error across the evaluated checkpoints, while repeated 99-sample generator estimates exhibit Monte Carlo variability. Thus, VaR/CVaR can be computed from learned lower-tail quantiles rather than estimated empirically from a finite set of generated returns.

Additional exploration/risk-sensitive baselines. As requested, we added QR-D4PG+RND [2] as a simpler curiosity-driven baseline and continuous-action IQN [1] with the same CVaR actor objective; RACER remains as a recent risk-sensitive actor-critic baseline. GAN-D4PG achieves stronger final CVaR than RND in both hedging settings, both F1TENTH tasks, and stochastic MuJoCo. Relative to IQN, GAN-D4PG is stronger in both hedging settings, on F1TENTH-Bumpy, and on MuJoCo, while IQN is slightly better on F1TENTH-Flat.

AuthorsOpenReview ↗

Part 2 of the response

Stochastic MuJoCo evaluation. We agree that an additional continuous-control benchmark is useful, and we added a stochastic MuJoCo InvertedPendulum-v4 experiment with state-dependent reward uncertainty. We also note that state dimensionality alone does not capture the sources of risk in the original tasks: option hedging contains stochastic arrivals, overlapping positions, transaction costs, and volatility-dependent tail risk, while F1TENTH is a 12-dimensional continuous-control task with nonlinear vehicle dynamics and rollover failures.

The MuJoCo experiment provides a complementary source of risk: Gaussian reward perturbations are applied only when the cart enters a designated risky region. All methods are trained for 500,000 environment interactions across six seeds under the same evaluation protocol. Figure 4 shows that GAN-D4PG reaches strong lower-tail performance early and remains among the best methods during training; SAC is better at several intermediate checkpoints, but GAN-D4PG recovers and achieves the highest final CVaR. In the final evaluation, GAN-D4PG achieves the strongest mean return and CVaR0.90\mathrm{CVaR}_{0.90} $498.0\pm4.8$ and 468.0±44.4468.0\pm44.4 and the lowest fall rate 8.0±2.18.0 \pm 2.1%. The corresponding final CVaR values for QR-D4PG, IQN, QR-D4PG+RND, RACER, and SAC are 194.5,102.5,272.2,25.2194.5, 102.5, 272.2, 25.2, and 371.1371.1, respectively.

Minor: distributional RL versus risk-sensitive optimization. We agree that distributional RL by itself does not imply a risk-sensitive objective. The revised text now distinguishes learning the return distribution from optimizing an explicit risk functional. GAN-D4PG, QR-D4PG, IQN, and RACER use CVaR-based policy objectives, whereas SAC optimizes expected return.

Minor: citation formatting. We reviewed the manuscript and corrected inconsistent citation commands so that parenthetical and textual citations use the appropriate (\citep/\citet) form consistently.

Minor: quantile representation. We agree that quantile regression is the learning procedure, not the parameterization. The revised background now describes the critic as a finite set of equally weighted Dirac masses located at learned quantile values, with those values learned using the quantile-regression loss.

References

[1] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning (ICML), pages 1096–1105, 2018.

[2] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations (ICLR), 2019.

[3] Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.08617, 2018.

[4] Luchen Li and A. Aldo Faisal. Bayesian distributional policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, 35(10):8429–8437, 2021.

Reviewer_upPHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for sample-efficient risk-sensitive control that adds an uncertainty-driven exploration bonus to a distributional actor-critic method. The authors propose a generator/encoder/discriminator architecture, and the KL divergence between the encoder’s posterior for a Bellman-target sample and for a predicted return sample serves as an information gain that is added as a decaying intrinsic reward. A separate distributional critic is then trained with this new reward and used for policy optimization of a risk-sensitive actor. Empirically, they compare their method on European option hedging and autonomous driving environments and report improved tail performance relative to distributional, risk-sensitive, and risk-neutral baselines.

Critical strengths
  • Sample efficiency in risk-sensitive RL is an important and under-studied problem.
  • The paper's overall approach of combining an exploration bonus with a distributional actor-critic method is sound.
Critical weaknesses
  • The notation in Figure 1 has many discrepancies with the notation in the text, which makes the paper difficult to read. Minor: citations are often missing parentheses.
  • D4PG is used as the backbone of the method, yet it is neither described nor cited in the main text.
  • The generator, encoder, and discriminator are used only to calculate the exploration bonus. However, the authors do not show how useful this extra machinery is compared to simpler exploration bonuses such as RND or critic-ensemble disagreement. The only relevant comparison is QR-D4PG, which has no exploration bonus.
  • Comparing GAN-D4PG with QR-D4PG in Figure 3, the proposed method does not seem convincingly superior in sample efficiency or final performance, certainly not enough to justify all of the extra components needed for the exploration bonus.
  • RACER's CVaR in option hedging suggests that the method was not properly tuned for that environment.
  • Strong risk-sensitive baselines, such as an IQN critic with a CVaR actor, are missing from the experiments.
  • An analysis of computational cost, especially compared to simpler approaches, is missing.
  • The network architecture, batch size, and buffer size of the D4PG-based methods differ from those of the RACER and SAC baselines, and no tuning protocol is described for these baselines, which makes it unclear whether the comparison to them is fair.
Questions for authors

Could the authors explain in more detail how their work differs from BDPG (Li & Faisal, 2021), especially beyond state-action conditioning and the CVaR critic? Although BDPG is cited, the two works appear to share many components, and the paper mentions it only briefly.

2 comments
AuthorsOpenReview ↗

Part 1 of the response

We thank the reviewer for the detailed and constructive feedback. We substantially revised the manuscript to improve notation, baseline coverage, reproducibility, and the positioning of the method. In particular, we revised Fig. 1, added the original D4PG citation and implementation details, added IQN and QR-D4PG+RND baselines, added a stochastic MuJoCo experiment, expanded the exploration ablations, and added computational-cost and tail-estimation analyses.

Notation in Fig. 1 and citation formatting. We revised Fig. 1 so that its notation now matches the main text and explicitly separates the four stages: generative return modeling, adversarial posterior inference, posterior-guided exploration, and risk-sensitive control. We also corrected citation formatting throughout the manuscript.

D4PG backbone. We now explicitly cite the original D4PG work [1] when introducing GAN-D4PG in Sec. 5. Appendix C.1 provides the D4PG-based implementation details, including actor/critic architectures, target-network schedule, (n)-step returns, batch size, learning rates, and exploration noise.

Need for the generator/encoder/discriminator versus simpler exploration. We agree that the additional machinery should be justified against a simpler exploration mechanism. Appendix C.4 reports a direct ablation in which GAN-D4PG retains the full generative/inference architecture and disables only the intrinsic reward by setting (ηt=0)(\eta_t=0). We also added QR-D4PG+RND, which keeps the same quantile-based risk-sensitive D4PG backbone but uses Random Network Distillation (RND) [2] as a simpler prediction-error bonus.

GAN-D4PG achieves higher final CVaR than QR-D4PG+RND in both hedging settings −7.34-7.34 vs. −8.00-8.00, −16.44-16.44 vs. −17.44-17.44, on F1TENTH Flat 787.7787.7 vs. 756.9756.9 and Bumpy 639.1639.1 vs. 480.5480.5, and on stochastic MuJoCo 468.0468.0 vs. 272.2272.2. Together with the six-run $\eta_t=0 $ablations, this shows that the observed gains are not reproduced simply by adding a generic prediction-error intrinsic reward. We agree that critic-ensemble disagreement would be a useful complementary comparison, but RND provides a controlled simpler alternative within the same risk-sensitive learner.

Figure 3 / sample efficiency relative to QR-D4PG. We agree that Fig. 3 does not show uniform dominance over QR-D4PG, and we revised the discussion to state this explicitly. On Offroad-Flat, GAN-D4PG improves steadily and reaches strong tail performance within the first half of training, but QR-D4PG and IQN are better at some intermediate checkpoints. On Offroad-Bumpy, GAN-D4PG becomes increasingly favorable in the second half of training and has its largest advantage near the end. We therefore no longer characterize the F1TENTH result as uniformly faster learning. The final held-out evaluation is more favorable to GAN-D4PG relative to QR-D4PG: on Flat, CVaR improves from 681.29681.29 to 787.73787.73 and rollover decreases from 2.80% to 0.40%; on Bumpy, CVaR improves from 571.40571.40 to 639.11639.11 and rollover decreases from 2.80% to 1.60%. We revised the manuscript to distinguish these final lower-tail/safety gains from the more modest and nonuniform training-time advantage on F1TENTH.

RACER on option hedging. The reviewer’s observation prompted us to re-examine the RACER configuration and rerun the comparison. The final configuration is reported explicitly in Appendix C.1: a 151-atom categorical critic over [−250,150][-250,150], a twin ensemble, batch size 128, update-to-data ratio 8, and the same task-specific CVaR objective and interaction budget. After rerunning the comparison, RACER still exhibits substantial variability on hedging, particularly at σ=0.7\sigma=0.7, where CVaR0.95=−247.72±171.94\mathrm{CVaR}_{0.95}=-247.72\pm171.94. We therefore do not rely on RACER alone to establish the advantage of GAN-D4PG; the revised evaluation also includes the more tightly matched QR-D4PG, IQN, and QR-D4PG+RND comparisons.

Missing IQN baseline. We agree and added a continuous-action IQN baseline [3] using the same D4PG-style backbone and actor architecture, with an implicit quantile critic and the same CVaR actor objective. GAN-D4PG obtains stronger final CVaR than IQN in both hedging settings −7.34-7.34 vs. −7.98-7.98 at σ=0.3\sigma=0.3, and −16.44-16.44 vs. −103.12-103.12 at σ=0.7\sigma=0.7, on F1TENTH-Bumpy 639.1639.1 vs. 345.2345.2, and on stochastic MuJoCo 468.0468.0 vs. 102.5102.5. IQN is slightly better on F1TENTH-Flat 802.8802.8 vs. 787.7787.7, which we now state explicitly rather than claiming uniform superiority.

AuthorsOpenReview ↗

Part 2 of the response

Computational cost. We agree that the additional generator, encoder, and discriminator introduce training overhead, and we added a computational-cost analysis in Appendix C.2 covering all evaluated methods and domains. GAN-D4PG requires 0.47±0.020.47\pm0.02 h for the full hedging training budget, compared with 0.16±0.030.16\pm0.03 h for QR-D4PG; the corresponding costs are also reported for IQN, QR-D4PG+RND, RACER, and SAC. This additional computation is learner-side: the generator, encoder, discriminator, and critic are not required for action selection after training, and GAN-D4PG deploys the same actor architecture as the other D4PG-based methods. We therefore view the method as trading additional training computation for improved environment-sample efficiency and lower-tail performance, rather than as a computationally cheaper learner.

Fairness of RACER/SAC comparisons. We clarified the baseline configurations in Appendix C.1. GAN-D4PG, QR-D4PG, IQN, and QR-D4PG+RND provide the most tightly controlled comparisons: they share the same D4PG-style off-policy framework, actor architecture, batch size, target-update schedule, interaction budget, and evaluation protocol, and differ primarily in return representation and/or exploration mechanism. RACER and SAC are broader method-level baselines and retain their method-specific architectures rather than being forced into the D4PG architecture. The revision reports their selected architectures and optimization settings, including batch size, replay capacity, update-to-data ratio, target updates, RACER's categorical support and ensemble size, and task-specific objectives. All methods are evaluated under the same environment-interaction budget and evaluation protocol.

Question: How does GAN-DRL differ from BDPG? We agree that BDPG [4] and GAN-DRL share important ingredients, including adversarial latent inference and generative return modeling. In BDPG, samples from the generative return model are used to construct the return and advantage estimates that drive the policy-gradient update. In GAN-DRL, the generative return model instead provides the posterior-guided exploration signal. The return distribution used directly for policy optimization, Zφ(s,a)Z_\varphi(s,a), is represented by an independently parameterized quantile critic trained with the exploration-augmented reward. Thus, the two representations serve distinct roles: the generative model supports exploration, while the quantile representation directly supports lower-tail VaR/CVaR control.

Appendix C.5 empirically examines this design choice. We compare CVaR0.95\mathrm{CVaR}_{0.95} from the quantile critic with an empirical estimate from 99 generator samples, matching the critic's 99 quantiles, against a reference computed from 5,000 Bellman-return samples. Across the evaluated checkpoints, the quantile critic has lower absolute CVaR estimation error, while repeated 99-sample generator estimates exhibit Monte Carlo variability. This supports computing the risk objective from learned lower-tail quantiles rather than relying on a finite-sample empirical tail estimate from the generative model.

References

[1] Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint arXiv:1804.08617, 2018.

[2] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations (ICLR), 2019.

[3] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning (ICML), pages 1096–1105, 2018.

[4] Luchen Li and A. Aldo Faisal. Bayesian distributional policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, 35(10):8429–8437, 2021.

Reviewer_e2suOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GAN-DRL, a framework for sample-efficient risk-sensitive reinforcement learning. It combines a generative return model, an encoder trained through adversarial inference, and a quantile-based critic used to optimize VaR or CVaR. Its main exploration mechanism compares the latent distributions inferred from a sampled return prediction and its Bellman target. Their KL divergence supplies an intrinsic reward, intended to encourage informative exploration. Instantiated on D4PG, the method is evaluated on option hedging and autonomous driving, where it achieves improved tail performance and faster learning relative to the selected baselines.

Critical strengths

a) Learning policies with good tail performance under limited interaction budgets is important, particularly in financial and safety-sensitive applications. The two experimental domains are well aligned with this motivation.

b) Interesting algorithmic combination. Coupling generative return modeling with exploration and a risk-sensitive distributional critic is novel and a plausible approach to improving learning efficiency.

c) Promising empirical results. The experiments report improved CVaR and fewer vehicle rollovers, alongside learning curves and variability across six runs.

Critical weaknesses

a) The exploration bonus compares latent distributions conditioned on two different return samples, rather than explicitly comparing Bayesian beliefs before and after incorporating an observation. It may remain positive even when the return distribution is learned accurately, because the sampled returns themselves remain random. The authors should clarify this.

b) The connection between the exploration bonus and tail-risk learning is unclear. The bonus does not specifically target uncertainty in the lower tail. The paper does not establish why this exploration mechanism should be particularly effective for VaR or CVaR optimization.

c) The empirical comparisons do not sufficiently isolate the contribution. QR-D4PG and RACER are relevant baselines, but comparisons against simpler prediction-error or ensemble-disagreement bonuses would help justify the additional generative and adversarial machinery. Where is the ablation study?

d) The positioning within risk-sensitive RL is incomplete. Additional discussion and, where appropriate, comparisons with explicit CVaR methods and generative approaches (there are a couple of papers on this at NeurIPS last year) would strengthen the assessment.

Questions for authors
  1. Why should the KL divergence between latent distributions conditioned on a sampled return prediction and its Bellman target measure epistemic information gain?

  2. Can you establish the benefit of the proposed construction over simpler exploration bonuses, such as Bellman prediction error or ensemble disagreement, using the same risk-sensitive learner?

3 comments
AuthorsOpenReview ↗

Part 1 of the response

We thank the reviewer for the careful reading and constructive feedback. We substantially revised the manuscript to clarify the interpretation of the exploration signal and strengthen the empirical evaluation. The revision now (i) describes the KL signal as a Bellman-induced latent posterior shift rather than exact Bayesian information gain, (ii) adds IQN and QR-D4PG+RND baselines, (iii) adds a stochastic MuJoCo experiment, (iv) expands the posterior-guided exploration ablation across all three domains, and (v) adds a tail-estimation diagnostic for the quantile critic.

(a) Interpretation of the posterior-shift signal. We agree with the reviewer that the KL term should not be interpreted as an exact Bayesian belief update. In our construction, gtg_t is a sample from the current return model at (st,at)(s_t,a_t), whereas xtx_t is the corresponding Bellman-target sample that becomes available after observing the transition. We pass both through the same posterior encoder and use ut=DKL!(qψˉ(z∣xt,st,at)∣∣qψˉ(z∣gt,st,at))u_t = D_{\mathrm{KL}}!\left( q_{\bar\psi} (z\mid x_t,s_t,a_t) || q_{\bar\psi} (z\mid g_t,s_t,a_t) \right) to measure how strongly the Bellman target changes the inferred latent return representation. Because stochastic return outcomes can produce nonzero differences even after the return model is well learned, we no longer equate utu_t with exact Bayesian information gain. The revised manuscript consistently describes it as a Bellman-induced latent posterior-shift signal, motivated by posterior-based exploration (Sections 3.3 and 4.4).

(b) Connection to tail-risk learning. We agree that the exploration bonus is not explicitly lower-tail-specific. Its role is to guide data acquisition toward state-action regions where the learned generative return representation changes strongly under new Bellman information. Tail sensitivity enters separately through the independently parameterized quantile critic and the VaR/CVaR actor objective. Thus, the exploration mechanism is distribution-aware rather than explicitly tail-aware. The ablations in Appendix C.4 support this separation. Relative to disabling the exploration bonus, η0=0.15\eta_0=0.15 improves final CVaR from −7.58-7.58 to −7.34-7.34 and from −16.71-16.71 to −16.44-16.44 in the two hedging settings, and from 570.4570.4 to 787.7787.7 and from 498.4498.4 to 639.1639.1 on F1TENTH Flat and Bumpy. On Bumpy, mean return decreases from 893.0893.0 to 879.0879.0 while CVaR improves substantially, showing that the effect is not simply an increase in average return. The six-run MuJoCo ablation shows the same qualitative effect, with both nonzero exploration settings substantially improving mean return and CVaR0.90\mathrm{CVaR}_{0.90} over ηt=0\eta_t=0.

AuthorsOpenReview ↗

Part 2 of the response

(c) Ablations and simpler exploration baselines. We expanded the empirical analysis specifically to address this concern. Appendix C.4 reports six-run ablations across option hedging, F1TENTH, and stochastic MuJoCo. GAN-D4PG retains the same generative return model, encoder, discriminator, quantile critic, actor, and training configuration while disabling only the intrinsic exploration reward by setting ηt=0\eta_t=0.

We also added QR-D4PG+RND, which uses the same quantile-based risk-sensitive D4PG backbone but replaces the proposed posterior-shift signal with Random Network Distillation (RND) [1] prediction error. GAN-D4PG achieves higher final CVaR than QR-D4PG+RND in both hedging settings -7.34 vs. -8.00, and -16.44 vs. -17.44, on F1TENTH Flat 787.7 vs. 756.9 and Bumpy 639.1 vs. 480.5, and on stochastic MuJoCo 468.0 vs. 272.2. Together, the no-exploration ablation and matched RND comparison show that the observed gains are not reproduced simply by adding a generic prediction-error intrinsic reward. We do not claim that this comparison exhausts all possible exploration mechanisms, including critic-ensemble disagreement, but it provides a controlled comparison with a widely used simpler prediction-error baseline using the same risk-sensitive learner.

(d) Positioning within risk-sensitive and generative RL. We expanded the related-work discussion to cover explicit CVaR/risk-sensitive methods and recent generative approaches. The revised paper now discusses QR-based CVaR methods, IQN [2], RACER, BDPG [3], GAN-based distributional RL, and recent generative approaches including Value Diffusion RL [4] and Adversarial Diffusion [5] for robust/risk-sensitive control. Empirically, we added IQN to complement QR-D4PG and RACER. GAN-D4PG achieves the strongest final CVaR in both hedging settings and on F1TENTH-Bumpy, while IQN is slightly better on F1TENTH-Flat. GAN-D4PG also achieves the strongest final CVaR in the added stochastic MuJoCo experiment.

Question: Why should the KL divergence measure epistemic information gain? We agree that it should not be interpreted as exact epistemic information gain. As clarified in (a), the revised manuscript describes it as a Bellman-induced latent posterior shift. Since stochastic return variation can also contribute to the discrepancy, we explicitly avoid equating it with a Bayesian prior-to-posterior information gain.

Question: Can you establish the benefit over simpler exploration bonuses using the same risk-sensitive learner? Yes. QR-D4PG+RND retains the same quantile-based risk-sensitive D4PG learner while replacing the proposed posterior-shift signal with a standard prediction-error intrinsic reward. GAN-D4PG achieves higher final CVaR than QR-D4PG+RND in both hedging settings, both F1TENTH tasks, and stochastic MuJoCo. Together with the ηt=0\eta_t=0 ablations, this provides evidence that the improvement is not reproduced simply by adding a generic prediction-error exploration bonus.

AuthorsOpenReview ↗

References

[1] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations (ICLR), 2019.

[2] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning (ICML), pages 1096–1105, 2018.

[3] Luchen Li and A. Aldo Faisal. Bayesian distributional policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, 35(10):8429–8437, 2021.

[4] Xiaoliang Hu, Fuyun Wang, Tong Zhang, and Zhen Cui. Value diffusion reinforcement learning. In Advances in Neural Information Processing Systems, volume 38, 2025.

[5] Daniele Foffano, Alessio Russo, and Alexandre Proutiere. Adversarial diffusion for robust reinforcement learning. In Advances in Neural Information Processing Systems, volume 38, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.