acceptodds
Under review as a conference paper at ICLR 2027

Deep Epistemic Value Functions for Optimistic Exploration

Abstract

Principled exploration in reinforcement learning requires an agent to quantify its epistemic uncertainty and act to resolve it. Uncertainty over the value function provides a natural signal for exploration, yet existing deep approximations remain brittle and perform inconsistently. The central challenge is therefore to scale these ideas robustly. We conduct a systematic empirical study of how epistemic uncertainty is represented, propagated, and optimized in deep epistemic value functions, and uncover distinct failure modes along each of these axes. These findings motivate DEVOTE, a model-free reinforcement learning algorithm that controls how uncertainty generalizes beyond observed data, stabilizes its temporal propagation, and preserves adaptation to the resulting non-stationary exploration objective. Across reward-free exploration and challenging continuous-control tasks, DEVOTE reaches novel states more effectively and achieves higher task return than strong model-free and model-based exploration baselines. These results provide evidence that deep epistemic value functions are a promising path toward scalable, principled exploration.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Exploration, Uncertainty Quantification, Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zEubOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DEVOTE, a model-free RL algorithm for optimistic exploration with deep "epistemic value functions", built on SAC with two exploit/explore actors. Traditional apporaches to deep epistemic value functions might fail for 3 separate reasons: i) uncertainty is badly represented away from the data; ii) uncertainty estimates get "corrupted" when propagated through TD learning; iii) the networks lose plasticity (dead neurons) as exploration target moves. DEVOTE proposes one fix for each failure: i) a ENN with Randomized Fourier Netwrok prior instead of ReLU prior quantifies uncertianty better away from the data; ii) Temporal Uncertainty Difference learning break down the critic into a reward only base critic, a corrector that cancels the prior uncertainty on visited (s,a), and a residual bootstrap critic that quantifies the remaining novelty; iii) application of soft resets to the exploration critic and actor networks to prevent plasticity loss.

Critical strengths
  1. The paper presents a very clear diagnosis of the issues with deep uncertainty-driven exploration in model-free RL. It cleanly breaks the problem down into 3 separate failure modes (ie, basically estimation, propagation and optimization), and tests proposed fixes for each of the failure modes in a controlled setting before the full benchmarking. The main novel two fixes are the ENN with RFN prior, that offers a simple way to improve OOD detection (rather than calibration) over ReLU ENN or DeepEns, and the TUD split of the critic

  2. The gains are overall well supported by the empirical studies, including the ablations. in particular, the ablations show that each of the 3 fixes matters, and that which one matters most depends on the environment.

Critical weaknesses
  1. I wonder if the current baselines might be too weak. There are directly comparable methods that are missing from the benchmarking. Algorithnms like BRO and Optimistic Actor Critic are cited and referenced but not included in any of the evaluations. Also, as I understand, only DEVOTE gets soft resets. Althoguht soft resets are a general fix for loss of plasticity. I wonder how the baselines would compare if they also were to get soft resets.
Questions for authors
  • Does DEVOTE margin of gains still hold or how much the gains erodes under baselines specifications with matched soft resets? How would it compare with stronger baselines like BRO?
1 comment

Thank you for the constructive review and recognizing the clarity of our diagnosis and the value of our empirical study.

W1: Current baselines might be too weak.

The baselines cover the main families of exploration signals: undirected, entropy-driven exploration (SAC); posterior sampling over value functions (TS); novelty bonuses from prediction error (RND); and optimism derived from epistemic uncertainty in a learned dynamics model (SOMBRL). All baselines use the same modern SAC backbone as DEVOTE, including RMS normalization and a categorical critic (Appendix D.1), hence the gains we report are not due to a stronger base agent.

To strengthen the evaluation further, we added two baselines: BRO [1] and MaxInfoSAC [2], the latter suggested by Reviewer Pg3j in the new Appendix E.5 (Figure 17). In pure exploration (final coverage after 5M steps, averaged over PointMaze and Cheetah), BRO performs on par with SAC (17.9% vs. 17.3%), indicating the optimism in BRO is insufficient for directed exploration. MaxInfoSAC is the strongest new baseline and matches SOMBRL (38.4% vs. 38.5%), but remains clearly below DEVOTE (50.7%). In complex control (maximum normalized return, averaged over AntMaze and PickAndPlace), both new baselines improve upon SAC (0.42 for both vs. 0.35), but remain below RND (0.49) and clearly below DEVOTE (0.74).

Finally, Optimistic Actor Critic [3] reportedly improves the performance of SAC only slightly in continuous control, and we expect it to underperform with respect to currently evaluated baselines.

Q1: I wonder how the baselines would compare if they also were to get soft resets.

We agree that this is an important ablation. Figure 6 already provides partial evidence: all ablations except "no soft resets" keep soft resets and only disable other DEVOTE components, and none matches DEVOTE. Soft resets are thus helpful but not sufficient; the RFN prior and TUD learning each contribute substantially.

To answer the question directly, we added DEVOTE's soft resets to SAC, TS, RND, and SOMBRL (Figure 7, Section 5). Soft resets improve all baselines, most notably RND (22.7% → 26.2%) and SOMBRL (38.5% → 43.2%), but the gap to DEVOTE remains: the strongest baseline with soft resets, SOMBRL, still trails DEVOTE by 7.5 percentage points (43.2% vs. 50.7%). This suggests that soft resets are a generally useful tool against plasticity loss, while DEVOTE's advantage stems from its exploration signal.

—

Thank you for your supportive and constructive review. We hope that our additional results and clarifications addressed your comments on baselines, and are happy to answer any further questions.

[1] Nauman et al. 2024: “Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control”

[2] Sukhija et al. 2024: “MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization”

[3] Ciosek et al. 2019: “Better Exploration with Optimistic Actor-Critic”

Reviewer_Pg3jOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper develops a principled approach to exploration in deep reinforcement learning by developing uncertainty over value function. They use epistemic neural network for uncertainty quantification.
  • Based on careful analysis, they develop DEVOTE, a deep model free RL algorithm that uses ENN-based uncertainty estimates and optimism.
  • Each choice in the algorithm is developed to address a specific issue. In limited evaluation, they show that DEVOTE can achieve better state coverage and trade-off exploration-exploitation more efficiently that the selected baselines.
Critical strengths
  • They paper introduces a principled approach towards uncertainty estimates with value function. Each algorithmic decision in motivated to address specific issues investigated in the paper.
  • Despite hundreds of papers, principled exploration in deep RL remains an active topic of interest and the empirical investigations in the paper contribute towards better understanding of deep RL exploration.
Critical weaknesses
  • Weak Evaluation:

    • RFN Length Scale: As shown in Figure 12, the coverage performance of DEVOTE is very sensitive to RFN length scale in 2 of 3 examples. The paper does not report normalized return vs RFN length scale and reports only coverage vs RFN length scale. While good coverage is required for efficient exploration, it does not necessarily imply higher return (as demonstrated in Figure 7 and 13, RND and DEVOTE have similar coverage but not similar returns.)

    • Computational Time: The paper does not discuss compute time. Given that the algorithm introduces multiple actors/critic, this is key to evaluate the algorithm.

    • Limited Environments: The algorithm is evaluated on very few environments. Most of the experiments are in pure exploration setting, hence the better performance of DEVOTE over baselines remain unclear.

    • Baselines: MaxInfoRL (MaxInfoSAC) seems to be a better suited baselines for this work. MEX [1] which incorporates optimism without epistemic uncertainty would also be a good baseline to compare with (Particularly the model-free variant).

[1] Liu, Zhihan, et al. "Maximize to explore: One objective function fusing estimation, planning, and exploration." Advances in Neural Information Processing Systems 36 (2023): 22151-22165.

Questions for authors
  • The DEVOTE algorithm is developed in a principled manner but its empirical evaluation and comparison with other baselines is very limited. The compute time is also not reported. My primary concerns with this paper are related to the weak evaluation described above. I will consider increasing the score if authors provide a expanded empirical evaluation of the algorithm, a full benchmark evaluation on DMC proprio would be a minimal expectation, though there are many more possible continuous control benchmarks.

  • Do the baselines use same configuration across all environments? Can the authors expand on the modification of implemented SOMBRL compared to the method proposed in the SOMBRL paper? Does it use same exploration coefficient (intrinsic reward coefficient in the SOMBRL paper) across all environments?

5 comments

Thank you for your thorough review and for recognizing that our paper contributes towards better understanding in deep reinforcement learning. We address the weaknesses and questions individually.

W1: The paper does not report normalized return vs RFN length scale

Thank you for suggesting this ablation. We have added normalized returns of DEVOTE across RFN length scales on the complex control tasks (Figure 16, Appendix E.4). In AntMaze, the most complex prior ℓ=1\ell=1 never finds the detour around the wall, whereas every ℓ≥2.5\ell\geq 2.5 solves the task. In PickAndPlace, the trend reverses, and the smoothest prior ℓ=7.5\ell=7.5 fails to solve the task reliably. These trends match the coverage of the same runs and mirror the pure-exploration sweep (Figure 14): the best length scale depends on the environment, and the intermediate ℓ=5\ell=5 succeeds in both tasks. Importantly, averaged over both tasks, every length scale still achieves a higher maximum normalized return than all baselines (Figure 17); even the weakest setting, ℓ=1\ell=1, reaches 0.62 compared to 0.49 for the strongest baseline, RND. DEVOTE’s advantage therefore does not hinge on a carefully tuned length scale: the choice of ℓ\ell adapts the prior to the environment, but every RFN prior provides a useful novelty signal.

W1/Q1: The algorithm is evaluated on very few environments. A full benchmark evaluation on DMC proprio would be a minimal expectation

We agree that a broader evaluation is valuable and added six DMC tasks (Appendix E.5): the sparse-reward AcrobotSwingupSparse, CartpoleSwingupSparse, FingerTurnHard, and ReacherHard, as well as the dense-reward HopperHop and CheetahRun. Averaged over all tasks (Figure 19), DEVOTE achieves the highest coverage without rewards throughout training, while all methods converge to a similar return later in training. The per-task results (Figures 20 and 21) show that most of these tasks pose only a limited exploration challenge: even SAC with undirected exploration covers the relevant state space of Acrobot, Cartpole, and Reacher within about 1M steps, and matches the return of the directed exploration methods in most sparse-reward tasks. In the dense-reward Hopper and Cheetah, DEVOTE and TS lose return later in training as they continue to explore, reflecting the inherent exploration–exploitation trade-off. We observe similar trends for RND and SOMBRL, respectively in AcrobotSwingUp and Reacher.

Due to the relative ease of exploration in DMC, our main evaluation focuses on pure exploration (Figure 5) and complex control with uniformative shaped rewards (Figure 8), where exploration is necessary to solve the task.

W1: Baselines: MaxInfoRL (MaxInfoSAC) seems to be a better suited baseline for this work. MEX would also be a good baseline.

Thank you for proposing these baselines. We added a comparison of DEVOTE against MaxInfoSAC [1], as well as BRO [2] as suggested by Reviewer zEub in the new Appendix E.5 (Figure 17). In pure exploration (final coverage after 5M steps, averaged over PointMaze and Cheetah), BRO performs on par with SAC (17.9% vs. 17.3%), indicating the optimism in BRO is insufficient for directed exploration. MaxInfoSAC is the strongest new baseline and matches SOMBRL [3] (38.4% vs. 38.5%), but remains clearly below DEVOTE (50.7%). In complex control (maximum normalized return, averaged over AntMaze and PickAndPlace), both new baselines improve upon SAC (0.42 for both vs. 0.35), but remain below RND (0.49) and clearly below DEVOTE (0.74).

The similar performance of MaxInfoSAC and SOMBRL is the reason why we originally focused on SOMBRL: both derive their intrinsic reward from model-based epistemic uncertainty, and SOMBRL, the more recent method, was shown to be more sample efficient than MaxInfoSAC in some settings (Appendix C of [3]). Finally, MEX [4] reportedly improves the performance of TD3 only slightly in continuous control, so we expect it to underperform the currently evaluated baselines.

W1: The paper does not discuss compute time.

Thank you for pointing this out. We added a discussion of the computational runtime in Appendix D.8. On an RTX 4090, DEVOTE needs about 37 minutes per 1M environment steps, roughly 1.2 times the runtime of SAC (32 minutes). This is comparable to TS and RND (35 minutes each) and faster than SOMBRL (50 minutes). The overhead comes mainly from the ensemble used to implement the ENN. Because the 8 ensemble heads are evaluated in parallel in JAX, the cost grows far less than linearly with the number of heads.

Q2: Do the baselines use the same configuration across environments? How does the SOMBRL implementation differ from the original, and is its intrinsic reward coefficient shared across environments?

Yes, all baselines share the same SAC backbone and use the same configuration across all environments (Appendix D). For SOMBRL, we compute the intrinsic reward as in the original paper [1], from the epistemic uncertainty of a Dreamer-based dynamics model, but optimize it with the same SAC actor-critic as all other baselines instead of on model rollouts, so that the comparison isolates the exploration signal. The intrinsic reward coefficient is not tuned per environment; it is adjusted automatically during training, analogous to SAC's entropy temperature, with the same settings across all environments.


Thank you for your suggestions, which we have directly implemented in order to strengthen our evaluation. Please, let us know if your concerns are addressed and whether you would consider updating your score through this rebuttal. We are happy to answer any standing questions.

[1] Sukhija et al. 2024: "MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization"

[2] Nauman et al. 2024: "Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control"

[3] Sukhija et al. 2025: "SOMBRL: Scalable and Optimistic Model-Based RL"

[4] Liu et al. 2023: "Maximize to Explore: One objective function fusing estimation, planning, and exploration"

Dear Reviewer,

We wanted to follow up before the discussion period ends on November 18. We hope our clarifications have addressed your concerns, and we would be grateful if you would consider reconsidering your evaluation in light of them. Please let us know if you have any remaining questions. Thank you again for taking the time to carefully review our paper and for your helpful suggestions!

Reviewer_dmgTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that deep epistemic value estimation methods like ensemble and epistemic neural networks(ENNs) fail at estimating uncertainty on out of distribution, in addition using TD learning destabilizing the training of the RL agent by telescoping prior variance. The paper proposes random fourier feature (RFN) prior as a function space prior for epistemic neural networks. The paper claims to stabilize agent learning by decoupling task-value learning from uncertainty propagation (which they call TUD update). Finally, they use soft resets for inducing plasticity. The resulting method DEVOTE shows improved coverage rate on three pure-exploration tasks and two sparse-reward tasks compared to RND, SOMBRL and other ensemble methods.

Critical strengths
  1. The paper analyzes the quality of uncertainty estimation of ensemble and Epistemic neural network and compares different priors with a synthetic toy example which is informative about what exactly these priors assume about the underlying function and how that shapes the uncertainty away from the data.. It clearly shows the Randomized fourier prior (RFN) clearly increase the uncertainty estimate within data points compared to ensemble and MLP prior ENNs.

  2. TUD decomposition is simple and the combination of the ideas does improve coverage on exploration tasks.

Critical weaknesses
  1. Most component ideas have been proposed in literature before. RND[1] proposed prior-corrector residual similar to TUD and the bellman propagation of uncertainty proposed in [2]. Soft-resets were proposed in [3].

  2. The RFN's uncertainty profile is not clearly desirable, and the comparison is not fair. In Fig 2, the baselines (ensemble, ENN with MLP prior) rank uncertainty as in-data < in-between < extrapolation, whereas the RFN saturates and treats gaps between clusters like regions far from the data, making interior states as novel as the frontier. Table 5 tests only in-between splits. Moreover, ℓ (equivalent to input rescaling) is tuned per environment, the MLP prior gets no matching tuning, and the toy uses the RFN's own kernel.

  3. Evaluation is too narrow. Its not clear from the results that using RFN prior over MLP prior would always help exploration across environments because only three environments have been used for evaluation in terms of coverage and the baselines don't get the same treatment as DEVOTE like soft resets.

[1] Y. Burda, H. Edwards, A. Storkey, O. Klimov. Exploration by Random Network Distillation. ICLR 2019.

[2] B. O'Donoghue, I. Osband, R. Munos, V. Mnih. The Uncertainty Bellman Equation and Exploration. ICML 2018.

[3] P. D'Oro, M. Schwarzer, E. Nikishin, P.-L. Bacon, M. G. Bellemare, A. Courville. Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier. ICLR 2023.

Questions for authors
  1. How does DEVOTE compare with stronger/enhanced baselines like RND with and RFN target, soft resets, ENN with MLP prior with scaling?

  2. Evaluating RFN prior wrt MLP priors along with other baselines on harder exploration tasks like DMC sparse-reward tasks.

  3. RFN prior uses tuned ℓ, we should compare it with a MLP prior whose input scale (or first-layer weight/bias scale) and output scale are tuned per environment with the same budget.

  4. An analysis of using different activation functions for the prior would also be informative.

3 comments

Thank you for your extensive review of our paper. We are happy to address each comment and answer each question.

W1: Most component ideas have been proposed in literature before.

We view our primary contributions as understanding of empirical failure modes of exploration algorithms, and finding simple ideas to address them. Thus, when existing techniques are effective (e.g. in the case of soft resets), they are also implemented by DEVOTE. However, the majority of components are, to the best of our knowledge, novel: RFN priors for ENNs were not previously explored, and the TUD updates were developed independently. The latter can be connected to RND, as we describe in appendix A.3, but provides a more grounded understanding of the objective as TD learning with a specific ENN parameterization.

W2: The RFN's uncertainty profile is not clearly desirable [...] RFN saturates and treats gaps between clusters like regions far from the data.

We generally agree: no uncertainty profile is desirable a priori. Following your example, deciding whether unvisited interior states or the frontier should be prioritized is likely to be a domain-dependent decision. We however believe that both choices are reasonable, and that ultimately downstream validation is what determines desirability. For this reason, we validate the uncertainty profile induced by RFN priors by measuring coverage and success rates in Figure 5 and 6. Results suggest that DEVOTE can drive deep exploration, and we can thus conclude that, while potentially not always desirable, the uncertainty profile is generally meaningful.

W2/Q3/Q4: RFN prior uses tuned ℓ, we should compare it with a MLP prior whose input scale (or first-layer weight/bias scale) and output scale are tuned per environment with the same budget. An analysis of using different activation functions for the prior would also be informative.

Thank you for raising the importance of tuning the MLP prior. We agree that this is an important ablation to confirm the ineffectiveness of this prior, and have added an ablation in which DEVOTE’s RFN prior is replaced by an MLP prior (DEVOTE-MLP) and tuned per environment over its activation (ReLU, Sigmoid, Tanh) and input scale (0.5, 1, 2.5). These nine configurations exceed the tuning budget of the RFN length scale, which we vary over four values. Averaged over PointMaze and Cheetah, all MLP configurations reach between 28.5% and 32.6% final coverage, well below DEVOTE with the RFN prior (50.7%) (Figures 17 and 18, Appendix E.5). The choice of activation has little effect (32.0–32.6% for the best input scale per activation), and neither does the input scale (33.4–34.2% for the best activation per input scale). In contrast, the RFN length scale changes exploration substantially (Figure 14), as it directly sets how quickly prior functions decorrelate between states. The RFN prior therefore provides both a stronger novelty signal and a more interpretable way to adapt the method to an environment.

W3/Q2: Evaluation is too narrow.

We agree that a broader evaluation is valuable and added six DMC tasks (Appendix E.5): the sparse-reward AcrobotSwingupSparse, CartpoleSwingupSparse, FingerTurnHard, and ReacherHard, as well as the dense-reward HopperHop and CheetahRun. Averaged over all tasks (Figure 19), DEVOTE achieves the highest coverage without rewards throughout training, while all methods converge to a similar return later in training. The per-task results (Figures 20 and 21) show that most of these tasks pose only a limited exploration challenge: even SAC with undirected exploration covers the relevant state space of Acrobot, Cartpole, and Reacher within about 1M steps, and matches the return of the directed exploration methods in most sparse-reward tasks. In the dense-reward Hopper and Cheetah, DEVOTE and TS lose return later in training as they continue to explore, reflecting the inherent exploration–exploitation trade-off. We observe similar trends for RND and SOMBRL, respectively in AcrobotSwingUp and Reacher. Due to the relative ease of exploration in DMC, our main evaluation focuses on pure exploration (Figure 5) and complex control with uniformative shaped rewards (Figure 8), where exploration is necessary to solve the task.

Q1: How does DEVOTE compare with stronger/enhanced baselines like RND with and RFN target, soft resets?

DEVOTE’s local novelty signal is an RND-style prediction error: the corrector cc is trained to cancel a fixed random prior gg, and the residual ∣c+g∣|c+g| is large wherever the predictor has not yet been fit. RND with an RFN target therefore corresponds to DEVOTE without soft resets, which is part of our component ablation (Figure 6, “–Resets”). Replacing RND’s MLP target with the RFN improves final coverage from 22.7% to 32.9% (averaged over PointMaze and Cheetah), mostly in the higher-dimensional Cheetah (36.4% → 54.2%), where a well-designed prior is essential for a meaningful novelty signal. In PointMaze, the gain is small (9.0% → 11.6%): the spiral continually shifts the exploration frontier, and without resets the exploration policy loses the plasticity to follow it. The RFN target is thus an important ingredient, but it does not by itself explain DEVOTE’s performance (50.7%). Conversely, adding DEVOTE’s soft resets to the baselines (Figure 7) improves all of them, most notably RND (22.7% → 26.2%) and SOMBRL (38.5% → 43.2%). Even so, the strongest baseline with soft resets trails DEVOTE by 7.5 percentage points.

Taken together, neither enhancement alone closes the gap: RND gains 3.5 points from soft resets and 10.2 points from the RFN target, while combining both in DEVOTE yields 50.7%, 28 points above RND. The RFN prior provides a meaningful novelty signal, and soft resets keep the exploration policy plastic enough to follow it as the frontier moves, which is why the two components are most effective in combination.


We would like to thank the reviewer for raising precise and actionable questions. Following these suggestions, we were able to ablate the MLP prior, further reinforcing the effectiveness of the proposed RFN prior, and to expand our evaluation to five new environments. We hope that these changes, coupled with our clarifications, addressed remaining concerns, and support an update to the score. In any case, we remain available for further discussion through the rebuttal phase.

Dear Reviewer,

We wanted to follow up before the discussion period ends on November 18. We hope our clarifications have addressed your concerns, and we would be grateful if you would consider reconsidering your evaluation in light of them. Please let us know if you have any remaining questions. Thank you again for taking the time to carefully review our paper and for your helpful suggestions!

Comments on the paper

3 comments
~Haque_M_Ishfaq1OpenReview ↗

Missing related works on Thompson Sampling/Randomized Exploration in RL

Hi Authors,

We did a series of work on Thompson Sampling which we think would be relevant. I am listing the papers below. We would really appreciate if you could cite them and consider using them as baseline (the codes for all are open-sourced), especially in Deep Reinforcement Learning subsection of your Related Works section.

  1. Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo (ICLR 2024): In this work, we propose a very simple way to implement approximate Thompson sampling using Langevin Monte Carlo. We provide regret bound under linear MDP assumption and also propose LMC-DQN algorithm for deep RL that performs much better than bootstrapped DQN and other variants of posterior sampling for deep RL that you discuss in your paper.

  2. More Efficient Randomized Exploration for Reinforcement Learning via Approximate Sampling (RLC 2024): This work extends the previous paper's proof to general function approximation setting by combining Feel-Good Thompson sampling (Zhang et al. 2021) with Langevin Monte Carlo and proposes deep RL algorithms based on various sampling algorithms.

  3. Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning (ICLR 2025): This work extends the previous two papers to continuous control setting.

Thanks for the suggestion. While our focus is on representing, propagating, and learning the value posterior, sampling from it is also an important problem, which we now mention among related works.

Dear Area Chair,

We summarize the discussion period below.

The reviewers found the paper clear and easy to follow (all reviewers) and valued its diagnosis of failure modes in deep uncertainty-driven exploration, with each algorithmic choice motivated by a specific issue (Pg3j, zEub). They also found the analysis of priors informative (dmgT) and the gains well supported by ablations (zEub).

We address the main concerns below together with further changes to the paper.

Scope of the evaluation (dmgT, Pg3j, zEub)

Reviewers asked for more environments and stronger baselines. Our main evaluation deliberately targets settings where exploration is the bottleneck: three pure-exploration tasks and two complex control tasks with uninformative shaped rewards. We extended it in two ways. First, as requested by dmgT and Pg3j, we added six DMC tasks: four sparse-reward and two dense-reward (App. E.5). DEVOTE achieves the highest coverage and the highest average final normalized return (0.594, vs. 0.565 for RND, the strongest baseline, and 0.510 for SAC). The margins are small, because most of these tasks pose only a limited exploration challenge. This supports our focus on harder exploration problems in the main evaluation. Second, we added MaxInfoSAC (Pg3j) and BRO (zEub) (App. E.5). Both remain clearly below DEVOTE in pure exploration (38.4% and 17.9% vs. 50.7% final coverage) and in complex control (0.42 for both vs. 0.74 maximum normalized return).

Fairness of the comparison (dmgT, zEub).

Reviewers asked whether DEVOTE’s gains come from more tuning or from components the baselines did not receive. To test the prior, we replaced the RFN prior with an MLP prior and tuned it per environment over activation and input scale. These nine configurations are more than the four values we tried for ℓ\ell, yet they reach only 28.5–32.6% final coverage, compared with 50.7% for DEVOTE (App. E.5, Fig. 17, 18). To test soft resets, we added them to all baselines (Fig. 7). This improves every baseline, but the strongest, SOMBRL, still trails DEVOTE by 7.5 points (43.2% vs. 50.7%). RND with an RFN target and no resets reaches only 32.9%, so DEVOTE’s advantage comes from combining the RFN prior with soft resets, not from either one alone.

Novelty (dmgT).

Reviewer dmgT noted that some components relate to prior work. Our main contributions are a systematic empirical analysis of the failure modes of deep exploration, simple fixes that address each one, and DEVOTE, an exploration method that combines these fixes and performs well across pure-exploration and complex control tasks. The RFN prior for ENNs and the TUD update are, to our knowledge, new (also noted by zEub), and we reuse existing techniques such as soft resets where they work.

Additional changes:

  • We added a discussion of runtime (App. D.8) showing that DEVOTE takes about 1.2 times the runtime of SAC, on par with TS and RND and faster than SOMBRL (Pg3j).
  • We added normalized returns across RFN length scales (App. E.4): every ℓ\ell outperforms all baselines, so DEVOTE’s advantage does not depend on a carefully tuned length scale (Pg3j).

We believe these additions substantially strengthen the evaluation, and we thank all reviewers for their detailed and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.