- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DEVOTE, a model-free RL algorithm for optimistic exploration with deep "epistemic value functions", built on SAC with two exploit/explore actors. Traditional apporaches to deep epistemic value functions might fail for 3 separate reasons: i) uncertainty is badly represented away from the data; ii) uncertainty estimates get "corrupted" when propagated through TD learning; iii) the networks lose plasticity (dead neurons) as exploration target moves. DEVOTE proposes one fix for each failure: i) a ENN with Randomized Fourier Netwrok prior instead of ReLU prior quantifies uncertianty better away from the data; ii) Temporal Uncertainty Difference learning break down the critic into a reward only base critic, a corrector that cancels the prior uncertainty on visited (s,a), and a residual bootstrap critic that quantifies the remaining novelty; iii) application of soft resets to the exploration critic and actor networks to prevent plasticity loss.
Critical strengths
-
The paper presents a very clear diagnosis of the issues with deep uncertainty-driven exploration in model-free RL. It cleanly breaks the problem down into 3 separate failure modes (ie, basically estimation, propagation and optimization), and tests proposed fixes for each of the failure modes in a controlled setting before the full benchmarking. The main novel two fixes are the ENN with RFN prior, that offers a simple way to improve OOD detection (rather than calibration) over ReLU ENN or DeepEns, and the TUD split of the critic
-
The gains are overall well supported by the empirical studies, including the ablations. in particular, the ablations show that each of the 3 fixes matters, and that which one matters most depends on the environment.
Critical weaknesses
- I wonder if the current baselines might be too weak. There are directly comparable methods that are missing from the benchmarking. Algorithnms like BRO and Optimistic Actor Critic are cited and referenced but not included in any of the evaluations. Also, as I understand, only DEVOTE gets soft resets. Althoguht soft resets are a general fix for loss of plasticity. I wonder how the baselines would compare if they also were to get soft resets.
Questions for authors
- Does DEVOTE margin of gains still hold or how much the gains erodes under baselines specifications with matched soft resets? How would it compare with stronger baselines like BRO?
Thank you for the constructive review and recognizing the clarity of our diagnosis and the value of our empirical study.
The baselines cover the main families of exploration signals: undirected, entropy-driven exploration (SAC); posterior sampling over value functions (TS); novelty bonuses from prediction error (RND); and optimism derived from epistemic uncertainty in a learned dynamics model (SOMBRL). All baselines use the same modern SAC backbone as DEVOTE, including RMS normalization and a categorical critic (Appendix D.1), hence the gains we report are not due to a stronger base agent.
To strengthen the evaluation further, we added two baselines: BRO [1] and MaxInfoSAC [2], the latter suggested by Reviewer Pg3j in the new Appendix E.5 (Figure 17). In pure exploration (final coverage after 5M steps, averaged over PointMaze and Cheetah), BRO performs on par with SAC (17.9% vs. 17.3%), indicating the optimism in BRO is insufficient for directed exploration. MaxInfoSAC is the strongest new baseline and matches SOMBRL (38.4% vs. 38.5%), but remains clearly below DEVOTE (50.7%). In complex control (maximum normalized return, averaged over AntMaze and PickAndPlace), both new baselines improve upon SAC (0.42 for both vs. 0.35), but remain below RND (0.49) and clearly below DEVOTE (0.74).
Finally, Optimistic Actor Critic [3] reportedly improves the performance of SAC only slightly in continuous control, and we expect it to underperform with respect to currently evaluated baselines.
We agree that this is an important ablation. Figure 6 already provides partial evidence: all ablations except "no soft resets" keep soft resets and only disable other DEVOTE components, and none matches DEVOTE. Soft resets are thus helpful but not sufficient; the RFN prior and TUD learning each contribute substantially.
To answer the question directly, we added DEVOTE's soft resets to SAC, TS, RND, and SOMBRL (Figure 7, Section 5). Soft resets improve all baselines, most notably RND (22.7% → 26.2%) and SOMBRL (38.5% → 43.2%), but the gap to DEVOTE remains: the strongest baseline with soft resets, SOMBRL, still trails DEVOTE by 7.5 percentage points (43.2% vs. 50.7%). This suggests that soft resets are a generally useful tool against plasticity loss, while DEVOTE's advantage stems from its exploration signal.
—
Thank you for your supportive and constructive review. We hope that our additional results and clarifications addressed your comments on baselines, and are happy to answer any further questions.
[1] Nauman et al. 2024: “Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control”
[2] Sukhija et al. 2024: “MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization”
[3] Ciosek et al. 2019: “Better Exploration with Optimistic Actor-Critic”