- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
VEX is an exploration add-on for DreamerV3. Ensemble of next-latent predictors output candidate successor states. Their disagreement (measure of uncertainty) is either used directly (ρ=0) or priced through the agent's reward, continuation and value heads. The resulting score is added to the actor's normalised advantage, scaled phased out over the last phases of training. Theoreticsl results support interpolation between uncertainty only and value priced allocation according to how reliable the sensitivity estimates are. Few experiments on DMC tasks support the claims.
Critical strengths
The main critical strengths of the paper in my opinion are:
-
VEX is a clean isolated add-on. It leaves the world model objective untouched (and also the other components e.g. the critic target and the replay buffer). The method reduces to Dreamer at and the baselines all share the same Dreamer backbone for a fair unconfounded comparison.
-
The theory partly supports the main claims, and the experiments, while limited, seem to partly improve on Dreamer and baselines.
Critical weaknesses
-
As I understand, the headline mechanism, which is the value based "pricing", is not really supported by the (limited) empirical results.. VEX (\rho = 0) seems to beat VEX (priced) on IQM and AUC in Table1. On Cheetah, the priced variant AUC is lower than Dreamer and RND in Table 6. Are there results that support task-relevant states in the paper? Otherwise the limited empirical evidence a supports more an uncertainty driven advantage based approach rather than value based exploration.
-
Experimentation is limited but mainly statistical power in the experiments is generally quite low. Also, table 2 seems to suggest biggest improvement by the adding of phase out/withdrawal (+71 points and +37, no confidence intervals reported). I wonder whether in the experiments this would pull the authors method back with the baselines.
Questions for authors
I have some clarification questions:
-
Directly linked to W2 above: are the baselines receiving the same phase-out/withdrawal schedule as VEX that seem to bring about the most gains from ablation? Do VEX gains over baseline holds under a matched schedule?
-
About the experiments, given the limited empirical evaluation, what I think it is missing is:
- Value priced exploration empirical validation: is there any evidence that VEX (priced) actually explores more task-relevant states than VEX (\rho = 0)?
- Statistical power and CIs: evaluation seems to be limited to 3 DMC tasks and 3 seeds. Table 2 ablations are missing CIs.