acceptodds
Under review as a conference paper at ICLR 2027

Exploring What Matters: Value-Aware Exploration for Dreamer

Abstract

Dreamer improves data efficiency by learning a world model from real experience and training its actor and critic on imagined trajectories. These imagined updates depend on the experience collected, making efficient exploration essential under a limited interaction budget. Predictive disagreement can waste interaction on dynamics with little effect on return. Immature value estimates can also overlook useful experience, limiting task-relevant model learning and policy improvement in imagination. We propose Value-aware Exploration (VEX), which directs exploration in DreamerV3 through value-priced predictive uncertainty. VEX evaluates alternative transition predictions through a shared task readout of reward, continuation, and future value, measuring their disagreement in terms of task return. We derive a Bayesian allocation priority under imperfectly observed task sensitivity, recovering uncertainty-only exploration in the uninformative limit. VEX’s exploration score directly weights imagined actor updates to guide experience collection. Across three continuous-control tasks, both VEX operating points improve aggregate sample efficiency and final return over DreamerV3 and same-backbone exploration baselines. Ablations demonstrate the contributions of its key components.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, World Models, Intrinsic Motivation, Value-Aware Exploration, Latent Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_zAEVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VEX is an exploration add-on for DreamerV3. Ensemble of next-latent predictors output candidate successor states. Their disagreement (measure of uncertainty) is either used directly (ρ=0) or priced through the agent's reward, continuation and value heads. The resulting score is added to the actor's normalised advantage, scaled phased out over the last phases of training. Theoreticsl results support interpolation between uncertainty only and value priced allocation according to how reliable the sensitivity estimates are. Few experiments on DMC tasks support the claims.

Critical strengths

The main critical strengths of the paper in my opinion are:

  1. VEX is a clean isolated add-on. It leaves the world model objective untouched (and also the other components e.g. the critic target and the replay buffer). The method reduces to Dreamer at η=0\eta = 0 and the baselines all share the same Dreamer backbone for a fair unconfounded comparison.

  2. The theory partly supports the main claims, and the experiments, while limited, seem to partly improve on Dreamer and baselines.

Critical weaknesses
  1. As I understand, the headline mechanism, which is the value based "pricing", is not really supported by the (limited) empirical results.. VEX (\rho = 0) seems to beat VEX (priced) on IQM and AUC in Table1. On Cheetah, the priced variant AUC is lower than Dreamer and RND in Table 6. Are there results that support task-relevant states in the paper? Otherwise the limited empirical evidence a supports more an uncertainty driven advantage based approach rather than value based exploration.

  2. Experimentation is limited but mainly statistical power in the experiments is generally quite low. Also, table 2 seems to suggest biggest improvement by the adding of phase out/withdrawal (+71 points and +37, no confidence intervals reported). I wonder whether in the experiments this would pull the authors method back with the baselines.

Questions for authors

I have some clarification questions:

  1. Directly linked to W2 above: are the baselines receiving the same phase-out/withdrawal schedule as VEX that seem to bring about the most gains from ablation? Do VEX gains over baseline holds under a matched schedule?

  2. About the experiments, given the limited empirical evaluation, what I think it is missing is:

    • Value priced exploration empirical validation: is there any evidence that VEX (priced) actually explores more task-relevant states than VEX (\rho = 0)?
    • Statistical power and CIs: evaluation seems to be limited to 3 DMC tasks and 3 seeds. Table 2 ablations are missing CIs.
Reviewer_aPXiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce a novel exploration framework based on pricing predictive transition uncertainty according to its sensitivity to task return, Value-aware Exploration (VEX). The framework is specifically built around DreamerV3 RSSM. The framework is evaluated on three continuous control tasks from the DeepMind Control Suite.

Critical strengths

None

Critical weaknesses

While the paper clearly identifies a problem in the RL exploration literature where exploration strategies are not properly informed by the value of the task, the proposed method is not generally applicable, it's not well evaluated, and the theory is not strongly related to the empirical method.

  • The empirical evaluation is based on only three mostly dense-reward tasks with only three seeds. Meaning that both the exploration hardness and statistical validity have not been properly assessed.
Questions for authors

The core hypothesis of the paper is that Bellman value pricing outperforms task-agnostic uncertainty, however, the uninformative limit VEX (ρ=0\rho = 0) outperforms the priced version of VEX on two of the three tasks and achieves higher aggregate IQM scores. In the ablation you mention "This pattern supports a gradual shift from collecting exploratory experience to refining the task policy, rather than maintaining the same correction strength throughout training". I believe that this is an artifact of the specific tasks you selected as they are dense-reward and cannot really benefit from prolonged exploration. What do you think?

Reviewer_wGoDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work focuses on improving exploration of the DreamerV3 world model in the online model-based reinforcement learning setting. The paper proposes VEX: Value-aware Exploration, which weights the actor's updates in imagination by value-priced uncertainty. The prioritization is achieved by training an ensemble of next-state predictors and utilizing a task readout. The algorithm design is justified with theoretical analysis. Two instances, VEX (priced) and VEX (ρ\rho=0), are evaluated on three DMC tasks with DreamerV3, OWM, disagreement, and RND as baselines.

Critical strengths
  • [S1] In the last couple of years, many papers have shown that prioritization in different components of MBRL algorithms can lead to performance improvement. This paper brings another notion of prioritization, directly to imagined actor updates.
  • [S2] The paper provides theoretical justification for the algorithm design.
  • [S3] All baselines are built on the same DreamerV3 backbone (R2-Dreamer codebase), hence isolating the effect of the proposed modifications from architectural design.
Critical weaknesses
  • [W1] Insufficient Evaluation: The evaluation compares the baselines across 3 DMC environments (Acrobot swing-up, Cheetah run, and Walker run) on 3 seeds each. The number of tasks evaluated is much lower than in similar papers on this topic (SOMBRL and OWM). To clearly establish improved exploration, a significantly broader set of DMC tasks is required. It is unclear why these three tasks were chosen when the benchmark includes tasks such as Cartpole swingup sparse, which are arguably more suitable to evaluate exploration. On the three evaluated environments, the performance improvement is not outside statistical noise in most cases. A significantly higher number of environments with 5 seeds would be needed to establish performance improvement.

  • [W2] To establish better exploration, the authors should consider exploration-centric environments such as sparse DMC environments and Atari games (either the Atari100K benchmark or exploration-centric Atari games including Montezuma's Revenge and Private Eye).

  • [W3] In the limited set of tasks, VEX (ρ\rho=0) performs better than VEX (priced) in 2 out of 3 tasks.

  • [W4] Computational Time Study: The paper does not provide a comparison of wall-clock time for each baseline.

  • [W5] The IQM metric is computed on only 9 runs (tasks x seeds), hence providing an insufficient basis for comparison.

Questions for authors
  • The current evaluation on three DMC tasks is insufficient to establish a robust and clear performance improvement. The authors are requested to provide results on a significantly expanded set of tasks. The current rating (1) of the manuscript is primarily due to insufficient empirical evidence.
  • Is VEX (ρ=0\rho=0) a consistently better algorithm choice than VEX (priced)? In that case, what is the advantage of the priced method? The withdrawal contributes toward a large share of the performance gain. Can the withdrawal be introduced in the RND and disagreement baselines as well?
  • Many works, such as SOMBRL and OWM, have shown that the baseline DreamerV3 lacks efficient exploration, which justifies building on DreamerV3. However, the general principle of VEX is not tied to DreamerV3, and the authors can consider positioning VEX as a general exploration algorithm for online MBRL and then provide results for instances that include DreamerV3, STORM, and MBPO. Have the authors explored the idea beyond DreamerV3? The paper might benefit from being positioned as a general exploration algorithm for model-based RL rather than a DreamerV3 exploration algorithm.
  • Clarifications: As the paper uses R2-Dreamer backbone, can the authors confirm that the DreamerV3 is used with reconstruction and not without (R2-Dreamer)?
Reviewer_SHGROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to improve the exploration of DreamerV3. Generally this work falls into the framework where an exploration bonus is added to the policy learning objective so that exploration is encouraged in data collection. What makes this work different is that the exploration bonus is value aware, in the sense that they don't uniformly care about predictive disagreement on next (latent) states, but weigh these disagreements based on how much they affect predicted rewards and values. The motivation is that an exploration bonus based on raw predictive disagreement may encourage learning task-irrelevant dynamics, making inefficient use of the interaction budget. We should instead prioritise exploration of transitions whose predictive disagreements have greater consequence for estimated task return.

The proposed algorithm, called value-aware exploration, was tested in three DMC environments. Theoretical analysis was done to motivate the value-aware prioritised exploration of transitions. Improvement was observed when compared to baseline DreamerV3 and three exploration baselines on top of DreamerV3.

Critical strengths

The proposed algorithm seems straightforward to integrate into DreamerV3. To compute the exploration bonus, it requires training an ensemble of transition predictors, but it reuses the same heads for reward, continuation and value prediction. This design makes the method potentially more practically usable, which positively affects my overall assesment.

Critical weaknesses

Insufficient discussion to related work:

  • Bayes-adaptive model-based RL provides a principled framework that connects uncertainty on dynamics, rewards and future returns for efficient exploration. I find this line of work closely related to this work but they are not discussed at all. For example, see [1] and [2].
  • Furthermore, [3] takes a pretty similar approach to constructing value-aware exploration bonus by measuring disagreement in ensemble reward/value predictions but is not discussed in this paper.

The experimental results do not convincingly show the benefit of this approach:

  • If I understand correctly, results are only based on three random seeds per task.
  • The method is only tested in three argubly simple DMC tasks. Testing in more tasks will make the results more convincing.
  • The gain observed from the learning curves seems rather marginal.
  • The approach is motivated by identifying a pretty specific failure mode - wasting interactions or samples on transitions that are highly uncertain but do not matter so much to the task. However, the experiments do not directly test this hypothesis in any way. The conclusions are entirely based on the final task performance. This leaves a big gap in the research.

[1] Ghavamzadeh, M., Mannor, S., Pineau, J., and Tamar, A. (2015). Bayesian Reinforcement Learning: A Survey.
[2] Guez, A., Silver, D., and Dayan, P. (2012). Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search.
[3] Zhou, Q., Li, H., and Wang, J. (2020). Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimization.

Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.