- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper makes model-based online RL more efficient by directing exploration toward states that are critical for the downstream reward and states that polarize the return (significantly make the return more positive or negative). They measure criticality and polarity by training a transformer that predicts a trajectory return using the latent variables; the attention of the transformer to different tokens is a measure of how critical that state is. They then train a model that, for any state, predicts its criticality times polarity and use that model as a weight for determining how much exploration is needed in states. In those states that are more critical and polar, more exploration noise is added. They evaluate their method across navigation and manipulation tasks.
Critical strengths
The paper uses an interesting idea to evaluate how critical a certain state is for determining reward. The use of transformer attention to measure this metric is novel and interesting. The paper is relatively well-written, although the writing could still be more simplified. The paper systematically studies whether states with estimated high criticality are actually critical or not, and this analysis is interesting and shows the faithfulness and rigor of the authors.
Critical weaknesses
The emperical experiments are relatively limited. The original Dreamer paper showcases more challenging tasks such as Minecraft, but the paper only evaluates its method on relatively low-dimensional tasks.
The architecture seems overly complicated and requires training two new models; I am not convinced that this is the best way to get return-driven exploration. For instance, how does this method compare to training an ensemble of Q functions and only exploring in those states where we have more uncertainty about the Q by measuring disagreement across ensembles? Also, it seems that in order to have a good estimate of how critical a state is, one should have already seen that state many times and have a good estimate of return for trajectories containing that state, and only then attempt to explore that state more; it is not clear if this method will explore the frontier of exploration or not.
Questions for authors
Can you scale your method to more high-dimensional tasks? For instance, Minecraft?
Can you justify whether this is the best way to do return-focused exploration? What are some simpler approaches, and why won't those methods work?
5 comments
Response to Reviewer aj51 (W2)
Response to Weakness 2: Architectural Complexity, Uncertainty-Based Exploration, and Exploration Frontiers
Thank you for these insightful comments. We clarify the necessity of our additional modules, their distinction from uncertainty-based exploration, and how FocusDreamer extends exploration beyond previously collected experience.
(1) Architectural complexity and the necessity of two predictors. Our design addresses a fundamental distinction between learning where exploration matters from completed outcomes and making exploration decisions online. Under limited interaction budgets, the key is not simply collecting more data, but identifying decision states where additional experience is most valuable for task improvement. Accordingly, the trajectory-level return predictor learns state relevance from completed real trajectories and their empirical episodic returns, leveraging long-range temporal context to identify consequential states. Since these retrospective signals are unavailable during ongoing interaction, a lightweight online head distills them into state-conditioned predictions for real-time exploration. These two modules are complementary components of a unified return-relevance mechanism, rather than additional world models or critics. They preserve DreamerV3's original optimization objectives while introducing only 5.4% more parameters and 12.8% additional training time (Appendix G), providing a practical trade-off for task-directed exploration.
(2) Comparison with Q-Ensemble exploration. We agree that Q-Ensemble is a meaningful return-aware alternative. However, whether disagreement arises from uncertain action-value estimates or world-model predictions, it often reflects insufficient data coverage or distribution shifts, rather than the actual importance of these regions to task completion. Uncertainty-driven exploration prioritizes poorly understood regions because they are uncertain, but reducing such uncertainty does not necessarily improve the policy. For a world model trained from scratch under limited interaction budgets, broadly collecting data to cover unfamiliar regions can be prohibitively expensive and task-inefficient. In contrast, FocusDreamer prioritizes exploration according to return relevance rather than uncertainty, directing additional experience toward decision states that matter for task outcomes. We have additionally included Q-Ensemble Disagreement in the state-selection comparison in Appendix E (Table 7). Its results are comparable to world-model Ensemble Uncertainty (return: 628.3 vs. 624.7; Spearman: 0.30 vs. 0.28), while Criticality achieves substantially better results (748.0; 0.72).
(3) Generalization to unseen states and exploration frontiers. FocusDreamer does not require repeated visits to individual states to estimate their Criticality. Instead, the return predictor learns a shared relevance function from completed trajectories, which is distilled into an online head to predict relevance for newly encountered states. Appendix C supports this generalization, reporting 91.5% polarity sign accuracy and 0.82 Spearman correlation on held-out trajectories. Importantly, online exploration primarily occurs within the evolving state distribution induced by the current policy. Although stochastic actions and policy updates introduce distribution shifts, the online head can leverage previously learned representations to identify potentially critical states without requiring their exact prior visitation. FocusDreamer then applies bounded local action perturbations to explore underrepresented behavioral alternatives, while continually updating both predictors using newly completed trajectories.
More fundamentally, we argue that exhaustively exploring and modeling the entire state–action space is prohibitively expensive for a world model trained from scratch under limited interaction budgets, and is not necessarily required for effective task learning. Rather than broadly exploring uncertain regions, FocusDreamer adopts a task-directed strategy: identifying return-relevant decision states within the currently supported distribution and using them as starting points to discover previously underexplored behavioral alternatives. Although relevance estimates may become less reliable far beyond collected experience, our method does not require identifying every unknown critical state in advance. By retaining standard Dreamer exploration and incorporating newly collected experience, FocusDreamer progressively expands task-relevant behavioral coverage and improves the world model where additional data is most valuable for policy optimization.
Response to Reviewer aj51 (Q2)
Response to Question 2: Return-Focused Exploration and Alternative Approaches
(1) Why is our approach appropriate for return-focused exploration? We do not claim universal optimality. Instead, FocusDreamer addresses a specific limitation of world-model RL: potentially better behavioral alternatives may remain insufficiently explored, causing their real return advantages to be underestimated in imagination. Therefore, our goal is to identify decision states with substantial real local improvement opportunities and prioritize additional exploration around them. Importantly, Table 3(a) directly validates this objective through real-environment interventions, showing that local improvement opportunities increase from 0.028 to 0.108 across Criticality quartiles. Furthermore, Table 2 demonstrates that relevance-guided exploration improves average return from 453.3 under random allocation to 593.3, supporting its practical benefit for policy learning.
(2) What simpler alternatives exist, and why may they be insufficient? Directly prioritizing high-return trajectories or high-value states tends to reinforce already successful behaviors without necessarily discovering better alternatives. TD-error prioritization and Q-Ensemble disagreement focus on value-estimation errors or uncertainty, which need not correspond to actual improvement opportunities. Simpler return-based signals, including RUDDER/RRD contributions, prefix return changes, and last-layer attention, capture outcome attribution or predictive progression but may not reliably identify states where alternative actions offer substantial gains. The central limitation is that these signals do not necessarily distinguish where the model is uncertain or return-predictive from where additional exploration is genuinely valuable for improving the policy.
(3) What evidence supports our design? Appendix E (Table 7) evaluates alternative state-selection signals under matched learning budgets while keeping the allocation pipeline unchanged. Ensemble Uncertainty, Q-Ensemble Disagreement, RUDDER, RRD, Prefix Change, and Last-Layer Attention achieve average returns of 624.7, 628.3, 651.8, 666.4, 681.7, and 707.6, respectively, compared with 748.0 for Criticality. Moreover, Criticality achieves the strongest correlation with real local improvement opportunities (Spearman 0.72) and the highest top-25% opportunity ratio (1.77×). These results demonstrate that, among the evaluated signals, Attention-Rollout Criticality more effectively identifies states with larger real improvement opportunities and achieves better downstream performance, supporting our return-relevance-guided resource allocation.
Response to Reviewer aj51 (W1, Q1)
Response to Weakness 1 and Question 1: Minecraft Evaluation
Thank you for this important suggestion. We agree that evaluating more complex environments, particularly Minecraft, is valuable for assessing the scalability of FocusDreamer.
Our original experiments already cover three complementary pixel-based visual RL benchmarks: eight DMControl tasks involving diverse continuous-control challenges, four Meta-World robotic manipulation tasks, and five Atari100K games involving discrete actions and different temporal dependencies (Appendix B).
To directly address this concern, we have extended FocusDreamer to Minecraft by integrating our return-relevance-guided resource allocation into ResDreamer (ICML 2026), a hierarchical world-model framework. We evaluate two challenging combat tasks, Combat Ghast and Combat Wolf, over 1M environment steps. As shown in Fig. 4 (Sec. 5.2), FocusDreamer achieves final success rates of 93.0% and 84.7%, respectively, compared with 82.2% and 66.1% for ResDreamer (50M×2). Notably, FocusDreamer continues to improve when the original methods exhibit limited progress, demonstrating the effectiveness of our allocation mechanism in more challenging open-world environments.
Importantly, this experiment goes beyond evaluating another environment: it examines whether FocusDreamer's exploration, model-learning, and imagination allocation mechanisms remain beneficial when the underlying world model already incorporates more advanced hierarchical visual reasoning. The observed improvements provide further evidence that return-relevance-guided allocation is complementary to world-model architecture improvements and can generalize to more challenging visual control settings.
We have incorporated these additional results and clarified the experimental scope in the revised manuscript.