acceptodds
Under review as a conference paper at ICLR 2027

FocusDreamer: Task-Directed Model-Based Reinforcement Learning Guided by Return Relevance

Abstract

Model-based reinforcement learning improves interaction efficiency through learned world models and imagination, but model bias can distort return differences among local behaviors and misguide policy optimization. When local alternatives are weakly supported by experience, their predictions may be biased toward familiar latent dynamics, causing real return advantages to shrink or even reverse in imagination. Therefore, under limited interaction budgets, it is more sensible to prioritize states where local behavioral distinctions matter more for return. We propose FocusDreamer, which learns state-level return relevance from completed real trajectories and uses it to focus exploration, model learning, and imagination without altering the original Dreamer objectives. Return relevance is characterized by criticality, which measures its relative strength, and polarity, which reflects the direction of return-predictive progression. Criticality determines where learning resources should be concentrated, while polarity modulates policy-side intervention. Across visual DMControl and Meta-World, FocusDreamer improves performance and sample efficiency, achieving a 53.1% average return improvement over DreamerV3 on DMControl.

Primary area:
reinforcement learning
Keywords:
Visual Reinforcement Learning, Model-Based Reinforcement Learning, World Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aj51OpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper makes model-based online RL more efficient by directing exploration toward states that are critical for the downstream reward and states that polarize the return (significantly make the return more positive or negative). They measure criticality and polarity by training a transformer that predicts a trajectory return using the latent variables; the attention of the transformer to different tokens is a measure of how critical that state is. They then train a model that, for any state, predicts its criticality times polarity and use that model as a weight for determining how much exploration is needed in states. In those states that are more critical and polar, more exploration noise is added. They evaluate their method across navigation and manipulation tasks.

Critical strengths

The paper uses an interesting idea to evaluate how critical a certain state is for determining reward. The use of transformer attention to measure this metric is novel and interesting. The paper is relatively well-written, although the writing could still be more simplified. The paper systematically studies whether states with estimated high criticality are actually critical or not, and this analysis is interesting and shows the faithfulness and rigor of the authors.

Critical weaknesses

The emperical experiments are relatively limited. The original Dreamer paper showcases more challenging tasks such as Minecraft, but the paper only evaluates its method on relatively low-dimensional tasks.

The architecture seems overly complicated and requires training two new models; I am not convinced that this is the best way to get return-driven exploration. For instance, how does this method compare to training an ensemble of Q functions and only exploring in those states where we have more uncertainty about the Q by measuring disagreement across ensembles? Also, it seems that in order to have a good estimate of how critical a state is, one should have already seen that state many times and have a good estimate of return for trajectories containing that state, and only then attempt to explore that state more; it is not clear if this method will explore the frontier of exploration or not.

Questions for authors

Can you scale your method to more high-dimensional tasks? For instance, Minecraft?

Can you justify whether this is the best way to do return-focused exploration? What are some simpler approaches, and why won't those methods work?

5 comments
AuthorsOpenReview ↗

Response to Reviewer aj51 (W1, Q1)

Response to Weakness 1 and Question 1: Minecraft Evaluation

Thank you for this important suggestion. We agree that evaluating more complex environments, particularly Minecraft, is valuable for assessing the scalability of FocusDreamer.

Our original experiments already cover three complementary pixel-based visual RL benchmarks: eight DMControl tasks involving diverse continuous-control challenges, four Meta-World robotic manipulation tasks, and five Atari100K games involving discrete actions and different temporal dependencies (Appendix B).

To directly address this concern, we have extended FocusDreamer to Minecraft by integrating our return-relevance-guided resource allocation into ResDreamer (ICML 2026), a hierarchical world-model framework. We evaluate two challenging combat tasks, Combat Ghast and Combat Wolf, over 1M environment steps. As shown in Fig. 4 (Sec. 5.2), FocusDreamer achieves final success rates of 93.0% and 84.7%, respectively, compared with 82.2% and 66.1% for ResDreamer (50M×2). Notably, FocusDreamer continues to improve when the original methods exhibit limited progress, demonstrating the effectiveness of our allocation mechanism in more challenging open-world environments.

Importantly, this experiment goes beyond evaluating another environment: it examines whether FocusDreamer's exploration, model-learning, and imagination allocation mechanisms remain beneficial when the underlying world model already incorporates more advanced hierarchical visual reasoning. The observed improvements provide further evidence that return-relevance-guided allocation is complementary to world-model architecture improvements and can generalize to more challenging visual control settings.

We have incorporated these additional results and clarified the experimental scope in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer aj51 (W2)

Response to Weakness 2: Architectural Complexity, Uncertainty-Based Exploration, and Exploration Frontiers

Thank you for these insightful comments. We clarify the necessity of our additional modules, their distinction from uncertainty-based exploration, and how FocusDreamer extends exploration beyond previously collected experience.

(1) Architectural complexity and the necessity of two predictors. Our design addresses a fundamental distinction between learning where exploration matters from completed outcomes and making exploration decisions online. Under limited interaction budgets, the key is not simply collecting more data, but identifying decision states where additional experience is most valuable for task improvement. Accordingly, the trajectory-level return predictor learns state relevance from completed real trajectories and their empirical episodic returns, leveraging long-range temporal context to identify consequential states. Since these retrospective signals are unavailable during ongoing interaction, a lightweight online head distills them into state-conditioned predictions for real-time exploration. These two modules are complementary components of a unified return-relevance mechanism, rather than additional world models or critics. They preserve DreamerV3's original optimization objectives while introducing only 5.4% more parameters and 12.8% additional training time (Appendix G), providing a practical trade-off for task-directed exploration.

(2) Comparison with Q-Ensemble exploration. We agree that Q-Ensemble is a meaningful return-aware alternative. However, whether disagreement arises from uncertain action-value estimates or world-model predictions, it often reflects insufficient data coverage or distribution shifts, rather than the actual importance of these regions to task completion. Uncertainty-driven exploration prioritizes poorly understood regions because they are uncertain, but reducing such uncertainty does not necessarily improve the policy. For a world model trained from scratch under limited interaction budgets, broadly collecting data to cover unfamiliar regions can be prohibitively expensive and task-inefficient. In contrast, FocusDreamer prioritizes exploration according to return relevance rather than uncertainty, directing additional experience toward decision states that matter for task outcomes. We have additionally included Q-Ensemble Disagreement in the state-selection comparison in Appendix E (Table 7). Its results are comparable to world-model Ensemble Uncertainty (return: 628.3 vs. 624.7; Spearman: 0.30 vs. 0.28), while Criticality achieves substantially better results (748.0; 0.72).

(3) Generalization to unseen states and exploration frontiers. FocusDreamer does not require repeated visits to individual states to estimate their Criticality. Instead, the return predictor learns a shared relevance function from completed trajectories, which is distilled into an online head to predict relevance for newly encountered states. Appendix C supports this generalization, reporting 91.5% polarity sign accuracy and 0.82 Spearman correlation on held-out trajectories. Importantly, online exploration primarily occurs within the evolving state distribution induced by the current policy. Although stochastic actions and policy updates introduce distribution shifts, the online head can leverage previously learned representations to identify potentially critical states without requiring their exact prior visitation. FocusDreamer then applies bounded local action perturbations to explore underrepresented behavioral alternatives, while continually updating both predictors using newly completed trajectories.

More fundamentally, we argue that exhaustively exploring and modeling the entire state–action space is prohibitively expensive for a world model trained from scratch under limited interaction budgets, and is not necessarily required for effective task learning. Rather than broadly exploring uncertain regions, FocusDreamer adopts a task-directed strategy: identifying return-relevant decision states within the currently supported distribution and using them as starting points to discover previously underexplored behavioral alternatives. Although relevance estimates may become less reliable far beyond collected experience, our method does not require identifying every unknown critical state in advance. By retaining standard Dreamer exploration and incorporating newly collected experience, FocusDreamer progressively expands task-relevant behavioral coverage and improves the world model where additional data is most valuable for policy optimization.

AuthorsOpenReview ↗

Response to Reviewer aj51 (Q2)

Response to Question 2: Return-Focused Exploration and Alternative Approaches

(1) Why is our approach appropriate for return-focused exploration? We do not claim universal optimality. Instead, FocusDreamer addresses a specific limitation of world-model RL: potentially better behavioral alternatives may remain insufficiently explored, causing their real return advantages to be underestimated in imagination. Therefore, our goal is to identify decision states with substantial real local improvement opportunities and prioritize additional exploration around them. Importantly, Table 3(a) directly validates this objective through real-environment interventions, showing that local improvement opportunities increase from 0.028 to 0.108 across Criticality quartiles. Furthermore, Table 2 demonstrates that relevance-guided exploration improves average return from 453.3 under random allocation to 593.3, supporting its practical benefit for policy learning.

(2) What simpler alternatives exist, and why may they be insufficient? Directly prioritizing high-return trajectories or high-value states tends to reinforce already successful behaviors without necessarily discovering better alternatives. TD-error prioritization and Q-Ensemble disagreement focus on value-estimation errors or uncertainty, which need not correspond to actual improvement opportunities. Simpler return-based signals, including RUDDER/RRD contributions, prefix return changes, and last-layer attention, capture outcome attribution or predictive progression but may not reliably identify states where alternative actions offer substantial gains. The central limitation is that these signals do not necessarily distinguish where the model is uncertain or return-predictive from where additional exploration is genuinely valuable for improving the policy.

(3) What evidence supports our design? Appendix E (Table 7) evaluates alternative state-selection signals under matched learning budgets while keeping the allocation pipeline unchanged. Ensemble Uncertainty, Q-Ensemble Disagreement, RUDDER, RRD, Prefix Change, and Last-Layer Attention achieve average returns of 624.7, 628.3, 651.8, 666.4, 681.7, and 707.6, respectively, compared with 748.0 for Criticality. Moreover, Criticality achieves the strongest correlation with real local improvement opportunities (Spearman 0.72) and the highest top-25% opportunity ratio (1.77×). These results demonstrate that, among the evaluated signals, Attention-Rollout Criticality more effectively identifies states with larger real improvement opportunities and achieves better downstream performance, supporting our return-relevance-guided resource allocation.

Reviewer_Fu8XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds on top of Dreamer (MBRL) to mitigate the phenomenon where the model pulls unfamiliar exploratory actions back toward familiar trajectories during imagination. The authors formulate this phenomenon and connect it theoretically to “criticality” and “polarity”. Following the theoretical analysis, the authors train a predictor for the episodic return to extract these quantities. Then, the author suggests leveraging these to focus exploration noise, prioritize replay sampling and prioritize starting imagination states.

Critical strengths
  1. The formulation of the value-gap contraction and the following theoretical analysis on criticality (Section 4) are very informative and explain the motivation for the approach quite well.
  2. The proposed three-stage mechanism is well motivated to improve resource allocation and keeps the optimization objective of Dreamer the same while introducing minimal computational overhead.
  3. The empirical analysis and extended ablation study support the claims made throughout the paper and are informative.
Critical weaknesses
  1. [Critical] Missing Statistical Deviation and Problems in Evaluation Protocol: All tables in the paper report single point estimates without standard deviation and no information about statistical significance or repeated seeds were given. This is especially concerning since the reported returns for the baselines seem quite low compared to the returns in the original works. For example The returns of DreamerV3 in Table 1 are lower than those reported in the original paper. Additionally, the authors evaluated their approach and baselines on subsets of popular benchmarks without explaining the specific task selection rationale.
  2. [Minor] Overstated Theoretical Guarantees: The theoretical results in Section 4 rely on very strong assumptions - a proper discussion on the limitations/realism of such assumption is missing.
  3. [Minor] Figure 1: I found FIgure 1 to not convey the paper’s core concept well, as the frames across (a)-(d) look much alike. Further visualization of the phenomenon, and visualizing for example “critical” states can help clarity.
Questions for authors

No further questions

2 comments
AuthorsOpenReview ↗

Response to Reviewer Fu8X (W1, W2)

Response to Weakness 1: Statistical Reliability and Evaluation Protocol

Thank you for these constructive comments.

Statistical reliability. All reported FocusDreamer results are averaged over five independent random seeds, rather than relying on single runs. The shaded regions in the original Fig. 3 already represent the standard deviations across seeds. To make the statistical reliability more explicit, we have additionally reported mean ± standard deviation in the revised tables.

DreamerV3 baseline and implementation. The original experiments were conducted using the earlier DreamerV3 v1 backbone, with the corresponding DreamerV3 reference results drawn from the 2023 benchmark. We acknowledge that this comparison does not reflect the stronger, subsequently updated DreamerV3 implementation. Therefore, we have additionally evaluated both DreamerV3 v2 and FocusDreamer built upon v2 under the same experimental protocol, and included these results in the revised Appendix H. FocusDreamer continues to achieve substantial performance improvements on the updated backbone. These consistent gains suggest that the issue motivating our method, where insufficiently supported better alternatives may have their return advantages underestimated in imagination, remains relevant in the updated DreamerV3 implementation. The original v1-based ablations are retained to preserve their internally consistent experimental setting.

Task selection. We adopt established visual RL benchmarks and representative tasks widely used in prior studies. Specifically, the eight DMControl tasks cover complementary challenges involving locomotion, dynamic balance, underactuated control, fine-grained manipulation, and goal-directed reaching. The four Meta-World tasks further examine robotic object manipulation, while five Atari100K tasks evaluate discrete-action control with different temporal dependencies. These settings provide complementary evidence across diverse control behaviors, task characteristics, and action spaces. We have clarified this selection rationale in the revised manuscript.

Response to Weakness 2: Theoretical Assumptions and Their Applicability

Thank you for raising this important concern. Our theoretical analysis is established under explicit and well-motivated local sufficient conditions, designed to characterize the relationship between return relevance and local policy improvement.

Specifically, Proposition 1 adopts a local familiarity-attraction model to characterize the tendency of learned world models to bias weakly supported behavioral alternatives toward familiar dynamics. This provides a mathematical explanation for how genuine return advantages can be contracted or reversed in imagination. Theorem 1 further establishes the connection between Criticality and local improvement opportunities through several interpretable assumptions. Local smoothness and bounded Jacobians are standard regularity conditions for analyzing small perturbations. Attention stability and non-cancellation ensure that attention propagation preserves meaningful predictive contributions, while bounded predictor-gradient error explicitly accommodates approximation imperfections in the learned return predictor. Local action controllability further connects return sensitivity to achievable improvements through nearby action choices. Together, these conditions yield an explicit improvement bound that accounts for predictor approximation error.

Importantly, our empirical observations provide direct support for the theoretical analysis. The value-gap contraction illustrated in Fig. 5 is consistent with Proposition 1. Furthermore, controlled real-environment interventions in Sec. 5.4, using 32 local alternatives and 15-step rollouts, show that real improvement opportunities increase from 0.028 to 0.108 across Criticality quartiles, with a Spearman correlation of 0.72. FocusDreamer also reduces the better-alternative miss rate from 56.5% to 24.0% in the highest-Criticality quartile. These findings establish a strong empirical connection between the theoretical characterization of return relevance and its practical effectiveness in preserving behavioral advantages. We have further clarified the motivation and applicability of these assumptions in Sec. 4 and Appendix A.

AuthorsOpenReview ↗

Response to Reviewer Fu8X (W3)

Response to Weakness 3: Figure 1 Visualization

Thank you for this helpful suggestion. We would like to clarify that Fig. 1 illustrates how DreamerV3's imagination can become biased toward familiar behaviors when evaluating insufficiently explored alternatives. In the revised figure, the left and right panels present real-environment outcomes and their corresponding world-model imaginations, respectively. The upper and lower rows represent the reference actions generated by the current DreamerV3 policy (red) and alternative actions that achieve higher returns in the real environment (green), respectively. Starting from the same initial state, the alternative produces a more favorable behavioral outcome in the real environment. However, the world model faithfully reproduces the familiar reference behavior while predicting the better alternative to remain close to the reference.

This discrepancy between real and imagined behavioral differences is precisely the alternative value-gap contraction motivating FocusDreamer. The learned world model may underestimate insufficiently explored action-conditioned alternatives, causing their real return advantages to be compressed or even reversed in imagination. In this example, the alternative's return advantage changes from +0.0739 in the real environment to −0.0259 in imagination. We have revised Fig. 1 with clearer behavioral comparisons, color-coded annotations, and explicit prediction-fidelity indicators to make this phenomenon more intuitive. The revised visualization complements the quantitative counterfactual analyses in Sec. 5.4.

Reviewer_BV1eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FocusDreamer, a model-based RL framework that uses trajectory-derived return relevance to concentrate exploration, model learning and imagination. This concentration is determined by two state-level signals: criticality and polarity, which is theoretically secured to be connected with local improvement opportunity and return progression. Empirical results on DMC and Meta-world show improved performance and sample efficiency.

Critical strengths
  1. The idea of identifying states with larger chance of local improvement is interesting and makes sense. The states more relevant to return prediction, have a larger chance of positive local policy changes. Use this signal to allocate exploration, model-learning and imagination budget is potentially beneficial.

  2. The empirical analysis and ablations are comprehensive and helps understanding the value-gap contraction in DreamerV3.

Critical weaknesses
  1. The method proposed uses two signals to assess the relevance between states and predicted returns. However, they are not derived directly from environment expected returns but some proxy metrics within return predictor. The theoretical proof is derived under strong hypothesis, among which the predictor-gradient fidelity is especially not necessarily true.

  2. The empirical results on DMC lack confidence intervals or error bars. Results in Fig.3 seems to be on par with baselines or in the error bar, especially in Hammer, Stick Pull and Pick Place Wall.

Questions for authors
  1. Related to W1, the criticality identifies the relevance between state and return prediction, does this relate to true environment returns or only return predictors?

  2. Does the warmup process guarantee accessible return predictor within small updates? Its seems that the criticality and polarity is dependent heavily on the return predictor. Is the return predictor trained on the same distribution with data collected with focused exploration online?

  3. How is the two relevance signal related to causality-discovery literature which identifies the causality between state/action and returns?

4 comments
AuthorsOpenReview ↗

Response to Reviewer BV1e (W1, W2, Q1)

Response to Weakness 1: Return Relevance and Theoretical Assumptions

Thank you for raising this important concern. Although Criticality and Polarity are derived from the return predictor, the predictor is supervised by actual episodic returns from completed environment trajectories, rather than manually designed proxy rewards. More importantly, their effectiveness is independently validated through real-environment interventions. In Sec. 5.4, real local improvement opportunities increase from 0.028 to 0.108 across Criticality quartiles, while random selection remains nearly unchanged. Appendix E further reports a Spearman correlation of 0.72 with real improvement opportunities, outperforming ensemble uncertainty (0.28) and last-layer attention (0.57). Polarity provides complementary information: negative-Polarity states exhibit 29% larger improvement opportunities than positive-Polarity states (0.121 vs. 0.094). Removing Polarity also reduces average return from 748.0 to 698.0, while the full FocusDreamer reduces the better-alternative miss rate from 56.5% to 24.0% in the highest-Criticality quartile.

Regarding the theoretical assumptions, predictor-gradient fidelity permits bounded approximation error rather than requiring exact agreement with the true return gradient. Corollary 1 (Appendix A.6) explicitly incorporates this error and establishes a positive local-improvement lower bound when Criticality exceeds a derived threshold. The non-cancellation assumption likewise provides a sufficient condition for relating attention propagation to predictor sensitivity. These are conditional guarantees rather than universally valid properties of learned predictors. We have further clarified their interpretation, applicability, and limitations in the revised Sec. 4 and Appendix A.

Response to Weakness 2: Statistical Reliability and Meta-World Performance

Thank you for highlighting the need for clearer statistical reporting and task-specific comparisons. Our FocusDreamer results were already averaged over five independent random seeds, and the shaded regions in the original Fig. 3 represent cross-seed standard deviations. We have additionally reported mean ± standard deviation in the revised result tables.

Regarding the Meta-World results, Fig. 3 demonstrates different but complementary advantages across tasks. FocusDreamer achieves a clear advantage on Hand Insert. On Stick Pull and Pick Place Wall, it also consistently outperforms the other model-based baselines, although TACO, a model-free method, achieves a higher final success rate on Stick Pull. On Hammer, FocusDreamer and several strong baselines approach the performance ceiling, limiting further separation among the strongest methods. More importantly, FocusDreamer achieves the highest average return of 778.0 among all compared methods on the eight DMControl tasks, ranking first or tied-first on six tasks and second on the remaining two. It outperforms the recent 2026 methods Knowledge Boundary (686.9) by 13.3% and R2-Dreamer (684.0) by 13.7%.

Response to Question 1

Thank you for this important question. Our relevance signals are not merely internal indicators of the return predictor; they are empirically associated with actual return differences in the environment. The return predictor is trained on empirical episodic returns from completed real trajectories, rather than artificial proxy rewards. Criticality identifies states relevant to predicting these outcomes, while Polarity captures the direction of return-predictive progression.

More importantly, we independently validate this relationship through real-environment interventions (Sec. 5.4). By restoring identical simulator states and evaluating 32 local alternative actions over a 15-step horizon, we directly measure real return differences without relying on the predictor. Local improvement opportunities increase from 0.028 to 0.108 across Criticality quartiles (Spearman 0.72), while random selection shows no meaningful separation. Moreover, negative-Polarity states exhibit 29% larger real improvement opportunities than positive-Polarity states (0.121 vs. 0.094). These results confirm that our signals capture behaviorally meaningful return relevance beyond internal predictor statistics. We have clarified this relationship in the revised Sec. 3.2 and Sec. 5.4.

AuthorsOpenReview ↗

Response to Reviewer BV1e (Q2, Q3)

Response to Question 2: Warm-up, Predictor Reliability, and Online Data Distribution

Thank you for these insightful questions regarding the reliability and adaptability of our return-relevance learning mechanism.

(1) Warm-up reliability. The 10,000-update warm-up serves as an empirical initialization stage rather than a theoretical convergence guarantee. To assess early prediction reliability, we evaluate the return predictor across different training stages on the evolving online replay distribution (Appendix C). Across two representative tasks, the Pearson correlation between predicted and empirical episodic returns increases from 0.588 at 5,000 updates to 0.984 at 10,000 updates, reaching 0.985 at 20,000 updates. Meanwhile, relative MAE decreases from 51.97% to 28.25% and 20.44%, respectively. These results demonstrate that the return predictor captures strong return-predictive relationships within relatively few updates, while its numerical prediction accuracy continues to improve throughout online training.

(2) Reliability of Criticality and Polarity. The reliability of our relevance estimation is supported by the close coupling between online data collection and continual predictor adaptation. During interaction, states are generated sequentially through environment transitions under the current policy, rather than sampled arbitrarily from the entire state space. Since the policy evolves through incremental updates and newly collected trajectories are continually incorporated into relevance learning, the predictor is regularly exposed to the state distribution induced by the evolving policy. Moreover, bounded local action perturbations keep exploration actions close to the current policy, reducing abrupt deviations in action selection. Together, these mechanisms allow the online head to leverage previously learned representations and updated experience to identify critical states without requiring repeated visits to identical states. Appendix C empirically supports this reliability: on held-out trajectories, the online head achieves 0.016 MAE, 91.5% polarity sign accuracy, and 0.82 Spearman correlation with retrospective relevance targets.

(3) Adaptation to focused exploration data. Yes, newly collected trajectories from Focused Exploration are incorporated into subsequent relevance learning. The return predictor is periodically updated using completed episodes generated by the evolving collection policy, including focused exploration experience. The resulting relevance annotations are refreshed, and the online head is updated accordingly. This establishes a continuous feedback process in which newly explored behaviors contribute additional return supervision and improve subsequent relevance estimation. We have clarified these mechanisms in the revised Sec. 3.2 and Appendix G.

Response to Question 3

Thank you for this insightful question. Our approach shares the motivation of causal discovery and causal credit assignment: identifying states or actions that are consequential for long-term returns. However, causal methods typically seek to identify causal dependencies or estimate the effects of interventions, whereas FocusDreamer focuses on predictive return relevance rather than causal attribution. Specifically, Criticality measures each state's contribution to episodic-return prediction through attention rollout, while Polarity captures the directional change in predicted return as new trajectory information becomes available, conceptually related to prediction-difference attribution in RUDDER. Neither signal alone establishes a causal relationship, as clarified in Appendix A.3 and A.9.

Importantly, we further connect predictive relevance to actual action consequences through controlled real-environment interventions (Sec. 5.4). By restoring identical simulator states and evaluating alternative actions, we directly measure real return differences independently of the predictor. Criticality increases with real local improvement opportunities (0.028 to 0.108), while negative-Polarity states exhibit 29% larger opportunities than positive-Polarity states. Thus, rather than discovering causal structure, FocusDreamer uses predictive relevance to identify where interventions on alternative actions are potentially most valuable. This provides a complementary perspective to causal credit assignment, supported by independent counterfactual evaluation. We have clarified this relationship in the revised related-work discussion and Sec. 3.2 and 5.4.

AuthorsOpenReview ↗

Additional Clarification on W1 and W2

Thank you again for your careful review. We would like to highlight two further improvements in the revised manuscript that directly address your main concerns.

W1: Reliability of Return Relevance and Theoretical Assumptions. We have further reorganized the experimental evidence and incorporated key validation results into the main text for easier inspection. Specifically, Sec. 5.3 (Table 3) now provides a complete evaluation of the online relevance predictor on held-out trajectories, including its generalization to previously unseen states. Sec. 5.4 (Table 4) independently validates the relationship between Criticality and actual local improvement opportunities through controlled real-environment interventions.

Regarding the theoretical assumptions, although our guarantees are conditional, the assumptions are well-motivated and appropriate for the local behavioral analysis considered in this work. Local smoothness and bounded Jacobians are standard conditions for analyzing small perturbations, while attention stability, non-cancellation, and bounded predictor-gradient error characterize when predictive relevance can meaningfully reflect return sensitivity. Importantly, the independent real-environment interventions provide empirical support for the practical implications of these assumptions, without requiring them to hold universally.

W2: Experimental Reliability and Generalization. We have further clarified the original Meta-World results in Sec. 5.2, providing a more detailed task-specific discussion of FocusDreamer's performance relative to the baselines. In particular, we explain its clear advantage on Hand Insert, its improvements over the other model-based baselines on Stick Pull and Pick Place Wall, and the performance saturation observed on Hammer.

To further strengthen the experimental evidence and demonstrate effectiveness in more challenging environments, we have additionally extended FocusDreamer to two long-horizon Minecraft tasks, Combat Ghast and Combat Wolf (Sec. 5.2, Fig. 4). Compared with ResDreamer (ICML 2026), a recent hierarchical world-model approach, FocusDreamer improves success rates from 82.2% to 93.0% and from 66.1% to 84.7%, respectively. The learning curves also include standard-deviation shading across independent random seeds. Together with the five-seed statistics reported in our original benchmark evaluations, these additional experiments further support the effectiveness and general applicability of return-relevance-guided resource allocation.

We sincerely appreciate your constructive feedback, which has helped us strengthen the experimental validation and presentation of our work. We hope these additional clarifications help address your remaining concerns.

AuthorsOpenReview ↗

Further Clarifications on Q1–Q3

Thank you again for your thoughtful questions. We have included several additional analyses and clarifications in the revised manuscript that complement our earlier responses.

Q1: Relationship Between Return Relevance and Actual Environment Returns.

In our earlier response, we presented real-environment intervention results showing the relationship between Criticality and actual local improvement opportunities (Sec. 5.4, Table 4). We have also extended the evaluation to alternative relevance measures in Appendix E (Table 7). Under matched learning budgets, we compare Q-Ensemble Disagreement, RUDDER, RRD, Prefix Change, and Last-Layer Attention while keeping the resource-allocation pipeline unchanged. Attention-Rollout Criticality achieves the highest average return (748.0) and a top-25% improvement-opportunity ratio of 1.77, compared with 628.3 and 1.31 for Q-Ensemble Disagreement, and 707.6 and 1.61 for Last-Layer Attention. Together with the earlier intervention results, these comparisons show that the proposed relevance signal provides more effective localization of improvement opportunities and better downstream policy performance than the evaluated alternatives.

Q2: Warm-up Reliability and Adaptation to Online Exploration.

Our earlier response reported the return predictor's reliability during warm-up and explained how newly collected trajectories are incorporated into subsequent relevance learning. An important aspect of this process is the different roles of relevance estimation across the three allocation stages. During real interaction, the lightweight online head controls the magnitude of bounded action perturbations, whereas focused world-model learning and imagination use retrospective relevance annotations obtained from completed trajectories.

Meanwhile, FocusDreamer retains Dreamer's original exploration mechanism and directs additional exploration toward local behavioral alternatives at return-relevant states. As the policy evolves, newly collected trajectories provide further supervision for relevance estimation. This allows exploration and relevance learning to progress together, gradually expanding the coverage of task-relevant behaviors. The corresponding design is described in Sec. 3.3 and Appendix G.

Q3: Relationship to Causal Discovery and Attribution.

As discussed in our earlier response, Criticality and Polarity characterize predictive return relevance, while controlled real-environment interventions independently evaluate the consequences of alternative actions. Related methods such as RUDDER (Arjona-Medina et al., 2019) and RRD (Ren et al., 2022) similarly leverage trajectory-level returns to construct informative learning signals, while counterfactual credit assignment (Mesnard et al., 2021) examines the contributions of alternative decisions to outcomes. FocusDreamer shares the broader motivation of identifying consequential decisions, but uses predictive relevance to guide exploration and learning resource allocation rather than directly estimating causal effects.

We have further organized the counterfactual analyses in Appendix D to compare the original value-gap contraction in DreamerV3 with the recovery achieved by FocusDreamer. For alternatives that achieve higher returns in the real environment, their mean return advantage is 0.0808. DreamerV3 reduces this advantage to 0.0051 in imagination, whereas FocusDreamer preserves 0.0397. This comparison complements the real-environment intervention results and illustrates how relevance-guided allocation improves the preservation of behaviorally meaningful return differences, providing practical support for predictive relevance as a learning signal while remaining distinct from causal attribution.

We appreciate your constructive questions and the opportunity to further clarify these aspects of our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Experimental Evidence

Dear Area Chair and Reviewers,

We would like to summarize the motivation, methodology, and experimental evidence for FocusDreamer.

1. Motivation and Methodology

Our work starts from a limitation of imagination-based world-model RL: when potentially better behavioral alternatives lack sufficient real experience, the world model may predict their outcomes toward familiar dynamics, contracting or even reversing their actual return advantages. Since policy optimization relies on imagination, this bias can discourage further exploration of these alternatives, limiting corrective experience and reinforcing prediction errors.

Existing uncertainty-guided approaches mainly explore poorly modeled regions to improve model reliability and behavioral coverage. However, comprehensively exploring and modeling the state–action distribution requires substantial interaction and computation, while many regions may contribute little to policy improvement. Our central idea is to use task-return relevance to identify consequential decision states and allocate limited learning resources where alternative behaviors could meaningfully improve the policy.

To this end, we propose FocusDreamer, which trains a lightweight Transformer on completed real trajectories to predict empirical episodic returns. From this predictor, we derive Criticality through attention rollout to estimate each state's relative relevance to return prediction, and Polarity through predicted-return changes between consecutive trajectory prefixes to characterize favorable or unfavorable progression. These signals jointly guide focused exploration, world-model learning, and imagination, directing additional resources toward consequential decisions while preserving Dreamer's original optimization objectives.

2. Experimental Results and Mechanism Validation

On eight visual DMControl tasks, FocusDreamer achieves an average return of 778.0, outperforming the 2026 world-model methods Knowledge Boundary (686.9) and R2-Dreamer (684.0) by 13.3% and 13.7%, respectively. We also observe improvements on representative Meta-World and Atari100K tasks. During the rebuttal, we further extend FocusDreamer to two long-horizon Minecraft tasks, improving success rates over ResDreamer (ICML 2026) from 82.2% to 93.0% and from 66.1% to 84.7%, respectively (Sec. 5.2, Fig. 4). These evaluations demonstrate the effectiveness of our approach across multiple benchmarks, task types, and world-model architectures.

We also examine the proposed mechanism through complementary experiments. ​

First, validating the original problem and FocusDreamer's improvement. Counterfactual comparisons (Appendix D, Figs. 5 and 6) show that DreamerV3 substantially underestimates the return advantages of insufficiently supported but genuinely better actions, whereas FocusDreamer recovers a considerable portion of these lost advantages. Second, evaluating whether Criticality identifies valuable decision states. Controlled real-environment interventions (Sec. 5.4, Table 4) show that Criticality identifies states with larger local improvement opportunities more effectively than random selection and uncertainty-based measures. Further comparisons with Q-Ensemble Disagreement, return-decomposition methods, and alternative attention-based relevance signals (Appendix E, Table 7) support its advantages in opportunity localization and downstream policy performance.​ Finally, evaluating generalization to unseen states. Held-out trajectory experiments (Sec. 5.3, Table 3) show that the online relevance predictor can estimate return relevance for previously unseen states within the evaluated policy distribution, supporting its use during online exploration.

We appreciate the reviewers' constructive feedback, which has helped strengthen these experimental analyses.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.