acceptodds
Under review as a conference paper at ICLR 2027

Building World Models from Intrinsic Motivation

Abstract

In the absence of extrinsic rewards or task supervision, human behavior is guided by intrinsic motivation—neural signals that drive curiosity, creativity, play, and other essential facets of intelligence. Intuitively, intrinsic motivation holds a fundamental role in developing robust internal models of the world that enable generalization to new tasks from little to no task-directed experience. However, this hypothesis has not been carefully explored: existing approaches often focus on model-free reinforcement learning, compare only a small number of intrinsic motivations, use inconsistent reinforcement learning algorithms, and fail to isolate the benefits of representation learning from those of policy learning. This makes it impossible to compare different hypotheses for intrinsic motivation, and more importantly, ignores the interplay between intrinsic motivation and world-model building. To close these gaps, we integrate 24 state-of-the-art self-supervised exploration algorithms—spanning a breadth of methods across curiosity, entropy, and skill-discovery—with a common model-based reinforcement learning backbone, and evaluate how well these intrinsic motivations learn an effective world model. We test our agents on zero-shot, few-shot, and multi-task control tasks in locomotion, manipulation, and open-ended gaming environments using three competing world-model representation learning hypotheses: observation reconstruction, latent prediction, and redundancy reduction. Across all experiments, we find that curiosity-driven exploration with observation reconstruction has the strongest aggregate family profile across representations, but the best individual pairing can depend on the specific experiment. Reconstruction is particularly effective for visual zero-shot control, while the reconstruction-free models are achieve competitive transfer and few-shot adaptation. These findings motivate evaluating intrinsic drives by the reusable world models they build and designing intrinsic motivation and representation learning jointly. Our approach demonstrates that world models provide a powerful tool to measure the quality of intrinsically motivated experience, and identify intrinsic motivation as a fundamental ingredient for building world models just as animals do—autonomously.

Primary area:
reinforcement learning
Keywords:
intrinsic motivation, reinforcement learning, world model, curiosity

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_ZBXEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares 24 intrinsic-motivation objectives within a JAX Dreamer framework and three world-model representations: DreamerV3, NE-Dreamer, and R2-Dreamer. It evaluates zero-shot control, transfer, and adaptation in DMC, Craftax, and RoboKitchen. Curiosity has the strongest aggregate family profile, but rankings vary across representations, modalities, and tasks; simple intrinsic rewards sometimes outperform more elaborate ones.

Critical strengths
  1. Scope and common evaluation framework. A common Dreamer setting reduces cross-paper comparison confounds and could provide a useful, reusable benchmark of 24 objectives.

  2. Useful evaluation distinctions. Disentangling exploration from downstream control and testing transfer and adaptation exposes modality-dependent rankings (Figs. 3–5). The competitiveness of simple drives is practically relevant.

Critical weaknesses
  1. Insufficiently established novelty and scientific significance. Plan2Explore established curiosity-driven world-model pretraining; URLB benchmarked intrinsic objectives. This paper combines and expands these directions, but most ingredients are established, and its main findings—curiosity often works well, rankings depend on context, and simple intrinsic rewards can compete—remain descriptive. The rDADS variant and cross-representation observations add some novelty, but the paper lacks a compelling explanation or generalizable, actionable insight. Benchmark/software work is legitimate at ICLR; the question is whether the new knowledge, rather than the JAX integration effort itself, is sufficiently consequential. The current paper does not convince me that it is.

  2. Evidence does not fully support the world-model interpretation or efficiency narrative. The paper infers world-model quality from downstream returns without measuring predictive accuracy or rollout reliability (Sec. 4); returns also reflect coverage and task-head optimization. The backbones differ in auxiliary predictors and loss scales (Sec. 2.1), preventing isolated representation comparisons. Approximately three seeds and method-specific hyperparameters weaken rankings; the many adaptations also require careful validation against the source implementations. Fig. 5 also excludes pretraining/offline costs: Craftax's 6.22× adaptation ratio follows 5M pretraining plus 0.5M adaptation steps, versus ~3.11M from scratch to matched return. Reusable pretraining may justify this expense, but it is not 6.22× total interaction efficiency. Model diagnostics, uncertainty on headline comparisons, and amortized-cost analysis are needed to substantiate the interpretation.

Questions for authors
  1. Which finding establishes new understanding beyond Plan2Explore/URLB, and what testable prediction follows?

  2. Do methods achieving higher downstream performance also learn more accurate world models, as measured by held-out prediction or imagined-rollout reliability?

  3. Which rankings survive seed/implementation variation, and how were implementations validated? How does efficiency change when pretraining costs are counted or amortized?

7 comments
AuthorsOpenReview ↗

Response to Reviewer ZBXE

We thank the reviewer for their time in evaluation our paper and providing constructive feedback. We found the feedback to be very helpful in improving our paper, and have incorporated the reviewer's concerns into our revisions. We provide a response to each identified weakness and concern below.

Response to Weaknesses

W1: We agree that it was difficult to clearly follow our specific contributions and novelty in the initial submission. Our revisions carefully address this concern by centering around a consistent narrative throughout the introduction, results, and discussion section. The goals of our paper are as follows, each of which a novel contributions:

  • The primary goal of our work is to propose world models as a tool for evaluating intrinsic motivations, which are typically evaluated according to generic criteria that is not well-motivated, such as state-space coverage or model-free adaptation. We also jointly study three different world model representations to understand the interplay between exploration and representation learning.
  • The secondary goal of our work is to present a breadth of algorithms in a single framework, both to offer the community a glance at the current state of self-supervised reinforcement learning and to study algorithms that have previously not received attention in the community. To the former point, most researchers are familiar with the basics of self-supervised RL approaches, such as a self-supervised prediction reward, but many algorithms related to empowerment are not common knowledge. To the latter point, we find that a relatively obscure algorithm (NGU) that combines episodic and lifelong memory shows immense promise indicates clear mechanisms for future research on improving both self-supervised RL algorithms and world model representation learning.
  • The third goal is to show that current state-of-the-art world models still struggle on vision-based tasks with compositional structure (the walker transfer experiments, going from stand to walk to run, and Crafter, which has complex achievement structures) . Thus, the novelty of our paper is not the Jax integration effort or minor experiments like the rDADS estimator experiment. Rather, we offer a large-scale study of intrinsic motivations and world-model learning that shows curiosity-driven exploration is the most robust pretraining recipe across modalities and experiments. Further, we use the NGU algorithms exceptional performance to motivate future research into incorporating episodic memory into world models in order to push performance on complex tasks with compositional structure. We believe the revised Introduction, Results, and Discussion section make clear the novelty of these contributions beyond previous work the reviewer mentions, which study only a handful of objectives, don’t identify actionable insight for improving world model pretraining, or study world model representation learning.

W2: Our paper does not propose a world-model efficiency narrative. You are correct that there is a cost of pretraining and that zero-shot performance does not come for free. However, we do not suggest otherwise in our paper. Rather, we are following an established line of work in reinforcement learning to understand how pretraining from task-agnostic, self-supervised objectives can support zero-shot control. We are not studying how to push the pretraining efficiency of any single objective.

Regarding world model backbones and intrinsic motivation hyperparameters, we do not believe this is a significant weakness. We choose to adopt the strongest settings proposed by the authors of each original algorithm in order to assess robustness of the algorithm. Our goal is not to independently tune each algorithm, which would obscure an understanding of which intrinsic motivation family and world model backbone works across the board.

Regarding using downstream control to assess world-model quality, we believe this is the strongest and most relevant measure of a world model. While it is true that we can also study the world model predictive accuracy on held-out observations, this is a weaker proxy for whether these predictions can successfully generate behavior via reinforcement learning. This is the crux of building world models from intrinsic motivation—to study the exploration and representation learning jointly through behavior. However, if the reviewer would prefer us to include an analysis of the world-model independently of control, we are happy to accommodate this.

...(continued in next comment)

AuthorsOpenReview ↗

Response to Reviewer Questions

Response to Questions

Q1: Which finding establishes new understanding beyond Plan2Explore/URLB, and what testable prediction follows?

A1: Plan2Explore only tests three intrinsic motivations, all of which curiosity, on a single world model representation. URLB tests a few more algorithms, but in a model-free setting. Thus, neither prior work establishes a large-scale study of zero-shot, model-based control from intrinsic motivation. Our study is the first to compare 24 algorithms broadly across the self-supervised RL literature in a single framework, alongside comparing each of these with an alternative word model representation. Our revisions center the paper around several actionable insights.

  • Reconstruction-free world models struggle in vision-based tasks, hindering the benefits of self-supervised pretraining from intrinsic motivation.
  • Curiosity is the single-most robust intrinsic motivation family across representations and environment modalities. We propose this is due to its alignment with the Dreamer world model representation, both relying on a predictive latent state for future control. We utilize this observation to propose alternative world model objectives that align with competency algorithms for future work to identify if alignment between policy and world-model learning is a driving factor.
  • Using our transfer experiments in Walker and Craftax, which both possess a compositional structure, we identify NGU as a particularly robust algorithm. We propose this results from NGU’s dependence on an explicit episodic and lifelong memory that complement a generic world model memory through intrinsic motivation. This suggests a mechanism to design world models with explicit memory structures that intrinsic motivation can benefit from.
  • Finally, we observe that Dreamer-based world models exhibit a scaling mismatch between offline imagination and online adaptation in Craftax, but not Walker. In particular, more pretraining in imagination improves zero-shot and few-shot adaptation in physics-based tasks where the dynamics are low-dimensional and constrained, but the opposite effect holds in Craftax where the dynamics are unconstrained by physics and possess complex, long timescale dependencies between achievements. Thus, we point out that even state-of-the-art model-based RL agents still struggle with cognitive tasks. We use our observations about NGU’s episodic memory to suggest a general mechanism for improving world model performance in these tasks for future work.

Q2: Do methods achieving higher downstream performance also learn more accurate world models, as measured by held-out prediction or imagined-rollout reliability?

A2: Yes, this is precisely the case. There is a direct correlation between zero-shot performance and world-model prediction error. However, this correlation does not hold for few-shot adaptation, where the policy is allowed to collect data that corrects the world model on task-relevant observations with high prediction error. Thus, algorithms that exhibit the best zero-shot performance do not necessarily exhibit the best few-shot adaptation.

Q3: Which rankings survive seed/implementation variation, and how were implementations validated? How does efficiency change when pretraining costs are counted or amortized?

A3: The rankings reported already follow seed variation, both in the network and environment seeding. We do not propose that building world models from intrinsic motivation is more efficient in a cost-of-compute or total sample complexity sense, we are simply interested in whether a self-supervised, task-agnostic objective can lead to task-optimal performance. Thus, in our experiments, there is no efficiency gain argument made, as we do not attempt to achieve zero-shot performance by fewer self-supervised pretraining steps than supervised steps.

Reviewer_ZBXEOpenReview ↗

Response to Authors

Thank you for the substantial revisions. The clearer scientific positioning and new Figure 8 improve the manuscript.

One central concern remains. In response to Q2, you report a direct correlation between zero-shot performance and world-model prediction error, but I could not identify the corresponding measurements in the revised manuscript.

Could you provide the prediction-error metric, correlation results, and supporting evidence across methods? This would materially strengthen the paper's central interpretation and inform my final recommendation.

Reviewer_7rzDOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3(+2)
Soundness
2(−1)
Read review
Paper summary

This paper proposes the use of intrinsic motivation (IM) approaches in Reinforcement Learning as a method to learn robust world models and to evaluate IM approaches by the quality of the world models that they build. To test theyr hypothesis, the authors test 24 self-supervised exploration algorithms on a range of RL settings that include zero-shot generalization, few-shot learning and multi-task learning. Furthermore the authors study three different representation learning methods for world models.

Critical strengths
  • The authors provide a large scale evaluation of intrinsic motivation objectives
  • The authors test 3 different latent representation auxiliary objectives.
  • Multiple RL settings were considered
Critical weaknesses

Major:

  • There is no dedicated related work section in the main document. Important work is missed, in particular [2], which already proposes a model-based backbone to URL based on Dreamer. The IM approaches are only introduced in the supplementary materials and in a very high level manner. In my opinion it is essential to have at least a description of the IM families and the differences between them in the main text.
  • The main weakness of this work is its presentation and communication of the main findings. The same large scale evaluation that can be considered a strength is also the reason why the research questions that motivate this paper become unclear for the reader. Is the idea to do a large-scale study of IM approaches or to study their impact on WM learning? Or is it to use World Model learning for evaluating IM approaches?
  • It is also unclear what conclusions should be drawn from each experiment. The authors do a good job at including different test cases but the reasoning behind the choice of an experiment in regards to how it helps answer the research questions should be made clear. Furthermore, results descriptions should not be limited to stating the numbers that are in the plots but rather to interpret what they teach us in therms of the design choices of the different methods and their influence the outcomes of the experiments. This, together with the previous point made it hard for me to interpret the scientific outcomes of this work.
  • Choosing States as main modality of experiments. While the paper places a lot of emphasis on the study of 3 alternative feature learning auxiliary objectives, it then focuses its experiments on state inputs instead of pixels (25 tasks vs. 4 tasks in 2 domains). When using states, the agent receives a “near-optimal representation”, as described in [1]. The impact of feature learning techniques is better understood when pixels are used as input.
  • Some of the results are presented in a convoluted way that is hard to interpret. For example: “Transfer gain is return minus the same-backend, same modality Stand-supervised reference, divided by task-supervised DreamerV3 return on the evaluation task”. Why not simply present the rewards? There are other such examples throughout the paper.

Minor:

  • All links to figures, references and sections are broken, and there are many typos throughout the text.

References:

[1] Laskin, M., Yarats, D., Liu, H., Lee, K., Zhan, A., Lu, K., Cang, C., Pinto, L. & Abbeel, P. (2021). Urlb: Unsupervised reinforcement learning benchmark. 35th Conference on Neural Information Processing Systems.

[2] Rajeswar, S., Mazzaglia, P., Verbelen, T., Piché, A., Dhoedt, B., Courville, A., & Lacoste, A. (2023, July). Mastering the unsupervised reinforcement learning benchmark from pixels. In International Conference on Machine Learning (pp. 28598-28617). PMLR.

Questions for authors
  • For the zero shot experiments it would be useful to have a reading on how much the zero-shot task execution improves over the pure intrinsic reward pretraining. This is because even during pretraining, due to the design (and density) of the rewards, the agents generate extrinsic reward, even though they are unaware of it. So zero-shot results are only meaningful in terms of how much they improve over the baseline extrinsic reward that is generated during the pretraining stage.
  • How does the zero-shot adaptation work? Is it like in Plan2Explore where the agent is allowed to adapt in imagination? Or are the checkpoints simply deployed with task reward but frozen?
  • How many random seeds were used to obtain the results? In the appendix it is mentioned that three seeds are used at some point, but it is not clear whether this is the case for all experiments. This is an important factor and it should be explicit in the main document how many seeds are used for each experiment.

Note: due to the high number of papers for review it was not possible for me to thoroughly read the Supplementary Materials. My appologies if some of these questions have an answer there.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 7rzD

Note to Reviewer

We thank the reviewer for taking the time to evaluate our paper. While we appreciate the important of concision when possible, the reviewer provided such constructive and detailed feedback that greatly improved the quality of our paper, so we felt a thorough response that addresses each and every point was deserved. Below, we respond to each of the reviewer's points in the order they appear.

Response to Weaknesses

W1: We agree with the reviewers concerns here, and have made significant efforts to address each one in the revisions.

  • First, we have expanded the related work in Appendix A to include [2] and explain our contributions as they relate to this paper, which are 1) the breadth of algorithms considered, 2) joint comparisons with world model representations, 3) and leveraging these insights, as well as novel experiments (Fig. 8), to identify ingredients of intrinsic motivation that support pertaining in complex environments.

  • Second, we agree that the paper should include an explanation of the IM families in the main text. We have added a section (sec. 2.3, lines 218-248) in the revisions that gives an intuitive explanation of each IM family and how they relate to each other, as well as important differences within IM families such as the Competency family.

  • Third, we have included a fuller discussion of related work, including [2], and where our paper fits in. This Appendix section is referenced in the introduction, which has also been rewritten to discuss how our contributions go beyond related work. It is organized this way due to the breadth of our paper across 24 intrinsic motivations, three world model representations, and several experimental designs that each need explanation in the main text. We believe this a natural choice in context of the new revisions we provide. However, if the reviewer feels strongly that the related work must appear in the main text, we are happy to make the necessary adjustments to accommodate.

W2: We agree that the original presentation of the work obscured a clear narrative in the paper (we detail changes in next section).

  • The primary goal of our work is to propose world models as a tool for evaluating intrinsic motivations, which are typically evaluated according to generic criteria that is not well-motivated, such as state-space coverage or model-free adaptation. We also jointly study three different world model representations to understand the interplay between exploration and representation learning.
  • The secondary goal of our work is to present a breadth of algorithms in a single framework, both to offer the community a glance at the current state of self-supervised reinforcement learning and to study algorithms that have previously not received attention in the community. To the former point, most researchers are familiar with the basics of self-supervised RL approaches, such as a self-supervised prediction reward, but many algorithms related to empowerment are not common knowledge. To the latter point, we find that a relatively obscure algorithm (NGU) that combines episodic and lifelong memory shows immense promise indicates clear mechanisms for future research on improving both self-supervised RL algorithms and world model representation learning.
  • The third goal is to show that current state-of-the-art world models still struggle on vision-based tasks with compositional structure (the walker transfer experiments, going from stand to walk to run, and Crafter, which has complex achievement structures) .

You are correct that these ideas were lost throughout the paper. We have made significant improvements in the revisions to maintain a clear narrative throughout the paper that communicates each of these goals.

  • In the introduction, we have added prose to make our position on using world models to evaluation intrinsic motivations a clear goal of this paper (lines 81-102).
  • In the introduction, we added a contributions section that reiterates the limitations of previous work in this area, and our specific contributions and findings regarding all three of the goals above.
  • In the methods section, we have removed any redundant explanations or irrelevant information to make room for an intuitive explanation of each IM family (section 2.3). This follows from a concise explanation of where the IDM sits in our architecture, and then uses the taxonomy in figure 1 to explain the main idea behind each IM and any important nuances of specific algorithms.
  • In the results section, we have replaced any unhelpful explanations of figures that cite numbers with concise explanations in prose. Further, each experiment section ends with the main takeaway to maintain a clear narrative around the three goals of the paper.

...(continued in the next comment)

AuthorsOpenReview ↗

Continuation of Response to Reviewer 7rzD

Response to W2 Continued...

  • In the results section, we have removed some of the idiosyncratic results that detract from the main ideas. In its place, we include a new figure and an accompanying clear explanation related the third goal of the paper and motivates future work on world model design.
  • In the discussion section, we have centered our narrative around these three goals and their implications for intrinsic motivation and world model research. Specifically, we have reinterpreted our results around the specific performance deficits arising from commonalities between tasks with a compositional structure and impose strong constraints on episodic memory. We discuss examples of recent work in world models from the vision community that include mechanisms to address these issues, and how these mechanisms can be incorporated into model-based RL agents.
  • The limitations and future work included a fuller discussion of key opportunities for extending the paradigm to robotic control, pretrained world models, and alternative world model architectures.

W3: We agree that the initial presentation of results did not illustrate the key findings. We have made significant improvements to the results section to address these issues. Each experiment section begins by summarizing the actual takeaway from the figures, and ends with the takeaway for broader narrative of the paper. Further, we refrain from arbitrarily referencing plot numbers, but rather using specific results only when relevant to explanation. Please see section 3 in the revisions to appreciate these changes, particularly in the last paragraph of each section (or the last few lines when the section is a single paragraph) that interprets the results in light of the broader goals of the paper.

  • The revisions to the discussion (section 4) and limitations and future work (section 5) also reiterate these points and make the scientific outcomes of the work clear. Please see lines 483-538 in the revision.
  • We have also revised both the background section, where the choice of each experiment is motivated, and the results section to clearly reiterate what each experiment answers about our research question.

W4: We agree that the importance of feature learning is best understood in high-dimensional tasks, such as pixels. As such, the focus of our experiments was not on states, but rather a comparison between states and vision in general. The full state-based 25-environment DMC suite was used only for Figure 1, and all following experiments were a one-to-one comparison between states and vision. Specifically, the vision experiments in the Walker and Craftax were the main source of insights, and propriocpection experiments were used primarily as a baseline. However, we recognize that the initial presentation of the paper obscured the three goals of our paper discussed above, especially surrounding the interpretation of results that primarily rely on the vision experiments. The narrative in our revisions now includes these takeaways from the vision experiments clearly in each experimental section, as well as reiterates them in the introduction and discussion.

We also want to point out that in many DMC environments, the proprioceptive state is not actually a sufficient state for optimal control. Although the RL community typically views these environments as solved, our experiments point out that there are many state-based DMC environments that DreamerV3 performs very poorly on (see Fig. 8). Thus, while we agree with the reviewers point that representation learning is primarily an object of interest in vision, proprioceptive control is still a relevant area of research to the general community with many opportunities for improvement. Our narrative in the revisions reflects these ideas more generally as part of the third goal of our work, exposing the limitations of current world models even in simple environments.

W5: We completely agree with the reviewer’s feedback here, and have addressed this in our revisions, simplifying each reporting metric where relevant. We report things relative to a task-optimal agent in general to avoid reporting raw rewards that are difficult to interpret without careful explanation of each environment. The relative metrics are also more direct, since ultimately the experiments aim to elucidate how pretraining from intrinsic motivation compares with training under the task reward. The goal was to make the results more intuitive and consistent across environments, but this was counteracted by some other poor choices in reporting metrics you have pointed out. With the new revisions, we believe this general strategy now achieves its original intention of being easy to read, including to non-experts. However, if the reviewer still feels strongly that we should report raw rewards, please let us know.

W6: We sincerely apologize for the broken links. We have fixed this and typos in the new revisions.

AuthorsOpenReview ↗

Response to Reviewer Questions

Response to Reviewer Questions

Q1: For the zero shot experiments it would be useful to have a reading on how much the zero-shot task execution improves over the pure intrinsic reward pretraining. This is because even during pretraining, due to the design (and density) of the rewards, the agents generate extrinsic reward, even though they are unaware of it. So zero-shot results are only meaningful in terms of how much they improve over the baseline extrinsic reward that is generated during the pretraining stage.

A1: We agree with the reviewer that the important metric is how much intrinsic pretraining benefits beyond the baseline extrinsic reward agent, and this is exactly why we report our results relative to the task-supervised agent rather than raw rewards. All experiments, including the zero-shot results, are reported in this way. For example, in Figure 3, we see that Vanilla Curiosity achieves ~50% zero-shot of the agent trained with the extrinsic task reward. Similarly, transfer gain in figure 4 reports the percent improvement relative to the agent trained with extrinsic task reward.

Q2: How does the zero-shot adaptation work? Is it like in Plan2Explore where the agent is allowed to adapt in imagination? Or are the checkpoints simply deployed with task reward but frozen?

A2: Their are two notions of zero-shot adaptation you may be referring to. In the transfer experiments, the policies are fine-tuned from training in imagination (where the world-model is frozen) using the task reward, as in Plan2Explore. In the few-shot adaptation experiments, the policy is first evaluated zero-shot without any fine-tuning (we refer to this in the paper as step-0), and then subsequently evaluated after successive stages of fine-tuning in imagination as in the previous experiment.

Q3:: How many random seeds were used to obtain the results? In the appendix it is mentioned that three seeds are used at some point, but it is not clear whether this is the case for all experiments. This is an important factor and it should be explicit in the main document how many seeds are used for each experiment.

A3: All experiments in the main text use three seeds. We also point out that the training is parallelized over at least 16 environment instances that are independently seeded, thus providing variation in the training data as well. We have clarified this point in the revisions in the main text where experiments are described (section 2.4).

Reviewer_a2DpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that the interaction between world modelling and intrinsic motivation objectives remains underexplored. It presents an empirical study of intrinsic motivation objectives through the lens of world modelling, viewing intrinsic rewards as objectives for collecting experience (but not the world-model representations) that affect the generalization of the world model. The paper integrates 24 intrinsic exploration objectives with a Dreamer-based world model architecture and investigates how these objectives affect the generalization and transfer of the world model to different tasks under different adaptation assumptions (zero-shot, few-shot, and multi-task control). Moreover, the paper investigates the interplay between world model representation learning and intrinsic objectives, and how this affects performance across different generalization and transfer learning settings.

The results show that curiosity-based methods built on top of DreamerV3 achieve strong mean performance across all generalization and transfer learning settings. However, environment-specific results show that no single method consistently outperforms the others across all tasks.

Critical strengths
  • The analysis is thorough and covers a wide variety of intrinsic reward objectives.
  • The analysis covers different settings of generalization and transfer learning and highlights when some of the intrinsic rewards might be more useful depending on the setting, constraints, and observation type (image observations vs. proprioceptive states)
  • Assuming the codebase will be released (which is one of the major paper's contributions), this will be a great resource for studying the interaction between world models and intrinsic rewards.
Critical weaknesses
  • An interesting intrinsic reward objective (surprise minimization) is missing from the analysis.
  • Most environments are continuous control; only one is for open-ended exploration.
Questions for authors
  • The multi-task transfer experimental setup is confusing: can you clarify how the data is collected? Is it collected by running the intrinsic exploration agent on the source environment? What do you mean by task-driven exploration?
  • By "Stand-supervised experience," do you mean training only on the task reward? It would help if the wording were clarified.
  • Can the authors run surprise minimization experiments on Craftax-Classic? Since surprise minimization is about controlling the environment, it makes sense to run it on Craftax-Classic, where it has higher natural entropy than DMC.
3 comments
AuthorsOpenReview ↗

Response to Reviewer a2Dp

We thank the reviewer for taking the time to evaluate our paper, and for providing constructive feedback allowing us to improve our work. Below we provide responses to the weaknesses and questions you propose.

Response to Weaknesses

W1: An interesting intrinsic reward objective (surprise minimization) is missing from the analysis.

  • All of our experiments do include surprise minimization, which is implemented by ICM and Vanilla Curiosity. Both algorithms compute the surprise of a future observation under a learned internal model.

W2: Most environments are continuous control; only one is for open-ended exploration.

  • While we agree with this observation, this is simply a reflection of the availability of such environments. While the community has created numerous continuous control environments with the goal of robotic control, Crafter/Craftax is the primary standard reinforcement learning environment for open-ended exploration. Minecraft is a compelling exception, but current RL environments for Minecraft are extremely slow, computationally expensive, and have not been adopted at large by the RL community, thus precluding reasonable baselines and validation. Further, Craftax is an active research environment that already captures the essential properties of Minecraft. It has not yet been solved by model-based RL algorithms, and to the best of our knowledge, our work is the large-scale study of training world models in Craftax with intrinsic motivation.

Response to Questions

Q1: The multi-task transfer experimental setup is confusing: can you clarify how the data is collected? Is it collected by running the intrinsic exploration agent on the source environment? What do you mean by task-driven exploration?

A1: Yes, your understanding is correct. In the multi-task experiment, the source environment refers to the environment that the intrinsic motivation policy uses to collect data and train the world model. For your question on “task-driven exploration”, we believe you are referring to the sentence (line 268): “‘Stand source’ identifies the environment in which data are collected, not the task driving exploration.” Here, we are emphasizing that the task reward is not used in the source environment, so we are not collecting data under a task-driven policy (i.e. the stand-supervised policy, optimized to collect extrinsic reward). We thank the reviewer for pointing out that this the phrasing is confusing, so we have corrected this sentence in the revisions to make this distinction clear (lines 298-300).

Q2: By "Stand-supervised experience," do you mean training only on the task reward? It would help if the wording were clarified.

A2: Again, your understanding is correct. Stand-supervised experience refers to the data collected under the task policy trained to maximize extrinsic reward. We have rephrased our explanation so this is clear (lines 300-302).

Q3:: Can the authors run surprise minimization experiments on Craftax-Classic? Since surprise minimization is about controlling the environment, it makes sense to run it on Craftax-Classic, where it has higher natural entropy than DMC.

A3:: As we mentioned in our response to W1, our experiments already include surprise minimization via the algorithms ICM and Vanilla Curiosity.

Reviewer_a2DpOpenReview ↗

Response to Authors

Thank you for your response.

| All of our experiments do include surprise minimization, which is implemented by ICM and Vanilla Curiosity. Both algorithms compute the surprise of a future observation under a learned internal model.

I am not sure the authors understood my point here; I was referring to an environment control objective via surprise minimization[1]. The idea behind these kinds of objectives is to encourage the agent to control the entropy of the environment, but these objectives assume that the environment has a natural entropy; for example, in Craftax, the agent would be encouraged to build a shelter and preserve their life in front of the enemies.

| About the continuous control and open-ended exploration.

You raised a fair point; sorry, maybe I mis-specified my point. In some sense, the continuous-control environment requires less structured exploration than an environment such as Craftax. I mentioned Craftax because it requires more structured exploration and understanding of how one task/achievement might unlock the path for other achievements. Some environments in the MiniGrid benchmark, such as door-key, have this property. I am not asking you to run experiments on this benchmark, but just to be aware that there are different "kinds" of exploration and your paper is mostly about the more unstructured type of exploration common in continuous control, and there are some environments that you can experiment with which require more structured exploration (MiniGrid, Craftax).

| A1, A2

Thank you for clarifying; it's less confusing now.

| A3

I refer you to my first comment.

References: [1] Berseth, Glen, et al. "Smirl: Surprise minimizing reinforcement learning in unstable environments." arXiv preprint arXiv:1912.05510 (2019).

AuthorsOpenReview ↗

Response to Reviewer Clarification

Thank you for your clarification regarding surprise minimization. Without the specific reference, we misunderstood your comment. We agree that the smirl algorithm is indeed an interesting alternative, where entropy is explicitly minimized as the policy objective. ICM and other forms of curiosity minimize surprise of the forward-model, but the policy itself is adversarial in the sense of wanting to maximize this surprise. For now, we have begun a focused experiment on Craftax using smirl based on the reviewer's interest, but should the paper be accepted, smirl will be included in the final version for all environments across the experiment suite. The focused experiment on Craftax should be complete tomorrow and we can share the results with the reviewer.

Regarding your comment on open-ended exploration environments, thank you for the further clarification. It is definitely true that there are differing environmental constraints between continuous control, which has strong physical dynamical constraints, and abstract environments like Craftax. These differences indeed require different kinds of exploration, as evidenced by the fact the most performant algorithm is different between them. This is why we chose Crafax as our canonical example, since it is both open-ended and not physics-based. Many Craftax achievements have compositional structure matching or even more complex than the door-and-key example. In fact, our new revisions use this compositionality as a central idea to interpret the success of intrinsic motivations that use episodic memory where the novelty of such door-and-key observations can be tracked explicitly over the episode. We do recognize that other environments, such as mini-grid, could be tested for this kind of exploration, but we believe that Craftax is the most compelling of these environments.

Reviewer_rZpCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores the relationship between intrinsic motivation and world model construction. Prior approaches mostly focus on model-free reinforcement learning, compare only a small set of intrinsic motivations, employ inconsistent reinforcement learning algorithms, and fail to disentangle the benefits of representation learning and policy learning. To address these issues, the paper integrate 24 state-of-the-art self-supervised exploration algorithms (including curiosity, entropy, and skill discovery methods) into a unified model-based reinforcement learning framework to evaluate the ability of different intrinsic motivations to learn effective world models. Zero-shot, few-shot and multi-task control tests are conducted across locomotion, manipulation and open-ended game environments, and three hypotheses for world model representation learning are compared: observation reconstruction, latent variable prediction, and redundancy reduction.

Critical strengths

The paper systematically integrate 24 state-of-the-art self-supervised exploration algorithms—spanning curiosity, entropy, and skill-discovery families—with a single model-based RL backbone, and to evaluate them under three competing representation-learning hypotheses (observation reconstruction, latent prediction, redundancy reduction).

Critical weaknesses

None

Questions for authors

Why is observation-based reconstruction DreamerV3 most effective in visual zero-shot control, while reconstruction-free models have an advantage in proprioceptive transfer? Given only about three training seeds, each algorithm retained its original hyperparameters, and different implementation choices were adopted, under what conditions might the conclusion of this paper that "the curiosity family is the strongest overall" not hold?

2 comments
AuthorsOpenReview ↗

Response to Reviewer rZpC

We thank the reviewer for taking the time to evaluate our paper, and provide responses to your questions below.

Q1: Why is observation-based reconstruction DreamerV3 most effective in visual zero-shot control, while reconstruction-free models have an advantage in proprioceptive transfer?

A1: This reflects an important shortcoming of the reconstruction-free representation learning algorithms ability to learn a sufficient latent state from images. It is no surprise that DreamerV3 is the most performant, in general, given the information preserving properties of observation reconstruction. There are two possible reasons why reconstruction-free representation learning is successful in proprioception-based experiments. First, for many continuous control tasks, the proprioceptive state is already a low-dimensional, nearly-sufficient representation for optimal control, so little extra work has to be done to learn a state abstraction. Second, the continuous control environments largely involve locomotion or similar tasks that can be described as stabilizing a nonlinear dynamical system, the physics of which impose strong dynamical constraints the observations and their corresponding representations. Thus, the environment/task itself constrain the representations such that reconstruction-free world models are in an optimal regime and observation reconstruction imposes additional unnecessary learning pressures.

Q2: Given only about three training seeds, each algorithm retained its original hyperparameters, and different implementation choices were adopted, under what conditions might the conclusion of this paper that "the curiosity family is the strongest overall" not hold?

A2: Although we seed the network three times, we first note that the training is parallelized over many environments (typically 16) that are independently seeded, thus providing sufficient variation in the training data. Second, we retrain the hyperparameters of the IDMs and world-models introduced by their original papers in order to faithfully represent the optimal parameter tuning the original authors’ chose. Thus, we believe the cases where the conclusions that “the curiosity family is the strongest overall” have less to do with seeds and hyperparameters, but rather other design decisions regarding the type of world-model Dreamer uses and some of our implementation choices. First, Dreamer’s goal is to learn a latent representation that allows training policies by Monte Carlo rollouts of the recurrent world model. One of the inductive biases of Dreamer is a KL constraint between posterior and prior dynamics, which is closely related to the prediction-error objectives used by curiosity algorithms. Thus, with a different world model algorithm, such as those based on successor representations, it could be that entropy or competency algorithms, whose objectives are closely related to successor and occupancy representations, would confer a greater benefit than curiosity. Second, several competency algorithms use skills, which are temporal abstractions similar to options. In our implementation, we retain the Dreamer world model architecture to put all algorithms on equal footing and thus do not allow a skill-conditioned world model. However, allowing the world model to use the skills discovered by the competency algorithm for planning may confer better performance. Our revisions discuss these possibilities in the Limitations and Future Work section.

Thank you for your response,I basically agree with the author's explanations and analyses. For Q2,Are the relevant conclusions supported by comparative ablation studies?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Overall Changes

We thank the reviewer for their time in evaluating our paper. Below, we first describe some important improvements to the overall writing and explanation in light of the general reviewer feedback. Then, we respond to the reviewer-specific questions.

Summary of Changes

We agree with the weaknesses pointed out by Reviewers ZBXE and 7rzD concerning the overall lack of explanations that 1) obscured the main takeaways from each experiment, 2) clearly motivates the work and how this fits into broader literature, 3) explanations of the intrinsic motivation algorithms, and 4) the overall communication of the findings, both in terms of interpreting figures and the main ideas of the introduction and discussion.

We take this feedback very seriously, and have significantly improved each and every one of these aspects in the new revisions. While we will save pointing out every change to the reviewer-specific response, we encourage the reviewers to view the revisions and appreciate the following general improvements:

  1. Introduction, where both the motivation and specific contributions are explicated clearly and in more detail.
  2. Section 2.3, where we provide an intuitive explanation of each intrinsic motivation family.
  3. Results section, where for each main experiment, we both describe the main findings in clearly written prose and conclude each section with the main takeaway and interpretation of the result.
  4. Included a new main figure 8. related to the scaling mismatch result between world-model imagination and environment experience.
  5. Simplifying figure captions and reporting simpler axes (e.g., figure 8).
  6. A revised discussion section that interprets our results, and provides the key intuitions for how our takeaways can improve pertaining future world models by relating it to two specific ideas: memory and structured representations.
  7. An expanded limitations and future work, where we discuss several interesting opportunities to expand the idea of using world models to evaluate intrinsic motivations to more complex robotic tasks and more complex world models.
  8. A fuller discussion of related work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.