- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper compares 24 intrinsic-motivation objectives within a JAX Dreamer framework and three world-model representations: DreamerV3, NE-Dreamer, and R2-Dreamer. It evaluates zero-shot control, transfer, and adaptation in DMC, Craftax, and RoboKitchen. Curiosity has the strongest aggregate family profile, but rankings vary across representations, modalities, and tasks; simple intrinsic rewards sometimes outperform more elaborate ones.
Critical strengths
-
Scope and common evaluation framework. A common Dreamer setting reduces cross-paper comparison confounds and could provide a useful, reusable benchmark of 24 objectives.
-
Useful evaluation distinctions. Disentangling exploration from downstream control and testing transfer and adaptation exposes modality-dependent rankings (Figs. 3–5). The competitiveness of simple drives is practically relevant.
Critical weaknesses
-
Insufficiently established novelty and scientific significance. Plan2Explore established curiosity-driven world-model pretraining; URLB benchmarked intrinsic objectives. This paper combines and expands these directions, but most ingredients are established, and its main findings—curiosity often works well, rankings depend on context, and simple intrinsic rewards can compete—remain descriptive. The rDADS variant and cross-representation observations add some novelty, but the paper lacks a compelling explanation or generalizable, actionable insight. Benchmark/software work is legitimate at ICLR; the question is whether the new knowledge, rather than the JAX integration effort itself, is sufficiently consequential. The current paper does not convince me that it is.
-
Evidence does not fully support the world-model interpretation or efficiency narrative. The paper infers world-model quality from downstream returns without measuring predictive accuracy or rollout reliability (Sec. 4); returns also reflect coverage and task-head optimization. The backbones differ in auxiliary predictors and loss scales (Sec. 2.1), preventing isolated representation comparisons. Approximately three seeds and method-specific hyperparameters weaken rankings; the many adaptations also require careful validation against the source implementations. Fig. 5 also excludes pretraining/offline costs: Craftax's 6.22× adaptation ratio follows 5M pretraining plus 0.5M adaptation steps, versus ~3.11M from scratch to matched return. Reusable pretraining may justify this expense, but it is not 6.22× total interaction efficiency. Model diagnostics, uncertainty on headline comparisons, and amortized-cost analysis are needed to substantiate the interpretation.
Questions for authors
-
Which finding establishes new understanding beyond Plan2Explore/URLB, and what testable prediction follows?
-
Do methods achieving higher downstream performance also learn more accurate world models, as measured by held-out prediction or imagined-rollout reliability?
-
Which rankings survive seed/implementation variation, and how were implementations validated? How does efficiency change when pretraining costs are counted or amortized?
7 comments
Response to Reviewer Questions
Response to Questions
Q1: Which finding establishes new understanding beyond Plan2Explore/URLB, and what testable prediction follows?
A1: Plan2Explore only tests three intrinsic motivations, all of which curiosity, on a single world model representation. URLB tests a few more algorithms, but in a model-free setting. Thus, neither prior work establishes a large-scale study of zero-shot, model-based control from intrinsic motivation. Our study is the first to compare 24 algorithms broadly across the self-supervised RL literature in a single framework, alongside comparing each of these with an alternative word model representation. Our revisions center the paper around several actionable insights.
- Reconstruction-free world models struggle in vision-based tasks, hindering the benefits of self-supervised pretraining from intrinsic motivation.
- Curiosity is the single-most robust intrinsic motivation family across representations and environment modalities. We propose this is due to its alignment with the Dreamer world model representation, both relying on a predictive latent state for future control. We utilize this observation to propose alternative world model objectives that align with competency algorithms for future work to identify if alignment between policy and world-model learning is a driving factor.
- Using our transfer experiments in Walker and Craftax, which both possess a compositional structure, we identify NGU as a particularly robust algorithm. We propose this results from NGU’s dependence on an explicit episodic and lifelong memory that complement a generic world model memory through intrinsic motivation. This suggests a mechanism to design world models with explicit memory structures that intrinsic motivation can benefit from.
- Finally, we observe that Dreamer-based world models exhibit a scaling mismatch between offline imagination and online adaptation in Craftax, but not Walker. In particular, more pretraining in imagination improves zero-shot and few-shot adaptation in physics-based tasks where the dynamics are low-dimensional and constrained, but the opposite effect holds in Craftax where the dynamics are unconstrained by physics and possess complex, long timescale dependencies between achievements. Thus, we point out that even state-of-the-art model-based RL agents still struggle with cognitive tasks. We use our observations about NGU’s episodic memory to suggest a general mechanism for improving world model performance in these tasks for future work.
Q2: Do methods achieving higher downstream performance also learn more accurate world models, as measured by held-out prediction or imagined-rollout reliability?
A2: Yes, this is precisely the case. There is a direct correlation between zero-shot performance and world-model prediction error. However, this correlation does not hold for few-shot adaptation, where the policy is allowed to collect data that corrects the world model on task-relevant observations with high prediction error. Thus, algorithms that exhibit the best zero-shot performance do not necessarily exhibit the best few-shot adaptation.
Q3: Which rankings survive seed/implementation variation, and how were implementations validated? How does efficiency change when pretraining costs are counted or amortized?
A3: The rankings reported already follow seed variation, both in the network and environment seeding. We do not propose that building world models from intrinsic motivation is more efficient in a cost-of-compute or total sample complexity sense, we are simply interested in whether a self-supervised, task-agnostic objective can lead to task-optimal performance. Thus, in our experiments, there is no efficiency gain argument made, as we do not attempt to achieve zero-shot performance by fewer self-supervised pretraining steps than supervised steps.
Response to Authors
Thank you for the substantial revisions. The clearer scientific positioning and new Figure 8 improve the manuscript.
One central concern remains. In response to Q2, you report a direct correlation between zero-shot performance and world-model prediction error, but I could not identify the corresponding measurements in the revised manuscript.
Could you provide the prediction-error metric, correlation results, and supporting evidence across methods? This would materially strengthen the paper's central interpretation and inform my final recommendation.
Response to Reviewer ZBXE
We thank the reviewer for their time in evaluation our paper and providing constructive feedback. We found the feedback to be very helpful in improving our paper, and have incorporated the reviewer's concerns into our revisions. We provide a response to each identified weakness and concern below.
Response to Weaknesses
W1: We agree that it was difficult to clearly follow our specific contributions and novelty in the initial submission. Our revisions carefully address this concern by centering around a consistent narrative throughout the introduction, results, and discussion section. The goals of our paper are as follows, each of which a novel contributions:
W2: Our paper does not propose a world-model efficiency narrative. You are correct that there is a cost of pretraining and that zero-shot performance does not come for free. However, we do not suggest otherwise in our paper. Rather, we are following an established line of work in reinforcement learning to understand how pretraining from task-agnostic, self-supervised objectives can support zero-shot control. We are not studying how to push the pretraining efficiency of any single objective.
Regarding world model backbones and intrinsic motivation hyperparameters, we do not believe this is a significant weakness. We choose to adopt the strongest settings proposed by the authors of each original algorithm in order to assess robustness of the algorithm. Our goal is not to independently tune each algorithm, which would obscure an understanding of which intrinsic motivation family and world model backbone works across the board.
Regarding using downstream control to assess world-model quality, we believe this is the strongest and most relevant measure of a world model. While it is true that we can also study the world model predictive accuracy on held-out observations, this is a weaker proxy for whether these predictions can successfully generate behavior via reinforcement learning. This is the crux of building world models from intrinsic motivation—to study the exploration and representation learning jointly through behavior. However, if the reviewer would prefer us to include an analysis of the world-model independently of control, we are happy to accommodate this.
...(continued in next comment)