- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies representation learning for Dreamer-style world models under partial observability. It argues that losses anchored to the current observation, such as pixel reconstruction, do not pressure the latent state to retain information needed later. The proposed NE-Dreamer removes the pixel decoder and trains a causal transformer, used only during training, to predict the encoder embedding of the next observation from the RSSM history. The prediction is aligned with a stop-gradient target through a Barlow Twins loss, and the actor-critic and imagination pipeline of Dreamer are kept unchanged. Experiments on bsuite, MiniGrid Memory, eight DMLab tasks and twenty DeepMind Control tasks report the largest gains over DreamerV3 on long-delay memory and navigation tasks, with continuous-control results close to the Dreamer-family baselines. The authors conclude that the temporal placement of the representation target is a lever complementary to architectural memory.
Critical strengths
S1. The objective is simple and inexpensive. The predictor is used only during training, the actor-critic is unchanged, and the wall-clock overhead on Rooms Select is about two per cent (Table 8). NE-Dreamer improves over its DreamerV3 base on most of the eight DMLab memory and navigation tasks (Fig. 3), with the clearest margins on Explore Obstructed Goals and Rooms Keys Doors Puzzle. This strength matters for the assessment, since existing Dreamer-style agents can adopt the change at negligible cost.
Critical weaknesses
W1. The contribution is minor. The core objective predicts the next encoder embedding against a stop-gradient target without a decoder. With mean-squared error in place of Barlow Twins, this is the joint-embedding prediction loss of TD-MPC2 (Hansen et al., ICLR 2024), which the paper uses as a baseline without discussing the overlap. The same recipe has become common in 2026, for example LeWorldModel (LeWM, Maes et al., 2026) and Dreamer-CDP (Hauri and Zenke, 2026), a reconstruction-free Dreamer with a JEPA-style predictor. The one-step temporal shift that the paper presents as its key idea also exists in Dreamer. Its RSSM prior predicts the latent of the incoming observation from history and is trained towards a posterior computed from the observation embedding (Sec. 3.2). Without the transformer, the shift raises the mean score on two of four tasks and lowers it on the other two (Table 3, rows A and B). The authors need to demonstrate what NE-Dreamer adds beyond these components.
W2. The evaluation does not match the claims. For partial observability, it omits the memory benchmarks used by world models with explicit memory, such as Memory Maze and POPGym reported by R2I (Samsami et al., 2024). Occlusion settings such as ViZDoom or occluded manipulation are also absent. For the general claim in the title and Sec. 4.5, it omits Atari 100k and Crafter, both reported by DreamerV3. The baselines are also dated. DMLab compares against DreamerV3 only, and decoder-free baselines appear on just two tasks (Appendix B.1). No comparison is made with recent world models such as Horizon Imagination (ICLR 2026), the transformer world model of Dedieu et al. (ICML 2025), TWISTER (ICLR 2025) or LeWM. These benchmarks and baselines are needed before the generality of the method can be assessed.
Questions for authors
Q1 (clarification). Next-embedding prediction against a stop-gradient target is already the joint-embedding loss of TD-MPC2. Which component of NE-Dreamer is new beyond TD-MPC2, R2-Dreamer and the Dreamer prior? An ablation that replaces the causal transformer with a one-step latent transition predictor, as in TD-MPC2, would isolate the contribution of the history readout.
Q2 (requested experiments). Can the authors report results on Memory Maze, POPGym and an occlusion setting, as well as on Atari 100k and Crafter? The comparison should include recent world models such as Horizon Imagination, Dedieu et al. (2025), TWISTER and LeWM. The DMLab study should also cover all eight tasks against R2-Dreamer and DreamerPro. The reviewer's assessment depends on whether the gains hold under these settings.
4 comments
Q1. We implemented the requested one-step local predictor and separated target timing from direct history readout. The Transformer controls match capacity, loss, actions, readout positions and budget; a parameter-matched MLP tests the feedforward substitution. All configurations retain RSSM recurrence, with history attention providing direct access to earlier states.
Table 4. Finite-deck HigherLower, later 1M-decision cohort. Five seeds, 256 held-out episodes per policy; mean ± sample SD.
| Direct predictor input / architecture | Current target | Next target |
|---|---|---|
| Current RSSM state / Transformer | 0.486 ± 0.005 | 0.343 ± 0.071 |
| Current RSSM state / parameter-matched MLP | 0.487 ± 0.004 | 0.386 ± 0.064 |
| RSSM-state history / Transformer | 0.486 ± 0.005 | 0.502 ± 0.005 (NE) |
The MLP is matched in parameter count and data/update budgets. The history Transformer with the next-embedding target achieves the highest return in this comparison.
We also probed the current RSSM feature supplied to the actor using five frozen 10M models per method and common random-action trajectories.
Table 5. Remaining-rank-count recovery from the policy-visible RSSM feature. Values are R², reported as mean ± sample SD across models.
| Task | NE-Dreamer R² | R2-Dreamer R² | CDP-style R² | Dreamer R² |
|---|---|---|---|---|
| HigherLowerEasy | 0.975 ± 0.004 | 0.774 ± 0.014 | 0.547 ± 0.050 | 0.623 ± 0.046 |
| HigherLowerMedium | 0.883 ± 0.016 | 0.852 ± 0.039 | 0.720 ± 0.016 | 0.797 ± 0.011 |
These results show that remaining-rank counts are more recoverable from NE-Dreamer's policy-visible RSSM feature.
Q2. W2 reports the added original POPGym comparisons, and Q1 provides the requested local-predictor control. The additional DMLab comparison is reported below.
Table 6. DMLab final returns at 10M agent decisions, all five training seeds. Shared 64px BF16 protocol, 256 held-out episodes per policy; ± is sample SD.
| Task | NE-Dreamer | R2-Dreamer | CDP-style | Dreamer |
|---|---|---|---|---|
| Keys | 25.950 ± 13.926 | 30.306 ± 1.825 | 30.552 ± 1.416 | 22.745 ± 2.208 |
Four NE-Dreamer seeds leads other methods with scores 30.7–35.6, while one scores 1.3; R2/CDP-style have higher means across all five seeds.
Together, the paper and rebuttal cover four broad types of environments: structured capability and partial-observation tasks (bsuite and POPGym), 2D visual control (MiniGrid), 3D memory and navigation (DMLab), and continuous control (DMC).
Evaluating every requested baseline across all environments, configurations and training seeds would require an experimental grid beyond what we can complete during the rebuttal period. The added native POPGym comparison alone covers 15 settings × four methods × five training seeds.
The broader comparisons, matched predictor controls and representation diagnostics strengthen the case for a practical, task-dependent representation objective. We believe this evidence addresses your main concerns and ask you to reconsider your recommendation in light of these results. We welcome further discussion and are ready to run targeted follow-up experiments to address any remaining concerns.
The reviewer thanks the authors for the additional experiments. They were addressed to Reviewer_zr7h rather than to this review, and the reviewer has read them with interest. They cover part of W2 and of Q1, namely the five POPGym tasks at ten million agent decisions with five seeds, and the capacity-matched comparison of configurations A, B, C and NE. The matched control is a useful addition.
A number of the requests in Q2 remain open, and the reviewer would welcome them within the discussion period. Atari 100k and Crafter were asked for as the test of the generality claimed in the title and in Section 4.5. Memory Maze and an occlusion setting are also absent. None of Horizon Imagination, Dedieu et al. (2025), TWISTER or LeWM enters the new tables, where the baselines remain R2-Dreamer, configuration C and Dreamer. The DMLab study adds Keys alone rather than the remaining tasks against R2-Dreamer and DreamerPro. TD-MPC2 was the overlap named in W1, and the reviewer would value its place in the comparison table.
Two clarifications on the new results would also help. On DMLab Keys the table places R2-Dreamer at 30.306 +/- 1.825 ahead of NE-Dreamer at 25.950 +/- 13.926, while the accompanying text refers to four of the five tabulated seeds. The interaction table reports point differences without intervals, and the NE - C column is modest at +0.0171, +0.0291 and +0.0132, which leaves the reported interaction resting on B - A being negative. Intervals over the full seed set would settle how much the target choice contributes.
On the present evidence the reviewer's assessment is unchanged. The reviewer would reconsider it should the outstanding comparisons arrive within the discussion period.
The five-task comparison referenced in your follow-up comes from our reply to zr7h. Our two replies in your thread, both posted on 18 November, already include the broader requested POPGym comparison (15 configurations) and the local-predictor tests: W1/W2 · Q1/Q2.
- W2, Tables 2–3 report 15 native POPGym settings at 10M decisions with five seeds per method, including the added CDP-style objective in the shared agent. NE has the highest mean return on seven of the nine memory and reasoning settings among the four methods shown.
- W1, Table 1 includes TD-MPC2 in the first row of the component comparison. The paper also reports TD-MPC2 results on MiniGrid and results from its discrete-action port on two DMLab tasks.
- Q1, Tables 4–5 report the local-predictor control and RSSM probes. On HigherLower at 1M decisions, the next-target history Transformer scores 0.502, compared with 0.343 for the same-capacity local Transformer and 0.386 for the parameter-matched local MLP. All configurations retain RSSM recurrence.
“On DMLab Keys the table places R2-Dreamer at 30.306 ± 1.825 ahead of NE-Dreamer at 25.950 ± 13.926, while the accompanying text refers to four of the five tabulated seeds.”
We meant that NE-Dreamer has a higher return than baselines in four of the five paired seeds. Its low return on the remaining seed increases its standard deviation and lowers its five-seed mean below baselines.
Intervals for interaction table
| 10M task | NE−C (95% paired-t interval) | Interaction (95% paired-t interval) |
|---|---|---|
| HigherLowerEasy | +0.017 [−0.001, +0.035] | +0.081 [−0.036, +0.199] |
| RepeatPreviousEasy | +0.029 [−0.016, +0.075] | +0.066 [−0.179, +0.311] |
| AutoencodeEasy | +0.013 [−0.100, +0.127] | +0.067 [−0.079, +0.212] |
All intervals use five pairs unless otherwise noted.
We also ran visual-task comparisons in ViZDoom and Atari 100k within the remaining rebuttal time. The table reports results at 100,000 agent decisions as mean return ± sample SD across matched training seeds. My Way Home uses four development evaluation episodes per model; Atari uses 100 held-out evaluation episodes per policy.
| Environment | Training seeds per method | NE-Dreamer | R2-Dreamer | Dreamer |
|---|---|---|---|---|
| My Way Home — ViZDoom | 5 | 0.694 ± 0.359 | 0.368 ± 0.579 | 0.704 ± 0.511 |
| Alien — Atari 100k | 2 | 831.95 ± 257.32 | 1,102.05 ± 123.53 | 679.75 ± 19.16 |
| MsPacman — Atari 100k | 2 | 1,501.75 ± 32.17 | 2,190.85 ± 216.02 | 709.75 ± 155.35 |
On My Way Home, NE-Dreamer’s mean is 0.326 above R2-Dreamer’s and within 0.010 of Dreamer’s. In the preliminary two-seed Atari comparisons, NE-Dreamer exceeds Dreamer on both games and reaches 2.12× Dreamer’s mean return on MsPacman. R2-Dreamer leads both games.
Adding Crafter, Memory Maze, occlusion tests and native-agent comparisons would substantially expand the experimental scope of this work. Such breadth has not been covered by previous related work, with the exception of Dreamer. The ICLR Reviewer Guide asks that additional experiments be “limited in scope” and validate existing results. With addition to main paper experiments the matched control and the 15-setting comparison test our claim within the current family of methods. ICLR 2027 Reviewer Guide.
We believe this evidence supports acceptance for the paper’s representation contribution. Please take both replies in your thread into account.
Thank you for the review. We completed the requested local-predictor control and broadened the comparisons with original POPGym results and representation diagnostics, strengthening the evidence for our objective and clarifying its scope.
W1. Our contribution is a history-conditioned next-embedding representation objective for Dreamer, together with a controlled representation study of target timing and predictor history. The methods you identify establish the broader idea of predictive learning. The table below clarifies the differences in targets, predictor inputs and agent design.
Table 1. Target timing, predictor readout and agent design.
NE-Dreamer removes image reconstruction and adds a training-only causal predictor while retaining Dreamer's RSSM and actor–critic. The predictor is discarded at inference. Our contribution is this particular objective and its controlled study. A recurrent feature already contains history; direct history attention provides access to earlier states rather than introducing recurrence.
These methods differ substantially in architecture, training procedure, offline versus online learning, and action selection.
W2. The added original POPGym comparisons extend the evidence beyond Memory Length to sequence storage, delayed recall, history aggregation, associative memory and partial-observation control. We also added another decoder-free baseline, CDP-style, implementing continuous deterministic next-embedding prediction in the shared Dreamer agent. The R2/CDP-style comparisons test representation objectives within that agent; they are not full native-agent reproductions.
Tables 2–3 report final returns as mean ± sample SD across five training seeds, with 256 held-out episodes per policy.
Table 2. Original POPGym memory and reasoning tasks, 10M agent decisions.
NE-Dreamer achieves the highest mean return on seven of these nine settings. This broader comparison supports a useful representation objective with capability-dependent benefits.
Table 3. Other original POPGym tasks: position-only control, noisy observations and adaptation, 10M agent decisions.
These are native POPGym tasks. No single method leads on every task, and NE-Dreamer achieves competitive performance across these settings.