acceptodds
Under review as a conference paper at ICLR 2027

Next Embedding Prediction Makes World Models Stronger

Abstract

Dreamer-style world models process trajectories sequentially, yet their representation losses are usually anchored to the current observation through pixel reconstruction or same-timestep latent alignment. This mismatch is limiting under partial observability, where successful control depends on retaining compact information about cues, landmarks, and spatial context long after they disappear. We introduce NE-Dreamer, a decoder-free world model that moves the representation target one step into the future. Given the history-conditioned latent state, a lightweight causal predictor forecasts the next encoder embedding, and training aligns this prediction to a stop-gradient target with a redundancy-reduction objective. This converts representation learning from same-timestep description into history-conditioned prediction, directly pressuring the latent state to preserve information useful beyond the current frame. Across bsuite diagnostics, MiniGrid Memory, and DMLab, NE-Dreamer improves most strongly in long-delay memory and navigation settings, while matching strong decoder-based and decoder-free baselines on the DeepMind Control Suite. These results identify temporal target placement as a distinct and complementary lever to architectural memory for improving partially observable model-based reinforcement learning.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, representation learning, training in imagination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_sPFiOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies representation learning for Dreamer-style world models under partial observability. It argues that losses anchored to the current observation, such as pixel reconstruction, do not pressure the latent state to retain information needed later. The proposed NE-Dreamer removes the pixel decoder and trains a causal transformer, used only during training, to predict the encoder embedding of the next observation from the RSSM history. The prediction is aligned with a stop-gradient target through a Barlow Twins loss, and the actor-critic and imagination pipeline of Dreamer are kept unchanged. Experiments on bsuite, MiniGrid Memory, eight DMLab tasks and twenty DeepMind Control tasks report the largest gains over DreamerV3 on long-delay memory and navigation tasks, with continuous-control results close to the Dreamer-family baselines. The authors conclude that the temporal placement of the representation target is a lever complementary to architectural memory.

Critical strengths

S1. The objective is simple and inexpensive. The predictor is used only during training, the actor-critic is unchanged, and the wall-clock overhead on Rooms Select is about two per cent (Table 8). NE-Dreamer improves over its DreamerV3 base on most of the eight DMLab memory and navigation tasks (Fig. 3), with the clearest margins on Explore Obstructed Goals and Rooms Keys Doors Puzzle. This strength matters for the assessment, since existing Dreamer-style agents can adopt the change at negligible cost.

Critical weaknesses

W1. The contribution is minor. The core objective predicts the next encoder embedding against a stop-gradient target without a decoder. With mean-squared error in place of Barlow Twins, this is the joint-embedding prediction loss of TD-MPC2 (Hansen et al., ICLR 2024), which the paper uses as a baseline without discussing the overlap. The same recipe has become common in 2026, for example LeWorldModel (LeWM, Maes et al., 2026) and Dreamer-CDP (Hauri and Zenke, 2026), a reconstruction-free Dreamer with a JEPA-style predictor. The one-step temporal shift that the paper presents as its key idea also exists in Dreamer. Its RSSM prior predicts the latent of the incoming observation from history and is trained towards a posterior computed from the observation embedding (Sec. 3.2). Without the transformer, the shift raises the mean score on two of four tasks and lowers it on the other two (Table 3, rows A and B). The authors need to demonstrate what NE-Dreamer adds beyond these components.

W2. The evaluation does not match the claims. For partial observability, it omits the memory benchmarks used by world models with explicit memory, such as Memory Maze and POPGym reported by R2I (Samsami et al., 2024). Occlusion settings such as ViZDoom or occluded manipulation are also absent. For the general claim in the title and Sec. 4.5, it omits Atari 100k and Crafter, both reported by DreamerV3. The baselines are also dated. DMLab compares against DreamerV3 only, and decoder-free baselines appear on just two tasks (Appendix B.1). No comparison is made with recent world models such as Horizon Imagination (ICLR 2026), the transformer world model of Dedieu et al. (ICML 2025), TWISTER (ICLR 2025) or LeWM. These benchmarks and baselines are needed before the generality of the method can be assessed.

Questions for authors

Q1 (clarification). Next-embedding prediction against a stop-gradient target is already the joint-embedding loss of TD-MPC2. Which component of NE-Dreamer is new beyond TD-MPC2, R2-Dreamer and the Dreamer prior? An ablation that replaces the causal transformer with a one-step latent transition predictor, as in TD-MPC2, would isolate the contribution of the history readout.

Q2 (requested experiments). Can the authors report results on Memory Maze, POPGym and an occlusion setting, as well as on Atari 100k and Crafter? The comparison should include recent world models such as Horizon Imagination, Dedieu et al. (2025), TWISTER and LeWM. The DMLab study should also cover all eight tasks against R2-Dreamer and DreamerPro. The reviewer's assessment depends on whether the gains hold under these settings.

4 comments

Thank you for the review. We completed the requested local-predictor control and broadened the comparisons with original POPGym results and representation diagnostics, strengthening the evidence for our objective and clarifying its scope.

W1. Our contribution is a history-conditioned next-embedding representation objective for Dreamer, together with a controlled representation study of target timing and predictor history. The methods you identify establish the broader idea of predictive learning. The table below clarifies the differences in targets, predictor inputs and agent design.

Table 1. Target timing, predictor readout and agent design.

MethodAgent / backboneTargetDirect predictor inputRepresentation termImage decoder in training
TD-MPC2Latent-dynamics model + MPCNext observation's encoding across a multi-step unrollLatent transition unroll and actionsMSE, stop-gradient targetNo
TWISTERTransformer world model + actor–criticFuture stochastic latentsTransformer state and future actionsContrastive CPCYes
R2-DreamerDreamer RSSM + actor–criticCurrent encoder embeddingCurrent RSSM featureBarlow Twins, stop-gradient targetNo
Dreamer-CDPDreamer RSSM + actor–criticNext continuous encoder embeddingMLP on next deterministic recurrent stateCosine alignment, stop-gradient targetNo
LeWorldModelOffline latent model + MPCNext encoder embeddingCausal embedding history and actionsMSE + SIGReg, no stop-gradientNo
NE-DreamerDreamer RSSM + actor–criticNext encoder embeddingRSSM-state history and actionsBarlow Twins, stop-gradient targetNo

NE-Dreamer removes image reconstruction and adds a training-only causal predictor while retaining Dreamer's RSSM and actor–critic. The predictor is discarded at inference. Our contribution is this particular objective and its controlled study. A recurrent feature already contains history; direct history attention provides access to earlier states rather than introducing recurrence.

These methods differ substantially in architecture, training procedure, offline versus online learning, and action selection.

W2. The added original POPGym comparisons extend the evidence beyond Memory Length to sequence storage, delayed recall, history aggregation, associative memory and partial-observation control. We also added another decoder-free baseline, CDP-style, implementing continuous deterministic next-embedding prediction in the shared Dreamer agent. The R2/CDP-style comparisons test representation objectives within that agent; they are not full native-agent reproductions.

Tables 2–3 report final returns as mean ± sample SD across five training seeds, with 256 held-out episodes per policy.

Table 2. Original POPGym memory and reasoning tasks, 10M agent decisions.

TaskNE-DreamerR2-DreamerCDP-styleDreamer
AutoencodeMedium−0.433 ± 0.017−0.479 ± 0.007−0.485 ± 0.017−0.503 ± 0.003
AutoencodeEasy−0.327 ± 0.029−0.430 ± 0.058−0.414 ± 0.049−0.491 ± 0.022
RepeatPreviousEasy0.930 ± 0.0160.847 ± 0.0500.699 ± 0.4030.603 ± 0.029
RepeatPreviousMedium−0.346 ± 0.010−0.350 ± 0.082−0.482 ± 0.029−0.486 ± 0.012
HigherLowerEasy0.510 ± 0.0130.484 ± 0.0090.182 ± 0.1070.369 ± 0.019
HigherLowerMedium0.473 ± 0.0510.467 ± 0.0150.149 ± 0.1010.269 ± 0.152
ConcentrationEasy0.791 ± 0.016−0.043 ± 0.2830.791 ± 0.0140.806 ± 0.006
ConcentrationMedium−0.946 ± 0.029−0.745 ± 0.0400.429 ± 0.060−0.032 ± 0.193
MineSweeperEasy0.679 ± 0.0580.658 ± 0.0720.644 ± 0.0990.619 ± 0.054

NE-Dreamer achieves the highest mean return on seven of these nine settings. This broader comparison supports a useful representation objective with capability-dependent benefits.

Table 3. Other original POPGym tasks: position-only control, noisy observations and adaptation, 10M agent decisions.

TaskNE-DreamerR2-DreamerCDP-styleDreamer
PositionOnlyCartPoleEasy0.975 ± 0.0551.000 ± 0.0001.000 ± 0.0000.847 ± 0.335
NoisyPositionOnlyCartPoleEasy0.895 ± 0.0330.940 ± 0.0670.874 ± 0.0350.644 ± 0.127
NoisyPositionOnlyCartPoleMedium0.411 ± 0.0190.495 ± 0.0370.417 ± 0.0220.293 ± 0.022
PositionOnlyPendulumEasy0.910 ± 0.0010.910 ± 0.0020.910 ± 0.0030.907 ± 0.001
NoisyPositionOnlyPendulumMedium0.675 ± 0.0140.672 ± 0.0160.662 ± 0.0270.660 ± 0.007
MultiarmedBanditEasy0.519 ± 0.0330.197 ± 0.0820.554 ± 0.0360.064 ± 0.073

These are native POPGym tasks. No single method leads on every task, and NE-Dreamer achieves competitive performance across these settings.

Q1. We implemented the requested one-step local predictor and separated target timing from direct history readout. The Transformer controls match capacity, loss, actions, readout positions and budget; a parameter-matched MLP tests the feedforward substitution. All configurations retain RSSM recurrence, with history attention providing direct access to earlier states.

Table 4. Finite-deck HigherLower, later 1M-decision cohort. Five seeds, 256 held-out episodes per policy; mean ± sample SD.

Direct predictor input / architectureCurrent targetNext target
Current RSSM state / Transformer0.486 ± 0.0050.343 ± 0.071
Current RSSM state / parameter-matched MLP0.487 ± 0.0040.386 ± 0.064
RSSM-state history / Transformer0.486 ± 0.0050.502 ± 0.005 (NE)

The MLP is matched in parameter count and data/update budgets. The history Transformer with the next-embedding target achieves the highest return in this comparison.

We also probed the current RSSM feature supplied to the actor using five frozen 10M models per method and common random-action trajectories.

Table 5. Remaining-rank-count recovery from the policy-visible RSSM feature. Values are R², reported as mean ± sample SD across models.

TaskNE-Dreamer R²R2-Dreamer R²CDP-style R²Dreamer R²
HigherLowerEasy0.975 ± 0.0040.774 ± 0.0140.547 ± 0.0500.623 ± 0.046
HigherLowerMedium0.883 ± 0.0160.852 ± 0.0390.720 ± 0.0160.797 ± 0.011

These results show that remaining-rank counts are more recoverable from NE-Dreamer's policy-visible RSSM feature.

Q2. W2 reports the added original POPGym comparisons, and Q1 provides the requested local-predictor control. The additional DMLab comparison is reported below.

Table 6. DMLab final returns at 10M agent decisions, all five training seeds. Shared 64px BF16 protocol, 256 held-out episodes per policy; ± is sample SD.

TaskNE-DreamerR2-DreamerCDP-styleDreamer
Keys25.950 ± 13.92630.306 ± 1.82530.552 ± 1.41622.745 ± 2.208

Four NE-Dreamer seeds leads other methods with scores 30.7–35.6, while one scores 1.3; R2/CDP-style have higher means across all five seeds.

Together, the paper and rebuttal cover four broad types of environments: structured capability and partial-observation tasks (bsuite and POPGym), 2D visual control (MiniGrid), 3D memory and navigation (DMLab), and continuous control (DMC).

Evaluating every requested baseline across all environments, configurations and training seeds would require an experimental grid beyond what we can complete during the rebuttal period. The added native POPGym comparison alone covers 15 settings × four methods × five training seeds.


The broader comparisons, matched predictor controls and representation diagnostics strengthen the case for a practical, task-dependent representation objective. We believe this evidence addresses your main concerns and ask you to reconsider your recommendation in light of these results. We welcome further discussion and are ready to run targeted follow-up experiments to address any remaining concerns.

The reviewer thanks the authors for the additional experiments. They were addressed to Reviewer_zr7h rather than to this review, and the reviewer has read them with interest. They cover part of W2 and of Q1, namely the five POPGym tasks at ten million agent decisions with five seeds, and the capacity-matched comparison of configurations A, B, C and NE. The matched control is a useful addition.

A number of the requests in Q2 remain open, and the reviewer would welcome them within the discussion period. Atari 100k and Crafter were asked for as the test of the generality claimed in the title and in Section 4.5. Memory Maze and an occlusion setting are also absent. None of Horizon Imagination, Dedieu et al. (2025), TWISTER or LeWM enters the new tables, where the baselines remain R2-Dreamer, configuration C and Dreamer. The DMLab study adds Keys alone rather than the remaining tasks against R2-Dreamer and DreamerPro. TD-MPC2 was the overlap named in W1, and the reviewer would value its place in the comparison table.

Two clarifications on the new results would also help. On DMLab Keys the table places R2-Dreamer at 30.306 +/- 1.825 ahead of NE-Dreamer at 25.950 +/- 13.926, while the accompanying text refers to four of the five tabulated seeds. The interaction table reports point differences without intervals, and the NE - C column is modest at +0.0171, +0.0291 and +0.0132, which leaves the reported interaction resting on B - A being negative. Intervals over the full seed set would settle how much the target choice contributes.

On the present evidence the reviewer's assessment is unchanged. The reviewer would reconsider it should the outstanding comparisons arrive within the discussion period.

The five-task comparison referenced in your follow-up comes from our reply to zr7h. Our two replies in your thread, both posted on 18 November, already include the broader requested POPGym comparison (15 configurations) and the local-predictor tests: W1/W2 · Q1/Q2.

  • W2, Tables 2–3 report 15 native POPGym settings at 10M decisions with five seeds per method, including the added CDP-style objective in the shared agent. NE has the highest mean return on seven of the nine memory and reasoning settings among the four methods shown.
  • W1, Table 1 includes TD-MPC2 in the first row of the component comparison. The paper also reports TD-MPC2 results on MiniGrid and results from its discrete-action port on two DMLab tasks.
  • Q1, Tables 4–5 report the local-predictor control and RSSM probes. On HigherLower at 1M decisions, the next-target history Transformer scores 0.502, compared with 0.343 for the same-capacity local Transformer and 0.386 for the parameter-matched local MLP. All configurations retain RSSM recurrence.

“On DMLab Keys the table places R2-Dreamer at 30.306 ± 1.825 ahead of NE-Dreamer at 25.950 ± 13.926, while the accompanying text refers to four of the five tabulated seeds.”

We meant that NE-Dreamer has a higher return than baselines in four of the five paired seeds. Its low return on the remaining seed increases its standard deviation and lowers its five-seed mean below baselines.


Intervals for interaction table

10M taskNE−C (95% paired-t interval)Interaction (95% paired-t interval)
HigherLowerEasy+0.017 [−0.001, +0.035]+0.081 [−0.036, +0.199]
RepeatPreviousEasy+0.029 [−0.016, +0.075]+0.066 [−0.179, +0.311]
AutoencodeEasy+0.013 [−0.100, +0.127]+0.067 [−0.079, +0.212]

All intervals use five pairs unless otherwise noted.


We also ran visual-task comparisons in ViZDoom and Atari 100k within the remaining rebuttal time. The table reports results at 100,000 agent decisions as mean return ± sample SD across matched training seeds. My Way Home uses four development evaluation episodes per model; Atari uses 100 held-out evaluation episodes per policy.

EnvironmentTraining seeds per methodNE-DreamerR2-DreamerDreamer
My Way Home — ViZDoom50.694 ± 0.3590.368 ± 0.5790.704 ± 0.511
Alien — Atari 100k2831.95 ± 257.321,102.05 ± 123.53679.75 ± 19.16
MsPacman — Atari 100k21,501.75 ± 32.172,190.85 ± 216.02709.75 ± 155.35

On My Way Home, NE-Dreamer’s mean is 0.326 above R2-Dreamer’s and within 0.010 of Dreamer’s. In the preliminary two-seed Atari comparisons, NE-Dreamer exceeds Dreamer on both games and reaches 2.12× Dreamer’s mean return on MsPacman. R2-Dreamer leads both games.

Adding Crafter, Memory Maze, occlusion tests and native-agent comparisons would substantially expand the experimental scope of this work. Such breadth has not been covered by previous related work, with the exception of Dreamer. The ICLR Reviewer Guide asks that additional experiments be “limited in scope” and validate existing results. With addition to main paper experiments the matched control and the 15-setting comparison test our claim within the current family of methods. ICLR 2027 Reviewer Guide.

We believe this evidence supports acceptance for the paper’s representation contribution. Please take both replies in your thread into account.

Reviewer_zr7hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes NE-Dreamer, a decoder-free variant of Dreamer that changes when the representation target is taken. Most existing objectives ask the latent to explain the current observation. NE-Dreamer instead trains a small causal transformer to predict the next observation’s encoder embedding from the latent history, using a Barlow Twins loss against a stop-gradient target. The idea is that predicting the future forces the latent to keep information that is no longer visible. The method improves clearly on memory and navigation tasks that require holding information over long delays, such as bsuite memory tasks and DMLab, while staying competitive on standard continuous control.

Critical strengths
  1. The idea is simple and clean: moving the representation target one step into the future directly pushes the latent to keep information that will matter later. It requires no decoder, no negative samples, and only a light training-time predictor.

  2. The gains appear exactly where they should, on long-delay memory and navigation tasks, while standard control performance is maintained. The ablations are informative and show that temporal alignment and the stop-gradient are both necessary.

Critical weaknesses
  1. Incremental novelty. Predicting future latent representations is a well-explored idea (e.g., SPR, JEPA-style objectives, TWISTER). The main change here is the choice of target and loss, which makes the contribution feel modest.

  2. Selective gains. The improvement is concentrated on memory length, while memory capacity and exploration do not improve. NE-Dreamer also underperforms on some standard control tasks.

Questions for authors
  1. Would predicting embeddings further ahead help on tasks with longer delays or larger memory demands? If the gains vanish beyond one step, the method’s role as a general memory mechanism would be more limited than claimed.

  2. How much of the improvement comes from the extra transformer capacity rather than from the future target itself? A capacity-matched baseline, for example the same predictor trained on a current-step target, would make the source of the gain clearer.

4 comments

Thank you for the review. We completed the requested capacity and horizon controls and broadened the comparisons, strengthening the evidence for our representation objective and clarifying its scope.


W1. Our contribution is a history-conditioned next-embedding representation objective for Dreamer, together with a controlled representation study of target timing and predictor history.

SPR, JEPA-style objectives and TWISTER establish relative predictive-learning idea, with architecture and training changes. Our main contribution - representation objective that extend memory capabilities, while keeping all other parts of Dreamer unchanged. We provide deeper comparison in table below.

Table 1. Comparison of mentioned methods in comparison with Dreamer-style world models.

MethodDreamer-derived?Change in Dreamer architecture?Change in Dreamer Representation objective?
SPRNoN/AN/A
JEPA-style pretrainingNoN/AN/A
TWISTERYesYesYes
NE-DreamerYesNoYes

NE-Dreamer uses next-encoder-embedding alignment as a decoder-free objective while retaining Dreamer’s RSSM and actor–critic. TWISTER combines contrastive future-latent prediction with reconstruction and a transformer state-space model.

With capacity, loss, action inputs and training budget matched, shifting to a next-embedding target improves mean return with history attention but lowers it with current-state inputs on all three diagnostic tasks. This observed interaction supports the importance of combining target timing with temporal context, beyond additional predictor capacity.

The added storage, recall, history-aggregation and adaptation results broaden the objective’s practical relevance beyond memory length, while showing that benefits remain task-dependent.

W2. The added results extend beyond the original Memory Length diagnostic to complementary memory capabilities, visual control, noisy observations and adaptation. Scores are final returns, mean ± sample SD across five training seeds. Evaluations use 256 held-out episodes per policy. C is the same decoder-free history Transformer as NE, with the same capacity and Barlow loss, trained on the current embedding instead of the next one, notation here follows paper.

Table 2. Visual control, noisy observations and adaptation.

Task / settingWhat the task testsNE-DreamerR2-DreamerCDreamer
DMLab KeysExploration and matching task25.950 ± 13.92630.306 ± 1.8253.784 ± 2.27922.745 ± 2.208
Noisy position-only CartPole, σ=0.1, 1MState estimation from noisy, partial observations.0.968 ± 0.0230.947 ± 0.0130.942 ± 0.0460.469 ± 0.132
Noisy position-only Pendulum, 1MInfer hidden angular velocity for swing-up.0.698±0.010‾\underline{0.698 ± 0.010}0.677 ± 0.0300.701 ± 0.0090.639 ± 0.019
Resampled Multiarmed Bandit, 1MAdapt to arm payoffs resampled each episode.0.346 ± 0.0880.209 ± 0.1000.034 ± 0.0420.041 ± 0.072

On different visual tasks not directly related to memory capacity NE-Dreamer matches or exceed other methods. Keys shows a benefit over the matched current-target control, and win all methods on 4 from training 5 seeds, and diverge on the last one. The noise and bandit settings extend the evidence to state estimation and within-episode adaptation.

Table 3. Memory-related capabilities: 10M agent decisions for every row. Task descriptions follow the POPGym paper and its environment implementations.

Task / settingWhat the task testsNE-DreamerR2-DreamerCDreamer
AutoencodeMediumSequence storage and reverse-order recall−0.433 ± 0.017−0.479 ± 0.007−0.486 ± 0.004−0.503 ± 0.003
AutoencodeEasySequence storage and reverse-order recall.−0.327 ± 0.029−0.430 ± 0.058−0.340 ± 0.103−0.491 ± 0.022
RepeatPreviousEasyFixed-delay recall while new cards arrive.0.930 ± 0.0160.847 ± 0.0500.901 ± 0.0260.603 ± 0.029
HigherLowerEasyHistory aggregation and card counting for next-rank prediction.0.510 ± 0.0130.484 ± 0.0090.493 ± 0.0050.369 ± 0.019
ConcentrationEasyAssociative memory of hidden card values and locations.0.791±0.016‾\underline{0.791 ± 0.016}−0.043 ± 0.2830.090 ± 0.9400.806 ± 0.006

NE-Dreamer has the highest mean on four of the five memory settings in Table 3, and competitive performance on last memory setting, which supports the objective for memory-dependent tasks.

Q1. Forecast offset and required memory duration are distinct: even a next-observation target may require information from much earlier in the history. We compared forecast offsets on observation-gap Pendulum using one head and equal capacity.

Task / budgetOffset 1 final returnOffset 5 final returnPaired gain ± SD95% paired-t CI
Observation-gap Pendulum, 1M0.636 ± 0.0320.666 ± 0.011+0.0300 ± 0.0360+0.0300 ± 0.0447

The performance change is modest and the paired interval includes zero. This demonstrates feasibility of a longer forecast offset in this setting, rather than improved performance under larger memory demands. A systematic study of prediction horizons remains outside the scope of this work.

Q2. We separated target choice from predictor capacity using four configurations with the same Transformer capacity, loss, action inputs and training budget:

ConfigurationPredictor inputPrediction target
ACurrent RSSM stateCurrent embedding
BCurrent RSSM stateNext embedding
CHistory of RSSM statesCurrent embedding
NE-Dreamer (Ours)History of RSSM statesNext embedding

C is the capacity-matched control you suggest. Both input settings retain Dreamer’s recurrent memory; history attention additionally gives the predictor direct access to earlier states.

Results at 10M agent decisions. NE−C measures the effect of changing the target with history attention. B−A measures the same change for the local predictor. Their difference measures whether the next target helps more with history attention.

EnvironmentNE − CB − AInteraction: (NE−C)−(B−A)
HigherLowerEasy+0.0171−0.0642+0.0813
RepeatPreviousEasy+0.0291−0.0367+0.0658
AutoencodeEasy+0.0132−0.0289+0.0421

In this comparison, the next target improves mean return with history attention and lowers it with the local predictor, while showing synergy effect for both changes. These results support contribution from target choice beyond additional predictor capacity.


The matched controls and broader comparisons strengthen the case for a focused representation-learning contribution. We believe this evidence supports acceptance and ask you to reconsider your rating accordingly. We welcome further discussion and are ready to run targeted follow-up experiments to address any remaining concerns.

Thanks for the experiments. The gain attributable to the next-step target itself seems small, and the novelty concern remains, so I’m keeping my score.

NE-Dreamer’s novelty lies in a representation objective for imagination that couples causal history with next-embedding prediction. This objective has not appeared in prior work on imagination-based models.

The matched 2×2 experiment shows synergy on HigherLower at 1M decisions. With the same Transformer capacity, switching to the next-embedding target increases mean final return from 0.486 to 0.502 when the predictor has direct access to earlier states, but lowers it from 0.486 to 0.343 when the predictor reads only the current recurrent state. The target-by-history interaction is +0.1592 ± 0.0859 across five seeds (paired 95% t interval). A parameter-matched local MLP with the next-embedding target scores 0.386. These results support both the synergy between the two components and the method’s practical value.

The direct gain from the next-embedding target reaches +0.050 on RepeatPreviousEasy at 50M decisions. NE-Dreamer scores 0.979 vs. 0.930 for C, with a gain in all five seed pairs (paired 95% t interval: +0.0496 ± 0.0226). C uses the same history Transformer and Barlow Twins loss, with the current embedding as its target. NE-Dreamer’s 50M runs continue the five runs that averaged 0.930 at 10M. Together, these results show clear gains for the full NE-Dreamer recipe over the baselines and the simple local next-embedding predictor, further supporting our representation objective.

Mean final-return differences at matched budgets, using five seeds and 256 evaluation episodes per policy:

POPGym settingNE−CNE−R2-DreamerNE−Dreamer
RepeatPreviousEasy, 50M+0.050+0.161+0.346
MultiarmedBanditEasy, 10M+0.548+0.322+0.455
AutoencodeMedium, 10M+0.053+0.046+0.070
AutoencodeEasy, 50M−0.066+0.120+0.189

Additional 10M comparison on RepeatPreviousHard

MethodReturn at 10MNE advantageNE improvement (%)
NE-Dreamer−0.4580——
C−0.5063+0.0483+9.54%
R2-Dreamer−0.4966+0.0386+7.77%
Dreamer−0.4956+0.0376+7.59%

We also added RepeatPreviousHard, the hardest version of POPGym’s RepeatPrevious memory task, where NE-Dreamer shows a clear lead. Percentages express the return gain relative to the absolute baseline return.

Comments on the paper

3 comments

Thank you for the review. We broadened the decoder-free comparisons, added matched target controls and loss-weight sensitivity results, and clarified the statistical limits of the evidence.

W1. The added comparisons with R2-Dreamer test whether NE-Dreamer's gains extend beyond decoder removal.

Table 1. Native POPGym comparisons at 10M agent decisions. Scores are final returns, reported as mean ± sample SD across five training seeds, with 256 held-out episodes per policy.

Task / capabilityNE-DreamerR2-DreamerDreamer
AutoencodeEasy / sequence storage and recall−0.327 ± 0.029−0.430 ± 0.058−0.491 ± 0.022
RepeatPreviousEasy / delayed recall0.930 ± 0.0160.847 ± 0.0500.603 ± 0.029
HigherLowerEasy / history aggregation0.510 ± 0.0130.484 ± 0.0090.369 ± 0.019
ConcentrationEasy / associative memory0.791 ± 0.016−0.043 ± 0.2830.806 ± 0.006
PositionOnlyCartPoleEasy / partial-state control0.975 ± 0.0551.000 ± 0.0000.847 ± 0.335
MineSweeperEasy / hidden-board inference0.679 ± 0.0580.658 ± 0.0720.619 ± 0.054

NE-Dreamer achieves the highest mean return among the three methods on four of these six additional tasks, which test different model capabilities, and remains competitive on the other two. These results show benefits beyond removing the decoder.

Table 2. DMLab final returns at 10M agent decisions, using all five training seeds.

TaskNE-DreamerR2-DreamerDreamer
KeysDoorsPuzzle25.950 ± 13.92630.306 ± 1.82522.745 ± 2.208

We added DMLab's KeysDoorsPuzzle to broaden the baseline comparison. On Keys, four NE-Dreamer seeds reach 30.7–35.6, while one fails at 1.3. In the originally reported DMLab Rooms Select experiment, R2-Dreamer's lower mean likewise reflected poor performance in two of five seeds.

W2. The benchmark tables use reference loss weights of NE=1 and R2=0.05; the latter is R2-Dreamer's published default, reported in Appendix F.

We tested three weights per method on Keys using the same development budget: 5M agent decisions, two training seeds and 64 final evaluation episodes per policy. Final returns are reported as mean ± sample SD across the two development seeds.

Table 3. Keys loss-weight sensitivity.

MethodLoss weightFinal return
NE-Dreamer0.129.313 ± 1.061
NE-Dreamer128.805 ± 2.906
NE-Dreamer1033.219 ± 1.326
R2-Dreamer0.00529.047 ± 1.613
R2-Dreamer0.0525.148 ± 4.563
R2-Dreamer0.530.844 ± 2.210

These results indicate some sensitivity to the loss weight, and the best value may vary across tasks. Neither method shows a large performance drop within the tested range.

Q1. The bsuite results were obtained over five seeds using the evaluation code from the original bsuite repository. We followed its default evaluation configuration, which does not automatically report uncertainty.

Q2. The added controls separate target timing from direct predictor access to history. The four Transformer configurations match capacity, loss, action inputs and training budget:

ConfigurationDirect predictor inputTarget
ACurrent RSSM stateCurrent embedding
BCurrent RSSM stateNext embedding
CRSSM-state historyCurrent embedding
NE-DreamerRSSM-state historyNext embedding

All configurations retain RSSM recurrence. NE−C measures the effect of changing the prediction target with history, B−A measures the same effect with a local predictor, and their difference measures the interaction.

Table 4. Contrasts at 10M agent decisions.

TaskNE−CB−AInteraction
HigherLowerEasy+0.0171−0.0642+0.0813
RepeatPreviousEasy+0.0291−0.0367+0.0658
AutoencodeEasy+0.0132−0.0289+0.0421

Across these comparisons, predicting the next embedding improves mean return with history attention and lowers it with the local predictor, yielding positive interaction estimates. These results support a contribution from target choice beyond additional predictor capacity.

Q3. We extended the evaluation to different types of tasks, as detailed in W1. The POPGym results do not follow a simple memory-versus-control split: NE-Dreamer leads several memory tasks, although it is not uniformly strongest, and remains competitive on control tasks. We retain the DMC regressions as limitations on standard fully observed environments. At the same time, the competitive POPGym control results argue against a general loss of control performance.

The broader comparisons, explicit sensitivity results and matched controls strengthen the case for a practical, capability-dependent representation objective. We believe this evidence addresses your main concerns and ask you to adjust your score in light of these results. We welcome further discussion and are ready to run targeted follow-up experiments to address any remaining concerns.

We thank reviewers and AC for rebuttal stage. Below is a table of addressed concerns and where they were addressed.

ConcernReviewerWhere addressed
Contribution relative to prior workzr7h, sPFizr7h W1; sPFi W1/Q1
Capacity, target timing and local predictorzr7h, 1g6N, sPFizr7h Q2; 1g6N Q2; sPFi Q1
Broader task coverage and stronger baselineszr7h, 1g6N, sPFizr7h W2; 1g6N W1/Q3; sPFi W2/Q2
Policy-visible informationsPFisPFi Q1
Loss-weight sensitivity1g6N1g6N W2
Seed counts and variability1g6N1g6N W1/Q1; sPFi Q2
Prediction beyond one stepzr7hzr7h Q1

Additionally, we updated the paper based on the reviews and rebuttal results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.