- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DreamSPI is on-policy model-based agent built on DeepSPI. After each batch of real experience it freezes the encoder and the world model, rolls out short imagined trajectories in latent space and updates only the linear actor-critic head on them. The latent space uses continuous token embeddings and the world model predicts discrete codes from a finite codebook. The authors tie this design to safe policy improvement (SPI) bounds. On the full Atari-57 benchmark DreamSPI reaches a higher IQM human-normalized score than DeepSPI, PPO and DeepMDP (0.81 vs 0.72 for DeepSPI), and it also comes out on top on the Octax suite. Ablations on five Atari games cover the representation the optimizer and other design choices.
Critical strengths
- Latent imagination without a replay buffer, working on all of Atari-57. The DeepSPI authors found imagination less effective than real data alone. This paper makes it work at full scale at about 2.4 hours per run on one H200. The evaluation is careful - 57 games, 5 seeds, bootstrap intervals, many negative ablations and an honest account of the games where the method loses. Code is provided.
- A representation choice that matters a lot. In the ablations pairing continuous token embeddings for the actor with a discrete codebook for the model reaches an Atari-5 IQM of about 2.07 against about 1.2 for Hadamard embeddings and below 0.4 for the categorical and DeepSPI-style representations. Other latent-model agents could use that pairing.
Critical weaknesses
- It is not shown that the gain comes from imagination. The Atari-57 baselines are older DeepSPI runs with Adam, ReLU, no LayerNorm and a different discount and number of environments, while DreamSPI uses Kron, Mish and LayerNorm. DreamSPI's ablations show that it depends heavily on these choices (switching to Adam drops the Atari-5 IQM from 2.07 to 0.10). The closest control is DreamSPI without imagination. It is run only on Atari-5 with 3 seeds and there it does about as well (IQM 1.9 vs 2.07, overlapping intervals, better median). So the Atari-57 result may come from a better training setup rather than from imagination. On Octax, DreamSPI picks one of two configurations per game while the baselines use one and its interval still overlaps PPO's. This is the main reason for my score.
- The SPI theory does not cover the agent as trained. The bounds need each policy update to stay within , but the agent clips at 0.1 on real data and 0.01 on imagined data, and the logged KL is two to three orders of magnitude above what that radius allows. The codebook Lipschitz bound only becomes useful when all codebook entries are almost equally far apart, and that condition is never checked. The theory is good motivation, but the title and the "principled design" claim present it as a guarantee.
Questions for authors
Clarifications
- Can you run DreamSPI without imagination (with its own tuned learning rates) on Atari-57, and ideally the baselines with Kron, Mish and LayerNorm? Imagination also gives the actor-critic about 16 times more updates per iteration, so a control with the same number of updates on real data would help too. If DreamSPI stays clearly ahead, I would raise my score.
- Were the Atari-57 baselines rerun, or taken from the DeepSPI paper? Their confidence intervals look almost zero width, which is surprising with 5 seeds.
- How were the two Octax configurations assigned to games, and what do the results look like with one configuration for every game?
Suggestions (future work, not decisive)
- Treat the theory as motivation, and report the actual policy-change radius and codebook distances during training.
- Related model-based methods with improvement guarantees or Lipschitz models should be discussed:
- ME-TRPO (Kurutach et al., 2018)
- SLBO (Luo et al., 2019)
- On-Policy Model Errors in RL (Fröhlich et al., 2022)
- MnM (Eysenbach et al., 2022)
- Dreaming (Okada & Taniguchi, 2021).
- Minor - some Octax reference scores are almost equal to random, which can inflate normalized scores. Section titles such as "The Golden Ticket to Pure Imagination" make the paper harder to navigate.
6 comments
References
[1] Roger Creus Castanyer, Johan S. Obando-Ceron, Lu Li, Pierre-Luc Bacon, Glen Berseth, Aaron C. Courville, Pablo Samuel Castro: Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning. NeurIPS 2025
[2] Pablo Samuel Castro: The Formalism-Implementation Gap in Reinforcement Learning Research. CoRR abs/2510.16175 (2025)
[3] Jakub Grudzien Kuba, Christian A. Schröder de Witt, Jakob N. Foerster: Mirror Learning: A Unifying Framework of Policy Optimisation. ICML 2022: 7825-7844
[4] Carles Gelada et al. DeepMDP: Learning Continuous Latent Space Models for Representation Learning. ICML, 2019.
[5] Ishaan Gulrajani et al. Improved Training of Wasserstein GANs. NeurIPS, 2017.
[6] Cem Anil, James Lucas, and Roger Grosse. Sorting Out Lipschitz Function Approximation. ICML, 2019.
[7] Jacob E. Kooi, Mark Hoogendoorn, Vincent François-Lavet: Latent Assistance Networks: Rediscovering Hyperbolic Tangents in RL. CoRR abs/2406.09079 (2024)
[8] Matthew Aitchison, Penny Sweetser, Marcus Hutter: Atari-5: Distilling the Arcade Learning Environment down to Five Games. ICML 2023: 421-438
Revision: additional experiments
As you requested, we performed a large number of new experiments to address your concerns. You may find them in our revised paper. All our additions are in purple. All our new experiments are over 5 seeds.
- We updated Figure 7(a) (DreamSPI dyna vs. no imagination vs. imagination only) over seeds.
- We updated the broken Figures (4, 5, 16);
- We performed DreamSPI without imagination on Atari 57, and we fully analyzed the effect of isolating the contribution of imagination to DreamSPI's performance in Appendix P. The new Figures 45 to 48 compare DreamSPI with DeepSPI including all our contributions except from imagination. We performed an analysis by splitting games into group to identify where imagination helps exactly when compared to the "no-imagination" variant ("augmented" DeepSPI).
- In the same analysis, we added an Atari 5 analysis isolating our added components: vanilla [DeepSPI; PPO]+Kron and "augmented" PPO, i.e., PPO+[Kron, Mish, LayerNorm, token emeddings+codebook, simple projections]. We included PPO because DeepSPI is roughly "PPO + world model learning."
- Appendix P.3 includes an analysis where we replace imagined updates with the same number of updates on the real batch. We performed the analysis on Atari-57 and we've also split the set to identify exactly where each variant helps or hurts.
- Appendix L.5.1 now includes a clear explanation on the Octax configuration, with plots showing both the transition losses of the games for which we switched to a Hadamard representation. We also provided the Octax aggregation plots for the same configuration on all games, which barely changes the result (roughly 0.02 IQM). As mentioned, the motivation was mainly to fix "broken" transition losses.
- Appendix N.1 plots policy ratios during training (Figures 36 and 37) for real and imagination steps, and we show in mean, median, IQM that the ratios lie in the theoretical neighbourhood with our proposed configuration.
- Appendix N.2 provides a new Atari 10 evaluation with tighter clipping (where we use the theoretical clip
epsilon=0.005instead of our proposed practical configuration). The results show that clipping close to the theoretical radius does not necessarily hurt the performance and is thus a viable choice when a closer theoretical alignment matters. In practice, this is a trade-off to explore according to the environment. - We unfortunately can't report codebook distances during training: exact extrema are intractable to compute during training as the space of the categorical codes ( in Fig.3) is (the transition model predicts categorical variables, each with possible classes).
Thank you for the careful reading and for recognizing our contribution to latent imagination, our representation design, and the care taken in our experiments. Below, we address your concerns.
The gains of DreamSPI. Our experiments compare DreamSPI with baselines using their best configuration reported in the corresponding papers. Fig. 7(a) compares DreamSPI with DeepSPI augmented by all our improvements, obtained by removing imagination. The ablations performed in Sect. 4 show those make latent imagination possible in the deep SPI setting: we pass from DreamSPI-v0 (poor performance) to DreamSPI (ours), where imagination alone (
dyna=False) provides consistent performance in a non-trivial setting. Ultimately, that was the main goal of the paper: enable latent imagination witin the SPI framework.That being said, we expect DeepSPI to be improved by our contributions:
Furthermore, we see the notable improvements of DeepSPI, augmented by our contributions, as a positive result as this is consistent with our theoretical motivation. We believe that understanding where imagination helps and where it does not is more informative than aggregate scores alone (cf. [2]). We will therefore add a per-game analysis of DreamSPI (dyna) vs. DeepSPI (augmented) to clarify its benefits and limitations.
SPI bounds. DreamSPI builds upon PPO, which has a rigorous interpretation within mirror-learning theory (cf. [3]). Our approach inherits this principled foundation and extends it to world-model learning and latent imagination. As in DeepSPI, we approximate the exact SPI constraints via PPO-style clipping. The KL curves approximate avg. policy displacement but doesn’t identify the ratio deviations; for this reason, we are running experiments to measure the magnitude of policy-ratio with DreamSPI in practice. We’re also running experiments on a tighter clipping to measure the performance trade-off of a more constrained setting (
epsilon=5e-3, aligned with the theoretical clip). Note that even tighter, PPO-style clipping doesn’t enforce the pointwise constraint. That’s the usual trade-off between theory and practice; by “principled” algorithm, we thus mean that the theory guides our design and identifies the quantities to control. We’ll clarify this in the main text.Codebook design. Claim 1/Thm. 1 establish finite regularity constants without requiring equidistant embeddings, although satisfying the SPI conditions also depends on their size. To better control regularity, we also ran extensive experiments and theoretical analysis with gradient penalties [4], WGAN-GP [5], and spectral and Björck normalization [6] (Appendix M.2). These methods aim to limit how much the model’s predictions change between nearby latent states. Some variants were promising, but none matched DreamSPI (Fig. 31). These results illustrate a trade-off: stronger regularity control can bring the model closer to the theory, but comes at a cost in performance.
This also motivated our second Octax configuration. As reported in Appendix F.2, we observed high-magnitude transition losses on some games. We therefore used Hadamard representation [7] to bound the encoder outputs and model predictions, limiting latent distances and the codebook diameter. While stronger constraints can reduce performance, they offer a way to control these distances when training becomes unstable; we also pointed to this in Sect. 4. This illustrates how our theory guides practical design choices.
Question 1: We acknowledge your concern on Fig. 7(a). We are currently running those experiments. But we’d like to stress that Atari-57x5 represents ~450h on A40s. Given our available compute, we are only running Atari-57 on a selected subset of your request and do the others on Atari-5x5 seeds. Note that Atari-5 is the right benchmark to perform ablations: it provides a representative subset of Atari [8].
Question 2: Thanks for spotting this. We relaunched the experiments for the submission, but indeed the Figure is broken, related to the way we gathered the data. We’ve re-generated it and corrected the plot in the upcoming revision.
Question 3: As mentioned in Appendix F.2 and above, we used the alternative representation to avoid exploding transition losses (the exact two-split is in F2). We'll provide the requested Figures in an upcoming revision. Note that the baselines do not suffer from this explosion, coming from the representation of DreamSPI.
We finally thank you for the suggestions, we will take them into account when revising our paper.