acceptodds
Under review as a conference paper at ICLR 2027

DreamSPI: World Models for Safe Policy Improvement through Latent Imagination

Abstract

In model-based reinforcement learning (MBRL), world models promise to support policy improvement with fewer environment interactions. Safe policy improvement (SPI) can reason about such updates, but its guarantees are local to the behavior policy and difficult to combine with changing encoders, deep representations, and replay-heavy model learning. We introduce DreamSPI, a principled on-policy MBRL algorithm that translates SPI locality into practical update proxies while updating the encoder, model, and latent actor-critic. It tokenizes spatial encoder outputs, predicts categorical next-token embeddings, imagines fixed-horizon latent rollouts from fresh vectorized data, and stabilizes coupled updates with Kronecker-factored optimization. On ALE-57 and Octax, DreamSPI outperforms strong baselines while ablations identify on-policy latent imagination and our token-embedding interface as the key design.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, world models, representation learning, safe policy improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LHASOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DreamSPI is on-policy model-based agent built on DeepSPI. After each batch of real experience it freezes the encoder and the world model, rolls out short imagined trajectories in latent space and updates only the linear actor-critic head on them. The latent space uses continuous token embeddings and the world model predicts discrete codes from a finite codebook. The authors tie this design to safe policy improvement (SPI) bounds. On the full Atari-57 benchmark DreamSPI reaches a higher IQM human-normalized score than DeepSPI, PPO and DeepMDP (0.81 vs 0.72 for DeepSPI), and it also comes out on top on the Octax suite. Ablations on five Atari games cover the representation the optimizer and other design choices.

Critical strengths
  1. Latent imagination without a replay buffer, working on all of Atari-57. The DeepSPI authors found imagination less effective than real data alone. This paper makes it work at full scale at about 2.4 hours per run on one H200. The evaluation is careful - 57 games, 5 seeds, bootstrap intervals, many negative ablations and an honest account of the games where the method loses. Code is provided.
  2. A representation choice that matters a lot. In the ablations pairing continuous token embeddings for the actor with a discrete codebook for the model reaches an Atari-5 IQM of about 2.07 against about 1.2 for Hadamard embeddings and below 0.4 for the categorical and DeepSPI-style representations. Other latent-model agents could use that pairing.
Critical weaknesses
  1. It is not shown that the gain comes from imagination. The Atari-57 baselines are older DeepSPI runs with Adam, ReLU, no LayerNorm and a different discount and number of environments, while DreamSPI uses Kron, Mish and LayerNorm. DreamSPI's ablations show that it depends heavily on these choices (switching to Adam drops the Atari-5 IQM from 2.07 to 0.10). The closest control is DreamSPI without imagination. It is run only on Atari-5 with 3 seeds and there it does about as well (IQM 1.9 vs 2.07, overlapping intervals, better median). So the Atari-57 result may come from a better training setup rather than from imagination. On Octax, DreamSPI picks one of two configurations per game while the baselines use one and its interval still overlaps PPO's. This is the main reason for my score.
  2. The SPI theory does not cover the agent as trained. The bounds need each policy update to stay within ϵ≤1/γ−1≈0.005\epsilon \le 1/\gamma - 1 \approx 0.005, but the agent clips at 0.1 on real data and 0.01 on imagined data, and the logged KL is two to three orders of magnitude above what that radius allows. The codebook Lipschitz bound only becomes useful when all codebook entries are almost equally far apart, and that condition is never checked. The theory is good motivation, but the title and the "principled design" claim present it as a guarantee.
Questions for authors

Clarifications

  1. Can you run DreamSPI without imagination (with its own tuned learning rates) on Atari-57, and ideally the baselines with Kron, Mish and LayerNorm? Imagination also gives the actor-critic about 16 times more updates per iteration, so a control with the same number of updates on real data would help too. If DreamSPI stays clearly ahead, I would raise my score.
  2. Were the Atari-57 baselines rerun, or taken from the DeepSPI paper? Their confidence intervals look almost zero width, which is surprising with 5 seeds.
  3. How were the two Octax configurations assigned to games, and what do the results look like with one configuration for every game?

Suggestions (future work, not decisive)

  1. Treat the theory as motivation, and report the actual policy-change radius and codebook distances during training.
  2. Related model-based methods with improvement guarantees or Lipschitz models should be discussed:
  • ME-TRPO (Kurutach et al., 2018)
  • SLBO (Luo et al., 2019)
  • On-Policy Model Errors in RL (Fröhlich et al., 2022)
  • MnM (Eysenbach et al., 2022)
  • Dreaming (Okada & Taniguchi, 2021).
  1. Minor - some Octax reference scores are almost equal to random, which can inflate normalized scores. Section titles such as "The Golden Ticket to Pure Imagination" make the paper harder to navigate.
6 comments

Thank you for the careful reading and for recognizing our contribution to latent imagination, our representation design, and the care taken in our experiments. Below, we address your concerns.

The gains of DreamSPI. Our experiments compare DreamSPI with baselines using their best configuration reported in the corresponding papers. Fig. 7(a) compares DreamSPI with DeepSPI augmented by all our improvements, obtained by removing imagination. The ablations performed in Sect. 4 show those make latent imagination possible in the deep SPI setting: we pass from DreamSPI-v0 (poor performance) to DreamSPI (ours), where imagination alone (dyna=False) provides consistent performance in a non-trivial setting. Ultimately, that was the main goal of the paper: enable latent imagination witin the SPI framework.

That being said, we expect DeepSPI to be improved by our contributions:

  • the tokenization+codebook representations come with regularity guarantees (Claim 1), a key conditions for SPI (Eq. 6);
  • the simple projection layer provides controlled policy updates (Claim 2), which helps preventing drift, as required by SPI (Eq. 2); and
  • Kron may also benefit DeepSPI, whose world-model also faces non-stationarity [1].

Furthermore, we see the notable improvements of DeepSPI, augmented by our contributions, as a positive result as this is consistent with our theoretical motivation. We believe that understanding where imagination helps and where it does not is more informative than aggregate scores alone (cf. [2]). We will therefore add a per-game analysis of DreamSPI (dyna) vs. DeepSPI (augmented) to clarify its benefits and limitations.

SPI bounds. DreamSPI builds upon PPO, which has a rigorous interpretation within mirror-learning theory (cf. [3]). Our approach inherits this principled foundation and extends it to world-model learning and latent imagination. As in DeepSPI, we approximate the exact SPI constraints via PPO-style clipping. The KL curves approximate avg. policy displacement but doesn’t identify the ratio deviations; for this reason, we are running experiments to measure the magnitude of policy-ratio with DreamSPI in practice. We’re also running experiments on a tighter clipping to measure the performance trade-off of a more constrained setting (epsilon=5e-3, aligned with the theoretical clip). Note that even tighter, PPO-style clipping doesn’t enforce the pointwise constraint. That’s the usual trade-off between theory and practice; by “principled” algorithm, we thus mean that the theory guides our design and identifies the quantities to control. We’ll clarify this in the main text.

Codebook design. Claim 1/Thm. 1 establish finite regularity constants without requiring equidistant embeddings, although satisfying the SPI conditions also depends on their size. To better control regularity, we also ran extensive experiments and theoretical analysis with gradient penalties [4], WGAN-GP [5], and spectral and Björck normalization [6] (Appendix M.2). These methods aim to limit how much the model’s predictions change between nearby latent states. Some variants were promising, but none matched DreamSPI (Fig. 31). These results illustrate a trade-off: stronger regularity control can bring the model closer to the theory, but comes at a cost in performance.

This also motivated our second Octax configuration. As reported in Appendix F.2, we observed high-magnitude transition losses on some games. We therefore used Hadamard representation [7] to bound the encoder outputs and model predictions, limiting latent distances and the codebook diameter. While stronger constraints can reduce performance, they offer a way to control these distances when training becomes unstable; we also pointed to this in Sect. 4. This illustrates how our theory guides practical design choices.

Question 1: We acknowledge your concern on Fig. 7(a). We are currently running those experiments. But we’d like to stress that Atari-57x5 represents ~450h on A40s. Given our available compute, we are only running Atari-57 on a selected subset of your request and do the others on Atari-5x5 seeds. Note that Atari-5 is the right benchmark to perform ablations: it provides a representative subset of Atari [8].

Question 2: Thanks for spotting this. We relaunched the experiments for the submission, but indeed the Figure is broken, related to the way we gathered the data. We’ve re-generated it and corrected the plot in the upcoming revision.

Question 3: As mentioned in Appendix F.2 and above, we used the alternative representation to avoid exploding transition losses (the exact two-split is in F2). We'll provide the requested Figures in an upcoming revision. Note that the baselines do not suffer from this explosion, coming from the representation of DreamSPI.

We finally thank you for the suggestions, we will take them into account when revising our paper.

AuthorsOpenReview ↗

References

[1] Roger Creus Castanyer, Johan S. Obando-Ceron, Lu Li, Pierre-Luc Bacon, Glen Berseth, Aaron C. Courville, Pablo Samuel Castro: Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning. NeurIPS 2025

[2] Pablo Samuel Castro: The Formalism-Implementation Gap in Reinforcement Learning Research. CoRR abs/2510.16175 (2025)

[3] Jakub Grudzien Kuba, Christian A. Schröder de Witt, Jakob N. Foerster: Mirror Learning: A Unifying Framework of Policy Optimisation. ICML 2022: 7825-7844

[4] Carles Gelada et al. DeepMDP: Learning Continuous Latent Space Models for Representation Learning. ICML, 2019.

[5] Ishaan Gulrajani et al. Improved Training of Wasserstein GANs. NeurIPS, 2017.

[6] Cem Anil, James Lucas, and Roger Grosse. Sorting Out Lipschitz Function Approximation. ICML, 2019.

[7] Jacob E. Kooi, Mark Hoogendoorn, Vincent François-Lavet: Latent Assistance Networks: Rediscovering Hyperbolic Tangents in RL. CoRR abs/2406.09079 (2024)

[8] Matthew Aitchison, Penny Sweetser, Marcus Hutter: Atari-5: Distilling the Arcade Learning Environment down to Five Games. ICML 2023: 421-438

AuthorsOpenReview ↗

Revision: additional experiments

As you requested, we performed a large number of new experiments to address your concerns. You may find them in our revised paper. All our additions are in purple. All our new experiments are over 5 seeds.

  • We updated Figure 7(a) (DreamSPI dyna vs. no imagination vs. imagination only) over 55 seeds.
  • We updated the broken Figures (4, 5, 16);
  • We performed DreamSPI without imagination on Atari 57, and we fully analyzed the effect of isolating the contribution of imagination to DreamSPI's performance in Appendix P. The new Figures 45 to 48 compare DreamSPI with DeepSPI including all our contributions except from imagination. We performed an analysis by splitting games into group to identify where imagination helps exactly when compared to the "no-imagination" variant ("augmented" DeepSPI).
  • In the same analysis, we added an Atari 5 analysis isolating our added components: vanilla [DeepSPI; PPO]+Kron and "augmented" PPO, i.e., PPO+[Kron, Mish, LayerNorm, token emeddings+codebook, simple projections]. We included PPO because DeepSPI is roughly "PPO + world model learning."
  • Appendix P.3 includes an analysis where we replace imagined updates with the same number of updates on the real batch. We performed the analysis on Atari-57 and we've also split the set to identify exactly where each variant helps or hurts.
  • Appendix L.5.1 now includes a clear explanation on the Octax configuration, with plots showing both the transition losses of the games for which we switched to a Hadamard representation. We also provided the Octax aggregation plots for the same configuration on all games, which barely changes the result (roughly 0.02 IQM). As mentioned, the motivation was mainly to fix "broken" transition losses.
  • Appendix N.1 plots policy ratios during training (Figures 36 and 37) for real and imagination steps, and we show in mean, median, IQM that the ratios lie in the theoretical neighbourhood with our proposed configuration.
  • Appendix N.2 provides a new Atari 10 evaluation with tighter clipping (where we use the theoretical clip epsilon=0.005 instead of our proposed practical configuration). The results show that clipping close to the theoretical radius does not necessarily hurt the performance and is thus a viable choice when a closer theoretical alignment matters. In practice, this is a trade-off to explore according to the environment.
  • We unfortunately can't report codebook distances during training: exact extrema are intractable to compute during training as the space of the categorical codes (uωu^{\omega} in Fig.3) is 323232^{32} (the transition model predicts 3232 categorical variables, each with 3232 possible classes).
Reviewer_ZBRWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work focuses on model-based reinforcement learning, aiming to stably learn both a dynamics model and a policy through environment interactions and imagined trajectories.

Its design incorporates several implementation improvements and techniques. In particular, it relies on on-policy experience without replaying data from previous iterations. It alternates between learning from real-environment trajectories and training the policy on imagined trajectories, while updating the transition model only from real-environment data. It also uses the Kron optimizer, categorical next-state prediction, and other implementation choices.

Critical strengths

The motivation to carefully coordinate updates to the dynamics model and policy to support safe policy improvement and achieve better performance is reasonable.

The experiments cover various Atari games and Octax tasks, demonstrate higher IQM than DeepSPI, the closest related baseline, and include a broad range of ablation studies to validate the design choices.

Critical weaknesses

The writing has considerable room for improvement. The introduction mixes the motivation, challenges, and novelty with extensive descriptions of the method, making these points difficult to distinguish. The background and problem statement could also be more concise and focused. As a result, the paper’s readability is quite limited.

It would be helpful to state the novelty of the proposed method more clearly. My understanding is that it extends DeepSPI to make more effective use of imagined trajectories for policy learning. However, the main methodological contributions should be clearly distinguished from implementation improvements, and their differences from related work should be explicitly explained.

The experiments lack comparisons with strong, established model-based baselines, such as DreamerV3 and MuZero.

It is unclear how much of the performance gain over the baselines comes from Kron versus the authors’ proposed design. It would be fairer to use matched optimization settings for PPO and DeepSPI, including Kron, and to clearly report these settings.

The limitations and trade-offs would benefit from a clearer, more explicit discussion in the main text, which would help readers understand the method more comprehensively.

Questions for authors

Please see the weaknesses above for details.

I am curious whether there is a theoretical guarantee of safe policy improvement for the actual training procedure, and how the reported KL divergence and loss curves relate to the conditions required by that guarantee.

4 comments

We thank the reviewer for their thoughtful assessment and for recognizing the motivation and empirical contributions of our work. We appreciate their detailed suggestions, which will help us improve the clarity and presentation of the paper.

Below, we clarify our methodological contributions, experimental choices, and theoretical guarantees.

About the writing: we regret to read that you found our introduction and background difficult to read and we will revise the writing and leverage the extra page for the rebuttal to improve on the clarity and the readability of our paper. We will also strengthen our discussion on the trade-off and limitation of our approach, as you suggested.

Contribution and novelty. Indeed, DreamSPI builds upon DeepSPI [1], which uses the world model only for representation learning. Our main contribution is to enable on-policy latent imagination within the SPI framework, supported by theoretical results on model regularity and policy-ratio control. This is not a straightforward extension: a naive implementation explored in [1] fell short of the baselines ("DreamSPI-v0" in our experiments). We distinguish our methodological contributions from the additional design choices that make them effective in practice below.

  1. On-policy latent imagination. We introduce a principled MBRL algorithm that enables guided SPI via imagination, using data from vectorized environments and w/o replay buffers. Unlike replay-based methods, this keeps model learning tied to the current policy distribution, as required by SPI theory.
  2. Representation with regularity guarantees. We introduce a token-based representation with a learned codebook, providing explicit regularity guarantees for the discrete latent dynamics (Claim 1/Thm. 1). This addresses a key condition for SPI (Eq. 6), with practical benefits in Fig. 8(c).
  3. Controlled policy updates. We show that simple projection layers provide a sufficient condition to control policy ratios during latent imagination (Claim 2/Thm. 3). Intuitively, freezing the representation and updating only the policy readout allows us to limit policy drift, as required by the SPI neighborhood (Eq. 2). See Fig. 7(b).

Additional design choices make the approach effective in practice:

  1. Simple world-model learning. We learn the model directly via latent predictions, without reconstruction, stop-gradient targets, or EMA. Our objective is motivated by SPI theory (Sect. 2), and its effectiveness is supported by Fig. 8(a,b).
  2. Handling non-stationarity. World-models amplify the non-stationarity inherent to RL. We adopt Kron, following [2], to stabilize training (Fig. 7(c)).

About the baselines. We deliberately focus our comparisons for on-policy methods compatible with vectorized environments and SPI theory. DreamerV3 and MuZero rely on replay buffers (+tree search for MuZero), introducing different training distributions that do not fit our theoretical setting (cf. Appendices B+K). Moreover, their computational requirements differ considerably: ~2.44h for DreamSPI on a single H200 for 40M frames (Appendix G, Table 4), v.s. ~7.7 GPU-days for DreamerV3 on A100 and ~12h for MuZero using 6 TPU-v3 devices for 200M Atari frames. While different hardware and data budgets prevent direct speed comparisons, MuZero's reported substantially more hardware than a single-GPU run. Finally, our replay-free approach offers a simpler training pipeline, straightforward to parallelize through vectorized environments, without replay management or distributed search. Our goal is to demonstrate that latent imagination is possible in RL under these lightweight conditions.

Comparisons to augmented DeepSPI and PPO. Please note that Fig. 7(a) shows Dream SPI without imagination, which is strictly equivalent to DeepSPI+all the improvements of this paper (Kron included). We are also currently running targeted PPO experiments.

Answer to your Question

As DeepSPI, DreamSPI build upon PPO, which has a rigorous interpretation within mirror-learning theory (cf. [3]). Our approach inherits this principled foundation and extends it to world-model learning and latent imagination. As in DeepSPI, we approximate the exact SPI constraints through PPO-style clipping and regularization, while projection-only updates help control policy drift. Stronger guarantees could be obtained through stricter constrained updates, at the cost of efficiency and potentially performance.

Empirical evidence. Appendix N reports average KL to monitor policy displacement and model-prediction losses related to the error terms in our SPI bounds (Equation 1). We are currently running additional experiments to measure how closely the empirical policy ratios satisfy the theoretical neighborhood in practice, despite using more permissive clipping thresholds.

We will post a revision containing the mentioned experiments when they are ready.

AuthorsOpenReview ↗

References

[1] Florent Delgrange, Raphaël Avalos, Willem Röpke: Deep SPI: Safe Policy Improvement via World Models. ICLR 2026

[2] Roger Creus Castanyer, Johan S. Obando-Ceron, Lu Li, Pierre-Luc Bacon, Glen Berseth, Aaron C. Courville, Pablo Samuel Castro: Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning. NeurIPS 2025

[3] Jakub Grudzien Kuba, Christian A. Schröder de Witt, Jakob N. Foerster: Mirror Learning: A Unifying Framework of Policy Optimisation. ICML 2022: 7825-7844

AuthorsOpenReview ↗

Revision: additional experiments

As promised, we've realized several new experiments to address your concerns.

  • Appendix P now contains an entire, thorough analysis and ablation of our proposed design (Figures 43 to 48). In particular, Appendix P.1 compares DreamSPI with PPO, DeepSPI (which is, roughly, "PPO + world model"), PPO + Kron, DeepSPI + Kron, and both with the addition of our proposed representation learning designs (Section 3). Ablations are on Atari-5 (5 seeds)
  • Appendix P.2 and P.3 show targetted ablations on Atari-57 (5 seeds). In particular, P.2 isolates imagination: we split the benchmark per group of environments (per kinds and induced behaviours) to evaluate where imagination helps compared to our proposed design without imagination.
  • Appendix N.1 now plots policy ratios during training (Figures 36 and 37) for real and imagination steps, and we show in mean, median, and IQM that the ratios lie in the theoretical neighbourhood with our proposed configuration.
  • (edit) As you suggested, we've substantially revised our introduction and added a limitation section to improve on the clarity and the readability of our paper, especially to better highlight and convey the contributions, novelty, limitations and trade-offs.

Dear Reviewer, we hope that our response and revisions address your concerns. If so, we would greatly appreciate it if you would consider updating the score. As the end of the discussion period is close, please let us know if any concerns remain so that we can address them. In any case, we’ll be happy to continue the discussion until the end of this period.

Reviewer_mNvTOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper extends DeepSPI (safe policy improvement) by also training the policy in the imagination of the world model (as in Dreamer, etc). The key technical innovation is to ensure that training the poicy on imaginary rollouts still results in improvement in the real world, by bounding how much the model can change at each step. This is achieved by using a discrete latent space (obtained by passing patch embeddings z coming from a CNN through a series of K softmaxes), followed by a linear map back to a continuous latent prediction z'; the codebook bounds the Lipschitz constant, and the contious space defines a notion of distance in the latent space. The latent dynamics are trained using next latent state prediction and reward prediction, not reconstruction (similar to TD-MPC), using real data.

Crucially the actor and critic are just linear readouts of this frozen encoding and dynamics model, so that PPO policy updates on imaginary rollouts maintain the theoretical guarantees of DeepSPI. Since the dynamics model is a cheap Markovian model, imaginary rollouts are fast (so no replay buffer is needed) and on-policy.

They combine this training procedure with the Kron second order optimizer to handle non-stationarity. They show substantial empirical gains on ALE and Octax (a Jax port of some ALE games) and do interesting ablations, such as adding reconstruction loss or EMA/sg to prevent latent collapse, or using muon or adam instead of Kron. (All these variants are worse.) Experimental results are based on multiple trials and include error bars so it is easy to assess statistical significance.

Critical strengths

Theoretically sound and novel . Empirically strong. Relatively simple to implement.

Critical weaknesses

Markovian latent dynamics is a limitation for environments with more partial observability (even on top of 4-frame padding).

Questions for authors

none

1 comment

Thank you for the careful reading and for highlighting the representation, imagination updates, and breadth of the evaluation.

Concerning the weakness (partial observability), we acknowledge this is a limitation of our framework: in this work, we focused on the Markovian case (cf. Section 2 and the text associated with Figure 1 where we make the Markovian assumption explicit). As shown in the paper, MDPs in the context of safe policy improvement already come with their own set of challenges. Furthermore, note that extending SPI-type guarantees to POMDPs is not a minor variation of our current setting; standard pure planning theoretical results already show a complexity jump from MDPs (P-time) to POMDPs (PSPACE-hard or even undecidable in general).

We also discussed this limitation in Appendix O and identified histories- or belief-states based policies as possible extensions. So we see this limitation as an exciting, non-trivial avenue for future work, which naturally follows up to our work. We will also provide a short limitation paragraph in the main text to re-state that limitation explicitly.

One additional small clarification: Octax contains CHIP-8 games with a different simulator and game dynamics; it is not a JAX port of the ALE games; the benchmark suite is different and the games have their own mechanics. So the Octax analysis is complementary to ALE.

Comments on the paper

1 comment

Dear Reviewers and ACs,

We thank you once again for reviewing and assessing our paper. The author-reviewer discussion is coming to an end. To facilitate further discussions, we summarize below the changes we made to address your comments and the additional experiments we conducted. Our revisions are highlighted in purple in the paper. For transparency, we also report the GPU-hours required for each requested experiment.

  • Abstract, introduction, and limitations (for reviewers ZBRW and LHAS). We rewrote the introduction to better highlight our contributions, explain how DreamSPI enables latent imagination, and clarify what the SPI theory guarantees. We added a limitations section after the conclusion to discuss our assumptions and trade-offs more explicitly.
  • Imagination ablation, Fig. 7(a) (for LHAS, Question 1). We expanded the Atari-5 comparison to five runs per game (20.06 GPU-hours)
  • Octax, Appendix L.5.1 (for LHAS, Question 3). We added a single-configuration comparison and showed how monitoring transition losses led us to adopt the Hadamard configuration on some games.
  • Policy ratios, Appendix N.1 (for ZBRW and LHAS, concerning the SPI neighborhood in practice). We plotted diverse metrics (mean, median, and IQM ±\pm std, 25-75 percentiles, and 95% stratified bootstrap CIs) of the policy ratios against the theoretical bounds for real and imagined updates (52.54 GPU-hours)
  • Tighter clipping, Appendix N.2 (for LHAS, concerning the application of the SPI bounds in practice). We tested a clipping radius of 0.005 on Atari-10 (corresponding to the theoretical one). The two settings (original and more constrained) achieved comparable performance overall (again compared on diverse metrics), so tighter clipping is a viable option when staying close to the theoretical bounds matters most (116.88 GPU-hours)
  • Optimizer and representation controls, Appendix P.1 (for ZBRW and LHAS, Question 1). We added PPO+, vanilla PPO with Kron, and original DeepSPI with Kron on Atari-5 (5 seeds) to assess what the optimizer and our representation contribute (46.03 GPU-hours)
  • Where imagination helps, Appendix P.2 (for LHAS, Question 1). We compared DreamSPI and DeepSPI+ on Atari-57. Imagination helped most in frequent-reward and reactive-control games and struggled in sparse-reward games, in concordance with our theory (219.49 GPU-hours)
  • Extra real updates, Appendix P.3 (for LHAS, Question 1). We replaced 256 imagined updates with 256 updates on the same real batch. At the same update budget, the two variants helped different groups of games (679.94 GPU-hours)

(In total, this corresponds to 1,135.31 GPU-hours, spread across H200s (62.45%), H100s (31.26%), and A40s (6.29%).)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.