acceptodds
Withdrawn from ICLR 2027

ReSPRIT: Reset-Aware Imagination from Self-Predictive Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model-free agents such as BBF achieve strong performance on Atari 100K, yet use their predictive models only to improve representations. Can imagination from a compact predictive model improve an already strong off-policy learner? We introduce Reset-aware Self-Predictive Representation Imagination Training (ReSPRIT), which adds reward and continuation prediction heads to BBF's existing latent prediction model to form a compact world model. Short imagined rollouts provide auxiliary actor and value learning alongside updates from replayed experience. Reset-aware scheduling limits the influence of unreliable imagined targets after parameter resets, while reduced imagination-loss weights balance learning from real and imagined experience. On Atari 100K, ReSPRIT improves the human-normalized interquartile mean (IQM) score from for the matched baseline without imagination to , the highest reported canonical IQM among the compared agents. It also achieves a higher game-level IQM than EfficientZero V2, while requiring no lookahead search at inference. In our Asterix comparison, ReSPRIT completes training and evaluation in roughly one-third the time of EfficientZero V2. Ablation studies support the benefits of imagination and the importance of reset-aware scheduling and loss weighting.

Primary area:
reinforcement learning
Keywords:
Sample-efficient reinforcement learning, world models, Atari 100K

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BnBPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReSPRIT extends a BBF/SAC-BBF learner by reusing its SPR for imagination-based actor and value learning. Reward and continuation heads enable five-step imagined rollouts, whose λ-returns provide auxiliary actor and C51 Q-learning targets. The method retains these prediction heads across parameter resets and temporarily suppresses, then gradually restores, imagination losses with small final weights. Evaluation covers 26 Atari 100K games with ten seeds per game. ReSPRIT improves canonical IQM from 1.025 to 1.177 over the matched baseline, with additional ablations examining imagination scheduling and loss weights.

Critical strengths
  1. The matched-baseline evaluation provides meaningful evidence that a compact SPR model can improve an already strong replay-based learner.

  2. The approach adds relatively little machinery by reusing existing latent dynamics and Q-values, without requiring pixel reconstruction, a separate state-value network, or inference-time search. This makes the demonstrated performance gain practically interesting and provides a useful example of extending a representation-learning module to support behavior learning.

Critical weaknesses
  1. The methodological contribution is incremental, and its distinction from closely related approaches is insufficiently established. Imagination-based learning and parameter resets are established techniques. In particular, Mind the Model, Not the Agent: The Primacy Bias in Model-based RL already combines model-based learning with world-model resets, while Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning connects predicted latent states to value learning. A more direct comparison of objectives, gradient flow, reset choices, and training schedules is needed to establish the specific advance offered by ReSPRIT.

  2. The existing ablations do not sufficiently isolate the proposed mechanisms. Removing the schedule eliminates both initialization warm-up and post-reset suppression, so it does not establish the importance of reset-specific coordination. Imagined actor and Q-learning updates are not evaluated separately, and retaining reward/continuation heads is not compared with resetting them. Moreover, the weight ablation changes both coefficients simultaneously by large factors. These experiments support the complete configuration but leave the necessity and independent contribution of its key choices unclear.

Questions for authors
  1. What is the principal methodological distinction from world-model resetting approaches and value-consistent learning on predicted latent states? Please clarify which aspects constitute new mechanisms and which are adaptations of established components.

  2. Does post-reset suppression provide additional benefit over a schedule that uses warm-up only at initialization? Do prediction-error measurements around resets support the proposed explanation that imagined targets become unreliable and subsequently recover?

  3. Can separate imagined-actor and imagined-Q ablations, together with retained-versus-reset prediction-head controls, identify which components are responsible for the improvement?

  4. Can confidence intervals for the performance difference against the matched baseline be provided? The reported intervals characterize ReSPRIT’s performance but do not directly quantify uncertainty in the improvement.

Reviewer_CwWcOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReSPRIT, a new method for using imagined experience to improve a strong model-free Atari agent (built on SAC-BBF). The key idea is to add a reward head and a continuation head to the latent transition model the agent already trains for self-predictive representations (SPR), turning it into a small world model whose short imagined rollouts provide auxiliary training for the existing actor and Q-network

Critical strengths
  1. Well-documented implementation and experimental details.
  2. The core idea is rather simple, elegant (at heart), and intuitive (predict reward and termination, train in imagination when data-sparse).

I do not find the paper very strong overall, so don't report many substantive strengths.

Critical weaknesses
  1. Ablations don't fully support claims. The Abstract (Line 25) claims that ablations support "the benefits of imagination and the importance of reset-aware scheduling and loss weighting," and the Conclusion (Line 459) and Contribution 1 (Lines 84–86) make similar claims about scheduling and loss weighting. However, the ablations on reset-aware scheduling (App. B.2) and the imagination-loss weights (App. B.3) are each run on a different six-game subset, with a single alternative setting each and no confidence intervals (Table 4, Line 703). This makes it hard to tell how robust these design choices are, or how sensitive the method is to them. In addition, the matched baseline reduces the final replay n-step horizon from 3 to 1 (App. A.2), while imagination supplies multi-step returns, so part of the reported gain may come from restoring multi-step targets rather than from the world model itself (see Question 1). Several design choices (e.g., keeping the reward and continuation heads through resets, Line 307) are also not ablated.

  2. Uncertainty is not reported for comparisons. The paper reports CIs in Appendix C.1 (Table 6, Line 783) but does not report them in Tables 1–3 (Lines 379, 392, 649). While these tables are labelled as point estimates, omitting intervals makes the paper appear weaker. CIs are only given for ReSPRIT, not for the baseline or any ablation (related to Con 1), even though the authors have per-run scores for these. ReSPRIT's IQM interval contains the point estimates of ITC and SAC-BBF, so "the highest reported canonical IQM among the compared agents" (Line 21) is less compelling. Reporting intervals for the baseline and ablations, or a paired comparison such as probability of improvement, would make the results much easier to trust.

  3. I did not find the paper very clearly written. In particular, there is a lengthy discussion of the method's implementation details from pages 3–7 (Section 3, Lines 156–363). These pages, while useful, contain a lot of information that might be better left to the Appendix (e.g., the boundary masking in Eqs. 5–7 and the PER/schedule details in Section 3.4). Meanwhile, the matched-baseline table, all ablations, and the runtime results are in the appendix, and the main-text experiments section is about one page. To be fair, the paper does justify many choices locally; however, I personally found the ratio of implementation minutiae to overarching intuition to be high enough to damage reading clarity. I would suggest moving these details to the appendix and bringing the baseline comparison and ablations into the main text.

  4. No limitations section. In particular, all results are on a single benchmark (Atari 100K), the world model is deterministic, performance appears sensitive to the imagination-loss weights (unit weights substantially reduce performance, App. B.3), and the runtime comparison against EZ-V2 (Line 91, App. E) is based on a single game. A short discussion of these points would help readers judge where the method is likely to work.

Questions for authors
  1. Can you please run the baseline with baseline and ReSPRIT with n = 3?
  2. Does the runtime comparison hold when you use multiple games rather than just Asterix?
  3. How accurate is the world model, e.g., its reward and continuation predictions over the 5-step rollouts?
Reviewer_2EWeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds upon BBF and SAC-BBF, adding elements from Dreamer-like world model training. They use the transition model from SAC-BBF to train an actor (REINFORCE) and critic (C51) on both sampled replays and imagined rollouts of length H=5H=5. Additionally, they add prediction heads that output the reward and continuation from the internal latent encoding of environment states to aid the policy training. These heads are trained without allowing gradients to flow to the representation learning. They also add a gradient update schedule to make training more stable in periods after the frequent parameter reinitializations inherited from BBF, fully suppressing imagination based gradient updates to the policy and critic during imagination for a period of W=4kW=4k gradient steps, and then linearly increasing the scheduling weight of those updates from 00 to 11 in the following W=4kW=4k steps. The authors also report three bugs in the official BBF code, which SAC-BBF inherits, and construct a matched baseline from the bug-fixed code with several configuration changes. On Atari 100K, ReSPRIT reaches a canonical IQM of 1.177 compared to 1.025 for this baseline, reports the highest canonical IQM among the compared agents, and requires roughly one third of EfficientZero V2’s wall-clock time on Asterix.

Critical strengths
  • Simple yet thought out combination of model free BBF/SAC-BBF and Dreamer like imagination training, reusing the existing SPR transition model and Q-network and only adding two small heads. The computation overhead is modest (+18% runtime over their baseline on Asterix)
  • Good explanation of the method, easy to follow and reasonings made clear, e.g. the masking at terminal transitions and the motivation for the reset schedule
  • Transparent reporting of their findings, listing the results of all 10 seeds of their runs and providing raw scores and code.
  • If substantiated, the reported bugs in the BBF and SAC-BBF code base are a useful contribution to the community
  • The ablation covers the two main design choices of the paper, the reset schedule and the loss weights.
  • Strong results on Atari100k, IQM of 1.177 with the lower CI bound above the human level, all without look ahead search.
Critical weaknesses
  • limited novelty. The world model is basically the SPR transition model plus two heads, and combining self-predictive latent dynamics with reward/value heads is not new (EfficientZero’s model is structurally very similar, also TD-MPC/TD-MPC2 [1]). Using short model rollouts to help an off-policy learner is the MVE/STEVE/MBPO [2] line, and Muesli [3] uses a learned model for policy learning without search. None of this is discussed in the related work, so the only really new part is the reset schedule
  • The matched baseline is weaker than the method it replaces. After fixing the bugs they build a new baseline with four hand picked config changes, which gets 1.025, below the reported SAC-BBF (1.088) and their own SAC-BBF RR4 reproduction (1.092). The +0.15 headline gain is against this weaker baseline; against SAC-BBF the gain is ~0.09 and both SAC-BBF values lie inside ReSPRIT’s 95% CI. No CIs are given for the baseline or SAC-BBF
  • The most relevant comparison (SAC-BBF, BBF) is only in the appendix, Table 1 leaves both out. Also all ablations are in the appendix, the experiment section in the main text is only about 1.5 pages
  • The world model is never validated as a model. SPR only aligns projected predictions via cosine similarity, so nothing forces the predicted latents to match the encoder outputs that the actor and critic are trained on, and the transition model was only trained on replay actions, not policy actions. Still, imagination evaluates the actor and target critic on these latents. No reward/continuation errors, multi-step latent errors or Q(z^)Q(\hat z) vs Q(eθ(s))Q(e_\theta(s)) gaps are reported
  • No per game comparison to the baselines, so it’s unclear on which kind of games imagination helps. They only state that ReSPRIT beats the baseline on 18/26 games without saying which. From Table 5 the profile looks like the usual BBF one, strong on reactive dense reward games, weak on exploration heavy ones (PrivateEye, Seaquest, Hero, MsPacman). Seed level failures like Freeway (4/10 seeds at exactly 0) are not discussed

Minor:

  • the abstract is hard to read, it assumes the reader knows BBF, IQM and EZ-V2 and never clearly states the gap
  • runtime comparison between baselines and this work is only on one game and across different codebases

[1] Hansen, N., Su, H., Wang, X. TD-MPC2: Scalable, Robust World Models for Continuous Control. ICLR 2024. arXiv:2310.16828

[2] e.g. this paper: Janner, M., Fu, J., Zhang, M., Levine, S. When to Trust Your Model: Model-Based Policy Optimization. NeurIPS 2019. arXiv:1906.08253

[3] Hessel, M., Danihelka, I., Viola, F., Guez, A., Schmitt, S., Sifre, L., Weber, T., Silver, D., van Hasselt, H. Muesli: Combining Improvements in Policy Optimization. ICML 2021. arXiv:2104.06159

Questions for authors
  • What does SAC-BBF score with only the bug fixes and its original config? Please also report CIs for the baseline and SAC-BBF and the probability of improvement.
  • What is the effect of each bug fix on its own, and can you give more evidence that these are actually bugs (e.g. a small example where the old code gives wrong targets/masks)?
  • How were H, W and the loss weights chosen, and on which games? Was the baseline config tuned with similar effort? How sensitive is the method to W?
  • What happens on Freeway where 4/10 seeds score exactly 0, and on Kangaroo with its bimodal results? Does the baseline show the same behavior?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.