- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ReSPRIT extends a BBF/SAC-BBF learner by reusing its SPR for imagination-based actor and value learning. Reward and continuation heads enable five-step imagined rollouts, whose λ-returns provide auxiliary actor and C51 Q-learning targets. The method retains these prediction heads across parameter resets and temporarily suppresses, then gradually restores, imagination losses with small final weights. Evaluation covers 26 Atari 100K games with ten seeds per game. ReSPRIT improves canonical IQM from 1.025 to 1.177 over the matched baseline, with additional ablations examining imagination scheduling and loss weights.
Critical strengths
-
The matched-baseline evaluation provides meaningful evidence that a compact SPR model can improve an already strong replay-based learner.
-
The approach adds relatively little machinery by reusing existing latent dynamics and Q-values, without requiring pixel reconstruction, a separate state-value network, or inference-time search. This makes the demonstrated performance gain practically interesting and provides a useful example of extending a representation-learning module to support behavior learning.
Critical weaknesses
-
The methodological contribution is incremental, and its distinction from closely related approaches is insufficiently established. Imagination-based learning and parameter resets are established techniques. In particular, Mind the Model, Not the Agent: The Primacy Bias in Model-based RL already combines model-based learning with world-model resets, while Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning connects predicted latent states to value learning. A more direct comparison of objectives, gradient flow, reset choices, and training schedules is needed to establish the specific advance offered by ReSPRIT.
-
The existing ablations do not sufficiently isolate the proposed mechanisms. Removing the schedule eliminates both initialization warm-up and post-reset suppression, so it does not establish the importance of reset-specific coordination. Imagined actor and Q-learning updates are not evaluated separately, and retaining reward/continuation heads is not compared with resetting them. Moreover, the weight ablation changes both coefficients simultaneously by large factors. These experiments support the complete configuration but leave the necessity and independent contribution of its key choices unclear.
Questions for authors
-
What is the principal methodological distinction from world-model resetting approaches and value-consistent learning on predicted latent states? Please clarify which aspects constitute new mechanisms and which are adaptations of established components.
-
Does post-reset suppression provide additional benefit over a schedule that uses warm-up only at initialization? Do prediction-error measurements around resets support the proposed explanation that imagined targets become unreliable and subsequently recover?
-
Can separate imagined-actor and imagined-Q ablations, together with retained-versus-reset prediction-head controls, identify which components are responsible for the improvement?
-
Can confidence intervals for the performance difference against the matched baseline be provided? The reported intervals characterize ReSPRIT’s performance but do not directly quantify uncertainty in the improvement.