- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TT-MBRL adapts the world model and actor–critic within each evaluation episode, using current-episode experience mixed with training replay. Parameters reset between episodes; the encoder and decoder remain frozen during adaptation.
Critical strengths
The method improves memory-intensive control across recurrent, Transformer, and state-space world models. On the largest 3D maze, R2I improves from 42.0 to 54.1 at the 400M-step training budget (p. 7). Prediction-quality measurements complement return results. The train-only compute control, replay-mixture ablations, and update-frequency study provide useful evidence about why adaptation helps (pp. 8, 17).
Critical weaknesses
- World-model and policy adaptation occur together. Separate world-model-only and actor–critic-only ablations are needed to establish how much improvement specifically comes from encoding episodic information in world-model parameters.
- Deployment cost remains substantial: the reported R2I evaluation takes 0.65 hours per episode on Memory Maze 15×15 (p. 9). The compute control rules out simply continuing training on old data, but does not establish superiority over other ways of spending the same inference budget.
- Gains are concentrated in larger mazes. Additional memory tasks with different structures would strengthen the general claim, especially against explicit episodic-memory or mapping baselines.
- Missing comparisons to similar relevant approaches (see questions).
Questions for authors
- Can you separate world-model-only adaptation from actor–critic-only adaptation, or provide evidence that distinguishes their contributions?
- What is evaluation latency without adaptation on the same hardware, and what return is achievable at a deployment budget comparable to the baseline?
- How does TT-MBRL differ conceptually and algorithmically from prior gradient-based online dynamics adaptation methods, particularly GrBAL (Nagabandi et al., 2019) and MOLe (Nagabandi et al., 2019)? Both already adapt predictive dynamics models using test-time trajectories. The paper should clarify what is novel beyond applying this established paradigm to Dreamer-style world models, and ideally compare against corresponding adaptation baselines.
Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., & Finn, C. (2019). Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning. International Conference on Learning Representations (ICLR). Nagabandi, A., Finn, C., & Levine, S. (2019). Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL. International Conference on Learning Representations (ICLR).