acceptodds
Withdrawn from ICLR 2027

Encoding Memory in World Models with Test-Time Model-Based Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model-based reinforcement learning (MBRL) approaches have demonstrated strong performance across diverse domains in recent years, learning accurate world models from agent trajectories. However, despite this progress, current approaches still struggle to surpass human performance on tasks that require long-term memory and planning. In this work, we address the problem of long-term memory in MBRL by moving beyond the commonly studied paradigm of architecture design. We introduce Test-Time Model-Based Reinforcement Learning (TT-MBRL), using gradient descent at test time to encode the memory of episode observations into world model parameters. This enables the agent to learn a distinct world model for each episode, improving memory capabilities compared to traditional approaches that solely rely on contextual memory. We first establish a proof of concept on a simplified two-dimensional variant of the Memory Maze environment, which we refer to as Memory Maze 2D, showing consistent improvements across different world model architectures. We then demonstrate the effectiveness of TT-MBRL in the 3D domain, achieving new state-of-the-art results on the Memory Maze benchmark. Finally, we show that TT-MBRL maintains strong performance on non-memory benchmarks such as DeepMind Control and Atari 100k. Our code will be made publicly available for full reproducibility.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, world models, test-time training, memory maze

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eqEfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

TT-MBRL adapts the world model and actor–critic within each evaluation episode, using current-episode experience mixed with training replay. Parameters reset between episodes; the encoder and decoder remain frozen during adaptation.

Critical strengths

The method improves memory-intensive control across recurrent, Transformer, and state-space world models. On the largest 3D maze, R2I improves from 42.0 to 54.1 at the 400M-step training budget (p. 7). Prediction-quality measurements complement return results. The train-only compute control, replay-mixture ablations, and update-frequency study provide useful evidence about why adaptation helps (pp. 8, 17).

Critical weaknesses
  • World-model and policy adaptation occur together. Separate world-model-only and actor–critic-only ablations are needed to establish how much improvement specifically comes from encoding episodic information in world-model parameters.
  • Deployment cost remains substantial: the reported R2I evaluation takes 0.65 hours per episode on Memory Maze 15×15 (p. 9). The compute control rules out simply continuing training on old data, but does not establish superiority over other ways of spending the same inference budget.
  • Gains are concentrated in larger mazes. Additional memory tasks with different structures would strengthen the general claim, especially against explicit episodic-memory or mapping baselines.
  • Missing comparisons to similar relevant approaches (see questions).
Questions for authors
  1. Can you separate world-model-only adaptation from actor–critic-only adaptation, or provide evidence that distinguishes their contributions?
  2. What is evaluation latency without adaptation on the same hardware, and what return is achievable at a deployment budget comparable to the baseline?
  3. How does TT-MBRL differ conceptually and algorithmically from prior gradient-based online dynamics adaptation methods, particularly GrBAL (Nagabandi et al., 2019) and MOLe (Nagabandi et al., 2019)? Both already adapt predictive dynamics models using test-time trajectories. The paper should clarify what is novel beyond applying this established paradigm to Dreamer-style world models, and ideally compare against corresponding adaptation baselines.

Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., & Finn, C. (2019). Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning. International Conference on Learning Representations (ICLR). Nagabandi, A., Finn, C., & Levine, S. (2019). Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL. International Conference on Learning Representations (ICLR).

Reviewer_aooZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work studies whether Dreamer-style agents can improve their memory by continuing to learn during evaluation. At the start of an episode, the agent loads its pretrained parameters; it then trains both the world model and actor-critic using experience collected in that episode together with replay from training. The image encoder and decoder are kept fixed throughout this process. In the authors' account, the updated weights retain information about the environment being explored, helping the agent remember over longer periods. Experiments cover DreamerV3, TWISTER, and R2I in the 2D and 3D versions of Memory Maze, with larger benefits reported on the more difficult mazes. DreamerV3 is also tested on DMC and Atari, where the aggregate results improve by a small amount.

Critical strengths

The result I find most convincing is R2I on the harder 3D setting: after 400M training steps, its score on Memory Maze 15x15 rises from 42.0 to 54.1. The other backbones benefit as well. This gives some evidence that the improvement carries across world-model architectures.

The ablations help explain which parts of the adaptation procedure affect performance. They cover the mixture of replay data, keeping the visual networks fixed, dropping individual loss terms, and the frequency of updates. In Table 5, the authors compare against continued learning from training data with the same compute budget, and also against adaptation from test experience alone. These results indicate that both the fresh experience and how it is mixed into replay affect the outcome. Another useful feature is that the approach works with already trained Dreamer-style agents and requires no separate meta-training phase.

Critical weaknesses

My recommendation is rejection, mainly because I find the contribution to the method too small. The learning procedure retains Dreamer's world-model losses and actor-critic updates on imagined experience, running them further within each test episode. What is added is a mixture of training and episode replay, fixed visual networks, and a return to the pretrained parameters before the next episode. Neither a new memory component nor a new training objective is introduced. Test-time adaptation of models has already been explored. In light of these methods, it remains unclear to me what separates TT-MBRL at the algorithmic level.

I also find the evidence for the main memory claim incomplete. The direct memory tests are restricted to two variants of one maze task family. Adding DMC and Atari broadens the evaluation, but those results do not directly measure long-term memory. The comparison in most cases is to the same agent without adaptation. Since PAD changes different components, that baseline does not answer how TT-MBRL compares with another approach that updates the world model at test time. A further missing experiment would hold the world model fixed and train only the actor-critic. This control matters. Without it, the higher returns leave open how much of the benefit can be attributed to information stored in the world-model weights and how much to changes in the policy.

Questions for authors

1 Which part of TT-MBRL constitutes an algorithmic change over continuing to fine-tune a pretrained Dreamer agent online?

2 Could you report an actor-critic-only adaptation experiment, holding the world model fixed and using the same experience under matched update budget? The reverse experiment in Table 9, where the actor-critic is frozen, does not settle this point.

3 Can you provide results on a long-term-memory problem outside this maze family, or a comparison showing an advantage over another world-model adaptation method with similar access to data and compute? Neither question is answered by the average scores on DMC and Atari.

Reviewer_YxoAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the long-term memory problem in the partially observable MBRL setting. The key idea is to perform gradient-based test-time training (TTT) rather than storing history in the context. It is important to mix the train and test datasets for this TT update of the world model. An actor-critic policy is then trained again on this updated model. In experiments, the authors augment existing MBRL models Dreamer v3, TWISTER, and R2I with this TT-MBRL and shows the gain increases as the map size and episode length increases. Performance on DMC and Atari is preserved, though evaluation becomes considerably slower.

Critical strengths
  1. The proposed method is simple and architecture-agnostic. It requires no change to the underlying sequence model and gives consistent gains across RNN-, Transformer-, and SSM-based world models.

  2. The method works reasonably well, especially in the larger mazes and longer episodes where memory matters most. That said, the result is somewhat expected: adapting the model on the very episode it is evaluated on should naturally help, so the finding itself is not too surprising.

Critical weaknesses
  1. Limited novelty. The approach is essentially test-time training, or dynamic evaluation, applied to a Dreamer-style world model. The positive result is largely what one would expect from adapting on the evaluation episode itself.

  2. Fairness and cost. The proposed agent keeps training during evaluation while the baselines do not, which blurs the line between memory and online learning. The overhead is also substantial (about 0.65 hours per 3D episode), which limits its practicality.

Questions for authors
  1. How does TT-MBRL compare with a simpler online-adaptation baseline, such as fine-tuning only the actor-critic, or the full agent with standard online RL updates on the test episode?

  2. How sensitive are the results to the test-time hyperparameters (learning rate, update frequency, train/test mixing ratio)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.