- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses catastrophic forgetting in the continual adaptation of generalist robot policies. In open-ended deployment, a policy must learn new tasks one after another. Both weight-space fine-tuning and noise-space steering (DSRL) suffer from catastrophic forgetting. Steering only shifts the forgetting from the policy itself to the shared steering module. The authors propose Dream to Remember (D2R), which builds on noise-space steering: the generative policy and the world model are both kept frozen, and only a small module that steers the policy’s input noise is trained. To prevent forgetting, the world model regenerates rollouts of previously learned tasks from a few stored initial frames, and a KL penalty anchors the current module to its earlier steering behavior on these imagined states. The most interesting idea in the paper is that a world model needs lower fidelity to serve as “memory” than to serve as a “simulator.” When the world model is accurate enough, new tasks are learned entirely in imagination with PPO. When it is not, new tasks are learned in the environment with SAC, and the world model is used only for rehearsal.
Critical strengths
- The paper is well written, the core idea is clearly presented, and the method is simple.
- The ablation design is well thought out. The three variants (instruction KL, first-obs KL, and replay KL) differ only in the states on which the rehearsal penalty is applied. Together they show clearly that the memory must provide states that the retained tasks actually visit, but that these states need not be real or stored. The finding that instruction KL works on CALVIN, where all tasks share a scene, but fails on LIBERO, where scenes differ across tasks, is also convincing.
- The experiments cover two simulation benchmarks and a real robot. The appendix is detailed and the hyperparameters are fully reported.
Critical weaknesses
- The key claim that rehearsal needs less world-model fidelity than optimization is not yet well supported. This is the paper’s most interesting claim, but the evidence for it is a little bit weak.
- The headline LIBERO result: the +1.6 retention is measured on the four original LIBERO suites. A2World was fine-tuned on these suites and is shown to be accurate on them (Fig. S.7). So this result shows that rehearsal works with an accurate world model, not an inaccurate one.
- The real test is forgetting on the LIBERO-90 tasks, where A2World is inaccurate. Here D2R forgets 8.3 points vs. 11.6 for replay KL, but this comes from a single training run with no confidence intervals.
- The claim that A2World cannot serve as a simulator is indirect. It relies on a behavior-cloning proxy (training on imagined vs. real demos) rather than on actually trying to learn inside A2World. The proxy may also be confounded, because imagined frames are paired with real expert actions that may no longer match them.
Questions for authors
See weakness. Here are some suggestions that might be helpful:
- Report the results separately: split the LIBERO retention results into tasks where the world model is accurate (the four suites) and tasks where it is not (the learned LIBERO-90 tasks)
- Interpolate the rehearsal horizon. DSRL + first-obs KL is effectively the extreme case of a rehearsal horizon of zero, while D2R uses the full horizon (200 action chunks). The authors could test several intermediate horizons on the LIBERO-90 tasks. If forgetting keeps decreasing as the imagined horizon grows and world-model errors accumulate, this would directly show that inaccurate long-horizon imagined states are still useful.
- Directly test the claim that A2World cannot serve as a simulator (moderate cost). Train a simple reward model in A2World for the LIBERO-90 tasks, attempt to optimize the steering module in imagination, and report the real success rate as well as the gap between the success rate predicted in imagination and the real one. This would turn the claim from an inference into a direct measurement.
- Construct a series of world models with decreasing fidelity, and for each one measure two things: the real success rate when it serves as a simulator (learning in imagination), and the forgetting when it serves as memory (learning in the environment, rehearsing in the world model). I would like to make sure that I don’t expect the authors do all of the them during the rebuttal, but more evidences are always better.
- I also find some concurrent work: R2-WAM (arXiv:2610.04913) and Dream Rehearsal (arXiv 2607.19749). Maybe authors could discuss them in the related work.
2 comments
Thanks a lot for the revision. Most of my concerns are addressed and I therefore remain my original positive score.
Response to Reviewer 3mWR
We thank the reviewer for the careful assessment and the concrete suggestions. Everything the reviewer requested is now in the revised paper, marked in blue, and discussed point by point below.
Q1. Results split by world model accuracy. Table 2 already separates the two cases, but its headers hid this. On LIBERO, retention is measured on the four suites A2World was fine-tuned on, and success and forgetting on the eight LIBERO-90 tasks it never saw. The retention the reviewer refers to (+1.6, now +1.3 over three training runs) is therefore the accurate case, and forgetting is the test with an inaccurate model. There D2R forgets 9.3 points, no more than rehearsal on stored real states (11.4), against 52.5 points with first observations only and 78.3 without rehearsal. Revision. The caption of Table 2 now states which tasks each LIBERO column is measured on, and the introduction reports the forgetting on the LIBERO-90 tasks.
Q2. Rehearsal horizon. We varied the horizon T of the imagined rehearsal rollouts on the first four tasks of the LIBERO sequence. At T=0 only the stored initial frame is rehearsed (first-obs KL), and T=200 is D2R.
Forgetting falls with every increase of the horizon. Prediction errors of A2World accumulate along a rollout, so the late states are its least accurate ones. Yet they matter most. Extending the horizon from 100 to 200 removes almost all remaining forgetting, from 23.7 to 2.6 points. Revision. New Appendix S.6.6 with Table S.9.
Q3. Learning directly in A2World. We trained the steering module entirely inside A2World and evaluated it in the environment, on four tasks from each group (success in %).
On the unseen tasks the module succeeds in imagination but not in the environment. On the fine-tuned suites the gap is 1 to 12 points for the three short-horizon tasks, and the long-horizon one collapses. This also avoids the confound of the behavior-cloning proxy. The module acts on its own imagined rollouts, and no expert actions are involved. Revision. Section 4.4 reports this result in place of the earlier statement, with the per-task numbers in the new Table 3 and details in Appendix S.6.1.
Q4. World models of decreasing fidelity. We ran this series on CALVIN, with the world model taken at 10%, 25% and 50% of its training and used in both roles. Forgetting is measured on a sequence of four tasks.
As a simulator, the result improves with fidelity, by 10 points from the weakest to the strongest model. As a memory, forgetting stays low at every level and shows no trend. LIBERO shows the same pattern within one model. As a simulator, A2World falls far short on the tasks it never saw (Q3), while as a memory it remains sufficient on them (Q1, Q2). Revision. New Appendix S.6.7 with Table S.10.
Q5. Concurrent work. Dream Rehearsal (Nijjer, 2026) is the closest, as it also rehearses earlier tasks in a world model. It trains a DreamerV3 agent from scratch on MiniGrid, keeps a replay buffer of all real experience, and selects imagined rollouts with the reward and value heads, so its world model must judge behavior. We adapt a frozen pretrained robot policy, keep only initial frames, and our KL anchor needs no reward, which is why it tolerates a coarse world model. R2-WAM (Xu et al., 2026) improves a world action model on its training tasks and does not study forgetting. It first repairs the fidelity of the video model so that it can judge actions, which matches our observation that this role needs an accurate model while rehearsal does not. Revision. Both works are now in the related work, in the paragraphs on continual learning and on improving pretrained policies.
Independent training runs. The reviewer is right that the LIBERO-90 forgetting came from a single training run. We repeated the full sequence with three independent runs for D2R and the strongest baseline, DSRL + replay KL. Both reach 72.9% success (± 4.4 and ± 6.5), with a forgetting of 9.3 and 11.4 points and a retention of +1.3 and +1.9 points. Revision. Table 2 reports these numbers in the rows marked with a dagger. We will rerun the remaining LIBERO methods in the same way for the final version.
We are happy to clarify any remaining point and hope these changes can be considered in the final assessment.