acceptodds
Under review as a conference paper at ICLR 2027

Dream to Remember: Continual Steering of Generalist Robot Policies with World Models

Abstract

Generalist robot policies are trained with imitation learning and are bounded by the quality and coverage of their demonstrations. Reinforcement learning can improve existing skills and add new ones, either by fine-tuning the policy weights or by training a small module to steer the input noise of a frozen generative policy. In open-ended deployment, tasks arrive sequentially, and both approaches suffer from catastrophic forgetting. Fine-tuning on a task degrades performance on prior tasks, and steering only shifts forgetting to the shared steering module. We propose Dream to Remember (D2R), which continually steers a frozen generalist policy toward new tasks, one after another, while retaining its earlier skills. As the steering module learns a new task, a frozen pretrained world model generates imagined rollouts of prior tasks, anchoring the module to its earlier behavior. Prior tasks are thus rehearsed without a replay buffer or new data. When the world model is faithful enough, D2R learns the new task entirely in imagination. Otherwise, it learns the task in the environment while using the world model only to rehearse prior tasks. On CALVIN, D2R improves eight tasks in sequence entirely in imagination, raising average success from 63.7% to 74.8%, while LoRA fine-tuning on the same sequence drops it to 11.7%. With a frozen off-the-shelf generalist policy and world model, and A2World, D2R learns unseen LIBERO-90 tasks in the environment while keeping 's performance on the LIBERO suites it was fine-tuned on, even though the world model is not accurate enough for policy optimization. Finally, on a real Franka robot, D2R improves three tasks in sequence entirely in imagination while retaining the earlier improvements.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, Continual Learning, World Models, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3mWROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses catastrophic forgetting in the continual adaptation of generalist robot policies. In open-ended deployment, a policy must learn new tasks one after another. Both weight-space fine-tuning and noise-space steering (DSRL) suffer from catastrophic forgetting. Steering only shifts the forgetting from the policy itself to the shared steering module. The authors propose Dream to Remember (D2R), which builds on noise-space steering: the generative policy and the world model are both kept frozen, and only a small module that steers the policy’s input noise is trained. To prevent forgetting, the world model regenerates rollouts of previously learned tasks from a few stored initial frames, and a KL penalty anchors the current module to its earlier steering behavior on these imagined states. The most interesting idea in the paper is that a world model needs lower fidelity to serve as “memory” than to serve as a “simulator.” When the world model is accurate enough, new tasks are learned entirely in imagination with PPO. When it is not, new tasks are learned in the environment with SAC, and the world model is used only for rehearsal.

Critical strengths
  • The paper is well written, the core idea is clearly presented, and the method is simple.
  • The ablation design is well thought out. The three variants (instruction KL, first-obs KL, and replay KL) differ only in the states on which the rehearsal penalty is applied. Together they show clearly that the memory must provide states that the retained tasks actually visit, but that these states need not be real or stored. The finding that instruction KL works on CALVIN, where all tasks share a scene, but fails on LIBERO, where scenes differ across tasks, is also convincing.
  • The experiments cover two simulation benchmarks and a real robot. The appendix is detailed and the hyperparameters are fully reported.
Critical weaknesses
  • The key claim that rehearsal needs less world-model fidelity than optimization is not yet well supported. This is the paper’s most interesting claim, but the evidence for it is a little bit weak.
  • The headline LIBERO result: the +1.6 retention is measured on the four original LIBERO suites. A2World was fine-tuned on these suites and is shown to be accurate on them (Fig. S.7). So this result shows that rehearsal works with an accurate world model, not an inaccurate one.
  • The real test is forgetting on the LIBERO-90 tasks, where A2World is inaccurate. Here D2R forgets 8.3 points vs. 11.6 for replay KL, but this comes from a single training run with no confidence intervals.
  • The claim that A2World cannot serve as a simulator is indirect. It relies on a behavior-cloning proxy (training on imagined vs. real demos) rather than on actually trying to learn inside A2World. The proxy may also be confounded, because imagined frames are paired with real expert actions that may no longer match them.
Questions for authors

See weakness. Here are some suggestions that might be helpful:

  • Report the results separately: split the LIBERO retention results into tasks where the world model is accurate (the four suites) and tasks where it is not (the learned LIBERO-90 tasks)
  • Interpolate the rehearsal horizon. DSRL + first-obs KL is effectively the extreme case of a rehearsal horizon of zero, while D2R uses the full horizon (200 action chunks). The authors could test several intermediate horizons on the LIBERO-90 tasks. If forgetting keeps decreasing as the imagined horizon grows and world-model errors accumulate, this would directly show that inaccurate long-horizon imagined states are still useful.
  • Directly test the claim that A2World cannot serve as a simulator (moderate cost). Train a simple reward model in A2World for the LIBERO-90 tasks, attempt to optimize the steering module in imagination, and report the real success rate as well as the gap between the success rate predicted in imagination and the real one. This would turn the claim from an inference into a direct measurement.
  • Construct a series of world models with decreasing fidelity, and for each one measure two things: the real success rate when it serves as a simulator (learning in imagination), and the forgetting when it serves as memory (learning in the environment, rehearsing in the world model). I would like to make sure that I don’t expect the authors do all of the them during the rebuttal, but more evidences are always better.
  • I also find some concurrent work: R2-WAM (arXiv:2610.04913) and Dream Rehearsal (arXiv 2607.19749). Maybe authors could discuss them in the related work.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 3mWR

We thank the reviewer for the careful assessment and the concrete suggestions. Everything the reviewer requested is now in the revised paper, marked in blue, and discussed point by point below.

Q1. Results split by world model accuracy. Table 2 already separates the two cases, but its headers hid this. On LIBERO, retention is measured on the four suites A2World was fine-tuned on, and success and forgetting on the eight LIBERO-90 tasks it never saw. The retention the reviewer refers to (+1.6, now +1.3 over three training runs) is therefore the accurate case, and forgetting is the test with an inaccurate model. There D2R forgets 9.3 points, no more than rehearsal on stored real states (11.4), against 52.5 points with first observations only and 78.3 without rehearsal. Revision. The caption of Table 2 now states which tasks each LIBERO column is measured on, and the introduction reports the forgetting on the LIBERO-90 tasks.

Q2. Rehearsal horizon. We varied the horizon TT of the imagined rehearsal rollouts on the first four tasks of the LIBERO sequence. At T=0T = 0 only the stored initial frame is rehearsed (first-obs KL), and T=200T = 200 is D2R.

Horizon TT02060100200
Forgetting ↓\downarrow80.764.134.423.72.6
Success (%) ↑\uparrow21.940.860.869.775.8

Forgetting falls with every increase of the horizon. Prediction errors of A2World accumulate along a rollout, so the late states are its least accurate ones. Yet they matter most. Extending the horizon from 100 to 200 removes almost all remaining forgetting, from 23.7 to 2.6 points. Revision. New Appendix S.6.6 with Table S.9.

Q3. Learning directly in A2World. We trained the steering module entirely inside A2World and evaluated it in the environment, on four tasks from each group (success in %).

Trained inBaseA2WorldA2WorldEnv.Gap
Evaluated inEnv.A2WorldEnv.Env.
LIBERO suites A2World was fine-tuned on83.180.071.493.622.2
LIBERO-90, unseen by A2World4.779.420.392.271.9

On the unseen tasks the module succeeds in imagination but not in the environment. On the fine-tuned suites the gap is 1 to 12 points for the three short-horizon tasks, and the long-horizon one collapses. This also avoids the confound of the behavior-cloning proxy. The module acts on its own imagined rollouts, and no expert actions are involved. Revision. Section 4.4 reports this result in place of the earlier statement, with the per-task numbers in the new Table 3 and details in Appendix S.6.1.

Q4. World models of decreasing fidelity. We ran this series on CALVIN, with the world model taken at 10%, 25% and 50% of its training and used in both roles. Forgetting is measured on a sequence of four tasks.

World model training10%25%50%
As simulator, success (%) ↑\uparrow71.778.581.5
As memory, forgetting ↓\downarrow4.40.53.9

As a simulator, the result improves with fidelity, by 10 points from the weakest to the strongest model. As a memory, forgetting stays low at every level and shows no trend. LIBERO shows the same pattern within one model. As a simulator, A2World falls far short on the tasks it never saw (Q3), while as a memory it remains sufficient on them (Q1, Q2). Revision. New Appendix S.6.7 with Table S.10.

Q5. Concurrent work. Dream Rehearsal (Nijjer, 2026) is the closest, as it also rehearses earlier tasks in a world model. It trains a DreamerV3 agent from scratch on MiniGrid, keeps a replay buffer of all real experience, and selects imagined rollouts with the reward and value heads, so its world model must judge behavior. We adapt a frozen pretrained robot policy, keep only initial frames, and our KL anchor needs no reward, which is why it tolerates a coarse world model. R2-WAM (Xu et al., 2026) improves a world action model on its training tasks and does not study forgetting. It first repairs the fidelity of the video model so that it can judge actions, which matches our observation that this role needs an accurate model while rehearsal does not. Revision. Both works are now in the related work, in the paragraphs on continual learning and on improving pretrained policies.

Independent training runs. The reviewer is right that the LIBERO-90 forgetting came from a single training run. We repeated the full sequence with three independent runs for D2R and the strongest baseline, DSRL + replay KL. Both reach 72.9% success (±\pm 4.4 and ±\pm 6.5), with a forgetting of 9.3 and 11.4 points and a retention of +1.3 and +1.9 points. Revision. Table 2 reports these numbers in the rows marked with a dagger. We will rerun the remaining LIBERO methods in the same way for the final version.

We are happy to clarify any remaining point and hope these changes can be considered in the final assessment.

Thanks a lot for the revision. Most of my concerns are addressed and I therefore remain my original positive score.

Reviewer_9EzaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper experimentally explores a method for retaining performance when tuning a generalist policy across multiple different tasks. It proposes Dream to Remember (D2R), which adapts a frozen stochastic policy through a modified Markov Decision Process, which uses the noise input of the policy as its action space, by utilizing world models either for memorizing or for optimization, depending on its fidelity. Its main contribution is the experimental verification: On CALVIN, they show that D2R improves a sequence of eight tasks using only interactions through the world model, never interacting with the simulator itself. On LIBERO, using a world model too coarse to be used for optimization, they show that such a setup can benefit through rehearsal, interacting with the simulator for improvement but using the world model to retain previously learned skills. The authors also show that the setup transfers to the real world, by training a Franka Emika Panda robot entirely in imagination by a world model trained on teleoperated play data.

Critical strengths
  • Thorough, Reproducible Evaluation: The pipeline is evaluated in two simulation experiments and one real-world experiment, reporting sensible metrics. Simulation experiments report these for multiple seeds and episodes. Although some 95% CIs are quite large, it is apparent that the method improves task retention in the provided settings.
  • Presentation: The method is easy to follow and the paper does a good job presenting the problem, the method's components and the results.
Critical weaknesses
  • Novelty/Focus: The individual components are all well established, the novelty therefore lies in the evaluation of the proposed architecture. As such, the focus should lie in the experimental understanding of the components of the methods. I would therefore like to see more ablations of those components (the paper ablates only the rehearsal penalty, with some others like task order and steering module initialization deferred to the appendix). Specifically:

    • ablations towards the world model, what makes a world model fine-grained enough, a test of different world models, and
    • ablations towards the base policies, which source of stochasticity (gaussian, diffusion, ...) works best and how noisy or deterministic they can be would be interesting.
  • Applicability: We are in a setting with low number of tasks (evaluation is for at most 8 tasks) and the steering module is said to be small. The motivation for using a single steering policy and dealing with between-task forgetting as opposed to one policy per task and just not having that problem altogether is unclear. While the paper addresses the point of memory accumulation of per-task adaptation, the cost of that seems small in the low-number-of-tasks setting where the method is evaluated and evaluation in settings where this may prove to be useful (many or generalization to unseen tasks) is missing. A clear motivation for this setup is missing.

Questions for authors
  • What are the benefits of using this architecture in the settings evaluated in the paper as opposed to the task specific adaptations evaluated against?
  • Does the method hold up under longer task sequences or when generalizing to never-before seen tasks?
  • Are there quantitative evaluations of when a world model is too coarse to be usable? S.5.7 claims that a success rate of 19.6% in imagination vs. 74.9% in real demos is too large of a gap, whereas 70.3% vs. 74.5% is claimed to be faithful. Are there experiments showing which gap is too large to be usable?
  • What is the sensitivity of the method to different frozen policies?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 9Eza

We thank the reviewer for the careful assessment. Everything the reviewer requested is now in the revised paper, marked in blue, and discussed point by point below.

Q1. Why a single steering module. Per-task modules are the reference setting of Table 2. They cannot forget, and at eight tasks they reach the highest success (81.8% on CALVIN and 91.3% on LIBERO, against 74.8% and 72.9% for D2R). We do not claim that a single module is the better choice for a short and fixed list of tasks. Our goal is continual adaptation that does not grow with the number of tasks, neither in the model nor in a buffer of stored trajectories.

  • The model does not grow. D2R keeps one module of fixed size and 50 to 150 initial frames per task. With one module per task, the robot becomes a growing collection of specialists.
  • No task identity is needed. Per-task modules must be told at test time which module to load, and when to load none because the task was never adapted. A single module decides from the instruction and the observation.
  • It can transfer. The D2R module learned to open the top drawer in one scene, and on the same instruction in a scene it never saw, the success rises from 3.3% to 30.0%. A task that was never adapted has no per-task module.

The price is rehearsal time, about half an hour per retained task on LIBERO and about two minutes on CALVIN. Revision. The introduction now states this motivation, and the cost is reported in Section 4.3 and Appendix S.6.4.

Q2. Longer sequences and unseen tasks. We evaluated the final D2R module on the 82 LIBERO-90 tasks outside our sequence, none of which is learned or rehearsed. It keeps the success of the base policy on average (12.5% against 13.0%), while a steering module trained on a single task without rehearsal lowers it to 5.6%. Over all 90 tasks, D2R raises the success from 12.8% to 18.0%. We also ran a longer CALVIN sequence of twelve tasks. Forgetting stays low at every stage (between 2.9 and 6.0 points), and the average success stays above the base policy (58.7% against 51.1% after twelve tasks). Revision. New Appendices S.6.5 and S.6.8.

Q3. When a world model is too coarse. The behavior-cloning comparison measured this only indirectly, so we now measure it directly. We trained the steering module entirely inside A2World and evaluated it in the environment, on four tasks from each group (success in %).

Trained inBaseA2WorldA2WorldEnv.Gap
Evaluated inEnv.A2WorldEnv.Env.
LIBERO suites A2World was fine-tuned on83.180.071.493.622.2
LIBERO-90, unseen by A2World4.779.420.392.271.9

As a simulator, A2World is usable where training in it comes close to training in the environment. This holds for the three short-horizon tasks of the fine-tuned suites, with gaps of 1 to 12 points. It fails on their long-horizon task and on all four unseen tasks, with gaps of 35 to 93 points. As a memory, the same coarse model is sufficient. Over the LIBERO-90 sequence, D2R forgets 9.3 points, no more than rehearsal on stored real states (11.4).

A graded series on CALVIN shows the same pattern. With the world model taken at 10%, 25% and 50% of its training, success as a simulator rises from 71.7% to 78.5% and 81.5%, while forgetting as a memory, measured on a sequence of four tasks, stays low throughout (4.4, 0.5 and 3.9 points).

Revision. Section 4.4 reports the direct measurement, with the per-task numbers in the new Table 3 and details in Appendix S.6.1. The fidelity series is in the new Appendix S.6.7.

Q4. Sensitivity to the frozen policy. Steering acts on the input noise, so it requires a generative policy that turns noise into actions. A Gaussian policy, which predicts the mean and variance of the action directly, has no such input and cannot be steered this way. The paper covers both common families, a diffusion policy on CALVIN and on the real robot and the flow-matching model π0\pi_0 on LIBERO. To test how noisy the policy can be, we varied the DDIM parameter η\eta of the CALVIN policy, which sets how much fresh noise the sampler adds after the module has acted.

SamplerBaseD2RGain
DDIM, η=0\eta = 0 (paper)63.779.8+16.1
DDIM, η=0.5\eta = 0.562.470.9+8.5
DDIM, η=1\eta = 160.659.2-1.4

The gain shrinks as the sampler becomes more stochastic and disappears at η=1\eta = 1. D2R therefore needs a sampler in which the initial noise determines the action. This is the case for DDIM with η=0\eta = 0 and for the ODE sampler of flow matching, and choosing it requires no retraining of the policy. Revision. Section 3.1 now states this requirement, and the ablation is in the new Appendix S.6.2.

We are happy to clarify any remaining point and hope these changes can be considered in the final assessment.

Thank you for the clarifications! They largely address my concerns, so I have increased my score!

Reviewer_xjpZOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

D2R adapts a frozen generative robot policy by learning a small network that selects its input noise. At each new task, the steering network is trained from scratch while being regularized toward the preceding network on prior-task states generated by a frozen world model. Unadapted skills are anchored to the original Gaussian noise prior. A sufficiently accurate world model supports both learning and rehearsal; otherwise the new task is learned through environment interaction and imagination is used only for rehearsal. The evaluation includes sequential adaptation on CALVIN and LIBERO and a three-task Franka experiment.

Critical strengths
  1. The rehearsal ablations identify a meaningful ingredient beyond freezing the large policy. On LIBERO, DSRL finishes at 16.9% success, first-observation KL at 36.4%, replay KL at 73.9%, and D2R at 74.9% (Table 2). The progression supports the need for states along retained-task trajectories and shows that generated states can replace stored trajectories in this setting. The method also preserves the four pretrained LIBERO suites, which is a stronger retention test than evaluating only newly learned tasks.

  2. The distinction between a world model used for optimization and one used for rehearsal is useful. The LIBERO results show that imagined states can still support behavioral anchoring when behavior cloning from imagined demonstrations transfers poorly. Together with CALVIN and the real-robot experiment, this gives the paper more substance than a single favorable simulator result. Appendix S.5.3 also tests the important choice to reinitialize the steering module, and S.5.4 checks four CALVIN task orders.

Critical weaknesses
  1. The evidence for the stronger statement that the coarse world model cannot support policy optimization is indirect. Section 4.4 and Appendix S.5.7 compare behavior cloning from real and imagined demonstration replays, not RL optimization in the same world model. A large cloning gap demonstrates a consequential modeling mismatch, but does not by itself establish that a steering policy trained there must exploit errors or fail. The result supports successful rehearsal with an imperfect model; the boundary between simulator-quality and memory-quality models remains less precisely established.

  2. The headline comparisons leave two uncertainties about the practical advantage. The sequential weight-space baseline is LoRA without the corresponding imagined-rehearsal control, although DiWA with rehearsal is included for single-task adaptation. Thus the large gap to LoRA does not isolate steering from rehearsal in the sequential setting. In addition, LIBERO's intervals come from evaluation seeds for one trained model, not independent training runs. Since D2R regenerates prior-task rollouts at each update, its storage savings also exchange stored trajectories for recurring world-model computation. Table 2 quantifies the former but does not give the resulting wall-clock trade-off as the retention set grows.

Questions for authors
  1. Have you directly trained the steering module in A2World on the same LIBERO-90 tasks? A comparison against environment training would test the simulator-versus-memory claim more directly than behavior cloning alone. Otherwise, please narrow the claim to what the cloning experiment establishes.
  2. How does sequential LoRA with the same world-model rehearsal and an appropriately tuned anchoring coefficient compare with D2R? The single-task DiWA control is helpful but does not answer this sequential question.
  3. Can you report independent LIBERO training runs and the cost of adaptation versus rehearsal as more tasks are retained? I am interested in the joint performance, storage, and time trade-off, rather than an additional benchmark for its own sake.
3 comments
AuthorsOpenReview ↗

Response to Reviewer xjpZ (1/3): Q1

We thank the reviewer for the careful and constructive assessment. We are glad the reviewer agrees with the central finding of the paper, that rehearsal does not need stored trajectories because states imagined by a world model can take their place. The questions point to real gaps in our evidence, and we ran new experiments for each.

  • Q1. We trained the steering module directly inside A2World on LIBERO and evaluated it in the environment.
  • Q2. We added sequential LoRA with the same world model rehearsal, on both CALVIN and LIBERO.
  • Q3a. We report three independent LIBERO training runs for D2R and the strongest baseline.
  • Q3b. We report the time and storage cost of adaptation and rehearsal per stage as the retained set grows.

We answer each point below. All new results are in the revised paper, with changes marked in blue.

Q1. Training the steering module directly in A2World

Following the reviewer's suggestion, we optimized the steering module entirely in imagination inside A2World and evaluated it in the real LIBERO environment. We did this in the two regimes the paper distinguishes. In the faithful regime, four tasks from the suites on which A2World was fine-tuned. In the coarse regime, four of the eight LIBERO-90 tasks of our sequence (Table 2), which A2World never saw. The reward is a per-task success classifier trained on real and A2World-generated frames, with 100% precision and 93% average recall on held-out real rollouts (details at the end of our reply). The module is trained with SAC for 200k gradient steps per task, the same algorithm and budget in both settings, so the two differ only in where the transitions come from. Imagined success is the success predicted by the classifier on imagined rollouts, real success is measured in the environment. Both are over 90 episodes (30 episodes, 3 seeds).

Steering trained in A2World against steering trained in the environment, success rate (%). Imagined is the success predicted in imagination, real is the success in the LIBERO environment. Gap = env real minus A2World real.

TaskBaseA2World, imaginedA2World, realEnv, realGap
Spatial T490.0100.085.697.812.2
Goal T382.298.988.990.01.1
Object T394.482.296.7100.03.3
Long T865.638.914.486.772.3
Faithful, average83.180.071.493.622.2
LIBERO-90 T00.0100.03.396.793.4
LIBERO-90 T70.027.854.488.934.5
LIBERO-90 T2018.990.023.393.370.0
LIBERO-90 T350.0100.00.090.090.0
Coarse, average4.779.420.392.271.9

Real success. In the coarse regime, steering trained in A2World reaches 20.3% on average, against 92.2% when trained in the environment. On three of the four tasks it barely improves the base policy, by less than 5 points, and only Task 7 improves substantially (54.4%), still 35 points short of environment training. In the faithful regime the picture is different. Three of the four tasks end within 12 points of environment training, while the Long task collapses to 14.4%, far below its base of 65.6%. Its episodes take about 500 steps, roughly twice as long as in the other three suites.

Imagined against real success. In the faithful regime the success predicted in imagination is 80.0% on average against 71.4% in the real environment. In the coarse regime the predicted success is 79.4% against 20.3% real. The module thus learns to succeed in imagination in both regimes, but in the coarse regime this success does not carry over to the real environment.

Taken together, A2World works as a simulator only for short-horizon tasks on which it was fine-tuned. Outside that range, which includes all LIBERO-90 tasks we adapt in the paper, training in imagination falls far short of training in the environment.

The same A2World is nevertheless sufficient for rehearsal on these tasks. Over the eight-task LIBERO-90 sequence of Table 2, from which the four coarse tasks are taken, D2R forgets 9.3 points, no more than rehearsal on stored real states (11.4). This is the gap between a simulator-quality and a memory-quality world model that the paper argues for.

Revision. Section 4.4 now reports this measured result instead of the statement that A2World cannot serve as a simulator. The table is in the main text as Table 3, with details in Appendix S.6.1, and the behavior-cloning comparison (S.5.7) remains as supporting evidence only.

AuthorsOpenReview ↗

Response to Reviewer xjpZ (2/3): Q2 and Q3a

Q2. Sequential LoRA with the same world model rehearsal

We ran this control on both benchmarks. LoRA is fine-tuned over the same eight-task sequence and rehearsed on the same imagined rollouts as D2R. Rehearsal needs a penalty that keeps the current policy close to the previous one on these states. In D2R this is a KL divergence between two Gaussian steering modules. A LoRA-tuned diffusion policy has no closed-form action density, so we instead feed the same state and the same noise to the previous and the current policy and penalize the squared L2 distance between the two resulting action chunks, as in our single-task DiWA + rehearsal control. On CALVIN we sweep the weight λ\lambda of this penalty over four values. On LIBERO, where every run trains in the environment over the full sequence, a sweep was computationally infeasible within the rebuttal period, so we ran only λ=0.1\lambda = 0.1, the best value on CALVIN.

Sequential adaptation over eight tasks, LoRA with and without world model rehearsal (%). LIBERO rows report a single training run.

MethodCurrent-task ↑\uparrowSuccess ↑\uparrowRetention ↑\uparrowForgetting ↓\downarrow
CALVIN
LoRA58.2 ±\pm 5.911.7 ±\pm 1.9-52.053.2
LoRA + rehearsal (λ=0.025\lambda = 0.025)49.4 ±\pm 4.712.9 ±\pm 2.2-50.842.2
LoRA + rehearsal (λ=0.05\lambda = 0.05)54.7 ±\pm 2.217.8 ±\pm 7.9-45.941.6
LoRA + rehearsal (λ=0.1\lambda = 0.1)73.2 ±\pm 5.437.3 ±\pm 6.5-26.441.1
LoRA + rehearsal (λ=0.5\lambda = 0.5)56.3 ±\pm 3.931.7 ±\pm 3.4-32.028.1
D2R (ours)79.8 ±\pm 3.874.8 ±\pm 3.7+11.15.7
LIBERO
LoRA54.7 ±\pm 1.61.0 ±\pm 1.6-41.661.4
LoRA + rehearsal (λ=0.1\lambda = 0.1)44.2 ±\pm 2.926.9 ±\pm 1.6-14.119.7
D2R (ours)82.1 ±\pm 2.774.9 ±\pm 2.6+1.68.3

Rehearsal clearly helps LoRA. Final success rises from 11.7% to 37.3% on CALVIN and from 1.0% to 26.9% on LIBERO, and forgetting drops from 53.2 to between 28.1 and 42.2 points on CALVIN and from 61.4 to 19.7 points on LIBERO. But at its best coefficient LoRA + rehearsal still reaches only half the final success of D2R on CALVIN (37.3% against 74.8%) and about a third on LIBERO (26.9% against 74.9%). Final success peaks at λ=0.1\lambda = 0.1 and drops again at λ=0.5\lambda = 0.5, so a stronger anchor does not close the gap.

Rehearsal therefore explains only part of D2R's advantage over LoRA. Even with the same rehearsal, LoRA reaches at most half of D2R's final success, so steering in noise space contributes on its own. This mirrors the single-task result in the paper, where DiWA + rehearsal recovers most but not all of the retention loss (-12.7 points against +1.6 for D2R). The rehearsal penalty constrains a fine-tuned policy only on the imagined states, while its changed weights can still alter behavior elsewhere. Steering leaves the policy weights untouched, so every action still comes from the same pretrained policy.

The new Figure 4 shows this on LIBERO. Adding rehearsal to LoRA recovers only part of the retention on the pretrained suites (from -41.6 to -14.1 points). Adding it to steering restores retention fully (from -30.8 to +1.3 points) and raises the success on the new tasks from 16.9% to 72.9%.

Revision. LoRA + rehearsal is now a row of Table 2 on both benchmarks and a panel of Figure 3, and Figure 4 is new. The sweep is in Appendix S.6.3.

Q3a. Independent LIBERO training runs

The reviewer is right that the LIBERO intervals in Table 2 come from evaluation seeds of a single trained model. Within the rebuttal period we repeated the full eight-task sequence with three independent training runs for D2R and for the strongest baseline, DSRL + replay KL.

LIBERO sequential adaptation over three independent training runs (%), mean over runs, with the 95% confidence interval for success.

MethodSuccess ↑\uparrowRetention ↑\uparrowForgetting ↓\downarrow
DSRL + replay KL72.9 ±\pm 6.5+1.911.4
D2R (ours)72.9 ±\pm 4.4+1.39.3

The pattern of the paper holds across training runs. With a single run, D2R and DSRL + replay KL reached 74.9% and 73.9% success, a retention of +1.6 and +2.0 points, and a forgetting of 8.3 and 11.6 points. Over three runs both reach 72.9% success, with a retention of +1.3 and +1.9 points and a forgetting of 9.3 and 11.4 points. The two methods thus remain on par in success and retention, and D2R forgets no more than rehearsal on stored real states.

Revision. We have updated Table 2 in the revised PDF with the three-run results for D2R and DSRL + replay KL. For the final version we will rerun the remaining LIBERO methods with three independent training runs as well.

AuthorsOpenReview ↗

Response to Reviewer xjpZ (3/3): Q3b and reward model details

Q3b. Cost as the number of retained tasks grows

The reviewer is right that D2R exchanges stored trajectories for world model computation, and that the paper reported only the storage side. The table below gives time, storage and performance together as the retained set grows.

Cost and performance as the number of retained tasks grows. Frames are the stored frames. Replay KL needs no generation time. All times are measured on a single NVIDIA A100 GPU. Success is from a single training run.

StageRetained tasksD2R rehearsalFrames, D2RFrames, replay KLSuccess, D2R (%)Success, replay KL (%)
LIBERO, A2World (video model), adaptation about 4 h per stage
1000021.723.1
210.5 h50743532.828.6
431.5 h1502230545.849.8
873.5 h3505204574.973.9
CALVIN, RSSM (latent model), adaptation about 1 h per stage
870.25 h10506321074.874.0

Storage. D2R keeps 50 seed frames per retained task on LIBERO, replay KL keeps about 7400 frames of stored rollouts. Both grow linearly with the number of retained tasks, but D2R stores about 150 times less at every stage.

Time. This saving is paid for with generation time. On LIBERO the rollouts are generated once at the start of each stage, which takes 0.5 h per retained task, so the cost grows linearly and reaches 3.5 h at the last stage, on top of about 4 h of adaptation. The rollouts of the four pretrained suites are generated once, before the sequence, at the same rate. Replay KL has no such cost. How large the cost is depends on the world model. A2World is a video model, and every imagined step is expensive. On CALVIN the world model is a latent one, and rehearsal takes about two minutes per retained task. Even at the last stage, with all seven previous tasks retained, this is only 0.25 h of a 1 h stage.

Performance. The two methods stay on par at every stage, and end at 74.9% and 73.9% on LIBERO and at 74.8% and 74.0% on CALVIN.

In short, at equal performance D2R stores about 150 times fewer frames and instead spends generation time that grows linearly with the retained set and is set by the speed of the world model.

Revision. Section 4.3 now states the time cost next to the storage numbers, and the cost table is in Appendix S.6.4, so that both sides of the trade-off are visible.

Reward model details for Q1

We train one success classifier per task directly on image frames, using a mix of real and A2World-generated data. The real part is 100 rollouts of the task, where the last 5 frames of each rollout are positives and 10 randomly sampled frames are negatives. For the generated part, A2World replays the actions of 25 rollouts, and its frames are labeled in the same way. Training on both ensures the classifier is valid on the generated frames it sees during imagination. On 50 held-out real rollouts, with 5 positives and 5 negatives each, the classifiers reach a precision of 100% on all eight tasks and a recall between 86.4% and 100%. The reward model is therefore unlikely to be what limits training in A2World. On held-out real frames it never rewards a non-success, and recall does not explain the failures. LIBERO-90 Task 35 has a recall of 95.6% and 0% real success, while Task 7 has the lowest recall in the coarse regime (87.2%) and the highest success.

We hope these additions address the reviewer's concerns. We are happy to clarify any remaining point, and would be grateful if the reviewer would consider these changes in their final assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful and constructive assessments. We ran all the experiments the reviewers asked for and revised the paper accordingly. All changes are marked in blue in the revised PDF. We reply to each reviewer individually, and summarize the main takeaways here.

What the new experiments show

  • A world model can be too coarse to learn in and still good enough to remember with (xjpZ Q1, 3mWR Q3, 9Eza Q3). We trained the steering module directly inside A2World. On tasks the model never saw, it succeeds in imagination but fails in the environment (20% against 92% when trained in the environment). On the same tasks, rehearsal in this model prevents forgetting as well as rehearsal on stored real states. Table 3, Section 4.4, Appendix S.6.1.

  • This holds across fidelity levels and along the rollout (3mWR Q2 and Q4, 9Eza Q3). With weaker world models, success as a simulator drops, while forgetting as a memory stays low. Forgetting also keeps falling as the imagined rollouts get longer, although their late states are the least accurate ones. Appendices S.6.6 and S.6.7.

  • Rehearsal alone does not explain the result, steering matters too (xjpZ Q2). The same world model rehearsal reduces the forgetting of LoRA fine-tuning, but its final success stays at most half of that of D2R. Table 2, Figures 3 and 4, Appendix S.6.3.

  • The results are stable across training runs (xjpZ Q3a, 3mWR). Over three independent runs on LIBERO, D2R remains on par with rehearsal on stored real trajectories. Table 2.

  • D2R trades storage for generation time (xjpZ Q3b, 9Eza Q1). It stores about 150 times fewer frames than replay and instead spends time that grows linearly with the number of retained tasks and depends on the world model. Section 4.3, Appendix S.6.4.

  • A single module leaves other tasks intact (9Eza Q1 and Q2). On the 82 LIBERO-90 tasks it never learned, the D2R module keeps the success of the base policy, while steering without rehearsal lowers it. Appendix S.6.5.

  • Steering needs a deterministic sampler (9Eza Q4). The gain from steering shrinks as the sampler of the frozen policy becomes more stochastic. Such a sampler is available for diffusion and flow-matching policies without retraining. Section 3.1, Appendix S.6.2.

  • D2R holds up on a longer sequence (9Eza Q2). When the CALVIN sequence is extended beyond the eight tasks of the paper, forgetting stays low and the policy stays above the base policy at every stage. Appendix S.6.8.

Changes in the main text

  • Introduction. States why we aim for a single steering module that does not grow with the number of tasks, and reports the forgetting on the LIBERO-90 tasks next to the retention on the suites.
  • Related work. Discusses the concurrent work on rehearsal in a world model and cites R2-WAM.
  • Section 3.1. States that steering requires a deterministic sampler.
  • Table 2. The caption says which tasks each LIBERO column is measured on. A new row for LoRA + rehearsal, and three-run results for D2R and DSRL + replay KL.
  • Figure 3. Two new panels for LoRA and LoRA + rehearsal.
  • Table 3 and Figure 4. Both new. A2World as a simulator, and learning against retention with and without rehearsal.
  • Section 4.3. Discusses LoRA + rehearsal and reports the time cost of rehearsal next to the storage numbers.
  • Section 4.4. Replaces the statement that A2World cannot serve as a simulator with the measured result.

We thank the reviewers again for their time and are happy to clarify any remaining point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.