acceptodds
Under review as a conference paper at ICLR 2027

Continual Adaptation of Multitask World Model with Embedding Retrieval

Abstract

Recent advances in world models have enabled generalist agents to learn diverse continuous control tasks within a single model. However, joint training on all desired tasks is not always feasible, as limited resources constrain access to collected data and simulation environments at a given time. An important goal is therefore to adapt pretrained multitask agents to newly accessible tasks while retaining their existing capabilities. We propose MARR, a framework for multitask world model adaptation with capability retention. Given a short probing trajectory, our framework retrieves a task-relevant dynamics representation from the pretrained world model to initialize adaptation, enabling the agent to learn multiple new tasks efficiently. We adapt world-model weights through a factorized parameterization that learns new combinations of fixed pretrained bases, while exemplar-based distillation helps retain prior knowledge and skills. Experiments across diverse continuous control tasks show that our approach learns new tasks while retaining performance on previously learned tasks and outperforms a range of existing baselines. These results support multitask adaptation as a promising approach to broadening the capabilities of a multitask model.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Multitask Adaptation, Catastrophic Forgetting, World Models, Continuous Control, Model-Based Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dsi6OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on a continual learning scenario in RL: adapting a pretrained multitask world model to a batch of newly accessible tasks. Building on the TD-MPC2 pretrained on 70 tasks across DeepMind Control Suite (DMControl) and Meta-World, the authors propose MARR, a framework combining three mechanisms for adaptation to 10 new tasks while retaining performance on previously learned tasks:

  1. Embedding Retrieval: Using a short random-action probing context (500 transitions per new task), MARR retrieves a dynamics embedding by minimizing latent consistency loss for the encoder and dynamics modules, and a control embedding by minimizing reward and temporal-difference losses for the task-related modules.

  2. Exemplar-Based Component Distillation: To mitigate catastrophic forgetting on previously learned tasks, MARR caches the highest-return trajectories of the old tasks and regularizes the encoder, dynamics, reward, value, and policy prior outputs during adaptation.

  3. SVD-Based Weight Parameterization: Linear weight matrices WW are decomposed via singular value decomposition (W=UΣV⊤W = U \Sigma V^\top), freezing the left and right orthogonal bases (U,V)(U, V) while optimizing a dense core matrix CC initialized to Σ\Sigma.

Experiments show that MARR outperforms several baselines, including vanilla TD-MPC2 fine-tuning and continual learning variants of Multi-SAC.

Critical strengths
  • Clear and Modular Formulation: The paper is clearly written and easy to follow. Decoupling the task conditioning of a world model into dynamics-related components (hθ,dθ)(h_{\theta}, d_\theta) and task/reward-related components (Rθ,πθ,Qθ)(R_\theta, \pi_\theta, Q_\theta) is an intuitive design choice for transfer learning in latent world models, as tasks may share physical embodiment dynamics while differing in reward structure (or vice versa).

  • Comprehensive Component-Wise Distillation: Rather than only distilling the final policy behavior (as in behavioral cloning on exemplars), regularizing the internal world-model predictions (encoder representations, latent transitions, rewards, and QQ-values) directly targets representation drift in shared latent spaces.

Critical weaknesses
  1. Conflation of Backbone RL Architecture with Adaptation/Retention Mechanisms in Baselines: All continual learning baselines with retention mechanisms are built on a model-free Multi-SAC backbone, whereas MARR is built on TD-MPC2. There are no continual learning baselines implemented on the TD-MPC2 backbone to provide a meaningful comparison to the adaptation mechanisms this paper proposes. As shown in Figure 3 (Left) and Figure 4, all Multi-SAC variants—including unconstrained Multi-SAC FT—fail to learn the 10 new tasks within the step budget, whereas both TD-MPC2 variants learn well. This gap could just reflect the intrinsic sample-efficiency advantage of online MPC planning in TD-MPC2 or loss of plasticity in Multi-SAC network rather than the superiority of the proposed continual adaptation algorithm.

  2. Simplicity of Task Suite and Realism of the Single-Batch Adaptation Scenario: Despite framing the problem around continual adaptation of generalist robot foundation models, the evaluation only considers a single adaptation stage where n=10n=10 low-dimensional state-based simulated tasks arrive simultaneously and are trained jointly with task-balanced batches. This single-step joint batch expansion does not test sequential multi-stage continual learning (where forgetting compounds across multiple adaptation rounds) or high-dimensional visual/real-world robotic settings. Moreover, the 70 pretraining tasks already contain dense coverage of the exact same MuJoCo embodiments and manipulation objects, making the 10 held-out tasks very close interpolations of the pretraining distribution.

  3. Lack of System-Level Comparisons: The paper focuses on comparing performance under a fix budget, but lacks a system-level analysis of memory and compute overhead. Specifically, exemplar distillation requires storing trajectory records and cached original-model component outputs, which scales linearly with the number of prior tasks. Similarly, parameterizing weights as W′=UCV⊤W' = U C V^\top with frozen U,VU, V and dense cores CC increases the per-step forward/backward matrix multiplication cost compared to standard weight fine-tuning, yet wall-clock training time and memory footprints are not taken into account in the comparison or even reported.

Questions for authors
  1. Clarification on TD-MPC2 FT with retrieval: IIUC, TD-MPC2 with retrieval means that the new task embeddings are initialized from manually chosen existing tasks? In addition, the same task is used for initializing both dynamics and control embedding? Is this understanding correct? If so, does the benefit of embedding retrieval solely come from decoupling task for dynamics and control embeddings? Could you provide an ablation comparing TD-MPC2 with decoupled task embeddings for dynamics and control? And why does Figure 3 (Right) attribute 5,000 retrieval steps to this baseline?

  2. Clarification on distillation: Section 3.3 introduces Lhdist,Lddist,LRdist,\mathcal{L}_h^{\text{dist}}, \mathcal{L}_d^{\text{dist}}, \mathcal{L}_R^{\text{dist}}, and LQdist\mathcal{L}_Q^{\text{dist}} without giving their explicit mathematical formulas. Are these simply ℓ2\ell_2 (MSE) losses on the encoder/dynamics latents and scalar reward/value predictions?

  3. Role of Full-Rank Dense SVD and Controlled Ablation: Given that retaining almost all singular vectors and training a dense rl×rlr_l \times r_l matrix CC leaves 4.6M trainable parameters and does not restrict the rank or subspace of the 512×512512 \times 512 hidden matrices (Table 8), what is the theoretical or empirical mechanism by which W′=UCV⊤W' = U C V^\top improves over standard weight optimization? Why does the ablation of w/o SVD Decomposition in Table 6 not use the same embedding parameterization and distillation weights? Can you provide a strictly controlled ablation of w/o SVD Decomposition? Is there any comparison to LoRA?

  4. Evaluation Step Discrepancies (0.6M vs. 1.0M): Why is Table 4 evaluated at 0.6M total steps while Figure 3, Figure 4, Table 5, and Table 6 are evaluated at 1.0M total steps? What are the retention percentages of all baselines in Table 4 at the final 1.0M step checkpoint? It seems like the retention percentage degrades when trained longer?

2 comments
AuthorsOpenReview ↗

Response to Reviewer dsi6

We thank the reviewer for recognizing our work's and methodology's value. Below, we address each concern one by one.

[Q1] Based on your question, your understanding for initializing both dynamics and control embedding is correct. We made a General Response (4/4) for analysis of decoupled task embeddings for dynamics and control.

The 5,000 environment steps refer to the total probing budget across the 10 new tasks (500 steps per task), rather than 5,000 steps per task. As detailed in Appendix A.2 and Table 7, MARR uses this budget for embedding retrieval, while ClonEx is allocated the same interaction budget for policy selection (with 4,900 steps actually used). However, not all baselines perform interaction-based retrieval; for example, Multi-SAC fine-tuning uses randomly initialized embeddings. We acknowledge that the annotation in Figure 3 (Right) could be misleading and will revise it to clarify the initialization procedures and interaction budgets of the individual baselines. We thank the reviewer for pointing out this ambiguity.

[Q2] Clarification of Component-wise Distillation Losses.**

We thank the reviewer for pointing out the lack of clarity in the distillation objectives and we have revised this part in paper. Specifically, the component-wise distillation losses for each exemplar transition are defined as follows:

  • Encoder: Lhdist=MSE⁡(z,zˉ)\mathcal{L}_h^{\mathrm{dist}} = \operatorname{MSE}(z,\bar{z}).

  • Dynamics: Lddist=MSE⁡(dθ(z,a),dˉ(zˉ,a))\mathcal{L}_d^{\mathrm{dist}} = \operatorname{MSE}(d_\theta(z,a),\bar{d}(\bar{z},a)).

  • Reward: LRdist=CE⁡(Rˉ(zˉ,a),Rθ(z,a))\mathcal{L}_R^{\mathrm{dist}} = \operatorname{CE}(\bar{R}(\bar{z},a),R_\theta(z,a)).

  • Value: LQdist=CE⁡(Qˉ(zˉ,a),Qθ(z,a))\mathcal{L}_Q^{\mathrm{dist}} = \operatorname{CE}(\bar{Q}(\bar{z},a),Q_\theta(z,a)).

  • Policy: Lπdist=DKL(πˉ(⋅∣zˉ)∥πθ(⋅∣zˉ))\mathcal{L}_\pi^{\mathrm{dist}} = D_{\mathrm{KL}}(\bar{\pi}(\cdot\mid\bar{z})\parallel\pi_\theta(\cdot\mid\bar{z})).

Here, MSE, CE, and KL divergence denote mean squared error, cross-entropy between the softmax distributions of prediction logits, and Kullback--Leibler divergence, respectively. The barred quantities denote outputs of the frozen pretrained world model, which serve as fixed distillation targets.

[Q3] We thank the reviewer for raising this important concern. We have provided a detailed explanation in General Response (3/4) and hope it addresses this question. The dense core provides the greatest adaptation flexibility within the fixed SVD bases by allowing interactions and recombinations across singular directions. However, for non-square weight matrices under compact SVD, updates remain constrained to the subspaces spanned by the retained bases. We emphasize that the effectiveness of this parameterization depends on the richness and diversity of pretrained capabilities, the relatedness between pretrained and new tasks, and the number of tasks being adapted. Our work proposes an empirical framework rather than claiming that a particular core parameterization is universally optimal. Establishing formal theoretical guarantees for these relationships remains an important direction for future work.

Regarding the controlled ablation, we have updated the w/o SVD Decomposition results in General Response (3/4), where only SVD-based parameterization is removed while all other settings remain unchanged.

[Q4] [Q] Evaluation Step Discrepancies (0.6M vs. 1.0M)

We thank the reviewer for pointing out this inconsistency. The original Table 4 reported retention at an intermediate checkpoint of 0.6M total environment steps, whereas the adaptation experiments extended to 1.0M steps. We agree that reporting retention at the final checkpoint provides a more consistent evaluation. To address this concern, we additionally evaluated all baselines at 1.0M steps, as shown below.

Table: Old-task capability retention at different adaptation checkpoints (%).

MethodRetention @ 0.6M (%)Retention @ 1.0M (%)
MARR (Ours)93.50 ± 3.9386.54 ± 4.33
TD-MPC2 FT with Retrieval34.56 ± 4.4622.45 ± 2.86
TD-MPC2 + ER42.00 ± 4.7835.87 ± 3.43
TD-MPC2 + EWC91.00 ± 1.3387.79 ± 2.11
Multi-SAC with ClonEx90.98 ± 0.8782.12 ± 3.91
Multi-SAC with Retrieval and EWC88.75 ± 2.9184.22 ± 3.66
Multi-SAC Fine-tuning10.637.59 ± 2.14
AuthorsOpenReview ↗

Response to Weakness

Response to Weakness [W1] We made a General Response (2/4) and provide additional experiments.

[W2] We thank the reviewer for raising this important concern. Our work is motivated by the observation that classical continual learning typically considers tasks arriving sequentially, with the model adapting to one task at a time. While this setting is important, we believe that task-by-task adaptation may limit the efficiency and scalability of expanding the capabilities of pretrained multitask world models. Therefore, rather than focusing on sequential adaptation, we investigate a complementary setting in which multiple new tasks become available and are learned jointly, leveraging multitask learning to facilitate more efficient capability expansion while retaining previously acquired skills.

We acknowledge that our current evaluation focuses on a single adaptation stage using state-based simulated tasks, many of which share embodiments and dynamics with the pretraining tasks. This setting allows us to investigate the reuse of previously learned representations for new-task adaptation, but does not fully capture the challenges of sequential multi-stage continual learning or adaptation to substantially different embodiments and environments. Extending our framework to multiple sequential adaptation stages, high-dimensional visual observations, and real-world robotic systems represents an important and valuable direction for future work.

**[W3]**We thank the reviewer for raising this important concern regarding computational and memory overhead.

Regarding SVD-based adaptation, we clarify that our dense-core parameterization does not increase the number of trainable parameters compared with standard full weight fine-tuning. For non-square weight matrices(our model weights), the fixed SVD bases additionally constrain the parameter updates to their retained subspaces. Our framework also allows alternative parameterizations, such as diagonal or sparse cores, offering different trade-offs between adaptation flexibility and computational efficiency. We provide further discussion of these design choices in General Response (3/4).

Regarding wall-clock training time, environment simulation and interaction constitute a substantial part of online reinforcement learning, and we parallelize environment execution in our implementation to improve training efficiency. We acknowledge that SVD-based parameterization introduces additional matrix multiplication overhead, and that reporting wall-clock training time and memory consumption would provide a more comprehensive system-level comparison. Although computational efficiency is not the primary focus of this work, we will consider including these measurements in the appendix.

Regarding exemplar memory, we agree that storing historical trajectories and cached pretrained model outputs introduces memory overhead that grows with the number of previously learned tasks. While our current experiments use a limited number of exemplars per task, we view more evaluation on larger-scale, more realistic robotic settings as valuable directions for future work.

Reviewer_xUkMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MARR, a framework that adapts a pretrained multitask world model to multiple new tasks while retaining its capabilities on existing tasks in Multitask RL. MARR retrieves dynamics-related and task-related task embeddings from a short probing trajectory and uses them as initializations for the new tasks, and adapts trainable core matrices on fixed SVD bases. It further employs exemplar-based distillation to retain capabilities on existing tasks. In the experiments, TD-MPC2 pretrained on 70 tasks is simultaneously adapted to 10 new tasks, and the results are favorable in terms of both new-task performance and retention of existing capabilities.

Critical strengths
  • The problem setting of adapting a pretrained multitask world model to new tasks while retaining its existing capabilities is well motivated and is meaningful from the perspective of continually extending generalist agents.
  • The design of retrieving different existing task embeddings for the dynamics-related components and the task-related components is interesting, and it is intuitively reasonable in that it exploits different kinds of task similarity.
  • The experimental results are favorable overall; in particular, the fact that the method achieves both adaptation to new tasks and retention of existing tasks supports the effectiveness of the proposed method.
Critical weaknesses
  • While MARR consists of multiple modules, including embedding retrieval, SVD-based adaptation, and exemplar-based distillation, ablations that sufficiently disentangle the contribution of each are lacking. In particular, it is somewhat unclear which components play the main role in improving new-task performance and capability retention.
  • Although SVD-based adaptation itself builds on prior work, using it for capability retention is an interesting aspect of this paper. However, since the "w/o SVD Decomposition" variant in Table 6 also differs in embedding parameterization and distillation weights, it is difficult to directly assess the contribution of SVD-based adaptation itself to retention.
  • The effect of the separate retrieval for the dynamics-related and task-related components, a relatively distinctive element of this paper, is also not sufficiently isolated from the other components. TD-MPC2 FT with retrieval shares a single embedding across all components, but at the same time it uses neither SVD-based adaptation nor exemplar distillation. Therefore, while the novelty of the proposed method is understandable, the extent to which each new element contributes to the final performance is not fully clear from the current experiments.
Questions for authors
  • In the "w/o SVD Decomposition" variant of Table 6, settings other than SVD are also changed. Given this confounding, how do the authors interpret the role that SVD-based adaptation plays in capability retention?
  • Retrieving embeddings separately for the dynamics-related and task-related components appears to be a distinctive design choice of MARR. However, the current experiments do not isolate its effect: the “w/o Retrieval” ablation removes retrieval altogether, and TD-MPC2 FT with retrieval also differs in other components. Could the authors clarify which claims about separate retrieval are supported by the current results, and limit the claims in the paper accordingly?
2 comments
AuthorsOpenReview ↗

Response to Reviewer xUkM

We sincerely thank the reviewer for the constructive feedback and for recognizing the practical value of our work. Below, we address each of the raised questions point by point.

[Q1] To clarify the motivation of SVD parameterization, we provide a analysis of test gradients during update at 1M interactions, the cosine between the mean gradients of two DMC tasks, or of a DMC and a Meta-World task, is 0.00 on average (ranges −0.13 to +0.08 and −0.01 to +0.02, respectively), and the same holds at every checkpoint from 5k to 1M. The exception is the Meta-World tasks, which share the same robot: their pairwise cosines range from −0.50 to +0.56, so some pairs are aligned and others opposed. The new-task objective and the distillation objective are only weakly aligned (per-update cosine +0.17 ± 0.06 at 1M) and point in opposing directions in about one fifth of updates (21%), which indicates gradients of different new tasks may be nearly orthogonal.

To clarify the contribution of SVD-based adaptation, we conducted a strictly controlled ablation in which only the SVD-based weight parameterization was removed, while all other settings, including embedding parameterization, distillation weights, and training configurations, remained unchanged. We have updated Table 6 accordingly and corrected the misleading description in the original caption.

Table 6. Performance for Ablations. Normalized scores after 1M total adaptation interactions. We report mean ± sample standard deviation over three training seeds.

MethodOriginal 70 TasksNew 10 Tasks
MARR (Ours)43.45 ± 2.4173.41 ± 6.88
w/o Exemplar Distillation17.11 ± 1.1468.73 ± 3.97
w/o Retrieval38.02 ± 2.3167.35 ± 6.60
w/o SVD Decomposition28.33 ± 0.8372.91 ± 4.57

We also clarify in General Response (3/4) the synergistical design of Component-wise distillation and SVD-based adaptation and how they work together to retain previously learned tasks behavior.

[Q2] We made a "General Response (4/4)Analysis of Split Embedding Retrieval" and hope it can address your concern on this aspect.

The additional controlled ablations substantially address my original concerns regarding SVD-based adaptation and split embedding retrieval. The results now more clearly support their respective contributions to capability retention and new-task adaptation.

While several components build on existing techniques, the main novelty lies in their integration for multitask world-model adaptation with capability retention. Given the strong empirical results and the clarified ablations, I maintain my Weak Accept recommendation.

Reviewer_EGtcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a method for continually adapting a pretrained multitask TD-MPC2 world model to new tasks while trying to prevent catastrophic forgetting on the original pretraining tasks. The approach first collects random steps on each new task and picks two existing task embeddings that minimize TD-MPC2's consistency and TD losses on the collected data. They use an approach based on SVD to only update parts of the model weights in the adaptation phase. They store high-return trajectories for each old task in a buffer and use a weighted loss to optimize for old and new tasks. The authors compare perf against naive fine-tuning of TD-MPC2 and three Multi-SAC baselines.

Critical strengths
  1. The problem setup, ie, taking an existing pretrained multitask world model such as TD-MPC2 and extending it to support new tasks without retraining on the full environment from scratch is interesting and can be of interest to the WM community.
Critical weaknesses
  1. The adaptation mechanism and loss functions are vaguely described, and there is not enough technical details in Section 3 about how the model is trained. The distillation losses in Sec 3.3 are never defined. The pseudocode in appendix mentions updating the old task embeddings which is confusing given that the goal is preserving the behavior on old tasks. In table 8 and appendix A.3 the authors mention almost 5 million new learnable parameters for a 5-million parameter TDMPC2 checkpoint which defeats the purpose of adaptation (doubles the number of parameters).

  2. The experimental evaluation is flawed. The baselines are model-free SAC methods that are not directly comparable to model-based TDMPC2. And there is no baseline that combines TDMPC w/ common continual learning approaches such as EWC or replay (only naive finetuning of TDMPC2 is done).

Questions for authors
  1. Why do you need to update the old task embeddings and how can doing that help preserve the old tasks perf after adaptation?

  2. How can you justify doubling the number of model params using the SVD-based fine-tuning?

  3. Can you provide model-based continual adaption baselines for a fair comparison?

1 comment
AuthorsOpenReview ↗

Response to Reviewer EGtc

We thank the reviewer for these questions and the opportunity to clarify our design.

[Q1] Updating old-task embeddings.

We thank the reviewer for this question. Our objective is to find an adapted model that retains sufficient plasticity to learn new tasks while preserving previously acquired capabilities, rather than enforcing exact equivalence to the pretrained model’s input–output mapping for each task differentiating by task embedding. In our framework, the new-task learning objectives and old-task exemplar distillation objectives are optimized jointly. Distillation provides a soft, sample-based regularization toward the pretrained model’s behavior, rather than a hard requirement that its outputs remain unchanged.

Accordingly, we allow old-task embeddings to adjust alongside the shared parameters, giving the joint optimization additional flexibility to find a solution that accommodates both old and new tasks. Their updates remain subject to the old-task distillation objectives, whose pretrained-model targets are fixed throughout adaptation. Thus, the purpose of updating these embeddings is to support coordinated adaptation under an explicit retention objective; preserving old-task capabilities does not require freezing the embeddings or reproducing every pretrained output exactly. To investigate this, we compare MARR with a variant that freezes the old-task embeddings while keeping the new-task embeddings trainable. The shared-model adaptation mechanism, distillation objectives, and other training settings remain unchanged.

MethodOld 70 Tasks Retention (%) at 0.6MOld 70 Tasks Retention (%) at 1.0M
MARR: Trainable Old-task Embeddings93.50 ± 3.9386.54 ± 4.33
MARR: Frozen Old-task Embeddings91.76 ± 2.4483.11 ± 2.88

[Q2] Parameter count of SVD-based fine-tuning.

We clarify that the dense core replaces the original weight matrix as the trainable parameter, rather than introducing an additional trainable copy of that matrix. For a weight matrix with m rows and n columns, the dense core contains r × r trainable coefficients, where r = min(m, n), which is no greater than the original m × n coefficients. The left and right singular-vector bases remain frozen. Accordingly, Table 8 reports approximately 4.6M trainable core coefficients for the 5M model, with task embeddings trained separately; the SVD parameterization does not double the number of trainable parameters.

For non-square matrices under compact SVD, these bases also restrict the available update directions. The design rationale and the strictly controlled ablation supporting its contribution to retention are discussed in General Response (3/4).

[Q3] Model-based continual adaptation baselines.

We thank the reviewer for suggesting these comparisons. We have added TD-MPC2 with Elastic Weight Consolidation (EWC) and TD-MPC2 with Experience Replay (ER), both built on the same pretrained TD-MPC2 backbone, to evaluate established retention mechanisms within the same model-based learning framework. Please see General Response (2/4) for the experimental settings, quantitative results, and discussion.

Response to weakness Thanks for pointing out concers.we have revised section 3 to better present details. We also provide additional experiments as stated in response to question 3.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response (1/4): Our Contributions and Intuitive Overview of Methodology

We thank all three reviewers for their constructive feedback. The common concerns regarding the fairness of continual learning baselines, the roles and contributions of exemplar-based distillation and SVD-based adaptation, and the effectiveness of split embedding retrieval are important questions for understanding and evaluating MARR. We address these concerns through additional experiments, analyses, and methodological clarifications below. We first briefly highlight our motivation, methodology, and contributions.

Why this matters

Generalist world models can learn many tasks jointly. However, joint training on all desired tasks is not always feasible, as limited resources constrain access to collected data and simulation environments at a given time. How can one agent efficiently adapt to several diverse new tasks by reusing a trained model to acquire new capabilities while preserving what it already knows? We hope this work provide a potential to extend multitask model's ability for continuous control.

The idea

Our approach follows a simple intuition: explore briefly, recognize which prior knowledge is useful, and use it to initialize adaptation. We retrieve relevant dynamics and control representations from previously learned tasks, then build on them to learn multiple new tasks simultaneously.

Methodology

Within a multitask world model, each component can be viewed as a task-conditioned function and task indices are mapped to learnable embedding vectors, enabling shared components to produce task-specific outputs and behaviors. Our design addresses two complementary goals: initializing learning on new tasks with relevant representations and function behaviors acquired from previous tasks, and preserving the component outputs and behaviors associated with those previous tasks. We address these goals as follows:

  1. To provide an effective initialization for adaptation, we design an embedding retrieval mechanism that separately retrieves dynamics-related and control-related representations from previously learned tasks. The retrieved embeddings allow the model to reuse relevant learned functional behaviors when adapting to new tasks.

  2. To retain capabilities on original tasks, exemplar-based distillation and SVD-based adaptation are designed to work together. Exemplar-based distillation uses stored exemplars and original-model outputs to regularize the model's component outputs and behaviors on previous tasks, introducing additional loss terms for capability retention. Combined with the learning objectives of multiple new tasks, these terms turn adaptation into a multitask optimization problem. SVD-based adaptation parameterizes the selected weights using fixed pretrained singular-vector bases and trainable core matrices, which are optimized under this combined objective.

Contributions

  1. Multitask adaptation with retention: From a random interaction context, MARR provides an effective initialization and adapts a single agent to multiple new tasks jointly while preserving its capabilities.

  2. Adaptation across diverse tasks: The new tasks span locomotion and manipulation.

  3. Scaling with model capacity: We examine whether larger models provide a stronger foundation for adaptation.

AuthorsOpenReview ↗

General Response (2/4): Model-Based Continual Learning Baselines

We thank the reviewers for raising this important concern. To address it, we provide additional experiments comparing MARR with TD-MPC2 equipped with established continual learning methods, including Elastic Weight Consolidation (EWC) and Experience Replay (ER). We have also incorporated these two baselines into the main paper's results in Figure 3.

Zero-shot performance. Since EWC and ER operate during the optimization stage, while zero-shot evaluation is conducted before any gradient updates, neither mechanism affects zero-shot performance.

New-task adaptation and old-task retention. We find that TD-MPC2 with ER can efficiently learn new tasks but exhibits substantial forgetting of previously learned capabilities. Under our problem setting, access to large amounts of data from previously learned tasks is not feasible. We attribute this to extrapolation error in off-policy learning from a small, fixed memory (Fujimoto et al., 2019; Kumar et al., 2020). The TD targets bootstrap from value estimates at actions proposed by the current policy, which lie outside the support of the stored trajectories, and the replayed policy-improvement step then optimizes the original-task policies toward these unconstrained estimates. Because TD-MPC2 plans with its learned reward and value functions, such errors translate directly into poor actions. Exemplar distillation avoids both issues: it regresses onto original model's behabior, which requires no bootstrapping, and it constrains the policy explicitly through a KL divergence to the original policy.

In contrast, TD-MPC2 with EWC achieves retention comparable to MARR, but its ability to learn new tasks is substantially limited. EWC mitigates forgetting by penalizing changes to parameters estimated to be important for previously learned tasks, using a Fisher-information-based regularization term. In our multitask world model, tasks may share embodiments and reuse the same encoder, latent dynamics, and task-conditioned reward, value, and policy networks. Parameters important for old tasks may also be essential for learning new ones. Moreover, parameter-wise regularization does not directly distinguish between parameter changes that disrupt old-task behavior and those that preserve it. This may lead to overly restrictive updates, limiting trade off retention against plasticity when the protected and the needed coordinates overlap. In contrast, exemplar distillation regularizes the model for function behavior by aligning its predictions and policy behavior on stored old-task trajectories, allowing more flexibility in parameter adaptation while preserving previously learned capabilities.

Numerical results are provided in the following table:

MethodNew 10 Tasks (Normalized Score)Old 70 Tasks Retention (%)
TD-MPC2 + FT62.42 ± 3.5134.56 ± 4.46
TD-MPC2 + ER71.41 ± 6.8842.00 ± 4.78
TD-MPC2 + EWC56.51 ± 5.4191.00 ± 1.33
MARR (Ours)73.41 ± 6.8893.50 ± 3.93

Except MARR, others use randomly initialized task embeddings for the new tasks here and are evaluated on 1M steps as well. Interestingly, we observe that TD-MPC2 + ER achieves better new-task learning performance than naive TD-MPC2 fine-tuning. We think that this comes from the repeated replay of stored data alongside newly collected experience from the new tasks, effectively combining offline data from previously learned tasks with online data from new tasks during adaptation. Such mixed replay may provide additional learning signals and help stabilize the shared world model, thereby facilitating new-task learning.

References

[1] Fujimoto, S., Meger, D., & Precup, D. (2019). Off-Policy Deep Reinforcement Learning without Exploration. Proceedings of the 36th International Conference on Machine Learning (ICML), 2052–2062. https://proceedings.mlr.press/v97/fujimoto19a.html

[2] Kumar, A., Zhou, A., Tucker, G., & Levine, S. (2020). Conservative Q-Learning for Offline Reinforcement Learning. Advances in Neural Information Processing Systems (NeurIPS), 33. https://proceedings.neurips.cc/paper/2020/hash/0d2b2061826a5df3221116a5085a6052-Abstract.html

AuthorsOpenReview ↗

General Response (3/4): Clarification on Design of Component-wise Distillation and SVD-based Adaptation

Component-wise distillation and SVD-based adaptation are synergistically designed to support the retention of previously acquired capabilities. Component-wise distillation aims to preserve the functional behavior of the pretrained world model rather than directly constrain its parameters. Meanwhile, as joint adaptation to multiple new tasks updates the shared model weights, SVD-based adaptation parameterizes these updates within the structured singular-vector bases learned during pretraining, with the goal of reusing previously acquired representations while mitigating interference with existing capabilities. We refine our writing on section 3.3 EXEMPLAR-BASED COMPONENT DISTILLATION to make training process clearer.

Specifically, a diagonal core restricts adaptation to adjusting the original singular-direction pairs, whereas a dense core additionally allows interactions and recombinations across different singular directions, providing greater flexibility within the retained bases. For non-square weight matrices (our model weights) under compact SVD, dense core still remain constrained by the subspaces spanned by the fixed bases. Importantly, neither parameterization is universally preferable and also our framework allows other forms of sparse core matrix: the appropriate balance between structural constraints and adaptation flexibility depends on the richness and diversity of the pretrained capabilities, the relatedness between pretrained and new tasks, and the number of tasks being adapted.

Our framework is motivated by the empirical premise that foundation models pretrained across diverse tasks and robot embodiments can acquire rich, reusable representations that support subsequent adaptation. We do not claim universal guarantees regarding the effectiveness of a particular core parameterization. Establishing theoretical guarantees that characterize how task relatedness and the numbers of pretrained and newly introduced tasks affect adaptation and retention remains an important direction for future work.

We also provide a analysis of test gradients during update at 1M interactions, the cosine between the mean gradients of two DMC tasks, or of a DMC and a Meta-World task, is 0.00 on average (ranges −0.13 to +0.08 and −0.01 to +0.02, respectively), and the same holds at every checkpoint from 5k to 1M. The exception is the Meta-World tasks, which share the same robot: their pairwise cosines range from −0.50 to +0.56, so some pairs are aligned and others opposed. The new-task objective and the distillation objective are only weakly aligned (per-update cosine +0.17 ± 0.06 at 1M) and point in opposing directions in about one fifth of updates (21%), which indicates gradients of different new tasks may be nearly orthogonal.

To clarify the contribution of SVD-based adaptation, we conducted a strictly controlled ablation in which only the SVD-based weight parameterization was removed, while all other settings, including embedding parameterization, distillation weights, and training configurations, remained unchanged. We have updated Table 6 accordingly and corrected the misleading description in the original caption.

Table 6. Performance for Ablations. Normalized scores after 1M total adaptation interactions. We report mean ± sample standard deviation over three training seeds.

MethodOriginal 70 TasksNew 10 Tasks
MARR (Ours)43.45 ± 2.4173.41 ± 6.88
w/o Exemplar Distillation17.11 ± 1.1468.73 ± 3.97
w/o Retrieval38.02 ± 2.3167.35 ± 6.60
w/o SVD Decomposition28.33 ± 0.8372.91 ± 4.57

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.