- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on a continual learning scenario in RL: adapting a pretrained multitask world model to a batch of newly accessible tasks. Building on the TD-MPC2 pretrained on 70 tasks across DeepMind Control Suite (DMControl) and Meta-World, the authors propose MARR, a framework combining three mechanisms for adaptation to 10 new tasks while retaining performance on previously learned tasks:
-
Embedding Retrieval: Using a short random-action probing context (500 transitions per new task), MARR retrieves a dynamics embedding by minimizing latent consistency loss for the encoder and dynamics modules, and a control embedding by minimizing reward and temporal-difference losses for the task-related modules.
-
Exemplar-Based Component Distillation: To mitigate catastrophic forgetting on previously learned tasks, MARR caches the highest-return trajectories of the old tasks and regularizes the encoder, dynamics, reward, value, and policy prior outputs during adaptation.
-
SVD-Based Weight Parameterization: Linear weight matrices are decomposed via singular value decomposition (), freezing the left and right orthogonal bases while optimizing a dense core matrix initialized to .
Experiments show that MARR outperforms several baselines, including vanilla TD-MPC2 fine-tuning and continual learning variants of Multi-SAC.
Critical strengths
-
Clear and Modular Formulation: The paper is clearly written and easy to follow. Decoupling the task conditioning of a world model into dynamics-related components and task/reward-related components is an intuitive design choice for transfer learning in latent world models, as tasks may share physical embodiment dynamics while differing in reward structure (or vice versa).
-
Comprehensive Component-Wise Distillation: Rather than only distilling the final policy behavior (as in behavioral cloning on exemplars), regularizing the internal world-model predictions (encoder representations, latent transitions, rewards, and -values) directly targets representation drift in shared latent spaces.
Critical weaknesses
-
Conflation of Backbone RL Architecture with Adaptation/Retention Mechanisms in Baselines: All continual learning baselines with retention mechanisms are built on a model-free Multi-SAC backbone, whereas MARR is built on TD-MPC2. There are no continual learning baselines implemented on the TD-MPC2 backbone to provide a meaningful comparison to the adaptation mechanisms this paper proposes. As shown in Figure 3 (Left) and Figure 4, all Multi-SAC variants—including unconstrained
Multi-SAC FT—fail to learn the 10 new tasks within the step budget, whereas both TD-MPC2 variants learn well. This gap could just reflect the intrinsic sample-efficiency advantage of online MPC planning in TD-MPC2 or loss of plasticity in Multi-SAC network rather than the superiority of the proposed continual adaptation algorithm. -
Simplicity of Task Suite and Realism of the Single-Batch Adaptation Scenario: Despite framing the problem around continual adaptation of generalist robot foundation models, the evaluation only considers a single adaptation stage where low-dimensional state-based simulated tasks arrive simultaneously and are trained jointly with task-balanced batches. This single-step joint batch expansion does not test sequential multi-stage continual learning (where forgetting compounds across multiple adaptation rounds) or high-dimensional visual/real-world robotic settings. Moreover, the 70 pretraining tasks already contain dense coverage of the exact same MuJoCo embodiments and manipulation objects, making the 10 held-out tasks very close interpolations of the pretraining distribution.
-
Lack of System-Level Comparisons: The paper focuses on comparing performance under a fix budget, but lacks a system-level analysis of memory and compute overhead. Specifically, exemplar distillation requires storing trajectory records and cached original-model component outputs, which scales linearly with the number of prior tasks. Similarly, parameterizing weights as with frozen and dense cores increases the per-step forward/backward matrix multiplication cost compared to standard weight fine-tuning, yet wall-clock training time and memory footprints are not taken into account in the comparison or even reported.
Questions for authors
-
Clarification on
TD-MPC2 FT with retrieval: IIUC, TD-MPC2 with retrieval means that the new task embeddings are initialized from manually chosen existing tasks? In addition, the same task is used for initializing both dynamics and control embedding? Is this understanding correct? If so, does the benefit of embedding retrieval solely come from decoupling task for dynamics and control embeddings? Could you provide an ablation comparing TD-MPC2 with decoupled task embeddings for dynamics and control? And why does Figure 3 (Right) attribute 5,000 retrieval steps to this baseline? -
Clarification on distillation: Section 3.3 introduces and without giving their explicit mathematical formulas. Are these simply (MSE) losses on the encoder/dynamics latents and scalar reward/value predictions?
-
Role of Full-Rank Dense SVD and Controlled Ablation: Given that retaining almost all singular vectors and training a dense matrix leaves 4.6M trainable parameters and does not restrict the rank or subspace of the hidden matrices (Table 8), what is the theoretical or empirical mechanism by which improves over standard weight optimization? Why does the ablation of
w/o SVD Decompositionin Table 6 not use the same embedding parameterization and distillation weights? Can you provide a strictly controlled ablation ofw/o SVD Decomposition? Is there any comparison to LoRA? -
Evaluation Step Discrepancies (0.6M vs. 1.0M): Why is Table 4 evaluated at 0.6M total steps while Figure 3, Figure 4, Table 5, and Table 6 are evaluated at 1.0M total steps? What are the retention percentages of all baselines in Table 4 at the final 1.0M step checkpoint? It seems like the retention percentage degrades when trained longer?
2 comments
Response to Weakness
Response to Weakness [W1] We made a General Response (2/4) and provide additional experiments.
[W2] We thank the reviewer for raising this important concern. Our work is motivated by the observation that classical continual learning typically considers tasks arriving sequentially, with the model adapting to one task at a time. While this setting is important, we believe that task-by-task adaptation may limit the efficiency and scalability of expanding the capabilities of pretrained multitask world models. Therefore, rather than focusing on sequential adaptation, we investigate a complementary setting in which multiple new tasks become available and are learned jointly, leveraging multitask learning to facilitate more efficient capability expansion while retaining previously acquired skills.
We acknowledge that our current evaluation focuses on a single adaptation stage using state-based simulated tasks, many of which share embodiments and dynamics with the pretraining tasks. This setting allows us to investigate the reuse of previously learned representations for new-task adaptation, but does not fully capture the challenges of sequential multi-stage continual learning or adaptation to substantially different embodiments and environments. Extending our framework to multiple sequential adaptation stages, high-dimensional visual observations, and real-world robotic systems represents an important and valuable direction for future work.
**[W3]**We thank the reviewer for raising this important concern regarding computational and memory overhead.
Regarding SVD-based adaptation, we clarify that our dense-core parameterization does not increase the number of trainable parameters compared with standard full weight fine-tuning. For non-square weight matrices(our model weights), the fixed SVD bases additionally constrain the parameter updates to their retained subspaces. Our framework also allows alternative parameterizations, such as diagonal or sparse cores, offering different trade-offs between adaptation flexibility and computational efficiency. We provide further discussion of these design choices in General Response (3/4).
Regarding wall-clock training time, environment simulation and interaction constitute a substantial part of online reinforcement learning, and we parallelize environment execution in our implementation to improve training efficiency. We acknowledge that SVD-based parameterization introduces additional matrix multiplication overhead, and that reporting wall-clock training time and memory consumption would provide a more comprehensive system-level comparison. Although computational efficiency is not the primary focus of this work, we will consider including these measurements in the appendix.
Regarding exemplar memory, we agree that storing historical trajectories and cached pretrained model outputs introduces memory overhead that grows with the number of previously learned tasks. While our current experiments use a limited number of exemplars per task, we view more evaluation on larger-scale, more realistic robotic settings as valuable directions for future work.
Response to Reviewer dsi6
We thank the reviewer for recognizing our work's and methodology's value. Below, we address each concern one by one.
[Q1] Based on your question, your understanding for initializing both dynamics and control embedding is correct. We made a General Response (4/4) for analysis of decoupled task embeddings for dynamics and control.
The 5,000 environment steps refer to the total probing budget across the 10 new tasks (500 steps per task), rather than 5,000 steps per task. As detailed in Appendix A.2 and Table 7, MARR uses this budget for embedding retrieval, while ClonEx is allocated the same interaction budget for policy selection (with 4,900 steps actually used). However, not all baselines perform interaction-based retrieval; for example, Multi-SAC fine-tuning uses randomly initialized embeddings. We acknowledge that the annotation in Figure 3 (Right) could be misleading and will revise it to clarify the initialization procedures and interaction budgets of the individual baselines. We thank the reviewer for pointing out this ambiguity.
[Q2] Clarification of Component-wise Distillation Losses.**
We thank the reviewer for pointing out the lack of clarity in the distillation objectives and we have revised this part in paper. Specifically, the component-wise distillation losses for each exemplar transition are defined as follows:
Encoder: Lhdist=MSE(z,zˉ).
Dynamics: Lddist=MSE(dθ(z,a),dˉ(zˉ,a)).
Reward: LRdist=CE(Rˉ(zˉ,a),Rθ(z,a)).
Value: LQdist=CE(Qˉ(zˉ,a),Qθ(z,a)).
Policy: Lπdist=DKL(πˉ(⋅∣zˉ)∥πθ(⋅∣zˉ)).
Here, MSE, CE, and KL divergence denote mean squared error, cross-entropy between the softmax distributions of prediction logits, and Kullback--Leibler divergence, respectively. The barred quantities denote outputs of the frozen pretrained world model, which serve as fixed distillation targets.
[Q3] We thank the reviewer for raising this important concern. We have provided a detailed explanation in General Response (3/4) and hope it addresses this question. The dense core provides the greatest adaptation flexibility within the fixed SVD bases by allowing interactions and recombinations across singular directions. However, for non-square weight matrices under compact SVD, updates remain constrained to the subspaces spanned by the retained bases. We emphasize that the effectiveness of this parameterization depends on the richness and diversity of pretrained capabilities, the relatedness between pretrained and new tasks, and the number of tasks being adapted. Our work proposes an empirical framework rather than claiming that a particular core parameterization is universally optimal. Establishing formal theoretical guarantees for these relationships remains an important direction for future work.
Regarding the controlled ablation, we have updated the w/o SVD Decomposition results in General Response (3/4), where only SVD-based parameterization is removed while all other settings remain unchanged.
[Q4] [Q] Evaluation Step Discrepancies (0.6M vs. 1.0M)
We thank the reviewer for pointing out this inconsistency. The original Table 4 reported retention at an intermediate checkpoint of 0.6M total environment steps, whereas the adaptation experiments extended to 1.0M steps. We agree that reporting retention at the final checkpoint provides a more consistent evaluation. To address this concern, we additionally evaluated all baselines at 1.0M steps, as shown below.
Table: Old-task capability retention at different adaptation checkpoints (%).