acceptodds
Under review as a conference paper at ICLR 2027

Exploiting Model Embedding Memory in Policy Learning

Abstract

Adapting a pretrained model to downstream tasks produces a collection of task-specific adaptations that can serve as reusable model memory. However, existing weight-space representations, such as full-parameter or low-rank, are primarily designed for optimization rather than memory organization: they lack a shared representation in which task-specific adaptations can be compactly stored, compared, and retrieved. In this work, we propose a model embedding space, a low-dimensional representation of task adaptations constructed in alignment with the intrinsic structure of pretrained weights. This space provides a shared coordinate system for organizing task-specific knowledge into a model embedding memory, enabling compact storage and similarity-based retrieval across adapted models. To make this memory both efficient to search and easy to access from other modalities, we further introduce a model embedding index: a lightweight representation that preserves retrieval-relevant information while forming a compact interface between model embeddings and language representations. We demonstrate the framework in robot policy learning, where task-specific policies are encoded as model embeddings and retrieved from memory for target tasks. We further enable language-guided retrieval by mapping LLM embeddings to the model embedding index, showing that the model embedding space provides a structured memory for storing and reusing task-specific policies.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
memory, policy learning, retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TUk4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes a model embedding memory for robot policy learning. It uses the SVD of pretrained weights to build a shared embedding space, where task-specific adapters can be stored and compared. A compact index is used for efficient retrieval, and language instructions can also be mapped into this space to retrieve relevant policies. Experiments on Meta-World, LIBERO, and RoboCasa show promising results in parameter efficiency, retrieval, and task performance.

Critical strengths
  • The idea of using task-specific adapters as a searchable memory is interesting and practical. It connects parameter-efficient fine-tuning with policy retrieval in a simple framework.

  • The experimental results are promising. The method achieves comparable performance to LoRA with fewer parameters, and the retrieval experiments show clear efficiency gains.

Critical weaknesses
  • The novelty seems somewhat limited. SVD-based fine-tuning has already been explored in works like SVDiff and Spectral Adapter, while model and skill retrieval have also been studied. The main contribution seems to be combining these ideas, but the paper does not clearly show why this particular combination is better than existing alternatives.

  • Some important baselines and ablations are missing. For example, using a LoRA adapter bank with the same retrieval mechanism would be a natural baseline. Also, it is unclear whether the pretrained SVD basis is actually necessary, since a shared random orthogonal basis might provide similar geometric properties. The current experiments do not fully isolate where the improvements come from.

Questions for authors
  • How does the method compare with Spectral Adapter or a LoRA adapter bank using the same retrieval mechanism and task-specific training setup?

  • Have you tried using a random fixed orthogonal basis instead of the pretrained SVD basis? Would the task clustering and retrieval performance still hold?

4 comments
AuthorsOpenReview ↗

Rebuttal (Part1)

We thank the reviewer for their constructive feedback. We address the comments below and have revised the manuscript accordingly.

(W1, Q1) What is the contribution of this paper compared to existing work? Compare with the spectral-adaptation methods and other retrieval methods.

A: Our contribution is investigating how to construct a structured model embedding space in which task-specific adaptations can be learned, represented (Sec. 2.1), organized, and retrieved (Sec. 2.2-2.4, Sec. 3) within a shared coordinate system derived from the pretrained model.

Comparison with Spectral-adaptation methods. Let the pretrained weight matrix have SVD W0=UΣV⊤W_0=U\Sigma V^\top. Model embedding directly optimizes task-specific coordinates Mi(h)M^{(h)}_i within the subspace, spanned by the pretrained bases U(h)U^{(h)} and V(h)V^{(h)}: ΔWi=∑U(h)Mi(h)V(h)⊤.\Delta W_i= \sum U^{(h)} M^{(h)}_i V^{(h)\top}.

  • Spectral Adapter introduces additive and rotational adaptations of selected singular vectors. Its formulation is Wi=(U+ΔUi)Σ(V+ΔVi)⊤W_i=(U+\Delta U_i) \Sigma (V+\Delta V_i)^\top, where only selected singular-vector columns are modified. In contrast, model embedding keeps UU and VV fixed.

  • SVDiff adapts pretrained weights by updating only their singular values while keeping the singular-vector bases fixed Wi=Udiag⁡(ReLU⁡(σ+δi))V⊤.W_i=U\operatorname{diag}(\operatorname{ReLU}(\sigma+\delta_i))V^\top. In contrast, Model Embedding learns MiM_i within structured pretrained subspaces, enabling interactions between spectral directions and providing greater adaptation expressivity than SVDiff's diagonal-only parameterization.

We further evaluated Spectral Adapter and SVDiff on LIBERO to investigate the differences between these formulations. All results are obtained using oracle task-ID selection, without retrieval, to isolate task-specific adaptation performance.

MethodRankSpatial ↑Object ↑Goal ↑LIBERO-10 ↑Overall ↑Params ↓
Spectral Adapter1686.6%75.4%87.4%71.6%80.3%0.9M
Spectral Adapter6490.9%86.0%89.1%73.7%84.9%3.6M
SVDiff-25.0%7.8%35.6%0.0%17.1%1.5K
Model Embedding6491.5%88.5%90.5%74.0%86.1%0.9M

At a matched parameter budget of 0.9M, our method improves average success over Spectral Adapter. SVDiff provides a much smaller adaptation budget but achieves substantially lower success, indicating that its restricted adaptation capacity is insufficient for these robotic policy-learning tasks under the evaluated configuration.

Same retrieval mechanism for other adaptation methods. To further isolate the contribution of the proposed representation from task-specific adaptation and retrieval, we conducted additional controlled comparisons against Spectral Adapter, LoRA, and full fine-tuning on LIBERO, using the same MTACT backbone and experimental protocol described in Sec. 4.2.

We consider two settings: (i) oracle task-ID selection, where the correct task-specific adapter is directly selected; and (ii) using the same language-guided retrieval mechanism to select the adaptation.

Oracle task-ID selection:

MethodSpatialObjectGoalLIBEO-10OverallTrainable Params
LoRA89.8%81.6%90.2%71.4%83.25%2.4M
Spectral Adapter86.6%75.4%87.4%71.6%80.3%0.9M
Full Fine-tuning94.4%88.4%96.4%82.4%90.4%53M
Model Embedding91.5%88.5%90.5%74.0%86.12%0.9M

Language-guided retrieval mechanism:

MethodSpatialObjectGoalLIBERO-10OverallTrainable Params
LoRA89.8%81.6%90.2%69.4%82.75%2.4M
Spectral Adapter86.6%75.4%87.4%69.1%79.6%0.9M
Full Fine-tuning94.4%88.4%96.4%78.4%89.4%53M
Model Embedding91.5%88.5%90.5%71.0%85.4%0.9M

The performance gap between oracle task-ID selection and actual retrieval is small, as the language-guided router achieves nearly perfect task identification. Only one task in LIBERO-10 is assigned an incorrect adapter, leading to a slight performance degradation.

Model Embedding demonstrates effective task adaptation with much fewer parameters. Under oracle selection, it achieves 86.1% overall success, outperforming LoRA and Spectral Adapter. It also approaches full fine-tuning performance with significantly fewer parameters. These results isolate the contribution of the adaptation representation. The performance advantage persists under actual retrieval. With the shared retrieval mechanism, Model Embedding achieves 85.4% overall success compared to the LoRA adapter bank, while retaining a smaller per-task parameter budget.

These results demonstrate that our structured model embedding space provides a favorable trade-off between adaptation performance and parameter efficiency.

AuthorsOpenReview ↗

Rebuttal (Part2)

(W2, Q2) Compare with the shared random orthogonal basis.

A: To investigate whether pretrained spectral directions are necessary, we replace the SVD bases with fixed random orthogonal bases while preserving the block structure and trainable parameter budget.

MethodSpatialObjectGoalLIBERO-10OverallParams
Model Embedding (Random basis)79.0%56.0%68.0%23.0%56.5%0.9M
Model Embedding (SVD basis)91.5%88.5%90.5%74.0%86.1%0.9M

Replacing the pretrained SVD basis with a random orthogonal basis reduces overall success from 86.1% to 56.5%, with consistent degradation across all four LIBERO suites. Since both variants use identical block structures and trainable parameter counts, this result isolates the contribution of the basis choice rather than model capacity.

The results indicate that the pretrained SVD basis provides task-effective adaptation directions that cannot be replaced by an arbitrary shared orthogonal basis without substantial performance degradation under the evaluated configuration.

To further assess the impact of pretrained versus random orthogonal bases on retrieval performance, we compare language-guided retrieval accuracy under both settings.

MethodTop-1Top-3Top-5
Ours (random basis)92.5%97.0%98.0%
Ours (SVD basis)94.0%99.50%99.50%

The pretrained SVD basis consistently outperforms the random basis, improving Top-1, Top-3, and Top-5 retrieval accuracy. Notably, the random basis still achieves strong retrieval accuracy, suggesting that a shared coordinate system supports task discrimination even without pretrained spectral alignment. However, the pretrained basis provides additional benefits for retrieval, indicating that its structure contributes to more effective task representations.

Combined with the adaptation results, these findings demonstrate that pretrained spectral structure benefits both task adaptation and retrieval, with a substantially larger impact on adaptation performance.

Additional experimental results: In our response to reviewer wAsr, we also include the other experiments.

  • Spectral-based methods (wAsr, Part1): Additionally, we also compare our method with Spectral Rewiring.

  • Retrieval experiments (wAsr, Part3): We compare our language-guided retrieval method with baselines including text-based nearest-neighbor retrieval, a supervised task classifier, LoraRetriever, RAMoLE, Arrow-style routing, and a reusable LoRA adapter library (MBC).

Thank you for the detailed response and additional experiments. The comparisons with Spectral Adapter, SVDiff, and LoRA under the same retrieval setting address my main concerns. In particular, the random orthogonal basis experiment provides convincing evidence that the pretrained spectral structure is important for effective task adaptation, rather than simply providing a shared coordinate system.

While I still think the methodological novelty is somewhat incremental, the new results better establish the benefits of the proposed representation in terms of adaptation performance and parameter efficiency. My main concerns have been largely addressed, and I am raising my rating to weak acceptance.

Thank you for your positive feedback and support. We sincerely appreciate your time and thoughtful evaluation of our work.

Reviewer_wAsrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes model embedding memory (MEM), a framework for storing and retrieving task-specific policy adaptations in a shared spectral coordinate system. For each pretrained weight W=UΛV⊤W=U\Lambda V^\top, a task update is parameterized as ΔW=UMV⊤\Delta W=UMV^\top, with block-diagonal trainable matrices MM serving as the model embedding. The authors define Frobenius/cosine geometry over these embeddings, store per-task embeddings in a memory, and introduce a compact index formed from a small subset of embedding coordinates. A learned projection from MiniLM text embeddings to this index enables language-conditioned adapter retrieval. The framework is evaluated on a toy two-joint reacher, Meta-World MT50, five LIBERO suites, and a qualitative RoboCasa composition example. The reported results indicate that the proposed adaptation can approach LoRA performance with fewer parameters, same-task checkpoints cluster in embedding space, the compact index correlates with full-embedding similarity, and closed-set language retrieval reaches 94.0% top-1 and 99.5% top-3 accuracy. On LIBERO, per-task MEM adapters substantially improve the MTACT backbone. The paper also provides a local Lipschitz argument relating behavioral distance to model-embedding distance and a linear-Gaussian illustration of why in-subspace updates may reduce unnecessary drift.

Critical strengths
  1. The paper presents a coherent and practically attractive systems view: task-specific policy adaptations are placed in a common pretrained spectral coordinate system, stored separately, and accessed through a small index. If the results hold under stronger controls, this could be a useful design for modular robot-policy deployment because it separates a frozen backbone, compact skills, and routing.
  2. The empirical scope is broader than that of many adapter papers. Meta-World tests parameter efficiency and seed-level retrieval geometry, LIBERO tests language routing and policy performance, and RoboCasa illustrates temporal skill switching. The reported parameter savings and LIBERO improvements are substantial enough to motivate further investigation.
Critical weaknesses
  1. The novelty is not yet established relative to the closest literature. Spectral Adapter already uses SVD bases of pretrained weights for parameter-efficient adaptation and multi-adapter organization[1]. More directly, Spectral Rewiring represents projected updates in the same form, ΔW∗=UMV⊤\Delta W^*=UMV^\top, and interprets MM as a compact coordinate matrix in the pretrained spectral basis[2]. Existing work also retrieves task LoRAs from inputs or language[3][4], builds reusable adapter libraries from parameter similarity[5], represents models and tasks for model selection[6], retrieves and adapts robot skills[7], or maps task descriptions directly to adapter parameters[8]. The present paper's potentially novel contribution is therefore the particular combination of a dense/block spectral-coordinate adapter, a small weight-derived index, and robot-policy retrieval. That combination may be worthwhile, but the manuscript neither discusses several of the closest works[1]-[8] nor compares against their retrieval or routing principles, making the claimed methodological novelty appear incremental.
  2. The experiments do not isolate the contribution of the proposed representation. The Meta-World retrieval protocol stores seven seeds and queries with three independently trained embeddings from the same tasks. Obtaining the query embedding already requires training on the target task, so this is closer to task identification than useful skill retrieval. The language experiment trains on canonical descriptions of the same 40 tasks and tests five paraphrases, without strong text-nearest-neighbor, supervised-classifier, LoraRetriever/Arrow-style routing[3][4], or reusable LoRA-library baselines based on parameter similarity[5]. Existing work on model selection and robot-skill retrieval also provides relevant retrieval principles for comparison[6][7]. LIBERO compares against published generalist-policy numbers while the proposed method adds one 0.9M-parameter adapter per task, so training data, backbone, tuning budget, and total storage are not clearly controlled. An oracle task-ID adapter, a LoRA library with the same router, full per-task fine-tuning, and task-held-out transfer experiments are needed. RoboCasa is qualitative, and the toy reacher leaves the hardest task essentially unsolved. Together, these issues weaken both the empirical support and the claimed scope of the memory/retrieval contribution.

[1] Zhang F, Pilanci M. Spectral adapter: Fine-tuning in spectral space[J]. arXiv preprint arXiv:2405.13952, 2024. [2] Zhang Z, Yu H, Gao H, et al. Spectral Rewiring for Exploration, Purification, and Model Merging[J]. arXiv preprint arXiv:2607.03065, 2026. [3] Zhao Z, Gan L, Wang G, et al. Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild[C]//Findings of the Association for Computational Linguistics: ACL 2024. 2024: 4447-4462. [4] Zhao Z, Gan L, Wang G, et al. Retrieval-augmented mixture of lora experts for uploadable machine learning[J]. arXiv preprint arXiv:2406.16989, 2024. [5] Ostapenko O, Su Z, Ponti E M, et al. Towards modular llms by building and reusing a library of loras[J]. arXiv preprint arXiv:2405.11157, 2024. [6] Zhang Y K, Huang T J, Ding Y X, et al. Model spider: Learning to rank pre-trained models efficiently[J]. Advances in Neural Information Processing Systems, 2023, 36: 13692-13719. [7] Guo Y, Tang B, Akinola I, et al. Srsa: Skill retrieval and adaptation for robotic assembly tasks[C]//International Conference on Learning Representations. 2025, 2025: 17082-17111. [8] Charakorn R, Cetin E, Tang Y, et al. Text-to-lora: Instant transformer adaption[J]. arXiv preprint arXiv:2506.06105, 2025.

Questions for authors
  1. Please compare precisely with the closest spectral-adaptation methods. Spectral Adapter already performs parameter-efficient fine-tuning in the spectral basis of pretrained weights[5], while Spectral Rewiring more directly uses the coordinate representation ΔW∗=UMV⊤\Delta W^*=UMV^\top[6]. What is mathematically new about learning MM in ΔW=UMV⊤\Delta W=UMV^\top? How does the block-diagonal restriction differ from these methods in expressivity, optimization, and storage? Was arXiv:2607.03065 considered concurrent work?
  2. Please evaluate retrieval on genuinely new task-held-out targets. Can you evaluate retrieval on task-held-out targets for which no query adapter has already been trained? For example, retrieve a source skill for a new task, then report zero-shot performance and sample efficiency after adaptation. Please compare against text-only nearest-neighbor retrieval, a supervised task classifier, LoraRetriever/Arrow-style routing[1][2], a reusable LoRA adapter library[3], robot-skill retrieval[4], LoRA parameter similarity, and an oracle task ID.
  3. Please reconcile the experimental inconsistencies. Please reconcile the following issues: 512-dimensional versus 16-dimensional LIBERO indices; MT50 adaptation versus the MT10 description in Section C.4; the missing action-chunk length kk in Table 8; the different LoRA parameter counts in Tables 1 and 2; and the absence of index accuracy in Table 2. Please also report total memory as the number of tasks NN grows and provide quantitative RoboCasa results with per-skill and compositional baselines.

[1] Zhao Z, Gan L, Wang G, et al. Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild[C]//Findings of the Association for Computational Linguistics: ACL 2024. 2024: 4447-4462. [2] Zhao Z, Gan L, Wang G, et al. Retrieval-augmented mixture of lora experts for uploadable machine learning[J]. arXiv preprint arXiv:2406.16989, 2024. [3] Ostapenko O, Su Z, Ponti E M, et al. Towards modular llms by building and reusing a library of loras[J]. arXiv preprint arXiv:2405.11157, 2024. [4] Guo Y, Tang B, Akinola I, et al. Srsa: Skill retrieval and adaptation for robotic assembly tasks[C]//International Conference on Learning Representations. 2025, 2025: 17082-17111. [5] Zhang F, Pilanci M. Spectral adapter: Fine-tuning in spectral space[J]. arXiv preprint arXiv:2405.13952, 2024. [6] Zhang Z, Yu H, Gao H, et al. Spectral Rewiring for Exploration, Purification, and Model Merging[J]. arXiv preprint arXiv:2607.03065, 2026.

4 comments
AuthorsOpenReview ↗

Rebuttal (Part1)

We thank the reviewer for their constructive feedback. We address the comments below and have revised the manuscript accordingly.

(W1, Q1): What is the contribution of this paper compared to existing work? Compare with the spectral-adaptation methods and other retrieval methods.

A: Our contribution is investigating how to construct a structured model embedding space in which task-specific adaptations can be learned, represented (Sec. 2.1), organized, and retrieved (Sec. 2.2-2.4, Sec. 3) within a shared coordinate system derived from the pretrained model.

Comparison with Spectral-adaptation methods. Let the pretrained weight matrix have SVD W0=UΣV⊤W_0=U\Sigma V^\top. Model embedding directly optimizes task-specific coordinates Mi(h)M^{(h)}_i within the subspace, spanned by the pretrained bases U(h)U^{(h)} and V(h)V^{(h)}: ΔWi=∑U(h)Mi(h)V(h)⊤.\Delta W_i= \sum U^{(h)} M^{(h)}_i V^{(h)\top}.

  • Spectral Adapter introduces additive and rotational adaptations of selected singular vectors. Its formulation is Wi=(U+ΔUi)Σ(V+ΔVi)⊤W_i=(U+\Delta U_i) \Sigma (V+\Delta V_i)^\top, where only selected singular-vector columns are modified. In contrast, model embedding keeps UU and VV fixed.

  • Spectral Rewiring first obtains an update through full parameter fine-tuning ΔWi=Witrained−W0\Delta W_i=W_i^{\mathrm{trained}}-W_0. It then extracts MiM_i through post-hoc projection: Mi=U⊤ΔWiVM_i=U^\top\Delta W_i V. In contrast, model embedding directly optimizes on MiM_i.

We further evaluated Spectral Adapter and Spectral Rewiring on LIBERO to investigate the practical differences between these formulations. All results are obtained using oracle task-ID selection, without retrieval, to isolate task-specific adaptation performance.

MethodRankSpatial ↑Object ↑Goal ↑LIBERO-10 ↑Overall ↑Params ↓
Spectral Adapter1686.6%75.4%87.4%71.6%80.3%0.9M
Spectral Adapter6490.9%86.0%89.1%73.7%84.9%3.6M
Spectral Rewiring1280.2%5.6%8.2%0.0%3.5%0.5M
Spectral Rewiring2565.4%18.2%13.0%1.8%9.6%2.2M
Spectral Rewiring51282.0%56%76.0%40.6%63.7%8.9M
Model Embedding6491.5%88.5%90.5%74.0%86.1%0.9M

These results demonstrate that our block-structured model embedding achieves a favorable trade-off between adaptation performance and parameter efficiency. We specifically compare these three perspectives:

  • Expressivity: Our model embeddings enable adaptations within pretrained spectral subspaces, achieving higher success rates than Spectral Adapter and Spectral Rewiring at comparable parameter budgets.

  • Optimization: Unlike Spectral Adapter, which modifies selected singular vectors, and Spectral Rewiring, which projects fully fine-tuned updates, our method directly optimizes task-specific coordinates within fixed pretrained bases, avoiding costly full fine-tuning and post-hoc projection.

  • Storage: Our method stores only compact task-specific coordinates (0.9M parameters) per task. Spectral Rewiring additionally requires full fine-tuning (53M parameters) before projection.

Distinction from existing adapter retrieval methods.

The distinction of our contribution lies in the representation being retrieved and its relationship to the learned adaptation. To enhance practical applicability, we further introduce a language-to-index interface that enables language-guided retrieval of stored model embeddings.

Methods such as LoraRetriever, RAMoLE, Arrow, and reusable LoRA libraries organize or route existing adapters. Model Spider and SRSA focus on model selection or identifying transferable source models, while Text-to-LoRA predicts adaptation parameters from task descriptions. We will include more discussion in the related work.

AuthorsOpenReview ↗

Rebuttal (Part2)

(W2) How the experiments isolate the contribution of the proposed representation. Provide additional experiments on baseline retrieval methods.

A: Our experiments investigate three complementary aspects: (i) whether model embeddings provide a meaningful representation of learned tasks, (ii) whether stored knowledge can be retrieved without training a target query adapter, and (iii) whether the resulting adaptations support effective policy learning and knowledge reuse. We clarify these evaluation objectives below and provide additional controlled comparisons.

Meta-World experiment. Our intention in Meta-World is to investigate the intrinsic properties of the proposed model embedding space, rather than demonstrate retrieval for unseen tasks.

Specifically, the independently trained embeddings provide a way to evaluate whether task-specific adaptations exhibit consistent geometric relationships despite differences in training initialization. This experiment examines whether the proposed coordinates provide meaningful task representations that can be compared and organized in memory. Additionally, the experiment investigates the parameter efficiency of storing multiple task-specific adaptations in a shared pretrained coordinate system.

Language-guided retrieval. Unlike Meta-World's retrieval adapters using a query adapter, our language-guided retrieval does not require training a query adapter for the target task. Instead, a language query is mapped to the model embedding index to retrieve a previously learned skill from memory. It mainly aims to provide a practical use of our model embedding space.

LIBERO experiment. To isolate the contribution of the proposed representation from task-specific adaptation and retrieval, we conduct additional controlled comparisons against LoRA and full fine-tuning under two settings: (i) oracle task-ID selection, where the correct task-specific adapter is directly selected; and (ii) using the same language-guided retrieval mechanism to select the adaptation.

Oracle task-ID selection:

MethodSpatialObjectGoalLIBEO-10OverallTrainable Params
LoRA89.8%81.6%90.2%71.4%83.25%2.4M
Model Embedding91.5%88.5%90.5%74.0%86.12%0.9M
Full Fine-tuning94.4%88.4%96.4%82.4%90.4%53M

Language-guided retrieval mechanism:

MethodSpatialObjectGoalLIBERO-10OverallTrainable Params
LoRA89.8%81.6%90.2%69.4%82.75%2.4M
Model Embedding91.5%88.5%90.5%71.0%85.4%0.9M
Full Fine-tuning94.4%88.4%96.4%78.4%89.4%53M

Model embedding achieves favorable adaptation performance and parameter efficiency compared with LoRA and Full. It demonstrates that the proposed structured adaptation space maintains strong task-learning capability with a smaller parameter budget. The model embedding space enables more memory-efficient storage, retrieval, and reuse of previously learned task-specific adapters.

(W2) Explanation for RoboCasa and Toy reacher results.

RoboCasa: The RoboCasa experiment addresses a distinct question from Meta-World and LIBERO: whether previously learned atomic skills can be retrieved and reused during the execution of tasks that were not individually trained and stored in memory. Specifically, the memory contains adaptations learned for atomic tasks, while the evaluation involves compositional tasks constructed from these atomic skills. During execution, the system retrieves relevant stored skills according to the current task context. This setting extends beyond identifying an existing task from a set of trained adapters, since the target composition itself is not stored as a dedicated adaptation. It illustrates how model embedding memory can support modular skill reuse and temporal switching.

Toy reacher: This experiment uses a small neural network in a controlled setting to investigate how variations in task dynamics relate to the geometry of learned model embeddings. Its primary purpose is to illustrate the structural properties of the model embedding space, rather than to demonstrate strong robotic control performance.

AuthorsOpenReview ↗

Rebuttal (Part3)

(W2, Q2) Additional experiments on task-held-out targets.

A: We clarify that our primary objective is to organize and reuse knowledge acquired from previously learned tasks, rather than to directly learn a new task without training. Specifically, our framework stores task-specific adaptations in a compact, structured model embedding space and retrieves relevant knowledge for subsequent use. Importantly, while constructing the memory requires trained source adapters, retrieval for a new target task does not necessarily require training a corresponding query adapter.

Comparison with existing retrieval methods. To better evaluate the effectiveness of our retrieval framework, we first conduct additional comparisons against a set of baselines, including text-based nearest-neighbor retrieval, a supervised task classifier, LoraRetriever, RAMoLE, Arrow-style routing, and a reusable LoRA adapter library (MBC). We follow the same experimental setting as Tab.3 in our paper. Extra retrieval index refers to the storage required for method-specific parameters, embeddings, and metadata used to retrieve adapters at inference.

MethodTop-1Top-3Top-5Extra retrieval index
Text nearest neighbor94.0%99.5%99.5%61.4 KB
Supervised classifier94.0%99.5%100.00%61.6 KB
LoraRetriever92.5%96.5%98.0%1.24 MB
RAMoLE router92.5%96.5%98.0%1.24 MB
Arrow-style10.0%25.0%40.0%4.44 MB
MBC92.5%97.0%98.0%73.8 KB
Ours94.0%99.50%99.50%88.6 KB

From the table, our language-guided retrieval achieves 94% Top-1 accuracy, matching text nearest-neighbor retrieval, supervised classifer and outperforming the other evaluated routing approaches. This demonstrates that our model embedding memory supports competitive language-guided retrieval.

Zero-shot transfer and sample efficiency on new targets. We further investigate whether retrieving a previously learned adapter can accelerate adaptation to a new target task.

We first evaluate zero-shot transfer of retrieved source adapters. Among the 40 LIBERO tasks, we hold out 8 as unseen targets and construct a memory bank using adapters trained on the remaining 32 tasks. Across all evaluated retrieval methods, the zero-shot success rate is approximately 0%, indicating a substantial gap between the retrieved source skills and the target-specific policies. These results highlight a limitation of direct skill transfer: retrieving a related source adapter alone is insufficient to solve an unseen robotic task without further adaptation.

We therefore investigate whether retrieved adapters provide a useful initialization for learning the target task. Specifically, we compare training from scratch against initialization using adapters retrieved through Text nearest neighbors and our language-guided retrieval. We measure the number of training steps required to reach 50%, 80%, and 100% of the success rate achieved by an independently trained target-specific adapter. This independently trained adapter serves as the reference performance for each target task.

Target fractionScratchText nearest neighborOurs
50% of target500 steps250 steps250 steps
80% of target750 steps375 steps250 steps
100% of target1,500 steps2,750 steps2,000 steps

Retrieved adapters substantially accelerate initial adaptation. Notably, our method reaches 80% of target performance in 250 steps, compared with 375 steps for text nearest-neighbor retrieval. This suggests that retrieval accuracy alone may not fully capture the transferability of selected source skills, motivating the use of model-space relationships for knowledge retrieval.

However, training from scratch reaches the full target performance in fewer steps than either retrieval-based initialization. This suggests that while source adapters provide useful transferable knowledge, they may also introduce task-specific biases that limit subsequent optimization.

Generalization to unseen compositional tasks in RoboCasa. Beyond retrieving source skills for new targets, we have evaluated knowledge reuse on genuinely unseen compositional tasks using RoboCasa in our paper. In this setting, the memory contains model embeddings trained on atomic skills, while evaluation is conducted on compositional tasks that were not individually trained or stored in memory. The quantitative results are shown below.

NBase model+MEM
Atomic Task6549.062.2 (+13.2)
Compositional Task2218.726.3 (+7.6)

The 7.6% improvement in compositional tasks demonstrates that the memory supports knowledge reuse beyond simply identifying and executing previously trained tasks.

AuthorsOpenReview ↗

Rebuttal (Part4)

(Q3) Address the inconsistency between the main paper and the appendix.

(Q3.1) 512-dimensional versus 16-dimensional LIBERO indices;

A: The correct dimensionality of the LIBERO model embedding index is 16. The reported 512-dimensional representation refers to the hidden feature dimension of the MTACT backbone, rather than the model embedding index. We have corrected Table 7 to distinguish these dimensions.

(Q3.2) MT50 adaptation versus the MT10 description in Section C.4;

A: The experiment in Appendix C.4 uses Meta-World MT50, rather than MT10. The reference to MT10 was a typo, which we have corrected to MT50.

(Q3.3) Report action-chunk length kk in Table 8;

A: The action-chunk length is (k=10). Each MTACT forward pass predicts 10 future actions.

(Q3.4) Clarification of Tab.2

A: Tab.1 and Tab.2 report different quantities: Tab.1 reports the trainable parameters stored in the LoRA adapters, whereas Tab.2 reports the size of the reconstructed dense weight updates used for parameter-based retrieval.

Breakdown for Tab.1. For a two-block transformer containing Q/K/V/O attention projections and MLP layers, the stored LoRA parameters at rank (r=16) are: 2[4∗16∗128∗2+16∗(128+256)∗2]=57,344.2\left[4 * 16 * 128 * 2+16 * (128+256) * 2\right]=57,344.

Breakdown for Tab.2. In contrast, parameter-based retrieval reconstructs the dense updates ΔW=BA\Delta W=BA. The total number of elements in these reconstructed matrices is: 2[4∗128∗128+2∗128∗256]=262,144.2\left[4 * 128 * 128 + 2 * 128 * 256 \right]=262,144.

Therefore, the two values describe different representations of the same adaptation: 57,344 trainable LoRA parameters vs. 262,144 elements in the reconstructed dense updates used for retrieval.

In the original Tab. 2, "accuracy" refers to the* success rate* on a Meta-World task using the retrieved adapter, rather than retrieval accuracy. Since LoRA and Model Embedding represent executable adaptations, their retrieved adapters can be evaluated directly to get the success rate. In contrast, the Model Embedding Index is a compact representation designed solely for retrieval and cannot be executed to get a success rate; thus, we don't include the number there.

Accordingly, we have revised Table 2 to replace the success-rate column with retrieval accuracy:

MethodParams. for RetrievalRetrieval Time (ms)FLOPsRetrieval Accuracy
LoRA262,1441.93(2rl+4N)Ld2(2r_l+4N)Ld^297.5%
Model Embedding6,1440.134NHLr24NHLr^2100%
Model Embedding Index2560.114Nr24Nr^2100%

The results demonstrate that the compact Model Embedding Index achieves 100% Top-1 retrieval accuracy, matching retrieval using the full model embeddings. Furthermore, the Top-1 agreement between full-embedding retrieval and index-based retrieval is 100% on the evaluated queries.

(Q3.5) Report quantitative RoboCasa results.

A: To further evaluate the scalability and knowledge-reuse capability of our model embedding memory, we provide additional quantitative results on RoboCasa.

The total memory requirement for RoboCasa is: 6.28+0.004N GB,6.28+0.004N\ \mathrm{GB}, where NN is the number of stored atomic-task adapters. For 65 atomic tasks, the storage of adapters requires only 0.004∗65=0.26GB0.004*65=0.26GB, while compositional tasks reuse these stored adapters without requiring additional task-specific storage.

We further evaluate the effectiveness of model embedding memory on 65 atomic tasks and 22 compositional tasks:

NBase model+MEM
Atomic Task6549.062.2(+13.2)
Compositional Task2218.726.3(+7.6)

Model embedding memory improves success rates on both atomic and compositional tasks. Notably, the compositional tasks have no dedicated adapters in memory; instead, the framework retrieves and reuses previously learned atomic skills during task execution. The 7.6% improvement demonstrates that the memory supports knowledge reuse beyond simply identifying and executing previously trained tasks.

Reviewer_DYwuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Model Embedding Memory, which represents the weight changes of policies trained for individual tasks as compact embeddings, and stores and retrieves these embeddings for reuse. Task-specific adaptation is expressed through block-wise learnable matrices on top of a shared basis obtained from the singular value decomposition of the pretrained weights. A subset of the full embedding serves as a retrieval index, and language embeddings are mapped into the index space so that the policy matching a natural language instruction can be selected.

Experiments are conducted in three environments. On Meta-World, the paper evaluates adaptation performance, parameter efficiency, and retrieval behavior across policies trained with different random seeds. On LIBERO, it reports task success rates and language-based retrieval performance, and on RoboCasa, it presents execution examples that select and switch among stored task-specific policies during compositional task execution. I understand the core contribution to be managing task-specific adaptations in a shared coordinate system and connecting them to policy retrieval and reuse.

Critical strengths

1. Achieving comparable adaptation performance with few trainable parameters is practically valuable.

In Table 1(a), the proposed method achieves a success rate of 87.90% with 6,144 parameters, comparable to the 87.63% of LoRA, which uses 14,336 parameters. The paper also reports how performance and capacity vary with the number of blocks, allowing readers to examine the trade-off between storage cost and adaptation performance. This is a meaningful advantage in settings where many task-specific policies must be stored.

2. Unifying policy storage, comparison, and language-based selection in a single framework is a notable strength.

Figure 3 shows a high correlation between the index and the full embedding, and Figure 4 shows the task-wise similarity structure across policies trained with different seeds. Together, these support the potential of the representation as a retrievable one. In addition, the paper reports 94.0% top-1 language retrieval accuracy on 200 held-out instruction variants covering the existing 40 tasks, suggesting its potential as an interface for accessing stored policies.

Critical weaknesses

1. The comparative experiments are insufficient to assess the independent contribution of the proposed representation and retrieval scheme.

The performance gain in Table 4 is impressive, but the proposed method additionally trains and stores an adapter for each task. Therefore, the current results make it difficult to tell whether the improvement over MTACT stems from task-specific training or from the proposed embedding structure and retrieval scheme. A comparison with a LoRA adapter memory that uses the same backbone, data, training budget, and retriever would make the core contribution clearer. Moreover, Spectral Adapter is closely related prior work in that it also leverages the spectral information of pretrained weights. I believe it is important to show what additional benefits the proposed method offers over this approach in terms of memory construction and retrieval.

2. The connection between the formal definition and the actual implementation needs further clarification.

The main text defines the update using an orthogonal SVD basis, whereas Appendix B.1 describes inserting a learnable matrix between existing attention projections. It is unclear how these two formulations correspond in the actual experiments, which makes it difficult to judge whether the theoretical properties of the embedding distance hold in the implementation. In addition, the main text and Appendix C.5 state that MiniLM embeddings are projected directly into the index space, while Appendix C.7 states that discrete task tokens are predicted first. Clarifying the implemented retrieval procedure and reconciling these descriptions would help readers interpret the results and reproduce the method.

Questions for authors

The following questions concern points that directly affect the interpretation and evaluation of the current results.

  1. Does the performance advantage persist when compared with a LoRA adapter memory that uses the same backbone, task-specific demonstration data, training budget, and language retriever? Could the authors also report the performance gap between directly selecting the adapter of the ground-truth task and using actual retrieval?

  2. In the actual experiments, is the learnable matrix inserted between the UU and VV obtained from SVD, or between existing attention projections as described in Appendix B.1? It would be helpful if the authors could explain the relationship between these two settings and the scope in which the theoretical distance-preservation property applies.

  3. Is language retrieval performed by directly projecting MiniLM embeddings, or by first predicting discrete task tokens? In addition, could the authors clarify which settings the 16-dimensional output in Appendix C.6 and the 512-dimensional index in Table 7 respectively refer to?

  4. Could the authors report the top-1 retrieval accuracy of the Model Embedding Index in Table 2, as well as its top-1 agreement rate with retrieval using the full embedding? Reporting the retrieval-index storage separately from the full adapter storage required for execution would also help clarify the efficiency claims.

4 comments
AuthorsOpenReview ↗

Rebuttal (Part1)

We thank the reviewer for their constructive feedback. We address the comments below and have revised the manuscript accordingly.

(W1, Q1) What is the contribution of this paper? Compare with LoRA and Spectral Adapter.

Our contribution is investigating how to construct a structured model embedding space in which task-specific adaptations can be learned, represented (Sec. 2.1), organized, and retrieved (Sec. 2.2-2.4, Sec. 3) within a shared coordinate system derived from the pretrained model.

Experiments on LoRA and Spectral Adapter. Our original Meta-World experiments already included comparisons with LoRA, demonstrating the parameter efficiency of Model Embedding and its ability to capture task-specific relationships in a shared representation space.

To further isolate the contribution of the proposed representation from task-specific adaptation and retrieval, we conducted additional controlled comparisons against Spectral Adapter, LoRA, and full fine-tuning on LIBERO, using the same MTACT backbone and experimental protocol described in Sec. 4.2.

We consider two settings: (i) oracle task-ID selection, where the correct task-specific adapter is directly selected; and (ii) using the same language-guided retrieval mechanism to select the adaptation. The best-performing LoRA-based language-guided retrieval method achieves a retrieval accuracy of 97.5%, matching that of our model-embedding-based language-guided retrieval approach.

Oracle task-ID selection:

MethodSpatialObjectGoalLIBEO-10OverallTrainable Params
LoRA89.8%81.6%90.2%71.4%83.25%2.4M
Spectral Adapter86.6%75.4%87.4%71.6%80.3%0.9M
Full Fine-tuning94.4%88.4%96.4%82.4%90.4%53M
Model Embedding91.5%88.5%90.5%74.0%86.12%0.9M

Language-guided retrieval mechanism:

MethodSpatialObjectGoalLIBERO-10OverallTrainable Params
LoRA89.8%81.6%90.2%69.4%82.75%2.4M
Spectral Adapter86.6%75.4%87.4%69.1%79.6%0.9M
Full Fine-tuning94.4%88.4%96.4%78.4%89.4%53M
Model Embedding91.5%88.5%90.5%71.0%85.4%0.9M

The performance gap between oracle task-ID selection and actual retrieval is small, as the language-guided router achieves nearly perfect task identification. All adapters are correctly retrieved in LIBERO-Spatial, LIBERO-Object, and LIBERO-Goal, resulting in identical success rates under both settings. In LIBERO-10, only one task is assigned an incorrect adapter, leading to a slight performance degradation.

Expressiveness of the proposed representation. Model Embedding demonstrates effective task adaptation with substantially fewer parameters. Under oracle selection, it achieves 86.1% overall success, outperforming LoRA and Spectral Adapter. It also approaches full fine-tuning performance with significantly fewer parameters. Since retrieval is excluded, these results isolate the contribution of the adaptation representation.

Second, the performance advantage persists under actual retrieval. With the shared retrieval mechanism, Model Embedding achieves 85.4% overall success compared with 82.75% for the LoRA adapter bank, while retaining its smaller per-task parameter budget.

These results demonstrate that our structured model embedding space provides a favorable trade-off between adaptation performance and parameter efficiency, independently of the retrieval mechanism. Beyond efficient adaptation, the learned coordinates also serve as executable task representations that can be organized, indexed, and retrieved within model embedding memory.

Compare the formula with Spectral Adapter. Let the pretrained weight matrix have SVD W0=UΣV⊤W_0=U\Sigma V^\top. Model embedding directly optimizes task-specific coordinates Mi(h)M^{(h)}_i within the subspace, spanned by the pretrained bases U(h)U^{(h)} and V(h)V^{(h)}: ΔWi=∑U(h)Mi(h)V(h)⊤.\Delta W_i= \sum U^{(h)} M^{(h)}_i V^{(h)\top}.

Spectral Adapter introduces additive and rotational adaptations of selected singular vectors. Its formulation is Wi=(U+ΔUi)Σ(V+ΔVi)⊤W_i=(U+\Delta U_i) \Sigma (V+\Delta V_i)^\top, where only selected singular-vector columns are modified. In contrast, model embedding keeps UU and VV fixed.

AuthorsOpenReview ↗

Rebuttal (Part2)

(W2, Q2) The implementation details. Does the analysis hold for MHA?

Our SVD-based formulation and MHA implementation follow the same principle of learning task-specific model embeddings within fixed projection structures derived from the pretrained model.

MLP. In the general linear-layer formulation, we construct the model embedding space using fixed pretrained singular-vector bases: ΔWi=∑hU(h)Mi(h)V(h)⊤.\Delta W_i=\sum_h U^{(h)}M_i^{(h)}V^{(h)\top}.

MHA. This block-wise structure is motivated by multi-head attention (MHA), where individual attention heads naturally define separate projection components (Sec. B.2). In our MHA implementation, we directly insert the learnable model embedding matrices between existing fixed attention projections (Eq. (66)), rather than explicitly recomputing their SVD bases. This design treats the attention block as a unified structure rather than adapting its components independently.

Both formulations optimize task-specific coordinates while sharing fixed projection factors across tasks. The SVD formulation provides an orthogonal coordinate system for theoretical analysis, while the MHA implementation leverages the architecture's inherent block structure. In our experiments, we employ the MHA-based design for attention blocks and the MLP-based design exclusively for feed-forward network (FFN) blocks.

Analysis for MHA. For the MHA implementation, consider the effective weight update ΔWi=WvMiWo\Delta W_i = W_v M_i W_o, where WvW_v and WoW_o are the fixed projection matrix and MiM_i contains the task-specific block coordinates. The corresponding distances satisfy ∣∣ΔWi−ΔWj∣∣F≤∣∣Wv∣∣2∣∣Wo∣∣2∣∣Mi−Mj∣∣F.||\Delta W_i-\Delta W_j||_F \leq ||W_v||_2 ||W_o||_2 ||M_i-M_j||_F.

As WvW_v and WoW_o are fixed pretrained weights, i.e., ∣∣Wv∣∣2||W_v||_2 and ∣∣Wo∣∣2||W_o||_2 are constant. Thus, the MHA implementation satisfies a more general bounded-distance relationship. It is aligned with Proposition 2.2, with $C=\left( \mathbb{E}_{s\sim\mu}[L(s)^2] \right)^{1/2} ||W_v||_2 ||W_o||_2 $.

(Q3) Address the inconsistency between main text and appendix.

A: Both descriptions in Appendix C.5 and C.7 refer to the same language-guided retrieval procedure. Specifically, MiniLM first encodes the task description into a continuous language embedding, which we refer to as a task token in Appendix C.7. This embedding is then projected into the model embedding index space for adapter retrieval. Thus, the method does not involve predicting discrete task tokens as an intermediate step.

The 512-dimensional representation reported in Table 7 refers to the MTACT backbone's hidden feature dimension, not the model embedding index. The actual retrieval index is 16-dimensional, consistent with Appendix C.6.

(Q4) Clarification of Tab.2

A: In the original table, "accuracy" refers to the success rate on a Meta-World task using the retrieved adapter, rather than retrieval accuracy. Since LoRA and Model Embedding represent executable adaptations, their retrieved adapters can be evaluated directly to get the success rate. In contrast, the Model Embedding Index is a compact representation designed solely for retrieval and cannot be executed to get a success rate; thus, we don't include the number there. We agree that the retrieval accuracy provides a more appropriate comparison across all three methods. Accordingly, we have revised Table 2 to replace the success-rate column with retrieval accuracy:

MethodParams. for RetrievalRetrieval Time (ms)FLOPsRetrieval Accuracy
LoRA262,1441.93(2rl+4N)Ld2(2r_l+4N)Ld^297.5%
Model Embedding6,1440.134NHLr24NHLr^2100%
Model Embedding Index2560.114Nr24Nr^2100%

The results demonstrate that the compact Model Embedding Index achieves 100% Top-1 retrieval accuracy, matching retrieval using the full model embeddings. Furthermore, the Top-1 agreement between full-embedding retrieval and index-based retrieval is 100% on the evaluated queries.

We further report below the storage of the retrieval index and full adapter:

Model EmbeddingModel Embedding Index
Params.6,144256

Thank you for taking the time to write such a detailed response. Your clarifications and additional results have addressed my concerns. I believe the paper makes a valuable contribution, and the revisions will strengthen it further.

I will maintain my current score.

Thank you for your positive feedback and support. We sincerely appreciate your time and thoughtful evaluation of our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Reviewers and AC,

As the discussion period approaches its end, we would like to briefly summarize the clarifications and experimental evidence provided in our rebuttal. Specifically, we have:

  1. Clarified our contribution from existing methods: We emphasized that our contribution is to establish a structured model embedding space for representing, retrieving, and reusing learned task adaptations, rather than solely proposing another PEFT or adapter retrieval method. We distinguished our method from Spectral Adapter, Spectral Rewiring, SVDiff, and existing LoRA retrieval methods through comparisons of their underlying formulations and additional empirical evaluations. (DYwu W1/Q1; wAsr W1/Q1; TUk4 W1/Q1).

  2. Demonstrated the effectiveness of our representation: We evaluated model embedding against LoRA, Spectral Adapter, and full fine-tuning under both oracle task-ID selection and a shared retrieval mechanism, isolating adaptation performance from retrieval effects. On LIBERO, our method achieves 86.1% success with only 0.9M parameters, outperforming LoRA and Spectral Adapter, while approaching full fine-tuning (90.4%, 53M). Using the same retrieval method, our method also outperforms LoRA (85.4% vs. 82.75%). Furthermore, replacing pretrained SVD bases with random orthogonal bases reduces success from 86.1% to 56.5%, under identical parameter budgets and block structures, demonstrating the importance of pretrained spectral structure (DYwu W1; wAsr W1/W2; TUk4 W1/W2/Q2).

  3. Expanded retrieval evaluations and investigated generalization to held-out tasks: We added comparisons against text nearest-neighbor retrieval, supervised classification, LoraRetriever, RAMoLE, Arrow-style routing, and reusable LoRA libraries. Our language-guided retrieval achieves 94.0% Top-1 accuracy, matching the strongest evaluated baselines. We further evaluated transfer to held-out LIBERO tasks, showing that retrieved embeddings accelerate early adaptation. We further demonstrated the ability of our method to generalize to unseen compositional tasks through evaluations on RoboCasa. (wAsr W2/Q2).

  4. Demonstrated knowledge reuse on unseen compositional tasks: We provided quantitative RoboCasa experiments involving atomic tasks and unseen compositional tasks. Model Embedding Memory improves success from 49.0% to 62.2% on atomic tasks and from 18.7% to 26.3% on compositional tasks. Importantly, the compositional tasks have no dedicated adapters stored in memory, demonstrating the potential to reuse previously acquired knowledge beyond retrieving already learned tasks (wAsr W2/Q2).

  5. Strengthened theoretical analysis and resolved implementation concerns: We clarified the relationship between the SVD-based formulation and MHA implementation, including the bounded-distance relationship between model embeddings and their corresponding parameter updates (DYwu W2/Q2). We also corrected and clarified the retrieval-index dimensionality, Meta-World experimental settings, parameter accounting, and retrieval accuracy metrics, and updated the corresponding tables and manuscript descriptions (DYwu Q3/Q4; wAsr Q3).

We hope these responses address the reviewer's concerns. We sincerely appreciate your time and thoughtful evaluation.

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.