Exploiting Model Embedding Memory in Policy Learning

Adapting a pretrained model to downstream tasks produces a collection of task-specific adaptations that can serve as reusable model memory. However, existing weight-space representations, such as full-parameter or low-rank, are primarily designed for optimization rather than memory organization: they lack a shared representation in which task-specific adaptations can be compactly stored, compared, and retrieved. In this work, we propose a model embedding space, a low-dimensional representation of task adaptations constructed in alignment with the intrinsic structure of pretrained weights. This space provides a shared coordinate system for organizing task-specific knowledge into a model embedding memory, enabling compact storage and similarity-based retrieval across adapted models. To make this memory both efficient to search and easy to access from other modalities, we further introduce a model embedding index: a lightweight representation that preserves retrieval-relevant information while forming a compact interface between model embeddings and language representations. We demonstrate the framework in robot policy learning, where task-specific policies are encoded as model embeddings and retrieved from memory for target tasks. We further enable language-guided retrieval by mapping LLM embeddings to the model embedding index, showing that the model embedding space provides a structured memory for storing and reusing task-specific policies.