Structured Model Embeddings as Representations of Task Adaptations

Fine-tuning a shared pretrained model produces a collection of task-specific parameter updates. Beyond serving as independent adapters, these updates offer a way to represent learned task behavior. A useful representation must do more than store parameters: it should retain adaptation capacity, expose meaningful task relationships, and support interpretable changes in behavior. We study this problem through a structured model embedding space whose coordinates are defined by the pretrained model. Each embedding specifies a parameter update through fixed bases, providing both a compact description of an adaptation and a direct means of executing it. We organize the study around three properties: adaptation fidelity, relational geometry, and operational usefulness. We establish exact properties of the coordinate system and conditional bounds connecting embedding changes to functional changes and task arithmetic. A controlled MNIST generation study is designed to examine these properties with explicitly specified task relations, followed by evaluations using ViT on image datasets and GPT-2 on NLP datasets. We further investigate the construction of held-out task adaptations from existing embeddings as a test of the representation's practical scope.