MergeCodec: Learning to Compress and Compose Task Vectors

Model merging aims to combine experts derived from a shared pretrained model, and yet preserving their capabilities across varying task compositions and numbers of experts remains challenging. Existing methods typically rely on hand-crafted merging rules or combination-specific coefficient optimization. We introduce MergeCodec, which learns a reusable merging map through task supervision. MergeCodec encodes task vectors into compact latent representations, composes selected experts by summing their codes, and decodes the result into a merged model update. Training end-to-end over subsets with varying task identities and cardinalities encourages the latent space to support robust composition. Once trained, the encoder can be reused for arbitrary expert combinations without additional task data, gradient updates, or per-combination calibration. Across 20 tasks and 3 CLIP backbones, a single MergeCodec per backbone successfully merges any number of experts, achieving performance comparable to multi-task models trained separately for each subset. Despite never being trained on single subsets, MergeCodec also largely preserves the accuracy of individual experts. Moreover, MergeCodec generalizes to out-of-distribution settings, successfully merging experts unseen during training, and its generalization improves as the training expert bank expands. These results demonstrate that model merging can be learned as a reusable compositional mapping, yielding task-vector representations that generalize across both novel expert combinations and previously unseen experts.