FARMerge: Learning Functional Task Representations for Source-Data-Free Adapter Merging

Low-Rank Adaptation (LoRA) enables modular task adaptation for Large Language Models (LLMs), yet composing multiple task-specific adapters into a unified model remains challenging. Existing merging methods operate predominantly in parameter space using heuristics such as sign consensus or magnitude pruning. In our experiments, parameter similarity does not reliably track functional activation-space compatibility or downstream interference. We introduce FARMerge (Functional Adapter Representation Merge), a representation-learning framework for multi-adapter merging. During one-time meta-training on a disjoint development task bank, a layer-aware set encoder maps multi-layer probe activations into a metric space predictive of empirical pairwise interference. FARMerge is source-data-free at merge time: for novel adapters, 64 generic probe sequences drive interference prediction and a learned gate over rank-1 SVD components before fixed-rank synthesis. Across four LLM families (LLaMA-3-8B, Mistral-7B, Qwen-3.5-4B, OLMo-2-7B) and 12 downstream tasks, FARMerge averages 98.69% single-task-oracle retention on LLaMA-3-8B and 98.39% across all four model families, gaining $+3.72$/$+1.24$/$+1.61$/$+4.85$ points over DARE-TIES/TaDA-inspired/CT-Merging/LoraHub on LLaMA-3-8B ($+4.01$/$+1.95$/$+2.31$/$+5.00$ across families; Table tab:main_summary); on LLaMA-3-8B, merging 12 adapters takes 18.4 seconds. Code and reproduction scripts will be released publicly upon acceptance.