Higher-order Inter-task Interference Modeling for Flexible Model Merging

Model merging combines multiple task-specific fine-tuned models into a single multi-task model without additional training. However, it faces a fundamental dilemma: merging all N models into one degrades accuracy due to inter-task interference, whereas maintaining N independent experts alongside a router incurs an N-fold increase in storage costs. We argue that this dilemma can be resolved by utilizing K < N experts, where tasks with minimal mutual interference share the same expert, and K serves as a user-defined storage budget. This flexible strategy adapts seamlessly to various application scenarios with diverse memory constraints. The core of this approach lies in estimating inter-task compatibility to group tasks with low mutual interference. Our empirical study reveals that inter-task interference does not accumulate consistently: two tasks that exhibit only mild interference in pairwise merges may suffer severe degradation when a third is introduced, whereas three tasks with strong pairwise conflicts can exhibit a stabilizing effect. To address this, we introduce Interference-Aware Flexible Merging (IAFM), a framework that constructs a task interference graph whose edge weights aggregate information from both pairwise and triple task merging experience, thereby capturing higher-order interactions overlooked by purely pairwise analysis. We evaluate IAFM on 8,14, and 20 vision tasks using OpenCLIP ViT-B/32, and 6 NLP tasks using Llama3-8B models, benchmarking it against the state-of-the-art model merging methods.