Flexible and Evolving Model Merging via Task Interference Modeling

Training-free model merging builds multi-task models from existing finetuned checkpoints, but every practical method faces a tension between accuracy and storage. Monolithic merges are storage-cheap but suffer from inter-task interference; expert routers avoid interference but store N independent copies. We argue that a principled middle ground exists: partition N tasks into K user-chosen groups and keep one expert per group, trading accuracy against memory. The central problem then becomes grouping tasks to minimize intra-group interference, and we make two observations. First, we reveal that pairwise interference is insufficient to characterize multi-task interactions: two tasks that exhibit only mild interference in pairwise merges may suffer severe degradation when a third is introduced. Second, interference modeling should not be static: as more empirical evidence about task compatibilities accumulates—in the form of cheap random clustering trials—our understanding of which tasks belong together can evolve and improve. From these insights we build Task Interference Mining (TIM), a two-stage framework. Stage 1 builds a task interference graph from pairwise and triple-merging outcomes and produces an initial grouping via spectral clustering. Stage 2 evolves the partition by iteratively mining random clustering trials and applying count-preserving correction moves drawn from three complementary strategies, guaranteeing monotonic improvement as more trial evidence is leveraged. We evaluate TIM on three vision benchmarks of increasing size (8, 14, and 20 tasks) and on six NLI tasks. TIM substantially outperforms similarity-based and random grouping baselines, recovering most of the gap between full merging and per-task expert performance.