Weight-Space Graph Signal Processing for Multi-Resolution Model Merging

Model merging often suffers from destructive interference arising from conflicting parameter updates across tasks. Prior work has proposed various methods in Euclidean parameter space to mitigate this issue. However, it remains unclear how to identify which components of the parameters cause interference and which can be safely shared. To address this, we study model merging with graph signal processing and propose a novel model merging method, Spectral Graph Merging (SGM), based on a multi-resolution merging process. By viewing weight matrices as bipartite neuron-interaction graphs, we apply graph wavelet transformation to decompose the task vectors into low-frequency components capturing shared global structure and higher-frequency components associated with task-specific conflicts. SGM then reduces the interference in the high-frequency components while aggregating the low-frequency components. Evaluation results on MergeBench and OptMerge benchmarks demonstrate that SGM outperforms state-of-the-art merging methods on both benchmarks, while maintaining robust reasoning and safety under highly conflicting task combinations. Our results also indicate that exploiting spectral structure in weight space provides a principled and effective mechanism for mitigating destructive interference in large-scale model merging.