Conflict-Guided Data-Free Model Merging

Data-free model merging fuses several fine-tuned models into a single network. Recently proposed data-free methods estimate task statistics from model weights and apply regularization to reduce the interference between the fine-tuned specialized models. However, these approaches typically apply uniform regularization strength across layers, even though expert disagreement varies across the network. We introduce conflict-guided model merging, a data-free model merging scheme that assigns regularization strength to each layer based on a conflict score derived from the task vectors. This score quantifies how well a single shared update can accommodate competing experts. The conflict score guides both Ridge regularization and an orthogonalized iterative solver, while maintaining the existing task-statistics estimator. Experimental results show that regularization alone improves merging accuracy, and further gains come from adapting regularization strength to layer-specific conflict. Conflict-guided Ridge improves accuracy over ACTMat on CLIP ViT-B/32, with statistically significant gains when merging 14 and 20 tasks. For input-adaptive merging, selective storage has accuracy close to ours MASS baseline with  50% of its per-task factor storage.