Merging What Matters: Input-Adaptive and Layer-Aware Model Merging

Model merging provides an efficient approach to consolidating the complementary capabilities of multiple task-specific models without retraining from scratch. Existing methods often combine complete task updates or weight entire experts without accounting for the different contributions of LoRA-adapted layers, hence introducing parameter conflicts and cross-task interference. To address these limitations, in this paper, we propose Merging What Matters (MWM), an input-adaptive, layer-aware model merging framework. MWM first identifies task-critical LoRA updates within each expert and attenuates less informative auxiliary updates. It then employs a lightweight router to estimate input-dependent expert relevance and select the most appropriate experts for each input. Finally, the selected experts are dynamically integrated using differentiated layer-wise rules that preserve the core knowledge of the primary expert while conservatively incorporating complementary information from other experts. Extensive experiments across natural language processing and computer vision benchmarks demonstrate that MWM consistently outperforms the SOTA model merging baselines, achieves performance comparable to or better than task-specific fine-tuned experts, and retains competitive out-of-domain generalization. Code is provided in the supplementary material.