Dynamic Model Merging Made Slim

Model merging combines fine-tuned models into a unified one without joint training or access to original data. Dynamic merging improves flexibility by selectively activating task-relevant parameters, but existing methods either maintain a full shared model with tiny experts or allocate excessive capacity to experts, leading to suboptimal accuracy–efficiency trade-offs. We propose **DiDi-Merging**, a slim dynamic merging framework that uses **differentiable rank allocation** to balance shared and per-task expert parameters. The method casts parameter budgeting as differentiable rank optimization over low-rank modules, supervised data-free by the original task vectors, and applies a brief refinement stage to recover task fidelity. Across vision, language, and multimodal benchmarks, DiDi-Merging matches prior dynamic baselines at **1.24×** the parameters of a single fine-tuned model and surpasses them at **1.4×**, substantially more compact than methods requiring ≥2× storage.