When Privacy Hurts Mergeability: Geometry-Aware Model Merging under Differential Privacy

Model merging promises to construct a single multi-task model from independently fine-tuned task models without accessing the original task data. This makes it attractive when task data cannot be centralized, but released task models may still leak private fine-tuning data. Differential privacy (DP) provides a principled mechanism for limiting such leakage, yet its interaction with model merging remains poorly understood. In this paper, we study the geometry of differentially private model merging and identify two geometric factors associated with poor mergeability: 1) local sharpness, which increases the sensitivity of task losses to the parameter displacement induced by merging, and 2) reference drift, which captures how far private task models move from the shared pretrained initialization and can contribute to cross-task discrepancy. We propose DP-Merging, a geometry-aware framework that improves the mergeability of differentially private task models. DP-Merging uses a DP-compatible sharpness-aware objective to guide each private task model toward flatter loss regions, and a reference-based alignment regularizer to keep task models close to the shared pretrained initialization. Our analysis relates merge-induced loss increases to local loss sensitivity and parameter displacement. Experiments on vision and language tasks across multiple privacy budgets show that DP-Merging consistently improves private merged-model performance while preserving the privacy guarantees of the underlying DP fine-tuning procedures.