Loss-Aware Optimization for Model Merging in a Low-Dimensional Subspace

Model merging aims to combine multiple task-specific models into a single model without joint retraining. Many existing methods construct merged updates from geometric or structural information, leaving their relation to downstream task losses largely indirect. To address this limitation, we formulate model merging as local optimization of the average supervised loss around a strong merged anchor. Using calibration subsets, we linearize each task output at the anchor and apply a second-order Taylor expansion to the resulting tangent loss, yielding a quadratic surrogate based on generalized Gauss-Newton curvature. This curvature captures the sensitivity of each task loss to perturbations along different parameter directions, distinguishing directions that are costly to change from those that can accommodate larger deviations. We further restrict optimization to a low-dimensional subspace spanned by structured components of merged updates from strong existing methods. This restriction retains the structural information encoded by these updates, regularizes the optimization by constraining the search space, and reduces the quadratic problem to a low-dimensional linear system with a closed-form solution. Our method can be integrated with a broad range of existing merging methods and jointly leverage updates from multiple sources. Extensive experiments on benchmarks demonstrate notable improvements over strong baselines across model architectures and task collections.