Optimizing over Mode Connectivity for Multi-Task Learning and Machine Unlearning

Mode connectivity refers to the phenomenon that independently trained neural networks can be connected by continuous low-loss paths in the parameter space, revealing a family of high-quality candidate models rather than isolated optima. While existing theoretical studies have focused exclusively on the single-task learning setting, the role of mode connectivity in more challenging learning paradigms remains largely unexplored. In this paper, we provide the first theoretical study of mode connectivity-based multi-task learning and machine unlearning. We consider overparameterized fully connected multilayer perceptrons (MLPs) trained from a shared initialization in the Neural Tangent Kernel (NTK) regime, which yields closed-form expressions for the loss along a path up to a bounded error. We then extend these results to finite-width nonlinear networks. We establish theoretical conditions under which quadratic Bézier curves achieve lower loss than linear interpolation, demonstrating the advantage of optimizing over nonlinear connectivity manifolds when balancing competing learning objectives. Finally, we conduct experiments on both NTK-linearized models and standard finite-width neural networks, and the results closely validate our theoretical predictions.