The Coupling Is the Lever: Cross-Group Geometry for Joint Optimization

Multi-task models contain heterogeneous parameter groups, including shared and task-specific components as well as experts and routers in mixture-of-experts architectures. Even when individual groups are well conditioned, optimization can remain slow or unstable because important directions may emerge only from their interaction. We show that these hidden joint modes arise from cross-group curvature that is missed when parameter groups are preconditioned independently. Near local minima, they create slow directions, while near saddles, individually stable groups can become unstable jointly. For a fixed multi-task objective, we characterize when accounting for this joint geometry improves local convergence and accelerates saddle escape. We then develop CoPEN (Coupling-aware Preconditioning for Efficient Network Optimization), which augments tensor-wise preconditioning with a low-dimensional model of cross-group curvature without forming the full Hessian. CoPEN applies to both shared-task-specific architectures and input-dependent expert-router interactions in mixture-of-experts models. Controlled and model-scale experiments show improved optimization progress and validation performance over the evaluated optimizer and multi-task baselines.