COMET: Continual Model Merging via Adaptation to Mismatch in Distributional Tails

Continual model merging requires preserving past solution expertise (stability) while retaining the ability to adapt (plasticity), without access to past data. Recent approaches cast this problem as recursive posterior inference, representing accumulated expertise as a model prior and incorporating new knowledge through the target task likelihood. The resulting posterior often assumes a Gaussian structure, with task-specific curvature statistics accumulated in its precision matrix. Our analysis reveals a previously overlooked geometric interpretation of this formulation. Incorporating knowledge from a new model distribution can be viewed as updating the mode of the existing model distribution along a set of orthogonal latent directions in the inner-product space of its precision matrix. Each direction is associated with a unit scale of update that acts as a base speed controlling how fast new knowledge should be incorporated along that direction. This reveals a novel geometric characterization of stability and plasticity that is largely independent of the model architecture. Importantly, the appropriate unit scale along each direction should adapt to the relative tail behaviors of the existing and incoming model distributions as their disagreement determines how strongly newly acquired knowledge should alter the existing solution. Gaussian structures, however, ignore such tail behavior and consequently impose fixed unit scales of update that are empirically suboptimal. Replacing the Gaussian structure with an energy based distributional structure whose tail decay rate is adaptive restores this missing dependence. Extensive experiments with billion-parameter foundation models show that our approach consistently outperforms state-of-the-art methods.