Beyond the Merged Vector: Fixed-Budget Directional Memory for Data-Free Continual Model Merging

Data-free continual model merging aims to progressively integrate sequentially arriving task models into a unified model without accessing historical data or retaining previous task models. Existing methods typically construct historical subspaces from the cumulative merged task vector, but such implicit historical representations progressively degrade as tasks accumulate. To address this limitation, we propose **CUSP-Merge** (**CU**mulative **S**hared–**P**rivate Memory with Adaptive Protection for Continual Model Merging), a fixed-budget Shared–Private directional memory framework that replaces the implicit historical subspace with explicitly maintained historical task directions. Specifically, CUSP-Merge consolidates directions repeatedly supported across tasks into Shared Memory, retains task-specific residual directions in Private Memory, and dynamically promotes private directions to Shared Memory according to cumulative cross-task support, thereby enabling fine-grained organization of historical directions. Furthermore, this structured memory enables direction-level adaptive protection to balance stability and plasticity. Extensive experiments across multiple model architectures and task configurations demonstrate that CUSP-Merge achieves the highest average accuracy in all settings, with more pronounced gains on longer task sequences, while maintaining a favorable performance–storage trade-off.