Bilinear and Pairing-Aware Continual Model Merging

Continual model merging (CMM) combines independently fine-tuned task models as they arrive, without returning to earlier checkpoints or task data. Preserving the parameter directions of past tasks is only part of this problem. The merger must also retain enough information to correct later updates. We introduce BiPaCo (Bilinear admission with Pairing-Aware Correction), a training-free method that stores consolidated input and output bases and a bounded ledger of paired spectral records. These records preserve the within-task coupling between input and output directions. Bilinear admission discards the incoming component aligned with both previous bases, while pairing-aware correction re-queries surviving records in the spectral blocks of the provisional delta and adjusts selected blocks toward historical targets in closed form. Using one configuration, BiPaCo leads training-free methods in accuracy across all nine CLIP-ViT settings and achieves better backward transfer than the baselines designed for continual model merging in each setting. In the measured 8-task ViT-B/32 setting, merging is faster than the optimization-based baselines. Correct pairings improve accuracy in every tested order of the 8- and 20-task ViT-B/32 streams. Flan-T5 and Gemma-2-2B results extend the evaluation to language models.