PLEAT: Progressive LoRA Expert Aggregation and Transport for Continual Model Merging

Continual learning aims to acquire knowledge from new tasks while retaining previously learned capabilities. One way to pursue this goal is through model merging, which combines the weights of independently trained experts without additional training. A central challenge in continual model merging is to integrate new experts without retaining an expanding expert pool. In resource-constrained settings, merging experts trained with low-rank adaptation (LoRA) must also preserve their compact representation. Existing merging strategies face three limitations in this setting: (1) joint merging requires retaining and revisiting an expanding collection of historical experts; (2) methods designed for fully fine-tuned models typically operate on dense updates, giving up LoRA’s compact representation when applied directly to LoRA experts; and (3) fixed-rank LoRA merging preserves compactness but can become a capacity bottleneck as knowledge accumulates over long task sequences. We propose PLEAT, Progressive LoRA Expert Aggregation and Transport, a training-free approach that exploits the low-rank structure of LoRA experts to maintain a factorized merged update with progressively adjustable capacity. PLEAT incrementally updates a shared input basis under a prescribed rank schedule, aligns each incoming expert to the updated representation, and transports historical coefficients before aggregation. Experiments across multiple backbone models show that PLEAT outperforms a range of existing merging methods, while retaining the parameter-efficient representation of LoRA and enabling continual merging without revisiting historical expert weights.