BiFilt: Bilateral Subspace Filtering for Data-Free Continual Model Merging

Data-free continual model merging (DFCMM) aims to sequentially integrate independently fine-tuned experts into a single model without access to task data or previously seen experts. Existing subspace-based methods typically protect historical knowledge through input-side filtering, imposing $ZV=0$ and thereby suppressing all update responses along selected historical input directions. We show that preserving the selected historical input-output core requires only the weaker condition $U^{\top }ZV=0$, revealing cross-subspace updates that are removed by one-sided filtering despite leaving the protected core unchanged. Based on this observation, we introduce BiFilt, a bilateral subspace filtering framework that decomposes each incoming update into four input-output interaction blocks, removes the jointly historical component, preserves the complementary component, and independently controls the two directional cross components. We derive BiFilt as the unique solution of a convex quadratic objective, yielding a simple closed-form filter and a fixed symmetric configuration without task-dependent tuning. Controlled task-pair analyses show that the released cross direction contains substantial new-task information while causing only limited historical degradation, and complete continual-merging experiments across multiple CLIP backbones and task horizons consistently demonstrate the benefit of partial cross-block retention over one-sided filtering. Further analyses reveal a stability-plasticity trade-off: partial retention improves task acquisition and final accuracy on standard sequences, whereas stricter filtering can provide stronger retention over longer horizons. These results show that distinguishing input-output interaction directions provides a more flexible parameter-only principle for continual model merging.