- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates data-free and training-free continual model merging (CMM) and proposes BiPaCo to mitigate knowledge interference and forgetting during sequential model merging. The method consists of two key components: Bilinear Admission, which uses bilinear projection onto historical input and output subspaces to filter conflicting components of incoming task updates, and Pairing-Aware Correction, which preserves historical paired spectral information and applies closed-form corrections in the current spectral space to mitigate parameter drift. Experiments on CLIP-ViT, Flan-T5, and Gemma-2-2B demonstrate competitive accuracy and improved retention of previously acquired knowledge across multiple task settings.
Critical strengths
Well-motivated and coherent method design. BiPaCo addresses the loss of historical task information in continual model merging by combining bilinear admission with pairing-aware correction. The two components complement each other by reducing interference from incoming task updates and correcting accumulated parameter drift, respectively.
Strong empirical performance and comprehensive evaluation. The experiments cover multiple CLIP-ViT architectures, task stream lengths, and language models. BiPaCo achieves the best average accuracy among training-free methods across all nine CLIP-ViT settings and consistently improves backward transfer over the main continual merging baselines. The ablation studies further support the effectiveness of the proposed components.
Critical weaknesses
The novelty of the spectral correction mechanism relative to existing work is not sufficiently established. The proposed pairing-aware correction shares a similar high-level motivation with Singular Value Calibration (SVC, ICML 2026), which also uses historical task contributions in the merged spectral space to correct model merging results without additional training. Although BiPaCo extends this idea to continual model merging and performs block-wise coefficient correction rather than solely rescaling singular values, the paper does not sufficiently clarify the conceptual and technical differences from SVC. In particular, the independent benefits of pairing-aware block correction over simpler spectral calibration strategies remain unclear.
Bilinear admission may over-protect historical subspaces and suppress novel task updates. Bilinear admission removes the component jointly aligned with the consolidated input and output subspaces. However, this constraint protects all combinations of historical input and output directions, rather than only the observed within-task pairings emphasized by the paper. Consequently, an incoming update containing a novel input–output pairing may be completely removed even if it is orthogonal to the previously observed task updates. The paper does not sufficiently investigate whether this conservative constraint limits plasticity as the historical subspaces expand.
Questions for authors
Regarding novelty relative to existing spectral calibration methods: How does the proposed pairing-aware correction differ fundamentally from SVC (ICML 2026), beyond its continual setting and historical record storage? Since the correction reduces to a scalar coefficient adjustment for singleton spectral blocks, could the authors report the contribution of singleton versus multi-dimensional block corrections, or compare against a simpler online spectral calibration baseline under the same memory budget?
Regarding the consistency between bilinear admission and pairing-aware modeling: If preserving within-task input–output pairings is a central motivation, why does bilinear admission remove the entire component jointly aligned with the historical input and output subspaces, including combinations that may never have appeared in previous tasks? Could this over-protection suppress novel task-specific directions, and how would a pairing-aware admission alternative affect the stability–plasticity trade-off?
Author- Reviewer Discussion
Dear Reviewers, the authors have submitted their response. Kindly review it and update your review where appropriate.