acceptodds
Under review as a conference paper at ICLR 2027

Bilinear and Pairing-Aware Continual Model Merging

Abstract

Continual model merging (CMM) combines independently fine-tuned task models as they arrive, without returning to earlier checkpoints or task data. Preserving the parameter directions of past tasks is only part of this problem. The merger must also retain enough information to correct later updates. We introduce BiPaCo (Bilinear admission with Pairing-Aware Correction), a training-free method that stores consolidated input and output bases and a bounded ledger of paired spectral records. These records preserve the within-task coupling between input and output directions. Bilinear admission discards the incoming component aligned with both previous bases, while pairing-aware correction re-queries surviving records in the spectral blocks of the provisional delta and adjusts selected blocks toward historical targets in closed form. Using one configuration, BiPaCo leads training-free methods in accuracy across all nine CLIP-ViT settings and achieves better backward transfer than the baselines designed for continual model merging in each setting. In the measured 8-task ViT-B/32 setting, merging is faster than the optimization-based baselines. Correct pairings improve accuracy in every tested order of the 8- and 20-task ViT-B/32 streams. Flan-T5 and Gemma-2-2B results extend the evaluation to language models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual model merging, Training-free model merging, Bilinear admission, Pairing-aware correction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MgbFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates data-free and training-free continual model merging (CMM) and proposes BiPaCo to mitigate knowledge interference and forgetting during sequential model merging. The method consists of two key components: Bilinear Admission, which uses bilinear projection onto historical input and output subspaces to filter conflicting components of incoming task updates, and Pairing-Aware Correction, which preserves historical paired spectral information and applies closed-form corrections in the current spectral space to mitigate parameter drift. Experiments on CLIP-ViT, Flan-T5, and Gemma-2-2B demonstrate competitive accuracy and improved retention of previously acquired knowledge across multiple task settings.

Critical strengths

Well-motivated and coherent method design. BiPaCo addresses the loss of historical task information in continual model merging by combining bilinear admission with pairing-aware correction. The two components complement each other by reducing interference from incoming task updates and correcting accumulated parameter drift, respectively.

Strong empirical performance and comprehensive evaluation. The experiments cover multiple CLIP-ViT architectures, task stream lengths, and language models. BiPaCo achieves the best average accuracy among training-free methods across all nine CLIP-ViT settings and consistently improves backward transfer over the main continual merging baselines. The ablation studies further support the effectiveness of the proposed components.

Critical weaknesses

The novelty of the spectral correction mechanism relative to existing work is not sufficiently established. The proposed pairing-aware correction shares a similar high-level motivation with Singular Value Calibration (SVC, ICML 2026), which also uses historical task contributions in the merged spectral space to correct model merging results without additional training. Although BiPaCo extends this idea to continual model merging and performs block-wise coefficient correction rather than solely rescaling singular values, the paper does not sufficiently clarify the conceptual and technical differences from SVC. In particular, the independent benefits of pairing-aware block correction over simpler spectral calibration strategies remain unclear.

Bilinear admission may over-protect historical subspaces and suppress novel task updates. Bilinear admission removes the component jointly aligned with the consolidated input and output subspaces. However, this constraint protects all combinations of historical input and output directions, rather than only the observed within-task pairings emphasized by the paper. Consequently, an incoming update containing a novel input–output pairing may be completely removed even if it is orthogonal to the previously observed task updates. The paper does not sufficiently investigate whether this conservative constraint limits plasticity as the historical subspaces expand.

Questions for authors

Regarding novelty relative to existing spectral calibration methods: How does the proposed pairing-aware correction differ fundamentally from SVC (ICML 2026), beyond its continual setting and historical record storage? Since the correction reduces to a scalar coefficient adjustment for singleton spectral blocks, could the authors report the contribution of singleton versus multi-dimensional block corrections, or compare against a simpler online spectral calibration baseline under the same memory budget?

Regarding the consistency between bilinear admission and pairing-aware modeling: If preserving within-task input–output pairings is a central motivation, why does bilinear admission remove the entire component jointly aligned with the historical input and output subspaces, including combinations that may never have appeared in previous tasks? Could this over-protection suppress novel task-specific directions, and how would a pairing-aware admission alternative affect the stability–plasticity trade-off?

Reviewer_UK5UOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BiPaCo, a training-free method for continual model merging that sequentially integrates task-specific models without accessing previous task data or checkpoints. BiPaCo maintains consolidated input and output subspace bases and a bounded memory of paired spectral records. It uses bilinear admission to reduce interference with previous tasks and pairing-aware correction to adjust the merged model based on historical information. Experiments on CLIP-ViT, Flan-T5, and Gemma-2-2B show competitive accuracy and improved backward transfer compared with existing continual merging methods, without requiring gradient-based optimization.

Critical strengths
  1. BiPaCo achieves the highest average accuracy among training-free methods across all nine CLIP-ViT settings and consistently outperforms existing continual merging baselines in backward transfer. It also demonstrates competitive results on language models. These improvements are achieved without gradient-based optimization or additional inference overhead, highlighting its practical value for continual model merging.

  2. The method combines minimum-change bilinear projection with pairing-aware correction to balance the preservation of previous tasks and the integration of new ones. The ablation studies show substantial performance drops when correction is removed and consistent degradation when historical singular-vector pairings are scrambled, providing empirical support for the proposed design.

Critical weaknesses
  1. BiPaCo requires approximately 625 MiB of additional historical state on 20-task ViT-B/32. In Table 4, scrambled pairings still recover 14.9 of the full method's 19.5 percentage-point gain over no correction. Without comparisons against simpler history-based correction methods under matched memory budgets, it remains unclear how much of the improvement comes from the pairing-aware mechanism rather than additional historical information.

  2. The consolidated bases grow monotonically and are never pruned, even when historical records are evicted by FIFO. As both bases approach their rank limits, bilinear admission can completely suppress incoming updates in square layers and reduce to one-sided projection in rectangular layers. The experiments cover at most 20 vision tasks and do not analyze basis saturation or the fraction of incoming updates retained, leaving the method's effectiveness over longer task streams uncertain.

Questions for authors

Can the authors compare BiPaCo against simpler history-based correction methods, such as the running mean task vector mentioned in the Introduction, under matched memory budgets to isolate the contribution of pairing-aware correction?

Reviewer_bPX4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies strict-online, data-free continual model merging (CMM), where independently fine-tuned task models arrive sequentially and past task checkpoints and training data cannot be revisited. BiPaCo retains consolidated input/output subspace bases together with a bounded ledger of paired spectral records. Its bilinear admission step removes only the component of an incoming task matrix that is aligned with both historical input and output subspaces. A subsequent pairing-aware correction re-queries past spectral records in the provisional merged model's current basis and pulls selected blocks toward a history-derived target. The method requires no merge-time gradients, task samples, or adapters. The paper reports substantial improvements across 8/14/20-task CLIP-ViT streams, Flan-T5/GLUE, and exploratory Gemma-2-2B domains. My principal concern is not whether BiPaCo performs well in its stated protocol—the results and important ablations are convincing—but whether the comparisons sufficiently isolate the value of its specific paired correction mechanism from (i) retaining considerably more per-task historical information and (ii) converting originally offline spectral methods into memoryless pairwise online operators.

Critical strengths
  1. Clear and relevant problem formulation. Strict-online merging is genuinely different from offline merging. By requiring that the next task be merged without returning to past data or checkpoints, the paper targets a practical constraint that ordinary task-vector aggregation does not resolve. BiPaCo's separation of admission and correction is conceptually sensible: the former screens interference from the incoming delta, while the latter can modify distortions already present in the running merge. The fact that this process is gradient-free at merge time is accurately described; I do not regard it as test-time adaptation.

  2. Strong results across multiple scales and task orders. For ViT-B/32, Table 1 reports ACC of 84.0% versus 75.5% for OPCM on eight tasks, and 72.2% versus 65.7% on twenty tasks. The eight- and twenty-task BWT values (2.0% and -3.2%) also exceed OPCM (-6.3% and -7.8%). Results extend to ViT-B/16, ViT-L/14, Flan-T5, and limited Gemma experiments. Reporting ten vision task orders and separately analyzing first-task degradation makes the evidence more informative than a single final ACC number.

  3. The paper provides unusually useful component controls. Table 4 shows that removing correction lowers eight-task ACC from 84.0% to 66.2%, while removing admission projection lowers it to 69.5%. Scrambling historical input/output pairings lowers ACC to 79.4% (and 67.6% rather than 72.2% on twenty tasks). Supplement B.4 reports paired 95% confidence intervals of [3.4, 5.8] and [3.2, 6.0] percentage points for the full-minus-scrambled contrasts. These are real positive evidence that the historical representation contains useful task information and that admission and correction should not be dismissed as cosmetic components.

  4. Helpful disclosure of operational trade-offs. The paper reports 46.9 seconds for eight-task ViT-B/32 merging compared with 41.8 seconds for OPCM and distinguishes merge construction from inference. It also gives component-level persistent-state costs, including bases, ledger, and merged delta (Supplement Table 15), rather than claiming zero storage overhead.

Critical weaknesses
  1. The main spectral baseline comparison does not separate better online merging from additional historical information. Supplement B.3 states that ISO-C, TSV-M, and KnOTS-TIES are adapted to strict-online operation by applying their operators to only (D^(k-1), tau_k) at each task arrival. This loses individual old task vectors and their original singular directions. BiPaCo, in contrast, explicitly retains consolidated bases and a per-task spectral ledger. The gap in Table 1 thus measures both the benefit of the proposed operations and the benefit of retaining richer history. Figure 2(a) usefully acknowledges this issue: when applied to all task vectors offline, TSV-M is approximately one point better than BiPaCo at twenty tasks, whereas its pairwise online version is dramatically worse. That is not an unfair comparison within the strict-online definition, nor does it invalidate BiPaCo; however, it weakens any broader inference that its spectral merging mechanism is intrinsically superior. The critical missing experiment is an online, historical-state-budget-matched spectral alternative, such as an incremental TSV-style method or a compact recent-task spectral cache.

  2. The comparison omits Iso-CTS, a particularly relevant extension of the included ISO-C baseline. The ICML 2025 Iso-CTS method explicitly retains common and task-specific singular subspaces; its original paper shows improvements over Iso-C particularly as the task count grows. This overlaps conceptually with BiPaCo's claim that discarding task-specific spectral information is harmful. I recognize that the published Iso-CTS is an offline method: simply applying it to all past task vectors would violate the strict-online protocol and should NOT be demanded as an apples-to-apples primary baseline. Nevertheless, the paper should explain why a bounded-memory online analogue cannot be constructed, or evaluate one under matched state constraints. Comparing only to the weaker Iso-C variant makes it difficult to establish how much of the gain specifically requires within-task left/right pairings rather than the more general strategy of preserving task-specific spectral structure. Relevant prior work: Marczak et al., ICML 2025, "No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces."

  3. The method's persistent historical-state cost is significant and not matched across training-free baselines. Supplement Table 15 reports for twenty-task ViT-B/32: 474.7 MiB of bases, 150.5 MiB of ledger, and 324.0 MiB of merged delta, for 949.2 MiB total. Thus the state is approximately 2.93 times the size of the merged delta alone. For eight-task Flan-T5-base, the reported persistent state is 2,004.1 MiB, of which 791.6 MiB is bases and 362.4 MiB is ledger. Importantly, the nominal rho=0.5 cap applies only to the ledger, not the potentially expanding bases. These numbers are much smaller than storing all complete task vectors, but they do not establish an advantage over a simpler method permitted to use the same storage. A matched-memory benchmark e.g., OPCM with a compact subspace/low-rank history, or a FIFO task-vector cache, would be more diagnostic than comparison solely with baselines that retain no corresponding history. The paper's retention claim beyond twenty tasks is also constrained by both basis saturation and FIFO eviction.

  4. The ablations demonstrate that pairing matters, but do not yet establish that the full history-target and correction-strength construction is necessary. Scrambling right singular vectors is a meaningful control for the correctness of stored pairings, but it is a deliberately corrupted-history baseline, not a competitive alternative that stores correct information in a different form. Furthermore, Supplement Table 12 reports that replacing the historical target with a simple mean yields ACC 84.3% versus 84.0% for the full method on eight tasks and 72.3% versus 72.2% on twenty tasks. Removing the agreement term also yields 84.3% and 72.3%; a fixed correction strength lambda=0.5 gives 83.6% and 72.0%. There may be modest BWT advantages for the full design, but the large ACC improvement appears to rely more on admission plus access to past spectral information and some correction than on several proposed weighting components. The paper would be stronger if it identified a minimal competitive variant and showed why paired-history correction improves over simple history-aware interpolation under the same memory budget.

  5. Reproducibility and generalization of the sequential benchmark need a clearer accounting. Supplement B.3 specifies that the CLIP Table 1 baseline values are taken from Qiu et al. (2026), whereas BiPaCo and some additional controls are run for this submission. Although the authors state that checkpoints and task orders match, this reuse makes detailed parity of implementations, scaling schedules, precision, and resource limits important to verify. The alpha and rho hyperparameters are tuned on disjoint task orders G11-G13, but these orders reuse the same task collection as G1-G10; this checks order overfitting, not transfer to unseen task identities. Appendix B.5 also shows noticeable sensitivity to alpha (eight-task development ACC 69.0%, 84.0%, 81.1%, and 75.0% for alpha=0.2, 0.4, 0.6, 0.8). Ten-order vision experiments are valuable, but the narrower language evidence and absence of longer recurring/conflicting streams leave open whether the historical correction rule remains reliable when the task family changes or old ledger records are evicted.

Questions for authors
  1. Strict-online, equal-memory comparison: Can the authors compare BiPaCo with a baseline that stores roughly the same total historical-state bytes (bases + ledger), for example an OPCM-style projection augmented by compressed previous-task SVD records, a bounded low-rank replay cache, or an incremental TSV/ISO-style method? Please keep the same arrival order, original task vector, checkpoint initialization, and storage budget, and report ACC/BWT and merging time. This would most directly establish whether the gain is from pairing-aware correction rather than simply retaining more history.

  2. Iso-CTS versus ISO-C: Why was Iso-CTS omitted when its task-specific-subspace preservation directly relates to the claimed motivation? If a strictly online Iso-CTS analogue is infeasible or incompatible with the memory budget, please explain which computational/data requirement prevents a fair adaptation. Reporting original offline Iso-CTS only as a contextual upper/reference point would also help, but I would not treat such an offline number as an equal-constraint comparison.

  3. Simplest sufficient correction: Supplement Table 12 suggests that simple-mean targets, removal of agreement, and fixed lambda retain essentially all of the full method's ACC. What specific failure mode requires the full recurrence/relevance/agreement-dependent target and strength rather than a simpler history-aware weighted interpolation? Can the authors show paired confidence intervals or targeted examples where the added complexity improves older-task retention or resolves conflicting task records?

  4. State cost and long-stream behavior: Please provide matched-budget ACC/BWT curves for total persistent state—not ledger budget alone—and indicate basis widths, ledger occupancy, and FIFO eviction events over the twenty-task streams. When the basis approaches full rank and older records are evicted, how does the admission/correction method protect early tasks, and are longer or cyclic task streams available?

  5. Baseline/protocol parity and hyperparameter dependence: For the CLIP baselines imported from Qiu et al. (2026), can the authors state exact checkpoint/order identity, implementation versions, scaling coefficients, rank/precision settings, and whether any baseline was retuned for the new comparisons? How sensitive is performance to alpha on previously unseen task collections, rather than only unseen permutations of the same tasks?

Comments on the paper

2 comments
Area_Chair_6ASAOpenReview ↗

Author- Reviewer Discussion

Dear Reviewers, the authors have submitted their response. Kindly review it and update your review where appropriate.

AuthorsOpenReview ↗

Summary of Discussion and Manuscript Revisions

We thank the Area Chair for facilitating the discussion and all three reviewers for their careful reviews and follow-up comments. Differences are computed by subtracting the corresponding ACC or BWT values. Brackets indicate paired 95% confidence intervals across the ten task orders.

1. Correction with the same records and admission (bPX4, MgbF)

We compared update rules while keeping bilinear admission, the spectral records, and the FIFO policy fixed. (Appendix B.9)

On the 20-task ViT-B/32 stream, BiPaCo achieved 5.3% [4.2, 6.4] higher ACC than raw parameter-space interpolation using the same records, with higher ACC in all ten orders. BiPaCo also achieved 2.7% [2.0, 3.5] higher ACC than shrink-only, which rescales the provisional delta without consulting the records. At the same β=0.05\beta=0.05, raw interpolation, which adds the mean of the reconstructed records, yielded 2.6% [2.2, 3.0] lower ACC than shrink-only. On 8 tasks, BiPaCo exceeded shrink-only by 3.2% [2.7, 3.7] in ACC and 4.4% [3.7, 5.1] in BWT.

On 20 tasks, raw interpolation had 1.7% [0.6, 2.6] higher BWT than BiPaCo (−1.5% and −3.2%, respectively). The BWT difference (BiPaCo minus shrink-only) was −0.8% [−1.6, 0.1], with zero in the interval. On the first 19 tasks, mean ACC at arrival and at the end of the stream was 75.9% and 72.7% for BiPaCo, and 68.9% and 67.4% for raw interpolation.

Scrambling only the pairings between input and output directions lowered ACC by 4.6% on both 8 and 20 tasks. These controls provide evidence that the update rule and the preservation of pairings each contribute to final accuracy in the evaluated settings.

Removing the agreement term or using a simple-mean target changed ACC by at most 0.3%. On 20 tasks, both variants had 0.3% lower BWT, with confidence intervals of [0.2, 0.5] and [0.2, 0.4], respectively. This suggests that these choices mainly help retain performance on earlier tasks. (Section 4.2, Table 4, and Appendices B.4 and B.9)

SVC rescales the singular values of the merged matrix. BiPaCo uses both the left and right singular vectors of the provisional delta to query paired records and correct selected spectral blocks. (Section 2)

With the same records and memory cap, BiPaCo exceeded the SVC-based control from our response to MgbF by 4.5% [3.4, 5.7] in BWT on 8 tasks, and by 1.0% [0.6, 1.4] in ACC and 1.1% [0.2, 2.1] in BWT on 20 tasks. The 8-task ACC difference was 0.4% [−0.2, 1.0], and the interval included zero.

2. Comparisons under the same auxiliary memory cap (bPX4, UK5U)

Under the same 648 MiB cap for bases and history on 20 tasks, BiPaCo exceeded the compressed running-mean control by 2.5% [1.5, 3.5] in ACC and 3.1% [2.3, 4.2] in BWT. It exceeded OPCM with history by 4.0% [3.4, 4.6] in ACC and 2.8% [1.7, 4.0] in BWT.

The BWT advantage over the running-mean control also held at 567 MiB and 810 MiB. On 30 tasks, under the same cap, BiPaCo exceeded the running-mean control by 2.4% [1.6, 3.2] in ACC and 4.9% [4.1, 5.7] in BWT. (Section 4.3 and Appendices B.6 and B.7)

3. Bilinear admission and plasticity in longer streams (MgbF, UK5U, bPX4)

On 20 tasks, bilinear admission achieved 17.4% [14.5, 20.3] higher ACC and 20.6% [17.2, 24.0] higher BWT than the observed-pair admission control, which removes only components corresponding to observed pairings. (Appendix B.4, Table 13)

At the final task arrival, rectangular MLP layers retained 54.4% of the incoming update energy after admission in the 20-task stream and 52.2% in the 30-task stream. (Appendix B.6) Square attention projections admitted little update energy late in the original 20-task stream. (Figure 4) The Conclusion identifies how basis saturation affects plasticity well beyond 30 tasks as an open question.

BiPaCo used the same (α,ρ)(\alpha, \rho) across 8, 14, 20, and 30 tasks. The comparisons under matched memory caps and those using the same records support preserving task-wise pairings and using them in the spectral blocks of the current merge.

Changes in the Revised Paper (marked in blue)

  • Section 2 explains how BiPaCo's correction differs from SVC.
  • Section 4.2, Table 4, and Appendix B.9, Tables 20 to 22 present the update controls, β\beta selection, and the analysis of update magnitude.
  • Section 4.3 and Appendix B.7, Tables 17 to 19 report comparisons under matched memory caps, results across caps, and merging times.
  • Appendix B.3 reports baseline reruns in Table 12 and explains the exclusion of the pairwise application of Iso-CTS.
  • Appendix B.4, Table 13 reports the observed-pair admission control.
  • Appendix B.6, Figure 4, and the Conclusion cover basis width, update energy, 30-task results, and basis saturation in longer streams.

We thank all three reviewers again for their thoughtful questions. We would be glad to address any further points during the remainder of the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.