acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Merged Vector: Fixed-Budget Directional Memory for Data-Free Continual Model Merging

Abstract

Data-free continual model merging aims to progressively integrate sequentially arriving task models into a unified model without accessing historical data or retaining previous task models. Existing methods typically construct historical subspaces from the cumulative merged task vector, but such implicit historical representations progressively degrade as tasks accumulate. To address this limitation, we propose **CUSP-Merge** (**CU**mulative **S**hared–**P**rivate Memory with Adaptive Protection for Continual Model Merging), a fixed-budget Shared–Private directional memory framework that replaces the implicit historical subspace with explicitly maintained historical task directions. Specifically, CUSP-Merge consolidates directions repeatedly supported across tasks into Shared Memory, retains task-specific residual directions in Private Memory, and dynamically promotes private directions to Shared Memory according to cumulative cross-task support, thereby enabling fine-grained organization of historical directions. Furthermore, this structured memory enables direction-level adaptive protection to balance stability and plasticity. Extensive experiments across multiple model architectures and task configurations demonstrate that CUSP-Merge achieves the highest average accuracy in all settings, with more pronounced gains on longer task sequences, while maintaining a favorable performance–storage trade-off.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual model merging, data-free model merging, task vectors, directional memory, subspace methods, fixed-budget memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PzJZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that a cumulative merged task vector loses some directions that matter to earlier tasks. CUSP-Merge therefore maintains a fixed-size memory of historical directions, with shared directions for repeated cross-task support and private directions for task-specific information. The memory also sets direction-level protection strengths when a new model is merged. Experiments measure final accuracy, forgetting, directional coverage, and storage across CLIP backbones and task orders.

Critical strengths
  1. The paper separates loss caused by a fixed rank from loss caused by reconstructing history from the merged vector. The equal-rank coverage analysis supports this distinction.
  2. The shared–private design has task-level evidence: on ViT-B/32 with 20 tasks, it raises worst-task accuracy from 35.43% for Single Memory to 43.52%.
  3. The growing-memory comparison makes the fixed-budget trade-off measurable across sequence lengths.
Critical weaknesses
  1. The average gain over NUFILT is modest in several 8- and 14-task settings. The paper does not report paired significance tests for this strongest baseline.
  2. CUSP-Merge has more negative BWT because it reaches higher accuracy when tasks first arrive.
  3. The full performance-versus-storage analysis focuses on Growing Memory. It does not give the same accounting for NUFILT and the other strongest continual-merging baselines.
  4. Directional coverage is measured on task vectors. Higher coverage is consistent with better retention, but it is not a direct measure of which directions preserve task accuracy.
  5. The experiments use related CLIP vision encoders and classification tasks. The memory rule’s behavior with substantially different model structures remains untested.
Questions for authors

Please refer to weakness.

Reviewer_7msYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CUSP-Merge for data-free continual model merging with a fixed-budget Shared–Private directional memory. The paper reports consistent accuracy improvements across three CLIP backbones and task sequences of different lengths.

Critical strengths
  1. The paper addresses a well-motivated problem. Explicitly maintaining historical directions offers a plausible alternative to reconstructing historical subspaces from the cumulative merged vector.

  2. The proposed framework combines cumulative direction promotion with adaptive protection. The memory construction and merging objective provide a coherent approach to balancing historical retention and current-task adaptation.

  3. Experiments cover multiple architectures, task-sequence lengths, and ten task orders. The performance–storage comparisons, early-task retention analysis, and BWT decomposition provide useful additional evidence.

Critical weaknesses

1. The source of the accuracy improvements remains insufficiently isolated.

CUSP-Merge retains 320 historical directions per layer, whereas the NUFILT proxy in the motivating diagnostic uses rank 128. Figure 7 extends the coverage comparison to rank 320, but no corresponding accuracy comparison with rank-320 NUFILT is reported. How much of the improvement comes from explicitly maintaining historical directions rather than increasing the historical subspace capacity? Please provide a rank-controlled accuracy comparison with comparable compensation and optimization settings.

Table 2 supports adaptive protection through the fixed-protection variants. However, “w/o Cumulative Promotion” also disables adaptive protection, so its 1.97-point decrease cannot be attributed to cumulative promotion alone. Similarly, Figure 8 compares 256 Shared directions against 256 Shared plus 64 Private directions, which does not isolate the partition’s benefit at fixed total capacity. The update and protection rules of Single Memory also need clarification.

2. Shared Memory consolidation needs a more precise specification.

Section 4.1 describes combining existing Shared directions with newly promoted directions and retaining principal components through SVD. However, it does not specify the matrix being decomposed or how stored spectral energies enter it.

Appendix A.1 states that shared and private directions are mutually orthogonal. If the existing Shared directions and newly promoted Private directions are concatenated without weighting, their nonzero singular values would all be equal, leaving no spectral preference for truncation. What weighting is used, how are historical energies carried forward, and how is orthogonality maintained? An explicit update equation or pseudocode would make this central operation reproducible.

3. The first-stage coverage discrepancy needs clarification.

Section 3.1 initializes θ1merged=θ1\theta^{\mathrm{merged}}_1=\theta_1. The Merged-Vector Proxy and the rank-128 Joint Full-History Subspace should therefore both be constructed from the first task alone. With identical inputs, analyzed layers, and effective ranks, their coverage under Eq. (1) should coincide.

However, Section 3.2 reports first-stage proxy coverage of 0.7342, while the reference curves in Figure 1 begin at approximately 0.87. What explains this difference? Please clarify the inputs, ranks, and stage indexing of each branch. Otherwise, the later gaps may partly reflect differences in diagnostic construction rather than information loss from multi-task aggregation.

4. The practical role of cumulative support remains unclear.

With the defaults minit=1m^{\mathrm{init}}=1 and M=2M=2, a private direction is promoted after a single subsequent task whenever st,j≥ηs_{t,j}\geq\eta. Weaker supports can instead accumulate across several tasks. What fractions of directions follow each path, and how many are evicted before promotion?

These statistics would clarify whether promotion primarily reflects one strong subsequent overlap or evidence accumulated across multiple tasks. The sensitivity curves cannot distinguish these effects, since changing MM or η\eta also changes private protection before promotion. An ablation holding the protection rule fixed would help establish the independent benefit of cumulative promotion.

Questions for authors

see Weaknesses

Reviewer_ysxaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies data-free continual model merging, pointing out that existing methods implicitly reconstruct historical subspaces from accumulated merged task vectors, which gradually lose historical directions. It proposes CUSP-Merge: an approach that explicitly maintains a fixed-budget Shared-Private directional memory, divides directions into shared and private memory based on cross-task support, and uses direction-level adaptive protection intensity to balance stability and plasticity. The paper provides a clear analysis of its motivation, presents a comprehensive method, and conducts experiments covering the CLIP ViT-B/32, B/16, L/14 and 8/14/20 task sequences. It utilises 10 randomised task sequences and reports standard deviations, whilst also including ablation studies, hyperparameter tuning, memory-performance trade-offs, and coverage analysis.

Critical strengths
  1. The method addresses a clearly motivated problem. The paper analyzes how historical-subspace coverage degrades when reconstructed from the cumulative merged vector, then proposes explicit directional memory that distinguishes cross-task shared directions from private directions that have not yet received sufficient support.
  2. The experiments support the main claims and include useful analyses. CUSP-Merge achieves the highest average accuracy across all nine backbone and task-sequence configurations. The paper also evaluates memory budgets and component variants, and analyzes retention of early-arriving tasks and worst-task performance. The results suggest that fixed-budget memory is useful for longer task sequences.
Critical weaknesses

The statistical support for some of the reported gains is limited. The paper reports mean accuracy and standard deviation over ten task orders, but does not provide paired tests or confidence intervals for the main comparisons with NUFILT, despite evaluating the methods on the same orders. This makes it difficult to assess the reliability of the smaller gains, particularly in the 8- and 14-task settings.

Questions for authors
  1. What are the per-task merging time and peak GPU memory of CUSP-Merge compared with NUFILT, OPCM, and DOP?
  2. Could the authors report paired confidence intervals or paired statistical tests across the ten shared task orders for CUSP-Merge versus NUFILT, especially for the 8- and 14-task settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.