acceptodds
Withdrawn from ICLR 2027

PALM-Merge: Preserving Task Residuals for Data-Free Continual Model Merging

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Data-free continual model merging can continually extend deployed foundation models without retaining task data or retraining, but it must integrate new checkpoints while preserving earlier capabilities. Existing methods fold successive task deltas into shared weights and control interference by projection or filtering. However, forcing heterogeneous responses into one map creates an approximation floor and can overwrite directions required by earlier tasks. To address this limitation, we propose PALM-Merge (Preserve-and-Activate Low-rank Merging), which stores each task residual as an independent low-rank expert and activates experts sparsely from inputs, separating response preservation from access and avoiding cross-task parameter overwriting. Specifically, truncated SVD converts each task update into a frozen low-rank expert whose compression error is controlled by the discarded singular energy, while routing each activation to the most relevant low-rank expert at inference without access to any training data. Theoretically, we show that a historical prediction can change only when a newly added expert changes the routing decision. Forgetting can then occur only if the resulting perturbation is large enough to overturn the original classification margin. This yields a cumulative accuracy-loss bound for task \(c\) of \(\min{1,\sum_t=c+1^Tp_t,c}\). Across nine FusionBench CLIP-ViT settings, achieves \(89.8%\) average accuracy. On a controlled eight-task stream, it outperforms NUFILT by \(4.85%\) and reduces positive final forgetting by \(65.9%\), while reaching \(82.75%\) on Flan-T5 GLUE-6. These gains require \(2.10\times\) storage but only \(1.24\times\) active-tensor cost.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Model Merging, Data-Free Method, Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Exu7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper considers the continual model merging setting (i.e. where experts and tasks arrive in a continuous stream) and proposes, instead of incrementally merging all the tasks into a single weight set, to store every new model as a low-rank update. Then, at inference time, they decide which expert processes each token with a data-free router. The routing is based on the subspace spanned by the top right singular vectors of the updates themselves. It also includes formalization of the setting from a theoretical perspective.

Critical strengths
  1. The decomposition of final accuracy in arrival task accuracy and retention is an intuitive and useful formalization.
  2. The method is truly data-free, and label-free.
  3. Results cover a sufficient variety of tasks, modalities and models.
  4. Theory is correct.
Critical weaknesses

The paper does not include the mandatory AI use statement and has the wrong ICLR2026 template.

The paper is self-contained, but it overlooks an extensive part of the Merging [1-3] and MoErging [4-7] literature to which it closely aligns, once this literature is included the novelty reduces to the setting (that is not novel by itself, but the missing comparable methods were never tested in this scenario) and the formalization only:

  1. the static merging methods considered in Table1 and Table 2 are not the state-of-the-art spectral ones [1-3]. These methods make use of the same SVD of the layer updates but are not cited in the paper. The interference these methods [1-3] define overlaps with PaLM-Merge's definition. In their case the setting motivated different solutions; they should be included in the comparison, and acknowledged.
  2. The low-rank experts the paper proposed correspond to TSV-C proposed by [1] and it should be acknowledged in the text. More a similar SVD task compression method is TwinMerging [6] that is not mentioned.
  3. The routing mechanism proposed is not novel and coincides with what MASS [4] proposed, except that MASS makes one routing decision per sample and pays roughly 2x active tensor cost. MASS is not cited and SMILE [5] is cited without discussing the differences with the method.
  4. Both MASS [4] and SMILE [5] should be included in Table 1 and Table 2, and the differences should be explained clearly.
  5. MASS [4] and SMILE [5] provide a much fairer comparison ground than the considered methods, as they both suffer from the same storage penalty by keeping the same low-rank updates and routing on them, have similar computation drawbacks, and they can be extended to the continual merging setup naturally without modifications.
  6. I also recommend that the authors align with the task arithmetic and spectral merging notation for the theoretical part to make the paper easier to understand.

*References below.

Questions for authors

See weaknesses.

  1. How does PALM-Merge differ from SMILE [5] and MASS [4] (except for the continual merging setting) ?
  2. How does the proposed method compare against SMILE [5], MASS [4], TSVM [1], IsoC, IsoCTS [2] and KnOTS [3]?
  3. Extending MASS and SMILE to the continual setting should be trivial; while static merging methods should serve as a bound for performances in the continual setting. Could the authors report the performance of MASS and SMILE when extended to the continual merging setup?

[1] Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, Emanuele Rodolà: Task Singular Vectors: Reducing Task Interference in Model Merging. CVPR 2025: 18695-18705

[2] Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartlomiej Twardowski, Andrew D. Bagdanov, Joost van de Weijer: No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces. ICML 2025

[3] George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, Judy Hoffman: Model merging with SVD to tie the Knots. ICLR 2025

[4] Donato Crisostomi, Alessandro Zirilli, Antonio Andrea Gargiulo, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, Iacopo Masi, Emanuele Rodolà: MASS: MoErging through Adaptive Subspace Selection. ICLR 2026

[5] Anke Tang, Li Shen, Yong Luo, Shuai Xie, Han Hu, Lefei Zhang, Bo Du, Dacheng Tao: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models. IEEE Trans. Pattern Anal. Mach. Intell. 48(2): 1145-1157 (2026)

[6] Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, Yu Cheng: Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging. NeurIPS 2024

Reviewer_e4HXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PALM-Merge, a data-free continual model merging framework that preserves task-specific knowledge by storing task updates as independent low-rank experts and sparsely routing inputs to relevant experts at inference. It provides theoretical guarantees on historical prediction stability and cumulative forgetting, showing that forgetting depends on routing changes and whether the resulting perturbations exceed the original classification margins. Experiments on nine FusionBench CLIP-ViT settings, an eight-task continual learning stream, and Flan-T5 GLUE-6 demonstrate improved accuracy and reduced forgetting compared with existing methods, at the cost of increased storage.

Critical strengths
  1. The paper addresses the practical challenge of integrating new task-specific checkpoints without retaining training data while preserving previously acquired capabilities.

  2. PALM-Merge combines frozen low-rank residual experts with input-dependent sparse routing, directly targeting parameter interference and historical knowledge retention.

Critical weaknesses
  1. Limited novelty. The core components, low-rank residual decomposition and activation-based expert routing, are well-established techniques. The paper should more clearly distinguish its novel contributions from this prior method, particularly beyond the continual-merging formulation and theoretical analysis.
  2. The retention analysis provides useful sufficient conditions for prediction preservation, but the cumulative forgetting bound depends on routing-change and margin-related probabilities that may be difficult to estimate or control in practice. Its practical predictive value remains unclear.
  3. Routing effectiveness remains insufficiently validated. The appendix provides a detailed mathematical analysis of projection-based routing, including its limitations, but does not empirically establish how reliably it selects suitable experts. Comparisons with alternative routing strategies and analyses of routing switches and their impact on prediction errors would strengthen the method's justification.
  4. Unresolved storage and comparison fairness concerns. The expert bank increases total storage from 2.10× at TA8 to 3.75× at TA20. The appendix explicitly acknowledges that the existing comparisons are not memory-matched and that separating the benefits of storage from those of routing requires controlled comparisons. This leaves it unclear how much of the performance gain comes from the proposed design rather than its additional parameter capacity.
  5. Incomplete efficiency and ablation evaluation. The appendix provides useful complexity analysis and studies expert rank and top-1 routing, but the evaluation remains limited in practical efficiency and routing design. Direct runtime measurements, tests of alternative gate ranks or top-kk settings, and a stronger analysis of storage–accuracy trade-offs would make the empirical conclusions more convincing. The claimed computational benefits are currently supported mainly by analytical cost estimates rather than measured latency.
Questions for authors

See weaknesses.

Reviewer_QjJkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses data-free continual model merging: integrating successive checkpoints into a frozen foundation model without retaining task data or retraining, while preserving earlier capabilities. Existing methods project or filter later task deltas into shared weights, which the authors argue leaves an approximation floor and can overwrite directions required by earlier tasks. PALM-Merge (Preserve-and-Activate Low-rank Merging) stores each arriving task update as an independent frozen low-rank expert and activates experts sparsely from inputs, avoiding cross-task parameter overwriting.

Critical strengths
  1. Useful performance decomposition and dual-modality coverage. Splitting final performance into arrival accuracy and backward transfer (Eq. 1), and analyzing integration error separately from retention error, is a helpful analytical framing for data-free merging. The paper also reports results on both vision (CLIP-ViT) and language (Flan-T5) streams, broadening the empirical surface compared with vision-only merging studies.
Critical weaknesses
  1. Poor readability: theory and definitions are stacked without a guiding narrative. Sections 2–4 compress several analytical threads—shared-map approximation floor, selection error, routing stability, classification margin, and cumulative forgetting—into a dense sequence of equations (Eqs. 2, 3, 11–14) and definitions without an intuitive bridge or a worked numerical example. Figure 1's three panels map onto Sections 2, 3, and 4 only after repeated back-and-forth reading, and the essential distinction between PALM-Merge and existing sparse-expert, expert-bank, and adapter-routing schemes is covered in a single Related Work sentence. As written, a reviewer outside this narrow sub-area cannot reconstruct the problem motivation, design rationale, or failure modes in one pass; the manuscript needs a restructured narrative with a running example, explicit intuition for each theorem, and clearer positioning against prior merging and Mixture-of-Experts work.
  2. Motivation and method positioning are unclear; the link between theory and design choices is not established. The paper does not clearly answer why each arriving task must be frozen as an independent low-rank expert rather than reusing established sparse-expert, adapter, or projection-based merging schemes (OPCM, DOP, NUFILT, AdaMerging, and FLEM are cited but not contrasted on the exact constraints PALM-Merge claims to add). It also remains unclear whether the analysis in Sections 2 and 4 actually derives the non-trivial design choices—expert rank r = 64, query size q = 16, top-k = 1—or whether these choices are post-hoc and the theory serves as retrospective justification. Without this link, the theoretical contribution and the experimental contribution are difficult to evaluate together.
Questions for authors
  1. What concrete failure case in existing data-free continual merging motivated this work? Compared with the straightforward baseline of storing each task delta as an independent expert and routing by input similarity, what single constraint or capability does PALM-Merge's problem definition add that prior methods did not cover?
  2. Why choose the truncated-SVD low-rank expert as the minimal preservation unit, rather than a full checkpoint, an adapter vector, or a projection subspace? Which quantity in the analysis—compression error C_r or selection error S_g—dictates the default rank r = 64, query q = 16, and top-k = 1, and how should these values change as the task stream grows?
  3. Section 4 states that a historical prediction is preserved unless a new expert first changes the routing path. How does this claim differ in mechanism from established Mixture-of-Experts routing stability analyses, including Shazeer et al. (2017) and later expert-routing work? How should the geometric routing-stability condition h^{\top}(P_i - P_j)h > 0 guide the actual choice of rank, routing score, and top-k in practice?
  4. Figure 2 decomposes the 4.85% final-accuracy gap into 3.30% stronger integration and 1.55% better retention. What is the exact accounting definition behind this split? Table 3 shows added storage growing from 61.46% at rank 32 to 401.37% at rank 256: how much of the integration gain is attributable to the larger expert bank (more stored parameters) rather than the preserve-and-activate mechanism?
  5. The routing score uses only the projection norm of the current activation h onto checkpoint-derived subspaces G_e, with no task labels or historical calibration. On streams with large cross-domain distribution shift, what does a misrouting failure look like, and is there a recovery or fallback path when no expert scores above a threshold?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.