- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper considers the continual model merging setting (i.e. where experts and tasks arrive in a continuous stream) and proposes, instead of incrementally merging all the tasks into a single weight set, to store every new model as a low-rank update. Then, at inference time, they decide which expert processes each token with a data-free router. The routing is based on the subspace spanned by the top right singular vectors of the updates themselves. It also includes formalization of the setting from a theoretical perspective.
Critical strengths
- The decomposition of final accuracy in arrival task accuracy and retention is an intuitive and useful formalization.
- The method is truly data-free, and label-free.
- Results cover a sufficient variety of tasks, modalities and models.
- Theory is correct.
Critical weaknesses
The paper does not include the mandatory AI use statement and has the wrong ICLR2026 template.
The paper is self-contained, but it overlooks an extensive part of the Merging [1-3] and MoErging [4-7] literature to which it closely aligns, once this literature is included the novelty reduces to the setting (that is not novel by itself, but the missing comparable methods were never tested in this scenario) and the formalization only:
- the static merging methods considered in Table1 and Table 2 are not the state-of-the-art spectral ones [1-3]. These methods make use of the same SVD of the layer updates but are not cited in the paper. The interference these methods [1-3] define overlaps with PaLM-Merge's definition. In their case the setting motivated different solutions; they should be included in the comparison, and acknowledged.
- The low-rank experts the paper proposed correspond to TSV-C proposed by [1] and it should be acknowledged in the text. More a similar SVD task compression method is TwinMerging [6] that is not mentioned.
- The routing mechanism proposed is not novel and coincides with what MASS [4] proposed, except that MASS makes one routing decision per sample and pays roughly 2x active tensor cost. MASS is not cited and SMILE [5] is cited without discussing the differences with the method.
- Both MASS [4] and SMILE [5] should be included in Table 1 and Table 2, and the differences should be explained clearly.
- MASS [4] and SMILE [5] provide a much fairer comparison ground than the considered methods, as they both suffer from the same storage penalty by keeping the same low-rank updates and routing on them, have similar computation drawbacks, and they can be extended to the continual merging setup naturally without modifications.
- I also recommend that the authors align with the task arithmetic and spectral merging notation for the theoretical part to make the paper easier to understand.
*References below.
Questions for authors
See weaknesses.
- How does PALM-Merge differ from SMILE [5] and MASS [4] (except for the continual merging setting) ?
- How does the proposed method compare against SMILE [5], MASS [4], TSVM [1], IsoC, IsoCTS [2] and KnOTS [3]?
- Extending MASS and SMILE to the continual setting should be trivial; while static merging methods should serve as a bound for performances in the continual setting. Could the authors report the performance of MASS and SMILE when extended to the continual merging setup?
[1] Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, Emanuele Rodolà: Task Singular Vectors: Reducing Task Interference in Model Merging. CVPR 2025: 18695-18705
[2] Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartlomiej Twardowski, Andrew D. Bagdanov, Joost van de Weijer: No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces. ICML 2025
[3] George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, Judy Hoffman: Model merging with SVD to tie the Knots. ICLR 2025
[4] Donato Crisostomi, Alessandro Zirilli, Antonio Andrea Gargiulo, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, Iacopo Masi, Emanuele Rodolà: MASS: MoErging through Adaptive Subspace Selection. ICLR 2026
[5] Anke Tang, Li Shen, Yong Luo, Shuai Xie, Han Hu, Lefei Zhang, Bo Du, Dacheng Tao: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models. IEEE Trans. Pattern Anal. Mach. Intell. 48(2): 1145-1157 (2026)
[6] Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, Yu Cheng: Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging. NeurIPS 2024