- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a merging method in the more challenging and realistic setting of pruned models, i.e. where the capacity of the weight matrices is reduced by pruning, a practice needed in many practical deployment scenarios. They do so by decomposing the update into (i) a low-rank "spectral core" that differently from previous approaches is allocated model-wise, and (ii) a task-specific "sparse tail" that recovers the largest-magnitude entries of the residuals after the low-rank dense part. It is evaluated under two settings: a single merged backbone (after a whitening applied to the spectral cores, while the sparse tails are summed to the merged model directly at inference) and the quality of the compressed updates, stored and applied independently.
Critical strengths
The setting is clear, novel (to my knowledge), interesting and well-posed. I like the general idea, how it is presented, the notation following existing literature, the clarity and the narrative in general.
Critical weaknesses
The paper has a strong base, but I've found many critical aspects that motivate my score. The authors present several propositions (and prove them) however, from my understanding:
- Proposition 1 asserts that pruning reduces the effective update dimension, but this is neither shown empirically nor studied. Moreover, the isotropic assumption conflicts with the low-rank nature of updates proven in [2] and that the method itself exploits.
- Proposition 2 can be replaced by the Eckart-Young theorem.
Concerning other weaknesses not related to the theoretical aspect:
- The pruning effect on the ranks of task vectors and on the interference should be measured empirically, e.g. subspace alignment ratio [3] or singular task interference [2] (or better both).
- An analysis of singular values and the network-wise allocated budget should be included. Moreover, the singular values are compared blindly across layers; the previous analysis should clarify if this is principled.
- How Ada-HiSS distributes ranks is not studied sufficiently.
- In the shared-backbone setting, the sparse tail is applied in an oracle way from my understanding. Results without this contribution should be reported (in the merging literature is accepted to use the head in an oracle fashion, but usually weights must be fixed).
- Table 3 should report the occupied memory; maybe a test using TSV-C at a matched occupied budget with the proposed approach should be considered to validate if the different rank allocation is beneficial.
- Baselines exclude other state-of-the-art spectral merging methods like ISO-C [3] and ISO-CTS [3] and KnOTS [4].
- Several baseline accuracies look below the literature. Every table should report the finetuning and zero-shot reference rows (subscript normalised accuracy would be appreciated). A task arithmetic reference might help as well.
- Table 18 questions the impact of the sparse tail, thus the gain over TSV-C seems to be due to the different rank allocation strategy that, however, at the moment is not investigated enough in my opinion.
- The information on allocated memory and a test at a matching memory budget are needed to consolidate the method's specific gain.
*References in the questions.
Questions for authors
See weaknesses.
- How is the spectral budget k chosen? Can you show how ranks get allocated model-wise?
- Can you report used memory per task for each method in Table 3?
- Replicate the ablation of Table 18 for the shared-backbone scenario.
- Can you show singular value magnitudes across layers? If their magnitude differs how does this relate to the spectral allocation budget?
- Propositions 2 and 3 should be linked to the Eckart-Young theorem, are they just a rephrasing?
[1] Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, Ali Farhadi: Editing models with task arithmetic. ICLR 2023
[2] Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, Emanuele Rodolà: Task Singular Vectors: Reducing Task Interference in Model Merging. CVPR 2025: 18695-18705
[3] Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartlomiej Twardowski, Andrew D. Bagdanov, Joost van de Weijer: No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces. ICML 2025
[4] George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, Judy Hoffman: Model merging with SVD to tie the Knots. ICLR 2025