acceptodds
Withdrawn from ICLR 2027

Ada-HiSS: Spectral-Sparse Model Merging in the Small-Room Regime

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model merging consolidates multiple fine-tuned experts into a single shared backbone without joint retraining. Most merging rules, however, are largely developed in dense, overparameterized regimes, where models have enough redundancy to absorb interference among task updates. We consider a deployment-oriented setting in which task experts are fine-tuned from a structurally pruned backbone rather than from a dense model, reflecting edge and resource-constrained scenarios. In this setting, we discover the Small-Room problem where pruning reduces the effective degrees of freedom available for consolidation, causing task updates to interfere more as the number of experts grows. To address this problem, we propose Ada-HiSS, an Adaptive Hybrid Spectral-Sparse merging method. Ada-HiSS decomposes each task update into a globally allocated low-rank spectral core and a sparse residual tail. The spectral core captures dominant low-rank structure in each task update, while the sparse tail preserves localized task-specific changes discarded by the spectral core. We evaluate Ada-HiSS in two complementary settings: shared-backbone merging and task-update compression. In both settings, Ada-HiSS substantially improves over existing parameter-space and spectral baselines, with the largest gains on interference-sensitive fine-grained and scene-recognition tasks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merging, Model Compression, Pruning, Spectral Decomposition, Low-Rank Approximation, Multi-Task Learning, Parameter-Efficient Learning, Resource-Constrained Deep Learning, Edge AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AT3sOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a merging method in the more challenging and realistic setting of pruned models, i.e. where the capacity of the weight matrices is reduced by pruning, a practice needed in many practical deployment scenarios. They do so by decomposing the update into (i) a low-rank "spectral core" that differently from previous approaches is allocated model-wise, and (ii) a task-specific "sparse tail" that recovers the largest-magnitude entries of the residuals after the low-rank dense part. It is evaluated under two settings: a single merged backbone (after a whitening applied to the spectral cores, while the sparse tails are summed to the merged model directly at inference) and the quality of the compressed updates, stored and applied independently.

Critical strengths

The setting is clear, novel (to my knowledge), interesting and well-posed. I like the general idea, how it is presented, the notation following existing literature, the clarity and the narrative in general.

Critical weaknesses

The paper has a strong base, but I've found many critical aspects that motivate my score. The authors present several propositions (and prove them) however, from my understanding:

  • Proposition 1 asserts that pruning reduces the effective update dimension, but this is neither shown empirically nor studied. Moreover, the isotropic assumption conflicts with the low-rank nature of updates proven in [2] and that the method itself exploits.
  • Proposition 2 can be replaced by the Eckart-Young theorem.

Concerning other weaknesses not related to the theoretical aspect:

  • The pruning effect on the ranks of task vectors and on the interference should be measured empirically, e.g. subspace alignment ratio [3] or singular task interference [2] (or better both).
  • An analysis of singular values and the network-wise allocated budget should be included. Moreover, the singular values are compared blindly across layers; the previous analysis should clarify if this is principled.
  • How Ada-HiSS distributes ranks is not studied sufficiently.
  • In the shared-backbone setting, the sparse tail is applied in an oracle way from my understanding. Results without this contribution should be reported (in the merging literature is accepted to use the head in an oracle fashion, but usually weights must be fixed).
  • Table 3 should report the occupied memory; maybe a test using TSV-C at a matched occupied budget with the proposed approach should be considered to validate if the different rank allocation is beneficial.
  • Baselines exclude other state-of-the-art spectral merging methods like ISO-C [3] and ISO-CTS [3] and KnOTS [4].
  • Several baseline accuracies look below the literature. Every table should report the finetuning and zero-shot reference rows (subscript normalised accuracy would be appreciated). A task arithmetic reference might help as well.
  • Table 18 questions the impact of the sparse tail, thus the gain over TSV-C seems to be due to the different rank allocation strategy that, however, at the moment is not investigated enough in my opinion.
  • The information on allocated memory and a test at a matching memory budget are needed to consolidate the method's specific gain.

*References in the questions.

Questions for authors

See weaknesses.

  1. How is the spectral budget k chosen? Can you show how ranks get allocated model-wise?
  2. Can you report used memory per task for each method in Table 3?
  3. Replicate the ablation of Table 18 for the shared-backbone scenario.
  4. Can you show singular value magnitudes across layers? If their magnitude differs how does this relate to the spectral allocation budget?
  5. Propositions 2 and 3 should be linked to the Eckart-Young theorem, are they just a rephrasing?

[1] Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, Ali Farhadi: Editing models with task arithmetic. ICLR 2023

[2] Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, Emanuele Rodolà: Task Singular Vectors: Reducing Task Interference in Model Merging. CVPR 2025: 18695-18705

[3] Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartlomiej Twardowski, Andrew D. Bagdanov, Joost van de Weijer: No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces. ICML 2025

[4] George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, Judy Hoffman: Model merging with SVD to tie the Knots. ICLR 2025

Reviewer_ZVYKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work the authors investigate model merging in capacity constrained settings, focusing on structurally pruned models. The authors introduce the small-room problem, arguing that pruning reduces the effective dimensionality available, thereby increasing the cross-task interference. To address this issue, they propose Ada-HiSS, a spectral-space approach that decomposes task vectors into a low-rank spectral core, allocated globally across tasks and layers, and a sparse residual component to preserve task-specific information.

The method is evaluated under two protocols: Shared-bakcbone merging, where spectral components are built into a common backbone with task specific residuals, and Task update compression, where task vectors are compressed without cross-task aggregation.

Critical strengths
  • Relevant and underexplored setting. The paper investigates model merging under structured pruning, an interesting scenario that has recieved limited attention. Understanding how reduced model capacity affects task specific knowledge is an important research direction.

  • Simple and intuitive methodological design. The combination of globally allocated low-rank spectral components and sparse residual corrections is straightforward and does not require additional training or data during merging.

  • Evaluation across multiple pruning levels and task configuration. The experiments consider 8, 14, and 20 tasks across different pruning ratios, including additional results on pruned DeiT models.

Critical weaknesses
  1. Unfair comparison against TSVM and missing task-conditioned baselines. In the shared backbone protocol, Ada-HiSS applies task specific sparse residuals selected using the task identity at inference time (Eq. 19), whereas TSVM, its only baseline in Tab. 2, produces a single task-independent merged backbone. While both use task-specific classification heads, Ada-HiSS additionally modifies the backbone itself depending on the task. Thus, the reported improvements cannot be attributed solely to the proposed spectral merging strategy. A more appropriate evaluation should include methods operating under similar assumptions, such as EMR-Merging (which is present only in the compression comparison), TALL-masks (Wang, Ke, et al. "Localizing task information for improved model merging and compression." 2024), or other task-dependent merging methods or MoE based approaches. TSVC would also be a relevant comparison in this setting. Additionally, the sparse residuals introduce non-negligible overhead, reaching around 40% of the shared backbonbe size in the 20-task setting, so comparisons under same storage budgets should be considered.

  2. Limited connection between theoretical analysis and experimental validation. A central contribution of the paper is the theoretical characterization of the small-room effect. However, the connection between the proposed theoretical model and the experimental results remains insufficient. Proposition 1 demonstrates that the expected squared projection of isotropic task vectors scales inversely with the effective subspace dimension. While mathematically correct under the stated assumptions, this result does not directly establish increased task interference in terms of downstream loss. Specifically, the loss-based formulation in Eq. (3) depends on the task-specific curvature matrix, whereas Proposition 1 considers Euclidean projections onto task-vector directions. The relationship between these quantities is not established. Additionally, the theoretical quantities introduced in the analysis, particularly the effective update-space dimension and projected interference, are never measured experimentally. The evaluation primarily reports classification accuracy across pruning ratios, without investigating whether the observed degradation is actually associated with increased task-vector overlap or reduced effective dimensionality. Similar concerns apply to Propositions 2 and 3, which establish optimality for Frobenius reconstruction error rather than merging performance or interference reduction. Overall, the theoretical analysis remains largely disconnected from both the proposed method and its experimental validation.

  3. Insufficient ablation studies and unclear contribution of the sparse residual. The paper does not provide adequate ablations to isolate the contributions of its main components in the Shared-Backbone Protocol. In particular, there is no comparison between Ada-HiSS without task-specific residuals and TSVM, nor an evaluation of TSVM augmented with equivalent sparse residual corrections. This makes it difficult to determine whether the improvements originate from the proposed spectral allocation or the additional task-specific capacity. Moreover, the ablation results in Appendix F, Table 18, raise further concerns about the sparse residual component. Under the reported compression budget, the spectral-core-only variant achieves 77.16% average accuracy, compared to 77.17% for the complete method. Interestingly, the core-only variant also performs slightly better on Cars (67.73% vs. 67.50%) and SUN397 (39.89% vs. 39.63%), despite these being highlighted as tasks where sparse residuals are particularly beneficial. These findings weaken the empirical motivation for the sparse residual and are not adequately coeherent with the interpretation of the main results.

Questions for authors
  1. Fairness of the Shared-Backbone comparison. Given that Ada-HiSS relies on task-specific sparse residuals at inference time, why is TSVM, which produces a task-independent merged backbone, selected as the only baseline? Could the authors provide comparisons against task-conditioned methods such as EMR-Merging, TALL-masks, or TSVC under comparable storage budgets?

  2. Experimental validation of the Small-Room theory. Could the authors provide empirical evidence connecting the theoretical quantities introduced in Proposition 1 to the observed degradation in merging performance? In particular, how does the effective dimensionality of the task-update space change with pruning, and does the projected interference defined in Eq. (5) actually increase as predicted? Furthermore, what assumptions connect the Euclidean projection analyzed in Proposition 1 to the curvature-weighted loss introduced in Eq. (3)?

  3. Evaluation consistency across protocols. The Task-Update Compression Protocol explicitly reconstructs independent task-specific updates without cross-task aggregation. Could the authors clarify how TIES and EMR are implemented under this protocol, particularly whether they produce independent task-specific reconstructions or conventional merged models? This distinction is important to ensure that all methods are evaluated under equivalent assumptions. Additionally, could the authors clarify why improvements in task-update reconstruction should be interpreted as evidence of reduced cross-task interference?

Reviewer_SS6pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies consolidation of experts fine-tuned from a common structurally pruned backbone. Ada-HiSS allocates spectral components across tasks and layers, then supplements the low-rank representation with sparse residuals. It evaluates shared-backbone merging with task-specific tails and independent task-update compression, reporting gains over spectral baselines. The theory provides a projected-interference model and reconstruction optimality results.

Critical strengths
  1. Pruning before expert fine-tuning is a plausible resource-constrained workflow. Allocating budget according to update spectra is more flexible than fixing each layer's rank. Separating a shared core from task residuals also offers a straightforward storage–accuracy trade-off.
  2. Table 2 reports improvements across dense and three pruned configurations at three task counts, including 64.02% to 67.92% for Taylor 0.7 with 20 tasks. The authors explicitly describe shared-backbone-plus-tail deployment and report storage overhead, validation selection, and limitations.
Critical weaknesses
  1. Proposition 1 describes expected projected overlap for isotropic, normalized updates in a low-dimensional space; Propositions 2–3 instead optimize reconstruction error. Retaining updates by singular-value energy does not directly imply less task conflict, and spectral alignment is inherited. The analysis therefore does not establish why the new allocation should alleviate pruning-induced interference.
  2. Compression ablations give 77.16% for the spectral core alone and 77.17% for the full method, with the former slightly better on Cars and SUN397. This does not substantiate Section 5.3's attribution of fine-grained gains to tails. Shared-protocol controls for core-only merging and TSVM with identical tails are also missing, although 20-task tails add about 40% storage. The disclosed overhead still leaves representation benefits and additional task state confounded.
  3. Section 4.5 and Table 1 exclude cross-task aggregation from compression, yet Table 22's pruned TIES merging diagnostic exactly matches compression Table 23 task by task. How TIES and EMR are adapted to independent compression needs explanation; the description does not establish that they solve the same problem. Moreover, spectral-component counts do not equal storage, and the compressed methods lack storage accounting that includes factors and sparse indices.
Questions for authors
  1. What information about task interference does global singular-value ranking use? Beyond reducing update reconstruction error, how does it relate to the projected overlap described in Proposition 1?

  2. What parameter objects do TIES and EMR store and evaluate under compression? Why are the TIES results identical in Tables 22–23? Please report actual storage including factors, sparse indices, and non-matrix parameters.

  3. Under the shared protocol, what are the accuracy and actual storage of the global spectral core alone, TSVM with equally budgeted task tails, and the full method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.