Ada-HiSS: Spectral-Sparse Model Merging in the Small-Room Regime

Model merging consolidates multiple fine-tuned experts into a single shared backbone without joint retraining. Most merging rules, however, are largely developed in dense, overparameterized regimes, where models have enough redundancy to absorb interference among task updates. We consider a deployment-oriented setting in which task experts are fine-tuned from a structurally pruned backbone rather than from a dense model, reflecting edge and resource-constrained scenarios. In this setting, we discover the Small-Room problem where pruning reduces the effective degrees of freedom available for consolidation, causing task updates to interfere more as the number of experts grows. To address this problem, we propose Ada-HiSS, an Adaptive Hybrid Spectral-Sparse merging method. Ada-HiSS decomposes each task update into a globally allocated low-rank spectral core and a sparse residual tail. The spectral core captures dominant low-rank structure in each task update, while the sparse tail preserves localized task-specific changes discarded by the spectral core. We evaluate Ada-HiSS in two complementary settings: shared-backbone merging and task-update compression. In both settings, Ada-HiSS substantially improves over existing parameter-space and spectral baselines, with the largest gains on interference-sensitive fine-grained and scene-recognition tasks.