- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Merging What Matters (MWM), a three-stage framework for merging LoRA-adapted task-specific models. First, Layer Importance-based Expert Refinement (LIER) uses layer-wise ablation on validation data to split each expert’s updates into task-critical core layers and lower-impact auxiliary layers. Second, Strengthened Expert Routing (SER) trains a lightweight MLP router to predict per-input expert relevance and select the top-k most relevant experts. Third, Layer-Adaptive Dynamic Merging (LDM) asymmetrically combines selected experts: the top-ranked primary expert contributes its full core updates, while other selected experts contribute only attenuated auxiliary updates to reduce cross-task interference. The authors evaluate MWM on 8 NLP GLUE tasks (Qwen2.5 models) and 10 CV classification tasks (ViT models), demonstrating consistent improvements over a wide range of state-of-the-art merging baselines, performance matching or exceeding individual fine-tuned models, scalability to 7B language models, and retained out-of-domain generalization on MMLU. The paper also includes formal theoretical analysis supporting the non-uniform contribution of LoRA layers and the expressivity advantage of layer-wise merging.
Critical strengths
-
The unified three-stage MWM framework addresses three key under-explored limitations of existing model merging methods—uniform layer treatment within experts, static expert weighting, and undifferentiated layer integration—in a single cohesive pipeline. Empirical results across both NLP and CV modalities consistently outperform over 15 state-of-the-art merging baselines, and even surpass individual task-specific fine-tuned models on average while preserving out-of-domain generalization. This is consequential because it delivers clear, consistent performance gains on a widely relevant problem, with evaluation rigor spanning multiple modalities, model scales, and settings.
-
The paper provides extensive formal theoretical analysis that rigorously justifies MWM’s core design choices. It proves that LoRA layers contribute non-uniformly to task specialization, that layer-wise merging has strictly greater representational capacity than model-level merging when layers prefer different expert mixtures, and that top-k routing and asymmetric core/auxiliary integration have controlled approximation error. This is consequential because most existing model merging work relies on empirical heuristics without formal grounding, and this analysis both validates the proposed method and provides a reusable theoretical framework for future layer-aware merging research.
Critical weaknesses
-
The core conceptual insights of the work are largely incremental and well-established in prior literature. The observation that different LoRA layers contribute unevenly to task-specific performance has been widely documented in work on parameter-efficient fine-tuning and importance-aware model merging, and the claim that layer-wise merging expands the feasible solution space relative to model-level merging is a straightforward structural property. This limits the novelty of the paper’s foundational motivation.
-
The proposed method aligns more closely with a dynamic mixture-of-experts (MoE) paradigm than with traditional model merging, undermining its framing as a merging approach. Unlike standard merging methods that produce a single fixed parameter set with no inference-time overhead beyond the base model, MWM retains separate expert parameters, requires a dedicated routing network, and performs per-input dynamic parameter composition. It does not reduce runtime parameter count or deliver the deployment simplicity that is a core motivation of the model merging paradigm.
Questions for authors
-
How do the authors respond to the characterization of MWM as a dynamic mixture-of-experts variant rather than a model merging method? Specifically, what unique advantages does the approach offer over standard MoE architectures, given that it retains separate expert parameters and requires per-input routing?
-
Since uneven layer contribution across tasks is already a well-documented observation in the LoRA and model merging literature, what uniquely new conceptual or technical insight does this work’s theoretical analysis provide that was not established by prior work on importance-aware merging?
-
Can the authors quantify the full end-to-end computational cost of the MWM pipeline—including LIER ablation preprocessing and SER router training—relative to the cost of training the original LoRA experts? How does inference latency compare to both static merging methods and standard MoE baselines of similar capacity?
4 comments
I thank the authors for their detailed rebuttal and additional experiments, which help clarify the design rationale, computational efficiency, and empirical advantages of MWM. I acknowledge that MWM achieves substantial performance improvements over the evaluated MoE baselines while maintaining relatively low inference overhead. However, my primary concern remains unresolved. In my view, a fundamental motivation of model merging is to integrate task-specific knowledge into a single unified model in a simple, efficient, and scalable manner, ideally without retaining separate experts or introducing additional inference-time mechanisms. Despite its effective layer-wise parameter composition, MWM still relies on task-specific expert parameters, a learned routing network, and input-dependent dynamic merging. Consequently, I consider MWM conceptually closer to an efficient MoE-style adaptation framework than to conventional model merging. While this design is practically valuable, its increased complexity and departure from the original motivation of model merging limit my enthusiasm for the contribution, particularly given the relatively incremental nature of its underlying methodological insights. Therefore, after carefully considering the authors' responses and additional evidence, I have decided to maintain my original rating.
Thank you for acknowledging MWM’s practical value. We respectfully disagree that retaining task-specific parameters, using a learned router, and performing input-dependent composition are sufficient grounds to place MWM outside model merging. These features distinguish dynamic merging from static merging, but do not establish a boundary between model merging and MoE:
- Twin-Merging (NeurIPS 2024) retains compressed task-specific residuals, trains a router, and dynamically combines those residuals with shared parameters. It therefore exhibits all three characteristics raised in the review. MWM instead derives component partitions from source-task layer ablations and changes each expert’s contributing components according to its primary or secondary role. [1]
- FREE-Merging (ICCV 2025) retains lightweight task-specific experts, trains a router, and dynamically adds selected experts to a merged backbone—again sharing all three characteristics. Its components come from frequency-domain filtering and magnitude-based extraction; MWM uses measured layer-level performance contributions and primary-core/secondary-auxiliary composition. [2]
- DaWin (ICLR 2025) retains model weights for sample-dependent interpolation, demonstrating that inference-time parameter adaptation is also studied within model merging. MWM additionally performs role-dependent selection of layer components. [3]
MWM’s innovation is to make layer selection conditional on an expert’s role for each input. LIER identifies core and auxiliary layers through source-task ablations; LDM then retains the primary expert’s core updates and introduces attenuated auxiliary updates from secondary experts. Thus, the same expert contributes different components when its role changes, extending dynamic weighting to role-dependent component selection. This differs from Twin-Merging’s weighting of compressed residuals, FREE-Merging’s addition of extracted task experts, and DaWin’s sample-dependent interpolation. We therefore ask that MWM be assessed on this specific mechanism, rather than treating static merging’s deployment properties as necessary conditions for model merging generally.
References
[1] Lu et al. Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging. NeurIPS 2024.
[2] Zheng et al. FREE-Merging: Fourier Transform for Efficient Model Merging. ICCV 2025.
[3] Oh et al. DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation. ICLR 2025.
Dear Reviewer,
With fewer than one day left in the discussion period, we would appreciate any further thoughts you may have on our responses and additional results. Your feedback would help us understand whether our explanations have resolved your concerns. Should any questions remain, we would welcome the opportunity to clarify them before the discussion closes.
Thank you again for your time and thoughtful review.
Q1 / W2
Sorry for the confusion regarding the relationship between MWM and MOE. MWM goes beyond expert routing by selectively composing layer updates to preserve task-critical knowledge and reuse complementary capabilities without retraining source experts. LIER identifies critical layers, SER selects relevant experts, and LDM combines the primary expert’s core updates with attenuated auxiliary updates from others, limiting cross-expert interference rather than uniformly mixing complete experts.
Following your suggestion, we added MOE comparisons using the paper’s backbones/tasks: Qwen2.5-0.5B-Instruct/eight NLP tasks; ViT-Base-Patch16-224/ten CV tasks.
MOE values average three runs; MWM values are from the main tables. Following sparse gating[1], CV uses eight width-655 FFN experts in the final block to match MWM’s total parameter count; NLP uses shared dense weights plus eight LoRA experts in all 24 blocks. Both use token-level top-k output mixing, AdamW (2e-4), and task loss plus 0.01 times load-balancing loss. CV trains experts/router for 200 steps (batch 4), freezing the remaining backbone/heads; NLP trains LoRA/router for three epochs, freezing dense weights and selecting checkpoints by validation target-token loss. MWM exceeds the best tested CV/NLP MOE by 16.88/10.11 points in these configurations.
Q2 / W1
Thanks for your question.
Our theoretical contribution quantifies when layer-specific expert mixtures improve the objective and how their additional flexibility can be controlled. AdaLoRA[2] allocates adaptation budgets by importance; AdaMerging[3] learns task/layer coefficients shared across inputs. MWM instead links source-task importance to an expert’s role for each input. Specifically:
(1) For a block-separable quadratic model, Ql≻0, and interior unconstrained optima, D.3.2, Eq. (64), gives
Jg∗−Jlw∗=21∑l∥αl∗−αg∗∥Ql2.
The gain measures curvature-weighted disagreement in layer-preferred mixtures: it vanishes when all layers prefer the same coefficients. Appendix D.3.3 transfers this gain to smooth risk when it exceeds Taylor remainders.
(2) With bounded loss and independent validation data, D.4.1 gives complexity terms log M versus L log M for M shared versus M^L layer-wise candidates. This motivates masks, shared routing and one attenuation parameter; it is not a full-pipeline generalization guarantee.
(3) D.3.8 separates low source-task importance from beneficial transfer. For auxiliary update A and H_x-smooth conditional risk,
Rx(θc+λA)−Rx(θc)≤λ⟨∇Rx(θc),A⟩+21Hxλ2∥A∥22.
Negative alignment and sufficiently small λ>0 ensure improvement, providing a theoretical basis for attenuating auxiliary updates. Table 6 further supports the effectiveness of LIER and LDM empirically.
Q3
Thanks for your thoughtful question regarding the full computational cost and inference efficiency of MWM.
The full pipeline (loading, LIER, and SER preprocessing/training/selection/saving) costs 428.5 s in NLP (LIER: 418.2 s; SER: 1.1 s), or 10.3 s excluding offline LIER, and 128.2–147.4 s in CV. Across three CV runs, the mean reuse cost is 137.34 s versus 553.6 s for training all source experts, adding 24.8% overhead (690.9 s total; 1.2× expert training). Offline LIER/SER preparation is reusable for fixed experts; online routing remains necessary.
Batch latency (ms/batch; Table 12 instead reports dataset-level times):
MWM adds 2–7 ms (NLP) and 1–2 ms (CV) over static averaging; it is 2.2–6.9 times faster than tested NLP MOEs. Where effective-weight composition avoids repeated full FFN branches, MWM pays routing/composition overhead instead of k nonlinear expert evaluations per token. NLP uses experts in 24 blocks, CV only one, explaining the smaller CV benefit.
[1] Shazeer et al. Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017.
[2] Zhang et al. AdaLoRA. ICLR 2023.
[3] Yang et al. AdaMerging. ICLR 2024.