acceptodds
Under review as a conference paper at ICLR 2027

Merging What Matters: Input-Adaptive and Layer-Aware Model Merging

Abstract

Model merging provides an efficient approach to consolidating the complementary capabilities of multiple task-specific models without retraining from scratch. Existing methods often combine complete task updates or weight entire experts without accounting for the different contributions of LoRA-adapted layers, hence introducing parameter conflicts and cross-task interference. To address these limitations, in this paper, we propose Merging What Matters (MWM), an input-adaptive, layer-aware model merging framework. MWM first identifies task-critical LoRA updates within each expert and attenuates less informative auxiliary updates. It then employs a lightweight router to estimate input-dependent expert relevance and select the most appropriate experts for each input. Finally, the selected experts are dynamically integrated using differentiated layer-wise rules that preserve the core knowledge of the primary expert while conservatively incorporating complementary information from other experts. Extensive experiments across natural language processing and computer vision benchmarks demonstrate that MWM consistently outperforms the SOTA model merging baselines, achieves performance comparable to or better than task-specific fine-tuned experts, and retains competitive out-of-domain generalization. Code is provided in the supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Merging, LLM, NLP, CV

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oeZzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Merging What Matters (MWM), a three-stage framework for merging LoRA-adapted task-specific models. First, Layer Importance-based Expert Refinement (LIER) uses layer-wise ablation on validation data to split each expert’s updates into task-critical core layers and lower-impact auxiliary layers. Second, Strengthened Expert Routing (SER) trains a lightweight MLP router to predict per-input expert relevance and select the top-k most relevant experts. Third, Layer-Adaptive Dynamic Merging (LDM) asymmetrically combines selected experts: the top-ranked primary expert contributes its full core updates, while other selected experts contribute only attenuated auxiliary updates to reduce cross-task interference. The authors evaluate MWM on 8 NLP GLUE tasks (Qwen2.5 models) and 10 CV classification tasks (ViT models), demonstrating consistent improvements over a wide range of state-of-the-art merging baselines, performance matching or exceeding individual fine-tuned models, scalability to 7B language models, and retained out-of-domain generalization on MMLU. The paper also includes formal theoretical analysis supporting the non-uniform contribution of LoRA layers and the expressivity advantage of layer-wise merging.

Critical strengths
  1. The unified three-stage MWM framework addresses three key under-explored limitations of existing model merging methods—uniform layer treatment within experts, static expert weighting, and undifferentiated layer integration—in a single cohesive pipeline. Empirical results across both NLP and CV modalities consistently outperform over 15 state-of-the-art merging baselines, and even surpass individual task-specific fine-tuned models on average while preserving out-of-domain generalization. This is consequential because it delivers clear, consistent performance gains on a widely relevant problem, with evaluation rigor spanning multiple modalities, model scales, and settings.

  2. The paper provides extensive formal theoretical analysis that rigorously justifies MWM’s core design choices. It proves that LoRA layers contribute non-uniformly to task specialization, that layer-wise merging has strictly greater representational capacity than model-level merging when layers prefer different expert mixtures, and that top-k routing and asymmetric core/auxiliary integration have controlled approximation error. This is consequential because most existing model merging work relies on empirical heuristics without formal grounding, and this analysis both validates the proposed method and provides a reusable theoretical framework for future layer-aware merging research.

Critical weaknesses
  1. The core conceptual insights of the work are largely incremental and well-established in prior literature. The observation that different LoRA layers contribute unevenly to task-specific performance has been widely documented in work on parameter-efficient fine-tuning and importance-aware model merging, and the claim that layer-wise merging expands the feasible solution space relative to model-level merging is a straightforward structural property. This limits the novelty of the paper’s foundational motivation.

  2. The proposed method aligns more closely with a dynamic mixture-of-experts (MoE) paradigm than with traditional model merging, undermining its framing as a merging approach. Unlike standard merging methods that produce a single fixed parameter set with no inference-time overhead beyond the base model, MWM retains separate expert parameters, requires a dedicated routing network, and performs per-input dynamic parameter composition. It does not reduce runtime parameter count or deliver the deployment simplicity that is a core motivation of the model merging paradigm.

Questions for authors
  1. How do the authors respond to the characterization of MWM as a dynamic mixture-of-experts variant rather than a model merging method? Specifically, what unique advantages does the approach offer over standard MoE architectures, given that it retains separate expert parameters and requires per-input routing?

  2. Since uneven layer contribution across tasks is already a well-documented observation in the LoRA and model merging literature, what uniquely new conceptual or technical insight does this work’s theoretical analysis provide that was not established by prior work on importance-aware merging?

  3. Can the authors quantify the full end-to-end computational cost of the MWM pipeline—including LIER ablation preprocessing and SER router training—relative to the cost of training the original LoRA experts? How does inference latency compare to both static merging methods and standard MoE baselines of similar capacity?

4 comments

Q1 / W2

Sorry for the confusion regarding the relationship between MWM and MOE. MWM goes beyond expert routing by selectively composing layer updates to preserve task-critical knowledge and reuse complementary capabilities without retraining source experts. LIER identifies critical layers, SER selects relevant experts, and LDM combines the primary expert’s core updates with attenuated auxiliary updates from others, limiting cross-expert interference rather than uniformly mixing complete experts.

Following your suggestion, we added MOE comparisons using the paper’s backbones/tasks: Qwen2.5-0.5B-Instruct/eight NLP tasks; ViT-Base-Patch16-224/ten CV tasks.

MethodCV Acc. (%)NLP Acc. (%)
MOE Top-164.459.5
MOE Top-265.270.7
MOE Top-565.372.1
MOE Top-866.373.5
MWM83.283.6

MOE values average three runs; MWM values are from the main tables. Following sparse gating[1], CV uses eight width-655 FFN experts in the final block to match MWM’s total parameter count; NLP uses shared dense weights plus eight LoRA experts in all 24 blocks. Both use token-level top-k output mixing, AdamW (2e-4), and task loss plus 0.01 times load-balancing loss. CV trains experts/router for 200 steps (batch 4), freezing the remaining backbone/heads; NLP trains LoRA/router for three epochs, freezing dense weights and selecting checkpoints by validation target-token loss. MWM exceeds the best tested CV/NLP MOE by 16.88/10.11 points in these configurations.

Q2 / W1

Thanks for your question.

Our theoretical contribution quantifies when layer-specific expert mixtures improve the objective and how their additional flexibility can be controlled. AdaLoRA[2] allocates adaptation budgets by importance; AdaMerging[3] learns task/layer coefficients shared across inputs. MWM instead links source-task importance to an expert’s role for each input. Specifically:

(1) For a block-separable quadratic model, Ql≻0Q_l\succ0, and interior unconstrained optima, D.3.2, Eq. (64), gives

Jg∗−Jlw∗=12∑l∥αl∗−αg∗∥Ql2.J_g^*-J_{\mathrm{lw}}^*=\tfrac12\sum_l\|\alpha_l^*-\alpha_g^*\|_{Q_l}^2.

The gain measures curvature-weighted disagreement in layer-preferred mixtures: it vanishes when all layers prefer the same coefficients. Appendix D.3.3 transfers this gain to smooth risk when it exceeds Taylor remainders.

(2) With bounded loss and independent validation data, D.4.1 gives complexity terms log M versus L log M for M shared versus M^L layer-wise candidates. This motivates masks, shared routing and one attenuation parameter; it is not a full-pipeline generalization guarantee.

(3) D.3.8 separates low source-task importance from beneficial transfer. For auxiliary update A and H_x-smooth conditional risk,

Rx(θc+λA)−Rx(θc)≤λ⟨∇Rx(θc),A⟩+12Hxλ2∥A∥22.R_x(\theta_c+\lambda A)-R_x(\theta_c)\le\lambda\langle\nabla R_x(\theta_c),A\rangle+\tfrac12H_x\lambda^2\|A\|_2^2.

Negative alignment and sufficiently small λ>0\lambda>0 ensure improvement, providing a theoretical basis for attenuating auxiliary updates. Table 6 further supports the effectiveness of LIER and LDM empirically.

Q3

Thanks for your thoughtful question regarding the full computational cost and inference efficiency of MWM.

The full pipeline (loading, LIER, and SER preprocessing/training/selection/saving) costs 428.5 s in NLP (LIER: 418.2 s; SER: 1.1 s), or 10.3 s excluding offline LIER, and 128.2–147.4 s in CV. Across three CV runs, the mean reuse cost is 137.34 s versus 553.6 s for training all source experts, adding 24.8% overhead (690.9 s total; 1.2× expert training). Offline LIER/SER preparation is reusable for fixed experts; online routing remains necessary.

Batch latency (ms/batch; Table 12 instead reports dataset-level times):

MethodNLP B1/B8/B32CV B1/B8/B32
Static averaging11.3/12.8/38.54.5/5.3/14.2
MWM13.3/16.8/45.55.5/6.3/16.2
MOE Top-188.8/84.6/99.27.1/9.6/18.3
MOE Top-282.6/86.3/127.07.1/9.1/18.2
MOE Top-583.7/91.9/212.17.2/9.0/18.7
MOE Top-886.0/107.4/313.17.3/9.3/19.9

MWM adds 2–7 ms (NLP) and 1–2 ms (CV) over static averaging; it is 2.2–6.9 times faster than tested NLP MOEs. Where effective-weight composition avoids repeated full FFN branches, MWM pays routing/composition overhead instead of k nonlinear expert evaluations per token. NLP uses experts in 24 blocks, CV only one, explaining the smaller CV benefit.

[1] Shazeer et al. Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017.

[2] Zhang et al. AdaLoRA. ICLR 2023.

[3] Yang et al. AdaMerging. ICLR 2024.

I thank the authors for their detailed rebuttal and additional experiments, which help clarify the design rationale, computational efficiency, and empirical advantages of MWM. I acknowledge that MWM achieves substantial performance improvements over the evaluated MoE baselines while maintaining relatively low inference overhead. However, my primary concern remains unresolved. In my view, a fundamental motivation of model merging is to integrate task-specific knowledge into a single unified model in a simple, efficient, and scalable manner, ideally without retaining separate experts or introducing additional inference-time mechanisms. Despite its effective layer-wise parameter composition, MWM still relies on task-specific expert parameters, a learned routing network, and input-dependent dynamic merging. Consequently, I consider MWM conceptually closer to an efficient MoE-style adaptation framework than to conventional model merging. While this design is practically valuable, its increased complexity and departure from the original motivation of model merging limit my enthusiasm for the contribution, particularly given the relatively incremental nature of its underlying methodological insights. Therefore, after carefully considering the authors' responses and additional evidence, I have decided to maintain my original rating.

Thank you for acknowledging MWM’s practical value. We respectfully disagree that retaining task-specific parameters, using a learned router, and performing input-dependent composition are sufficient grounds to place MWM outside model merging. These features distinguish dynamic merging from static merging, but do not establish a boundary between model merging and MoE:

  • Twin-Merging (NeurIPS 2024) retains compressed task-specific residuals, trains a router, and dynamically combines those residuals with shared parameters. It therefore exhibits all three characteristics raised in the review. MWM instead derives component partitions from source-task layer ablations and changes each expert’s contributing components according to its primary or secondary role. [1]
  • FREE-Merging (ICCV 2025) retains lightweight task-specific experts, trains a router, and dynamically adds selected experts to a merged backbone—again sharing all three characteristics. Its components come from frequency-domain filtering and magnitude-based extraction; MWM uses measured layer-level performance contributions and primary-core/secondary-auxiliary composition. [2]
  • DaWin (ICLR 2025) retains model weights for sample-dependent interpolation, demonstrating that inference-time parameter adaptation is also studied within model merging. MWM additionally performs role-dependent selection of layer components. [3]

MWM’s innovation is to make layer selection conditional on an expert’s role for each input. LIER identifies core and auxiliary layers through source-task ablations; LDM then retains the primary expert’s core updates and introduces attenuated auxiliary updates from secondary experts. Thus, the same expert contributes different components when its role changes, extending dynamic weighting to role-dependent component selection. This differs from Twin-Merging’s weighting of compressed residuals, FREE-Merging’s addition of extracted task experts, and DaWin’s sample-dependent interpolation. We therefore ask that MWM be assessed on this specific mechanism, rather than treating static merging’s deployment properties as necessary conditions for model merging generally.

References

[1] Lu et al. Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging. NeurIPS 2024.

[2] Zheng et al. FREE-Merging: Fourier Transform for Efficient Model Merging. ICCV 2025.

[3] Oh et al. DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation. ICLR 2025.

Dear Reviewer,

With fewer than one day left in the discussion period, we would appreciate any further thoughts you may have on our responses and additional results. Your feedback would help us understand whether our explanations have resolved your concerns. Should any questions remain, we would welcome the opportunity to clarify them before the discussion closes.

Thank you again for your time and thoughtful review.

Reviewer_8pVbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MWM, an input-adaptive model-merging framework that combines layer-importance-based expert refinement (LIER), a lightweight supervised router (SER), and layer-adaptive dynamic merging (LDM). It separates task-critical and auxiliary LoRA updates, selects relevant experts per input, and integrates their updates asymmetrically. Experiments cover NLP and computer vision benchmarks.

Critical strengths
  • The framework addresses three distinct sources of interference in model merging: non-uniform layer importance, input-dependent expert relevance, and conflicts between expert updates.

  • The paper provides broad experiments across NLP and vision tasks, with ablations indicating that all three proposed components contribute to performance.

Critical weaknesses
  • LIER estimates layer importance through individual layer ablation. Although the paper provides theoretical analysis and component ablations, the stability of core-layer selections across validation samples and different values of (\rho) remains insufficiently evaluated. Comparisons with alternative layer-importance criteria would further validate the approach.

  • SER is trained to predict source-expert identities, which may encourage task classification rather than identify complementary expertise required by individual inputs. Although the paper evaluates existing input-adaptive merging baselines and out-of-domain generalization, it remains unclear whether SER can effectively handle inputs requiring knowledge from multiple experts.

  • LDM's ablation results support the benefit of differentiated integration over uniform merging, but do not establish the necessity of its specific asymmetric rule. Comparisons with alternatives that combine core updates from multiple experts or adaptively weight all selected experts are needed to isolate the contribution of this design.

Minor Comment: Duplicate Reference. The paper “Language Models Are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch” appears twice in the reference list, once as an arXiv preprint and once as a conference publication.

Questions for authors

Q1. How stable are LIER's core-layer selections across validation samples and different values of (\rho)? Can the authors compare LIER with alternative layer-importance criteria?

Q2. Can the authors disentangle SER's task-classification ability from its ability to identify complementary expertise? In particular, how well does it handle inputs requiring knowledge from multiple experts?

Q3. Can the authors compare LDM's asymmetric integration rule with alternatives that combine core updates from multiple experts or adaptively weight all selected experts?

3 comments

W1 / Q1

Thanks for your insightful suggestions on calibration sensitivity, the choice of ρ\rho, and alternative importance criteria. LIER's accuracy improves with more calibration data, depends on ρ\rho, and exceeds alternative importance criteria under matched retention settings.

We varied only calibration samples, fixing ρ=0.8\rho=0.8, SER seed 42, the merging rule and evaluation set. Results average five repetitions per sample size.

Calibration examples/taskNLP Acc. (%)CV Acc. (%)
1680.780.2
3281.581.6
6482.081.9
Full calibration set82.682.1

With 64 examples, accuracy is within 0.6/0.2 points of the full-calibration reference on NLP/CV. These results measure downstream sensitivity to sampling, not core-set agreement.

For fixed rankings, varying ρ\rho produces nested core sets. Below we report all prespecified settings without test-based selection, averaging three router seeds.

ρ\rhoNLP Acc. (%)CV Acc. (%)
0.2564.270.6
0.5074.274.9
0.7582.081.7
0.9081.980.1

Accuracy peaks at ρ=0.75\rho=0.75 among these settings; retaining more layers does not invariably help.

We compared importance criteria at ρ=0.8\rho=0.8, fixing retained-layer counts, SER and evaluation data. Gradient scoring uses the same calibration samples as LIER and loss derivatives at block-update gates.

CriterionNLP Acc. (%)CV Acc. (%)
Update Frobenius norm [1]79.774.6
Random layers79.675.6
Absolute gradient × update [2]80.377.0
LIER ablation82.682.1

LIER exceeds the strongest alternative by 2.3/5.1 points on NLP/CV, supporting selection based on measured source-task performance contributions.

W2 / Q2

Thanks for your comment. Our paired-task controls separate expert retrieval from integration and show gains beyond single-expert selection.

We evaluated 447 paired-task NLP inputs covering 28 task pairs and 360 paired-image CV inputs covering 45 pairs. All methods share the inputs and, for CV, identical task-known dual heads. One NLP example was excluded by a common length limit. Results average three seeds; joint accuracy requires both answers to be correct.

Metric (%)NLPCV
Top-5 coverage of both source experts100.065.6
SER top-1 joint accuracy26.66.7
Primary core only, coefficient fixed26.27.4
Random auxiliary experts27.77.7
MWM joint accuracy30.29.7

MWM gains 3.7/3.0 points over SER top-1 on NLP/CV. Fixing the primary expert and its coefficient, auxiliary integration adds 4.0/2.3 points. Its 2.6/2.0-point advantage over random auxiliaries further supports SER's selection of useful complementary experts on these paired-task inputs.

W3 / Q3

Thanks for your insightful suggestions on evaluating LDM’s specific design choices. With routing fixed, MWM outperforms multi-core fusion and adaptive full-expert weighting.

All methods share SER outputs, top-5 experts, LIER masks and test samples:

Integration ruleNLP Acc. (%)CV Acc. (%)
Primary core + attenuated secondary auxiliary (MWM)83.683.2
Adaptively weighted cores of all selected experts81.380.6
Adaptively weighted full updates of all selected experts79.779.9
Primary core + attenuated secondary core82.582.0
Full primary + attenuated secondary auxiliary81.481.2

MWM exceeds weighted cores by 2.3/2.6 points and weighted full updates by 3.9/3.3 points on NLP/CV. Replacing secondary auxiliary updates with secondary cores, while preserving the primary coefficient and λ=0.1\lambda=0.1, reduces accuracy by 1.1/1.2 points. Using full primary updates instead of the primary core reduces accuracy by 2.2/2.0 points. These controls support both component choices in LDM's asymmetric rule under fixed routing.

We will include these settings and results in the revision and consolidate the duplicate “Language Models Are Super Mario” references, retaining the conference version and updating citations. Thank you again for your constructive feedback.

[1] Han et al. “Learning both Weights and Connections for Efficient Neural Network.” NeurIPS 2015.

[2] Molchanov et al. “Pruning Convolutional Neural Networks for Resource Efficient Inference.” ICLR 2017.

Thank you for the authors' detailed response and clarification. I appreciate the effort in addressing my concerns. I will maintain my score of 3.

Thank you for considering our response and clarifications. If our rebuttal has resolved the main concerns, could we kindly ask you to reconsider the rating in light of the additional evidence? Your feedback and support mean a great deal to us, and we respect your independent judgment.

Reviewer_ZeQuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1(−1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces a new method for model merging aimed for reducing parameter conflicts and cross-task interference. In particular, the paper proposes Merging What Matters (MWM), an input-adaptive and layer-aware framework for merging task-specific LoRA experts. MWM consists of three components. First, Layer Importance-based Expert Refinement (LIER) measures the validation-performance change caused by ablating individual LoRA-adapted layers and partitions each expert into task-critical core updates and auxiliary updates. Second, Strengthened Expert Routing (SER) trains a lightweight router to predict expert relevance for each input. Finally, Layer-Adaptive Dynamic Merging (LDM) preserves the core updates of the highest-ranked expert while incorporating attenuated auxiliary updates from other selected experts. Experiments across NLP and CV benchmarks report improvements over static, structure-aware, and input-adaptive merging baselines, with additional experiments on larger models, out-of-domain tasks, and fully fine-tuned experts.

Critical strengths
  • The paper aims to tackle an important problem (i.e., parameter conflicts and cross-task interference) in model merging by jointly considering non-uniform importance within an expert and input-dependent relevance across experts. The decomposition into expert refinement, routing, and asymmetric integration is intuitive and practically motivated.
  • The empirical evaluation is broad, covering NLP and CV tasks, multiple model scales, a relatively large collection of merging baselines, out-of-domain evaluation, efficiency analysis, and component-wise ablations. The extension to fully fine-tuned experts also suggests that the framework is not restricted to LoRA.
Critical weaknesses
  • The novelty relative to existing dynamic and structure-aware merging methods is unclear
    • The paper states that prior router-based approaches commonly operate over complete expert updates and positions input-dependent layer-wise integration as a key distinction. However, WEMoE [A] already dynamically integrates task-specific modules according to the input, while MoW-Merging [B] combines an input-conditioned gating network with different merging strategies across layers based on parameter interference. Earlier work such as TALL-Mask [H] and TSV-Merge [I] also establishes that task-specific information and interference are highly non-uniform across parameters/layers. Moreover, MIN-Merging [F] appears particularly close in combining importance-based filtering, input-dependent routing, and differentiated layer-wise merging. The paper should therefore clarify more precisely what is novel beyond the specific LIER importance criterion and primary-core/secondary-auxiliary integration rule.
  • Important task-unknown dynamic merging baselines are missing
    • While the paper notes handling unknown task identity as a central motivation of MWM, the paper lacks discussions and experimental comparisons against DaWin [C], SiM [D], and ReTeX [E]. DaWin performs training-free per-sample dynamic interpolation based on model uncertainty, SiM performs training-free task routing using low-rank task manifolds, and ReTeX combines router-free task identification with recovery of task-specific experts from a single merged checkpoint. These methods address closely related task-unknown deployment settings and provide important accuracy/efficiency reference points for evaluating SER and MWM.
  • The large SER gain may primarily reflect task identification rather than expert merging
    • SER is explicitly trained to predict the source expert identity. Because the main experiments mix examples from distinct known datasets, a substantial part of its advantage may come from learning a dataset/task classifier and selecting the corresponding expert. The ablation showing a large degradation without SER therefore does not isolate the benefit of combining information from multiple experts. Router accuracy, an oracle-task baseline, and a simple predicted top-1-expert baseline are necessary to determine how much of the gain actually comes from layer-adaptive merging rather than identifying the correct source expert.
  • The data and offline computation requirements are not sufficiently accounted for
    • LIER requires labeled source-task validation data and repeated evaluation of layer-ablated experts, while SER requires additional supervised router training. This differs substantially from data-free or training-free merging approaches such as DaWin [C], and from SiM [D], which requires only a small support set and no router training. Moreover, recent work on adaptive LoRA merging [G] finds that when task-specific data are available for adaptive merging, training a new LoRA on the same data can be a surprisingly strong baseline. Reporting the total LIER/SER preprocessing cost and including a data- and compute-matched new-adapter or multi-task adaptation baseline would better establish the practical benefit of MWM.
  • The evaluation does not fully establish generalization beyond recognizable source tasks
    • The MMLU experiment shows that MWM largely preserves pretrained-model performance, but this does not directly demonstrate that SER can meaningfully route inputs from unseen tasks. Since SER is trained as a closed-set classifier over source-expert identities, its behavior when none of the experts corresponds to the input distribution remains unclear. SiM [D] and ReTeX [E] explicitly examine task-unknown and unseen-task settings, making routing accuracy, confidence, and expert selection on genuinely unseen distributions important additional evaluations for MWM.

[A] Tang et al., Merging Multi-Task Models via Weight-Ensembling Mixture of Experts. ICML 2024.

[B] Ye et al., Dynamic Model Merging With Mixture of Weights. IEEE Transactions on Circuits and Systems for Video Technology 2025.

[C] Oh et al., DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation. ICLR 2025.

[D] Son et al., Training-Free Task Classification for Multi-Task Model Merging. ECCV 2026.

[E] Jung et al., Learning to Recover Task Experts from a Multi-Task Merged Model. ECCV 2026.

[F] Liang et al., MIN-Merging: Merge the Important Neurons for Model Merging. arXiv 2025.

[G] Liu et al., The Appeal and Reality of Recycling LoRAs with Adaptive Merging. ICML 2026.

[H] Wang et al., Localizing Task Information for Improved Model Merging and Compression. ICML 2024.

[I] Gargiulo et al., Task Singular Vectors: Reducing Task Interference in Model Merging. CVPR 2025.

Questions for authors
  • Could the authors clarify the methodological distinction between MWM and prior dynamic/layer-aware approaches such as WEMoE [A] and MoW-Merging [B]? In particular, what aspect of input-dependent layer-wise integration is unique to MWM?
  • Could the authors discuss and distinguish MWM from MIN-Merging [F]? Although [F] is a preprint rather than a peer-reviewed publication, its combination of importance-based expert refinement, routing, and differentiated layer-wise merging appears closely related to the proposed framework.
  • Could the authors compare against task-unknown dynamic methods such as DaWin [C], SiM [D], and ReTeX [E] under matched settings? These seem particularly important given that unknown task identity is a central motivation of the paper.
  • Could the authors report SER's expert/task classification accuracy and include oracle-expert and predicted top-1-expert baselines? This would clarify how much of the improvement comes from identifying the appropriate expert versus actually integrating complementary information from multiple experts.
  • What is the total offline cost of LIER, including all layer-ablation evaluations, and how much data are used to train SER? Given the findings of [G], could the authors additionally compare against training a new LoRA or multi-task adapter using the same data and a comparable compute budget?
  • How does SER behave on inputs belonging to none of the source tasks? Reporting routing confidence and selected experts on unseen distributions would make the out-of-domain generalization claims more convincing.
6 comments

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, with less than two days remaining, we would like to ensure that we have adequately addressed all your concerns. If you have any additional questions or feedback, please feel free to let us know. Your insights are invaluable to us, and we would be happy to address any remaining concerns to further improve our work.

Thank you for your time and effort in reviewing our paper.

I would like to thank the authors for the rebuttal. But, several concerns remain. For one, how are the compared methods actually implemented? The performance is substantially different from the reported, such as DaWin and ReTeX.

AuthorsOpenReview ↗

Implementation Details and the Rationale for Adaptation

Thank you for your question. To ensure a consistent comparison, we adapted the baseline implementations to our experimental setting. These adaptations allow us to evaluate the methods on the same pretrained backbone, source experts, and evaluation protocol as MWM. The published configurations use different backbones, expert checkpoints, task suites, and prediction interfaces. Retaining those configurations would compare different underlying systems rather than isolate the performance of merging methods on our expert pools. In particular, their feature-extraction and parameter-recovery interfaces require adaptation to our backbones and LoRA checkpoints. We therefore implement the baselines within our experimental setting, retaining their key methodological components while adapting them to our backbones and LoRA checkpoints. Below, we specify these adaptations and other implementation choices, including changes to feature extraction, parameter recovery, and prediction interfaces.

All baseline implementation code has been uploaded with the supplementary materials for inspection and reproduction. We will also incorporate the implementation details and adaptation rationale described below into the appendix of the revised manuscript. Upon acceptance, we will publicly release all code.

The implementations are as follows:

MoW-WA. We retain the structural principle of selecting blocks according to expert-weight similarity and applying input-dependent merging to the selected blocks. Specifically, we compute the mean pairwise cosine similarity of the full expert weights, including their LoRA updates, and dynamically merge the five least-similar blocks. Other blocks use uniform weight averaging. We use the same SER as MWM to control for the routing component, allowing this comparison to examine the effect of the layer-selection and integration strategy under a common router.

SiM. We preserve its training-free task identification through SVD projection residuals. Each task uses 32 source-training support examples to construct a mean-centered subspace from normalized features. At inference, the task with the smallest projection residual is selected, and its update is recovered through EMR-style masks and rescaling. The necessary backbone adaptation is the feature source: we use the final prompt-token representation for Qwen and the CLS representation for ViT. EMR operates on the actual LoRA update ΔW=BA\Delta W=BA, which is the parameter change being merged. Averaging or masking the two factors independently would not implement the same operation on model weights.

ReTeX. We retain the combination of task identification and learned expert-offset recovery. Starting from the mean-merged updates, we learn task-conditioned offsets that reconstruct each source expert. Recovery is restricted to the matrices modified by LoRA, because the remaining backbone parameters are shared across the experts. We use rank-8 offsets with an eight-dimensional task embedding, optimized for 1,000 steps using parameter-reconstruction loss; checkpoint selection uses reconstruction error. Task identification uses SVD subspaces built from 32 support examples per task using mean-merged-model features.

DaWin. We retain entropy-based, input-dependent interpolation, using update coefficients 0.3N softmax(−H)0.3N\,\mathrm{softmax}(-H), where NN is the number of experts. The output distribution must match the backbone’s prediction interface: for NLP, entropy is computed over candidate-label probabilities derived from sequence scores; for CV, it is computed using the same supplied task-specific classification head for every expert. This requires no supervised router training. We evaluate per-input interpolation directly, without the mixture-model approximation, and account for the additional expert forward passes when measuring inference cost.

These adaptations make the comparisons executable on the same expert pools.

We welcome any further questions and would be happy to provide additional clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.