acceptodds
Under review as a conference paper at ICLR 2027

Learning Not to Interfere: Adapting Experts for Model Merging

Abstract

Model merging combines experts fine-tuned from a shared pretrained initialization into a single multitask model, but cross-task interference often degrades their specialized capabilities. We argue that this interference stems from how experts are fine-tuned. Standard fine-tuning optimizes each expert on its own task distribution without constraining how its parameter update affects predictions on other tasks. These parameter updates, known as task-vectors, represent changes from the shared initialization and can disrupt other experts when combined. We propose Resolving Interference (RI), a lightweight framework that independently adapts experts before merging using unlabeled task data. RI distills the original expert's predictions on its own task to preserve specialization, while matching the shared pretrained model's predictions on other tasks. The pretrained model provides a reference for behavior before specialization, allowing RI to restrict each task vector's influence beyond its own task distribution. RI complements existing merging methods, including those using test-time adaptation, and improves performance across vision and language benchmarks. For language-model merging, RI surpasses the strongest evaluated baselines by up to 2.9%. Additionally, quantitative consistency evaluations and qualitative generations confirm RI's ability to better preserve the original experts' behavior across tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Model Merging, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UuRoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Resolving Interference (RI), a pre-merging adaptation framework that uses unlabeled task data to make independently fine-tuned experts more compatible. RI preserves each expert's predictions on its own task while encouraging it to match the shared pretrained model on other tasks. Experiments on vision and language benchmarks show that RI improves multiple existing merging methods and helps preserve the behavior of individual experts after merging.

Critical strengths
  • The paper provides a clear motivation for addressing cross-task interference at the expert-adaptation stage rather than only correcting parameter conflicts during or after merging.

  • RI is compatible with multiple merging methods and introduces no additional inference-time modules. Experiments across vision and language tasks, together with behavioral-consistency and data-efficiency analyses, provide encouraging evidence for its effectiveness.

Critical weaknesses
  • Unclear computational cost-effectiveness for LLMs. Although RI is described as lightweight, the paper does not report actual adaptation time or peak GPU memory for Llama-3.1-8B and Gemma-2-9B. Since RI requires additional optimization for each expert, including distillation over prompts and generated completions, its total computational overhead may be substantial. Given the relatively modest gains over strong merging baselines, particularly on Gemma, it remains unclear whether these improvements justify the additional cost compared with standard merging methods that require no extra adaptation. A direct comparison of end-to-end compute cost and performance would better establish RI's practical advantage.

  • Limited comparison with recent training-based merging methods. Although the paper compares RI with several optimization-based merging methods, recent approaches such as Expert Merging [1] and ProDistill [2] also leverage data and additional optimization to improve model merging. The paper does not establish whether RI offers consistent advantages over these approaches under comparable data-access and computational budgets. Comparisons with recent methods would better clarify the specific contribution and practical benefits of pre-merging expert adaptation.

[1] Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking. ICLR 2026.

[2] Scalable Model Merging with Progressive Layer-wise Distillation. ICML 2025.

Questions for authors
  • Can the authors report the actual adaptation time, peak GPU memory, and total GPU-hours for RI on Llama and Gemma, including completion generation and caching? Given that standard merging methods do not require this additional adaptation stage, how do the performance gains justify the extra computational cost, and what is the end-to-end cost-effectiveness of RI compared with these methods?

  • Given that RI uses unlabeled task-specific inputs, can the authors compare it with multi-expert distillation and other data-driven merging approaches under matched data-access and computational budgets? Does RI provide consistent gains over these alternatives when using the same available inputs?

4 comments
AuthorsOpenReview ↗

Response to Reviewer-3 UuRo Part-1

We thank the reviewer for recognizing the clear motivation for addressing cross-task interference through expert adaptation, RI’s compatibility with multiple merging methods. We also appreciate their positive assessment of our experiments across vision and language tasks, including our behavioral-consistency and data-efficiency analyses. Below, we address each weakness and question, providing additional analyses and clarification.

R3W1 & R3Q1: Computational cost and cost-effectiveness for language models.

We thank the reviewer for requesting this comparison. We report the cost of generating expert completions, performing RI adaptation for 500 steps per expert, and downstream merging.

All paired values below denote Llama-3.1-8B-Instruct / Gemma-2-9B-Instruct.

MethodGPU-hours per expertWall-clock time per expert (h)Peak GPU memory (GB)
RI adaptation + expert generations1.6 / 2.81.6 / 2.8127 / 125

For five experts, the RI stage requires 8.2 GPU-hours for Llama and 13.9 GPU-hours for Gemma experts. Since each expert can be adapted independently, we can adapt them in parallel, reducing elapsed time.

The end-to-end cost is the five-expert RI cost plus the downstream merging cost, as detailed in the table below. For static merging methods—Task Arithmetic (TA), TIES, and TSV-M—we include both parameter merging and tuning of the merging coefficient, with tuning accounting for most of their computational cost.

Downstream merging cost and end-to-end comparison

Merging methodMerging GPU-hoursMerging wall-clock time (h)Merging peak GPU memory (GB)Total GPU-hours with RIPerformance gain from RI (Normalized Accuracy)
TA (20-α grid)5.1 / 5.65.1 / 5.6125 / 13113.3 / 19.5+2.8 / +4.2
TIES (10-α grid)2.7 / 2.92.7 / 2.9125 / 13110.9 / 16.8+4.8 / +4.9
TSV-M (10-α grid)4.9 / 5.14.9 / 5.1125 / 13113.1 / 19.0+2.0 / +1.6
AdaMerging1.3 / 2.00.6 / 1.0149 / 1809.5 / 15.9+17.3 / +9.5
AdaRank1.4 / 2.10.7 / 0.5240 / 2909.6 / 16.0+10.2 / +6.7

α denotes the downstream merging scaling coefficient; performance is the mean expert-normalized score, with the individual-expert reference set to 100.

While RI does add a one-time adaptation cost, it can be justified based of the following:

[1] Significant performance improvement: Experts merged with RI do not just improve but outperform the individual experts in terms of average performance. The largest gains occur with AdaMerging (17.3%/9.5%) and AdaRank (10.2%/6.7%). The lower gains with Gemma, as pointed out by the reviewer, is simply a result of an already saturated baseline.

[2] Improved consistency: As seen in Figures 4, 5, and 10 (in the main paper), the merged models are not just more accurate but are also more consistent - by 5.6% with the Llama experts, this is an important and often overlooked aspect concerning merging.

[3] Recycling Experts: The additional compute cost is a one-time cost for adapting all five experts; these adapted checkpoints can subsequently be reused across merging methods without repeating RI.

We thank the reviewer for requesting this end-to-end comparison and will include the measurements and associated compute–performance trade-off in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer-3 UuRo Part-2

R3W2 & R3Q2: Comparison with data-adaptation baselines under matched compute and data.

We thank the reviewer for suggesting these comparisons. We additionally evaluate Expert Merging, ProDistill, and Multi-Expert Distillation on the five-task Llama-3.1-8B benchmark, using RI + AdaMerging as our reference configuration. All methods receive the same full unlabeled test inputs, without ground-truth labels. We use a batch size of 32 and increase the baselines’ optimization steps to obtain comparable computational budgets of 9.0–9.5 GPU-hours on NVIDIA H200 GPUs.

Merging MethodIfevalMBPP+GSM8KXSTestArc_MLAverage AccuracyAverage Normalized AccuracyGPU Hours on Nvidia H200
Original Experts60.858.782.184.646.566.5100-
Prodistill41.062.482.990.343.464.095.09.4
Expert Merging42.261.481.189.744.863.895.09.1
Multi-Expert Distillation61.058.084.784.647.067.1100.79.0
RI+Adamerging60.161.483.493.745.968.9102.99.5

As seen from the table above, RI + AdaMerging achieves the highest average accuracy and expert-normalized accuracy, exceeding ProDistill, Expert Merging, and multi-expert distillation by 7.9%, 7.9%, and 2.2% in average normalized accuracy, respectively. These results demonstrate that the aggregate advantage of pre-merge adaptation persists under matched input access and comparable computational budgets. We will include these comparisons and their implementation details in the revised paper.

We thank the reviewer for prompting these additional analyses. The explicit end-to-end compute measurements and comparisons with recent data-driven methods directly address the concerns about computational cost and comparative performance. We hope this new evidence resolves your remaining reservations and provides a basis for upgrading your assessment of our work. We welcome any further questions.

Thank you for the authors' detailed response and clarification. My concerns have been sufficiently addressed, and I will raise my score to 3.

Thank you for following up and for raising your score. We appreciate your constructive feedback, which motivated the additional compute measurements and comparisons with data-driven merging methods, and are glad that our responses sufficiently addressed your concerns. Thank you again for your time and support.

Reviewer_a6quOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Resolving Interference (RI), a method that adapts individual experts before merging to reduce cross-task interference. RI uses two distillation losses to preserve each expert's behavior on its own task while limiting its influence on other tasks. Experiments on vision and language models show that RI consistently improves the performance of several existing merging methods.

Critical strengths

1.The paper is well motivated. The analysis in Section 3 and Figure 2 clearly shows how task vectors can interfere with other experts, providing good support for the proposed method.

2.The method is simple and easy to apply. It works with different merging methods and does not introduce additional inference-time overhead.

3.The experiments are fairly comprehensive, covering different model architectures, numbers of tasks, and merging methods. The improvements are consistent across most settings.

Critical weaknesses

1.The paper lacks ablation studies on the two loss terms. It would be useful to see how each objective contributes to the final performance. A sensitivity analysis of λ is also missing.

2.RI assumes that experts should have minimal influence on other task distributions. However, tasks may share useful knowledge, and suppressing cross-task effects could potentially hurt performance. The paper does not sufficiently explore this case.

3.The computational cost is not fully evaluated, especially for language models. RI requires additional training for each expert, but it is unclear whether the performance gains justify this overhead. Results with multiple random seeds would also make the conclusions more convincing.

Questions for authors

please see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer-2 a6qu Part-1

We thank the reviewer for recognizing the clear motivation and supporting analysis behind RI, as well as its simplicity and compatibility with different merging methods. We also appreciate their acknowledgment of our comprehensive experiments and consistent improvements across architectures, task counts, and merging methods. Below, we address each weakness and question, providing additional analyses and clarification.

R2W1.1: Ablation study of RI loss terms

We thank the reviewer for suggesting this analysis. As recommended, we separately evaluate task-preservation, interference-reduction loss, and their combination on ViT-B/32 in the eight-task setting. The table reports mean classification accuracy (%) across five merging baselines.

Adaptation objectiveTSV-MIso-CTSAdaMergingAdaRankSurgery
Task Preservation only85.786.781.187.785.0
Interference Reduction only87.886.186.589.886.9
Both objectives (RI)88.987.686.889.887.9

From the table above, we first see that task-preservation alone matches the unadapted merging baselines: adaptation starts from the original expert, which already satisfies the preservation objective. Secondly, interference-reduction loss alone provides the core adaptation signal and improves four merging methods by 1.9%–5.4%, the only exception is Iso-CTS, whose accuracy drops by 0.6%. This is caused by forgetting the original task during adaptation, which is not recovered by the merging logic. Finally, combining both objectives forms our method RI, which improves over interference reduction alone by 0.3%–1.5%. These results support the complementary roles of the losses: interference-reduction encourages compatibility, while task-preservation constrains changes to specialized behavior. We will add this ablation to the main paper.

R2W1.2: Sensitivity analysis of λ

We found RI to be stable across different values of λ and therefore fixed λ=1 for all paper experiments. At Reviewer 2’s request, we provide an explicit sensitivity analysis on ViT-B/32 in the eight-task setting across five merging methods. Across the tested λ values, merged-model accuracy has a standard deviation of 0.2–0.3 percentage points. Furthermore, performance with λ=1 is within 0.55 percentage points of every method’s best observed accuracy, supporting its use as a practical default without extensive tuning.

λTSVMIso-CTSAdaMergingAdaRankSurgery
1/888.888.186.589.487.0
1/489.087.986.689.687.4
1/289.187.886.589.687.8
189.287.686.889.887.7
3/289.487.586.889.887.9
289.387.487.089.887.7
489.487.487.289.988.1
889.087.187.390.087.6
Mean ± std89.2 ± 0.287.6 ± 0.386.8 ± 0.389.7 ± 0.287.6 ± 0.3

R2W2: Could RI suppress useful knowledge shared between tasks?

We agree that related tasks can share useful knowledge. We discuss this briefly in L268-269 of the main paper but expand on it here. RI balances two soft objectives: reducing prediction changes on other task distributions while preserving the expert’s own-task predictions. The preservation objective therefore discourages removing capabilities shared with the original task, although it does not explicitly protect every beneficial cross-task interaction.

For example, we observe this in the ViT-B/32 eight-task vision setting, where two tasks, namely MNIST and SVHN, are both digit-classification tasks but finetuned on different data distributions. Naturally, one can expect shared knowledge between the two. The heatmap in Figure 2 of the main paper shows cross-task influence measured via the Total Variation (TV) distance measure, which goes down close to zero across most task-pairs after RI adaptation, with the SVHN-MNIST pair being the exception. We share a short snapshot of the heatmap in Figure 2 of the main paper for reference:

ExpertEvaluation dataTV before RITV after RI
SVHNMNIST0.740.69
MNISTSVHN0.720.39

From the table above, we see substantial cross-task influence exists between MNIST and SVHN, and a large part of this cross-task shared/useful influence is retained post RI adaptation as well. This analysis shows that RI preserves shared knowledge between related tasks rather than enforcing complete separation. Additionally, in the language model setting, we consistently observe RI-adapted merged Llama3-8B and Gemma2-9B models outperform their own experts, further demonstrating the retention and positive knowledge transfer. We will add this extended discussion to the main paper.

AuthorsOpenReview ↗

Response to Reviewer-2 a6qu Part-2

W3.1: Computational cost for language models.

We appreciate the request for explicit language-model computation statistics. In the table below, we share the computational cost for generating expert completions and RI adaptation for 500 steps on an NVIDIA H200 GPU for Llama and Gemma expert models.

Model familyGPU-hours per expertPeak GPU memory
Llama-3.1-8B-Instruct1.64127 GB
Gemma-2-9B-Instruct2.77125 GB

For five experts, the adaptation stage requires 8.2 and 13.9 GPU-hours across Llama3.1-8B and Gemma2-9B experts, respectively. An advantage of RI is that each expert can be adapted independently in parallel.

While RI adaptation does add an additional step, its computation cost is justified by the significant performance improvement. For example, in the language model setting, RI consistently offers merged models that don’t just improve but outperform the individual experts themselves. Notably, RI improves average Llama/Gemma benchmark scores by 10.5%/5.8% with AdaMerging and 5.9%/4.1% with AdaRank, making the one-time compute cost justifiable. We will report the extended computation profile across language models in the main paper.

W3.2: Results across random seeds.

We thank the reviewer for their suggestion to test RI’s robustness across random seeds. We evaluate across 5 different seeds and share revised performance with the corresponding mean and standard deviation for the Llama 3.1 8B benchmark.

MethodIFEvalMBPP+GSM8KXSTestARC-mlAvgNorm
Individual60.858.782.184.646.566.5100
TA48.859.181.190.945.865.197.1
TIES44.461.384.689.743.464.796
TSV-M4961.984.29045.16698.4
AdaMerging18.157.78490.342.158.485.5
AdaRank41.355.982.993.739.862.792.1
TA + RI56.8 ± 1.361.9 ± 0.381.9 ± 1.489.9 ± 1.245.1 ± 0.367.1 ± 0.5100.4 ± 0.7
TIES + RI54.4 ± 2.661.3 ± 1.084.0 ± 1.291.2 ± 1.245.0 ± 0.367.2 ± 1.0100.1 ± 1.5
TSV-M + RI57.2 ± 1.361.0 ± 0.584.0 ± 0.791.5 ± 0.545.7 ± 0.367.9 ± 0.3101.3 ± 0.5
AdaMerging + RI59.6 ± 1.261.6 ± 0.583.4 ± 0.392.9 ± 0.746.0 ± 0.268.7 ± 0.4102.7 ± 0.6
AdaRank + RI60.6 ± 0.861.1 ± 0.483.6 ± 0.791.9 ± 0.543.0 ± 0.268.1 ± 0.2101.4 ± 0.2

After evaluating across random seeds, we continue to observe consistent improvement in performance across all merging baselines, so the trend in the main paper continues to hold reliably across seeds. We will run a similar evaluation with the Gemma models and add both to the main paper.

Finally, we appreciate your positive assessment and hope the additional evidence addresses your remaining concerns and strengthens your support for acceptance. We welcome any further questions.

AuthorsOpenReview ↗

Final Remarks

Thank you again for your thoughtful review and helpful suggestions. In response, we have provided ablations of both loss terms, a λ sensitivity analysis, an expanded discussion of shared knowledge across tasks, explicit language-model compute measurements, and results across five random seeds. We hope this additional evidence strengthens your confidence in RI’s effectiveness, robustness, and practical value.

If you find your concerns sufficiently addressed, we would be grateful if you would consider upgrading your recommendation. We greatly appreciate your time and consideration.

Reviewer_iMchOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses cross‑task interference in model merging: fine‑tuned expert task vectors alter predictions for unrelated tasks and degrade merged‑model performance. The authors diagnose that standard fine‑tuning does not constrain task‑vector influence outside its target task. They propose Resolving Interference (RI) method, which is a lightweight pre‑merging expert‑adaptation framework using unlabeled data. RI jointly optimizes two distillation losses: preserving original‑expert predictions on its own task, and matching pre‑trained‑model outputs for other tasks to localize task‑vector functional impact. RI works compatibly with nearly all existing merging algorithms without adding inference modules. Evaluations cover vision and language benchmarks across diverse merging baselines. RI consistently boosts merged performance for LLMs, and improves consistency retaining original expert correct predictions. Ablations study data efficiency, distillation targets, metrics, and computational overhead.

Critical strengths

The work offers a compelling perspective by treating cross-task interference as a property of the individual expert updates rather than exclusively as a problem of parameter aggregation. The proposed dual-distillation objective is conceptually simple and directly connected to this motivation. Importantly, RI can be integrated with different merging operators without modifying their underlying algorithms or adding inference-time modules.

Critical weaknesses
  1. The trade‑off weight λ balances task preservation and interference reduction. The paper uses a fixed λ=1.0 for all experiments without a principled selection procedure. λ mis‑setting risks catastrophic trade‑offs: either failing to suppress cross‑task interference or erasing expert specialized capability. Without a validation‑based tuning protocol for λ, real‑world practical deployment is less straightforward.
  2. While overall consistency improves, several method‑task pairs show consistency drops after RI adaptation. The paper observes these regressions but provides limited diagnosis on why certain tasks/merging operators suffer degradation. It is unclear what properties of tasks or expert updates trigger these residual failures, limiting deeper understanding and reliable real‑world adoption.
Questions for authors
  1. You fix λ=1.0 across all experiments. Could you elaborate why this fixed value generalizes well across vision/language, different model sizes, and varying task numbers? Is there a feasible validation‑driven λ‑selection pipeline for practical usage?
  2. In Appendix C.4, auxiliary non‑task data yields gains but underperforms in‑domain unlabeled task data. What minimal quality requirements should auxiliary datasets satisfy to avoid harming RI adaptation?
2 comments
AuthorsOpenReview ↗

Response to Reviewer-1 (R1) iMch

We’re glad the reviewer found our work on treating cross-task interference as a property of the individual expert to be compelling, recognizing the method as conceptually simple and compatible across a wide range of merging methods, all without adding any inference-time modules. Below, we address each weakness and question, providing additional analyses and clarification.

R1W1 and R1Q1: λ=1 & Sensitivity to λ. We found RI to be stable across different values of λ and therefore fixed λ=1 for all paper experiments. At Reviewer 1’s request, we provide an explicit sensitivity analysis on ViT-B/32 in the eight-task setting across five merging methods. Across the tested λ values, merged-model accuracy has a standard deviation of 0.2%–0.3%. Furthermore, performance with λ=1 is within 0.55% of every method’s best observed accuracy, supporting its use as a practical default without extensive tuning.

λTSVMIso-CTSAdaMergingAdaRankSurgery
1/888.888.186.589.487.0
1/489.087.986.689.687.4
1/289.187.886.589.687.8
189.287.686.889.887.7
3/289.487.586.889.887.9
289.387.487.089.887.7
489.487.487.289.988.1
889.087.187.390.087.6
Mean ± std89.2 ± 0.287.6 ± 0.386.8 ± 0.389.7 ± 0.287.6 ± 0.3

R1W2: Residual consistency drops. Thanks for your question. As the reviewer points out, RI significantly improves overall consistency by 5.6% on average across all pairs of tasks and merging methods. While consistency helps assess retention of expert successes, it is a complementary diagnostic, our primary objective is strong merged-model performance.

Often, when merging reduces performance compared to individual experts, consistency can be a useful proxy objective to study to examine whether the merged model has similar behavior to the original expert. However, in our case, the few situations 4/25 where our merged model reduces consistency are scenarios where our merged model increases performance compared to the corresponding individual expert, so consistency is less critical in this case. For example, TSV-M + RI exceeds the original expert’s performance on XSTest (89.4% versus 84.6%) and MBPP+ (60.4% versus 58.6%), despite the observed minor regression in consistency relative to merging without RI. We will clarify this distinction and discuss the residual regressions in the main paper.

R1Q2: What quality should auxiliary data satisfy? Thank you for prompting this useful analysis. All six tested auxiliary sources, including Gaussian noise, improve performance over merging without RI, although the gains depend on the source. Our additional analysis shows that lower embedding-space discrepancy of the auxiliary data source with respect to task data indicates closer distributions. This is further associated with higher merged accuracy, as seen in the table below. We use Squared Maximum Mean Discrepancy (MMD²) to measure discrepancy against reference embeddings from task data, where the embeddings are generated from the shared pretrained model. We use TSVM as the reference merging method.

Data sourceMean MMD² ↓Mean accuracy (%) ↑
Gaussian noise1.7685.8
Leaves1.3286.0
Shaders21k0.8286.5
ImageNet0.5687.0
MS COCO0.5687.2
OpenImages0.5487.2
Task-data reference0.0088.7

While we do not establish a universal threshold guaranteeing no degradation. These results suggest auxiliary data may be used in the absence of even unlabeled task data, where auxiliary data closer to the task distribution yields larger gains. A practical selection criterion is therefore low embedding MMD² against held-out unlabeled task-validation samples, when available. We will add this analysis and the resulting guidance to the paper.

Finally, we appreciate your positive assessment and hope the additional evidence addresses your remaining concerns and strengthens your support for acceptance. We welcome any further questions.

Thank you again for your thoughtful review and constructive suggestions. As the discussion period draws to a close, we would like to highlight our additional λ sensitivity experiments, clarification of the residual consistency drops, and auxiliary-data analysis providing practical guidance for data selection. We hope these additions address your concerns and strengthen the paper’s empirical support and practical usefulness.

We greatly appreciate your time and consideration. If you find your concerns sufficiently addressed, we would be grateful if you would consider upgrading your recommendation.

Comments on the paper

1 comment
Area_Chair_nVr5OpenReview ↗

Author- Reviewer Discussion

Dear Reviewers, the authors have submitted their response. Kindly review it and update your review where appropriate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.