- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Resolving Interference (RI), a pre-merging adaptation framework that uses unlabeled task data to make independently fine-tuned experts more compatible. RI preserves each expert's predictions on its own task while encouraging it to match the shared pretrained model on other tasks. Experiments on vision and language benchmarks show that RI improves multiple existing merging methods and helps preserve the behavior of individual experts after merging.
Critical strengths
-
The paper provides a clear motivation for addressing cross-task interference at the expert-adaptation stage rather than only correcting parameter conflicts during or after merging.
-
RI is compatible with multiple merging methods and introduces no additional inference-time modules. Experiments across vision and language tasks, together with behavioral-consistency and data-efficiency analyses, provide encouraging evidence for its effectiveness.
Critical weaknesses
-
Unclear computational cost-effectiveness for LLMs. Although RI is described as lightweight, the paper does not report actual adaptation time or peak GPU memory for Llama-3.1-8B and Gemma-2-9B. Since RI requires additional optimization for each expert, including distillation over prompts and generated completions, its total computational overhead may be substantial. Given the relatively modest gains over strong merging baselines, particularly on Gemma, it remains unclear whether these improvements justify the additional cost compared with standard merging methods that require no extra adaptation. A direct comparison of end-to-end compute cost and performance would better establish RI's practical advantage.
-
Limited comparison with recent training-based merging methods. Although the paper compares RI with several optimization-based merging methods, recent approaches such as Expert Merging [1] and ProDistill [2] also leverage data and additional optimization to improve model merging. The paper does not establish whether RI offers consistent advantages over these approaches under comparable data-access and computational budgets. Comparisons with recent methods would better clarify the specific contribution and practical benefits of pre-merging expert adaptation.
[1] Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking. ICLR 2026.
[2] Scalable Model Merging with Progressive Layer-wise Distillation. ICML 2025.
Questions for authors
-
Can the authors report the actual adaptation time, peak GPU memory, and total GPU-hours for RI on Llama and Gemma, including completion generation and caching? Given that standard merging methods do not require this additional adaptation stage, how do the performance gains justify the extra computational cost, and what is the end-to-end cost-effectiveness of RI compared with these methods?
-
Given that RI uses unlabeled task-specific inputs, can the authors compare it with multi-expert distillation and other data-driven merging approaches under matched data-access and computational budgets? Does RI provide consistent gains over these alternatives when using the same available inputs?
4 comments
Response to Reviewer-3 UuRo Part-2
R3W2 & R3Q2: Comparison with data-adaptation baselines under matched compute and data.
We thank the reviewer for suggesting these comparisons. We additionally evaluate Expert Merging, ProDistill, and Multi-Expert Distillation on the five-task Llama-3.1-8B benchmark, using RI + AdaMerging as our reference configuration. All methods receive the same full unlabeled test inputs, without ground-truth labels. We use a batch size of 32 and increase the baselines’ optimization steps to obtain comparable computational budgets of 9.0–9.5 GPU-hours on NVIDIA H200 GPUs.
| Merging Method | Ifeval | MBPP+ | GSM8K | XSTest | Arc_ML | Average Accuracy | Average Normalized Accuracy | GPU Hours on Nvidia H200 |
|---|---|---|---|---|---|---|---|---|
| Original Experts | 60.8 | 58.7 | 82.1 | 84.6 | 46.5 | 66.5 | 100 | - |
| Prodistill | 41.0 | 62.4 | 82.9 | 90.3 | 43.4 | 64.0 | 95.0 | 9.4 |
| Expert Merging | 42.2 | 61.4 | 81.1 | 89.7 | 44.8 | 63.8 | 95.0 | 9.1 |
| Multi-Expert Distillation | 61.0 | 58.0 | 84.7 | 84.6 | 47.0 | 67.1 | 100.7 | 9.0 |
| RI+Adamerging | 60.1 | 61.4 | 83.4 | 93.7 | 45.9 | 68.9 | 102.9 | 9.5 |
As seen from the table above, RI + AdaMerging achieves the highest average accuracy and expert-normalized accuracy, exceeding ProDistill, Expert Merging, and multi-expert distillation by 7.9%, 7.9%, and 2.2% in average normalized accuracy, respectively. These results demonstrate that the aggregate advantage of pre-merge adaptation persists under matched input access and comparable computational budgets. We will include these comparisons and their implementation details in the revised paper.
We thank the reviewer for prompting these additional analyses. The explicit end-to-end compute measurements and comparisons with recent data-driven methods directly address the concerns about computational cost and comparative performance. We hope this new evidence resolves your remaining reservations and provides a basis for upgrading your assessment of our work. We welcome any further questions.
Thank you for the authors' detailed response and clarification. My concerns have been sufficiently addressed, and I will raise my score to 3.
Thank you for following up and for raising your score. We appreciate your constructive feedback, which motivated the additional compute measurements and comparisons with data-driven merging methods, and are glad that our responses sufficiently addressed your concerns. Thank you again for your time and support.
Response to Reviewer-3 UuRo Part-1
We thank the reviewer for recognizing the clear motivation for addressing cross-task interference through expert adaptation, RI’s compatibility with multiple merging methods. We also appreciate their positive assessment of our experiments across vision and language tasks, including our behavioral-consistency and data-efficiency analyses. Below, we address each weakness and question, providing additional analyses and clarification.
R3W1 & R3Q1: Computational cost and cost-effectiveness for language models.
We thank the reviewer for requesting this comparison. We report the cost of generating expert completions, performing RI adaptation for 500 steps per expert, and downstream merging.
All paired values below denote Llama-3.1-8B-Instruct / Gemma-2-9B-Instruct.
For five experts, the RI stage requires 8.2 GPU-hours for Llama and 13.9 GPU-hours for Gemma experts. Since each expert can be adapted independently, we can adapt them in parallel, reducing elapsed time.
The end-to-end cost is the five-expert RI cost plus the downstream merging cost, as detailed in the table below. For static merging methods—Task Arithmetic (TA), TIES, and TSV-M—we include both parameter merging and tuning of the merging coefficient, with tuning accounting for most of their computational cost.
Downstream merging cost and end-to-end comparison
α denotes the downstream merging scaling coefficient; performance is the mean expert-normalized score, with the individual-expert reference set to 100.
While RI does add a one-time adaptation cost, it can be justified based of the following:
[1] Significant performance improvement: Experts merged with RI do not just improve but outperform the individual experts in terms of average performance. The largest gains occur with AdaMerging (17.3%/9.5%) and AdaRank (10.2%/6.7%). The lower gains with Gemma, as pointed out by the reviewer, is simply a result of an already saturated baseline.
[2] Improved consistency: As seen in Figures 4, 5, and 10 (in the main paper), the merged models are not just more accurate but are also more consistent - by 5.6% with the Llama experts, this is an important and often overlooked aspect concerning merging.
[3] Recycling Experts: The additional compute cost is a one-time cost for adapting all five experts; these adapted checkpoints can subsequently be reused across merging methods without repeating RI.
We thank the reviewer for requesting this end-to-end comparison and will include the measurements and associated compute–performance trade-off in the revised paper.