acceptodds
Withdrawn from ICLR 2027

Higher-order Inter-task Interference Modeling for Flexible Model Merging

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model merging combines multiple task-specific fine-tuned models into a single multi-task model without additional training. However, it faces a fundamental dilemma: merging all N models into one degrades accuracy due to inter-task interference, whereas maintaining N independent experts alongside a router incurs an N-fold increase in storage costs. We argue that this dilemma can be resolved by utilizing K < N experts, where tasks with minimal mutual interference share the same expert, and K serves as a user-defined storage budget. This flexible strategy adapts seamlessly to various application scenarios with diverse memory constraints. The core of this approach lies in estimating inter-task compatibility to group tasks with low mutual interference. Our empirical study reveals that inter-task interference does not accumulate consistently: two tasks that exhibit only mild interference in pairwise merges may suffer severe degradation when a third is introduced, whereas three tasks with strong pairwise conflicts can exhibit a stabilizing effect. To address this, we introduce Interference-Aware Flexible Merging (IAFM), a framework that constructs a task interference graph whose edge weights aggregate information from both pairwise and triple task merging experience, thereby capturing higher-order interactions overlooked by purely pairwise analysis. We evaluate IAFM on 8,14, and 20 vision tasks using OpenCLIP ViT-B/32, and 6 NLP tasks using Llama3-8B models, benchmarking it against the state-of-the-art model merging methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Model Merging, Task Interference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iyGeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the conflict phenomenon existing in model merging and takes a balanced solution between performance and storage. N models are merged into K groups instead of the traditional merging into 1 group or MoE merging into N groups. This indeed solves an important problem in model merging, the motivation is good.

Critical strengths

Motivation has practical value, and the solution of grouping is interesting. It is relatively novel to do clustering by combining two-task and three-task methods.

Critical weaknesses
  1. The presentation of the paper is a disaster compared to the paper itself. The main text lacks a visualization diagram that can fully explain the method, and the quality of the diagrams in the appendix is relatively low.
  2. There are some papers about multitask learning are close to this paper's core contribution [1], it will be better if author can clarify the contributions of this paper. For example, I think the two-task and three-task settings are a good contribution, but this needs to be further clarified.

[1] Li, D., Ju, H., Sharma, A., & Zhang, H. R. (2023, August). Boosting multitask learning on graphs through higher-order task affinities. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 1213-1222).

Questions for authors
  1. See the Critical Weaknesses 1 & 2.
  2. Why inter-task interference is defined Eq. (7), rather than as a performance drop relative to the single-task expert?
Reviewer_q2CPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper tackles the classic tension in model merging—full merging tanks accuracy, independent experts cost too much storage—by proposing K shared experts as a middle ground. The nice insight is that task interference doesn't simply add up: two tasks that look fine together can suddenly collapse once a third joins in, which is a neat finding. That said, the method needs to exhaustively check triple combinations to build its interference graph, and that cost scales cubically, so it feels like it'd get rough fast once the task count grows.

Critical strengths
  1. A convincing finding: pairwise task interference doesn't simply add up—rankings can reverse once a third task is added. It's backed by both large-scale statistics (336 triples) and concrete counterexamples, so it's fairly solid.

  2. First to use triple-merging experiments to correct the task-compatibility graph before grouping—a novel idea, and framing K as a tunable storage budget is practically useful.

Critical weaknesses
  1. Triple-merging scales as O(N³), and while 20 tasks takes 1-2 GPU-hours, this grows cubically—likely a bottleneck at hundreds of tasks. The paper doesn't discuss any approximation/sampling scheme for large N.

  2. Evaluation stops at triples; quartets are dismissed as "not worth it" with no supporting data, just a one-line claim. More importantly, the paper never offers a mechanistic explanation for why triples outperform pairs—just a vague "more rescue/harm interactions become visible" without real analysis of the underlying cause.

Questions for authors
  1. Have the authors analyzed why triple-merging helps—e.g. via null-space/singular-vector overlap or gradient conflict analysis—to see if "rescue" vs "collapse" triples correspond to some identifiable geometric pattern (like knowledge conflict in shared subspaces)?
  2. What is the projected cost of building the interference graph at N~100-1000 tasks, and how does it compare to the near-linear preprocessing cost of standard merging baselines (TA, TIES)?
  3. Eq. (10) indexes ordered triples (s,x,k), but the merge of {τ_s,τ_x,τ_k} itself is symmetric in x and k. Does the implementation avoid recomputing the same merged model for permuted orderings, or is some of the reported cost redundant?
  4. How sensitive are the final groupings/accuracy to the choice of α and β in Eq. (11)? Only β=0 vs β=1 is ablated—was any other weighting tried, and does the method require re-tuning them per dataset/scale?
Reviewer_3ZLDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies model merging under a storage budget. IAFM combines pairwise and triple-task evaluations into an affinity matrix, then uses spectral clustering to group tasks into merged experts. Experiments report accuracy–storage trade-offs on vision and natural language inference benchmarks.

Critical strengths

Balancing task performance and storage costs in model merging is a practically relevant problem.

Critical weaknesses
  • Limited methodological novelty. The core pipeline is closely related to Boosting Multitask Learning on Graphs through Higher-Order Task Affinities (KDD 2023). That work already considers higher-order task affinities and task grouping under a deployment budget. The authors should establish what is new beyond adapting this approach to model merging.
  • Limited and dated evaluation. Experiments rely on ViT-B/32 and Llama-3-8B, with three nested vision task suites and six NLI datasets. This provides limited evidence of generalization across model architectures, scales, and task types.
  • Serious inconsistencies in the reported results. For eight tasks, Table 1 reports 86.53% expert accuracy, 90.89% IAFM accuracy at (K=4), and 97.47% normalized accuracy. These values cannot simultaneously satisfy the stated metric definitions. The TSV baseline is also substantially below its published result on the same eight-task benchmark and backbone (53.05% vs. 85.86%), despite the claimed use of the original setup and public checkpoints. These discrepancies require explanation, verification of the implementation, and corrected per-task results.
  • Poor writing and inadequate quality control. The related work is superficial, and the experimental analysis mostly repeats table entries without explaining the findings. The manuscript also uses nonstandard formatting, including the missing “Under review as a conference paper at ICLR 2027” header. Together with the numerical inconsistencies, these issues raise serious concerns about whether the manuscript was adequately checked before submission.

In its current form, it is not ready for peer review.

Questions for authors

no

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.