acceptodds
Under review as a conference paper at ICLR 2027

Flexible and Evolving Model Merging via Task Interference Modeling

Abstract

Training-free model merging builds multi-task models from existing finetuned checkpoints, but every practical method faces a tension between accuracy and storage. Monolithic merges are storage-cheap but suffer from inter-task interference; expert routers avoid interference but store N independent copies. We argue that a principled middle ground exists: partition N tasks into K user-chosen groups and keep one expert per group, trading accuracy against memory. The central problem then becomes grouping tasks to minimize intra-group interference, and we make two observations. First, we reveal that pairwise interference is insufficient to characterize multi-task interactions: two tasks that exhibit only mild interference in pairwise merges may suffer severe degradation when a third is introduced. Second, interference modeling should not be static: as more empirical evidence about task compatibilities accumulates—in the form of cheap random clustering trials—our understanding of which tasks belong together can evolve and improve. From these insights we build Task Interference Mining (TIM), a two-stage framework. Stage 1 builds a task interference graph from pairwise and triple-merging outcomes and produces an initial grouping via spectral clustering. Stage 2 evolves the partition by iteratively mining random clustering trials and applying count-preserving correction moves drawn from three complementary strategies, guaranteeing monotonic improvement as more trial evidence is leveraged. We evaluate TIM on three vision benchmarks of increasing size (8, 14, and 20 tasks) and on six NLI tasks. TIM substantially outperforms similarity-based and random grouping baselines, recovering most of the gap between full merging and per-task expert performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merging, LLM, Task Interference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SPHUOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents TIM, an approach to merge models into groups of larger sizes than a single monolithic merge. TIM achieves this by considering conflicts beyond pairwise interference analysis and leverages random clustering trials to evolve partitions. The paper provides strong evidence of the insufficiency of pairwise analysis, where involving a third task overturns the decision of pairwise analysis in either directions (collapse or rescue). Experiments across both vision and language modalities involving up to 20 tasks show improvements resulting from the accuracy-storage trade-off.

Critical strengths

The analysis of pairwise merging provides novel empirical insights to the model merging community. The design of the second stage is also principled in improving over static merged groups.

However, unfortunately, this alone is insufficient for an acceptance recommendation and the weaknesses below far outweigh the strengths.

Critical weaknesses
  1. My main critic of the paper is that it ignores most-related prior work in this area [1,2], which has not been cited, discussed or compared against. This comes as a surprise, given the striking similarity of the title, exposition and motivation with [2].

This results in several technical considerations being absent from the paper. While [2] targets a data-free setting, TIM targets a data-based setting. This is crucial as a data-based setting elicits comparison with other data-based approaches, including Surgery [3], Surgery-v2, etc. which already recover most of the accuracy gap w.r.t individual experts while needing minimal extra storage. Secondly, both [1,2] allow different groupings per layer, while TIM forces same groups in each layer hurting flexibility. Thirdly, [2] is able to generate fractional-sized models, a capability not available in TIM.

  1. My second main critic of the paper is that it does not accurately represent the size of router-based methods. Twin-Merging does not store all N models but only small rank compressed deltas alongside the merged model. Thus, Twin-Merging's storage is rather close to one monothilic merge than N models. Similarly, Surgery, requires storing only small additional adapters. In comparison, TIM stores KK full models, which is a vastly higher storage requirement. Thus the resulting accuracy gains are obvious. A fair comparison representing the exact storage in GBs vs. accuracy attained for all methods is missing from the paper.

  2. Finally, the paper provides no analysis of overheads of the approach in the main section. The Appendix agrees that the default method is not scalable as N grows and proposes several amendments to improve efficiency, none of which are alluded to in the main paper.

References

[1] Zhang, M., Liu, J., Ding, G., Ou, L., Yu, X. and Zhuang, B., 2025, April. Channel merging: Preserving specialization for merged experts. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 21, pp. 22479-22487).

[2] Dhasade, A., Jhunjhunwala, D., Vujasinovic, M., Joshi, G. and Kermarrec, A.M., 2026, April. Navigating the accuracy-size trade-off with flexible model merging. In International Conference on Learning Representations (Vol. 2026, pp. 81010-81041).

[3] Yang, E., Shen, L., Wang, Z., Guo, G., Chen, X., Wang, X. and Tao, D. (2024) Representation surgery for multi-task model merging, Proceedings of the 41st International Conference on Machine Learning, 235, pp. 56332–56356.

Questions for authors
  1. How is the validation dataset split between tuning λ\lambda and performing accuracy evaluations during stage 1 and stage 2?
  2. Which merging algorithm M\mathcal{M} is used in the evaluations of the main section? (Tables 1, 2 and Figure 2)
  3. How does the choice of M\mathcal{M} affect the performance of TIM?
Reviewer_BbE8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TIM, a two-stage model merging method. Given N task vectors, TIM partitions the tasks into groups and merges each group using existing merging methods. Experiments on several vision and NLP tasks show the effectiveness of the proposed method.

Critical strengths
  • This paper is well-written and easy to follow.
  • The problem of router-based merging is meaningful, making this paper well-motivated.
  • The experiments are extensive.
Critical weaknesses
  • This paper claim that the proposed method is cheap and traning-free. However, the authors do not provide experiments on the computational cost, such as wall-clock time. I also suggest the authors can plot a figure to compare the trade-off between accuracy and computation across different methods.
  • Partitioning tasks into groups is a good idea. However, I do not consider this as an important question, making the proposed method less significant for me.
  • I think this paper should compare a variation using the STI metric. Because in the related work, the authors say " STI essentially measures the correlation/orthogonality among tasks; consequently, a non-zero STI score cannot distinguish between interfering and compatible task relationships", which should be supported by experiments.
Questions for authors
  • What is the sensitivity of \alpha?
  • What happens in Stage 2 when the reference pool is empty at initialization?
  • Is \lambda tuned per group/expert independently for TIM, and identically for the grouping baselines?
Reviewer_ZzFhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper discusses an approach to model merging that provides an accuracy-to-memory trade-off by partitioning NN tasks into KK groups and keeping one expert per group. The central problem is how to identify KK groups of tasks that minimize negative task interference.

To address this, the authors propose a two-stage approach called Task Interference Modeling (TIM). In the first stage, the model builds a task interference graph based on pairwise and triple-merging analyses and produces an initial grouping via spectral clustering. The second stage takes an active-learning approach, in which the partion evolves by iteratively mining random clustering trials and applying correction moves.

Critical strengths
  • The paper is well written and easy to understand.
  • The problem is interesting, as it effectively addresses the task-grouping problem in the multi-task learning literature.
  • The paper provides strong empirical evidence that pairwise task compatibility does not reliably predict higher-order merging behavior.
  • The evaluation across vision and NLP tasks, together with the use of multiple model-merging operators, is well-designed.
Critical weaknesses
  1. The experimental framework could be more robust. The results do not sufficiently demonstrate the robustness of the proposed approach. How the approach holds across multiple random subset of groupings should be checked, variance/standard deviation of the performances should be reported.

  2. The results do not establish the generalizability of the proposed approach. How different architectures affect the computed task interference from stage 1? Can the authors point me to any specific analysis if it exists?

  3. The computational cost of Stage 1 could limit the scalability of TIM. The method requires pairwise and triple-merging evaluations, with the latter growing cubically with the number of tasks. The paper should provide a clearer analysis of this cost and discuss its implications for larger task collections. Also, the improvements from Stage 2 are relatively small, and the paper does not report whether these gains are statistically significant or robust across different random trials.

  4. From my understanding, the paper is effectively a new addition to the task-grouping literature in the multi-task domain. Several existing approaches propose different ways to compute the pairwise interference scores, but the paper does not discuss them. For example,

  • Fifty et al. (2021) introduce inter-task-affinity scores computed for one model and transfer them to groups. The paper mentions this approach.

  • Li et al. (2024) derive pairwise task affinities by estimating the performance of random task groups and averaging their resulting performance.

  • Ayman et al. (2026) propose a pairwise task-affinity scores that serve as an initial scoring strategy, which is later refined using the performance of random groups in a data-driven manner.

  • Song et al. (2022) propose a solely data-driven approach that also employs active learning to predict optimal groupings for multi-task models. Their active learning strategy picks the groups that have the maximum predicted gains and refine their model with the newly appointed training group.

I think the paper should mention or compare stage 1 of TIM against some of these existing pairwise scoring approaches to strengthen its claims.

References:

  • Li, D., Sharma, A., & Zhang, H. R. (2024, August). Scalable multitask learning using gradient-based estimation of task affinity. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 1542-1553).
  • Song, Xiaozhuang, et al. "Efficient and effective multi-task grouping via meta learning on task combinations." Advances in Neural Information Processing Systems 35 (2022): 37647-37659.
  • Ayman, A., Mukhopadhyay, A., & Laszka, A. (2026). Ensemble Prediction of Task Affinity for Efficient Multi-Task Learning. arXiv preprint arXiv:2602.18591.
Questions for authors

See weakness.

Comments on the paper

6 comments

The authors' response does not adequately address my concerns. I elaborate on the remaining issues below.

  1. FlexMerge supports different groups across layers, whereas TIM does not. FlexMerge can also produce models with fractional sizes, unlike TIM. These represent concrete limitations of TIM. The distinction emphasized by the authors appears to primarily concern the choice of similarity metric, which is not fundamental to either framework. Both FlexMerge and Channel Merge can, in principle, be implemented with alternative similarity metrics.

I also remain unconvinced by the authors' claim that the underlying methodologies and core assumptions of the cited methods are "fundamentally different," given that all these approaches address the same general problem of merging models into architectures larger than a single constituent model. Moreover, both cited methods are data-free, unlike TIM, which makes them applicable in settings where access to data is limited or unavailable.

  1. I disagree with the authors' characterization of prior work as "orthogonal." A central objective of model merging is to achieve a favorable trade-off between model size and predictive accuracy. Compression, as employed in Twin-Merge, is therefore not a confounding factor but a legitimate approach to achieving this objective. Without properly accounting for the parameter footprints of the compared methods, Table 1 does not provide a fair assessment of their relative performance. If Twin-Merge already achieves a superior accuracy–size trade-off, the additional benefit of TIM remains unclear.

Finally, beyond the comparisons with FlexMerge and Channel Merge, the authors have not provided comparisons against other data-based methods, as suggested in my original review.

Given that these concerns remain unresolved, I am unable to revise my score.

AuthorsOpenReview ↗

The 2-nd Response

We sincerely thank the reviewer for their persistent feedback.

Q1: The distinction emphasized by the authors appears to primarily concern the choice of similarity metric, which is not fundamental to either framework.

A1: FlexMerge and Channel Merging rely on parameter-space geometric proxies (e.g., cosine similarity of weight vectors). Even if one were to substitute cosine similarity with another geometric metric, it remains a static, algebraic approximation of compatibility. Crucially, as the reviewer correctly noted, FlexMerge does not model or discuss task interference. In contrast, the core contribution of TIM is explicitly modeling empirical task interference, specifically the higher-order "inconsistent interference accumulation" (collapse and rescue phenomena) we rigorously demonstrate in Section 5.6. Geometric metrics, by definition, cannot predict how the addition of a third task dynamically alters the compatibility of two others. TIM bypasses this limitation by using actual validation accuracy drops (from pairwise and triple merging outcomes) to build its grouping signal. This moves the paradigm from static parameter correlation to real-world performance compatibility.

Q 2: Without properly accounting for the parameter footprints of the compared methods, Table 1 does not provide a fair assessment of their relative performance. If Twin-Merge already achieves a superior accuracy–size trade-off, the additional benefit of TIM remains unclear.

A 2: We sincerely thank the reviewer for emphasizing the importance of the accuracy-size trade-off. To provide the fair, footprint-aware assessment the reviewer rightly requested, we have added three detailed comparison tables that directly pit TIM against Twin-Merging across varying storage constraints. The empirical results decisively demonstrate that TIM achieves a strictly superior accuracy-size trade-off when storage is constrained.

1. Empirical Evidence: TIM’s Superior Trade-off Under Storage Constraints As shown in the new tables below, while Twin-Merging may hold a marginal advantage at very large storage budgets (e.g., 4×), its accuracy collapses catastrophically as the storage budget tightens, whereas TIM’s performance degrades very gracefully.

Table A: 20-task suite Comparison

Storage4×2×1×0.5×0.25×
TIM (K=10)90.4890.1989.5588.0985.51
TIM (K=5)89.0188.6487.7986.0983.37
TwinMerge91.0987.7680.8173.3364.75

Table B: 14-task suite Comparison

Storage2.8×1.4×0.7×0.35×0.175×
TIM (K=7)89.6189.3788.6386.7483.17
TIM (K=4)88.2887.7886.8485.0582.35
TwinMerge90.7187.5781.5775.3667.85

Table C: 8-task suite Comparison

Storage1.6×0.8×0.4×0.2×0.1×
TIM (K=4)91.9991.5790.3388.1883.79
TIM (K=2)90.4389.7787.9784.8479.89
TwinMerge92.8990.3385.0978.5769.76

Data Highlight: On the 20-task suite at 0.25× storage, Twin-Merging’s accuracy plummets to 64.75, while TIM maintains 85.51—a massive 20.76% absolute advantage. This trend is consistent across all suites.

2. Why TIM Achieves a Better Trade-off: The Role of Interference Modeling This stark contrast is exactly due to the task interference that TIM explicitly models, but Twin-Merging ignores. Twin-Merging's Bottleneck: It attempts to compress the exclusive residuals of all N tasks globally. When tasks interfere heavily, these residuals contain complex, high-frequency information to correct the shared expert's compromises. Forcing a low-rank approximation on such high-interference residuals inevitably discards critical information, causing the accuracy collapse seen in the tables. TIM's Advantage: By first partitioning the N tasks into K groups with minimal intra-group interference, TIM ensures that the tasks within each group are highly compatible. Consequently, the exclusive residuals within a TIM group are genuinely clean and strictly low-rank.

AuthorsOpenReview ↗

More experiments

We sincerely apologize for the slight delay in providing this specific comparison in our previous response; we were actively running the requested TSV + Surgery baseline to ensure a strictly fair, apples-to-apples evaluation aligned with our experimental setup. We have now completed this experiment, and the results further solidify our argument.

Merging Accuracy Comparison (Vision Tasks: 14 & 20 Tasks, ViT-B/32)

Method14 Tasks (Norm. Acc.)20 Tasks (Norm. Acc.)
Experts (Upper Bound)100.00%100.00%
Twin-Merging98.80%98.50%
Surgery (Data-based)96.27%96.61%
TIM (Ours, K=N/2K=N/2)98.54%98.57%
TIM (Ours, K=N/4K=N/4)96.43%95.92%

Crucially, from a pure performance perspective, Surgery's accuracy is demonstrably inferior to both Twin-Merging and our TIM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.