- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents TIM, an approach to merge models into groups of larger sizes than a single monolithic merge. TIM achieves this by considering conflicts beyond pairwise interference analysis and leverages random clustering trials to evolve partitions. The paper provides strong evidence of the insufficiency of pairwise analysis, where involving a third task overturns the decision of pairwise analysis in either directions (collapse or rescue). Experiments across both vision and language modalities involving up to 20 tasks show improvements resulting from the accuracy-storage trade-off.
Critical strengths
The analysis of pairwise merging provides novel empirical insights to the model merging community. The design of the second stage is also principled in improving over static merged groups.
However, unfortunately, this alone is insufficient for an acceptance recommendation and the weaknesses below far outweigh the strengths.
Critical weaknesses
- My main critic of the paper is that it ignores most-related prior work in this area [1,2], which has not been cited, discussed or compared against. This comes as a surprise, given the striking similarity of the title, exposition and motivation with [2].
This results in several technical considerations being absent from the paper. While [2] targets a data-free setting, TIM targets a data-based setting. This is crucial as a data-based setting elicits comparison with other data-based approaches, including Surgery [3], Surgery-v2, etc. which already recover most of the accuracy gap w.r.t individual experts while needing minimal extra storage. Secondly, both [1,2] allow different groupings per layer, while TIM forces same groups in each layer hurting flexibility. Thirdly, [2] is able to generate fractional-sized models, a capability not available in TIM.
-
My second main critic of the paper is that it does not accurately represent the size of router-based methods. Twin-Merging does not store all N models but only small rank compressed deltas alongside the merged model. Thus, Twin-Merging's storage is rather close to one monothilic merge than N models. Similarly, Surgery, requires storing only small additional adapters. In comparison, TIM stores full models, which is a vastly higher storage requirement. Thus the resulting accuracy gains are obvious. A fair comparison representing the exact storage in GBs vs. accuracy attained for all methods is missing from the paper.
-
Finally, the paper provides no analysis of overheads of the approach in the main section. The Appendix agrees that the default method is not scalable as N grows and proposes several amendments to improve efficiency, none of which are alluded to in the main paper.
References
[1] Zhang, M., Liu, J., Ding, G., Ou, L., Yu, X. and Zhuang, B., 2025, April. Channel merging: Preserving specialization for merged experts. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 21, pp. 22479-22487).
[2] Dhasade, A., Jhunjhunwala, D., Vujasinovic, M., Joshi, G. and Kermarrec, A.M., 2026, April. Navigating the accuracy-size trade-off with flexible model merging. In International Conference on Learning Representations (Vol. 2026, pp. 81010-81041).
[3] Yang, E., Shen, L., Wang, Z., Guo, G., Chen, X., Wang, X. and Tao, D. (2024) Representation surgery for multi-task model merging, Proceedings of the 41st International Conference on Machine Learning, 235, pp. 56332–56356.
Questions for authors
- How is the validation dataset split between tuning and performing accuracy evaluations during stage 1 and stage 2?
- Which merging algorithm is used in the evaluations of the main section? (Tables 1, 2 and Figure 2)
- How does the choice of affect the performance of TIM?
The authors' response does not adequately address my concerns. I elaborate on the remaining issues below.
I also remain unconvinced by the authors' claim that the underlying methodologies and core assumptions of the cited methods are "fundamentally different," given that all these approaches address the same general problem of merging models into architectures larger than a single constituent model. Moreover, both cited methods are data-free, unlike TIM, which makes them applicable in settings where access to data is limited or unavailable.
Finally, beyond the comparisons with FlexMerge and Channel Merge, the authors have not provided comparisons against other data-based methods, as suggested in my original review.
Given that these concerns remain unresolved, I am unable to revise my score.