acceptodds
Under review as a conference paper at ICLR 2027

Is Flat Merging Enough? Evolutionary Search for Tree-Structured Model Merging

Abstract

Model merging provides an efficient way to integrate independently fine-tuned models without costly retraining or access to their original training data. Most existing methods, however, focus primarily on designing merging operators while combining a fixed set of candidate experts through a typically flat aggregation structure. This leaves a largely overlooked structural design space: how experts should be organized, which experts should participate, and whether experts should be reused throughout the merging process. Through systematic empirical studies, we show that, even when the candidate experts and merging operator are held fixed, changing only the merging structure can substantially affect performance, with markedly different sensitivities across operator families. Identifying effective configurations in this structural space is challenging, as merging topology, expert selection, and expert reuse jointly define a large discrete and combinatorial search space. We therefore propose EvoTM, an evolutionary framework based on genetic programming for optimizing tree-structured model merging. EvoTM represents a complete merging procedure as a hierarchical tree, where leaves correspond to expert models and internal nodes apply a given merging operator. Structure-aware evolutionary operators jointly explore tree topology and expert composition, naturally accommodating expert selection and reuse within a unified framework. Experiments across diverse merging operators and both vision and language models show that EvoTM can improve over conventional flat merging in multiple settings. Further analyses reveal operator-dependent structural sensitivities and provide additional insights into how different classes of merging operators respond to tree-structured composition. These results highlight merging structure as a complementary and previously underexplored design dimension of model merging.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Merging, Tree-Structured Model Merging, Evolutionary Optimization, Genetic Programming, Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fHMqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes EvoTM, a genetic-programming method that searches how expert models are grouped and recursively merged. They use tree structure to model the merging process. Leaves represent experts and internal nodes apply a fixed merging operator. EvoTM searches these trees using tournament selection, subtree crossover, and mutations for grouping, expert selection, and reuse. Experiments cover three ViT backbones and Qwen2.5-1.5B LoRA experts evaluated on eight source-disjoint target tasks. The paper reports improvements over fixed flat merging across several operators.

Critical strengths
  • The paper includes useful controls experiments.
  • The idea is interesing and the abstract / introduction is clear to understand.
Critical weaknesses
  • The approach appears to be a direct and trivial application of standard Genetic Programming, where leaves are naturally terminal symbols (expert models) and internal nodes are functions (merging operators). While the figures seem complex, the textual description omits formal mathematical formulations and merely repurposes existing evolutionary algorithms. It would be better for the authors to clarify the technical depth and originality of their method.

  • Regarding linear weighting methods in model merging, recursive averaging is mathematically equivalent to a flat weighted average (for methods like Averaging and Task Arithmetic). The authors have already demonstrated in Table 6 that this method is almost equivalent to carefully optimized flat merging methods, which seems to render the complex tree structure meaningless. Therefore, the authors should compare their method with carefully tuned flat coefficients and TA scaling under the same search budget in the main experiments, rather than relying on flat baselines with fixed coefficients.

  • Figure 7 shows that using a simple search metric like subspace distance already identifies the best TIES pairing 60% of the time, suggesting that employing a complex and costly evolutionary algorithm may not yield a high return on investment (ROI).

  • The computational cost remains unclear. Trees contain different numbers of internal merges, and recursive statistics-based or adaptive operators may require repeated feature collection or optimization. Moreover, the authors only conduct experiments on small models like ViT or Qwen-LoRA (LoRA is significantly smaller than a normal backbone). The authors should conduct scaling experiments and report wall-clock time, GPU-hours, operator executions, and intermediate-state memory.

  • It is confusing that the performance change substantially between the Table 2 and Table 5 experiments, which raises serious concerns regarding the soundness of the experiments.

OperatorTable 2 ValTable 2 TestTable 5 Flat
Task Arithmetic23.9921.4034.16
TSV-M36.3232.5328.61
  • There is a severe scarcity of experiments and supporting tabular data (only three tables are in the main text, although many are in the appendix, they lack clear logic for the reader). There is also a lack of valid baselines and unfair comparisons. The authors should compare EvoTM with existing evolutionary merging methods like EvoGM [1] and Evolutionary Optimization of Model Merging Recipes [2]. Alternatively, they should compare it with other search methods under the same initial population and evaluation budget.

[1] EvoGM: Learning to Merge LLMs via Evolutionary Generative Optimization, ICML 2026

[2] Evolutionary Optimization of Model Merging Recipes, Nature Machine Intelligence

Questions for authors

See Weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer fHMq

We thank the reviewer for the detailed assessment and address each point below.

W1. Technical contribution and originality.

We respectfully disagree that EvoTM’s contribution is limited to applying established GP operations. As illustrated in Figure 1, we formulate merging structure as an explicit optimization problem: with the candidate expert pool, merging operator, and its hyperparameters fixed, how should experts be grouped into intermediate models, which experts should participate, and whether they should be reused?

This formulation is grounded in the analysis in Section 3. Controlled experiments show that structural effects differ across operators and identify their sources: implicit reweighting and repeated scaling for additive operators, and path-dependent intermediate transformations for nonlinear operators. The quartet experiments further isolate grouping effects while holding tree shape and nominal leaf weights fixed. These findings explain when and why merging structure matters.

We represent these choices as complete, variable-size multi-ary trees. Internal structure specifies hierarchical grouping, while leaf identities and multiplicities encode expert selection and reuse. Eqs. (2), (4), and (5) formalize recursive execution, feasibility constraints, and the optimization objective. GP provides a practical way to search this discrete, combinatorial space.

We acknowledge that tree encoding and generic GP operations are established techniques. Our contribution lies in the systematic analysis of structural effects and the unified formulation for optimizing these choices.

W2. Does equivalence to flat weighting make tree search redundant, and how does it compare with tuned flat merging?

Explaining how structure affects different merging operators is precisely the purpose of Section 3. For Averaging and TA, the effect arises from implicit reweighting and repeated scaling, yielding equivalent flat coefficients. Table 7 confirms this explanation: controlling effective weights or repeated scaling largely removes the differences across structures, as intended in these experiments.

However, these mechanisms explain only part of the structural effects. In TIES, grouping affects intermediate trimming and sign resolution; in TSV-M, it affects subspace construction and truncation. These intermediate transformations can change the final result even with fixed operator hyperparameters and nominal leaf weights. Structural effects therefore cannot all be attributed to reweighting and repeated scaling.

We agree that direct coefficient optimization is an essential comparison for additive operators. We compare EvoTM with Averaging and TA against EvoGM [1] and modelwise/blockwise Recipe+TA [2] on the same 20 experts and tasks. These methods provide comparable approaches to optimizing effective expert contributions: EvoTM determines coefficients implicitly through tree structures, while the baselines optimize them explicitly.

MethodVal ACC (%)Test ACC (%)Runtime (h)
EvoGM60.40 ± 0.0860.40 ± 0.030.68 ± 0.01
EvoTM + Averaging61.82 ± 0.0161.64 ± 0.170.64 ± 0.01
EvoTM + TA62.25 ± 0.2961.85 ± 0.070.78 ± 0.22
Recipe + TA (modelwise)61.93 ± 0.2961.61 ± 0.100.63 ± 0.02
Recipe + TA (blockwise)61.85 ± 1.4261.50 ± 1.570.62 ± 0.03
  • All methods use the same number of validation evaluations, identical validation/test splits, and the same device. Results are mean ± SD over five seeds. Under this evaluation budget, EvoTM achieves competitive test accuracy at comparable runtimes, with EvoTM+TA obtaining the highest mean test accuracy.

W3. Does the heuristic result in Figure 7 justify replacing evolutionary search?

We agree that the 60% result is informative: subspace distance identifies the best TIES pairing in 12 of 20 sampled quartets, compared with 33.3% for random choice. However, Figure 7 (Figure 8 in the revision) tests a local decision among three pairings of four fixed experts under a fixed depth-2 balanced tree. We evaluated several metrics motivated by different merging mechanisms across all operators (such as sign conflict, Gram distance, and subspace distance for TIES, RegMean, and TSV-M, respectively), but each metric was evaluated for every operator. Thus, the 60% is one result from these comparisons, not the performance of a defined rule for selecting metrics and constructing trees.

Constructing a full merge tree requires additional decisions about grouping and how to handle intermediate models, which become inputs to subsequent merges. EvoTM also searches topology, expert selection, and reuse, all held fixed in the quartet experiment. Thus, Figure 8 in the revised manuscript motivates using static metrics to guide search, but cannot establish that they achieve comparable final performance or better computational return.

AuthorsOpenReview ↗

Response to Reviewer fHMq

W4. Computational cost and scaling.

Regarding model scale, our evaluation includes full-weight merging of ViT-B/32, ViT-B/16, and ViT-L/14 experts. In particular, Appendix A.4.2, Table 12 reports merging 20 fully fine-tuned ViT-L/14 experts, each with approximately 303.2M vision-encoder parameters. These experiments merge complete vision encoders. To measure scaling under a controlled workload, we additionally evaluate EvoTM with Averaging on the same 10 tasks, using 10 experts per architecture. Each run evaluates 352 candidate trees on one NVIDIA A100 GPU.

BackboneParams./expert (M)Runtime (h)
ViT-B/3287.50.331
ViT-B/1685.80.936
ViT-L/14303.23.740

We also added accuracy–runtime comparisons for merging 20 ViT-B/32 experts, covering Flat, four other fixed structures, and EvoTM M0–M3. The table below summarizes M3 results for five operators; the revised manuscript provides the complete comparisons and visualizations (acc-runtime trade-off plots).

OperatorTest accuracy (%)Gain over Flat (pp)Total runtime (h)
Averaging61.38+1.120.682
TA61.74+26.080.623
TIES61.75+12.083.825
Fisher61.58+3.304.820
RegMean62.28+1.882.282
  • Results are averaged over five seeds, with one A100 GPU per run.

We acknowledge that structural search can require substantial additional computation, and its practical value depends on the operator, accuracy gain, and available budget. The comparisons in W2 further place this cost alongside other evolutionary merging approaches. All reported timings use a single GPU for consistent comparison; candidate trees within each generation could be evaluated in parallel across multiple GPUs to reduce wall-clock time.

W5. Inconsistent results in Tables 2 and 5.

We thank the reviewer for identifying this discrepancy. Both experiments use the same Qwen base model and ten LoRA experts, but their protocols differ. The original Table 2 merged LoRA factors AA and BB separately and used the original dev/test splits. Table 5 (Table 6 in the revision) merged reconstructed updates, ΔW=(α/r)BA\Delta W=(\alpha/r)BA, and evaluated a randomly sampled 50% subset of the original test set to reduce the cost of the preliminary topology study. Factor-wise and update-space merging are not equivalent, so the discrepancy cannot be attributed solely to evaluation data. We apologize for not clearly distinguishing these protocols.

We have rerun Flat and all M0–M3 searches using dense-update merging, retaining Table 2’s original dev/test splits, evaluator, and search budgets. Each internal node merges dense updates, and the root update is added to the pretrained model. Tree selection uses dev data exclusively; the selected models are evaluated on the full original test set. Table 2 has been replaced with these results.

Within both the original and revised Table 2, Flat and EvoTM share the same merging representation and evaluation protocol. The revised tables now share the merging representation, but their test sets still differ; their absolute scores should therefore not be compared directly. We have explicitly stated these distinctions in the experimental setup.

W6. Baselines and presentation.

We would like to clarify that our primary comparison concerns merging structure under a fixed operator, as explained in W1. EvoGM [1] optimizes merging coefficients, while Merging Recipes [2] optimizes merging hyperparameters and layer-level composition. These methods aim to introduce new merging operators; while our work examines a complementary dimension: how merging operations are organized. We compare flat merging with fixed tree structures, investigate their operator-dependent effects, and introduce EvoTM to optimize topology, expert selection, and reuse.

For additive operators, structural search induces effective coefficient changes, making direct weight optimization particularly relevant. We have therefore added comparisons with EvoGM and Merging Recipes, as reported in W2. We also compare GP with random and greedy structural search under matched candidate-evaluation budgets; representative accuracy–runtime comparisons are presented in Section 5.3, Figure 7.

I appreciate the authors’ detailed response and additional experiments. The new baselines and runtime measurements are helpful. However, I still have three main concerns:

  1. I remain unconvinced that the methodological novelty is sufficient. The structural analysis is interesting, but the additional methodological contribution remains limited.
  2. The reported search takes several hours on an A100. I therefore remain concerned about the practical accuracy–cost trade-off, particularly as model size and merging complexity increase.
  3. Unclear benefit of GP over simpler structural search. W6 mentions comparisons with random and greedy search in the revised manuscript, but the response does not include their numerical results.
Reviewer_ECNnOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the structural limitations of traditional flat model merging and proposes EvoTM, a tree-structured model merging framework based on genetic programming. This method treats expert models as leaf nodes and merging operators as internal nodes, jointly optimizing the merging topology, expert selection, and expert reuse through evolutionary search. Experiments were conducted on various merging operators, the CLIP-based vision models, and the Qwen2.5-1.5B LoRA experts, revealing that the merging structure affects the final performance, and different merging operators exhibit varying sensitivities to structural changes.

Critical strengths

S1. This paper extends model merging from traditional flat aggregation to a structured optimization problem, unifying the three design dimensions of merging topology, expert selection, and expert reuse. By analyzing the effects of implicit reweighting and path-dependent expert interactions, the sensitivity of different merging operators to changes in hierarchical structure is revealed.

S2. This paper proposes the EvoTM framework based on genetic programming, utilizing a tree structure to represent the complete model merging process and jointly searching for different merging configurations through subtree crossover and structure-aware mutation operations. This method explores more flexible expert combinations and hierarchical merging strategies without modifying the underlying merging operator.

Critical weaknesses

W1. The core method has limited innovation, and its differentiation from existing automatic model merging research is not sufficiently prominent. Existing research optimizes model merging configurations through evolutionary search, considering expert selection, merging order, and parameter tuning. EvoTM primarily applies tree encoding, crossover, and mutation operations from standard genetic programming to model merging, lacking a novel search mechanism. Although the search space of general merging trees differs to some extent, the paper does not sufficiently demonstrate that this structured search has an irreplaceable advantage over existing methods, nor does it provide a direct comparison with strongly competing methods.

[1] Akiba et al., Evolutionary Optimization of Model Merging Recipes, Nature Machine Intelligence, 2025

[2] Bolton et al., SimMerge: Learning to Select Merge Operators from Similarity Signals, 2026

[3] Jiang et al., EvoGM: Learning to Merge LLMs via Evolutionary Generative Optimization, ICML 2026

W2. The main performance improvements are significantly confounded by implicit reweighting and repeated scaling. Appendix A.2.2 of the paper shows that after controlling for expert weights, the structural performance difference of Averaging decreased from 4.59 to 0.01 percentage points; after controlling for repeated scaling, the difference in Task Arithmetic decreased from 24.58 to 0.01. This indicates that some significant gains are essentially attributable to changes in weights and scaling, rather than the unique role of the complex tree structure. Due to the lack of a fair comparison with flat merging methods that optimize weights and scaling coefficients, the actual gains of EvoTM cannot be fully demonstrated at present.

W3. There are significant issues with the necessity of joint search and the reliability of the experimental results. With the same 20-generation budget, the M320\mathrm{M3}{20} (Full) search outperforms the M0 (Topology) search in only 5 out of 13 operator settings for visual and language models, indicating that expanding the search space does not bring stable gains. Furthermore, ΔFlat\Delta{\mathrm{Flat}} in Tables 1 and 2 selects the best validation and test results from different search configurations, rather than uniformly selecting a model based on the validation set. For example, in Qwen Averaging, the 20-generation version of M3 is only 0.02 test percentage points higher than Flat, while the reported best test gain is 0.60. Such aggregation of results may overestimate the actual achievable performance improvement.

Questions for authors

Q1. Since the structural sensitivity of Averaging and Task Arithmetic almost disappears after controlling weights or repeated scaling, can the authors compare it with the Flat method, which optimizes the merging of weights and scaling factors, under the same computational budget? Otherwise, how can it be proven that the main improvement comes from the tree structure, rather than implicit parameter tuning?

Q2. Why do joint search topology, expert selection, and reuse not consistently outperform topology-only, and even show significant degradation in some operators? Can the authors prove that these three search dimensions are truly complementary, rather than simply expanding the search space and increasing optimization difficulty?

Q3. EvoTM uses conventional genetic programming operations; why is such a complex evolutionary search necessary? Can the authors compare it with random tree search, greedy structure search, and existing automatic merging methods under the same number of merging evaluations and GPU time, demonstrating the inherent advantage of its search strategy?

Q4. Why are the best test results selected from different search configurations to calculate ΔFlat\Delta_{\mathrm{Flat}} in Tables 1 and 2? The actual model selection should not use the test set. Can the authors uniformly determine the optimal structure based on the validation set, then report its corresponding test performance, and provide statistical results for multiple random seeds?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ECNn

We thank the reviewer for the detailed assessment and address each point below.

W1. Contribution and distinction from automatic merging methods.

We respectfully disagree that EvoTM’s contribution is limited to applying established GP operations. As illustrated in Figure 1, we explicitly formulate merging structure as a design dimension encompassing hierarchical grouping, expert selection, and expert reuse, with the merging operator and its hyperparameters held fixed. The cited methods optimize different aspects of merging. Merging Recipes [1] optimizes DARE–TIES hyperparameters and assembles layers from different models into an inference path. EvoGM [3] optimizes task-vector mixing coefficients and updates its expert pool with elite merged models. SimMerge [2] selects model subsets, merge order, and per-step operators within a chain, repeatedly merging the accumulated result with the next model. EvoTM fixes the operator and its hyperparameters and represents each candidate as a complete multi-ary tree: separate branches can form intermediate models before they are combined, while leaf identities and multiplicities encode expert selection and reuse.

Our study also analyzes why structural effects differ across operators: effective weighting and repeated scaling for additive operators, and path-dependent interactions for nonlinear operators. Controlled quartet experiments isolate grouping effects and show that the tested pairwise metrics do not consistently predict the best grouping. These findings motivate explicitly searching merging structures. The resulting space of variable-size trees is discrete and combinatorial, making GP a suitable search tool. Our contribution lies in the systematic analysis and joint formulation of these structural choices; we do not claim new generic GP operations. To assess their practical value, we compare EvoTM with existing automatic model-merging methods [1], [3] (W2/Q1), as well as alternative search strategies (Q3), under comparable experimental settings.

W2/Q1. Are the gains attributable to implicit reweighting, and how does EvoTM compare with optimized flat merging?

The reviewer’s observation is consistent with Section 3, where we identify implicit reweighting and repeated scaling as mechanisms through which structure affects additive operators. For Averaging and TA, controlling these factors largely removes the differences across structures, as Table 7 shows. This is the intended conclusion of these controlled experiments.

However, this explanation does not extend to all operators. For TIES and TSV-M, structure also determines which updates are combined at each intermediate step, affecting trimming and sign resolution in TIES and subspace construction and truncation in TSV-M. These intermediate transformations can change the final result even with fixed operator hyperparameters and nominal leaf weights. Structural effects therefore cannot all be attributed to reweighting and repeated scaling.

For additive operators, the practical question is whether tree search offers benefits over directly optimizing flat coefficients. We therefore compare EvoTM with Averaging and TA against EvoGM [3] and modelwise/blockwise Recipe+TA [1] on the same 20 experts and tasks. These methods provide comparable approaches to optimizing effective expert contributions: EvoTM determines coefficients implicitly through tree structures, while the baselines optimize them explicitly.

MethodVal ACC (%)Test ACC (%)Runtime (h)
EvoGM60.40 ± 0.0860.40 ± 0.030.68 ± 0.01
EvoTM + Averaging61.82 ± 0.0161.64 ± 0.170.64 ± 0.01
EvoTM + TA62.25 ± 0.2961.85 ± 0.070.78 ± 0.22
Recipe + TA (modelwise)61.93 ± 0.2961.61 ± 0.100.63 ± 0.02
Recipe + TA (blockwise)61.85 ± 1.4261.50 ± 1.570.62 ± 0.03
  • All methods use the same number of validation evaluations, identical validation/test splits, and the same device. Results are mean ± SD over five seeds. Under this evaluation budget, EvoTM achieves competitive test accuracy at comparable runtimes, with EvoTM+TA obtaining the highest mean test accuracy.
AuthorsOpenReview ↗

Response to Reviewer ECNn

W3/Q2. Why does joint search not consistently outperform topology-only search?

M0–M3 are designed to examine how different structural choices affect a fixed merging operator. M0 searches topology; M1 additionally allows selection; M2 allows reuse while retaining all experts; and M3 enables both. The purpose is to understand when these choices help, rather than to assume that each additional freedom improves performance. The observation that M3 does not consistently outperform M0 therefore highlights the operator-dependent benefits that these experiments investigate.

The mechanisms in Section 3 help interpret this variation. For Averaging and TA, topology already changes effective expert weights, so selection and reuse may provide limited additional benefit. For TSV-M, recursive merges construct and truncate task-vector subspaces, making the retained directions sensitive to both grouping and expert participation. Retaining all experts while allowing reuse, as in M2, may therefore be useful for some settings. In the Qwen TSV-M results in Table 2, M2 achieves 34.61% test accuracy at 20 generations, compared with 34.45% for M0 and 33.45% for M3. Search budget may also contribute: M3 explores a larger space under the same evaluation budget, and extending it to 30 generations raises its accuracy to 35.23% in this example. This observation suggests a possible budget effect, without establishing a general advantage for joint search. Our results support treating topology, selection, and reuse as operator-dependent design choices; they do not establish consistent complementary gains from enabling all three.

Q3. Why use GP, and how does it compare with simpler search strategies?

GP is a natural fit for EvoTM’s large, discrete, combinatorial search space of variable-size merge trees, with validation accuracy as the objective and no directly available gradients. Internal structure encodes hierarchical grouping, while leaf identities and multiplicities encode expert selection and reuse. GP acts directly on this representation: fitness-based selection retains promising trees, subtree crossover recombines their groupings, and mutation modifies topology and expert participation. This motivates our choice of GP.

To assess whether this search strategy improves over simpler alternatives, we added random and greedy structure search in the same feasible space, using shared initial reference candidates and matched budgets of 352 evaluations, running on one NVIDIA A100 GPU.

Merge operatorSearch methodVal (%)Test (%)Runtime (h)
TAEvoTM62.0661.740.623
TArandom60.8360.240.608
TAgreedy61.2060.940.635
TIESEvoTM62.1861.753.825
TIESrandom61.7561.234.165
TIESgreedy62.0361.444.574
FisherEvoTM62.1161.584.820
Fisherrandom60.9560.384.810
Fishergreedy61.5760.825.741
RegMeanEvoTM62.8462.282.282
RegMeanrandom60.9160.562.177
RegMeangreedy61.5161.242.416
TSVMEvoTM74.9774.9018.160
TSVMrandom73.1472.6618.150
TSVMgreedy74.0273.8918.080
  • EvoTM uses M3 (Full). Under the matched evaluation budget, GP achieves higher test accuracy than random and greedy search across all five tested operators, with comparable runtimes.

Q4. (i) Why best results summarized across configurations? The original summaries were intended to show which search configuration benefited each merging operator most, reflecting the operator-dependent effects studied in our ablation. We agree that the test-side maximum can give a misleading impression of performance and have revised this reporting in the updated manuscript.

(ii) Can selection consistently use validation performance? Yes. Within each configuration, EvoTM already selects the best tree by validation accuracy and reports its corresponding test accuracy. We now also select the configuration summarized for each operator using validation accuracy only. The reported gain is the test accuracy of the validation-selected configuration minus the Flat baseline. For ViT-B/32, results are averaged over five seeds.

(iii) Can the authors provide multiple-seed statistics? Yes. We have added ViT-B/32 experiments with five random seeds. In the revised manuscript, Table 1 reports the five-seed means, with standard deviations provided in Appendix A.4.1, Tables 10 and 11.

Reviewer_gw9XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Model merging traditionally uses flat application of merging operators on the underlying experts to obtain the resulting model(s). However, recent works explore applying the operator in a fixed tree like structure, which raises a question how this tree can be optimized in order to obtain a better performing resulting model, and the process of this tree optimization is the primary research question of this paper. The proposed solution, EvoTM, starts with a set of set trees and applies evolutionary algorithms on this set, each iteration applying merges and mutations while keeping the candidates with the highest fitness across the population, before returning one final merges model at the end. The results show clear advantage over flat merging, especially prominent across simpler merging algorithms such as TA and TIES.

Critical strengths
  • The paper is very well written and easy to follow, especially motivation, as well as mathematical formulation of the problem. However, I believe Section 4.2 would benefit from visualization of some of the mentioned operations.
  • EvoTM shows clear benefits over flat merging, especially on simpler methods.
  • I like the attention to programmatical optimization as shown through caching explored trees. I am however curious about the cache hit rate and whether this optimization is really beneficial (very minor point, please focus on others, and leave this for the end if there is time left).
  • Hyperparameter sensitivity study is A.4.4 is useful.
  • Predominantly good evaluation section: Extensive ablations in Tables 1 and 2, interesting study in Table 3. A number of merging operators tested, effects on generalization considered.
Critical weaknesses

My main concerns of the paper revolves around the cost of running it. Model merging as we know it is useful because it is cheap and in some cases can even be done just in time on quite cheap hardware (even CPUs). However, if we stack so many space search optimizations on top at some point this may become more expensive than fine tuning the pretrained model on all datasets collectively. As such we have to ask ourselves how much is each unit of accuracy worth in terms of compute cost, and this question of compute cost is entirely ignored by the paper. This comes in two forms in the paper: firstly the search process is expensive on its own, and secondly it introduces a number of hyperparameters (some of them being number of iterations, population size, n-arity, the set of allowed operations) that also take compute to explore and depending on the number of iterations in hyper parameter tuning may dominate the cost.

I like the idea and the way paper presents it, hence the positive rating. However I am open to considering to lower my rating if the issue of the computational cost is not properly discussed and evaluated.

Questions for authors
  • How is hyper parameter search exactly performed?
  • The computational cost of the algorithm is unclear beyond the number of iterations, inclusion of accuracy against compute budget (GFLOPs or time across the same hardware) would paint a much clearer picture. Inclusion of simple fixed structure hierarchical merging approaches in such plot would also be a good reference.
  • In general comparison in terms of compute and performance against existing fixed hierarchical merging approaches is omitted (Tables 1 and 2 for example). Without such comparison the question whether the usage of EvoTM is worth the trade-off in practice remains under a question mark. Please include hyper parameter search cost into compute of EvoTM as it's clearly affecting its performance.
  • All shown models are either relatively small, or use PEFT which raises a concern how does the search time scale against the number of trainable parameters. Have there been any attempts to evaluate the algorithm on models in 100s million parameter range such as T5-base/large?
  • Considering there are no mentions of reproducibility, nor the code, are the authors committed to releasing their implementation in case the paper is accepted?

Clarifying questions:

  • What models/tasks are used in Section 3.3?
  • In practice some may want to optimize for a different metric than plain averaging of individual task performances shown in Eq. (5). Has this been explored, are there any hints that the performance of EvoTM remains high across the change?
  • In Structure-aware mutation (Section 4.2) it is unclear where new nodes are introduced within the tree structure? Can this be clarified?
  • (Minor remark) amaxa_\text{max} appears in Eq. (4) for the first time yet it is introduced only in Table 3, please consider introducing this earlier.
  • Why is validation accuracy consistently higher than test accuracy, is the validation set ever seen by the algorithm during the search process?
5 comments
AuthorsOpenReview ↗

Response to Reviewer gw9X

We sincerely thank the reviewer for the careful review and encouraging assessment. We greatly appreciate your thoughtful suggestions and address each point below.

Q1. How were the hyperparameters determined?

We used the fixed EvoTM settings reported in Appendix A.1.3: population size 32, 16 offspring per generation, crossover/mutation probabilities 0.8/0.2, and a maximum node arity equal to the expert count. No separate hyperparameter search or task-/operator-specific tuning was performed. Appendix A.4.5 (Fig.11) reports sensitivity analyses, which were not used to select the main settings. During each run, the merging operator and its parameters remained fixed, and EvoTM searched only the merge-tree structure.

Q2-Q3. Computational cost and practical trade-offs.

We agree that iteration counts alone do not adequately characterize computational cost. We have therefore added accuracy–runtime comparisons covering Flat, 4 other fixed structures, and 4 EvoTM configurations, M0–M3 with 20 iterations.
The table below summarizes representative results for Flat, Hierarchical, and EvoTM M3 when merging 20 ViT-B/32 experts.

The revised manuscript provides the complete results and visualizations (acc-runtime trade-off plots) for all configurations in a new subsection "Computational Cost and Practical Trade-offs".

OperatorFlatHierarchicalEvoTM M3
TA35.66 / 0.09860.27 / 0.09561.74 / 0.623
TIES49.67 / 0.11859.71 / 0.11961.75 / 3.825
Fisher58.28 / 0.12559.76 / 0.12961.58 / 4.820
RegMean60.40 / 0.12560.74 / 0.12862.28 / 2.282
  • Each entry reports mean test accuracy (%) / total runtime (h) over 5 seeds. All measurements use one NVIDIA A100 GPU. As clarified in Q1, no separate hyperparameter search was used to select the main settings.

We acknowledge that structural search can require substantial additional computation, and its practical value depends on the merging operator and the available budget. Fixed structures offer lower-cost alternatives, while EvoTM achieves higher accuracy in the reported comparisons. Search cost remains a practical limitation of evolutionary methods; evaluating candidate trees in parallel across multiple GPUs within each generation could reduce wall-clock time.

Q4. Model scale and runtime.

Appendix A.4.2 (Table 12) reports full-weight merging of 20 fully fine-tuned ViT-L/14 experts, each with approximately 303.2M vision-encoder parameters. To assess runtime across model scales, we additionally benchmark 3 backbones on the same 10 tasks, with 10 experts per backbone. All runs use one A100 GPU, the Averaging operator, and EvoTM M3 (full), with a population size of 32, 16 offspring per generation, and 20 generations.

BackboneParams./expert (M)Runtime (h)
ViT-B/3287.50.331
ViT-B/1685.80.936
ViT-L/14303.23.740

Search takes longer on ViT-L/14, but parameter count alone does not determine runtime: ViT-B/16 has a similar parameter count to ViT-B/32 yet requires more validation computation because its smaller patches produce more image tokens.

Q5. Code release.

Yes, we will release the implementation upon acceptance.

AuthorsOpenReview ↗

Response to Reviewer gw9X (C denotes Clarifying questions)

C1. Setup in Section 3.3.

Both analyses use the same pool of 20 task-specific CLIP ViT-B/32 experts, independently fine-tuned from a common pretrained initialization, with one expert per task. The 20 tasks are SUN397, Stanford Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, DTD, Oxford Flowers 102, PCAM, FER2013, Oxford-IIIT Pet, STL10, CIFAR100, CIFAR10, Food101, Fashion-MNIST, EMNIST Letters, KMNIST, and Rendered SST-2. We have clarified this setup in Sections 3.2 and 3.3 and provided the model details and complete task list in Appendix.

C2. Performance with alternative optimization objectives.

EvoTM supports alternative objectives by changing its fitness function. We evaluate two variants on 10 CLIP ViT-B/32 experts. All accuracies below are measured on held-out test data.

Task-weighted fitness uses a weighted mean of validation accuracies, assigning weight 4 to the target task (SUN397 or DTD) and weight 1 to each remaining task. U/W denote uniform/weighted fitness.

Operator / targetTarget U→W (%)10-task mean U→W (%)
Averaging / SUN39759.62→63.2365.35→64.66
Averaging / DTD50.35→57.7465.35→64.02
TIES / SUN39757.11→63.4268.77→68.57
TIES / DTD47.75→58.8768.77→68.02
  • All four settings improve target accuracy, with a modest decrease in the unweighted mean.

Structure-penalized fitness subtracts a penalty for each internal merge node beyond the root from mean validation accuracy, aiming to reduce merging cost. The following results use Averaging.

Topology / objectiveMerge nodesMean test accuracy (%)
Flat163.03
Chain951.70
EvoTM: mean accuracy365.35
EvoTM: node penalty164.99
  • Penalizing merge nodes reduces the selected tree from 3 nodes to 1.

C3. Where are new nodes introduced?

Split samples an internal node with k≥3k \geq 3 children, including the root, and groups 2 to k−1k-1 of its children under a new internal child of that node, e.g., (A,B,C,D)→((A,B),C,D)(A,B,C,D)\rightarrow((A,B),C,D). Regroup rebuilds a selected subtree and replaces it at the same position. For leaf insertion in Restore, Duplicate, and Relocate, a target node is sampled from the tree, including the root. If the target is an internal node with available capacity, the leaf is added as its child; otherwise, a new binary parent is inserted above the target, with the target subtree and the leaf as its children. If the target is the root, this parent becomes the new root. Candidates violating the active search-space constraints are rejected. We have clarified these insertion locations in Section 4.2 and Appendix.

C4. Symbol definitions.

Thank you for pointing this out. In the revised manuscript, we define amax⁡a_{\max} as the maximum allowed number of children per internal node when it first appears in Eq. (4).

C5. Val vs. Test accuracy.

Yes, validation accuracy is used to evaluate and select candidate trees during search. For each task, we divide the original test set into fixed, disjoint validation and held-out test subsets. All methods use the same held-out test subset. Test accuracy is evaluated only after tree selection and never guides search. Repeated selection based on validation accuracy can make the selected tree's validation score optimistic; differences in sample composition between the two subsets may also contribute to the gap.

I thank the authors for answering my questions and better highlighting limitations of their work in the manuscript. In the revised manuscript, particularly in Table, 4 Flat Averaging significantly outperforms more complex methods (including TA, TIES and Fisher), which contradicts what is shown in the rest of the literature. How would the authors explain this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.