- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes EvoTM, a genetic-programming method that searches how expert models are grouped and recursively merged. They use tree structure to model the merging process. Leaves represent experts and internal nodes apply a fixed merging operator. EvoTM searches these trees using tournament selection, subtree crossover, and mutations for grouping, expert selection, and reuse. Experiments cover three ViT backbones and Qwen2.5-1.5B LoRA experts evaluated on eight source-disjoint target tasks. The paper reports improvements over fixed flat merging across several operators.
Critical strengths
- The paper includes useful controls experiments.
- The idea is interesing and the abstract / introduction is clear to understand.
Critical weaknesses
-
The approach appears to be a direct and trivial application of standard Genetic Programming, where leaves are naturally terminal symbols (expert models) and internal nodes are functions (merging operators). While the figures seem complex, the textual description omits formal mathematical formulations and merely repurposes existing evolutionary algorithms. It would be better for the authors to clarify the technical depth and originality of their method.
-
Regarding linear weighting methods in model merging, recursive averaging is mathematically equivalent to a flat weighted average (for methods like Averaging and Task Arithmetic). The authors have already demonstrated in Table 6 that this method is almost equivalent to carefully optimized flat merging methods, which seems to render the complex tree structure meaningless. Therefore, the authors should compare their method with carefully tuned flat coefficients and TA scaling under the same search budget in the main experiments, rather than relying on flat baselines with fixed coefficients.
-
Figure 7 shows that using a simple search metric like subspace distance already identifies the best TIES pairing 60% of the time, suggesting that employing a complex and costly evolutionary algorithm may not yield a high return on investment (ROI).
-
The computational cost remains unclear. Trees contain different numbers of internal merges, and recursive statistics-based or adaptive operators may require repeated feature collection or optimization. Moreover, the authors only conduct experiments on small models like ViT or Qwen-LoRA (LoRA is significantly smaller than a normal backbone). The authors should conduct scaling experiments and report wall-clock time, GPU-hours, operator executions, and intermediate-state memory.
-
It is confusing that the performance change substantially between the Table 2 and Table 5 experiments, which raises serious concerns regarding the soundness of the experiments.
| Operator | Table 2 Val | Table 2 Test | Table 5 Flat |
|---|---|---|---|
| Task Arithmetic | 23.99 | 21.40 | 34.16 |
| TSV-M | 36.32 | 32.53 | 28.61 |
- There is a severe scarcity of experiments and supporting tabular data (only three tables are in the main text, although many are in the appendix, they lack clear logic for the reader). There is also a lack of valid baselines and unfair comparisons. The authors should compare EvoTM with existing evolutionary merging methods like EvoGM [1] and Evolutionary Optimization of Model Merging Recipes [2]. Alternatively, they should compare it with other search methods under the same initial population and evaluation budget.
[1] EvoGM: Learning to Merge LLMs via Evolutionary Generative Optimization, ICML 2026
[2] Evolutionary Optimization of Model Merging Recipes, Nature Machine Intelligence
Questions for authors
See Weakness
5 comments
Response to Reviewer fHMq
W4. Computational cost and scaling.
Regarding model scale, our evaluation includes full-weight merging of ViT-B/32, ViT-B/16, and ViT-L/14 experts. In particular, Appendix A.4.2, Table 12 reports merging 20 fully fine-tuned ViT-L/14 experts, each with approximately 303.2M vision-encoder parameters. These experiments merge complete vision encoders. To measure scaling under a controlled workload, we additionally evaluate EvoTM with Averaging on the same 10 tasks, using 10 experts per architecture. Each run evaluates 352 candidate trees on one NVIDIA A100 GPU.
| Backbone | Params./expert (M) | Runtime (h) |
|---|---|---|
| ViT-B/32 | 87.5 | 0.331 |
| ViT-B/16 | 85.8 | 0.936 |
| ViT-L/14 | 303.2 | 3.740 |
We also added accuracy–runtime comparisons for merging 20 ViT-B/32 experts, covering Flat, four other fixed structures, and EvoTM M0–M3. The table below summarizes M3 results for five operators; the revised manuscript provides the complete comparisons and visualizations (acc-runtime trade-off plots).
| Operator | Test accuracy (%) | Gain over Flat (pp) | Total runtime (h) |
|---|---|---|---|
| Averaging | 61.38 | +1.12 | 0.682 |
| TA | 61.74 | +26.08 | 0.623 |
| TIES | 61.75 | +12.08 | 3.825 |
| Fisher | 61.58 | +3.30 | 4.820 |
| RegMean | 62.28 | +1.88 | 2.282 |
- Results are averaged over five seeds, with one A100 GPU per run.
We acknowledge that structural search can require substantial additional computation, and its practical value depends on the operator, accuracy gain, and available budget. The comparisons in W2 further place this cost alongside other evolutionary merging approaches. All reported timings use a single GPU for consistent comparison; candidate trees within each generation could be evaluated in parallel across multiple GPUs to reduce wall-clock time.
W5. Inconsistent results in Tables 2 and 5.
We thank the reviewer for identifying this discrepancy. Both experiments use the same Qwen base model and ten LoRA experts, but their protocols differ. The original Table 2 merged LoRA factors and separately and used the original dev/test splits. Table 5 (Table 6 in the revision) merged reconstructed updates, , and evaluated a randomly sampled 50% subset of the original test set to reduce the cost of the preliminary topology study. Factor-wise and update-space merging are not equivalent, so the discrepancy cannot be attributed solely to evaluation data. We apologize for not clearly distinguishing these protocols.
We have rerun Flat and all M0–M3 searches using dense-update merging, retaining Table 2’s original dev/test splits, evaluator, and search budgets. Each internal node merges dense updates, and the root update is added to the pretrained model. Tree selection uses dev data exclusively; the selected models are evaluated on the full original test set. Table 2 has been replaced with these results.
Within both the original and revised Table 2, Flat and EvoTM share the same merging representation and evaluation protocol. The revised tables now share the merging representation, but their test sets still differ; their absolute scores should therefore not be compared directly. We have explicitly stated these distinctions in the experimental setup.
W6. Baselines and presentation.
We would like to clarify that our primary comparison concerns merging structure under a fixed operator, as explained in W1. EvoGM [1] optimizes merging coefficients, while Merging Recipes [2] optimizes merging hyperparameters and layer-level composition. These methods aim to introduce new merging operators; while our work examines a complementary dimension: how merging operations are organized. We compare flat merging with fixed tree structures, investigate their operator-dependent effects, and introduce EvoTM to optimize topology, expert selection, and reuse.
For additive operators, structural search induces effective coefficient changes, making direct weight optimization particularly relevant. We have therefore added comparisons with EvoGM and Merging Recipes, as reported in W2. We also compare GP with random and greedy structural search under matched candidate-evaluation budgets; representative accuracy–runtime comparisons are presented in Section 5.3, Figure 7.
I appreciate the authors’ detailed response and additional experiments. The new baselines and runtime measurements are helpful. However, I still have three main concerns:
- I remain unconvinced that the methodological novelty is sufficient. The structural analysis is interesting, but the additional methodological contribution remains limited.
- The reported search takes several hours on an A100. I therefore remain concerned about the practical accuracy–cost trade-off, particularly as model size and merging complexity increase.
- Unclear benefit of GP over simpler structural search. W6 mentions comparisons with random and greedy search in the revised manuscript, but the response does not include their numerical results.
Response to Reviewer fHMq
We thank the reviewer for the detailed assessment and address each point below.
W1. Technical contribution and originality.
We respectfully disagree that EvoTM’s contribution is limited to applying established GP operations. As illustrated in Figure 1, we formulate merging structure as an explicit optimization problem: with the candidate expert pool, merging operator, and its hyperparameters fixed, how should experts be grouped into intermediate models, which experts should participate, and whether they should be reused?
This formulation is grounded in the analysis in Section 3. Controlled experiments show that structural effects differ across operators and identify their sources: implicit reweighting and repeated scaling for additive operators, and path-dependent intermediate transformations for nonlinear operators. The quartet experiments further isolate grouping effects while holding tree shape and nominal leaf weights fixed. These findings explain when and why merging structure matters.
We represent these choices as complete, variable-size multi-ary trees. Internal structure specifies hierarchical grouping, while leaf identities and multiplicities encode expert selection and reuse. Eqs. (2), (4), and (5) formalize recursive execution, feasibility constraints, and the optimization objective. GP provides a practical way to search this discrete, combinatorial space.
We acknowledge that tree encoding and generic GP operations are established techniques. Our contribution lies in the systematic analysis of structural effects and the unified formulation for optimizing these choices.
W2. Does equivalence to flat weighting make tree search redundant, and how does it compare with tuned flat merging?
Explaining how structure affects different merging operators is precisely the purpose of Section 3. For Averaging and TA, the effect arises from implicit reweighting and repeated scaling, yielding equivalent flat coefficients. Table 7 confirms this explanation: controlling effective weights or repeated scaling largely removes the differences across structures, as intended in these experiments.
However, these mechanisms explain only part of the structural effects. In TIES, grouping affects intermediate trimming and sign resolution; in TSV-M, it affects subspace construction and truncation. These intermediate transformations can change the final result even with fixed operator hyperparameters and nominal leaf weights. Structural effects therefore cannot all be attributed to reweighting and repeated scaling.
We agree that direct coefficient optimization is an essential comparison for additive operators. We compare EvoTM with Averaging and TA against EvoGM [1] and modelwise/blockwise Recipe+TA [2] on the same 20 experts and tasks. These methods provide comparable approaches to optimizing effective expert contributions: EvoTM determines coefficients implicitly through tree structures, while the baselines optimize them explicitly.
W3. Does the heuristic result in Figure 7 justify replacing evolutionary search?
We agree that the 60% result is informative: subspace distance identifies the best TIES pairing in 12 of 20 sampled quartets, compared with 33.3% for random choice. However, Figure 7 (Figure 8 in the revision) tests a local decision among three pairings of four fixed experts under a fixed depth-2 balanced tree. We evaluated several metrics motivated by different merging mechanisms across all operators (such as sign conflict, Gram distance, and subspace distance for TIES, RegMean, and TSV-M, respectively), but each metric was evaluated for every operator. Thus, the 60% is one result from these comparisons, not the performance of a defined rule for selecting metrics and constructing trees.
Constructing a full merge tree requires additional decisions about grouping and how to handle intermediate models, which become inputs to subsequent merges. EvoTM also searches topology, expert selection, and reuse, all held fixed in the quartet experiment. Thus, Figure 8 in the revised manuscript motivates using static metrics to guide search, but cannot establish that they achieve comparable final performance or better computational return.