- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates training-free heterogeneous LLM merging through dimensional adaptation and weighted model averaging. The proposed approach uses truncation or expansion to align parameter dimensions before interpolation. Experiments across Qwen, Llama, and Gemma model families examine merging performance and the effects of interpolation ratios on different tasks.
Critical strengths
- The paper addresses a clear research question with straightforward presentation. Dimensional adaptation followed by weighted interpolation provides a simple and useful baseline for studying heterogeneous model merging.
- The evaluation covers multiple model families and 16 benchmarks, including fixed-ratio results, perturbation controls, and comparisons with other merging methods. The results demonstrate that simple weight interpolation can improve performance for certain model pairs.
Critical weaknesses
-
The dependence on interpolation ratios deserves further investigation. The Potential values emphasized in the main tables are obtained by selecting the best interpolation ratio separately for each task, rather than evaluating a single merged checkpoint. Although Appendix C reports fixed-ratio results, including configurations that achieve genuine improvements, their gains are generally smaller than Potential. Selecting a fixed ratio using held-out validation data and evaluating its test performance would help clarify the improvements achievable in practice.
-
The architectural compatibility requirements could be clarified further. Although the experiments cover Qwen, Llama, and Gemma, merging is primarily performed between models of different scales within the same family. Appendix D also acknowledges that dimensional adaptation relies on architectural compatibility. Clarifying the specific requirements of Truncate and Expand, and discussing their applicability to other heterogeneous architectures, would better establish the scope of the approach.
Questions for authors
- If a fixed interpolation ratio is selected using held-out validation data, how does its test performance compare with the reported Potential values?
- What specific architectural compatibility conditions are required by Truncate and Expand, and how do these conditions affect the applicability of the approach?
2 comments
Additional cross-family result: Llama-3.3-70B-Instruct Qwen2.5-3B
Thank you again for highlighting practical ratio selection and cross-family applicability. In addition to the fixed-ratio results in our rebuttal, we completed the task-wise IM Potential sweep for Llama-3.3-70B-Instruct Qwen2.5-3B. The table reports the corresponding Qwen2.5-3B baseline from the same evaluation:
| Dataset | 3B baseline | Cross-family IM Potential |
|---|---|---|
| GSM8K | 65.66 | 65.0 |
| MATH-500 | 38.40 | 38.6 |
| HumanEval+ | 43.90 | 37.2 |
| MBPP+ | 58.99 | 60.6 |
| IFEval | 40.41 | 41.1 |
| BoolQ | 70.80 | 74.3 |
| COPA | 78.00 | 81.0 |
| MNLI | 45.98 | 49.4 |
| PIQA | 75.79 | 75.9 |
| QNLI | 52.86 | 61.2 |
| RTE | 64.26 | 81.2 |
| WinoGrande | 63.38 | 64.2 |
| ARC | 34.42 | 39.0 |
| HellaSwag | 50.94 | 67.6 |
| MMLU | 44.96 | 45.3 |
| BBH | 44.63 | 46.9 |
| Average | 54.59 | 58.0 |
The cross-family Potential improves the matched average by 3.41 points and improves 14 of 16 tasks. We emphasize that this is a task-wise oracle envelope, not a single deployable checkpoint; its purpose is to quantify cross-family headroom. The positive fixed-ratio result already reported in our rebuttal separately shows that cross-family transfer is attainable with a single checkpoint.
Together with the validation-selected and compatibility results in our rebuttal, this evidence shows that same-family membership is not necessary for positive transfer, while architectural compatibility alone does not guarantee it. If these results address your concerns, we would appreciate an updated assessment; otherwise, we would be grateful for any remaining question about the method's scope.
From Oracle Potential to Practical Gains: Validation-Based Selection and Cross-Family Compatibility
Response to Reviewer z8bw: Validation-Based Ratio Selection and Architectural Compatibility
We thank the reviewer for these questions. We conducted held-out validation experiments, completed a cross-family evaluation on all 16 benchmarks, and will clarify the requirements of Truncate and Expand.
R3.1. Performance after validation-based ratio selection
We split each task once into 30% validation and 70% test data. All ratios used the same split, and test data were never used for selection. We selected one global ratio using the unweighted mean over the 16 validation tasks.
Validation selected μ=0.02. We evaluated this single checkpoint on held-out test data and also tested task-aware selection, where each task uses only its validation data to select a checkpoint.
The validation-selected global checkpoint gains +1.10, comparable to the paper's best fixed-checkpoint gain of +0.95. Task-wise validation selection gains +2.09, approaching the +2.57 oracle gain represented by Potential. Because the new results use the 70% test split while the paper uses the full set, we compare gains over the corresponding 3B baselines, not absolute scores across sets.
Global validation selection produces one deployable checkpoint; task-wise validation selection uses task identity but not test performance; Potential uses test performance and is therefore an oracle upper bound. We will clarify this distinction.
R3.2. Compatibility requirements and cross-family applicability
Our compatibility rule is selective, recipient-anchored parameter injection. One model is the donor and the other is the recipient. We apply three rules:
For Truncate, the donor tensor must be no smaller along the adapted dimensions and reducible to the recipient shape along predefined architectural axes. For Expand, the donor tensor is injected into the mapped part of the recipient tensor, while the remainder retains recipient values. Neither operation uses semantic alignment or learned matching.
Shape compatibility alone is insufficient. Components with incompatible parameterizations or scales are excluded. For example, Gemma and Qwen RMSNorm parameters have markedly different scales, so we retain the recipient's normalization parameters.
To test whether the method is restricted to within-family merging, we completed all 16 original benchmarks for a 2×2 donor-recipient design.
The positive Llama-to-Qwen result shows that same-family membership is not necessary for positive transfer; the absence of an average Llama-to-Llama gain shows that it is not sufficient. Cross-family merging is applicable when enough components admit the required mappings; incompatible components are skipped.
Compatibility makes injection well-defined but does not guarantee improvement. With few compatible components, fewer donor parameters are transferred. Outcomes also depend on the recipient, tasks, and ratio. Gains are more consistent with Qwen2.5-3B as recipient here, but the tested pairs do not establish the cause. We will state these requirements and limitations explicitly.