acceptodds
Under review as a conference paper at ICLR 2027

SMAT: Simple and Efficient Merge-Aware Training

Abstract

Model merging integrates the capabilities of multiple experts without joint re-training, but standard expert training optimizes task loss alone and does not guarantee good performance after merging. Merge-aware training (MAT) aims to improve merged performance, but existing methods do not fully account for common merging operations and add training cost. We observe that, from an expert's perspective, common merging methods can be described by three operations: Scale reweights its own update, Mask removes selected coordinates, and Perturb adds updates from other experts. Based on this view, we introduce SMAT (Simple MAT), which jointly optimizes expert loss and expected loss at simulated merged parameters generated by sampling scaling coefficients, masks, and additive noise. We further introduce periodic scheduling, kernel fusion, and parameter storage switching to make SMAT efficient, with one forward and one backward pass per step. Across four language and vision-language backbones, SMAT improves the mean score across five merging methods by 1.07–2.16 points over the strongest baseline for each backbone, with less than 2% training-time overhead over standard fine-tuning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merge, Merge-Aware Training, Task Vector

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_meYROpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SMAT, a merge-aware training (MAT) method for experts that will later be merged. The authors observe that, from one expert's point of view, common data-free mergers (Task Arithmetic, TIES, DARE, DELLA, weight averaging) apply three operations to its update: Scale (reweighting its own task vector), Mask (dropping selected coordinates), and Perturb (adding the other experts' updates). SMAT trains each expert on a mix of its ordinary task loss and the expected loss at simulated merged parameters, obtained by sampling a random scaling coefficient, a DARE-style rescaled Bernoulli mask on attention/MLP weights, and isotropic uniform noise as a stand-in for the other experts.

Critical strengths
  1. The method is simple, and its Scale and Mask components match the mergers it targets. The Scale/Mask/Perturb view (Eq. 1, 3) is easy to understand. Each component is ablated. The gains are largest for WA on CLIP, which fits Scale training for small own-coefficients (αmin=0.1\alpha_{min}=0.1 vs. WA's 1/81/8). This explanation is plausible and a practitioner could reuse it.
Critical weaknesses
  1. Perturb is a generic flatness regularizer rather than a simulation of merging, and flatness-based merge-aware training already exists. A real merge adds structured, non-zero-mean updates from other experts, but SMAT replaces them with i.i.d. uniform noise (l.195-214). By the paper's own expansion (App. C, Eq. 14), this penalizes tr⁡Hi\operatorname{tr}H_i, and the authors attribute the gains to loss smoothing (l.460-463). That flatter experts merge better is the premise of SAFT (Lee et al., 2025), which is the ASAM baseline here. Fig. 3 (uniform ≈\approx Gaussian ≈\approx Laplace) suggests that the structure of other experts' updates does not matter, and Prop. 3 leaves exactly this mismatch unbounded (l.790-793). Perturb is also the dominant component (−2.10 in Table 4; Perturb-only already beats MergOPT), while Scale and Mask together add about 1 point. SMAT therefore reads as a cheaper flatness regularizer rather than a new merge-aware principle, and its speed advantage comes mostly from periodic scheduling, which the baselines could also use (Q3).

  2. Hyperparameter selection and statistical support are unclear. SMAT is very sensitive to σ\sigma: Fig. 3 shows that σ=2×10−3\sigma=2\times10^{-3} is a sharp optimum, and at 4×10−34\times10^{-3} Perturb-only falls below FT. The paper does not say how (αmin,σ)(\alpha_{min},\sigma) or the baselines' hyperparameters (Table 5) were chosen, or which split Fig. 3 uses; if it is the eval split, σ\sigma was tuned on test data. Tables 1-4 report no SD, although the LLM gains over the strongest baseline are only about 1.1 points and the SDs in Fig. 3 reach 0.5-1 point. Because the main claim rests on margins of 1.07-2.16 points, this concern weighs heavily in my rating.

Questions for authors
  1. How were (αmin,σ)(\alpha_{min},\sigma), pp, tt, and the baseline hyperparameters in Table 5 selected? Was it on the dev split or the final eval split, and with what grid size per method? Specifically, was Fig. 3 computed on dev or on the released eval split?
  2. Please report SD over the 3 seeds for every entry of Tables 1-4, and state whether FT and the ablation variants also use 3 seeds. Paired tests over seeds, or a per-task breakdown for the LLM margins over OrthoReg, would help. Relatedly, what does the "Without Perturb" reference line in Fig. 3 (about 43) correspond to, given that FT is 42.02 (Table 1) and w/o Perturb is 43.67 (Table 4)?
  3. Sec. 5.2 (l.376) says Appendix E describes "the software difference affecting ASAM timing", but I could not find this in App. E. What is it, and how much does it change ASAM's 1.7-2.9x?
8 comments
AuthorsOpenReview ↗

Response to meYR (1/5): Method Contribution and Merge-Specific Components

Thank you for carefully reviewing our work and for your suggestions. We respond to your questions point by point below.


W-1a: Merge-aware contribution and flatness

Perturb is a generic flatness regularizer rather than a simulation of merging, and flatness-based merge-aware training already exists. […] Perturb is also the dominant component […], while Scale and Mask together add about 1 point.

Thank you for this discussion. We explain SMAT's contribution in terms of performance and efficiency, theoretical understanding, and the concrete benefits of simulating merging operations.

1. SMAT provides effective merge-aware training at a cost close to standard fine-tuning. In Tables 1 and 2, it improves over the strongest baseline by 1.07–2.16 points with only 0.2%–1.9% training-time overhead. We also apply the same periodic scheduling, fused sampling, and parameter storage switching to MergOPT and measure mean step time on C-STANCE after warmup. For 1B, accelerated MergOPT and SMAT take 97.61 and 97.88 ms, with merging averages of 43.09 and 45.77. For 8B, their times are 266.19 and 268.35 ms, with scores of 55.77 and 58.49. With engineering optimizations matched, SMAT still improves by 2.68 and 2.72 points at comparable speed. SMAT thus makes merge-robust training practical at near-FT cost while delivering consistent performance gains.


2. We characterize how merging operations shape robustness training and explain why simple random perturbations can be effective. SAFT (Lee et al., 2025, §4, §5, and Appendix D) connects merge-induced parameter offsets to ASAM and explains approximate joint-task loss linearity through dominant Hessian eigenvalues. Our Appendix C analyzes the complete Scale–Mask–Perturb state and characterizes the variation introduced by each operation. Its covariance is:

\underbrace{v_\alpha\Delta_i\Delta_i^\top}_{\mathrm{Scale}} +\underbrace{s_\alpha\operatorname{Diag} \left(\Delta_{i,k}^{2}\frac{p_k}{1-p_k}\right)}_{\mathrm{Mask}} +\underbrace{\sigma^2I}_{\mathrm{Perturb}},$$ Here $v_\alpha=\operatorname{Var}(\alpha_i)$ and $s_\alpha=\mathbb E[\alpha_i^2]$. Under the local smoothness conditions in Appendix C, the simulated loss has the expansion: $$\mathbb E\mathcal L_i(\widetilde\theta_i) \approx\mathcal L_i(\bar\theta_i) +\frac12\operatorname{tr}\left[H_i(\bar\theta_i)C_i\right], \qquad \bar\theta_i=\theta_0+\mathbb E[\alpha_i]\Delta_i.$$ This identifies the robustness requirements associated with the three components: **Scale trains tolerance to changes along the expert's own task vector. Mask trains tolerance to missing task-update coordinates. Perturb extends training to unknown external change directions.** The complete SMAT objective therefore incorporates directional structure associated with merging mechanisms. Loss smoothing explains the source of robustness, while merging operations determine which parameter changes training should accommodate. For Perturb, Appendix C further shows that, under smoothness and small-perturbation conditions, zero-mean uniform, Gaussian, and Laplace noise with the same covariance share the same second-order smoothing term. This explains their similar trends in Fig. 3. In our additional experiments, real task-vector perturbations achieve a merging average of **45.44**, SMAT uniform perturbations achieve **45.77**, and Laplace perturbations fitted to the empirical FT task-vector distribution achieve only **41.41**. These results show that **effective merge robustness does not require precisely fitting the distribution of other experts' task vectors: simple uniform perturbations combined with explicit simulation of merging operations can achieve competitive performance.** --- **3. Scale and Mask provide merger-specific benefits that an average across mergers can obscure.** In the Llama-1B AdamW ablation evaluation, averaging across seven tasks for each merger gives the following differences (ablated variant minus Full SMAT, in score points): | Merger | w/o Scale | w/o Mask | |---|---:|---:| | WA | −2.42 | +1.16 | | TA | +0.29 | −1.24 | | TIES | −2.07 | −2.35 | | DARE | +0.20 | −1.38 | | DELLA | −2.25 | −1.54 | | **Overall average** | **−1.25** | **−1.07** | Removing Scale decreases WA, TIES, and DELLA by **2.42, 2.07, and 2.25 points**. Removing Mask decreases TIES by **2.35 points**, and TA, DARE, and DELLA by **1.24, 1.38, and 1.54 points**. Different mergers introduce different parameter changes, and Scale and Mask improve experts' ability to accommodate these changes. An average benefit of approximately one point therefore includes improvements exceeding two points for specific mergers, demonstrating the practical value of training for scaling and coordinate removal.
AuthorsOpenReview ↗

Response to meYR (2/5): Periodic Scheduling and Accelerated Baselines


W-1b: Periodic scheduling and accelerating baselines

[…] its speed advantage comes mostly from periodic scheduling, which the baselines could also use (Q3).

Periodic scheduling is the main source of acceleration and transfers to baselines. With acceleration matched, SMAT still offers a better performance–efficiency tradeoff. We add a progressive ablation of the acceleration components and apply the same acceleration scheme to MergOPT.

1. Periodic scheduling provides the largest speed gain, with kernel fusion and parameter storage switching further reducing overhead. On Llama-3.2-1B-Instruct with AdamW, progressively adding these components gives:

ConfigurationStep time (ms/update)Reduction from previous row
Base implementation, t=1t=1155.36—
+periodic scheduling, t=4t=4110.7428.72%
+kernel fusion98.2911.24%
+parameter storage switching97.880.42%

Periodic scheduling reduces simulation frequency and step time by 28.72%. Kernel fusion further reduces sampling and parameter-processing overhead, and parameter storage switching reduces weight copying. Together, the three components reduce step time from 155.36 to 97.88 ms, a 37.00% reduction.


2. The same acceleration brings MergOPT close to SMAT's speed, while SMAT retains higher merging performance. We retain MergOPT's original Laplace parameters and coefficient-sampling rules and add periodic scheduling (t=4t=4), fused sampling, and parameter storage switching. Step times below are means over 200 timed steps on C-STANCE after warmup:

ModelMethodStep time (ms/update)Merged average
1BOriginal MergOPT, t=1t=1145.9944.11
1BAccelerated MergOPT, t=4t=497.6143.09
1BSMAT, t=4t=497.8845.77
8BOriginal MergOPT, t=1t=1471.1457.16
8BAccelerated MergOPT, t=4t=4266.1955.77
8BSMAT, t=4t=4268.3558.49

Efficiency is measured on the same node and with matched hardware and software within each model, using one GPU for 1B and two for 8B. We warm up for 20 steps and time 200 C-STANCE steps, including data loading and optimizer updates. Merging averages come from complete TRACE seven-task training, dev-based parameter selection, and five-merger evaluation.

After acceleration, MergOPT's merging averages decrease from 44.11 to 43.09 on 1B and from 57.16 to 55.77 on 8B. At the same perturbation interval and approximately the same speed, SMAT still leads by 2.68 and 2.72 points. Thus, periodic scheduling and the accompanying optimizations provide a transferable acceleration scheme, while SMAT's merge-aware training design maintains higher merging performance under this low-overhead training scheme.

AuthorsOpenReview ↗

Response to meYR (3/5): Hyperparameter Selection and Data Splits


W-2/Q-1: Hyperparameter selection and data splits

How were (α_min, σ), p, t, and the baseline hyperparameters in Table 5 selected? […] Specifically, was Fig. 3 computed on dev or on the released eval split?

Both training and merging hyperparameters are selected on dev data, and Fig. 3 also reports preliminary dev-set results. We clarify the splits, search ranges, and selection procedure below.

1. All methods share an independent dev split. As described in §5.1 and Appendix A, approximately 50–200 examples per task are held out from training for dev evaluation. We deduplicate by normalized prompt or RGB-image identity so that training, dev, and final evaluation sets do not overlap. Final performance is reported on the released eval splits for language tasks and test sets for vision tasks.


2. We first select training hyperparameters, then select merging parameters for the resulting experts. FT uses a fixed base training configuration. For the three baselines, the candidate values are:

Model familyASAM: radius ρ\rhoMergOPT: Laplace scale bbOrthoReg: regularization weight λreg\lambda_{\rm reg}
Llama (1B, 8B)0.125, 0.25, 0.5, 1, 20.00025, 0.0005, 0.0010.003, 0.01, 0.02, 0.03, 0.05, 0.1
ViT (B/32, L/14)0.125, 0.25, 0.375, 0.50.0000125, 0.000025, 0.00005, 0.000075, 0.0001, 0.00020.01, 0.03, 0.05, 0.1, 0.2, 0.3, 1

For SMAT, we fix t=4t=4, initialize αmin⁡=0.2\alpha_{\min}=0.2, perturbation RMS σ=2×10−3\sigma=2\times10^{-3}, and p=0.5p=0.5, and search sequentially:

  • Perturbation strength: σ∈{1×10−3,2×10−3,5×10−3}\sigma\in\{1\times10^{-3},2\times10^{-3},5\times10^{-3}\}.
  • Scale lower bound: αmin⁡∈{0.1,0.2,0.3}\alpha_{\min}\in\{0.1,0.2,0.3\}.
  • Mask probability: p∈{0.1,0.2,0.5}p\in\{0.1,0.2,0.5\}.

At each stage, we hold the other parameters fixed, retain the value with the best dev performance, and proceed to the next parameter. Training hyperparameters are compared using mean dev performance across tasks with the TA coefficient fixed at 0.3. The final (αmin⁡,σ,p)(\alpha_{\min},\sigma,p) is (0.2,0.002,0.5)(0.2,0.002,0.5) for both language models and (0.1,0.001,0.5)(0.1,0.001,0.5) for both vision models.

After fixing training configurations, we calibrate merger sparsity as in Appendix A and share it within each model family. For each model and training method's expert set, TA, TIES, DARE, and DELLA then independently select γ∈{0.1,0.2,…,1.0}\gamma\in\{0.1,0.2,\ldots,1.0\} by the mean dev score across tasks. WA directly averages weights and requires no search.


3. Fig. 3 is a preliminary dev experiment examining perturbation distributions and scales. It compares three noise distributions and four RMS values for Perturb-only training on Llama-1B with AdamW, reporting the five-merger average on dev. This is a separate experiment from the sequential search for full SMAT above. The original caption did not explicitly identify the split. We will add “evaluated on the dev split” in the revision to distinguish these dev results from the final evaluation results in the main tables.

Reviewer_r3AkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents SMAT, a merge-aware training algorithm to improve downstream merging performance. Different from prior work, SMAT models merging as a combination of three operations: scale, mask and perturb. By simulating these operations into a separate merge loss during training, SMAT robustifies trained parameters to downstream merge. To maintain high efficiency, SMAT introduces periodic scheduling, kernel fusion and parameter storage switching. Thanks to periodic scheduling, SMAT only needs to apply the simulated merge loss every once in a while. Results on vision and language benchmarks show consistent improvements in accuracy over existing Merge-Aware-Training methods while capping overheads to at most 2% compared to standard fine-tuning.

Critical strengths

The paper's idea of modeling merging as three operations captures several merging methods while being simple at the same time. The simplicity of the method is a key strength.

The paper demonstrates that the improvements post merging not only come from accuracy retention after merging but also through maintaining expert quality in the first place. This is not direct / obvious and counts as another key strength.

Finally, periodic scheduling also counts as one of the strengths, allowing SMAT to skip evaluating the costly simulated merging loss at every step.

Critical weaknesses

The paper's contribution can be assessed on two fronts, the efficiency front and the accuracy front.

On the Efficiency Front. My main critic of the paper is that the design of SMAT involves an expensive memory read operation, whose criticality may only be uncovered at scale. Specifically, to compute θ~i\tilde\theta_i, it has to read the pre-trained weights θ0\theta_0 which are stored on CPU RAM. These have to be read before every simulated-loss update into the GPU buffer. Thus the efficiency of SMAT fully depends upon the PCIe bandwidth between the CPU and GPU, and the model size θ\theta. When evaluated for large models, the PCIe bandwidth will be further shared among multiple tensor parallel workers. The authors mention that the weights can be prefetched during ordinary steps, however, how much latency hiding happens depends on both the frequency and computation time of the ordinary steps.

Importantly, the baseline MergOPT incurs no such extra memory read. Thus it is unclear whether MergOPT will still be slower compared to SMAT if the kernel fusion and storage switching optimizations are enabled for it. Since these optimizations are fairly simple to execute, a fair comparison calls for enabling these optimizations for both baselines.

In the same light, MergOPT seems implementable via only a single copy of fine-tuned weights. The perturbations can be temporarily applied and reverted before the gradient update. Thus, while SMAT requires holding two copies of model weights, MergOPT may require only one. Since memory is a scarce resource during large-scale training, MergOPT may actually be more memory efficient than SMAT at comparable time efficiency (i.e. when the CPU-GPU transfers start becoming costly for SMAT at scale).

To conclude, the authors should demonstrate the efficiency gains of SMAT at scale (e.g. 70B models) and while enabling storage switching and kernel fusion optimizations for MergOPT. Furthermore, the evaluations should dissect the exact time spent in CPU-GPU transfers at this scale.

On the Accuracy Front. Compared to most efficient baseline, MergeOPT, the overall average accuracy gains of SMAT are within 1.3-2.5 percentage points. However, these relative gains seem to reduce as bigger models are used. For instance, the gains are only 1.3 points for LLama-3.1-8B. Similar to the efficiency evaluations, the paper might thus benefit from demonstrating if significant gains persist at larger model scales.

Questions for authors
  1. Why does the training time of MergOPT become drastically worse for the Llama evaluations (1.718x, 1.543x) compared to the ViT evaluations (1.146x, 1.025x)? Furthermore, why are the relative overheads of fine-tuning Llama-3.2-1B higher than fine-tuning Llama-3.2-8B for MergeOPT (1.718x vs. 1.543x)?
5 comments
AuthorsOpenReview ↗

Response to r3Ak (1/3): Base-Weight Transfer and Prefetching

Thank you for carefully reviewing our work and for your suggestions. We respond to your questions point by point below.


W-1: Base-weight transfer and prefetching overhead

[…] the design of SMAT involves an expensive memory read operation […]. […] how much latency hiding happens depends on both the frequency and computation time of the ordinary steps.

Thank you for highlighting the potential cost of base-weight transfer. We supplement our experiments with a prefetching comparison on Llama-3.1-8B-Instruct using 2, 4, and 8 GPUs, AdamW, and model parallelism that assigns consecutive layers to each GPU. Training hyperparameters follow the main experiment.

1. Periodic scheduling provides a sufficient prefetching window for base-weight transfer to finish before the simulation step. SMAT simulates merging every four steps. During the preceding three ordinary training steps, the frozen base weights can be prefetched from pinned CPU memory into a GPU buffer, overlapping transfer with training computation. We measure both transfer duration and the time the simulation step actually waits for transfer:

GPUsPrefetchingH2D duration (ms)Three ordinary steps (ms)Remaining wait (ms)Four-step cycle (ms)
2Off144.72707.15144.7291249.09
2On145.03718.740.0031113.81
4Off82.02826.3782.0331480.56
4On82.03824.030.0031396.22
8Off93.641116.2393.6512141.43
8On52.941100.360.0032055.74

H2D (Host-to-Device, from CPU memory to GPU memory) duration is accumulated separately on each GPU, after which we report the maximum cumulative duration across GPUs. Remaining wait is the time the simulation step waits for the base transfer to finish. The three ordinary steps and the four-step cycle include computation, queueing, and host submission gaps.

With prefetching, remaining wait falls to approximately 0.003 ms for all three GPU counts. The mean windows from when prefetching can begin to when the base is needed are approximately 637–638, 727–752, and 961–986 ms across GPUs, respectively, sufficient to cover transfer. The weights are therefore ready on the GPU when the simulation step needs them, avoiding the serial cost of waiting for transfer before simulation.


2. Prefetching reduces the complete training cycle and remains beneficial with multiple GPUs. The four-step cycles on 2, 4, and 8 GPUs decrease by 135.28, 84.34, and 85.69 ms, or 10.83%, 5.70%, and 4.00%, respectively. Although prefetching increases the three ordinary steps by 11.59 ms in the two-GPU setting, the complete cycle still decreases by 135.28 ms. The benefit of overlapping computation and transfer is thus reflected in measured training time.

We also measure transfer bandwidth with multiple GPUs. With eight GPUs, the range of per-GPU mean effective bandwidths is 21.30–38.99 GiB/s without prefetching and 40.66–51.50 GiB/s with it. The corresponding maximum cumulative H2D duration falls from 93.64 to 52.94 ms, explaining the reduced transfer duration in this setting.

AuthorsOpenReview ↗

Response to r3Ak (2/3): MergOPT with Matched Acceleration


W-2: Fair efficiency comparison with MergOPT

[…] it is unclear whether MergOPT will still be slower compared to SMAT if the kernel fusion and storage switching optimizations are enabled for it. […] MergOPT seems implementable via only a single copy of fine-tuned weights.

Thank you for this suggestion. We apply the same acceleration scheme to MergOPT and implement the proposed in-place perturbation and restoration. The acceleration transfers to MergOPT, while SMAT still achieves better merging performance at comparable training cost.

1. The same acceleration scheme brings MergOPT's training speed close to SMAT's. We add periodic scheduling (t=4t=4), fused sampling, and parameter storage switching to MergOPT while retaining its original Laplace parameters and coefficient-sampling rules. An alternative implementation directly adds noise to the current weights, subtracts it after the forward and backward passes, and then performs the optimizer update. Step times below are means over 200 timed steps on C-STANCE after warmup.

ModelMethodStep time (ms/update)Peak memory (GiB/GPU)Merged average
1BAccelerated MergOPT97.6115.5043.09
1BAccelerated MergOPT+in-place restoration98.3015.4942.51
1BSMAT97.8815.4945.77
8BAccelerated MergOPT266.1939.1155.77
8BAccelerated MergOPT+in-place restoration269.1239.1156.01
8BSMAT268.3539.1158.49

Experiments use Llama-3.2-1B-Instruct, Llama-3.1-8B-Instruct, and AdamW. All methods in the table use t=4t=4. Efficiency is measured on the same node and with matched hardware and software within each model: one GPU for 1B and two for 8B. On C-STANCE, we warm up for 20 steps and time 200 steps, including data loading and optimizer updates. Memory is the maximum per-GPU peak allocated memory during training. Merging averages come from complete seven-task training, dev-based parameter selection, and evaluation with five merging methods.


2. SMAT retains higher merging performance at approximately the same speed. Compared with MergOPT using the same acceleration scheme, SMAT improves by 2.68 and 2.72 points on 1B and 8B. In-place restoration also does not close this gap. With engineering optimizations matched, SMAT offers a better performance–efficiency tradeoff: it produces experts that merge better at a training speed close to standard fine-tuning.


3. In-place restoration removes the weight backup but does not reduce peak memory in this implementation. Restoring weights by subtracting noise requires retaining the complete noise tensor, which is the same size as the weights. Noise storage therefore replaces the weight backup. Measured peak memory is essentially identical for the two MergOPT implementations and SMAT: approximately 15.49 GiB/GPU for 1B and 39.11 GiB/GPU for 8B.

AuthorsOpenReview ↗

Response to r3Ak (3/3): 70B Results and Training-Time Ratios


W-3: Efficiency and performance on a larger model

[…] the authors should demonstrate the efficiency gains of SMAT at scale (e.g. 70B models) […]. […] demonstrating if significant gains persist at larger model scales.

Thank you for this suggestion. We add LoRA experiments with Llama-3.1-70B-Instruct on the seven TRACE tasks. SMAT continues to improve merging performance at 70B with a manageable training overhead.

Experiments use eight H800 80GB GPUs and AdamW, with LoRA rank 16, alpha 32, and dropout 0, targeting q/k/v/o_proj and gate/up/down_proj. Other training hyperparameters follow the main 8B configuration. MergOPT uses the accelerated periodic implementation with in-place restoration. Both MergOPT and SMAT use t=4t=4. Each method is trained once, merging parameters are selected using the original dev procedure, and performance is averaged over three decoding runs.

MethodExpertMerged averageStep time (s/update)Relative to FTPeak memory (GiB/GPU)
FT64.4453.521.7891.000×42.89
Accelerated MergOPT63.7354.541.9971.116×59.94
SMAT65.0655.422.0501.146×59.94

1. SMAT's merging gains extend to 70B while maintaining expert capability. Its average over five merging methods is 55.42, exceeding FT and accelerated MergOPT by 1.90 and 0.88 points, respectively. It outperforms MergOPT on four of five mergers, with gains of 1.06 and 3.01 points on DARE and DELLA. Its independent-expert score is also the highest at 65.06, showing that the merging improvement accompanies maintained expert quality.


2. In 70B LoRA training, SMAT achieves better merging performance at a cost close to accelerated MergOPT's. Its step time is 2.050 seconds, 14.6% above FT. Compared with accelerated MergOPT, it adds only 0.053 seconds per step, approximately 2.7%, with the same peak memory. Thus, SMAT obtains better merged results at comparable time and memory cost in this 70B setting.


Q-1: Differences in relative training time across models

Why does the training time of MergOPT become drastically worse for the Llama evaluations […] compared to the ViT evaluations […]? Furthermore, why are the relative overheads of fine-tuning Llama-3.2-1B higher than fine-tuning Llama-3.2-8B […]?

The time ratio depends on the fraction of base training time taken by additional perturbation operations. Absolute overhead can therefore increase with model size while the relative time ratio decreases. Appendix E.2 (Training-Time Measurement), Table 7 (Training time for complete expert sets) reports absolute training costs. Mean step times are obtained by dividing total training time across complete expert sets by their total update counts: seven tasks for language and eight for vision. Timing includes data loading, initial warmup, and logging. We convert these values to milliseconds and show MergOPT's additional time over FT:

ModelFT step time (ms)MergOPT step time (ms)Additional time (ms)Relative to FT
Llama-1B68.7118.049.31.718×
Llama-8B354.3546.9192.61.543×
ViT-B/32125.5143.818.31.146×
ViT-L/141449.41486.236.81.025×

1. Parameter-processing overhead and base training computation differ across models, changing the relative cost of perturbations. MergOPT adds noise sampling, perturbation application, and weight restoration at each step. Their cost depends on the number of parameters processed, whereas base training cost also depends on architecture, batch size, and input length. These costs do not grow proportionally. Our language models use full-parameter training with batch size 8, whereas vision experiments train only the CLIP vision encoder with batch size 128. Measured additional time is 49.3–192.6 ms for language models versus 18.3–36.8 ms for vision models, and it accounts for a larger fraction of base training time in the language experiments.


2. The lower ratio for 8B arises because base training time grows faster than additional overhead. From 1B to 8B, MergOPT's overhead increases from 49.3 to 192.6 ms, approximately 3.9-fold, whereas FT step time increases from 68.7 to 354.3 ms, approximately 5.2-fold. Consequently, relative overhead decreases from 71.8% to 54.3%. Appendix E.1 also specifies that 8B uses layerwise model parallelism across two GPUs and 1B uses one GPU. These times correspond to their actual training configurations.

Vision models show the same pattern. From ViT-B/32 to ViT-L/14, additional time approximately doubles, whereas base training time increases 11.5-fold, reducing relative overhead from 14.6% to 2.5%. The larger base training cost amortizes the additional parameter operations.

Reviewer_NAPfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents the simple merge‑aware training (SMAT) approach to produce experts that perform well after model merging. The authors observe mainstream merging algorithms can be decomposed into three primitive operations: Scale, Mask, Perturb. SMAT simulates these operations during each expert’s independent training, optimizing for both original‑task loss and expected loss under simulated merged parameters. Periodic scheduling, kernel fusion and parameter‑storage switching ensure SMAT imposes fewer training‑time overhead vs standard fine‑tuning. Evaluations span four backbones (Llama‑1B/8B, CLIP ViT‑B/32, ViT‑L/14), tested across five representative merging baselines. SMAT outperforms prior MAT methods (MergOPT, OrthoReg, ASAM) on merged‑model average scores by 1.07‑2.16 points. Analysis attributes gains to loss‑surface smoothing, creating broader low‑loss regions along merge‑relevant parameter directions; ablation validates each of Scale/Mask/Perturb contributes performance gains.

Critical strengths

The work provides a coherent formulation of merge-aware training by decomposing common parameter-merging operations into Scale, Mask, and Perturb. This formulation is intuitive, applicable to independently trained experts, and does not require access to other experts' checkpoints during training. In additional, the proposed implementation addresses a significant practical limitation of existing merge-aware training approaches: computational overhead. By combining periodic scheduling, kernel fusion, and parameter storage switching, SMAT requires only one forward and backward pass per training step and incurs approximately 0.2–1.9% additional training time over standard fine-tuning.

Critical weaknesses

A central premise of SMAT is that randomly sampled scaling coefficients, masks, and independent additive noise adequately approximate the parameter changes introduced by real model merging. However, actual merging operations are often structured, data-dependent, and correlated across parameters and experts. In contrast, SMAT uses a scalar scaling coefficient, independently sampled coordinate masks, and isotropic uniform noise. Although Appendix C provides a theoretical characterization of expected-loss differences through distributional moments, this analysis does not establish that the sampled perturbation distribution closely approximates actual merged parameter states. Similarly, the loss-landscape experiments demonstrate robustness along selected merging directions but do not directly quantify the correspondence between simulated and actual merging distributions.

Questions for authors
  1. Could the authors provide a quantitative comparison between simulated and actual merge-induced parameter changes, for example using update norms, coordinate correlations, or directional similarity? Additionally, would sampling perturbations based on empirical task-vector statistics produce meaningfully different results from the current independent uniform-noise formulation?
  2. All SMAT hyperparameters are manually set per‑model‑family. Could you describe a practical validation‑driven hyperparameter selection workflow for new unseen model/task suites?
4 comments
AuthorsOpenReview ↗

Response to NAPf (1/3): Real Task-Vector Perturbations

Thank you for carefully reviewing our work and for your suggestions. We respond to your questions point by point below.


W-1/Q-1: Merge simulation and empirical perturbations

Could the authors provide a quantitative comparison between simulated and actual merge-induced parameter changes […]? Additionally, would sampling perturbations based on empirical task-vector statistics produce meaningfully different results […]?

Thank you for this suggestion. We supplement our study with real task-vector perturbations, perturbations fitted to empirical statistics, and quantitative comparisons between simulated and actual merge changes. We address this question from three perspectives.

1. Without accessing other experts, SMAT achieves merging performance comparable to training with real task-vector perturbations.

On Llama-3.2-1B-Instruct with AdamW and the seven TRACE tasks, we use a frozen FT expert bank to construct pi=0.3∑j≠i(θj−θ0)p_i=0.3\sum_{j\ne i}(\theta_j-\theta_0) for task ii. This control preserves the directions, signs, and coordinate structure of the other experts' task vectors. All other training settings are identical to SMAT.

TrainingWATATIESDAREDELLAOverall average
FT39.88 ± 0.2842.64 ± 0.6042.93 ± 0.5543.50 ± 1.0141.17 ± 0.4642.02 ± 0.49
SMAT w/ real task-vector perturbations40.86 ± 0.5748.41 ± 0.8548.23 ± 0.1547.60 ± 0.4942.09 ± 2.7245.44 ± 0.74
SMAT w/ uniform perturbations42.59 ± 0.1946.11 ± 0.7247.17 ± 0.5746.90 ± 0.6346.09 ± 0.4345.77 ± 0.39

Results are means ± sample standard deviations over three independent training runs. Real task-vector perturbations and SMAT improve over FT by 3.42 and 3.75 points, respectively, achieving comparable overall performance. Real task-vector perturbations perform better with TA, TIES, and DARE, whereas SMAT performs better with WA and DELLA.

The directional analysis is shown in Appendix F, Fig. 7. The left panel compares the cosine similarity between pip_i and the actual merge change rmi=θm−θir_{mi}=\theta_m-\theta_i for each task. The right panel decomposes the actual change sequentially into the direction of pip_i, the component of the expert's own task vector orthogonal to pip_i, and the remaining directions, reporting squared-norm shares averaged over seven tasks.

Real task-vector perturbations are more targeted to TA, TIES, and DARE because the offsets adapted to during training align more closely with the changes these methods introduce. The squared-norm shares of actual merge changes along pip_i are 76.8%, 63.3%, and 45.1% for TA, TIES, and DARE, respectively, all higher than for WA and DELLA. Training with a fixed pip_i prepares experts for prominent change directions in these methods, leading to better merged performance with them.

SMAT repeatedly resamples uniform perturbations, exposing experts to a broader range of change directions and helping improve WA and DELLA. The main changes introduced by these methods depart from the fixed pip_i. WA reduces the coefficient of the expert's own task vector from 1 to 1/71/7: the component along its own task vector orthogonal to pip_i accounts for 80.1% of the change, while the component along pip_i accounts for only 16.4%. For DELLA, 58.7% lies outside the plane spanned by pip_i and the expert's own task vector. Both training variants retain Scale and Mask. By resampling additive noise, SMAT exposes training to offsets beyond the fixed pip_i, making experts better prepared for merge changes that depart from that direction.

AuthorsOpenReview ↗

Response to NAPf (2/3): Robustness to Unknown Merge Changes


W-1/Q-1: Merge simulation and empirical perturbations (continued)

2. Actual merge changes are diverse. SMAT aims to improve tolerance to unknown merge changes.

Starting from the current expert parameters, we compare the complete SMAT change s=θ~i−θis=\widetilde\theta_i-\theta_i with the actual merge change r=θm−θir=\theta_m-\theta_i, computing cosine similarity on one million sampled parameter coordinates. We also randomly select two pairs of two-task combinations and compare their TA update directions relative to the pretrained base:

ComparisonCosine similarity
SMAT seven-task change vs actual seven-task TA change0.0467
Actual two-task TA perturbation A vs B0.0363
Actual two-task TA perturbation C vs D0.0985

A, B, C, and D denote the TA updates of the following task combinations relative to the base. For a task set SS, the update is uS=θTA,S−θ0=γ∑j∈S(θj−θ0)u_S=\theta_{\mathrm{TA},S}-\theta_0=\gamma\sum_{j\in S}(\theta_j-\theta_0), where γ\gamma is the TA merging coefficient:

  • A: {C-STANCE, 20Minuten}, and B: {NumGLUE-cm, NumGLUE-ds}.
  • C: {C-STANCE, NumGLUE-ds}, and D: {NumGLUE-cm, 20Minuten}.

These comparisons show that:

  • Actual merge changes from different task combinations themselves have low directional similarity. The two pairs have cosine similarities of only 0.0363 and 0.0985, demonstrating the diversity of directions introduced by actual merging.
  • The cosine similarity of 0.0467 between SMAT and actual TA changes is of the same order as the similarity between different actual changes. Low directional similarity is therefore also observed between actual merge changes, rather than being specific to simulated changes.

During independent expert training, the task vectors of other experts are unknown. SMAT therefore optimizes the expected loss over a variety of simulated parameter states:

=(1-\lambda)\mathcal L_i(\theta_i) +\lambda\mathbb E_{\alpha_i,m_i,\epsilon_i} \left[\mathcal L_i(\widetilde\theta_i)\right], \qquad \widetilde\theta_i =\theta_0+\alpha_i m_i\odot\Delta_i+\epsilon_i.$$ Conditioning on Scale and Mask, let $\theta_\star$ denote the simulated parameters before adding noise. For zero-mean uniform perturbations with covariance $\sigma^2I$, the smoothness and small-perturbation conditions in Appendix C give: $$\mathbb E_\epsilon\mathcal L_i(\theta_\star+\epsilon) \approx\mathcal L_i(\theta_\star) +\frac{\sigma^2}{2}\operatorname{tr}H_i(\theta_\star).$$ **Uniform perturbation training encourages a broader low-loss region around an expert, making unknown merge changes less likely to cause a large increase in task loss.** The expansion shows that optimizing the expected perturbed loss constrains both the task loss and the total local curvature. Repeatedly sampling different directions trains the expert to perform well around its parameters. When an unknown merge moves the parameters in a direction not specified in advance, the expert is consequently more likely to remain in a low-loss region and retain its task capability. SMAT thus improves tolerance to unknown merge changes by reducing local loss sensitivity.
AuthorsOpenReview ↗

Response to NAPf (3/3): Empirical Perturbations and Hyperparameter Search


W-1/Q-1: Merge simulation and empirical perturbations (continued)

3. The empirical distribution and its fitted scale do not yield the best training performance.

Empirical distribution and scale calibration. Based on task-vector statistics, MergOPT (Yang et al., 2026, §3.2.3 and Appendix D.1.1) uses a Laplace distribution to describe the empirical task-vector distribution. Our analysis of FT task vectors yields a consistent observation. We therefore adopt this distribution and fit its zero-centered scale using the mean absolute deviation of task vectors from our FT expert bank, obtaining b≈1.77×10−4b\approx1.77\times10^{-4} and a perturbation RMS of approximately 2.50×10−42.50\times10^{-4}.

Results using the empirical distribution and fitted scale directly. On Llama-3.2-1B-Instruct with AdamW and seven TRACE tasks, we change only Perturb and retain the other main-experiment settings. Means over three independent training runs are:

TrainingOverall average
FT42.02
Empirical Laplace41.41
SMAT uniform perturbations45.77

Empirical Laplace scores 41.41, below FT (42.02) and SMAT uniform perturbations (45.77). Fitting the task-vector distribution does not directly improve merging performance.

Fig. 3 further shows that a larger RMS improves Laplace's training performance, while uniform noise achieves a slightly higher best score. Our preliminary Perturb-only experiments on the dev set also examine Laplace noise and compare it with uniform and Gaussian noise at matched RMS:

Perturb RMSUniformGaussianLaplace
0.000542.6443.6843.04
0.00143.9044.0343.90
0.00244.9244.8144.68
0.00438.8538.5639.50

Laplace scores 43.90–44.68 at RMS 1×10−31\times10^{-3} to 2×10−32\times10^{-3}, approximately 4–8 times the fitted RMS. Its best score is 44.68 at RMS 2×10−32\times10^{-3}, where uniform noise scores 44.92, or 0.24 points higher. Thus, neither the distributional form nor the scale fitted from empirical task vectors is optimal for merge-aware training.


Q-2:Validation-driven hyperparameter selection for new models and task suites

All SMAT hyperparameters are manually set per‑model‑family. Could you describe a practical validation‑driven hyperparameter selection workflow for new unseen model/task suites?

We clarify our selection procedure: first select each training method's hyperparameters on dev data, then select merging coefficients for the resulting experts. We hold out approximately 50–200 examples per task from the training data for dev evaluation and deduplicate by prompt or image identity so that training, dev, and final evaluation sets do not overlap. All methods use the same splits.

Training hyperparameter search. FT uses a fixed base training configuration. For the three baselines, we search the following parameters:

Model familyASAM: radius ρ\rhoMergOPT: Laplace scale bbOrthoReg: regularization weight λreg\lambda_{\rm reg}
Llama (1B, 8B)0.125, 0.25, 0.5, 1, 20.00025, 0.0005, 0.0010.003, 0.01, 0.02, 0.03, 0.05, 0.1
ViT (B/32, L/14)0.125, 0.25, 0.375, 0.50.0000125, 0.000025, 0.00005, 0.000075, 0.0001, 0.00020.01, 0.03, 0.05, 0.1, 0.2, 0.3, 1

SMAT uses sequential hyperparameter search. Starting from αmin⁡=0.2\alpha_{\min}=0.2, perturbation RMS σ=2×10−3\sigma=2\times10^{-3}, and p=0.5p=0.5, we fix t=4t=4 and search in this order:

  1. Perturbation strength: σ∈{1×10−3,2×10−3,5×10−3}\sigma\in\{1\times10^{-3},2\times10^{-3},5\times10^{-3}\}.
  2. Scale lower bound: αmin⁡∈{0.1,0.2,0.3}\alpha_{\min}\in\{0.1,0.2,0.3\}.
  3. Mask probability: p∈{0.1,0.2,0.5}p\in\{0.1,0.2,0.5\}.

At each stage, we hold the other parameters fixed, retain the value with the best dev performance, and proceed to the next parameter. Training hyperparameters are selected by TA performance on dev with the merging coefficient fixed at 0.3.

Merging parameter search: calibrate sparsity first, then select merging coefficients.

MergerParameterCandidatesSelected value
WANoneDirect averaging—
TIESRetention0.1, 0.2, 0.3, 0.50.5 for language and vision
DAREDrop probability0.1, 0.3, 0.5, 0.7, 0.90.3 for language and vision
DELLADrop probabilitySame as above0.5 for language, 0.1 for vision

The selected sparsity settings are shared across training methods and model sizes within each family. For each model and training method's expert set, TA, TIES, DARE, and DELLA then independently select γ∈{0.1,0.2,…,1.0}\gamma\in\{0.1,0.2,\ldots,1.0\} by the mean dev score across tasks. WA directly averages weights and requires no search.

References

Enneng Yang, Qun Yang, Peng Wang, Anke Tang, Guibing Guo, Xiaochun Cao, and Li Shen. MergOPT: A Merge-Aware Optimizer for Robust Model Merging. ICLR, 2026.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Thank you again for reviewing our work. We have posted our point-by-point responses and additional experimental results under your review. As the discussion period is approaching its end, we would appreciate any feedback on whether our responses address your concerns, or any remaining questions we can clarify.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Key Clarifications and Additional Results

We thank the reviewers and the Area Chair for their careful assessment and suggestions. Our point-by-point responses provide additional experiments and analyses. We summarize the key findings below.


1. Effective merge-robust training does not require precisely fitting other experts' task-vector distributions. Without accessing other experts during training, SMAT achieves a merging average of 45.77, comparable to 45.44 when training with real task-vector perturbations. Our theoretical analysis further characterizes the parameter changes associated with Scale, Mask, and Perturb and their roles in training. Component ablations show that Scale and Mask can each provide gains exceeding 2 points for specific merging methods, demonstrating the value of explicitly simulating merging operations.

2. SMAT retains a better performance–efficiency tradeoff with matched acceleration, and its gains extend to 70B. Applying the same acceleration scheme to MergOPT brings its training speed close to SMAT's, while SMAT still leads by 2.68 and 2.72 points on 1B and 8B, respectively. In the additional 70B LoRA experiments, SMAT improves over FT and accelerated MergOPT by 1.90 and 0.88 points, respectively, further supporting its effectiveness on a larger model.

3. The main experimental gains are consistent across training runs. We report means and sample standard deviations over three independent training runs for the original main tables and component ablations. For Llama-1B with AdamW and ViT-B/32 with Adam, SMAT outperforms OrthoReg under all three paired training seeds. Per-task analyses also show improvements on a majority of tasks in both settings.


Complete experimental settings, results, and analyses are provided in our responses under each review. The revised paper adds Appendix F, Table 8, and Fig. 7, presenting the performance comparison with real task-vector perturbations and the directional analysis. We welcome any remaining questions or requests for clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.