- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SMAT, a merge-aware training (MAT) method for experts that will later be merged. The authors observe that, from one expert's point of view, common data-free mergers (Task Arithmetic, TIES, DARE, DELLA, weight averaging) apply three operations to its update: Scale (reweighting its own task vector), Mask (dropping selected coordinates), and Perturb (adding the other experts' updates). SMAT trains each expert on a mix of its ordinary task loss and the expected loss at simulated merged parameters, obtained by sampling a random scaling coefficient, a DARE-style rescaled Bernoulli mask on attention/MLP weights, and isotropic uniform noise as a stand-in for the other experts.
Critical strengths
- The method is simple, and its Scale and Mask components match the mergers it targets. The Scale/Mask/Perturb view (Eq. 1, 3) is easy to understand. Each component is ablated. The gains are largest for WA on CLIP, which fits Scale training for small own-coefficients ( vs. WA's ). This explanation is plausible and a practitioner could reuse it.
Critical weaknesses
-
Perturb is a generic flatness regularizer rather than a simulation of merging, and flatness-based merge-aware training already exists. A real merge adds structured, non-zero-mean updates from other experts, but SMAT replaces them with i.i.d. uniform noise (l.195-214). By the paper's own expansion (App. C, Eq. 14), this penalizes , and the authors attribute the gains to loss smoothing (l.460-463). That flatter experts merge better is the premise of SAFT (Lee et al., 2025), which is the ASAM baseline here. Fig. 3 (uniform Gaussian Laplace) suggests that the structure of other experts' updates does not matter, and Prop. 3 leaves exactly this mismatch unbounded (l.790-793). Perturb is also the dominant component (−2.10 in Table 4; Perturb-only already beats MergOPT), while Scale and Mask together add about 1 point. SMAT therefore reads as a cheaper flatness regularizer rather than a new merge-aware principle, and its speed advantage comes mostly from periodic scheduling, which the baselines could also use (Q3).
-
Hyperparameter selection and statistical support are unclear. SMAT is very sensitive to : Fig. 3 shows that is a sharp optimum, and at Perturb-only falls below FT. The paper does not say how or the baselines' hyperparameters (Table 5) were chosen, or which split Fig. 3 uses; if it is the eval split, was tuned on test data. Tables 1-4 report no SD, although the LLM gains over the strongest baseline are only about 1.1 points and the SDs in Fig. 3 reach 0.5-1 point. Because the main claim rests on margins of 1.07-2.16 points, this concern weighs heavily in my rating.
Questions for authors
- How were , , , and the baseline hyperparameters in Table 5 selected? Was it on the dev split or the final eval split, and with what grid size per method? Specifically, was Fig. 3 computed on dev or on the released eval split?
- Please report SD over the 3 seeds for every entry of Tables 1-4, and state whether FT and the ablation variants also use 3 seeds. Paired tests over seeds, or a per-task breakdown for the LLM margins over OrthoReg, would help. Relatedly, what does the "Without Perturb" reference line in Fig. 3 (about 43) correspond to, given that FT is 42.02 (Table 1) and w/o Perturb is 43.67 (Table 4)?
- Sec. 5.2 (l.376) says Appendix E describes "the software difference affecting ASAM timing", but I could not find this in App. E. What is it, and how much does it change ASAM's 1.7-2.9x?
8 comments
Response to meYR (2/5): Periodic Scheduling and Accelerated Baselines
W-1b: Periodic scheduling and accelerating baselines
[…] its speed advantage comes mostly from periodic scheduling, which the baselines could also use (Q3).
Periodic scheduling is the main source of acceleration and transfers to baselines. With acceleration matched, SMAT still offers a better performance–efficiency tradeoff. We add a progressive ablation of the acceleration components and apply the same acceleration scheme to MergOPT.
1. Periodic scheduling provides the largest speed gain, with kernel fusion and parameter storage switching further reducing overhead. On Llama-3.2-1B-Instruct with AdamW, progressively adding these components gives:
| Configuration | Step time (ms/update) | Reduction from previous row |
|---|---|---|
| Base implementation, | 155.36 | — |
| +periodic scheduling, | 110.74 | 28.72% |
| +kernel fusion | 98.29 | 11.24% |
| +parameter storage switching | 97.88 | 0.42% |
Periodic scheduling reduces simulation frequency and step time by 28.72%. Kernel fusion further reduces sampling and parameter-processing overhead, and parameter storage switching reduces weight copying. Together, the three components reduce step time from 155.36 to 97.88 ms, a 37.00% reduction.
2. The same acceleration brings MergOPT close to SMAT's speed, while SMAT retains higher merging performance. We retain MergOPT's original Laplace parameters and coefficient-sampling rules and add periodic scheduling (), fused sampling, and parameter storage switching. Step times below are means over 200 timed steps on C-STANCE after warmup:
| Model | Method | Step time (ms/update) | Merged average |
|---|---|---|---|
| 1B | Original MergOPT, | 145.99 | 44.11 |
| 1B | Accelerated MergOPT, | 97.61 | 43.09 |
| 1B | SMAT, | 97.88 | 45.77 |
| 8B | Original MergOPT, | 471.14 | 57.16 |
| 8B | Accelerated MergOPT, | 266.19 | 55.77 |
| 8B | SMAT, | 268.35 | 58.49 |
Efficiency is measured on the same node and with matched hardware and software within each model, using one GPU for 1B and two for 8B. We warm up for 20 steps and time 200 C-STANCE steps, including data loading and optimizer updates. Merging averages come from complete TRACE seven-task training, dev-based parameter selection, and five-merger evaluation.
After acceleration, MergOPT's merging averages decrease from 44.11 to 43.09 on 1B and from 57.16 to 55.77 on 8B. At the same perturbation interval and approximately the same speed, SMAT still leads by 2.68 and 2.72 points. Thus, periodic scheduling and the accompanying optimizations provide a transferable acceleration scheme, while SMAT's merge-aware training design maintains higher merging performance under this low-overhead training scheme.
Response to meYR (3/5): Hyperparameter Selection and Data Splits
W-2/Q-1: Hyperparameter selection and data splits
How were (α_min, σ), p, t, and the baseline hyperparameters in Table 5 selected? […] Specifically, was Fig. 3 computed on dev or on the released eval split?
Both training and merging hyperparameters are selected on dev data, and Fig. 3 also reports preliminary dev-set results. We clarify the splits, search ranges, and selection procedure below.
1. All methods share an independent dev split. As described in §5.1 and Appendix A, approximately 50–200 examples per task are held out from training for dev evaluation. We deduplicate by normalized prompt or RGB-image identity so that training, dev, and final evaluation sets do not overlap. Final performance is reported on the released eval splits for language tasks and test sets for vision tasks.
2. We first select training hyperparameters, then select merging parameters for the resulting experts. FT uses a fixed base training configuration. For the three baselines, the candidate values are:
| Model family | ASAM: radius | MergOPT: Laplace scale | OrthoReg: regularization weight |
|---|---|---|---|
| Llama (1B, 8B) | 0.125, 0.25, 0.5, 1, 2 | 0.00025, 0.0005, 0.001 | 0.003, 0.01, 0.02, 0.03, 0.05, 0.1 |
| ViT (B/32, L/14) | 0.125, 0.25, 0.375, 0.5 | 0.0000125, 0.000025, 0.00005, 0.000075, 0.0001, 0.0002 | 0.01, 0.03, 0.05, 0.1, 0.2, 0.3, 1 |
For SMAT, we fix , initialize , perturbation RMS , and , and search sequentially:
- Perturbation strength: .
- Scale lower bound: .
- Mask probability: .
At each stage, we hold the other parameters fixed, retain the value with the best dev performance, and proceed to the next parameter. Training hyperparameters are compared using mean dev performance across tasks with the TA coefficient fixed at 0.3. The final is for both language models and for both vision models.
After fixing training configurations, we calibrate merger sparsity as in Appendix A and share it within each model family. For each model and training method's expert set, TA, TIES, DARE, and DELLA then independently select by the mean dev score across tasks. WA directly averages weights and requires no search.
3. Fig. 3 is a preliminary dev experiment examining perturbation distributions and scales. It compares three noise distributions and four RMS values for Perturb-only training on Llama-1B with AdamW, reporting the five-merger average on dev. This is a separate experiment from the sequential search for full SMAT above. The original caption did not explicitly identify the split. We will add “evaluated on the dev split” in the revision to distinguish these dev results from the final evaluation results in the main tables.
Response to meYR (1/5): Method Contribution and Merge-Specific Components
Thank you for carefully reviewing our work and for your suggestions. We respond to your questions point by point below.
W-1a: Merge-aware contribution and flatness
Thank you for this discussion. We explain SMAT's contribution in terms of performance and efficiency, theoretical understanding, and the concrete benefits of simulating merging operations.
1. SMAT provides effective merge-aware training at a cost close to standard fine-tuning. In Tables 1 and 2, it improves over the strongest baseline by 1.07–2.16 points with only 0.2%–1.9% training-time overhead. We also apply the same periodic scheduling, fused sampling, and parameter storage switching to MergOPT and measure mean step time on C-STANCE after warmup. For 1B, accelerated MergOPT and SMAT take 97.61 and 97.88 ms, with merging averages of 43.09 and 45.77. For 8B, their times are 266.19 and 268.35 ms, with scores of 55.77 and 58.49. With engineering optimizations matched, SMAT still improves by 2.68 and 2.72 points at comparable speed. SMAT thus makes merge-robust training practical at near-FT cost while delivering consistent performance gains.
2. We characterize how merging operations shape robustness training and explain why simple random perturbations can be effective. SAFT (Lee et al., 2025, §4, §5, and Appendix D) connects merge-induced parameter offsets to ASAM and explains approximate joint-task loss linearity through dominant Hessian eigenvalues. Our Appendix C analyzes the complete Scale–Mask–Perturb state and characterizes the variation introduced by each operation. Its covariance is:
\underbrace{v_\alpha\Delta_i\Delta_i^\top}_{\mathrm{Scale}} +\underbrace{s_\alpha\operatorname{Diag} \left(\Delta_{i,k}^{2}\frac{p_k}{1-p_k}\right)}_{\mathrm{Mask}} +\underbrace{\sigma^2I}_{\mathrm{Perturb}},$$ Here $v_\alpha=\operatorname{Var}(\alpha_i)$ and $s_\alpha=\mathbb E[\alpha_i^2]$. Under the local smoothness conditions in Appendix C, the simulated loss has the expansion: $$\mathbb E\mathcal L_i(\widetilde\theta_i) \approx\mathcal L_i(\bar\theta_i) +\frac12\operatorname{tr}\left[H_i(\bar\theta_i)C_i\right], \qquad \bar\theta_i=\theta_0+\mathbb E[\alpha_i]\Delta_i.$$ This identifies the robustness requirements associated with the three components: **Scale trains tolerance to changes along the expert's own task vector. Mask trains tolerance to missing task-update coordinates. Perturb extends training to unknown external change directions.** The complete SMAT objective therefore incorporates directional structure associated with merging mechanisms. Loss smoothing explains the source of robustness, while merging operations determine which parameter changes training should accommodate. For Perturb, Appendix C further shows that, under smoothness and small-perturbation conditions, zero-mean uniform, Gaussian, and Laplace noise with the same covariance share the same second-order smoothing term. This explains their similar trends in Fig. 3. In our additional experiments, real task-vector perturbations achieve a merging average of **45.44**, SMAT uniform perturbations achieve **45.77**, and Laplace perturbations fitted to the empirical FT task-vector distribution achieve only **41.41**. These results show that **effective merge robustness does not require precisely fitting the distribution of other experts' task vectors: simple uniform perturbations combined with explicit simulation of merging operations can achieve competitive performance.** --- **3. Scale and Mask provide merger-specific benefits that an average across mergers can obscure.** In the Llama-1B AdamW ablation evaluation, averaging across seven tasks for each merger gives the following differences (ablated variant minus Full SMAT, in score points): | Merger | w/o Scale | w/o Mask | |---|---:|---:| | WA | −2.42 | +1.16 | | TA | +0.29 | −1.24 | | TIES | −2.07 | −2.35 | | DARE | +0.20 | −1.38 | | DELLA | −2.25 | −1.54 | | **Overall average** | **−1.25** | **−1.07** | Removing Scale decreases WA, TIES, and DELLA by **2.42, 2.07, and 2.25 points**. Removing Mask decreases TIES by **2.35 points**, and TA, DARE, and DELLA by **1.24, 1.38, and 1.54 points**. Different mergers introduce different parameter changes, and Scale and Mask improve experts' ability to accommodate these changes. An average benefit of approximately one point therefore includes improvements exceeding two points for specific mergers, demonstrating the practical value of training for scaling and coordinate removal.