- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses cross-task interference in data-free model merging and proposes Gated Task Arithmetic (GTA). Unlike existing methods that directly manipulate task vector parameters, GTA introduces a task-preserving and interference-suppressing gate (PSI-Gate) on the input side for each task vector, which selectively preserves input representations of the target task and suppresses those of other tasks to reduce merging interference at the input information flow level. To enable data-free training, the paper derives surrogate input representations from the singular value decomposition of task vectors and designs a corresponding PSI surrogate loss to optimize the gates. Theoretically, it proves that finite-step optimization initialized from the identity matrix induces a preservation-oriented spectral bias, and its limiting convergent solution corresponds to the existing closed-form merging method ACTMat. Experiments are conducted across ViT vision models, LLaMA and Qwen2 language models (covering classification and reasoning tasks) with various model scales. GTA delivers consistent performance improvements, with task recovery rate improved by more than 16 percentage points over existing methods in the best case. Multiple ablation studies verify the interference suppression effect, task preservation capability, and hyperparameter robustness of the gating mechanism.
Critical strengths
-
The core idea is clean, novel, and practically appealing. Instead of only manipulating aggregated task vectors, GTA controls the input information flow supplied to each individual task vector via task-specific input gates. The method is fully data-free, and since the learned gates are absorbed into the task vector parameters, it introduces no extra inference cost, no input-dependent routing, and no additional parameters at deployment. This makes it a practically attractive approach that advances the design paradigm of data-free model merging.
-
The work combines solid theoretical analysis with strong empirical validation. Theorem 2 nicely frames finite-step gate optimization from identity initialization as preservation-biased, and formally establishes the connection between GTA and the closed-form ACTMat method as the K \to \infty limit, which deepens the understanding of data-free merging mechanisms. Empirically, GTA achieves consistent performance gains across ViT, LLaMA, and Qwen2 architectures covering both vision and language tasks, outperforming a wide range of baselines including TA, TIES, TSV, WUDI, and ACTMat.
Critical weaknesses
-
Related work and baseline coverage are incomplete. The paper discusses TSV, Subspace Boosting, WUDI, and ACTMat, but omits highly relevant methods: Isotropic Merging (Marczak et al., 2025), which explicitly handles common vs. task-specific subspaces in a closely related manner, and CAT Merging (Luo et al., 2023), which also leverages activation and task information to improve task arithmetic. Including these baselines and discussing their relation to GTA would clarify the paper’s positioning and better contextualize its improvements.
-
The surrogate input derivation relies on SGD assumptions (Theorem 1), while all fine-tuned models in experiments use AdamW. The paper acknowledges this approximation gap but does not quantify how much it degrades the quality of surrogate inputs or affects final merging performance. This weakens the link between the theoretical motivation and the empirical results.
Questions for authors
- Why were Isotropic Merging and CAT Merging excluded from the baseline comparison? Given their close conceptual relevance to GTA, adding these results would significantly strengthen the paper’s positioning and help readers assess the magnitude of the proposed improvements.
- Can the authors quantify the approximation gap between SGD-derived surrogate inputs and the actual input representations under AdamW fine-tuning? Have you run any experiments with SGD fine-tuned models to verify whether GTA performance aligns more closely with theoretical predictions in that setting?
4 comments
Response to Reviewer 7kVw (2/4)
Relation to CAT Merging.
CAT Merging (Sun et al., 2025) shares the objective of preserving task-specific information while reducing interference. It collects layer inputs through forward passes on a small number of unlabeled exemplars per task and uses these features to identify and trim conflict-prone task-vector components. Although CAT Merging requires no additional training, it uses task data during merging. We therefore discuss it as related work rather than include it in our data-free baseline comparison.
Standard GTA constructs surrogate inputs and learns PSI-Gates solely from model checkpoints, without accessing task data or fine-tuning trajectory information during merging. Actual activations are used only for diagnostic analyses and the activation-span projection ablation (Appendix A.3).
We thank the reviewer again for encouraging these comparisons. The added results strengthen the empirical evaluation of GTA, while the discussion clarifies its relationship to Isotropic Merging and the different data requirements of CAT Merging.
ㅤ
W2 and Q2: Surrogate Approximation under AdamW and SGD Fine-Tuning
We thank the reviewer for highlighting the gap between the original SGD-based analysis and AdamW fine-tuning, and for suggesting an SGD comparison. We have extended Theorem 1 to AdamW with row-shared adaptive scaling, introduced Approximation 1 for general entrywise AdamW with zero weight decay, and added quantitative analyses of activation-span representability, merging accuracy, and task-specific input alignment.
Controlled Adam(W) experiments yield mean numerical projection errors of 0.97–5.02%. For the original ViT-B/32 checkpoints, projecting the surrogates onto sampled activation spans changes mean merging accuracy by at most 0.13 percentage points. We also observe higher average alignment with same-task than other-task input statistics (Section 3.3; Appendices A.2–A.3, Tables 4–7).
Theorem 1 and Approximation 1 under AdamW.
Let be the compact SVD of a layer's task vector, and let collect input activations along the fine-tuning trajectory.
Under the stated linear-layer gradient assumptions, zero-initialized moments, zero weight decay, and row-shared adaptive scaling , the extended Theorem 1 establishes
for an appropriate coefficient matrix . Plain SGD is recovered with and (Section 3.3, Eq. (4); Appendix A.2).
For general entrywise AdamW, Lemma 1 instead establishes the residual-inclusive representation
where the residuals arise from weight decay and deviations from row-shared adaptive scaling (Appendix A.2, Eqs. (23)–(26)).
Our LLaMA and Qwen2 checkpoints use AdamW with zero weight decay, so , while may remain nonzero. The FusionBench ViT checkpoints use Adam. Thus, the exact row-shared condition is not generally satisfied by the practical optimizers.
For AdamW-trained checkpoints with zero weight decay, Approximation 1 assumes approximate representability within the training-path activation span:
where denotes the Moore–Penrose pseudoinverse (Section 3.3, Eq. (7); Appendix A.3.1, Eqs. (63)–(65)).
Let be the orthogonal projector onto this span. Under zero weight decay, Lemma 1 gives
Thus, Approximation 1 requires only the component of outside the training-path activation span to be small, rather than the entire adaptive-scaling residual (Appendix A.3, Eqs. (60)–(61)).
Response to Reviewer 7kVw (3/4)
Projection errors under controlled fine-tuning.
We compare SGD with Adam on ViT-B/32 and with AdamW on LLaMA-3.2-1B, using 100 training samples per task across eight tasks (seed 42).
Each task is fine-tuned separately with an effective batch size of 100. One or five passes over the same samples yield or optimizer updates, respectively. All runs use FP32 precision, a learning rate of , zero weight decay, and no warmup or learning-rate schedule.
Token-level input activations collected before each update form . Let contain the orthonormal singular directions retained by the numerical rank criterion. We measure
and average equally across the evaluated weight matrices. The rank criterion and numerical projection procedure are specified (Appendix A.3.1, Eqs. (67)–(70)).
| Model | Updates | SGD | Adam(W) |
|---|---|---|---|
| ViT-B/32 | 1 | 0.1003 | 1.1704 |
| ViT-B/32 | 5 | 0.0703 | 0.9694 |
| LLaMA-3.2-1B | 1 | 0.2873 | 5.0244 |
| LLaMA-3.2-1B | 5 | 0.2422 | 1.3851 |
Table 3. Mean relative numerical projection error (%) onto training-path activation spans. Adam(W) denotes Adam for ViT-B/32 and AdamW for LLaMA-3.2-1B. Results are reproduced from the revised manuscript (Appendix A.3.1, Table 4).
SGD yields smaller mean projection errors in all four comparisons, while Adam(W) yields errors of 0.97–5.02% (Appendix A.3.1, Table 4).
Because the retained numerical span is contained in the full activation span, upper-bounds the minimum relative representation error within the full span. Numerical truncation can produce nonzero errors even under SGD, where exact representability holds under the theorem's assumptions (Appendix A.3.1, Eqs. (65)–(71)).
These results support approximate activation-span representability under Adam-family fine-tuning in the evaluated controlled settings. This experiment measures representation errors; it does not compare final merging accuracy using fully fine-tuned SGD and Adam(W) experts.
Effect of activation-span projection on merging accuracy.
We next use the original fine-tuned ViT-B/32 checkpoints and collect input activations from 100 sampled training examples per task. These activations come from fixed fine-tuned checkpoints, rather than the fine-tuning trajectory.
Let contain an orthonormal basis for the retained numerical activation span. We construct
Here, is defined using sampled checkpoint activations and is distinct from the optimizer-induced and the training-path projection residual (Appendix A.3.2, Eqs. (75)–(78)).
For each task-count setting, we independently learn PSI-Gates using either the original surrogate or its projected counterpart , keeping the remaining GTA settings unchanged.
| Tasks | (%) | Original | Projected | (pp) |
|---|---|---|---|---|
| 8 | 1.0562 | 86.51 | 86.63 | |
| 14 | 1.0078 | 81.60 | 81.73 | |
| 20 | 1.1176 | 77.39 | 77.38 |
Table 4. Mean relative projection residual norm across evaluated tasks and weight matrices, and mean GTA accuracy (%) on ViT-B/32. Original and Projected use and , respectively. denotes projected minus original accuracy in percentage points. Results are reproduced from the revised manuscript (Appendix A.3.2, Table 6).
The mean relative residual norm is approximately 1% in all three settings. Using the projected surrogate changes mean accuracy by , , and percentage points for 8, 14, and 20 tasks, respectively (Appendix A.3.2, Table 6).
These results indicate limited sensitivity to removing components outside the sampled activation spans in the evaluated configurations. They support using the original surrogate without additional activation-span projection during merging.
Response to Reviewer 7kVw (4/4)
Alignment with actual checkpoint inputs.
We further examine whether checkpoint-derived surrogates contain input information more closely associated with their corresponding tasks than with other tasks.
Using 100 training examples per task (seed 42), we collect mean-pooled input activations at six representative attention and MLP weight matrices per model. ViT activations are pooled over vision tokens, including the CLS token. LLM activations are pooled over non-padding tokens from prompts without reference answers.
We compare the uncentered second-order statistics of surrogate and actual inputs using Frobenius cosine similarity. Same-task alignment pairs a surrogate with inputs from its corresponding task; other-task alignment pairs it with inputs from different tasks. Results are averaged over the corresponding task pairs and evaluated weight matrices (Appendix A.3.3, Eqs. (79)–(82)).
The mean same-task versus other-task similarities are 0.1508 versus 0.0572 for ViT-B/32, 0.2017 versus 0.1480 for LLaMA-3.2-1B, and 0.1268 versus 0.0902 for Qwen2-7B (Appendix A.3.3, Table 7).
The higher average same-task alignment across all three models provides additional evidence of task-dependent input information in , although the absolute alignment remains limited.
Together, the controlled projection-error measurements, activation-span projection ablation, and input-alignment analysis provide complementary empirical support for Approximation 1.
Actual activations are used only for diagnostic analyses and the projection ablation. Standard GTA constructs surrogates and learns PSI-Gates from checkpoint-derived , without accessing task data, fine-tuning trajectory information, or computing activation-span projections (Appendix A.3.4).
We thank the reviewer again for motivating these analyses. The revisions clarify the exact theoretical conditions and the scope of the practical approximation, while providing quantitative empirical support for the surrogate in the evaluated Adam-family fine-tuning settings.
ㅤ
We thank the reviewer again for the constructive feedback. The additional comparisons strengthen GTA's empirical evaluation and clarify its relationship to related merging methods. The revised theoretical analysis and quantitative experiments distinguish the conditions for exact input-activation representability from the practical approximation, and provide complementary empirical support for checkpoint-derived surrogates in the evaluated fine-tuning settings.
Response to Reviewer 7kVw (1/4)
We thank the reviewer for the careful assessment and constructive suggestions on baseline coverage and surrogate validity. In response, we have added comparisons with Iso-C and Iso-CTS (Appendix D, Tables 13–17) and clarified GTA's relationship to CAT Merging, including their different data requirements.
We have also extended Theorem 1 to AdamW with zero weight decay and row-shared adaptive scaling, and introduced Approximation 1 for general entrywise scaling under zero weight decay. To examine the practical approximation, we provide controlled projection-error measurements under SGD and Adam(W), an activation-span projection ablation on merging accuracy, and an analysis of task-specific input alignment (Section 3.3; Appendices A.2–A.3, Tables 4–7). We address each concern below.
ㅤ
We thank the reviewer for suggesting comparisons with Isotropic Merging and CAT Merging. We have added comparisons with Iso-C and Iso-CTS alongside TSV and Subspace Boosting (SB), using the same checkpoints and evaluation settings as GTA (Appendix D, Tables 13–17). We also discuss GTA's relationship to CAT Merging below. For Iso-CTS, the common-space ratio is fixed at 0.8 across all vision and language settings.
Comparison on vision tasks.
We evaluate ViT-B/32, ViT-B/16, and ViT-L/14 on 8, 14, and 20 tasks using the FusionBench checkpoints. For Iso-C and Iso-CTS, we use the merging coefficients reported in the original Isotropic Merging paper (its Appendix D.3, Table 5), without additional tuning.
Table 1. Mean accuracy (%) on vision tasks using the same FusionBench checkpoints and evaluation settings. Results are reproduced from the revised manuscript (Appendix D, Table 13).
GTA achieves the highest mean accuracy among the listed methods in all nine settings, exceeding the strongest listed baseline in each setting by 0.37–4.88 percentage points (Appendix D, Table 13).
Comparison on language tasks.
We further evaluate LLaMA-3.2-1B, LLaMA-3.2-3B, and LLaMA-3.1-8B on eight classification tasks, and Qwen2-7B on five reasoning tasks (Appendix D, Tables 14–17).
The original Isotropic Merging paper does not report merging coefficients for these specific language-model settings. We therefore use α=1 for both Iso-C and Iso-CTS without additional tuning.
Table 2. Mean accuracy (%) on eight classification tasks for LLaMA-3.2-1B/3B and LLaMA-3.1-8B, and mean exact match (EM, %) on five reasoning tasks for Qwen2-7B. Results are summarized from the revised manuscript (Appendix D, Tables 14–17).
GTA achieves the highest mean accuracy or EM among the listed methods in all four settings (Appendix D, Tables 14–17). These results characterize performance under the evaluated configurations; the Iso-C and Iso-CTS results use untuned coefficients and do not establish their performance after hyperparameter tuning.
Methodologically, Isotropic Merging flattens singular-value spectra in a common subspace, with Iso-CTS additionally incorporating task-specific directions. GTA instead learns task-specific PSI-Gates using task-vector-derived surrogate inputs Ht=VtΣt. Its PSI surrogate loss balances task preservation and interference suppression, and the gates are optimized for a finite number of Adam steps from identity initialization.