acceptodds
Under review as a conference paper at ICLR 2027

Gated Task Arithmetic: Controlling Task-Specific Input Information Flow in Model Merging

Abstract

Model merging has emerged as an effective approach for consolidating multiple task-specific fine-tuned models into a single unified model. Among existing approaches, data-free merging is particularly attractive as it does not require access to the original training data. A central challenge in model merging is to preserve task-specific knowledge while mitigating interference among task vectors. However, existing merging methods mainly manipulate task vectors themselves, without explicitly controlling the input representations on which they operate. After merging, each task vector may therefore operate on input representations associated with other tasks, potentially introducing cross-task interference. In this work, we propose **Gated Task Arithmetic (GTA)**, a data-free model merging method that introduces a task-specific input gate for each task vector. GTA constructs a gated task vector as and performs task arithmetic over the resulting gated task vectors. We design as a **task-preserving and interference-suppressing gate (PSI-Gate)**, which preserves input representations associated with its corresponding task while suppressing those associated with other tasks. To learn the PSI-Gates without the original training data, we introduce a **task-preserving and interference-suppressing surrogate loss (PSI surrogate loss)** using task-vector-derived surrogate input representations. Under task-vector Gram covariance estimates and the stated convergence conditions, finite-step gate optimization from identity induces a preservation-oriented spectral bias, while its limiting solution corresponds to closed-form activation-aware merging. We extensively evaluate GTA across ViT models with varying numbers of vision tasks, as well as LLaMA and Qwen2 models fine-tuned on diverse natural language tasks. GTA consistently achieves strong merging performance across different model architectures, scales, and modalities, improving the recovery rate of task-vector performance by more than **16 percentage points** over existing merging methods in the best case.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging, data free, merging

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_7kVwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses cross-task interference in data-free model merging and proposes Gated Task Arithmetic (GTA). Unlike existing methods that directly manipulate task vector parameters, GTA introduces a task-preserving and interference-suppressing gate (PSI-Gate) on the input side for each task vector, which selectively preserves input representations of the target task and suppresses those of other tasks to reduce merging interference at the input information flow level. To enable data-free training, the paper derives surrogate input representations from the singular value decomposition of task vectors and designs a corresponding PSI surrogate loss to optimize the gates. Theoretically, it proves that finite-step optimization initialized from the identity matrix induces a preservation-oriented spectral bias, and its limiting convergent solution corresponds to the existing closed-form merging method ACTMat. Experiments are conducted across ViT vision models, LLaMA and Qwen2 language models (covering classification and reasoning tasks) with various model scales. GTA delivers consistent performance improvements, with task recovery rate improved by more than 16 percentage points over existing methods in the best case. Multiple ablation studies verify the interference suppression effect, task preservation capability, and hyperparameter robustness of the gating mechanism.

Critical strengths
  1. The core idea is clean, novel, and practically appealing. Instead of only manipulating aggregated task vectors, GTA controls the input information flow supplied to each individual task vector via task-specific input gates. The method is fully data-free, and since the learned gates are absorbed into the task vector parameters, it introduces no extra inference cost, no input-dependent routing, and no additional parameters at deployment. This makes it a practically attractive approach that advances the design paradigm of data-free model merging.

  2. The work combines solid theoretical analysis with strong empirical validation. Theorem 2 nicely frames finite-step gate optimization from identity initialization as preservation-biased, and formally establishes the connection between GTA and the closed-form ACTMat method as the K \to \infty limit, which deepens the understanding of data-free merging mechanisms. Empirically, GTA achieves consistent performance gains across ViT, LLaMA, and Qwen2 architectures covering both vision and language tasks, outperforming a wide range of baselines including TA, TIES, TSV, WUDI, and ACTMat.

Critical weaknesses
  1. Related work and baseline coverage are incomplete. The paper discusses TSV, Subspace Boosting, WUDI, and ACTMat, but omits highly relevant methods: Isotropic Merging (Marczak et al., 2025), which explicitly handles common vs. task-specific subspaces in a closely related manner, and CAT Merging (Luo et al., 2023), which also leverages activation and task information to improve task arithmetic. Including these baselines and discussing their relation to GTA would clarify the paper’s positioning and better contextualize its improvements.

  2. The surrogate input derivation relies on SGD assumptions (Theorem 1), while all fine-tuned models in experiments use AdamW. The paper acknowledges this approximation gap but does not quantify how much it degrades the quality of surrogate inputs or affects final merging performance. This weakens the link between the theoretical motivation and the empirical results.

Questions for authors
  1. Why were Isotropic Merging and CAT Merging excluded from the baseline comparison? Given their close conceptual relevance to GTA, adding these results would significantly strengthen the paper’s positioning and help readers assess the magnitude of the proposed improvements.
  2. Can the authors quantify the approximation gap between SGD-derived surrogate inputs and the actual input representations under AdamW fine-tuning? Have you run any experiments with SGD fine-tuned models to verify whether GTA performance aligns more closely with theoretical predictions in that setting?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 7kVw (1/4)

We thank the reviewer for the careful assessment and constructive suggestions on baseline coverage and surrogate validity. In response, we have added comparisons with Iso-C and Iso-CTS (Appendix D, Tables 13–17) and clarified GTA's relationship to CAT Merging, including their different data requirements.

We have also extended Theorem 1 to AdamW with zero weight decay and row-shared adaptive scaling, and introduced Approximation 1 for general entrywise scaling under zero weight decay. To examine the practical approximation, we provide controlled projection-error measurements under SGD and Adam(W), an activation-span projection ablation on merging accuracy, and an analysis of task-specific input alignment (Section 3.3; Appendices A.2–A.3, Tables 4–7). We address each concern below.

ㅤ

W1 and Q1: Comparison with Isotropic Merging and Relation to CAT Merging

We thank the reviewer for suggesting comparisons with Isotropic Merging and CAT Merging. We have added comparisons with Iso-C and Iso-CTS alongside TSV and Subspace Boosting (SB), using the same checkpoints and evaluation settings as GTA (Appendix D, Tables 13–17). We also discuss GTA's relationship to CAT Merging below. For Iso-CTS, the common-space ratio is fixed at 0.8 across all vision and language settings.

Comparison on vision tasks.

We evaluate ViT-B/32, ViT-B/16, and ViT-L/14 on 8, 14, and 20 tasks using the FusionBench checkpoints. For Iso-C and Iso-CTS, we use the merging coefficients reported in the original Isotropic Merging paper (its Appendix D.3, Table 5), without additional tuning.

ModelTasksTSVSBIso-CIso-CTSGTA
ViT-B/32883.0784.0482.5182.7186.51
ViT-B/321478.7977.3278.0579.4481.60
ViT-B/322073.2270.1971.1673.4177.39
ViT-B/16887.1089.0387.8788.3290.28
ViT-B/161482.1381.7582.4584.4985.22
ViT-B/162078.3173.5575.9178.5083.38
ViT-L/14890.5791.5792.1492.7293.09
ViT-L/141488.2188.3389.6390.5391.13
ViT-L/142083.1980.7183.4585.7589.37

Table 1. Mean accuracy (%) on vision tasks using the same FusionBench checkpoints and evaluation settings. Results are reproduced from the revised manuscript (Appendix D, Table 13).

GTA achieves the highest mean accuracy among the listed methods in all nine settings, exceeding the strongest listed baseline in each setting by 0.37–4.88 percentage points (Appendix D, Table 13).

Comparison on language tasks.

We further evaluate LLaMA-3.2-1B, LLaMA-3.2-3B, and LLaMA-3.1-8B on eight classification tasks, and Qwen2-7B on five reasoning tasks (Appendix D, Tables 14–17).

The original Isotropic Merging paper does not report merging coefficients for these specific language-model settings. We therefore use α=1\alpha=1 for both Iso-C and Iso-CTS without additional tuning.

MethodLLaMA-1BLLaMA-3BLLaMA-8BQwen2-7B
TSV63.1473.4077.8180.14
SB62.1573.4477.9080.45
Iso-C48.3058.7466.4477.82
Iso-CTS48.0558.3464.1677.61
GTA67.0978.0781.7881.21

Table 2. Mean accuracy (%) on eight classification tasks for LLaMA-3.2-1B/3B and LLaMA-3.1-8B, and mean exact match (EM, %) on five reasoning tasks for Qwen2-7B. Results are summarized from the revised manuscript (Appendix D, Tables 14–17).

GTA achieves the highest mean accuracy or EM among the listed methods in all four settings (Appendix D, Tables 14–17). These results characterize performance under the evaluated configurations; the Iso-C and Iso-CTS results use untuned coefficients and do not establish their performance after hyperparameter tuning.

Methodologically, Isotropic Merging flattens singular-value spectra in a common subspace, with Iso-CTS additionally incorporating task-specific directions. GTA instead learns task-specific PSI-Gates using task-vector-derived surrogate inputs H^t=VtΣt\widehat{\boldsymbol{H}}_t=\boldsymbol{V}_t\boldsymbol{\Sigma}_t. Its PSI surrogate loss balances task preservation and interference suppression, and the gates are optimized for a finite number of Adam steps from identity initialization.

AuthorsOpenReview ↗

Response to Reviewer 7kVw (2/4)

Relation to CAT Merging.

CAT Merging (Sun et al., 2025) shares the objective of preserving task-specific information while reducing interference. It collects layer inputs through forward passes on a small number of unlabeled exemplars per task and uses these features to identify and trim conflict-prone task-vector components. Although CAT Merging requires no additional training, it uses task data during merging. We therefore discuss it as related work rather than include it in our data-free baseline comparison.

Standard GTA constructs surrogate inputs and learns PSI-Gates solely from model checkpoints, without accessing task data or fine-tuning trajectory information during merging. Actual activations are used only for diagnostic analyses and the activation-span projection ablation (Appendix A.3).

We thank the reviewer again for encouraging these comparisons. The added results strengthen the empirical evaluation of GTA, while the discussion clarifies its relationship to Isotropic Merging and the different data requirements of CAT Merging.

ㅤ

W2 and Q2: Surrogate Approximation under AdamW and SGD Fine-Tuning

We thank the reviewer for highlighting the gap between the original SGD-based analysis and AdamW fine-tuning, and for suggesting an SGD comparison. We have extended Theorem 1 to AdamW with row-shared adaptive scaling, introduced Approximation 1 for general entrywise AdamW with zero weight decay, and added quantitative analyses of activation-span representability, merging accuracy, and task-specific input alignment.

Controlled Adam(W) experiments yield mean numerical projection errors of 0.97–5.02%. For the original ViT-B/32 checkpoints, projecting the surrogates onto sampled activation spans changes mean merging accuracy by at most 0.13 percentage points. We also observe higher average alignment with same-task than other-task input statistics (Section 3.3; Appendices A.2–A.3, Tables 4–7).

Theorem 1 and Approximation 1 under AdamW.

Let τ=WK−W0=UΣV⊤\boldsymbol{\tau}=\boldsymbol{W}_K-\boldsymbol{W}_0 =\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^{\top} be the compact SVD of a layer's task vector, and let Htrain\boldsymbol{H}^{\mathrm{train}} collect input activations along the fine-tuning trajectory.

Under the stated linear-layer gradient assumptions, zero-initialized moments, zero weight decay, and row-shared adaptive scaling Dk=ak1⊤\boldsymbol{D}_k=\boldsymbol{a}_k\boldsymbol{1}^{\top}, the extended Theorem 1 establishes

VΣ=τ⊤U=HtrainA,\boldsymbol{V}\boldsymbol{\Sigma} =\boldsymbol{\tau}^{\top}\boldsymbol{U} =\boldsymbol{H}^{\mathrm{train}}\boldsymbol{A},

for an appropriate coefficient matrix A\boldsymbol{A}. Plain SGD is recovered with β1=0\beta_1=0 and Dk=11⊤\boldsymbol{D}_k=\boldsymbol{1}\boldsymbol{1}^{\top} (Section 3.3, Eq. (4); Appendix A.2).

For general entrywise AdamW, Lemma 1 instead establishes the residual-inclusive representation

VΣ+R=HtrainA,R=Rwd+Rscale,\boldsymbol{V}\boldsymbol{\Sigma}+\boldsymbol{R} =\boldsymbol{H}^{\mathrm{train}}\boldsymbol{A}, \qquad \boldsymbol{R} =\boldsymbol{R}_{\mathrm{wd}}+\boldsymbol{R}_{\mathrm{scale}},

where the residuals arise from weight decay and deviations from row-shared adaptive scaling (Appendix A.2, Eqs. (23)–(26)).

Our LLaMA and Qwen2 checkpoints use AdamW with zero weight decay, so Rwd=0\boldsymbol{R}_{\mathrm{wd}}=\boldsymbol{0}, while Rscale\boldsymbol{R}_{\mathrm{scale}} may remain nonzero. The FusionBench ViT checkpoints use Adam. Thus, the exact row-shared condition is not generally satisfied by the practical optimizers.

For AdamW-trained checkpoints with zero weight decay, Approximation 1 assumes approximate representability within the training-path activation span:

H^=VΣ≈HtrainA⋆,A⋆=(Htrain)+VΣ,\widehat{\boldsymbol{H}} =\boldsymbol{V}\boldsymbol{\Sigma} \approx \boldsymbol{H}^{\mathrm{train}}\boldsymbol{A}^{\star}, \qquad \boldsymbol{A}^{\star} =(\boldsymbol{H}^{\mathrm{train}})^+ \boldsymbol{V}\boldsymbol{\Sigma},

where (Htrain)+(\boldsymbol{H}^{\mathrm{train}})^+ denotes the Moore–Penrose pseudoinverse (Section 3.3, Eq. (7); Appendix A.3.1, Eqs. (63)–(65)).

Let PH=Htrain(Htrain)+\boldsymbol{P}_H =\boldsymbol{H}^{\mathrm{train}} (\boldsymbol{H}^{\mathrm{train}})^+ be the orthogonal projector onto this span. Under zero weight decay, Lemma 1 gives

Rproj=(I−PH)VΣ=−(I−PH)Rscale.\begin{aligned} \boldsymbol{R}_{\mathrm{proj}} &=(\boldsymbol{I}-\boldsymbol{P}_H)\boldsymbol{V}\boldsymbol{\Sigma}\\\\ &=-(\boldsymbol{I}-\boldsymbol{P}_H)\boldsymbol{R}_{\mathrm{scale}}. \end{aligned}

Thus, Approximation 1 requires only the component of Rscale\boldsymbol{R}_{\mathrm{scale}} outside the training-path activation span to be small, rather than the entire adaptive-scaling residual (Appendix A.3, Eqs. (60)–(61)).

AuthorsOpenReview ↗

Response to Reviewer 7kVw (3/4)

Projection errors under controlled fine-tuning.

We compare SGD with Adam on ViT-B/32 and with AdamW on LLaMA-3.2-1B, using 100 training samples per task across eight tasks (seed 42).

Each task is fine-tuned separately with an effective batch size of 100. One or five passes over the same samples yield K=1K=1 or K=5K=5 optimizer updates, respectively. All runs use FP32 precision, a learning rate of 10−510^{-5}, zero weight decay, and no warmup or learning-rate schedule.

Token-level input activations collected before each update form Htrain\boldsymbol{H}^{\mathrm{train}}. Let QH\boldsymbol{Q}_H contain the orthonormal singular directions retained by the numerical rank criterion. We measure

erel=100∥(I−QHQH⊤)VΣ∥F∥VΣ∥F,e_{\mathrm{rel}} =100\frac{ \\\|(\boldsymbol{I}-\boldsymbol{Q}_H\boldsymbol{Q}_H^{\top}) \boldsymbol{V}\boldsymbol{\Sigma}\\\|_F }{ \\\|\boldsymbol{V}\boldsymbol{\Sigma}\\\|_F },

and average equally across the evaluated weight matrices. The rank criterion and numerical projection procedure are specified (Appendix A.3.1, Eqs. (67)–(70)).

ModelUpdates KKSGDAdam(W)
ViT-B/3210.10031.1704
ViT-B/3250.07030.9694
LLaMA-3.2-1B10.28735.0244
LLaMA-3.2-1B50.24221.3851

Table 3. Mean relative numerical projection error erele_{\mathrm{rel}} (%) onto training-path activation spans. Adam(W) denotes Adam for ViT-B/32 and AdamW for LLaMA-3.2-1B. Results are reproduced from the revised manuscript (Appendix A.3.1, Table 4).

SGD yields smaller mean projection errors in all four comparisons, while Adam(W) yields errors of 0.97–5.02% (Appendix A.3.1, Table 4).

Because the retained numerical span is contained in the full activation span, erel/100e_{\mathrm{rel}}/100 upper-bounds the minimum relative representation error within the full span. Numerical truncation can produce nonzero errors even under SGD, where exact representability holds under the theorem's assumptions (Appendix A.3.1, Eqs. (65)–(71)).

These results support approximate activation-span representability under Adam-family fine-tuning in the evaluated controlled settings. This experiment measures representation errors; it does not compare final merging accuracy using fully fine-tuned SGD and Adam(W) experts.

Effect of activation-span projection on merging accuracy.

We next use the original fine-tuned ViT-B/32 checkpoints and collect input activations H~\widetilde{\boldsymbol{H}} from 100 sampled training examples per task. These activations come from fixed fine-tuned checkpoints, rather than the fine-tuning trajectory.

Let Q~\widetilde{\boldsymbol{Q}} contain an orthonormal basis for the retained numerical activation span. We construct

H^proj=Q~Q~⊤VΣ,Robs=VΣ−H^proj,ϵobs=100∥Robs∥F∥VΣ∥F.\begin{aligned} \widehat{\boldsymbol{H}}_{\mathrm{proj}} &=\widetilde{\boldsymbol{Q}}\widetilde{\boldsymbol{Q}}^{\top} \boldsymbol{V}\boldsymbol{\Sigma},\\\\ \boldsymbol{R}_{\mathrm{obs}} &=\boldsymbol{V}\boldsymbol{\Sigma} -\widehat{\boldsymbol{H}}_{\mathrm{proj}},\\\\ \epsilon_{\mathrm{obs}} &=100\frac{\\\|\boldsymbol{R}_{\mathrm{obs}}\\\|_F} {\\\|\boldsymbol{V}\boldsymbol{\Sigma}\\\|_F}. \end{aligned}

Here, Robs\boldsymbol{R}_{\mathrm{obs}} is defined using sampled checkpoint activations and is distinct from the optimizer-induced Rscale\boldsymbol{R}_{\mathrm{scale}} and the training-path projection residual Rproj\boldsymbol{R}_{\mathrm{proj}} (Appendix A.3.2, Eqs. (75)–(78)).

For each task-count setting, we independently learn PSI-Gates using either the original surrogate VΣ\boldsymbol{V}\boldsymbol{\Sigma} or its projected counterpart H^proj\widehat{\boldsymbol{H}}_{\mathrm{proj}}, keeping the remaining GTA settings unchanged.

Tasksϵˉobs\bar\epsilon_{\mathrm{obs}} (%)OriginalProjectedΔacc\Delta_{\mathrm{acc}} (pp)
81.056286.5186.63+0.12+0.12
141.007881.6081.73+0.13+0.13
201.117677.3977.38−0.01-0.01

Table 4. Mean relative projection residual norm ϵˉobs\bar\epsilon_{\mathrm{obs}} across evaluated tasks and weight matrices, and mean GTA accuracy (%) on ViT-B/32. Original and Projected use VΣ\boldsymbol{V}\boldsymbol{\Sigma} and H^proj\widehat{\boldsymbol{H}}_{\mathrm{proj}}, respectively. Δacc\Delta_{\mathrm{acc}} denotes projected minus original accuracy in percentage points. Results are reproduced from the revised manuscript (Appendix A.3.2, Table 6).

The mean relative residual norm is approximately 1% in all three settings. Using the projected surrogate changes mean accuracy by +0.12+0.12, +0.13+0.13, and −0.01-0.01 percentage points for 8, 14, and 20 tasks, respectively (Appendix A.3.2, Table 6).

These results indicate limited sensitivity to removing components outside the sampled activation spans in the evaluated configurations. They support using the original surrogate without additional activation-span projection during merging.

AuthorsOpenReview ↗

Response to Reviewer 7kVw (4/4)

Alignment with actual checkpoint inputs.

We further examine whether checkpoint-derived surrogates contain input information more closely associated with their corresponding tasks than with other tasks.

Using 100 training examples per task (seed 42), we collect mean-pooled input activations at six representative attention and MLP weight matrices per model. ViT activations are pooled over vision tokens, including the CLS token. LLM activations are pooled over non-padding tokens from prompts without reference answers.

We compare the uncentered second-order statistics of surrogate and actual inputs using Frobenius cosine similarity. Same-task alignment pairs a surrogate with inputs from its corresponding task; other-task alignment pairs it with inputs from different tasks. Results are averaged over the corresponding task pairs and evaluated weight matrices (Appendix A.3.3, Eqs. (79)–(82)).

The mean same-task versus other-task similarities are 0.1508 versus 0.0572 for ViT-B/32, 0.2017 versus 0.1480 for LLaMA-3.2-1B, and 0.1268 versus 0.0902 for Qwen2-7B (Appendix A.3.3, Table 7).

The higher average same-task alignment across all three models provides additional evidence of task-dependent input information in VΣ\boldsymbol{V}\boldsymbol{\Sigma}, although the absolute alignment remains limited.

Together, the controlled projection-error measurements, activation-span projection ablation, and input-alignment analysis provide complementary empirical support for Approximation 1.

Actual activations are used only for diagnostic analyses and the projection ablation. Standard GTA constructs surrogates and learns PSI-Gates from checkpoint-derived VΣ\boldsymbol{V}\boldsymbol{\Sigma}, without accessing task data, fine-tuning trajectory information, or computing activation-span projections (Appendix A.3.4).

We thank the reviewer again for motivating these analyses. The revisions clarify the exact theoretical conditions and the scope of the practical approximation, while providing quantitative empirical support for the surrogate in the evaluated Adam-family fine-tuning settings.

ㅤ

We thank the reviewer again for the constructive feedback. The additional comparisons strengthen GTA's empirical evaluation and clarify its relationship to related merging methods. The revised theoretical analysis and quantitative experiments distinguish the conditions for exact input-activation representability from the practical approximation, and provide complementary empirical support for checkpoint-derived surrogates in the evaluated fine-tuning settings.

Reviewer_sAocOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an improved variant of Task Arithmetic that learns a dense, input-side linear gate for each task vector. These gates are then absorbed into the merged model weights prior to inference. The authors report performance gains on 20-task vision benchmarks compared to the strongest existing baselines.

Critical strengths
  • The experiments are extensive, covering nested 8/14/20-task vision settings, LLaMA models across three different sizes, and generative reasoning tasks.
  • The method achieves performance improvements on 20-task vision classification over competitive baselines.
Critical weaknesses
  • Theorem 2's theoretical limit M∞M_\infty is exactly ACTMat. The finite-step GD mathematically equivalent to early-stopped optimization of the existing ACTMat objective, which does not establish a new theoretical bound. Furthermore, the theoretical claims heavily rely on vanilla SGD/GD, but the language checkpoints use AdamW and the actual gates are optimized using Adam.

  • Performance improvements on language tasks are marginal. For example, 0.61 average EM points on Qwen2-7B (Table 2), and +1.45, −0.02, and +0.39 accuracy points over ACTMat on the three LLaMA scales (Tables 4–6). Table 8 shows a 2.80-point average loss for 1B experts, with drastic drops on MRPC (−4.29) and CommonsenseQA (−4.26). Table 3 also shows a −2.00 drop on OpenBookQA.

  • It is confusing that Equation 10 and Figures 3–5 compare GTA only with unscaled Task Arithmetic (α\alpha=1), whereas the main experiments (Section 4.1) use scaled Task Arithmetic (α=1/n\alpha=1/n) . Comparing against an unscaled baseline artificially amplifies interference, exaggerating the gate's benefit.

  • While there is no inference overhead, the offline cost of Algorithm 1 (compact SVDs + dense-gate optimization per task/matrix) in merging process is non-trivial and unreported. The author should provide end-to-end merging wall-clock time and peak memory comparisons against baselines.

Questions for authors

Section 4.1 does not specify the actual reasoning prompts, answer markers, decoding settings, or harness configuration. Figure 6 reports its highest observed accuracy at the chosen learning rate without identifying the split used to select it. ACTMat hyperparameters and pseudoinverse handling are also absent from the comparison-method setup.

9 comments
AuthorsOpenReview ↗

Response to Reviewer sAoc (1/9)

We thank the reviewer for the careful assessment and constructive feedback on GTA's relationship to ACTMat, theoretical scope, empirical evaluation, and computational cost. In response, we have extended Theorems 1 and 2 and introduced Approximation 1 and Empirical Support 1 to clarify the distinction between exact theoretical results and practical Adam-family optimization. We have also expanded the analyses of surrogate validity, individual task-vector preservation, and interference with Scaled TA and ACTMat, quantified offline merging costs, and clarified evaluation protocols, sensitivity analysis, and ACTMat implementation settings. We address each concern below, referring to the corresponding revisions in the manuscript.

ㅤ

W1-1: Theorem 1 and Surrogate Validity under AdamW Fine-Tuning

We thank the reviewer for highlighting the mismatch between the original SGD-based theoretical analysis and the AdamW fine-tuning used for our language checkpoints. In response, we have (1) extended Theorem 1 to AdamW under zero weight decay and row-shared adaptive scaling, (2) established a residual-inclusive representation for general entrywise AdamW and introduced Approximation 1, and (3) added three quantitative analyses of surrogate validity. These analyses show mean numerical projection errors of 0.97–5.02% under controlled Adam(W) fine-tuning, absolute changes in average merging accuracy of at most 0.13 percentage points after removing components outside sampled checkpoint activation spans, and higher same-task than other-task alignment with actual input statistics (Section 3.3; Appendices A.1–A.3, Tables 4–7).

Theoretical scope under AdamW.

The extended Theorem 1 considers AdamW with zero weight decay, zero-initialized moments, and row-shared adaptive scaling:

Dk=ak1⊤.\boldsymbol{D}_k=\boldsymbol{a}_k\boldsymbol{1}^{\top}.

The scaling factors may vary across output rows and update steps but are shared across input coordinates within each row. Consequently, Dk⊙M^k=diag⁡(ak)M^k\boldsymbol{D}_k\odot\widehat{\boldsymbol{M}}_k =\operatorname{diag}(\boldsymbol{a}_k)\widehat{\boldsymbol{M}}_k, preserving the gradient outer-product structure (Appendix A.1, Eqs. (15)–(16)).

Assuming that the layer input is independent of the layer weights and that the loss depends on these weights only through the linear-layer output, Theorem 1 establishes

VΣ=τ⊤U=HtrainA,\boldsymbol{V}\boldsymbol{\Sigma} =\boldsymbol{\tau}^{\top}\boldsymbol{U} =\boldsymbol{H}^{\mathrm{train}}\boldsymbol{A},

where τ=WK−W0=UΣV⊤\boldsymbol{\tau}=\boldsymbol{W}_K-\boldsymbol{W}_0 =\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^{\top} is the compact SVD of the task vector and Htrain\boldsymbol{H}^{\mathrm{train}} collects input activations along the fine-tuning trajectory. Plain SGD is recovered with β1=0\beta_1=0 and Dk=11⊤\boldsymbol{D}_k=\boldsymbol{1}\boldsymbol{1}^{\top} (Section 3.3, Eq. (4); Appendix A.2).

For general entrywise AdamW under the same linear-layer assumptions, Lemma 1 gives the residual-inclusive representation

VΣ+R=HtrainA,R=Rwd+Rscale,\boldsymbol{V}\boldsymbol{\Sigma}+\boldsymbol{R} =\boldsymbol{H}^{\mathrm{train}}\boldsymbol{A}, \qquad \boldsymbol{R} =\boldsymbol{R}_{\mathrm{wd}}+\boldsymbol{R}_{\mathrm{scale}},

where the two residual terms arise from weight decay and deviations from row-shared adaptive scaling, respectively (Appendix A.2, Eqs. (23)–(26)).

Our LLaMA and Qwen2 checkpoints use AdamW with zero weight decay, so Rwd=0\boldsymbol{R}_{\mathrm{wd}}=\boldsymbol{0}, while Rscale\boldsymbol{R}_{\mathrm{scale}} may remain nonzero. The FusionBench ViT checkpoints use Adam. Thus, the exact representation in Theorem 1 is not assumed to hold for the original checkpoints.

For AdamW-trained checkpoints with zero weight decay, Approximation 1 instead assumes that the surrogate is approximately representable within the training-path activation span:

H^=VΣ≈HtrainA⋆,A⋆=(Htrain)+VΣ,\widehat{\boldsymbol{H}}=\boldsymbol{V}\boldsymbol{\Sigma} \approx\boldsymbol{H}^{\mathrm{train}}\boldsymbol{A}^{\star}, \qquad \boldsymbol{A}^{\star} =(\boldsymbol{H}^{\mathrm{train}})^+ \boldsymbol{V}\boldsymbol{\Sigma},

where A⋆\boldsymbol{A}^{\star} is a least-squares coefficient matrix (Section 3.3, Eq. (7); Appendix A.3.1, Eqs. (63)–(64)).

Let PH=Htrain(Htrain)+\boldsymbol{P}_H =\boldsymbol{H}^{\mathrm{train}}(\boldsymbol{H}^{\mathrm{train}})^+ denote the orthogonal projector onto this span. Under zero weight decay, Lemma 1 implies

Rproj=(I−PH)VΣ=−(I−PH)Rscale\begin{aligned} \boldsymbol{R}_{\mathrm{proj}} &=(\boldsymbol{I}-\boldsymbol{P}_H)\boldsymbol{V}\boldsymbol{\Sigma}\\\\ &=-(\boldsymbol{I}-\boldsymbol{P}_H)\boldsymbol{R}_{\mathrm{scale}} \end{aligned}

(Appendix A.3, Eqs. (60)–(61)). Therefore, Approximation 1 requires only the component of Rscale\boldsymbol{R}_{\mathrm{scale}} outside the training-path activation span to be small relative to the surrogate, rather than the entire adaptive-scaling residual.

AuthorsOpenReview ↗

Response to Reviewer sAoc (2/9)

Projection errors under controlled Adam(W) fine-tuning.

We compare SGD with Adam on ViT-B/32 and with AdamW on LLaMA-3.2-1B. For each model, we use 100 training samples per task across eight tasks (seed 42), fine-tuning each task separately. Each update uses an effective batch of 100 samples, with K=1,5K=1,5 updates over the same samples, FP32 precision, learning rate 10−510^{-5}, zero weight decay, and no warmup or learning-rate schedule.

Token-level activations collected before each update form Htrain\boldsymbol{H}^{\mathrm{train}}. Using an orthonormal basis QH\boldsymbol{Q}_H of its retained numerical column span, we measure

erel=100∥(I−QHQH⊤)VΣ∥F∥VΣ∥F.e_{\mathrm{rel}} =100\frac{ \\\|(\boldsymbol{I}-\boldsymbol{Q}_H\boldsymbol{Q}_H^{\top}) \boldsymbol{V}\boldsymbol{\Sigma}\\\|_F }{ \\\|\boldsymbol{V}\boldsymbol{\Sigma}\\\|_F }.

The numerical rank criterion and averaging procedure are specified (Appendix A.3.1, Eqs. (67)–(71)).

ModelUpdates KKSGDAdam(W)
ViT-B/3210.10031.1704
ViT-B/3250.07030.9694
LLaMA-3.2-1B10.28735.0244
LLaMA-3.2-1B50.24221.3851

Table 1. Mean relative numerical projection error erele_{\mathrm{rel}} (%) onto training-path activation spans, averaged equally across evaluated weight matrices. ViT uses Adam; LLaMA uses AdamW.

Adam(W) yields mean numerical projection errors of 0.97–5.02% in these controlled settings (Appendix A.3.1, Table 4). Because the retained numerical span is contained in the full activation span, these measurements upper-bound the minimum relative representation errors over the corresponding full spans. Numerical truncation can also produce nonzero errors under SGD, even when the exact representation holds under the stated assumptions. The results support approximate activation-span representability in the evaluated controlled settings.

Effect of projection residual removal on merging accuracy.

Using the original ViT-B/32 checkpoints, we collect activations H~\widetilde{\boldsymbol{H}} from 100 training samples per task with the corresponding fine-tuned checkpoint. We construct

H^proj=Q~Q~⊤VΣ,Robs=VΣ−H^proj,\widehat{\boldsymbol{H}}_{\mathrm{proj}} =\widetilde{\boldsymbol{Q}}\widetilde{\boldsymbol{Q}}^{\top} \boldsymbol{V}\boldsymbol{\Sigma}, \qquad \boldsymbol{R}_{\mathrm{obs}} =\boldsymbol{V}\boldsymbol{\Sigma} -\widehat{\boldsymbol{H}}_{\mathrm{proj}},

where Q~\widetilde{\boldsymbol{Q}} spans the retained numerical column space of H~\widetilde{\boldsymbol{H}}.

These activations come from fixed fine-tuned checkpoints, rather than the training trajectories. Accordingly, Robs\boldsymbol{R}_{\mathrm{obs}} is a residual relative to the sampled checkpoint activation span and is distinct from the optimizer-induced Rscale\boldsymbol{R}_{\mathrm{scale}} in Lemma 1. We independently learn PSI-Gates using the original and projected surrogates under otherwise identical settings.

Tasksϵˉobs\bar\epsilon_{\mathrm{obs}} (%)OriginalProjectedΔacc\Delta_{\mathrm{acc}} (pp)
81.056286.5186.63+0.12+0.12
141.007881.6081.73+0.13+0.13
201.117677.3977.38−0.01-0.01

Table 2. Mean relative projection residual norm ϵˉobs\bar\epsilon_{\mathrm{obs}}, with ϵobs=100∥Robs∥F/∥VΣ∥F\epsilon_{\mathrm{obs}} =100\\\|\boldsymbol{R}_{\mathrm{obs}}\\\|_F/ \\\|\boldsymbol{V}\boldsymbol{\Sigma}\\\|_F, and average task accuracy (%) on ViT-B/32. Original and Projected denote GTA using VΣ\boldsymbol{V}\boldsymbol{\Sigma} and H^proj\widehat{\boldsymbol{H}}_{\mathrm{proj}}, respectively. Δacc\Delta_{\mathrm{acc}} denotes projected minus original accuracy in percentage points.

The mean relative residual norm is approximately 1% in all three settings. Removing these components changes average accuracy by +0.12+0.12, +0.13+0.13, and −0.01-0.01 percentage points for 8, 14, and 20 tasks, respectively (Appendix A.3.2, Tables 5–6). These results indicate limited sensitivity to the observed out-of-span components and support using the original surrogate without additional activation-based projection in the evaluated settings.

AuthorsOpenReview ↗

Response to Reviewer sAoc (3/9)

Alignment with actual task inputs.

We further examine whether the surrogates retain task-dependent input information in the original fine-tuned checkpoints: Adam-trained ViT-B/32 and AdamW-trained LLaMA-3.2-1B and Qwen2-7B.

Using 100 training samples per task (seed 42), we collect activations at six representative attention/MLP weight matrices per model. ViT activations are mean-pooled over vision tokens, including the CLS token; LLM activations are mean-pooled over non-padding tokens using prompts without reference answers.

We compare uncentered second-order statistics of actual and surrogate inputs using Frobenius cosine similarity. Averaged over the corresponding task pairs and evaluated weight matrices, same-task versus other-task similarities are 0.1508 versus 0.0572 for ViT-B/32, 0.2017 versus 0.1480 for LLaMA-3.2-1B, and 0.1268 versus 0.0902 for Qwen2-7B (Appendix A.3.3, Table 7).

The consistently higher same-task alignment suggests that the surrogates contain task-dependent input information, although the absolute alignment remains limited.

Together, these analyses provide complementary empirical support for Approximation 1 in the evaluated settings. They do not establish that the full adaptive-scaling residual Rscale\boldsymbol{R}_{\mathrm{scale}} is small under general AdamW fine-tuning. Actual activations are used only for these diagnostic analyses and the projection ablation. Standard GTA constructs its surrogate directly from checkpoint-derived VΣ\boldsymbol{V}\boldsymbol{\Sigma} and learns PSI-Gates without accessing the original task data or fine-tuning trajectories.

We thank the reviewer again for the constructive feedback, which helped us clarify the theoretical scope of Theorem 1 and strengthen the quantitative empirical support for Approximation 1.

ㅤ

W1-2: Relationship to ACTMat and Finite-Step Optimization under Adam

We thank the reviewer for highlighting GTA's relationship to ACTMat and the distinction between the original GD-based analysis and the Adam optimizer used in practice. In response, we have (1) extended Theorem 2 to row-shared Adam-type updates, with GD as a special case, and (2) added Empirical Support 1, based on experiments using GTA's actual entrywise Adam optimizer. We summarize these revisions below (Section 3.5; Appendices A.4–A.5, Tables 8–9).

Relationship to ACTMat and the extended Theorem 2.

We agree that finite-step GD is early-stopped optimization of the quadratic PSI surrogate, whose optimal gates recover ACTMat under the same task-vector-based covariance estimates. Specifically, let Ct=H^tH^t⊤=τt⊤τt\boldsymbol{C}_t=\widehat{\boldsymbol{H}}_t\widehat{\boldsymbol{H}}_t^\top =\boldsymbol{\tau}_t^\top\boldsymbol{\tau}_t and S=∑iCi≻0\boldsymbol{S}=\sum_i\boldsymbol{C}_i\succ\boldsymbol{0}. The unique optimal PSI-Gate is

Gt∗=CtS−1.\boldsymbol{G}_t^*=\boldsymbol{C}_t\boldsymbol{S}^{-1}.

Merging task vectors with these optimal gates yields the closed-form ACTMat merge under the stated covariance estimates, independently of the gate optimizer (Section 3.5; Appendix A.4, Eqs. (83)–(85)).

The extended Theorem 2 characterizes finite-step gate optimization with fixed surrogate representations, zero weight decay, zero-initialized first moments, and positive row-shared adaptive scaling, Dt,k=at,k1⊤\boldsymbol{D}_{t,k}=\boldsymbol{a}_{t,k}\boldsymbol{1}^{\top}. With identity initialization, it establishes

Gt(K)=Gt∗+∑j=1dinejej⊤(I−Gt∗)pt,j,K(S),\boldsymbol{G}_t^{(K)} = \boldsymbol{G}_t^* + \sum_{j=1}^{d_{\mathrm{in}}} \boldsymbol{e}_j\boldsymbol{e}_j^\top (\boldsymbol{I}-\boldsymbol{G}_t^*)p_{t,j,K}(\boldsymbol{S}),

where ej\boldsymbol{e}_j is the jjth standard basis vector and pt,j,Kp_{t,j,K} is a row-specific polynomial determined by the bias-corrected first-moment updates and scaling factors (Section 3.5, Eq. (10); Appendix A.4, Eqs. (86)–(90)).

For any fixed finite KK and realized scaling sequence, these polynomials satisfy pt,j,K(0)=1p_{t,j,K}(0)=1 and lim⁡x→0pt,j,K(x)=1\lim_{x\to0}p_{t,j,K}(x)=1. Consequently, displacement from identity is attenuated along sufficiently small-eigenvalue directions of S\boldsymbol{S}, conditional on that scaling sequence. This characterizes a finite-step identity-preserving bias without assuming convergence to Gt∗\boldsymbol{G}_t^* or asserting monotone decay under arbitrary row-shared scaling (Appendix A.4, Eqs. (122)–(126)).

Setting β1=0\beta_1=0 and Dt,k=11⊤\boldsymbol{D}_{t,k}=\boldsymbol{1}\boldsymbol{1}^{\top} recovers GD, for which the representation reduces to

Gt(K)=Gt∗+(I−Gt∗)(I−2ζS)K\boldsymbol{G}_t^{(K)} = \boldsymbol{G}_t^* + (\boldsymbol{I}-\boldsymbol{G}_t^*) (\boldsymbol{I}-2\zeta\boldsymbol{S})^K

(Section 3.5, Eq. (11); Appendix A.4, Eqs. (92)–(93)).

Reviewer_fAC1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Gated Task Arithmetic (GTA), a data-free model merging method designed to reduce cross-task interference by controlling the input information on which each task vector operates. For each task vector, GTA introduces a task-specific PSI-Gate that aims to preserve representations associated with the corresponding task while suppressing representations associated with other tasks. Since the original training data and activations are unavailable, the method derives surrogate input representations from the task vectors through their scaled right singular vectors, (V\Sigma), and optimizes the gates using a preservation-and-suppression surrogate loss. The resulting gates are absorbed into the task vectors, so no additional routing or modules are required at inference time. The paper further analyzes finite-step optimization from identity initialization and connects its limiting solution to ACTMat. Experiments across ViT, LLaMA, and Qwen2 models show strong merging performance across vision, classification, and reasoning tasks, together with reduced measured interference and largely preserved individual task-vector performance.

Critical strengths

1.The paper provides a clear and useful formulation of cross-task interference in model merging. 2.The empirical evaluation is broad and includes mechanism-level validation rather than only final benchmark accuracy.

Critical weaknesses

1.The theoretical justification for using (V\Sigma) as a surrogate for task-specific input representations relies on assumptions that do not match the main experimental setting. 2.It is not fully isolated whether GTA’s advantage over ACTMat requires iterative gate optimization or mainly comes from identity-biased regularization. 3.The merging-time computational cost is insufficiently characterized.

Questions for authors

1.Can the authors compare GTA with a simple regularized or interpolated ACTMat baseline, such as (G_\alpha=(1-\alpha)I+\alpha G_{\mathrm{ACTMat}})? 2.Can the authors report the merging-time computational cost of GTA? 3.Can the authors empirically quantify how well (V\Sigma) captures task-specific input information for the AdamW-trained models?

7 comments
AuthorsOpenReview ↗

Response to Reviewer fAC1 (1/7)

We thank the reviewer for the careful assessment and constructive comments on surrogate validity, identity bias, and GTA's computational cost. Under zero weight decay, we have extended Theorem 1 to AdamW with row-shared adaptive scaling and introduced Approximation 1 for general entrywise scaling. We also provide quantitative surrogate validation, comparisons with identity–ACTMat interpolation, empirical analyses of finite-step Adam optimization, and measurements of merging runtime, peak GPU memory, and accuracy–cost trade-offs. We address each concern below, with detailed analyses and results in the revised manuscript (Section 3.3; Appendices A.2–A.3, A.5, H, and J).

ㅤ

W1 and Q3: Validity of Surrogate Input Representations under AdamW

We thank the reviewer for highlighting the mismatch between the original SGD-based analysis and practical AdamW fine-tuning, and for requesting quantitative validation of task-specific input information.

To address these concerns, we have (1) extended Theorem 1 to AdamW under zero weight decay and row-shared adaptive scaling, (2) established a residual-inclusive representation for general entrywise AdamW and introduced a new Approximation 1, and (3) added three complementary empirical analyses. The results show small activation-span projection errors under controlled Adam(W) fine-tuning (0.97–5.02%), changes in merging accuracy of at most 0.13 percentage points after removing observed projection residuals, and higher same-task alignment between surrogate and actual input statistics. These revisions are provided (Section 3.3; Appendices A.1–A.3, Tables 4–7).

Extended Theorem 1 and New Approximation 1.

We distinguish an exact representation under restricted AdamW updates from an approximation for practical entrywise adaptive scaling.

Extended Theorem 1 under AdamW. Let τ=WK−W0=UΣV⊤\tau=W_K-W_0=U\Sigma V^\top be the compact SVD of a layer's task vector, and let HtrainH^{\mathrm{train}} collect input activations along the fine-tuning trajectory.

Standard AdamW uses an entrywise adaptive scaling matrix DkD_k. The extended Theorem 1 considers row-shared adaptive scaling,

Dk=ak1⊤,D_k=a_k\mathbf{1}^{\top},

where the scaling factors may vary across output rows and optimization steps but remain shared across input coordinates within each row.

Under this condition,

Dk⊙M^k=diag⁡(ak)M^k,D_k\odot\widehat M_k = \operatorname{diag}(a_k)\widehat M_k,

where M^k\widehat M_k is the bias-corrected first moment (Appendix A.1, Eqs. (15)–(16)).

Under the stated linear-layer gradient assumptions, zero-initialized moments, zero weight decay, and row-shared adaptive scaling, the extended Theorem 1 establishes

VΣ=τ⊤U=HtrainAV\Sigma=\tau^\top U=H^{\mathrm{train}}A

for an appropriate coefficient matrix AA (Section 3.3, Eq. (4); Appendix A.2).

The key observation is that AdamW's first-moment accumulation remains a weighted combination of training-path gradients, which have the outer-product form

Γk=1B∑b=1Bgk,bhk,b⊤.\Gamma_k = \frac{1}{B}\sum_{b=1}^{B} g_{k,b}h_{k,b}^{\top}.

Row-shared scaling preserves this structure, so the accumulated task vector remains representable using training-path input activations. Plain SGD is recovered as a special case with β1=0\beta_1=0 and Dk=11⊤D_k=\mathbf{1}\mathbf{1}^{\top}.

Residual-inclusive representation under general AdamW. The residual-free representation in Theorem 1 is not guaranteed under general entrywise adaptive scaling. To characterize this difference, Lemma 1 decomposes the scaling matrix as

Dk=ak1⊤+Ek,D_k=a_k\mathbf{1}^{\top}+E_k,

where EkE_k captures deviations from row-shared scaling (Appendix A.2, Eqs. (20)–(21)).

Lemma 1 establishes

VΣ+R=HtrainA,R=Rwd+Rscale,V\Sigma+R=H^{\mathrm{train}}A, \qquad R=R_{\mathrm{wd}}+R_{\mathrm{scale}},

where the residual terms arise from weight decay and entrywise adaptive scaling, respectively (Appendix A.2, Eqs. (23)–(26)).

Our LLaMA and Qwen2 checkpoints use AdamW with zero weight decay, while the FusionBench ViT checkpoints use Adam. For the zero-weight-decay AdamW setting, Rwd=0R_{\mathrm{wd}}=0, but RscaleR_{\mathrm{scale}} may remain nonzero.

AuthorsOpenReview ↗

Response to Reviewer fAC1 (2/7)

New Approximation 1 for practical AdamW. Importantly, a nonzero adaptive-scaling residual does not necessarily prevent approximate activation-span representation.

For this practical setting, Approximation 1 assumes approximate representability within the training-path activation span:

H^=VΣ≈HtrainA⋆,A⋆=(Htrain)+VΣ,\widehat H=V\Sigma \approx H^{\mathrm{train}}A^\star, \qquad A^\star=(H^{\mathrm{train}})^+V\Sigma,

where ++ denotes the Moore–Penrose pseudoinverse and A⋆A^\star is a least-squares coefficient matrix (Section 3.3, Eq. (7); Appendix A.3.1, Eqs. (63)–(64)).

Let PH=Htrain(Htrain)+P_H=H^{\mathrm{train}}(H^{\mathrm{train}})^+ denote the orthogonal projector onto the training-path activation span. The minimum representation residual is

Rproj=VΣ−HtrainA⋆=(I−PH)VΣ=−(I−PH)Rscale,\begin{aligned} R_{\mathrm{proj}} &=V\Sigma-H^{\mathrm{train}}A^\star\\\\ &=(I-P_H)V\Sigma\\\\ &=-(I-P_H)R_{\mathrm{scale}}, \end{aligned}

under zero weight decay (Appendix A.3, Eqs. (60)–(61)).

Consequently,

∥Rproj∥F≤∥Rscale∥F.\\\|R_{\mathrm{proj}}\\\|_F \leq \\\|R_{\mathrm{scale}}\\\|_F.

Thus, Approximation 1 requires only the component of RscaleR_{\mathrm{scale}} outside the training-path activation span to be small relative to ∥VΣ∥F\\\|V\Sigma\\\|_F, rather than the entire residual. A small projection residual therefore does not establish that RscaleR_{\mathrm{scale}} itself is small.

Projection Errors under Controlled Adam(W) Fine-Tuning.

We first examine activation-span representability through controlled fine-tuning experiments. We compare SGD with Adam on ViT-B/32 and with AdamW on LLaMA-3.2-1B, using 100 samples per task across eight tasks (seed 42).

Each update uses an effective batch of 100 samples, with K=1,5K=1,5 updates, FP32 precision, learning rate 10−510^{-5}, zero weight decay, and no warmup or learning-rate schedule.

Token-level activations collected before each update form HtrainH^{\mathrm{train}}. Using an orthonormal basis QHQ_H of its numerical column span, we measure

erel=100∥(I−QHQH⊤)VΣ∥F∥VΣ∥F,e_{\mathrm{rel}} = 100\frac{ \\\|(I-Q_HQ_H^\top)V\Sigma\\\|_F }{\\\|V\Sigma\\\|_F},

averaged equally across evaluated weight matrices. The numerical rank criterion is described (Appendix A.3.1).

ModelUpdates KKSGDAdam(W)
ViT-B/3210.10031.1704
ViT-B/3250.07030.9694
LLaMA-3.2-1B10.28735.0244
LLaMA-3.2-1B50.24221.3851

Table 1. Mean relative numerical projection error erele_{\mathrm{rel}} (%) onto training-path activation spans. ViT uses Adam; LLaMA uses AdamW.

As shown above, Adam(W) yields numerical projection errors of 0.97–5.02% in the evaluated settings (Appendix A.3.1, Table 4).

For each evaluated matrix, erel/100e_{\mathrm{rel}}/100 upper-bounds the minimum relative representation error within the corresponding full activation span (Appendix A.3.1, Eq. (71)). Although the errors are higher than those under SGD, they support approximate activation-span representability under Adam-family fine-tuning in these controlled settings.

AuthorsOpenReview ↗

Response to Reviewer fAC1 (3/7)

Effect of Projection Residual Removal on Merging Accuracy.

We further examine whether removing surrogate components outside sampled checkpoint-activation spans affects GTA's merging performance.

For the original ViT-B/32 experts, we collect checkpoint activations H~\widetilde H from 100 samples per task and construct

H^proj=Q~Q~⊤VΣ,Robs=VΣ−H^proj,\widehat H_{\mathrm{proj}} =\widetilde Q\widetilde Q^\top V\Sigma, \qquad R_{\mathrm{obs}}=V\Sigma-\widehat H_{\mathrm{proj}},

where Q~\widetilde Q spans the numerical column space of H~\widetilde H.

These sampled checkpoint activations differ from the training-path activations HtrainH^{\mathrm{train}} used above. Consequently, RobsR_{\mathrm{obs}} is distinct from the optimizer-induced residual RscaleR_{\mathrm{scale}} (Appendix A.3.2, Eqs. (72)–(78)). We independently learn PSI-Gates using VΣV\Sigma and H^proj\widehat H_{\mathrm{proj}}, while keeping other settings unchanged.

Tasksϵˉobs\bar\epsilon_{\mathrm{obs}} (%)GTA (VΣV\Sigma)GTA (H^proj\widehat H_{\mathrm{proj}})Δacc\Delta_{\mathrm{acc}} (pp)
81.056286.5186.63+0.12+0.12
141.007881.6081.73+0.13+0.13
201.117677.3977.38−0.01-0.01

Table 2. Mean relative projection residual norm ϵˉobs\bar\epsilon_{\mathrm{obs}} and mean task accuracy (%) on ViT-B/32. Here, ϵobs=100∥Robs∥F/∥VΣ∥F\epsilon_{\mathrm{obs}}=100\\\|R_{\mathrm{obs}}\\\|_F/\\\|V\Sigma\\\|_F, and ϵˉobs\bar\epsilon_{\mathrm{obs}} denotes its mean over evaluated tasks and weight matrices. Δacc\Delta_{\mathrm{acc}} denotes projected minus original accuracy in percentage points.

Across the three settings, the mean relative residual norm is approximately 1%, and removing this component changes mean accuracy by +0.12, +0.13, and -0.01 percentage points, respectively (Appendix A.3.2, Tables 5–6).

These results indicate limited sensitivity to removing the observed out-of-span components in the evaluated ViT configurations. They also support using the original surrogate without activation-based projection during merging.

Task-Specific Alignment with Actual Inputs.

To directly address Q3, we additionally analyze the original AdamW-trained LLaMA-3.2-1B and Qwen2-7B checkpoints, using Adam-trained ViT-B/32 as a reference.

We collect activations from 100 samples per task (seed 42) at six representative attention/MLP matrices per model. For ViT, activations are mean-pooled over vision tokens, including CLS; for LLMs, they are mean-pooled over non-padding tokens using prompts without reference answers.

Let Hobs,tH_{\mathrm{obs},t} contain the pooled actual inputs for task tt. We compare the uncentered second-order statistics

Cact,t=Hobs,tHobs,t⊤,Csur,i=(ViΣi)(ViΣi)⊤C_{\mathrm{act},t}=H_{\mathrm{obs},t}H_{\mathrm{obs},t}^{\top}, \qquad C_{\mathrm{sur},i}=(V_i\Sigma_i)(V_i\Sigma_i)^{\top}

using Frobenius cosine similarity,

si,t=⟨Csur,i,Cact,t⟩F∥Csur,i∥F∥Cact,t∥F.s_{i,t}= \frac{\langle C_{\mathrm{sur},i},C_{\mathrm{act},t}\rangle_F} {\\\|C_{\mathrm{sur},i}\\\|_F\\\|C_{\mathrm{act},t}\\\|_F}.

Same-task alignment uses i=ti=t, whereas other-task alignment uses i≠ti\neq t. Results are averaged over the corresponding task pairs and evaluated weight matrices (Appendix A.3.3, Eqs. (79)–(82), Table 7).

The same-task versus other-task similarities are 0.1508 versus 0.0572 for ViT-B/32, 0.2017 versus 0.1480 for LLaMA-3.2-1B, and 0.1268 versus 0.0902 for Qwen2-7B.

Across all three models, higher same-task alignment suggests that VΣV\Sigma retains task-dependent input information, although the absolute alignment remains limited (Appendix A.3.3, Table 7).

Together, these analyses support the practical use of VΣV\Sigma as a task-dependent input surrogate in the evaluated settings, without establishing that the full optimizer-induced residual RscaleR_{\mathrm{scale}} is small. Actual activations are used only for diagnostic analyses and the projection ablation; standard GTA requires neither original task data nor fine-tuning trajectories.

We hope these clarifications and additional analyses address your concerns regarding the use of surrogate input representations under AdamW fine-tuning.

Reviewer_zCqzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Gated Task Arithmetic (GTA), a data-free model merging framework that explicitly controls the input information processed by individual task vectors. Instead of directly summing task vectors, GTA introduces task-specific linear gates, termed PSI-Gates, which aim to preserve task-relevant representations while suppressing cross-task interference. Since original training activations are unavailable, the method constructs surrogate input representations from the right singular vectors and singular values of task vectors. The gates are optimized using a preservation-and-suppression objective and subsequently absorbed into their corresponding task vectors before merging. Theoretical analysis characterizes the relationship between task vectors and input activations under SGD assumptions and derives an identity-preserving bias for finite-step gradient descent. Experiments across ViT, LLaMA, and Qwen2 demonstrate competitive or improved merging performance, with additional analyses of activation-level interference and individual task-vector preservation.

Critical strengths
  1. An interesting input-side perspective on data-free model merging, supported by direct interference measurements: the main methodological contribution is to interpret task interference as not only a conflict between parameter updates but also a problem of task vectors operating on input representations associated with other tasks, and GTA addresses this through independently optimized PSI-Gates. Equations (2)–(3) and Proposition 1 (p. 4) formalize layer-level task interference and establish that ideal PSI-Gates eliminate this interference when the stated preservation and suppression conditions are satisfied. Figure 3 (p. 8) measures interference on real Qwen2-7B input activations and shows that GTA reduces interference relative to ungated Task Arithmetic, while Figures 4–5 (pp. 25–26) extend the analysis to three LLaMA model sizes and CLIP ViT-B/32, providing evidence across different architectures. Table 3 (p. 9) examines individual task-vector performance after gating on Qwen2-7B: the average exact-match score decreases by only 0.16 percentage points, from 81.80% to 81.64%, with an average task recovery rate of 99.3%. This combination of a data-free construction and direct activation-level validation provides a useful contribution to understanding the relationship between task-specific parameter updates and input-side interference.
  2. Broad empirical evaluation across model architectures, scales, and task types: the paper evaluates GTA beyond a single model family or task configuration, providing a relatively comprehensive assessment of its practical merging performance. Table 1 (p. 7) evaluates three ViT backbones across 8, 14, and 20 vision tasks, and GTA achieves the highest reported average accuracy in all nine configurations; in the 20-task setting, it reaches 77.39%, 83.38%, and 89.37% on ViT-B/32, ViT-B/16, and ViT-L/14, respectively. Table 2 (p. 7) evaluates five reasoning tasks on Qwen2-7B, where GTA achieves an average exact-match score of 81.21% and a task recovery rate of 97.3%, outperforming the strongest reported baseline in average performance. Tables 4–6 (pp. 23–24) extend the evaluation to LLaMA models at 1B, 3B, and 8B scales, and GTA achieves average accuracies of 67.09%, 78.07%, and 81.78%, respectively, demonstrating competitive performance across substantially different model sizes. The breadth of the evaluation, together with the activation-level and individual-vector analyses, makes the empirical contribution valuable for further research on data-free model merging.
Critical weaknesses
  1. The theoretical assumptions underlying PSI-Gate construction and finite-step optimization do not fully match the actual experimental implementation, and while the theoretical analysis provides a useful interpretation of the proposed method, its applicability to the experimental settings remains insufficiently established. Theorem 1 (Sec. 3.3, p. 4; Appendix A.1, pp. 17–18) establishes the exact identity between task-vector-derived surrogate representations and gradient-weighted input activations under direct SGD without momentum, weight decay, or additional gradient transformations; however, Appendix C (p. 22) states that the LLaMA and Qwen2 checkpoints are fine-tuned using AdamW, and Appendix A.1.1 introduces a residual term and establishes only an approximate relationship when the projected residual is sufficiently small, without quantitatively validating this condition across the evaluated models. Theorem 2 (Sec. 3.5, p. 5; Appendix A.2, pp. 19–20) derives the identity-preserving spectral residual under gradient descent and a specific positive-definiteness and learning-rate condition; however, Algorithm 1 (Appendix B, p. 21) and the actual experiments use Adam for gate optimization, and the manuscript acknowledges this difference in Appendix G (p. 29), describing the experimental results as qualitative support rather than direct verification of the theorem’s spectral dynamics. Equation (3) (p. 4) requires each ideal gate to preserve its own task’s surrogate input representations while completely suppressing those of all other tasks, but these requirements are not necessarily jointly feasible when different task representations have overlapping subspaces, and the manuscript does not provide sufficient conditions for their joint feasibility or quantify the extent of such overlap. These issues weaken the theoretical justification for interpreting the learned gates as task-preserving and interference-suppressing operators under the actual optimization conditions, and stronger validation is needed to establish how much of the theoretical behavior survives in practical AdamW-based fine-tuning and Adam-based gate optimization.
  2. The computational scalability of full-dimensional PSI-Gates and their practical advantage over ACTMat remain insufficiently characterized, and GTA introduces a potentially substantial merging-time optimization burden, particularly when applied independently across many tasks and weight matrices in large language models. Section 3.2 (p. 3) defines each gate as a full matrix, and Algorithm 1 (p. 21) requires a separate gate for each task and optimizes it through repeated updates, increasing the computational burden as the number of tasks and matrix dimensions grow. Section 4.1 (p. 6) uses 300 Adam updates with a learning rate of 10^{-3} for the gates; however, the manuscript does not provide systematic runtime, peak-memory, or computational-scaling comparisons against ACTMat and other data-free merging methods. The practical improvement over ACTMat is not uniform in the language experiments: Table 6 (p. 24) reports 78.07% for GTA versus 78.09% for ACTMat on LLaMA-3.2-3B, and Table 5 (p. 23) shows an improvement of only 0.39 percentage points on LLaMA-3.1-8B, so although GTA achieves larger improvements in other configurations, the benefit does not consistently increase with model scale. Since the limiting PSI-Gate solution coincides with ACTMat under the stated theoretical assumptions, the practical value of finite-step optimization should be considered together with its additional computational cost; without such an analysis, it is difficult to assess whether the observed improvements justify the potentially expensive gate optimization process, particularly for larger models.
Questions for authors
  1. Can you quantitatively evaluate the residual term in Appendix A.1.1 for the actual AdamW-fine-tuned LLaMA and Qwen2 checkpoints, and how closely does (V\Sigma) approximate the gradient-weighted input representation when the SGD assumptions of Theorem 1 are violated? This matters because the surrogate representation is the foundation of the entire PSI-Gate construction, and if the approximation is poor under the actual fine-tuning procedure, the theoretical interpretation of the gates becomes less convincing. Please provide residual-norm measurements or comparisons between the derived surrogate representations and actual input activations across representative layers and models.
  2. Under what conditions can a linear gate simultaneously preserve its own task’s surrogate input subspace and suppress all other tasks’ surrogate subspaces, and how does the method behave when different tasks have substantially overlapping surrogate representation subspaces? This matters because the ideal conditions in Equation (3) may be incompatible when task subspaces intersect, which directly affects the extent to which the proposed mechanism can achieve its intended preservation and suppression objectives. Please characterize task-subspace overlap, provide conditions for exact or approximate feasibility, and report how the corresponding overlap affects preservation and suppression losses.
  3. Can you provide a systematic comparison of GTA’s merging-time runtime and peak GPU memory against ACTMat across different model sizes and numbers of merged tasks, and additionally, how much of GTA’s performance improvement remains when the two methods are compared under a comparable computational budget? This matters because GTA optimizes a separate full-dimensional gate for each task and weight matrix, and its additional optimization cost could become substantial for larger models, particularly when the performance advantage over ACTMat is relatively small. Please report the computational cost of gate construction and optimization, discuss the scaling behavior with respect to model dimensions and task counts, and clarify whether structured or low-rank gates could reduce this overhead without substantially sacrificing merging performance.
8 comments
AuthorsOpenReview ↗

Response to Reviewer zCqz (1/8)

We thank the reviewer for the careful assessment and constructive comments on the applicability of our theoretical analysis, ideal PSI-Gate feasibility, and GTA's computational cost. In response, we have extended Theorem 1 to AdamW and Theorem 2 to Adam-type updates, both under the stated row-shared adaptive scaling conditions. We have also introduced Approximation 1 and Empirical Support 1 to clarify the connection to practical AdamW fine-tuning and entrywise Adam gate optimization. Furthermore, we establish a necessary and sufficient condition for exact PSI-Gate feasibility and provide additional quantitative analyses of surrogate validity, finite-step optimization, subspace overlap, and offline accuracy–cost trade-offs. We address each concern below, with references to the corresponding revisions in the manuscript.

ㅤ

W1-1 and Q1: Surrogate Input Representations under AdamW Fine-Tuning

We thank the reviewer for highlighting the gap between the original SGD-based analysis and the AdamW fine-tuning used in our language experiments. In response, we have (1) extended Theorem 1 to AdamW with row-shared adaptive scaling, (2) introduced Approximation 1 for general entrywise AdamW, and (3) added three complementary empirical analyses. These analyses show numerical training-path projection errors of 0.97–5.02% under controlled Adam(W) fine-tuning, changes in average GTA accuracy of at most 0.13 percentage points after activation-span projection, and higher alignment with same-task input statistics (Section 3.3; Appendices A.2–A.3, Tables 4–7).

Extended Theorem 1 and New Approximation 1.

Under the stated linear-layer gradient assumptions, we extend Theorem 1 to AdamW with zero weight decay, zero-initialized moments, and row-shared adaptive scaling Dk=ak1⊤\boldsymbol{D}_k=\boldsymbol{a}_k\boldsymbol{1}^{\top}. The scaling factors may vary across output rows and update steps, while remaining shared across input coordinates within each row. Consequently,

Dk⊙M^k=diag⁡(ak)M^k,\boldsymbol{D}_k\odot\widehat{\boldsymbol{M}}_k =\operatorname{diag}(\boldsymbol{a}_k)\widehat{\boldsymbol{M}}_k,

which preserves the gradient outer-product structure (Appendix A.1, Eqs. (15)–(16)).

Let τ=WK−W0=UΣV⊤\boldsymbol{\tau}=\boldsymbol{W}_K-\boldsymbol{W}_0 =\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^{\top} be the compact SVD of the task vector, and let Htrain\boldsymbol{H}^{\mathrm{train}} collect input activations along the fine-tuning trajectory. The extended Theorem 1 establishes that a coefficient matrix A\boldsymbol{A} exists such that

VΣ=τ⊤U=HtrainA.\boldsymbol{V}\boldsymbol{\Sigma} =\boldsymbol{\tau}^{\top}\boldsymbol{U} =\boldsymbol{H}^{\mathrm{train}}\boldsymbol{A}.

Plain SGD is included as a special case (Section 3.3, Eq. (4); Appendix A.2).

For general entrywise AdamW with zero weight decay, Lemma 1 gives

VΣ+Rscale=HtrainA,\boldsymbol{V}\boldsymbol{\Sigma}+\boldsymbol{R}_{\mathrm{scale}} =\boldsymbol{H}^{\mathrm{train}}\boldsymbol{A},

where Rscale\boldsymbol{R}_{\mathrm{scale}} arises from deviations from row-shared scaling (Section 3.3, Eq. (6); Appendix A.2). Our LLaMA and Qwen2 checkpoints use entrywise AdamW with zero weight decay, while the FusionBench ViT checkpoints use Adam. The exact row-shared condition therefore need not hold for the practical checkpoints. Computing Rscale\boldsymbol{R}_{\mathrm{scale}} requires fine-tuning gradients and optimizer states, which are unavailable from the checkpoints alone.

We therefore introduce Approximation 1, which assumes that the surrogate H^=VΣ\widehat{\boldsymbol{H}}=\boldsymbol{V}\boldsymbol{\Sigma} is approximately representable within the training-path activation span:

H^≈HtrainA⋆,A⋆=(Htrain)+VΣ,\widehat{\boldsymbol{H}} \approx\boldsymbol{H}^{\mathrm{train}}\boldsymbol{A}^{\star}, \qquad \boldsymbol{A}^{\star} =(\boldsymbol{H}^{\mathrm{train}})^+ \boldsymbol{V}\boldsymbol{\Sigma},

where (⋅)+(\cdot)^+ denotes the Moore–Penrose pseudoinverse and A⋆\boldsymbol{A}^{\star} is a least-squares solution (Section 3.3, Eq. (7); Appendix A.3.1, Eqs. (63)–(65)).

Let PH=Htrain(Htrain)+\boldsymbol{P}_H=\boldsymbol{H}^{\mathrm{train}} (\boldsymbol{H}^{\mathrm{train}})^+ be the orthogonal projector onto this span. The projection residual satisfies

Rproj=(I−PH)VΣ=−(I−PH)Rscale,\begin{aligned} \boldsymbol{R}_{\mathrm{proj}} &=(\boldsymbol{I}-\boldsymbol{P}_H)\boldsymbol{V}\boldsymbol{\Sigma}\\\\ &=-(\boldsymbol{I}-\boldsymbol{P}_H)\boldsymbol{R}_{\mathrm{scale}}, \end{aligned}

under zero weight decay. Thus, Approximation 1 requires only the component of Rscale\boldsymbol{R}_{\mathrm{scale}} outside the training-path activation span to be small relative to the surrogate norm; it does not require the full optimizer-induced residual to be small (Appendix A.3, Eqs. (60)–(61)).

AuthorsOpenReview ↗

Response to Reviewer zCqz (2/8)

Quantitative Validation.

We examine the approximation through three complementary analyses with distinct activation sources.

Training-path projection errors. We compare SGD with Adam on ViT-B/32 and with AdamW on LLaMA-3.2-1B, using 100 samples per task across eight tasks (seed 42) and K∈1,5K\in\\{1,5\\} fine-tuning updates. All runs use FP32, a learning rate of 10−510^{-5}, zero weight decay, and no warmup or learning-rate schedule. We collect token-level input activations before each update and measure the relative Frobenius projection error onto their retained numerical column span.

ModelUpdates KKSGD (%)Adam(W) (%)
ViT-B/3210.10031.1704
ViT-B/3250.07030.9694
LLaMA-3.2-1B10.28735.0244
LLaMA-3.2-1B50.24221.3851

Table 1. Mean relative numerical projection error under controlled fine-tuning. ViT uses Adam, while LLaMA uses AdamW.

The mean numerical projection errors under Adam(W) are 0.97–5.02% in these controlled settings. Because the retained numerical span is contained in the full collected activation span, these errors upper-bound the corresponding minimum relative representation errors. Numerical truncation can also produce nonzero errors under SGD despite its exact representation under the stated assumptions (Appendix A.3.1, Eq. (71), Table 4).

Effect on merging accuracy. Using the original ViT-B/32 checkpoints, we project the surrogate representations onto numerical spans of input activations from 100 sampled training examples per task. These activations are collected at the fixed fine-tuned checkpoints, so their spans differ from the training-path spans above. The removed components are therefore observed projection residuals, distinct from Rscale\boldsymbol{R}_{\mathrm{scale}}. We then independently learn PSI-Gates using the projected surrogates, keeping the remaining GTA settings unchanged.

The mean relative residual norms are 1.0562%, 1.0078%, and 1.1176% for the 8-, 14-, and 20-task settings, respectively. The corresponding changes in average GTA accuracy are +0.12, +0.13, and −0.01-0.01 percentage points. These results indicate limited sensitivity to removing components outside the sampled activation spans in the evaluated configurations (Appendix A.3.2, Tables 5–6).

Alignment with actual task inputs. We compare the uncentered second-order statistics of surrogate representations and actual checkpoint inputs, using 100 examples per task at six representative attention and MLP weight matrices per model. The same-task versus other-task Frobenius cosine similarities are 0.1508 versus 0.0572 for ViT-B/32, 0.2017 versus 0.1480 for LLaMA-3.2-1B, and 0.1268 versus 0.0902 for Qwen2-7B. The higher same-task alignment across all three models supports the presence of task-dependent input information in the surrogates, although the absolute alignment remains limited (Appendix A.3.3, Table 7).

These analyses provide empirical support for the surrogate in the evaluated settings, without establishing that Rscale\boldsymbol{R}_{\mathrm{scale}} is small under general AdamW. Actual activations are used only for these diagnostic experiments and the projection ablation. Standard GTA constructs its surrogates directly from task vectors and learns PSI-Gates without access to the original training data or fine-tuning trajectory (Appendix A.3.4).

We thank the reviewer again for this constructive suggestion, which helped us clarify the theoretical scope of the surrogate and strengthen its empirical validation under Adam-family fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer zCqz (3/8)

W1-2: Finite-Step Gate Optimization under Adam

We thank the reviewer for highlighting the mismatch between the original GD-based analysis and the Adam optimizer used in GTA. In response, we have (1) extended Theorem 2 to row-shared Adam-type updates, with GD as a special case, and (2) introduced Empirical Support 1 based on additional experiments with GTA's actual entrywise Adam optimizer. The theorem characterizes finite-step identity preservation under row-shared adaptive scaling. The experiments show that, in the evaluated settings, finite-step Adam can achieve a more favorable empirical preservation–suppression balance than further minimization of the PSI surrogate (Section 3.5; Appendices A.4–A.5, Tables 8–9).

Extended Theorem 2 under Row-Shared Adam-Type Updates.

We extend the analysis to full-batch optimization of the fixed PSI surrogate using bias-corrected first-moment accumulation and row-shared adaptive scaling, Dt,k=at,k1⊤\boldsymbol{D}_{t,k}=\boldsymbol{a}_{t,k}\boldsymbol{1}^{\top}. The positive scaling factors may vary across rows and update steps, while remaining shared across input coordinates within each row (Section 3.5, Eq. (9); Appendix A.4, Eqs. (86)–(87)).

The extended theorem establishes the following result:

Extended Theorem 2 (Identity-Preserving Bias of Finite-Step Optimization).

Let Ci=H^iH^i⊤\boldsymbol{C}_i=\widehat{\boldsymbol{H}}_i\widehat{\boldsymbol{H}}_i^{\top}, S=∑i∈TCi≻0\boldsymbol{S}=\sum_{i\in\mathcal{T}}\boldsymbol{C}_i\succ\boldsymbol{0}, and Gt∗=CtS−1\boldsymbol{G}_t^*=\boldsymbol{C}_t\boldsymbol{S}^{-1}. Keep the surrogate representations fixed.

Consider the row-shared Adam-type updates with zero weight decay, zero-initialized first moments, β1∈[0,1)\beta_1\in[0,1), learning rate ζ 0\zeta\>0, positive scaling factors, and identity initialization Gt(0)=I\boldsymbol{G}_t^{(0)}=\boldsymbol{I}. After KK updates,

Gt(K)=Gt∗+∑j=1dinejej⊤(I−Gt∗)pt,j,K(S),\boldsymbol{G}_t^{(K)} = \boldsymbol{G}_t^* + \sum_{j=1}^{d_{\mathrm{in}}} \boldsymbol{e}_j\boldsymbol{e}_j^{\top} (\boldsymbol{I}-\boldsymbol{G}_t^*)p_{t,j,K}(\boldsymbol{S}),

where ej\boldsymbol{e}_j is the jjth standard basis vector and pt,j,Kp_{t,j,K} is a row-specific polynomial determined by the first-moment updates and scaling factors.

For every fixed finite KK and fixed realized scaling sequence,

pt,j,K(0)=1,lim⁡x→0pt,j,K(x)=1.p_{t,j,K}(0)=1, \qquad \lim_{x\to0}p_{t,j,K}(x)=1.

Thus, conditional on this sequence, displacement from identity is attenuated along sufficiently small-eigenvalue directions of S\boldsymbol{S}.

The polynomial recurrence and proof are provided (Appendix A.4, Eqs. (89)–(90), (94)–(126)). This finite-step characterization does not require convergence to the optimal gate and does not establish monotone decay of residual multipliers or convergence under arbitrary row-shared adaptive scaling.

Setting β1=0\beta_1=0 and Dt,k=11⊤\boldsymbol{D}_{t,k}=\boldsymbol{1}\boldsymbol{1}^{\top} recovers full-batch GD:

Gt(K)=Gt∗+(I−Gt∗)(I−2ζS)K.\boldsymbol{G}_t^{(K)} = \boldsymbol{G}_t^* + (\boldsymbol{I}-\boldsymbol{G}_t^*) (\boldsymbol{I}-2\zeta\boldsymbol{S})^K.

Thus, GD remains a special case of the extended theorem (Section 3.5, Eq. (11); Appendix A.4, Eqs. (92)–(93)). The optimum Gt∗\boldsymbol{G}_t^* is independent of the optimizer; under task-vector Gram covariance estimates, merging these optimal gates yields closed-form ACTMat (Section 3.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.