acceptodds
Under review as a conference paper at ICLR 2027

FARMerge: Learning Functional Task Representations for Source-Data-Free Adapter Merging

Abstract

Low-Rank Adaptation (LoRA) enables modular task adaptation for Large Language Models (LLMs), yet composing multiple task-specific adapters into a unified model remains challenging. Existing merging methods operate predominantly in parameter space using heuristics such as sign consensus or magnitude pruning. In our experiments, parameter similarity does not reliably track functional activation-space compatibility or downstream interference. We introduce FARMerge (Functional Adapter Representation Merge), a representation-learning framework for multi-adapter merging. During one-time meta-training on a disjoint development task bank, a layer-aware set encoder maps multi-layer probe activations into a metric space predictive of empirical pairwise interference. FARMerge is source-data-free at merge time: for novel adapters, 64 generic probe sequences drive interference prediction and a learned gate over rank-1 SVD components before fixed-rank synthesis. Across four LLM families (LLaMA-3-8B, Mistral-7B, Qwen-3.5-4B, OLMo-2-7B) and 12 downstream tasks, FARMerge averages 98.69% single-task-oracle retention on LLaMA-3-8B and 98.39% across all four model families, gaining /// points over DARE-TIES/TaDA-inspired/CT-Merging/LoraHub on LLaMA-3-8B (/// across families; Table tab:main_summary); on LLaMA-3-8B, merging 12 adapters takes 18.4 seconds. Code and reproduction scripts will be released publicly upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Deep Learning, Large Language Model (LLM), Low-Rank Adaptation (LoRA), Parameter-Efficient Fine-Tuning, Multi-Adapter Merging

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_S6maOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes FARMerge, a method for merging multiple task-specific LoRA adapters into a single fixed-rank adapter without accessing their source training data at merge time. It learns functional task representations from adapter-induced activation changes on generic unlabeled probes and uses them to predict compatibility between adapters. These predictions guide the weighting and recombination of SVD components. The representation encoder and merging gate are trained offline on a development task bank. Experiments across four model families show improved task performance retention over existing merging methods.

Critical strengths
  1. The paper presents a reasonable and potentially useful approach to adapter merging: learning functional representations that predict interference and using these predictions to guide component-level weighting.

  2. The reported evaluation is substantial, covering multiple backbones, tasks, random seeds, and fixed-rank comparisons. The consistently strong retention results support the potential usefulness of the approach.

Critical weaknesses
  1. The presentation makes the method difficult to follow. The workflow is spread across several sections: encoder and interaction-head training are described in Section 3.2, gate training in Section 3.3, and the complete merge-time algorithm in Appendix A. The dense notation—for example, SiS_i, ziz_i, y^ij\hat y_{ij}, I^ij\hat I_{ij}, γirm\gamma_{ir}^{m}, and ρirm\rho_{ir}^{m}—also makes it difficult to understand how the different components connect.

  2. Lack of ablation study isolating the contributions of the two supervision objectives. Table 3 evaluates removing interaction-head outputs from the gate, but does not separately establish the effects of the retention loss Lret\mathcal L_{\mathrm{ret}} and the non-additivity loss Lint\mathcal L_{\mathrm{int}} on representation learning.

  3. Although FARMerge requires no additional training at merge time, it still involves forward passes through the base and adapted models on the probe set, activation extraction, and SVD-based decomposition and synthesis.

Questions for authors

Why does removing SVD sign canonicalization cause such a large drop in average retention, from 98.39% to 95.39% in Table 3? This is particularly surprising because paired sign flips preserve the rank-1 component matrices and the gate features defined in the paper. Could you provide further analysis explaining this substantial degradation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer S6ma (1/2): Presentation & Objective isolation

Response to Reviewer S6ma (part 1/2)

Submission 16474: FARMerge

Thank you for the thoughtful review and the positive assessment of our evaluation. We address each point below. Results marked (†) are new rebuttal experiments. The manuscript revisions in §1, the withdrawal of the Section 5.2 interpretation of canonicalization, and the removal of the "0 GPU-hours" / "regardless of KK" phrases are all in the revised PDF.

1. Presentation, workflow, and notation (W1)

Section 3 and Appendix A are revised to give a single presentation:

  • Unified notation. Component gates are now g_ir throughout (the α_ir variant is removed), with R* for target rank and s_i(x) for probe activation residuals. σ\sigma is reserved for singular values and the logistic non-linearity is written sigm(⋅)\mathrm{sigm}(\cdot) in Eqs. 7 and 11. The first paragraph of Section 3 states this convention and points to the notation table; the 10-page limit keeps the table itself in Appendix A.
  • Two-stage algorithm stated in Section 3. The opening of Section 3 and a callout after Eq. 12 now state the two stages and point to Algorithm 1, which the 10-page limit keeps in Appendix A. The two stages are:
    • Stage 1 (offline, once per backbone): probe collection on dev tasks (Eqs. 1–2), signature extraction (Eq. 3), pairwise supervision (Eqs. 5–6), encoder and interaction-head training via L_rep (representation loss) (Eq. 7), then gate training on merge episodes via L_gate (gate loss) (Eq. 12) with representations frozen.
    • Stage 2 (merge time, source-data-free): probe pass on novel adapters, pairwise interaction inference, SVD, feature computation (γ\gamma, ρ\boldsymbol{\rho}), gating, and fixed-rank synthesis.
  • Training flow clarified. g_φ and the heads f_ret and f_int are trained first under L_rep (representation loss); the logistic gate G_ω is trained next under L_gate (gate loss) with the encoder and heads frozen, so L_gate (gate loss) does not backpropagate into g_φ. The §3.2 sentence that implied joint training is replaced by this stage separation, and no "end-to-end meta-training" wording remains in the paper.

2. Isolating the two supervision objectives (W2)

We agree that ablating gate inputs alone does not isolate the roles of the retention loss and the non-additivity loss. We ran four variants that vary the training loss and the merge-time gate inputs independently. Mean retention is across the four backbones (same convention as Table 3). Pair-prediction correlation rr is on the 45 held-out Super-NI pairs.

VariantTraining lossMerge gate inputs4-backbone retentionPair rr
Full FARMergeLret+λILint\mathcal{L}_{\text{ret}} + \lambda_I \mathcal{L}_{\text{int}}all (zi,yˉi,Iˉi,γ,ρ,emz_i, \bar{y}_i, \bar{I}_i, \gamma, \boldsymbol{\rho}, e_m)98.39% ±\pm 0.51rI^=0.912r_{\hat{I}} = 0.912, ry^=0.884r_{\hat{y}} = 0.884
Full encoder, Iˉi\bar{I}_i ablated at merge (†)Lret+λILint\mathcal{L}_{\text{ret}} + \lambda_I \mathcal{L}_{\text{int}}drop Iˉi\bar{I}_i97.44% ±\pm 0.54rI^=0.912r_{\hat{I}} = 0.912
Retention only (†)Lret(λI=0)\mathcal{L}_{\text{ret}} (\lambda_I = 0)drop Iˉi\bar{I}_i97.82% ±\pm 0.55ry^=0.762r_{\hat{y}} = 0.762
Non-additivity only (†)λILint(no Lret)\lambda_I \mathcal{L}_{\text{int}} (\text{no } \mathcal{L}_{\text{ret}})drop yˉi\bar{y}_i96.94% ±\pm 0.58rI^=0.635r_{\hat{I}} = 0.635
Calibration only (†)gate only, no encoderγ,ρ,em\gamma, \boldsymbol{\rho}, e_m only96.77% ±\pm 0.59n/a

Findings.

  1. The retention loss is the primary anchor. Removing it (Non-additivity only variant) drops retention by 1.45 points (98.39 → 96.94).
  2. The non-additivity loss sharpens interaction prediction. Removing it (Retention only variant) drops retention by 0.57 points (98.39 → 97.82) and pair correlation from 0.884 to 0.762.
  3. The interaction summary must reach the gate. Training with the non-additivity loss but ablating its summary at merge gives 97.44%, slightly below the retention-only variant (97.82%) — a mild train/merge mismatch. The full gain of the non-additivity loss (+0.95 over row 2, +0.57 over row 3) appears only when its predictions also enter the gate.

Section 5.3 presents the non-additivity loss as an auxiliary regularizer that sharpens interaction forecasting in the full pipeline, rather than as a standalone objective.

AuthorsOpenReview ↗

Response to Reviewer S6ma (2/2): Merge-time cost & Sign canonicalization

Response to Reviewer S6ma (part 2/2)

3. Merge-time computational and memory cost (W3)

"Source-data-free" does not mean zero merge-time cost; it means no target-task data at merge time. We have replaced "0 GPU-hours" with "no merge-time training" and "regardless of KK" with the explicit scaling below.

Complexity. O ⁣((K+1)Cf+KCenc+K2Chead+KMdr(r+NT+R)+K2Mdr2).\mathcal{O}\!\left((K+1)C_f + K C_{\mathrm{enc}} + K^2 C_{\mathrm{head}} + K M d r (r + N T + R) + K^2 M d r^2\right). For K ≤ 12, probe forward passes dominate; pairwise terms scale quadratically.

Wall-clock (†) (1× A100-80GB, LLaMA-3-8B, N=64N=64, R=16R=16, mean of 3 runs):

KK24812
Probe forward passes (s)3.2 ± 0.15.8 ± 0.110.4 ± 0.214.8 ± 0.2
Feature extraction & SVD (s)0.6 ± 0.01.0 ± 0.01.6 ± 0.12.2 ± 0.1
Gating & truncated synthesis (s)0.3 ± 0.00.5 ± 0.00.9 ± 0.01.4 ± 0.1
Total (s)4.1 ± 0.17.3 ± 0.112.9 ± 0.218.4 ± 0.2

At K=12K=12, 18.4 s is close to TaDA-inspired (16.2 s) and ~6.8× faster than Fisher Merging (124.5 s).

Memory. Peak GPU memory at K=12K=12 is 18.8 GB: 15.6 GB for the FP16 LLaMA-3-8B backbone plus 3.2 GB of merge-time working memory (activations, SVD factors, gate). Table 12 reports 3.2 GB working overhead — the same convention as the baselines (KnOTS 2.3 GB, TaDA 3.0 GB, Fisher 16.8 GB). The revised §5 and Table 12 caption now also state the total peak VRAM with the backbone loaded.

4. Sign canonicalization (Q1)

You are right: paired sign flips (u,v)→(−u,−v)(u,v)\to(-u,-v) leave c=σuv⊤c = \sigma u v^\top and every gate feature (Eqs. 9–11) invariant. There is no mathematical mechanism by which canonicalization could alter the merge, so the 95.39% row must be an implementation artifact rather than a property of the algorithm.

Origin of the row. Our development log shows the ablation script applied an unpaired reflection — flipping vv without the matching flip on uu — which turns cc into −σuv⊤-\sigma u v^\top for the affected components and injects a cancellation the released code never performs. It evaluated a different operator, not an ablation of our method. Re-running the released code with _canonicalise_svd disabled reproduces Full exactly: 98.69 / 98.15 / 98.92 / 97.80 across the four backbones, mean 98.39%. The faulty branch only runs when canonicalization is off, so Full keeps canonicalization on, the baselines do not use this code, and the other rows of Table 3 change a single input or component (probe set, rank, gate inputs, encoder) without touching SVD signs — none are affected. The 95.39% row is removed and the Section 5.2 claim built on it is withdrawn.

Paired-flip test (†). 20 trials of random paired sign flips on fixed checkpoints and adapters gave maximum deviations of 3.8×10−73.8 \times 10^{-7} on gate values, 4.6×10−74.6 \times 10^{-7} on reconstructed updates (relative ∥ΔW∥F\|\Delta W\|_F), and 0.00 points on retention — floating-point noise, as the invariance predicts.

Summary of new results (†)

EvaluationResult
Paired-flip test (20 trials)∣Δgate∣≤3.8×10−7\lvert\Delta\text{gate}\rvert \le 3.8\times10^{-7}, ∥ΔW∥F\|\Delta W\|_F rel. ≤4.6×10−7\le 4.6\times10^{-7}, Δret=0\Delta\text{ret}=0
Rerun with canonicalization disabled98.39% mean (98.69 / 98.15 / 98.92 / 97.80)
Retention only (λI=0\lambda_I=0)97.82% ± 0.55; ry^=0.762r_{\hat{y}} = 0.762
Non-additivity only (no L_ret)96.94% ± 0.58; rI^=0.635r_{\hat{I}} = 0.635
Calibration only (no encoder)96.77% ± 0.59
Wall-clock (K=2/4/8/12K = 2 / 4 / 8 / 12)4.1 / 7.3 / 12.9 / 18.4 s on A100-80GB
Peak memory (K=12K=12)18.8 GB total (3.2 GB working overhead)

Thanks for your rebuttal. Note that Figure 2 still uses α\alpha for the gate outputs and weighted synthesis, while Eqs. (11)–(13) use gg. Please update the figure to match the main text.

AuthorsOpenReview ↗

Response to Reviewer S6ma

Thank you for catching this. We have updated Figure 2 to match the main text:

  • Gate outputs and weighted synthesis: the figure now uses girmg_{ir}^{m} instead of αirℓ\alpha_{ir}^{\ell}, consistent with Eqs. (11)–(13).
  • Module index: per-module quantities in the figure now use mm instead of ℓ\ell, as in the main text.

The revised PDF has been uploaded.

Reviewer_1zbyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FARMerge, an amortized framework for merging multiple LoRA adapters by learning functional representations that predict cross-task interference. A layer-aware set encoder maps these functional signatures into task embeddings. Pairwise prediction heads are trained to estimate retention changes and activation non-additivity. At merge time, the predicted interactions, probe-based component contributions, and subspace-collision statistics are used by a linear logistic gate to weight individual rank-1 SVD components. The weighted updates are then compressed into a fixed-rank merged adapter. The paper reports some evaluation results on several traks.

Critical strengths
  1. The paper focus on functional behavior rather than only parameter geometry.
  2. The experimental analysis covers several practically tasks.
Critical weaknesses
  1. Table 3 reports that removing SVD sign canonicalization reduces average retention from 98.39% to 95.39%, a three-point difference. The manuscript further interprets this as evidence that sign canonicalization prevents severe interference. How to describe this result?
  2. How to define 'source-data-free'? What's the meaning of it?
Questions for authors
  1. How can SVD sign canonicalization affect the output of the stated algorithm?
  2. How does FARMerge behave when the probe domains are unknown or incorrectly specified?
  3. What is the true end-to-end merge-time complexity as the number of adapters increases?
  4. How much of the gain comes from learning cross-task interaction predictions rather than offline supervised calibration of the component gate?
  5. How well does the learned interaction geometry generalize to new adapter-generation pipelines?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1zby (1/3): Sign canonicalization & Source-data-free

Response to Reviewer 1zby (part 1/3)

Submission 16474: FARMerge

Thank you for the careful review. Prompted by your Q1, we re-audited our ablations and traced the 95.39% row to an inadvertent bug in a diagnostic script, not to FARMerge itself: the released code reproduces 98.39% exactly. The row is removed, its claim withdrawn, and no other result is affected (§1). "Source-data-free" is defined as a merge-time property, with the offline supervision stated explicitly (§2). We add probe, complexity, attribution and transfer evidence (§3–§6).

Conventions: (†) marks new rebuttal-time experiments, reported here with per-backbone values. ± in Table 3 and this response is the standard deviation across the four backbones; ± in Tables 1–2 is across seeds.

1. Sign canonicalization (Table 3)

Q1 — How can SVD sign canonicalization affect the output? It cannot; the invariance is provable. A thin SVD is unique only up to paired sign flips: uu and vv flip together, so the rank-1 term c=σuv⊤c = \sigma u v^\top is identical. Every quantity the algorithm reads off the components is a function of cc and therefore invariant: γirm∝E[∥σuv⊤a∥]\gamma_{ir}^m \propto \mathbb{E}[\|\sigma u v^\top a\|] (Eq. 9); q=cos⁡(u,u′)cos⁡(v,v′)q = \cos(u,u')\cos(v,v') is the normalized Frobenius inner product of the rank-1 matrices (Eq. 10). The gate inputs (Eq. 11), gate values, and synthesized adapter (Eq. 13) are all unchanged. _canonicalise_svd is a serialization convention that fixes a representative sign for reproducibility; by construction it cannot alter the merge.

W1 — How should the 98.39% → 95.39% result be described? As an artifact of an invalid diagnostic run, now removed. Tracing the row to our development logs, the ablation applied an unpaired reflection — flipping vv without the matching flip on uu — which turns cc into −σuv⊤-\sigma u v^\top for the affected components and injects a cancellation the released code never performs. It evaluated a different operator, not an ablation of our method. Disabling _canonicalise_svd in the released code reproduces Full exactly on every backbone (98.69 / 98.15 / 98.92 / 97.80; mean 98.39%).

We have (i) removed the row from Table 3 and (ii) withdrawn the main-text claim that canonicalization prevents interference. The faulty branch ran only with canonicalization disabled, so it can affect only that row: Full keeps canonicalization on, the baselines do not use this code, and each of the other Table 3 rows changes a single input or component (probe set, rank, gate inputs, or encoder) without touching SVD signs.

Paired-flip test (†). 20 random paired sign flips on fixed checkpoints and adapters produced maximum deviations of 3.8×10−73.8\times 10^{-7} on gate outputs, 4.6×10−74.6\times 10^{-7} on reconstructed updates (∥ΔW∥F\|\Delta W\|_F, relative), and 0.00 points on retention — floating-point noise, as the proof predicts.

2. Definition of "source-data-free" (W2)

Definition. Given a base model and KK target adapters, a merge is source-data-free if it uses no training examples, labels, gradients, validation splits, or task-specific prompts from any of the KK target tasks, and evaluates no trial merge on a benchmark. It is not a claim that no data is used anywhere in the pipeline.

FARMerge satisfies this: at merge time it reads only the KK adapter weights and passes 64 generic, author-written, unlabeled probe prompts (Appendix H) through the model. No probe is drawn from or conditioned on the target tasks.

The revised §2.2 separates the two stages:

StageInputs
Offline, once per backbone (amortized)Labeled Super-NI development tasks (60 meta-training, 10 validation, 10 test; Appendix F), their adapters, and the probe dictionary; used to train gϕg_\phi, the interaction heads and GωG_\omega. Non-overlap with every evaluation benchmark is verified in Appendix F.
Merge time (per request, source-data-free)Base model, KK target adapter weights, 64 generic probes, and frozen gϕg_\phi, heads and GωG_\omega.

Supervision enters only offline (32.8 GPU-hours per backbone; Appendix I) and never sees the target tasks. This is the amortization assumption of any learned merger; we now state it explicitly rather than implying end-to-end data-freeness.

Adjacent categories (Appendix B; "merge-time data" column of Table 12): data-free (Task Arithmetic, TIES, DARE, KnOTS, CtM, CT-Merging) uses adapter weights only; source-data-free (FARMerge, TaDA-inspired) passes task-agnostic probes through the model; data-dependent methods use unlabeled batches (NSC, RegMean), labeled few-shot (LoraHub), or test batches (AdaMerging).

Scope. The practical benefit is merging adapters for proprietary tasks without supplying their data at merge time. This is not a formal privacy guarantee — adapter weights can carry information about their training data — and the revised Limitations state this explicitly.

AuthorsOpenReview ↗

Response to Reviewer 1zby (2/3): Probe domains & Merge-time complexity

Response to Reviewer 1zby (part 2/3)

3. Unknown or misspecified probe domains (Q2)

Mean retention over the four model families. Rows without (†) are in Table 3, which also gives per-backbone values.

Probe conditionRetentionΔ vs. FullRelative to baselines (Table 1)
Full (N=64N=64, 4 balanced domains)98.39% ±\pm 0.51—+1.95 over TaDA-inspired (96.44%, Table 1)
Blind C4/Dolma stratified (N=64N=64)97.68% ±\pm 0.48−0.71above every baseline
No Code (N=48N=48)96.48% ±\pm 0.57−1.91on par with TaDA-inspired (96.44%)
No Math (N=48N=48)96.08% ±\pm 0.58−2.31on par with CT-Merging (96.08%)
Wrong domain: PubMed only (N=64N=64) (†)95.72% ±\pm 0.40−2.67below TaDA-inspired and CT-Merging; above KnOTS (95.07%)
No Math or Code (N=32N=32)94.55% ±\pm 0.56−3.84below KnOTS (95.07%); above DARE-TIES (94.38%)

Per-backbone values for the new PubMed condition (†): LLaMA-3 95.98%, Mistral 95.50%, Qwen 96.12%, OLMo 95.28%.

What the probes need: broad domain coverage, not task data. Domain-blind text sampled from general pre-training corpora keeps FARMerge above every baseline (97.68%). Surface rewording does not matter: translating the probes (English, Spanish, French), changing their length, or substituting synonyms changes retention by less than 0.15 points (Appendix D.4). Removing a whole domain, or drawing all probes from a single mismatched domain, degrades performance gradually. FARMerge then loses its margin over the probe-based TaDA-inspired baseline but stays above parameter-space baselines such as KnOTS. The only setting that falls below KnOTS is the most truncated one, with two domains removed and N=32N=32.

Specialized domains (Appendix H.1). For a clinical MedQA adapter merged with the 12 primary adapters, generic probes give 96.82% MedQA retention, and adding medical text to the probe dictionary raises it to 99.22%. We therefore state broad domain coverage as an operating requirement rather than claim robustness to arbitrary domains. Adversarially constructed probes are not tested.

4. End-to-end merge-time complexity (Q3)

Merge time grows with KK. We removed "regardless of KK" and "arbitrary KK in 18.4 s" from the abstract and main text and replaced them with the scaling below.

Theoretical complexity. From Algorithm 1, let MM be the number of adapted modules, rr the input rank, RR the target rank, N=64N=64 the probe count, TT the probe token length, dd the hidden dimension, and CfC_f the cost of one forward pass over the NN probes. Total merge-time cost is O((K+1)Cf+K⋅Cenc+K2⋅Chead+K⋅M⋅d⋅r⋅(r+N⋅T+R)+K2⋅M⋅d⋅r2).\mathcal{O}\left((K+1)C_f + K \cdot C_{\text{enc}} + K^2 \cdot C_{\text{head}} + K \cdot M \cdot d \cdot r \cdot (r + N \cdot T + R) + K^2 \cdot M \cdot d \cdot r^2\right). For moderate KK (≤12\le 12) the probe forward sweeps dominate. For large KK the pairwise interaction heads and the subspace-collision terms (ρ\boldsymbol{\rho}) grow quadratically.

Measured wall-clock time (†) (1× NVIDIA A100-80GB, LLaMA-3-8B, N=64N=64, R=16R=16, mean of 3 runs):

KK (adapters)24812
Probe forward sweeps (s)3.2 ±\pm 0.15.8 ±\pm 0.110.4 ±\pm 0.214.8 ±\pm 0.2
Feature extraction & SVD (s)0.6 ±\pm 0.01.0 ±\pm 0.01.6 ±\pm 0.12.2 ±\pm 0.1
Gating & synthesis (s)0.3 ±\pm 0.00.5 ±\pm 0.00.9 ±\pm 0.01.4 ±\pm 0.1
Total (s)4.1 ±\pm 0.17.3 ±\pm 0.112.9 ±\pm 0.218.4 ±\pm 0.2

At K=12K=12, 18.4 s is close to TaDA-inspired (16.2 s) and 6.8× faster than Fisher Merging (124.5 s) (Table 12).

Memory. Peak GPU memory at K=12K=12 is 18.8 GB: 15.6 GB for the FP16 LLaMA-3-8B backbone plus 3.2 GB of merge-time working memory (activations, SVD factors, gate). Table 12 reports the working-memory figure (3.2 GB). It is comparable to KnOTS (2.3 GB) and TaDA-inspired (3.0 GB) and well below Fisher Merging (16.8 GB). The revised §5 and Table 12 caption now also state the total (18.8 GB).

Reviewer_qCPnOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors of this paper argue that parameter-space similarity in current multi-adapter merging methods may not capture functional activation-space compatibility or downstream interference as shown in their experiments. Thus they propose FARMerge, a representation-learning framework for multi-adapter merging. FARMerge learns functional representations from probe activations and predicts pairwise interactions to guide SVD component gating and fixed-rank synthesis. Across four model families and twelve tasks, the method reports 98.39% average single-task performance retention, outperforming evaluated baselines and predicting degradation on unseen task pairs.

Critical strengths
  • The motivation is clear, and amortizing interaction learning offers practical value for merging new adapters.
  • Evaluation spans multiple model families and tasks, with rank-matched comparisons and component ablations.
  • The distinction between offline training and source-data-free merging is clearly established.
Critical weaknesses
  • The sign canonicalization ablation conflicts with the formulation. The cosine product in Eq. (10), rank-1 components, and functional contributions are invariant to simultaneous sign flips of paired singular vectors. Removing sign canonicalization should therefore preserve the gated updates, yet Table 3 reports a retention drop from 98.39% to 95.39%.
  • Proposition 1 lacks a necessary assumption. An anisotropic covariance can have a nullspace within (v⊥v^\perp), allowing zero functional discrepancy. The claimed lower bound using the smallest positive eigenvalue requires additional restrictions.
  • Non-additivity is not established as a cause of performance interference. Eq. (6) measures deviation from activation interpolation, which need not imply harmful interference. Figure 3 supports predictive association, but the shared representation also receives direct performance supervision, leaving the independent role of non-additivity unclear.
  • The contribution of functional representations is insufficiently isolated. Comparing a supervised functional predictor against raw parameter similarities confounds representation choice with learning. Jointly removing both interaction summaries also fails to isolate the benefit of the non-additivity objective.
Questions for authors
  1. What exactly changes in the sign canonicalization ablation? Can a controlled paired sign-flip test reconcile Table 3 with Eqs. (9)–(11)?
  2. Should Proposition 1 assume positive definiteness on (v^\perp), or restrict perturbations to the positive-eigenvalue subspace?
  3. How does measured non-additivity correlate with performance degradation? What happens when training uses only retention supervision?
  4. Do functional features outperform learned parameter-based predictors under matched supervision and capacity? How well do predictions for pairwise averaging transfer to the final multi-adapter merge?
2 comments
AuthorsOpenReview ↗

Response to Reviewer qCPn (1/2): Sign canonicalization & Proposition 1

Response to Reviewer qCPn (part 1/2)

Submission 16474: FARMerge

Thank you for the detailed and technically precise review. We address each point below. Results marked (†) are new rebuttal experiments; corrections to Proposition 1 and the withdrawn claims are already in the revised PDF.

1. Sign canonicalization (Q1, W1)

You are correct: a paired sign flip (u,v)→(−u,−v)(u,v)\to(-u,-v) leaves the rank-1 component c=σuv⊤c=\sigma u v^\top unchanged, so every quantity the algorithm reads from the components is invariant. In Eq. 9, γirm∝E[∥σuv⊤a∥]\gamma_{ir}^m \propto \mathbb{E}[\|\sigma u v^\top a\|] depends only on cc. In Eq. 10, q=cos⁡(u,u′)cos⁡(v,v′)q=\cos(u,u')\cos(v,v') is the Frobenius cosine between the rank-1 outer products, also invariant. The gate inputs (Eq. 11), gate values, and synthesized adapter (Eq. 13) therefore cannot change.

We have withdrawn the Section 5.2 interpretation that canonicalization prevents interference and the word "sign-aware" in Appendix B.

Origin of the 95.39% row. The row did not run the released algorithm with canonicalization disabled: our development log shows an early ablation script applied an unpaired reflection — flipping vv without the matching flip on uu — which turns cc into −σuv⊤-\sigma u v^\top for the affected components and injects a cancellation the released code never performs. Re-running the released code with _canonicalise_svd disabled reproduces Full exactly (98.69/98.15/98.92/97.80 across the four backbones; mean 98.39%). The bug sits on the branch that runs only when canonicalization is off, so Full and every other row of Table 3 — which change a single input or component without touching SVD signs — are unaffected. The 95.39% row is removed.

Paired-flip test (†). 20 trials of random paired sign flips on fixed checkpoints and adapters gave maximum deviations 3.8×10−73.8\times10^{-7} on gate values, 4.6×10−74.6\times10^{-7} on reconstructed updates (∥ΔW∥F\|\Delta W\|_F, relative), and 0.000.00 points on retention — floating-point noise, as the invariance predicts.

2. Proposition 1 (Q2, W2)

You are right that Σx\Sigma_x can have null directions within v⊥v^\perp, so Dfunc=0\mathcal{D}_{\text{func}} = 0 is reachable while λmin⁡+\lambda_{\min}^+ remains strictly positive. The revised Proposition 1 and Appendix D state the two regimes explicitly:

  1. General bound (arbitrary δv∈v⊥\delta v \in v^\perp): λmin⁡(Σx∣v⊥)ϵ2≤Dfunc≤λmax⁡(Σx∣v⊥)ϵ2\lambda_{\min}(\Sigma_x|_{v^\perp})\epsilon^2 \le \mathcal{D}_{\text{func}} \le \lambda_{\max}(\Sigma_x|_{v^\perp})\epsilon^2, where the lower bound can be zero when Σx∣v⊥\Sigma_x|_{v^\perp} is rank-deficient.
  2. Strictly positive lower bound: λmin⁡+(Σx∣v⊥)ϵ2\lambda_{\min}^+(\Sigma_x|_{v^\perp})\epsilon^2 holds when δv\delta v is restricted to the range of Pv⊥ΣxPv⊥P_{v^\perp}\Sigma_x P_{v^\perp}. Case 2 in the proof now selects the eigenvector of λmin⁡+(Σx∣v⊥)\lambda_{\min}^+(\Sigma_x|_{v^\perp}) directly rather than indexing by d−1d{-}1.

The possibility of Dfunc=0\mathcal{D}_{\text{func}}=0 actually strengthens the motivation: identical parameter distance can give zero functional discrepancy, which is the disconnect we want to highlight.

AuthorsOpenReview ↗

Response to Reviewer qCPn (part 2/2)

Response to Reviewer qCPn (part 2/2)

3. Non-additivity vs. interference (Q3, W3)

We agree that Eq. 6 measures departure from linear interpolation, not causation, and that the non-additivity loss and the retention loss both shape the latent representation. Two controlled evaluations isolate the signal:

  • Raw Iˉ\bar{I} vs. empirical drop, no learned model (†). Pearson r=0.573r = 0.573 on 45 held-out Super-NI pairs and r=0.481r = 0.481 on 66 primary pairs. Raw activation non-additivity carries a moderate signal before any learning.
  • Retention-only training, λI=0\lambda_I=0 (†). With the non-additivity loss removed from training and its summary removed from the gate, 4-backbone mean retention is 97.82% (vs. 98.39% for Full). Pair correlation on retention drops from r=0.884r = 0.884 (Full, via y^\hat{y}) to r=0.762r = 0.762 (retention-only, via y^\hat{y}). Full reaches r=0.912r = 0.912 on its dedicated I^\hat{I} head.

Training with the non-additivity loss but ablating its summary only at merge time gives 97.44% (†) — below retention-only by 0.38 points, consistent with a distributional mismatch: the encoder relies on features the gate no longer sees. The benefit of the non-additivity loss materializes when its predictions are also passed to the gate.

The revised Section 5.3 and Appendix E.4 frame non-additivity as a predictive auxiliary signal, not an established mechanistic cause.

4. Isolating the functional representation (Q4a, W4)

We built a capacity-matched parameter-space predictor (†): a 4-layer Transformer encoder (dmodel=256d_{\text{model}}=256, 4 heads, same 2-layer MLP interaction heads) consuming the top singular vectors of ΔW\Delta W per layer, with ~4.35M parameters — matching the functional set encoder's budget — and identical loss, pairs, and training steps.

  • Parameter-based predictor: 97.24% ± 0.52 retention, r=0.684r = 0.684 on held-out pairs.
  • Functional predictor (Ours): 98.39% ± 0.51 retention, r=0.912r = 0.912 on held-out pairs.

Under matched supervision and capacity the functional representation gains +1.15 points of retention and +0.228 in pair rr. This matched baseline is reported in Appendix E.4, and the main claim is calibrated around it.

5. From pairwise averaging to the K-way merge (Q4b)

We correlated each adapter's predicted pairwise burden Iˉi=1K−1∑j≠iI^ij\bar{I}_i = \frac{1}{K-1}\sum_{j\ne i}\hat{I}_{ij} with its empirical degradation in K=12K{=}12 merges. To avoid confounding the gate with the burden prediction, we evaluated under uniform averaging (pre-gate).

Across 12 tasks × 4 backbones (48 points): Pearson r=0.742r = 0.742. Per-backbone: LLaMA-3 r=0.768r=0.768, Mistral r=0.731r=0.731, Qwen r=0.755r=0.755, OLMo r=0.714r=0.714. The 48 points are not fully independent (each task appears across all four families); we treat the aggregate as a summary and report per-backbone correlations beside it. Pairwise predictions reliably forecast which adapters degrade in the 12-way merge.

Summary of new results (†)

EvaluationResult
Paired-flip test (20 trials)∣Δgate∣≤3.8×10−7\lvert\Delta\text{gate}\rvert \le 3.8\times10^{-7}, ∥ΔW∥F\|\Delta W\|_F rel. ≤4.6×10−7\le 4.6\times10^{-7}, Δret=0\Delta\text{ret}=0
Rerun with canonicalization disabled98.39% mean (98.69 / 98.15 / 98.92 / 97.80)
Raw Iˉ\bar{I} vs. dropr=0.573r = 0.573 (45 pairs), r=0.481r = 0.481 (66 pairs)
Retention-only training (λI=0\lambda_I{=}0)97.82% mean; ry^=0.762r_{\hat y}=0.762 (vs. Full ry^=0.884r_{\hat y}=0.884, rI^=0.912r_{\hat I}=0.912)
Matched parameter predictor (~4.35M)97.24%, r=0.684r=0.684 (vs. functional 98.39%, r=0.912r=0.912)
Pairwise burden vs. K=12K{=}12 drop (uniform)r=0.742r=0.742; per-backbone [0.714,0.768][0.714, 0.768]

We would be happy to have further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.