- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes FARMerge, a method for merging multiple task-specific LoRA adapters into a single fixed-rank adapter without accessing their source training data at merge time. It learns functional task representations from adapter-induced activation changes on generic unlabeled probes and uses them to predict compatibility between adapters. These predictions guide the weighting and recombination of SVD components. The representation encoder and merging gate are trained offline on a development task bank. Experiments across four model families show improved task performance retention over existing merging methods.
Critical strengths
-
The paper presents a reasonable and potentially useful approach to adapter merging: learning functional representations that predict interference and using these predictions to guide component-level weighting.
-
The reported evaluation is substantial, covering multiple backbones, tasks, random seeds, and fixed-rank comparisons. The consistently strong retention results support the potential usefulness of the approach.
Critical weaknesses
-
The presentation makes the method difficult to follow. The workflow is spread across several sections: encoder and interaction-head training are described in Section 3.2, gate training in Section 3.3, and the complete merge-time algorithm in Appendix A. The dense notation—for example, , , , , , and —also makes it difficult to understand how the different components connect.
-
Lack of ablation study isolating the contributions of the two supervision objectives. Table 3 evaluates removing interaction-head outputs from the gate, but does not separately establish the effects of the retention loss and the non-additivity loss on representation learning.
-
Although FARMerge requires no additional training at merge time, it still involves forward passes through the base and adapted models on the probe set, activation extraction, and SVD-based decomposition and synthesis.
Questions for authors
Why does removing SVD sign canonicalization cause such a large drop in average retention, from 98.39% to 95.39% in Table 3? This is particularly surprising because paired sign flips preserve the rank-1 component matrices and the gate features defined in the paper. Could you provide further analysis explaining this substantial degradation?
4 comments
Response to Reviewer S6ma (2/2): Merge-time cost & Sign canonicalization
Response to Reviewer S6ma (part 2/2)
3. Merge-time computational and memory cost (W3)
"Source-data-free" does not mean zero merge-time cost; it means no target-task data at merge time. We have replaced "0 GPU-hours" with "no merge-time training" and "regardless of " with the explicit scaling below.
Complexity. For K ≤ 12, probe forward passes dominate; pairwise terms scale quadratically.
Wall-clock (†) (1× A100-80GB, LLaMA-3-8B, , , mean of 3 runs):
| 2 | 4 | 8 | 12 | |
|---|---|---|---|---|
| Probe forward passes (s) | 3.2 ± 0.1 | 5.8 ± 0.1 | 10.4 ± 0.2 | 14.8 ± 0.2 |
| Feature extraction & SVD (s) | 0.6 ± 0.0 | 1.0 ± 0.0 | 1.6 ± 0.1 | 2.2 ± 0.1 |
| Gating & truncated synthesis (s) | 0.3 ± 0.0 | 0.5 ± 0.0 | 0.9 ± 0.0 | 1.4 ± 0.1 |
| Total (s) | 4.1 ± 0.1 | 7.3 ± 0.1 | 12.9 ± 0.2 | 18.4 ± 0.2 |
At , 18.4 s is close to TaDA-inspired (16.2 s) and ~6.8× faster than Fisher Merging (124.5 s).
Memory. Peak GPU memory at is 18.8 GB: 15.6 GB for the FP16 LLaMA-3-8B backbone plus 3.2 GB of merge-time working memory (activations, SVD factors, gate). Table 12 reports 3.2 GB working overhead — the same convention as the baselines (KnOTS 2.3 GB, TaDA 3.0 GB, Fisher 16.8 GB). The revised §5 and Table 12 caption now also state the total peak VRAM with the backbone loaded.
4. Sign canonicalization (Q1)
You are right: paired sign flips leave and every gate feature (Eqs. 9–11) invariant. There is no mathematical mechanism by which canonicalization could alter the merge, so the 95.39% row must be an implementation artifact rather than a property of the algorithm.
Origin of the row. Our development log shows the ablation script applied an unpaired reflection — flipping without the matching flip on — which turns into for the affected components and injects a cancellation the released code never performs. It evaluated a different operator, not an ablation of our method. Re-running the released code with _canonicalise_svd disabled reproduces Full exactly: 98.69 / 98.15 / 98.92 / 97.80 across the four backbones, mean 98.39%. The faulty branch only runs when canonicalization is off, so Full keeps canonicalization on, the baselines do not use this code, and the other rows of Table 3 change a single input or component (probe set, rank, gate inputs, encoder) without touching SVD signs — none are affected. The 95.39% row is removed and the Section 5.2 claim built on it is withdrawn.
Paired-flip test (†). 20 trials of random paired sign flips on fixed checkpoints and adapters gave maximum deviations of on gate values, on reconstructed updates (relative ), and 0.00 points on retention — floating-point noise, as the invariance predicts.
Summary of new results (†)
| Evaluation | Result |
|---|---|
| Paired-flip test (20 trials) | , rel. , |
| Rerun with canonicalization disabled | 98.39% mean (98.69 / 98.15 / 98.92 / 97.80) |
| Retention only () | 97.82% ± 0.55; |
| Non-additivity only (no L_ret) | 96.94% ± 0.58; |
| Calibration only (no encoder) | 96.77% ± 0.59 |
| Wall-clock () | 4.1 / 7.3 / 12.9 / 18.4 s on A100-80GB |
| Peak memory () | 18.8 GB total (3.2 GB working overhead) |
Thanks for your rebuttal. Note that Figure 2 still uses for the gate outputs and weighted synthesis, while Eqs. (11)–(13) use . Please update the figure to match the main text.
Response to Reviewer S6ma
Thank you for catching this. We have updated Figure 2 to match the main text:
- Gate outputs and weighted synthesis: the figure now uses instead of , consistent with Eqs. (11)–(13).
- Module index: per-module quantities in the figure now use instead of , as in the main text.
The revised PDF has been uploaded.
Response to Reviewer S6ma (1/2): Presentation & Objective isolation
Response to Reviewer S6ma (part 1/2)
Submission 16474: FARMerge
Thank you for the thoughtful review and the positive assessment of our evaluation. We address each point below. Results marked (†) are new rebuttal experiments. The manuscript revisions in §1, the withdrawal of the Section 5.2 interpretation of canonicalization, and the removal of the "0 GPU-hours" / "regardless of K" phrases are all in the revised PDF.
1. Presentation, workflow, and notation (W1)
Section 3 and Appendix A are revised to give a single presentation:
2. Isolating the two supervision objectives (W2)
We agree that ablating gate inputs alone does not isolate the roles of the retention loss and the non-additivity loss. We ran four variants that vary the training loss and the merge-time gate inputs independently. Mean retention is across the four backbones (same convention as Table 3). Pair-prediction correlation r is on the 45 held-out Super-NI pairs.
Findings.
Section 5.3 presents the non-additivity loss as an auxiliary regularizer that sharpens interaction forecasting in the full pipeline, rather than as a standalone objective.