- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes FCP as a model merging method that protects pretrained weights along dominant input-feature directions to reduce interference between downstream tasks. The method uses unlabeled calibration data without backpropagation and improves several existing merging methods on vision and language benchmarks.
Critical strengths
- The method is simple and practical because it can improve existing model merging approaches without additional training or labeled data.
- The experiments cover several merging methods, model architectures, and task settings, showing consistent improvements over the original methods and the CSP baseline.
Critical weaknesses
- The theoretical analysis only shows that large gradients make a model more sensitive to merging errors, but it does not establish that preserving these directions is better than applying the correct task updates. This leaves the main motivation for FCP insufficiently justified.
- The paper assumes that dominant input-feature directions are good proxies for important gradient directions, but the mathematical relationship only guarantees that gradient directions lie within the feature space. The experiments compare gradient energy without directly measuring subspace alignment or showing that the selected directions correspond to harmful merging errors.
- The experiments mainly use vision classification models and a relatively small Flan-T5 model on GLUE tasks, without evaluating modern large language models or instruction-following capabilities. This makes it difficult to judge whether the method generalizes to realistic foundation model merging, where preserving task-specific capabilities is particularly challenging.
Questions for authors
-
Can the authors show that protecting high-gradient directions reduces actual merging errors rather than simply removing useful task updates?
-
Can the authors directly measure the alignment between the FCP and GCP subspaces and explain why high feature variance should identify directions that are harmful to merge?
-
Can the authors provide results on larger language models with instruction-following or reasoning tasks to demonstrate that FCP works beyond the current benchmarks?
3 comments
Response to Reviewer F7DJ (Part 2)
Q3.2: Can the authors directly measure the alignment between the FCP and GCP subspaces, and explain why high feature variance should identify directions that are harmful to merge?
A3.2:
Thank you for this suggestion. We directly quantify the alignment between the input-side FCP and GCP subspaces using the following three metrics:
- Subspace overlap: , where and are orthonormal bases of the rank- FCP and GCP subspaces. Higher values indicate greater alignment.
- Mean principal Cosine: the average of Cosine similarity of principal angles, i.e., , where indicates the -th singular value of . Higher values indicate closer alignment.
- GCP energy coverage: the fraction of energy in GCP's leading eigendirections captured by the rank- FCP subspace , where are the leading eigenpairs of . Higher values indicate greater alignment.
As shown in the table below, the FCP and GCP subspaces align across ranks well above the random-subspace baseline, indicating that the dominant input-feature directions can serve as informative proxies for gradient-sensitive directions.
| Backbone | Rank () | Subspace Overlap | Mean Principal Cosine | Coverage of GCP Energy |
|---|---|---|---|---|
| ViT-B/32 | 1 | 54.67% | 0.6868 | 37.41% |
| ViT-B/32 | 2 | 50.17% | 0.6177 | 47.13% |
| ViT-B/32 | 5 | 47.84% | 0.5878 | 58.63% |
| ViT-B/32 | 10 | 50.35% | 0.6169 | 68.50% |
| ViT-B/32 | Random (=10) | 1.14% | 0.0877 | 1.14% |
| ViT-L/14 | 1 | 34.81% | 0.5218 | 22.70% |
| ViT-L/14 | 2 | 37.91% | 0.5024 | 31.18% |
| ViT-L/14 | 5 | 40.33% | 0.5131 | 43.47% |
| ViT-L/14 | 10 | 47.10% | 0.5751 | 54.14% |
| ViT-L/14 | Random (=10) | 0.85% | 0.0757 | 0.85% |
| Flan-T5-base | 1 | 33.30% | 0.5263 | 28.73% |
| Flan-T5-base | 2 | 36.60% | 0.5035 | 40.12% |
| Flan-T5-base | 5 | 33.67% | 0.4578 | 60.34% |
| Flan-T5-base | 10 | 35.68% | 0.4805 | 68.47% |
| Flan-T5-base | Random (=10) | 0.88% | 0.0757 | 1.21% |
We also evaluate the output-side subspaces as a comparison. Output-side alignment is substantially weaker, consistent with the input-side focus of our analysis:
| Backbone | Rank () | Subspace Overlap | Mean Principal Cosine | Coverage of Top-10 GCP Energy |
|---|---|---|---|---|
| ViT-B/32 | 10 | 3.35% | 0.1270 | 3.63% |
| ViT-L/14 | 10 | 2.87% | 0.1191 | 2.98% |
| Flan-T5-base | 10 | 2.12% | 0.1071 | 2.53% |
The above results support that high feature variance is a proxy for high gradient variance. Combined with A3.1, which shows that high gradient variance identifies directions harmful to merging, this implies that feature variance identifies the harmful directions.
We have included these results in Section 5 of the main text and Section D.8 of the Appendix in the revised manuscript.
Q3.3: Does FCP extend to larger language models and instruction following or reasoning?
A3.3:
Thanks for the suggestion.
We extend our evaluation to Qwen2.5-7B-Instruct on four agent benchmarks covering household planning (ALFWorld), function calling (BFCL), compositional crafting (TextCraft), and web shopping (WebShop). We report task success rates (%) and their unweighted average.
As shown in the table below, FCP improves both merging methods on all four benchmarks, increasing average performance from 20.47% to 31.05% for Task Arithmetic and from 14.23% to 34.18% for TSV. These results support FCP's effectiveness on large language model benchmarks.
| Method | ALFWorld | BFCL | TextCraft | WebShop | Avg. |
|---|---|---|---|---|---|
| Pretrained | 14.80 | 10.60 | 6.80 | 0.80 | 8.25 |
| Task Arithmetic | 38.81 | 11.25 | 14.00 | 17.80 | 20.47 |
| Task Arithmetic + FCP | 57.46 | 18.13 | 24.00 | 24.60 | 31.05 |
| TSV | 38.81 | 3.13 | 4.00 | 11.00 | 14.23 |
| TSV + FCP | 58.95 | 19.37 | 32.00 | 26.40 | 34.18 |
Fine-tuned Qwen2.5-7B-Instruct checkpoints stem from: Zhang K, Chen X, Liu B, et al. Agent learning via early experience. ICML 2026.
We have included these results in Section 6.2 of the main text.
Dear Reviewer F7DJ, thanks again for your comments. As we near the Nov 18 (AoE) discussion deadline, could you let us know if there are any further suggestions or unresolved concerns? Should our response have addressed your concerns, we would be grateful for your consideration of an updated score.
Response to Reviewer F7DJ (Part 1)
Q3.1: Can the authors show that protecting high-gradient directions reduces actual merging errors rather than simply removing useful task updates?
A3.1:
We thank the reviewer for raising an interesting question.
In this paper, we only conclude that high-gradient directions are highly sensitive to perturbations such as those induced by merging. As shown below, we perturb each task expert Wt in either a random subspace or the FCP core subspace, with equal perturbation norm. The FCP core subspace is highly sensitive:
Whether merging interference outweighs the loss of useful task updates is highly scenario-dependent, making a general proof difficult to establish.
On standard model-merging benchmarks, merging interference outweighs the loss of useful task updates. This persists under severe cross-model conflicts in real-world settings (See A3.3 below). Consistent with this, FCP's gain over baselines grows with task conflict (e.g., growing the number of merged tasks), as shown below.
We have included these results in Section D.9 of the Appendix in the revised manuscript.