acceptodds
Under review as a conference paper at ICLR 2027

Feature Core Preservation: Protecting Task-Sensitive Parameter Subspaces for Model Merging

Abstract

Model merging has emerged as an efficient parameter-editing paradigm for augmenting pretrained models with fine-tuned downstream weights. A central challenge is that merging task updates can overwrite task-shared knowledge encoded in the pretrained model and degrade downstream performance. Core Subspace Preservation (CSP) mitigates this issue by protecting the dominant singular subspace of pretrained weights, along which the pretrained model is already locally optimal. However, CSP leaves directions highly sensitive to downstream task updates unprotected. In this work, we revisit subspace protection from a sensitivity perspective and show, both analytically and empirically, that directions with high downstream gradient energy have low perturbation tolerance and are therefore vulnerable to merging interference. To make such protection practical, we establish a connection between parameter gradients and layer input features, and propose Feature Core Preservation (FCP), which protects pretrained weights along dominant input-feature subspaces. FCP is a lightweight plug-and-play module that can be applied after arbitrary merging rules without requiring costly backward propagation. Extensive experiments on vision and language benchmarks, across different task scales and fine-tuning regimes, show that FCP consistently improves strong model merging baselines.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merging, Knowledge Conflict, Multi-Task Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_F7DJOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes FCP as a model merging method that protects pretrained weights along dominant input-feature directions to reduce interference between downstream tasks. The method uses unlabeled calibration data without backpropagation and improves several existing merging methods on vision and language benchmarks.

Critical strengths
  • The method is simple and practical because it can improve existing model merging approaches without additional training or labeled data.
  • The experiments cover several merging methods, model architectures, and task settings, showing consistent improvements over the original methods and the CSP baseline.
Critical weaknesses
  • The theoretical analysis only shows that large gradients make a model more sensitive to merging errors, but it does not establish that preserving these directions is better than applying the correct task updates. This leaves the main motivation for FCP insufficiently justified.
  • The paper assumes that dominant input-feature directions are good proxies for important gradient directions, but the mathematical relationship only guarantees that gradient directions lie within the feature space. The experiments compare gradient energy without directly measuring subspace alignment or showing that the selected directions correspond to harmful merging errors.
  • The experiments mainly use vision classification models and a relatively small Flan-T5 model on GLUE tasks, without evaluating modern large language models or instruction-following capabilities. This makes it difficult to judge whether the method generalizes to realistic foundation model merging, where preserving task-specific capabilities is particularly challenging.
Questions for authors
  • Can the authors show that protecting high-gradient directions reduces actual merging errors rather than simply removing useful task updates?

  • Can the authors directly measure the alignment between the FCP and GCP subspaces and explain why high feature variance should identify directions that are harmful to merge?

  • Can the authors provide results on larger language models with instruction-following or reasoning tasks to demonstrate that FCP works beyond the current benchmarks?

3 comments
AuthorsOpenReview ↗

Response to Reviewer F7DJ (Part 1)

Q3.1: Can the authors show that protecting high-gradient directions reduces actual merging errors rather than simply removing useful task updates?

A3.1:

We thank the reviewer for raising an interesting question.

In this paper, we only conclude that high-gradient directions are highly sensitive to perturbations such as those induced by merging. As shown below, we perturb each task expert WtW_t in either a random subspace or the FCP core subspace, with equal perturbation norm. The FCP core subspace is highly sensitive:

BackbonePerturbationSUN397CarsRESISC45EuroSATSVHNGTSRBMNISTDTDAverage
ViT-B-32Random79.11 ± 0.2977.52 ± 0.3396.01 ± 0.0399.77 ± 0.0297.42 ± 0.0598.70 ± 0.0399.68 ± 0.0179.43 ± 0.1690.96 ± 0.07
ViT-B-32Random in FCP core17.43 ± 20.696.67 ± 8.7818.12 ± 18.4424.90 ± 14.6837.18 ± 27.6116.23 ± 20.3795.86 ± 5.9242.45 ± 25.9032.35 ± 17.34
ViT-L-14Random84.79 ± 0.0092.45 ± 0.0797.39 ± 0.0199.74 ± 0.0098.13 ± 0.0099.21 ± 0.0199.70 ± 0.0184.23 ± 0.0494.45 ± 0.00
ViT-L-14Random in FCP core70.86 ± 15.9956.15 ± 29.5443.32 ± 27.7965.36 ± 22.7278.61 ± 24.0746.42 ± 28.1099.29 ± 0.5773.73 ± 11.1466.72 ± 29.99

Whether merging interference outweighs the loss of useful task updates is highly scenario-dependent, making a general proof difficult to establish.

On standard model-merging benchmarks, merging interference outweighs the loss of useful task updates. This persists under severe cross-model conflicts in real-world settings (See A3.3 below). Consistent with this, FCP's gain over baselines grows with task conflict (e.g., growing the number of merged tasks), as shown below.

Metric8 tasks14 tasks20 tasks
Avg. improvement of FCP over baselines on ViT-B/322.662.843.61
Avg. improvement of FCP over baselines on ViT-B/162.622.723.01
Avg. improvement of FCP over baselines on ViT-L/142.422.462.98

We have included these results in Section D.9 of the Appendix in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer F7DJ (Part 2)

Q3.2: Can the authors directly measure the alignment between the FCP and GCP subspaces, and explain why high feature variance should identify directions that are harmful to merge?

A3.2:

Thank you for this suggestion. We directly quantify the alignment between the input-side FCP and GCP subspaces using the following three metrics:

  • Subspace overlap: ∣UrX⊤Urgrad∣F2/r|{U_r^X}^\top U_r^{grad}|_F^2/r, where UrXU_r^X and UrgradU_r^{grad} are orthonormal bases of the rank-rr FCP and GCP subspaces. Higher values indicate greater alignment.
  • Mean principal Cosine: the average of Cosine similarity of principal angles, i.e., σi(UrX⊤Urgrad)\sigma_i({U_r^X}^\top U_r^{grad}), where σi(W)\sigma_i(W) indicates the ii-th singular value of WW. Higher values indicate closer alignment.
  • GCP energy coverage: the fraction of energy in GCP's leading eigendirections captured by the rank-rr FCP subspace ∑j=1rλjgrad∥(UrX)⊤ujgrad∥22∑j=1rλjgrad\frac{\sum_{j=1}^{r} \lambda_j^{grad} \|(U_r^X)^\top u_j^{grad}\|_2^2}{\sum_{j=1}^{r}\lambda_j^{grad}}, where (λjgrad,ujgrad)(\lambda_j^{grad},u_j^{grad}) are the leading eigenpairs of ∑tGt⊤Gt\sum_t G_t^\top G_t. Higher values indicate greater alignment.

As shown in the table below, the FCP and GCP subspaces align across ranks well above the random-subspace baseline, indicating that the dominant input-feature directions can serve as informative proxies for gradient-sensitive directions.

BackboneRank (rr)Subspace OverlapMean Principal CosineCoverage of GCP Energy
ViT-B/32154.67%0.686837.41%
ViT-B/32250.17%0.617747.13%
ViT-B/32547.84%0.587858.63%
ViT-B/321050.35%0.616968.50%
ViT-B/32Random (rr=10)1.14%0.08771.14%
ViT-L/14134.81%0.521822.70%
ViT-L/14237.91%0.502431.18%
ViT-L/14540.33%0.513143.47%
ViT-L/141047.10%0.575154.14%
ViT-L/14Random (rr=10)0.85%0.07570.85%
Flan-T5-base133.30%0.526328.73%
Flan-T5-base236.60%0.503540.12%
Flan-T5-base533.67%0.457860.34%
Flan-T5-base1035.68%0.480568.47%
Flan-T5-baseRandom (rr=10)0.88%0.07571.21%

We also evaluate the output-side subspaces as a comparison. Output-side alignment is substantially weaker, consistent with the input-side focus of our analysis:

BackboneRank (rr)Subspace OverlapMean Principal CosineCoverage of Top-10 GCP Energy
ViT-B/32103.35%0.12703.63%
ViT-L/14102.87%0.11912.98%
Flan-T5-base102.12%0.10712.53%

The above results support that high feature variance is a proxy for high gradient variance. Combined with A3.1, which shows that high gradient variance identifies directions harmful to merging, this implies that feature variance identifies the harmful directions.

We have included these results in Section 5 of the main text and Section D.8 of the Appendix in the revised manuscript.

Q3.3: Does FCP extend to larger language models and instruction following or reasoning?

A3.3:

Thanks for the suggestion.

We extend our evaluation to Qwen2.5-7B-Instruct on four agent benchmarks covering household planning (ALFWorld), function calling (BFCL), compositional crafting (TextCraft), and web shopping (WebShop). We report task success rates (%) and their unweighted average.

As shown in the table below, FCP improves both merging methods on all four benchmarks, increasing average performance from 20.47% to 31.05% for Task Arithmetic and from 14.23% to 34.18% for TSV. These results support FCP's effectiveness on large language model benchmarks.

MethodALFWorldBFCLTextCraftWebShopAvg.
Pretrained14.8010.606.800.808.25
Task Arithmetic38.8111.2514.0017.8020.47
Task Arithmetic + FCP57.4618.1324.0024.6031.05
TSV38.813.134.0011.0014.23
TSV + FCP58.9519.3732.0026.4034.18

Fine-tuned Qwen2.5-7B-Instruct checkpoints stem from: Zhang K, Chen X, Liu B, et al. Agent learning via early experience. ICML 2026.

We have included these results in Section 6.2 of the main text.

Dear Reviewer F7DJ, thanks again for your comments. As we near the Nov 18 (AoE) discussion deadline, could you let us know if there are any further suggestions or unresolved concerns? Should our response have addressed your concerns, we would be grateful for your consideration of an updated score.

Reviewer_a2riOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Feature Core Preservation (FCP), a method for preserving pretrained model capabilities during model merging. The authors argue that dominant singular directions of pretrained weights may not accurately reflect task-loss sensitivity. They first introduce gradient-based Core Preservation (GCP) and then propose FCP, which uses input activation features to approximate important gradient directions without backpropagation. The resulting feature subspaces are used to project task updates during merging. Experiments on vision and language models show improvements over existing merging methods.

Critical strengths

The transition from gradient-based protection to activation-based protection is practically motivated. FCP avoids backpropagation and requires only a small amount of unlabeled calibration data.

The experiments show consistent improvements over CSP on several model and merging settings, suggesting that activation-based protection is a promising direction.

Critical weaknesses

The theoretical motivation for FCP is not sufficiently established. The paper uses the relationship between weight gradients and input features to motivate replacing gradient-based protection with activation-based protection. However, showing that the gradient subspace is contained in the input feature subspace does not establish that their dominant directions are aligned. The gradient also depends on the output-side loss derivatives, which may substantially change the relative importance of different input directions. The paper needs stronger evidence that the principal activation subspace is a reliable proxy for the gradient-sensitive subspace.

The motivation for protecting high-gradient directions is incomplete. A large gradient indicates that the loss is sensitive to parameter changes, but it does not necessarily mean that task updates in that direction are harmful. Such directions may also be important for downstream adaptation. The proposed protection may remove both harmful merging interference and useful task updates. The current analysis does not sufficiently explain when the former outweighs the latter.

The novelty over existing projection-based methods is unclear. FCP uses an activation-derived subspace to project task updates, which is closely related to activation-based null-space projection and feature-subspace protection methods. The main difference from CSP is the source of the protected subspace, while the projection operation itself remains similar. The paper should better distinguish FCP from existing activation-based projection methods and clarify what is technically new beyond applying this idea to model merging.

The contribution of input-side protection needs further investigation. The ablation results suggest that most of the improvement comes from input-side projection, while output-side protection contributes relatively little. This raises the question of whether the proposed method essentially reduces to a standard activation-based projection. A direct comparison with existing single-sided activation projection methods would help establish the contribution of FCP.

The improvements may partly result from reducing the magnitude of task updates. Orthogonal projection generally reduces or preserves the Frobenius norm of the task vector. Since merging performance is sensitive to the scaling coefficient, the gains may partly come from reducing update magnitude rather than protecting particularly important feature directions. The random-subspace comparison is useful, but norm-matched controls and comparisons with optimally rescaled task vectors are needed to distinguish these explanations.

The evaluation does not directly establish preservation of pretrained capabilities. The paper motivates FCP as a way to protect pretrained knowledge, but the main results focus on downstream task performance. These results do not directly demonstrate that the merged model retains the original pretrained model's general capabilities or suffers less degradation on general-purpose data. Additional evaluations of pretrained capability retention would strengthen the central claim.

Questions for authors

Can the authors quantify the alignment between the principal activation subspace used by FCP and the gradient-sensitive subspace used by GCP? For example, can they report principal angles, subspace overlap, or gradient-energy coverage across different ranks?

Why should high-gradient directions be protected rather than used for downstream adaptation? Can the authors provide evidence that the benefit of suppressing harmful merging interference outweighs the loss of useful task updates in these directions?

How does FCP compare with existing activation-based null-space projection methods when applied directly to task-vector merging? What specific technical contribution distinguishes FCP from these methods?

Can the authors provide norm-matched comparisons with Task Arithmetic and random-subspace projection, including tuning the merging coefficient for each baseline? This would help determine whether the gains arise from direction selection or simply from reducing the magnitude of task updates.

Can the authors evaluate whether FCP actually preserves pretrained capabilities, using general-purpose held-out data, output consistency with the original model, or other appropriate measures?

9 comments
AuthorsOpenReview ↗

Response to Reviewer a2ri (Part 1)

Q2.1: How closely do the FCP and GCP subspaces align?

A2.1:

Thank you for this suggestion. We directly quantify the alignment between the input-side FCP and GCP subspaces using the following three metrics:

  • Subspace overlap: ∣UrX⊤Urgrad∣F2/r|{U_r^X}^\top U_r^{grad}|_F^2/r, where UrXU_r^X and UrgradU_r^{grad} are orthonormal bases of the rank-rr FCP and GCP subspaces. Higher values indicate greater alignment.
  • Mean principal Cosine: the average of Cosine similarity of principal angles, i.e., σi(UrX⊤Urgrad)\sigma_i({U_r^X}^\top U_r^{grad}), where σi(W)\sigma_i(W) indicates the ii-th singular value of WW. Higher values indicate closer alignment.
  • GCP energy coverage: the fraction of energy in GCP's leading eigendirections captured by the rank-rr FCP subspace ∑j=1rλjgrad∥(UrX)⊤ujgrad∥22∑j=1rλjgrad\frac{\sum_{j=1}^{r} \lambda_j^{grad} \|(U_r^X)^\top u_j^{grad}\|_2^2}{\sum_{j=1}^{r}\lambda_j^{grad}}, where (λjgrad,ujgrad)(\lambda_j^{grad},u_j^{grad}) are the leading eigenpairs of ∑tGt⊤Gt\sum_t G_t^\top G_t. Higher values indicate greater alignment.

As shown in the table below, the FCP and GCP subspaces align across ranks well above the random-subspace baseline, indicating that the dominant input-feature directions can serve as informative proxies for gradient-sensitive directions.

BackboneRank (rr)Subspace OverlapMean Principal CosineCoverage of GCP Energy
ViT-B/32154.67%0.686837.41%
ViT-B/32250.17%0.617747.13%
ViT-B/32547.84%0.587858.63%
ViT-B/321050.35%0.616968.50%
ViT-B/32Random (rr=10)1.14%0.08771.14%
ViT-L/14134.81%0.521822.70%
ViT-L/14237.91%0.502431.18%
ViT-L/14540.33%0.513143.47%
ViT-L/141047.10%0.575154.14%
ViT-L/14Random (rr=10)0.85%0.07570.85%
Flan-T5-base133.30%0.526328.73%
Flan-T5-base236.60%0.503540.12%
Flan-T5-base533.67%0.457860.34%
Flan-T5-base1035.68%0.480568.47%
Flan-T5-baseRandom (rr=10)0.88%0.07571.21%

We also evaluate the output-side subspaces as a comparison. Output-side alignment is substantially weaker, consistent with the input-side focus of our analysis:

BackboneRank (rr)Subspace OverlapMean Principal CosineCoverage of Top-10 GCP Energy
ViT-B/32103.35%0.12703.63%
ViT-L/14102.87%0.11912.98%
Flan-T5-base102.12%0.10712.53%

We have included these results in Section 5 of the main text and Section D.8 of the Appendix in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer a2ri (Part 2)

Q2.2: Why should high-gradient directions be protected rather than used for downstream adaptation?

A2.2:

Good question.

This is because high-gradient directions are highly sensitive to perturbations such as those induced by merging. As shown below, we perturb each task expert WtW_t in either a random subspace or the FCP core subspace, with equal perturbation norm. The FCP core subspace is highly sensitive:

BackbonePerturbationSUN397CarsRESISC45EuroSATSVHNGTSRBMNISTDTDAverage
ViT-B-32Random79.11 ± 0.2977.52 ± 0.3396.01 ± 0.0399.77 ± 0.0297.42 ± 0.0598.70 ± 0.0399.68 ± 0.0179.43 ± 0.1690.96 ± 0.07
ViT-B-32Random in FCP core17.43 ± 20.696.67 ± 8.7818.12 ± 18.4424.90 ± 14.6837.18 ± 27.6116.23 ± 20.3795.86 ± 5.9242.45 ± 25.9032.35 ± 17.34
ViT-L-14Random84.79 ± 0.0092.45 ± 0.0797.39 ± 0.0199.74 ± 0.0098.13 ± 0.0099.21 ± 0.0199.70 ± 0.0184.23 ± 0.0494.45 ± 0.00
ViT-L-14Random in FCP core70.86 ± 15.9956.15 ±29.5443.32 ± 27.7965.36 ± 42.7278.61 ± 24.0746.42 ± 28.1099.29 ± 0.5773.73 ± 11.1466.72 ± 29.99

Whether merging interference outweighs the loss of useful task updates is highly scenario-dependent, making a general theoretical proof difficult to establish. Nevertheless, we empirically find this holds in standard and real-world merging benchmarks:

For standard model-merging benchmarks, merging interference does outweigh the loss of useful task updates. Consistent with this, FCP's gain over baselines grows with task conflict (e.g., growing the number of merged tasks), as shown below.

Metric8 tasks14 tasks20 tasks
Avg. improvement of FCP over baselines on ViT-B/322.662.843.61
Avg. improvement of FCP over baselines on ViT-B/162.622.723.01
Avg. improvement of FCP over baselines on ViT-L/142.422.462.98

For challenging real-world tasks, we extend our evaluation to Qwen2.5-7B-Instruct on four agent benchmarks covering household planning (ALFWorld), function calling (BFCL), compositional crafting (TextCraft), and web shopping (WebShop). We report task success rates (%) and their unweighted average.

As shown in the table below, FCP improves both merging methods on all four benchmarks, increasing average performance from 20.47% to 31.05% for Task Arithmetic and from 14.23% to 34.18% for TSV. These results support that merging interference outweighs the loss of useful task updates in real-world settings.

MethodALFWorldBFCLTextCraftWebShopAvg.
Pretrained14.8010.606.800.808.25
Task Arithmetic38.8111.2514.0017.8020.47
Task Arithmetic + FCP57.4618.1324.0024.6031.05
TSV38.813.134.0011.0014.23
TSV + FCP58.9519.3732.0026.4034.18

We have included these results in Section 6.2 of the main text and Section D.9 of the Appendix.

Q2.3: What specific technical contribution distinguishes FCP from null-space-projection based methods?

A2.3:

We thank the reviewer for this question. We agree that activation-based null-space projection is a standard technique, and we do not claim the projection operator itself as a contribution. Our contribution is better understood as a sensitivity-based formulation of where pretrained knowledge should be preserved during model merging.

Technically, compared with activation-based methods such as RegMean and LOT Merging, FCP differs in three aspects:

  • Different goals. Existing methods use activation statistics to protect fine-tuned knowledge encoded in the task vectors. FCP instead protects pretrained knowledge encoded in the initial checkpoint to which task vectors are added.
  • Different protected objects. Consequently, existing methods derive their subspaces from fine-tuned-model activations, whereas FCP derives them from pretrained-model activations, projecting the merged update away from the dominant input-feature subspace of WpreW_{pre}.
  • Different extents of preservation. Existing methods typically operate on the full feature/null space, whereas FCP protects only the top-rr principal input-feature directions, which our analysis links to high downstream gradient energy and low perturbation tolerance.

Beyond these differences, we provide both theoretical motivation and empirical evidence for protecting the dominant input-feature directions of the pretrained model, which is the central contribution of this work.

AuthorsOpenReview ↗

Response to Reviewer a2ri (Part 3)

Q2.4: Does input-side protection provide most of the benefit?

A2.4:

Yes, as we note in Sections 5.1 and 6.3, input-side protection accounts for most of the benefit.

The output-side protection is included because it yields slight gains in performance. It is orthogonal to our core motivation and is not claimed as a main contribution of this paper.

Q2.5: Could task-vector shrinkage explain the gains of FCP?

A2.5:

No, the gains of FCP cannot be explained by task-vector shrinkage.

As shown below, rescaling the merged task vector to match FCP's per-layer norm does not improve performance, nor do three random projections with the same norm. This is because the merging coefficient λλ of each merging method is already tuned to its optimum (see the second table).

UpdateNorm treatmentViT-B/32 accuracyViT-L/14 accuracy
TSVOriginal84.1%91.7%
TSVMatched to FCP per layer84.0%90.9%
TSV w/ Random projection (seed 1)Matched to FCP per layer83.9%90.8%
TSV w/ Random projection (seed 2)Matched to FCP per layer83.5%90.8%
TSV w/ Random projection (seed 3)Matched to FCP per layer83.7%90.9%
TSV w/ FCPProjected86.2%93.3%
Merging Coefficient0.10.20.30.40.50.60.70.80.91.01.11.21.31.41.51.61.71.81.92
ViT-B/3255.9463.3069.3774.6378.2280.9082.6883.7284.1384.14 (selected)83.8283.0381.9780.5978.8176.7474.3771.7569.0466.34
ViT-L/1472.1477.7782.2585.4887.5588.9989.9590.7191.1991.4891.66 (selected)91.6491.4791.2090.8090.3289.8389.2188.5087.55

We have included these results in Section D.11 of the Appendix in the revised manuscript.

Q2.6: Does FCP preserve general pretrained capabilities?

A2.6:

Thanks for the suggestion. To test whether FCP preserves general pretrained capabilities, we evaluate it on unseen tasks (i.e., tasks not included in merging), whose performance serves as a direct proxy for retained general capability.

The results below are on the 20-task ViT-B/32 benchmark. As shown, FCP consistently improves both seen and unseen performance, indicating that it preserves general pretrained capabilities:

Seen (first 8 tasks)Unseen (last 12 tasks)Seen (first 14 tasks)Unseen (last 6 tasks)
ISO-CTS78.84%58.75%78.12%51.93%
ISO-CTS w/ FCP79.64%59.43%79.1152.64%
TSV83.08%55.04%78.38%50.68%
TSV w/ FCP85.56%58.05%82.00%51.19%

We have included these results in Section D.10 of the Appendix in the revised manuscript.

Reviewer_7RX9OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to preserve pretrained knowledge during model merging by identifying parameter subspaces that are particularly sensitive to downstream task perturbations. It first introduces Gradient Core Preservation (GCP), which protects high-gradient parameter directions based on downstream gradient statistics. To avoid the backward-pass overhead of GCP, the authors propose Feature Core Preservation (FCP), which uses dominant input-feature subspaces extracted through forward passes of the pretrained model as proxies for gradient-sensitive directions. The merged task update is projected away from these protected subspaces, preserving the pretrained mapping along selected input directions. An optional output-side protection component is also investigated. Experiments cover ViT models with up to 20 vision tasks and Flan-T5-base models under full fine-tuning and LoRA settings. Results show consistent improvements when FCP is combined with several existing merging methods.

Critical strengths
  1. The paper extends subspace protection beyond the dominant singular directions of pretrained weights by investigating parameter directions that exhibit high downstream gradient energy, which offers a useful complementary perspective to existing parameter-space preservation methods. Equations (5)–(8) (pp. 4–5) decompose the projected task update into an ideal adaptation component and a merging-error component, and the resulting perturbation analysis establishes that the worst-case first-order loss sensitivity increases with the magnitude of the projected gradient. Figure 2 and Table 1 (p. 5) provide empirical evidence supporting GCP: for Task Arithmetic, GCP increases average accuracy from 69.1% to 74.0% on ViT-B/32 and from 84.5% to 88.5% on ViT-L/14, and the results also show that preserving randomly selected moderate-gradient subspaces can degrade performance. Figure 3 (p. 6) demonstrates that the FCP core captures substantially more gradient energy than the CSP core, reaching levels comparable to GCP on the evaluated vision models. This analysis provides a useful conceptual extension of existing pretrained-core preservation approaches and motivates the use of gradient sensitivity as an additional criterion for identifying protected parameter subspaces.
  2. FCP translates the gradient-based protection principle into a practical procedure using only pretrained-model forward features, without requiring backward propagation or calibration labels, and Equation (13) (p. 6) establishes that the right gradient subspace of a linear layer is contained in its input-feature span, providing the mathematical foundation for using input features to approximate gradient-sensitive directions. Tables 2–4 (pp. 7–8) demonstrate that FCP improves Task Arithmetic, LOT, ISO, and TSV across ViT-B/32 and ViT-L/14, as well as configurations involving 8, 14, and 20 tasks; for example, TSV improves from 84.1% to 86.2% on ViT-B/32 and from 91.7% to 93.3% on ViT-L/14. Table 5 (p. 9) extends the evaluation to Flan-T5-base under full fine-tuning and LoRA, where FCP improves average performance for Task Arithmetic from 78.9% to 80.9% under full fine-tuning and from 77.4% to 79.3% under LoRA, and TSV also improves from 82.8% to 83.8% and from 81.8% to 82.5%, respectively. The plug-and-play design and consistent improvements across different merging rules make FCP a practically interesting complementary component for existing model-merging pipelines.
Critical weaknesses
  1. The approximation of gradient-sensitive subspaces using dominant input-feature directions is not sufficiently characterized across different model families: the core theoretical argument establishes a relationship between the input-feature span and the gradient span, but this does not necessarily imply that their leading singular subspaces coincide, and this distinction is important because FCP protects only the dominant feature directions rather than the complete input-feature span. Equation (13) (p. 6) establishes (G=(∂L/∂Z)X^⊤), implying that the right gradient subspace is contained in the input-feature span; however, this relationship alone does not guarantee that the leading singular vectors of (X) align with those of (G), since the output-gradient term can reweight different input directions. Section 5 (p. 6) explicitly acknowledges that the leading feature and gradient subspaces may differ because of the loss derivatives; nevertheless, the claim that this discrepancy has limited practical impact is primarily supported by the gradient-energy comparisons in Figure 3 and Appendix Figures 4–5 (pp. 6, 18–19). Appendix Figures 4–5 (pp. 18–19) provide detailed evidence across vision tasks, ViT backbones, and layer types, but the manuscript does not provide an equivalent quantitative comparison between FCP and GCP cores for the Flan-T5 language experiments; no direct evidence was found in the manuscript. This limitation affects the central justification for FCP: although the empirical results demonstrate its effectiveness, the extent to which dominant feature directions reliably approximate gradient-sensitive directions across different architectures and task distributions remains insufficiently established.
  2. The practical data requirements and validation procedure are not fully clarified relative to the label-free and lightweight nature of the proposed method: FCP avoids backward propagation and calibration labels, but it still requires task-specific calibration examples and validation-based rank selection, and these requirements should be more clearly distinguished from fully data-free merging. Algorithm 1 (Appendix A, p. 13) explicitly requires calibration sets from every downstream task and performs forward passes through the pretrained model to collect input and output features. Section 6.1 (p. 7) specifies that each task contributes 64 unlabeled calibration examples drawn from its training set and that the protection rank is selected from {1, 2} using a validation set. Appendix D.2 (p. 16) demonstrates strong robustness to calibration-set size on vision tasks, including configurations with only one exemplar per task; however, this analysis does not establish the same calibration efficiency for language models or clarify whether the validation-based rank selection requires access to task labels. Consequently, FCP is clearly label-free in its feature-collection procedure, but it is not entirely independent of downstream task data, and this distinction is relevant when considering deployment in settings where the original task datasets are unavailable or where obtaining representative calibration examples is difficult.
Questions for authors
  1. Can the authors provide quantitative comparisons between the FCP and GCP subspaces on Flan-T5-base, particularly across attention and feed-forward layers under both full fine-tuning and LoRA? This matters because the central mechanism relies on dominant input-feature directions approximating gradient-sensitive directions, the mathematical containment result does not guarantee alignment of their leading singular subspaces, and the current quantitative evidence is concentrated on ViT models. Please report the rank-normalized gradient energy captured by FCP and GCP across representative Flan-T5 layers, and investigate whether their principal-subspace alignment varies between full fine-tuning and LoRA.
  2. Can the authors clarify whether the validation-based selection of protection rank (r) requires labeled validation data, and can FCP maintain its reported performance when calibration examples come from an unlabeled public or external dataset rather than the original training data of each downstream task? This matters because the practical applicability of FCP depends on whether task-specific calibration data and labeled validation information are required, and these requirements determine how broadly the method can be used in settings where downstream training datasets are unavailable. Please specify the complete data-access requirements of FCP, clarify the role of labels during rank selection, and evaluate whether a fixed or label-free rank-selection strategy can achieve comparable results.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 7RX9 (Part 1)

Q1.1: Please report the rank-normalized gradient energy captured by FCP and GCP on Flan-T5-base.

A1.1:

Thank you for this helpful suggestion. The rank-normalized gradient energy captured by FCP and GCP across all 8 NLP tasks is as follows:

TaskCSPFCPGCP
CoLA9.588e-056.464e-039.586e-03
MNLI4.004e-024.483e+004.581e+00
MRPC1.881e-035.638e-029.278e-02
QNLI5.138e-038.307e-019.830e-01
QQP3.234e-033.828e-014.371e-01
RTE6.056e-037.960e-011.020e+00
SST-24.818e-043.395e-024.820e-02
STS-B3.970e+002.059e+022.091e+02

Rank-normalized gradient energy across layer types on Flan-T5-base:

Layer typeCSPFCPGCP
Attn-Q2.654e+008.869e+011.081e+02
Attn-K3.358e-021.271e+001.583e+00
Attn-V1.953e-022.156e+005.496e+00
Attn-Out2.132e-021.296e+001.539e+00
MLP-FC12.024e-017.433e+006.459e+00
MLP-FC23.326e-012.580e+013.320e+01

The above results are consistent with our findings on ViT, where the gradient energy in FCP is on the same order as GCP and substantially larger than CSP.

Important note: Both the FCP and GCP core subspaces are computed on the pretrained Flan-T5-base parameters. Therefore, LoRA fine-tuning and full fine-tuning share the same subspaces.

We have included the above results in Figures 5 and 7 of the Appendix in the revised manuscript.

Q1.2: Investigate whether GCP and FCP subspaces are aligned.

A1.2:

Thank you for this suggestion. We directly quantify the alignment between the input-side FCP and GCP subspaces using the following three metrics:

  • Subspace overlap: ∣UrX⊤Urgrad∣F2/r|{U_r^X}^\top U_r^{grad}|_F^2/r, where UrXU_r^X and UrgradU_r^{grad} are orthonormal bases of the rank-rr FCP and GCP subspaces. Higher values indicate greater alignment.
  • Mean principal Cosine: the average of Cosine similarity of principal angles, i.e., σi(UrX⊤Urgrad)\sigma_i({U_r^X}^\top U_r^{grad}), where σi(W)\sigma_i(W) indicates the ii-th singular value of WW. Higher values indicate closer alignment.
  • GCP energy coverage: the fraction of energy in GCP's leading eigendirections captured by the rank-rr FCP subspace ∑j=1rλjgrad∥(UrX)⊤ujgrad∥22∑j=1rλjgrad\frac{\sum_{j=1}^{r} \lambda_j^{grad} \|(U_r^X)^\top u_j^{grad}\|_2^2}{\sum_{j=1}^{r}\lambda_j^{grad}}, where (λjgrad,ujgrad)(\lambda_j^{grad},u_j^{grad}) are the leading eigenpairs of ∑tGt⊤Gt\sum_t G_t^\top G_t. Higher values indicate greater alignment.

As shown in the table below, the FCP and GCP subspaces align across ranks well above the random-subspace baseline, indicating that the dominant input-feature directions can serve as informative proxies for gradient-sensitive directions.

BackboneRank (rr)Subspace OverlapMean Principal CosineCoverage of GCP Energy
ViT-B/32154.67%0.686837.41%
ViT-B/32250.17%0.617747.13%
ViT-B/32547.84%0.587858.63%
ViT-B/321050.35%0.616968.50%
ViT-B/32Random (rr=10)1.14%0.08771.14%
ViT-L/14134.81%0.521822.70%
ViT-L/14237.91%0.502431.18%
ViT-L/14540.33%0.513143.47%
ViT-L/141047.10%0.575154.14%
ViT-L/14Random (rr=10)0.85%0.07570.85%
Flan-T5-base133.30%0.526328.73%
Flan-T5-base236.60%0.503540.12%
Flan-T5-base533.67%0.457860.34%
Flan-T5-base1035.68%0.480568.47%
Flan-T5-baseRandom (rr=10)0.88%0.07571.21%

We have included these results in Section 5 of the main text and Section D.8 of the Appendix in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 7RX9 (Part 2)

Q1.3: Are validation labels required, and what does “label-free” mean?

A1.3:

Good point. We clarify the following two points:

  • We clarify that “label-free” in our paper refers specifically to the feature-collection and projection steps. We do use a labeled validation set to select the protection rank r, as stated in §6.1. We have revised the related statements in the manuscript to make this distinction explicit.
  • When labeled validation is unavailable, we recommend the default r=2r=2 (Appendix D.5). This may cause a slight performance drop in some scenarios (see Table below), but remains a practical option.

We would like to emphasize that using a validation set for hyperparameter selection is standard practice in model merging, including for data-free methods such as CSP and Task Arithmetic.

MethodViT-B/32 (8)ViT-B/32 (14)ViT-B/32 (20)ViT-L/14 (8)ViT-L/14 (14)ViT-L/14 (20)
Task Arithmetic69.5065.7660.6382.0877.5370.98
Task Arithmetic w/ CSP71.5067.3761.8483.6078.7171.68
Task Arithmetic w/ FCP76.6071.2165.3687.3483.1575.26
Task Arithmetic w/ FCP (r=2r=2)76.6071.2164.3987.3483.1575.13
TSV83.0878.3873.2090.5787.8482.81
TSV w/ CSP84.4379.7475.0691.6488.9185.48
TSV w/ FCP85.5682.0078.3392.6890.2088.52
TSV w/ FCP (r=2r=2)85.5681.9177.7592.6890.2088.46

We have included the above results in Table 11 of the Appendix in the manuscript, and revised the statements related to “label-free”.

Q1.4: Can external data replace validation data to locate the optimal value of rr?

A1.4:

We thank the reviewer for the suggestion. We analyze whether external data can be used to select rr through the following two experiments.

Experiment 1: OOD data as external data. We conduct experiments on 20 tasks, where 8 tasks are merged, and select rr using either unseen tasks (not involved in merging) as external data or seen-task performance. As shown below, the optimal rr differs substantially from that obtained from seen tasks.

MethodNum. of tasksEvaluation Datar=1r=2r=3r=4r=5r=6r=7r=8r=9r=10
ISO-CTS + FCP8Unseen Task (last 12 tasks)59.3059.5659.5959.7159.8159.8559.8759.9959.9960.03
ISO-CTS + FCP8Seen task79.6478.3777.9077.4577.1576.7876.4676.2175.9875.77
TSV + FCP8Unseen Task (last 12 tasks)57.6258.2358.6158.8259.0159.3859.6059.5559.6059.75
TSV + FCP8Seen task85.3985.5685.3885.1884.9884.7484.5384.3684.2284.03

Experiment 2: In-distribution data as external data. We slightly perturb the seen-task evaluation data (adding Gaussian noise) and use it as external data. In this case, the optimal rr is close to that obtained from the seen task.

MethodNum. of tasksEvaluation Datar=1r=2r=3r=4r=5r=6r=7r=8r=9r=10
ISO-CTS + FCP8Seen task with slight perturbation75.3476.7675.7874.9874.3574.2173.6973.5673.1772.93
ISO-CTS + FCP8Seen task79.6478.3777.9077.4577.1576.7876.4676.2175.9875.77
TSV + FCP8Seen task with slight perturbation83.1283.3183.0282.9182.6582.4782.1882.1081.8681.72
TSV + FCP8Seen task85.3985.5685.3885.1884.9884.7484.5384.3684.2284.03

Therefore, whether external data can be used to select rr depends on how similar it is to the seen data. If the external data is in-distribution with the seen data, it can be used to choose rr; otherwise, the selected rr may be far from optimal.

We have included these results in Section D.12 of the Appendix in the revised manuscript.

Q1.5: What is the robustness to exemplar-set size on Flan-T5-base?

A1.5:

As shown in the table below, FCP (equipping to Task Arithmetic) is robust to the exemplar-set size on Flan-T5-base under both full fine-tuning and LoRA fine-tuning.

Num. of exemplars1248163264
Full fine-tune80.780.880.880.880.980.980.9
LoRA fine-tune79.179.279.379.279.279.279.3

We have included these results in Table 11 in the revised manuscript.

Dear Reviewer 7RX9, Thanks again for your comments. As the discussion deadline (Nov 18, AoE) approaches, do you have any further suggestions or remaining concerns? If our response has addressed your concerns, we would be grateful if you could consider updating your score.

As the discussion deadline is now only a few hours away, we wanted to send a final gentle reminder. If our responses have addressed your concerns, we would be very grateful if you could consider updating your score before the deadline. We sincerely appreciate your time and feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to AC and all Reviewers

We thank the Area Chair and all reviewers for their constructive feedback and valuable suggestions. We are pleased that the reviewers recognize the sensitivity-based perspective of our paper and its analytical and empirical support (7RX9). Reviewers F7DJ and a2ri find our proposed method practical, simple, and low-overhead. They (7RX9, F7DJ, and a2ri) acknowledge its consistent improvements over CSP, and 7RX9 and F7DJ further recognize the breadth of our evaluation.

We have revised the manuscript accordingly and marked all changes in blue. We provide point-by-point responses to each reviewer below and look forward to further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.