acceptodds
Under review as a conference paper at ICLR 2027

Preserving Anchored Cores in Task-Vectors for Model Merging

Abstract

Model merging has emerged as a training-free alternative to multi-task learning, aiming to combine multiple task-specific fine-tuned models into a single multi-task model. Most existing model merging approaches follow the Task Arithmetic paradigm, which decomposes fine-tuned weights into pre-trained parameters and task vectors, and performs merging exclusively in the task-vector space. The effectiveness of this paradigm implicitly relies on the assumption that task-specific knowledge is encoded solely within task vectors. In this paper, we argue that this assumption generally does not hold due to the intrinsic task preferences of pre-trained models. Specifically, we identify Load-Bearing Wall (LBW) dimensions, namely some task-critical knowledge that remains embedded in the pre-trained weights rather than being fully transferred into task vectors. We characterize LBW dimensions from both scalar-weight and subspace perspectives, thereby covering the major paradigms of existing model merging methods. Our analysis reveals that, by ignoring LBW dimensions, task-vector-based approaches fail to fully resolve task conflicts and may inadvertently damage task-specific knowledge encoded in the pre-trained model, leading to degradation. To address this issue, we propose PACT, which preserves the anchored task-specific cores (i.e., LBW components) within task vectors by aligning their orthogonal complements with the subspace of the pre-trained weights. These aligned subspace components are then removed from the task vectors before applying existing model merging algorithms. Furthermore, we develop an efficient variant based on randomized SVD to improve scalability. Since PACT approaches model merging from a complementary perspective, it can be seamlessly integrated with existing methods. Extensive experiments across multiple benchmarks demonstrate that PACT consistently enhances mainstream model merging approaches and establishes new state-of-the-art performance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merging, Anchored Cores

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aGGQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper challenges the prevailing Task Arithmetic assumption that task-specific knowledge is fully encoded in task vectors. Through Fisher-information-based perturbation experiments and subspace geometry analysis, the authors identify "Load-Bearing Wall" (LBW) dimensions: task-critical parameters anchored in pre-trained weights that receive negligible fine-tuning updates yet are essential for task performance. To address the performance degradation caused by ignoring LBW dimensions during merging, the authors propose PACT, a data-free, plug-and-play filtering framework that constructs a global orthogonal shield for each task from the LBW subspaces of all other tasks, and projects out the components of its task vector that would invade those protected directions. The method is simple yet effective, and an efficient randomized SVD variant is further provided for scalability. Extensive experiments on vision and NLP benchmarks show consistent improvements over mainstream merging methods.

Critical strengths
  1. The identification of the LBW phenomenon brings a valuable conceptual insight to the model merging community. It challenges the long-standing implicit assumption of the task vector paradigm and reveals that task-critical knowledge can remain anchored in pre-trained weights with negligible fine-tuning updates. This finding helps explain the performance degradation of existing merging methods and opens up a new direction for designing more robust merging algorithms.

  2. The proposed PACT framework has good practical applicability. It is data-free, simple to implement, and can be seamlessly integrated with both scalar-based and subspace-based mainstream merging methods. The randomized SVD variant effectively reduces computational complexity and improves scalability. Consistent performance gains across multiple vision and NLP settings demonstrate its effectiveness as a plug-and-play enhancement for existing merging pipelines.

Critical weaknesses
  1. The work lacks sufficient distinction from the concurrent CSP method (Sun et al., ICML 2026), which also recognizes that pre-trained weights contain task-critical structures that should be preserved during merging. A reader could reasonably view PACT as a refinement of CSP rather than an independent contribution. Although a comparison is provided in Appendix D.1, the paper does not clearly articulate the fundamental differences in core motivation, design logic and applicable scenarios between the two methods, which weakens the perceived originality of this work.

  2. There is a disconnect between the motivating experiments and the proposed method. Section 3.2.1 identifies LBW dimensions at the scalar parameter level using Fisher information, but the PACT method only relies on the spectral proxy defined in Eq. 3. The paper does not show what fraction of the Fisher-identified crucial parameters lie within the span of V_{t,LBW}. Without this validation, the Fisher-based motivation experiments and the spectral-based method design read as two separate stories, and the spectral proxy cannot be firmly justified as an effective surrogate for true LBW dimensions.

Questions for authors

Could you more explicitly elaborate the core differences between PACT and the concurrent CSP work? What are the essential distinctions in terms of protection targets, design rationale, and applicable scenarios? What is the key incremental contribution of PACT compared to CSP?

What fraction of the LBW parameters identified by Fisher information fall within the span of the spectral LBW subspace V_{t,LBW} used in PACT? To what extent can the spectral proxy reliably represent the true LBW dimensions identified in the motivation experiments?

Have you evaluated how PACT performs as the number of tasks T grows significantly? How do you address the potential issue that the protection subspace becomes overly high-dimensional with more tasks? Is there an adaptive scheme to control the dimension of the protection space?

5 comments

We sincerely thank Reviewer aGGQ for the constructive and insightful comments, and we are glad that Reviewer recognize our paper to be insightful, practical, and easy to follow. To address Reviewer’s concerns, we provide pointwise responses below.

W1&Q1: Could you more explicitly elaborate the core differences between PACT and the concurrent CSP work? What are the essential distinctions in terms of protection targets, design rationale, and applicable scenarios? What is the key incremental contribution of PACT compared to CSP?

A1: Thank you for encouraging us to clarify this distinction. PACT and CSP differ first in their motivation, which leads to different protection targets and mechanisms.

Motivation. CSP starts from evidence that pre-trained weights are near-optimal within a dominant core subspace under multi-task losses, motivating preservation of that shared core during merging. PACT starts from a different question: whether fine-tuning update magnitude adequately reflects the importance of parameters or directions to a downstream task. Our perturbation and ablation experiments show that some components remain important despite receiving little update during fine-tuning. These LBW components motivate protecting pre-trained dependencies that are barely represented in a task’s update.

Protection targets and mechanisms. CSP determines its protection space from the pre-trained weights, without incorporating each task’s adaptation directions. PACT additionally uses each task’s leading update directions to construct its LBW subspace:

\left[ \left(I-V_{t,k}V_{t,k}^{\top}\right)V_{0,K} \right].$$ Here, $V_{0,K}$ contains the leading right singular vectors of the pre-trained weights, and $V_{t,k}$ contains those of task $t$’s update. PACT then filters each task update against the combined LBW subspaces of the other tasks before merging. Its protection space is therefore conditioned on task updates rather than determined by pre-trained weights alone. This distinction between task-conditioned shields and a static global core was also recognized by Reviewer 4FGQ, who noted that the S-PACT ablation supports task-conditioned protection. **Design assumptions and applicable setting.** Both methods support merging without training data. PACT’s construction does not require the pre-trained core to be near-optimal; its theoretical bound instead uses local smoothness around the fine-tuned model and the projected gradient and update magnitudes. The distinction concerns the rationale and construction of protection within the model-merging setting. **Incremental contribution and empirical evidence.** PACT contributes the identification of important but weakly updated pre-trained components, together with a task-conditioned mechanism for protecting them during merging. As reported in Table 11, PACT-Iso-C and PACT-TSV-M outperform their CSP counterparts in average absolute accuracy in all nine backbone/task-scale settings, while PACT-TA does so in seven of nine. To further examine the contribution of task conditioning and mutual protection, we compare PACT with our static-protection ablation, S-PACT, which filters each task update using only the pre-trained right-singular core: $\tilde{\Delta}_t^{\text{S-PACT}} = \Delta_t (I - V_{0,K} V_{0,K}^T)$ As shown in Table 13, PACT achieves higher average absolute accuracy than S-PACT in all nine Iso-C settings and eight of nine TA settings. For example, on ViT-B/16 with 14 tasks, accuracy increases from 71.2% to 76.8% with TA and from 84.2% to 86.1% with Iso-C. These results support the practical value of combining task-conditioned LBW construction with mutual protection. **Table 13. Static core-protection ablation.** Entries report average absolute accuracy, with average normalized accuracy in parentheses; both are expressed in percent. Column labels specify the backbone and number of merged tasks. |Method|ViT-B/32, 8|ViT-B/32, 14|ViT-B/32, 20|ViT-B/16, 8|ViT-B/16, 14|ViT-B/16, 20|ViT-L/14, 8|ViT-L/14, 14| ViT-L/14, 20 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Task Arithmetic | 70.8 (76.5) | 65.3 (72.1) | 60.5 (66.8) | 75.4 (79.6) | 70.5 (75.9) | 65.8 (70.8) | 84.9 (88.7) | 79.4 (84.0) | 74.0 (78.1) | |S-PACT-TA | 70.8 (76.3) | 65.8 (72.7) | 61.7 (68.3) | 76.2 (80.5) | 71.2 (76.7) | 66.4 (71.6) | 85.0 (88.7) | 80.3 (85.1) | 74.7 (78.9) | |PACT-TA| 75.2 (81.0) | 66.4 (73.3) | 59.9 (66.4) | 85.5 (90.3) | 76.8 (82.7) | 68.7 (74.0) | 92.8 (96.9) | 87.0 (92.0) | 83.0 (87.7) | |Iso-C| 86.6 (93.3) | 79.5 (87.3) | 75.1 (82.2) | 90.8 (96.0) | 84.4 (90.7) | 79.7 (85.4) | 94.7 (98.9) | 89.2 (94.3) | 87.7 (92.4) | |S-PACT-Iso-C| 86.5 (93.2) | 79.5 (87.2) | 74.8 (81.7) | 90.8 (96.0) | 84.2 (90.4) | 79.7 (85.6) | 94.7 (98.9) | 89.3 (94.4) | 87.7 (92.4) | | PACT-Iso-C | 87.3 (94.1) | 81.3 (89.0) | 78.7 (86.0) | 92.0 (97.2) | 86.1 (92.3) | 83.4 (89.3) | 95.1 (98.3) | 90.5 (95.5) | 90.3 (95.0) |

W2&Q2: What fraction of the LBW parameters identified by Fisher information ...

A2: Task Fisher information depends on the input distribution through gradient statistics. In our data-free setting, only pretrained and fine-tuned weights are available, making high-Fisher parameters difficult to identify without assumptions about task data. PACT therefore constructs a protection subspace from pretrained directions and task-update geometry. We conducted additional experiments examining its coordinate-level correspondence with Fisher-selected parameters and its functional role in protecting them.

We analyze coverage on eight standard vision tasks using CLIP ViT-B/16. For each task, we estimate diagonal Monte Carlo model Fisher information using 1,000 training examples, one label sampled per example from the fine-tuned model’s predictive distribution, and seed 0. In each weight matrix, we intersect the top 30% of coordinates by Fisher information with the top or bottom 30% by absolute task-update magnitude. We compare the large-update/high-Fisher group with Vt,kV_{t,k}, and the small-update/high-Fisher group with Vt,LBWV_{t,\mathrm{LBW}}, using (K,k)=(15,8)(K,k)=(15,8).

Since parameter coordinates and spectral directions differ, we quantify correspondence through coordinate-projection energy, rather than a binary fraction of recovered parameters. For a binary mask M∈{0,1}m×nM\in\{0,1\}^{m\times n} and an orthonormal basis Q∈Rn×dQ\in\mathbb{R}^{n\times d}, we define

C(M,Q)=∑i,jMij(QQT)jj∑i,jMij.C(M,Q)=\frac{\sum_{i,j}M_{ij}(QQ^T)_{jj}}{\sum_{i,j}M_{ij}}.

This measures selected coordinate directions’ average projection energy. The expected coverage of uniformly random coordinates in the same matrix is B=d/nB=d/n.

Across eight tasks, the large-update/high-Fisher group has mean coverage of 0.949% in Vt,kV_{t,k}, compared with a random expectation of 0.865%. The small-update/high-Fisher group has mean coverage of 1.740% in Vt,LBWV_{t,\mathrm{LBW}}, compared with 1.621% for random coordinates. Both comparisons show low absolute coordinate-projection coverage, which should be interpreted relative to their low-dimensional random baselines. Both averages exceed random expectations, indicating a positive but limited coordinate-level association.

To test whether Fisher selection adds correspondence beyond low update magnitude, we compare MlowM_{\mathrm{low}}, containing the bottom 30% of coordinates by update magnitude, with Mlow+FM_{\mathrm{low+F}}, which also requires membership in the top 30% by Fisher information. Both masks are evaluated with the same LBW basis. Relative enrichment over random is

G(M)=(C(M,Q)‾B‾−1)×100%.G(M)=\left(\frac{\overline{C(M,Q)}}{\overline{B}}-1\right)\times100\%.

ScopeRandom expectationMlowM_{\mathrm{low}}Mlow+FM_{\mathrm{low+F}}G(Mlow)G(M_{\mathrm{low}})G(Mlow+F)G(M_{\mathrm{low+F}})
Global average1.6211%1.6365%1.7395%+0.95%+7.30%
Attention input projection1.9531%1.9389%2.3222%−0.73%+18.90%

Coverage and random expectation are arithmetic means; enrichment is computed from these means.

Adding Fisher selection increases global coverage from 1.6365% to 1.7395%, and enrichment over random from 0.95% to 7.30%. For attention input projections, enrichment rises from −0.73% to 18.90%. These results support partial correspondence between Fisher-selected LBW coordinates and the spectral construction.

We also test its functional significance through fixed Fisher-mask attacks. In the SUN397 experiment with Cars-derived perturbations, we decompose the full PACT correction into components inside and outside the original Fisher-selected low-update mask. The table reports the loss reduction retained with only the correction inside the mask, relative to that achieved by full PACT. Both reductions use the same unprotected attack as reference.

Attack strength α\alphaK=15K=15K=30K=30K=60K=60
130.1%29.7%33.5%
2070.7%71.0%76.0%
3065.7%68.2%76.2%

The task-update rank is k=8. These ratios measure retained loss reduction in the controlled attack experiment, not a causal decomposition of overall merging gains.

Under stronger attacks, corrections restricted to Fisher-selected LBW coordinates retain 65.7%–76.2% of the full PACT loss reduction. They contribute less under the weakest attack; corrections outside the mask also contribute. Thus, limited coordinate-projection coverage can accompany substantial functional benefits at Fisher-selected LBW coordinates.

This supports our motivation: parameters that change little during fine-tuning can remain task-critical and benefit from protection against interfering updates. Fisher analysis identifies this phenomenon using data; the spectral construction protects using weights alone. We will clarify this relationship as partial geometric correspondence supported by controlled functional evidence, and investigate more direct data-free protection of Fisher-identified LBW parameters in future work.

Q3: Have you evaluated how PACT performs as the number of tasks T grows significantly? How do you address the potential issue that the protection subspace becomes overly high-dimensional with more tasks? Is there an adaptive scheme to control the dimension of the protection space?

A3: Thank you for raising this issue. Our existing experiments cover 8, 14, and 20 tasks, including the simultaneous merging of 20 independently fine-tuned models.

These experiments show that PACT remains effective at this scale. Across multiple baseline–backbone combinations, its gains over the original merging methods are even larger than in the 8-task case, as shown in Table 4. These results indicate that the protection mechanism retains practical value as the number of tasks increases within the evaluated settings.

To further address the reviewer’s concern, we additionally evaluated PACT on 50 tasks, adopting the CIFAR100/50 class partition used in [1]. This setting partitions CIFAR-100 into 50 two-class tasks, with each expert independently fine-tuned from the same pre-trained model. It complements our existing experiments across different datasets. We report both 100-way accuracy, which requires prediction among all 100 classes without task identity, and task-aware 2-way accuracy, which restricts prediction to the two classes of the known task.

Method100-way accuracy (%)Task-aware 2-way accuracy (%)
Pre-trained66.9597.54
TA64.0097.72
PACT-TA69.8698.12
TIES-Merge50.4296.11
PACT-TIES71.1198.14
TSV-Merge71.1399.12
PACT-TSV73.5298.71
Iso-C68.9698.81
PACT-Iso-C74.8998.87

PACT improves 100-way accuracy across all four baselines, with gains of 5.86, 20.69, 2.39, and 5.93 percentage points for TA, TIES-Merge, TSV-Merge, and Iso-C, respectively. Task-aware accuracy also improves for three baselines, while PACT-TSV exhibits a small decrease of 0.41 percentage points. These results provide additional empirical evidence that PACT exhibits a degree of robustness to increasing task count on a practical model-merging benchmark.

Regarding protection-space growth, Eq. (6) in the main text constructs the joint span of the other tasks’ LBW subspaces. Linearly dependent directions do not enlarge this span, but this does not guarantee that the protection dimension remains small as the number of tasks increases. A larger protection space can restrict useful task updates, with its impact depending on the removed components and their task importance. Our additional 50-task results provide empirical evidence of robustness to increased task count, while the possibility of over-constraint at larger scales remains open. The current configuration does not impose an adaptive budget on the joint protection space. We will clarify this scope of applicability in the discussion and investigate adaptive, importance-aware dimension control to identify which LBW directions require joint protection and balance their preservation against the retention of useful task updates as future work.

Reference:

[1] MagMax: Leveraging Model Merging for Seamless Continual Learning. ECCV 2024.

Reviewer_4FGQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper questions a core assumption behind task-vector-based model merging — that task-specific knowledge lives entirely in the fine-tuning updates. The authors show that certain task-critical knowledge actually remains anchored in the pre-trained weights, which they term Load-Bearing Wall dimensions, and demonstrate through perturbation and subspace ablation experiments that standard merging can damage these components and degrade performance. To address this, they propose PACT, a data-free plug-in method that shields each task's anchored subspace from interference by other task vectors before merging, with a randomized-SVD variant for scalability. Experiments across vision and language benchmarks, covering both full and LoRA fine-tuning, show that PACT consistently improves existing merging methods and achieves state-of-the-art results.

Critical strengths
  • The LBW phenomenon addresses a real blind spot in the task-vector paradigm. The perturbation experiments (Table 1, Figure 2) and especially the subspace ablation (Table 3, with degradation 458× larger than a matched random control) are convincing and cleanly designed causal evidence, not just correlational analysis.
  • PACT is essentially orthogonal projection against a union of other tasks' anchor subspaces — easy to understand, data-free, and composable with existing methods (TA, TSV-M, Iso-C). The distinction from concurrent CSP (task-conditioned dynamic shields vs. a static global core) is clearly articulated, and the S-PACT ablation supports the necessity of task-conditioned protection.
  • The randomized-SVD variant with explicit complexity/memory analysis makes the method practically usable, addressing an obvious objection to a T+1-SVD-per-layer pipeline.
Critical weaknesses
  • Table 6 is concerning: PACT-Iso-C lowers absolute average accuracy relative to Iso-C (0.7164 → 0.7093) yet is reported as winning on normalized accuracy (85.32 → 86.46). Individual task columns are also erratic (e.g., MNLI drops sharply from 0.7873 to 0.5856 while CoLA jumps from 0.3013 to 0.4960). The paper claims "PACT outperforms all other merging methods" for NLP, which is not well supported as stated. This should be clarified — at minimum, the discrepancy between absolute and normalized metrics and the MNLI collapse deserve discussion.
  • The paper empirically establishes the phenomenon but offers little mechanistic explanation for why task-critical knowledge remains anchored in pre-trained weights. Is this an artifact of short fine-tuning, small learning rates, or CLIP-style encoders? Some analysis (e.g., how LBW mass varies with fine-tuning epochs/steps) would substantially strengthen the paper's claim that this is a fundamental rather than incidental property.
  • The choice (K,k)=(15,8) (and (7,4) for LoRA) appears somewhat arbitrary. Since V_protect is built from top-K pre-trained singular vectors, results could be sensitive to these ranks, to the steepness of the pre-trained spectrum, and to the thresholding choices in the scalar mask construction (30%/30% → ~9% → <2%). Ablations exist in the appendix but the main text gives little guidance.
  • All experiments are on ViT-B/L, RoBERTa-base, and GPT-2. Given that much of the current merging literature targets decoder LLMs (7B+), at least one experiment in that regime would meaningfully broaden the claim of generality.
Questions for authors

Please refer to weakness.

3 comments

We sincerely thank Reviewer 4FGQ for the constructive and insightful comments, and we are glad that Reviewer recognize our paper to be cleanly motivated, practical, and easy to follow. To address Reviewer’s concerns, we provide pointwise responses below.

W1: Table 6 is concerning: PACT-Iso-C lowers absolute average accuracy relative to Iso-C (0.7164 → 0.7093) yet is reported as winning on normalized accuracy (85.32 → 86.46). Individual task columns are also erratic (e.g., MNLI drops sharply from 0.7873 to 0.5856 while CoLA jumps from 0.3013 to 0.4960). The paper claims "PACT outperforms all other merging methods" for NLP, which is not well supported as stated. This should be clarified — at minimum, the discrepancy between absolute and normalized metrics and the MNLI collapse deserve discussion.

A1: Thank you for pointing this out. We agree that these results require a distinction between aggregate improvements and task-level trade-offs. PACT-Iso-C reduces the average absolute score from 0.7164 to 0.7093, with MNLI accuracy decreasing from 78.73% to 58.56%, while CoLA improves from 0.3013 to 0.4960. We retain these outcomes alongside the improvements to provide a complete account of the method’s benefits and limitations.

The normalized average nevertheless increases from 85.32% to 86.46%. Because this metric measures each task’s score relative to its individually fine-tuned reference, gains on tasks such as CoLA and RTE contribute differently than under absolute averaging. This indicates an aggregate relative-performance improvement, but does not negate the absolute-average decrease or the MNLI regression.

A possible explanation for the MNLI degradation is the interaction between PACT and Iso-C’s spectral equalization. Iso-C computes singular directions from the aggregated updates and assigns them a common singular value. PACT changes this aggregate before decomposition, potentially altering the influence of task-relevant directions after reconstruction. Marczak et al., 2025 MNLI covers multiple genres and diverse inference phenomena, including negation and quantification; changes to directions supporting these semantic distinctions may therefore affect its performance substantially. Williams et al., 2018 We regard this as a plausible explanation rather than a confirmed cause. The interaction is method-dependent: PACT improves MNLI with TA and TSV-M in the same table.

To address your concern about broader applicability, we additionally evaluated PACT-TIES and PACT-DARE-TIES. Both improve their corresponding baselines in absolute and normalized averages, although individual-task regressions remain. The complete results are provided below.

MethodCoLASST-2MRPCSTSBQQPMNLIQNLIRTEAvg.Norm Avg.
Individual0.55110.94720.87990.87700.89460.87450.92440.66430.8266-
Weight Averaging0.00.55280.75490.42010.39930.41730.59820.47290.451952.54
TA w/ DARE0.00.80730.78110.30590.71240.64010.77720.55600.572566.19
Ties-Merging0.11390.91170.7134-0.04760.72450.79310.82780.41520.556564.54
PACT-Ties0.27260.80160.82580.32880.8090.58050.7430.4260.598370.83
TIES w/ DARE0.06560.86010.78030.14640.72630.6660.790.49460.566265.67
PACT-Dare-Ties0.10080.83490.83270.69070.79120.55260.56320.49460.607670.85
Task Arithmetic0.00.83140.78410.29580.70250.65230.77850.53430.572466.05
PACT-TA0.17630.90600.77990.12550.77110.770.81550.53790.610371.75
TSV-M0.20200.93000.78950.49770.82600.74560.82880.51260.666578.22
PACT-TSV-M0.44530.92550.77020.64890.79710.75420.85410.51260.713585.62
Iso-C0.30130.90940.77550.62060.84700.78730.85100.63900.716485.32
PACT-Iso-C0.49600.88990.77280.63060.81100.58560.79540.69310.709386.46

Results of merging RoBERTa models on eight GLUE tasks. Task scores and their arithmetic average are reported on their original metric scales; normalized averages are percentages. Bold denotes the best result among merging methods, excluding the individually fine-tuned reference.

We will replace the statement “PACT outperforms all other merging methods” with the more precise description: On the evaluated RoBERTa/GLUE setting, PACT improves the normalized average for all five tested merging baselines and the absolute average for four of them. These aggregate improvements coexist with task-specific regressions, including a substantial MNLI decline for PACT-Iso-C.

W2: The paper empirically establishes the phenomenon but offers little mechanistic explanation for why task-critical knowledge remains anchored in pre-trained weights...

A2: Thank you for raising this question. A plausible explanation is that fine-tuning builds on computations already learned during pre-training. When a pre-trained component already provides useful functionality, adaptation may primarily adjust how that component is used, allowing it to remain important without requiring a large update.

Recent studies of large language models support this reuse-based interpretation. [1] find that fine-tuning enhances an existing entity-tracking circuit while largely preserving its functionality. [2] observe substantial overlap in circuit nodes before and after fine-tuning, alongside significant changes in their connections. These findings suggest that adaptation can reuse pre-existing computational components, although they do not directly establish preservation of the corresponding weights.

To address your concern directly in our setting, we extended fine-tuning to four times the original training duration and increased the learning rate up to fivefold on Cars, DTD, and EuroSAT. We held the identified LBW subspaces fixed and measured both their subsequent updates and the effects of restoring or removing their weight components, testing whether their importance persists beyond the original training budget.

1. Experimental design.

We use CLIP ViT-B/16 on Cars, DTD, and EuroSAT, whose standard training lengths are E=35, 76, and 12 epochs, respectively. For each task, we train for 4E epochs with peak learning rates of 10−510^{-5}, 2×10−52\times10^{-5}, and 5×10−55\times10^{-5}. We also train an EE-epoch reference at 10−510^{-5} for each task, giving 12 training runs.

All runs use the same pretrained checkpoint, seed 0, AdamW with weight decay 0.1, effective batch size 128 with microbatches of 32, gradient clipping at norm 1, FP32 training, and a frozen classification head. Each run uses 500 warmup optimizer steps followed by cosine decay over its full training horizon. Checkpoints are collected at ⌈0.25E⌉,⌈0.5E⌉,E,2E,4E\lceil0.25E\rceil,\lceil0.5E\rceil,E,2E,4E. Validation sets contain 814 Cars, 188 DTD, and 5,400 EuroSAT examples.

For each run, we construct the LBW subspace at its epoch-EE checkpoint:

QE=orth⁡[(I−VE,kVE,k⊤)V0,K],PE=QEQE⊤,Q_E=\operatorname{orth}\left[(I-V_{E,k}V_{E,k}^{\top})V_{0,K}\right], \qquad P_E=Q_EQ_E^\top,

using K=15,k=8. We analyze all 50 eligible visual parameter matrices, including positional embeddings and the visual projection. Every observed LBW space has dimension 15. The subspace is held fixed at subsequent checkpoints, so persistence is evaluated without selecting new low-update directions.

2. Do these fixed directions remain functionally important?

To test whether task performance continues to depend on their pretrained values, we evaluate two interventions at E,2E,4EE,2E,4E:

Wsrestore=Ws−(Ws−W0)PE,Wsremove=Ws−WsPE.W_s^{\mathrm{restore}}=W_s-(W_s-W_0)P_E, \qquad W_s^{\mathrm{remove}}=W_s-W_sP_E.

Restoration removes only the learned update within the subspace; removal deletes its complete weight component. The random control applies the same removal to a dimension-matched Gaussian subspace sampled from the full input space, orthonormalized with seed 0 and held fixed across checkpoints. Interventions affect all eligible matrices simultaneously, with the remaining parameters and classification head unchanged.

TaskPeak LRAccuracy (%)Δrestore\Delta_{restore}Δremove\Delta_{remove}Δrandom\Delta_{random}
Cars10−510^{-5}87.71+0.86+0.86−87.35-87.35−3.32-3.32
2×10−52\times10^{-5}87.10−0.49-0.49−86.86-86.86−2.83-2.83
5×10−55\times10^{-5}84.64+0.37+0.37−83.42-83.42−2.09-2.09
DTD10−510^{-5}77.13+0.53+0.53−73.40-73.40−0.53-0.53
2×10−52\times10^{-5}79.790.000.00−73.94-73.94−1.06-1.06
5×10−55\times10^{-5}67.55−1.60-1.60−49.47-49.47−0.53-0.53
EuroSAT10−510^{-5}99.00+0.06+0.06−79.43-79.43−0.33-0.33
2×10−52\times10^{-5}98.69+0.09+0.09−57.46-57.46−0.13-0.13
5×10−55\times10^{-5}98.67+0.04+0.04−29.48-29.48−0.02-0.02

After extended training, restoring pretrained values changes accuracy by only −1.60 to +0.86 points, whereas removing the complete LBW component causes 29.48–87.35 points of degradation. Random removal causes only 0.02–3.32 points of degradation. Thus, the pretrained values in these directions remain sufficient to preserve nearly the same performance, while the directions themselves remain important.

3. Do these fixed directions accumulate substantial updates?

We measure both the fraction of total update energy within the fixed LBW space and its update magnitude relative to the pretrained component:

ρfixed(s)=∥(Ws−W0)QE∥F2∥Ws−W0∥F2,rfixed(s)=∥(Ws−W0)QE∥F∥W0QE∥F.\rho_{\text{fixed}}(s)=\frac{\|(W_s-W_0)Q_E\|_F^2}{\|W_s-W_0\|_F^2}, \qquad r_{\text{fixed}}(s)=\frac{\|(W_s-W_0)Q_E\|_F}{\|W_0Q_E\|_F}.

TaskLR 10−510^{-5}LR 2×10−52\times10^{-5}LR 5×10−55\times10^{-5}
Cars2.902.923.00
DTD2.902.943.05
EuroSAT2.442.652.99

At 4E4E, only 2.44–3.05% of task-update energy lies within the fixed LBW spaces on average, including under the fivefold learning-rate increase.

TaskLR 10−510^{-5}LR 2×10−52\times10^{-5}LR 5×10−55\times10^{-5}
Cars1.633.268.29
DTD0.832.125.78
EuroSAT1.222.516.34

Relative update magnitudes increase from 0.83–1.63% at LR 10−510^{-5} to 5.78–8.29% at LR 5×10−55\times10^{-5}. These directions therefore do change under more aggressive optimization, but their updates remain limited relative to their pretrained components, and reverting those updates has little effect on accuracy.

The strength of the dependence can also change: on EuroSAT at LR 5×10−55\times10^{-5}, the removal-induced accuracy drop decreases from 45.54 points at E to 29.48 points at 4E, while remaining substantial.

These results support persistent reuse of task-critical pretrained components under longer training and larger learning rates in the evaluated settings.

Reference:

[1] Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking. ICLR 2024.

[2] Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis. ICML 2025.

W3: The choice (K,k)=(15,8) (and (7,4) for LoRA) appears somewhat arbitrary. Since VprotectV_{protect} is built from top-K pre-trained singular vectors, results could be sensitive to these ranks, to the steepness of the pre-trained spectrum, and to the thresholding choices in the scalar mask construction (30%/30% → 9% → <2%). Ablations exist in the appendix but the main text gives little guidance.

A3: We acknowledge that the selection of the hyperparameters (K, k) is indeed rather empirical. However, as can be seen from the merging performance under various settings in Table 16 of the appendix, the PACT algorithm is robust to different rank configurations, achieving consistent merging results across a fairly wide range. Meanwhile, as noted in our response to the first question from Reviewer pFTQ, hyperparameter combinations that yield better performance also exhibit lower average intrusion energy on the LBW space—a phenomenon that may serve as a direction for future research. We admit that the current work lacks theoretical guidance for the hyperparameter selection of the PACT algorithm, and the choices are empirical. Providing such theoretical guidance would require a more in-depth and detailed analysis of the LBW phenomenon, which we leave as future work.

W4: All experiments are on ViT-B/L, RoBERTa-base, and GPT-2. Given that much of the current merging literature targets decoder LLMs (7B+), at least one experiment in that regime would meaningfully broaden the claim of generality.

A4: Thank you for this suggestion. To address your concern, we conducted additional experiments on MergeBench [1] using Llama-8B, extending our evaluation to an 8B-parameter decoder-only LLM. Given data-access and computational constraints, we focus on four domains: instruction following, mathematics, multilingual understanding, and coding.

As shown below, PACT improves TA’s mathematics and multilingual performance, with a modest increase in the reported four-domain average from 39.10 to 39.77. These gains involve trade-offs: instruction-following performance decreases, while coding performance remains nearly unchanged. The results provide additional evidence of PACT’s applicability to larger decoder-only models, although its benefits remain task-dependent.

TaskTAPACT-TA
Instruction following17.3813.12
Math37.3141.10
Multilingual51.3854.57
Coding50.3450.30
Avg. Acc.39.1039.77

Due to the rebuttal timeline and computational budget, this additional evaluation currently focuses on Llama-8B with TA as the baseline. We will include these results in the revised manuscript and provide more extensive experiments, including comparisons with additional merging baselines, in future versions of the paper.

Reference:

[1] MergeBench: A Benchmark for Merging Domain-Specialized LLMs. NeurIPS 2025.

Reviewer_pFTQOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether task-critical knowledge can remain embedded in pretrained weights rather than being fully represented by fine-tuning updates, and how such knowledge can be damaged during model merging. It characterizes Load-Bearing Wall (LBW) components through Fisher-based parameter analysis and an SVD-based subspace construction. Building on this observation, the authors propose PACT, which constructs task-conditioned orthogonal protection subspaces and removes interfering components from task vectors before applying existing merging algorithms. A randomized SVD variant is introduced to reduce computational complexity. Experiments cover multi-task vision merging with CLIP/ViT, LoRA-adapted vision models, and language models on GLUE benchmarks. Additional studies investigate component contributions, filtering depth, hyperparameter sensitivity, and computational efficiency.

Critical strengths
  1. An interesting and potentially important observation about pretrained knowledge in model merging.The paper challenges the assumption that task vectors fully capture task-specific knowledge and investigates the possibility that important task dependencies remain embedded in pretrained weights.Table 1 (p. 5) demonstrates that perturbing selected small-magnitude, high-Fisher parameters causes substantially greater performance degradation than perturbing safe or random parameter subsets.Table 2 (p. 5) shows that restoring selected parameters to their pretrained values can recover performance after task-vector merging, providing evidence of destructive interference.Table 10 (Appendix C.2, p. 23) extends the subspace ablation analysis across multiple datasets, showing substantial degradation after removing the identified LBW subspaces.This investigation provides a meaningful perspective on the limitations of task-vector-based merging and motivates further research into preserving pretrained representations.
  2. A modular geometric protection mechanism with broad empirical evaluation.PACT introduces a task-conditioned orthogonal projection mechanism that can be integrated with existing merging algorithms without additional model training.Equations (6)–(7) (p. 7) provide a clear formulation of the mutual protection mechanism through aggregation of other tasks' LBW subspaces and orthogonal projection.Table 4 (p. 8) demonstrates substantial improvements for PACT-Iso-C across three ViT backbones and different task scales, with particularly notable gains in larger multi-task settings.Tables 13–17 (Appendix E, pp. 29–32) investigate dynamic protection, filtering depth, component selection, hyperparameter sensitivity, and randomized SVD, providing useful evidence regarding the method's design.The combination of a relatively simple projection operation and compatibility with multiple merging frameworks makes the proposed approach practically interesting.
Critical weaknesses
  1. The identification and validation of LBW subspaces rely on an insufficiently established approximation: the central mechanism assumes that directions outside the leading task-vector subspace contain negligible fine-tuning updates and therefore represent knowledge anchored in pretrained weights, but this interpretation is not fully established by the presented experiments. Equation (3) (p. 5) constructs the LBW subspace by projecting the pretrained top-K subspace onto the orthogonal complement of the task vector’s top-k right singular vectors, yet being orthogonal to the leading singular directions does not necessarily imply that the remaining task-vector components have negligible energy. Equation (4) (p. 6) and Appendix Section A.3 (pp. 17–18) explicitly rely on the approximation that the task update is predominantly concentrated in its leading singular subspace, but the manuscript does not sufficiently quantify the residual update energy or directly validate this approximation across layers and tasks. Moreover, Table 3 (p. 6) compares LBW subspace removal against a random control subspace restricted to the orthogonal complement of the pretrained top-K subspace; since the proposed LBW subspace is itself derived from the pretrained core, this comparison does not establish whether the observed degradation is specifically attributable to task-conditioned LBW directions rather than the general importance of dominant pretrained directions. This issue affects the central interpretation of LBW and, consequently, the justification for PACT’s protection mechanism, and the theoretical analysis in Appendix A also provides conditional bounds rather than establishing that the relevant projection terms are sufficiently small in practice.
  2. The claimed consistent improvements are not fully supported by all reported evaluation metrics, because although PACT performs well in many settings, some results reveal non-negligible trade-offs that deserve further investigation. Table 4 (p. 8) shows that PACT-TA obtains 59.9% on ViT-B/16? More precisely, in the ViT-B/32, 20-task setting, it achieves 59.9%, compared with 60.5% for Task Arithmetic, indicating that improvements are not universal. Table 6 (p. 9) reports an average absolute metric of 0.7093 for PACT-Iso-C, lower than the 0.7164 obtained by Iso-C, despite an improvement in normalized average from 85.32 to 86.46. The same table shows that PACT-Iso-C reduces MNLI performance from 0.7873 to 0.5856 and QNLI performance from 0.8510 to 0.7954, suggesting that improvements in aggregate normalized performance can coexist with substantial degradation on individual tasks. These results do not invalidate the overall effectiveness of PACT, but they make the broad claim of consistent improvement less convincing, and a clearer analysis of task-level trade-offs is needed to establish the reliability and practical benefits of the approach.
Questions for authors
  1. Can you provide quantitative measurements of the residual task-vector energy outside the leading k-dimensional subspace, specifically the ratio between the energy of Δt(I − Vt,kVt,kᵀ) and the total energy of Δt across different layers and tasks? This is important because Equation (3) assumes that the resulting LBW subspace contains directions receiving negligible fine-tuning updates, and without validating this assumption, the interpretation of these directions as anchored task-specific knowledge remains uncertain. Please provide residual-energy statistics and investigate whether the LBW construction remains valid under different k values or adaptive rank-selection strategies.
  2. Why is the random control subspace in Table 3 restricted to the orthogonal complement of the pretrained top-K subspace? Could you additionally compare LBW ablation against randomly selected subspaces of identical dimensionality within the pretrained top-K subspace? This matters because the current comparison does not fully distinguish task-conditioned LBW importance from the general importance of dominant pretrained directions, and an additional matched control experiment within the pretrained core would help establish whether the identified LBW directions have genuinely task-specific importance.
  3. How do you explain the substantial MNLI degradation and lower average absolute performance of PACT-Iso-C in Table 6 despite its higher normalized average? Are these trade-offs consistent across different random seeds and validation configurations? Understanding whether PACT improves overall multi-task performance at the expense of individual tasks is important for evaluating its practical usefulness and the claim of consistent improvements. Please provide a task-level analysis of performance changes and clarify the evaluation criterion used to support the overall improvement claims.
3 comments

We sincerely thank Reviewer pFTQ for the constructive and insightful comments, and we are glad that Reviewer recognize our paper to be interesting and insightful, empirically strong, and easy to follow. To address Reviewer’s concerns, we provide pointwise responses below.

W1&Q1: Can you provide quantitative measurements of the residual task-vector energy outside the leading k-dimensional subspace, specifically the ratio between the energy of Δt(I−Vt,kVt,k⊤)\Delta_t(I - V_{t,k}V_{t,k}^\top) and the total energy of Δt\Delta_t across different layers and tasks? ...

A1: Thank you for this suggestion. We would like to clarify that the construction in Eq. (3) does not require negligible global residual energy. It constructs a particular subspace by projecting the pretrained core away from the leading task-update directions. Substantial residual energy outside those leading directions does not necessarily imply substantial update energy within the constructed LBW subspace.

To address your concern, we conducted three quantitative analyses: measuring the global residual energy, directly measuring update energy within the LBW subspace, and examining sensitivity to the construction ranks. All experiments use ViT-B/16. We report arithmetic means of per-matrix energy ratios, excluding zero-update matrices and retaining 50 matrices per task.

1. Measuring residual energy outside the leading task-update subspace.

To provide the requested residual-energy statistics, we measured

$ \rho_{\mathrm{tail}}(k) = \frac{|\Delta_t(I-V_{t,k}V_{t,k}^{\top})|_F^2}{|\Delta_t|_F^2} $

for k=1,...,20k=1,...,20 across eight tasks: Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SVHN, and SUN397. Each entry below averages 400 task–matrix pairs.

kρtail\rho_{\mathrm{tail}}(%)kρtail\rho_{\mathrm{tail}} (%)
189.371159.84
284.081258.26
379.811356.79
476.171455.42
573.031554.14
670.261652.93
767.781751.79
865.521850.70
963.461949.67
1061.572048.69

The global residual is indeed substantial: 65.52% at k=8 and 48.69% at k=20. We therefore directly examine the particular LBW subspace, rather than infer its update energy from global low-rank concentration.

2. Directly measuring update energy within the constructed LBW subspace.

To address whether the constructed directions receive limited fine-tuning updates, we measured

$ \rho_{\mathrm{LBW}}(K,k)=\frac{|\Delta_tV_{t,\mathrm{LBW}}|_F^2}{|\Delta_t|_F^2}. $

We evaluated all 20 tasks using (K,k)=(15,8)(K,k)=(15,8), the same configuration as the Table 3 ablation. Each task-level result averages its 50 nonzero-update matrices.

TaskρLBW\rho_{\mathrm{LBW}} (%)TaskρLBW\rho_{\mathrm{LBW}} (%)
Cars2.69Flowers1022.46
DTD2.73CIFAR1002.66
EuroSAT2.13PCAM1.94
GTSRB2.16FER20132.28
MNIST1.68CIFAR102.47
RESISC452.57Food1012.74
SVHN1.96FashionMNIST2.10
SUN3972.98RenderedSST21.75
STL102.71EMNIST1.76
OxfordIIITPet2.72KMNIST1.92
Mean across all 20 tasks2.32

Task-level means range from 1.68% to 2.98%, averaging 2.32%. Thus, despite the substantial global residual, only a small fraction of task-update energy lies within the constructed LBW subspaces.

3. Examining sensitivity to the construction ranks.

To address your concern about rank selection, we varied both the pretrained-core rank KK and the task-filtering rank kk. We also measured the update energy within the unfiltered pretrained core,

$ \rho_{\mathrm{core}}(K)=\frac{|\Delta_tV_{0,K}|_F^2}{|\Delta_t|_F^2}, $

to assess the effect of task-conditioned filtering. All entries below are percentages averaged over the same 400 task–matrix pairs in the eight-task setting.

SubspaceK=15K=23K=30K=38
Unfiltered pretrained core5.957.759.2110.74
LBW, k=1k=14.005.586.868.19
LBW, k=2k=23.514.986.187.44
LBW, k=4k=42.944.275.356.48
LBW, k=8k=82.363.474.395.35
LBW, k=12k=122.012.973.774.63
LBW, k=16k=161.762.613.324.09
LBW, k=20k=201.562.332.973.67
LBW, k=24k=241.412.102.693.33
LBW, k=32k=321.181.772.272.82

Task-conditioned filtering reduces update energy relative to the unfiltered core in every tested configuration. Increasing K admits more update energy, while increasing kk consistently suppresses it. For example, (K,k)=(15,8),(23,20),(30,32), and (38,32) yield 2.36%, 2.33%, 2.27%, and 2.82%, respectively.

Together with the performance degradation under LBW removal in Table 3, these measurements support the two properties motivating our construction: limited fine-tuning updates within the selected subspace and substantial importance for task performance.

Q2: Why is the random control subspace in Table 3 restricted to the orthogonal complement ...

A2: The random control in Table 3 is restricted to the orthogonal complement of the pretrained core to compare LBW removal with removing the same number of non-core directions. To distinguish general core importance from task-conditioned protection, we added a dimension-matched core ablation and a core-rank sweep, and examined both update energy and merging performance.

1. Dimension-matched ablation within the pretrained core.

Eq. (3) projects the pretrained core away from the task-active subspace and orthonormalizes the result. This changes its directions without necessarily reducing its dimension; the resulting LBW space is generally not contained within the original core.

With (K,k)=(15,8), the observed LBW dimension is 15 in every analyzed layer. A dimension-matched subspace within the 15-dimensional pretrained core is therefore the entire core, regardless of the randomly selected basis. We implement this control as

$ W_t^{\mathrm{abl}}=W_t(I-V_{0,K}V_{0,K}^{\top}). $

TaskOriginal accuracy (%)Core ablated (%)Accuracy drop (pp)
Cars86.490.6185.87
DTD98.402.1396.28
EuroSAT99.819.0490.78
GTSRB99.920.7199.21
MNIST99.5010.3089.20
RESISC4597.042.8094.23
SVHN96.7616.6480.12
SUN39781.320.3081.02
STL1099.609.6090.00
OxfordIIITPet94.843.2691.58
Flowers10297.061.9695.10
CIFAR10091.080.8290.26
PCAM97.8660.2837.58
FER201344.4316.0628.36
CIFAR1098.429.9688.46
Food10189.561.2288.34
FashionMNIST95.282.7292.56
RenderedSST277.3144.6532.66
EMNIST99.7810.9088.88
KMNIST99.9410.3489.60
Average92.2210.7281.50

The mean drop is 81.50 percentage points across 20 tasks. On the 18 tasks shared with Appendix Table 10, full-core removal causes about 80.56 points of degradation, compared with 80.03 points for LBW removal and 1.74 points for random non-core removal. These results establish that both the pretrained core and the constructed LBW space are functionally important.

To examine ablation-size sensitivity, we swept the number of removed core directions from r=1 to 15. For each matrix and seed, we randomly permuted the original 15 pretrained singular vectors and selected the first rr, producing nested subsets. Results were averaged over three seeds and 20 tasks.

Ablated directions rr1234571015
Accuracy (%)59.1135.3217.8412.4611.0510.9010.4810.72
Accuracy drop (pp)33.1156.9074.3879.7681.1781.3281.7481.50

Removing one direction per matrix already causes a 33.11-point drop; removing five causes an 81.17-point drop. For r=5,...,15, the mean drop fluctuates between 80.68 and 82.07 points, indicating a plateau. Only r=15 is dimension-matched to LBW; the smaller ranks characterize sensitivity to ablation size.

2. What task conditioning changes.

The pretrained core is fixed for each base model. LBW instead uses each task’s update to construct directions that retain functional importance while receiving limited updates from that task.

We measured the fraction of each task’s update energy within the two spaces. On the eight-task setting with (K,k)=(15,8), task-conditioned filtering reduces the mean ratio from 5.95% in the pretrained core to 2.36% in LBW, averaged over 400 task–matrix pairs. Together with the ablation results, this supports the intended property of LBW: important directions with reduced overlap with the task’s own updates.

3. Whether protecting the pretrained core alone is sufficient.

At the merging level, Appendix E.1 compares PACT with S-PACT, which protects the static pretrained core by filtering every task update as

Δ~tS-PACT=Δt(I−V0,KV0,KT)\tilde{\Delta}_t^{\text{S-PACT}} = \Delta_t (I - V_{0,K} V_{0,K}^T)

Full PACT instead constructs task-conditioned LBW spaces and filters each task update against the combined LBW spaces of the other tasks.

BackboneTasksS-PACT-TAPACT-TAS-PACT-Iso-CPACT-Iso-C
ViT-B/32870.875.286.587.3
ViT-B/321465.866.479.581.3
ViT-B/322061.759.974.878.7
ViT-B/16876.285.590.892.0
ViT-B/161471.276.884.286.1
ViT-B/162066.468.779.783.4
ViT-L/14885.092.894.795.1
ViT-L/141480.387.089.390.5
ViT-L/142074.783.087.790.3

PACT achieves higher absolute accuracy in 8 of 9 TA settings and all 9 Iso-C settings. These comparisons support the practical value of the full task-conditioned, cross-task protection design over static core protection. The distinction is not that LBW removal must be more destructive than core removal, but that PACT uses task updates to construct protection spaces that are more effective for merging in most tested settings.

W2&Q3: How do you explain the substantial MNLI degradation and lower average absolute performance of PACT-Iso-C in Table 6 despite its higher normalized average? Are these trade-offs consistent across different random seeds and validation configurations? Understanding whether PACT improves overall multi-task performance at the expense of individual tasks is important for evaluating its practical usefulness and the claim of consistent improvements. Please provide a task-level analysis of performance changes and clarify the evaluation criterion used to support the overall improvement claims.

A3: Thank you for highlighting this trade-off. PACT-Iso-C reduces MNLI accuracy from 78.73% to 58.56% and the average absolute score from 0.7164 to 0.7093. We retain these results alongside the improvements to provide a complete account of PACT’s benefits and limitations.

A possible explanation is the interaction between Iso-C’s spectral equalization and MNLI’s semantic requirements. Iso-C computes singular directions from the aggregated task updates and assigns them a common singular value. PACT changes this aggregate before decomposition, potentially changing which directions receive substantial influence in the reconstructed model. MNLI covers ten genres and diverse inference phenomena, including negation and quantification. If this spectral redistribution weakens directions supporting those distinctions, MNLI performance may decrease even while other tasks benefit. We consider this a plausible explanation, rather than a confirmed cause of the observed drop. Nevertheless, PACT-Iso-C improves the normalized average from 85.32% to 86.46%, reflecting gains on tasks such as CoLA and RTE relative to their individually fine-tuned references.

To address your concern about broader applicability, we additionally evaluated PACT-TIES and PACT-DARE-TIES. Their absolute averages improve from 0.5565 to 0.5983 and 0.5662 to 0.6076, respectively; their normalized averages increase from 64.54% to 70.83% and 65.67% to 70.85%. The complete task-level results, including both improvements and regressions, are presented below.

MethodCoLASST-2MRPCSTSBQQPMNLIQNLIRTEAvg.Norm Avg.
Individual0.55110.94720.87990.87700.89460.87450.92440.66430.8266-
Weight Averaging0.00.55280.75490.42010.39930.41730.59820.47290.451952.54
TA w/ DARE0.00.80730.78110.30590.71240.64010.77720.55600.572566.19
Ties-Merging0.11390.91170.7134-0.04760.72450.79310.82780.41520.556564.54
PACT-Ties0.27260.80160.82580.32880.8090.58050.7430.4260.598370.83
TIES w/ DARE0.06560.86010.78030.14640.72630.6660.790.49460.566265.67
PACT-Dare-Ties0.10080.83490.83270.69070.79120.55260.56320.49460.607670.85
Task Arithmetic0.00.83140.78410.29580.70250.65230.77850.53430.572466.05
PACT-TA0.17630.90600.77990.12550.77110.770.81550.53790.610371.75
TSV-M0.20200.93000.78950.49770.82600.74560.82880.51260.666578.22
PACT-TSV-M0.44530.92550.77020.64890.79710.75420.85410.51260.713585.62
Iso-C0.30130.90940.77550.62060.84700.78730.85100.63900.716485.32
PACT-Iso-C0.49600.88990.77280.63060.81100.58560.79540.69310.709386.46

Across the five tested methods, PACT improves the normalized average in all five and the absolute average in four.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Request for Reviewer Feedback

Dear Reviewers,

Thank you again for your constructive feedback. We have provided detailed responses and additional experiments addressing your concerns. For ease of reference, we summarize the main updates below.

  1. LBW construction and persistence (to Reviewers pFTQ and 4FGQ). We added residual-energy measurements, rank-sensitivity analyses, and experiments extending fine-tuning to 4× the original duration with learning rates up to 5× higher. The constructed LBW spaces contain an average of 2.32% of task-update energy across 20 tasks. The extended-training experiments further support persistent dependence on their pretrained components in the evaluated settings.
  2. Distinction from pretrained-core protection and CSP (to Reviewers pFTQ and aGGQ). We added dimension-matched core ablations and a core-rank sweep, and clarified PACT’s motivation and task-conditioned, cross-task protection mechanism. Comparisons with static protection support the practical value of this design.
  3. Connection to Fisher-based motivation (to Reviewer aGGQ). We added coordinate-projection measurements and controlled Fisher-mask attack experiments. These provide evidence of partial geometric correspondence and functional protection, while clarifying that the spectral construction does not recover Fisher-selected parameters coordinate by coordinate.
  4. Evaluation beyond the original settings (to Reviewers aGGQ and 4FGQ). To address Reviewer aGGQ’s scalability concern, we evaluated 50 two-class tasks partitioned from CIFAR-100, where PACT improves 100-way accuracy across TA, TIES, TSV, and Iso-C by 2.39–20.69 percentage points. To address Reviewer 4FGQ’s request for larger decoder models, we added Llama-8B experiments on MergeBench, where PACT-TA improves the reported average accuracy across the four evaluated domains from 39.10% to 39.77% compared with TA.
  5. Task-level trade-offs and evaluation criteria (to Reviewers pFTQ and 4FGQ). We added RoBERTa results for PACT-TIES and PACT-DARE-TIES and clarified absolute versus normalized averages. Across five tested baselines, normalized averages improve in all five and absolute averages in four. We explicitly acknowledge individual-task regressions, including the MNLI degradation with PACT-Iso-C, and qualify our overall improvement claims accordingly.

The full experimental details and results are included in our individual responses. We would appreciate your feedback on these additions and would be happy to address any remaining questions during the discussion period. Thank you for your time and consideration.

The Authors of Paper 2862

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.