- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper challenges the prevailing Task Arithmetic assumption that task-specific knowledge is fully encoded in task vectors. Through Fisher-information-based perturbation experiments and subspace geometry analysis, the authors identify "Load-Bearing Wall" (LBW) dimensions: task-critical parameters anchored in pre-trained weights that receive negligible fine-tuning updates yet are essential for task performance. To address the performance degradation caused by ignoring LBW dimensions during merging, the authors propose PACT, a data-free, plug-and-play filtering framework that constructs a global orthogonal shield for each task from the LBW subspaces of all other tasks, and projects out the components of its task vector that would invade those protected directions. The method is simple yet effective, and an efficient randomized SVD variant is further provided for scalability. Extensive experiments on vision and NLP benchmarks show consistent improvements over mainstream merging methods.
Critical strengths
-
The identification of the LBW phenomenon brings a valuable conceptual insight to the model merging community. It challenges the long-standing implicit assumption of the task vector paradigm and reveals that task-critical knowledge can remain anchored in pre-trained weights with negligible fine-tuning updates. This finding helps explain the performance degradation of existing merging methods and opens up a new direction for designing more robust merging algorithms.
-
The proposed PACT framework has good practical applicability. It is data-free, simple to implement, and can be seamlessly integrated with both scalar-based and subspace-based mainstream merging methods. The randomized SVD variant effectively reduces computational complexity and improves scalability. Consistent performance gains across multiple vision and NLP settings demonstrate its effectiveness as a plug-and-play enhancement for existing merging pipelines.
Critical weaknesses
-
The work lacks sufficient distinction from the concurrent CSP method (Sun et al., ICML 2026), which also recognizes that pre-trained weights contain task-critical structures that should be preserved during merging. A reader could reasonably view PACT as a refinement of CSP rather than an independent contribution. Although a comparison is provided in Appendix D.1, the paper does not clearly articulate the fundamental differences in core motivation, design logic and applicable scenarios between the two methods, which weakens the perceived originality of this work.
-
There is a disconnect between the motivating experiments and the proposed method. Section 3.2.1 identifies LBW dimensions at the scalar parameter level using Fisher information, but the PACT method only relies on the spectral proxy defined in Eq. 3. The paper does not show what fraction of the Fisher-identified crucial parameters lie within the span of V_{t,LBW}. Without this validation, the Fisher-based motivation experiments and the spectral-based method design read as two separate stories, and the spectral proxy cannot be firmly justified as an effective surrogate for true LBW dimensions.
Questions for authors
Could you more explicitly elaborate the core differences between PACT and the concurrent CSP work? What are the essential distinctions in terms of protection targets, design rationale, and applicable scenarios? What is the key incremental contribution of PACT compared to CSP?
What fraction of the LBW parameters identified by Fisher information fall within the span of the spectral LBW subspace V_{t,LBW} used in PACT? To what extent can the spectral proxy reliably represent the true LBW dimensions identified in the motivation experiments?
Have you evaluated how PACT performs as the number of tasks T grows significantly? How do you address the potential issue that the protection subspace becomes overly high-dimensional with more tasks? Is there an adaptive scheme to control the dimension of the protection space?
5 comments
W2&Q2: What fraction of the LBW parameters identified by Fisher information ...
A2: Task Fisher information depends on the input distribution through gradient statistics. In our data-free setting, only pretrained and fine-tuned weights are available, making high-Fisher parameters difficult to identify without assumptions about task data. PACT therefore constructs a protection subspace from pretrained directions and task-update geometry. We conducted additional experiments examining its coordinate-level correspondence with Fisher-selected parameters and its functional role in protecting them.
We analyze coverage on eight standard vision tasks using CLIP ViT-B/16. For each task, we estimate diagonal Monte Carlo model Fisher information using 1,000 training examples, one label sampled per example from the fine-tuned model’s predictive distribution, and seed 0. In each weight matrix, we intersect the top 30% of coordinates by Fisher information with the top or bottom 30% by absolute task-update magnitude. We compare the large-update/high-Fisher group with , and the small-update/high-Fisher group with , using .
Since parameter coordinates and spectral directions differ, we quantify correspondence through coordinate-projection energy, rather than a binary fraction of recovered parameters. For a binary mask and an orthonormal basis , we define
This measures selected coordinate directions’ average projection energy. The expected coverage of uniformly random coordinates in the same matrix is .
Across eight tasks, the large-update/high-Fisher group has mean coverage of 0.949% in , compared with a random expectation of 0.865%. The small-update/high-Fisher group has mean coverage of 1.740% in , compared with 1.621% for random coordinates. Both comparisons show low absolute coordinate-projection coverage, which should be interpreted relative to their low-dimensional random baselines. Both averages exceed random expectations, indicating a positive but limited coordinate-level association.
To test whether Fisher selection adds correspondence beyond low update magnitude, we compare , containing the bottom 30% of coordinates by update magnitude, with , which also requires membership in the top 30% by Fisher information. Both masks are evaluated with the same LBW basis. Relative enrichment over random is
| Scope | Random expectation | ||||
|---|---|---|---|---|---|
| Global average | 1.6211% | 1.6365% | 1.7395% | +0.95% | +7.30% |
| Attention input projection | 1.9531% | 1.9389% | 2.3222% | −0.73% | +18.90% |
Coverage and random expectation are arithmetic means; enrichment is computed from these means.
Adding Fisher selection increases global coverage from 1.6365% to 1.7395%, and enrichment over random from 0.95% to 7.30%. For attention input projections, enrichment rises from −0.73% to 18.90%. These results support partial correspondence between Fisher-selected LBW coordinates and the spectral construction.
We also test its functional significance through fixed Fisher-mask attacks. In the SUN397 experiment with Cars-derived perturbations, we decompose the full PACT correction into components inside and outside the original Fisher-selected low-update mask. The table reports the loss reduction retained with only the correction inside the mask, relative to that achieved by full PACT. Both reductions use the same unprotected attack as reference.
| Attack strength | |||
|---|---|---|---|
| 1 | 30.1% | 29.7% | 33.5% |
| 20 | 70.7% | 71.0% | 76.0% |
| 30 | 65.7% | 68.2% | 76.2% |
The task-update rank is k=8. These ratios measure retained loss reduction in the controlled attack experiment, not a causal decomposition of overall merging gains.
Under stronger attacks, corrections restricted to Fisher-selected LBW coordinates retain 65.7%–76.2% of the full PACT loss reduction. They contribute less under the weakest attack; corrections outside the mask also contribute. Thus, limited coordinate-projection coverage can accompany substantial functional benefits at Fisher-selected LBW coordinates.
This supports our motivation: parameters that change little during fine-tuning can remain task-critical and benefit from protection against interfering updates. Fisher analysis identifies this phenomenon using data; the spectral construction protects using weights alone. We will clarify this relationship as partial geometric correspondence supported by controlled functional evidence, and investigate more direct data-free protection of Fisher-identified LBW parameters in future work.
Q3: Have you evaluated how PACT performs as the number of tasks T grows significantly? How do you address the potential issue that the protection subspace becomes overly high-dimensional with more tasks? Is there an adaptive scheme to control the dimension of the protection space?
A3: Thank you for raising this issue. Our existing experiments cover 8, 14, and 20 tasks, including the simultaneous merging of 20 independently fine-tuned models.
These experiments show that PACT remains effective at this scale. Across multiple baseline–backbone combinations, its gains over the original merging methods are even larger than in the 8-task case, as shown in Table 4. These results indicate that the protection mechanism retains practical value as the number of tasks increases within the evaluated settings.
To further address the reviewer’s concern, we additionally evaluated PACT on 50 tasks, adopting the CIFAR100/50 class partition used in [1]. This setting partitions CIFAR-100 into 50 two-class tasks, with each expert independently fine-tuned from the same pre-trained model. It complements our existing experiments across different datasets. We report both 100-way accuracy, which requires prediction among all 100 classes without task identity, and task-aware 2-way accuracy, which restricts prediction to the two classes of the known task.
| Method | 100-way accuracy (%) | Task-aware 2-way accuracy (%) |
|---|---|---|
| Pre-trained | 66.95 | 97.54 |
| TA | 64.00 | 97.72 |
PACT-TA | 69.86 | 98.12 |
| TIES-Merge | 50.42 | 96.11 |
PACT-TIES | 71.11 | 98.14 |
| TSV-Merge | 71.13 | 99.12 |
PACT-TSV | 73.52 | 98.71 |
| Iso-C | 68.96 | 98.81 |
PACT-Iso-C | 74.89 | 98.87 |
PACT improves 100-way accuracy across all four baselines, with gains of 5.86, 20.69, 2.39, and 5.93 percentage points for TA, TIES-Merge, TSV-Merge, and Iso-C, respectively. Task-aware accuracy also improves for three baselines, while PACT-TSV exhibits a small decrease of 0.41 percentage points. These results provide additional empirical evidence that PACT exhibits a degree of robustness to increasing task count on a practical model-merging benchmark.
Regarding protection-space growth, Eq. (6) in the main text constructs the joint span of the other tasks’ LBW subspaces. Linearly dependent directions do not enlarge this span, but this does not guarantee that the protection dimension remains small as the number of tasks increases. A larger protection space can restrict useful task updates, with its impact depending on the removed components and their task importance. Our additional 50-task results provide empirical evidence of robustness to increased task count, while the possibility of over-constraint at larger scales remains open. The current configuration does not impose an adaptive budget on the joint protection space. We will clarify this scope of applicability in the discussion and investigate adaptive, importance-aware dimension control to identify which LBW directions require joint protection and balance their preservation against the retention of useful task updates as future work.
Reference:
[1] MagMax: Leveraging Model Merging for Seamless Continual Learning. ECCV 2024.
We sincerely thank Reviewer aGGQ for the constructive and insightful comments, and we are glad that Reviewer recognize our paper to be insightful, practical, and easy to follow. To address Reviewer’s concerns, we provide pointwise responses below.
A1: Thank you for encouraging us to clarify this distinction. PACT and CSP differ first in their motivation, which leads to different protection targets and mechanisms.
Motivation. CSP starts from evidence that pre-trained weights are near-optimal within a dominant core subspace under multi-task losses, motivating preservation of that shared core during merging. PACT starts from a different question: whether fine-tuning update magnitude adequately reflects the importance of parameters or directions to a downstream task. Our perturbation and ablation experiments show that some components remain important despite receiving little update during fine-tuning. These LBW components motivate protecting pre-trained dependencies that are barely represented in a task’s update.
Protection targets and mechanisms. CSP determines its protection space from the pre-trained weights, without incorporating each task’s adaptation directions. PACT additionally uses each task’s leading update directions to construct its LBW subspace:
\left[ \left(I-V_{t,k}V_{t,k}^{\top}\right)V_{0,K} \right].$$ Here, $V_{0,K}$ contains the leading right singular vectors of the pre-trained weights, and $V_{t,k}$ contains those of task $t$’s update. PACT then filters each task update against the combined LBW subspaces of the other tasks before merging. Its protection space is therefore conditioned on task updates rather than determined by pre-trained weights alone. This distinction between task-conditioned shields and a static global core was also recognized by Reviewer 4FGQ, who noted that the S-PACT ablation supports task-conditioned protection. **Design assumptions and applicable setting.** Both methods support merging without training data. PACT’s construction does not require the pre-trained core to be near-optimal; its theoretical bound instead uses local smoothness around the fine-tuned model and the projected gradient and update magnitudes. The distinction concerns the rationale and construction of protection within the model-merging setting. **Incremental contribution and empirical evidence.** PACT contributes the identification of important but weakly updated pre-trained components, together with a task-conditioned mechanism for protecting them during merging. As reported in Table 11, PACT-Iso-C and PACT-TSV-M outperform their CSP counterparts in average absolute accuracy in all nine backbone/task-scale settings, while PACT-TA does so in seven of nine. To further examine the contribution of task conditioning and mutual protection, we compare PACT with our static-protection ablation, S-PACT, which filters each task update using only the pre-trained right-singular core: $\tilde{\Delta}_t^{\text{S-PACT}} = \Delta_t (I - V_{0,K} V_{0,K}^T)$ As shown in Table 13, PACT achieves higher average absolute accuracy than S-PACT in all nine Iso-C settings and eight of nine TA settings. For example, on ViT-B/16 with 14 tasks, accuracy increases from 71.2% to 76.8% with TA and from 84.2% to 86.1% with Iso-C. These results support the practical value of combining task-conditioned LBW construction with mutual protection. **Table 13. Static core-protection ablation.** Entries report average absolute accuracy, with average normalized accuracy in parentheses; both are expressed in percent. Column labels specify the backbone and number of merged tasks. |Method|ViT-B/32, 8|ViT-B/32, 14|ViT-B/32, 20|ViT-B/16, 8|ViT-B/16, 14|ViT-B/16, 20|ViT-L/14, 8|ViT-L/14, 14| ViT-L/14, 20 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Task Arithmetic | 70.8 (76.5) | 65.3 (72.1) | 60.5 (66.8) | 75.4 (79.6) | 70.5 (75.9) | 65.8 (70.8) | 84.9 (88.7) | 79.4 (84.0) | 74.0 (78.1) | |S-PACT-TA | 70.8 (76.3) | 65.8 (72.7) | 61.7 (68.3) | 76.2 (80.5) | 71.2 (76.7) | 66.4 (71.6) | 85.0 (88.7) | 80.3 (85.1) | 74.7 (78.9) | |PACT-TA| 75.2 (81.0) | 66.4 (73.3) | 59.9 (66.4) | 85.5 (90.3) | 76.8 (82.7) | 68.7 (74.0) | 92.8 (96.9) | 87.0 (92.0) | 83.0 (87.7) | |Iso-C| 86.6 (93.3) | 79.5 (87.3) | 75.1 (82.2) | 90.8 (96.0) | 84.4 (90.7) | 79.7 (85.4) | 94.7 (98.9) | 89.2 (94.3) | 87.7 (92.4) | |S-PACT-Iso-C| 86.5 (93.2) | 79.5 (87.2) | 74.8 (81.7) | 90.8 (96.0) | 84.2 (90.4) | 79.7 (85.6) | 94.7 (98.9) | 89.3 (94.4) | 87.7 (92.4) | | PACT-Iso-C | 87.3 (94.1) | 81.3 (89.0) | 78.7 (86.0) | 92.0 (97.2) | 86.1 (92.3) | 83.4 (89.3) | 95.1 (98.3) | 90.5 (95.5) | 90.3 (95.0) |