- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Pretrained Aware Merger Lifting (PAL), which incorporates pretrained-model output sensitivity into existing model merging methods through an invertible parameter transformation. PAL improves aggregate performance across CLIP, RoBERTa, and Llama-3.2-3B merging benchmarks.
Critical strengths
-
PAL introduces a reusable, invertible, and rank-preserving transformation that incorporates pretrained-model output sensitivity into existing merging algorithms. This provides a practical way to leverage pretrained information without task-specific data or additional inference overhead.
-
PAL improves aggregate performance across multiple merging algorithms and model families, including CLIP, RoBERTa, and Llama-3.2-3B, demonstrating the practical value and broad applicability of the proposed framework.
Critical weaknesses
-
High parameter sensitivity does not necessarily imply that the corresponding task-vector updates should be prioritized during merging. Moreover, the reliability of the local sensitivity approximation for finite fine-tuning updates remains unclear.
-
Several existing methods have explored pretrained-model importance, parameter sensitivity, and feature-aware merging. The distinctive contribution of PAL relative to these approaches is not sufficiently established.
Questions for authors
Q1. PAL assigns greater reconstruction weight to high-sensitivity parameter updates, but higher sensitivity does not necessarily imply greater task importance or lower interference. For example, AIM [1] instead emphasizes preserving important pretrained weights. Could the authors provide stronger theoretical or empirical justification for prioritizing high-sensitivity updates during merging?
Q2. Eqs. (4)–(5) rely on a first-order Taylor approximation around the pretrained model, although actual task vectors may not represent sufficiently small perturbations, potentially leading to substantial approximation errors. Could the authors quantify these errors for actual task vectors, particularly at different update magnitudes, beyond the ranking correlations reported in Table 7?
Q3. Since the true output response in Eq. (3) can be directly measured using forward passes on unlabeled data, the necessity of approximating it through pretrained sensitivity is not fully justified. Could the authors compare their actual computational cost and effectiveness?
Q4. Several closely related studies deserve more detailed discussion and comparison, including AIM [1], Sens-Merging [2], Towards Minimizing Feature Drift in Model Merging [3], and MAGIC [4], which additionally investigates magnitude-sensitive layers. Could the authors clarify PAL's distinctive contributions and provide comparisons on compatible benchmarks?
[1] Activation-Informed Merging of Large Language Models.
[2] Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models.
[3] Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration.
[4] MAGIC: Achieving Superior Model Merging via Magnitude Calibration.
4 comments
Response to Reviewer Eu6u (2/2)
Q3. Since Eq. (3)'s true response can be measured by forward passes on unlabeled data, why approximate it with pretrained sensitivity? Please compare computational cost and effectiveness. (Q3)
A3. Direct response measurement and PAL address different levels of the problem. The former measures the output change caused by a given update; the latter converts pretrained output sensitivity into a reusable parameter metric, allowing pretrained response information to participate in the internal parameter operations of existing mergers.
Table 7 of the paper shows that PAL's practical transform retains most of the response ranking information provided by diagonal sensitivity. Table 6 and the main merging experiments further demonstrate improvements in output feature reconstruction and downstream merging performance. Together, these results support the effectiveness of PAL's response metric and its use in model merging.
For computational cost, PAL's metric is constructed only once per pretrained checkpoint; Section 5.4 of the paper reports a construction cost of 5.77 seconds for ViT-B/32. Following the response evaluation in Section 5.2 of the paper, we use 40 held-out images to compare the time required to measure a given update's response through forward passes with the time required to score that update using the cached PAL metric. Table 2 below shows substantially lower per-update scoring cost after the PAL metric has been constructed.
| Backbone | Direct Response Measurement (ms) | Cached PAL Scoring (ms) |
|---|---|---|
| ViT-B/32 | 56.63 | 5.50 |
| ViT-B/16 | 110.22 | 5.48 |
| ViT-L/14 | 433.73 | 12.06 |
Table 2. Evaluation time per update on an NVIDIA RTX 4090 D, reported as the median over repeated timings.
Q4. Please clarify PAL's distinctive contributions relative to AIM, Sens-Merging, Towards Minimizing Feature Drift in Model Merging, and MAGIC, and compare them on compatible benchmarks. (W2, Q4)
A4. PAL constructs a reusable metric of how parameter updates affect pretrained outputs and incorporates it into the internal parameter operations of existing mergers through an invertible transform. AIM aims to mitigate forgetting by limiting changes to important pretrained weights; Sens-Merging assigns merging coefficients based on task sensitivity and cross-task transferability; MAGIC calibrates deviations in weight and feature magnitudes caused by merging.
We compare the gains from PAL, AIM, Sens-Merging, and MAGIC with the same base mergers. LOT reduces feature drift between experts and the merged model through layer-wise optimization and is itself a standalone merging algorithm, so it is not included in this comparison of methods that enhance existing mergers. Table 3 below shows that PAL achieves the highest average accuracy in all nine settings.
| Method | TSV-M | Iso-C | ESM | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Tasks | 8 | 14 | 20 | 8 | 14 | 20 | 8 | 14 | 20 |
| Base | 85.81 | 80.04 | 76.61 | 86.31 | 80.00 | 75.21 | 88.53 | 84.21 | 81.13 |
| AIM | 86.37 | 80.92 | 78.01 | 87.49 | 81.07 | 76.56 | 89.10 | 84.85 | 82.46 |
| MAGIC | 85.14 | 80.49 | 76.92 | 86.37 | 80.88 | 76.68 | 85.88 | 81.40 | 77.43 |
| Sens-Merging | 85.84 | 80.07 | 76.67 | 86.32 | 80.00 | 75.25 | 88.54 | 84.20 | 81.15 |
| PAL | 87.13 | 82.03 | 78.96 | 88.32 | 81.84 | 77.76 | 89.88 | 85.13 | 83.17 |
Table 3. Merging 8, 14, and 20 tasks on CLIP ViT-B/32 with TSV-M, Iso-C, and ESM as base mergers. We report average task Top-1 accuracy (%). Sens-Merging is adapted to these base mergers using its task sensitivity and cross-task coefficient mechanism.
Thanks for your careful and detailed responses, and most of my questions have been addressed. But I'm still concerned that the first-order Taylor approximation error may undermine the value of your theoretical analysis. As you note in Table 1, I don't think it's a small error. Therefore, I will keep my score.
Response to Reviewer Eu6u
Thank you for taking the time to consider our responses and share your perspective. In Section 3.2 of the paper, we construct a parameter metric from the local output sensitivity of the pretrained model. As discussed in our earlier response to Q2, PAL uses this metric to guide the parameter operations of existing mergers without requiring accurate predictions of the nonlinear output changes produced by full task vectors.
Here, “local” refers to where we evaluate output sensitivity, namely at the pretrained parameters. The resulting metric can be applied directly to task vectors at their original magnitudes. Table 1 of our response examines how accurately this local information predicts the response to finite updates, and we appreciate your concern about the magnitude of the prediction errors. In Table 6 of the paper, we measured reconstruction error directly on the model’s output features through forward passes and found lower error with PAL. Our main experiments also showed gains in downstream merging performance.
Response to Reviewer Eu6u (1/2)
Thank you for your questions and suggestions. We address them individually below.
Q1. Why should high-sensitivity updates receive greater reconstruction weight when sensitivity need not imply task importance or lower interference? Please justify this theoretically or empirically and distinguish PAL from AIM. (W1, Q1)
A1. PAL does not equate pretrained output sensitivity with task importance or task interference, nor does it directly amplify task vectors with high sensitivity.
PAL assigns greater reconstruction error weights to sensitive parameter locations because the same magnitude of parameter reconstruction error at these locations may cause larger changes in pretrained outputs. PAL uses this observation to construct a parameter metric (Eqs. (3)–(6)), allowing reconstruction, projection, and other operations in existing mergers to account for pretrained output responses while preserving their merging rules.
This differs from AIM's objective. AIM focuses on protecting important pretrained weights, whereas PAL incorporates response information into the merger's parameter operations without aiming to minimize changes in pretrained outputs.
Table 6 in Section 5.1 of the paper provides direct evidence. At the same retained SVD rank, PAL can achieve lower actual output feature reconstruction error despite potentially higher parameter reconstruction error, supporting the use of response sensitivity rather than parameter distance alone.
Q2. Please quantify first-order Taylor approximation errors for actual task vectors at different update magnitudes, beyond Table 7's ranking correlations. (W1, Q2)
A2. PAL uses local pretrained output sensitivity to construct a parameter metric for existing mergers, rather than aiming for globally accurate predictions of the nonlinear responses of full task vectors.
Appendix A.1 of the paper provides a theoretical error bound for the local approximation. Table 7 of the paper validates the response sensitivity information across parameter directions and the PAL transform's ability to retain it while controlling the update norm.
We further quantify the first-order Taylor approximation error for actual task vectors at different update magnitudes. Table 1 below shows the discrepancy between measured responses and full first-order Jacobian predictions, with error generally increasing as the update magnitude grows.
Normalized RMSE ↓
Table 1. First-order prediction error for 150 actual update directions across three CLIP backbones. Normalized RMSE is normalized by the root mean square of measured responses; lower is better.
PAL's practical value lies in using local response sensitivity to improve the parameter operations of existing mergers; it does not require globally accurate response prediction. Table 6 of the paper verifies improved output feature reconstruction through actual model forward passes, and the main merging experiments further demonstrate the metric's effectiveness on downstream tasks.