acceptodds
Under review as a conference paper at ICLR 2027

Pretrained Aware Merger Lifting: Bringing Pretrained Responses into Model Merging

Abstract

Model merging combines models fine-tuned for different tasks without joint retraining. Task vector values alone do not describe how the shared pretrained model responds to parameter updates. A controlled experiment shows that identical perturbation values can produce substantially different output responses at different parameter locations, revealing information that update values alone do not capture. We introduce Pretrained Aware merger Lifting (PAL), which estimates pretrained sensitivity from a small set of generic unlabeled inputs and constructs an invertible parameter transform. PAL applies the original merging rules in transformed coordinates and maps the merged update back to the original parameter space. The sensitivity estimate and transform require no downstream task data and are computed once per checkpoint for reuse across task sets and mergers. PAL substantially improves recent state of the art mergers across CLIP vision tasks, RoBERTa on GLUE, and Llama-3.2-3B capability merging, with higher aggregate scores for every evaluated base merger.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging, task arithmetic

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Eu6uOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Pretrained Aware Merger Lifting (PAL), which incorporates pretrained-model output sensitivity into existing model merging methods through an invertible parameter transformation. PAL improves aggregate performance across CLIP, RoBERTa, and Llama-3.2-3B merging benchmarks.

Critical strengths
  1. PAL introduces a reusable, invertible, and rank-preserving transformation that incorporates pretrained-model output sensitivity into existing merging algorithms. This provides a practical way to leverage pretrained information without task-specific data or additional inference overhead.

  2. PAL improves aggregate performance across multiple merging algorithms and model families, including CLIP, RoBERTa, and Llama-3.2-3B, demonstrating the practical value and broad applicability of the proposed framework.

Critical weaknesses
  1. High parameter sensitivity does not necessarily imply that the corresponding task-vector updates should be prioritized during merging. Moreover, the reliability of the local sensitivity approximation for finite fine-tuning updates remains unclear.

  2. Several existing methods have explored pretrained-model importance, parameter sensitivity, and feature-aware merging. The distinctive contribution of PAL relative to these approaches is not sufficiently established.

Questions for authors

Q1. PAL assigns greater reconstruction weight to high-sensitivity parameter updates, but higher sensitivity does not necessarily imply greater task importance or lower interference. For example, AIM [1] instead emphasizes preserving important pretrained weights. Could the authors provide stronger theoretical or empirical justification for prioritizing high-sensitivity updates during merging?

Q2. Eqs. (4)–(5) rely on a first-order Taylor approximation around the pretrained model, although actual task vectors may not represent sufficiently small perturbations, potentially leading to substantial approximation errors. Could the authors quantify these errors for actual task vectors, particularly at different update magnitudes, beyond the ranking correlations reported in Table 7?

Q3. Since the true output response in Eq. (3) can be directly measured using forward passes on unlabeled data, the necessity of approximating it through pretrained sensitivity is not fully justified. Could the authors compare their actual computational cost and effectiveness?

Q4. Several closely related studies deserve more detailed discussion and comparison, including AIM [1], Sens-Merging [2], Towards Minimizing Feature Drift in Model Merging [3], and MAGIC [4], which additionally investigates magnitude-sensitive layers. Could the authors clarify PAL's distinctive contributions and provide comparisons on compatible benchmarks?

[1] Activation-Informed Merging of Large Language Models.

[2] Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models.

[3] Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration.

[4] MAGIC: Achieving Superior Model Merging via Magnitude Calibration.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Eu6u (1/2)

Thank you for your questions and suggestions. We address them individually below.

Q1. Why should high-sensitivity updates receive greater reconstruction weight when sensitivity need not imply task importance or lower interference? Please justify this theoretically or empirically and distinguish PAL from AIM. (W1, Q1)

A1. PAL does not equate pretrained output sensitivity with task importance or task interference, nor does it directly amplify task vectors with high sensitivity.

PAL assigns greater reconstruction error weights to sensitive parameter locations because the same magnitude of parameter reconstruction error at these locations may cause larger changes in pretrained outputs. PAL uses this observation to construct a parameter metric (Eqs. (3)–(6)), allowing reconstruction, projection, and other operations in existing mergers to account for pretrained output responses while preserving their merging rules.

This differs from AIM's objective. AIM focuses on protecting important pretrained weights, whereas PAL incorporates response information into the merger's parameter operations without aiming to minimize changes in pretrained outputs.

Table 6 in Section 5.1 of the paper provides direct evidence. At the same retained SVD rank, PAL can achieve lower actual output feature reconstruction error despite potentially higher parameter reconstruction error, supporting the use of response sensitivity rather than parameter distance alone.

Q2. Please quantify first-order Taylor approximation errors for actual task vectors at different update magnitudes, beyond Table 7's ranking correlations. (W1, Q2)

A2. PAL uses local pretrained output sensitivity to construct a parameter metric for existing mergers, rather than aiming for globally accurate predictions of the nonlinear responses of full task vectors.

Appendix A.1 of the paper provides a theoretical error bound for the local approximation. Table 7 of the paper validates the response sensitivity information across parameter directions and the PAL transform's ability to retain it while controlling the update norm.

We further quantify the first-order Taylor approximation error for actual task vectors at different update magnitudes. Table 1 below shows the discrepancy between measured responses and full first-order Jacobian predictions, with error generally increasing as the update magnitude grows.

Normalized RMSE ↓\downarrow

Update ScaleViT-B/32ViT-B/16ViT-L/14
0.25×0.250.120.12
1.00× (Original)0.390.420.27
2.00×0.450.420.54

Table 1. First-order prediction error for 150 actual update directions across three CLIP backbones. Normalized RMSE is normalized by the root mean square of measured responses; lower is better.

PAL's practical value lies in using local response sensitivity to improve the parameter operations of existing mergers; it does not require globally accurate response prediction. Table 6 of the paper verifies improved output feature reconstruction through actual model forward passes, and the main merging experiments further demonstrate the metric's effectiveness on downstream tasks.

AuthorsOpenReview ↗

Response to Reviewer Eu6u (2/2)

Q3. Since Eq. (3)'s true response can be measured by forward passes on unlabeled data, why approximate it with pretrained sensitivity? Please compare computational cost and effectiveness. (Q3)

A3. Direct response measurement and PAL address different levels of the problem. The former measures the output change caused by a given update; the latter converts pretrained output sensitivity into a reusable parameter metric, allowing pretrained response information to participate in the internal parameter operations of existing mergers.

Table 7 of the paper shows that PAL's practical transform retains most of the response ranking information provided by diagonal sensitivity. Table 6 and the main merging experiments further demonstrate improvements in output feature reconstruction and downstream merging performance. Together, these results support the effectiveness of PAL's response metric and its use in model merging.

For computational cost, PAL's metric is constructed only once per pretrained checkpoint; Section 5.4 of the paper reports a construction cost of 5.77 seconds for ViT-B/32. Following the response evaluation in Section 5.2 of the paper, we use 40 held-out images to compare the time required to measure a given update's response through forward passes with the time required to score that update using the cached PAL metric. Table 2 below shows substantially lower per-update scoring cost after the PAL metric has been constructed.

BackboneDirect Response Measurement (ms)Cached PAL Scoring (ms)
ViT-B/3256.635.50
ViT-B/16110.225.48
ViT-L/14433.7312.06

Table 2. Evaluation time per update on an NVIDIA RTX 4090 D, reported as the median over repeated timings.

Q4. Please clarify PAL's distinctive contributions relative to AIM, Sens-Merging, Towards Minimizing Feature Drift in Model Merging, and MAGIC, and compare them on compatible benchmarks. (W2, Q4)

A4. PAL constructs a reusable metric of how parameter updates affect pretrained outputs and incorporates it into the internal parameter operations of existing mergers through an invertible transform. AIM aims to mitigate forgetting by limiting changes to important pretrained weights; Sens-Merging assigns merging coefficients based on task sensitivity and cross-task transferability; MAGIC calibrates deviations in weight and feature magnitudes caused by merging.

We compare the gains from PAL, AIM, Sens-Merging, and MAGIC with the same base mergers. LOT reduces feature drift between experts and the merged model through layer-wise optimization and is itself a standalone merging algorithm, so it is not included in this comparison of methods that enhance existing mergers. Table 3 below shows that PAL achieves the highest average accuracy in all nine settings.

MethodTSV-MIso-CESM
Tasks814208142081420
Base85.8180.0476.6186.3180.0075.2188.5384.2181.13
AIM86.3780.9278.0187.4981.0776.5689.1084.8582.46
MAGIC85.1480.4976.9286.3780.8876.6885.8881.4077.43
Sens-Merging85.8480.0776.6786.3280.0075.2588.5484.2081.15
PAL87.1382.0378.9688.3281.8477.7689.8885.1383.17

Table 3. Merging 8, 14, and 20 tasks on CLIP ViT-B/32 with TSV-M, Iso-C, and ESM as base mergers. We report average task Top-1 accuracy (%). Sens-Merging is adapted to these base mergers using its task sensitivity and cross-task coefficient mechanism.

Thanks for your careful and detailed responses, and most of my questions have been addressed. But I'm still concerned that the first-order Taylor approximation error may undermine the value of your theoretical analysis. As you note in Table 1, I don't think it's a small error. Therefore, I will keep my score.

AuthorsOpenReview ↗

Response to Reviewer Eu6u

Thank you for taking the time to consider our responses and share your perspective. In Section 3.2 of the paper, we construct a parameter metric from the local output sensitivity of the pretrained model. As discussed in our earlier response to Q2, PAL uses this metric to guide the parameter operations of existing mergers without requiring accurate predictions of the nonlinear output changes produced by full task vectors.

Here, “local” refers to where we evaluate output sensitivity, namely at the pretrained parameters. The resulting metric can be applied directly to task vectors at their original magnitudes. Table 1 of our response examines how accurately this local information predicts the response to finite updates, and we appreciate your concern about the magnitude of the prediction errors. In Table 6 of the paper, we measured reconstruction error directly on the model’s output features through forward passes and found lower error with PAL. Our main experiments also showed gains in downstream merging performance.

Reviewer_Qh3HOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on bringing the information of pretrained model into model merging. To this end, the authors propose PAL, which estimates the diagonal sensitivity of pretrained weights. Then, PAL reweighs merging of different parameters on top of existing methods. Several experiments on VIT and Llama show the effectiveness of the proposed method.

Critical strengths
  • This paper is well-written and easy to follow.
  • Using the pretrained information to navigate the merging process is well-motivated.
  • The experiments are extensive.
Critical weaknesses
  • PAL only works for model merging methods with SVD truncation. And this paper does not provide experiments when PAL is applied to Task arithmetic, Ties Merging, or PCB Merging.
  • This paper does not mention a closely related baseline method CSP [1]. CSP also protects pretrained information by projecting the merged task vector into the non-essential subspaces. This paper should compare CSP in its techniques, motivations, and performance.
  • PAL generates sensitivity from 80 ImageNet or WikiText inputs. However, this paper does not analyze how this relates to downstream tasks.
  • PAL needs to calculate the gradient of pretrained weights, which requires high computational cost.

[1] Revisiting the Role of Pretrained Weights in Model Merging: On Near-Optimality within the Core Subspace (ICML 2026)

Questions for authors
  • What about using OOD input samples in PAL?
  • How should we select input sampels for PAL?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Qh3H

Thank you for your questions and suggestions. We address them individually below.

Q1. Is PAL limited to SVD truncation? Please provide results for Task Arithmetic, TIES, and PCB. (W1)

A1. PAL is not limited to SVD truncation, as described in Section 3.4 and Appendices A.3–A.4 of the paper. WUDI optimizes an interference objective based on task vector inner products and norms, while ESM projects task updates onto a subspace constructed from activations and applies semantic weights to the projected components. Tables 3–4 of the paper report performance improvements from applying PAL to both methods.

For Task Arithmetic, fixed-coefficient linear merging causes the transform and its inverse to cancel exactly, as proved in Appendix A.3, Eq. (25). For TIES, Table 1 below shows that PAL(TIES) achieves higher average accuracy across the nine settings than the TIES results reported in the literature.

MethodViT-B/32ViT-B/16ViT-L/14Avg
Tasks814208142081420
TIES75.1068.0063.4079.7073.2068.2086.9079.5075.7074.41
+ PAL75.0968.4164.0679.4773.1769.1687.0480.5076.9374.87

Table 1. Merging 8, 14, and 20 tasks on CLIP ViT-B/32, ViT-B/16, and ViT-L/14. We report average task Top-1 accuracy (%).

For PCB, PAL changes its parameter-dependent scoring and selection, which does not guarantee better merging performance; our additional experiments did not show gains under the same settings.

Q2. How does PAL compare with CSP's projection into non-essential subspaces in motivation, technique, and performance? (W2)

A2. PAL constructs a reusable response sensitivity metric based on how parameter updates affect pretrained model outputs and incorporates it into the parameter operations of existing mergers. CSP constrains merged updates using the core subspace of pretrained weights to protect pretrained knowledge, focusing on the selection of update directions.

Table 2 below shows that PAL outperforms CSP in all nine settings across the three base mergers.

MethodTSV-MIso-CESM
Tasks814208142081420
CSP86.7181.2678.4886.6980.7876.1788.4884.8082.19
PAL87.1382.0378.9688.3281.8477.7689.8885.1383.17

Table 2. Merging 8, 14, and 20 tasks on CLIP ViT-B/32 with TSV-M, Iso-C, and ESM as base mergers. We report average task Top-1 accuracy (%).

Q3. How do the 80 ImageNet/WikiText calibration inputs relate to downstream tasks? What happens with OOD inputs, and how should inputs be selected? (W3, Q1–Q2)

A3. Tables 8–9 in Section 5.3 of the paper analyze how the source and number of estimation inputs affect response prediction and merging performance. We further compare the effects of different input sources on final merging accuracy.

Table 3 below shows similar merging performance across natural image sources, with weaker performance from pure random noise. This supports using generic unlabeled natural inputs that match the model's modality, without relying on downstream task data.

Calibration InputViT-B/32
Tasks81420
ImageNet-1k (Original)88.3281.8477.76
Downstream Images88.2081.9677.92
ImageNet-R88.2381.8477.79
Class-matched Natural Images88.2881.7277.86
Random Noise86.6380.3276.43

Table 3. Merging 8, 14, and 20 tasks on CLIP ViT-B/32 with Iso-C as the base merger. We report average task Top-1 accuracy (%) for different calibration inputs.

Q4. PAL needs to calculate the gradient of pretrained weights, which requires high computational cost. (W4)

A4. PAL requires gradient computation only for a one-time offline construction for each pretrained checkpoint. Section 5.4 of the paper reports the actual construction costs, summarized for three models in Table 4 below. The resulting transform can be reused across task combinations and base mergers, so construction need not be repeated for every merge, and the merged model incurs no additional inference overhead.

ModelConstruction Time
CLIP ViT-B/325.77 s
RoBERTa-base3.70 s
Llama-3.2-3B3.66 min

Table 4. One-time PAL transform construction cost using 80 generic unlabeled inputs on an NVIDIA RTX 4090 D.

Thanks for the feedback. My concerns have been resolved. And I would like to keep my positive score.

Reviewer_CMdrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAL estimates diagonal sensitivity of normalized pretrained features from generic unlabeled inputs, constructs invertible row/column scaling, and runs existing mergers in transformed coordinates before mapping back. One transform is reused across task sets and mergers sharing a checkpoint. Evaluations cover three CLIP backbones, RoBERTa-base on GLUE, and Llama-3.2-3B on MergeBench.

Critical strengths

Reusable design: Sensitivity estimation needs no downstream task data, and the merged model incurs no additional inference operations.

Consistent aggregate gains: Tables 1–5 report improvements for all 45 vision comparisons, six GLUE mergers, and four Llama mergers.

Explicit mathematical boundaries: This work distinguishes local response approximation, optimality under the fitted metric, and cancellation for fixed-coefficient linear merging.

Critical weaknesses
  1. Positioning misses close precedents. AIM [1] already uses generic calibration and pretrained activations to enhance existing mergers; GFWSVD [2] performs sensitivity-weighted SVD through two-sided transformation and inversion for compression, closely related to Eqs. (12) and (26). Section 2 discusses neither. PAL's shared pretrained Jacobian metric differs, but its incremental value needs a controlled comparison.
  2. Mechanism attribution remains incomplete. No activation-scale or shuffled-scale controls isolate the proposed sensitivity signal. The existing experiments do not separate sensitivity-specific improvements in downstream merging from benefits of generic coordinate rescaling.
  3. Key implementation and evaluation details are missing. Sections 3.3 and A.2–A.4 omit the actual probe count/distribution, damping coefficient, and row/column fitting objective and algorithm. Validation-only selection is stated, but hyperparameter grids, selected values, and separate GLUE selection/evaluation splits are unidentified.
  4. Estimator stability is unquantified. Main results use one 80-input estimate per checkpoint; Table 9 averages other sample counts over five sets without reporting dispersion. Coding and multilingual scores decrease for every Llama merger (Table 5).

References:

[1] Activation-Informed Merging of Large Language Models. In NeurIPS 2026.

[2] Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity. In ICML 2026.

Questions for authors
  1. With identical experts, calibration inputs, and tuning budgets, how does PAL compare with AIM and an activation-derived transform around the same merger?
  2. For actual full-model merged updates at native magnitudes, does the practical metric predict measured pretrained responses better than parameter norms alone?
  3. On representative configurations, across independent input/probe seeds, what are the paired-gain uncertainties and per-domain changes, particularly for Llama coding and multilingual performance?
4 comments
AuthorsOpenReview ↗

Response to Reviewer CMdr (1/2)

Thank you for your questions and suggestions. We address them individually below.

Q1. What distinguishes PAL from AIM and GFWSVD's two-sided, sensitivity-weighted SVD, and how does PAL compare with AIM using identical experts, calibration inputs, tuning budgets, and the same merger? (W1, Q1)

A1. PAL constructs a reusable pretrained response metric and uses an invertible transform to guide existing merging operations, addressing the fact that task vector values alone do not fully capture their output effects. AIM aims to mitigate forgetting by limiting changes to important pretrained weights, whereas PAL uses the response metric to guide the combination and approximation of task updates without aiming to constrain weight changes. GFWSVD uses Fisher-weighted low-rank approximation for model compression, a different objective from PAL's multitask merging.

AIM and PAL both enhance existing merging methods, allowing their gains to be compared with the same base merger; GFWSVD targets model compression and falls outside this comparison setting. Table 1 below shows that PAL outperforms AIM in all nine settings.

MethodTSV-MIso-CESM
Tasks814208142081420
AIM86.3780.9278.0187.4981.0776.5689.1084.8582.46
PAL87.1382.0378.9688.3281.8477.7689.8885.1383.17

Table 1. AIM and PAL with TSV-M, Iso-C, and ESM as base mergers on CLIP ViT-B/32 for 8, 14, and 20 tasks. We report average task accuracy (%).

Q2. Does sensitivity improve downstream merging beyond generic rescaling? Please provide activation-derived and shuffled-scale controls with matched experts, calibration inputs, tuning budgets, and the same merger. (W2, Q1)

A2. The invertible transform TT is a core component of PAL that implements the pretrained output response metric, rather than an additional heuristic parameter rescaling. Eq. (26) shows that PAL minimizes low-rank reconstruction error under the constructed response metric while preserving the base merger's operational rules.

We add an Activation-derived Transform based on the second moments of pretrained activations, and a Permutation control that preserves the scale distribution while randomly shuffling the correspondence between sensitivity and parameter locations. Table 2 below shows that PAL outperforms the Activation-derived Transform in all nine settings and exceeds all five random permutations in each setting, supporting the value of response sensitivity and its correspondence to parameter locations for merging performance.

MethodB/32B/16L/14
Tasks814208142081420
Activation Tran.88.0281.6177.4291.5785.7881.2594.6790.0888.69
Permutation85.67±0.12\small\begin{array}{c}\mathsf{85.67}\\\\\mathsf{\pm 0.12}\end{array}79.39±0.20\small\begin{array}{c}\mathsf{79.39}\\\\\mathsf{\pm 0.20}\end{array}75.31±0.19\small\begin{array}{c}\mathsf{75.31}\\\\\mathsf{\pm 0.19}\end{array}90.26±0.07\small\begin{array}{c}\mathsf{90.26}\\\\\mathsf{\pm 0.07}\end{array}84.21±0.14\small\begin{array}{c}\mathsf{84.21}\\\\\mathsf{\pm 0.14}\end{array}79.75±0.12\small\begin{array}{c}\mathsf{79.75}\\\\\mathsf{\pm 0.12}\end{array}94.24±0.05\small\begin{array}{c}\mathsf{94.24}\\\\\mathsf{\pm 0.05}\end{array}89.33±0.12\small\begin{array}{c}\mathsf{89.33}\\\\\mathsf{\pm 0.12}\end{array}87.56±0.11\small\begin{array}{c}\mathsf{87.56}\\\\\mathsf{\pm 0.11}\end{array}
PAL88.3281.8477.7691.9186.2781.8194.7990.1788.86

Table 2. PAL and two controls with Iso-C as the base merger on CLIP ViT-B/32, ViT-B/16, and ViT-L/14 for 8, 14, and 20 tasks. We report average task Top-1 accuracy (%); Permutation reports the mean ±\pm standard deviation over five independent permutations.

AuthorsOpenReview ↗

Response to Reviewer CMdr (2/2)

Q3. At native magnitudes, does PAL's practical metric predict measured pretrained responses to full-model merged updates better than parameter norms? (Q2)

A3. We compare PAL's practical metric with Parameter Norm (computed as ∥Δ∥22\|\Delta\|_2^2) for predicting the responses of actual full-model merged updates. Table 3 below shows that, under the same calibration procedure on expert updates, PAL achieves lower response prediction error in all nine settings.

Metric Parameter Norm   PAL  
Relative RMSE ↓\downarrow113.3619.07

Table 3. Output responses of actual full-model merged updates at native magnitudes on CLIP ViT-B/32, ViT-B/16, and ViT-L/14 for 8, 14, and 20 tasks. We report the mean Relative RMSE across the nine settings (dimensionless; lower is better).

Q4. Please specify probe count/distribution, damping, the row/column fitting objective and algorithm, hyperparameter grids and selected values, and the GLUE selection/evaluation splits. (W3)

A4. Eqs. (8)–(12) and Appendices A.1–A.4 of the paper define the sensitivity estimator and invertible transform and establish their mathematical properties. The implementation uses 80 generic unlabeled inputs, eight sets of normalized Rademacher probes, a damping coefficient of 10−310^{-3}, and root mean square normalization. The row and column factors are fitted by least squares in the log domain and obtained in closed form from the row, column, and overall means. Merging hyperparameters are selected on validation data and frozen before final evaluation. Vision experiments use separate validation and test sets; GLUE follows the reporting data and evaluation procedures of existing public implementations, with hyperparameters selected on a predefined validation subset. We will release the full code and experiment configurations to provide the implementation and reproduction details.

Q5. How stable are gains across independent input/probe seeds and estimation sample counts? Report paired-gain uncertainty and per-domain changes, especially the consistent Llama coding and multilingual declines. (W4, Q3)

A5. We add repeated experiments that vary the input samples, random probes, or both. Table 4 below shows positive gains in every repeat with low variation, supporting PAL's stability to randomness in sensitivity estimation under these representative settings.

BackboneInput Seed (Mean ± SD)Probe Seed (Mean ± SD)Joint Seed (Mean ± SD)
ViT-B/32+1.97±0.03+1.97 \pm 0.03+1.97±0.02+1.97 \pm 0.02+1.97±0.03+1.97 \pm 0.03
ViT-B/16+1.33±0.03+1.33 \pm 0.03+1.32±0.01+1.32 \pm 0.01+1.33±0.02+1.33 \pm 0.02

Table 4. Five repeats per setting on CLIP ViT-B/32 and ViT-B/16 with eight tasks and Iso-C as the base merger. Merging hyperparameters are fixed while input samples, random probes, or both are varied. We report paired accuracy gains over Iso-C as the mean ±\pm standard deviation in percentage points.

We also perform five repeats with different numbers of estimation inputs. In representative B/32 and B/16 settings, the accuracy standard deviations for 5, 20, and 160 inputs are all at most 0.09 percentage points, further supplementing the stability analysis in Table 9 of the paper.

For Llama, Table 5 of the paper reports the complete results by domain. Although the domains do not all improve together, PAL improves the average performance across the five domains for all four base mergers and consistently improves Math and Safety.

Thank you for the additional experiments and clarifications. The AIM, activation-transform, and permutation comparisons support PAL’s added value. The repeated vision experiments and lower mean response-prediction error also improve my assessment. Please clarify the GLUE selection/evaluation split, hyperparameter search settings, and Relative RMSE definition and calibration. The Llama coding and multilingual declines should remain explicit limitations. Overall, the rebuttal addresses most of my main concerns. I raise my score from 2 to 3.

AuthorsOpenReview ↗

Response to Reviewer CMdr

Thank you for carefully reviewing our additional experiments and clarifications. As noted in our earlier response, we followed the evaluation protocols in the corresponding public implementations. We will release the full experimental configurations with our code, including the hyperparameter search ranges and selected values. We appreciate your constructive feedback throughout the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.