acceptodds
Under review as a conference paper at ICLR 2027

Predicting and Repairing Merge Collapse in Large Language Models

Abstract

Large language models fine-tuned from a shared base can be merged by averaging their task vectors, but some merges collapse far below the base model, and common merge operators give no warning before evaluation. We show that one statistic of the specialists' task vectors both predicts this collapse and calibrates its repair. The power that averaging removes equals the variance of the task vectors across specialists, our measure of interference. Under a working noise model, the disturbance that a merge injects grows with the merge coefficient and with interference, yielding a pre-merge score. In our experiments on twenty-two merge configurations from four model families, only destructive merges exceed a threshold on this score. We find that statistics of sign conflict between specialists, a common target of existing merge operators, are anti-predictive. We then predicted the outcomes of fourteen merges before evaluating them, and twelve predictions were correct, including the destructive outcome of a specialist pair pushed past the threshold by continued pretraining. To address this collapse, we introduce PRISM, an operator that averages the task vectors first and then soft-thresholds each layer at a level set by the layer's interference. Without data or tuning, PRISM keeps all five destructive merges above the threshold within evaluation noise of the base model, where plain averaging falls at least 14.4 points below it or collapses entirely. We apply PRISM only above the threshold and keep the plain average for merges below it, which include all fifteen harmless ones. Code will be released upon publication.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merging, Large Language Models, Task Arithmetic, Task Vectors, Parameter Interference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hMV3OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies destructive merging of language-model specialists derived from a shared base. It defines interference as the coordinate-wise variance of scaled task vectors and motivates a pre-merge risk score​ , through a shared-signal, independent-disturbance model. It also proposes PRISM, which soft-thresholds the averaged task vector using layer-wise interference, with a minimum-retention floor and a cancellation-based bypass gate. The evaluation covers 22 configurations across four model families. A fixed score threshold identifies five of seven destructive configurations without flagging any of the fifteen harmless ones; the authors additionally report twelve correct predictions out of fourteen prospective evaluations. PRISM keeps the five above-threshold destructive merges near base-model performance. Further experiments examine coefficient sensitivity, norm-matched shrinkage, specialist count, and a controlled continued-pretraining trajectory.

Critical strengths
  1. The paper addresses a practically useful problem and includes prospective and controlled evidence.

Checking collapse risk from weights before constructing and evaluating a merged model is a useful objective. The reported prospective predictions strengthen the evidence beyond retrospective correlation.

  1. The paper evaluates several merge operators, coefficient sweeps, multiple model families, and both general capability and generation benchmarks. Importantly, it includes the unmerged base, a norm-matched uniform-shrinkage control, and floor/threshold ablations.
Critical weaknesses
  1. The main repair results do not sufficiently isolate the contribution of interference-calibrated thresholding from fixed-quantile sparsification and conservative shrinkage.

Appendix D.2 states that, on the main Qwen2.5-7B Math+Coder setting, the 0.1% retention floor binds in every layer at the default κ=1. According to Algorithm 1, when this happens, the final threshold is replaced by the magnitude of the relevant order statistic of the averaged task vector. The available controls point the same way: the uniform norm-matched control (68.1) and the unmerged base (68.0) match PRISM on the six-task mean. Interference thus enters only through the gate, not through the repair itself.

  1. The predictive evidence does not establish an advantage of interference over simpler update-magnitude statistics, and its generalization remains limited.

Table 18 reports that total power and signal power both achieve AUC 1.00 and completely separate the 22 configurations, whereas the proposed interference score has AUC 0.98 and misses two destructive merges. The signal-power score is directly related to the RMS magnitude of the averaged update. Thus, the current evidence is also consistent with a simpler interpretation: large applied updates are dangerous, without task-vector disagreement being the essential predictive quantity.

Questions for authors
  1. Interference calibration versus the retention floor. For each of the five above-threshold destructive configurations, what fraction of matrices has an active floor? Can the authors compare PRISM against fixed-0.1%-quantile soft thresholding under identical normalization, gating, parameter scope, and coefficient settings? On the main setting, should these methods be exactly equivalent at the default?

  2. Choice of screening statistic. Given the stronger separation obtained by total and signal power in Table 18, what evidence favors interference as the screening statistic? Can the authors compare these alternatives using thresholds fixed on the same initial configurations and evaluated on the same prospective cases?

  3. Missing base row in Table 1. Please add the unmerged base model to Table 1. By the numbers in App. D.11, PRISM is below the base on all four generation metrics, and this should be stated and discussed in the main text rather than left in the appendix.

15 comments
AuthorsOpenReview ↗

Response to Reviewer hMV3 (1/9)

We thank the reviewer for the close read and the precise questions.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

In short, three points.

1, W1, Q1. Floor versus interference and shrinkage. PRISM's interference thresholds repair the five destructive merges without the floor, so interference is used in the repair itself, not only through the gate. With the floor disabled, PRISM's thresholds keep all five above-threshold destructive merges no more than 0.4pp below the base. The floor is therefore not required for repair. At κ≤0.5\kappa\le0.5, no floor binds in any of the five. At PRISM's norm, concentrating the update on the kept 0.1% outscores uniform shrinkage on all four EvalPlus metrics, so coordinate placement adds a coding benefit beyond update size (Table R6, Table R2 and Table 3).
2, W2, Q2. Choice of screening statistic and shared drift. The recipe flags all fifteen destructive merges and none of the twenty-one harmless ones. Cancellation-selected screening therefore covers the shared-drift misses and selects a repair instead of the destructive average. The screening comparison covers Tables 4 and R7 and revised Tables 22 and 23. Its predictions were correct on all fourteen public pairs scored before evaluation, five in Table R7 and nine in revised Tables 22 and 23. Retaining the largest 0.1% of each matrix's averaged update keeps all nine flagged merges in which little cancels within 1.7pp of the base (Tables R1 and R7, revised Table 22).
3, Q3. Missing base comparison. PRISM matches the base's six-task mean within 0.2pp and scores above the base on all four EvalPlus coding metrics, as the comparison with the new base row in Section 5.2 states. Table 1 now includes the base and makes PRISM's deficits on all four harness generation metrics explicit. The main-text comparison therefore distinguishes general-capability recovery from remaining generation deficits.

AuthorsOpenReview ↗

Response to Reviewer hMV3 (2/9)

W1, Q1. Repair without the floor.

"Appendix D.2 states that, on the main Qwen2.5-7B Math+Coder setting, the 0.1% retention floor binds in every layer at the default κ=1."

Answer. PRISM calibrates each layer's repair threshold with interference and repairs all five above-threshold destructive merges with the floor disabled. It does not use interference only through the gate. On all five above-threshold destructive merges, PRISM without the floor stays no more than 0.4pp below the base. To check whether the retention floor binds in every layer, we added measurements of floor activation across projection matrices, destructive merges and threshold scales. Where the floor does not bind, the interference threshold alone sets every cut. Appendix D.2, in the paragraph before Table 6, gives the main-merge measurements.

To compare repair where no floor binds with naive averaging, the table below collects the main-merge scores from Tables 4 and 6.

Main-merge six-task meanScore
Interference thresholds with no active floor67.4 and 67.3
Naive averaging49.5

The κ≤0.5\kappa\le0.5 runs therefore isolate repair by the interference threshold in all five merges.

Main merge without floorValue
Coordinates kept by the interference thresholds0.0081%
Update norm relative to the average1.4%
Six-task mean68.6
Base six-task mean68.0

The interference thresholds alone recover the base's mean on the main merge.

Where the floor binds. We count each projection matrix separately with the floor on, as Algorithm 1 applies it.

Table R2. Matrices in which the 0.1% floor sets the cut, out of the 84 MLP projection matrices. Rows are the five above-threshold destructive merges of Table 4. All use K=2 and c=0.5. Keep without floor is the share of coordinates the interference threshold alone keeps at κ=1\kappa=1.

Settingρ\rhoKeep without floorκ=0.25\kappa=0.25κ=0.5\kappa=0.5κ=1\kappa=1κ=2\kappa=2 and 3
Qwen2.5-1.5B ja + med, constructed0.3420.0021%008484
Qwen2.5-1.5B Math + Coder0.2460.081%006684
Qwen2.5-7B Math + Coder0.2930.0081%008384
Qwen2.5-7B R1-Distill + Coder0.2910.0085%008384
Qwen2.5-7B Math-PRM + Coder0.2850.011%008384

At κ=1\kappa=1, λl\lambda_l exceeds the 99.9th percentile of ∣δˉ∣|\bar\delta| in 83 of the 84 MLP projection matrices (Table R2). At κ≤0.5\kappa\le0.5, where the interference threshold keeps more than 0.1% of a matrix, no floor binds in any matrix of the five merges.

The matrix-level counts identify where the floor binds and demonstrate repair where interference thresholds alone set every cut.

The matrix counts are added to the revised PDF as the bottom part of Table 6.

AuthorsOpenReview ↗

Response to Reviewer hMV3 (3/9)

W1, Q1. Continued. Floor-free repair across all five merges.

"The main repair results do not sufficiently isolate the contribution of interference-calibrated thresholding from fixed-quantile sparsification and conservative shrinkage."

With the floor disabled, every above-threshold destructive merge stays no more than 0.4pp below the base. To isolate the contribution of interference-calibrated thresholding from fixed-quantile sparsification, we ran a new experiment on destructive merges with the retention floor disabled.

Table R6. PRISM with the 0.1% floor disabled, on the five above-threshold destructive merges of Table 4. It runs at κ=1\kappa=1 and c=0.5. Six-task means (%). Base, Task Arithmetic and PRISM are from Table 4. PPL is the floor-free merge's perplexity, with the base's in parentheses.

SettingBaseTask ArithmeticPRISMPRISM without floorPPL without floor
Qwen2.5-1.5B ja + med, constructed56.135.855.956.013.5 (13.0)
Qwen2.5-1.5B Math + Coder56.337.155.855.913.1 (13.0)
Qwen2.5-7B Math + Coder68.049.567.868.68.9 (8.6)
Qwen2.5-7B R1-Distill + Coder68.149.767.768.18.9 (8.6)
Qwen2.5-7B Math-PRM + Coder68.153.767.667.98.9 (8.6)

Result without the floor. On all five above-threshold destructive merges, PRISM without the floor stays no more than 0.4pp below the base of Table 4. It also scores 14.2 to 20.2pp above Task Arithmetic. On each merge, its six-task mean is within evaluation uncertainty of the floored operator's, 0.1 to 0.8pp above it. Table R6 and Table 6 of the revised PDF report these means.

Disabling the floor lowers the six-task mean on none of the five merges.

The floor only lowers the cut of a matrix in which the interference threshold would keep fewer than 0.1% of the coordinates, so it adds to the update and removes none of it. At κ≤0.5\kappa\le0.5, the interference thresholds set every cut.

Without the floor, the update of the interference thresholds has 1.4% of the average's norm, less than the uniform control's 2.2%. We compare this update with that control on all four EvalPlus metrics (Appendix D.2, uniform control from Table 3).

EvalPlus pass@1 (%)HumanEvalHumanEval+MBPPMBPP+
PRISM without floor, norm 1.4%63.457.370.660.8
Uniform control, norm 2.2%62.257.369.659.0

The floor-free update matches or beats the uniform control on every metric with a smaller update.

The repair itself comes from the interference thresholds, which recover all five merges on their own.

The revised PDF adds these means as the Mean without floor column of Table 6 and states them in Appendix D.2.

Reviewer_obyqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the theoretical causes of performance collapse in model merging and possible remedies. Under a common-signal model with independent, zero-mean disturbances, the theoretical analysis shows that averaging reduces disturbance variance and motivates using cross-expert task-vector variance as a proxy for the disturbance scale introduced by merging. Based on this analysis, the authors propose PRISM and conduct extensive experiments. The results demonstrate its effectiveness in many of the evaluated settings and show improvements over several relevant baselines.

Critical strengths
  1. Predicting and repairing model-merging collapse without requiring calibration datasets or model evaluation addresses an important and theoretically meaningful research problem. Preserving model capabilities during merging is a central and challenging issue in this field.
  2. The experimental evaluation is extensive, and the paper reports and analyzes failure cases.
Critical weaknesses
  1. Although the overall reasoning is relatively clear, the balance of emphasis across the paper is poorly judged: the method and theoretical analysis sections are too brief and underdeveloped, while the introduction is inadequately organized and its statement of the main contributions lacks clarity.
  2. The proposed variance-based criterion has important limitations. It can miss destructive merges with low cross-expert variance or low cancellation ratios, including the reported Math+Finance and Coder+Coder-Instruct configurations. More generally, task vectors may share a harmful drift that is not captured by their cross-expert variance. In low-cancellation cases, the gate also bypasses shrinkage, leaving potentially destructive updates unrepaired. These limitations restrict the method’s applicability and show that it does not yet provide a comprehensive solution to model-merging collapse.
  3. The experiments do not yet establish variance-based calibration as the principal mechanism behind the repair gains. Recovery of base-model performance is more directly supported by the reduction in update magnitude: the norm-matched uniform-shrinkage control also avoids collapse, and the default operator is dominated by the retention floor in the central experimental setting. Variance appears useful as a prediction and automatic-calibration signal, and some comparisons suggest additional repair benefits. However, its independent contribution has not been sufficiently isolated experimentally from update shrinkage and magnitude-based selection.
Questions for authors
  1. Table 11 shows that variance-based calibration performs better than activation-based calibration when both use denoising. Could the authors rescale the activation-based denoising update to match PRISM’s final update norm and compare their performance in the same setting? Reporting the six-task mean and perplexity would help clarify whether variance-based calibration provides an additional benefit beyond reducing the update magnitude.
10 comments
AuthorsOpenReview ↗

Response to Reviewer obyq (1/7)

We thank the reviewer for recognizing that predicting and repairing collapse without calibration data addresses an important problem, and for noting the extensive evaluation.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

In short, six points.

1, W1. Introduction, method and theory. The revision expands Sections 1, 3 and 4. The contributions, the working model's scope and the procedure are therefore explicit in the main text. It adds a contribution list and the working model and argument of Appendix B in Section 3.2. It also adds a summary of Algorithm 1 in Section 4.
2, W2. Harmful drift missed by variance. The recipe screens merges in which little cancels with total power. Shared-drift collapse is therefore detected even when cross-expert variance stays low. Total power measures shared drift in the same decomposition, and the other merges are screened with interference, as submitted. The recipe flags all fifteen destructive merges and none of the twenty-one harmless ones of Tables 4 and R7 and revised Tables 22 and 23.
3, W2. Gate bypass of repair. The recipe's repair keeps all nine evaluated destructive low-cancellation merges within 1.7pp of the base. A flag therefore leads to repair rather than the destructive average. For these flagged merges, it retains the largest 0.1% of each matrix's averaged update instead of bypassing shrinkage. Seven of these merges were flagged before evaluation and two are in DeepSeek-7B (Tables R1 and R7 and revised Table 22).
4, W3. Dependence on the retention floor. PRISM's interference thresholds repair all five destructive merges without the floor. With the floor disabled, PRISM stays no more than 0.4pp below the base of Table 4 on all five above-threshold destructive merges. Repair therefore persists after fixed-quantile sparsification is removed. On the main merge, its interference thresholds alone score 68.6 against the base's 68.0. Table R6 gives the five merges, and Appendix D.2 gives the main merge. At κ≤0.5\kappa\le0.5, no floor binds in any of the five above-threshold destructive merges. There the main merge scores 67.4 and 67.3. Naive averaging scores 49.5. These runs isolate repair by interference thresholds where the floor never sets a cut (Tables 4 and 6).
5, W3. Uniform shrinkage as the repair mechanism. At PRISM's norm, concentrating the update on the kept 0.1% scores higher than spreading it uniformly on all four EvalPlus metrics. Coordinate placement therefore improves coding after update size is controlled (Table 3).
6, W3, Q1. Activation calibration at the same norm. Rescaled to PRISM's norm, the activation-calibrated update of Table 11 scores below PRISM on all four EvalPlus metrics. Variance-based repair therefore adds a coding benefit beyond matched shrinkage. Both six-task means stay within 0.4pp of the base, so this benefit preserves near-base general capability (Table R5).

AuthorsOpenReview ↗

Response to Reviewer obyq (2/7)

W1. Expanded introduction, method and theory.

"the method and theoretical analysis sections are too brief and underdeveloped, while the introduction is inadequately organized and its statement of the main contributions lacks clarity."

Answer. The revised PDF expands all three parts that the review names. They are the introduction, the method and the theory, and Section 1 now also states the contributions in a list.

Within the page limit of the submission, we gave most of the space to the experimental results behind the screen and the repair. As these results accumulated, they made the paper harder to follow and left the method and theory sections shorter than they should have been. We did not give the presentation of these sections the time it needed, and the reviewer's reading showed us where more explanation was required. The revised PDF expands these sections as follows.

  • Section 1 ends with three contributions, and the first is the interference score as a pre-merge screen with its prospective test. The second is PRISM as a data-free repair calibrated by the same variance. The third is the controlled titration that links the score to specialist training.
  • Section 3.2 adds a paragraph, The working model and what it leaves out. It states the argument of Appendix B for why averaging keeps the shared drift and divides the disturbance variance by K. It also states why a drift that all specialists share leaves interference unchanged.
  • Section 4 now opens with a summary of Algorithm 1, which gives the order of normalization, averaging, the threshold of each layer, the gate and the floor. This summary comes before the details of Sections 4.1 to 4.3.

We also revised the rest of the main text section by section for length and wording. These additions make the working model's scope and the algorithm's sequence explicit in the main text.

The main text now states the contributions, the argument behind the screen and the order of the algorithm.

We thank the reviewer for this point, which the paper needed.

The additions appear in the revised PDF's Introduction, “The working model and what it leaves out,” and the opening summary of the algorithm.

AuthorsOpenReview ↗

Response to Reviewer obyq (3/7)

W2. Screening and repairing both named merges.

"More generally, task vectors may share a harmful drift that is not captured by their cross-expert variance. In low-cancellation cases, the gate also bypasses shrinkage, leaving potentially destructive updates unrepaired."

Answer. The recipe flags Math+Finance and Coder+Coder-Instruct and keeps them near the base. To compare screens for harmful shared drift missed by cross-expert variance, Table R1 collects thresholds and counts of correct predictions from revised Appendix J. The repaired merges score 67.9 and 67.0. The base scores 68.1. On the same merges, naive averaging scores 37.0 and 39.3. Table R7 gives these scores.

To compare destructive and harmless shared drift, the two tables below collect scores from Table 4 and revised Appendix J and Table 23.

Low-cancellation mergeρ\rhoTA six-task meanBase six-task meanOutcome
Qwen2.5-7B Math + Finance0.03637.068.1destructive
Qwen2.5-7B Coder + Coder-Instruct0.01339.368.1destructive
Mistral-7B MetaMath + Code0.09359.957.4harmless
Llama-3.1-8B Tulu-3 + Tulu-3-DPO0.00364.559.3harmless
Qwen2.5-7B-Instruct + Uncensored0.02368.868.1harmless
Total-power score (×10⁻³)Value
Math + Finance8.21
Coder + Coder-Instruct6.97
The two harmless low-cancellation merges of Table 40.047 and 0.129
Instruct + Uncensored0.101

Total power therefore flags these harmful shared drifts despite their low variance, without flagging the harmless low-cancellation merges.

Why the variance misses them. The two configurations named in the review are parent-descendant pairs. Coder-Instruct is trained from Coder, and the Finance model descends from Qwen2.5-Math-7B through DeepSeek-R1-Distill-Qwen-7B. Section 3.1 states that identical scaled task vectors give zero interference regardless of their shared drift. The shared drift is measured by total power P, the sum of signal power and interference in Equation 1.

A shared drift alone is not destructive. One example is Qwen2.5-7B-Instruct + Uncensored, where Uncensored descends from Instruct. It was predicted harmless before evaluation. Low cancellation and descent therefore do not trigger a flag when the shared update has low power.

What the recipe does. The recipe screens each merge with the term that its cancellation share selects. Where ρ≥τg\rho\ge\tau_g, it keeps the interference threshold of 1.9×10⁻³. Where ρ<τg\rho<\tau_g, it flags a merge whose total power exceeds 1.5×10⁻³, a threshold placed on the same nine initial configurations as the paper's. The split reuses the gate τg=0.1\tau_g=0.1 of the submitted operator, so it adds no parameter beyond the total-power threshold.

Table R1. Thresholds placed on the nine initial configurations (two destructive, seven harmless) and applied to the fourteen prospective configurations of Section 5.7 and the twenty-two of Table 4. The predictions made before evaluation used interference, and the other rows apply their thresholds to the same configurations afterwards. The paper's interference threshold of 1.9×10⁻³ gives the same counts as 0.98×10⁻³.

ScreenThreshold (×10⁻³)Prospective correctTable 4 correct
Interference c√(I/K)0.9812 of 1420 of 22
Total power c√(P/K)1.514 of 1422 of 22
Signal power c√(S/K)1.114 of 1422 of 22
Recipe, interference where ρ≥τg\rho\ge\tau_g and total power where ρ<τg\rho<\tau_g1.9 or 1.514 of 1422 of 22

The split corrects the cases that interference alone misses in Table 4 and the prospective set.

The recipe flags all seven destructive merges of Table 4 and none of the fifteen harmless ones. Applied afterwards to the fourteen prospective configurations, it is correct on all fourteen (Table R1, Appendix J).

The recipe adds no parameter beyond the total-power threshold, and its repair keeps both merges near the base.

Total power from Equation 1 flags both named merges, and the revised recipe repairs them near the base.

The recipe is in revised Section 5.7.

Reviewer_ru18OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes that the variance of task vectors across specialists, a measure of interference computable from the weights alone, both predicts when merging fine-tuned LLMs will collapse and calibrates how to repair it. Across merge configurations from four model families, a pre-merge interference score exceeds a fixed threshold only for destructive merges, and 12 of 14 predictions made before evaluation were correct; sign-conflict statistics, by contrast, turn out to be anti-predictive. The authors also propose PRISM, a data-free operator that averages task vectors and then soft-thresholds each layer at a level set by its interference.

Critical strengths
  1. The problem is novel and interesting. While most prior work asks how to merge models, this paper asks when models should be merged, which is a relatively new question.
  2. The experimental setup is fair, including predictions of outcomes for 14 new configurations made before evaluation.
  3. The method is simple and straightforward, making it easy to use in practice.
Critical weaknesses
  1. The score exceeds the threshold for only five of the seven destructive merges. The two it misses (Qwen2.5-7B Math+Finance and Coder+Coder-Instruct) score far below the threshold and are also gated by ρ, so under the proposed recipe they would be merged with plain averaging and collapse, and PRISM cannot repair them either.
  2. Although the paper reports 22 configurations, if I understand correctly, the samples that actually determine the conclusions are much fewer. Only 7 of the 22 are destructive, and the threshold of 1.9×10^{-3} was set on the nine initially measured configurations, of which only two (Qwen2.5-1.5B and 7B Math+Coder) are destructive. With so few positive examples, the threshold and the reported AUC values may be unstable.
  3. All seven destructive configurations come from Qwen2.5, the same faily, and most involve Qwen2.5-Coder or Math+Coder combinations, while the Llama-3.1, Mistral and DeepSeek configurations are all harmless. The claim that exceeding the threshold predicts collapse has therefore only been validated on Qwen2.5, and it remains unclear whether the threshold transfers to other model families.
Questions for authors

Please refer to the weaknesses section

11 comments
AuthorsOpenReview ↗

Response to Reviewer ru18 (1/8)

We thank the reviewer for finding the question of when to merge novel and the prospective setup fair.

Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.

In short, three points.

1, W1. Missed low-cancellation collapse. The recipe flags and repairs both merges that the review names. These cases therefore no longer pass to a destructive plain average, while harmless merges remain unflagged. It screens merges in which little cancels (ρ<τg\rho<\tau_g) with total power from the same decomposition, and the others with interference, as submitted. Both named merges are parent-descendant pairs with ρ<τg\rho<\tau_g. The recipe flags all fifteen destructive merges of Tables 4, R7 and R8, including both named ones, and none of the twenty-one harmless ones of Tables 4, R8 and R9. Before evaluation, total power at this threshold flagged five further Qwen2.5-7B pairs and two DeepSeek-7B pairs, and all proved destructive. Retaining the largest 0.1% of each matrix's averaged update keeps all nine merges of this kind within 1.7pp of the base. Naive averaging scores 37.0 and 39.3 on the two of Table 4 (Tables R1, R7 and R8).
2, W2. Few positive examples and threshold stability. Both thresholds preserve their decisions across ranges, so the conclusions do not depend on their exact initial placement. Every interference threshold from 0.70 to 4.17×10⁻³ gives the same decisions on all twenty-two configurations and the first titration endpoint (Table R4). The submitted interference threshold can therefore be divided by up to 2.7 or multiplied by up to 2.2 without changing a decision. Every total-power threshold from 0.324 to 6.97×10⁻³ gives the same decisions on every evaluated merge of two or more specialists with ρ<τg\rho<\tau_g. The total-power threshold can therefore be divided or multiplied by up to 4.6 without changing a decision (revised Appendix J).
3, W3. Transfer beyond Qwen2.5 and code specialists. The unchanged screen predicts destructive and harmless outcomes in a second model family. Collapse prediction therefore extends beyond Qwen2.5 in these tests, and destructive merges do not require a code specialist. On four further public DeepSeek-7B pairs scored before evaluation, the recipe predicted all four outcomes, three destructive and one harmless. Its repair keeps the three destructive merges within 1pp of the base (Table R8). Inside Qwen2.5, five pairs of Qwen2.5-7B-Instruct and models derived from it, also scored before evaluation, were all predicted harmless and are harmless (Table R9). On Llama-3.1-8B and Mistral-7B, five further public pairs scored before evaluation were predicted harmless and are harmless (Tables R10 and R11). Seven of the twelve destructive merges of Table 4 and Table R7 involve no code specialist.

AuthorsOpenReview ↗

Response to Reviewer ru18 (2/8)

W1. Screening and repairing both named merges.

"The two it misses (Qwen2.5-7B Math+Finance and Coder+Coder-Instruct) score far below the threshold and are also gated by ρ, so under the proposed recipe they would be merged with plain averaging and collapse, and PRISM cannot repair them either."

Answer. Under the revised recipe, neither merge is returned as a plain average. To compare screening of the destructive merges missed by interference, Table R1 collects thresholds and counts of correct predictions from revised Appendix J. The total-power screen flags both. The recipe's repair keeps them at 67.9 and 67.0, close to the base's 68.1 (Table R7). Naive averaging scores 37.0 and 39.3 on the same merges.

To compare destructive and harmless shared drift, the table below collects total-power scores c√(P/K) from Table 4.

Low-cancellation mergesTotal-power scores
Math+Finance and Coder+Coder-Instruct8.21 and 6.97×10⁻³
The two harmless merges0.047 and 0.129×10⁻³

All nine destructive merges that interference misses have ρ<τg\rho<\tau_g, and total power flags all nine.

Why interference misses them. Coder-Instruct is trained from Coder, and the Finance model descends from Qwen2.5-Math-7B through DeepSeek-R1-Distill-Qwen-7B. In each pair, the scaled task vectors nearly coincide. Section 3.1 states that identical scaled task vectors give zero interference, regardless of the size of their shared drift. For Coder+Coder-Instruct ρ\rho = 0.013, and for Math+Finance ρ\rho = 0.036. Little cancels in both pairs (Table 4).

What flags them. The size of the shared drift is measured by total power P, the sum of signal power and interference in Equation 1.

What the recipe does. The recipe (revised Section 5.7) screens each merge with the term that its cancellation share selects. Where ρ≥τg\rho\ge\tau_g, it keeps the interference threshold of 1.9×10⁻³. Where ρ<τg\rho<\tau_g, it flags a merge whose total power exceeds 1.5×10⁻³, the geometric midpoint of the total-power gap on the same nine initial configurations. The split reuses the gate τg=0.1\tau_g=0.1 of the submitted operator, so it adds no parameter beyond the total-power threshold.

Table R1. Thresholds placed on the nine initial configurations (two destructive, seven harmless). They are applied to the fourteen prospective configurations of Section 5.7 and the twenty-two of Table 4. The predictions made before evaluation used interference. The other rows apply their thresholds to the same configurations afterwards. The paper's interference threshold of 1.9×10⁻³ gives the same counts as the threshold in the Interference row.

ScreenThreshold (×10⁻³)Prospective correctTable 4 correct
Interference c√(I/K)0.9812 of 1420 of 22
Total power c√(P/K)1.514 of 1422 of 22
Signal power c√(S/K)1.114 of 1422 of 22
Recipe, interference where ρ≥τg\rho\ge\tau_g and total power where ρ<τg\rho<\tau_g1.9 or 1.514 of 1422 of 22

Selecting the statistic by cancellation share corrects the interference-only screen's misses in Table 4 and in the prospective tests.

Result. The recipe flags all seven destructive merges of Table 4 and none of the fifteen harmless ones. Applied afterwards to the fourteen prospective configurations, it is correct on all fourteen (Table R1). Where averaging cancels (ρ≥τg\rho\ge\tau_g), interference flags all six destructive merges of Tables 4 and R8 and none of the eighteen harmless ones of Tables 4, R8 and R9. The submitted screen's predictions made before evaluation were correct on all eleven prospective configurations of this kind.

The gate that the submitted operator already uses decides which term to read, so the recipe needs no new measurement and adds one threshold. Interference remains the screen wherever averaging cancels, and there it separates all twenty-four merges of Tables 4, R8 and R9.

Both named merges are flagged and repaired by the revised recipe, whose screening statistic is already part of Equation 1. The revised Appendix J reports the screening comparison and low-cancellation repairs.

AuthorsOpenReview ↗

Response to Reviewer ru18 (3/8)

W1. Continued. Prospective predictions and repairs.

"PRISM cannot repair them either."

The recipe repairs both named merges and correctly predicts five further destructive pairs. To test whether low-cancellation merges can be repaired, we evaluated five further public Qwen2.5-7B pairs after scoring them against the total-power threshold. For low-cancellation merges, the recipe retains the largest 0.1% of each matrix's averaged update instead of returning the plain average.

Setup. All five have ρ<τg\rho<\tau_g and score above it. In each, both specialists belong to the Qwen2.5-Math-7B lineage, like the Finance pair.

Seven-merge comparisonResult
Recipe repair, six-task mean range66.4 to 68.5
Base six-task mean68.1
Naive-average perplexity range (Table R7)142 to 10,485
Naive-average means on the two pairs of Table 437.0 and 39.3

The low-cancellation repair recovers near-base capability where naive averaging collapses.

Table R7. Merges with ρ<τg\rho<\tau_g that the total-power screen flags, added to the revised PDF as Table 19 (Qwen2.5-7B, K=2, c=0.5). Scores are c√(x/K) ×10⁻³. The gate of the denoising operator returns Task Arithmetic (TA) for every pair. The recipe's repair (PRISM columns) retains the largest 0.1% of each MLP projection matrix's averaged update, soft-thresholded at that magnitude. This is the operator that the κ≥2\kappa\ge2 row of Table 6 evaluates on the main merge. Six-task mean (%) and WikiText-2 perplexity (PPL). The base scores 68.1 at perplexity 8.6.

Pairρ\rhoInterferenceTotal powerTA PPLPRISM meanPRISM PPL
Math + Finance (Table 4, TA mean 37.0)0.0360.6538.2110,48567.910.4
Coder + Coder-Instruct (Table 4, TA mean 39.3)0.0130.1906.97141.767.09.3
R1-Distill + Math-PRM, scored before evaluation0.0500.8537.681,14366.810.3
Math + Math-PRM, scored before evaluation0.0380.6537.681,19366.710.4
Math + Math-Instruct, scored before evaluation0.0380.6537.681,19266.410.4
R1-Distill + Math, scored before evaluation0.0360.6438.208,61168.110.2
R1-Distill + Light-R1, scored before evaluation0.0030.0528.209,28468.510.2

The repair brings all seven merges of Table R7 to within 1.7pp of the base.

Total power predicted the collapse of all five new merges before they were evaluated.

The kind of merge that the review names is now both flagged and repaired.

Revised Section 5.7 and Table 19 report them.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General Response (1/5)

We thank the reviewers for their careful reading. Reviewer ru18 finds the question of when to merge, rather than how, novel and the prospective setup fair. Reviewer hMV3 values a check of collapse risk from the weights before any merged model is built, and notes that the prospective predictions go beyond retrospective correlation. Reviewer obyq finds the problem important and the evaluation extensive. The reviews raise three main points, and the revised PDF answers each with new results.


1. Floor-free repair and same-norm alternatives (hMV3, obyq)

Answer. PRISM's interference thresholds repair the five above-threshold destructive merges without the retention floor. On the five above-threshold destructive merges, PRISM without the retention floor stays no more than 0.4pp below the base. At the same norm, PRISM's default update scores above both uniform shrinkage and the activation-calibrated update. This holds on all four EvalPlus metrics of the main merge.

We disable the floor, vary the interference threshold and compare updates at matched norms. The existing tables report the following measurements.

Floor or threshold comparisonMeasurementSource
Main merge without floor, coordinates kept0.0081%Appendix D.2 and Table 3
Main merge without floor, six-task mean / base68.6 / 68.0Appendix D.2 and Table 3
Main merge without floor, perplexity8.9Appendix D.2
All five destructive merges without floor, largest shortfall to base0.4ppTable 4, Table R6, Table 6
Floor-free gain over Task Arithmetic14.2 to 20.2ppTable R6
At κ≤0.5\kappa\le0.5, floor-active matrices in each of the five mergesNone of 84 MLP projection matricesTable 6, bottom
Main merge at κ=0.25\kappa=0.2567.4Tables 4 and 6
Main merge at 0.567.3Tables 4 and 6
Naive average on the same merge49.5Table 4
Coordinates kept by the interference threshold at default without floor0.002 to 0.08%Table 6, bottom

The interference thresholds repair the merges on their own. The floor-free six-task means remain within evaluation uncertainty of the floored operator. The floor binds at default because the interference threshold would keep fewer coordinates.

At PRISM's norm, we compare its concentrated update with uniform shrinkage and with LEWIS-CTP rescaled from Table 11. The latter comparison evaluates both updates in one environment.

The first three rows come from Table 3 and the last four from one evaluation environment of revised Table 11, in which the coding scores of PRISM and the base differ from Table 3 by at most 1.8 points.

EvalPlus pass@1 (%)HumanEvalHumanEval+MBPPMBPP+Source
PRISM, norm 0.022, kept 0.1%65.959.171.261.1Table 3
Uniform control, norm 0.02262.257.369.659.0Table 3
Base62.857.969.859.5Table 3
PRISM, one environment64.657.371.261.1Table R5, revised Table 11
Floor-free PRISM, norm 0.014, one environment62.857.370.660.8Table R5, revised Table 11
Rescaled LEWIS-CTP, norm 0.022, one environment61.656.168.559.3Table R5, revised Table 11
Base, one environment62.257.370.159.3Table R5, revised Table 11
Same-norm measurementValueSource
Six-task-mean gap bound to base for PRISM or rescaled LEWIS-CTP0.4ppTable R5
Floor-free PRISM norm relative to the average1.4%Table R5

PRISM's default update exceeds both same-norm alternatives on all four EvalPlus metrics. Floor-free PRISM also exceeds the rescaled LEWIS-CTP update on all four metrics.

PRISM's interference thresholds, not the floor, do the repair. The revised PDF reports the floor-free results in Appendix D.2 and the rescaled-activation comparison in Appendix D.7.

AuthorsOpenReview ↗

General Response (2/5)

2. Screening with the cancellation-selected statistic (hMV3, ru18, obyq)

Answer. The recipe screens each merge with the decomposition term selected by its cancellation share. It flags all fifteen destructive merges of Tables 4, R7 and R8 and none of the twenty-one harmless merges of Tables 4, R8 and R9. Its repair keeps all fifteen within 1.7pp of the base.

Where averaging cancels. Interference flags all six destructive merges and none of the eighteen harmless ones. These are the twenty-four evaluated merges with ρ≥τg\rho\ge\tau_g in Tables 4, R8 and R9. Where averaging cancels part of each update (ρ≥τg\rho\ge\tau_g), the recipe screens with interference at 1.9×10⁻³, as in the submission. The predictions that the submitted screen made before evaluation were correct on all eleven prospective configurations with ρ≥τg\rho\ge\tau_g of Section 5.7. They were also correct on all six of Tables R8 and R9. All nine destructive merges that interference misses have ρ<τg\rho<\tau_g. Total power flags all nine.

To compare the screening statistics, Table R1 collects thresholds and counts of correct predictions from revised Appendix J.

Table R1. Thresholds placed on the nine initial configurations (two destructive, seven harmless) and applied to the fourteen prospective configurations of Section 5.7 and the twenty-two of Table 4. The predictions made before evaluation used interference, and the other rows apply their thresholds to the same configurations afterwards. The paper's interference threshold of 1.9×10⁻³ gives the same counts as 0.98×10⁻³.

ScreenThreshold (×10⁻³)Prospective correctTable 4 correct
Interference c√(I/K)0.9812 of 1420 of 22
Total power c√(P/K)1.514 of 1422 of 22
Signal power c√(S/K)1.114 of 1422 of 22
Recipe, interference where ρ≥τg\rho\ge\tau_g and total power where ρ<τg\rho<\tau_g1.9 or 1.514 of 1422 of 22

Where little cancels. Total power also separates all thirty-six merges of Tables 4, R7, R8 and R9. It is the sum of both terms of Equation 1 and is computed in the same pass. Every destructive merge scores more than 21 times the total power of every harmless one. The recipe reads ρ\rho to decide which repair runs. The repair is PRISM's interference thresholds where averaging cancels, and the 0.1% floor where little cancels.

The screen uses interference where averaging cancels and total power where shared drift leaves interference low.

The cancellation share selects both the screening term and the repair. The revised recipe and screening comparisons are in Appendix J.

AuthorsOpenReview ↗

General Response (3/5)

2, continued. Low-cancellation repair and additional model families

Low-cancellation merges use total-power screening and quantile repair when flagged.

To compare low-cancellation repair with naive averaging, the table below collects the results from Table R7, revised Section 5.7 and Table 19.

Low-cancellation comparisonResult
Recipe repair on all seven merges, six-task mean66.4 to 68.5
Base six-task mean68.1
Naive-average perplexity on the same merges142 to 10,485

The low-cancellation branch therefore restores capability where the submitted gate would return a destructive average.

Where little cancels (ρ<τg\rho<\tau_g), a drift that the specialists share leaves interference low. There the recipe screens with total power P, the sum of signal power and interference in Equation 1. The screen uses 1.5×10⁻³, the geometric midpoint of the total-power gap on the same nine initial configurations. The split reuses the gate τg=0.1\tau_g=0.1 of the submitted operator. It therefore adds no parameter beyond the total-power threshold. The screen flags all seven destructive merges of Table 4 and none of the fifteen harmless ones. Applied afterwards to the fourteen prospective configurations, it is correct on all fourteen (Table R1, revised Table 18 bottom). Five further public pairs with ρ<τg\rho<\tau_g were flagged by total power at this threshold before evaluation. All five proved destructive under naive averaging (revised Section 5.7 and Table 19). Total power thus detects shared-drift collapse that the interference-only screen would pass to plain averaging.

Repair. On all seven merges of Table R7, the recipe's low-cancellation repair stays within 1.7pp of the base. A flagged merge with ρ≥τg\rho\ge\tau_g is repaired by PRISM as in the submission. A flagged merge with ρ<τg\rho<\tau_g has little to cancel. The recipe therefore retains the largest 0.1% of each matrix's averaged update, soft-thresholded at that magnitude. This is the operator that the κ≥2\kappa\ge2 row of Table 6 evaluates on the main merge.

Why little cancels in these pairs. In each of these pairs, one specialist descends from the other by fine-tuning, or both descend from a third specialist. Qwen2.5-Coder-7B-Instruct is trained from Qwen2.5-Coder-7B. The Finance specialist descends from Qwen2.5-Math-7B. Its path runs through DeepSeek-R1-Distill-Qwen-7B. In both pairs, the scaled task vectors nearly coincide, so little cancels (Table 4). Their values of ρ\rho are 0.013 and 0.036. Total power measures this shared drift.

Other model families. On a second model family, the recipe predicted all four outcomes of four public DeepSeek-7B pairs scored before evaluation. Its repair keeps the three destructive merges within 1pp of the base. Table R8 and revised Table 22 report these pairs. Inside Qwen2.5, five pairs were also scored before evaluation. These are pairs of Qwen2.5-7B-Instruct and four public models derived from it. All five were predicted harmless and are harmless (Table R9, revised Table 23). They include Instruct + Uncensored, a parent-descendant pair with ρ\rho = 0.023. On Mistral-7B, two further public pairs scored before evaluation were predicted harmless. They are harmless (Table R10, revised Table 24). On Llama-3.1-8B, three more public pairs scored before evaluation were predicted harmless and are harmless (Table R11). These predictions extend screening and repair to a second family while keeping the tested harmless pairs unflagged.

To test whether the screen holds on public pairs scored before evaluation, we ran new experiments on pairs from four model families.

Public pairs scored before evaluationPairsOutcomePredictions correctSource
DeepSeek-7Bfourthree destructive, one harmlessall fourTable R8
Qwen2.5-7B-Instruct and its descendantsfiveall harmlessall fiveTable R9
Mistral-7Btwoboth harmlessbothTable R10
Llama-3.1-8Bthreeall harmlessall threeTable R11

Interference covers the merges where averaging cancels. Total power covers those where a shared drift leaves interference low. The cancellation share that the operator already computes chooses between them. The recipe therefore adds one threshold and no new measurement.

The two terms of the decomposition together cover every evaluated destructive merge. The revised Appendix J reports the low-cancellation repair and model-family tests.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.