- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies destructive merging of language-model specialists derived from a shared base. It defines interference as the coordinate-wise variance of scaled task vectors and motivates a pre-merge risk score , through a shared-signal, independent-disturbance model. It also proposes PRISM, which soft-thresholds the averaged task vector using layer-wise interference, with a minimum-retention floor and a cancellation-based bypass gate. The evaluation covers 22 configurations across four model families. A fixed score threshold identifies five of seven destructive configurations without flagging any of the fifteen harmless ones; the authors additionally report twelve correct predictions out of fourteen prospective evaluations. PRISM keeps the five above-threshold destructive merges near base-model performance. Further experiments examine coefficient sensitivity, norm-matched shrinkage, specialist count, and a controlled continued-pretraining trajectory.
Critical strengths
- The paper addresses a practically useful problem and includes prospective and controlled evidence.
Checking collapse risk from weights before constructing and evaluating a merged model is a useful objective. The reported prospective predictions strengthen the evidence beyond retrospective correlation.
- The paper evaluates several merge operators, coefficient sweeps, multiple model families, and both general capability and generation benchmarks. Importantly, it includes the unmerged base, a norm-matched uniform-shrinkage control, and floor/threshold ablations.
Critical weaknesses
- The main repair results do not sufficiently isolate the contribution of interference-calibrated thresholding from fixed-quantile sparsification and conservative shrinkage.
Appendix D.2 states that, on the main Qwen2.5-7B Math+Coder setting, the 0.1% retention floor binds in every layer at the default κ=1. According to Algorithm 1, when this happens, the final threshold is replaced by the magnitude of the relevant order statistic of the averaged task vector. The available controls point the same way: the uniform norm-matched control (68.1) and the unmerged base (68.0) match PRISM on the six-task mean. Interference thus enters only through the gate, not through the repair itself.
- The predictive evidence does not establish an advantage of interference over simpler update-magnitude statistics, and its generalization remains limited.
Table 18 reports that total power and signal power both achieve AUC 1.00 and completely separate the 22 configurations, whereas the proposed interference score has AUC 0.98 and misses two destructive merges. The signal-power score is directly related to the RMS magnitude of the averaged update. Thus, the current evidence is also consistent with a simpler interpretation: large applied updates are dangerous, without task-vector disagreement being the essential predictive quantity.
Questions for authors
-
Interference calibration versus the retention floor. For each of the five above-threshold destructive configurations, what fraction of matrices has an active floor? Can the authors compare PRISM against fixed-0.1%-quantile soft thresholding under identical normalization, gating, parameter scope, and coefficient settings? On the main setting, should these methods be exactly equivalent at the default?
-
Choice of screening statistic. Given the stronger separation obtained by total and signal power in Table 18, what evidence favors interference as the screening statistic? Can the authors compare these alternatives using thresholds fixed on the same initial configurations and evaluated on the same prospective cases?
-
Missing base row in Table 1. Please add the unmerged base model to Table 1. By the numbers in App. D.11, PRISM is below the base on all four generation metrics, and this should be stated and discussed in the main text rather than left in the appendix.
15 comments
Response to Reviewer hMV3 (2/9)
W1, Q1. Repair without the floor.
"Appendix D.2 states that, on the main Qwen2.5-7B Math+Coder setting, the 0.1% retention floor binds in every layer at the default κ=1."
Answer. PRISM calibrates each layer's repair threshold with interference and repairs all five above-threshold destructive merges with the floor disabled. It does not use interference only through the gate. On all five above-threshold destructive merges, PRISM without the floor stays no more than 0.4pp below the base. To check whether the retention floor binds in every layer, we added measurements of floor activation across projection matrices, destructive merges and threshold scales. Where the floor does not bind, the interference threshold alone sets every cut. Appendix D.2, in the paragraph before Table 6, gives the main-merge measurements.
To compare repair where no floor binds with naive averaging, the table below collects the main-merge scores from Tables 4 and 6.
| Main-merge six-task mean | Score |
|---|---|
| Interference thresholds with no active floor | 67.4 and 67.3 |
| Naive averaging | 49.5 |
The runs therefore isolate repair by the interference threshold in all five merges.
| Main merge without floor | Value |
|---|---|
| Coordinates kept by the interference thresholds | 0.0081% |
| Update norm relative to the average | 1.4% |
| Six-task mean | 68.6 |
| Base six-task mean | 68.0 |
The interference thresholds alone recover the base's mean on the main merge.
Where the floor binds. We count each projection matrix separately with the floor on, as Algorithm 1 applies it.
Table R2. Matrices in which the 0.1% floor sets the cut, out of the 84 MLP projection matrices. Rows are the five above-threshold destructive merges of Table 4. All use K=2 and c=0.5. Keep without floor is the share of coordinates the interference threshold alone keeps at .
| Setting | Keep without floor | and 3 | ||||
|---|---|---|---|---|---|---|
| Qwen2.5-1.5B ja + med, constructed | 0.342 | 0.0021% | 0 | 0 | 84 | 84 |
| Qwen2.5-1.5B Math + Coder | 0.246 | 0.081% | 0 | 0 | 66 | 84 |
| Qwen2.5-7B Math + Coder | 0.293 | 0.0081% | 0 | 0 | 83 | 84 |
| Qwen2.5-7B R1-Distill + Coder | 0.291 | 0.0085% | 0 | 0 | 83 | 84 |
| Qwen2.5-7B Math-PRM + Coder | 0.285 | 0.011% | 0 | 0 | 83 | 84 |
At , exceeds the 99.9th percentile of in 83 of the 84 MLP projection matrices (Table R2). At , where the interference threshold keeps more than 0.1% of a matrix, no floor binds in any matrix of the five merges.
The matrix-level counts identify where the floor binds and demonstrate repair where interference thresholds alone set every cut.
The matrix counts are added to the revised PDF as the bottom part of Table 6.
Response to Reviewer hMV3 (3/9)
W1, Q1. Continued. Floor-free repair across all five merges.
"The main repair results do not sufficiently isolate the contribution of interference-calibrated thresholding from fixed-quantile sparsification and conservative shrinkage."
With the floor disabled, every above-threshold destructive merge stays no more than 0.4pp below the base. To isolate the contribution of interference-calibrated thresholding from fixed-quantile sparsification, we ran a new experiment on destructive merges with the retention floor disabled.
Table R6. PRISM with the 0.1% floor disabled, on the five above-threshold destructive merges of Table 4. It runs at and c=0.5. Six-task means (%). Base, Task Arithmetic and PRISM are from Table 4. PPL is the floor-free merge's perplexity, with the base's in parentheses.
| Setting | Base | Task Arithmetic | PRISM | PRISM without floor | PPL without floor |
|---|---|---|---|---|---|
| Qwen2.5-1.5B ja + med, constructed | 56.1 | 35.8 | 55.9 | 56.0 | 13.5 (13.0) |
| Qwen2.5-1.5B Math + Coder | 56.3 | 37.1 | 55.8 | 55.9 | 13.1 (13.0) |
| Qwen2.5-7B Math + Coder | 68.0 | 49.5 | 67.8 | 68.6 | 8.9 (8.6) |
| Qwen2.5-7B R1-Distill + Coder | 68.1 | 49.7 | 67.7 | 68.1 | 8.9 (8.6) |
| Qwen2.5-7B Math-PRM + Coder | 68.1 | 53.7 | 67.6 | 67.9 | 8.9 (8.6) |
Result without the floor. On all five above-threshold destructive merges, PRISM without the floor stays no more than 0.4pp below the base of Table 4. It also scores 14.2 to 20.2pp above Task Arithmetic. On each merge, its six-task mean is within evaluation uncertainty of the floored operator's, 0.1 to 0.8pp above it. Table R6 and Table 6 of the revised PDF report these means.
Disabling the floor lowers the six-task mean on none of the five merges.
The floor only lowers the cut of a matrix in which the interference threshold would keep fewer than 0.1% of the coordinates, so it adds to the update and removes none of it. At , the interference thresholds set every cut.
Without the floor, the update of the interference thresholds has 1.4% of the average's norm, less than the uniform control's 2.2%. We compare this update with that control on all four EvalPlus metrics (Appendix D.2, uniform control from Table 3).
| EvalPlus pass@1 (%) | HumanEval | HumanEval+ | MBPP | MBPP+ |
|---|---|---|---|---|
| PRISM without floor, norm 1.4% | 63.4 | 57.3 | 70.6 | 60.8 |
| Uniform control, norm 2.2% | 62.2 | 57.3 | 69.6 | 59.0 |
The floor-free update matches or beats the uniform control on every metric with a smaller update.
The repair itself comes from the interference thresholds, which recover all five merges on their own.
The revised PDF adds these means as the Mean without floor column of Table 6 and states them in Appendix D.2.
Response to Reviewer hMV3 (1/9)
We thank the reviewer for the close read and the precise questions.
Before writing this response, we ran a number of new experiments for the reviews and restructured the paper around their results, which took much of the discussion period. We realize that little time remains, and we respectfully ask the reviewer to read our response and the new experiments.
In short, three points.
1, W1, Q1. Floor versus interference and shrinkage. PRISM's interference thresholds repair the five destructive merges without the floor, so interference is used in the repair itself, not only through the gate. With the floor disabled, PRISM's thresholds keep all five above-threshold destructive merges no more than 0.4pp below the base. The floor is therefore not required for repair. At κ≤0.5, no floor binds in any of the five. At PRISM's norm, concentrating the update on the kept 0.1% outscores uniform shrinkage on all four EvalPlus metrics, so coordinate placement adds a coding benefit beyond update size (Table R6, Table R2 and Table 3).
2, W2, Q2. Choice of screening statistic and shared drift. The recipe flags all fifteen destructive merges and none of the twenty-one harmless ones. Cancellation-selected screening therefore covers the shared-drift misses and selects a repair instead of the destructive average. The screening comparison covers Tables 4 and R7 and revised Tables 22 and 23. Its predictions were correct on all fourteen public pairs scored before evaluation, five in Table R7 and nine in revised Tables 22 and 23. Retaining the largest 0.1% of each matrix's averaged update keeps all nine flagged merges in which little cancels within 1.7pp of the base (Tables R1 and R7, revised Table 22).
3, Q3. Missing base comparison. PRISM matches the base's six-task mean within 0.2pp and scores above the base on all four EvalPlus coding metrics, as the comparison with the new base row in Section 5.2 states. Table 1 now includes the base and makes PRISM's deficits on all four harness generation metrics explicit. The main-text comparison therefore distinguishes general-capability recovery from remaining generation deficits.