acceptodds
Under review as a conference paper at ICLR 2027

An Error Decomposition for Merging and Pruning Neural Networks

Abstract

While pruning removes units from a trained network, and merging combines several trained networks into one, many methods from both fields can be expressed as linear layer-wise compression methods. We introduce an error decomposition for such methods, which splits the layer-wise compression error exactly into a reconstruction error, which arises from feature information lost through compression, and a commutation error, which arises from the method's failure to commute with the network nonlinearities. Propagating these two local errors through the neural network gives a calibration-data based estimate of the output deviation between the compressed and initial network(s), which can be used to rank methods. Further the analysis reveals that the decoder—the linear operation that reconstructs the original features from the compressed representation—can be fitted independently in closed form once the encoder is fixed, because only the encoder is constrained to commute with the nonlinearity. Paired with the fitted decoder, a simple neuron-importance selection encoder, which has no commutation error, outperforms the considered merging baselines, including ZipIt and PLeaS, on same-task benchmarks across MLP, convolutional, and residual architectures. It performs on par with selecting the encoder using the predicted output deviation, showing that the decoder accounts for most of the downstream improvement. For Llama models, our experiments show that the derived reconstruction and calibration-selected axis allocation changes improve over FLAT-LLM constructions at matched parameter counts.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Model Merging, Structured Pruning, Model Compression, Post-training Compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NSuHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies generalized pruning, a framework that unifies structured pruning and model merging as layer-wise encoder–decoder compression of a larger model/ensemble. The main contribution is an error decomposition that separates the layer-wise deviation into three terms: reconstruction error (information lost by the encoder–decoder projection), commutation error (failure of the encoder to commute with the nonlinearity), and propagation error (carrying previous-layer deviation forward). The paper proves an exact recursion and an output-deviation bound, then uses the decomposition to derive practical tools: a generalized PCA/“priced” selection encoder, a closed-form locally optimal least-squares decoder, and a calibration-based predictor of output deviation via linearized error propagation. Empirically, the paper evaluates on vision merging/pruning tasks (MLPs, VGG11, ResNet18, with/without BatchNorm or Fixup) and on structured Llama pruning as adjustments to FLAT-LLM. The decoder yields large gains over native decoders, priced selection is competitive, and the predictor ranks methods well on non-normalized/Fixup models but less reliably on BatchNorm/residual models and under extreme compression. The Llama experiments show modest but consistent improvements over FLAT-LLM at ~0.80× parameters.

Critical strengths
  1. The separation into reconstruction, commutation, and propagation error is exact and conceptually illuminating. Lemma 1 and Theorem 2 give a clean local identity; Theorem 3 unrolls it into an output bound. The paper is careful to note that individual terms have appeared before and positions the contribution as the exact separation plus its exploitation.

  2. The generalized PCA result (Lemma 4) and the commutation characterization (Lemma 5) explain the trade-off between flexible dense kernels and restricted selection kernels. The closed-form decoder in Theorem 7 is elegant and empirically effective: fitting the decoder substantially improves structured pruning at 50% unit pruning (e.g., VGG11 from 49.08% to 86.48%, ResNet18 from 1.00% to 64.95%).

  3. The calibration-based estimator in Section 5 can compare candidate encoders from cached activations of the large model without constructing each compressed model. Table 1 shows perfect rank correlation for MLP/VGG11/Fixup settings and useful correlations for BatchNorm models when candidate-specific statistics are used.

  4. The paper evaluates six merging/pruning families, multiple architectures, and controlled ablations (pre- vs post-build decoder, common decoder across encoders, importance-priced selection, split-task merging). The Llama experiments, while narrower, include controlled adjustments and ablations.

  5. The paper provides detailed experimental settings, calibration protocols, and states that code, configurations, seeds, and raw results are released. The AI-use and reproducibility statements are transparent.

Critical weaknesses
  1. The paper acknowledges that Theorem 3’s bound can overshoot by orders of magnitude. The useful estimator is Eq. 15, which linearizes ReLU propagation and can fail when sign flips occur. The flip-mass diagnostic and exact fallback help, but the predictor’s reliability on BatchNorm/residual models is limited (Table 1: ResNet18-BN rank correlation 0.58, predicted/measured ratio 1.28). The paper notes failure under extreme compression (e.g., VGG11 at 90% pruning). A clearer characterization of when the predictor is trustworthy would strengthen the contribution.

  2. Table 6 shows that keeping model A and fitting only the readout or all layers changes mean accuracy by at most 0.11 points on same-task pairs. This suggests that, in these same-task settings, much of the post-refit accuracy may not require features from both models. The split-task results in Appendix C.3 are more favorable and show gains over one-model controls, but the main text’s same-task framing weakens the claim that the decomposition explains substantial merging benefits. The paper should clarify when merging genuinely requires both models and how the decomposition predicts that.

  3. The Llama results are limited to FLAT-LLM-style structured pruning at ~0.80× parameters on Llama-2-7B/13B and Llama-3-8B. The 13B comparison is against a published cross-harness number, not a controlled in-harness baseline. The 7B/8B improvements are modest (e.g., 6.415 → 6.217 perplexity and 59.44% → 61.19% accuracy for Llama-2-7B). Broader comparisons with SliceGPT, MoDeGPT, and other structured pruning methods, and larger models, would better support the generality of the approach.

  4. For BatchNorm models, the predictor must re-estimate candidate-specific running statistics, which requires constructing each candidate enough to collect statistics. This partially undermines the “compare without building” advantage. The post-build decoder is sequential and depends on the already-updated model; while the paper ablates pre- vs post-build, the computational overhead and sensitivity to calibration size/distribution are not fully quantified.

  5. Priced selection is not uniformly best: CCA is higher on ResNet18-Fixup in Table 8, and tuned ZipIt beats priced selection in split-task merging. The paired Jacobian objective improves accuracy on three architectures but does not reduce output deviation on any (Table 10). The paper is transparent about these results, but the practical recommendation—when to use which encoder and how much tuning is allowed—could be sharper.

  6. The main text treats feedforward networks with coordinatewise activations. Appendix G discusses convolutions, normalization, residual streams, attention, and gated FFNs, but the decomposition’s guarantees do not directly cover softmax attention or LayerNorm/RMSNorm. The Llama construction is a sensible engineering adaptation (RoPE-pair preservation, value-output refit, Wanda selection), but it is not fully derived from the decomposition. This limits the theoretical claim for transformer pruning.

Questions for authors
  1. Can the flip-mass diagnostic be used to automatically reject unreliable predictions? How often does the predictor select a clearly suboptimal encoder in the vision and Llama settings, and what is the downstream cost?

  2. Given Table 6, how much of the same-task merging gain is due to using both models’ features versus post-hoc decoder fitting? Do the split-task results change the paper’s main conclusion about merging?

  3. How sensitive are the post-build decoder and the predictor to calibration set size and distribution? For BatchNorm models, is candidate-specific recalibration necessary, or can a cheaper approximation retain the ranking ability?

1 comment

Thank you for your review of our paper and mentioning the weaknesses [W1]-[W6] and questions [Q1]-[Q3]

[W1, Q1]: We agree that the worst-case bound is too loose for practical prediction and that ReLU flips can affect the linearized estimate. We added an analysis of whether these prediction errors lead to meaningful accuracy losses by measuring the test-accuracy gap between the selected encoder and the best evaluated encoder. We find that the selected encoders generally perform close to the best candidate. However we also find that for Fixup models we can identify a failure case. The flip-mass diagnostic does not provide a clear indication, as its relationship with the prediction error varies across architectures. For details see Appendix C.2. For vision models, the selection uses calibration data throughout and the best test candidate is used only for diagnosis. For Llama, we use held-out training data to select the attention allocation (Appendices D.3–D.4).

[W2, Q2]: We agree that the same-task results do not establish that features from both networks are needed. Keeping one model’s hidden features and fitting only the final layer gives strong results and additionally fitting its hidden layers has little effect. We also evaluate specialists trained on different class subsets and here both weighted selection and decoder-refitted ZipIt outperform keeping one model and fitting all its layers, on every model pair. This shows a benefit of retaining representations from both models. We do not yet have a general criterion for predicting when both models are necessary. Appendix C.6 gives the results and protocol, including the final-interface-only decoder fit used for the merged models.

[W4, Q3]: We agree that for BatchNorm models the prediction is less computationally attractive because it requires candidate-specific recalibration. Using only the original models' statistics ranks candidates poorly in our experiments. A cheaper replacement that reliably preserves the rankings is an open problem. In Appendix C.4, we have added runtime and memory measurements. These quantify the additional cost of prediction-based selection over using weighted selection directly, which remains a competitive alternative without evaluating multiple candidates.

[W4, Q3]: We also added calibration-size and distribution experiments on VGG11-BN and ResNet18-BN. We vary the calibration budget for encoder statistics separately from that for decoder fitting and BatchNorm recalibration. The results show that too few samples for decoder-fitting and recalibration can substantially reduce accuracy, while encoder statistics are less sensitive. Class-imbalanced calibration also lowers accuracy for both selection rules, so the calibration distribution matters even when the sample count is unchanged. Larger calibration sets do not uniformly improve encoder selection. Appendix C.3 gives the accuracy, output-deviation and selection results, with variability across calibration draws and the recorded construction failure.

[W5]: While weighted selection is not uniformly best, it provides a consistently competitive choice across the evaluated architectures, without any accuracy-based encoder tuning. CCA performs better on Fixup in the same-task experiments, while tuned ZipIt performs better on split-dataset tasks. Rather than establish a universally optimal method, our experiments aim to show that the theory can be used to improve existing methods and explain why some methods perform better than others. We also agree that output deviation and task accuracy are different objectives, as the Jacobian-weighting ablation illustrates.

[W3]: FLAT-LLM compares against SliceGPT, SVD-LLM, FLAP and LLM-Pruner and reports better overall accuracy and perplexity at 20% compression. We therefore use it as a strong baseline and test whether changes derived from our theory can improve it further. Table 18 and Appendix D.4 additionally include direct comparisons with SliceGPT and MoDeGPT on Llama-2-7B using the same evaluation harness. In this setting our uniform-layer construction improves both perplexity and average accuracy over these baselines. The 13B FLAT-LLM comparison remains cross-harness, because the released paths failed for the rotary-attention implementation. A newer LLM family and a full compression-ratio sweep remain limitations of the current evaluation.

[W6]: Our added transformer merging results show how the theory can guide method design and improvements even where the complete architecture is not covered by our theory. Appendix G explains the architectural restrictions and the local adaptations used in these experiments.

Reviewer_kS1WOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper expands on the "generalized pruning" framework from [1]. [1] unifies structured pruning and model ensemble merging as applications of (linear) encoder and decoder kernels to every weight matrix in a neural network. Since the decoder and encoder can't perfectly reconstruct the original model weights, this procedure introduces noise to the weights and thus a deviation (error) on the network's output. This paper introduces a decomposition of the layerwise error and use this decomposition for further analysis and method improvements.

The authors appear to derive two main claims about their error decomposition: 1. the decomposition allows the derivation of a closed-form MSE-minimizing decoder for a given encoder (section 4.1) and 2. while the optimal encoder still requires data, the error decomposition only requires data to be passed through the large model and not to the smaller model.

The decomposition introduced in the paper is novel and appears interesting, and the optimal decoder has implications for practitioners and researchers alike. However, the paper has many weaknesses, both in its structure and in its experimental validation. I therefore recommend this paper be rejected

References (also for weaknesses): 1: Fabian Morelli and Stephan Eckstein (2026) "Partial Fusion of neural networks: Efficient trade-offs between ensembles and weight aggregation" 2: AdaRound 3: SlimGPT

Critical strengths

The decomposition introduced in sections 3, 4 and 5 is interesting and can potentially be useful in studying how error propagates through a network.

The individual steps in the decomposition itself well motivated and well derived, with much supporting material in the main text and appendices, making each step easy to follow, but see weaknesses.

Critical weaknesses
  1. While the meat of the paper is spent on deriving and experimentally validating the error decomposition, and the each step in the derivation is easy to follow, it is unclear what the bigger picture is, and specifically what implications we can derive from the decomposition. The derivation of the locally optimal decoder depends only on notation introduced in the beginning of section 3, and none of the error derivations introduced between the introduction of notation and the derivation of the optimal decoder.

  2. The paper is poorly structured, making the main arguments hard to follow and muddying the bigger picture. E.g., the locally optimal decoder introduced in section 4.1 is preceded by a detour about a "metric G that prices each direction by its downstream effect", to later conclude that the optimal decoder is independent of "pricing".

  3. It is unclear what the benefit is of the final error approximation in section 5., estimation, as section 6 shows the estimation is far from perfect, and it should be relatively cheap to evaluate different compression approaches by comparing layerwise error.

  4. Experimental validation of the model is weak, and scattered throughout the main paper and appendices, making it hard to understand what the central conclusions are.

  5. Much text is spent on validating the decomposition on obsolete networks (VGG11, ResNet18) The experimental setup in section 6 is not clear: What are the "three disjoint endpoints pairs" trained for each architecture and what compression method is used?

  6. In section 6.2, three obsolete LLM architectures are studied, but no recent (open source) LLMs are included in their experimental analysis.

  7. An interesting ablation on pruning % vs test accuracy is presented in appendix C.2 figure 3. It would strengthen the paper to present similar experimental analyses on modern LLMs in the main paper and focus experimental validation much more on the practical implications and validations of the error decomposition

  8. Overall, I would recommend the authors restructure and refocus their paper, providing more experimental validation of the main claims in the main paper, and using more recent networks

  9. A comparable analysis on the effect of local error and compounding error was introduced by AdaRound [2] and a similar derivation was later used in SparseGPT (which uses unstructured compression and can thus not directly compared) and SlimGPT (which uses structured compression).

Certain terms need explanation before introduction:

  • Line 212: "Priced second moment" and other uses of "priced"/"prices" etc (e.g. line 319)
  • Line 355 and elsewhere in section 6: "Endpoint pairs"

Other notes:

  • Line 367: type "wiht" should be "with"
Questions for authors
  1. In the introduction the authors claim that "the encoder and decoder can be independently fitted". Does this not follow directly from the encoder-decoder kernel formulation of generalized pruning?
  2. Would an extension of the "generalized pruning" framework to other compression methods (e.g. quantization, semi-structured (2:4) pruning, unstructured pruning, etc.)?
2 comments

Thank you for your review of our paper and for raising weaknesses [W1]–[W9] and questions [Q1] and [Q2].

[W4, W5, W8]: We use commonly used models for merging (see Git Re-Basin, OT Fusion, CCA Fusion) and pruning (see FLAT-LLM, SparseGPT) to compare different methods. Our goal is to understand how these methods differ and test whether our theory is well aligned with downstream performance. Our added transformer merging experiments also show that the theory can be applied to more modern architectures. The newly tested transformer models give a direct comparison to existing work on merging differently initialized transformers. We discuss the goals of our paper further in the overall response.

[W4, W6]: For LLM pruning, we use the evaluation protocols and models used by FLAT-LLM, which already compares against SliceGPT, SVD-LLM, FLAP and LLM-Pruner and reports better overall accuracy and perplexity in its 20% compression evaluation (Table 1). We therefore use it as a strong baseline and test whether changes derived from our theory can improve it further.

[W3]: Regarding the benefit of the output-deviation approximation, local errors can be amplified or suppressed by subsequent layers, and their contributions can reinforce or cancel each other. The approximation accounts for these downstream effects. Our added comparison shows that this improves encoder selection over local fitting scores on ResNet18-BN, while both rules perform equally well on VGG11-BN. We also find that, despite imperfect predictions, the selected encoders generally perform close to the best candidate. Appendix C.2 gives these results and the failure cases.

[W1, Q1]: We would also like to clarify the role of the decoder in our theory and address your question about the independence of encoder and decoder. As you correctly stated, the encoder-decoder formulation of generalized pruning already allows the decoder to be chosen separately from the encoder. It does not have to be the inverse or close to the inverse of the encoder, as is commonly assumed in merging methods (see e.g. Git Re-Basin, CCA Fusion). However, the formulation alone does not tell us how to choose the two kernels. Our decomposition shows that the encoder faces a trade-off between reconstruction and commutation error, while an unrestricted decoder can be fitted to minimize the local quadratic reconstruction objective. Theorem 7 gives this locally optimal decoder in closed form. The fitted decoder still depends on the encoder through the compressed model's features.

[W2]: Regarding the use of the metric G, for a fixed encoder and upstream features, the same unregularized fitted decoder minimizes the objective for every metric G, while the preferred encoder and regularized versions of the fitted decoder can be influenced by this metric (see Remark 1). In our experiments we see a clear positive effect of using a suitable metric. This is why we included the metric in the reconstruction loss, which includes both the encoder and the decoder. The theory thus tells us to use the importance metric for the encoder, but not necessarily for the unrestricted decoder.

[W9]: Thank you for the suggestions of literature we missed. We have updated our related-work section and summarize it here. AdaRound already studies local reconstruction and accounts for upstream quantization errors through its asymmetric reconstruction objective. SparseGPT reconstructs weights to compensate for pruning, and SlimGPT applies second-order reconstruction to structured pruning and addresses error accumulation. The novelty of our paper is not in the local reconstruction alone. Our analysis separates reconstruction and commutation error and transports their effects through depth, within the same formulation for pruning and merging. As with PLeaS for merging, our theory can help explain why these practically developed methods work well.

[Q2]: Regarding your second question, we think that this is an interesting direction for further research, especially the connection between pruning and quantization. Extending our decomposition to quantization would require accounting for errors that are not represented by the layer-wise linear encoder–decoder kernels considered here. We have added a related sentence to the conclusion.

We have also replaced “priced” with “weighted” and explain the weighting when first introduced. “Three disjoint model pairs” means six independently trained models, each used in one pair, which we now state explicitly.

I thank the authors for their response. Unfortunately, my concerns, especially with regards to the structure of the paper, have not been taken away and I will not change my rating

Reviewer_kg3gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper focuses on decomposing errors of the layer-wise pre-activation deviation between the full model and the compressed one, during pruning and merging trained networks within the generalized pruning framework of Morelli and Eckstein. Their error decomposition captures the reconstruction and commutation errors at each layer's pre-activation deviation; these feed into a recursive transport of the error that is essentially used to rank methods and select encoders (its linearization). The decomposition unlocks certain encoder and decoder choices that can be made principled. For example, the priced selection encoder falls out of Lemmas 4 and 5 nicely. The empirical results in the paper cover classical networks and transformers (pruning only), providing empirical evidence to justify the paper's theoretical framing, such as outperforming ZipIt and PLeaS across a half dozen architectures.

Critical strengths

The mathematical formulation and results in the paper seem to be clear and well within the contemporary generalized pruning framework in the literature. The decomposition and subsequent transport of the error lead to interesting results that enable principled design choices. For example, since it is shown that only the encoder needs to commute with the non-linearity, the decoder can be solved in closed form (some examples of this are given). Although these theoretical results do largely summarize and integrate numerous recent results, the resulting framework is compelling and uniquely integrates merging and pruning in one formalism. The output-deviation predictor seems like it will have practical impact on the space.

It is important to note that the empirical results, especially those on classical networks (MLP/VGG/ResNet) are mostly compelling with respect to the state of the art. Table 9 shows where ZipIt outperforms priced selection.

Critical weaknesses

Although the integrated error decomposition and transport framework provide certain insights, it is largely an integration of existing results into one formulation. This is ok, but important to note here. The authors do not overclaim this in any noticeable way. (This disclaimer from Appendix A probably should be moved into the main paper.) As said above, the resulting framework uniquely integrates pruning and merging into one formulation.

The lack of addressing merging on transformer models is a limitation of the work. It is an acknowledged structural limitation of the work (Appendix G), but it is a limitation.

(Minor) Non-technical but important point to make: The use of color in the figures 1 and 2 does help. But it is a poor way to distinguish pieces of a figure as color blind individuals will not be able to see the differences. The figures should choose an additional or alternative way to support the need to disinguish other parts of a figure.

Questions for authors

The paper's contribution is sound. Some clarity on the two weakpoints mentioned could be helpful. The paper is rather forthright in its limitations, which is commendable. But there are other aspects of the work (like the uselessness of Theorem 3's bound) that limit a stronger read.

2 comments

Thank you for your review of our paper. We agree with the importance of transparency regarding the novelty of our approach and have clarified the scope of our novelty claims in the conclusion.

Our new transformer merging experiments in the overall response address the open question about transformer merging.

Regarding Theorem 3, we agree that its bound is too loose for practical numerical prediction. The approximation in Section 5 is separate from this bound, and we have added an analysis of its accuracy gaps to the best candidate and failure cases in Appendix C.2.

We have also started updating our figures to make them more accessible to color-blind readers and plan to improve them further for the camera-ready version. Thank you for pointing this out.

Thanks for the comments. I read this and it seems to agree with my review, which differs a lot from the other two reviewers.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all the reviewers for their time and constructive criticism. We would like to address some general points about our empirical evaluation and the limitation regarding transformer merging, which came up in more than one review.

We initially focused on architectures where training-free merging methods were more established, as good results for merging transformers with different initializations have mostly been achieved in combination with fine-tuning or encoders and decoders learned using gradient descent (see Theus et al. (2025)). Motivated by the reviewers' comments, we have now evaluated transformer merging. We find that our original-width construction with fitted decoders substantially improves these weight-matching merges and that weighted selection further improves the ViT results. We have added these results to the paper and summarize them here.

Model / metricMean endpointWeight matching (Theus et al.)Weight matching + full LSWeighted selection + full LS
GPT-2, Tiny Shakespeare / perplexity ↓169.71 ± 2.10187.43 ± 4.24156.32 ± 2.14156.19 ± 1.74
ViT, CIFAR-10 / accuracy (%) ↑82.08 ± 0.3951.77 ± 3.5472.09 ± 1.3377.32 ± 0.17
ViT, CIFAR-100 / accuracy (%) ↑51.16 ± 2.3318.66 ± 6.8436.53 ± 4.1141.54 ± 3.74

Results are mean ± sample sd over three independently trained model pairs. Full LS denotes sequential post-build decoder fitting, without gradient-based fine-tuning.

With these and our other experiments, our primary goal is not to develop universally optimal methods, but to give researchers the theoretical foundations to understand existing methods and develop improved ones. We study merging and pruning methods in the generalized pruning framework to understand how they differ, what errors occur, how models can be aligned and how suitable sub-networks can be selected. Our experiments show how our findings help explain why methods such as PLeaS and FLAT-LLM work well in practice. With our adaptations of existing methods we demonstrate how the theory can be used to improve them.

We would also like to clarify that a merged model does not have to surpass the individual models at exactly the original model size for these improvements to be useful. Methods like Partial Fusion allow us to retain additional unmatched features and trade model size for accuracy. Merged models can also be further improved by fine-tuning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.