- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies generalized pruning, a framework that unifies structured pruning and model merging as layer-wise encoder–decoder compression of a larger model/ensemble. The main contribution is an error decomposition that separates the layer-wise deviation into three terms: reconstruction error (information lost by the encoder–decoder projection), commutation error (failure of the encoder to commute with the nonlinearity), and propagation error (carrying previous-layer deviation forward). The paper proves an exact recursion and an output-deviation bound, then uses the decomposition to derive practical tools: a generalized PCA/“priced” selection encoder, a closed-form locally optimal least-squares decoder, and a calibration-based predictor of output deviation via linearized error propagation. Empirically, the paper evaluates on vision merging/pruning tasks (MLPs, VGG11, ResNet18, with/without BatchNorm or Fixup) and on structured Llama pruning as adjustments to FLAT-LLM. The decoder yields large gains over native decoders, priced selection is competitive, and the predictor ranks methods well on non-normalized/Fixup models but less reliably on BatchNorm/residual models and under extreme compression. The Llama experiments show modest but consistent improvements over FLAT-LLM at ~0.80× parameters.
Critical strengths
-
The separation into reconstruction, commutation, and propagation error is exact and conceptually illuminating. Lemma 1 and Theorem 2 give a clean local identity; Theorem 3 unrolls it into an output bound. The paper is careful to note that individual terms have appeared before and positions the contribution as the exact separation plus its exploitation.
-
The generalized PCA result (Lemma 4) and the commutation characterization (Lemma 5) explain the trade-off between flexible dense kernels and restricted selection kernels. The closed-form decoder in Theorem 7 is elegant and empirically effective: fitting the decoder substantially improves structured pruning at 50% unit pruning (e.g., VGG11 from 49.08% to 86.48%, ResNet18 from 1.00% to 64.95%).
-
The calibration-based estimator in Section 5 can compare candidate encoders from cached activations of the large model without constructing each compressed model. Table 1 shows perfect rank correlation for MLP/VGG11/Fixup settings and useful correlations for BatchNorm models when candidate-specific statistics are used.
-
The paper evaluates six merging/pruning families, multiple architectures, and controlled ablations (pre- vs post-build decoder, common decoder across encoders, importance-priced selection, split-task merging). The Llama experiments, while narrower, include controlled adjustments and ablations.
-
The paper provides detailed experimental settings, calibration protocols, and states that code, configurations, seeds, and raw results are released. The AI-use and reproducibility statements are transparent.
Critical weaknesses
-
The paper acknowledges that Theorem 3’s bound can overshoot by orders of magnitude. The useful estimator is Eq. 15, which linearizes ReLU propagation and can fail when sign flips occur. The flip-mass diagnostic and exact fallback help, but the predictor’s reliability on BatchNorm/residual models is limited (Table 1: ResNet18-BN rank correlation 0.58, predicted/measured ratio 1.28). The paper notes failure under extreme compression (e.g., VGG11 at 90% pruning). A clearer characterization of when the predictor is trustworthy would strengthen the contribution.
-
Table 6 shows that keeping model A and fitting only the readout or all layers changes mean accuracy by at most 0.11 points on same-task pairs. This suggests that, in these same-task settings, much of the post-refit accuracy may not require features from both models. The split-task results in Appendix C.3 are more favorable and show gains over one-model controls, but the main text’s same-task framing weakens the claim that the decomposition explains substantial merging benefits. The paper should clarify when merging genuinely requires both models and how the decomposition predicts that.
-
The Llama results are limited to FLAT-LLM-style structured pruning at ~0.80× parameters on Llama-2-7B/13B and Llama-3-8B. The 13B comparison is against a published cross-harness number, not a controlled in-harness baseline. The 7B/8B improvements are modest (e.g., 6.415 → 6.217 perplexity and 59.44% → 61.19% accuracy for Llama-2-7B). Broader comparisons with SliceGPT, MoDeGPT, and other structured pruning methods, and larger models, would better support the generality of the approach.
-
For BatchNorm models, the predictor must re-estimate candidate-specific running statistics, which requires constructing each candidate enough to collect statistics. This partially undermines the “compare without building” advantage. The post-build decoder is sequential and depends on the already-updated model; while the paper ablates pre- vs post-build, the computational overhead and sensitivity to calibration size/distribution are not fully quantified.
-
Priced selection is not uniformly best: CCA is higher on ResNet18-Fixup in Table 8, and tuned ZipIt beats priced selection in split-task merging. The paired Jacobian objective improves accuracy on three architectures but does not reduce output deviation on any (Table 10). The paper is transparent about these results, but the practical recommendation—when to use which encoder and how much tuning is allowed—could be sharper.
-
The main text treats feedforward networks with coordinatewise activations. Appendix G discusses convolutions, normalization, residual streams, attention, and gated FFNs, but the decomposition’s guarantees do not directly cover softmax attention or LayerNorm/RMSNorm. The Llama construction is a sensible engineering adaptation (RoPE-pair preservation, value-output refit, Wanda selection), but it is not fully derived from the decomposition. This limits the theoretical claim for transformer pruning.
Questions for authors
-
Can the flip-mass diagnostic be used to automatically reject unreliable predictions? How often does the predictor select a clearly suboptimal encoder in the vision and Llama settings, and what is the downstream cost?
-
Given Table 6, how much of the same-task merging gain is due to using both models’ features versus post-hoc decoder fitting? Do the split-task results change the paper’s main conclusion about merging?
-
How sensitive are the post-build decoder and the predictor to calibration set size and distribution? For BatchNorm models, is candidate-specific recalibration necessary, or can a cheaper approximation retain the ranking ability?
Thank you for your review of our paper and mentioning the weaknesses [W1]-[W6] and questions [Q1]-[Q3]
[W1, Q1]: We agree that the worst-case bound is too loose for practical prediction and that ReLU flips can affect the linearized estimate. We added an analysis of whether these prediction errors lead to meaningful accuracy losses by measuring the test-accuracy gap between the selected encoder and the best evaluated encoder. We find that the selected encoders generally perform close to the best candidate. However we also find that for Fixup models we can identify a failure case. The flip-mass diagnostic does not provide a clear indication, as its relationship with the prediction error varies across architectures. For details see Appendix C.2. For vision models, the selection uses calibration data throughout and the best test candidate is used only for diagnosis. For Llama, we use held-out training data to select the attention allocation (Appendices D.3–D.4).
[W2, Q2]: We agree that the same-task results do not establish that features from both networks are needed. Keeping one model’s hidden features and fitting only the final layer gives strong results and additionally fitting its hidden layers has little effect. We also evaluate specialists trained on different class subsets and here both weighted selection and decoder-refitted ZipIt outperform keeping one model and fitting all its layers, on every model pair. This shows a benefit of retaining representations from both models. We do not yet have a general criterion for predicting when both models are necessary. Appendix C.6 gives the results and protocol, including the final-interface-only decoder fit used for the merged models.
[W4, Q3]: We agree that for BatchNorm models the prediction is less computationally attractive because it requires candidate-specific recalibration. Using only the original models' statistics ranks candidates poorly in our experiments. A cheaper replacement that reliably preserves the rankings is an open problem. In Appendix C.4, we have added runtime and memory measurements. These quantify the additional cost of prediction-based selection over using weighted selection directly, which remains a competitive alternative without evaluating multiple candidates.
[W4, Q3]: We also added calibration-size and distribution experiments on VGG11-BN and ResNet18-BN. We vary the calibration budget for encoder statistics separately from that for decoder fitting and BatchNorm recalibration. The results show that too few samples for decoder-fitting and recalibration can substantially reduce accuracy, while encoder statistics are less sensitive. Class-imbalanced calibration also lowers accuracy for both selection rules, so the calibration distribution matters even when the sample count is unchanged. Larger calibration sets do not uniformly improve encoder selection. Appendix C.3 gives the accuracy, output-deviation and selection results, with variability across calibration draws and the recorded construction failure.
[W5]: While weighted selection is not uniformly best, it provides a consistently competitive choice across the evaluated architectures, without any accuracy-based encoder tuning. CCA performs better on Fixup in the same-task experiments, while tuned ZipIt performs better on split-dataset tasks. Rather than establish a universally optimal method, our experiments aim to show that the theory can be used to improve existing methods and explain why some methods perform better than others. We also agree that output deviation and task accuracy are different objectives, as the Jacobian-weighting ablation illustrates.
[W3]: FLAT-LLM compares against SliceGPT, SVD-LLM, FLAP and LLM-Pruner and reports better overall accuracy and perplexity at 20% compression. We therefore use it as a strong baseline and test whether changes derived from our theory can improve it further. Table 18 and Appendix D.4 additionally include direct comparisons with SliceGPT and MoDeGPT on Llama-2-7B using the same evaluation harness. In this setting our uniform-layer construction improves both perplexity and average accuracy over these baselines. The 13B FLAT-LLM comparison remains cross-harness, because the released paths failed for the rotary-attention implementation. A newer LLM family and a full compression-ratio sweep remain limitations of the current evaluation.
[W6]: Our added transformer merging results show how the theory can guide method design and improvements even where the complete architecture is not covered by our theory. Appendix G explains the architectural restrictions and the local adaptations used in these experiments.