- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1(−1)
Read review
Paper summary
The paper explores the important topic of the performance cost of pruning and specifically the connection of degradation not captured in in-distribution accuracy with respect to the sharpness of the loss landscape and out of distribution performance.
The core constituents of the paper: Theoretical decomposition of masking and fine tuning steps in weight-space and curvature --- derived using linear paths between the dense-to-masked and masked-to-FT weight paths.
The experimental analysis of empirical sharpness (loss deviation in epsilon ball) on accuracy in distribution and under perturbative settings. Done in the image domain with cifar100/ tinyimagenet
The experimental accounting of the effect of different parameter groups --- according to the curvature contribution decomposition developed in the theoretical section, and in particular the saliency vs interaction scope partitioning to low saliency vs high saliency and low/high interaction as defined by two thresholds. Done in the low sparsity regime of 5% on cifar 100 (and some tinyimagenet experiments in appendix)
Finally the experimental extension to language models performs only masking and thus analyzes only the in distribution (wiki2) and out of distribution effect of pruning.
*** Update given rebuttal: *** I have updated the paper score to reject. The core concerns were not addressed, but rather exacerbated. And counterproductively, the additional evidence produced was done so with what seems to be little effort for synthesis and engaging with the core critiques with clarity. Most fundamentally, it does not seem that with proper baselines the main results survives (e.g. the pruned models anecdotally seem more robust, not less relative to added dense baseline), nor is it clear how this work advances our understanding or useful predictions in its current form.
Critical strengths
(a) Formulating a theoretical understanding of the connection between pruning and loss geometry, and in turn in-distribution and out of distribution performance as a function of the decomposed effects of masking and fine tuning in the pruning process is an important and still poorly understood topic. as such the effort in moving the field forward in this domain is important and insightful.
(b) Grounding the theoretical analysis in experimental evidence is crucial and insightful --- the results themselves are also insightful separately: while sharpness is well established showing it evolve with respect to sparsity is important, complementarily the curvature decomposition and FT recovery / offset is also a very useful view if sharpness plays a causal role in explaining out performance degradation not viewable by in-domain accuracy .
Critical weaknesses
(a) There are different components of the paper and it is simultaneously unnecessarily difficult to tease them out and more critically they are not sufficiently supported by the evidence. explicitly two primary ones are:
- Theoretical relationship between post training masked/FT pruning and geometry of loss and its curvature properties
- Relationship of parameter work-point and curvature (in this case recovery relative to dense) to performance and specifically in and out of distribution generalization as a measure of ~lost performance "beyond accuracy"
(b) The theory and experiments are not coherently supportive of each other.
-
The first experiment (tipping phase) shows an empirical increasing measure of sharpness with sparsity. But it does not show the corresponding out of distribution (lack of) degradation across pruning levels. nor does it tie it to the theoretical machinery developed Moreover, adversarial robustness (which is used as a measure of out of distribution generalization) is typically brittle even before pruning, unless explicitly trained for. So one need to compare at least two additional controls: robustness vs sharpness (which can be induced along pruning), and sharpness of adversarially-robust pruning ==> is the lost sparsity implied predicted by the theory?
-
The second experiment (parameter selection) has three limitations: (i) very limited dynamic range --- operating at the 5% sparsity and with little effect on accuracy. (ii) it does not establish convincingly the relationship with the theory --- for that to convincingly be demonstrated, predictive power should be established: quantitative theoretical prediction which can be evaluated of the form "predicted vs measured". (iii) it does not address a connection between the accuracy and the generalization gap central to the paper's main claim
-
the Curvature experiment also does not establish a relationship with theory and predictions. While it shows an offsetting curvature relationship post masking and FT it does not use the opportunity to tie this to decompose curvature vs non-curvature related effects on generalization. Is it the case that, as central to the paper question, generalization during pruning suffers beyond accuracy in a manner explained by the measured (and ideally theoretically predicted given the theoretical machinery developed) curvature?
-
The language modelling experiment does not engage with the theory --- which is OK if it still shared novel explanatory power or predictive power of the results.
-
The experiments lack representative data scale --- e.g. at the very least imagenet should be used in the vision domain. It is famously the case that cifar / tinyimagenet results , in the pruning domain in particular, do not readily carry over to imagenet.
-
The experiments also do not seem to use as baseline the gold standard of iterative pruning in the vision domain --- this seems a particularly important baseline as it maximizes accuracy retention versus sparsity. in Other words, if the geometric view proposed in this paper holds, one could conceptually show that the different methods of pruning (the IMP baseline missing vs the current implementation
Questions for authors
It would be very useful to see some predictive power from the theory to the experiments as outlines in full above.
While the topic is very important, at its current form, at least for this reviewer, I am left with interesting questions but little answers. The biggest one --- can one quantitatively answer "what is lost" and tie it to the theory developed. If it is robustness then at the very minimum we're missing the non-pruned and the adversarially robust trained baselines and their respective measures and predictions.
More fundamentally one would expect pruning to be at the limit optimal under some information theoretic definition --- what is the relationship of that to to the geometric view here. To give a counter example: if one has two extreme cases: one with partially overlapping domains (e.g. math and coding) and one with completely non overlapping domains. Can one give two predictions: (a) what would be the optimal curvature at maximal in-domain-lossless pruning? (b) What would the relationship between that and the performance in both domains be?
2 comments
Continuing our response.
For Weakness (b): loss beyond ID accuracy and its relationship to geometry, the added five-seed ResNet-50/CIFAR-100 evaluation finds a 3.90 percentage-point (pp) larger PGD-accuracy loss than in-distribution (ID) accuracy loss at 90% magnitude sparsity, relative to original dense: ID drops 0.52 pp and PGD 4.42 pp (paired 95% confidence interval (CI) for their difference: [1.95, 5.85]). Theorem 3.6 attributes curvature changes to pruning stages; our added joint measurements examine how those geometric changes accompany robustness losses beyond ID accuracy.
Submitted Section 4.2/Table 1 and Appendix C report sensitivity and robustness. We add maximum Hessian curvature on the same checkpoints and masks. ID/PGD use 1,000 fixed test images; PGD-20 has two restarts and radius , distinct from the submitted full-test PGD-10 protocol. Curvature uses 1,000 fixed training examples and compares sparse retained with dense full coordinates; sensitivity compares identical perturbable coordinates.
Table R2. Selected endpoints from the added joint sweep. Five-seed means; curvature is mean ± 95% CI half-width. Sparse states are fixed-mask fine-tuned models. Ratios use original dense.
| Method | Sparsity | ID (%) | PGD (%) | Corruption (%) | Sensitivity ratio | Curvature ratio |
|---|---|---|---|---|---|---|
| Dense | 0% | 76.80 | 17.08 | 48.85 | 1.00 | 1.00 |
| Magnitude | 20% | 74.16 | 13.98 | 46.48 | 1.25 | 3.70 ± 4.03 |
| Magnitude | 80% | 75.78 | 14.62 | 47.38 | 1.32 | 1.67 ± 1.10 |
| Magnitude | 90% | 76.28 | 12.66 | 46.94 | 1.37 | 1.87 ± 1.11 |
| Fisher | 20% | 73.94 | 14.02 | 45.86 | 1.23 | 4.06 ± 3.42 |
| Fisher | 80% | 75.92 | 13.88 | 47.15 | 1.31 | 3.14 ± 3.49 |
| Fisher | 90% | 74.76 | 12.02 | 46.03 | 1.37 | 4.61 ± 4.41 |
These endpoints combine higher mean sensitivity/curvature with lower PGD/corruption accuracy. Across the full sweep, however, geometry's correlation with robustness loss beyond ID loss is not consistently positive. Thus the measurements quantify the geometric changes accompanying the loss, while curvature alone does not explain its magnitude.
Optimizer intervention. In a separate five-seed comparison, SAM(AdamW) and AdamW start from identical 80%-magnitude masked checkpoints with equal fine-tuning epochs. SAM lowers curvature in every seed (mean paired ratio 0.584) and improves PGD by 1.98 pp (95% paired CI [1.18, 2.78]). This supports joint improvement under an optimizer intervention, which changes multiple properties of the solution and does not isolate curvature's causal role.
Matched non-pruned control. In a separate recovery experiment, dense receives the same additional clean-input training as the sparse models. PGD reaches 13.82% for dense, versus 14.50%/14.22% for magnitude/Fisher at 80% sparsity. Original-dense gaps therefore describe pruning plus adaptation; this matched comparison does not show an additional pruning penalty.
Adversarially robust reference. Five adversarially fine-tuned checkpoints start at 51.74% mean PGD accuracy. Matched additional training on clean inputs gives 28.26% without pruning and 32.84% with 80% magnitude pruning. Clean-loss curvature ratios to the robust reference are 1.22 ± 0.77 and 0.63 ± 0.64 (mean ± 95% CI half-width), respectively. Both arms lose robustness during clean-input adaptation, while the sparse arm has higher PGD accuracy and lower mean curvature than its matched dense control.
Iterative pruning. A separate five-seed comparison matches one-shot pruning plus five adaptation epochs against five magnitude-pruning rounds with one epoch each, without rewinding. At 90%, one-shot achieves 75.70% ID / 13.14% PGD, versus 74.84% ID / 11.54% PGD for iterative pruning; curvature ratios are 2.03/2.85. For the iterative schedule, PGD loss exceeds ID loss by 3.58 pp (paired 95% CI [1.26, 5.90]). This equal-budget control shows that the beyond-ID robustness gap also occurs under the tested iterative schedule.
Vision evaluation covers ResNet-50 and ViT-B/16 on CIFAR-100/Tiny-ImageNet. ImageNet-scale validation remains outside the present experiments.
Revision locations. We define the SAM-based directional estimate in Section 4.2 (pp. 6–7) and Appendix B.2 (p. 20). We report the joint sweep and four controls with 95% CIs in Appendix C.2 (Tables 12–16, pp. 26–27), with protocols in Appendix B.1 (p. 20). We evaluate curvature stages in Section 4.4 (pp. 8–9).
We welcome further feedback.
Thank you for emphasizing the theory–experiment connection. The submission analyzes immediate gradient changes and curvature changes across masking and fine-tuning. We add quantitative prediction tests and joint geometry/robustness measurements, together with optimizer, matched-training, adversarial-reference, and iterative-pruning controls.
Revision locations use the revised PDF; other manuscript numbers refer to the submission.
For Weakness (a): what the theory explains and predicts, the existing theory provides a dense-Hessian prediction of immediate retained-gradient changes, with explicit residual bounds. The curvature decomposition separately identifies the contributions of subspace restriction, masking, and fine-tuning.
Theorem 3.4 provides the dense-Hessian gradient predictor and bounds its error. Proposition 3.5 gives conditional group-specific response bounds, with mixed masks covered in Appendix A.5. Theorem 3.6 separates subspace restriction, masking, and fine-tuning effects on maximum retained-space curvature and bounds the stagewise shifts.
Quantitative prediction. From the dense model W0 and proposed mask M, we compute the prediction GM and compare it with the observed response GM after masking:
GM=−PMH0qM,GM=PM[g(WM)−g(W0)].
Here H0 is the dense loss Hessian, g is the loss gradient, qM contains removed weights, WM is the masked model, and PM selects retained coordinates. We compare vector norms, cosine alignment, and relative error ∥GM−GM∥2/∥GM∥2. Zero error is exact prediction; one is the error of predicting zero.
Submitted Section 4.3/Table 2 contains the 5% prediction comparison, and Appendix D/Table 9 contains observed 10% responses. We add the 10% predictions and vector-error diagnostics, plus practical magnitude/Fisher masks at 20/50/80% sparsity.
Table R1. Submitted 5% and added 10% prediction diagnostics, ResNet-50/CIFAR-100. Five-seed means. Low/high denotes the empirical-Fisher interaction group; responses are measured on retained coordinates.
At 10%, the five-seed mean predicted and observed response norms are both larger for the high-interaction group than for the low-interaction group. The large vector errors show that the leading term captures the group ordering better than response magnitude in this test. Masks match per-layer removal counts, but their removed-weight norms differ, so this comparison does not isolate interaction alone.
For practical masks across three checkpoints, mean cosine/relative error is 0.663/0.778 for magnitude at 50% and 0.027/1.000 for Fisher at 80%. Submitted ViT-B/16 high-interaction cosine is 0.020 at 5%. Together, these diagnostics locate informative and inaccurate approximations across masks and architectures; they assess prediction accuracy, not the tightness of the full path-dependent error bound.
Stagewise curvature. Submitted Section 4.4/Appendix E.2 shows that, for ResNet-50/Fisher, masking changes curvature little at 20% sparsity and fine-tuning raises it on average. At 80%, masking produces a large increase and fine-tuning reverses most of it. Applying the decomposition identifies whether the final increase arises mainly during masking or adaptation.
Language/code models. Submitted Appendix G.1/Table 16 tests the Qwen2-0.5B Wanda gradient predictor, with the protocol in Appendix B.5. New Qwen2.5-Coder-0.5B tests give cosine/relative error 0.805/0.715 at 20% and 0.474/1.637 at 40%, using one checkpoint and calibration draw. HumanEval separately measures downstream capability.
Optimal sparsity and domain overlap. Our theory analyzes a given mask and adaptation; it does not determine maximal in-domain-lossless sparsity or the curvature at that optimum. For two domains, their losses provide a concrete extension: evaluate each domain's local Hessian response under the same mask and regularity assumptions. Predicting optimal sparsity and performance additionally requires preservation criteria and assumptions about shared parameter dependence, which domain overlap alone does not specify.
Revision locations. We report added predictions and CIs in Appendix D.2 (Tables 19–21, pp. 28–29). We evaluate curvature stages in Section 4.4 (pp. 8–9), with the full sweep in Appendix E.2 (Table 24, p. 31). Coder predictions appear in Appendix F.2 (Table 28, p. 32), with HVP protocols in Appendix B.5 (pp. 22–23). We clarify optimal-sparsity and domain-specific scope in Section 5 (p. 10).