acceptodds
Under review as a conference paper at ICLR 2027

What We Lose in Pruning: A Generalization Perspective on Model Pruning

Abstract

Model pruning, which masks a fraction of model weights, primarily aims to preserve in-distribution accuracy; however, its hidden costs remain less understood. We ask: *Beyond accuracy, what do we lose when we prune a network?* We argue that pruning significantly reshapes the loss geometry, thereby damaging generalization and making the model more sensitive to data disturbances despite high clean accuracy. To theoretically investigate these effects, we analyze the consequences across two sequential stages: parameter masking and subsequent fine-tuning. First, immediately after masking, we characterize how the gradients of the remaining weights change through Hessian interactions. By grouping weights based on individual importance and interaction strength, we derive group-specific bounds on these gradient shifts. Second, after fine-tuning, we analyze sharpness via the maximum Hessian curvature with respect to the remaining weights. We decompose the change in sharpness relative to the original model into geometric shifts caused by masking and fine-tuning, showing how the final model becomes sharper. Through experiments on vision models and large language models across image classification, language modeling, commonsense reasoning, and code generation, we demonstrate that pruning incurs hidden costs in parameter sensitivity, sharpness, and robustness—costs that in-distribution performance fails to capture.

Primary area:
learning theory
Keywords:
Pruning, Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_g8oPOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

The paper explores the important topic of the performance cost of pruning and specifically the connection of degradation not captured in in-distribution accuracy with respect to the sharpness of the loss landscape and out of distribution performance.

The core constituents of the paper: Theoretical decomposition of masking and fine tuning steps in weight-space and curvature --- derived using linear paths between the dense-to-masked and masked-to-FT weight paths.

The experimental analysis of empirical sharpness (loss deviation in epsilon ball) on accuracy in distribution and under perturbative settings. Done in the image domain with cifar100/ tinyimagenet

The experimental accounting of the effect of different parameter groups --- according to the curvature contribution decomposition developed in the theoretical section, and in particular the saliency vs interaction scope partitioning to low saliency vs high saliency and low/high interaction as defined by two thresholds. Done in the low sparsity regime of 5% on cifar 100 (and some tinyimagenet experiments in appendix)

Finally the experimental extension to language models performs only masking and thus analyzes only the in distribution (wiki2) and out of distribution effect of pruning.

*** Update given rebuttal: *** I have updated the paper score to reject. The core concerns were not addressed, but rather exacerbated. And counterproductively, the additional evidence produced was done so with what seems to be little effort for synthesis and engaging with the core critiques with clarity. Most fundamentally, it does not seem that with proper baselines the main results survives (e.g. the pruned models anecdotally seem more robust, not less relative to added dense baseline), nor is it clear how this work advances our understanding or useful predictions in its current form.

Critical strengths

(a) Formulating a theoretical understanding of the connection between pruning and loss geometry, and in turn in-distribution and out of distribution performance as a function of the decomposed effects of masking and fine tuning in the pruning process is an important and still poorly understood topic. as such the effort in moving the field forward in this domain is important and insightful.

(b) Grounding the theoretical analysis in experimental evidence is crucial and insightful --- the results themselves are also insightful separately: while sharpness is well established showing it evolve with respect to sparsity is important, complementarily the curvature decomposition and FT recovery / offset is also a very useful view if sharpness plays a causal role in explaining out performance degradation not viewable by in-domain accuracy .

Critical weaknesses

(a) There are different components of the paper and it is simultaneously unnecessarily difficult to tease them out and more critically they are not sufficiently supported by the evidence. explicitly two primary ones are:

  • Theoretical relationship between post training masked/FT pruning and geometry of loss and its curvature properties
  • Relationship of parameter work-point and curvature (in this case recovery relative to dense) to performance and specifically in and out of distribution generalization as a measure of ~lost performance "beyond accuracy"

(b) The theory and experiments are not coherently supportive of each other.

  • The first experiment (tipping phase) shows an empirical increasing measure of sharpness with sparsity. But it does not show the corresponding out of distribution (lack of) degradation across pruning levels. nor does it tie it to the theoretical machinery developed Moreover, adversarial robustness (which is used as a measure of out of distribution generalization) is typically brittle even before pruning, unless explicitly trained for. So one need to compare at least two additional controls: robustness vs sharpness (which can be induced along pruning), and sharpness of adversarially-robust pruning ==> is the lost sparsity implied predicted by the theory?

  • The second experiment (parameter selection) has three limitations: (i) very limited dynamic range --- operating at the 5% sparsity and with little effect on accuracy. (ii) it does not establish convincingly the relationship with the theory --- for that to convincingly be demonstrated, predictive power should be established: quantitative theoretical prediction which can be evaluated of the form "predicted vs measured". (iii) it does not address a connection between the accuracy and the generalization gap central to the paper's main claim

  • the Curvature experiment also does not establish a relationship with theory and predictions. While it shows an offsetting curvature relationship post masking and FT it does not use the opportunity to tie this to decompose curvature vs non-curvature related effects on generalization. Is it the case that, as central to the paper question, generalization during pruning suffers beyond accuracy in a manner explained by the measured (and ideally theoretically predicted given the theoretical machinery developed) curvature?

  • The language modelling experiment does not engage with the theory --- which is OK if it still shared novel explanatory power or predictive power of the results.

  • The experiments lack representative data scale --- e.g. at the very least imagenet should be used in the vision domain. It is famously the case that cifar / tinyimagenet results , in the pruning domain in particular, do not readily carry over to imagenet.

  • The experiments also do not seem to use as baseline the gold standard of iterative pruning in the vision domain --- this seems a particularly important baseline as it maximizes accuracy retention versus sparsity. in Other words, if the geometric view proposed in this paper holds, one could conceptually show that the different methods of pruning (the IMP baseline missing vs the current implementation

Questions for authors

It would be very useful to see some predictive power from the theory to the experiments as outlines in full above.

While the topic is very important, at its current form, at least for this reviewer, I am left with interesting questions but little answers. The biggest one --- can one quantitatively answer "what is lost" and tie it to the theory developed. If it is robustness then at the very minimum we're missing the non-pruned and the adversarially robust trained baselines and their respective measures and predictions.

More fundamentally one would expect pruning to be at the limit optimal under some information theoretic definition --- what is the relationship of that to to the geometric view here. To give a counter example: if one has two extreme cases: one with partially overlapping domains (e.g. math and coding) and one with completely non overlapping domains. Can one give two predictions: (a) what would be the optimal curvature at maximal in-domain-lossless pruning? (b) What would the relationship between that and the performance in both domains be?

2 comments

Thank you for emphasizing the theory–experiment connection. The submission analyzes immediate gradient changes and curvature changes across masking and fine-tuning. We add quantitative prediction tests and joint geometry/robustness measurements, together with optimizer, matched-training, adversarial-reference, and iterative-pruning controls.

Revision locations use the revised PDF; other manuscript numbers refer to the submission.

For Weakness (a): what the theory explains and predicts, the existing theory provides a dense-Hessian prediction of immediate retained-gradient changes, with explicit residual bounds. The curvature decomposition separately identifies the contributions of subspace restriction, masking, and fine-tuning.

Theorem 3.4 provides the dense-Hessian gradient predictor and bounds its error. Proposition 3.5 gives conditional group-specific response bounds, with mixed masks covered in Appendix A.5. Theorem 3.6 separates subspace restriction, masking, and fine-tuning effects on maximum retained-space curvature and bounds the stagewise shifts.

Quantitative prediction. From the dense model W0W_0 and proposed mask MM, we compute the prediction G^M\widehat G_M and compare it with the observed response GMG_M after masking:

G^M=−PMH0qM,GM=PM[g(WM)−g(W0)].\widehat G_M=-P_MH_0q_M,\qquad G_M=P_M[g(W_M)-g(W_0)].

Here H0H_0 is the dense loss Hessian, gg is the loss gradient, qMq_M contains removed weights, WMW_M is the masked model, and PMP_M selects retained coordinates. We compare vector norms, cosine alignment, and relative error ∥GM−G^M∥2/∥GM∥2\|G_M-\widehat G_M\|_2/\|G_M\|_2. Zero error is exact prediction; one is the error of predicting zero.

Submitted Section 4.3/Table 2 contains the 5% prediction comparison, and Appendix D/Table 9 contains observed 10% responses. We add the 10% predictions and vector-error diagnostics, plus practical magnitude/Fisher masks at 20/50/80% sparsity.

Table R1. Submitted 5% and added 10% prediction diagnostics, ResNet-50/CIFAR-100. Five-seed means. Low/high denotes the empirical-Fisher interaction group; responses are measured on retained coordinates.

SparsityGroupPredicted normObserved normCosineRelative error
5%Low interaction0.0580.0690.7910.653
5%High interaction0.1640.2590.7370.803
10%Low interaction0.1650.1940.7180.823
10%High interaction0.6199.0520.4260.979

At 10%, the five-seed mean predicted and observed response norms are both larger for the high-interaction group than for the low-interaction group. The large vector errors show that the leading term captures the group ordering better than response magnitude in this test. Masks match per-layer removal counts, but their removed-weight norms differ, so this comparison does not isolate interaction alone.

For practical masks across three checkpoints, mean cosine/relative error is 0.663/0.778 for magnitude at 50% and 0.027/1.000 for Fisher at 80%. Submitted ViT-B/16 high-interaction cosine is 0.020 at 5%. Together, these diagnostics locate informative and inaccurate approximations across masks and architectures; they assess prediction accuracy, not the tightness of the full path-dependent error bound.

Stagewise curvature. Submitted Section 4.4/Appendix E.2 shows that, for ResNet-50/Fisher, masking changes curvature little at 20% sparsity and fine-tuning raises it on average. At 80%, masking produces a large increase and fine-tuning reverses most of it. Applying the decomposition identifies whether the final increase arises mainly during masking or adaptation.

Language/code models. Submitted Appendix G.1/Table 16 tests the Qwen2-0.5B Wanda gradient predictor, with the protocol in Appendix B.5. New Qwen2.5-Coder-0.5B tests give cosine/relative error 0.805/0.715 at 20% and 0.474/1.637 at 40%, using one checkpoint and calibration draw. HumanEval separately measures downstream capability.

Optimal sparsity and domain overlap. Our theory analyzes a given mask and adaptation; it does not determine maximal in-domain-lossless sparsity or the curvature at that optimum. For two domains, their losses provide a concrete extension: evaluate each domain's local Hessian response under the same mask and regularity assumptions. Predicting optimal sparsity and performance additionally requires preservation criteria and assumptions about shared parameter dependence, which domain overlap alone does not specify.

Revision locations. We report added predictions and CIs in Appendix D.2 (Tables 19–21, pp. 28–29). We evaluate curvature stages in Section 4.4 (pp. 8–9), with the full sweep in Appendix E.2 (Table 24, p. 31). Coder predictions appear in Appendix F.2 (Table 28, p. 32), with HVP protocols in Appendix B.5 (pp. 22–23). We clarify optimal-sparsity and domain-specific scope in Section 5 (p. 10).

Continuing our response.

For Weakness (b): loss beyond ID accuracy and its relationship to geometry, the added five-seed ResNet-50/CIFAR-100 evaluation finds a 3.90 percentage-point (pp) larger PGD-accuracy loss than in-distribution (ID) accuracy loss at 90% magnitude sparsity, relative to original dense: ID drops 0.52 pp and PGD 4.42 pp (paired 95% confidence interval (CI) for their difference: [1.95, 5.85]). Theorem 3.6 attributes curvature changes to pruning stages; our added joint measurements examine how those geometric changes accompany robustness losses beyond ID accuracy.

Submitted Section 4.2/Table 1 and Appendix C report sensitivity and robustness. We add maximum Hessian curvature on the same checkpoints and masks. ID/PGD use 1,000 fixed test images; PGD-20 has two restarts and ℓ∞\ell_\infty radius 1/2551/255, distinct from the submitted full-test PGD-10 protocol. Curvature uses 1,000 fixed training examples and compares sparse retained with dense full coordinates; sensitivity compares identical perturbable coordinates.

Table R2. Selected endpoints from the added joint sweep. Five-seed means; curvature is mean ± 95% CI half-width. Sparse states are fixed-mask fine-tuned models. Ratios use original dense.

MethodSparsityID (%)PGD (%)Corruption (%)Sensitivity ratioCurvature ratio
Dense0%76.8017.0848.851.001.00
Magnitude20%74.1613.9846.481.253.70 ± 4.03
Magnitude80%75.7814.6247.381.321.67 ± 1.10
Magnitude90%76.2812.6646.941.371.87 ± 1.11
Fisher20%73.9414.0245.861.234.06 ± 3.42
Fisher80%75.9213.8847.151.313.14 ± 3.49
Fisher90%74.7612.0246.031.374.61 ± 4.41

These endpoints combine higher mean sensitivity/curvature with lower PGD/corruption accuracy. Across the full sweep, however, geometry's correlation with robustness loss beyond ID loss is not consistently positive. Thus the measurements quantify the geometric changes accompanying the loss, while curvature alone does not explain its magnitude.

Optimizer intervention. In a separate five-seed comparison, SAM(AdamW) and AdamW start from identical 80%-magnitude masked checkpoints with equal fine-tuning epochs. SAM lowers curvature in every seed (mean paired ratio 0.584) and improves PGD by 1.98 pp (95% paired CI [1.18, 2.78]). This supports joint improvement under an optimizer intervention, which changes multiple properties of the solution and does not isolate curvature's causal role.

Matched non-pruned control. In a separate recovery experiment, dense receives the same additional clean-input training as the sparse models. PGD reaches 13.82% for dense, versus 14.50%/14.22% for magnitude/Fisher at 80% sparsity. Original-dense gaps therefore describe pruning plus adaptation; this matched comparison does not show an additional pruning penalty.

Adversarially robust reference. Five adversarially fine-tuned checkpoints start at 51.74% mean PGD accuracy. Matched additional training on clean inputs gives 28.26% without pruning and 32.84% with 80% magnitude pruning. Clean-loss curvature ratios to the robust reference are 1.22 ± 0.77 and 0.63 ± 0.64 (mean ± 95% CI half-width), respectively. Both arms lose robustness during clean-input adaptation, while the sparse arm has higher PGD accuracy and lower mean curvature than its matched dense control.

Iterative pruning. A separate five-seed comparison matches one-shot pruning plus five adaptation epochs against five magnitude-pruning rounds with one epoch each, without rewinding. At 90%, one-shot achieves 75.70% ID / 13.14% PGD, versus 74.84% ID / 11.54% PGD for iterative pruning; curvature ratios are 2.03/2.85. For the iterative schedule, PGD loss exceeds ID loss by 3.58 pp (paired 95% CI [1.26, 5.90]). This equal-budget control shows that the beyond-ID robustness gap also occurs under the tested iterative schedule.

Vision evaluation covers ResNet-50 and ViT-B/16 on CIFAR-100/Tiny-ImageNet. ImageNet-scale validation remains outside the present experiments.

Revision locations. We define the SAM-based directional estimate in Section 4.2 (pp. 6–7) and Appendix B.2 (p. 20). We report the joint sweep and four controls with 95% CIs in Appendix C.2 (Tables 12–16, pp. 26–27), with protocols in Appendix B.1 (p. 20). We evaluate curvature stages in Section 4.4 (pp. 8–9).

We welcome further feedback.

Reviewer_3DpYOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is trying to understand the effect of pruning beyond the classically used classification performance or downstream performance to the robustness and the overall geometry of the loss landscape. The authors provide both insights in the vision space but also the language models.

Critical strengths
  • The considered problem and the corresponding research questions are very relevant and important.
  • The multitude of the models considered in the experimental results is illustrative of the effect and its potential extension to different families.
  • The overall process of theoretical analysis and then deriving experimental and empirical validation is very interesting and elegant.
Critical weaknesses
  • I am not sure about the worth of the theoretical study as it seems that both Theorem 3.4 and Theorem 3.6 are just decomposition of the immediate gradient response into a hessian-based leading term and remainder terms and a simple decomposition of the curvature change into subspace restriction, masking and fine-tuning effect. Maybe I am missing here the worth of the study, but from my understanding the theoretical study does not necessarily establish the paper’s central empirical observation that pruning produces sharper final models. Would be great if the authors can provide elements to clarify this.

  • I am also missing the link between the sharpness and adversarial robustness. While clearly there exists some correlation (one increases while the other descreases), but there was no established link on the potential causal link between the increased sharpness and if it causes the robustness to degrade.

  • There seems to also be a difference between the theoretical study and the emprical study on the sharpness measure. Specifically, the theory studies the maximum Hessian curvature in the retained parameter subspace while in terms of implementation the sensitivity measurement is using a single normalized gradient direction perturbation. From my understanding and perspective, this is rather a gradient-based sensitivity estimate rather than exact maximum over the perturbation ball. Can the authors clarify this please?

  • The pruning experiments considers only a limited set of methods, and specifically the magnitude and Fisher pruning for CV while LLm experiments use Wanda. More advanced pruning and newer one maybe differs in the overall message and key results?

Questions for authors

Please first refer to my previous weakness section to clarify some elements that I didn’t understand and it would be great if you could clarify those.

In addition, I would be interested in knowing more about your perspective on these questions:

  • Can the authors derive some theoretical study to connect how pruning increases the final curvature? As I see this as the main point to have in the theoretical study.
  • Can you clarify the causal relationship and maybe provide additional evidence on the increased sharpness and its contribution to robustness degradation?
  • Can you please try to provide elements demonstrating analogous increases in Hessian curvature or parameter sensitivity in the pruned language models?
  • Can you also provide elements using newer and more advanced pruning methods?
3 comments

Thank you for the constructive feedback. The submission provides a stagewise analysis of pruning-induced gradient and curvature changes. We clarify its quantitative implications and add optimizer and matched-training controls, direct LLM geometry measurements, and SparseGPT experiments.

Revision locations use the revised PDF; other manuscript numbers refer to the submission.

For Weakness 1 and Question 1: theoretical explanation of final curvature, the existing Theorem 3.6 yields a sufficient condition for a masking-induced curvature increase to persist after fine-tuning. The decomposition alone does not imply that every mask sharpens the final model; its stagewise bounds characterize when an increase can survive adaptation.

Theorem 3.4 predicts the immediate retained-gradient change from the trained dense model and proposed mask. Its error bound accounts for Hessian variation and gradient discontinuities along the masking path. Proposition 3.5 relates response bounds to saliency, influence on retained gradients, and cancellation.

Theorem 3.6 separates two effects: restricting perturbations to retained coordinates at unchanged dense weights can only reduce maximum curvature, while changing the weights through masking and adaptation can shift it in either direction. It bounds the magnitude of these stagewise shifts. This identifies which changes must overcome the subspace reduction for final sharpening to occur.

A persistence condition follows from its fine-tuning bound. Under Assumption 3.3, define the theorem's upper bound on the magnitude of the fine-tuning curvature change as

BFT:=LHrFT(M)+AH(χFT).B_{\mathrm{FT}} := L_H r_{\mathrm{FT}}(M) + A_H(\chi_{\mathrm{FT}}).

These terms control smooth Hessian variation and Hessian jumps along the straight-line path from masked to fine-tuned weights. Here, δFT:=ΛM(WFT)−ΛM(WM)\delta_{\mathrm{FT}}:=\Lambda_M(W_{\mathrm{FT}})-\Lambda_M(W_M) denotes the retained-space curvature change during fixed-mask fine-tuning. Since ∣δFT∣≤BFT|\delta_{\mathrm{FT}}|\le B_{\mathrm{FT}}, we have

\geq \Lambda_M(W_M)-\Lambda_0-B_{\mathrm{FT}}.$$ Here $\Lambda_0$ is dense full-space curvature, and $\Lambda_M(W_M)$ and $\Lambda_M(W_{\mathrm{FT}})$ are retained-space curvatures immediately after masking and after fine-tuning. In words, **a post-mask increase larger than the allowed fine-tuning change must survive adaptation**. This is a conditional guarantee; our experiments measure the stage changes but do not numerically certify this sufficient condition. Submitted Section 4.4/Appendix E.2 illustrates why the stages matter: for ResNet-50/Fisher, fine-tuning raises curvature at 20% sparsity, whereas at 80% it offsets most of a masking-induced increase on average. The decomposition distinguishes an increase generated during adaptation from one produced mainly by masking. **Revision locations.** We state the persistence condition after Theorem 3.7 in Section 3.3 (p. 6; submitted Theorem 3.6). We measure stagewise changes in Section 4.4 (pp. 8–9), with the full sparsity sweep in Appendix E.2 (Table 24, p. 31). We distinguish the theoretical geometric analysis from empirical robustness evaluation in the Introduction (p. 1). For **Weakness 2 and Question 2: curvature and robustness**, sharpness-aware minimization (SAM) lowers curvature and improves PGD accuracy in all five seeds. This intervention supports joint improvement; because the optimizer affects more than curvature, it does not isolate curvature's causal contribution. Submitted Section 4.2/Table 1 reports increased sensitivity and robustness losses despite comparatively preserved in-distribution (ID) accuracy. Our added intervention starts SAM(AdamW) and AdamW from identical 80%-magnitude masked ResNet-50/CIFAR-100 checkpoints, using equal fine-tuning epochs. **Table R1. SAM versus AdamW after the same mask; five-seed paired summaries with 95% confidence intervals (CI).** Accuracy differences are SAM minus AdamW, in percentage points (pp). PGD-20 uses two restarts, $\ell_\infty$ radius $1/255$, and 1,000 fixed test images. | Measure | Mean | 95% paired CI | |---|---:|---:| | Curvature ratio, SAM / AdamW | 0.584 | [0.401, 0.767] | | PGD accuracy gain (pp) | 1.98 | [1.18, 2.78] | | ID accuracy gain (pp) | 0.88 | [0.19, 1.57] | A separate **matched-training control** also loses PGD accuracy during additional clean-input training: dense reaches 13.82%, versus 14.50%/14.22% for 80%-sparse magnitude/Fisher under the matched schedule. Thus comparison with original dense describes the combined pruning-and-adaptation outcome; these controls do not show an additional pruning penalty relative to matched dense adaptation. **Revision locations.** We report the matched dense and SAM comparisons with 95% CIs in Appendix C.2 (Tables 13–14, pp. 26–27), with training and evaluation protocols in Appendix B.1 (p. 20).

Continuing our response.

For Weakness 3: measurement, submitted Section 4.4/Table 3 estimates the theoretical retained-subspace Hessian curvature. Section 4.2 uses a separate one-step SAM-based sharpness estimate; the revision explicitly defines its directional sensitivity measurement.

Curvature uses 16-step Lanczos with exact projected Hessian–vector products (HVPs). The SAM-based estimate averages mini-batch loss increases after one radius-0.05 step along the normalized retained-gradient direction. The revised Section 4.2 replaces the neighborhood-maximum definition with this implemented estimate and explicitly identifies it as directional parameter sensitivity.

Revision locations. We define the estimate in Section 4.2 (pp. 6–7) and Appendix B.2 (p. 20). We give curvature protocols in Appendix B.4 (p. 21), residual and initialization checks in Appendix E.4 (pp. 30–31), and LLM HVP protocols in Appendix B.5 (pp. 22–23).

For Weakness 4 and Question 4: newer pruning methods, added SparseGPT experiments show higher endpoint curvature under unstructured pruning at 40% and 60%, and lower curvature under 2:4 pruning at 50%, on Qwen2-0.5B. These outcomes extend the diagnostic to reconstruction-based pruning and reveal method-dependent geometric changes.

SparseGPT updates retained weights during reconstruction. We therefore evaluate the completed compensated models: endpoint curvature reflects masking and compensation together.

Table R2. Qwen2-0.5B SparseGPT after compensation. Curvature uses 64 WikiText-2 blocks × 128 tokens and compares sparse retained with dense full coordinates in the same attention/MLP scope. Curvature and PPL are ratios to dense; HellaSwag changes are percentage points (pp).

Pattern / sparsityCurvature ratioPPL ratioHellaSwag change (pp)
Unstructured, 40%1.9081.203−5.09
Unstructured, 60%3.7132.460−15.31
2:4, 50%0.6232.633−17.14

The same direction of curvature change holds across three evaluation subsets for these settings. All models use one calibration draw; the subsets are repeated evaluations of fixed models and masks.

The separate 16-block sweep uses the same models and calibration draw, but different evaluation data and a different eigensolver width for 2:4. At matched 50% sparsity, 2:4 has lower curvature but worse task scores than unstructured SparseGPT, illustrating why geometric diagnostics complement task evaluation.

Revision locations. We report these endpoints in Section 4.5 (Table 5, p. 10) and the 16-/64-block comparisons in Appendix F.2 (Tables 30–31, pp. 32–33). We give protocols in Appendix B.5 (pp. 22–23) and the compensation extension and its assumptions in Appendix A.9 (p. 19).

For Question 3: geometry in pruned language models, the added direct measurements show increased directional sensitivity at 40% Wanda sparsity in both Qwen and Coder, alongside the SparseGPT curvature increases in Table R2. Curvature and directional sensitivity capture different aspects of the endpoint geometry.

Submitted Section 4.5 reports task outcomes, and Appendices B.5/G.1 evaluate the dense-Hessian immediate-gradient predictor on Qwen2-0.5B. The new measurements below evaluate endpoint curvature and sensitivity directly.

Table R3. Wanda geometry and task outcomes. Ratios are sparse/dense; task changes are HellaSwag accuracy (Qwen) or HumanEval pass@1 (Coder), in pp. Sensitivity compares identical perturbable coordinates; curvature compares retained sparse with full dense coordinates in the attention/MLP scope. Coder denotes Qwen2.5-Coder-0.5B.

ModelSparsityCurvature ratioSensitivity ratioPPL ratioTask change (pp)
Qwen2-0.5B20%0.8740.9851.017−0.57
Qwen2-0.5B40%0.6981.0401.192−4.87
Coder-0.5B20%0.9590.8951.015−3.66
Coder-0.5B40%0.6411.0641.153−13.41

At 40%, sensitivity rises by 4.0%/6.4% for Qwen/Coder while maximum curvature remains below dense. These results distinguish increased finite-radius directional sensitivity from an increase in maximum Hessian curvature.

Geometry uses 64 WikiText-2 test blocks for Qwen and 42 MBPP validation blocks for Coder, each of 128 tokens. Coder PPL uses MBPP test text; HumanEval uses a new execution, separate from the submitted run. Each model has one checkpoint and calibration draw.

Revision locations. We summarize direct LLM geometry in Section 4.5 (p. 10) and report the Wanda endpoints in Appendix F.2 (Table 29, p. 32), with evaluation protocols in Appendix B.5 (pp. 22–23).

We welcome further feedback.

I thank the authors for the detailed rebuttal and for the additional experiments. While I would agree that the revision strengthens the paper, I believe there are some elements and limitations that still remain.

  • I appreciate the clarification regarding the theorems, but I believe that the theoretical results remain largely conditional decompositions and bounds, without establishing when pruning should increase the final curvature. The connection between the theoretical part and the central empirical observation still remains insufficiently demonstrated (at least in my opinion).
  • I appreciate the additional results using advanced pruning methods. The results seem to show that somehow the results are method-dependent with some pruned models exhibiting lower curvature despite the degraded task performance. In my opinion, this limits the generality of the proposed relationship between pruning, curvature and performance that the authors are trying to highlight.

While I still believe the manuscripts is missing a number of elements and that the overall paper’s story is not yet full and causal. I would increase my score given the efforts done by the author in the rebuttal that at least resolved some of my concerns.

Reviewer_1qgWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Prior studies on pruning primarily focused on the impact of sparsification on in-distribution accuracy (called 'clean accuracy' in this work). This work analyzes the impact of unstructured pruning (provided weights are selected in a way that rankings are similar to those obtained using the optimal brain surgeon saliency). In particular, using a two-part Hessian-based saliency (weight importance using the Hessian diagonals, and a interconnection score using the off-diagonal terms), the impact of pruning and subsequent finetuning on the models curvature and robustness to adversarial attack are studied. Specifically, in this setting, the effect of masking is quantified (Thm 3.4), and the effect of finetuning given a fixed mask is bounded (Thm 3.6). These results are validated on both vision and language models, and is compared to magnitude-based pruning on in-distribution and corrupted data.

Critical strengths

The paper is interesting, and has a few key strengths.

  • The main quesiton being addressed by this paper - that is, how does pruning impact metrics other than in-distribution accuracy - is interesting and particularly relevant in modern deep learning, where out-of-distribution performance/robustness are crucial.
  • This paper provides a formal analysis of how masking (unstructured) impacts the extent to which the loss function curvature changes, both before and after fine tuning, using standard tools.
  • The paper is quite well-written (with a few nitpicks) and easy to follow.
  • The interaction score sjs_j defined in Def. 3.2 is an interesting way to use the loss function curvature to identify which weights impact the gradients of other weights significantly, which is perhaps more informative than the OBS score.
Critical weaknesses

Weaknesses

  • Proposition 3.5, which concerns the impact of prunign on gradients, is restricted to saliency measures that generate masks that can fall into the categories defined in Def. 3.2 (Ilow,Clow,ChighI_{low}, C_{low},C_{high}). This is perhaps restrictive, and the results in A.5 are noticeably weaker, but more realistic for pruning. Perhaps those results should be highlighted, and Prop. 3.5. should be a corollary to them.
  • Prop 3.5. also seems to require that H0H_0 be PSD - could the authors clarify? This is a fairly strong assumption, and typically does not hold in practice.

Nitpicks (not affecting score)

  • 'Clean accuracy' is imprecise. The authors would be better served by directly calling it 'In-Distribution Accuracy'/
  • I don't think this work makes significant claims about generalization - this makes the title somewhat misleading. The key claims actually are about loss function geometry, and the experimental evaluation showcases robustness vs in-distribution accuracy.
Questions for authors
  • Have you thought about extending this work to structured (i.e. convolutional filter/weight matrix column/row) pruning, or semi-structured (N:M) pruning?
  • I think instead of using the Hessian, the authors use the expected Fisher Information for pruning (thus avoiding costly Hessian computation). But how does the quality of the saliency change when more or fewer samples are used to compute the expectation?
  • Most unstructured pruning approaches used today (i.e. WandA, SparseGPT) are very much activation based. How would one use the proposed machinery to analyze such methods?
  • See 'Weaknesses' as well.
3 comments

Thank you for the constructive feedback. The submission analyzes how a given pruning mask changes retained-weight gradients and how subsequent adaptation changes loss curvature. We clarify its applicability and add practical-mask diagnostics, calibration-size tests, and SparseGPT experiments.

Revision locations use the revised PDF; other manuscript numbers refer to the submission.

For Weakness 1: mixed masks, the general response analysis covers fixed coordinate masks under its stated assumptions, including masks spanning multiple taxonomy groups. Proposition 3.5 provides group-specific conditional bounds within this framework.

Appendix A.5 of the submitted manuscript generalizes Proposition 3.5's group-specific analysis to mixed masks, using the same retained-space projector for all group contributions. In particular, for the low-saliency, low-interaction, and high-interaction contributions indexed by K∈{I,L,H}K\in\{I,L,H\}, its upper bound is

∥GM∥2≤∑K∈{I,L,H}∥G^M,K∥2+εM.\lVert G_M \rVert_2 \leq \sum_{K\in\{I,L,H\}} \lVert \widehat{G}_{M,K} \rVert_2 + \varepsilon_M.

Here GMG_M is the actual retained-gradient change, G^M,K\widehat G_{M,K} is group KK's dense-Hessian prediction, and εM\varepsilon_M bounds the prediction error. All group contributions use the projector of the complete mask. The corresponding lower bound accounts for opposing contributions: weights with high interaction scores can have a smaller joint response when their effects cancel.

Submitted Appendix D/Table 8 reports the group composition of ordinary Fisher masks. We extend this diagnostic to magnitude and Fisher masks across three ResNet-50 checkpoints at 20%, 50%, and 80% sparsity. Fifteen of eighteen masks span multiple groups, and cross-group contributions partially cancel in all three checkpoints at 20% magnitude sparsity. These observations illustrate the relevance of the mixed-mask analysis to practical pruning.

Revision locations. We present the mixed-mask bounds before Proposition 3.6 in Section 3.2 (Corollary 3.5, p. 5). Derivations remain in Appendix A.5 (p. 17), and practical-mask diagnostics appear in Appendix D.2 (Table 21, p. 29).

For Weakness 2: PSD, this assumption is required only for Proposition 3.5(i)'s low-individual-saliency bound, including its use for a component of a mixed mask. It allows diagonal-Hessian saliency to control off-diagonal gradient influence.

Theorems 3.4 and 3.6, Proposition 3.5(ii)–(iii), and the general mixed-mask inequalities apply without PSD while retaining their other assumptions. The PSD premise concerns the dense loss Hessian; empirical-Fisher PSD does not establish it.

Revision locations. We specify the limited PSD premise in Section 3.2, Proposition 3.6(i) and its following paragraph (p. 5).

For Question 2: Fisher calibration size, increasing calibration from 125 to 500 examples improves mean mask agreement with the 1,000-example reference by 8.0 percentage points for ResNet-50 and 4.0 for ViT-B/16. The high-versus-low interaction response ordering holds at every tested sample size and draw. Thus sample size affects which weights are selected, while this qualitative response contrast is stable in our tests.

The submission uses observed-label empirical Fisher for scoring and grouping, and loss-Hessian–vector products for predictions and curvature. We additionally tested 125/250/500/1,000 calibration examples with three draws on one CIFAR-100 checkpoint per architecture. Within each draw, smaller sets are nested in its 1,000-example reference.

Table R1. Fisher mask agreement at 20% sparsity. Intersection over union (IoU) of removed-weight sets with the nested 1,000-example reference, in percent; three-draw mean ± sample SD.

Model125 examples250 examples500 examples
ResNet-5066.7 ± 5.470.6 ± 4.174.7 ± 4.0
ViT-B/1675.7 ± 3.579.3 ± 3.179.7 ± 5.4

The reference is finite, so IoU measures agreement rather than population-Fisher accuracy; between-draw stability is non-monotonic.

In a separate 5%-sparsity response experiment, high-interaction masks produce larger observed retained-gradient response norms than low-interaction masks in all 12 sample-size/draw combinations per architecture, including the 125-example settings. Evaluation examples are disjoint from calibration; paired masks match per-layer removal counts, with approximate matching of removed-weight energy.

Revision locations. We distinguish Fisher scoring from Hessian measurements in Section 4.1 (p. 6). We report mean and per-draw overlap, response ordering, between-draw comparisons, and sampling/evaluation protocols in Appendix B.6 (Tables 6–8, p. 23).

Continuing our response.

For Question 1: structured and semi-structured pruning, fixed coordinate-mask identities cover N:M supports, including 2:4, and zeroing entire rows, columns, or filters under the stated assumptions. This representation retains the original parameter coordinates and masked objective. Physical channel/filter deletion additionally requires a correspondence with the reduced architecture; block-specific bounds and reduced-model experiments remain outside the current evaluation.

We additionally compare unstructured and 2:4 SparseGPT at matched 50% sparsity and calibration budgets (Table R2). Both evaluate the complete procedure, including compensation.

Revision locations. We clarify N:M applicability and architectural reduction in Section 3.2 (p. 5). We summarize the comparison in Section 4.5 (p. 10) and give matched-50% results in Appendix F.2 (Table 30, p. 32), with protocols in Appendix B.5 (pp. 22–23).

For Question 3: Wanda and SparseGPT, the framework analyzes a selected mask independently of its selection criterion. For activation-aware Wanda, Theorem 3.4 predicts the immediate retained-gradient change using the dense loss Hessian and the removed weights.

Submitted Appendix G.1/Table 16 reports prediction–response cosines of 0.779/0.655/0.574 for Qwen2-0.5B Wanda masks at 20/30/40% sparsity. These show directional information alongside substantial vector errors; at 40%, relative prediction error is 1.051. Appendix B.5 specifies the HVP protocol, and Section 4.5 reports task outcomes.

For SparseGPT, we extend the curvature decomposition to its completed compensated model. Applying the final mask to the original dense weights defines a mathematical reference; the curvature change from that reference to the endpoint replaces the fine-tuning term. Because masking and compensation are interleaved, this reference need not be an actual intermediate state. The corresponding bounds require path regularity and displacement control. The measurements below evaluate masking and compensation together.

Table R2. Qwen2-0.5B SparseGPT after pruning and compensation. Curvature ratios compare sparse retained with dense full coordinates in the same attention/MLP weight scope, using 16 WikiText-2 blocks × 128 tokens. PPL ratios use 2,048-token test sequences; HellaSwag changes are percentage points (pp) relative to dense.

Sparsity / patternCurvature ratioPPL ratioHellaSwag change (pp)
20%, unstructured1.0031.020−0.74
30%, unstructured1.2111.076−2.22
40%, unstructured1.7151.203−5.09
50%, unstructured2.1941.521−9.37
60%, unstructured6.0082.460−15.31
50%, 2:40.8132.633−17.14

All settings share one calibration draw of 32 sequences × 128 tokens. The separate 64-block evaluation reported to Reviewer 2 uses the same compensated models, with different evaluation data and a different eigensolver width for 2:4; its curvature values therefore follow a distinct protocol.

At matched 50% sparsity, 2:4 has lower curvature but worse PPL and HellaSwag performance than unstructured SparseGPT. Reporting geometry and task scores together distinguishes outcomes that either measure alone would miss.

Revision locations. We summarize SparseGPT in Section 4.5 (Table 5, p. 10). We provide the compensation extension in Appendix A.9 (p. 19), separate 16-/64-block results in Appendix F.2 (Tables 30–31, pp. 32–33), and protocols in Appendix B.5 (pp. 22–23). The existing Wanda HVP analysis remains in Appendix G.1 (Table 32, p. 33).

Regarding terminology and the title, we use “in-distribution accuracy” in the main text and its abbreviation in Tables 1–3. We have revised the title to “What We Lose in Pruning: Understanding Changes in Loss Geometry and Robustness” to reflect the paper's focus more directly. The revised Introduction (p. 1) distinguishes the theoretical analysis of loss geometry from the empirical evaluation of robustness under input shifts.

We welcome further feedback.

I thank the authors for their response. While some of my concerns have been addressed, the response is rather unclear and feels AI-generated. A lot of terms are undefined - what is a prediction-response cosine? What is a directional response? Also, having read Reviewer g8oP and 3DPY's reviews and the issues raised there that I missed in my initial review, I do not think I can raise my score at this time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.