What We Lose in Pruning: A Generalization Perspective on Model Pruning

Model pruning, which masks a fraction of model weights, primarily aims to preserve in-distribution accuracy; however, its hidden costs remain less understood. We ask: *Beyond accuracy, what do we lose when we prune a network?* We argue that pruning significantly reshapes the loss geometry, thereby damaging generalization and making the model more sensitive to data disturbances despite high clean accuracy. To theoretically investigate these effects, we analyze the consequences across two sequential stages: parameter masking and subsequent fine-tuning. First, immediately after masking, we characterize how the gradients of the remaining weights change through Hessian interactions. By grouping weights based on individual importance and interaction strength, we derive group-specific bounds on these gradient shifts. Second, after fine-tuning, we analyze sharpness via the maximum Hessian curvature with respect to the remaining weights. We decompose the change in sharpness relative to the original model into geometric shifts caused by masking and fine-tuning, showing how the final model becomes sharper. Through experiments on vision models and large language models across image classification, language modeling, commonsense reasoning, and code generation, we demonstrate that pruning incurs hidden costs in parameter sensitivity, sharpness, and robustness—costs that in-distribution performance fails to capture.