Prune What Can Be Reconstructed: CARVE for Compensation-Aware Neuron Selection

This paper focuses on post-training structured pruning of MLP hidden neurons in pretrained vision backbones, without fine-tuning. Most modern approaches follow the same paradigm: they first rank neurons using a fixed importance score and then compensate for the neurons that are removed. We propose a different paradigm: neurons should be selected according to *what cannot be recovered after compensation*, rather than according to their importance before compensation. Following this principle, we introduce Covariance-Aware Reduction Via Elimination (CARVE), which greedily removes the neuron that causes the smallest increase in reconstruction error after optimal affine compensation through the surviving neurons. To the best of our knowledge, CARVE is the first post-training structured pruning method whose selection criterion is directly defined by the exact error remaining after optimal compensation and is updated as the set of surviving neurons changes. We further introduce a global allocation strategy that rescales local errors by the residual-stream signal for cross-block comparison. % makes these local % errors comparable across blocks by referencing each to the residual-stream % signal it perturbs. We evaluate CARVE on ImageNet-1k across nine vision backbones, from 5.7M to 632M parameters, and nine MLP sparsities from 10% to 90%. Without fine-tuning, CARVE outperforms the strongest competing baseline by **5.6 Top-1 accuracy points on average over the nine vision backbones** at sparsities of 50% and above, with gains reaching **20.7 points** at aggressive compression.