- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposed to remove the neurons in MLP based on the reconstruction error after optimal compensation, to reduce the size of NNs without training/finetuning. The method is straightforward and intuitive, and extensive experimental results on various model structures and sparsity levels show that it preserves better performance than baseline methods.
Critical strengths
- The idea is simple, straightforward, and intuitive. The pruning and compensation can be solved in closed form without fine-tuning.
- The experiments seem to be well-designed, covering a wide range of backbones and parameters (e.g., sparsity levels). The results are good and the ablation study is also quite detailed.
Critical weaknesses
The idea of “according to the compensation/reconstruction error to select neurons to remove” is similar to previous OBS (Optimal Brain Surgeon)/OBC (Optimal Brain Compression) works. Although CARVE applies the idea on structured neuron removing, the paper seems to emphasis that it is a new pruning idea. Further clarification of their specific differences can better state the contribution and novelty of this paper.
Questions for authors
Could you clarify the specific relationship and differences between CARVE and OBS/OBC?
3 comments
Optimal Brain Compression (OBC; Frantar & Alistarh, 2022). OBC makes OBS tractable by applying it layer by layer: instead of the global loss, it minimises the squared error between the outputs of the original and of the compressed layer on calibration data. This objective is quadratic, so the OBS formulas become exact for it. OBC removes weights greedily and updates the inverse Hessian in closed form after each removal. Since the error decomposes over the output rows of the layer, its core solver processes the rows independently (OBC also covers N:M and block sparsity, as well as quantisation).
CARVE and OBC both rely on an exact layer-wise quadratic objective, greedy removal and closed-form updates. They differ in three respects:
- Pruning constraint: OBC's core solver uses row-wise weight masks. CARVE removes a hidden neuron, that is, one row of and one column of , jointly in all the output rows, which yields a smaller dense network. All the rows then share one mask and one precision matrix, so the elimination of a block costs (Appendix O), whereas a core row-wise OBC pass on would cost .
- Compensation: OBC uses the uncentred second moment of the layer inputs and leaves the bias unchanged. CARVE uses an affine compensation, with the centred covariance and a bias update, so the mean contribution of a removed neuron is absorbed exactly by the bias. This matters for MLP hidden units, whose post-activation means are far from zero: a neuron with a large mean but little variance can be judged costly to remove without a bias update, whereas CARVE removes it at almost no error.
- Allocation across layers: OBC obtains non-uniform compression by compressing every layer at many levels, measuring the corresponding per-layer calibration losses, and solving a dynamic programme over these measurements. CARVE allocates a single neuron budget directly from the local post-compensation errors normalised by the residual-stream variance, without candidate-model evaluation or search.
We agree that the general principle of compensation-aware removal is not new and goes back to OBS/OBC. CARVE adapts this principle to post-training structured pruning of MLP neurons: it removes a complete hidden neuron, uses the exact marginal block-output error after affine compensation to update the selection as the surviving set changes, and uses the resulting errors for residual-aware allocation across blocks. We will revise the paper to make this positioning explicit and avoid wording that could suggest that the general principle itself is new.
In the revised version, we will add the comparison with OBS and OBC above to the related work (Section 2) and revise the abstract and introduction to attribute the general compensation-aware principle to OBS/OBC and state only the specific adaptation introduced by CARVE.
These changes clarify the positioning of the contribution; the method, analysis, and experimental conclusions are unchanged.
Thank you for your detailed response and the revisions to clarify the contributions and novelty relative to OBS/OBC. I will maintain my positive rating.
We thank the reviewer for the positive assessment and for this question, which helps us position the contribution more precisely. We summarise the relationship and the differences below.
Q1. Relationship and differences between CARVE and OBS/OBC
Optimal Brain Surgeon (OBS; Hassibi & Stork, 1992). OBS asks the following question: which weight can be deleted while increasing the training error the least? It approximates the variation of the error by a second-order Taylor expansion with respect to the weights, which involves the gradient and the Hessian of the loss over all the parameters of the network. It assumes that the network is trained to a local minimum, so that the gradient vanishes, and uses only the Hessian term to quantify the error. The removal of a weight is then accompanied by an update of all the other weights of the network, which is optimal for this quadratic approximation of the loss. The weight with the smallest increase is removed, and the procedure is repeated.
CARVE shares the philosophy of OBS: an element is judged by the error that remains once the others have been optimally adjusted, which is why we cite OBS as the origin of compensation in the introduction (lines 048-050). The two methods, however, differ in four respects: