acceptodds
Under review as a conference paper at ICLR 2027

Prune What Can Be Reconstructed: CARVE for Compensation-Aware Neuron Selection

17 views

Abstract

This paper focuses on post-training structured pruning of MLP hidden neurons in pretrained vision backbones, without fine-tuning. Most modern approaches follow the same paradigm: they first rank neurons using a fixed importance score and then compensate for the neurons that are removed. We propose a different paradigm: neurons should be selected according to *what cannot be recovered after compensation*, rather than according to their importance before compensation. Following this principle, we introduce Covariance-Aware Reduction Via Elimination (CARVE), which greedily removes the neuron that causes the smallest increase in reconstruction error after optimal affine compensation through the surviving neurons. To the best of our knowledge, CARVE is the first post-training structured pruning method whose selection criterion is directly defined by the exact error remaining after optimal compensation and is updated as the set of surviving neurons changes. We further introduce a global allocation strategy that rescales local errors by the residual-stream signal for cross-block comparison. % makes these local % errors comparable across blocks by referencing each to the residual-stream % signal it perturbs. We evaluate CARVE on ImageNet-1k across nine vision backbones, from 5.7M to 632M parameters, and nine MLP sparsities from 10% to 90%. Without fine-tuning, CARVE outperforms the strongest competing baseline by **5.6 Top-1 accuracy points on average over the nine vision backbones** at sparsities of 50% and above, with gains reaching **20.7 points** at aggressive compression.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Transformers, Structured pruning, Post-training compression, ImageNet, ConvNeXt

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_C6YdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed to remove the neurons in MLP based on the reconstruction error after optimal compensation, to reduce the size of NNs without training/finetuning. The method is straightforward and intuitive, and extensive experimental results on various model structures and sparsity levels show that it preserves better performance than baseline methods.

Critical strengths
  1. The idea is simple, straightforward, and intuitive. The pruning and compensation can be solved in closed form without fine-tuning.
  2. The experiments seem to be well-designed, covering a wide range of backbones and parameters (e.g., sparsity levels). The results are good and the ablation study is also quite detailed.
Critical weaknesses

The idea of “according to the compensation/reconstruction error to select neurons to remove” is similar to previous OBS (Optimal Brain Surgeon)/OBC (Optimal Brain Compression) works. Although CARVE applies the idea on structured neuron removing, the paper seems to emphasis that it is a new pruning idea. Further clarification of their specific differences can better state the contribution and novelty of this paper.

Questions for authors

Could you clarify the specific relationship and differences between CARVE and OBS/OBC?

3 comments

We thank the reviewer for the positive assessment and for this question, which helps us position the contribution more precisely. We summarise the relationship and the differences below.

Q1. Relationship and differences between CARVE and OBS/OBC

Optimal Brain Surgeon (OBS; Hassibi & Stork, 1992). OBS asks the following question: which weight can be deleted while increasing the training error the least? It approximates the variation of the error by a second-order Taylor expansion with respect to the weights, which involves the gradient and the Hessian of the loss over all the parameters of the network. It assumes that the network is trained to a local minimum, so that the gradient vanishes, and uses only the Hessian term to quantify the error. The removal of a weight is then accompanied by an update of all the other weights of the network, which is optimal for this quadratic approximation of the loss. The weight with the smallest increase is removed, and the procedure is repeated.

CARVE shares the philosophy of OBS: an element is judged by the error that remains once the others have been optimally adjusted, which is why we cite OBS as the origin of compensation in the introduction (lines 048-050). The two methods, however, differ in four respects:

  • Cost function: OBS minimises a second-order approximation of the global training loss, which is only valid near a local minimum and requires the Hessian over all the parameters. With modern networks, this Hessian is intractable, since its size is the square of the number of parameters (86M × 86M for DeiT-Base). CARVE minimises the exact reconstruction error of the output of one MLP block, which is quadratic in the compensated parameters: its cost involves no Taylor approximation and no assumption on the training state, and only requires the H×HH \times H covariance of the hidden activations of the block.
  • Compensation: As a consequence, the OBS update of all the weights of the network is only optimal within the second-order approximation. The CARVE compensation is the exact closed-form solution of its problem: the removed neuron is folded into the surviving neurons of the same block and into the bias.
  • Allocation across layers: In OBS, all the parameters compete within a single global loss, so the distribution of the budget across layers has no separate meaning. CARVE works block by block, which makes this allocation an explicit question, answered by the residual-normalised criterion of Section 3.2.
  • Unit and scope: OBS removes individual weights and applies to any model trained by gradient descent, at the price of the approximation above. CARVE removes whole hidden neurons, and exploits the structure of the MLP block to make its criterion exact and to produce a smaller dense network.

Optimal Brain Compression (OBC; Frantar & Alistarh, 2022). OBC makes OBS tractable by applying it layer by layer: instead of the global loss, it minimises the squared error between the outputs of the original and of the compressed layer on calibration data. This objective is quadratic, so the OBS formulas become exact for it. OBC removes weights greedily and updates the inverse Hessian in closed form after each removal. Since the error decomposes over the output rows of the layer, its core solver processes the rows independently (OBC also covers N:M and block sparsity, as well as quantisation).

CARVE and OBC both rely on an exact layer-wise quadratic objective, greedy removal and closed-form updates. They differ in three respects:

  • Pruning constraint: OBC's core solver uses row-wise weight masks. CARVE removes a hidden neuron, that is, one row of W1W_1 and one column of W2W_2, jointly in all the output rows, which yields a smaller dense network. All the rows then share one mask and one H×HH \times H precision matrix, so the elimination of a block costs O(H3+H2D)O(H^3 + H^2 D) (Appendix O), whereas a core row-wise OBC pass on W2W_2 would cost O(DH3)O(D H^3).
  • Compensation: OBC uses the uncentred second moment of the layer inputs and leaves the bias unchanged. CARVE uses an affine compensation, with the centred covariance and a bias update, so the mean contribution of a removed neuron is absorbed exactly by the bias. This matters for MLP hidden units, whose post-activation means are far from zero: a neuron with a large mean but little variance can be judged costly to remove without a bias update, whereas CARVE removes it at almost no error.
  • Allocation across layers: OBC obtains non-uniform compression by compressing every layer at many levels, measuring the corresponding per-layer calibration losses, and solving a dynamic programme over these measurements. CARVE allocates a single neuron budget directly from the local post-compensation errors normalised by the residual-stream variance, without candidate-model evaluation or search.

We agree that the general principle of compensation-aware removal is not new and goes back to OBS/OBC. CARVE adapts this principle to post-training structured pruning of MLP neurons: it removes a complete hidden neuron, uses the exact marginal block-output error after affine compensation to update the selection as the surviving set changes, and uses the resulting errors for residual-aware allocation across blocks. We will revise the paper to make this positioning explicit and avoid wording that could suggest that the general principle itself is new.

In the revised version, we will add the comparison with OBS and OBC above to the related work (Section 2) and revise the abstract and introduction to attribute the general compensation-aware principle to OBS/OBC and state only the specific adaptation introduced by CARVE.

These changes clarify the positioning of the contribution; the method, analysis, and experimental conclusions are unchanged.

Thank you for your detailed response and the revisions to clarify the contributions and novelty relative to OBS/OBC. I will maintain my positive rating.

Reviewer_AaLhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an post-training MLP pruning method for vision models; the main idea of the paper is that neurons should not be pruned via some proxy importance score followed by compensation, and they should instead be pruned according to whether or not they can be recovered after compensation. The method, CARVE, greedily selects neurons to remove based on the error they induce after optimal compensation is applied, and selects neurons non-uniformly across layers. The method is evaluated on vision models with ImageNet-1k, and on Qwen2.5 on wikitext. Results are strong on vision models on ImageNet classification, and are consistent with language modeling.

Critical strengths
  1. The overall optimization framing where the subset K is part of the minimization problem is a nice framing which is then solved by greedy selection. This is an interesting way to frame compression by jointly modeling the selection & compensation, which is the point of the proposed method.

  2. The paper is able to greedily select neurons without recomputing covariance by updating the compensated weights and precision matrix in closed form. Otherwise, recomputing variables for each greedy neuron selection would be perhaps prohibatively expensive.

Critical weaknesses
  1. In language models, MLPs are thought to contain parametric knowledge in terms of memorized information. When pruning MLPs specifically, it would be helpful to understand how they perform on knowledge-intensive tasks. The paper as it stands does test on Qwen, but it tests only on perplexity, which does not reflect a knowledge-intensive task. It would strengthen the paper to examine this effect, extending language modeling evaluation to knowledge intensive tasks where perhaps different ways of pruning MLPs would show greater performance deviation.

  2. Similarly, the majority of the paper's results are only on ImageNet, with limited additional results in the appendix. Despite using multiple image classification model backbones, the method's extensibility is largely limited to the image classification setting. However, it is the case that it does indeed perform well in this setting, but perhaps the paper should reflect that this is the case.

Questions for authors
  1. The cost of selecting neurons is limited due to closed form variable updates, but the cost is still linear in the number of pruned neurons. Do you have a computational cost comparison between CARVE and related methods?
5 comments

We thank the reviewer for the detailed review, and for appreciating the optimisation framing in which the kept subset is part of the minimisation problem. Here are our answers to the reviewer’s constructive feedback. The revised version of the paper has been uploaded, with all changes highlighted in blue.

W1. Knowledge-intensive evaluation of LLMs

We agree that perplexity alone does not fully capture factual knowledge retention. Motivated by the reviewer's suggestion, we extended our Qwen2.5-1.5B experiments to MMLU, NQ-open, and TriviaQA, comparing the same pruning approaches. Since the calibration data of a post-training method should cover the target tasks, every method is recalibrated for these benchmarks on multiple-choice questions from the MMLU auxiliary training set instead of WikiText-2, with the same data for all methods and without the items overlapping MMLU test questions.

Question answering on Qwen2.5-1.5B (lm-evaluation-harness, 5-shot; MMLU accuracy, NQ-open and TriviaQA exact match, in %). Every method is calibrated on 128 sequences of 2,048 tokens of multiple-choice questions from the MMLU auxiliary training set. Mean ± s.d. over three calibration seeds (MAG: one run; TriviaQA: one calibration draw). Best pruned model per column in bold.

Method10%: MMLU10%: NQ-open10%: TriviaQA30%: MMLU30%: NQ-open30%: TriviaQA
Dense60.914.240.060.914.240.0
MAG25.10.40.324.60.00.0
Wanda55.9 ± 0.110.1 ± 0.231.146.3 ± 0.04.5 ± 0.311.3
VBP49.4 ± 0.20.1 ± 0.10.528.5 ± 0.40.0 ± 0.00.3
DVBP50.6 ± 0.27.6 ± 0.417.934.8 ± 6.30.0 ± 0.00.1
CORP56.6 ± 0.310.7 ± 0.132.747.4 ± 0.14.9 ± 0.114.1
GRAIL56.3 ± 0.010.2 ± 0.330.246.8 ± 0.24.6 ± 0.113.3
CARVE_L57.1 ± 0.211.7 ± 0.234.848.8 ± 0.15.4 ± 0.114.9
CARVE54.0 ± 0.410.0 ± 0.529.842.9 ± 0.16.2 ± 0.216.4

These experiments confirm that knowledge-intensive tasks reveal behaviour that is not captured by perplexity alone. All methods experience substantial degradation on these tasks as sparsity increases. At 10% MLP sparsity, CARVE achieves the best perplexity, while CARVE_L performs best on MMLU, NQ-open, and TriviaQA. At 30% sparsity, CARVE_L remains best on MMLU, whereas CARVE achieves the best NQ-open and TriviaQA scores. Unlike perplexity, which measures average token-level prediction quality, these benchmarks probe more specific behaviours such as multiple-choice decision making and factual question answering. These results suggest that preserving language-model likelihood does not necessarily preserve all downstream behaviours equally well, and that different pruning allocations may affect these behaviours differently.

These findings should nevertheless be interpreted within the scope of the paper. Our main contribution concerns post-training structured pruning of MLP neurons in pretrained vision backbones, as stated in the abstract (lines 11-12) and introduction (lines 33-44). The Qwen2.5-1.5B experiment is included as an exploratory extension beyond vision, showing that the CARVE criterion can be applied to a decoder-only language model, rather than as a comprehensive study of LLM compression or knowledge preservation.

The additional results also highlight a broader aspect of task-agnostic structured MLP pruning. While such methods can effectively preserve generic model behaviour without task-specific supervision, they do not explicitly account for the specific behaviours probed by each downstream task. As a result, different pruning allocations may preserve these behaviours differently, particularly as sparsity increases. We therefore add the knowledge-intensive evaluation to Appendix M and discuss this task dependence explicitly. A systematic study of task-aware pruning and allocation for language models remains beyond the scope of this work.

W2. Evaluation beyond ImageNet classification

We chose ImageNet-1k as our primary benchmark because it provides a common evaluation setting with existing post-training vision pruning approaches (VBP, DVBP, CORP), enabling direct and controlled comparisons. Our evaluation nevertheless covers nine backbones, ranging from DeiT-Tiny to DeiT-III-Huge and ConvNeXt, as well as additional experiments on CIFAR-100. These results demonstrate that CARVE is effective across different vision backbone architectures and model scales, although they do not establish generalisation to all downstream vision tasks. Note also that the CARVE criterion does not depend on the task: it only reconstructs the output of each MLP block from unlabelled activations, and never uses the classification head.

To address this concern, we extend the evaluation beyond image classification by pruning DINOv2 and evaluating the resulting model on two dense prediction tasks, under the same post-training setting as the rest of the paper. We follow the setting of the DINOv2 experiments of CORP (same tasks, released heads and calibration images): the MLPs of the DINOv2 backbone are pruned with an unlabelled calibration set, the pretrained task heads are kept frozen, and no fine-tuning is applied. This setting differs substantially from classification: DINOv2 is pretrained without labels, and dense prediction produces spatially resolved outputs rather than a single image-level prediction.

NYUv2 depth RMSE in metres (lower is better) of DINOv2 backbones with the frozen DPT head, after one-shot MLP pruning without fine-tuning. Mean over three calibration seeds (s.d. ≤ 0.07, Appendix T). Best per row in bold.

Backbone (dense)SparsityMAGWandaVBPDVBPCORPGRAILCARVE
DINOv2-S (0.401)30%1.3742.0850.5810.4350.4130.4160.426
50%1.7342.1271.7650.5910.4750.4920.524
70%2.3172.0891.9821.3711.0060.7541.784
DINOv2-B (0.357)30%1.5902.2221.1060.4460.3760.3770.365
50%1.2492.3701.9320.6550.4470.5200.390
70%1.3802.1902.4210.8750.8360.9340.510
DINOv2-L (0.352)30%2.0962.0470.5730.3780.3540.3520.354
50%2.0972.0220.9490.4600.3940.3720.359
70%1.7781.7561.7110.5160.4800.4220.468

ADE20K mIoU in % (higher is better) of the same pruned backbones with the frozen linear head. Mean over three calibration seeds (s.d. ≤ 0.5, Appendix T). Best per row in bold.

Backbone (dense)SparsityMAGWandaVBPDVBPCORPGRAILCARVE
DINOv2-S (44.4)30%0.10.320.430.637.039.341.8
50%0.20.12.817.526.928.429.7
70%0.20.00.34.610.09.55.3
DINOv2-B (47.0)30%0.20.328.436.143.644.444.7
50%0.10.310.426.435.837.737.3
70%0.20.41.012.221.323.222.6
DINOv2-L (47.1)30%0.20.441.444.146.246.847.1
50%0.10.329.836.241.643.546.3
70%0.10.512.027.131.633.640.7

In the revised version, we report these results in Appendix T and summarise them in Section 4.2. CARVE remains competitive across both dense prediction tasks and all three DINOv2 scales. For example, on ADE20K it achieves 40.7 mIoU on DINOv2-L at 70% sparsity, 7.1 points above the strongest baseline, while on NYUv2 it reduces RMSE on DINOv2-B at 50% sparsity from 0.447 to 0.390.

We thank the reviewer again for this suggestion, which led us to broaden the evaluation beyond classification and, we believe, improves the completeness of the paper.

Q1. Computational cost of greedy elimination

We thank the reviewer for raising this point. The reviewer is right that the elimination performs one greedy step per removed neuron, so the number of steps grows linearly with the number of pruned neurons. However, each step is cheap, and no step re-inverts the covariance: removing a neuron is a closed-form rank-one update of the compensated weights and of the precision matrix, which costs O(kD+k2)O(kD + k^2) operations when kk neurons remain. Summed over all steps, the elimination of a block costs O(H3+H2D)=O(H3)O(H^3 + H^2 D) = O(H^3), since the hidden width HH exceeds the embedding dimension DD (H=4DH = 4D in ViTs). This is the same complexity as a single inversion of the H×HH \times H covariance matrix (Appendix O). Moreover, the blocks are processed independently.

In practice, compressing a backbone with CARVE takes a few minutes on a single GPU, even for the 632M-parameter DeiT-III-Huge, and a large part of this time is spent in the calibration pass that collects the activation statistics, which the other compensated methods also need.

Compression time in seconds at 50% MLP sparsity (one B200 GPU, mean ± s.d. over 4 runs). Calibration: statistics pass (SNIP: gradient pass on 64 labelled images); Inversion: covariance inversions and ridge solves; Elimination: greedy elimination of CARVE; "–": the method has no such step.

MethodDeiT-Base: CalibrationInversionEliminationTotalDeiT-III-Huge: CalibrationInversionEliminationTotal
MAG–––0.10 ± 0.01–––0.64 ± 0.04
SNIP0.43 ± 0.10––2.2 ± 0.11.2 ± 0.0––3.4 ± 0.1
VBP12.4 ± 0.3––16.9 ± 0.567.4 ± 0.8––72.4 ± 1.1
DVBP14.0 ± 0.10.20 ± 0.01–25.6 ± 0.499.1 ± 13.41.6 ± 0.1–176.8 ± 15.5
CORP14.5 ± 2.70.22 ± 0.03–17.4 ± 2.6112.9 ± 26.31.7 ± 0.1–121.5 ± 26.6
GRAIL13.9 ± 1.70.33 ± 0.01–17.3 ± 1.9105.7 ± 6.62.8 ± 0.1–116.4 ± 6.6
CARVE_L15.1 ± 1.90.88 ± 0.0323.8 ± 3.543.1 ± 1.8119.0 ± 19.97.4 ± 0.5103.6 ± 6.6240.7 ± 19.1
CARVE19.8 ± 8.50.95 ± 0.1022.1 ± 1.756.9 ± 8.8104.1 ± 8.17.5 ± 0.5109.4 ± 11.0277.9 ± 25.4

In the revised version, we add a new appendix that compares the end-to-end compression time of all methods under the same protocol, with the mean and standard deviation over repeated runs. We hope these clarifications address the reviewer's concerns.

Reviewer_GNtpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Covariance-Aware Reduction Via Elimination (CARVE) to prune the models in a post-training structured sheme. It evaluates neuron importance based on the reconstruction error remaining after optimal compensation. Moreover, the method employs greedy neuron elimination with efficient rank-one updates and introduces a residual-aware global allocation strategy to distribute the pruning budget across blocks. Experimental results on multiple vision backbones and benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths
  • The paper is generally well-written.
  • The experimental results show the effectiveness.
  • The theoretical proofs are comprehensive.
Critical weaknesses

Some technical details and experimental results require further clarification.

Questions for authors
  • The authors employ class-balanced sampling to construct the calibration set, although the proposed method is not explicitly category-dependent. Does the proposed method require a labeled calibration set?
  • Although the authors report reductions in parameters, MACs, and memory usage, the practical inference efficiency of the pruned models remains unclear. The authors should provide inference latency and throughput comparisons to demonstrate the practical acceleration achieved by CARVE.
  • CARVE employs iterative greedy neuron elimination, which may introduce additional computational overhead compared with previous pruning methods. Please provide the pruning efficiency comparisons with other pruning methods.
3 comments

We thank the reviewer for the careful reading of our paper, for finding it well written and its proofs comprehensive, and for the constructive questions. We answer each question below and describe the corresponding changes to the paper. The revised version of the paper has been uploaded, with all changes highlighted in blue.

Q1. Does CARVE require a labelled calibration set?

No, CARVE does not require labels to perform pruning. Its criterion only uses the mean and covariance of the hidden activations of each MLP, the outgoing weights, and the residual-stream variance, all of which are obtained from forward passes on unlabelled images.

Our ImageNet experiments do use a class-balanced calibration subset of 5,000 images, whose construction relies on labels. We agree that the motivation for this experimental choice was not sufficiently clear in the paper, and we clarify it in Section 4.1. Class balancing is not required by CARVE; it is a convenient way to obtain representative activation statistics with fewer calibration samples. Since calibration-based pruning methods estimate first- and second-order activation statistics from a finite sample, under-represented parts of the input distribution can lead to less stable estimates. On ImageNet, class-balanced sampling guarantees that every class contributes to these statistics, which allows them to stabilise with a smaller calibration set. A uniformly random sample requires no labels and converges to the same behaviour as its size increases.

This question is studied in Appendix Q, where we vary both the size of the calibration set and the sampling strategy, comparing class-balanced and uniformly random subsets. We acknowledge that the main paper did not make this sufficiently clear, as it mainly referred to this appendix as an ablation on the calibration-set size rather than on both the size and the sampling strategy. The results show that (i) the accuracy generally improves and then saturates as the calibration set becomes more representative; (ii) class-balanced sampling reaches this plateau earlier, typically between 1k and 4k images; and (iii) once the calibration set is large enough, the two sampling schemes yield essentially the same accuracy. Importantly, all methods are evaluated on the same calibration subset, and their relative performance is preserved across the two sampling strategies, so our conclusions do not depend on class-balanced sampling.

In the revised version, we state explicitly in the main paper (lines 323-337) that labels are only used to construct the class-balanced calibration subset, not by the pruning method itself, and we clarify that Appendix Q studies both the calibration-set size and the sampling strategy.

Q2. Practical inference efficiency: latency and throughput

Since CARVE removes whole neurons, the pruned model is a smaller dense network that runs with standard kernels. For the same final layer shapes, the inference speed therefore does not depend on the pruning method: at a given budget, all uniform-width methods (CARVE_L, VBP_L, CORP, GRAIL) produce exactly the same architecture, hence the same MACs and the same latency.

Following the reviewer's suggestion, we measured the inference speed of the pruned models on one NVIDIA B200 GPU, with the experimental setup of VBP (fp32, batch size 32) and in bf16 at batch size 256. For example, at 50% MLP sparsity, the DeiT-Base model pruned by CARVE runs 1.35× faster than the dense model. Since attention, embeddings, and the classifier are not pruned, this is close to the 1.46× upper bound set by the 31.7% reduction in MACs. The global allocation of CARVE runs within 0.04× of uniform widths.

The speed-up depends on the hardware and on the numerical precision. In bf16, GPU matrix kernels are most efficient when the hidden width is a multiple of 64, so the rounding of the widths matters more than the allocation: unaligned widths can be slower than the dense model, whereas rounding every width up to a multiple of 64 with null neurons, which leaves the accuracy unchanged, gives a speed-up of 1.13× to 1.40×.

Inference speed-up over the dense model (one B200 GPU, mean over 10 rounds, s.d. ≤ 0.01). Dense rows give the absolute throughput. Uniform: widths shared by all uniform-width methods; up64: widths rounded up to multiples of 64.

BackboneSparsityfp32, b32: Uniformfp32, b32: CARVEbf16, b256: Uniform up64bf16, b256: CARVEbf16, b256: CARVE up64
DeiT-Base0%1537 img/s15607 img/s
30%1.17×1.16×1.14×0.46×1.14×
50%1.38×1.35×1.24×0.64×1.24×
70%1.57×1.58×1.34×0.93×1.34×
DeiT-III-Huge0%163 img/s2235 img/s
30%1.23×1.20×1.15×0.44×1.13×
50%1.38×1.39×1.27×0.50×1.26×
70%1.68×1.66×1.40×0.66×1.40×

In the revised version, we have added the measured speed-up to Table 3 and a new appendix (Appendix R) that details the measurement setup and reports these results.

Q3. Computational overhead of greedy neuron elimination

We thank the reviewer for this valuable suggestion. Indeed, CARVE adds a step to the pruning process: in some concurrent methods, the ranking is almost instantaneous once the statistics are collected, whereas CARVE runs a greedy elimination. However, this elimination is cheap: each removal is a closed-form rank-one update, so the covariance is never re-inverted, and the whole elimination of a block has the same complexity, O(H3)O(H^3), as a single inversion of the covariance (Appendix O).

We measured the end-to-end compression time of every method at 50% MLP sparsity, on one B200 GPU. The calibration pass that collects the activation statistics dominates the compression time of the other compensated methods; for CARVE, the greedy elimination costs about as much as this pass. End to end, CARVE takes 56.9 s on DeiT-Base and 4.6 min on DeiT-III-Huge, against 17.4 s and 2.0 min for CORP: its compression time remains of the same order as that of the other compensated methods, i.e. a few minutes on a single GPU.

Compression time in seconds at 50% MLP sparsity (one B200 GPU, mean ± s.d. over 4 runs). Calibration: statistics pass (SNIP: gradient pass on 64 labelled images); Inversion: covariance inversions and ridge solves; Elimination: greedy elimination of CARVE; "–": the method has no such step.

MethodDeiT-Base: CalibrationInversionEliminationTotalDeiT-III-Huge: CalibrationInversionEliminationTotal
MAG–––0.10 ± 0.01–––0.64 ± 0.04
SNIP0.43 ± 0.10––2.2 ± 0.11.2 ± 0.0––3.4 ± 0.1
VBP12.4 ± 0.3––16.9 ± 0.567.4 ± 0.8––72.4 ± 1.1
DVBP14.0 ± 0.10.20 ± 0.01–25.6 ± 0.499.1 ± 13.41.6 ± 0.1–176.8 ± 15.5
CORP14.5 ± 2.70.22 ± 0.03–17.4 ± 2.6112.9 ± 26.31.7 ± 0.1–121.5 ± 26.6
GRAIL13.9 ± 1.70.33 ± 0.01–17.3 ± 1.9105.7 ± 6.62.8 ± 0.1–116.4 ± 6.6
CARVE_L15.1 ± 1.90.88 ± 0.0323.8 ± 3.543.1 ± 1.8119.0 ± 19.97.4 ± 0.5103.6 ± 6.6240.7 ± 19.1
CARVE19.8 ± 8.50.95 ± 0.1022.1 ± 1.756.9 ± 8.8104.1 ± 8.17.5 ± 0.5109.4 ± 11.0277.9 ± 25.4

In the revised version, we have added a new appendix (Appendix S) that compares the end-to-end compression time of every method under the same protocol.

We thank the reviewer again for the constructive questions on the calibration protocol, inference speed, and compression cost. We hope that our response and the corresponding additions to the revised paper (Appendices Q, R and S) have addressed them.

Best regards,

The Authors

Comments on the paper

1 comment

We thank the Area Chairs and reviewers for their careful and constructive feedback. We have uploaded a revised version of the paper, with all changes highlighted in blue. The main revisions directly address the concerns raised during review:

  • Evaluation beyond image classification (Reviewer AaLh; new Appendix T).
  • Evaluation beyond language-model perplexity (Reviewer AaLh; extended Appendix M).
  • Practical inference efficiency (Reviewer GNtp; Table 3 and new Appendix R).
  • End-to-end pruning cost (Reviewers GNtp and AaLh; Appendix S, Appendix O).
  • Clarification of the label-free calibration protocol (Reviewer GNtp; Appendix Q).
  • Clarification of the relationship to OBS and OBC (Reviewer C6Yd).

We hope that these revisions address the reviewers' questions and concerns. We thank the reviewers again for their constructive suggestions, which helped us improve the clarity, scope, and experimental support of the paper.

Sincerely,

The Corresponding authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.