- Rating
- 2
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper scores each neuron with a “Darwinian Score”. The score is a weighted geometric mean of three parts, each normalized within its layer:
- NDE: entropy of the neuron’s activation histogram plus its JS divergence from the layer’s average histogram.
- AGC: activation × gradient saliency.
- NAS: a score based on how the neuron’s weights moved during training.
NDC then prunes the lowest-scoring neurons in each layer. Experiments run from MNIST to ImageNet and report large gains at high sparsity.
Critical strengths
- The scoring rule is simple, and it combines three complementary signals. The robust normalization and the “must score well on all three” aggregation are deliberate, explained choices.
- Experimental coverage is broad: MLP up to DenseNet-121, Tiny-ImageNet, MobileNet-V2 on ImageNet, and both structured and unstructured sparsity.
Critical weaknesses
-
The theory basically assumes the conclusion. Theorem 3.6 relies on assumption (12): removing neurons perturbs a layer by at most times their normalized score. That is essentially the conclusion. Nothing in it is specific to this score, is never estimated, and the bound is never evaluated numerically. Lemma 3.5 does the same thing through Eq. (8).
-
The method is underspecified and contradicts itself. The abstract describes removing persistently weak neurons during training. Alg. 1 is a single pass after training with no persistence rule, and its NAS leaves out the r factor from Definition 3.3. NDC is defined on neurons, yet most tables report weight sparsity, with no explanation of how a neuron score produces it.
-
Reproducibility. No hyperparameter values or sensitivity analysis are reported and others. Most results are single runs.
Questions for authors
How are neural sparsity and weight sparsity defined? Please give parameter and FLOP counts for Tables 1, 11 and 15. Getting 93.76% at 90% neural sparsity is hard to square with 89.79% at 98% weight sparsity, because removing 90% of channels removes far more than 90% of weights.