acceptodds
Withdrawn from ICLR 2027

Neural Darwinism: A Theoretical Framework for Representation Evolution in Convolutional Neural Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce a Darwinian framework that mathematically formalizes representation evolution in deep networks, viewing each neuron as an adaptive entity competing for survival during training. In this perspective, learning is governed by a unified Darwinian Score that reflects three essential dimensions of neuronal fitness—informational diversity, functional contribution, and temporal adaptability. This score induces a principled constrained optimization objective that balances model compactness with predictive fidelity, supported by new approximation guarantees showing that preserving high-fitness neurons retains the network’s functional capacity. We then operationalize this framework through Neural Darwinism Culling (NDC), which serves as a practical instantiation of the Darwinian Score. NDC dynamically removes neurons with persistently low fitness while allowing high-value neurons to specialize. NDC captures the intrinsic evolutionary dynamics of neural representations: neurons with collapsed activations, negligible causal impact on loss reduction, or stagnant parameter trajectories are pruned, whereas differentiated and adaptable neurons are retained. This yields pruning decisions that are interpretable, layer-aware, and aligned with the competitive pressures naturally emerging across network depth. Experiments across diverse methodological settings demonstrate that NDC, as a direct application of the Darwinian Score, achieves substantially higher sparsity with improved generalization compared to SOTA methods, particularly under extreme compression. Ablations further confirm that the Darwinian Score is the key driver of these gains. Overall, our work provides both a general evolutionary lens for understanding representation dynamics and a practical, theory-grounded path toward efficient and adaptive deep learning.

Primary area:
learning theory
Keywords:
Neural Darwinism, Representation Evolution, Convolutional Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nhkrOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper scores each neuron with a “Darwinian Score”. The score is a weighted geometric mean of three parts, each normalized within its layer:

  • NDE: entropy of the neuron’s activation histogram plus its JS divergence from the layer’s average histogram.
  • AGC: activation × gradient saliency.
  • NAS: a score based on how the neuron’s weights moved during training.

NDC then prunes the lowest-scoring neurons in each layer. Experiments run from MNIST to ImageNet and report large gains at high sparsity.

Critical strengths
  • The scoring rule is simple, and it combines three complementary signals. The robust normalization and the “must score well on all three” aggregation are deliberate, explained choices.
  • Experimental coverage is broad: MLP up to DenseNet-121, Tiny-ImageNet, MobileNet-V2 on ImageNet, and both structured and unstructured sparsity.
Critical weaknesses
  1. The theory basically assumes the conclusion. Theorem 3.6 relies on assumption (12): removing neurons perturbs a layer by at most κl\kappa_\mathcal{l} times their normalized score. That is essentially the conclusion. Nothing in it is specific to this score, κl\kappa_\mathcal{l} is never estimated, and the bound is never evaluated numerically. Lemma 3.5 does the same thing through Eq. (8).

  2. The method is underspecified and contradicts itself. The abstract describes removing persistently weak neurons during training. Alg. 1 is a single pass after training with no persistence rule, and its NAS leaves out the r factor from Definition 3.3. NDC is defined on neurons, yet most tables report weight sparsity, with no explanation of how a neuron score produces it.

  3. Reproducibility. No hyperparameter values or sensitivity analysis are reported (μ,B,α,δfloor,ws,η)(\mu, B,\alpha,\delta_{floor},w_s,\eta) and others. Most results are single runs.

Questions for authors

How are neural sparsity and weight sparsity defined? Please give parameter and FLOP counts for Tables 1, 11 and 15. Getting 93.76% at 90% neural sparsity is hard to square with 89.79% at 98% weight sparsity, because removing 90% of channels removes far more than 90% of weights.

Reviewer_i2sNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a Darwinian Score for ranking neurons or CNN channels using activation-histogram entropy and divergence (NDE), activation–gradient magnitude (AGC), and directional weight-trajectory activity (NAS). Layerwise normalization and multiplicative aggregation combine these signals, and Neural Darwinism Culling (NDC) retains high-scoring channels. The paper develops conditional preservation and stability arguments and evaluates pruning on MNIST, CIFAR-10, Tiny-ImageNet, and ImageNet. Its intended contribution is a unified criterion connecting representation diversity, task contribution, and training dynamics to channel selection, supported by high-sparsity results and component ablations.

Critical strengths
  1. The combined criterion has relevant empirical support. Figure 2 compares removal of high-scoring, random, and low-scoring channels, supporting a relationship between DS rankings and functional importance in that experiment. Table 9 includes every single-component and pairwise variant, with full NDC performing best at all reported sparsities. These results support the usefulness of combining the signals, even though they do not establish the proposed theoretical interpretation or isolate the terms within NDE.

  2. The evaluation covers several architectures and substantial compression. The experiments span ResNet, VGG, DenseNet, MobileNet-V2, and an MLP, across four datasets. Reported results such as 55.90% ImageNet accuracy at 90% sparsity indicate potentially useful performance under aggressive pruning, subject to clarifying the comparison protocol. The training-duration and Leaky ReLU studies additionally examine whether the method's behavior persists under changes to optimization length and activation function.

Critical weaknesses
  1. The theoretical justification is not yet specific enough to the proposed score. Theorem A.1 assumes in Equation (12) that discarded normalized importance controls the effect of removing any channel subset; this relation, with useful constants, is not established from NDE, AGC, and NAS. Table 9 supports NDE as a component but does not separate entropy from the JS term, whose distance from the layer-average histogram alone need not indicate useful or non-redundant features. Two derivations also require clarification: Equations (13)–(14) introduce an additional activation-bound factor A after Equation (12) already bounds the activation perturbation, which can reduce the claimed bound when A < 1; Equation (32) appears to omit the population-shift term retained in Section A.1.5. These statements may be repairable by correcting the scaling and restricting the stability claim, but the score-specific link would still need justification or a narrower interpretation.
  2. The comparison protocol and implemented pruning procedure are insufficiently specified. The twenty baseline entries in Table 4 match UniPTS Table 1's MobileNet-V2 results, obtained in a limited-data post-training setting using 10,240 ImageNet images. The paper does not establish whether NDC uses the same data, starting checkpoint, and optimization budget. Consequently, the large margins cannot yet be interpreted as a controlled advantage; this does not establish that the comparison was actually unfair. UniPTS, Section 4.1 and Table 1. Moreover, Equation (3) includes a displacement ratio omitted from Algorithm 1. The introduction describes progressive pruning during training, whereas the algorithm starts from a trained network and ends with optional fine-tuning. Section A.3.1 and the epoch ablation do not supply the concrete optimization/pruning schedule or default E. These gaps make it difficult to determine which procedure and budget produced the results.
Questions for authors
  1. Can the authors justify contribution comparability for the actual DS with useful constants, or clarify the conditional scope of the theoretical contribution? Please explain the extra A factor in Equations (13)–(14), including A < 1, and reconcile Equation (32) with Section A.1.5.
  2. Did NDC's Table 4 results use the same 10,240-image post-training setting and starting checkpoint as the listed baselines? Please document data access, optimization steps, dense accuracy, and variation across runs. If budgets differ, what existing evidence supports an advantage under matched conditions?
  3. Which pruning schedule and NAS definition generated the results? Please specify the use of the displacement ratio, statistics-collection and mask-update times, layer-budget allocation, and default E. Please report the values and selection rules for μ, α, σ, and the component weights wsw_s, and clarify the value and role of η in the implementation. What were the NAS checkpoint cadence/window and AGC sample count and aggregation across batches? This request primarily concerns the configuration and selection procedure used to produce the reported results.
Reviewer_kLG2OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a pruning method inspired by evolutionary selection. Each neuron receives a score combining several signals like diversity of its activations and the change in its weights during training. Neurons with low scores are removed and then the network is finetuned. The authors also provide a theorem bounding the error after pruning. The results cover mainly CIFAR-10, Tiny-ImageNet, and MNIST against several pruning methods.

Critical strengths
  • The topic is timely and an interesting research direction.
  • The evaluation covers several architectures, datasets, and sparsity levels.
  • The ranking analysis in Section 4.1 is interesting and suggests that the proposed score does somewhat capture channel importance.
Critical weaknesses
  • The manuscript fail to clearly position the novelty compared to previous works. Existing approaches were listed without explaining precisely how NDC differs from them.
  • In my opinion, the whole evolutionary framing does not fully match the proposed method. The paper describes the method in terms of variation, competition, and selection. But the proposed algorithm just takes a trained network, computes a fixed score for each neuron and then keeps the top-scoring neurons in each layer. There is no population or selection over generations as far as I can tell. So basically the method could be viewed as just a combination of well established techniques.
  • Furthermore, the theoretical results do not convincingly support the method. Lemma 3.5 and Theorem 3.6 assume that the proposed importance scores control the effect of removing neurons which is a very strong assumption and actually what needs to be established to actually justify the approach.
  • I think the empirical comparisons are not fair since most baselines prune at or near initialization, whereas the proposed starts from a fully trained network.
Questions for authors
  1. Were all baselines evaluated using the same training and fine-tuning setup?
  2. How does NDC compare to standard pruning criteria such as magnitude,or movement pruning?
  3. Is pruning performed during training, as described in Sections 1 and 3, or after training, as in Algorithm 1?
  4. How is the contribution-comparability assumption in Theorem 3.6 justified for the proposed Darwinian Score?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.