Layer Activation Schedule: Novel Method for Efficient Structuring and Pruning of Neural Networks Using Monotonic Gating

Deep neural networks typically exhibit highly unstructured and entangled internal representations, which obscures interpretability, necessitates external regularization, and limits the hardware efficiency of post-training pruning due to irregular sparsity patterns. We introduce the Layer Activation Schedule, an end-to-end differentiable gating framework that imposes a strict structural hierarchy on neural network representations during training. By modulating pre-activations with a monotonically decaying survival probability mask, our method forces robust core features to accumulate in high-probability (leading) dimensions, while relegating highly specific details and noise to low-probability (trailing) dimensions. This self-organizing topology provides intrinsic regularization, yields highly interpretable, importance-ordered feature maps, and transforms network sparsification into a trivial structural truncation of trailing dimensions. Because representations are structurally sorted, pruning yields dense, smaller matrices without requiring unstructured masks or subsequent fine-tuning. Furthermore, our framework naturally produces a continuous family of nested sub-networks within a single trained model, enabling dynamic adjustments to computational capacity based on real-time deployment constraints without retraining. We validate our approach on ResNet and Transformer architectures, demonstrating massive reductions in parameters and Multiply-Accumulate operations (MACs) while maintaining competitive accuracy. For GPT-2 medium trained under constrained training budgets, our approach maintains good-level generation quality after removing 92.8% of FLOPs, extending the breaking point by 2.7$\times$ over the strongest baseline. On ResNet-50, it preserves 64.62% Top-1 accuracy at a 60.47% MAC reduction. Our method significantly outperforms traditional pruning techniques under strict computational constraints, offering a efficient paradigm for scalable AI deployment.