acceptodds
Under review as a conference paper at ICLR 2027

Layer Activation Schedule: Novel Method for Efficient Structuring and Pruning of Neural Networks Using Monotonic Gating

Abstract

Deep neural networks typically exhibit highly unstructured and entangled internal representations, which obscures interpretability, necessitates external regularization, and limits the hardware efficiency of post-training pruning due to irregular sparsity patterns. We introduce the Layer Activation Schedule, an end-to-end differentiable gating framework that imposes a strict structural hierarchy on neural network representations during training. By modulating pre-activations with a monotonically decaying survival probability mask, our method forces robust core features to accumulate in high-probability (leading) dimensions, while relegating highly specific details and noise to low-probability (trailing) dimensions. This self-organizing topology provides intrinsic regularization, yields highly interpretable, importance-ordered feature maps, and transforms network sparsification into a trivial structural truncation of trailing dimensions. Because representations are structurally sorted, pruning yields dense, smaller matrices without requiring unstructured masks or subsequent fine-tuning. Furthermore, our framework naturally produces a continuous family of nested sub-networks within a single trained model, enabling dynamic adjustments to computational capacity based on real-time deployment constraints without retraining. We validate our approach on ResNet and Transformer architectures, demonstrating massive reductions in parameters and Multiply-Accumulate operations (MACs) while maintaining competitive accuracy. For GPT-2 medium trained under constrained training budgets, our approach maintains good-level generation quality after removing 92.8% of FLOPs, extending the breaking point by 2.7 over the strongest baseline. On ResNet-50, it preserves 64.62% Top-1 accuracy at a 60.47% MAC reduction. Our method significantly outperforms traditional pruning techniques under strict computational constraints, offering a efficient paradigm for scalable AI deployment.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Structured Neural Networks, Structured Pruning, Model Compression, Dynamic Neural Networks, Representation Learning, Efficient Inference, Intrinsic Regularization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6d3QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Layer Activation Schedule trains networks with fixed, non-increasing survival probabilities so that trailing dimensions can be structurally removed without fine-tuning. It combines parameter slicing with either independent Bernoulli gating or contiguous-prefix sampling. Intra-Block Gating narrows internal dimensions; Extended Stream Gating also compresses the Transformer residual stream, while ResNet block outputs are scattered into full-width residual tensors. Experiments with GPT-2 medium on OpenWebText and ResNet-50 on ImageNet-1K report improved pruning robustness over the evaluated baselines, including MAUVE within 20% of the dense score at 92.8% FLOPs removed and 64.62% Top-1 accuracy at 60.47% MAC reduction. Activation statistics show greater concentration in leading channels.

Critical strengths
  1. A practical formulation for extracting smaller dense subnetworks without fine-tuning. The slicing rules and two gating variants explain how the method handles different layer types and residual connections, making the deployment mechanism concrete.

  2. Strong reported pruning robustness and evidence of activation concentration. At the interpolated 60% FLOP-reduction budget, Extended Stream achieves WikiText-2 perplexity 38.6 versus approximately 67,000 for SparseGPT. ResNet Intra-Block retains 67.94% accuracy at 55.44% MAC reduction versus 1.89% for the reported baseline. However, unpruned accuracy is lower: 70.49%/70.37% for Intra-Block/Extended Stream versus 72.02% for the baseline. Across 16 blocks, the mean share of activation power in the first 10% of channels is 37.9% for Extended Stream versus 9.9% for the baseline. This supports concentration, but does not establish a strict ordering of task importance.

Critical weaknesses
  1. The contribution over closely related training methods is insufficiently established. The prefix sampler closely resembles Nested Dropout and Ordered Dropout in FjORD; Universally Slimmable Networks and MatFormer also learn variable-width or nested models. Several predecessors are acknowledged, but none is experimentally compared. The independent sampler and architectural adaptations may differ, yet comparisons mainly against post-training pruning do not isolate their incremental benefit. A representative training-time comparison, constant-probability gating control, and compact dense models at matched inference budgets are needed. GPT-2 optimization also differs, the dense baseline uses peak learning rate 10−410^{-4} and clipping norm 0.1, versus 3×10−43\times10^{-4} and 1.0 for scheduled models. The reported baseline divergence at the higher learning rate justifies considering different settings, but controlled comparisons over stable settings and comparable tuning budgets are still needed. The ResNet baseline pruning operator and different curves across the two architectural comparisons also require clarification; different pruning scopes could explain them.

  2. Quality-preservation and efficiency claims exceed the supporting evidence. The 92.8% headline uses a 20% tolerance on MAUVE, which does not establish general preservation of language-model quality. At 60% FLOPs removed, Extended Stream’s MAUVE is 0.932 versus 0.942 for the dense reference, while WikiText-2 perplexity rises from 25.5 to 38.6, LAMBADA perplexity from 10.1 to 21.2, and LAMBADA accuracy falls from 0.410 to 0.303. The MAUVE evaluation settings and uncertainty are insufficiently documented. Figure 20 explicitly discloses an Intra-Block rebound from 0.015 at 82.4% removal to 0.332 at 87.2%, further motivating repeated evaluation. Table 1 describes a nine-task zero-shot mean of 0.510 for the dense model, while Appendix D/Table 3 describe five tasks. The five displayed values average to (0.440+0.379+0.661+0.783+0.690)/5=0.5906(0.440+0.379+0.661+0.783+0.690)/5=0.5906. Missing results or a corrected aggregation/description are needed; this discrepancy alone does not identify which value is wrong. Checkpoint-specific schedule/sampler choices and their sensitivity remain unclear despite the provided defaults. Accelerated training and hardware-efficiency claims also lack measured throughput, latency, and peak memory, particularly relevant to ResNet Extended Stream’s full-width residual tensors and scatter operations.

Questions for authors
  1. Which tasks and averaging rule produce Table 1’s zero-shot scores, and how should the five-versus-nine-task discrepancy be resolved?

  2. For each checkpoint, what schedule family, α\alpha, segment parameters, minimum probability, and sampler were used, and how were they selected? Were the samplers compared, given that matching marginals does not match their joint distributions? For the default ramp, α≤0.5\alpha\leq0.5 makes all probabilities one after scaling/clipping. How sensitive are results to these choices, and do configurations transfer between architectures without retuning?

  3. What selection rule and layer scope define each ResNet baseline? Do the different curves use the same checkpoint with different pruning scopes, and which is shown in Figure 6? Are BatchNorm statistics sliced or recalibrated, and is probability scaling retained after truncation?

  4. What MAUVE data, sample counts, generation lengths, decoding settings, feature model, and uncertainty estimates were used? Please report the other metrics at the 92.8% operating point and explain whether this threshold is measured or interpolated, including sensitivity to randomness and the chosen tolerance.

  5. Can the authors provide the training-time and compact-model comparisons above, plus constant-probability gating at comparable expected active width? Training and tuning budgets should be comparable, and optimization controls should account for the reported baseline divergence.

  6. What training throughput, inference latency and throughput, and peak memory are achieved at representative pruning ratios with specified hardware, batch sizes, and sequence lengths? Please include slicing and scatter overhead and explain FLOP accounting across pruning formats, including UniPruning (2:4) beyond 50% reported FLOP removal.

5 comments

We sincerely thank the reviewer for the constructive feedback, and for the insightful questions. Your feedback helped us identify and resolve several important presentation gaps in the revised manuscript.

Below, we address each of your questions and suggestions in detail.


Responses to Critical Weaknesses

Weakness 1: Contribution Over Related Training-Time Methods, Matched Optimization Controls, and ResNet Baseline Clarification

We have added a detailed comparison with Nested Dropout and Ordered Dropout in Appendix A ("Comparison with Prior Ordered and Nested Representation Methods", including Subsections A.1–A.3: "Flexible Continuous Survival Schedule vs. Rigid Profiles", "Generalization to Deep Residual Architectures (Intra-Block and Extended Stream Gating)", and "Direct Empirical Comparison of Training and Validation Losses") and reported the ResNet-50 baseline results in Appendix H ("ResNet-50 Dense Evaluation, Uniform Pruning Table and Per-Variant Budget Bars", Table 8 and Figures 45–46). Please note that the full training of Nested Dropout and Ordered Dropout is still in progress; once training is completed, we will evaluate them across the full benchmark suite and include the final results in the manuscript. At the current checkpoints, both of our scheduled variants already achieve lower training and validation losses than Ordered Dropout and Nested Dropout while training roughly 1.5×1.5\times faster than Nested Dropout.

Second, we trained our scheduled models using the exact same optimization hyperparameters as the dense baseline (ηmax⁡=1×10−4\eta_{\max} = 1 \times 10^{-4} and gradient clip 0.10.1; Appendix A.3, Tables 2–4). Under these baseline parameters, the validation loss of our scheduled models shows only a minor difference (≈0.01–0.05\approx 0.01\text{--}0.05) compared to our standard setting, while still achieving lower losses than Ordered Dropout and Nested Dropout (at every checkpoint for Intra-Block, and at 50k and 100k steps for Extended Stream).

Finally, regarding the ResNet-50 baseline: both Baseline curves come from the exact same unscheduled ResNet-50 model (72.02%72.02\% Top-1), evaluated under the two different pruning scopes corresponding to our two architectural variants (using the same positional trailing-index truncation 1…N′1 \dots N' without saliency sorting):

  • Intra-Block pruning scope: truncates only the internal bottleneck width CwC_w. At a 50%50\% drop ratio, this removes 55.44%55.44\% of MACs (1.89%1.89\% Baseline Top-1 vs. 67.94%67.94\% ours).
  • Extended Stream pruning scope: truncates both the internal width CwC_w and the block expansion output 4Cw4C_w. At a 50%50\% drop ratio, this removes 60.47%60.47\% of MACs (0.90%0.90\% Baseline Top-1 vs. 64.62%64.62\% ours).

Weakness 2: Scope of Quality-Preservation Claims, MAUVE Protocol, Zero-Shot Task Discrepancy, Checkpoint Settings, and Hardware Efficiency

We have updated and added comparisons across the relevant sections: we have clarified MAUVE quality claims, Appendix D.1 ("Activation Schedule Parameters") for the exact checkpoint configurations, Appendix E ("Additional GPT-2 Pruning Results", Table 6) for all nine zero-shot tasks, and Section 4.4 ("Hardware Utilization and Inference Speed-Up") alongside Appendix M ("Hardware Utilization and Inference Speed-Up Details", Tables 17–18) for the hardware benchmarks. The MAUVE evaluation settings and the Intra-Block rebound (now Figure 22 in Appendix F) are detailed in our answer to Question 4 below.

The results show that all four main checkpoints share the exact same schedule configuration without retuning. In the pervious version, we included five representative tasks in Table 3 for conciseness; Now, the full nine-task zero-shot average has been added, and physical truncation yields up to a 2.71×2.71\times decode speed-up and 89.8%89.8\% peak memory reduction on GPT-2 medium and up to a 2.86×2.86\times speed-up and 81.0%81.0\% weight memory reduction on ResNet-50.

Responses to Questions for Authors

Question 1: Zero-Shot Tasks, Averaging Rule, and Resolving the Five-vs.-Nine-Task Discrepancy

We have added the complete nine-task breakdown in Appendix E ("Additional GPT-2 Pruning Results", Table 6). The results show that Table 1's zero-shot score is the unweighted arithmetic mean across all nine zero-shot tasks (ARC-Easy, ARC-Challenge, HellaSwag, PIQA, WinoGrande, OpenBookQA, BoolQ, SciQ, and COPA), using length-normalised accuracy for ARC-Easy, ARC-Challenge, HellaSwag, PIQA, and OpenBookQA and plain accuracy for the other four (as stated in the Table 6 caption). The initial submission displayed only five of these tasks to save space; averaging all nine dense values gives exactly 0.5100.510, and our scheduled models outperform all baselines at 40% and 60% FLOPs removed.


Question 2: Checkpoint Configurations, Sampler Comparison, α\alpha Parameterization, and Sensitivity

We have added the full configuration details in Appendix D.1 ("Activation Schedule Parameters"). All four scheduled checkpoints use the same 3-segment piecewise linear schedule (ρ=[0.1,0.7,0.2]\rho = [0.1, 0.7, 0.2], decaying from 1.01.0 to 0.50.5, α=1.0\alpha = 1.0, pmin⁡=0p_{\min} = 0, mean active width 0.7250.725), fixed a priori and transferred between ResNet-50 and GPT-2 medium without retuning. The sampler is independent Bernoulli for ResNet-50 and contiguous prefix (slice) for GPT-2.

  • Samplers: as you note, the two samplers have identical marginals pip_i but different joint distributions. Independent Bernoulli sampling varies which subset of units co-activates on each step, whereas prefix sampling enforces strict nesting (m1≥m2≥⋯≥mNm_1 \ge m_2 \ge \dots \ge m_N) on every forward pass. We first developed the Bernoulli sampler and used it to train ResNet-50. On GPT-2, non-contiguous index gathering and scattering across the large hidden dimensions made it prohibitively slow, so we introduced the prefix sampler, which keeps memory accesses contiguous and matches the contiguous truncation operator used at deployment. We have not yet run a matched head-to-head comparison of the two samplers on the same architecture; the ≈1.5×\approx 1.5\times training speed-up reported in Appendix A.3 is relative to Nested Dropout, not to the Bernoulli sampler.
  • α\alpha parameterization: you are right that with a floor of 0.50.5, any α≤0.5\alpha \le 0.5 saturates all survival probabilities to 1.01.0 after scaling and clipping, so the non-degenerate range for our default ramp is α∈(0.5,1]\alpha \in (0.5, 1]. We introduced α∈(0,1]\alpha \in (0, 1] for schedule families whose trailing probabilities decay towards zero (such as the harmonic schedule, whose last segment ends at 1K+1\frac{1}{K+1}).
  • Sensitivity: during development, we evaluated several schedule families (harmonic, piecewise linear, and inverse-prime), segment counts Nmax⁡∈{2,…,10}N_{\max} \in \{2, \dots, 10\}, and scaling factors α∈[0.1,1.0]\alpha \in [0.1, 1.0] on smaller benchmarks (CIFAR-10 and EMNIST). Smoothly decaying profiles with a non-zero trailing floor consistently preserved high prunability (80–90%80\text{--}90\% unit removal within a 1%1\% accuracy drop) while avoiding representation collapse. Because this design space is a research direction in its own right, we are consolidating a comprehensive parameter and capacity study into a dedicated follow-up manuscript.

Question 3: ResNet-50 Baseline Definition, Pruning Scopes, BatchNorm Statistics, and Probability Scaling

The pruning operator is described in Section 3.4 ("Structured Pruning of the Scheduled Network") and Appendix C ("Detailed Formulation of Structured Pruning"), the inference-time probability scaling in Section 3.1 (paragraph "Inference Phase"), and the per-scope baseline results in Appendix H ("ResNet-50 Dense Evaluation, Uniform Pruning Table and Per-Variant Budget Bars", Table 8 and Figures 45–46).

  • Selection rule and layer scope: as explained in our answer to Weakness 1, both Baseline curves use the same unscheduled checkpoint (72.02%72.02\% Top-1) and the same positional trailing-index truncation (1…N′1 \dots N') as our scheduled models. The Intra-Block scope truncates only the internal width CwC_w, and the Extended Stream scope truncates both CwC_w and the block expansion output 4Cw4C_w. Figure 6 plots both scheduled models against the Intra-Block-scope Baseline, and Figures 45–46 show each variant against the Baseline under its matching scope.
  • BatchNorm statistics: no running statistics are tracked, so there is nothing to slice or recalibrate. Because active channel widths vary stochastically during training, all BatchNorm layers are built with track_running_stats=False; at evaluation, channels are normalized with batch statistics, identically for the Baseline and the scheduled checkpoints. The affine parameters (γ,β\gamma, \beta) are physically sliced to the truncated width N′N' (Appendix C).
  • Probability scaling after truncation: yes, it is retained. In ResNet-50, the deterministic schedule pp scales the output of each scheduled BatchNorm layer at inference (Section 3.1). When instantiating the physically truncated dense model, this static vector is folded into the sliced affine parameters (γ′=γ1:N′⊙p1:N′\gamma' = \gamma_{1:N'} \odot p_{1:N'} and β′=β1:N′⊙p1:N′\beta' = \beta_{1:N'} \odot p_{1:N'}), requiring zero extra runtime operations. In GPT-2, no probability scaling is applied during either training or inference.

Question 4: MAUVE Protocol, Uncertainty, the 92.8% Operating Point, and Measured vs. Interpolated Metrics

We have clarified the MAUVE criterion and report the full-sweep metrics in Section 4.1 ("Effective Pruning at Extreme Sparsity", Table 1), Appendix E ("Additional GPT-2 Pruning Results", Tables 5–6), Appendix F ("GPT-2 Pruning Curves: Structured Methods"), and Appendix G ("GPT-2 Pruning Curves: Semi-Structured and Unstructured Methods").

  • MAUVE settings: N=100N = 100 passages from the OpenWebText validation split, using the first 3535 tokens as the prompt; 256256 continuation tokens generated by ancestral sampling at temperature T=1.0T = 1.0 (top_k = None, seed 13371337); gpt2 as the feature model, with features quantized into 1010 clusters by MAUVE's default kk-means setting (⌊N/10⌋\lfloor N/10 \rfloor).
  • Measured vs. interpolated: the 92.8%92.8\% operating point is interpolated rather than a single measured configuration. It is the budget at which Extended Stream's MAUVE permanently drops below 80%80\% of the dense score (0.8×0.942≈0.750.8 \times 0.942 \approx 0.75). We take the first measured configuration below this threshold that is never followed by one above it, and linearly interpolate the crossing against the measured configuration before it. A noisy dip that recovers therefore does not end a curve, and a spike after a collapse cannot extend it.
  • Sensitivity to the tolerance: reading the same crossing at a stricter (90%90\% of dense) and a looser (55%55\% of dense) threshold keeps Extended Stream's breaking point within 92–95%92\text{--}95\% FLOPs removed and the strongest baseline's within 30–38%30\text{--}38\%, so the ranking does not depend on the chosen tolerance. At 60% FLOPs removed, all baselines are already at MAUVE≤0.033\text{MAUVE} \le 0.033 (Table 1).
  • The Figure 22 rebound (Figure 20 in the initial submission): at 87.2%87.2\% FLOPs removed, Intra-Block has already collapsed (validation loss 72.9172.91), and the severely truncated model generates erratic, degenerate text. In this regime, a clustering-based metric computed from N=100N = 100 samples and 1010 clusters becomes noisy and unreliable, which produces the spurious 0.3320.332 reading after MAUVE had already fallen to 0.0150.015 at 82.4%82.4\%. As stated in the Figure 22 caption, this point is not plotted, and it does not affect any breaking point.
Reviewer_TysQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work proposes Layer Activation Schedule which is an end-to-end differentiable gating method which forces a structural hierarchy on network representations ordered by importance during training. The method works by applying a monotonically decaying survival probability mask which enforces the network to place robust core features in the leading dimension while placing specifics and noise to later dimensions. The approach is evaluated on GPT-2 medium, and ResNet0-50. Because features are ordered by importance, pruning becomes a truncation of the later dimensions with no complex masks or finetuning required. This approach outperforms standard baselines at extreme sparsity, producing fully dense, nested sub-networks that maintain competitive accuracy.

Critical strengths
  1. The method provides a simple yet effective approach to pruning moving away from a computationally heavy post-hoc optimisation problem to a structural bias which is explicitly enforced during training. By forcing representations into a strict importance hierarchy, pruning is instead implemented as a simple tensor slice operation. This is highly impactful from my understanding, if it holds it entirely eliminates the inefficiencies of unstructured masks and the overhead of iterative fine-tuning by proposing an effective approach for existing hardware and implementations.
  2. The empirical studies also support the claim made, the performance on significant FLOP reduction on GPT-2 is impressive, and vastly outperforms benchmarked approaches. Specifically, the method maintains dense-level generation quality up to a 92.8% FLOP reduction, where the performance also holds on evaluation downstream tasks, and across the board the proposed avoided degenerate representations compared to standard baselines at 67% FLOP reduction.
  3. The paper is generally well written and presented throughout, clear use of figures and tables, and a logical narrative flow that makes it easily interpretable.
Critical weaknesses
  1. The fundamental concept of enforcing a dimensional importance hierarchy to enable nested sub-networks closely aligns to prior works that you identify (Nested Dropout, Ordered Dropout, and Matryoshka Representation Learning). While the proposed approach is a novel mechanism, you fail to empirically compare the method against these related training-time hierarchical techniques. Perhaps naive of me, but as the evaluations rely almost exclusively on comparisons to post-hoc pruning methods, it is impossible to determine if this specific scheduling approach offers a empirical improvement over existing hierarchical representation methods.
  2. The authors state that their GPT-2 models were trained on a constrained budget, which they do in fact admit prevents standard models from naturally structuring their representations and causes baselines to fail. This is a critical issue in the empirical evaluation in that it makes it impossible to determine whether the performance gap is an inherent advantage of the proposed schedule, or an result of under-training the dense baseline models. In addition, there is limited sensitivity analysis in the main text regarding the sensitivity to the chosen scheduling parameters and how the final performance is effected by scheduling parameter choice.
Questions for authors
  1. How sensitive is the networks final accuracy and sparsity to the exact shape of the decay schedule and the scaling parameter α\alpha? Do these require extensive tuning to prevent representation collapse in the trailing dimensions during training?
  2. While this approach eliminates the need for fine-tuning, is there a way or a conjectured benefit to applying this method to fine-tuning on an already-trained model? If not, is there a reasonable extension or approach that would allow your proposed method to be leveraged in this case?
4 comments

We sincerely thank the reviewer for the constructive feedback, and for the insightful questions. Your comments have helped us clarify and strengthen the presentation of our work.

Below, we address each of your questions and suggestions in detail.


Responses to Critical Weaknesses

Weakness 1: Comparison with Training-Time Hierarchical Methods (Nested Dropout, Ordered Dropout, and MRL)

We have added a direct comparison in Appendix A ("Comparison with Prior Ordered and Nested Representation Methods", including Subsections A.1–A.3: "Flexible Continuous Survival Schedule vs. Rigid Profiles", "Generalization to Deep Residual Architectures (Intra-Block and Extended Stream Gating)", and "Direct Empirical Comparison of Training and Validation Losses"). The results show that both of our scheduled variants consistently achieve lower training and validation losses than Ordered Dropout and Nested Dropout across all evaluated checkpoints, while training roughly 1.5×1.5\times faster than Nested Dropout. Please note that the full training runs for Nested Dropout and Ordered Dropout are still in progress; once training finishes, we will perform the complete benchmarking suite and update the manuscript.


Weakness 2: GPT-2 Training Budget, Baseline Convergence, and Hyperparameter Sensitivity

We have clarified the training budget, matched-hyperparameter controls, and schedule configuration details in Appendix A.3 ("Direct Empirical Comparison of Training and Validation Losses", Tables 2–4) and Appendix D ("Detailed Experimental Setup", including Subsections D.1 "Activation Schedule Parameters" and D.2 "GPT-2"), and we have removed the unclear phrasing about a constrained budget. Our dense baseline is not under-trained:

  • Training budget: as specified in Appendix D.2, every GPT-2 medium model is trained for 200,000200{,}000 steps at 480×1024=491,520480 \times 1024 = 491{,}520 tokens per step, i.e. 98.3B98.3\text{B} tokens. The Chinchilla compute-optimal budget for 354M354\text{M} parameters is ≈7.1B\approx 7.1\text{B} tokens (2020 tokens per parameter), so our models are trained for ≈14×\approx 14\times that budget.

  • Baseline convergence: our unpruned dense baseline reaches an OpenWebText validation loss of 2.682.68 (Appendix E, Table 5), which outperforms the gpt2-medium checkpoint trained on the same dataset at 2.842.84. When we extended baseline training beyond 200k200\text{k} steps (up to 500k–600k500\text{k}\text{--}600\text{k} steps), the validation loss plateaued.

  • Matched hyperparameters: under the dense baseline's conservative settings (ηmax⁡=1×10−4\eta_{\max} = 1 \times 10^{-4}, gradient clip 0.10.1), the validation loss of our scheduled models changes by only ≈0.01–0.05\approx 0.01\text{--}0.05, and they still outperform Ordered Dropout and Nested Dropout (Intra-Block at every checkpoint, Extended Stream at 50k and 100k steps; Appendix A.3, Tables 2–4).

  • Sensitivity: during development, we evaluated several schedule families (harmonic, piecewise linear, and inverse-prime), segment counts Nmax⁡∈{2,…,10}N_{\max} \in \{2, \dots, 10\}, and scaling factors α∈[0.1,1.0]\alpha \in [0.1, 1.0] on smaller benchmarks (CIFAR-10 and EMNIST). Smoothly decaying profiles with a non-zero trailing floor consistently preserved high prunability (80–90%80\text{--}90\% unit removal within a 1%1\% accuracy drop) while avoiding representation collapse. Because this design space is a research direction in its own right, we are consolidating a comprehensive parameter and capacity study into a dedicated follow-up manuscript.

Responses to Questions for Authors

Question 1: Sensitivity to Schedule Shape and Scaling Parameter α\alpha, and Preventing Collapse in Trailing Dimensions

  1. How Representation Collapse in Trailing Dimensions is Prevented: Representation collapse in trailing units happens only when survival probabilities approach zero (as in Nested Dropout's geometric tail ρi−1→0\rho^{i-1} \to 0, or when pmin⁡=0p_{\min}=0 with an overly steep decay). Our default ramp ends in a non-zero trailing floor (p=0.5p = 0.5 over the last 20%20\% of units; Appendix D.1), so every trailing channel remains active in 50%50\% of training steps and continues to receive gradient updates. As discussed in Appendix A.1, this floor keeps trailing units trainable so they capture fine-grained details without suffering representation collapse, while the monotonic slope (1.0→0.51.0 \to 0.5) is strong enough to enforce importance ordering: in our ResNet-50 channel analysis (Section 4.2), the first 10%10\% of channels concentrate 37.9%37.9\% of the activation power vs. 9.9%9.9\% in the baseline.

  2. Dedicated Follow-Up Study on Schedule Families and Parameters: More broadly, the design space of activation schedules spans many schedule families (e.g., linear, harmonic, polynomial, and step profiles) and parameterization choices (such as plateau width, decay slope, floor level, and scaling factor α\alpha). In our preliminary explorations across these parameters, we observed rich and systematic behaviors linking schedule geometry to network capacity, feature hierarchy, and task complexity. Because thoroughly characterizing this design space goes well beyond the scope of a single paper, we are dedicating a separate follow-up manuscript specifically to a comprehensive study of activation schedule families and their theoretical and empirical properties.


Question 2: Applying the Method to Fine-Tuning an Already-Trained Model

While our main focus in this paper is training ordered networks from scratch, there are several viable approaches to extend the Layer Activation Schedule to an already-trained model, one of them being the order-then-fine-tune procedure described below:

In a standard pretrained model, important features are scattered randomly across channel indices, so applying a decaying mask directly would drop useful features located at the end. One of the simplest ways to address this is a two-step order-then-fine-tune procedure: first, we rank the existing channels or neurons by a fast importance metric (such as weight norm or activation magnitude) and permute them so the most important units come first; second, we briefly fine-tune the ordered model under the Layer Activation Schedule so the representations adapt to the nested hierarchy. We plan to investigate it as a full framework in future work.

Thank you to the authors for their rebuttal and address the questions raised.

I look forward to seeing the updated results for the comparisons.

As for the clarification on the GPT-2 budget, thank you for this. I would advise there is a more clear and transparent reference to this in the main text to improve clarity and understanding of the empirical setup. My apologies for missing this.

I will wait to see the updated results you are running and provide an update following that. If the results hold against the baselines I would be inclined to increase my score.

We sincerely thank the reviewer for the encouraging follow-up. Ordered Dropout (FjORD) has now finished the full 200k-step training schedule, and we have evaluated both Ordered Dropout and the latest Nested Dropout checkpoint across our full benchmark suite. In the revised manuscript, Section 4.2 (Figure 7) compares WikiText-2 perplexity against removed FLOPs for all four ordered-width models. In the supplementary material, Appendix J.3 (Tables 13–15) updates the training and validation losses, and a new Appendix J.4 (Figures 60–61) plots validation loss, fixed-length perplexity, LAMBADA perplexity, WikiText-2 perplexity, PIQA, and the nine-task zero-shot average against removed FLOPs.

Against Ordered Dropout, our scheduled models achieve better unpruned quality and degrade smoothly across all widths. Unpruned, our Intra-Block model stays close to the dense GPT-2 medium baseline in WikiText-2 perplexity (25.89 vs. 25.51) and validation loss (2.71 vs. 2.68), whereas Ordered Dropout reaches 29.27 and 2.82. Under matched hyperparameters (ηmax⁡=1×10−4\eta_{\max} = 1\times10^{-4}, clip 0.10.1), Intra-Block also maintains a lower validation loss than Ordered Dropout at every checkpoint (2.7534 vs. 2.8134 at 200k steps; Tables 13–14). Under pruning, Extended Stream consistently outperforms Ordered Dropout, with validation losses of 2.85, 2.90, and 3.02 at 20%, 40%, and 60% FLOPs removed, compared to 2.87, 2.99, and 3.13 for Ordered Dropout. More importantly, because Ordered Dropout only trains k=5k=5 fixed widths ({1.0,0.8,0.6,0.4,0.2}\{1.0, 0.8, 0.6, 0.4, 0.2\}), it fails at intermediate widths: its WikiText-2 perplexity spikes above 1200 at 85–89% FLOPs removed (e.g., 1225.65 vs. our 91.21 at 87%, and 5872.27 vs. 147.91 on LAMBADA) before dropping back to 98.08 at its narrowest trained width (91.9% FLOPs removed). MAUVE shows the same instability, dropping to 0.10–0.33 between 82% and 89% FLOPs removed before recovering to 0.90 at 91.9%. In contrast, our models degrade smoothly and can be truncated to any arbitrary width.

Nested Dropout is still running because per-example geometric truncation prevents contiguous slicing, making training about 1.5×1.5\times slower. On the same 8×8\timesA100 node, Nested Dropout takes 25.97 hours per 50k steps compared to 16.87–17.48 hours for our models, reaching only 134k steps after roughly three days while our models and Ordered Dropout completed all 200k steps. At 100k steps under matched hyperparameters, Nested Dropout already trails our models with a validation loss of 2.9446, compared to 2.8253 for Intra-Block and 2.8991 for Extended Stream (Tables 13 and 15). Evaluated at its 134k checkpoint with 87% FLOPs removed, Nested Dropout performs substantially worse than Extended Stream across all language-modelling metrics and the zero-shot average, including WikiText-2 perplexity (91.21 vs. 122.20), LAMBADA perplexity (147.91 vs. 397.38), and nine-task zero-shot accuracy (0.426 vs. 0.415). We will update the manuscript with the final 200k-step Nested Dropout numbers once its run finishes.

Reviewer_iuG9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Layer Activation Schedule as a training method that encourages networks to concentrate useful information in leading channels, allowing trailing ones to be removed without post-pruning finetuning. Its main contributions are ordered representations, simple pruning into smaller dense networks, and support for multiple model widths from a single checkpoint. Experiments on GPT2 medium and Resnet50 show substantial compute reductions. GPT2 retains generation quality within the paper’s MAUVE threshold after removing 92.8% of FLOPs that doesn't directly establish equivalent wall-clock speedups, while Resnet50 retains 64.62% Top-1 accuracy after a 60.47% MAC reduction with activation analyses show information concentrating in leading channels.The results also suggest that the method makes GPT training more stable with less restrictive training settings. However, the GPT experiments used a limited training budget and do not show that all model abilities are equally preserved.

Critical strengths
  1. The paper establishes a practical connection between ordered representations and structured pruning. Training channels to follow an importance ordering makes compression possible through contiguous truncation, producing smaller dense operations. This is important because it addresses the difficulty of turning sparsity into usable computational savings.

  2. The paper is able to present evidence across two distinct architecture families. Experiments on GPT2 and Resnet50 show promising retention of performance under substantial pruning, supported by channel activation analysis. Although concerns about the comparisons remain, results across two different architecture families suggest the approach deserves further study.

  3. The paper’s method allows a single trained model to run at different sizes to suit different compute budgets. The ordered representations allow multiple model widths to be obtained through truncation without retraining or post-pruning finetuning. This is consequential because it could reduce the cost of maintaining separate models for devices with different compute budgets.

  4. The paper compares the effects of applying gating to different parts of the network. Comparing Intra-block and extended stream gating examines whether ordering internal channels alone is sufficient or whether the residual stream should also be constrained. This provides useful guidance on integrating the method into different architectures.

Critical weaknesses
  1. The method's added value over closely related work is insufficiently established. The paper mentions nested dropout as another way to learn ordered representations, but does not directly compare its performance with the proposed method. This makes it difficult to determine what the proposed schedule adds and whether that contribution explains the reported gains.

  2. The claim that the Layer ACtivation Schedule preserves “dense-level generation quality” after removing 92.8% of GPT-2 medium’s FLOPs relies on a limited criterion. Maintaining MAUVE within 20% of the dense score does not establish comparable language-modeling or downstream performance, which already deteriorate at lower pruning levels.

Questions for authors

Clarification requests:

  1. Did you compare the scheduled models and baseline using the same learning rate and gradient clipping settings? If so, do the gains remain?
  2. What does your method add beyond nested dropout, and do you have direct comparisons supporting that difference?
  3. At 92.8% FLOPs removal, what are the perplexity and downstream task scores? Does “dense-level quality” refer only to the stated MAUVE criterion?

Suggestions for future work:

  1. Measure actual inference speed and memory use to show whether fewer operations lead to practical savings.
  2. Remove leading and trailing channels separately and compare the performance loss to test whether leading channels are more important.
3 comments

We sincerely thank the reviewer for the constructive feedback, insightful questions, and positive recognition of our work's capabilities in single-checkpoint elastic deployment.

Below, we address each of your questions and suggestions in detail.

Responses to Critical Weaknesses

Weakness 1: Added value and direct comparison with Nested Dropout and Ordered Dropout

We have added a detailed methodological and empirical comparison in Appendix A ("Comparison with Prior Ordered and Nested Representation Methods", including Subsection A.3: "Direct Empirical Comparison of Training and Validation Losses").

The results show that both of our scheduled variants consistently achieve lower training and validation losses than Ordered Dropout and Nested Dropout across all evaluated checkpoints, while training roughly 1.5×1.5\times faster than Nested Dropout. Please note that the full training runs for Nested Dropout and Ordered Dropout are currently in progress; once training finishes, we will complete all downstream pruning benchmarks and update the manuscript accordingly.


Weakness 2: Clarification of the "Dense-Level Generation Quality" Claim at 92.8% FLOPs Removal

We have updated the wording and clarified the evaluation metrics in Section 4.1 ("Effective Pruning at Extreme Sparsity") and Appendix E ("Additional GPT-2 Pruning Results"). We replaced "dense-level" with "a decent level of generation quality (MAUVE within 20% of the dense model)" throughout the paper, and the results show that while token-level perplexity and zero-shot accuracy degrade earlier than MAUVE, our scheduled models still strongly outperform all baselines at 40% and 60% FLOPs removed.


Responses to Questions

Question 1: Matched Learning Rate and Gradient Clipping Settings

We have added a controlled comparison under identical optimization settings in Appendix A.3 ("Direct Empirical Comparison of Training and Validation Losses", Tables 2–4). The results show that under the dense baseline's conservative hyperparameters (ηmax⁡=1×10−4\eta_{\max} = 1 \times 10^{-4} and gradient clip 0.10.1), the validation loss of our scheduled models changes by only ≈0.01–0.05\approx 0.01\text{--}0.05 relative to our standard setting (Table 2), and both variants still achieve lower losses than Ordered Dropout and Nested Dropout trained under the same settings (Intra-Block at every checkpoint, Extended Stream at 50k and 100k steps). In our experiments, the structured-pruning advantage over the dense baseline also remains intact under these matched settings.


Question 2: Added Value Beyond Nested Dropout and Direct Comparisons

We have added a detailed comparison in Appendix A ("Comparison with Prior Ordered and Nested Representation Methods", including Subsections A.1–A.3: "Flexible Continuous Survival Schedule vs. Rigid Profiles", "Generalization to Deep Residual Architectures (Intra-Block and Extended Stream Gating)", and "Direct Empirical Comparison of Training and Validation Losses").

The results show that our continuous schedule avoids gradient starvation in wide layers, extends ordered gating to deep residual streams, and achieves consistently lower training and validation losses than both Nested Dropout and Ordered Dropout while training roughly 1.5×1.5\times faster than Nested Dropout. The full training of Nested Dropout and Ordered Dropout is still in progress; we will run all remaining benchmarks as soon as training finishes and update the manuscript.


Question 3: Perplexity and Downstream Scores at 92.8% FLOPs Removal

We have clarified the MAUVE criterion and reported the full language-modeling and zero-shot evaluations across pruning ratios in Section 4.1 ("Effective Pruning at Extreme Sparsity", Table 1), Appendix E ("Additional GPT-2 Pruning Results", Tables 5–6), Appendix F ("GPT-2 Pruning Curves: Structured Methods"), and Appendix G ("GPT-2 Pruning Curves: Semi-Structured and Unstructured Methods").

Yes, the phrase referred only to the MAUVE criterion. The 92.8%92.8\% figure marks the 20% MAUVE breaking point: the budget at which MAUVE permanently drops below 0.8×0.942≈0.750.8 \times 0.942 \approx 0.75, interpolated between the two measured configurations that bracket it. We do not claim dense-level perplexity or zero-shot accuracy at that depth. At the deepest matched table budget of 60% FLOPs removed, Extended Stream achieves 38.638.6 WikiText-2 perplexity and 0.4600.460 nine-task zero-shot accuracy (Table 1) and 21.221.2 LAMBADA perplexity (Table 5), whereas all baselines collapse to ≥2.3×104\ge 2.3 \times 10^4 WikiText-2 perplexity and near-chance zero-shot accuracy.

Responses to Suggestions

Suggestion 1: Practical Inference Speed and Memory Use

We have added practical hardware speed and memory comparisons in Section 4.4 ("Hardware Utilization and Inference Speed-Up", Figures 8–9) and Appendix M ("Hardware Utilization and Inference Speed-Up Details", including Subsections M.1 "ResNet-50 Hardware Benchmarks" and M.2 "GPT-2 Hardware Benchmarks", Tables 17–18).

The results show that physical truncation yields up to a 2.71×2.71\times decode speed-up and 89.8%89.8\% peak memory reduction on GPT-2 medium, and up to a 2.86×2.86\times inference speed-up and 81.0%81.0\% weight memory savings on ResNet-50.


Suggestion 2: Removing Leading vs. Trailing Channels Separately

Thank you for this very interesting suggestion, which aligns closely with our ongoing research. In addition to our activation power analysis (Section 4.2, showing that the leading 10% of channels hold 37.9% of total activation power vs. 9.9% in the baseline), we have been exploring this exact question by visualizing the attention mechanisms and separating channels into distinct groups to understand what features each group learns and how we can adapt the activation schedule for better results.Interestingly, our current hypothesis is that the very first trailing channels may also absorb high-variance or noisy signals from the data, though this behavior is currently under active investigation and we plan to report detailed findings in future work.

We would like to follow up on our initial response regarding the training-time ordered-width baselines. Ordered Dropout (FjORD) has now finished the full 200k-step training schedule, and we have evaluated both Ordered Dropout and the latest Nested Dropout checkpoint across our full benchmark suite. In the revised manuscript, Section 4.2 (Figure 7) compares WikiText-2 perplexity against removed FLOPs for all four ordered-width models. In the supplementary material, Appendix J.3 (Tables 13–15) updates the training and validation losses, and a new Appendix J.4 (Figures 60–61) plots validation loss, fixed-length perplexity, LAMBADA perplexity, WikiText-2 perplexity, PIQA, and the nine-task zero-shot average against removed FLOPs.

Against Ordered Dropout, our scheduled models achieve better unpruned quality and degrade smoothly across all widths. Unpruned, our Intra-Block model stays close to the dense GPT-2 medium baseline in WikiText-2 perplexity (25.89 vs. 25.51) and validation loss (2.71 vs. 2.68), whereas Ordered Dropout reaches 29.27 and 2.82. Under matched hyperparameters (ηmax⁡=1×10−4\eta_{\max} = 1\times10^{-4}, clip 0.10.1), Intra-Block also maintains a lower validation loss than Ordered Dropout at every checkpoint (2.7534 vs. 2.8134 at 200k steps; Tables 13–14). Under pruning, Extended Stream consistently outperforms Ordered Dropout, with validation losses of 2.85, 2.90, and 3.02 at 20%, 40%, and 60% FLOPs removed, compared to 2.87, 2.99, and 3.13 for Ordered Dropout. More importantly, because Ordered Dropout only trains k=5k=5 fixed widths ({1.0,0.8,0.6,0.4,0.2}\{1.0, 0.8, 0.6, 0.4, 0.2\}), it fails at intermediate widths: its WikiText-2 perplexity spikes above 1200 at 85–89% FLOPs removed (e.g., 1225.65 vs. our 91.21 at 87%, and 5872.27 vs. 147.91 on LAMBADA) before dropping back to 98.08 at its narrowest trained width (91.9% FLOPs removed). MAUVE shows the same instability, dropping to 0.10–0.33 between 82% and 89% FLOPs removed before recovering to 0.90 at 91.9%. In contrast, our models degrade smoothly and can be truncated to any arbitrary width.

Nested Dropout is still running because per-example geometric truncation prevents contiguous slicing, making training about 1.5×1.5\times slower. On the same 8×8\timesA100 node, Nested Dropout takes 25.97 hours per 50k steps compared to 16.87–17.48 hours for our models, reaching only 134k steps after roughly three days while our models and Ordered Dropout completed all 200k steps. At 100k steps under matched hyperparameters, Nested Dropout already trails our models with a validation loss of 2.9446, compared to 2.8253 for Intra-Block and 2.8991 for Extended Stream (Tables 13 and 15). Evaluated at its 134k checkpoint with 87% FLOPs removed, Nested Dropout performs substantially worse than Extended Stream across all language-modelling metrics and the zero-shot average, including WikiText-2 perplexity (91.21 vs. 122.20), LAMBADA perplexity (147.91 vs. 397.38), and nine-task zero-shot accuracy (0.426 vs. 0.415). We will update the manuscript with the final 200k-step Nested Dropout numbers once its run finishes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.