- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Layer Activation Schedule trains networks with fixed, non-increasing survival probabilities so that trailing dimensions can be structurally removed without fine-tuning. It combines parameter slicing with either independent Bernoulli gating or contiguous-prefix sampling. Intra-Block Gating narrows internal dimensions; Extended Stream Gating also compresses the Transformer residual stream, while ResNet block outputs are scattered into full-width residual tensors. Experiments with GPT-2 medium on OpenWebText and ResNet-50 on ImageNet-1K report improved pruning robustness over the evaluated baselines, including MAUVE within 20% of the dense score at 92.8% FLOPs removed and 64.62% Top-1 accuracy at 60.47% MAC reduction. Activation statistics show greater concentration in leading channels.
Critical strengths
-
A practical formulation for extracting smaller dense subnetworks without fine-tuning. The slicing rules and two gating variants explain how the method handles different layer types and residual connections, making the deployment mechanism concrete.
-
Strong reported pruning robustness and evidence of activation concentration. At the interpolated 60% FLOP-reduction budget, Extended Stream achieves WikiText-2 perplexity 38.6 versus approximately 67,000 for SparseGPT. ResNet Intra-Block retains 67.94% accuracy at 55.44% MAC reduction versus 1.89% for the reported baseline. However, unpruned accuracy is lower: 70.49%/70.37% for Intra-Block/Extended Stream versus 72.02% for the baseline. Across 16 blocks, the mean share of activation power in the first 10% of channels is 37.9% for Extended Stream versus 9.9% for the baseline. This supports concentration, but does not establish a strict ordering of task importance.
Critical weaknesses
-
The contribution over closely related training methods is insufficiently established. The prefix sampler closely resembles Nested Dropout and Ordered Dropout in FjORD; Universally Slimmable Networks and MatFormer also learn variable-width or nested models. Several predecessors are acknowledged, but none is experimentally compared. The independent sampler and architectural adaptations may differ, yet comparisons mainly against post-training pruning do not isolate their incremental benefit. A representative training-time comparison, constant-probability gating control, and compact dense models at matched inference budgets are needed. GPT-2 optimization also differs, the dense baseline uses peak learning rate and clipping norm 0.1, versus and 1.0 for scheduled models. The reported baseline divergence at the higher learning rate justifies considering different settings, but controlled comparisons over stable settings and comparable tuning budgets are still needed. The ResNet baseline pruning operator and different curves across the two architectural comparisons also require clarification; different pruning scopes could explain them.
-
Quality-preservation and efficiency claims exceed the supporting evidence. The 92.8% headline uses a 20% tolerance on MAUVE, which does not establish general preservation of language-model quality. At 60% FLOPs removed, Extended Stream’s MAUVE is 0.932 versus 0.942 for the dense reference, while WikiText-2 perplexity rises from 25.5 to 38.6, LAMBADA perplexity from 10.1 to 21.2, and LAMBADA accuracy falls from 0.410 to 0.303. The MAUVE evaluation settings and uncertainty are insufficiently documented. Figure 20 explicitly discloses an Intra-Block rebound from 0.015 at 82.4% removal to 0.332 at 87.2%, further motivating repeated evaluation. Table 1 describes a nine-task zero-shot mean of 0.510 for the dense model, while Appendix D/Table 3 describe five tasks. The five displayed values average to . Missing results or a corrected aggregation/description are needed; this discrepancy alone does not identify which value is wrong. Checkpoint-specific schedule/sampler choices and their sensitivity remain unclear despite the provided defaults. Accelerated training and hardware-efficiency claims also lack measured throughput, latency, and peak memory, particularly relevant to ResNet Extended Stream’s full-width residual tensors and scatter operations.
Questions for authors
-
Which tasks and averaging rule produce Table 1’s zero-shot scores, and how should the five-versus-nine-task discrepancy be resolved?
-
For each checkpoint, what schedule family, , segment parameters, minimum probability, and sampler were used, and how were they selected? Were the samplers compared, given that matching marginals does not match their joint distributions? For the default ramp, makes all probabilities one after scaling/clipping. How sensitive are results to these choices, and do configurations transfer between architectures without retuning?
-
What selection rule and layer scope define each ResNet baseline? Do the different curves use the same checkpoint with different pruning scopes, and which is shown in Figure 6? Are BatchNorm statistics sliced or recalibrated, and is probability scaling retained after truncation?
-
What MAUVE data, sample counts, generation lengths, decoding settings, feature model, and uncertainty estimates were used? Please report the other metrics at the 92.8% operating point and explain whether this threshold is measured or interpolated, including sensitivity to randomness and the chosen tolerance.
-
Can the authors provide the training-time and compact-model comparisons above, plus constant-probability gating at comparable expected active width? Training and tuning budgets should be comparable, and optimization controls should account for the reported baseline divergence.
-
What training throughput, inference latency and throughput, and peak memory are achieved at representative pruning ratios with specified hardware, batch sizes, and sequence lengths? Please include slicing and scatter overhead and explain FLOP accounting across pruning formats, including UniPruning (2:4) beyond 50% reported FLOP removal.
5 comments
Responses to Questions for Authors
Question 1: Zero-Shot Tasks, Averaging Rule, and Resolving the Five-vs.-Nine-Task Discrepancy
We have added the complete nine-task breakdown in Appendix E ("Additional GPT-2 Pruning Results", Table 6). The results show that Table 1's zero-shot score is the unweighted arithmetic mean across all nine zero-shot tasks (ARC-Easy, ARC-Challenge, HellaSwag, PIQA, WinoGrande, OpenBookQA, BoolQ, SciQ, and COPA), using length-normalised accuracy for ARC-Easy, ARC-Challenge, HellaSwag, PIQA, and OpenBookQA and plain accuracy for the other four (as stated in the Table 6 caption). The initial submission displayed only five of these tasks to save space; averaging all nine dense values gives exactly , and our scheduled models outperform all baselines at 40% and 60% FLOPs removed.
Question 2: Checkpoint Configurations, Sampler Comparison, Parameterization, and Sensitivity
We have added the full configuration details in Appendix D.1 ("Activation Schedule Parameters"). All four scheduled checkpoints use the same 3-segment piecewise linear schedule (, decaying from to , , , mean active width ), fixed a priori and transferred between ResNet-50 and GPT-2 medium without retuning. The sampler is independent Bernoulli for ResNet-50 and contiguous prefix (slice) for GPT-2.
- Samplers: as you note, the two samplers have identical marginals but different joint distributions. Independent Bernoulli sampling varies which subset of units co-activates on each step, whereas prefix sampling enforces strict nesting () on every forward pass. We first developed the Bernoulli sampler and used it to train ResNet-50. On GPT-2, non-contiguous index gathering and scattering across the large hidden dimensions made it prohibitively slow, so we introduced the prefix sampler, which keeps memory accesses contiguous and matches the contiguous truncation operator used at deployment. We have not yet run a matched head-to-head comparison of the two samplers on the same architecture; the training speed-up reported in Appendix A.3 is relative to Nested Dropout, not to the Bernoulli sampler.
- parameterization: you are right that with a floor of , any saturates all survival probabilities to after scaling and clipping, so the non-degenerate range for our default ramp is . We introduced for schedule families whose trailing probabilities decay towards zero (such as the harmonic schedule, whose last segment ends at ).
- Sensitivity: during development, we evaluated several schedule families (harmonic, piecewise linear, and inverse-prime), segment counts , and scaling factors on smaller benchmarks (CIFAR-10 and EMNIST). Smoothly decaying profiles with a non-zero trailing floor consistently preserved high prunability ( unit removal within a accuracy drop) while avoiding representation collapse. Because this design space is a research direction in its own right, we are consolidating a comprehensive parameter and capacity study into a dedicated follow-up manuscript.
Question 3: ResNet-50 Baseline Definition, Pruning Scopes, BatchNorm Statistics, and Probability Scaling
The pruning operator is described in Section 3.4 ("Structured Pruning of the Scheduled Network") and Appendix C ("Detailed Formulation of Structured Pruning"), the inference-time probability scaling in Section 3.1 (paragraph "Inference Phase"), and the per-scope baseline results in Appendix H ("ResNet-50 Dense Evaluation, Uniform Pruning Table and Per-Variant Budget Bars", Table 8 and Figures 45–46).
- Selection rule and layer scope: as explained in our answer to Weakness 1, both Baseline curves use the same unscheduled checkpoint ( Top-1) and the same positional trailing-index truncation () as our scheduled models. The Intra-Block scope truncates only the internal width , and the Extended Stream scope truncates both and the block expansion output . Figure 6 plots both scheduled models against the Intra-Block-scope Baseline, and Figures 45–46 show each variant against the Baseline under its matching scope.
- BatchNorm statistics: no running statistics are tracked, so there is nothing to slice or recalibrate. Because active channel widths vary stochastically during training, all BatchNorm layers are built with track_running_stats=False; at evaluation, channels are normalized with batch statistics, identically for the Baseline and the scheduled checkpoints. The affine parameters () are physically sliced to the truncated width (Appendix C).
- Probability scaling after truncation: yes, it is retained. In ResNet-50, the deterministic schedule scales the output of each scheduled BatchNorm layer at inference (Section 3.1). When instantiating the physically truncated dense model, this static vector is folded into the sliced affine parameters ( and ), requiring zero extra runtime operations. In GPT-2, no probability scaling is applied during either training or inference.
Question 4: MAUVE Protocol, Uncertainty, the 92.8% Operating Point, and Measured vs. Interpolated Metrics
We have clarified the MAUVE criterion and report the full-sweep metrics in Section 4.1 ("Effective Pruning at Extreme Sparsity", Table 1), Appendix E ("Additional GPT-2 Pruning Results", Tables 5–6), Appendix F ("GPT-2 Pruning Curves: Structured Methods"), and Appendix G ("GPT-2 Pruning Curves: Semi-Structured and Unstructured Methods").
- MAUVE settings: passages from the OpenWebText validation split, using the first tokens as the prompt; continuation tokens generated by ancestral sampling at temperature (top_k = None, seed ); gpt2 as the feature model, with features quantized into clusters by MAUVE's default -means setting ().
- Measured vs. interpolated: the operating point is interpolated rather than a single measured configuration. It is the budget at which Extended Stream's MAUVE permanently drops below of the dense score (). We take the first measured configuration below this threshold that is never followed by one above it, and linearly interpolate the crossing against the measured configuration before it. A noisy dip that recovers therefore does not end a curve, and a spike after a collapse cannot extend it.
- Sensitivity to the tolerance: reading the same crossing at a stricter ( of dense) and a looser ( of dense) threshold keeps Extended Stream's breaking point within FLOPs removed and the strongest baseline's within , so the ranking does not depend on the chosen tolerance. At 60% FLOPs removed, all baselines are already at (Table 1).
- The Figure 22 rebound (Figure 20 in the initial submission): at FLOPs removed, Intra-Block has already collapsed (validation loss ), and the severely truncated model generates erratic, degenerate text. In this regime, a clustering-based metric computed from samples and clusters becomes noisy and unreliable, which produces the spurious reading after MAUVE had already fallen to at . As stated in the Figure 22 caption, this point is not plotted, and it does not affect any breaking point.
We sincerely thank the reviewer for the constructive feedback, and for the insightful questions. Your feedback helped us identify and resolve several important presentation gaps in the revised manuscript.
Below, we address each of your questions and suggestions in detail.
Responses to Critical Weaknesses
Weakness 1: Contribution Over Related Training-Time Methods, Matched Optimization Controls, and ResNet Baseline Clarification
We have added a detailed comparison with Nested Dropout and Ordered Dropout in Appendix A ("Comparison with Prior Ordered and Nested Representation Methods", including Subsections A.1–A.3: "Flexible Continuous Survival Schedule vs. Rigid Profiles", "Generalization to Deep Residual Architectures (Intra-Block and Extended Stream Gating)", and "Direct Empirical Comparison of Training and Validation Losses") and reported the ResNet-50 baseline results in Appendix H ("ResNet-50 Dense Evaluation, Uniform Pruning Table and Per-Variant Budget Bars", Table 8 and Figures 45–46). Please note that the full training of Nested Dropout and Ordered Dropout is still in progress; once training is completed, we will evaluate them across the full benchmark suite and include the final results in the manuscript. At the current checkpoints, both of our scheduled variants already achieve lower training and validation losses than Ordered Dropout and Nested Dropout while training roughly 1.5× faster than Nested Dropout.
Second, we trained our scheduled models using the exact same optimization hyperparameters as the dense baseline (ηmax=1×10−4 and gradient clip 0.1; Appendix A.3, Tables 2–4). Under these baseline parameters, the validation loss of our scheduled models shows only a minor difference (≈0.01–0.05) compared to our standard setting, while still achieving lower losses than Ordered Dropout and Nested Dropout (at every checkpoint for Intra-Block, and at 50k and 100k steps for Extended Stream).
Finally, regarding the ResNet-50 baseline: both Baseline curves come from the exact same unscheduled ResNet-50 model (72.02% Top-1), evaluated under the two different pruning scopes corresponding to our two architectural variants (using the same positional trailing-index truncation 1…N′ without saliency sorting):
Weakness 2: Scope of Quality-Preservation Claims, MAUVE Protocol, Zero-Shot Task Discrepancy, Checkpoint Settings, and Hardware Efficiency
We have updated and added comparisons across the relevant sections: we have clarified MAUVE quality claims, Appendix D.1 ("Activation Schedule Parameters") for the exact checkpoint configurations, Appendix E ("Additional GPT-2 Pruning Results", Table 6) for all nine zero-shot tasks, and Section 4.4 ("Hardware Utilization and Inference Speed-Up") alongside Appendix M ("Hardware Utilization and Inference Speed-Up Details", Tables 17–18) for the hardware benchmarks. The MAUVE evaluation settings and the Intra-Block rebound (now Figure 22 in Appendix F) are detailed in our answer to Question 4 below.
The results show that all four main checkpoints share the exact same schedule configuration without retuning. In the pervious version, we included five representative tasks in Table 3 for conciseness; Now, the full nine-task zero-shot average has been added, and physical truncation yields up to a 2.71× decode speed-up and 89.8% peak memory reduction on GPT-2 medium and up to a 2.86× speed-up and 81.0% weight memory reduction on ResNet-50.