acceptodds
Under review as a conference paper at ICLR 2027

Bandstop: Emergent Compression via Residual Flow Throttling

Abstract

Residual connections turn network depth into a sequence of refinements to a shared representation. However, treating residual branches uniformly provides no structured way to exploit redundancy among these transformations. We introduce Bandstop, a gated residual architecture that imposes an inductive bias toward non-uniform residual branch scaling across depth. Without an auxiliary compression objective or prescribed pruning budget, gradient-based training learns two scalar cutoffs that delimit a contiguous intermediate region in which residual contributions are attenuated. Across architectures, this structure yields emergent compression: the attenuated region can be removed with limited loss in task performance. This learned depth structure improves robustness to post-hoc pruning by up to 8.6 percentage points in mean accuracy over the matched Residual network, with the advantage persisting even after layer replacement. In vision, Bandstop learns lower-rank representations while improving classification accuracy and robustness to input corruption. In language models, Bandstop counters the curse of depth reported for standard residual networks and shows the reasoning-primitive gains associated with depth-grown models. Our results demonstrate that non-uniform layer-wise throttling of residual flow is a simple mechanism for inducing compression during standard training, and establish layer-wise information flow as an important architectural degree of freedom for building efficient and adaptable deep networks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
neural network compression, layer pruning, residual networks, layer-wise gating

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qh6FOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Based on the observation that the middle layers of a neural network are redundant and nearly interchangeable, the paper proposes a method called Bandstop, which learns two parameters to attenuate the contributions of the residual flow, thereby reducing the depth of networks with residual layers. Experimentally, the authors compare networks whose depth has been reduced using Bandstop with the same residual network that has not been trained to be pruned. Experiments cover vision (MLP-Mixer, ViT-S) and language models (GPT-2 124M, SmolLM1 135M).

Critical strengths

Bandstop is a simple method with negligible additional cost. The approach introduces only two trainable scalars for the entire network, without requiring an auxiliary compression objective. The memory and computational costs during training are negligible with respect to a standard Residual training.

To remove layers, Bandstop does not use combinatorial search or importance estimation on a validation set: the layers to be removed are determined directly by reading the lowest values of βℓ\beta_\ell (no search and no validation access).

Critical weaknesses

The experimental comparisons in the main body are limited. The main comparison (Figures 1 and 3) is conducted almost exclusively against a standard residual baseline evaluated by an oracle. However, a conventional residual model is never trained to direct the calculation toward the identity at the center.

The paper omits closely related work that prepares models for layer skipping/removal. For example:

  • LayerSkip [1]: a related method that prepares for execution at a reduced depth, via progressive layer dropout and early-exit training.
  • LaCoOT [2]: a related method that enforces intermediate feature distributions toward identity collapse using optimal transport (Max-Sliced Wasserstein distance), enabling search-free block deletion post-training without combinatorial validation search

Bandstop is restricted to strictly constant-width, constant-resolution architectures. All experiments are based on models where the representation dimension is strictly constant (ViT, MLP-Mixer, GPT-2, SmolLM). The paper does not address widely used residual architectures with changes in scale or resolution across a stage boundary (for example: ResNet or ConvNeXt), for which contiguous deletion across a resolution jump mathematically breaks the identity operation.

The scaling of the method has not been demonstrated. The models evaluated are relatively small (≤135M\le 135\text{M} parameters) and are all trained from scratch. The cost of applying Bandstop to modern foundation models (7B+) is not discussed.

Questions for authors
  1. Missing baselines & positioning. How does Bandstop compare conceptually and empirically to recent training-time depth compression frameworks such as LayerSkip [1] and LaCoOT [2], which prepare networks for reduced-depth execution or block removal after training?

  2. Physical removal vs. masking. Is the removal a true removal of the block (computational elimination) or a masking? Figure 3 reports a “measured forward-pass speedup” without hardware or protocol, even though the limitations state that hardware performance is not measured. Can the authors report the actual latency, FLOPs, and memory usage, along with the hardware in at least one setup?

  3. Comparison to matched-depth models. Even though the value of compression methods lies in the fact that one does not know a priori what the optimal size of the neural network is for a given task, it would be interesting to see to what extent the proposed method achieves a compressed model with performance comparable to that of the same network trained from scratch. How does a pruned Bandstop subnetwork compare against a matched-depth model trained from scratch (and against a standard knowledge-distilled counterpart) across the four architectures?

  4. Scalability and applicability to a Bandstop fine-tuning version. Does Bandstop necessarily require full training from scratch? Would it be possible to initialize the thresholds cLc_L and cRc_R and fine-tune a standard pre-trained model (e.g. a LLaMA-7B) using a limited token budget to induce this retractable band?

  5. Low-pass vs Bandstop discrepancy. In Table 6, Low-pass appears to be more robust than Bandstop at ≥25 %\ge 25\,\% of the layers. Does this contradict the hypothesis that intermediate layers are the region most susceptible to removal and the claim of “emergent” compression?

  6. High-compression regime and the role of Bandstop-sharp. In Tables 8 and 9, Bandstop fails compared to Stochastic Depth, LayerDrop, or ACN under heavy pruning (e.g., ≥50 %\ge 50\,\% of the layers). What is the theoretical or practical justification for choosing between the standard profile and the Bandstop-sharp variant? Is it possible to learn the slope τ\tau rather than fixing it?

References:
[1] Elhoushi, et al. “Layerskip: Enabling early exit inference and self-speculative decoding”. ACL 2024
[2] Quétu, et al. “LaCoOT: Layer Collapse through Optimal Transport”. ICCV 2025

Given the current state of the paper, I am leaning toward a weak rejection. However, if the points raised are sufficiently addressed during the rebuttal, I am willing to increase my score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer qh6F

We thank the reviewer for recognizing Bandstop’s negligible overhead and search-free gate-based selection.

W1. Main-body baselines.

The main comparison (Figures 1 and 3) is conducted almost exclusively against a standard residual baseline evaluated by an oracle. However, a conventional residual model is never trained to direct the calculation toward the identity at the center.

We agree these baselines should be prominent in the main text. New Table 1 compares Bandstop with LayerSkip, LayerDrop, LaCoOT, ReZero, LayerScale and target-depth training; Figure 2 adds LayerSkip under the shared LLM-Streamline rule, with LaCoOT in Table 13. Stochastic Depth and ACN remain in the appendix.

W2. Related work.

The paper omits closely related work that prepares models for layer skipping/removal.

We agree these closely related methods should be discussed explicitly. The revision adds LayerSkip and LaCoOT to the introduction and App. A and compares them directly under Q1.

W3. Stage-wise architectures.

Bandstop is restricted to strictly constant-width, constant-resolution architectures.

We agree that stage-wise architectures are an important extension. Our experiments use constant-shape stacks; a natural design would use one gate per stage while keeping transition blocks active. Because this remains untested, §4 lists it as future work.

W4. Scale.

The scaling of the method has not been demonstrated.

Bandstop adds only two scalars, one scaling operation per block and no auxiliary loss, so relative overhead decreases with width. Our evidence addresses depth rather than parameter-count scaling. At half-depth, perplexity rises 5.6× vs. 14.2× on 12-layer GPT-2 and 2.7× vs. 208× on 30-layer SmolLM1. Q4 reports a preliminary retrofit. Matched 7B+ pretraining exceeded our compute, so §4 limits claims to at most 135M parameters.

Q1. LayerSkip and LaCoOT. Conceptually, the three methods differ in what they add to training and in where they place the removable depth:

  • LayerSkip combines layer dropout that increases with depth with an early-exit loss that evaluates the shared LM head at intermediate layers, following a curriculum. This prepares the network to run only a prefix of its layers, like our Low-pass profile.
  • LaCoOT adds an auxiliary optimal-transport loss, weighted by λ, that pulls each block's output distribution toward its input. It does not prescribe where collapse occurs, and λ is tuned per architecture.
  • Bandstop adds no loss term or pruning budget. Two learned scalars attenuate a contiguous middle band, the task loss alone organizes computation around the active prefix and suffix, and the removable window is read from the gate.

LayerSkip and LaCoOT prepare reduced depth through their training objectives. Bandstop instead uses the residual parameterization, requiring neither intermediate predictions nor a distribution regularizer and leaving the training loop unchanged.

Empirically, we trained both methods on MLP-Mixer and GPT-2 with architecture-specific recipes (LaCoOT with λ=1\lambda=1; Table 12 also reports λ=0.1\lambda=0.1 and 55). The table combines each method's own removal rule (Table 1; App. C, Table 7) with the shared LLM-Streamline rule (Figure 2, Table 13), which selects the window with the most similar input and output and optionally replaces it with a lightweight MLP. MLP-Mixer values average three seeds, GPT-2 values are single runs; bold marks the best method.

Method
MLP-Mixer / CIFAR-10, accuracy (%) ↑, blocks kept161284
Bandstop93.191.071.925.8
Bandstop-sharp92.292.289.138.8
LayerSkip91.490.988.668.6
LayerDrop90.789.486.472.1
LaCoOT90.489.983.449.9
GPT-2 124M, perplexity ↓, layers kept121186
Bandstop21.8724.8648.20122.26
Bandstop-sharp24.1124.1125.9635.67
LayerSkip22.1622.8028.8243.27
LayerDrop23.5924.2626.5731.18
LaCoOT24.0926.87364.69668.63
GPT-2 124M, mean accuracy (%) ↑, LLM-Streamline (shared rule), unhealed / healed121186
Residual40.6339.65 / 39.6833.89 / 35.9632.35 / 33.22
Bandstop40.9040.49 / 40.7237.35 / 38.7934.21 / 36.73
Bandstop-sharp39.3139.39 / 39.4838.84 / 39.0839.41 / 37.76
LayerSkip39.6339.30 / 38.8136.04 / 34.2032.48 / 32.48
LayerDrop39.6639.40 / 39.1937.67 / 37.0435.61 / 34.22
LaCoOT39.0138.70 / 38.1034.42 / 34.8433.94 / 33.83
AuthorsOpenReview ↗

Continued [1]

Q1 (continued). Reading the table.

  • Full depth. Among the pruning-aware methods, Bandstop gives the strongest full-depth model. Relative to Residual, it improves MLP-Mixer accuracy (93.1% vs. 91.7%) and essentially matches GPT-2 perplexity (21.87 vs. 21.79), while LayerSkip, LayerDrop and LaCoOT lose 0.3–1.3 points and add 0.4–2.3 to perplexity.
  • Where Bandstop wins and where the others get stronger. Under light removal, standard Bandstop leads the non-Bandstop methods on MLP-Mixer, while LayerSkip's early exit leads in GPT-2 perplexity. Under moderate removal, LayerSkip and LayerDrop overtake standard Bandstop. There, Bandstop-sharp is the strongest method in both tasks, and it also leads with 12 Mixer blocks kept.
  • Common rule and replacement. Under LLM-Streamline, Bandstop is the most accurate model with one layer removed, and Bandstop-sharp with 8 and 6 layers kept, healed or unhealed. Healing improves Bandstop at every budget, but not LayerSkip or LayerDrop. In mean accuracy, healed Bandstop beats every non-Bandstop method at every budget, and even unhealed it beats healed Residual.
  • Heavy removal. At 4 Mixer blocks, LaCoOT with λ = 5 leads overall (80.8%, Table 12) but falls to 83.1% at full depth; among the Table 1 configurations, LayerDrop leads with 72.1%. LayerDrop also leads half-depth GPT-2 perplexity. Bandstop-sharp instead leads half-depth GPT-2 accuracy and, without healing, retains its full-depth accuracy (39.41% vs. 39.31%).

Q2. Physical removal and inference cost. Removal is physical: the blocks are deleted, and the outputs are identical to those of the masked evaluation. Cost depends only on the number of retained layers, so Bandstop and Residual have the same parameters, FLOPs, latency and memory at each budget. App. C (Table 8) reports these on one NVIDIA A100 64 GB. The protocol is bf16 prefill of a batch of 8 full-length sequences (1,024 tokens for GPT-2, 2,048 for SmolLM1), with the minimum over 10 trials of 100 timed iterations after warm-up:

ModelLayers keptParamsFLOPs/tokenLatencySpeedupPeak memory
GPT-2 124M12124.5M285M15.1 ms1.00×1557 MiB
11117.4M268M14.1 ms1.07×1515 MiB
896.1M216M11.0 ms1.38×1390 MiB
681.9M181M8.9 ms1.70×1306 MiB
SmolLM1-135M30134.5M411M83.5 ms1.00×2379 MiB
24113.3M340M67.6 ms1.23×2256 MiB
1892.0M269M51.8 ms1.61×2133 MiB
1270.8M198M35.9 ms2.32×2009 MiB

Figure 2 (Figure 3 in the submission) now reports these end-to-end speedups instead of block-level ones, and the limitation on unmeasured hardware performance has been removed.

Q3. Matched-depth models and distillation. We trained models from scratch at each retained depth on three architectures: MLP-Mixer on CIFAR-10 and ViT-S/16 on ImageNet-100 (accuracy, three seeds), and GPT-2 (mean zero-shot accuracy); repeating SmolLM1's ~200B-token pretraining at several depths exceeded our rebuttal compute. Each such model needs its depth fixed before training and a separate pretraining run, whereas all Bandstop subnetworks come from one run. We compare them with Bandstop and Bandstop-sharp under β pruning, without fine-tuning (App. F.3, Table 15):

ModelLayers keptFrom scratchBandstopBandstop-sharp
MLP-Mixer1691.793.192.2
1291.691.092.2
891.271.989.1
ViT-S/161280.581.580.8
981.181.280.7
680.969.979.4
GPT-21240.6340.9039.31
1140.1440.4939.23
839.2736.1239.48
638.9632.8339.41

At full depth, the models trained from scratch are the Residual models.

  • Full depth: Bandstop exceeds every model trained from scratch, including the shallower ViTs that beat the 12-layer Residual.
  • Light removal: Bandstop stays comparable to the dedicated models with up to 8–25% of the layers removed (25% on MLP-Mixer and ViT, 8% on GPT-2).
  • Bandstop-sharp extends this range. It exceeds the 12-block Mixer and, in GPT-2 accuracy, the 8- and 6-layer models (which keep lower perplexity).
  • Half depth: the dedicated vision models lead Bandstop-sharp by 1.5 points on ViT and 2.1 on MLP-Mixer, whose from-scratch accuracy barely depends on depth (91.2–91.7%).

We did not train full per-depth distilled students, since each would add a teacher–student run for every depth. The closest counterpart in our results is Residual compressed with LLM-Streamline (Q1), whose replacement layer is distilled from the output of the removed blocks.

AuthorsOpenReview ↗

Continued [2]

Q4. Fine-tuning a pretrained model. A preliminary pilot shows that Bandstop can be introduced after pretraining, although we do not yet have a complete retrofit recipe. We inserted the gate into the pretrained Residual SmolLM1-135M using the fixed cutoffs of the separately pretrained Bandstop model. We faded it in, moving each βℓ\beta_\ell gradually from 1 (the unchanged Residual model) to the converged Bandstop gate, and continued training for 0.25B to 1B tokens (at most 0.5% of pretraining; three data orders). The control was Residual continued on the same tokens.

  • The retrofit makes moderate removal cheaper immediately. With 6 of 30 layers removed, the gate-selected window reaches PPL 19–24 versus 28.5 for the continued Residual's best window.
  • With 12 layers removed, the retrofitted model contains far better windows (PPL 41–109 vs. 294), although after the longer retrofits the gate no longer selects them.
  • Within this token budget, full-depth perplexity remains high (15.4 vs. 9.5 after 1B tokens). The model pretrained with the gate has full-depth perplexity 9.63 and reaches 20.4 with 12 layers removed.

Because this pilot borrows fixed cutoffs from the separately pretrained Bandstop model and does not recover full-depth perplexity, it is not yet a complete retrofit recipe. Learning the cutoffs during retrofit and scaling to larger models remain future work.

Q5. Low-pass and the middle-layer hypothesis. Low-pass's robustness is consistent with both claims. The least harmful layers of an ungated network lie in the middle, and Low-pass also compresses without a compression objective, only more strongly under heavy removal:

  • The removable depth lies in the middle. The Residual oracle, searching every contiguous window without a gate, removes an interior window. On SmolLM1 it never includes the first or last layer at any budget. On GPT-2 it includes neither until three quarters of the layers are removed, and on ViT-S/16 (ImageNet-100, three seeds) neither with up to 5 of 12 layers removed. With half of GPT-2 removed, the window is layers 4–9, the same as Bandstop's best window. Bandstop places its band where this removable depth already lies.
  • Compression emerges in both profiles. Low-pass is the same gate with a single front under the same task loss, so its robustness is emergent compression too, not a counterexample. Like LayerSkip and ACNs, it trains the network to remain complete when truncated, and this costs the full model: Low-pass falls below Residual on MLP-Mixer (90.7% vs. 91.7%) and GPT-2 (PPL 23.11 vs. 21.79), whereas Bandstop improves the full model on MLP-Mixer (93.1%) and stays close on GPT-2 (21.87).

At 25% removal the two profiles are on par (MLP-Mixer 90.95% vs. 90.75%; GPT-2 PPL 35.26 vs. 34.72). Low-pass has a clear advantage only at 50%, and this mainly reflects the suppression strength of the converged gate and layer selection rather than where the band lies (App. D): with matched suppression, Bandstop-sharp nearly closes the Mixer gap (89.1% vs. 90.2%), and on GPT-2, where Bandstop's β\beta window misses its oracle window by one layer, Bandstop is better under oracle selection (PPL 75.5 vs. 91.2).

By "emergent" we mean that the removable band forms under the task loss alone, without a compression objective or a prescribed budget; the profile's shape is a structural prior.

Q6. Standard vs. sharp, and learning τ\tau. The width τ\tau sets how strongly the band is suppressed (§2.3, "Full-depth quality and prunability"; Figure 8). Sharper transitions make the band easier to remove, while smoother ones favor the full model. This gives two operating points of one family:

  • Bandstop (τ=0.5\tau = 0.5 for vision, 0.750.75 for LMs) puts full depth first. It gives the strongest full-depth model among the pruning-aware methods (Q1) and stays comparable to training from scratch under light removal (Q3). It suits an unknown or mild reduction.
  • Bandstop-sharp (τ=0.25\tau = 0.25, βmin⁡=0\beta_{\min} = 0) puts prunability first. It is strongest at moderate removal and, in GPT-2 downstream accuracy, exceeds the 8- and 6-layer target-depth models, although those models retain lower perplexity.

Under heavy removal (Tables 1, 12), methods built for truncation are stronger but pay at full depth (Q1).

Learning τ\tau (App. E.3, Figure 9) is possible but softens the prior. We made τL\tau_L and τR\tau_R learnable on MLP-Mixer, all else unchanged (3 seeds):

  • Gate: the transitions widen until the band is a shallow dip, its lowest gate near 0.1.
  • Accuracy: the full model improves slightly (93.4% vs. 93.1%), but pruning falls at every budget, e.g. 62.0% vs. 71.9% with 8 of 16 blocks kept.

Cross-entropy alone spends the extra freedom on full-depth accuracy, so we fix τ\tau to choose the operating point.

Reviewer_4cuyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Bandstop, a residual architecture that scales each residual branch by a depth-dependent coefficient βℓ∈(0,1)\beta_\ell \in (0,1): xℓ=xℓ−1+βℓfℓ(xℓ−1)x_\ell = x_{\ell-1} + \beta_\ell f_\ell(x_{\ell-1}). The coefficient profile is built from two oppositely oriented sigmoid fronts over the layer index. Each front has a learned cutoff (cLc_L, cRc_R) and a fixed sharpness τ\tau. The fronts combine as g=sL+sR−sLsRg = s^L + s^R - s^L s^R, and a floor βmin⁡\beta_{\min} sets the lower bound of βℓ\beta_\ell. The resulting profile keeps a prefix and a suffix of layers active and attenuates a contiguous middle band. The two cutoffs are the sole added parameters and are trained together with the network weights under the task loss. After training, the paper removes layers by β\beta pruning: for a removal budget rr, it takes the rr layers with the lowest βℓ\beta_\ell, which form a contiguous window. This window is compared with the matched Residual network’s oracle window, obtained by exhaustive search on the validation set. Experiments cover four architectures: a 16-block MLP-Mixer on CIFAR-10/100, ViT-S/16 on ImageNet-100/1k, GPT-2 124M on ~10B FineWeb-Edu tokens, and SmolLM1-135M on ~200B SmolLM-corpus tokens. The authors report that β\beta-pruned Bandstop subnetworks reach higher accuracy (vision) and lower perplexity (language) than the Residual oracle subnetworks at matched retained depth, and that the β\beta-selected window overlaps with Bandstop’s own oracle window by 78–91%. For the language models, the paper applies the LLM-Streamline pruning-and-healing protocol and reports gains of up to 8.6 accuracy points and 17.7 stability points over Residual after direct removal. The paper also reports full-depth analyses. In the vision models, these cover effective rank, top-1 accuracy, and corruption robustness. In SmolLM1, they cover a logit-effect depth score, Tuned Lens top-5 overlap, reasoning-primitive accuracy, and few-shot sweeps. The appendices compare Bandstop with alternative sigmoid profiles (Low-, High-, Band-pass), frozen cutoffs, Stochastic Depth, LayerDrop, ACN, ReZero, and LayerScale.

Critical strengths

None

Critical weaknesses

W1. The reported gains follow from the imposed prior, and the paper’s own ablations do not support learning the profile during training.

The β\beta-pruned window consists of blocks trained with βℓ\beta_\ell near βmin⁡\beta_{\min} (10−410^{-4}–10−210^{-2}). Appendix B states that a small βℓ\beta_\ell attenuates both the block’s residual write and the gradient to its weights. Low-cost removal of these blocks is an expected consequence of the design, and the matched ungated Residual network is a baseline that the design wins by construction. The paper’s own results show where a benefit of training-time imposition should appear, and each points the other way:

  • Frozen cutoffs outperform learned cutoffs in 8 of 9 pruned settings (Table 7). The learned component, which the title frames as producing emergent compression, reduces prunability.
  • On GPT-2, models trained from scratch at the retained depth outperform healed Bandstop at the Mid and High budgets (Fig. 3; p.6 describes Bandstop as “the closest pruned model to the from-scratch reference” beyond the lowest budget).
  • Among the sigmoid profiles, Low-pass prunes better than Bandstop (Table 6: MLP-Mixer at 50% removal, 90.21 vs. 71.90; GPT-2 at 50% removal, PPL 91.21 vs. 122.26). The choice of Bandstop rests on full-depth performance.
  • The stated contribution that “previously proposed residual gating approaches fail to provide comparable gains” (p.2) rests on MLP-Mixer/CIFAR alone (Table 11). The language models have no ReZero or LayerScale comparison.
  • The later-layer analyses (Fig. 5) compare Bandstop with ungated Residual only. Without a frozen-cutoff or residual-scaling control, the higher depth score and the late Tuned Lens alignment are consistent with the suppression the design imposes.

Why it is consequential: Section 2.2 justifies the shape by citing evidence that middle layers of existing models are removable. The question the contribution depends on is why one should impose this shape during training rather than exploit the existing redundancy post hoc (e.g., LLM-Streamline, Structural Scalpel). The paper’s evidence leaves this question open.


W2. The main text is not self-contained, and the results that qualify its central claims appear only in the appendix.

The ICLR 2027 author guidelines state that reviewers are not required to read the appendix. Evaluated on the main text:

  • The metrics behind Section 3.3 and Fig. 3 (the logit-effect depth score, the reasoning-primitive tasks, and the stability score) are defined by citation. The depth score has no formula in the main text or in Appendix C. The reasoning-primitive tasks have no description or example anywhere in the paper.
  • The relationship to Structural Scalpel, which optimizes a two-sigmoid contiguous mask, appears in Appendix A (p.15). The main text cites it among post-training pruning methods (p.1).
  • Section 2.3 mentions the frozen-cutoff ablation without reporting its result. Tables 7 and 11, the two results most relevant to W1, appear in the appendix.
  • The layer-selection rule changes across figures and tables (β pruning, oracle search, LLM-Streamline selection, Every-Other, progressive readout). The same model at the same budget appears with different values: GPT-2 Bandstop at 50% removal has PPL 122.26 in Table 6 and 75.54 in Tables 7 and 9.

Why it is consequential: A reader of the main text reaches a different conclusion from a reader of the full 25 pages, and the evidence for W1 sits where the guidelines exempt reviewers from looking. Addressing this requires restructuring the main text and revising the framing of the title and abstract, which goes beyond the scope of a rebuttal revision. This sets the ceiling of my assessment.

Questions for authors

Questions that could change my assessment

  • Q1. What does imposing the profile during training provide that post-hoc selection on a standard model does not? Please reconcile this with Table 7 (frozen cutoffs outperform learned cutoffs in 8 of 9 pruned settings) and Fig. 3 (from-scratch shallow GPT-2 models outperform healed Bandstop at the Mid and High budgets). A from-scratch baseline at matched retained depth for MLP-Mixer/CIFAR-10 would address this within the rebuttal period.
  • Q2. For the language models, how do ReZero and LayerScale compare with Bandstop under the same training and tuning budget, both for contiguous pruning (the Fig. 1 protocol) and for the depth-score analysis (Fig. 5)? GPT-2 124M on FineWeb-Edu is a feasible setting. Without this comparison, the contribution stated on p.2 applies to MLP-Mixer/CIFAR.

Clarifications (these affect my Presentation score, not my rating)

  • C1. Please give the formula and aggregation of the depth score in Fig. 5, including which intervention and normalization from Csordás et al. (2025) you apply.

  • C2. Please describe each reasoning-primitive task in Table 4 with one example, and state how the mean is computed.

  • C3. Please give the formula of the stability metric in Fig. 3.

Satisfactory answers to Q1-Q2 would raise my assessment, but not to acceptance: the self-containedness issue in W2 requires restructuring the main text, which goes beyond the scope of a rebuttal revision.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4cuy

We thank the reviewer for the detailed assessment. Our intended setting is planned pretraining when several deployment depths may be needed: Bandstop replaces the Residual run with a single checkpoint evaluated jointly on full-depth quality and post-hoc removability. It improves vision accuracy, keeps language-modeling perplexity close to Residual, and its gate-selected subnetworks outperform Residual’s oracle-best contiguous subnetworks across all four architectures (Figure 1).

W1. Architectural objective and evidence.

The β\beta-pruned window consists of blocks trained with βℓ\beta_\ell near βmin⁡\beta_{\min} (10−410^{-4}–10−210^{-2}). Appendix B states that a small βℓ\beta_\ell attenuates both the block’s residual write and the gradient to its weights. Low-cost removal of these blocks is an expected consequence of the design, and the matched ungated Residual network is a baseline that the design wins by construction.

Attenuation alone does not make a block removable. To test this, we trained GPT-2 ReZero and LayerScale with the same recipe as Bandstop (new runs, Figure 11). ReZero provides a direct control: nine of its twelve scales converge to at most 0.05, below every gate in Bandstop’s six-block removal window. However, ReZero’s small scales occur almost everywhere and grow only in the last blocks, so they do not identify a removable region. Even with oracle window selection, ReZero has higher perplexity than Bandstop at every retained depth. LayerScale also degrades faster and is even worse than Residual’s oracle (Tables 1 and 16, Figure 11). What matters is therefore not suppression alone, but organizing it into a localized depth profile. The complete language-model and depth-score controls are reported under Q2.

Frozen cutoffs outperform learned cutoffs in 8 of 9 pruned settings. The learned component, which the title frames as producing emergent compression, reduces prunability.

Why use learned rather than frozen cutoffs? Learning allows the task loss to reclaim capacity from the initially suppressed band, which preserves full-depth quality, the first requirement for replacing Residual in planned pretraining. Across all three tested architectures, learned cutoffs outperform frozen cutoffs at full depth; on GPT-2, for example, they remain close to Residual in perplexity (21.87 vs. 21.79), whereas freezing raises it to 22.20. This gain comes from recruiting adjacent layers when they benefit the task, although those layers then become more costly to remove.

We therefore use learned cutoffs as the default operating point: not because they maximize prunability, but because they provide the better joint trade-off between full-depth quality and removal robustness. If prunability alone is prioritized, frozen cutoffs are stronger and still outperform Residual’s oracle after removal (Tables 10 and 11). The ablation thus compares a full-depth-oriented adaptive profile with a pruning-oriented fixed profile; it does not question the value of training-time preparation.

The two-sigmoid form is an imposed structural prior. By “emergent compression,” we mean that removability develops under the task loss alone, without a compression objective or prescribed target budget. We do not mean that the profile’s form is learned or that learning the cutoffs maximizes prunability.

Among the sigmoid profiles, Low-pass prunes better than Bandstop. The choice of Bandstop rests on full-depth performance.

This reflects the joint objective rather than a contradictory result. Low-pass is stronger under aggressive β\beta pruning, but at full depth it is worse than both Residual and Bandstop on MLP-Mixer and GPT-2 (Table 10). At 25% removal the two are on par. At 50% removal under oracle selection, Bandstop is more removable on GPT-2 (perplexity 75.5 vs. 91.2; see our reply to qh6F, Q5). Standard Bandstop is therefore our default when full-depth quality is required. Bandstop-sharp provides a separate operating point when prunability is prioritized, while frozen cutoffs and Low-pass serve as design ablations. Revised §2.3 now makes these trade-offs explicit.

The remaining W1 objections are addressed directly under Q1, which compares training-time preparation with post-hoc selection and matched-depth training, and Q2, which provides the requested language-model residual-scaling and depth-score controls.

AuthorsOpenReview ↗

Continued [1]

W2. Main-text completeness.

The main text is not self-contained, and the results that qualify its central claims appear only in the appendix.

The results that qualify the central claims were already reported in the submitted appendix: the depth profiles, frozen cutoffs, transition width and residual-scaling baselines. The revision brings them forward into the main text and fills the remaining gaps:

  • Trade-off (§2.3). A new paragraph summarizes the profile, cutoff and transition-width ablations: learned cutoffs favor full-depth quality, frozen ones favor removal. It defines the two operating points, standard Bandstop and Bandstop-sharp.
  • Baselines (Table 1). A new main-text table compares both operating points with target-depth training, LayerSkip, LayerDrop, LaCoOT, ReZero and LayerScale, including our new runs.
  • Selection rules (App. C, Table 7). Main-text captions state the rule used in each comparison, and one table lists all rules. App. C also explains that 122.26 and 75.54 come from the same GPT-2 checkpoint, under β\beta selection and under its oracle window.
  • Metrics (App. C). The depth score, the reasoning-primitive tasks (one example each) and the stability metric are now defined; see C1–C3 below.
  • Structural Scalpel. As discussed in the submitted App. A, it fits a fixed-width two-sigmoid mask to frozen pretrained weights before tuning adjacent layers. Bandstop instead trains the weights and two free endpoints jointly under the task loss, without prescribing a removal width or budget.

These changes fit within the existing structure of the paper, and the title and abstract remain accurate: "emergent compression" refers to removable depth that develops under the task loss alone (W1).

Q1. Training-time preparation versus post-hoc selection.

What does imposing the profile during training provide that post-hoc selection on a standard model does not? Please reconcile this with Table 7 (frozen cutoffs outperform learned cutoffs in 8 of 9 pruned settings) and Fig. 3 (from-scratch shallow GPT-2 models outperform healed Bandstop at the Mid and High budgets).

Training-time preparation changes the subnetworks available after training:

  • Contiguous selection. At a fixed budget, every contiguous post-hoc selector chooses among the windows evaluated by Residual’s oracle. Bandstop’s search-free window exceeds that oracle on all four architectures: removing 25% of ViT-S/16’s layers on ImageNet-1k gives 66.4% accuracy for Bandstop versus 30.8% for Residual’s best window (Figure 1).
  • Non-contiguous selection. SLEB and Block Influence also produce better subnetworks from Bandstop than from Residual (Table 16).
  • After repair. Unhealed Bandstop remains more accurate than healed Residual at every evaluated budget under LLM-Streamline (Figure 2).

The frozen-cutoff ablation (now Table 11) does not compare preparation with post-hoc selection: both variants are Bandstop models prepared during training and differ only in whether the cutoffs move. It identifies different full-depth/prunability operating points rather than testing whether preparation helps.

We also agree that dedicated shallow GPT-2 models outperform healed standard Bandstop at Mid and High removal. These models answer a different deployment question: each requires the target depth to be fixed before training and adds a separate pretraining run. Bandstop prepares multiple depths in one run.

A from-scratch baseline at matched retained depth for MLP-Mixer/CIFAR-10 would address this within the rebuttal period.

We added the requested MLP-Mixer/CIFAR-10 baseline and extended the comparison to ViT-S/16 on ImageNet-100 (Table 15), using β\beta selection without fine-tuning over three seeds:

Model / datasetLayers keptTarget-depth modelBandstopBandstop-sharp
MLP-Mixer / CIFAR-101691.793.192.2
1291.691.092.2
891.271.989.1
ViT-S/16 / ImageNet-1001280.581.580.8
981.181.280.7
680.969.979.4

How close the extracted subnetworks come depends on the operating point:

  • Standard Bandstop is on par with or above them up to 25% removal.
  • Bandstop-sharp extends this to larger budgets: on GPT-2, it reaches 39.48% vs. 39.27% at 8 layers and 39.41% vs. 38.96% at 6 layers in mean downstream accuracy, although the dedicated models retain lower perplexity (Tables 1 and 15); it also stays within 2.1 points of the half-depth vision models.

The distinction is deployment flexibility: each dedicated model requires a separate run at a depth fixed in advance, whereas one Bandstop checkpoint provides subnetworks at several deployment depths.

AuthorsOpenReview ↗

Continued [2]

Q2. Language-model residual-scaling controls.

For the language models, how do ReZero and LayerScale compare with Bandstop under the same training and tuning budget, both for contiguous pruning and for the depth-score analysis? GPT-2 124M on FineWeb-Edu is a feasible setting.

We trained GPT-2 ReZero and LayerScale under the same training and tuning budget as Bandstop: the same data, token budget, recipe and seed, with their published initializations. Like Bandstop, neither received GPT-2-specific tuning. Table 1 and Figure 11 report full-depth performance and contiguous pruning using Bandstop’s β\beta selection and oracle windows for the other models. Bandstop has lower perplexity than both controls at full depth and every reported retained depth:

GPT-2 perplexity ↓12 layers11 layers8 layers6 layers
Residual (oracle window)21.7925.05126.63308.61
Bandstop (β\beta window)21.8724.8648.20122.26
ReZero (oracle window)24.9427.0771.02162.68
LayerScale (oracle window)22.0527.30196.95714.84

We also added frozen-cutoff and residual-scaling depth-score controls (Table 14):

TaskResidualBandstopFrozen BandstopReZeroLayerScale
MATH3.534.554.564.704.26
MQuAKE3.584.474.514.584.37

All gated models score above Residual, so down-weighting residual branches generally shifts computation toward later layers, and frozen and learned cutoffs score similarly. This control isolates Bandstop’s distinct contribution: its structured depth allocation turns that shift into robust compression. Among these controls, only the Bandstop variants combine later-layer influence with competitive full-depth quality and removable contiguous depth.

C1. Depth score (App. C). We use the layer-skipping intervention with which Csordás et al. (2025, Sec. 3.2) measure a layer’s importance for future predictions. For block ℓ\ell and an answer position tt, we skip the block at positions before tt except the first, rerun the later blocks, and record the largest change in the output distribution at positions p≥tp\geq t:

\left\lVert \operatorname{softmax}(z_p)-\operatorname{softmax}(z_p^{\mathrm{clean}}) \right\rVert_2.$$ We use this logit effect rather than their relative change in each later layer’s contribution, so no per-layer normalization is applied. We take the maximum over four answer positions per example and, following Kapl et al. (2025), the mean over 100 examples, giving $\bar e_\ell$. The depth score is the effect-weighted mean layer: $$D=\frac{\sum_{\ell=0}^{L-1}\ell\,\bar e_\ell} {\sum_{\ell=0}^{L-1}\bar e_\ell},$$ whose denominator is the only normalization. A higher value means that later layers have greater influence on the prediction. **C2. Reasoning primitives** (App. C). The five depth-0 tasks of Saunshi et al. (2024) comprise: - copying sequences of random strings or real words; and - retrieving a variable’s value written as a basic list, equations or code. For example, copying presents `pum nyj gdq ocu rzk jbw mlz eny kyx uni rzk jbw mlz eny kyx`, whose next word is `uni`; variable assignment presents `o=14 s=4 u=8 m=10 q=12 m=`, with expected output `10`. Each query follows five solved examples and is scored using the highest-likelihood candidate. The reported mean is the unweighted average of the five task accuracies. Table 6 provides one example for each task. **C3. Stability** (Chen et al., 2025; App. C). Stability is the confidence-weighted fraction of examples on which the full and pruned models agree in being correct or incorrect: $$S=\frac{\sum_{i=1}^{N}e^{s_i}\mathbf{1}[i\in\mathrm{TP}\cup\mathrm{TN}]} {\sum_{i=1}^{N}e^{s_i}},$$ where TP and TN are the examples both models answer correctly and incorrectly, respectively, and $s_i$ is the standard deviation of the full model’s probabilities over the answer choices. LAMBADA uses uniform weights, and we report 100 times the unweighted mean over the six tasks.
Reviewer_5wwcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BandStop, which is a pruning-aware architecture and training technique that learns a continguous intermediate set of layers in Transformers where the contribution of those residual components are attenuated, meaning that they can later be pruned easily from the residual stream. This is accomplished via a learned gating mechanism that applies to each block within the residual stream; this gating mechanism includes an inductive bias where two "fronts" are learned to define a contiguous layer region of low coefficients. Results are shown for layer pruning with bandstop versus a vanilla Transformer baseline with oracle contiguous layer pruning, where BandStop is shown to improve upon the baseline.

Critical strengths
  1. Uncompressed performance remains strong: It is encouraging to see that this architectural modification maintains high performance before pruning, like in Table 2. This means BandStop is also just a good architectural change outside of the context of pruning as well.

  2. Solid experiments demonstrating later layer influence: The use of logit-effect depth score and tune lens predictions is useful to try to understand that despite the attentuation of middle layers, BandStop is reallocating depth and computation effectively in this way. It is nice to both propose a novel method and have some sort of investigative explanation as to why this still might be working out.

Critical weaknesses
  1. Point of outcompeting oracle-best pruning: The paper repeatedly refers to outperforming an oracle-best contiguous subnetworks, but this does not seem like a fair comparison or even a strong goal to achieve. Oracle-best contiguous subnetwork pruning does inclur L - r + 1 evaluations, but it does not require retraining models from scratch like Bandstop, and these evaluations are parallelizable as well. Additionally, this baseline improves with depth given prior work on layer pruning, but this paper evaluates models only up to 12-16 layers, which may only reflect a range of sizes in which BandStop may outperform this baseline, limiting its extensibility. This training from scratch weakness is the papers largest weakness; more motivation and explanation of applicability of a pruning-aware pretraining method would strengthen the paper.

  2. Clarity and motivation behind the specific design choices in BandStop: There is limited experimental evidence and description that supports many of the decisions made in section 2.1, including the inductive bias toward the contiguous middle region of layers. The choice of middle layers is discussed via prior work in section 2.2, but experimental support for this claim, the specific setup of bandstop (sigmoid fronts, etc), and contiguity is lacking. Additionally, it would be great to understand more how the specific questions 2-4 in section 2.1 came to be, in terms of the method design.

Questions for authors
  1. Is it certainly the case that continguous depth needs to be considered? Recent layer pruning work suggests that non-contiguous pruning can be helpful as well (see: Pruning by Block Benefit: Exploring the Properties of Vision Transformer Blocks during Domain Adaptation).
  2. Why was (0,1) chosen as the range of the depth-dependent coefficient β\beta? Could it be the case that β>1\beta > 1 for some layers could "make up for" other layers being pruned?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 5wwc

We thank the reviewer for the constructive comments and for recognizing Bandstop's strong full-depth performance and its reallocation of computation across depth.

W1. Oracle comparison, training from scratch, and model depth.

The paper repeatedly refers to outperforming an oracle-best contiguous subnetworks, but this does not seem like a fair comparison or even a strong goal to achieve. Oracle-best contiguous subnetwork pruning does inclur L - r + 1 evaluations, but it does not require retraining models from scratch like Bandstop, and these evaluations are parallelizable as well. Additionally, this baseline improves with depth given prior work on layer pruning, but this paper evaluates models only up to 12-16 layers, which may only reflect a range of sizes in which BandStop may outperform this baseline, limiting its extensibility. This training from scratch weakness is the papers largest weakness; more motivation and explanation of applicability of a pruning-aware pretraining method would strengthen the paper.

Scope and applicability. Bandstop is a residual parameterization for models that will be pretrained anyway, particularly when the eventual deployment depth is unknown or several inference budgets must be supported. It replaces the planned Residual run while keeping the same architecture, data, training recipe and token budget, and adds only two learned scalars, with no auxiliary objective or prescribed pruning budget. One checkpoint then supplies subnetworks at several depths. Bandstop therefore belongs to the family of training-time methods for reduced-depth execution, including LayerDrop, LayerSkip and LaCoOT, which we compare in the general response and new Table 1. It is not intended to replace post-hoc pruning of an existing checkpoint; post-hoc methods remain complementary and produce stronger subnetworks when applied to Bandstop in our experiments. The revised introduction and discussion now make this setting explicit.

What the oracle comparison establishes. At each removal budget rr, the Residual oracle evaluates all L−r+1L-r+1 contiguous windows on validation data and retains the best one. The comparison therefore gives Residual the strongest possible contiguous selector on the held-out metric, whereas Bandstop reads its window directly from the learned gate. Nevertheless, Bandstop produces better subnetworks on all four architectures and at every budget in Figure 1, for example 66.4% versus 30.8% on ImageNet-1k after removing 25% of ViT-S/16's layers. We agree that exhaustive window search is cheap and parallelizable; the point of this comparison is not the search cost, but that preparation during pretraining changes the quality of the subnetworks available afterward. The advantage also persists under non-contiguous selection with SLEB [1] or Block Influence [2] (Q1; App. F.4). Under LLM-Streamline, even unhealed Bandstop beats healed Residual (§3.1, Figure 2).

A complementary and demanding reference is a model trained directly at the target depth, which requires fixing that depth before training and one run per depth. A single Bandstop run comes close to it at 25% removal, matching the 9-layer ViT-S/16 on ImageNet-100 (81.2% vs. 81.1%) and approaching the 12-block MLP-Mixer (91.0% vs. 91.6%), while at full depth it is above the models trained at every depth (81.5% and 93.1%; App. F.3, Table 15). Under larger removal a dedicated model is better, but it requires its own run for every size.

Model depth. The evaluation is not limited to 12–16 layers: SmolLM1-135M has 30 layers and was pretrained on approximately 200B tokens. On our 30-layer model, the observed gap is larger: removing half the layers raises perplexity 2.7× for Bandstop versus 208× for the Residual oracle, compared with 5.6× versus 14.2× on the 12-layer GPT-2 (Figure 1). On downstream tasks after direct removal, Bandstop is up to 8.6 points more accurate than Residual and up to 17.7 points higher in stability, the agreement with the unpruned model's correctness (Figure 2).

[1] Song et al. SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks. ICML 2024.

[2] Men et al. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect. Findings of ACL 2025.

W2. Motivation behind the design choices.

There is limited experimental evidence and description that supports many of the decisions made in section 2.1, including the inductive bias toward the contiguous middle region of layers. The choice of middle layers is discussed via prior work in section 2.2, but experimental support for this claim, the specifci setp of bandstop (sigmoid fronts, etdc), and contiguity is lacking. Additionally, it would be great to understand more how the specific questions 2-4 in section 2.1 came to be, in terms of the method design.

We read "questions 2–4" as Equations 2–4, whose components each serve a distinct role:

  • Sigmoid fronts (Equation 2). Each front smoothly separates active and attenuated regions. Its learned cutoff sets the boundary location, while fixed τ\tau sets sharpness; together, the fronts define the profile with only two learned parameters.
  • Union of the fronts (Equation 3). Their union preserves an active prefix and suffix while attenuating a contiguous middle band, reflecting evidence that endpoint blocks have more distinct roles. In the profile ablations, Bandstop gives the strongest full-depth model. Attenuating the output end (Low-pass) sacrifices full-depth quality, while attenuating the input end (Band-pass on both architectures, High-pass on GPT-2) makes removal substantially more damaging (App. D, Table 10).
  • Floor (Equation 4) and width (Equation 2). The floor βmin⁡\beta_{\min} sets how much the attenuated blocks still contribute: small values act like zero, while larger values make removal costlier. The width τ\tau sets suppression strength: sharper transitions favor prunability, smoother ones full-depth quality (Figure 8). Revised §2.3 makes this trade-off explicit.

The data also support placing the band in the middle: Residual's best removal window stays in the interior, never touching the first or last layer on SmolLM1, and on GPT-2 only once 9 of its 12 layers are removed.

Q1. Contiguity.

Is it certainly the case that continguous depth needs to be considered? Recent layer pruning work suggests that non-contiguous pruning can be helpful as well (see: Pruning by Block Benefit: Exploring the Properties of Vision Transformer Blocks during Domain Adaptation).

Contiguity is a structural prior rather than a pruning-time requirement. Non-contiguous selectors remain applicable after training, and App. F.4 evaluates ShortGPT’s Block Influence and SLEB without healing (Table 16).

Layers removedContiguous window*Block InfluenceSLEB
GPT-2, 1 of 1225.1 / 24.926.3 / 23.825.1 / 23.8
GPT-2, 4 of 12126.6 / 48.2108.9 / 74.271.1 / 42.2
GPT-2, 6 of 12308.6 / 122.3605.3 / 122.3266.4 / 75.5
SmolLM1, 6 of 3028.4 / 14.732.4 / 14.422.1 / 13.7
SmolLM1, 12 of 30295.5 / 20.4927.8 / 24.376.1 / 20.1
SmolLM1, 18 of 303,098 / 48.48.1·10⁵ / 116.4709.5 / 45.3

Each cell reports Residual / Bandstop perplexity. *Oracle window for Residual, β\beta window (no search) for Bandstop.

SLEB improves Residual, but Bandstop’s search-free window remains better at every budget. On Bandstop, SLEB largely rediscovers the learned band (12 SmolLM1 layers removed: layers 8 and 10–20 vs. β window 9–20). Bandstop therefore concentrates removable depth into one region while remaining compatible with non-contiguous selection. Pruning by Block Benefit operates within ViT blocks and is complementary (App. A).

Q2. Range of βℓ\beta_\ell.

Why was (0,1) chosen as the range of the depth-dependent coefficient β\beta? Could it be the case that β>1\beta > 1 for some layers could "make up for" other layers being pruned?

The range spans identity (βℓ→0\beta_\ell\to0) and a standard residual block (βℓ=1\beta_\ell=1). To test whether βℓ>1\beta_\ell>1 lets active layers compensate for removed ones, App. E.3 relaxes the gate constraints on the 16-block MLP-Mixer/CIFAR-10 using the Bandstop recipe and three seeds.

  • Learned upper bound. We relax the gate range from βℓ∈(βmin⁡,1)\beta_\ell\in(\beta_{\min},1) to βℓ∈(βmin⁡,βmax⁡)\beta_\ell\in(\beta_{\min},\beta_{\max}), where βmax⁡\beta_{\max} is a learned, unbounded scalar initialized at 1 and optimized without weight decay. Training instead drives βmax⁡\beta_{\max} to about 0.37, and pruning matches standard Bandstop.
  • Free per-layer scales. We remove the fronts and give each block its own unbounded scale, initialized at its Bandstop value. Only the last one or two blocks grow above one, improving full-depth accuracy slightly (93.8% vs. 93.1%). However, the middle reopens and pruning robustness deteriorates: with 8 of 16 blocks kept, accuracy is 45.9% (oracle) versus 76.0% for Bandstop's oracle window and 71.9% for its β window.

Thus, allowing scales above one does not improve pruning robustness; removability depends on the structured depth profile.

Thanks for your response.

I appreciate the author's response regarding β\beta values, which resolves that question.

However, my concern regarding the central motivation of the work as well as key baselines still stands. I find the comment that "models that will be pretrained anyway" to carry a lot of weight that is connected with the motivation and comparisons in this work. While it is the case that there are more post-training methods that look at layer removal and fewer that operate at initialization time, comparing these together seems unsatisfactory, and using the former to motivate the latter also seems unsatisfactory. The authors reiterate the utility of equations 2-4 (sorry for the typo(s)), but the stated weakness has more to do with what led up to the creations of the design choices of the method, rather than more surface level what they do. Additionally, the authors respond to post-training pruning in an non-contiguous manner, but do not address the potentiality/discuss that a method that does not require contiguousness but begins at train time is also possible. This also ties back to the decisions regarding the inductive bias created, which may again be unsufficiently motivated. Therefore, I maintain my current score.

AuthorsOpenReview ↗

Response to Reviewer 5wwc (follow-up)

We thank the reviewer for the follow-up and are glad the question on β\beta is resolved. We address the two remaining concerns in turn.

Motivation and key baselines ("models that will be pretrained anyway"). We agree that this phrase does not motivate the method by itself. When a checkpoint already exists, post-hoc pruning remains the natural choice. Bandstop targets a planned run whose model must later serve several depths, the setting of LayerDrop, LayerSkip and LaCoOT.

The motivation is that the task loss optimizes only the full-depth network. Shallower subnetworks and removable redundancy may arise incidentally, but neither their quality nor their placement is an objective. Post-hoc methods exploit what arises; training-time methods prepare the network for reduced depth through layer dropping or an auxiliary loss, often at some cost to the full model (Table 1). Bandstop asks whether a structured residual scale alone can do this while keeping full-depth quality. Post-hoc results enter only as evidence for where to attenuate.

The comparisons follow from this. Our key baselines are these training-time methods and models trained at each target depth (Table 1). Bandstop gives the strongest full model among the pruning-aware methods, above Residual on MLP-Mixer and close to it on GPT-2, and stays ahead of LayerSkip, LayerDrop and LaCoOT with 12 of 16 blocks kept on MLP-Mixer. Bandstop-sharp extends this lead to 8 of 16 blocks, and to 8 of 12 layers on GPT-2, while LayerDrop leads under the heaviest removal.

Figure 1 compares Bandstop with the same network trained without the gate, on the same data and training budget. Bandstop's β\beta-pruned window outperforms Residual's best window at every removal budget shown, on all four architectures, so training with the gate yields better subnetworks. Residual's window is selected post hoc only to give it its strongest subnetwork, not as a competing method. Figure 2 and App. F.4 apply the same post-hoc selectors to both models, and Bandstop stays ahead under each.

Design choices and non-contiguous training-time methods. We started from three requirements, each leading to one component of the gate:

  1. Keep both ends active (Eq. 3). Our own experiments point to the middle. In the matched Residual models, the cheapest removal window lies in the interior. Among the profiles the same fronts can define, attenuating the input end makes removal costly, while attenuating the output end weakens the full model (Table 10). This agrees with the distinct roles of layers near both ends, and with depth-growth methods that add adaptable capacity in the middle during training (§2.2). We therefore combine two opposite fronts that preserve the prefix and suffix.
  2. Keep the region intact without an extra loss (Eq. 2). The sigmoids tie every layer's scale to the two cutoffs, so layers outside them stay active and layers between them stay attenuated. Plain learnable gates would need a loss across layers to enforce these regions, or a parameterization that couples them, as the fronts do. Left free, the gates in the band rise and the middle reopens (App. E.3, Figure 10). With the fronts, training can only move the two edges, recruiting layers there (App. B), so the interior remains suppressed and the gate itself gives the removal window. τ\tau is fixed because learning it flattens the band (Figure 9).
  3. Bound the attenuation (Eq. 4). Restricting βℓ\beta_\ell to (βmin⁡,1)(\beta_{\min},1) attenuates the band softly, so its blocks stay in training and can be recruited.

We agree that non-contiguous preparation at training time is possible; our earlier answer covered only post-hoc selectors. Table 1 already includes different training-time priors: LayerSkip prepares prefix execution and removes a contiguous suffix, LayerDrop uses its non-contiguous Every-Other rule, and LaCoOT removes individually collapsed blocks.

To test contiguity within a structure like ours, we froze the gate in both arms, since two cutoffs cannot express an interleaved profile. We trained MLP-Mixer with the Bandstop gate and with the same 16 gate values reassigned to interleaved layers. Both keep gate mass, the first- and last-layer values, initialization and recipe, so only placement differs. Each prunes its lowest-β\beta layers.

Blocks removed (of 16)0481012
Frozen Bandstop gate (contiguous)92.891.584.869.645.3
Same gate values, interleaved92.590.381.355.944.9

Top-1 (%) on CIFAR-10, mean of three seeds.

Both placements tolerate removal, but arranging the suppression as one band makes removal cheaper at every budget up to 10 removed, until both models drop to about 45% at 12. Contiguity is therefore not required, but in this matched comparison on MLP-Mixer it is the better arrangement.

We hope this clarifies the reviewer's remaining concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their feedback. Below we restate the setting of Bandstop, summarize the main evidence, and list the changes in the revised PDF. The per-reviewer threads give the full tables.

Setting. Bandstop is a residual parameterization for a model that will be pretrained anyway, but whose deployment depth is unknown or must cover several budgets. It replaces the planned Residual run (same architecture, data and recipe) and adds two learned scalars, with no auxiliary loss and no pruning budget. It must therefore (i) keep the full model on par with Residual and (ii) yield, from that one checkpoint, subnetworks at several depths without a search. It does not replace post-hoc pruning, which remains complementary and works better on Bandstop.

Main evidence.

  • Full depth. Bandstop improves vision accuracy and keeps language-modeling quality on par with Residual, whereas every pruning-aware baseline we trained falls below Residual.
  • Subnetworks. On all four architectures, the subnetworks extracted from Bandstop outperform the best contiguous subnetworks of Residual. This holds when Residual's subnetworks are chosen by non-contiguous selectors (SLEB, Block Influence) and after LLM-Streamline repair, where even unhealed Bandstop subnetworks beat healed Residual ones.
  • Role of the structured profile. ReZero and LayerScale also learn small scales but degrade much faster after removal, and replacing the two fronts with free per-layer scales reopens the middle band.
  • One run versus one model per depth. Under light removal, Bandstop comes close to or matches models trained at the target depth. Bandstop-sharp extends this to larger budgets, while dedicated models keep the lead at half depth on vision and in GPT-2 perplexity.
  • Beyond compression. In vision, Bandstop learns lower-rank representations and is more robust to input corruption. In language models, its later layers have greater influence on predictions. It also improves aggregate reasoning-primitive accuracy, with larger in-context gains on variable assignment.

Changes in the revision.

  • Main text. A new Table 1 compares Bandstop and Bandstop-sharp with LayerSkip, LayerDrop, LaCoOT, ReZero, LayerScale and target-depth training. A new paragraph in §2.3 explains how the profile, the cutoffs and the transition width set the trade-off between full-depth quality and prunability. The introduction and related work now discuss LayerSkip and LaCoOT.
  • New experiments. We trained new baselines: LayerSkip and LaCoOT on MLP-Mixer and GPT-2 (App. F.1), ReZero and LayerScale on GPT-2 (App. F.2), and models trained from scratch at matched depth on MLP-Mixer and ViT (App. F.3). We also added non-contiguous selectors (App. F.4), relaxed-gate ablations (App. E.3) and measured inference cost (App. C).
  • Definitions. App. C defines the depth score, the reasoning-primitive tasks (one example each), the stability metric and the healing protocol. It also lists every layer-selection rule in one table.
  • Protocol correction. In the submitted Figure 3, the healed Bandstop and Residual models received a second, cross-entropy fine-tuning stage of the replacement MLP that LayerDrop did not. The revised Figure 2 heals every model with the first stage only. The conclusions are unchanged, except that unhealed Bandstop now also beats healed Residual at the highest GPT-2 budget.
AuthorsOpenReview ↗

Revised PDF after the discussion

We thank the reviewers for the discussion. We have uploaded a revised PDF that implements the changes committed in our follow-up replies. No other results or claims have changed.

  • Title and abstract. We removed "emergent" from the title, which is now "Bandstop: Compression via Residual Flow Throttling", and from the rest of the text. The abstract now separates the two roles discussed with Reviewers qh6F and 4cuy: the imposed profile attenuates a contiguous middle region, while training adapts its two cutoffs. The introduction and the discussion follow the same framing.
  • Localization of removable depth (§3.1, Table 2). Table 2 now reports the maximum regret of the gate-selected window over all budgets up to each removal level, replacing the mean regret and window overlap of the previous version (Reviewer qh6F). The localization claim is restricted to low and moderate budgets, and we note that it extends further when the attenuated band is wide. The contribution list and the discussion state the same scope.
  • Operating points (§2.3). We state that the choice between Bandstop and Bandstop-sharp reflects the expected range of removal (Reviewer qh6F).

The placement control with interleaved gate values is reported in our reply to Reviewer 5wwc.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.