- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Based on the observation that the middle layers of a neural network are redundant and nearly interchangeable, the paper proposes a method called Bandstop, which learns two parameters to attenuate the contributions of the residual flow, thereby reducing the depth of networks with residual layers. Experimentally, the authors compare networks whose depth has been reduced using Bandstop with the same residual network that has not been trained to be pruned. Experiments cover vision (MLP-Mixer, ViT-S) and language models (GPT-2 124M, SmolLM1 135M).
Critical strengths
Bandstop is a simple method with negligible additional cost. The approach introduces only two trainable scalars for the entire network, without requiring an auxiliary compression objective. The memory and computational costs during training are negligible with respect to a standard Residual training.
To remove layers, Bandstop does not use combinatorial search or importance estimation on a validation set: the layers to be removed are determined directly by reading the lowest values of (no search and no validation access).
Critical weaknesses
The experimental comparisons in the main body are limited. The main comparison (Figures 1 and 3) is conducted almost exclusively against a standard residual baseline evaluated by an oracle. However, a conventional residual model is never trained to direct the calculation toward the identity at the center.
The paper omits closely related work that prepares models for layer skipping/removal. For example:
- LayerSkip [1]: a related method that prepares for execution at a reduced depth, via progressive layer dropout and early-exit training.
- LaCoOT [2]: a related method that enforces intermediate feature distributions toward identity collapse using optimal transport (Max-Sliced Wasserstein distance), enabling search-free block deletion post-training without combinatorial validation search
Bandstop is restricted to strictly constant-width, constant-resolution architectures. All experiments are based on models where the representation dimension is strictly constant (ViT, MLP-Mixer, GPT-2, SmolLM). The paper does not address widely used residual architectures with changes in scale or resolution across a stage boundary (for example: ResNet or ConvNeXt), for which contiguous deletion across a resolution jump mathematically breaks the identity operation.
The scaling of the method has not been demonstrated. The models evaluated are relatively small ( parameters) and are all trained from scratch. The cost of applying Bandstop to modern foundation models (7B+) is not discussed.
Questions for authors
-
Missing baselines & positioning. How does Bandstop compare conceptually and empirically to recent training-time depth compression frameworks such as LayerSkip [1] and LaCoOT [2], which prepare networks for reduced-depth execution or block removal after training?
-
Physical removal vs. masking. Is the removal a true removal of the block (computational elimination) or a masking? Figure 3 reports a “measured forward-pass speedup” without hardware or protocol, even though the limitations state that hardware performance is not measured. Can the authors report the actual latency, FLOPs, and memory usage, along with the hardware in at least one setup?
-
Comparison to matched-depth models. Even though the value of compression methods lies in the fact that one does not know a priori what the optimal size of the neural network is for a given task, it would be interesting to see to what extent the proposed method achieves a compressed model with performance comparable to that of the same network trained from scratch. How does a pruned Bandstop subnetwork compare against a matched-depth model trained from scratch (and against a standard knowledge-distilled counterpart) across the four architectures?
-
Scalability and applicability to a Bandstop fine-tuning version. Does Bandstop necessarily require full training from scratch? Would it be possible to initialize the thresholds and and fine-tune a standard pre-trained model (e.g. a LLaMA-7B) using a limited token budget to induce this retractable band?
-
Low-pass vs Bandstop discrepancy. In Table 6, Low-pass appears to be more robust than Bandstop at of the layers. Does this contradict the hypothesis that intermediate layers are the region most susceptible to removal and the claim of “emergent” compression?
-
High-compression regime and the role of Bandstop-sharp. In Tables 8 and 9, Bandstop fails compared to Stochastic Depth, LayerDrop, or ACN under heavy pruning (e.g., of the layers). What is the theoretical or practical justification for choosing between the standard profile and the Bandstop-sharp variant? Is it possible to learn the slope rather than fixing it?
References:
[1] Elhoushi, et al. “Layerskip: Enabling early exit inference and self-speculative decoding”. ACL 2024
[2] Quétu, et al. “LaCoOT: Layer Collapse through Optimal Transport”. ICCV 2025
Given the current state of the paper, I am leaning toward a weak rejection. However, if the points raised are sufficiently addressed during the rebuttal, I am willing to increase my score.
5 comments
Continued [1]
Q1 (continued). Reading the table.
- Full depth. Among the pruning-aware methods, Bandstop gives the strongest full-depth model. Relative to Residual, it improves MLP-Mixer accuracy (93.1% vs. 91.7%) and essentially matches GPT-2 perplexity (21.87 vs. 21.79), while LayerSkip, LayerDrop and LaCoOT lose 0.3–1.3 points and add 0.4–2.3 to perplexity.
- Where Bandstop wins and where the others get stronger. Under light removal, standard Bandstop leads the non-Bandstop methods on MLP-Mixer, while LayerSkip's early exit leads in GPT-2 perplexity. Under moderate removal, LayerSkip and LayerDrop overtake standard Bandstop. There, Bandstop-sharp is the strongest method in both tasks, and it also leads with 12 Mixer blocks kept.
- Common rule and replacement. Under LLM-Streamline, Bandstop is the most accurate model with one layer removed, and Bandstop-sharp with 8 and 6 layers kept, healed or unhealed. Healing improves Bandstop at every budget, but not LayerSkip or LayerDrop. In mean accuracy, healed Bandstop beats every non-Bandstop method at every budget, and even unhealed it beats healed Residual.
- Heavy removal. At 4 Mixer blocks, LaCoOT with λ = 5 leads overall (80.8%, Table 12) but falls to 83.1% at full depth; among the Table 1 configurations, LayerDrop leads with 72.1%. LayerDrop also leads half-depth GPT-2 perplexity. Bandstop-sharp instead leads half-depth GPT-2 accuracy and, without healing, retains its full-depth accuracy (39.41% vs. 39.31%).
Q2. Physical removal and inference cost. Removal is physical: the blocks are deleted, and the outputs are identical to those of the masked evaluation. Cost depends only on the number of retained layers, so Bandstop and Residual have the same parameters, FLOPs, latency and memory at each budget. App. C (Table 8) reports these on one NVIDIA A100 64 GB. The protocol is bf16 prefill of a batch of 8 full-length sequences (1,024 tokens for GPT-2, 2,048 for SmolLM1), with the minimum over 10 trials of 100 timed iterations after warm-up:
| Model | Layers kept | Params | FLOPs/token | Latency | Speedup | Peak memory |
|---|---|---|---|---|---|---|
| GPT-2 124M | 12 | 124.5M | 285M | 15.1 ms | 1.00× | 1557 MiB |
| 11 | 117.4M | 268M | 14.1 ms | 1.07× | 1515 MiB | |
| 8 | 96.1M | 216M | 11.0 ms | 1.38× | 1390 MiB | |
| 6 | 81.9M | 181M | 8.9 ms | 1.70× | 1306 MiB | |
| SmolLM1-135M | 30 | 134.5M | 411M | 83.5 ms | 1.00× | 2379 MiB |
| 24 | 113.3M | 340M | 67.6 ms | 1.23× | 2256 MiB | |
| 18 | 92.0M | 269M | 51.8 ms | 1.61× | 2133 MiB | |
| 12 | 70.8M | 198M | 35.9 ms | 2.32× | 2009 MiB |
Figure 2 (Figure 3 in the submission) now reports these end-to-end speedups instead of block-level ones, and the limitation on unmeasured hardware performance has been removed.
Q3. Matched-depth models and distillation. We trained models from scratch at each retained depth on three architectures: MLP-Mixer on CIFAR-10 and ViT-S/16 on ImageNet-100 (accuracy, three seeds), and GPT-2 (mean zero-shot accuracy); repeating SmolLM1's ~200B-token pretraining at several depths exceeded our rebuttal compute. Each such model needs its depth fixed before training and a separate pretraining run, whereas all Bandstop subnetworks come from one run. We compare them with Bandstop and Bandstop-sharp under β pruning, without fine-tuning (App. F.3, Table 15):
| Model | Layers kept | From scratch | Bandstop | Bandstop-sharp |
|---|---|---|---|---|
| MLP-Mixer | 16 | 91.7 | 93.1 | 92.2 |
| 12 | 91.6 | 91.0 | 92.2 | |
| 8 | 91.2 | 71.9 | 89.1 | |
| ViT-S/16 | 12 | 80.5 | 81.5 | 80.8 |
| 9 | 81.1 | 81.2 | 80.7 | |
| 6 | 80.9 | 69.9 | 79.4 | |
| GPT-2 | 12 | 40.63 | 40.90 | 39.31 |
| 11 | 40.14 | 40.49 | 39.23 | |
| 8 | 39.27 | 36.12 | 39.48 | |
| 6 | 38.96 | 32.83 | 39.41 |
At full depth, the models trained from scratch are the Residual models.
- Full depth: Bandstop exceeds every model trained from scratch, including the shallower ViTs that beat the 12-layer Residual.
- Light removal: Bandstop stays comparable to the dedicated models with up to 8–25% of the layers removed (25% on MLP-Mixer and ViT, 8% on GPT-2).
- Bandstop-sharp extends this range. It exceeds the 12-block Mixer and, in GPT-2 accuracy, the 8- and 6-layer models (which keep lower perplexity).
- Half depth: the dedicated vision models lead Bandstop-sharp by 1.5 points on ViT and 2.1 on MLP-Mixer, whose from-scratch accuracy barely depends on depth (91.2–91.7%).
We did not train full per-depth distilled students, since each would add a teacher–student run for every depth. The closest counterpart in our results is Residual compressed with LLM-Streamline (Q1), whose replacement layer is distilled from the output of the removed blocks.
Continued [2]
Q4. Fine-tuning a pretrained model. A preliminary pilot shows that Bandstop can be introduced after pretraining, although we do not yet have a complete retrofit recipe. We inserted the gate into the pretrained Residual SmolLM1-135M using the fixed cutoffs of the separately pretrained Bandstop model. We faded it in, moving each gradually from 1 (the unchanged Residual model) to the converged Bandstop gate, and continued training for 0.25B to 1B tokens (at most 0.5% of pretraining; three data orders). The control was Residual continued on the same tokens.
- The retrofit makes moderate removal cheaper immediately. With 6 of 30 layers removed, the gate-selected window reaches PPL 19–24 versus 28.5 for the continued Residual's best window.
- With 12 layers removed, the retrofitted model contains far better windows (PPL 41–109 vs. 294), although after the longer retrofits the gate no longer selects them.
- Within this token budget, full-depth perplexity remains high (15.4 vs. 9.5 after 1B tokens). The model pretrained with the gate has full-depth perplexity 9.63 and reaches 20.4 with 12 layers removed.
Because this pilot borrows fixed cutoffs from the separately pretrained Bandstop model and does not recover full-depth perplexity, it is not yet a complete retrofit recipe. Learning the cutoffs during retrofit and scaling to larger models remain future work.
Q5. Low-pass and the middle-layer hypothesis. Low-pass's robustness is consistent with both claims. The least harmful layers of an ungated network lie in the middle, and Low-pass also compresses without a compression objective, only more strongly under heavy removal:
- The removable depth lies in the middle. The Residual oracle, searching every contiguous window without a gate, removes an interior window. On SmolLM1 it never includes the first or last layer at any budget. On GPT-2 it includes neither until three quarters of the layers are removed, and on ViT-S/16 (ImageNet-100, three seeds) neither with up to 5 of 12 layers removed. With half of GPT-2 removed, the window is layers 4–9, the same as Bandstop's best window. Bandstop places its band where this removable depth already lies.
- Compression emerges in both profiles. Low-pass is the same gate with a single front under the same task loss, so its robustness is emergent compression too, not a counterexample. Like LayerSkip and ACNs, it trains the network to remain complete when truncated, and this costs the full model: Low-pass falls below Residual on MLP-Mixer (90.7% vs. 91.7%) and GPT-2 (PPL 23.11 vs. 21.79), whereas Bandstop improves the full model on MLP-Mixer (93.1%) and stays close on GPT-2 (21.87).
At 25% removal the two profiles are on par (MLP-Mixer 90.95% vs. 90.75%; GPT-2 PPL 35.26 vs. 34.72). Low-pass has a clear advantage only at 50%, and this mainly reflects the suppression strength of the converged gate and layer selection rather than where the band lies (App. D): with matched suppression, Bandstop-sharp nearly closes the Mixer gap (89.1% vs. 90.2%), and on GPT-2, where Bandstop's window misses its oracle window by one layer, Bandstop is better under oracle selection (PPL 75.5 vs. 91.2).
By "emergent" we mean that the removable band forms under the task loss alone, without a compression objective or a prescribed budget; the profile's shape is a structural prior.
Q6. Standard vs. sharp, and learning . The width sets how strongly the band is suppressed (§2.3, "Full-depth quality and prunability"; Figure 8). Sharper transitions make the band easier to remove, while smoother ones favor the full model. This gives two operating points of one family:
- Bandstop ( for vision, for LMs) puts full depth first. It gives the strongest full-depth model among the pruning-aware methods (Q1) and stays comparable to training from scratch under light removal (Q3). It suits an unknown or mild reduction.
- Bandstop-sharp (, ) puts prunability first. It is strongest at moderate removal and, in GPT-2 downstream accuracy, exceeds the 8- and 6-layer target-depth models, although those models retain lower perplexity.
Under heavy removal (Tables 1, 12), methods built for truncation are stronger but pay at full depth (Q1).
Learning (App. E.3, Figure 9) is possible but softens the prior. We made and learnable on MLP-Mixer, all else unchanged (3 seeds):
- Gate: the transitions widen until the band is a shallow dip, its lowest gate near 0.1.
- Accuracy: the full model improves slightly (93.4% vs. 93.1%), but pruning falls at every budget, e.g. 62.0% vs. 71.9% with 8 of 16 blocks kept.
Cross-entropy alone spends the extra freedom on full-depth accuracy, so we fix to choose the operating point.
Response to Reviewer qh6F
We thank the reviewer for recognizing Bandstop’s negligible overhead and search-free gate-based selection.
W1. Main-body baselines.
We agree these baselines should be prominent in the main text. New Table 1 compares Bandstop with LayerSkip, LayerDrop, LaCoOT, ReZero, LayerScale and target-depth training; Figure 2 adds LayerSkip under the shared LLM-Streamline rule, with LaCoOT in Table 13. Stochastic Depth and ACN remain in the appendix.
W2. Related work.
We agree these closely related methods should be discussed explicitly. The revision adds LayerSkip and LaCoOT to the introduction and App. A and compares them directly under Q1.
W3. Stage-wise architectures.
We agree that stage-wise architectures are an important extension. Our experiments use constant-shape stacks; a natural design would use one gate per stage while keeping transition blocks active. Because this remains untested, §4 lists it as future work.
W4. Scale.
Bandstop adds only two scalars, one scaling operation per block and no auxiliary loss, so relative overhead decreases with width. Our evidence addresses depth rather than parameter-count scaling. At half-depth, perplexity rises 5.6× vs. 14.2× on 12-layer GPT-2 and 2.7× vs. 208× on 30-layer SmolLM1. Q4 reports a preliminary retrofit. Matched 7B+ pretraining exceeded our compute, so §4 limits claims to at most 135M parameters.
Q1. LayerSkip and LaCoOT. Conceptually, the three methods differ in what they add to training and in where they place the removable depth:
LayerSkip and LaCoOT prepare reduced depth through their training objectives. Bandstop instead uses the residual parameterization, requiring neither intermediate predictions nor a distribution regularizer and leaving the training loop unchanged.
Empirically, we trained both methods on MLP-Mixer and GPT-2 with architecture-specific recipes (LaCoOT with λ=1; Table 12 also reports λ=0.1 and 5). The table combines each method's own removal rule (Table 1; App. C, Table 7) with the shared LLM-Streamline rule (Figure 2, Table 13), which selects the window with the most similar input and output and optionally replaces it with a lightweight MLP. MLP-Mixer values average three seeds, GPT-2 values are single runs; bold marks the best method.