acceptodds
Withdrawn from ICLR 2027

BPAW-VIT: Block Pruning and Adaptive Width for Efficient Vision Transformer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In recent times, there has been a paradigm shift towards fine-tuned models rather than full-scale pre-training, owing to the availability of large corpora and pre-trained models on platforms like Hugging Face. The aim is to save significantly on computational costs during training. While Vision Transformers (ViTs) achieve remarkable performance, their deployment is hindered by high computational costs. Existing depth-Pruning methods such as block-dropping techniques like Entrodrop disrupt feature spaces, inducing severe loss in accuracy. Other depth-pruning methods, such as HeDEPT and NOSE, replace pruned layers with parameter-heavy linear substitutions. To achieve true parameter sparsity without architectural distortion, we propose a BPAW-ViT framework that unifies Alpha-Gated Progressive Layer Pruning (depth) with slimmable feed-forward networks (width Slicing). Guided by an entropy-alignment criterion and a dual-width training recipe, our framework dynamically reduces redundant blocks to pure identity mappings or slices intermediate capacities, requiring zero extra parameters. Finally, as a natural extension, we integrate this framework with Early-Exit architectures (EnViT). Because alpha-gated pruning preserves monotonic hidden states and intact 2D spatial resolution, our joint approach enables us to achieve a higher speedup (up to 2.45) with a significantly reduced memory footprint.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Transformer, Block Pruning, Adaptive Width, Early Exit, Memory Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NZBKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a systematic framework that targets real problems in the deployment of vision transformers under resource constraints, where it compresses the model at the macro level through progressive block dropping with entropy initialized gates, and at the micro level by dynamically slimming the FFN width. It then combines these approaches with established early exit techniques for additional system level benefits. Table 1 reports large accuracy gains for the macro level block dropping method against its peers, but unclear comparison budgets and text–table inconsistencies make these gains difficult to assess. Section 3.2 provides a more nuanced picture of the interaction between the two pruning methods that is worth further investigation. Finally, the reported results suggest that combining these methods with existing early exit techniques could yield system level benefits at approximately the same level of predictive performance, but there is ambiguity about how certain claimed benefits are measured, with limited supporting data or elaboration.

Critical strengths
  1. The research question is real and well motivated, and this paper provides a useful exploration of combinations of different methods that could each bring practical benefits.
  2. The method is tested on several different datasets and compared against competing methods, providing evaluation beyond a single benchmark.
Critical weaknesses
  1. Insufficient details are provided about the experimental results. Important information regarding comparison budgets, speedup definitions and timing measurements, memory reduction, training and recovery schedules, the detailed configurations of competing methods, and hardware is missing or unclear. This severely limits the verifiability and reproducibility of the claims.
  2. Insufficient ablation studies are provided to isolate the contributions of progressive block dropping and two-width training. While the exploratory investigation in Section 3.2 is useful, it doesn’t establish which parts of the training recipe account for the observed gains, and it limits the confidence in selecting reliable deployment configurations without further empirical validation.
  3. Several accuracy claims in the text are inconsistent with the paper’s own tables. Section 3.1 reports 91.37% on Food-101, while table 1 has 90.78%. The claimed improvements of 4.05% and 2% on Stanford Cars and Flowers-102 also don’t match the corresponding entries of the same table. Section 3.4’s claim of consistently outperforming EnViT in accuracy is again not supported by the Food-101 result in Table 3 (89.03% vs 89.15%). These inconsistencies on top of missing experimental details unfortunately raise concerns about the reliability of the reported results.
Questions for authors
  1. Could you provide more details on how these metrics are measured, what budgets and baseline settings are used, and how much the results vary across runs? This would help establish whether the claimed advantages are reliable.
  2. Beyond the exploratory analysis in Section 3.2, could you provide focused ablations of progressive block dropping and two-width training under comparable training and resource budgets? Further investigation of how they interact and when they may fail would help identify reliable configurations for deployment.
  3. Could you check the discrepancies between the text and Tables 1 and 3 against the experiment records? Please clarify the correct values and settings, and whether the corrections affect the conclusions.
Reviewer_CCsvOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper prunes ViT depth in the fine-tuning setting by wrapping each block with a learnable gate that interpolates between the identity and the block. Training runs in two phases: Phase 1 fine-tunes with near-frozen gates, and Phase 2 re-initializes the gates from an entropy-based criterion, adds a bimodal gate loss and a budget-pressure term, and progressively drops the weakest block. The FFNs of the surviving blocks are further made slimmable with a dual-width recipe, so that a single checkpoint yields a depth-and-width Pareto frontier. The pruned model is also combined with early-exit routing. Evaluation covers four transfer benchmarks fine-tuned from ImageNet-pretrained ViT-B and DeiT-B, against block-dropping, compensatory-layer pruning, and early-exit baselines.

Critical strengths
  1. Permanent block removal reduces worst-case memory compared with early-exit methods, and the pruned model remains compatible with early-exit routing.
  2. A single fine-tuned checkpoint yields a two-dimensional depth-and-width frontier, which is practical for deployment-oriented selection.
Critical weaknesses
  1. All results are on four small benchmarks, with no ImageNet-1K experiment and no hardware-measured latency or memory to support the edge-deployment claims.
  2. The method composes existing components, namely residual gates, entropy-based initialization, slimmable FFNs, and early-exit routing, and the writing contains duplicated sentences, malformed references, and inconsistent naming.
  3. The paper has no method overview figure, although the pipeline combines gating, two-phase pruning, and dual-width training.
Questions for authors
  1. Please report measured wall-clock latency and peak memory on a concrete device, rather than FLOPs-derived speedups.
  2. Please provide a sensitivity analysis of the progressive-drop thresholds and the minimum-keep constraint, since Table 4 hand-tunes them per dataset without studying their effect.
  3. Please add a framework diagram of the full two-phase pipeline.
Reviewer_buq6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article presents BPAW-ViT, a framework for vision transformers operating during the finetuning phase. The introduced method combines three mechanisms :

  1. a progressive block pruning (Alpha-Gated progressive layer pruning) relying on learnable scalar gates based on an entropy criterion which progressively reduces the less contributing blocks to identity mapping without adding compensating parameters
  2. a width reduction of FFN layers with slimmable networks trained with a two distinct forward passes (full-width and slim-width pass)
  3. the integration of the pruned architecture with an existing early-exit strategy (EnViT).

The experimental evaluation is performed on a ViT-B/16 on CIFAR-100, Food-101, Stanford Cars and Flowers-102.

Critical strengths

Relevance of the combination of approaches. The idea of combining deep structural pruning (which effectively reduces resident memory in the worst-case scenario), width-based elasticity (to adapt precisely to hardware budgets without retraining), and Early-Exit (which reduces average latency depending on the difficulty of the sample) is well-founded and consistent.

Critical weaknesses

Omission of a major area of the literature (Layer Collapse/Folding).

In lines 98-107 and in Section B.1, the authors contrast methods that involve the direct removal of blocks (Entrodrop, Cosine) with methods that replace them with linear layers (HeDEPT, NOSE). This presentation omits a line of work specifically dedicated to depth reduction without added parameters at inference. It includes methods that linearize non-linearities so that consecutive layers can be merged (Layer Folding [1], EGP [2]), and methods that remove entire blocks or collapse layers (LaCo [3], LaCoOT [4]). The mechanism described in lines 98-107 bears a strong resemblance to these approaches, which should be discussed and compared experimentally (particularly in comparison with LaCoOT).

Erroneous gradient analysis and inconsistencies in Equation 6 (budget loss).

The authors state that the "budget-pressure term" exerts a greater force on the gates that are farther from the mean. However, let αˉ\bar{\alpha} be the mean of the gates over the LL blocks, and TT the target TkeepT_{keep}. Eq. 6 is Lbudget=(max⁡(0,αˉ−Tkeep))2L_{budget} = (\max(0, \bar{\alpha} - T_{keep}))^2. The partial derivative with respect to any gate αk\alpha_k is ∂Lbudget/∂αk=(2/L)max⁡(0,αˉ−Tkeep)\partial L_{budget} / \partial \alpha_k = (2/L) \max(0, \bar{\alpha} - T_{keep}). This gradient is identical for all αk\alpha_k: the term applies a uniform downward push and does not penalize gates according to their distance from the mean.

Limited Experimental Evaluation (Datasets, Architectures, Edge Devices).

Architectures: All experiments are based on a single 12-layer topology (ViT-B/16 and DeiT-B share the same architectural structure). No validation is provided for other model sizes or for architectures such as Swin or DiT.

Datasets: The experiments focus exclusively on small- or medium-sized datasets (CIFAR-100, Food-101, Stanford Cars, Flowers-102), with no evaluation on ImageNet.

Edge Devices Deployment: Section 3.4 (line 420) highlights the suitability of the method for edge devices when compared to EnViT, but Section A.1 (line 621) indicates that all measurements were performed on workstation GPUs, with no actual testing on embedded hardware.

Large number of hyperparameters and lack of sensitivity analysis.

The pipeline introduces a considerable number of hyperparameters: the bound on αraw,l\alpha_{\text{raw},l} in [−6,6][-6, 6], the size of the calibration set (B=10B=10 batches), the normalization bounds [0.4,0.95][0.4, 0.95], the coefficients λ1\lambda_1, λ2\lambda_2 (λbim\lambda_{\text{bim}}) and λbudget\lambda_{\text{budget}}, the target budget TkeepT_{\text{keep}}, the minimum number of layers Lmin⁡L_{\min}, the inspection frequency KK, the cooldown period CC, as well as the decision thresholds smin⁡s_{\min}, ρ\rho, gmin⁡g_{\min}, and the width factor ww. No ablation study measures the method’s sensitivity to these parameters. Furthermore, Table 4 (Section A.2) shows that key hyperparameters vary significantly from one dataset to another (for example, Cooldown ranges from 800 to 300 to 100, λbudget\lambda_{\text{budget}} takes the values 0.5, 0.4, or 0.25, and the Keep Ratio ranges from 0.60 to 0.70), raising questions about the method’s robustness without extensive manual tuning.

Lack of Training Budget Evaluation.

The two-width training strategy (Section 2.4.2, Equation 9) requires two forward passes and two loss calculations per mini-batch throughout Phases 1 and 2. This additional training cost is never measured (wall-clock time or GPU training memory) or compared to baseline methods.

Lack of clarity and omissions in the tables and figures.

Table 1: The exact nature of the models being compared (number of blocks retained, MACs, FLOPs) is not specified.

Table 2: Standard deviations are not reported, even though an average over 3 runs (“3 runs avg.”) is mentioned. No computational cost metrics (FLOPs/MACs) are included in the table, even though the compared methods prune different numbers of layers (5, 6, 8, or 10 layers).

Line 431 and Section 3.5: The text refers to lowering a confidence threshold to 0.7 without ever having introduced or formalized this parameter in Section 2, and this value contradicts section A.2 (line 643), which specifies an inference threshold of 0.5.

Table 3 and Figure 3: In Table 3, neither accuracy nor speedup is held constant (fixed) across methods, making a direct comparison difficult (e.g. on CIFAR-100, EnViT outperforms BPAW-ENVIT in both accuracy and speedup). In Figure 3, the dataset used is not specified, and it is unclear whether the speedup corresponds to an actual hardware measurement or a theoretical MAC ratio.

References
[1] Dror, et al. « Layer Folding: Neural Network Depth Reduction using Activation Linearization » (BMVC 2022).
[2] Liao, et al. « Can Unstructured Pruning Reduce the Depth in Deep Neural Networks? » (ICCVW 2023).
[3] Yang, et al. « LaCo: Large Language Model Pruning via Layer Collapse » (EMNLP 2024).
[4] Quétu, et al. « LaCoOT: Layer Collapse through Optimal Transport » (ICCV 2025).

Questions for authors
  1. Positioning and Comparisons with Layer Collapse Methods. Can you discuss the methodological connections with Layer Folding [1], EGP [2], LaCo [3], and LaCoOT [4], and provide a direct experimental comparison with at least LaCoOT [4], which seems to be the closest to the proposed method?

  2. Architectural generalization. How does the method perform on other architectural configurations beyond ViT-B/DeiT-B (for example, on Swin or DiT)?

  3. Sensitivity to Hyperparameters and Training Cost. Can you provide a study on the sensitivity to the main hyperparameters (CC, ρ\rho, TkeepT_{\text{keep}}, λbudget\lambda_{\text{budget}}, calibration set of BB training batches) using a unified set of hyperparameters across the datasets, as well as a comparison of the training time (wall-clock time) resulting from the double forward pass?

  4. Clarification of Equation 6 and corrections to the tables. Can the authors clarify the theoretical explanation of the gradient of Lbudget\mathcal{L}_{\text{budget}} in lines 232-234, add the MACs in Table 1, complete Table 2 (standard deviations and linear probing), and specify the dataset and the protocol for measuring the speedup in Figure 3 (as well as on a real edge target)?

Although the combination of block pruning using learnable gates, slimable FFNs, and Early-Exit is conceptually promising, the current version of the paper has too many methodological and empirical shortcomings (omission of work on Layer Collapse, lack of sensitivity analysis despite numerous hyperparameters tuned per dataset and evaluation limited solely to the ViT-B/16 topology on small datasets) to recommend acceptance as is. That said, I would be willing to reconsider my position if the authors can provide convincing evidence that the concerns above are resolved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.