BPAW-VIT: Block Pruning and Adaptive Width for Efficient Vision Transformer

In recent times, there has been a paradigm shift towards fine-tuned models rather than full-scale pre-training, owing to the availability of large corpora and pre-trained models on platforms like Hugging Face. The aim is to save significantly on computational costs during training. While Vision Transformers (ViTs) achieve remarkable performance, their deployment is hindered by high computational costs. Existing depth-Pruning methods such as block-dropping techniques like Entrodrop disrupt feature spaces, inducing severe loss in accuracy. Other depth-pruning methods, such as HeDEPT and NOSE, replace pruned layers with parameter-heavy linear substitutions. To achieve true parameter sparsity without architectural distortion, we propose a BPAW-ViT framework that unifies Alpha-Gated Progressive Layer Pruning (depth) with slimmable feed-forward networks (width Slicing). Guided by an entropy-alignment criterion and a dual-width training recipe, our framework dynamically reduces redundant blocks to pure identity mappings or slices intermediate capacities, requiring zero extra parameters. Finally, as a natural extension, we integrate this framework with Early-Exit architectures (EnViT). Because alpha-gated pruning preserves monotonic hidden states and intact 2D spatial resolution, our joint approach enables us to achieve a higher speedup (up to 2.45$\times$) with a significantly reduced memory footprint.