Brain Data as a Block-Importance Metric for Depth Pruning

Brain-model alignment quantifies how well a deep neural network's internal representations correlate with human brain activity. It remains unclear however whether this alignment actually reflects the functional importance of individual deep network components. In this work, we investigate whether changes in this correlation between brain data and the model's internal representations can be used to infer block importance. Using per-block brain-model alignment scores computed on the Natural Scenes Dataset, we derive two model-agnostic block-importance metrics that capture the first derivative and second derivative of the alignment score, denoted R1D and R2D, respectively. We evaluate R1D and R2D through structured depth pruning by ranking blocks according to these metrics, removing the lowest-scoring blocks, and comparing the resulting performance against conventional pruning baselines. Across multiple models and vision tasks, R2D matches or outperforms several other block-importance metrics for pruning, with the largest margins at moderate compression rates. Notably, after pruning half the blocks of the Swin Small model using our brain-inspired metrics, it retains 61.5% accuracy on ImageNet without any additional finetuning, against 26.9% for the strongest depth pruning baseline. Our work demonstrate that brain-model alignment provides a direct signal that allows for more efficient model design.