When the Label Space Shrinks: Task-Adaptive Depth Compression via Recomputed Gradient-Residual Sensitivity

Modern neural networks are often trained to support a broad range of functionalities, while a particular deployment may require only a small subset of them. This mismatch raises a natural compression opportunity: if the required functionality becomes narrower, can the model itself become smaller? We study label-space reduction as a concrete instance of this setting, where a reference classifier is trained on a broad source label space but deployment focuses only on a smaller target subset. Rather than shrinking the classifier head, we ask whether reduced target requirements expose removable computation within the network backbone. We propose Task-ReGRIP, a training-free method for task-adaptive depth compression. For each removable residual block, Task-ReGRIP combines the representation change introduced by the block with the target-task loss gradient to estimate the first-order perturbation caused by bypassing that block. It then iteratively removes the least sensitive block and recomputes the sensitivities of all remaining candidates after every structural change. No recovery training or parameter updates are performed during compression. We evaluate Task-ReGRIP on CIFAR-100 using growth-trained ResNet and MobileNetV2 models across multiple target label-space sizes. Across 12 compression paths, the accuracy on the evaluation half of the selected classes' official test images at the maximum deletion budget ranges from $-1.2$ to $+1.0$ percentage points relative to the corresponding uncompressed reference model, while the ResNet models remove approximately one third of their blocks. These results show substantial depth removal with limited target-task accuracy change in the evaluated settings, without recovery training.