acceptodds
Under review as a conference paper at ICLR 2027

When the Label Space Shrinks: Task-Adaptive Depth Compression via Recomputed Gradient-Residual Sensitivity

Abstract

Modern neural networks are often trained to support a broad range of functionalities, while a particular deployment may require only a small subset of them. This mismatch raises a natural compression opportunity: if the required functionality becomes narrower, can the model itself become smaller? We study label-space reduction as a concrete instance of this setting, where a reference classifier is trained on a broad source label space but deployment focuses only on a smaller target subset. Rather than shrinking the classifier head, we ask whether reduced target requirements expose removable computation within the network backbone. We propose Task-ReGRIP, a training-free method for task-adaptive depth compression. For each removable residual block, Task-ReGRIP combines the representation change introduced by the block with the target-task loss gradient to estimate the first-order perturbation caused by bypassing that block. It then iteratively removes the least sensitive block and recomputes the sensitivities of all remaining candidates after every structural change. No recovery training or parameter updates are performed during compression. We evaluate Task-ReGRIP on CIFAR-100 using growth-trained ResNet and MobileNetV2 models across multiple target label-space sizes. Across 12 compression paths, the accuracy on the evaluation half of the selected classes' official test images at the maximum deletion budget ranges from to percentage points relative to the corresponding uncompressed reference model, while the ResNet models remove approximately one third of their blocks. These results show substantial depth removal with limited target-task accuracy change in the evaluated settings, without recovery training.

Primary area:
learning theory
Keywords:
Task-adaptive depth compression, Training-free structured pruning, Label-space specialization, Residual networks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QtLBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Task ReGRIP, a training free method for compressing neural networks by removing residual blocks that have minimal impact on a specified target task. It estimates block importance using the inner product of activation gradients and effective residuals, then recomputes scores after each deletion. The method is evaluated on growth trained ResNet and MobileNetV2 models on CIFAR 100. Across 12 compression paths, the method removes approximately one third of the ResNet blocks while keeping accuracy changes within −1.2 to +1.0 percentage points of the original models.

Critical strengths

Simple and training free method with promising results. Task ReGRIP combines gradient information with block residuals and recomputes sensitivity after each deletion. Experiments show that ResNet models can remove approximately one third of their blocks with limited accuracy degradation, without recovery training. This makes the method practically appealing when retraining is costly.

Critical weaknesses

Insufficient evidence of methodological novelty. The gradient and residual sensitivity criterion builds on established first order pruning principles, while iterative rescoring resembles existing block pruning approaches. The paper does not sufficiently establish how much this combination advances beyond prior methods, and the comparison with the SLEB style baseline is limited to selected deletion budgets and a specific implementation.

Questions for authors

Can the authors provide evidence that the gradient and residual sensitivity criterion predicts actual loss changes more reliably than simpler alternatives, such as residual magnitude or activation gradient magnitude alone?

2 comments

Thank you for your review and comments.

W1 Novelty and baseline evidence

The "deepest model" discussion in our depth-growth analysis motivates identifying effective depth under a changed target task. Task-ReGRIP turns this into whole-block deletion for explicit class subsets, evaluated without recovery training. GRS combines the target-loss gradient and effective block residual through an absolute per-example inner product. We will clarify its relation to established first-order sensitivity methods; our contribution is supported by task formulation, matched comparisons, actual-deletion diagnostics and rescoring ablations.

SLEB-style also deletes whole blocks, enabling comparisons on the same checkpoint, selection data, candidates and budgets. Theseus-style replacement is a control that allows training with a separate budget. Additional controls include block-level channel-Taylor and cosine adaptations, residual norm, gradient norm and random deletion. Could you recommend other baselines suited to target-class-subset compression with matched reference models and training budgets? We would welcome such comparisons.

The new comparison uses three split seeds and covers every deletion budget. Growth ResNet110's mean accuracy favors GRS across all 18 budgets by about 1.3-3.3 percentage points (pp). Growth ResNet398's gaps across 66 budgets stay within about +/-0.6 pp; MobileNetV2's within +/-0.14 pp. On growth ResNet542, SLEB-style leads by about 0.13-2.13 pp, with a 0.4 pp gap at maximum budget. SLEB-style's measured cumulative selection time is about 14/51/68 times GRS's on the three ResNets and 2.1 times on MobileNetV2, excluding final evaluation. These paths support an accuracy/selection-cost tradeoff.

Rescoring versus static GRS improves standard ResNet110 by about 1.5 pp on average, positive in all three paired splits. Growth-model rankings are more stable, allowing cheaper static variants: growth ResNet542 scores 73.9% static and 73.7% dynamic; standard MobileNetV2 shows no rescoring gain. Thus rescoring addresses changing importance after deletion, with model-dependent benefits.

Q1 Prediction of actual loss changes

We tested four growth-trained and two directly trained checkpoints, three split seeds, and three stages (no deletion, half budget, maximum budget): 54 states from six fixed models, not 54 independently trained models. In each state, every remaining candidate block is actually bypassed once. Loss changes are measured on images excluded from block selection.

The target quantity is the mean absolute per-example cross-entropy (CE) change. Each cell below is the mean block-ranking Spearman correlation across a model's nine states:

ModelGRSCosineResidual normTaylor adaptationGradient norm
Growth ResNet1100.9760.9580.8080.9260.305
Growth ResNet3980.9920.9630.9250.9600.196
Growth ResNet5420.9940.9640.9240.961-0.010
Growth MobileNetV20.7120.6200.2440.3580.313
Standard ResNet1100.9670.7450.8170.495-0.285
Standard MobileNetV20.9460.5570.6020.236-0.141
All 54 states0.9310.8010.7200.6560.063

GRS is strictly highest in 48 states and tied highest in 2. This directly supports combining residuals and gradients over using either magnitude alone to rank perturbation sizes in the tested settings.

We distinguish this quantity from the signed mean CE change. Correlations with that different target are:

Model or aggregateGRSGradient norm
All 54 states0.3900.262
Growth ResNet3980.2120.671
Growth ResNet5420.0790.579

Thus the strong absolute-perturbation ranking result does not imply universal superiority for signed net loss changes or final accuracy. The advantage is specific to the quantity GRS is designed to estimate.

AuthorsOpenReview ↗

Follow-up on actual-deletion diagnostics and baseline comparisons

Thank you again for your constructive review. We would like to follow up specifically on your question about whether GRS predicts actual loss changes more reliably than residual or gradient magnitude alone.

Our response reports actual-deletion diagnostics across 54 evaluated states from six fixed checkpoints. For the mean absolute per-example cross-entropy change, the average Spearman correlation is 0.931 for GRS, compared with 0.720 for residual norm and 0.063 for gradient norm. We also distinguish this objective from signed mean loss change, for which the advantage is not consistent, and report comparisons across deletion budgets and rescoring ablations.

We would appreciate your assessment of whether these diagnostics address your question, and which aspects of the methodological contribution or baseline evidence remain insufficiently established. We would be happy to clarify any remaining points.

Thank you for your time and consideration.

Reviewer_BBLYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a block saliency-based depth-compression algorithm to reduce computational cost in backbones of neural network classifiers when only a subset of the trained tasks is required at inference time. The paper proposes Gradient-Residual Sensitivity (GRS) as a block saliency method that uses both activation and gradient information, and Task-ReGRIP, a training-free iterative block-pruning algorithm. The authors empirically demonstrate that their proposed algorithm can perform depth removal with minimal loss of accuracy on retained classes in standard convolutional neural networks without any additional training.

Critical strengths

S1. Well Motivated

The motivation for this work is strong. The notion of reduced computation of generalist models on task-specific inference requirements is an important problem and raises several technical questions. Block sparsity is one way to exploit naturally occurring sparsity in neural networks, and the authors have identified this mechanism as a means to address task-specific inference.

Critical weaknesses

W1. Poor Technical Contribution

Much of the contribution of this work boils down to introducing the GRS as a simple block saliency measure, with many of the remaining parts of the algorithm being standard components of other compression techniques [1]. The discussion of the GRS lacks technical depth, and the heuristic first-order approximation presented in this work, which is also available in [2], does not provide sufficient evidence for the importance of the proposed saliency measure.

W2. Poor Baselining

The paper does not provide a suitable slate of experiments for the stated problem: task-specific compression. The experiments are baselined against SLEB, a block-pruning method designed for large language models (LLMs), and the authors do not provide a fair comparison of their algorithm on LLMs, even though there appears to be no algorithmic reason why the proposed method cannot be applied to them.

Questions for authors

Clarifications

  1. Can you specify what “growth-trained” models are?
  2. How are BatchNorm layers handled in the proposed methodology in the models tested in this work? In particular, are the stored statistics updated, and if so, how?
  3. How would these techniques extend to standard models like ResNet50, where only around 17 layers can be replaced by identity connections? Additionally, how would these techniques be useful in most standard LLMs, where every layer can be replaced with an identity layer?

Points to help strengthen the paper:

I gently disagree with the authors that the primary area of this work is "Learning Theory"; perhaps changing the area to "Deep Learning" would be more suitable. Additional review of related literature may also be beneficial. Works such as [3] have also addressed the removal of task-specific information to enable faster inference through the lens of Unlearning in classifier models.

Nitpicks not affecting the overall assessment:

  1. In line 302, supposed to say Bert-of-Theseus with a citation?

Score Justification

Given the short review period for this ICLR, I have kept my confidence lower than my level of familiarity with the field, and it is possible that I may not have understood a few nuances of this work. My review is constrained by the new review guidelines of this ICLR, and I would be pleased to expand on the points raised in the review. I am eager to engage with the authors in the rebuttal period to improve their manuscript and revise my score.

[1] Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks, Hoefler et al., 2021.

[2] Pruning Convolutional Neural Networks For Resource Efficient Inference, Molcanov et al., ICLR 2017.

[3] DisCEdit: Model Editing by Identifying Discriminative Components, Murti and Bhattacharyya, NeurIPS 2024.

1 comment

Thank you for your review and comments.

W1 Technical contribution

Our theoretical paper, "A Qualitative Test-Risk Mechanism for Scaling Behavior in Normalized Residual Networks" , discusses the "deepest model" through local depth saturation and task gradients. This motivates reassessing whole blocks for retained class subsets; deletion quality is established experimentally.

GRS averages absolute per-example inner products of the whole-block effective residual and target-loss output gradient. We will clarify its aggregation and relation to Molchanov et al. Across six checkpoints, three split seeds and three deletion stages (54 states), mean Spearman correlation with actual mean absolute per-example cross-entropy (CE) changes is 0.931, versus cosine 0.801, residual norm 0.720, channel-Taylor adaptation 0.656 and gradient norm 0.063. GRS is strictly highest in 48 states and tied highest in 2. This supports ranking, not exact loss or final-accuracy guarantees.

Rescoring improves standard ResNet110 by 1.8/0.6/2.0 percentage points (pp; mean 1.5). Growth rankings are stable (minimum correlation ~0.983, seed 2027), allowing static variants: growth ResNet542 is 73.9% static versus 73.7% dynamic; standard MobileNetV2 shows no gain. Target-only calibration is not consistently better (standard ResNet110: target/all/other classes give 64.9/64.9/66.3%).

W2 Baseline choice and language models

Label subsets give explicit task and accuracy/CE/cost definitions. LLM abilities overlap; perplexity alone does not define retained task subsets. We therefore focus on available growth-trained CNN classifiers.

SLEB-style's whole-block trial deletion also applies to compatible CNNs. Theseus-style replacement is a separate control that allows training. We added channel-Taylor/cosine adaptations, residual/gradient norms and random deletion. Methods share checkpoints, 500 selection images, candidates and budgets. Maximum-budget accuracy (%) over three split seeds is mean +/- sample SD:

Model / deleted blocksGRSSLEB-style
Growth ResNet110 / 1869.2 +/- 0.567.9 +/- 0.6
Growth ResNet398 / 6674.5 +/- 1.474.4 +/- 2.2
Growth ResNet542 / 9073.7 +/- 0.874.1 +/- 3.5
Growth MobileNetV2 / 562.7 +/- 0.162.7 +/- 0.3
Standard ResNet110 / 1864.9 +/- 0.867.0 +/- 0.7
Standard MobileNetV2 / 531.7 +/- 0.434.9 +/- 1.2

Growth ResNet110 favors GRS in all paired splits; other growth-model mean gaps are <=0.4 pp. SLEB-style has lower CE on growth ResNets and higher accuracy on standard models. Its measured selection time is about 14/51/68 times GRS's on ResNet110/398/542 and 2.1 times on MobileNetV2, excluding final evaluation. GRS uses one forward/backward pass per step; trial deletion uses a forward pass per candidate. Timings are implementation-dependent.

Single-run language-model tests use nonofficial Qwen3-config growth checkpoints (14->21->28 layers), matched validation selection, protected endpoints and independent WikiText-103 test sequences. Pairs are GRS/SLEB-style test perplexity:

ConfigurationOriginalDelete 3Delete 6Delete 9Selection seconds
0.6B34.536.0/35.839.5/37.152.0/41.8207/1762
1.7B23.123.8/23.724.6/24.427.4/26.6175/1519

GRS uses about 1/8.5 and 1/8.7 of the selection time; larger deletions favor SLEB-style in perplexity. These are supplemental architecture results, not standard pretrained-LLM task-subset benchmarks. Could you recommend other baselines suited to target-class-subset compression with matched checkpoints and training budgets? We would welcome comparisons.

Q1 Growth-trained models

We train a shallow model, insert blocks and continue training. LiGO (Linear Growth Operator; Wang et al., 2023) learns small-to-large parameter mappings; our CNNs use staged insertion with zero-output initialization, preserving the function at insertion. CIFAR ResNet grows from 8 to 18 BasicBlocks/stage (50->110 layers), then adds 4/stage per expansion through ResNet398 to ResNet542. The CIFAR-50-layer model differs from ImageNet's Bottleneck ResNet50. MobileNetV2 similarly grows identity-shortcut blocks. Stages use activation-gradient checks and epoch bounds; training/checkpoint selection use the 45k/5k training split, excluding official test images.

Q2 BatchNorm

Weights and BN running statistics stay fixed in eval mode during selection, deletion and evaluation; no BN recalibration is used.

Q3 Standard ResNet50 and LLMs

ResNet50 has 12 blocks with identity bypasses among 16; endpoint protection leaves 10 candidates. On ImageNet-1K, deleting 1 yields 77.2/76.5% accuracy (GRS/SLEB-style); deleting 5 reduces counted MACs by 26.7% but yields 17.9/41.6%. With many bypassable LLM layers, GRS can reduce selection cost, as the W2 results illustrate.

Reviewer_efwUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of label-space reduction, i.e., how to compress a pretrained model when it is deployed for a task involving only a subset of the original classes. The proposed Task-ReGRIP is a training-free method that estimates the first-order perturbation caused by bypassing a removable residual block, using both the resulting change in representation and the sensitivity of the target-task loss. The algorithm iteratively removes the least sensitive block and, after each structural change, recomputes the sensitivity of all remaining candidate blocks. Experiments with ResNet and MobileNetV2 on the CIFAR-100 dataset show that the proposed method improves performance over existing approaches.

Critical strengths
  • The paper proposes a mathematical formulation of gradient-residual sensitivity (GRS), which leverages the activation gradients of residual blocks to estimate the impact of removing each block. By incorporating GRS into an iterative routine of block deletion and rescoring, the proposed method achieves improved performance over existing approaches.
Critical weaknesses
  • Although the topic of label-space shrinkage is interesting, the current implementation appears to be applicable only to CNNs with residual structures, which limits its generality. Furthermore, the proposed method is compared only with SLEB, which was originally designed for transformer architectures. Adapting this idea to CNNs and using it as the sole baseline does not seem to constitute a fair comparison.

  • The experiments are conducted only on CIFAR-100, which is insufficient to demonstrate that the proposed method is generally applicable to diverse problems. Evaluations on larger and more diverse datasets (e.g., ImageNet) would strengthen the claims. In addition, Figure 4 shows that while the accuracy is high when only 10 classes are retained, it drops considerably as the number of classes increases. This trend raises concerns about the effectiveness of the proposed method in more realistic settings where a larger subset of classes must be preserved.

Questions for authors
  • Applicability beyond CNNs: The current formulation of GRS appears to rely on residual blocks in CNNs. Can Task-ReGRIP be extended to transformer architectures (e.g., ViT or DeiT), where SLEB was originally proposed? If so, how would its performance compare with SLEB in that setting?
  • Choice of baselines and datasets: Since SLEB was designed for transformers, could the authors include comparisons with pruning methods specifically designed for CNNs? In addition, could they provide results on larger and more diverse datasets, such as ImageNet, to demonstrate the generality of the proposed method?
  • Accuracy degradation with more classes: As shown in Figure 4, accuracy drops considerably as the number of retained classes increases. Could the authors explain the cause of this degradation and discuss whether Task-ReGRIP still offers a meaningful advantage over baselines when a larger subset of classes must be preserved?
2 comments

Thank you for your review and comments.

W1 and Q1 Architecture and task definition

Label-space shrinkage specifies which classes to retain, with target accuracy, cross-entropy (CE), and backbone cost as explicit metrics. General LLM abilities overlap, making task subsets and the degree of task compression harder to define uniformly. We therefore focus on existing growth-trained CNN classifiers, motivated by task-dependent effective depth in our depth-growth analysis.

GRS uses effective block residuals and target-loss activation gradients, without requiring convolutions. We added official pretrained ResNet50 and ViT-B/16 on ImageNet-1K, retaining 100 or 1000 classes. Each class's 50 validation images are split into disjoint 25-image selection and evaluation halves. Selection uses 5 images/class for 100 classes, or 1/class for 1000; batch size is 32. The table gives mean accuracy (%) over three split seeds; pairs are GRS/SLEB-style.

ModelOriginalDelete 1Delete 2Max blocksAt max
ResNet5080.877.2/76.570.9/72.0517.9/41.6
ViT-B/1681.077.2/77.366.7/72.1437.6/52.6

This establishes cross-architecture execution. At one deletion, accuracy is comparable or better; at larger budgets, SLEB-style performs better. ResNet50's residual-norm baseline reaches 32.5% at maximum budget, above GRS's 17.9%. The 100-class setting shows the same direction. Maximum budgets reduce counted MACs by 26.7%/33.1%, respectively. These official checkpoints are transfer controls; our main evidence uses growth-trained CNNs. A corresponding growth-trained ViT is not yet available.

W1, W2 and Q2 Baselines and datasets

Our depth-growth motivation leads to whole-block bypassing and comparisons on the same reference checkpoint. SLEB-style tests candidate deletions by calibration loss; this rule is applicable to CNNs with compatible bypasses. Theseus-style replacement is a control that allows training, with its budget reported separately.

We added block-level adaptations of channel Taylor and input-output cosine scores, residual norm, gradient norm, and random deletion. The new comparison corrects the original calibration mismatch: methods share 500 selection images, candidate blocks, deletion budgets and three split seeds. Maximum-budget accuracy (%) is:

Model / deleted blocksGRSSLEB-styleCosineResidual normTaylor
Growth ResNet110 / 1869.267.969.268.567.9
Growth ResNet398 / 6674.574.473.774.272.7
Growth ResNet542 / 9073.774.173.372.972.9
Growth MobileNetV2 / 562.762.762.762.362.5
Standard ResNet110 / 1864.967.060.461.759.9
Standard MobileNetV2 / 531.734.928.428.516.7

Growth ResNet110 favors GRS in all three paired splits; the other growth-model mean gaps are <=0.4 percentage points (pp). SLEB-style is better on directly trained checkpoints. Its cumulative selection time is about 14, 51 and 68 times GRS's on the three ResNets, and 2.1 times on MobileNetV2; these are measured selection times, excluding final evaluation. Taylor/cosine are adaptations, not full reproductions of their original pipelines. Could you recommend other baselines suited to target-class-subset compression under matched reference models and training budgets? We would welcome such comparisons. The ImageNet results above also address the dataset concern.

W2 and Q3 Retaining more classes

The full 100-way head is retained; class count changes the selection/evaluation subset. Original accuracy already varies: growth ResNet398 scores 75.6/70.9/71.1/72.2% on 10/30/50/100 classes. Thus absolute subset differences cannot all be attributed to compression.

Using split seed 2027, with 50 selection and 50 evaluation images/class, maximum-budget accuracy changes from each subset's uncompressed reference are (pp; columns list 30/50/100 classes):

ModelGRSResidual normRandom
ResNet110+0.5/-0.9/-1.4-0.4/-0.7/-1.3-64.9/-65.4/-66.2
ResNet398-0.3/-0.9/-0.8-0.8/-1.6/-1.4-64.3/-65.0/-65.4
ResNet542+0.1/-0.5/-0.4-0.4/-1.0/-1.1-60.9/-62.6/-63.8

GRS changes range from -1.4 to +0.5 pp, beating residual norm in 7/9 settings and differing by <=0.2 pp otherwise. ResNet110's SLEB-style changes are -1.7/-0.7/-1.7 pp. Compression remains usable with larger class sets, without a consistent worsening trend across the tested models. This does not establish a separate benefit from target-only calibration.

AuthorsOpenReview ↗

Follow-up on additional experiments and clarifications

Thank you again for your thoughtful comments. We would like to briefly follow up on our response, in which we added ImageNet experiments with ResNet50 and ViT-B/16, expanded the baseline comparisons under matched settings, and examined accuracy changes relative to each retained-class subset’s uncompressed reference.

These results help clarify both the applicability and limitations of our approach: performance is less favorable at larger deletion budgets on the standard ImageNet checkpoints, while the retained-class analysis distinguishes compression-induced changes from differences in the original model’s accuracy across subsets.

We would appreciate your perspective on whether these additions address your concerns about architectural scope, comparison fairness, and retaining more classes. If any concerns remain, your guidance on the most informative further comparison or clarification would be very helpful.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.