- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Task ReGRIP, a training free method for compressing neural networks by removing residual blocks that have minimal impact on a specified target task. It estimates block importance using the inner product of activation gradients and effective residuals, then recomputes scores after each deletion. The method is evaluated on growth trained ResNet and MobileNetV2 models on CIFAR 100. Across 12 compression paths, the method removes approximately one third of the ResNet blocks while keeping accuracy changes within −1.2 to +1.0 percentage points of the original models.
Critical strengths
Simple and training free method with promising results. Task ReGRIP combines gradient information with block residuals and recomputes sensitivity after each deletion. Experiments show that ResNet models can remove approximately one third of their blocks with limited accuracy degradation, without recovery training. This makes the method practically appealing when retraining is costly.
Critical weaknesses
Insufficient evidence of methodological novelty. The gradient and residual sensitivity criterion builds on established first order pruning principles, while iterative rescoring resembles existing block pruning approaches. The paper does not sufficiently establish how much this combination advances beyond prior methods, and the comparison with the SLEB style baseline is limited to selected deletion budgets and a specific implementation.
Questions for authors
Can the authors provide evidence that the gradient and residual sensitivity criterion predicts actual loss changes more reliably than simpler alternatives, such as residual magnitude or activation gradient magnitude alone?
2 comments
Follow-up on actual-deletion diagnostics and baseline comparisons
Thank you again for your constructive review. We would like to follow up specifically on your question about whether GRS predicts actual loss changes more reliably than residual or gradient magnitude alone.
Our response reports actual-deletion diagnostics across 54 evaluated states from six fixed checkpoints. For the mean absolute per-example cross-entropy change, the average Spearman correlation is 0.931 for GRS, compared with 0.720 for residual norm and 0.063 for gradient norm. We also distinguish this objective from signed mean loss change, for which the advantage is not consistent, and report comparisons across deletion budgets and rescoring ablations.
We would appreciate your assessment of whether these diagnostics address your question, and which aspects of the methodological contribution or baseline evidence remain insufficiently established. We would be happy to clarify any remaining points.
Thank you for your time and consideration.
Thank you for your review and comments.
W1 Novelty and baseline evidence
The "deepest model" discussion in our depth-growth analysis motivates identifying effective depth under a changed target task. Task-ReGRIP turns this into whole-block deletion for explicit class subsets, evaluated without recovery training. GRS combines the target-loss gradient and effective block residual through an absolute per-example inner product. We will clarify its relation to established first-order sensitivity methods; our contribution is supported by task formulation, matched comparisons, actual-deletion diagnostics and rescoring ablations.
SLEB-style also deletes whole blocks, enabling comparisons on the same checkpoint, selection data, candidates and budgets. Theseus-style replacement is a control that allows training with a separate budget. Additional controls include block-level channel-Taylor and cosine adaptations, residual norm, gradient norm and random deletion. Could you recommend other baselines suited to target-class-subset compression with matched reference models and training budgets? We would welcome such comparisons.
The new comparison uses three split seeds and covers every deletion budget. Growth ResNet110's mean accuracy favors GRS across all 18 budgets by about 1.3-3.3 percentage points (pp). Growth ResNet398's gaps across 66 budgets stay within about +/-0.6 pp; MobileNetV2's within +/-0.14 pp. On growth ResNet542, SLEB-style leads by about 0.13-2.13 pp, with a 0.4 pp gap at maximum budget. SLEB-style's measured cumulative selection time is about 14/51/68 times GRS's on the three ResNets and 2.1 times on MobileNetV2, excluding final evaluation. These paths support an accuracy/selection-cost tradeoff.
Rescoring versus static GRS improves standard ResNet110 by about 1.5 pp on average, positive in all three paired splits. Growth-model rankings are more stable, allowing cheaper static variants: growth ResNet542 scores 73.9% static and 73.7% dynamic; standard MobileNetV2 shows no rescoring gain. Thus rescoring addresses changing importance after deletion, with model-dependent benefits.
Q1 Prediction of actual loss changes
We tested four growth-trained and two directly trained checkpoints, three split seeds, and three stages (no deletion, half budget, maximum budget): 54 states from six fixed models, not 54 independently trained models. In each state, every remaining candidate block is actually bypassed once. Loss changes are measured on images excluded from block selection.
The target quantity is the mean absolute per-example cross-entropy (CE) change. Each cell below is the mean block-ranking Spearman correlation across a model's nine states:
GRS is strictly highest in 48 states and tied highest in 2. This directly supports combining residuals and gradients over using either magnitude alone to rank perturbation sizes in the tested settings.
We distinguish this quantity from the signed mean CE change. Correlations with that different target are:
Thus the strong absolute-perturbation ranking result does not imply universal superiority for signed net loss changes or final accuracy. The advantage is specific to the quantity GRS is designed to estimate.