- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a layer-wise profile, which weights whether filters in a layer are removed based on magnitude or on class-information criteria. This profile is based on an activation-weighted functional of the Bayes mutual information that decomposes into a magnitude component and a class-information component. They find that earlier layers should be pruned mainly based on magnitude and later layers should be pruned based on class information. Based on this weighting filters in each layer are removed within a per-layer budget, which determines how many parameters can be removed in each layer. The budget is determined by testing how much accuracy is lost when pruning each layer individually, weighed against the parameter savings.
Critical strengths
The paper quantifies the known intuition that earlier layers encode general feature information and later layers are class-discriminative. Together with the layer-wise pruning budget this gives an informed pruning method that does not rely on hyperparamter tuning.
Critical weaknesses
The usefulness of layer profiles and layer budgeting is not sufficiently shown, neither for understanding networks, nor for pruning them. The evaluation scope is very limited (e.g. single model checkpoints, few datasets, limited ResNet handling, very few re-training experiments, no ablations on the impact of layer profiles and layer budgeting individually), which weakens the empirical results, which in any case are not overly convincing.
While the theoretical part seems to be sound, the novelty is limited and as the authors mention, the underlying assumption that the class-conditionals are Gaussian is not necessarily given in ReLU models and not tested empirically.
These weaknesses are mostly acknowledged in the paper, but in my opinion a broader evaluation of the method showing a practical usefulness or new understanding of neural networks would be needed to accept the paper.
Questions for authors
- Could you clarify how many evaluations it takes to compute the layer-budget allocation? How does the wall-clock time of your method compare to other methods?
- How dependent is the theory on the Gaussian assumption?
- Is it possible to derive layer profiles using more realistic distributional assumptions that are based on empirical measurements?
Dear reviewer,
Thank you for the review and for recognizing that the method needs no hyperparameter tuning. Your central point, that the usefulness of the layer profiles was not shown either for understanding networks or for pruning, shaped the revision most. The paper now demonstrates both and positions itself accordingly. All overall changes to the paper can be found in the general response, along with all the new experiments R1-R9.
With regard to your addressed weaknesses:
Usefulness of the layer profiles for understanding networks: The revised paper presents the profile as an interpretable, per-layer measurement of how class evidence is encoded, and validates it against an estimate that does not use the Gaussian model (R5, new §4.5, App. F). Held-out probes trained on each layer's pooled responses measure class information directly. GlobalBMI's information component D agrees with them across 74 layers (Spearman rho = 0.85). When the same layer moves from CIFAR-10 to CIFAR-100, the change in D tracks the change in probe information (rho = 0.98 over 37 layer pairs, with depth held fixed), so D is not merely a proxy for depth. The probes also explain a finding that goes beyond "early general, late discriminative": alpha rises again in the final layer(s) of every network, although probe information there is maximal, because class evidence is carried by large, class-specific activations. This is consistent with neural collapse, which the revised paper discusses. Three independently trained networks per setting reproduce the profile to within 0.02 (R8). We also state the limits of this reading: within a single network D adds little beyond depth, and alpha does not predict a layer's prunability beyond depth (App. F).
Usefulness for pruning - profile and budget ablated separately: R1, R2, added to the paper in §4.4, on five partitions. The budget is where most of the gain comes from. Other criteria placed in BMIPrune's allocation gain 6–29 points on average at 70–85% removal (mean activation +17.6, GFI-AP +6.2, L1 and random +24–29), and with BMIPrune's widths even random channels come within one point of the full method up to 85% on average. The profile and criterion decide at the highest compression: at 90–95% random channels fall 6.3 points behind, LocalBMI has the best mean rank of seven criteria, and removing either signal costs 1.5–10 points at ≥ 70%. A constant alpha = 0.5 is a competitive default, which the paper now says. Across all 66 VGG settings, BMIPrune has the best mean rank of six methods and the smallest worst-case shortfall to the best method (7.8 vs. 17.6–62.0 points; new Table in §4.3).
Evaluation scope: The revised paper adds ImageNet-1k with two pretrained networks (R7), three independently trained checkpoints for each of two settings (R8), a native Taylor baseline (R9), training-set calibration with 10–75 images per class (R3), latency and matched FLOPs (R6), and the block->stage ResNet hierarchy on ImageNet ResNet-18. On retraining, the paper now frames BMIPrune as a fine-tuning-free method: after fine-tuning, methods with trainable masks recover to similar accuracy, and we report it that way.
Novelty: As you note, the intuition that early layers are general and late layers discriminative is known. The revised paper's contribution is to measure it per layer without training or tuning, to show that the measurement is faithful and reproducible (R5, R8), to reveal where it departs from the intuition (the final-layer rise), and to turn it directly into a one-shot pruning method that is the most consistent across networks.
Q1: Number of evaluations and wall-clock time: L*|Q|+1 BN-recalibrated forward passes over the calibration set (49 for VGG11, 79 for VGG16, 97 for VGG19), computed once for all targets: 87–289 s per network on a laptop GPU with 1.2–1.3 GB of GPU memory, versus 3–8 s for the one-shot baselines, and no gradients or training. With 10 images per class (R3) the cost falls further at unchanged accuracy. Added to App. A.
Q2: How dependent is the method on the Gaussian assumption? Weakly, in practice (R4, App. I). Replacing the Gaussian by a zero-inflated Gaussian or a 16-bin histogram leaves the LocalBMI ranking essentially unchanged (median ρ ≥ 0.998, worst layer ≥ 0.96). The Gaussian overstates information only in the sparsest layers (≥ 30% exact zeros), where the ranking is dominated by magnitude anyway; the revised limitations section quantifies this.
Q3: Profiles under more realistic distributions: Yes, and the revised appendix shows them (R4, App. I). Under the zero-inflated Gaussian and the histogram model, the first-layer alpha is unchanged (≈ 0.98); alpha shifts by at most ~0.1, towards magnitude, and only in the sparsest CIFAR-100 layers, while elsewhere the profiles stay close to the Gaussian one.