- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper addresses the conflict between pruning and robustness to natural input variations, starting from the observation that SAM-trained flatness in weight space does not survive the discrete structural changes induced by pruning. It proposes C-SSO, which concentrates its modifications in the mask-search stage of sparse robust training. It applies a learnable soft mask with percentile-scaled initialization, perturbed stochastically during training, and optimized under a proposed composite objective. The objective combines a stability loss that penalizes prediction variance across stochastic mask neighborhoods, a margin-ratio loss adapted from CC-Cert that penalizes prediction discrepancy under semantic transformations relative to the classification margin, and a KL consistency loss aligning the soft mask with its binarized form, together with an L1 sparsity term. Training follows the standard three-stage procedure of dense pretraining, mask search, and binarization with fine-tuning. Experiments on several small-scale image classification benchmarks with four architectures report that C-SSO consistently achieves higher probabilistically certified accuracy than magnitude-, adversarial-, SAM-based, and structured pruning baselines across 30–70% pruning ratios, at comparable task accuracy. The authors further report that pruned C-SSO can surpass unpruned models at a 50% pruning ratio.
Critical strengths
The proposed research question has clear real-world applications.
The range of experiments conducted is wide.
Experiments conducted on their settings shows certain promising results.
Critical weaknesses
Experimental Setting
I have very serious concerns on experimental setting.
Pruning rate. The paper evaluate on different sparsity settings, which is acknowledgeable. However, the selection (30%, 50%, 70%) is way too conservative and far from "Compact" and "Sparse". Early works (cited in your manuscript) were mostly evaluated on over 90% of pruning rate, with some reaching 99.5%. How does your proposed method work on really sparse settings? Given the edge-compression motivation, this is the regime that matters most for the compactness claim, and the only justification offered (the non-monotonicity/capacity trade-off) predicts but does not excuse the omission.
Scale of experiment. ResNet-18 and all other architectures & datasets in experiment are too small in terms of scale. At least one experiment at comparable scale (e.g., ImageNet or its subset with ResNet-50/DeiT-Small) is needed.
Claim on robustness. Each dataset is trained and certified under exactly one pre-declared transformation direction. The reported PCA is therefore average-case consistency along a single one-dimensional transformation segment, not robustness over semantic variations in any broader sense. Given the facial-authentication motivation, "robustness" should either be demonstrated under worst-case attribute selection, or the claims should be narrowed to consistency under the declared transformation. Targeting this, [1] introduced on-manifold adversarial example, which is precisely the worst-case semantic attacks this paper does not evaluate. The natural adversarial examples are also covered in ImageNet-A [2], which should be included in evaluation.
Since the article suggests that trained model performs badly under semantic perturbation. How about conversely? Does your method survive under adversarial perturbations, or some simple natural perturbations (e.g.: Gaussian / Poisson / Impulse noises) of certain intensity.
[1] Stutz et al. Disentangling Adversarial Robustness and Generalization, CVPR 2019.
[2] Hendrycks et al. Natural Adversarial Examples, CVPR 2021.
Theoretical Basis
The appendix proofs do not establish any property of the deployed model. A.2 restates the standard margin–argmax argument inherited from randomised smoothing and CC-Cert. A.3 is the textbook two-sample variance identity justifying a 2-sample estimator for the stability penalty. A.1 is a triangle-inequality decomposition whose role is to motivate the loss terms.
The decomposition bounds the discrepancy of the stochastic mask ensemble, whereas certification measures the single binarised model, whose variance terms vanish by construction. The looseness introduced by the triangle inequality and the relaxation is never quantified, and no statement connects the training losses to the certified accuracy of the deployed model.
Consequently, the claim of "a mathematical sufficiency condition enabling a rigorous margin-aware ratio loss" overstates what is derived. The theory is expository rather than supporting.
Questions for authors
See Weakness. Most of the concern regards the experiments.
Updated Comment
I thank the author for the effort. I raised my rating from 1 to 2. The rebuttal adds substantive experiments on pruning rate (90–99%), natural perturbations, and quantifies the relaxation terms, which I acknowledge quite.
On PCA, I accept the authors' correction that it is a per-input probabilistic certification criterion, not average agreement along a one-dimensional segment. My original wording was imprecise. But this clarification confirms rather than removes my substantive concern. PCA certifies correctness under a pre-specified transformation distribution, and in the experiments T is in practice a single declared direction (smile, fog). The unseen-transformation results merely substitute two other specific attributes. The claim that PCA "applies to multidimensional transformations" addresses the definition, not the experimental setting I criticized. Rejecting Stutz et al. and ImageNet-A as "incompatible with PCA" is a legitimate scope limitation, but its logical consequence is that the metric itself cannot capture worst-case semantic robustness. I therefore ask that the paper explicitly narrow its robustness claim to "certified consistency under the declared transformation distribution".
Two further concerns remain only partially addressed.
Experiment: the ResNet-50/Tiny-ImageNet experiment compares only against SAM-LMP, and at 90% pruning C-SSO's ACC is slightly below it. I don't see clear advantage of the proposed method, and comparisons to other methods are missing.
Theory: quantifying the relaxations is welcome, but no statement links the training losses to the deployed model's certified accuracy. The authors substitute an empirical ablation, and the "mathematical sufficiency condition" phrasing remains an overstatement.
I have reviewed and maintain a weak rejection.