acceptodds
Under review as a conference paper at ICLR 2027

Towards Compact and Robust DNNs via Compression-aware Structural Stability Optimization

Abstract

Sharpness-Aware Minimization (SAM) has recently emerged as an effective technique for improving DNN robustness to input variations. However, its interplay with the compactness requirements of on-device DNN deployments remains less explored. Simply pruning a SAM-trained model can undermine robustness, since flatness in the continuous parameter space does not necessarily translate to robustness under the discrete structural changes induced by pruning. Conversely, applying SAM after pruning may be fundamentally constrained by architectural limitations imposed by an early, robustness-agnostic pruning pattern. Motivated by this mismatch between robustness-aware training and compression, we propose Compression-aware Structural Stability Optimization (C-SSO). Rather than applying SAM before or after pruning, C-SSO optimizes pruning masks under stochastic structural perturbations, favoring subnetworks whose predictions remain stable under nearby connectivity changes while preserving robustness to input variations. Extensive experiments on CelebA-HQ, Flowers-102, CIFAR-10-C, and Tiny-ImageNet, spanning ResNet-18, GoogLeNet, MobileNet-V2, and DeiT-Ti, show that C-SSO consistently achieves higher certified robustness than competing baselines, with improvements of up to 42%, while maintaining task accuracy comparable to the corresponding unpruned models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
compactness, robustness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pXFBOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses the conflict between pruning and robustness to natural input variations, starting from the observation that SAM-trained flatness in weight space does not survive the discrete structural changes induced by pruning. It proposes C-SSO, which concentrates its modifications in the mask-search stage of sparse robust training. It applies a learnable soft mask with percentile-scaled initialization, perturbed stochastically during training, and optimized under a proposed composite objective. The objective combines a stability loss that penalizes prediction variance across stochastic mask neighborhoods, a margin-ratio loss adapted from CC-Cert that penalizes prediction discrepancy under semantic transformations relative to the classification margin, and a KL consistency loss aligning the soft mask with its binarized form, together with an L1 sparsity term. Training follows the standard three-stage procedure of dense pretraining, mask search, and binarization with fine-tuning. Experiments on several small-scale image classification benchmarks with four architectures report that C-SSO consistently achieves higher probabilistically certified accuracy than magnitude-, adversarial-, SAM-based, and structured pruning baselines across 30–70% pruning ratios, at comparable task accuracy. The authors further report that pruned C-SSO can surpass unpruned models at a 50% pruning ratio.

Critical strengths

The proposed research question has clear real-world applications.

The range of experiments conducted is wide.

Experiments conducted on their settings shows certain promising results.

Critical weaknesses

Experimental Setting

I have very serious concerns on experimental setting.

Pruning rate. The paper evaluate on different sparsity settings, which is acknowledgeable. However, the selection (30%, 50%, 70%) is way too conservative and far from "Compact" and "Sparse". Early works (cited in your manuscript) were mostly evaluated on over 90% of pruning rate, with some reaching 99.5%. How does your proposed method work on really sparse settings? Given the edge-compression motivation, this is the regime that matters most for the compactness claim, and the only justification offered (the non-monotonicity/capacity trade-off) predicts but does not excuse the omission.

Scale of experiment. ResNet-18 and all other architectures & datasets in experiment are too small in terms of scale. At least one experiment at comparable scale (e.g., ImageNet or its subset with ResNet-50/DeiT-Small) is needed.

Claim on robustness. Each dataset is trained and certified under exactly one pre-declared transformation direction. The reported PCA is therefore average-case consistency along a single one-dimensional transformation segment, not robustness over semantic variations in any broader sense. Given the facial-authentication motivation, "robustness" should either be demonstrated under worst-case attribute selection, or the claims should be narrowed to consistency under the declared transformation. Targeting this, [1] introduced on-manifold adversarial example, which is precisely the worst-case semantic attacks this paper does not evaluate. The natural adversarial examples are also covered in ImageNet-A [2], which should be included in evaluation.

Since the article suggests that ℓp\ell_p trained model performs badly under semantic perturbation. How about conversely? Does your method survive under adversarial perturbations, or some simple natural perturbations (e.g.: Gaussian / Poisson / Impulse noises) of certain intensity.

[1] Stutz et al. Disentangling Adversarial Robustness and Generalization, CVPR 2019.

[2] Hendrycks et al. Natural Adversarial Examples, CVPR 2021.

Theoretical Basis

The appendix proofs do not establish any property of the deployed model. A.2 restates the standard margin–argmax argument inherited from randomised smoothing and CC-Cert. A.3 is the textbook two-sample variance identity justifying a 2-sample estimator for the stability penalty. A.1 is a triangle-inequality decomposition whose role is to motivate the loss terms.

The decomposition bounds the discrepancy of the stochastic mask ensemble, whereas certification measures the single binarised model, whose variance terms vanish by construction. The looseness introduced by the triangle inequality and the ∞→2\infty\to 2 relaxation is never quantified, and no statement connects the training losses to the certified accuracy of the deployed model.

Consequently, the claim of "a mathematical sufficiency condition enabling a rigorous margin-aware ratio loss" overstates what is derived. The theory is expository rather than supporting.

Questions for authors

See Weakness. Most of the concern regards the experiments.

1 comment
Reviewer_pXFBOpenReview ↗

Updated Comment

I thank the author for the effort. I raised my rating from 1 to 2. The rebuttal adds substantive experiments on pruning rate (90–99%), natural perturbations, and quantifies the relaxation terms, which I acknowledge quite.

On PCA, I accept the authors' correction that it is a per-input probabilistic certification criterion, not average agreement along a one-dimensional segment. My original wording was imprecise. But this clarification confirms rather than removes my substantive concern. PCA certifies correctness under a pre-specified transformation distribution, and in the experiments T is in practice a single declared direction (smile, fog). The unseen-transformation results merely substitute two other specific attributes. The claim that PCA "applies to multidimensional transformations" addresses the definition, not the experimental setting I criticized. Rejecting Stutz et al. and ImageNet-A as "incompatible with PCA" is a legitimate scope limitation, but its logical consequence is that the metric itself cannot capture worst-case semantic robustness. I therefore ask that the paper explicitly narrow its robustness claim to "certified consistency under the declared transformation distribution".

Two further concerns remain only partially addressed.

Experiment: the ResNet-50/Tiny-ImageNet experiment compares only against SAM-LMP, and at 90% pruning C-SSO's ACC is slightly below it. I don't see clear advantage of the proposed method, and comparisons to other methods are missing.

Theory: quantifying the relaxations is welcome, but no statement links the training losses to the deployed model's certified accuracy. The authors substitute an empirical ablation, and the "mathematical sufficiency condition" phrasing remains an overstatement.

I have reviewed and maintain a weak rejection.

Reviewer_K6hpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper want a pruned model that is also robust to semantic change in the input, like a face starting to smile or a flower photo getting a sunrise glow, so they propose method named C-SAM, in which the key idea is to move the SAM-style perterbation from the weight to the pruning mask, where it add random noise to soft masks and trains them with a stability loss across random masks, a ratio loss derived from the CC-Cert certification condition, and a consistency loss between soft and hard masks. On CelebA-HQ, Flowers-102, and CIFAR-10-C with ResNet-18, GoogLeNet, and MobileNet-V2, the paper reports higher certified accuracy than the pruning and robust-training baselines at high pruning rate.

Critical strengths

I think one major strength from the idea side is that the method is intuitive, because on device models actually need to be both small in size and robust to natural variantions, and the design to move the perturbation from weight to pruning masks is a pretty clean way to make the compression pattern itself a robustness-aware pattern. The training objaective is also derived from the same certification situation that is used for the evaluation, as a result the method and the metric are consistent.

In addition the idea of the method, the evaluation cover both unstructured and structured compression which is great, and the inclusion of different model arch and dataset and the ablation and hyperparams sensitivety study, these make the evaluation result more comprehensive.

Critical weaknesses

One question that might concern: not sure on the baselines and the statistics's reliable enough for the current version althrough the model and data coverage is board.

At 50% unstructured sparsity, in Table 7, LMP drops MobileNet-V2 to 33.91% (comparing to 71.85% for Vanilla) and GoogLeNet to 44.12% (vs. 67.16%), which suggests under-tuned baselines, and the headline up to 42% gain comes from this GoogLeNet case. In addition, PCA is computed on only 100 test images in a single run, so its standard deviation is about 5 points, and many reported gaps (e.g., the 3 to 9 point drops in the ablation) fall within this noise.

So I would like to ask, wow can we be sure the gaps are not just noise? Could you report PCA on the full test set (or at least many more than 100 images), averaged over several seeds, with standard deviations or confidence intervals? Or, could you rerun the pruning baselines so that they start from the same pretrained weights and get the same 50-epoch fine-tuning as C-SAM? It would also help to add the stronger baselines to the cross-architecture comparison in Table 2, which currently only includes LMP.

Questions for authors

Please answer the question in the weakness section.

Reviewer_72PqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a robustness technique called Compression-aware Structural Stability Optimisation (C-SSO). This method uses a mask to preserve the model's connectivity structure during pruning. The main claim is that this pruning approach preserves robustness by preserving the model's connectivity. The authors show improvements over several baseline models across three datasets. They also provide a theoretical robustness certification. However, it remains probabilistic, selecting 100 test samples.

Critical strengths

The paper identifies key issues with pruning and robustness, such as unstructured pruning at the individual-weight level breaking the continuous parameter-space structure and linkage. This weakens a model's robustness. They therefore introduce a mask (at the structure level) based pruning. The key is that they design a loss function that introduces noise and binarises the mask, then optimise this loss (effectively the mask) to improve robustness for the pruned mask.

Critical weaknesses

Selective transformation was chosen. The main results are compared primarily against magnitude-based pruning methods, whereas their method trains and optimises the mask, which means dynamically reorganising the model's weight/connectivity space. Therefore, their comparison would be more suitable with dynamic pruning methods. Moreover, it mentions many other types of pruning baselines such as SynFlow

Questions for authors

What is xTx_T What is Aug. in Table 1? Why is Table 5 presented selectively? Also, why not include full GPU time, energy, etc., for training? Table 1 is very confusing. It mixes baseline types, which makes comparisons hard to follow.

1 comment
AuthorsOpenReview ↗

Gentle Reminder 3

Dear Reviewer,

Thank you for your time and consideration. We have submitted our rebuttal and would greatly appreciate any further feedback or questions you may have.

Best regards,

The Authors

Comments on the paper

1 comment
Area_Chair_DJPGOpenReview ↗

Reminder: Author–reviewer discussion ends today

Dear Reviewers,

As the author–reviewer discussion period ends today (November 18, AoE), please read the authors’ responses, any relevant manuscript updates, and the other reviewers’ comments, and engage with the authors before the deadline.

In particular, please clarify which of your concerns have been addressed and which remain unresolved, and raise any remaining questions promptly so the authors have an opportunity to respond. Even if your assessment is unchanged, a brief follow-up explaining your reasoning would be helpful.

Thank you for your time and thoughtful reviewing!

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.