acceptodds
Under review as a conference paper at ICLR 2027

Depth-Dependent Selection Geometry for Certified Structured Sparsification

Abstract

Structured pruning deletes whole channels, attention heads, or neurons, and a line of work makes such deletions happen during ordinary training by writing each weight as a product of D factors shared across a channel or head, so that weight decay drives whole groups to zero (deep gating). None of these methods says whether a particular deletion was justified: at a group whose factors are all zero, the training objective is stationary whatever the data say. Its curvature, however, depends sharply on depth. With two factors, the curvature at a deleted group, in that group's own coordinates, still sees the data in exactly the shape of the group lasso's optimality condition, so for almost every initialization a deletion that survives to a convergent gradient-descent limit carries a certificate that costs one gradient to check. With three or more factors, the data drop out of that curvature and a deleted group is locally stable in its own coordinates, justified or not. Because a network trained at one depth can be rewritten exactly at another, Certified Progressive Gating decides at depth two, records which decisions the certificate supports, and compresses deeper. On CIFAR-10 and CIFAR-100, it has the highest mean accuracy of eleven evaluated methods in five of six settings; on CIFAR-10, it matches a tuned dependency-aware pruner on two thirds of that pruner's training budget; on BERT-base, it has the highest mean accuracy of six evaluated structured-pruning methods, within noise. To our knowledge, it is the only method that records which of its deletions satisfy their optimality condition.

Primary area:
optimization
Keywords:
structured sparsification, deep gating, multiplicative decomposition, nonconvex sparse learning, optimization geometry

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_CghJOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how factorization depth changes the local optimization geometry of structured sparse learning with multiplicative gates. It distinguishes two-factor gating, where the curvature at an all-zero group reflects a group-lasso optimality condition, from deeper factorizations, where such a group can be locally stable even when that condition fails. Based on this analysis, Certified Progressive Gating first trains at depth two and records certificate labels for pruning decisions, then exactly reparameterizes the surviving weights at a greater factorization depth for further sparsification. Validation includes synthetic regression, CNNs on image-classification tasks, and BERT-base on SST-2 and QNLI.

Critical strengths

S1. Connection between parameterization and pruning behavior. The proposed distinction between two-factor and deeper gating offers a potentially useful explanation for why factorized sparse training can retain questionable zero groups. Connecting this behavior to an explicit groupwise optimality check is an interesting direction for understanding structured sparse learning. My assessment here concerns the conceptual motivation; I cannot independently verify the theoretical novelty or proofs.

Critical weaknesses

W1. Clarity of the practical algorithm. I found it difficult to connect the local geometric analysis to the complete pruning procedure. In particular, the phrase 'decide at D=2, compress deeper' initially suggests that the final support is selected in the first stage, although additional removals occur later. The relationship between the stage transition, near-zero commitment rule, regularization strength, and enforcement of a requested compression budget is spread across the main text and appendices. A concise worked example showing these steps, including the distinction between factorization depth and network depth, would make the method substantially easier to assess.

W2. Practical interpretation of the certificate. The paper carefully limits the certificate to a local, decision-time optimality condition, and the default BOUNDARY mode commits near-zero candidates even when they fail the test. I therefore remain uncertain what practical assurance the certificate ledger provides beyond documenting the decisions made by the depth schedule. Although the paper includes relevant controls and an example of using the certificate as a control signal, a clearer synthesis of what these results establish about its practical value would help me judge the significance of the contribution.

My provisional weak-rejection recommendation reflects these unresolved questions about exposition and practical interpretation, rather than an identified mathematical error. The central theory is outside my expertise, which I have flagged to the AC; an additional assessment by a reviewer with the relevant optimization background is needed.

Questions for authors
  1. Algorithm clarification: Could the authors walk through a concrete target-compression example, specifying when the D=2 stage ends, which groups are committed at that point, how further groups are removed at D>=3, and where the final MAC target is enforced? Please distinguish the abstract algorithm from the boundary and landing evaluation protocols.

  2. Meaning of certification: In the default BOUNDARY mode, what practical conclusion should a user draw from a group being recorded as certified or certificate-compatible, given that failed candidates are also committed and later training can change the risk gradient? I understand that the statement concerns the decision-time objective; I would appreciate a clearer explanation of what the ledger permits a practitioner to conclude.

  3. Evidence for the proposed mechanism: Among the existing controls, which comparison most directly isolates the benefit of beginning at D=2 from the effects of the per-cell penalty and other training refinements? A concise explanation connecting that result to the claimed selection-geometry advantage would materially help my assessment.

5 comments
AuthorsOpenReview ↗

Response to Reviewer CghJ

Thank you for the careful questions. We answer each weakness together with its question.

W1 / Q1 (clarity of the practical algorithm; a concrete run). We agree the procedure was spread across the text. §3.4 now states it in one place, with a Terms paragraph and a one-run example (App. E.2), and Algorithm 1 is restated as run. Two points you raised:

  • Factorization depth vs. network depth. D counts the multiplicative factors of each weight group, not layers; the architecture never changes.
  • "Decide at D=2, compress deeper" was imprecise, as you note. Under the default protocol every removal is made at the stage switch and the deeper stage only retrains; under the landing protocol the deeper stage removes further groups, recorded as uncertified. Fig. 1 now reads "decide at D=2, finish at D_max".

One run on CIFAR-10 ResNet-56, target 8× fewer MACs, default (boundary) protocol:

  1. Select at D=2. 16 SGD epochs with the group penalty (strength λ selected on validation, each channel weighted by its MAC cost), then 2 large-batch epochs. λ sets how many channels reach zero.
  2. Decide and record at the stage switch. The channels whose factors are numerically zero (about 640 of 1,008) are committed: removed permanently, pass or fail. For each, the certificate ‖∇_g L₀‖ ≤ λ_g is recorded; here 82% pass under the training gradient.
  3. Enforce the budget. About 220 of the smallest remaining channels are removed by magnitude, giving exactly 8× fewer MACs.
  4. Re-lift to D=3. Each surviving channel is rewritten with three factors in closed form; the network function is unchanged.
  5. Retrain at D=3 for 18 penalty-free epochs; removed channels stay at zero.

Steps 1, 2 and 4 are common to both protocols, which differ only in where the MAC target is enforced. The landing protocol skips step 3 and keeps the penalty in step 5, so the D=3 stage removes further channels and the target is met by magnitude at the end (ResNet-56 16×: 627 removals at the switch, 305 in the deep stage, 13 at the end).

W2 / Q2 (practical interpretation of the certificate).

  • Certified: when the removal was made, the data did not contest it under the selection stage's optimality condition. Certificate-compatible: the same within a small slack (δ = 0.05λ_g).
  • Uncertified: D=2 removals that fail the test, later-stage removals and budget-enforcing removals. It does not imply an incorrect removal.
  • The label is a decision-time statement: later training can change the gradient, but a committed group stays removed and its label is not re-evaluated. In the default mode it audits decisions; it does not change the network.

What the results establish about its practical value (Table 16, App. F.2, F.4):

  • It shows how much of the pruning the selection stage decided: 13%, 62% and 92% of removals at ResNet-18 8×, 16× and 32×, and 74% and 44% at ResNet-56 8× and 16×.
  • It can select λ. The certificate-compatible share reproduces the validation-selected λ on ResNet-18 without a validation split, and can be traded against accuracy along a frontier (Fig. 6).
  • On BERT, every removal is certified and lossless. Across the Table 2 CPG runs, all 310,173 removals pass before they are made, and removing them changes the outputs by < 6e-6.

Re-admitting failing channels gave no measurable accuracy benefit, so the label is recorded, not enforced. The accuracy comes from the schedule (Q3).

Q3 (which control isolates starting at D=2). The most direct one is fixed depth 3: the same per-group penalty, budget, protocol and λ search, without the D=2 start (new Table 3).

  • CPG vs. fixed depth 3, boundary protocol (CIFAR-10, 8×–16×): +0.011 to +0.027 (t=4.1–7.2). Fixed depth 3 is no better than fixed depth 2, so the gain is the change from 2 to 3, not depth.
  • CPG vs. 3→4, deepening without the D=2 start: +0.013 (t=5.6) and +0.015 (t=2.7) at 16× (App. F.6).
  • Same D=2 start, with vs. without the penalized deep stage (landing protocol): +0.096 and +0.047 at 16× on CIFAR-10, +0.099 on CIFAR-100 (t=6.0–12.0).

Penalty and compute are also controlled: Network Slimming with the same MAC-weighted penalty stays behind by 0.015 to 0.052 (Table 15), and on DepGraph's own budget CPG leads by +0.021 and +0.020 at 8× (Table 14).

Connection to the geometry. At D=2 a zero group that violates the optimality condition has a negative-curvature direction, so training can still undo a removal the data contest; at D≥3 its Hessian is positive definite, so whatever reaches zero stays. Starting at D=2 gives the deep stage a support the data could contest, consistent with the controls above.

On BERT, we corrected an evaluation error in the submitted Table 2 (it had evaluated unpruned model copies) and reran all six methods under one protocol; with the saliency-targeted penalty of the revision (§3.4), CPG has the highest mean accuracy in all four settings.

Reviewer_CghJOpenReview ↗

Response to rebuttal

Thank you for the detailed response and revisions. The worked example and distinction between the boundary and landing protocols substantially address my concern about the practical procedure. The additional matched controls also strengthen the empirical evidence for the depth schedule.

My remaining concern is the practical interpretation of certification. I now understand that it records a decision-time optimality condition, while failing candidates may still be removed. Could you clarify which gradient evaluation defines the reported certificate? In Table 16, for example, ResNet-56 at 8× has 634/852 removals committed at the switch, but 236/852 certified under the stored check, alongside an 82% pass rate under the training gradient. Distinguishing these quantities explicitly would help practitioners interpret the record.

I appreciate the disclosure and correction of the BERT evaluation error. The same-ranking hard-removal control appears to match CPG’s accuracy, so I interpret these results as supporting the allocation strategy’s accuracy and certification’s auditing role.

These revisions improve my assessment of the presentation and empirical support. As previously noted to the AC, I remain unable to independently assess the central theory’s correctness and novelty.

AuthorsOpenReview ↗

Response to follow-up: which gradient defines the certificate

Thank you for reading the revision so carefully, and for the clear summary of what it resolved.

Which gradient defines the certificate. The three numbers you quote for ResNet-56 at 8× measure different things:

quantityvaluewhat it measures
committed at the switch634 of 852 removals (74%)groups at zero when the D=2 stage ends; no gradient is involved
certified, stored check236 of the 634 (37%; 28% of all removals)‖∇_g L₀‖ ≤ λ_g with the gradient on 10k training images, without augmentation; this is the label written into the decision record at the switch
pass, training-objective check82% of the 634 (about 520)the same test with the gradient of the objective stage 1 actually minimizes: the full training split with the training augmentation

The certificate of Corollary 1 concerns the objective that is optimized. Stage 1 minimizes the loss on augmented data, so the training-objective check is the one that corresponds to the theory. The stored check evaluates the plain, unaugmented risk on a subsample; it costs one cheap pass at the switch and is the stricter reading. A practitioner can therefore read the record as follows: "committed" says the D=2 stage itself drove the group to zero; "certified" in the record is the conservative label; and the training-objective pass rate says how often the stage's own optimality condition held. The remaining 218 removals in this run are magnitude removals that meet the budget and carry no certificate.

We have made this explicit in the paper: §4.4 now reports both checks with their gradients (63–98% under the training objective, 26–92% under the stored check, across settings), and the caption of Table 16 defines each column and its denominator.

BERT. Yes, that is our reading as well, and App. F.4 states it: the allocation determines the accuracy, and the D=2 stage makes each removal lossless when it is made (logit change < 6e-6) and auditable.

Theory. We understand. Two parts of the paper may help without checking the proofs. First, §4.3 tests the central statements numerically: the D=2 group Hessian's smallest eigenvalue tracks λ − ‖∇_g L₀‖ and the D≥3 eigenvalue stays at 2λ/D, both to double precision over 6,750 points, and under full-batch gradient descent on ResNet-18 none of 1,366 D=2 removals violates the condition, against 12–13% at D≥3. Second, on novelty, the revision credits the prior work on depth-dependent geometry at zero and states our contribution as making that picture operational for structured pruning (§2, App. A).

Thank you again for the constructive engagement.

Reviewer_PoiFOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose Certified Progressive Gatting, which is a structured pruning method based on the optimisation geometry of multiplicative weight factorisation. CPG utilises the concept that stability of a zero group depends on the factorisation depth. The authors provide an evaluation of a two-stage approach which decides and certifies deletions at lower depths before re-lifting and compressing at deeper levels. They compare against fixed-depth regularisers and standard structural pruning baselines on standard benchmark datasets and architectures. Results show that dividing the pruning process across different depths achieves higher mean accuracy than fixed-depth approaches and uniquely provides a mathematical optimality certificate for the removed structure improving interpretability.

Critical strengths
  1. The insights from this work move away from simply proposing a novel sparsity schedule or regulariser to instead explaining how and why optimisation geometry traps zeroed weights based on factorisation depth. This presents insightful information regarding strict-saddle characteristics that I believe the wider community may take forward and lead to future research in optimisation theory and structural pruning in this area.
  2. The proposed method introduces a mathematically certified pruning schedule, and the authors directly acknowledge that the exact theoretical guarantees they rely on do move away from practical deep learning environments. While the exact mathematical certificates operate more as threshold-dependent heuristics in practice, they present this gap honestly and transparently.
  3. The exact canonical re-lift mechanism is a nice addition to change factorisation depth mid-training resulting in dynamic structural pruning during the initial training run. This elimination of the computational overhead of post-training threshold searches and fine-tuning cycles is a significant strength.
  4. Generally the empirical work is very robust including the generalisation performance across varied architectures (LeNet, ResNet, BERT), specifically the inclusion of targeted synthetic benchmarks to directly verify the mathematical claims and also the hardware timings which provides a practical empirical metric to support the theoretical MAC reduction.
  5. The certificate ledger approach used for auditing the deletions is a nice approach, mathematically justifying the deletions. While there are some caveats for this at deeper layers that uncertifieed removals are present to achieve competitive sparsity levels, the authors present an honest assessment of the realities of the approach.
  6. The presentation and writing is generally very good.
Critical weaknesses
  1. It is not entirely clear how the proposed method operates with stateful optimisers which they state in remark 1 they set the momentum buffers to 0 to prevent pruned weights from regrowing. Perhaps more of a question, but does this require full manual intervention? Or can this be programatically set to zero? Also, it is not clear from F.4 how “AdamW on the gates prunes 100% of FFN neurons” is not a collapse and therefore a concern. Can you provide clarity.
  2. As outlined quite extensively in the questions, the main concern is that the exact mathematical certificate that the author claims erst relies on zeros at convergence, however, this cannot be achieved in practice and therefore introduces a heuristic that is added which no longer preserves the mathematical guarantee the authors present.
  3. There is a stated limitation from the authors regarding the bias of the MAC counts for their approach, and I believe this should be stated more clearly in the main manuscript given the direct comparison to DepGraph in table 1. This bias does call into question the applicability of MAC reduction as a main comparative metric, and I would advise a separate metric is instead presented.
  4. ViT and other more modern transformer architectures are absent from the empirical study. While not a reason to reduce the weighting of the work, it does leave some more timely comparisons to be explored.
  5. It would be beneficial to see an ablation of the per-cell penalty on baseline approaches to understand the impact this penalty has over the depth schedule
Questions for authors
  1. Given the theoretical certificate relies on mathematical zeros at a convergent limit, does your use of arbitrary numerical thresholds still maintain this guarantee or instead result in a heuristic?
  2. If certificate does become a threshold heuristic during training, does the D=2 certification stage still provide a measurable empirical advantage over simpler heuristic methods like magnitude pruning?
  3. And how sensitive are the network's final sparsity and accuracy to the specific choices of ϵc\epsilon_c and δ\delta, do you need to tune these? And if tuned incorrectly does the network introduce poor certificates?
3 comments
AuthorsOpenReview ↗

Response to Reviewer PoiF

Thank you for the strong support and the constructive questions.

W1 (optimizer state).

  • Momentum. Handling is fully programmatic: when a gate's collapsed norm falls below 1e-8 of its layer's maximum, its momentum buffer is zeroed, once per stage. A re-lift creates new factor tensors and a fresh optimizer.
  • "AdamW removes 100% of FFN neurons." The precise figure is 99.85%: 57 of 36,864 remain above 1% of their layer maximum. The attention, residual and LayerNorm paths remain, and accuracy is 0.831 (one seed, soft gates) against 0.509 for the majority class, so this is not a collapse. App. F.4 now gives these figures.

W2/Q1 (exact zeros vs. thresholds). We measured each threshold's effect (App. F.6):

  1. Commit. The pre-commit and post-commit labels agree for at least 99.8% of the evaluated groups (largest change in γ: 0.031).
  2. ε_c. From 1e-4 to 1e-2 the final network is bit-identical in all but one (setting, seed) comparison: near-zero groups are the smallest ones, which magnitude pruning would remove anyway.
  3. δ. It only relabels; δ = 0.05λ adds 0–1.7 points over the exact test.
  4. Stationarity. When a tighter check is wanted, an 8-epoch settle in a 24-epoch stage 1 lowers the survivors' optimality residual by 25–43% with unchanged accuracy.

So the check stays exact, and the label is measured rather than assumed. At an exactly zero group, the zero-slack test directly verifies the stage-1 blockwise KKT condition, and this check does not require the entire network to be stationary. Corollary 1 states when the condition holds automatically; at finite training time we evaluate it for every group committed at the switch instead, and on CIFAR-10 63–98% pass using the training-objective gradient; Table 16 separately reports the stored check on 10k unaugmented images. On BERT, across the Table 2 CPG runs, all 310,173 removals are at zero and satisfy the certificate, and removing them changes the logits by < 6e-6.

Q2 (does the D=2 stage beat simpler heuristics?). Yes, in matched comparisons (new Table 3, App. F.6):

comparisonCPG margin
vs. the same D=2 start with penalty-free recovery, 16×, landing protocol+0.047 to +0.099 (t 6.0–12.0)
vs. fixed D=3 (no D=2 selection), CIFAR-10 8×–16×, boundary protocol+0.011 to +0.027 (t 4.1–7.2)
vs. deepening 3→4 without a D=2 start, 16×+0.013 (t 5.6), +0.015 (t 2.7)
vs. MAC-weighted Network Slimming (L1 + magnitude)+0.015 to +0.052
vs. DepGraph (group magnitude) on DepGraph's own budget, 8×+0.020 / +0.021

Q3 (sensitivity to ε_c, δ). We found no need to tune either threshold over the tested ranges (points 2 and 3). A mis-set δ cannot change the network: it only relabels removals, and δ is reported with the record.

W3 (cost). We adopted one rule for every method: each epoch is charged by the MACs of the network trained in it, with the measured gate overhead (dense-epoch equivalents, DEE).

  • DepGraph costs 24 DEE. Table 1's CPG configuration costs 39.4 (ResNet-18) and 46.8 (ResNet-56), with stage 2 charged at the compacted network's MACs (masked and compacted training compute the same function).
  • The revision therefore compares at equal compute (Table 14). Scaled down to DepGraph's budget at 8×, CPG leads by +0.021 (t 4.4) and +0.020 (t 2.8); given CPG's budget, DepGraph does not overtake it.
  • Other metrics. Beyond MACs, the paper reports measured speed-ups (Table 8: 2.2× on GPU at batch 512 and 5.7× on one CPU thread at batch 1, ResNet-18 16×) and parameter ratios (Tables 6–8). With a parameter budget in place of the MAC budget, CPG leads a matched fixed-D=2 control at three of four settings (App. F.1).

This statement is now in the main text (§4.1) and replaces the earlier budget-ratio claim.

W5 (per-group penalty on baselines).

  • D-Gating with the same MAC-weighted penalty is in Fig. 3 and Table 3.
  • Network Slimming with MAC weighting gains up to +0.035 but stays behind CPG in both settings under both protocols (Table 15).

So the gain is not the penalty alone.

W4 (transformers). ViT is a natural next step. On BERT, we identified and corrected an evaluation error in the submitted Table 2, which had evaluated unpruned model copies, and reran all six methods under a common protocol with assertions verifying the pruning state. The revision also introduces a saliency-targeted penalty allocation (§3.4, App. F.4). With it, CPG achieves the highest mean accuracy in all four settings, including gains of 4.7 and 2.2 points over the best baseline at 4×, with every removal certified and lossless (ablation of the allocation in new Table 10).

Other changes: the practical algorithm with both protocols and a one-run example (§3.4, Algorithm 1, App. E.2), a perturbed balanced initialization that satisfies Corollary 1's absolute-continuity requirement (App. E.3), and one consistent terminology.

Thank you to the authors for their detailed feedback.

I appreciate the clarifications made, and additional results. I would advise the authors integrate these finding and improve the clarity in places within the main text to address these concerns.

Given my already positive outlook I maintain my positive score.

AuthorsOpenReview ↗

Thank you; findings integrated into the main text

Thank you for the positive assessment and for the helpful suggestions throughout the discussion.

We agree that the key findings belong in the main text, and the revised PDF already moves most of them there:

  • Practical algorithm and protocols (§3.4): a Terms paragraph, both protocols and a one-run example.
  • Equal-compute comparison with DepGraph (§4.1), replacing the earlier budget-ratio claim, and measured speed-ups alongside MAC reduction (§4.2).
  • Controls isolating the depth schedule (§4.2, Table 3), including the per-group penalty given to the controls.
  • What the certificate measures (§4.4): the share of removals decided at D=2, with both the training-objective and the stored check, and the stability of the label under exact commitment.

For the final version, we will also bring into the main text short statements of the threshold robustness (ε_c and δ change labels, not the network) and of the programmatic optimizer-state handling, which are currently in App. F.6.

Thank you again for your support of the work.

Reviewer_uxSnOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The submission builds on D-Gating, which parametrizes the weights of each structured group (a channel, attention head or neuron) as D−1D-1 shared scalar gates times a vector of the same size as the original group. L2L_2 regularization or coupled weight decay on the factors then induces a structured sparsity penalty on the group norm raised to the power 2/D2/D, which is the convex group lasso at D=2D=2 and a non-convex structured bridge penalty at D≥3D\ge3.
The theoretical results mostly study a group whose factors are all zero. At such a balanced zero group, the curvature depends on the data at D=2D=2, but not at D≥3D\ge3 (Theorem 1). From this, the paper concludes that at a converged D=2D=2 solution, every removed group satisfies the group-lasso optimality condition ∥∇gL0∥2≤λ\lVert \nabla_g \mathcal{L}_0\rVert_2\le\lambda, which it calls the group’s “certificate” (Corollary 1). Proposition 1 derives the radius of the trapping region around zero at D≥3D\ge3, and Proposition 2 an exact balanced re-lift of a trained network to a larger depth. For the two-stage procedure, the appendix adds that a removed group stays at zero under SGD with coupled weight decay, so removals “committed” at D=2D=2 stay zero through the re-lift and the second stage (Lemma 2, Corollary 2).
Based on these results, the paper proposes Certified Progressive Gating (CPG). CPG first trains at D=2D=2, removes (near-)zero groups and records for each whether it passed the test, prunes the network to the compute target by magnitude, then re-lifts the non-zero groups to depth 3 or 4, and keeps training with the same λ\lambda. Each group’s penalty is scaled by its compute cost (MACs) to encourage computational efficiency. In the default mode, every (near-)zero group is removed, and its certificate is recorded in a per-removal “ledger”. Several variants include a short large-batch phase before the stage switch (the “settle”), slack tolerance in binary tests, and a three-stage schedule D=2→3→4D=2\to3\to4. The experiments cover synthetic group-sparse regression, ResNets on CIFAR-10/100, LeNet-5 on MNIST, ResNet-50 on ImageNet (one seed) and BERT-base.
The paper states three contributions: (i) a second-order classification of removed groups (the D=2D=2 certificate and the D≥3D\ge3 trapping barrier), (ii) the CPG algorithm with an exact re-lift between depths and a per-removal record of the certificate, and (iii) experiments in which CPG slightly outperforms comparison methods on almost all tasks.

Critical strengths

1. The theoretical results are correct and explain the qualitative changes in balanced zero groups across depths. The variational forms, the Hessian classification at a removed group, the closed forms of the trapping barrier, and the exact re-lift are all correct, to my understanding; the objective is clearly introduced, and the formal statements seem rigorously proven, though sometimes with potentially unnecessary complexity. Together they qualitatively paint a clear picture of why near-zero groups behave differently at D=2D=2 and D>2D>2: in factor space near the origin, the penalty’s gradient provably goes to zero slower than the loss gradient.

Why it matters: a sound analysis of the geometry at zero is a main contribution of the paper and gives a revised version a solid basis to build on.

2. Independently of its stated motivation, the two-stage design is a reasonable heuristic with classical precedents. It consists of a convex-penalty warm start followed by a non-convex refinement, plus mid-training commitment and MAC-weighted penalties. To explain why “train at q=1q=1, then continue at q<1q<1” might work regardless of a certificate, there are three plausible mechanisms, each with classical precedent:

  1. Recall without trapping. At q=1q=1 a zero block is not a local minimizer when it fails the KKT condition, and factor-space GD can escape at rate ∥∇gL0∥2−λ\lVert \nabla_g \mathcal{L}_0\rVert_2-\lambda. Stage 1 therefore retains groups with strong signal and, under over-selection, keeps true groups (on the synthetic problem it keeps 44 of 100 groups including essentially all 10 true ones). Stage 2 starts from a support that contains the truth and sparsifies it, instead of starting from a dense initialization and moving into the q<1q<1 traps.

  2. Bias removal. The convex penalty shrinks non-zero groups (fixed D=2D=2 keeps 0.40 of the true magnitude in the synthetic data experiment). The bridge penalty’s vanishing marginal penalty rate causes less parameter shrinkage and yields better estimation (0.87–0.91 in Figure 5c). This is the rationale of the adaptive lasso [1], one-step LLA [2], reweighted L1L_1 [3], and multi-stage convex relaxation [4].

  3. Runtime. Committing removals mid-training lets stage 2 train a smaller network (24–29 vs 49 dense-epoch equivalents).

None of these needs the KKT label. Mechanisms 1–2 depend on the stage-1 support and its recall. The theory of LLA [5] assumes folded-concave penalties with identically zero derivatives away from the origin (SCAD/MCP). Bridge penalties have vanishing but strictly positive derivatives, so those guarantees do not directly transfer. The design is hence plausible and classical, but heuristic. This suggests the intuition that stage 1 fixes a "failure" that is specific to factor-space (S)GD at q<1q<1, which is potentially a real benefit when one relies on differentiable training by (S)GD with weight decay, which is the practical appeal of D-Gating.

Why it matters: the method can potentially explain its (small to moderate) empirical gains through these mechanisms, which do not rely on the certificate and show that there likely is something worthwhile to be analyzed. The certificate framework the submission offers, however, is not convincing to me (see Critical Weakness 1).

References

  • [1] H. Zou. The adaptive lasso and its oracle properties. Journal of the American Statistical Association 101(476):1418–1429, 2006.

  • [2] H. Zou, R. Li. One-step sparse estimates in nonconcave penalized likelihood models. Annals of Statistics 36(4):1509–1533, 2008.

  • [3] E. J. Candès, M. B. Wakin, S. P. Boyd. Enhancing sparsity by reweighted ℓ1\ell_1 minimization. Journal of Fourier Analysis and Applications 14(5–6):877–905, 2008.

  • [4] T. Zhang. Analysis of multi-stage convex relaxation for sparse regularization. Journal of Machine Learning Research 11:1081–1107, 2010.

  • [5] J. Fan, L. Xue, H. Zou. Strong oracle optimality of folded concave penalized estimation. Annals of Statistics 42(3):819–849, 2014.

Critical weaknesses

Assessment of the stated contributions.

  • Contribution (i) is correct (Critical Strength 1) but largely known (W11). The certificate is the group-lasso KKT condition applied outside the assumptions of Corollary 1 (W1–W3), and the barrier holds only for a frozen, affine single-group model (W5). Strangely, the default CPG mode "BOUNDARY" permits every near-zero group to progress to stage 2 training, regardless of its certificate outcome, calling into question its importance. The optional "revival rule" is off by default and revives uncertified near-zero groups along the direction of steepest descent from the saddle. However, ablation studies show that this rule is unimportant or even decreases performance, undermining the role of the certificate.

  • Contribution (ii) is a plausible two-stage heuristic (Critical Strength 2). However, in the supplied code the stage-2 re-lift only reparametrizes an unpenalized gate, except in the synthetic data experiment (W4); the choice λ1=λ2\lambda_1=\lambda_2 for the q=1q=1 and q=2/Dmax⁡q=2/D_{\max} stages has no theoretical justification (W7); and Algorithm 1 does not describe the method as run (W8).

  • Contribution (iii): the experimental results as a whole are not convincing. The synthetic experiment seems inconclusive or shows only a marginal benefit for CPG (W10, W16), parts of the results cannot be reproduced with the supplied code (W4, W9, W10), and the exact settings are often unclear or insufficiently described (W8, W14). Many comparison methods are either for unstructured sparsity using a scalar factorization (spred, DWF, Hadamard) or for structured sparsity using D-Gating (fixed D=2D=2, D=3D=3, D=4D=4) at different depths, reducing the effective number of strong baseline comparison methods (W13, W17). Hyperparameters and training procedures are also not adequately reported (W8, W9), lending all in all little credibility to the experimental results.

The problems mainly fall into two groups:

  • statements whose assumptions silently exclude cases the paper explicitly admits

  • algorithmic definitions that do not encompass what the text says they do

1. The certificate and the trapping barrier are exact results, but applied outside their scope. A significant issue is that the results might be theoretically sound, but are largely inapplicable (and thus do not characterize the methods used in the experiments).

  • Corollary 1 does not cover the method as specified. It requires an initialization that is absolutely continuous on UcU_c and gradient descent, whereas CPG uses a balanced initialization, a Lebesgue-null set, and SGD. Corollaries 1 and 3 even have mutually exclusive initialization assumptions, and a special case of D-Gating shows how a balanced initialization can lead to convergence to a strict saddle under SGD (see W2, W3).

  • The certificate concerns a different objective. A “certified” zero group only satisfies the group-wise first-order condition of the L2,1L_{2,1} regularized objective, a penalty that is only used in stage 1 and has suboptimal selection properties and estimation bias. It says nothing about whether the zero group is a good or bad critical point of the deeper stage-2 objective, and under the assumptions of Corollary 1 every zero group would pass, in which case the test reduces to a convergence check for D=2D=2 (see W1).

  • The barrier belongs to a simplified problem. Proposition 1 is derived for a single-group subproblem with all other parameters fixed and an affine loss, and does not transfer to coupled objectives. The submission nevertheless tends to present such conditional results as unconditional properties of training, at times by omission (see W5, W12).

Why it matters: the certificate gives the method its name and is a main contribution. If its assumptions exclude every reported run and the condition it checks belongs to a different objective, the claim of certified pruning is not supported.

2. The method, its settings, and its experiments are presented confusingly and inconsistently, and the supplied code does not match the description.

  • Comprehensibility. The paper is difficult to read, and the difficulty mainly stems from the vocabulary used. The paper introduces many idiosyncratic terms, uses several in more than one (often non-standard) sense (“boundary”, “landing”, “ledger” and “cell” all have several meanings), and defines some only in the appendix or after first use. In these cases, a reader must reconstruct the method from captions, other parts of the paper, or appendix content (see W6).

  • Settings and procedures. The description of the procedure is incomplete and inconsistent. Algorithm 1 omits the “settle”, the certificate slack, the landing and boundary/landing protocols, “gated freezing” and revival, but includes a STRICT mode that the code does not implement and the paper does not consider. The choice of stage-wise strengths λ1=λ2\lambda_1=\lambda_2 has, to my knowledge, no theoretical justification. Further, it is not clear whether AdamW’s decoupled weight decay was erroneously used as the regularization, and the setting “gate off” is defined as “CPG, uniform λ\lambda, w/o settle”, yet Table 6 reports different results for the two (see W6–W8, W14).

  • Experiments. The empirical gains are rather small, but to the extent they exist, they could be attributable to the warm start, mid-training commitment and MAC-weighted penalties rather than to certification. Some baselines are named after different papers but in fact are the same element-wise family at different depths, and it is unclear how their unstructured sparsity is aggregated into structured sparsity. For details and other issues, see W10, W13, W16, W17.

  • Code and reproducibility. In the provided code, the CNN pipeline stage 2 trains without any penalty, although the paper states λ2=λ1\lambda_2=\lambda_1, so the deep stage never compresses. Instead, compression comes from magnitude pruning to the MAC target at the boundary. The supplementary material does not contain what the reproducibility statement entails: model implementations, hyperparameter sweeps, and the configurations of the reported runs are missing, so most tables cannot be faithfully reproduced. My reproduction of the synthetic benchmark matches the CPG rows but not the fixed-depth baselines, and even the README’s own demo output differs from the paper (see W4, W9, W10).

Why it matters: from the paper alone, it is often difficult to tell which procedure and which settings produced the reported numbers, and the released code does not resolve this. The paper’s case for acceptance, a certified pruning method that outperforms strong baselines, thus cannot be appropriately checked, and where it can be checked, it does not always hold as stated.




The following contains descriptions of specific weaknesses which are referred to in the other sections

  • W1. The result’s name “Certified pruning at depth two” overplays the role of the group-wise KKT conditions. A “certified” zero group is merely certified to be a first-order optimal point of the block-coordinate, single-group effective L2,1L_{2,1} regularized subproblem. Why partial fulfillment of the group-wise L2,1L_{2,1} first-order condition would be a useful criterion for pruning is not sufficiently motivated, especially since the L2,1L_{2,1} penalty is known to suffer from suboptimal selection properties and parameter shrinkage compared to L2,qL_{2,q} with q<1q<1 [6]. Using optimality conditions of this method as “certified pruning” is not convincing. The provided reasoning against fixed D>2D>2 is that the loss Hessian degenerates at balanced zero groups, implying that the regularizer’s positive definiteness ensures local minimality of a (balanced) zero group irrespective of the loss gradient – a well-known fact in sparse regularization and differentiable factorization methods [7, 8, 9]. Hence, the certificate provides no additional information on whether that zero group is a good or bad local solution to the D=3D=3 stage-2 objective, as it is an optimality condition for a different objective. If the assumptions of Cor. 1 held, all balanced zero groups would be “certified”, and its use would be limited mainly to checking convergence for D=2D=2 (q=1q=1).

    • For D=2D=2 and initialization according to Cor. 1, the certificate can be employed as a convergence indicator of the underlying L2,1L_{2,1} regularized objective.

    • With balanced initialization (Cor. 1 does not apply) and a stationary stage 1, an uncertified group at a balanced zero is a strict saddle of the D=2D=2 factorized objective that gradient descent cannot leave due to the absorbing state under (S)GD. Re-initializing these groups and continuing at D=2D=2 until every candidate passes (similar to an active-set loop) could constitute a use case in which the certificate decides something.

  • W2. Cor. 1 requires the initialization to be absolutely continuous w.r.t. the Lebesgue measure on UcU_c, which the proposed method fails to satisfy by using a balanced (null set) initialization constructed from a standard initialization for the effective parameter group. Therefore, the result might not apply to balanced initializations or SGD instead of GD, both of which are used repeatedly in theoretical results and experiments. While this does not invalidate the result, it makes it inapplicable. For scalar groups of size 1, a balanced initialization with the wrong effective sign at initialization cannot change sign, and it converges to the latent origin, a strict saddle that violates the D=2D=2 KKT condition [10].

  • W3. Corollaries 1 and 3 have mutually exclusive initialization assumptions. Cor. 1 requires an initialization that is absolutely continuous on UcU_c, whereas Cor. 3 assumes initialization on the balanced manifold [p. 19, l. 1023] (see W2). The method follows Cor. 3: Appendix E.2 starts stage 1 on the balanced manifold so that Lemma 1 holds at every stage [p. 24, l. 1298–1300], and the re-lift of Proposition 2 returns a balanced point [p. 6, l. 281–282]. Cor. 1 therefore does not cover any CPG run by design.

  • W4. Stage 2 uses no penalty in the code; the paper trains it with the Dmax⁡D_{\max} penalty. In the CNN pipeline the deep stage trains without any penalty, so the depth Dmax⁡D_{\max} compression never happens. Compression mainly comes from magnitude pruning to the MAC target at the boundary.

    • Paper.

      • Algorithm 1 step 5: “Compress. Train at Dmax⁡D_{\max}, then commit the remaining near-zeros as uncertified removals” [p. 24, l. 1293].

      • “Stage 2 reuses the stage-1 penalty, λ2=λ1\lambda_2=\lambda_1, which gives higher accuracy and a lower uncertified rate than a larger stage-2 penalty” [p. 32, l. 1713–1714].

      • Gated freezing works by “excluding survivors with γg<0.25\gamma_g<0.25 from the stage-2 penalty” [p. 33, l. 1776].

      • Sec. 3.4: “a deeper second stage compresses harder” [p. 6, l. 299].

    • Code.

      • The CNN trainer runs stage 2 with penalty=None.

      • The module docstring says “Stage 2 is penalty-free” (cpg/trainer.py:13), and README Sec. 1 step 4 says “recover without the penalty”.

      • No near-zero commit happens after the boundary.

      • The Dmax⁡D_{\max} re-lift only reparametrizes an unpenalized gate scale. The exponent 2/Dmax⁡2/D_{\max} that the paper credits with compression never enters an objective.

    • Consequence.

      • Several results cannot be produced with the released code: the Dmax⁡∈{3,4}D_{\max}\in\{3,4\} comparison, the three-stage row, gated freezing (F.5, Section 4.4) and the “no D=2D=2 stage” control 3→43\to4 (Table 4).

      • Unless the paper’s CNN runs used other code, the CIFAR results also do not test “compress at Dmax⁡D_{\max}”.

    • Note. The synthetic fit does keep the penalty in stage 2 (cpg/synthetic.py:81), i.e., the two demos implement two methods under the same name.

  • W5. The “trapping” barrier for D≥3D \geq 3 in Prop. 1 is derived for a loosely related objective, a single-group subproblem with all other parameters frozen and the loss assumed to be affine in that group. While this offers qualitative insights into initialization criteria for D=2D=2 and D>2D>2, the barrier is, similar to the “certificate”, an exact result applied to settings outside its appropriate scope. Without these restrictive assumptions and approximations, the result loses its sharpness and the usefulness of the exact analytical expression is limited.

  • W6. The paper is difficult to read, since it contains many non-standard terms, overloads their meaning, and defines some only in the appendix or after first use. The glossary is in Appendix C [p. 17, l. 866–870], not in the main text. Examples:

    • “Boundary” has various meanings:

      • the switch from D=2D=2 to Dmax⁡D_{\max} (“decides there, at the boundary” [p. 2, l. 084]; the title of Sec. 4.2), spelled out as “the D=2D=2 stage boundary” only in Definition 1 [p. 17, l. 906]

      • BOUNDARY mode, i.e., commit every candidate, as opposed to STRICT [p. 17, l. 910]

      • the boundary protocol, i.e., commit plus magnitude trim at the stage switch [p. 6, l. 321–323]

      • the “escape–trapping boundary of Proposition 1” [p. 29, l. 1538], i.e., the barrier radius

      • the “non-smooth stationarity boundary” [p. 22, l. 1165–1166], i.e., the KKT threshold

      • “the D=2D=2 versus D≥3D\ge3 boundary” [p. 14, l. 747–748]

      To read “Deciding at the boundary” one must first infer whether the stage switch or the trapping barrier is meant.

    Other examples of non-standard, confusing, or undefined terminology:

    • “Cell” is used in the introduction and Figure 1 [p. 2, l. 065, 075] before Sec. 3.1 defines it as an element of the partition [p. 3, l. 124–125]. It alternates with “group” and “unit”, and it also means an experimental setting: “the ten cells of Table 6” [p. 27, l. 1443], “every cell of Figure 3” [p. 27, l. 1452], “one seed per cell” [p. 32, l. 1719].

    • “Landing” names two things, neither of them defined:

      • a protocol: threshold at the end of training, then fine-tune [p. 6, l. 321–323]

      • the magnitude trim to the target inside the boundary protocol (“boundary landing” [p. 28, l. 1496]; Figure 6), which is not in Algorithm 1

    • Further non-standard or unclear terms are: certificate, ledger, commit (to what?), gated freezing (freezes nothing?), “the settle” (which are large-batch iterations toward the end of a stage), latent zero, majority-compatible operating point.

    • “gate off” is defined as “2→42\to4, uniform penalty, no settle or certificate” in the caption of Fig. 3, so it should correspond to the setting “CPG, uniform λ\lambda, w/o settle” in Table 6. But the results in Table 6 differ between “gate off” and “CPG, uniform λ\lambda, w/o settle”.

    • The statement appearing twice that two factors “turn spurious stationary points into strict saddles” is incorrect. For the convex group lasso, the non-smooth base problem has no spurious stationary points that a factorization could transform into anything. The factorization creates spurious stationary points at zero groups. At D=2D=2, these spurious stationary points are strict saddles [11].

  • W7. There is no theoretical justification for using λ1=λ2\lambda_1=\lambda_2, since stages 1 and 2 pose different optimization problems with different scales and ranges for λ\lambda. Despite being computationally more intensive, tuning the second-stage λ2\lambda_2 conditional on the selected stage-1 support seems reasonable.

  • W8. Algorithm 1 does not sufficiently characterize the proposed method. The algorithm is missing the important components of a large-batch phase at the end of training (the “settle”), a certificate slack δ=0.05λg\delta=0.05\lambda_g, a magnitude-pruning “landing protocol”, gated freezing, the “boundary protocol” (whose magnitude pruning at the stage switch is the “boundary landing”), and parameter revival. On the other hand, STRICT mode is included but not implemented in the code. Moreover, the supplied code is inconsistent with Alg. 1 in using a penalty-free stage 2 except for the synthetic application, despite the text stating λ1=λ2\lambda_1=\lambda_2.

  • W9. The supplied code does not align with what the reproducibility statement says it contains.

    • Paper. “Source code for CPG, the baselines and the evaluation pipeline is provided as anonymized supplementary material”, “with every configuration in the released code” [p. 10, l. 525–528].

    • Appendix F.6 promises hyperparameters, but none are given for the λ\lambda grid, the settle, the bisection, the full-batch run or the BERT schedules. In the code, README Sec. 8 states: “Not included: baseline implementations, the hyper-parameter sweeps, the BERT-base experiments and the analysis scripts of the paper.” This directly contradicts the paper’s claim above.

    • Also missing: LeNet-5, ImageNet and the per-cell configuration files; Table 9 controls (proximal solvers etc.), gated freezing, the three-stage schedule, the λ\lambda values of any reported run. Hence, most of the tables cannot be reproduced from the supplied code.

  • W10. Synthetic benchmark, not reproduced (5 seeds). The D-Gating baselines of the synthetic experiment do not reproduce. The code reproduces the paper’s CPG rows exactly but not its fixed-depth baselines. Fixed D=3D=3 recovers the support exactly on 4 of 5 seeds, and CPG differs from it only on the fifth. The README’s own demo output even gives fixed D=3D=3 F1 1.0, identical to CPG.

    • Setup. The released synthetic fit has the following defaults: p=1000p=1000, G=100G=100, s=10s=10, n=160n=160, λ=0.02\lambda=0.02, 12,000 steps, initialization 0.1⋅N(0,1)0.1\cdot\mathcal N(0,1), floor 10−310^{-3}. These match the Table 9 caption [p. 33, l. 1728–1729]. Seeds 0–4.

    • CPG rows. The CPG rows reproduce Table 9 exactly.

      • CPG. Kept 10.0±0.010.0\pm0.0, F1 1.0±0.01.0\pm0.0, violation rate 0.420; the paper reports 0.420.

      • CPG, Dmax⁡=4D_{\max}=4. F1 1.0, violation rate 0.196; the paper reports 0.196.

    • Fixed-depth baselines. The fixed-depth baseline results are inconsistent or do not reproduce. The submitted code gives a stronger fixed D=3D=3 baseline than the paper reports. The README contradicts Table 9 too. Its expected output in the README lists fixed D=3D=3 at kept 10.0, F1 1.0±0.01.0\pm0.0, rel. error 0.019, violation 0.374, identical to CPG. Seeds 0–2 of this run reproduce that table exactly (fixed D=2D=2 kept 42.0, F1 0.387). My reproduction calls the script unchanged, with the demo’s seed loop; only the seed count differs.

    • Consequence. Table 1’s “the only one that recovers the synthetic support exactly” [p. 7, l. 359] therefore rests on the seed configuration/count in the supplied code.

  • W11. The theoretical results mostly build upon/apply existing results with minor modifications. In parts, the paper suggests an overstatement of its own contributions: the depth dependence of the geometry at zero is known. The paper says that “to our knowledge none characterizes how the geometry at a zero depends on depth” [p. 3, l. 115–117] and that the second-order classification is “not established in prior work” [p. 14, l. 749–750]. Ziyin, Li and Meng [12] show that weight decay creates bad minima at zero in networks with more than one hidden layer, unlike networks with one. That is the D=2D=2 versus D≥3D\ge3 boundary, with the same mechanism: with three or more factors, the data term has no second-order contribution at the origin. They also discuss local minimum traps and establish numerical approximations of the trap radius. Kawaguchi [13] shows that deep linear networks with more than three layers have saddles whose Hessian has no negative eigenvalue. More directly, Kolb et al. [7], which is cited in the submission, already contains the group-structured version for general smooth losses. The submission cites it with the observation “that for depth k>2k>2 the zero state acquires positive curvature” [p. 14, l. 741], which is inconsistent with its own novelty sentences.

  • W12. The paper (silently by omission of limitations) overstates conditional, block-local, frozen-model, or linearized model results as general properties:

    • “a run that comes close to a zero is drawn into it whether or not the data would have kept the cell” [p. 4, l. 198]: in the frozen affine model a cell inside its data-dependent barrier returns to zero; outside it escapes. But the radius itself depends on the loss gradient, so it is not fully data-independent.

    • “the only method that records which of its deletions satisfy their optimality condition” [p. 1, l. 028–029]: the condition is the group-wise KKT condition for L2,1L_{2,1} regularization, which is an unconvincing concept of optimality.

  • W13. The baselines are named slightly confusingly with different method names and sources, but run as one method family. Table 1 lists “spred”, “DWF” and “Hadamard” [p. 6, l. 317–318]. Table 9 states “Element-wise D=2D=2 (spred-like)”, “D=3D=3 (DWF-like)” and “D=4D=4 (Hadamard-like)” [p. 33, l. 1750–1752]. They are therefore one element-wise factorization run at depths 2, 3 and 4, not three published methods (which would differ in, e.g., their initialization).

    • All three papers study multiplicative factorizations of the parameters. Kolb et al. [7] treat several parametrizations at any depth kk, including grouped ones (GHPPk\mathrm{GHPP}_k), so “Hadamard” does not uniquely identify a method. The related-work table reduces that paper to “element-wise, kk factors” [p. 16, l. 860].
  • W14. AdamW is used, but only coupled weight decay equals the required L2L_2 regularization of the differentiable D-Gating objective. It is not clear if the decoupled weight decay in AdamW was used as the regularization strength or if an explicit L2L_2 penalty was added on top of decoupled weight decay.

  • W15. The purpose and presentation of D.9 and Theorem 3 are not clear to me. The second half of the Theorem statement should be plain text below the theorem. The subsequent proof sketch is too vague and informal.

  • W16. It is unclear whether the synthetic experiment compares fair, tuned baselines. Table 9 runs “one λ\lambda for every row” [p. 33, l. 1729]. Table 1 repeats these numbers under the caption “strengths are selected on validation” [p. 7, l. 326]. It remains unclear if the wording in Table 9 means that the same λ\lambda was used across models or tuned separately. In the former case, the claim that CPG alone recovers the support exactly would be weak without separate per-method tuning.

  • W17. Some comparison methods (spred, DWF, Hadamard and Powerpropagation) are for unstructured (parameter-wise) sparsity and thus not a fair comparison for structured sparsification and compression targets. It is not documented how unstructured sparsity is converted into structured sparsity for these cases. This considerably weakens the experimental comparison, leaving fewer fair structured sparsity competitors.

References

  • [6] Y. Hu, C. Li, K. Meng, J. Qin, X. Yang. Group sparse optimization via ℓp,q\ell_{p,q} regularization. Journal of Machine Learning Research 18(30):1–52, 2017.

  • [7] C. Kolb, C. L. Müller, B. Bischl, D. Rügamer. Smoothing the edges: smooth optimization for sparse regularization using Hadamard overparametrization. Machine Learning 115:87, 2026.

  • [8] P.-L. Loh, M. J. Wainwright. Regularized M-estimators with nonconvexity: statistical and algorithmic theory for local optima. Journal of Machine Learning Research 16:559–616, 2015.

  • [9] B. D. Haeffele, R. Vidal. Global optimality in neural network training. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7331–7339, 2017.

  • [10] T. Jacobs, R. Burkholz. Mask in the mirror: implicit sparsification. International Conference on Learning Representations (ICLR), 2025.

  • [11] C. Poon, G. Peyré. Smooth bilevel programming for sparse regularization. Advances in Neural Information Processing Systems (NeurIPS) 34:1543–1555, 2021.

  • [12] L. Ziyin, B. Li, X. Meng. Exact solutions of a deep linear network. Advances in Neural Information Processing Systems (NeurIPS) 35, 2022.

  • [13] K. Kawaguchi. Deep learning without poor local minima. Advances in Neural Information Processing Systems (NeurIPS) 29, 2016.

Questions for authors

Clarification requests

  1. Could you explain the difference between “gate off” and “CPG, uniform λ\lambda, w/o settle” in Table 6, and why their results differ?

  2. The results in Figure 4 seem unsurprising, since only D=2D=2 matches the objective the certificate checks. Increasing DD yields a different objective, for which the D=2D=2 KKT conditions are clearly not applicable. What motivates this comparison of certificate ratios over removed groups? Is there something I am missing?

  3. What does “certified” mean for the final model? Under which assumptions does a structured bridge minimizer have its support inside the stage-1 support?

  4. How close is stage 1 to stationarity at the switch (measured in, e.g., γg\gamma_g)? The reported epoch numbers seem low for some applications.

  5. Which code in the supplementary material produced the reported synthetic experiment results, which I could not fully reproduce? Was λ\lambda for the Table 9 baselines selected per method (see W16)?

Suggestions (no need to reply)

  1. For better readability, I recommend providing a short glossary/notation table or defining all terms at first use, sticking to one meaning per term if possible, and using plain and established names (for example “stage switch” rather than “boundary”, “slack δ\delta” rather than “ledger on/off”, “parameter group” for “cell” or “large-batch iterations” for “settle”), with baselines named by parametrization, depth and grouping.

  2. How does CPG compare with proximal group-L2,qL_{2,q}, LLA or adaptive group lasso from the D=2D=2 solution, a SCAD/MCP second stage, and an independently tuned stage-2 λ2\lambda_2? Revival re-initializes failing groups along their descent direction but is off by default; why not continue at D=2D=2 until all removals pass (an active-set loop, see W1)?

  3. It would significantly improve the exposition if Algorithm 1 were stated as it is used in the experiments and not only in its rough form, with the optional variants explicitly included (the “settle”, the slack δ\delta, the candidate tolerance, the potential landing, and the stage-2 optimizer/penalty) (W4, W8–W10).

5 comments
AuthorsOpenReview ↗

Response to Reviewer uxSn (1/2): method as run, mechanism, experiments

Thank you for an exceptionally careful review, including running our code.

W4/W8 (stage 2; Algorithm 1). You are right. The released CNN trainer implements the boundary protocol, which produced the headline CIFAR-10 rows: the MAC target is met at the stage switch (near-zero groups plus the smallest survivors), and the D=3 stage retrains the pruned network without the penalty. λ₂=λ₁ is the landing protocol (CIFAR-100, Table 5 (old 4), synthetic), where the D=3 stage keeps compressing. The submission described only the second. §3.4 now states both, and Algorithm 1 is restated as run: settle, ε_c, δ, pruning to the target, the stage-2 objective per protocol, BN recalibration. STRICT is marked as unused, gated freezing is removed, and App. E.1 tabulates the configuration of the main CPG entries.

"A plausible heuristic: warm start, commitment, MAC weighting" (new Table 3). All rows share the MAC-weighted penalty, the 36-epoch budget and a λ search; paired rows branch from one stage-1 checkpoint; runs are deterministic. Entries are CPG minus the control (top-1).

controlR56 8×R56 16×R18 16×C100 R56 16×R18 32×
landing: same D=2 start, penalty-free stage 2+0.008+0.096 (t 6.0)+0.047 (t 8.1)+0.099 (t 12.0)—
landing: fixed D=2, λ searched+0.003+0.030 (t 3.4)+0.008+0.035 (t 7.1)+0.036 (t 5.5)
landing: fixed D=3, λ searched−0.000+0.020 (t 2.2)+0.004+0.023+0.006 (t 2.6)
boundary: fixed D=3, λ searched+0.011 (t 4.1)+0.027 (t 4.7)+0.011 (t 7.2)+0.014+0.006
boundary: fixed D=2 + settle, λ searched+0.008 (t 2.5)+0.017 (t 1.6)+0.008 (t 3.1)−0.022−0.000
  • Penalized continuation. The penalized deep stage does the compressing: at ResNet-56 16× it leaves 13 of 945 removals to magnitude pruning, against 491 without it.
  • Depth transition. Under the boundary protocol on CIFAR-10 at 8–16×, fixed D=3 is no better than fixed D=2, and re-lifting after the same switch adds +0.008 to +0.017. On CIFAR-100 and at 32×, CPG and fixed D=2 are level when paired at equal λ (−0.001, −0.000); with λ searched, fixed D=2 is ahead on CIFAR-100.
  • Matched penalty weighting. Network Slimming with the same MAC-weighted penalty stays behind under both protocols, by 0.015 to 0.052 (Table 15).
  • Equal compute. On DepGraph's own budget (24 dense-epoch equivalents), CPG leads by +0.021 (t 4.4) and +0.020 (t 2.8) at 8× (Table 14).

Convex-then-non-convex estimation (your refs [1–4]) is a close classical relative; the revision cites it. On the synthetic problem, adaptive group lasso and proximal ℓ_{2/3} started from CPG's D=2 point also recover the support (18/20, 19/20), confirming that the D=2 stage supplies the right starting point. CPG obtains both stages inside one SGD run through the exact re-lift.

W7 (λ₂=λ₁). Now presented as an empirical default: λ₂=2λ₁ is within 0.006 everywhere, and λ₂=0.5λ₁ is clearly worse only at ResNet-56 16×.

W9. App. F.7 now lists the hyperparameters; the updated anonymized supplement has the CPG pipeline with configurations, λ grids, seeds and outputs, and a map from each CPG table to its command; the reproducibility statement now matches the release.

W10/W16/Q5 (synthetic). Both codes see identical data but draw the initialization from different generators (CPU vs CUDA). Swapping the draws reproduces Table 12's (old 9) fixed-D=3 seeds [.44 .60 1 1 1] and yours [1 1 1 1 .6]. Fixed D=3 is seed-sensitive; CPG is exact on 5/5 under both. Over 20 seeds with λ tuned per method (Table 17), exact recoveries are CPG 19/20, proximal ℓ_q 18/20, fixed D=3 16/20.

W13/W17 (baselines; App. F.1).

  • spred, DWF, Hadamard and Powerpropagation are now described as element-wise factorization controls, made structured by landing on each filter's norm; Table 1 now uses Table 12's names ("Element-wise D=2 (spred-like)", …), and Table 4 lists Kolb et al. (2026) as element-wise or grouped.
  • Slimming and group lasso act on the same channel scale as the gates.
  • DepGraph searches a larger set that includes the residual stream.

BERT. We identified and corrected an evaluation error in the submitted Table 2, which had evaluated unpruned model copies, and reran all six methods under a common protocol with assertions verifying the pruning state. The revision also introduces a saliency-targeted penalty allocation (§3.4, App. F.4). With it, CPG achieves the highest mean accuracy in all four settings (+4.7 and +2.2 points at 4×). In the Table 2 CPG runs, all 310,173 removals satisfy the decision-time certificate, and removing them changes the logits by < 6e-6. New Table 10 ablates it: the MAC-proportional penalty of eq. (6) averages 0.819, and hard removal with the same ranking is level with CPG (|t| < 1). The allocation sets the accuracy; the D=2 stage makes every removal lossless and certified.

AuthorsOpenReview ↗

Response to Reviewer uxSn (2/2): scope of the theory, novelty, exposition

W1/Q3 (what "certified" means). We agree on the scope, and the revision states it in §3.2: the certificate is the groupwise KKT condition of the stage-1 (q=1) objective at the moment a group is removed. It audits the selection stage, whose job is recall without trapping (your Mechanism 1); at D≥3, convergence alone does not guarantee this condition. We do not claim that a minimizer of the stage-2 objective has its support inside the stage-1 support; CPG's final support is a subset of it by construction, since committed groups stay removed. Under Algorithm 1 on the synthetic problem, every group committed at the switch passes; across the Table 2 CPG runs on BERT, all 310,173 removals do.

W2/W3 (initialization). We narrowed this gap. The reported runs use the balanced start, and the revision adds, as an option, a perturbed balanced initialization, ω₀ = balanced + σ_p ξ (App. E.3), which is absolutely continuous as Corollary 1 requires and removes the scalar-gate sign invariance of Jacobs & Burkholz (2025). Measured:

  • the imbalance decays at 1.16–1.20× the Lemma-2 rate and is below 1e-10 long before the switch;
  • accuracy changes by ≤0.006 and the near-zero count by ≤8%;
  • ≤0.2% of gates end stage 1 with a negative scale;
  • synthetic CPG is bit-identical.

ReLU networks and stochastic gradients remain outside the hypotheses; Fig. 4 measures the condition for them directly.

Q4 (stationarity at the switch). The median optimality residual of the survivors is 0.76–1.03 and the relative gradient norm 2.5–4.2e-3. The test does not require global stationarity: at an exactly zero group, the zero-slack test directly verifies the stage-1 blockwise KKT condition, and Corollary 1 states when it holds automatically (§3.2). A longer settle (8 instead of 2 epochs in a 24-epoch stage 1) lowers the residual by 25–43% with unchanged accuracy. Pre- and post-commit labels agree for ≥99.8% of the groups.

W5/W12 (barrier). Now explicit: Proposition 1 is a frozen single-group, leading-order result (exact in the affine model, with a remainder bound otherwise). Its radius depends on the data, while the Hessian at zero does not. The sentence you quote now reads: a group within that radius "is held there by a curvature that the data do not enter". The contributions (§1) now say CPG records which removals satisfy "the group-lasso optimality condition of the stage that made them".

W11 (novelty). Agreed: depth-dependent geometry at zero is known in related settings, and the revision credits Ziyin et al. (2022), Kawaguchi (2016) and Kolb et al. (2026) in §2 and App. A; the App. B clause claiming no prior treatment is removed. Our contribution makes this picture operational for structured pruning:

  1. the exact shared-gate spectrum with the KKT threshold, turned into a per-removal certificate recorded when a group is removed;
  2. the leading-order size of the D≥3 trap;
  3. an exact closed-form re-lift, so one SGD run can change depth between the two roles;
  4. controlled evidence that this schedule produces the gains (part 1/2).

W6 (terminology). Revised throughout, with one meaning per term and a Terms paragraph in §3.4: cell → group, boundary (as a time) → stage switch, ledger → decision record, BOUNDARY mode → COMMIT-ALL mode.

The strict-saddle sentence now reads: a factorization creates stationary points at zero, and with two factors the non-optimal ones are strict saddles.

W14 (AdamW). No decoupled decay is used as the regularizer anywhere. On CIFAR the gate penalty is an explicit loss term optimized by SGD. On BERT the gates use SGD with the explicit penalty and the backbone AdamW with weight_decay=0; the AdamW-gate ablation keeps the explicit penalty in the loss.

W15. Theorem 3 now states only hypotheses and conclusion; its role (supplying the convergence Corollary 1 assumes) follows in the text.

Q1 (gate off vs. "CPG, uniform λ, w/o settle", Table 7 (old 6)). They are one configuration: at equal λ and seed they give bit-identical networks. Their two rows are independent λ selections in the submitted, non-deterministic runs, so the difference shows the run-to-run spread of the selection; the caption now says so. The new controls use deterministic kernels.

Q2 (Fig. 4). It tests the prediction outside the theory's hypotheses (SGD, ReLU, BN). D=2 removals satisfy the condition far more often than D≥3 removals: 97% vs 72–80% on LeNet-5 and 53% vs 4–5% on ResNet-18; under full-batch gradient descent, every D=2 removal does.

Suggestion 7 (active-set loop). Implemented (App. F.6). On the synthetic problem no near-zero group fails at the switch, so the loop is a no-op. On CIFAR, in three rounds of re-admitting failing groups off the balanced manifold, 63–89% collapse again within two epochs. Accuracy is unchanged, which supports recording the certificate without enforcing it.

Thank you again; we hope the revision resolves the soundness concerns.

Reviewer_uxSnOpenReview ↗

Response to rebuttal 1/2

Thank you for the detailed responses and additional experiments. Several specific points are now satisfactorily addressed (stage objectives, initialization discrepancy, optimizer details, baseline labels, stationarity diagnostics, and original λ sharing). Below, I address the remaining concerns across two comments. Note that when a weakness W concerns several issues, resolving or clarifying one does not necessarily resolve the broader criticism.

Synthetic results (W10/W16/Q5).

Both codes see identical data but draw the initialization from different generators (CPU vs CUDA).

Thank you for explaining this discrepancy. The clarified shared-λ\lambda design answers W16, and the newly tuned comparison is welcome. However, explaining the discrepancy does not resolve the concern about a substantial, robust comparative advantage. The reported 19/20 recoveries versus 16/20 and 18/20 are modest differences, while proximal continuation from the same D=2 performs in the same range. These counts alone do not establish a considerable benefit. My concern is the sensitivity and interpretation of the comparison, not a claim that CPG itself has been shown to be unstable. CPU/GPU computation, seed numbers, or number of repetitions should not change the results qualitatively.

What explains the gains? (mechanism concerns and W17).

All rows share the MAC-weighted penalty, the 36-epoch budget and a λ search; paired rows branch from one stage-1 checkpoint; runs are deterministic.

Convex-then-non-convex estimation (your refs [1–4]) is a close classical relative; the revision cites it.

Thank you for adding matched controls and the warm-started classical alternatives. These directly engage with my recommendations, and some reported gains are appreciable. They provide evidence about continuation, depth, weighting and compute. They do not, however, isolate a benefit from certification. The same-start proximal result also strengthens the alternative explanation that a good first-stage support and subsequent bias reduction could account for the performance. Correcting baseline names and disclosing their adaptation helps W13/W17, but the element-wise variants remain controls from one factorization family rather than several independent structured methods. Hence, comparing unstructured sparsity methods with a structured sparsity method on structured sparsity metrics renders these methods unsuitable, especially for more than a simple check/ablation. Moreover, the standard group lasso regularization is known to overselect, which has been conceded, but it can also push parameters in the true support to zero in relation to the shrinkage of convex relaxation, which non-convex penalties mostly avoid. This would mean the method also poses the risk of prematurely removing a group that a deeper stage would not have.

Reproducibility and penalty selection (W7/W9/Q5).

App. F.7 now lists the hyperparameters; the updated anonymized supplement has the CPG pipeline with configurations, λ grids, seeds and outputs, and a map from each CPG table to its command; the reproducibility statement now matches the release.

The added configurations help, but the supplied synthetic command still does not implement the claimed validation selection per method and seed and the README explicitly excludes sweep scripts. Reusing λ₂=λ₁ does not explain whether candidates were selected after stage 1 or after the complete pipeline. The selection procedure for the warm-started alternatives is also unclear. Please supply the selection runner, grid, validation criterion and selected values, including how continuation strengths were chosen. Thus the missing-code concern remains partly unresolved.

What is certified? (W1/Q3).

We do not claim that a minimizer of the stage-2 objective has its support inside the stage-1 support; CPG's final support is a subset of it by construction, since committed groups stay removed.

This clarifies the scope. My concern was whether the certificate justifies the selection or the final model, not whether committed removals remain removed. A stage-1 groupwise KKT check does not establish correct support selection or stage-2 optimality, and is well-known to require stringent assumptions for recovery in comparison to non-convex regularizers. Under COMMIT-ALL, removal does not depend on passing the check; the reported re-admission experiment leaves accuracy unchanged. Calling the certificate an audit is more accurate, but does not establish why it is central to the method's effectiveness. I struggle to see the value in having a piece of information that conveys no optimality property or actionable information for the solved objective, nor influences the procedure's default procedure.

Reviewer_pjFQOpenReview ↗
Rating
3
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes the second-order geometry of pruned groups in deep gating, where each disjoint structural group is parametrized as D−1D-1 shared scalar gates times a carrier vector so that weight decay induces a group ℓ2/D\ell_{2/D} penalty. The authors show a sharp depth dichotomy at the latent origin (all factors zero). At D=2D = 2, a zeroed group is a strict saddle in its own coordinates exactly when the group-lasso KKT condition ∥∇gL0∥2≤λ\|\nabla_g \mathcal{L}_0\|_2 \le \lambda fails (with ∇gL0\nabla_g \mathcal{L}_0 the unregularized loss gradient on collapsed weights), so under full-batch gradient descent on C2C^2 losses, deletions surviving to convergence carry a certificate checkable with one gradient evaluation—though D=2D = 2 shrinks kept weights. At D≥3D \ge 3, which shrinks survivors far less, every zeroed group is locally trapped regardless of the data.

Building on this and an exact, function-preserving re-lift between depths, the authors propose Certified Progressive Gating (CPG), consisting of five steps: (1) train at D=2D = 2 with a MAC-proportional per-group penalty, (2) check the certificate on near-zero groups, (3) permanently commit them while logging which pass, (4) re-lift survivors to D=3D = 3, and (5) compress further. Experiments verify the theory numerically and show that CPG achieves exact support recovery on synthetic data, attains the highest mean accuracy among 11 methods in five of six CIFAR-10/100 ResNet settings, matches a tuned DepGraph on CIFAR-10 at about two-thirds of its budget, and leads six baselines on average on BERT-base, within noise.

Critical strengths
  1. A clean, exact depth dichotomy. Because the factorized weight wg=(∏d=1D−1cg,d)vg\boldsymbol{w}_g = (\prod_{d=1}^{D-1} c_{g,d})\boldsymbol{v}_g is homogeneous of degree DD, the data gradient enters the Hessian at a pruned group only when D=2D = 2. There, a deletion is a strict saddle exactly when the group-lasso KKT condition ∥∇gL0∥2≤λ\|\nabla_g \mathcal{L}_0\|_2 \le \lambda fails. At D≥3D \ge 3 the Hessian is 2λDI≻0\frac{2\lambda}{D}\boldsymbol{I} \succ 0 regardless of the data. Together with the closed-form trapping barrier and the exact re-lift between depths, this gives a coherent theory. Why consequential: it recasts depth from a continuous penalty knob into a discrete switch between data-driven selection and data-independent trapping. This explains why deep factorizations get stuck in spurious zeros, and it is the paper's main conceptual contribution.

  2. Rigorous verification of the mechanism. The Hessian spectra, trapping barrier and gauge-attraction rate match the theory closely. Under full-batch GD on ResNet-18, none of 1,3661{,}366 removals at D=2D = 2 violate the KKT condition, against 1212–13%13\% at D≥3D \ge 3. On synthetic data, every D=2→Dmax⁡D = 2 \to D_{\max} schedule recovers the support exactly (F1=1.000F_1 = 1.000), which no fixed depth does. Why consequential: this shows that the progressive schedule fixes a real optimization failure of fixed-depth gating, rather than acting as one more hyperparameter.

Critical weaknesses

The certificate is a passive label without practical guarantees. In the default mode, all near-zero groups are removed whether or not they pass the certificate, and enforcing it hurts accuracy (ResNet-56: 0.8378→0.75210.8378 \to 0.7521). Under SGD only 55–44%44\% of removals pass, and on synthetic data 42%42\% of CPG's removals fail even though all of them are correct. The removals in the BERT comparison (Table 2) carry no certificate. Why consequential: "certified" sparsification is the core contribution, but the certificate neither guides pruning nor indicates whether a removal was correct.

Questions for authors
  1. What the certificate means in practice.

    • In Table 9, are the 4242\\% "violating" removals of CPG exactly its stage-2 removals? All of CPG's removals there are correct, so what decision should a practitioner draw from an "uncertified" label?
    • On CIFAR and BERT, what fraction of the final removals are certified stage-1 removals, as opposed to stage-2 or thresholded removals?
    • Why does reaching the latent origin on BERT (Table 8) cost about 10 points of SST-2 accuracy relative to Table 2?
  2. Group partition G\mathcal{G} and BatchNorm handling in ResNets (Appendix F.2): The gate overhead (0.053% of ResNet-18 parameters at D=4D=4, ~1,984 groups) suggests gating a subset of channels (e.g., interior block channels rather than coupled residual-stream channels). Could you clarify which Conv and BatchNorm parameters (γg,βg)(\gamma_g, \beta_g) belong to each group gg in ResNet-18/56 so that ωg=0\omega_g = 0 zeroes the channel (including BN shift) while keeping G\mathcal{G} disjoint, and confirm whether baselines (Network Slimming, Group lasso, DepGraph) prune over the same candidate structures or also prune residual-stream channels?

1 comment
AuthorsOpenReview ↗

Response to Reviewer pjFQ

Thank you for the positive assessment and the precise questions.

Weakness (the certificate as a passive label). The certificate audits each D=2 removal against the selection stage's own optimality condition at the moment the group is removed (§3.2). It is a record of the decision; the accuracy comes from the schedule it belongs to (new Table 3):

  • Under the landing protocol, from the same D=2 checkpoint, keeping the penalty in the D=3 stage beats warm start + recovery by +0.096 (ResNet-56 16×), +0.047 (ResNet-18 16×) and +0.099 (CIFAR-100), with t = 6.0–12.0.
  • Under the default (boundary) protocol on CIFAR-10 at 8×–16×, CPG beats a λ-searched fixed D=3 by +0.011 to +0.027.

We also tested enforcing it. Reviving failing groups at the switch gives the drop you cite (0.8378 → 0.7521, Table 7 (old 6)); re-admitting them inside stage 1 leaves accuracy unchanged, and 63–89% collapse again within two epochs (App. F.6). The default therefore records the certificate.

Q1a (the 42% in Table 12, old 9). Mostly, yes. Table 12 reports violations in the final model, from runs without a commitment step. To see when removals occurred, we additionally recorded decisions under Algorithm 1 (App. F.6):

  • 100% of the groups committed at the D=2 switch pass the test there;
  • 89% of the end-of-training violations are groups removed by the D=3 stage, which is uncertified by construction.

So "certified" means the removal agreed with the selection stage's optimality condition when it was made. "Uncertified" covers D=2 removals that fail the test, later-stage removals and budget-enforcing removals; it does not imply an incorrect removal, and by default calls for no action.

Q1b (share of final removals made and certified at D=2). In the BERT runs of Table 2, 100% (Q1c). On CIFAR-10, default boundary protocol (new Table 16):

R18 8×R18 16×R18 32×R56 8×R56 16×
final removals committed at the D=2 switch13%62%92%74%44%
of these, satisfying the certificate (training-objective gradient)87%98%—82%63%
final removals certified by the stored check (10k unaugmented images)5%45%84%28%11%
  • The two checks differ in gradient and denominator (definitions in Table 16).
  • Removals not committed at the D=2 switch are magnitude decisions that meet the MAC target.
  • Under the landing protocol the deep stage makes most further removals: at ResNet-56 16×, 627 at the switch, 305 in the deep stage, 13 by magnitude.

Q1c (BERT, Table 9 (old 8) vs Table 2). We identified and corrected an evaluation error in the submitted Table 2, which had evaluated unpruned model copies, and reran all six methods under a common protocol with assertions verifying the evaluated pruning state (new Table 2). The revision also introduces a saliency-targeted allocation of CPG's per-group penalty on transformers (§3.4, App. F.4).

  • Accuracy. With this allocation, CPG achieves the highest mean accuracy in all four settings, by 4.7 and 2.2 points over the best baseline at 4× (t = 7.0, 2.8).
  • Certified, lossless removals. Across the Table 2 CPG runs, all 310,173 removals satisfy the decision-time certificate. The removed groups are already at zero, so removing them changes the logits by < 6e-6.
  • Ablation (new Table 10). The allocation sets the accuracy: hard removal with the same ranking is level (|t| < 1). The D=2 stage adds that every removal is lossless and certified when it is made.

For the gap you noticed, new Table 11 changes one thing at a time on SST-2 at the in-place runs' compression (2.8×):

stepaccuracyΔ
in place, uniform penalty (Table 9, re-run)0.820—
pruning at the stage switch0.826+0.007
MAC-based per-group penalty0.863+0.036
third recovery epoch0.868+0.005
saliency-targeted penalty (as in Table 2)0.902+0.034

In these in-place runs, 99.9% of the removals pass the certificate. They score lower because their uniform penalty shrinks about 99% of the FFN gates to a scale of 5e-4, so most groups they keep are effectively inactive.

Q2 (groups in ResNets; App. F.1, F.6).

  • Each group is one scalar gate on an output channel of a residual block's first convolution, placed after BN and ReLU: 1,920 groups in ResNet-18 and 1,008 in ResNet-56. ResNet-18's 64 stem channels are also gated, but are never pruned to meet the target, and their removal is not credited toward the reported MAC reduction.
  • Because the gate follows the ReLU, ω_g = 0 silences the channel for every BN shift. The filter, BN (γ_g, β_g) and the matching input slice of the second convolution become removable, and the groups are disjoint.
  • Residual-stream channels are not pruned by CPG.
  • Slimming, group lasso, D-Gating and the element-wise controls prune the same block-internal candidates. DepGraph searches a larger set that includes the residual stream.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.