-
Comprehensibility. The paper is difficult to read, and the difficulty mainly stems from the vocabulary used. The paper introduces many idiosyncratic terms, uses several in more than one (often non-standard) sense (“boundary”, “landing”, “ledger” and “cell” all have several meanings), and defines some only in the appendix or after first use. In these cases, a reader must reconstruct the method from captions, other parts of the paper, or appendix content (see W6).
-
Settings and procedures. The description of the procedure is incomplete and inconsistent. Algorithm 1 omits the “settle”, the certificate slack, the landing and boundary/landing protocols, “gated freezing” and revival, but includes a STRICT mode that the code does not implement and the paper does not consider. The choice of stage-wise strengths λ1=λ2 has, to my knowledge, no theoretical justification. Further, it is not clear whether AdamW’s decoupled weight decay was erroneously used as the regularization, and the setting “gate off” is defined as “CPG, uniform λ, w/o settle”, yet Table 6 reports different results for the two (see W6–W8, W14).
-
Experiments. The empirical gains are rather small, but to the extent they exist, they could be attributable to the warm start, mid-training commitment and MAC-weighted penalties rather than to certification. Some baselines are named after different papers but in fact are the same element-wise family at different depths, and it is unclear how their unstructured sparsity is aggregated into structured sparsity. For details and other issues, see W10, W13, W16, W17.
-
Code and reproducibility. In the provided code, the CNN pipeline stage 2 trains without any penalty, although the paper states λ2=λ1, so the deep stage never compresses. Instead, compression comes from magnitude pruning to the MAC target at the boundary. The supplementary material does not contain what the reproducibility statement entails: model implementations, hyperparameter sweeps, and the configurations of the reported runs are missing, so most tables cannot be faithfully reproduced. My reproduction of the synthetic benchmark matches the CPG rows but not the fixed-depth baselines, and even the README’s own demo output differs from the paper (see W4, W9, W10).
-
W1. The result’s name “Certified pruning at depth two” overplays the role of the group-wise KKT conditions. A “certified” zero group is merely certified to be a first-order optimal point of the block-coordinate, single-group effective L2,1 regularized subproblem. Why partial fulfillment of the group-wise L2,1 first-order condition would be a useful criterion for pruning is not sufficiently motivated, especially since the L2,1 penalty is known to suffer from suboptimal selection properties and parameter shrinkage compared to L2,q with q<1 [6]. Using optimality conditions of this method as “certified pruning” is not convincing. The provided reasoning against fixed D>2 is that the loss Hessian degenerates at balanced zero groups, implying that the regularizer’s positive definiteness ensures local minimality of a (balanced) zero group irrespective of the loss gradient – a well-known fact in sparse regularization and differentiable factorization methods [7, 8, 9]. Hence, the certificate provides no additional information on whether that zero group is a good or bad local solution to the D=3 stage-2 objective, as it is an optimality condition for a different objective. If the assumptions of Cor. 1 held, all balanced zero groups would be “certified”, and its use would be limited mainly to checking convergence for D=2 (q=1).
-
For D=2 and initialization according to Cor. 1, the certificate can be employed as a convergence indicator of the underlying L2,1 regularized objective.
-
With balanced initialization (Cor. 1 does not apply) and a stationary stage 1, an uncertified group at a balanced zero is a strict saddle of the D=2 factorized objective that gradient descent cannot leave due to the absorbing state under (S)GD. Re-initializing these groups and continuing at D=2 until every candidate passes (similar to an active-set loop) could constitute a use case in which the certificate decides something.
-
W2. Cor. 1 requires the initialization to be absolutely continuous w.r.t. the Lebesgue measure on Uc, which the proposed method fails to satisfy by using a balanced (null set) initialization constructed from a standard initialization for the effective parameter group. Therefore, the result might not apply to balanced initializations or SGD instead of GD, both of which are used repeatedly in theoretical results and experiments. While this does not invalidate the result, it makes it inapplicable. For scalar groups of size 1, a balanced initialization with the wrong effective sign at initialization cannot change sign, and it converges to the latent origin, a strict saddle that violates the D=2 KKT condition [10].
-
W3. Corollaries 1 and 3 have mutually exclusive initialization assumptions. Cor. 1 requires an initialization that is absolutely continuous on Uc, whereas Cor. 3 assumes initialization on the balanced manifold [p. 19, l. 1023] (see W2). The method follows Cor. 3: Appendix E.2 starts stage 1 on the balanced manifold so that Lemma 1 holds at every stage [p. 24, l. 1298–1300], and the re-lift of Proposition 2 returns a balanced point [p. 6, l. 281–282]. Cor. 1 therefore does not cover any CPG run by design.
-
W4. Stage 2 uses no penalty in the code; the paper trains it with the Dmax penalty. In the CNN pipeline the deep stage trains without any penalty, so the depth Dmax compression never happens. Compression mainly comes from magnitude pruning to the MAC target at the boundary.
-
W5. The “trapping” barrier for D≥3 in Prop. 1 is derived for a loosely related objective, a single-group subproblem with all other parameters frozen and the loss assumed to be affine in that group. While this offers qualitative insights into initialization criteria for D=2 and D>2, the barrier is, similar to the “certificate”, an exact result applied to settings outside its appropriate scope. Without these restrictive assumptions and approximations, the result loses its sharpness and the usefulness of the exact analytical expression is limited.
-
W6. The paper is difficult to read, since it contains many non-standard terms, overloads their meaning, and defines some only in the appendix or after first use. The glossary is in Appendix C [p. 17, l. 866–870], not in the main text. Examples:
Other examples of non-standard, confusing, or undefined terminology:
-
“Cell” is used in the introduction and Figure 1 [p. 2, l. 065, 075] before Sec. 3.1 defines it as an element of the partition [p. 3, l. 124–125]. It alternates with “group” and “unit”, and it also means an experimental setting: “the ten cells of Table 6” [p. 27, l. 1443], “every cell of Figure 3” [p. 27, l. 1452], “one seed per cell” [p. 32, l. 1719].
-
“Landing” names two things, neither of them defined:
-
a protocol: threshold at the end of training, then fine-tune [p. 6, l. 321–323]
-
the magnitude trim to the target inside the boundary protocol (“boundary landing” [p. 28, l. 1496]; Figure 6), which is not in Algorithm 1
-
Further non-standard or unclear terms are: certificate, ledger, commit (to what?), gated freezing (freezes nothing?), “the settle” (which are large-batch iterations toward the end of a stage), latent zero, majority-compatible operating point.
-
“gate off” is defined as “2→4, uniform penalty, no settle or certificate” in the caption of Fig. 3, so it should correspond to the setting “CPG, uniform λ, w/o settle” in Table 6. But the results in Table 6 differ between “gate off” and “CPG, uniform λ, w/o settle”.
-
The statement appearing twice that two factors “turn spurious stationary points into strict saddles” is incorrect. For the convex group lasso, the non-smooth base problem has no spurious stationary points that a factorization could transform into anything. The factorization creates spurious stationary points at zero groups. At D=2, these spurious stationary points are strict saddles [11].
-
W7. There is no theoretical justification for using λ1=λ2, since stages 1 and 2 pose different optimization problems with different scales and ranges for λ. Despite being computationally more intensive, tuning the second-stage λ2 conditional on the selected stage-1 support seems reasonable.
-
W8. Algorithm 1 does not sufficiently characterize the proposed method. The algorithm is missing the important components of a large-batch phase at the end of training (the “settle”), a certificate slack δ=0.05λg, a magnitude-pruning “landing protocol”, gated freezing, the “boundary protocol” (whose magnitude pruning at the stage switch is the “boundary landing”), and parameter revival. On the other hand, STRICT mode is included but not implemented in the code. Moreover, the supplied code is inconsistent with Alg. 1 in using a penalty-free stage 2 except for the synthetic application, despite the text stating λ1=λ2.
-
W9. The supplied code does not align with what the reproducibility statement says it contains.
-
Paper. “Source code for CPG, the baselines and the evaluation pipeline is provided as anonymized supplementary material”, “with every configuration in the released code” [p. 10, l. 525–528].
-
Appendix F.6 promises hyperparameters, but none are given for the λ grid, the settle, the bisection, the full-batch run or the BERT schedules. In the code, README Sec. 8 states: “Not included: baseline implementations, the hyper-parameter sweeps, the BERT-base experiments and the analysis scripts of the paper.” This directly contradicts the paper’s claim above.
-
Also missing: LeNet-5, ImageNet and the per-cell configuration files; Table 9 controls (proximal solvers etc.), gated freezing, the three-stage schedule, the λ values of any reported run. Hence, most of the tables cannot be reproduced from the supplied code.
-
W10. Synthetic benchmark, not reproduced (5 seeds). The D-Gating baselines of the synthetic experiment do not reproduce. The code reproduces the paper’s CPG rows exactly but not its fixed-depth baselines. Fixed D=3 recovers the support exactly on 4 of 5 seeds, and CPG differs from it only on the fifth. The README’s own demo output even gives fixed D=3 F1 1.0, identical to CPG.
-
Setup. The released synthetic fit has the following defaults: p=1000, G=100, s=10, n=160, λ=0.02, 12,000 steps, initialization 0.1⋅N(0,1), floor 10−3. These match the Table 9 caption [p. 33, l. 1728–1729]. Seeds 0–4.
-
CPG rows. The CPG rows reproduce Table 9 exactly.
-
CPG. Kept 10.0±0.0, F1 1.0±0.0, violation rate 0.420; the paper reports 0.420.
-
CPG, Dmax=4. F1 1.0, violation rate 0.196; the paper reports 0.196.
-
Fixed-depth baselines. The fixed-depth baseline results are inconsistent or do not reproduce. The submitted code gives a stronger fixed D=3 baseline than the paper reports. The README contradicts Table 9 too. Its expected output in the README lists fixed D=3 at kept 10.0, F1 1.0±0.0, rel. error 0.019, violation 0.374, identical to CPG. Seeds 0–2 of this run reproduce that table exactly (fixed D=2 kept 42.0, F1 0.387). My reproduction calls the script unchanged, with the demo’s seed loop; only the seed count differs.
-
Consequence. Table 1’s “the only one that recovers the synthetic support exactly” [p. 7, l. 359] therefore rests on the seed configuration/count in the supplied code.
-
W11. The theoretical results mostly build upon/apply existing results with minor modifications. In parts, the paper suggests an overstatement of its own contributions: the depth dependence of the geometry at zero is known. The paper says that “to our knowledge none characterizes how the geometry at a zero depends on depth” [p. 3, l. 115–117] and that the second-order classification is “not established in prior work” [p. 14, l. 749–750]. Ziyin, Li and Meng [12] show that weight decay creates bad minima at zero in networks with more than one hidden layer, unlike networks with one. That is the D=2 versus D≥3 boundary, with the same mechanism: with three or more factors, the data term has no second-order contribution at the origin. They also discuss local minimum traps and establish numerical approximations of the trap radius. Kawaguchi [13] shows that deep linear networks with more than three layers have saddles whose Hessian has no negative eigenvalue. More directly, Kolb et al. [7], which is cited in the submission, already contains the group-structured version for general smooth losses. The submission cites it with the observation “that for depth k>2 the zero state acquires positive curvature” [p. 14, l. 741], which is inconsistent with its own novelty sentences.
-
W12. The paper (silently by omission of limitations) overstates conditional, block-local, frozen-model, or linearized model results as general properties:
-
“a run that comes close to a zero is drawn into it whether or not the data would have kept the cell” [p. 4, l. 198]: in the frozen affine model a cell inside its data-dependent barrier returns to zero; outside it escapes. But the radius itself depends on the loss gradient, so it is not fully data-independent.
-
“the only method that records which of its deletions satisfy their optimality condition” [p. 1, l. 028–029]: the condition is the group-wise KKT condition for L2,1 regularization, which is an unconvincing concept of optimality.
-
W13. The baselines are named slightly confusingly with different method names and sources, but run as one method family. Table 1 lists “spred”, “DWF” and “Hadamard” [p. 6, l. 317–318]. Table 9 states “Element-wise D=2 (spred-like)”, “D=3 (DWF-like)” and “D=4 (Hadamard-like)” [p. 33, l. 1750–1752]. They are therefore one element-wise factorization run at depths 2, 3 and 4, not three published methods (which would differ in, e.g., their initialization).
- All three papers study multiplicative factorizations of the parameters. Kolb et al. [7] treat several parametrizations at any depth k, including grouped ones (GHPPk), so “Hadamard” does not uniquely identify a method. The related-work table reduces that paper to “element-wise, k factors” [p. 16, l. 860].
-
W14. AdamW is used, but only coupled weight decay equals the required L2 regularization of the differentiable D-Gating objective. It is not clear if the decoupled weight decay in AdamW was used as the regularization strength or if an explicit L2 penalty was added on top of decoupled weight decay.
-
W15. The purpose and presentation of D.9 and Theorem 3 are not clear to me. The second half of the Theorem statement should be plain text below the theorem. The subsequent proof sketch is too vague and informal.
-
W16. It is unclear whether the synthetic experiment compares fair, tuned baselines. Table 9 runs “one λ for every row” [p. 33, l. 1729]. Table 1 repeats these numbers under the caption “strengths are selected on validation” [p. 7, l. 326]. It remains unclear if the wording in Table 9 means that the same λ was used across models or tuned separately. In the former case, the claim that CPG alone recovers the support exactly would be weak without separate per-method tuning.
-
W17. Some comparison methods (spred, DWF, Hadamard and Powerpropagation) are for unstructured (parameter-wise) sparsity and thus not a fair comparison for structured sparsification and compression targets. It is not documented how unstructured sparsity is converted into structured sparsity for these cases. This considerably weakens the experimental comparison, leaving fewer fair structured sparsity competitors.
-
[6] Y. Hu, C. Li, K. Meng, J. Qin, X. Yang. Group sparse optimization via ℓp,q regularization. Journal of Machine Learning Research 18(30):1–52, 2017.
-
[7] C. Kolb, C. L. Müller, B. Bischl, D. Rügamer. Smoothing the edges: smooth optimization for sparse regularization using Hadamard overparametrization. Machine Learning 115:87, 2026.
-
[8] P.-L. Loh, M. J. Wainwright. Regularized M-estimators with nonconvexity: statistical and algorithmic theory for local optima. Journal of Machine Learning Research 16:559–616, 2015.
-
[9] B. D. Haeffele, R. Vidal. Global optimality in neural network training. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7331–7339, 2017.
-
[10] T. Jacobs, R. Burkholz. Mask in the mirror: implicit sparsification. International Conference on Learning Representations (ICLR), 2025.
-
[11] C. Poon, G. Peyré. Smooth bilevel programming for sparse regularization. Advances in Neural Information Processing Systems (NeurIPS) 34:1543–1555, 2021.
-
[12] L. Ziyin, B. Li, X. Meng. Exact solutions of a deep linear network. Advances in Neural Information Processing Systems (NeurIPS) 35, 2022.
-
[13] K. Kawaguchi. Deep learning without poor local minima. Advances in Neural Information Processing Systems (NeurIPS) 29, 2016.
Response to Reviewer CghJ
Thank you for the careful questions. We answer each weakness together with its question.
W1 / Q1 (clarity of the practical algorithm; a concrete run). We agree the procedure was spread across the text. §3.4 now states it in one place, with a Terms paragraph and a one-run example (App. E.2), and Algorithm 1 is restated as run. Two points you raised:
One run on CIFAR-10 ResNet-56, target 8× fewer MACs, default (boundary) protocol:
Steps 1, 2 and 4 are common to both protocols, which differ only in where the MAC target is enforced. The landing protocol skips step 3 and keeps the penalty in step 5, so the D=3 stage removes further channels and the target is met by magnitude at the end (ResNet-56 16×: 627 removals at the switch, 305 in the deep stage, 13 at the end).
W2 / Q2 (practical interpretation of the certificate).
What the results establish about its practical value (Table 16, App. F.2, F.4):
Re-admitting failing channels gave no measurable accuracy benefit, so the label is recorded, not enforced. The accuracy comes from the schedule (Q3).
Q3 (which control isolates starting at D=2). The most direct one is fixed depth 3: the same per-group penalty, budget, protocol and λ search, without the D=2 start (new Table 3).
Penalty and compute are also controlled: Network Slimming with the same MAC-weighted penalty stays behind by 0.015 to 0.052 (Table 15), and on DepGraph's own budget CPG leads by +0.021 and +0.020 at 8× (Table 14).
Connection to the geometry. At D=2 a zero group that violates the optimality condition has a negative-curvature direction, so training can still undo a removal the data contest; at D≥3 its Hessian is positive definite, so whatever reaches zero stays. Starting at D=2 gives the deep stage a support the data could contest, consistent with the controls above.
On BERT, we corrected an evaluation error in the submitted Table 2 (it had evaluated unpruned model copies) and reran all six methods under one protocol; with the saliency-targeted penalty of the revision (§3.4), CPG has the highest mean accuracy in all four settings.