acceptodds
Under review as a conference paper at ICLR 2027

Predicting IMP Masks at Initialization under Finite-Time Linearized Dynamics

Abstract

Iterative magnitude pruning (IMP) selects sparse masks from weights obtained after training, whereas pruning at initialization selects masks before weight training. We formulate this difference as a **finite-time terminal-ranking problem**: can the magnitudes used by IMP be determined from initialization-side quantities? Under fixed-Jacobian squared-loss gradient descent, the finite-time response specifies terminal weights from the initialization, data, learning rate, and training horizon. Applying the same magnitude-pruning rule recovers one-shot and iterative linearized IMP masks when the top- boundary is stable. We validate this correspondence in controlled and functional whole-network fixed-Jacobian models. On correlated-feature tasks, the predictor recovers teacher supports where the tested static initialization scores degrade, and finite-time ablations show that the converged solution can select different masks. Beyond the exact regime, initialization-side scores combined with architecture-based layer allocation produce competitive sparse retraining accuracy in several nonlinear LeNet and ResNet settings. Repeated IMP discoveries provide a reference for interpreting coordinate agreement under training randomness. A matrix-free implementation evaluates the predictor without explicit pseudoinverses. Together, these results connect posterior IMP selection to initialization-side trajectory prediction and distinguish exact mask recovery from functional transfer to nonlinear training.

Primary area:
learning theory
Keywords:
Sparse Neural Networks, Iterative Magnitude Pruning, Pruning at Initialization, Linearized Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AuGKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether the terminal weight magnitudes used by Iterative Magnitude Pruning (IMP) can be predicted from initialization-side quantities alone, framing this as a "finite-time terminal-ranking problem." Under fixed-Jacobian (NTK-style linearized) squared-loss gradient descent, the authors derive a closed-form spectral-filter expression for the terminal weights after TT steps, and prove that applying IMP's magnitude-pruning rule to this closed form exactly reproduces both one-shot and iterative linearized IMP masks, given a positive top-kk boundary gap. They validate this exact correspondence at controlled and whole-network scale, show the predictor recovers data-dependent "teacher supports" in a correlated-feature synthetic task where static initialization-only pruning scores degrade, and test transfer to nonlinear sparse training (LeNet5, ResNet, VGG) via matched retraining protocols and repeated-IMP self-consistency diagnostics. They also show a matrix-free implementation that avoids materializing the Jacobian or Hessian.

Critical strengths
  • S1. The paper derives a closed-form initialization-side predictor that reproduces IMP's mask exactly under matched fixed-Jacobian dynamics, through the full iterative discovery sequence. This is further validated at real, whole-network scale to near machine precision.
  • S2. The paper includes an experiment demonstrating that residual information carries an important signal about which weights matter: specifically, they conduct a correlated-feature experiment using synthetic data that shows that static, initialization-only pruning scores degrade sharply as correlation increases, while their finite-time predictor recovers the true relevant features almost perfectly.
Critical weaknesses
  • W1. The correlated-feature experiment doesn't include actual linearized IMP as a baseline, leaving the section's central claim not fully self-contained.
  • W2. No computational cost comparison undercuts the paper's central practical motivation. The paper's framing is that predicting the mask at initialization avoids the need to train the network just to discover it, but no wall-clock/FLOP comparison is given between the predictor's pipeline (including the optional Lanczos acceleration) and simply running real train-prune-rewind IMP.
Questions for authors

Clarification Questions

  • Q1. In Figure 2, why is actual linearized IMP not plotted directly alongside the predictor? Can you confirm the exact correspondence holds specifically at the highest correlation levels tested (ρ≈0.95\rho\approx0.95), rather than relying on Section 5.1's validation under a different covariate structure?
  • Q2. What is the actual computational cost (wall-clock or FLOPs) of the initialization-side prediction pipeline compared to simply running real IMP to obtain a mask? Does the Lanczos acceleration provide substantial gains in the large T regime?
Reviewer_BZBmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the mask selected by IMP after training can be predicted from initialization-side quantities alone. Under fixed-Jacobian squared-loss gradient descent, terminal weights have a closed-form spectral filter depending only on initialization, data, learning rate, and horizon. Applying IMP's top-k rule to these predicted weights yields an exact correspondence with linearized IMP masks (Proposition 1), provided the top-k boundary gap is positive. Validation covers three regimes: exact mask recovery in fixed-Jacobian MLP and ResNet models, teacher-support recovery in a synthetic correlated-feature task where SNIP/GraSP degrade, and transfer to nonlinear sparse training using ERK layer budgets plus the linearized scores on MNIST, CIFAR-10, and CIFAR-100 with LeNet, ResNet-20/32, and VGG11-BN variants.

Critical strengths
  1. Expressing iterative IMP mask selection as a finite-time spectral ranking problem, with an explicit boundary-gap condition for mask stability, is a clean formulation.
  2. Using repeated IMP runs (IMP-IMP overlap) as a reference for reading coordinate agreement is a sensible practice. The paper is also upfront that nonlinear results show functional transfer, not exact mask recovery.
  3. The matrix-free implementation (no explicit Jacobian or pseudoinverse) is numerically validated against explicit spectral evaluation, and the horizon/learning-rate ablations (Fig. 3a) show that finite-time dynamics change the selected mask.
Critical weaknesses
  1. IMP requires multiple full train-prune-rewind cycles, while modern PaI methods (ProsPr, NTK-SAP, DPaI) report competitive sparse masks at initialization at far lower compute. IMP is a strong but expensive reference, and the paper offers no justification for why reproducing its masks, rather than matching it cheaply with PaI, is the right goal. It does not show that understanding IMP mask structure helps improve any modern PaI method. The contribution could be reframed as purely theoretical, but even then the impact is limited without experiments beyond basic models/datasets (MNIST, CIFAR-10/100 on LeNet/ResNet-20/32/VGG11-BN) and without showing how this framework advances the field beyond IMP.

  2. Table 2, the paper's main empirical result, compares only the predictor against IMP. SNIP, GraSP, and SynFlow appear only in the synthetic AR(1) task. ProsPr, NTK-SAP, and DPaI, which the paper cites in related work, appear in no accuracy table. Showing the predictor matches IMP on MNIST and CIFAR-10 with LeNet5 and ResNet20/32 (architectures from 1998 to 2016) does not establish practical competitiveness.

  3. Section 5.3 reports an allocation-matched random-mask control on LeNet5/MNIST, where random masks reach 99.17/98.92/98.67%, equal to or above the predictor's 99.17/98.84/98.43. The paper itself states: "this control does not show a general accuracy advantage from ranking alone under matched retraining and fixed layerwise keep counts." This control is not run for ResNet20/32, VGG11-BN, or CIFAR-10/100, where the headline claims rest, so there is no evidence that the linearized scores contribute beyond ERK layer allocation.

  4. Table 3 shows predictor-IMP coordinate overlap of only 0.130 at 90% sparsity on ResNet20, essentially identical to the IMP-IMP overlap (0.131), i.e ~87% coordinate mismatch. The paper says this is contextualized by IMP-IMP self-consistency, but the title and abstract promise IMP mask prediction, and no chance-level or allocation-only overlap baseline is reported for ResNet20, so the reader cannot tell whether 0.130 is meaningfully above chance (a uniform random mask would overlap roughly 0.10 at 90% sparsity). In the nonlinear regime, the evidence for mask prediction is therefore weak.

  5. Linearization validity is never tested for the networks used. LeNet5 and ResNet20 trained for 50-200 epochs at learning rates of 0.05-0.1 are plausibly far from the fixed-Jacobian regime. The paper provides no measurement of Jacobian drift and no comparison of linearized vs. nonlinear IMP masks as a function of width or learning rate. This is the experiment the title implies, and its absence leaves the nonlinear transfer claim without a mechanism.

  6. Ranking and allocation are confounded in the accuracy comparison. The predictor uses ERK layer budgets, but the layer allocation used by the IMP reference is not stated (it is described as "whole-network" IMP). The predictor also uses squared loss, one-hot targets, 1,000 examples, and T = 50, whereas IMP uses cross-entropy on full data. Without an ERK-allocated IMP reference, accuracy differences cannot be attributed to the scoring.

  7. Overall, the practical benefit is never quantified. The potential advantage of the predictor is cost (T = 50 curvature products on 1,000 examples vs. repeated full training), but no wall-clock time or FLOPs are reported for the predictor, IMP, or standard PaI methods.

Questions for authors

See Critical Weaknesses

Reviewer_yMLpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work poses the finite-time terminal-ranking problem, as the discrepancy between pruning masks computed at initialization and those computed after training. The paper presents a trajectory-aware Unstructured Pruning at Initialization (PaI) scheme that, under the assumption of fixed-Jacobian squared loss gradient descent, predicts weight-magnitude-based saliency rankings of trained networks at initialization using only information available before training. The paper empirically demonstrates that the PaI masks generated by the proposed method match those generated by iterated magnitude pruning on simple nonlinear classifiers.

Critical strengths

S1. Novel Approach

The result in the paper that simple linearized (first-order) dynamics provide a sufficient approximation for estimating weight-magnitude saliencies at initialization is novel and interesting. The matrix-free evaluation, which efficiently computes PaI saliencies and avoids extensive computation, strengthens the novelty of the proposed algorithm.

Critical weaknesses

W1. Poor Framing

Model compression has moved past unstructured magnitude-based saliencies, and modern reconstruction-based structured techniques far surpass them in both statistical and inference performance [1, 2, 3]. Matching to IMP masks at initialization may therefore yield suboptimal performance compared to such methods.

W2. Poor Technical Contribution

The paper offers little substantive theoretical insight into the connection between the simplified linearized dynamics and IMP masks. Proposition 1 (Lines 230-234) is not stated rigorously with relevant assumptions, making it difficult to fully understand the stated contribution of this work.

Questions for authors
  1. Can the authors provide a rigorous restatement of Proposition 1? It is also unclear where the proof of this statement ends without a marker.

Nitpicks not affecting the overall assessment:

  • At several instances (e.g., Line 196, 231, 258, etc), equation references are broken: they state "Eq. equation 8" instead of either "Eq. 8" or "equation 8".
  • I gently disagree with the authors that the primary area of this work is "Learning Theory".

Score Justification

Given the short review period for this ICLR, I have less confidence than my level of familiarity with this field would suggest, as I may not have fully understood this work. My review is constrained by the new review guidelines of this ICLR, and I would be pleased to expand on the points raised in the review. I am eager to engage with the authors in the rebuttal period to improve their manuscript and revise my score.

[1] MoDeGPT: Modular Decomposition for Large Language Model Compression. Lin et al. ICLR 2025

[2] SliceGPT: Compress Large Language Models by Deleting Rows and Columns. Ashkboos et al. ICLR 2024

[3] Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks. Hoefler et al, 2021

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.