- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies whether the terminal weight magnitudes used by Iterative Magnitude Pruning (IMP) can be predicted from initialization-side quantities alone, framing this as a "finite-time terminal-ranking problem." Under fixed-Jacobian (NTK-style linearized) squared-loss gradient descent, the authors derive a closed-form spectral-filter expression for the terminal weights after steps, and prove that applying IMP's magnitude-pruning rule to this closed form exactly reproduces both one-shot and iterative linearized IMP masks, given a positive top- boundary gap. They validate this exact correspondence at controlled and whole-network scale, show the predictor recovers data-dependent "teacher supports" in a correlated-feature synthetic task where static initialization-only pruning scores degrade, and test transfer to nonlinear sparse training (LeNet5, ResNet, VGG) via matched retraining protocols and repeated-IMP self-consistency diagnostics. They also show a matrix-free implementation that avoids materializing the Jacobian or Hessian.
Critical strengths
- S1. The paper derives a closed-form initialization-side predictor that reproduces IMP's mask exactly under matched fixed-Jacobian dynamics, through the full iterative discovery sequence. This is further validated at real, whole-network scale to near machine precision.
- S2. The paper includes an experiment demonstrating that residual information carries an important signal about which weights matter: specifically, they conduct a correlated-feature experiment using synthetic data that shows that static, initialization-only pruning scores degrade sharply as correlation increases, while their finite-time predictor recovers the true relevant features almost perfectly.
Critical weaknesses
- W1. The correlated-feature experiment doesn't include actual linearized IMP as a baseline, leaving the section's central claim not fully self-contained.
- W2. No computational cost comparison undercuts the paper's central practical motivation. The paper's framing is that predicting the mask at initialization avoids the need to train the network just to discover it, but no wall-clock/FLOP comparison is given between the predictor's pipeline (including the optional Lanczos acceleration) and simply running real train-prune-rewind IMP.
Questions for authors
Clarification Questions
- Q1. In Figure 2, why is actual linearized IMP not plotted directly alongside the predictor? Can you confirm the exact correspondence holds specifically at the highest correlation levels tested (), rather than relying on Section 5.1's validation under a different covariate structure?
- Q2. What is the actual computational cost (wall-clock or FLOPs) of the initialization-side prediction pipeline compared to simply running real IMP to obtain a mask? Does the Lanczos acceleration provide substantial gains in the large T regime?