- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces Second-Order Delta Space Search (SODSS), a method that leverages training checkpoints of a neural network to compute estimates of second-order saliency metrics, which are then used for pruning and/or quantisation.
The authors first develop their theory, highlighting how their estimators work and how they are linked with existing theory, such as optimal brain damage saliency. They then run experiments to validate their theoretical results and compare SODSS to other pruning schemes.
Critical strengths
- S1: The theoretical results and derivations, once contextualised with their limitations, appear to be solid.
Critical weaknesses
- W1: Experimental results are not convincing. In Figure 1, it appears that unless averaged over many epochs, the estimated curvature is actually quite far from the ground truth for SGD. In Figure 2, “Regime labels are plausible in all runs” is a very approximate statement, especially considering that regime thresholds and corresponding weights are assigned arbitrarily by the authors. In Table 1, SODSS performs below baselines in most cases. Cases where it performs best on MNIST and CIFAR-10 are levels of pruning so aggressive that the resulting network is at an unusable performance level. Finally, all experiments are only run on very small models.
- W2: The significance of the paper is very limited. It is unclear what exactly the authors try to achieve. Given that the performance of SODSS is not consistently better than the baselines, the paper does not establish when the method would actually be useful. It is unclear which theoretical developments are actually relevant, for example Table 2 seems to indicate that all developments in Section 3.4 are actually not important. Many results become invalid when moving from Gradient Descent to SGD or Adam, and rely on either noise being averaged out across many epochs or on various adaptations being performed.
- W3: The paper is generally unclear for a human reader. First, the structure does not allow one to understand which elements are important, and to build intuition into how the method works. The authors keep dropping names and equations without properly defining them beforehand, or even at all. Some symbols ($s$, $c$) are reused to represent very different things. Many numbers are put into the algorithm (for example in Section 3.3) without any context or justification. As mentioned in W2, some developments are included in the method even though experimental results have highlighted their lack of practical relevance. The experiments’ setup is not detailed enough (What kind of size is the char-Transformer?). In the results tables, best results are not shown in bold, despite the caption saying so.
Questions for authors
- Q1: What is the practical advantage of SODSS? It is not clearly outperforming the other methods, is it easier to compute? Even if one has to recompute the intermediate metrics for every snapshot? In which cases should it be preferred over the alternatives?
2 comments
Response to the authors' rebuttal
Dear authors,
I thank you for your response and revision. Having examined the revised manuscript you submitted, as well as your general response, it appears that the revision substantially diverges from the original submission with new claims, new experiments, different results and a major re-write of the paper.
In line with the ICLR 2027 Author Guidelines stating that “Area chairs and reviewers reserve the right to ignore changes that are significantly different from the original paper”, I shall exercise that right and will not be updating my rating.
Best regards,
Reviewer P3kD
Thank you; the revision is largely a response to this review.
W1: experiments. You were right: under SGD the curvature estimate is dominated by a noise bias, which we now derive and confirm within 3% (Table 29); it is a diagnostic only. The regime thresholds were set by hand, are reported with their sensitivity, and are removed from the method. On ResNets (three seeds) magnitude beats our score, and the paper says so.
W2: when to use it. Section 6 now answers concretely:
Releasing a small LM: keep one running sum of squared gradients, or a few optimiser states; it gave the best data-free masks and layer budgets on our GPT at 70–80% sparsity. Optimiser states, no data: prune with a global threshold on the stored states (65.2 vs 76.6 for the Squisher at 70%). Calibration data, high sparsity: give Wanda or SparseGPT the path's layer budget (106 → 85.0 at 80%). SGD-trained CNNs: magnitude. The gains are one-shot: fine-tuning evens them out at 70–80%, though at 90% the budget still helps (SparseGPT 91.1 → 74.6).
W3: clarity. Rewritten: method first, notation in one table with the clashes removed (Appendix A), model sizes in Table 4, and bold face restored (a font fallback had dropped it).
If the revision addresses your three weaknesses, we would be grateful if you would consider raising your score.