acceptodds
Under review as a conference paper at ICLR 2027

Second Order Delta Space Search (SODSS),

Abstract

Which weights of a trained network can be pruned or quantised is usu- ally decided from the loss landscape at convergence, using gradients on calibration data or extra Hessian-vector products. We show that the train- ing trajectory already contains this information and that it can be read from checkpoints alone. Along a gradient-descent path the second differ- ence of the parameters satisfies an exact identity, at =−η Htvt−1, with a path-averaged Hessian. We characterise what this gives: the curvature the optimiser experiences along its own path (a Rayleigh quotient), glob- ally and per block, with a bias under minibatch noise bounded by 1/(η∆) that vanishes with the checkpoint stride ∆; and what it does not: the Hessian diagonal per parameter, which we show is not identifiable. Second- Order Delta-Space Saliency (SODSS) builds on this. It detects second- order events in parameter, representation and token-embedding space with a scale-free statistic (the second difference of log-speed), segments training into temporal regimes, and scores each weight by a regime-weighted path- Fisher saliency θ2 N,j t wtv2 t,j , which we prove is an Optimal-Brain-Damage saliency with the empirical Fisher averaged over the path and which reduces to Synaptic Intelligence without the θ2 factor. The score needs no gradients, data or Hessian products and is computed in one streaming pass with O(d) memory. On SGD-trained networks (an MLP and a CNN) one-shot global pruning with SODSS is within a point of magnitude pruning up to 95% sparsity, beats gradient-based Fisher and SNIP criteria throughout, and is clearly best at 98%; the event detector recovers learning-rate schedule changes and representational phase changes without being told the sched- ule. Under AdamW the raw score fails, because the premise vt ∝gt does not hold; undoing the preconditioner with the second-moment state that checkpoints already contain restores it, and the corrected score then beats magnitude pruning on an AdamW-trained Transformer at every sparsity. We also report what does not work: per-parameter second-order statistics do not help, and regime weighting has no measurable effect.

Primary area:
foundation or frontier models, including LLMs
Keywords:
knowledge distillation, model compression, neural architecture search, training dynamics, second-order optimization, Hessian-based pruning, quantization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_P3kDOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Second-Order Delta Space Search (SODSS), a method that leverages training checkpoints of a neural network to compute estimates of second-order saliency metrics, which are then used for pruning and/or quantisation.

The authors first develop their theory, highlighting how their estimators work and how they are linked with existing theory, such as optimal brain damage saliency. They then run experiments to validate their theoretical results and compare SODSS to other pruning schemes.

Critical strengths
  1. S1: The theoretical results and derivations, once contextualised with their limitations, appear to be solid.
Critical weaknesses
  1. W1: Experimental results are not convincing. In Figure 1, it appears that unless averaged over many epochs, the estimated curvature is actually quite far from the ground truth for SGD. In Figure 2, “Regime labels are plausible in all runs” is a very approximate statement, especially considering that regime thresholds and corresponding weights are assigned arbitrarily by the authors. In Table 1, SODSS performs below baselines in most cases. Cases where it performs best on MNIST and CIFAR-10 are levels of pruning so aggressive that the resulting network is at an unusable performance level. Finally, all experiments are only run on very small models.
  2. W2: The significance of the paper is very limited. It is unclear what exactly the authors try to achieve. Given that the performance of SODSS is not consistently better than the baselines, the paper does not establish when the method would actually be useful. It is unclear which theoretical developments are actually relevant, for example Table 2 seems to indicate that all developments in Section 3.4 are actually not important. Many results become invalid when moving from Gradient Descent to SGD or Adam, and rely on either noise being averaged out across many epochs or on various adaptations being performed.
  3. W3: The paper is generally unclear for a human reader. First, the structure does not allow one to understand which elements are important, and to build intuition into how the method works. The authors keep dropping names and equations without properly defining them beforehand, or even at all. Some symbols ($s$, $c$) are reused to represent very different things. Many numbers are put into the algorithm (for example in Section 3.3) without any context or justification. As mentioned in W2, some developments are included in the method even though experimental results have highlighted their lack of practical relevance. The experiments’ setup is not detailed enough (What kind of size is the char-Transformer?). In the results tables, best results are not shown in bold, despite the caption saying so.
Questions for authors
  1. Q1: What is the practical advantage of SODSS? It is not clearly outperforming the other methods, is it easier to compute? Even if one has to recompute the intermediate metrics for every snapshot? In which cases should it be preferred over the alternatives?
2 comments

Thank you; the revision is largely a response to this review.

W1: experiments. You were right: under SGD the curvature estimate is dominated by a noise bias, which we now derive and confirm within 3% (Table 29); it is a diagnostic only. The regime thresholds were set by hand, are reported with their sensitivity, and are removed from the method. On ResNets (three seeds) magnitude beats our score, and the paper says so.

W2: when to use it. Section 6 now answers concretely:

Releasing a small LM: keep one running sum of squared gradients, or a few optimiser states; it gave the best data-free masks and layer budgets on our GPT at 70–80% sparsity. Optimiser states, no data: prune with a global threshold on the stored states (65.2 vs 76.6 for the Squisher at 70%). Calibration data, high sparsity: give Wanda or SparseGPT the path's layer budget (106 → 85.0 at 80%). SGD-trained CNNs: magnitude. The gains are one-shot: fine-tuning evens them out at 70–80%, though at 90% the budget still helps (SparseGPT 91.1 → 74.6).

W3: clarity. Rewritten: method first, notation in one table with the clashes removed (Appendix A), model sizes in Table 4, and bold face restored (a font fallback had dropped it).

If the revision addresses your three weaknesses, we would be grateful if you would consider raising your score.

Reviewer_P3kDOpenReview ↗

Response to the authors' rebuttal

Dear authors,

I thank you for your response and revision. Having examined the revised manuscript you submitted, as well as your general response, it appears that the revision substantially diverges from the original submission with new claims, new experiments, different results and a major re-write of the paper.

In line with the ICLR 2027 Author Guidelines stating that “Area chairs and reviewers reserve the right to ignore changes that are significantly different from the original paper”, I shall exercise that right and will not be updating my rating.

Best regards,

Reviewer P3kD

Reviewer_fKwiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper makes an observation about checkpoints: first- and second-order differences contain information about the optimization trajectory. This allows us to quantize and prune models.

Critical strengths

The strengths lie in the analysis of the second-order differences to give us a very nice measure of 'saliency' (very roughly how important is each parameter). Table 1 shows that the method has some promise in the 'one-shot' pruning setting.

Critical weaknesses

Experiments are too small scale to know if the method is useful in practice and can e.g. help with quantization. Additionally, no true 'quantization' is done.

We are also missing a comparison with cheap methods but 'multi-shot' methods that require e.g. extra data to do pruning.

Questions for authors
  • Can you re-run experiments on e.g. nanochat to show benefits of the method? Some public LLM runs may also have checkpoint information for you to use.
  • How did you decide on hyperparameters for the temporal regimes?
  • should the section title in 3.2 be 'What do second-order differences of a trajectory measure?'?
  • Can you maybe include a figure to show visual examples of the different temporal regimes?
1 comment

Thank you, including for checking our code against the tables. Both weaknesses were right, and the first led to the main result.

W1: relation to the Squisher and temperature pruning. Proposition 2 makes the relation exact: the stored accumulators tile the path and the Squisher is the last tile. Compared directly, with a global threshold the path sum is better in all three seeds at 50–80% sparsity (65.2 vs 76.6 at 70%); under a uniform budget neither is consistently better, so the gain is mostly between layers (Table 17). On Pythia the comparison improves with model size, and the path wins on Pythia-410M (Table 18). Our original SODSS-A does not beat the Squisher, and we say so. Under SGD there is no accumulator to reuse; there the checkpoint score is level with or ahead of the data-based Fisher (Table 5). Temperature pruning needs 200 extra epochs with data and scores input features; ours scores weights from the original run's checkpoints (Table 3).

W2: exact vs approximate. Proposition 4 is now an exact decomposition of the implemented Adam correction, with its three approximations named and measured (Table 23; rank correlation 0.997 with the exact first-moment sums). "Restores the premise" is gone. SODSS-V and SODSS-R need no approximation: Proposition 2 and Proposition 3 are identities, the latter checked numerically to 1e-15.

Curvature validation. Corrected on all three points (full objective, all parameters, segment-integrated Hessian); agreement is below 1e-10 at stride 1, and the wrong "within 1.5%" claim for blocks is fixed (Appendix F).

Numbers. code/check_claims.py tests each comparative claim against the raw result files in the supplement.

If this answers both weaknesses, we would be grateful if you would consider raising your score.

Reviewer_1yDjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how stored training trajectories can provide parameter-importance estimates for model compression without additional gradient computations or calibration data at analysis time. Its main saliency score combines squared final weights with accumulated squared displacements between checkpoints:

sj=θN,j2∑twt(θt,j−θt−1,j)2.s_j=\theta_{N,j}^{2}\sum_t w_t(\theta_{t,j}-\theta_{t-1,j})^{2}.

The method identifies events and regimes from changes in parameter and representation trajectories and uses them to determine the temporal weights (w_t). The paper also relates trajectory differences to curvature and connects the saliency score to gradient-based importance measures under specified assumptions.

Experiments on an MNIST MLP, a CIFAR-10 CNN, and a character-level Transformer show improvements in several high-sparsity, one-shot pruning settings. The raw score performs poorly under AdamW, while a correction using optimizer second-moment information substantially improves its performance. The paper also reports that regime weighting contributes little to pruning accuracy. The supplementary material provides the scoring implementation, experiment scripts, and raw results.

Critical strengths

S1. The problem setting is practically meaningful.

Estimating parameter importance from previously saved checkpoints is useful when training data and gradients are no longer accessible. In particular, the basic unweighted SGD score can be computed from weight checkpoints alone, giving it a distinct information requirement from methods that rely on saved adaptive-optimizer statistics. Although the full regime-aware pipeline and the Adam-corrected variant require additional recorded information, the basic setting offers a useful way to exploit training records that may otherwise remain unused.

S2. The experiments provide positive evidence for high-sparsity, one-shot pruning.

The results in Table 1 show several substantial improvements. At 98% sparsity on MNIST, SODSS achieves (65.6$\pm$4.8%) accuracy versus (48.1$\pm$11.6%) for magnitude pruning, with an improvement in each of the three seeds. On CIFAR-10 at 95% sparsity, the corresponding accuracies are 51.2% and 38.6%. On the character-level LM at 80% sparsity, the Adam-corrected score reduces one-shot loss from 3.410 for magnitude pruning to 2.735. These observations support the usefulness of the score in the tested settings. The single-seed CIFAR-10 and LM results limit confidence in broader generalization, but the reported improvements remain meaningful evidence.

S3. The supplementary material supports verification, and the paper reports informative negative results.

The code and raw JSON outputs allow the main numerical results and comparison procedures to be checked. Within each experiment, the methods use common dense models, matched global sparsities, and controlled fine-tuning procedures. The paper also reports the failure of the raw AdamW score, the small contribution of regime weighting, and additional stride and momentum results. These findings help identify the method’s effective scope. I checked the consistency of the relevant implementation, raw results, and reported tables, although I did not independently rerun model training.

Critical weaknesses

W1. The additional contribution relative to closely related importance-estimation methods is not yet sufficiently distinguished.

Squisher [1] already reuses an optimizer’s squared-gradient accumulator as a Fisher approximation and combines it with squared weights for pruning. Temperature-based pruning [2] uses squared parameter displacement as an importance signal in a different, structured-pruning setting. These methods overlap with the broader ideas of reusing training-history statistics and interpreting squared motion as parameter importance.

SODSS has real differences: it uses temporally sparse weight checkpoints, supports an SGD setting without adaptive-optimizer state, and includes trajectory diagnostics. However, the paper should more clearly establish what these differences allow us to conclude beyond the existing approaches and under which information constraints they provide an important advantage.

This distinction is particularly relevant to the Adam-corrected variant, which itself requires optimizer state. Its improvement over magnitude pruning supports its empirical usefulness, but does not isolate the additional value of checkpoint history relative to directly reusing optimizer statistics. Moreover, the ablation in Table 2 shows that regime weighting contributes little to pruning accuracy. This is an informative negative result, but it means that the diagnostic contribution of regime analysis should be distinguished from the effectiveness of the underlying saliency score.

The concern is therefore the scope and significance of the remaining contribution. A concise comparison of the required inputs, computed statistics, and conclusions supported by the existing results would substantially strengthen the paper’s positioning.

W2. Some theoretical interpretations extend beyond what the implemented statistics directly support.

The full-batch GD identity is valid under its stated assumptions, the noise analysis is explicitly approximate, and the connection to OBD in Proposition 3 includes the additional (H\approx\bar F) assumption. My concern is the extension of these relationships to the practical implementation.

For the Adam-corrected score, the supplementary function trajectory_scores multiplies the displacement over an entire checkpoint interval by the Adam denominator at the interval’s endpoint. In the LM experiment, checkpoints are saved every ten steps. This operation does not separately account for changes in the denominator within the interval, remove the AdamW weight-decay contribution, or convert the first moment into the instantaneous gradient. The description of this correction as restoring the original premise therefore requires explicit approximations. Exact gradient reconstruction is not necessary for a useful saliency score, and the reported improvement remains positive evidence; the theoretical interpretation should match the computation.

The curvature-validation discussion in Section 4.1 also needs qualification. The supplied HVP comparison uses a different data subset from the training objective, restores only the recorded weight parameters, and evaluates a checkpoint Hessian rather than the path-integrated Hessian in the exact identity. It therefore supports approximate empirical agreement. The claim of agreement within 1.5% across all tested strides and layer blocks should also be narrowed to match the raw results.

These concerns affect the theoretical interpretation and diagnostic contribution. They do not invalidate the main pruning results, since the curvature estimator is not directly used in the principal saliency score.

References

[1] Li et al. Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator. ICML, 2025.

[2] Lapenna et al. Temperature-based pruning for input features in Graph Neural Networks. The European Physical Journal Plus, 140, Article 880, 2025.

Questions for authors

Q1. What is the central contribution relative to Squisher and temperature-based pruning under clearly specified information constraints?

Please distinguish the methods in terms of their required inputs and computed statistics, particularly separating the basic SGD score that can use weight checkpoints alone from the Adam-corrected score that requires optimizer state. Which observations already reported in the paper support the distinctive usefulness of these variants?

Q2. Which relationships remain exact, and which require approximation, for the implemented Adam correction?

Under what assumptions can multiplying an interval’s displacement by the endpoint Adam denominator be interpreted as restoring the premise used to motivate the saliency? Please distinguish the exact SGD relationship from the claims supported theoretically or empirically for the implemented Adam-corrected score, accounting for the roles of denominator variation, weight decay, and first-moment averaging.

1 comment

Thank you, including for checking our code against the tables. Both weaknesses were right, and the first led to the main result.

W1: relation to the Squisher and temperature pruning. Proposition 2 makes the relation exact: the stored accumulators tile the path and the Squisher is the last tile. Compared directly, with a global threshold the path sum is better in all three seeds at 50–80% sparsity (65.2 vs 76.6 at 70%); under a uniform budget neither is consistently better, so the gain is mostly between layers (Table 17). On Pythia the comparison improves with model size, and the path wins on Pythia-410M (Table 18). Our original SODSS-A does not beat the Squisher, and we say so. Under SGD there is no accumulator to reuse; there the checkpoint score is level with or ahead of the data-based Fisher (Table 5). Temperature pruning needs 200 extra epochs with data and scores input features; ours scores weights from the original run's checkpoints (Table 3).

W2: exact vs approximate. Proposition 4 is now an exact decomposition of the implemented Adam correction, with its three approximations named and measured (Table 23; rank correlation 0.997 with the exact first-moment sums). "Restores the premise" is gone. SODSS-V and SODSS-R need no approximation: Proposition 2 and Proposition 3 are identities, the latter checked numerically to 1e-15.

Curvature validation. Corrected on all three points (full objective, all parameters, segment-integrated Hessian); agreement is below 1e-10 at stride 1, and the wrong "within 1.5%" claim for blocks is fixed (Appendix F).

Numbers. code/check_claims.py tests each comparative claim against the raw result files in the supplement.

If this answers both weaknesses, we would be grateful if you would consider raising your score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank the reviewers. We ran the larger experiments, baselines and rewrite you asked for. They did not support the original headline but led to a stronger result, and the revised paper is built around it (new title: "Second-Order Delta-Space Saliency: What a Training Run Leaves Behind for Compressing Small Language Models").

Main result. Compression methods read weight importance at the end of training, but a run leaves a record of its whole path, checkpoints and stored optimiser states, which releases such as Pythia publish. The revision shows exactly what that record contains and that it is a better guide to pruning.

  • Exact statements. Under SGD checkpoint differences are accumulated gradients (Proposition 1); stored Adam second moments tile the path, and the Squisher is its last tile (Proposition 2); under AdamW the final weights are a known discounted sum of all updates (Proposition 3), so weight decay sets which part of the path the model remembers.
  • Public checkpoints. On Pythia-410M, trained by others and run once with a fixed protocol, the path score beats the end point at every sparsity tested (45.8 vs 50.3 at 50%); across Pythia sizes it goes from behind (70M) to ahead (410M) (Figure 2).
  • GPT (51M). A global threshold on the stored states lowers one-shot perplexity at 70% sparsity from 76.3 to 65.6 (both seeds), and to 63.0 with the weight-decay weighting (SODSS-R, no free constant), which also beats the end point on held-out Pythia-410M (Table 20).
  • A layer budget without data. SparseGPT at 80%: 105 → 86.6, better than AlphaPruning (89.8) and OWL (104); it still helps after fine-tuning at 90%.
  • Practical. One running sum of squared gradients kept during training gave the best data-free masks and budgets on our main GPT runs at 70–80% sparsity.

What did not survive. Regime weighting (removed); the advantage over magnitude pruning on ResNets (magnitude wins, though a coarser checkpoint stride closes most of the gap); per-weight curvature from second-order differences (dominated by a noise bias that we now derive and confirm within 3%).

Limits. The gains are one-shot gains at high sparsity: a short fine-tune evens them out at 70–80%. AlphaPruning is the better budget on the GPT at 50–70%. SODSS-V and SODSS-R were developed during the revision; Appendix D states which experiments were designed afterwards, and Pythia-410M is the held-out test.

Verification. Every new number is generated from the raw results by a script, and code/check_claims.py (supplement) checks each comparative claim of the main text against them.

AuthorsOpenReview ↗

More expereme

Update: a third GPT seed. We added a third seed of the 51M GPT, trained after SODSS-R had been defined, and uploaded the updated PDF. Every statement in the paper holds over the three seeds, and the GPT tables now report means of three. Two results became stronger: SODSS-R now beats the Squisher in all 16 GPT cells, and from 70% sparsity on SparseGPT does better with the data-free running-sum budget than with its uniform budget (42.5 against 44.3 at 70%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.