acceptodds
Under review as a conference paper at ICLR 2027

Pruning Close to Home: Distance from Initialization impacts Lottery Tickets

Abstract

The Lottery Ticket Hypothesis states that there exist sparse subnetworks (called 'winning' Lottery Tickets) within dense randomly initialized networks that, when trained under the same regime, achieve similar or better validation accuracy as the dense network. It has been shown that for larger networks and more complex datasets, these Lottery Tickets cannot be found in random initializations, but that they require lightly pretrained weights. In this paper, we aim to circumvent this pretraining phase by exploring techniques to improve the quality of the pruning masks that emerge during the Mask Search phase. We start by demonstrating that a simple choice of hyperparameters during the search can significantly impact the trainability of the sparse network, and that the trends that hold for searching are contradictory with the best hyperparameters for training. This leads us to explore learning distance as an explanation for this, which is influenced by hyperparameter selection, and demonstrate its impact via explicit regularization. We find that a lower learning distance in combination with limited sign flips during pruning results in sparse networks that are better trainable, and that after several pruning rounds there even emerge useful features as-is in the untrained sparse network. Finally, we combine these observations to significantly reduce the extraction cost of a highly sparse network.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Lottery Ticket Hypothesis, Pruning, Efficiency, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_BD8ZOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the problem of trainability of lottery tickets. It is a well studied problem in the Lottery Ticket and sparse training literature that although the Lottery Ticket Hypothesis claims the existence of sparse networks, training them from scratch with the Iterative Magnitude Pruning method is hard. And in most cases the sparse networks found by this method for relatively large models are underperformant. The paper tries to address this issue of the trainability of sparse lottery tickets.

They propose to modify the iterative prune - reinitialize - train part of the IMP algorithm, by adding a regularization term which forces the active parameters to stay close to initialization. Additionally, they find that using no momentum and a smaller batch size aids this phase of training.

While the paper addresses a longstanding trainability problem, the experimental results are underwhelming. Results are mostly focused on a ResNet34 for Tiny Imagenet, which is not sufficient to validate the strong claims of trainability made in the paper.

Critical strengths
  1. The paper addresses a longstanding problem of making sparse lottery tickets trainable. They identify distance from initialization as a regularizer to aid trainability. This is an interesting and relevant finding.

  2. The paper also finds that the hyperparameter settings for the iterative pruning part of IMP must be different from the final sparse training phase.

Critical weaknesses
  1. My main concern with this paper is that the empirical evaluation is limited to just one setting. While I understand that evaluating large transformers might be difficult for an expensive method like IMP, I believe multiple CNN and small language or vision transformer settings could have been used to further strengthen the paper's findings.

  2. In Table 5, as the regularization strength increases, the effect of weight decay first worsens performance before improving it. Why does large weight decay improve performance and what happens at λ=1e−2\lambda = 1e-2. Given the trend it is possible that the weight decay alone could match the proposed regularization at least up to 90% sparsity. The broader point here is that at lower sparsities weight decay and the closeness to init regularizer are comparable and both are unable to reach the performance of the original network (so not a lottery ticket). Hence the distinction between these two regularizers is not completely obvious and might need more investigation.

  3. In the hyperparameter search, the paper only considers momentum and batch size, but there is no justification for choosing these two and not the learning rate or weight decay for example.

  4. Table 8 is not a fair comparison with the dense baseline, all sparse networks are first iteratively trained and then a linear layer is learned on top of it if I understand correctly, but for the dense network, a linear layer is learned on top of a random network. How can the results be interpreted and compared to the dense net in the context. I believe the correct baseline must be a trained backbone, which is the frozen and a linear layer is learned on top of it.

Questions for authors
  1. How does the closeness to initialization regularizer work in combination with rewinding to a partially trained point? This should improve the final performance of the sparse network significantly but this does not seem to have been studied in the paper.

  2. The speedup in able 6 is not clear to me. Why is the 200 epoch version not trained with a regularizer and why is the accuracy so much worse compared to the 20 epoch version?

4 comments

Dear reviewer BD8Z,

We thank you for your thorough review. However, we feel that some weaknesses are mischaracterized, which might be due to content in the appendix which we didn’t stress enough in the main paper. This will be resolved in the new version. Please find a specific rebuttal below to the weaknesses and questions listed in the provided review.

Kind regards,

the authors

Weaknesses

  1. This was mentioned in a single line in sec 3.3, but other settings were already explored in the appendix. We will make this more explicit by mentioning the specific combinations as well, and add a hard reference. FYI, these were: ResNet-18 + CIFAR10, ResNet-18 + CIFAR100, VGG16 + CIFAR100, Swin + TinyImageNet, ResNet-50 + ImageNet100. We chose ResNet-34 + TinyImageNet in the main body due to its balanced complexity, allowing us to have in-depth experiments, but still be computationally feasible.

  2. At λ=10−2\lambda=10^{-2}, dense networks are overregularized, and converge to the origin, resulting in effectively random pruning. We have since calculate the resulting accuracies and will add them to the paper including a short explanation. Weight decay somewhat impacts learning distance, as randomly initialized weights are quite close to the origin (see the second part of Table 4), and the ‘Different targets’ paragraph of Sec 4.2.

  3. We do have a motivation in Sec 2.4, where we refer to the existing work linking these hyperparameters to generalization. Additionally, we do explore the weight decay metric (see Table 5, and Sec 4.2 the ‘Explicit regularization’ paragraph). We will make this more explicit in Sec 2.4, and also mention that we have more hyperparameter values, as well as learning rate scaling in Sec C of the appendix.

  4. Actually, in this table, all networks consist of random weights with an optional sparsity mask, which are then probed. We show that certain masks extract useful features from the random weights depending on the Phase II hyperparameters, so in this case the random network is justified. In the appendix F we have an additional experiment where we transfer frozen trained tickets / networks, where a trained dense network is the correct baseline.

Answers:

  1. The main starting point of this paper was to avoid the partially trained rewind point, as stated in the contributions, which means that this is something we have not explored in detail. In Figure 2 we do however show that pretraining similarly impacts learning distance. If time allows us, we will include results for this in the appendix (as it does not directly fit in the main story), but to correctly frame this, it would require significant experiments. We do expect it to improve trainability of the tickets, but as pretraining also helps, the effect will likely be much less pronounced.

  2. In Table 6 our main goal was just to combine some different techniques to show their compatibility, not provide an exhaustive overview. To improve readability, we will add the lambda=0.0 values for some entries. Comparisons for individual hyperparameters (except for pruning rate) are in earlier tables in the paper (momentum, batch size, and budget in Table 2; regularization strength in Table 4). As in those tables, the accuracy is achieved by training a sparse network found under different Phase II hyperparameters with the same Phase III hyperparameters (see the first paragraph of sec 4.1). As such, the lower accuracy is explained by the fact that a higher budget finds worse trainable masks, which is supported by the central thesis in this research on learning distance.

Reviewer_BD8ZOpenReview ↗

Response to rebuttal

Thank you for providing clarifications,

Weaknesses

  1. I see that in the appendix additional experiments are provided for other datasets. But these are still small dataset, model size comparisons and for example ImageNet (with 1000 classes) on a ResNet50 or a vision transformer is missing. IMP is generally known to break down for larger model - dataset pairs, hence it is crucial to have analysis on these settings.

  2. If weight decay does somewhat affect learning distance, could you provide numbers to disentangle the effect of weight decay from the proposed method, otherwise it is difficult to evaluate.

  3. Thank you for clarifying this.

  4. It was not clear that the dense network is also random, it will help to add this explanation in the caption.

Questions

Q2 It is still not clear why the 200 epoch version is worse, considering longer training should improve performance. Is it because of the lack of regularization (weight decay), which was my initial question.

Dear reviewer BD8Z,

Thanks for considering our responses, and asking follow-ups.

  1. We do not have the computational capacity to run ImageNet-level experiments. Even a single training run would take almost a day on our architecture. As IMP is iteratively training, pruning, and rewinding; and we only use 20% pruning rate, this is simply not possible for us. We do have an experiment with ResNet-50 on ImageNet100 in the appendix (Table 16), which is a 100-class subset of ImageNet, and can serve to somewhat bridge the gap.

  2. We do provide these numbers, namely in Table 5 (block 1). Here, the regularization term is ∣∣M⊙θ∣∣2||M \odot \theta||_2 matching weight decay. We can see there compared to block 2 (M⊙θ−M⊙θ(0)M \odot \theta - M \odot \theta(0)_2) that increasing the regularization strength by having a higher λ\lambda negatively impacts the Weight decay approach suffers, while the proposed learning distance regularization does not. Additionally, the impact of weight decay on the learning distance (and accuracy) of a dense network is shown in Table 4, row 1

  3. We have added a sentence in the caption to make this more clear.

Q2. This is purely the budget of the Mask Search phase (Phase II). Once the mask is found, the sparse networks are all trained under exactly the same hyperparameter regime, including training epochs, and regularization (see appendix Sec G). The main difference is because during Mask Search, having more optimization steps before pruning means that the network has moved further from its initialization, leading to a higher learning distance. This is the main argument that we try to make, that a larger learning distance during Mask Search hinders the sparse networks' trainability.

Reviewer_BD8ZOpenReview ↗

Response to authors comments

Thank you for clarifying, I have raised my score.

Reviewer_D8NJOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Traditionally, finding a lottery ticket has 3 phases: Phase 1: pretrain a network for a short amount of time Phase 2: search for a mask through iterative magnitude pruning Phase 3: mask the pretrained network and train the remaining sparse network

Here they nicely show that because of their techniques, phase 1 is no longer necessary. Furthermore they decouple the training hyperparameters in phase 2 and phase 3, inspired by the nice idea that the optimal hyperparameters for finding a good mask may not be the same as the optimal hyperparameters required to train the subsequent sparse network. This allows them to do iterative magnitude pruning more aggressively in phase 2 resulting in finding lottery tickets as much as 50 times faster.

Critical strengths

The idea of decoupling hyperparameters across the phases is a powerful and simple idea.

Critical weaknesses

The logic and use of various diagnostic metrics in the paper, like sign flips, learning distance and stability to SGD noise, while interesting, could probably be tightened up and presented more crisply.

Questions for authors

Clarifying question: if you were to explain in a simple concise way what generalizable lessons you take away from your analysis of the above metrics, what would that be?

2 comments

Dear reviewer D8NJ,

We thank you for your positive assessment of our paper. To answer your question:

Our observation is that IMP suffers when it converges to a solution that is too far from the initialization. We show this in an implicit manner, by changing hyperparameters, which both lower the learning distance and increase the sparse network trainability; but to truly disentangle this, we also use explicit optimization on the learning distance. The generalizable lesson would then be that the lowest-magnitude connections for well-optimized dense networks are less important at that point, but more important in the learning process.

Kind regards, the authors

Thank you.

Reviewer_T17cOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to find trainable lottery tickets without pretraining the network.

The main observation is that dense-model training and mask search may need different hyperparameters. The authors focus on the distance from initialization. They use learning-distance regularization to keep the mask-search process close to the starting point.

The proposed recipe reduces ticket-extraction cost. It also keeps sparse-model accuracy at a competitive level.

Critical strengths

The paper makes a useful distinction between dense-model training and sparse-mask search. The best recipe for a dense model may not be the best recipe for finding a trainable ticket.

Learning distance provides a clearer explanation for why some masks are easier to retrain. This gives the analysis a useful mechanism.

The efficiency results are promising. The reported speedup could make lottery-ticket extraction more practical.

Critical weaknesses

Several design choices change at the same time. These include training length, pruning rate, regularization, and sign flips. It is difficult to identify the main source of improvement.

The method uses both a cheaper search schedule and more aggressive pruning. These choices may explain part of the gain.

The role of learning distance is not completely clear. Staying close to initialization may help retraining, but it may also limit useful exploration.

The paper does not clearly show whether lower learning distance consistently leads to better tickets. This relationship may depend on the model and dataset.

Questions for authors

A component-level ablation would help clarify the contribution of training length, pruning rate, regularization, and sign flips.

The reported speedup combines a cheaper search schedule with more aggressive pruning. Please separate the effects of these choices from the effect of learning-distance regularization.

Does learning-distance regularization still help when the pruning schedule and total search budget are fixed?

It would be useful to test whether lower learning distance consistently predicts better tickets across different models and datasets.

How should practitioners choose the target learning distance when applying the method to a new setting?

1 comment

Dear reviewer T17c,

We thank you for your thorough review. Please find our answers to the weaknesses and questions below. In general, we want to stress that sign flips is a diagnostic metric introduced by Renda et al. 2019 to explain why LRR outperforms IMP. Here in our paper, we build upon their reasoning and use it to flip the argument upon its head by showing how it changes based on different Phase II parameters. This means we do not directly optimize this metric.

Please find below a response to the different weaknesses and questions raised in the provided review.

Kind regards, the authors

Weaknesses / Questions

A component-level ablation... We do disentangle the hyperparameters (Table 4 for momentum, batch size, budget; Table 5 for regularization). In Table 6, we show the compatibility of these in a short form together with the pruning rate, which might be where this comment comes from.

Combines a cheaper search schedule with more aggressive pruning... This originates with table 6 as well. This table was mainly a thought experiment on how fast we could make IMP by combining different approaches we tested earlier in the paper. To accommodate this, we will add the 200eps search schedule with different pruning rates to this table as well.

Does learning-distance regularization still help when the pruning schedule and total search budget are fixed? Yes indeed, we show that in Table 5, where we start from the {momentum=0.9, bs=100, T=200eps} baseline, and use k=0.2 as throughout the rest of the paper. We will make this more explicit in the text.

It would be useful to test whether lower learning distance consistently predicts better tickets across different models and datasets. We do have additional dataset network combinations in the appendix, which we will stress more clearly in the paper. We show in this paper (1) that changing Phase II hyperparameters that lower learning distance (Table 3, Figure 1) have better performing tickets (Table 2), and (2) that explicitly optimizing for learning distance improves mask generalization (Table 4). Similar analysis is done in Sec C, D of the appendix for a variety of other models.

How should practitioners choose the target learning distance when applying the method to a new setting? Specific recommendations on the learning distance vary significantly based on the model, as it depends on the number of weights in the model (see Figure 12 in the appendix), so we cannot provide general recommendations. We can stress this a bit more in the conclusion. In our experiments we have seen that λ∈{10−3,10−2}\lambda \in \lbrace10^{-3}, 10^{-2}\rbrace performs best for various settings (Tables 5, 20), but we have not pushed for the upper limit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.