- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the problem of trainability of lottery tickets. It is a well studied problem in the Lottery Ticket and sparse training literature that although the Lottery Ticket Hypothesis claims the existence of sparse networks, training them from scratch with the Iterative Magnitude Pruning method is hard. And in most cases the sparse networks found by this method for relatively large models are underperformant. The paper tries to address this issue of the trainability of sparse lottery tickets.
They propose to modify the iterative prune - reinitialize - train part of the IMP algorithm, by adding a regularization term which forces the active parameters to stay close to initialization. Additionally, they find that using no momentum and a smaller batch size aids this phase of training.
While the paper addresses a longstanding trainability problem, the experimental results are underwhelming. Results are mostly focused on a ResNet34 for Tiny Imagenet, which is not sufficient to validate the strong claims of trainability made in the paper.
Critical strengths
-
The paper addresses a longstanding problem of making sparse lottery tickets trainable. They identify distance from initialization as a regularizer to aid trainability. This is an interesting and relevant finding.
-
The paper also finds that the hyperparameter settings for the iterative pruning part of IMP must be different from the final sparse training phase.
Critical weaknesses
-
My main concern with this paper is that the empirical evaluation is limited to just one setting. While I understand that evaluating large transformers might be difficult for an expensive method like IMP, I believe multiple CNN and small language or vision transformer settings could have been used to further strengthen the paper's findings.
-
In Table 5, as the regularization strength increases, the effect of weight decay first worsens performance before improving it. Why does large weight decay improve performance and what happens at . Given the trend it is possible that the weight decay alone could match the proposed regularization at least up to 90% sparsity. The broader point here is that at lower sparsities weight decay and the closeness to init regularizer are comparable and both are unable to reach the performance of the original network (so not a lottery ticket). Hence the distinction between these two regularizers is not completely obvious and might need more investigation.
-
In the hyperparameter search, the paper only considers momentum and batch size, but there is no justification for choosing these two and not the learning rate or weight decay for example.
-
Table 8 is not a fair comparison with the dense baseline, all sparse networks are first iteratively trained and then a linear layer is learned on top of it if I understand correctly, but for the dense network, a linear layer is learned on top of a random network. How can the results be interpreted and compared to the dense net in the context. I believe the correct baseline must be a trained backbone, which is the frozen and a linear layer is learned on top of it.
Questions for authors
-
How does the closeness to initialization regularizer work in combination with rewinding to a partially trained point? This should improve the final performance of the sparse network significantly but this does not seem to have been studied in the paper.
-
The speedup in able 6 is not clear to me. Why is the 200 epoch version not trained with a regularizer and why is the accuracy so much worse compared to the 20 epoch version?
4 comments
Response to rebuttal
Thank you for providing clarifications,
Weaknesses
-
I see that in the appendix additional experiments are provided for other datasets. But these are still small dataset, model size comparisons and for example ImageNet (with 1000 classes) on a ResNet50 or a vision transformer is missing. IMP is generally known to break down for larger model - dataset pairs, hence it is crucial to have analysis on these settings.
-
If weight decay does somewhat affect learning distance, could you provide numbers to disentangle the effect of weight decay from the proposed method, otherwise it is difficult to evaluate.
-
Thank you for clarifying this.
-
It was not clear that the dense network is also random, it will help to add this explanation in the caption.
Questions
Q2 It is still not clear why the 200 epoch version is worse, considering longer training should improve performance. Is it because of the lack of regularization (weight decay), which was my initial question.
Dear reviewer BD8Z,
Thanks for considering our responses, and asking follow-ups.
-
We do not have the computational capacity to run ImageNet-level experiments. Even a single training run would take almost a day on our architecture. As IMP is iteratively training, pruning, and rewinding; and we only use 20% pruning rate, this is simply not possible for us. We do have an experiment with ResNet-50 on ImageNet100 in the appendix (Table 16), which is a 100-class subset of ImageNet, and can serve to somewhat bridge the gap.
-
We do provide these numbers, namely in Table 5 (block 1). Here, the regularization term is matching weight decay. We can see there compared to block 2 (_2) that increasing the regularization strength by having a higher negatively impacts the Weight decay approach suffers, while the proposed learning distance regularization does not. Additionally, the impact of weight decay on the learning distance (and accuracy) of a dense network is shown in Table 4, row 1
-
We have added a sentence in the caption to make this more clear.
Q2. This is purely the budget of the Mask Search phase (Phase II). Once the mask is found, the sparse networks are all trained under exactly the same hyperparameter regime, including training epochs, and regularization (see appendix Sec G). The main difference is because during Mask Search, having more optimization steps before pruning means that the network has moved further from its initialization, leading to a higher learning distance. This is the main argument that we try to make, that a larger learning distance during Mask Search hinders the sparse networks' trainability.
Response to authors comments
Thank you for clarifying, I have raised my score.
Dear reviewer BD8Z,
We thank you for your thorough review. However, we feel that some weaknesses are mischaracterized, which might be due to content in the appendix which we didn’t stress enough in the main paper. This will be resolved in the new version. Please find a specific rebuttal below to the weaknesses and questions listed in the provided review.
Kind regards,
the authors
Weaknesses
This was mentioned in a single line in sec 3.3, but other settings were already explored in the appendix. We will make this more explicit by mentioning the specific combinations as well, and add a hard reference. FYI, these were: ResNet-18 + CIFAR10, ResNet-18 + CIFAR100, VGG16 + CIFAR100, Swin + TinyImageNet, ResNet-50 + ImageNet100. We chose ResNet-34 + TinyImageNet in the main body due to its balanced complexity, allowing us to have in-depth experiments, but still be computationally feasible.
At λ=10−2, dense networks are overregularized, and converge to the origin, resulting in effectively random pruning. We have since calculate the resulting accuracies and will add them to the paper including a short explanation. Weight decay somewhat impacts learning distance, as randomly initialized weights are quite close to the origin (see the second part of Table 4), and the ‘Different targets’ paragraph of Sec 4.2.
We do have a motivation in Sec 2.4, where we refer to the existing work linking these hyperparameters to generalization. Additionally, we do explore the weight decay metric (see Table 5, and Sec 4.2 the ‘Explicit regularization’ paragraph). We will make this more explicit in Sec 2.4, and also mention that we have more hyperparameter values, as well as learning rate scaling in Sec C of the appendix.
Actually, in this table, all networks consist of random weights with an optional sparsity mask, which are then probed. We show that certain masks extract useful features from the random weights depending on the Phase II hyperparameters, so in this case the random network is justified. In the appendix F we have an additional experiment where we transfer frozen trained tickets / networks, where a trained dense network is the correct baseline.
Answers:
The main starting point of this paper was to avoid the partially trained rewind point, as stated in the contributions, which means that this is something we have not explored in detail. In Figure 2 we do however show that pretraining similarly impacts learning distance. If time allows us, we will include results for this in the appendix (as it does not directly fit in the main story), but to correctly frame this, it would require significant experiments. We do expect it to improve trainability of the tickets, but as pretraining also helps, the effect will likely be much less pronounced.
In Table 6 our main goal was just to combine some different techniques to show their compatibility, not provide an exhaustive overview. To improve readability, we will add the lambda=0.0 values for some entries. Comparisons for individual hyperparameters (except for pruning rate) are in earlier tables in the paper (momentum, batch size, and budget in Table 2; regularization strength in Table 4). As in those tables, the accuracy is achieved by training a sparse network found under different Phase II hyperparameters with the same Phase III hyperparameters (see the first paragraph of sec 4.1). As such, the lower accuracy is explained by the fact that a higher budget finds worse trainable masks, which is supported by the central thesis in this research on learning distance.