- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper provides a computational evaluation of alternating between network pruning (for reducing parameters) and perforated backpropagation (for growing parameters) in order to obtain a more parameter-efficient neural network in terms of accuracy. The study evaluates the results of three experimental settings: two involving a same neural network from a 2012 paper but with different pruning criteria, and another one involving an industrial application. The study concludes that perforation is more beneficial under heavy pruning.
Critical strengths
Although not new, the idea of growing and reducing parameters is an interesting one. According to the authors, this has not been studied for perforated backpropagation.
Critical weaknesses
For a paper like this, I would expect one of the two possibilities below:
- An extensive study involving several neural networks based on one or more configurable architectures with varying sizes and trained on multiple datasets, where a statistically significant pattern is observed and may inform future developments related to network pruning and perforated backpropagation. However, the study is limited to two neural networks and does not provide actionable results. For example, consider the breadth of the experiments carried out in these two network pruning studies:
- https://proceedings.neurips.cc/paper_files/paper/2022/hash/d3303e0ca98a267164d905bbc7947f88-Abstract-Conference.html
- https://proceedings.neurips.cc/paper_files/paper/2022/hash/7087c949df293f13c0052ac825936e6f-Abstract-Conference.html
- Alternatively, a study like this could originate a method based on combining network pruning and perforated backpropagation that produces competitive results in terms of parameter-efficiency. Such a study would require evaluation on several neural networks again, but perhaps not as many as in the case above. However, there would be an expectation of using more recent architectures in this case.
This paper may potentially evolve into either, but as it stands it is not sufficiently developed to warrant publication.
Questions for authors
I do not have questions, but I welcome the authors to discuss the points above.