- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces PANCake, a post-training, pairwise neuron consolidation method that compresses neural network hidden representations by progressively consolidating redundant neurons. The method greedily identifies pairs of neurons with similar activation behavior and merges them by incorporating weight importance, an empirically weighted pair distance, and outgoing weight reconstruction. It adapts a refitting strategy from a previous structured pruning method, OSSCAR. The paper reports performance retention even after neuron consolidation on vision and language models.
Critical strengths
- The paper attempts to solve a well-known challenge in deep learning, which is to accelerate training. The method shows an interesting perspective on achieving savings by merging neurons instead of removing them completely. This is potentially important because, first, removing neurons accelerates training, resulting in potential savings; second, removing neurons completely can filter out important information propagation; merging them instead could retain some of it.
Critical weaknesses
- The contribution is incremental. Neuron pruning and merging formulations are well established in the literature, and the paper itself showed this in its literature review. What part of the contribution is new, and in what way it is exciting, is not clearly explained in the paper. A clear explanation of the novelty of the contribution could help in assessing the paper better.
- The empirical validation of the paper is incomplete. First, why the method often performs worse than magnitude and/or OSSCAR in multiple vision and language models is not explained. It also raises concerns about generalization. Second, the speed-up or acceleration gains are not properly explained. Table 2 shows a wall-clock cost, but PANCake is not the best-performing method in any setting. Overall, the magnitude approach looks better in terms of both performance and speed-up. Third, PANCake is compared against three baselines, but it is not directly compared against any modern state-of-the-art neuron merging techniques available in the literature. These limitations weaken the empirical evidence supporting the paper's central claims.
Questions for authors
- Can the author(s) clearly define the novelty of the contribution of the paper?
- What exactly does Table 1 represent? Does the first removal percentage for each model correspond to PANCake and the second to the baseline method? If so, how should the results be interpreted in relation to chance-level performance?
- Why does the method often perform worse than magnitude and/or OSSCAR in multiple vision and language models?
- Can the author(s) report direct comparison against any state-of-the-art neuron merging techniques?
- Can the author(s) explain how the method is accelerating deep learning training?