PANCake: Pairwise Neuron Consolidation

Memory and inference-time efficiency are major bottlenecks in scaling neural networks. Structured pruning reduces these costs by removing whole neurons or channels, but most methods formulate pruning as selecting a subset of units to delete, often followed by downstream reconstruction or fine-tuning to recover lost performance. We introduce PANCake (PAirwise Neuron Consolidation), a post-training structured pruning method that instead compresses hidden representations by progressively consolidating redundant neurons. PANCake greedily identifies pairs of neurons with similar activation behavior and merges them, producing a nested hierarchy of representations while avoiding combinatorial subset selection. Across image classification and language modeling, PANCake preserves performance particularly well without global downstream reoptimization. At 20% removal, PANCake retains 45.8% ImageNet top-1 accuracy on ResNet-50, compared with 14.0% when OSSCAR-selected neurons are deleted without downstream optimization and below 1.5% for magnitude pruning. On Pythia-2.8B, it achieves a perplexity of 19.1, compared with 60.3 for OSSCAR-selected deletion. With downstream repair, PANCake remains within 0.5 perplexity of OSSCAR on Pythia-2.8B, Qwen2.5-7B, and Llama-3.1-8B at 20% removal, despite selecting different neurons. These results show that explicitly exploiting redundancy through pairwise consolidation provides an effective alternative to deletion-based structured pruning.