acceptodds
Withdrawn from ICLR 2027

PANCake: Pairwise Neuron Consolidation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Memory and inference-time efficiency are major bottlenecks in scaling neural networks. Structured pruning reduces these costs by removing whole neurons or channels, but most methods formulate pruning as selecting a subset of units to delete, often followed by downstream reconstruction or fine-tuning to recover lost performance. We introduce PANCake (PAirwise Neuron Consolidation), a post-training structured pruning method that instead compresses hidden representations by progressively consolidating redundant neurons. PANCake greedily identifies pairs of neurons with similar activation behavior and merges them, producing a nested hierarchy of representations while avoiding combinatorial subset selection. Across image classification and language modeling, PANCake preserves performance particularly well without global downstream reoptimization. At 20% removal, PANCake retains 45.8% ImageNet top-1 accuracy on ResNet-50, compared with 14.0% when OSSCAR-selected neurons are deleted without downstream optimization and below 1.5% for magnitude pruning. On Pythia-2.8B, it achieves a perplexity of 19.1, compared with 60.3 for OSSCAR-selected deletion. With downstream repair, PANCake remains within 0.5 perplexity of OSSCAR on Pythia-2.8B, Qwen2.5-7B, and Llama-3.1-8B at 20% removal, despite selecting different neurons. These results show that explicitly exploiting redundancy through pairwise consolidation provides an effective alternative to deletion-based structured pruning.

Primary area:
optimization
Keywords:
Neural Network Pruning, Structured Pruning, Neuron Pruning, Neuron Merging / Neuron Consolidation, Model Compression, Representation Compression, Neuron Redundancy, Post-Training Pruning, Activation Similarity, Hierarchical Clustering, Greedy Optimization, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_qAxWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PANCake, a post-training, pairwise neuron consolidation method that compresses neural network hidden representations by progressively consolidating redundant neurons. The method greedily identifies pairs of neurons with similar activation behavior and merges them by incorporating weight importance, an empirically weighted pair distance, and outgoing weight reconstruction. It adapts a refitting strategy from a previous structured pruning method, OSSCAR. The paper reports performance retention even after neuron consolidation on vision and language models.

Critical strengths
  1. The paper attempts to solve a well-known challenge in deep learning, which is to accelerate training. The method shows an interesting perspective on achieving savings by merging neurons instead of removing them completely. This is potentially important because, first, removing neurons accelerates training, resulting in potential savings; second, removing neurons completely can filter out important information propagation; merging them instead could retain some of it.
Critical weaknesses
  1. The contribution is incremental. Neuron pruning and merging formulations are well established in the literature, and the paper itself showed this in its literature review. What part of the contribution is new, and in what way it is exciting, is not clearly explained in the paper. A clear explanation of the novelty of the contribution could help in assessing the paper better.
  2. The empirical validation of the paper is incomplete. First, why the method often performs worse than magnitude and/or OSSCAR in multiple vision and language models is not explained. It also raises concerns about generalization. Second, the speed-up or acceleration gains are not properly explained. Table 2 shows a wall-clock cost, but PANCake is not the best-performing method in any setting. Overall, the magnitude approach looks better in terms of both performance and speed-up. Third, PANCake is compared against three baselines, but it is not directly compared against any modern state-of-the-art neuron merging techniques available in the literature. These limitations weaken the empirical evidence supporting the paper's central claims.
Questions for authors
  1. Can the author(s) clearly define the novelty of the contribution of the paper?
  2. What exactly does Table 1 represent? Does the first removal percentage for each model correspond to PANCake and the second to the baseline method? If so, how should the results be interpreted in relation to chance-level performance?
  3. Why does the method often perform worse than magnitude and/or OSSCAR in multiple vision and language models?
  4. Can the author(s) report direct comparison against any state-of-the-art neuron merging techniques?
  5. Can the author(s) explain how the method is accelerating deep learning training?
Reviewer_WD9KOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to compress a neural network by combining neurons that perform similar functions. Instead of simply deleting neurons with low importance, PANCake repeatedly identifies a similar pair and replaces it with a single representative. This representative can be one of the original neurons or a weighted combination of their responses. The method also combines their outgoing weights and can optionally adjust the remaining weights to reduce the resulting error.

The main contributions are a procedure for selecting and consolidating neuron pairs, a theoretical analysis of the error introduced by an individual consolidation, and a hierarchy that allows different compression levels to be selected. The authors evaluate the method on several vision and language-model architectures, both with and without an additional weight-reconstruction step.

The strongest results occur on ResNet and Pythia models when the comparison methods delete neurons without compensating for their removal. With reconstruction, PANCake performs comparably to OSSCAR on several language models. Results vary across architectures, and the paper reports settings where other methods perform better.

Critical strengths

The paper addresses an important compression problem. Neurons may be redundant because other neurons perform similar functions, even when their individual activations or weights are large. Exploiting this similarity is a useful alternative to ranking neurons independently. The proposed procedure is conceptually simple: it repeatedly selects a similar pair, consolidates it, transfers the outgoing weights, and updates the pair's importance. This produces an understandable sequence of compression decisions, and the comparison with deleting both selected neurons provides evidence for the role of preserving a representative and transferring outgoing weights.

The paper combines a useful mathematical analysis with a broad empirical evaluation. The analysis characterizes the reconstruction error for a pair of neurons and bounds the additional error incurred when retaining an original neuron instead of using the ideal weighted representative. The experiments cover several model families, including convolutional networks, vision transformers, and multiple language models. The authors also investigate the contribution of weight reconstruction, acknowledge unfavorable results, and propose a hierarchy that could reduce the work required to explore multiple compression levels.

Critical weaknesses

The presentation and technical details need clarification. The introduction's four-stage explanation mixes definitions, descriptions of existing methods, and a stopping problem outside the paper's scope. Terms such as consolidation, restructuring, reconstruction, and repair are difficult to distinguish. More importantly, the proposed weighted representative is not necessarily implementable as a single neuron: a weighted average of nonlinear neuron responses generally cannot be reproduced by averaging their parameters. The paper should specify the exact transformations used to produce smaller networks, particularly for gated language-model layers, and clarify how repeated consolidations affect the validity of the theoretical guarantees. Important implementation details, including calibration data, sample counts, activation processing, and the reuse of downstream hierarchies when earlier layers change, also need to be documented.

The empirical evidence does not sufficiently establish the method's advantages or practical value. The main comparisons do not isolate the benefit of pair selection, since PANCake transfers outgoing weights while deletion baselines receive no equivalent compensation, and the repaired methods use different reconstruction objectives. Comparisons using a shared weight-transfer rule and reconstruction objective are needed. The novelty over existing neuron-surgery methods and activation-based Ward clustering also remains unclear. Finally, compression time across target widths does not establish the memory or inference benefits of a deployed model. The evaluation should report actual model size, latency, and throughput, and provide stronger evidence of robustness beyond WikiText-2 perplexity, including more repeated runs where needed.

Questions for authors
  1. How is the weighted combination implemented as a single neuron in the compressed network? Please provide the exact parameter transformations for each architecture, distinguish functional averaging from parameter averaging, and verify the resulting parameter and computational savings.

  2. Does PANCake retain its advantage when selection methods receive the same compensation and reconstruction? Can you compare the selection strategies using a shared weight-transfer rule and a shared reconstruction objective, while also reporting each method's standard complete pipeline?

  3. What does PANCake add beyond existing neuron-merging methods and activation-based Ward clustering? Can you compare these alternatives using the same models, calibration data, compression levels, and reconstruction procedure?

Reviewer_vrRNOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
1
Soundness
1
Read review
Paper summary

This paper uses a network merging logic to deal with model compression. In order to reduce the number of neuron and avoid structured pruning which requires fine-tuning and performance drop, this draft uses merging technique to achieve smaller neural network model. Comparing with baseline models, experiments show the proposed method is better than them.

Critical strengths
  1. Model compression is a critical topic especially for nowadays large-scale AI model deployment.
  2. Neuron merging is a wise choice to achieve the compression purpose.
  3. The draft is easy to follow and understand.
Critical weaknesses
  1. Writing is not clear. The first sentence, why test time efficiency is bottleneck for scaling neural network? this is a weird statement, the logic here should be carefully revised. This is an example happening at the very beginning of the draft, but other parts also need to be revised.
  2. Only OSSCAR is included as baseline for comparison, more methods are needed. The current experiments are not solid enough.
  3. Draft format is not ready. Fig1 is not informative. Fig2 and Tab1 are not arranged well. Why fig4 and tab2 are after the conclusion?
Questions for authors

Please check the section above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.