acceptodds
Under review as a conference paper at ICLR 2027

Layer Pruning as a Non-Cooperative Game: Conditional Exchanges for Large Language Models

Abstract

Structured layer pruning reduces the computational and memory costs of large language models, but the effect of removing a layer depends on which other layers remain. Static rankings may overlook this dependence, while surrogate-based approaches introduce an additional prediction stage. We formulate fixed-budget layer selection as a finite non-cooperative game with a shared utility and develop a conditional exchange procedure. Starting from standalone layer scores, the method evaluates exchanges that restore one removed layer and remove one retained layer. At each iteration, it accepts the exchange that most reduces the Kullback-Leibler divergence between dense and pruned output distributions on calibration data. The search terminates when no permitted exchange improves the objective beyond a numerical tolerance, without weight updates, surrogate training, or Monte Carlo Shapley estimation. Experiments across multiple model families, pruning budgets, and language-modeling benchmarks show lower perplexity than the comparison method in most evaluated configurations, while zero-shot results reveal budget-dependent trade-offs. Matched inference measurements show comparable latency, throughput, and allocated GPU memory. In a separate wall-clock evaluation on a seven-billion-parameter model, our method is about faster overall than the comparison method across four pruning budgets. These results support conditional exchange as a practical approach to selecting layers while accounting for their interactions.

Primary area:
optimization
Keywords:
Large Language Models, Layer Pruning, Structured Pruning, Model Compression, Non-Cooperative Games, Game Theory, Efficient LLMs, Post-Training Pruning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fS8hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a non-cooperative game theory based solution for pruning of LLMs. In this method, K selectors are initiated which select layers from LLMs for pruning. Their choices of layers for pruning are non-overlapping. In the process, these selectors swap layers in a pool of K layers until the performance gain received i.e. Δ>τ\Delta > \tau. The optimization terminates when a stable equillibrium is reached which the authors mention as one-exchange-stable removal sets. They provide empirical evaluations against other benchmarks and show that their method shows improvement in many experimental settings.

Critical strengths

The paper poses an important problem of pruning LLMs which has practical applications in numerous settings. It has many open problems and rich literature. This work provides a game theory based approach that is simple in comparison to many other approaches like [1], [2], [3], [4]. Their method has comparatively low computational complexity, the evaluation function is KLD which is easy to calculate and has a terminating condition that can be met in finite amount of time. They provide good experimental analysis for their work and comparison with other baselines. The hyper parameter settings along with other ablation study is also provided in the appendix.

Critical weaknesses
  • I would like to see more motivation in the paper which explains why this approach of game theory is worth pursuing in comparison to other pruning approaches. Although it mentions comparison with other methods like Wanda, this method in its core assumes a masked or switch on/off pruning of layers. It is not a pruning scenario where there is some pruning ratio to achieve certain level of sparsity in the network which is a more challenging problem setting.
  • There is no literature survey which explains how the field is moving in this problem setting and what were the problems that other approaches missed to address and this method is addressing them. Although it mentions about Ding et al. there is no comparison or improvement of this approach over EvoPress[2].
  • The novelty of this approach is limited in comparison to Curvature-Weighted Capacity Allocation [1] and EvoPress [2]. At its best, this is an interpretation of local search with little theoretical analysis of why it works. Although the paper mentions equilibrium characterization and finite termination conditions, I would like to see more theoretical analysis of this method.
  • The theoretical results are standard: assigning all selectors the same utility immediately gives an exact-potential gain, and strict improvement over finitely many feasible subsets guarantees termination. The equilibrium characterization simply restates that no one-for-one layer exchange improves the objective. Thus the Propositions 1–2 are therefore correct but offer limited theoretical novelty. None of the results provide any approximation guarantee, generalization bound, or polynomial-time convergence guarantee. Thus no novelty in theoretical section
  • The ablation mentions that this method does work very well in adversarial setting. I would like to see empirical and theoretical analysis of is this behaviour. Is this observation a result of layers being skipped completely instead of having sparsity ratios per layer. I would like to see what happens to the hidden representation of data distribution when you are turning off some layers of the network in this type of pruning.
  • It does not motivate why turning off layers is a good pruning setting. Although some papers suggest that calculating layer importance is not required for pruning, a good experiment would have been to test if there is actually any/no correlation between layer importance and the choices of layer numbers provided by this algorithm for pruning.
Questions for authors
  • The paper motivates its contribution through the limitations of standalone layer scores and the need to make better decisions under a fixed budget. Curvature-Weighted Capacity Allocation [1] does the same thing and the authors miss providing comparison against existing literature. I would like authors to provide a comparison against Curvature-Weighted Capacity Allocation [1] and EvoPress [2] and state what advantages their method provides.
  • Is any behaviour with respect to depth of layers observed in the obtained solution
  • It would be a good idea to redo Figure 2
  • Table 19 reports different PTB perplexities for identical final removal masks at budgets 3 and 6; Appendix F.1.3 acknowledges this discrepancy but does not explain its cause. Please identify its source and provide a controlled reevaluation using the same evaluation settings for random and standalone initialization.
  • There is mismatch between the values provided in Table 1 vs Table 4 on same setting. For example, Table 1 reports PTB PPL for Remove 9 for cooperative game as 76.9957 vs Table 4 reports it as 76.1299. What is the explanation for this difference?
  • Table 1 combines or reports published baseline scores with local evaluations without establishing equivalent evaluation protocols, and the PTB ranking at nine-layer removal reverses in the local comparison in Table 4. Please verify protocol equivalence or reevaluate the remaining baselines under the same settings, and base superiority claims on these controlled comparisons.
  • There is a strong overlap of the method proposed in this paper with EvoPress [2]. They use the same objective function of KL Divergence, swap mechanism, the block or layer level granularity as EvoPress [2]. The EvoPress [2] has the ability to escape local optima which is not examined in this method. The authors should cite this work and provide a comparison both intuitively and empirically with same settings to prove novelty over EvoPress [2].

References

[1] Amaefuna, T., Vaidya, H.U., Chhabra, A. & Mali, A.. (2026). Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization. <i>Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence</i>, in <i>Proceedings of Machine Learning Research</i> 337:62-86 Available from https://proceedings.mlr.press/v337/amaefuna26a.html.
[2] Sieberling, O., Kuznedelev, D., Kurtic, E. & Alistarh, D.. (2025). EvoPress: Accurate Dynamic Model Compression via Evolutionary Search. <i>Proceedings of the 42nd International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 267:55556-55590 Available from https://proceedings.mlr.press/v267/sieberling25a.html.
[3] Malettira, P. G., Nagaraj, M., Roy, A., Negi, S., & Roy, K. (2026, August). TraceNAS: Zero-shot search for LLM pruning via gradient trace correlation. Paper presented at the COLM 2026 Workshop on Efficient Reasoning. https://openreview.net/forum?id=lZVf95CjEz
[4] Askari, H., Gupta, S., Wang, F., Chhabra, A., & Chen, M. (2026). Layerif: Estimating layer quality for large language models using influence functions. Advances in Neural Information Processing Systems, 38, 125841-125868.

Reviewer_9e9JOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates fixed-budget transformer-layer pruning as a finite non-cooperative common-interest game. Each selector chooses one layer to remove, and unilateral deviations correspond to one-for-one exchanges between removed and retained layers. Starting from standalone layer-damage scores, the proposed method performs best-improvement local search using dense-to-pruned KL divergence on a small calibration set.

Critical strengths
  1. Simple and well-defined optimization procedure. The one-exchange neighborhood, explicit tolerance, and finite-termination argument make the method easy to understand and reproduce.
  2. Broad initial evaluation. The paper evaluates several model families, pruning budgets, perplexity datasets, downstream tasks, and efficiency measures, rather than relying on a single benchmark.
Critical weaknesses
  1. Limited and potentially overfit calibration protocol. Selection is based on ten short calibration sequences. The reported calibration-source analysis shows that different sources produce different layer sets and held-out results, but there is no systematic evaluation over independently sampled calibration sets or multiple seeds.

  2. Evidence for general superiority is mixed. The method improves perplexity in many settings, but downstream accuracy and ANLI often favor the cooperative baseline. The strongest efficiency claim is also based on a small number of repeats and an asymmetric timing boundary, so it does not yet establish a generally superior pruning pipeline.

Questions for authors
  1. Can the authors report results over multiple independently sampled calibration sets and seeds, including variance and statistical tests for perplexity and downstream accuracy?

  2. Can the authors provide a strictly matched wall-clock comparison in which both methods construct the same budgets under the same preprocessing, hardware, process, and repetition protocol?

  3. Can the authors release the implementation, selected masks, calibration samples, and evaluation scripts needed to reproduce all locally measured tables?

Reviewer_i6zPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a structured layer pruning method for large language models based on non-cooperative game theory, aiming to address the problem that traditional static importance ranking methods cannot adequately capture dependencies between layers. The authors formulate layer selection under a fixed pruning budget as a non-cooperative game with a shared utility function and propose a KL Divergence-based Conditional Exchange algorithm, which iteratively exchanges removed and retained layers to optimize the difference between the output distributions of the original and pruned models.

Critical strengths
  1. The research problem has practical significance.
  2. The paper is easy to follow.
Critical weaknesses

I have the following concerns about this paper:

  1. One very important issue is that I think the authors include too little content in the main paper. ICLR 2027 allows up to 9 pages for the main text, but this paper only has 8 pages of main content. I suggest moving some important content from the appendix to the main paper.
  2. Although the Introduction explains the differences between this work and Ding et al. (ICLR 2026) [1] in terms of the game formulation and layer selection strategy, it does not sufficiently clarify the substantive methodological innovations brought by these differences. Layer dependencies are not a new problem first identified by this paper, and the use of game theory for LLM layer pruning has already been explored in prior work [1].
  3. Regarding the method section, I think the game-theoretic framework provides limited algorithmic or theoretical advantages independent of local search. The authors assign the same utility function to all players, so the Exact Potential Game property in Proposition 1 follows almost directly from the definition, while the so-called Generalized Nash Equilibrium is essentially equivalent to a locally optimal condition under one-layer exchanges. In addition, Algorithm 1 is essentially a standard best-improvement local search and does not demonstrate additional advantages of the game-theoretic framework over traditional local search in terms of algorithm design or theoretical guarantees.
  4. The method figure looks quite messy, and placing it in the appendix makes it difficult for readers to understand.
  5. In the Experiments section, although the proposed method achieves better results than Cooperative Game in most PPL experiments, these advantages do not consistently translate into improvements in downstream task performance. For example, in Table 3, the MMLU accuracy of the proposed method is lower than that of Cooperative Game under all four pruning budgets. Moreover, in Table 1, when removing 12 layers from LLaMA-2-7B, the PTB PPL increases from 105.2407 for Cooperative Game to 148.6100 for the proposed method, showing a significant degradation. The locally paired evaluation in Table 4 also shows similar results. Although the authors acknowledge in the Discussion that reducing calibration KL Divergence does not necessarily lead to improvements in downstream performance, a more in-depth analysis of this phenomenon is still lacking.
  6. Some baseline results in Table 1 and Table 3 are directly taken from previous papers, while the results of the proposed method are obtained through local evaluations. Although the authors clarify the sources of these results, differences in experimental environments and evaluation settings may affect PPL and zero-shot accuracy.
  7. In Section 3.5, I think the reported 2.27× speedup needs more thorough validation. The authors compare the entire pruning pipeline, including model loading, layer selection, and checkpoint saving, rather than the search algorithm alone. Moreover, the experiment includes only two successful paired runs. Therefore, it is still unclear how much of the actual speedup comes from Conditional Exchange itself and how much comes from avoiding additional steps such as surrogate training in Cooperative Game.
  8. I think the ablation experiments in Section 3.5 are still insufficient to demonstrate the advantages of Conditional Exchange over simpler pruning strategies. Although Conditional Exchange consistently reduces KL Divergence on the calibration set, these improvements do not consistently translate into PPL gains on the test set. For example, in Table 17 of Appendix F.1, static ranking achieves lower PTB PPL than Conditional Exchange under all three settings of removing 6, 9, and 12 layers. In addition, when removing 9 layers, static ranking also achieves slightly better results on WikiText-2. This suggests that although the additional exchange search can improve the optimized calibration objective, it does not necessarily lead to better generalization performance.

References:

  1. Ding et al. "Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for Large Language Models." ICLR 2026.

Minors:

  1. Figure 1 is a results figure and would be more appropriately placed in the later comparison section.
  2. On Page 6, placing three tables on a single page makes it difficult for readers to read.
Questions for authors

Please see the Weaknesses above; the relevant questions are all included there.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.