- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a non-cooperative game theory based solution for pruning of LLMs. In this method, K selectors are initiated which select layers from LLMs for pruning. Their choices of layers for pruning are non-overlapping. In the process, these selectors swap layers in a pool of K layers until the performance gain received i.e. . The optimization terminates when a stable equillibrium is reached which the authors mention as one-exchange-stable removal sets. They provide empirical evaluations against other benchmarks and show that their method shows improvement in many experimental settings.
Critical strengths
The paper poses an important problem of pruning LLMs which has practical applications in numerous settings. It has many open problems and rich literature. This work provides a game theory based approach that is simple in comparison to many other approaches like [1], [2], [3], [4]. Their method has comparatively low computational complexity, the evaluation function is KLD which is easy to calculate and has a terminating condition that can be met in finite amount of time. They provide good experimental analysis for their work and comparison with other baselines. The hyper parameter settings along with other ablation study is also provided in the appendix.
Critical weaknesses
- I would like to see more motivation in the paper which explains why this approach of game theory is worth pursuing in comparison to other pruning approaches. Although it mentions comparison with other methods like Wanda, this method in its core assumes a masked or switch on/off pruning of layers. It is not a pruning scenario where there is some pruning ratio to achieve certain level of sparsity in the network which is a more challenging problem setting.
- There is no literature survey which explains how the field is moving in this problem setting and what were the problems that other approaches missed to address and this method is addressing them. Although it mentions about Ding et al. there is no comparison or improvement of this approach over EvoPress[2].
- The novelty of this approach is limited in comparison to Curvature-Weighted Capacity Allocation [1] and EvoPress [2]. At its best, this is an interpretation of local search with little theoretical analysis of why it works. Although the paper mentions equilibrium characterization and finite termination conditions, I would like to see more theoretical analysis of this method.
- The theoretical results are standard: assigning all selectors the same utility immediately gives an exact-potential gain, and strict improvement over finitely many feasible subsets guarantees termination. The equilibrium characterization simply restates that no one-for-one layer exchange improves the objective. Thus the Propositions 1–2 are therefore correct but offer limited theoretical novelty. None of the results provide any approximation guarantee, generalization bound, or polynomial-time convergence guarantee. Thus no novelty in theoretical section
- The ablation mentions that this method does work very well in adversarial setting. I would like to see empirical and theoretical analysis of is this behaviour. Is this observation a result of layers being skipped completely instead of having sparsity ratios per layer. I would like to see what happens to the hidden representation of data distribution when you are turning off some layers of the network in this type of pruning.
- It does not motivate why turning off layers is a good pruning setting. Although some papers suggest that calculating layer importance is not required for pruning, a good experiment would have been to test if there is actually any/no correlation between layer importance and the choices of layer numbers provided by this algorithm for pruning.
Questions for authors
- The paper motivates its contribution through the limitations of standalone layer scores and the need to make better decisions under a fixed budget. Curvature-Weighted Capacity Allocation [1] does the same thing and the authors miss providing comparison against existing literature. I would like authors to provide a comparison against Curvature-Weighted Capacity Allocation [1] and EvoPress [2] and state what advantages their method provides.
- Is any behaviour with respect to depth of layers observed in the obtained solution
- It would be a good idea to redo Figure 2
- Table 19 reports different PTB perplexities for identical final removal masks at budgets 3 and 6; Appendix F.1.3 acknowledges this discrepancy but does not explain its cause. Please identify its source and provide a controlled reevaluation using the same evaluation settings for random and standalone initialization.
- There is mismatch between the values provided in Table 1 vs Table 4 on same setting. For example, Table 1 reports PTB PPL for Remove 9 for cooperative game as 76.9957 vs Table 4 reports it as 76.1299. What is the explanation for this difference?
- Table 1 combines or reports published baseline scores with local evaluations without establishing equivalent evaluation protocols, and the PTB ranking at nine-layer removal reverses in the local comparison in Table 4. Please verify protocol equivalence or reevaluate the remaining baselines under the same settings, and base superiority claims on these controlled comparisons.
- There is a strong overlap of the method proposed in this paper with EvoPress [2]. They use the same objective function of KL Divergence, swap mechanism, the block or layer level granularity as EvoPress [2]. The EvoPress [2] has the ability to escape local optima which is not examined in this method. The authors should cite this work and provide a comparison both intuitively and empirically with same settings to prove novelty over EvoPress [2].
References
[1] Amaefuna, T., Vaidya, H.U., Chhabra, A. & Mali, A.. (2026). Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization. <i>Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence</i>, in <i>Proceedings of Machine Learning Research</i> 337:62-86 Available from https://proceedings.mlr.press/v337/amaefuna26a.html.
[2] Sieberling, O., Kuznedelev, D., Kurtic, E. & Alistarh, D.. (2025). EvoPress: Accurate Dynamic Model Compression via Evolutionary Search. <i>Proceedings of the 42nd International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 267:55556-55590 Available from https://proceedings.mlr.press/v267/sieberling25a.html.
[3] Malettira, P. G., Nagaraj, M., Roy, A., Negi, S., & Roy, K. (2026, August). TraceNAS: Zero-shot search for LLM pruning via gradient trace correlation. Paper presented at the COLM 2026 Workshop on Efficient Reasoning. https://openreview.net/forum?id=lZVf95CjEz
[4] Askari, H., Gupta, S., Wang, F., Chhabra, A., & Chen, M. (2026). Layerif: Estimating layer quality for large language models using influence functions. Advances in Neural Information Processing Systems, 38, 125841-125868.