- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes a method for solving the problem of optimal block removal via Constrained Binary Optimization with quadratic proxy. Specifically, the approach introduces auxiliary variables, corresponding to the presence/absence of a block, and removes the blocks leading to the minimal loss increase. Configurations are sought via combinatorial search or QUBO relaxation. The CBO method is validated on depth pruning of several language models.
Critical strengths
- The proposed method achieves strong performance on block removal consistently outperforming the baselines on a majority of the tasks.
Critical weaknesses
-
The method appears to be challenging to scale for large models due to the need to run backpropagation to large models and search over a huge number of configurations.
-
Several important baselines are missing:
- Pruning as a Cooperative Game [1] leveraging game-theoretic framework to solve the problem of optimal block dropping.
- LoRP [2] using hidden state similarity to guide the pruning decision.
- IG-Pruning [3] is input-guided block-wise pruning method that dynamically selects layer masks at inference time. Exhaustive comparison with the related work would be helpful to fully appreciate the efficacy of the proposed approach.
[1] Ding, Xuan, et al. "Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for Large Language Models." arXiv preprint arXiv:2602.07804 (2026).
[2] Yun, Vincent-Daniel, et al. "Locality-Aware Redundancy Pruning for LLM Depth Compression." arXiv preprint arXiv:2605.27786 (2026).
[3] Qiao, Kangyu, Shaolei Zhang, and Yang Feng. "Ig-pruning: Input-guided block pruning for large language models." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
Questions for authors
-
Have you tested the validity of the quadratic regularization? Is the loss increase predicted by a quadratic model correlated with the actual loss increase on some calibration dataset?
-
What is the runtime of the CBO method for configurations in Table 1? Do I get it right that some use exhaustive combinatorial search, whereas others (Llama-3.3-70B-Instruct) use QUBO solver? How does it compare to the baselines?
-
Would be interesting to see how far one can prune reasoning models (Qwen3.5, Gemma4, for instance) without significant performance degradation on popular reasoning benchmarks (GPQA-Diamond, AIME, MATH500, e.t.c.)
Weakness 1: Please see our responses to Q1 and Q2.
Weakness 2: more low-energy configurations
We have now extended the analysis to the 10 lowest-lying states for the Llama-3.1-8B-Instruct model when removing 12 blocks. The results show that the low-energy combinations are generally high-quality solutions with strong benchmark performance (considering that we have an aggressive compression ratio where other methods often collapse to random).
Ideally, we would like the energy to be a perfect cheap proxy for the average benchmark scores. However, it is only an approximate proxy (after all, there is a highly non-trivial relation ship between the calibration dataset and different downstream tasks). As such, the ground state is not guaranteed to have the best average score, and we do not claim it (see Fig. 3 of the paper and the related discussion). The energy is nevertheless a robust approximate proxy, since good solutions are found in the low-energy spectrum only (as far as we can check; we cannot check an exponentially large number of combianations).
We restrict ourselves to the ground state in the paper to remove any selection bias. However, we see that nearby excited states might score better on average (in this case, the 7th excited state has the highest score). We believe this is in fact a significant strength of our method, as it can identify multiple promising compression configurations compared to other methods, which tend to provide only a single option.
Weakness 3 (also see the answer to Q4): more compression rates and seeds
In the table below, we show another compression rate for Llama-3-70B, and two new models not in the paper: Qwen3-4B and Llama-2-13B-Chat. CBO achieves the highest average score in all instances. There is also a recurrent gap for MMLU, where CBO often achieves much higher scores than the other methods.