LLM Compression by Block Removal with Constrained Binary Optimization

Abstract: We formulate the compression of large language models (LLMs) by deleting Transformer blocks (“block removal”) as a constrained binary optimization (CBO) problem. This problem can be mapped to a physical system (Ising glass), whose energies we show provide a useful proxy for downstream model performance. Here, we focus on the lowest-energy state of the proxy CBO problem as the block combination to be removed, but in fact, other states in the low-energy spectrum also yield high-quality, non-trivial solutions that go beyond simple consecutive regions. Our technique outperforms other methods in 13 out of 20 model-benchmark configurations. Notably, we obtain particularly strong performance on MMLU: for example, for 37.5 and 50% compression of Llama-3.3-70B-Instruct and Llama-3.1-8B-Instruct respectively, we score 21.5 and 22.9 percentage points better than the second-best method. Interestingly, we never achieve the highest score on HellaSwag in our tested configurations. However, no single competing method consistently achieves the best results across all evaluated use cases, while our method obtains the $\text{highest average}$ score in all but one test case (e.g. 11.2 points better average for the 50% compression of Llama-3.3-70B-Instruct and 6.1 points better for 35.5% compression of Llama-3.1-8B-Instruct). In general, our approach is computationally efficient and requires only forward and backward passes on a calibration dataset to compute the gradient with respect to $N$ variables (where $N$ is the number of transformer blocks). Additionally, we demonstrate that using good heuristic solvers for the CBO problem provides solutions that perform well on downstream tasks within seconds, even when it becomes infeasible to solve the CBO exactly.