acceptodds
Under review as a conference paper at ICLR 2027

LLM Compression by Block Removal with Constrained Binary Optimization

Abstract

Abstract: We formulate the compression of large language models (LLMs) by deleting Transformer blocks (“block removal”) as a constrained binary optimization (CBO) problem. This problem can be mapped to a physical system (Ising glass), whose energies we show provide a useful proxy for downstream model performance. Here, we focus on the lowest-energy state of the proxy CBO problem as the block combination to be removed, but in fact, other states in the low-energy spectrum also yield high-quality, non-trivial solutions that go beyond simple consecutive regions. Our technique outperforms other methods in 13 out of 20 model-benchmark configurations. Notably, we obtain particularly strong performance on MMLU: for example, for 37.5 and 50% compression of Llama-3.3-70B-Instruct and Llama-3.1-8B-Instruct respectively, we score 21.5 and 22.9 percentage points better than the second-best method. Interestingly, we never achieve the highest score on HellaSwag in our tested configurations. However, no single competing method consistently achieves the best results across all evaluated use cases, while our method obtains the score in all but one test case (e.g. 11.2 points better average for the 50% compression of Llama-3.3-70B-Instruct and 6.1 points better for 35.5% compression of Llama-3.1-8B-Instruct). In general, our approach is computationally efficient and requires only forward and backward passes on a calibration dataset to compute the gradient with respect to variables (where is the number of transformer blocks). Additionally, we demonstrate that using good heuristic solvers for the CBO problem provides solutions that perform well on downstream tasks within seconds, even when it becomes infeasible to solve the CBO exactly.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Model Compression, Depth Pruning, Combinatorial Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9AGjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a method for solving the problem of optimal block removal via Constrained Binary Optimization with quadratic proxy. Specifically, the approach introduces auxiliary variables, corresponding to the presence/absence of a block, and removes the blocks leading to the minimal loss increase. Configurations are sought via combinatorial search or QUBO relaxation. The CBO method is validated on depth pruning of several language models.

Critical strengths
  • The proposed method achieves strong performance on block removal consistently outperforming the baselines on a majority of the tasks.
Critical weaknesses
  • The method appears to be challenging to scale for large models due to the need to run backpropagation to large models and search over a huge number of configurations.

  • Several important baselines are missing:

    • Pruning as a Cooperative Game [1] leveraging game-theoretic framework to solve the problem of optimal block dropping.
    • LoRP [2] using hidden state similarity to guide the pruning decision.
    • IG-Pruning [3] is input-guided block-wise pruning method that dynamically selects layer masks at inference time. Exhaustive comparison with the related work would be helpful to fully appreciate the efficacy of the proposed approach.

[1] Ding, Xuan, et al. "Pruning as a Cooperative Game: Surrogate-Assisted Layer Contribution Estimation for Large Language Models." arXiv preprint arXiv:2602.07804 (2026).

[2] Yun, Vincent-Daniel, et al. "Locality-Aware Redundancy Pruning for LLM Depth Compression." arXiv preprint arXiv:2605.27786 (2026).

[3] Qiao, Kangyu, Shaolei Zhang, and Yang Feng. "Ig-pruning: Input-guided block pruning for large language models." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Questions for authors
  • Have you tested the validity of the quadratic regularization? Is the loss increase predicted by a quadratic model correlated with the actual loss increase on some calibration dataset?

  • What is the runtime of the CBO method for configurations in Table 1? Do I get it right that some use exhaustive combinatorial search, whereas others (Llama-3.3-70B-Instruct) use QUBO solver? How does it compare to the baselines?

  • Would be interesting to see how far one can prune reasoning models (Qwen3.5, Gemma4, for instance) without significant performance degradation on popular reasoning benchmarks (GPQA-Diamond, AIME, MATH500, e.t.c.)

Reviewer_AU6HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper attaches a scalar gate to the attention and FFN residual branches of each transformer block. It approximates the Hessian of the loss with respect to these gates by the empirical Fisher, i.e., the averaged outer product of per-sample gradients on calibration data. Removing a fixed number of blocks then becomes a constrained binary quadratic program, whose off-diagonal terms capture the joint sensitivity of pairs of blocks. The removal set is found by exhaustive enumeration or, for the largest search space, by tabu search. No recovery training is applied. Experiments cover four models and five downstream benchmarks. The proposed method achieves the best score in 13 of the 20 model benchmark combinations and the highest average score for three of the four models. The appendix compares calibration datasets, lists the blocks removed by each method, and reports measured inference throughput and latency.

Critical strengths
  1. Joint block selection is clearly valuable. The quadratic objective is only N × N, where N is the number of blocks, so it is computed once and can be reused for any removal budget and any solver. More importantly, the off-diagonal terms matter.

  2. A clear advantage over the baselines at high compression. After half of the blocks of Llama-3.3-70B are removed, CBO reaches 76.9 on MMLU, versus 54.0 for the method BI. Its average score is 53.4, versus 42.2 for the next-best method. The value of CBO therefore lies in retaining more capability at a given budget.

Critical weaknesses

1、Eq. (3) in Section 3 drops the first-order term on the grounds that the weights are close to optimal. However, weights that are near-optimal for the original training objective do not imply that the average gradient of the calibration loss with respect to the gates is negligible. A complete local approximation should include both the linear and the quadratic term.

2、Figures 2–3 show only 11 configurations, all for a single model at a single compression rate: the 4 lowest-energy states, 3 random states and the 4 highest-energy states. This supports using the energy to identify a promising low-energy region. It is not enough, however, to establish a strong correlation in general. In particular, it says little about how well the energy ranks candidates within the low-energy region.

3、The main experiments report only one compression rate per model and no variation across calibration sampling seeds.

Questions for authors

1、On the actual calibration data, how large is the first-order term relative to the second-order term?

2、If the linear term is kept, do the selected removal sets and the downstream results change noticeably?

3、For the same models and removal budgets, what results do you obtain by (a) removing the last M blocks and (b) selecting the best contiguous range of blocks by calibration loss?

4、Does the main advantage persist under different calibration sampling seeds and different removal budgets?

4 comments

Weakness 1: Please see our responses to Q1 and Q2.

Weakness 2: more low-energy configurations

We have now extended the analysis to the 10 lowest-lying states for the Llama-3.1-8B-Instruct model when removing 12 blocks. The results show that the low-energy combinations are generally high-quality solutions with strong benchmark performance (considering that we have an aggressive compression ratio where other methods often collapse to random).

Ideally, we would like the energy to be a perfect cheap proxy for the average benchmark scores. However, it is only an approximate proxy (after all, there is a highly non-trivial relation ship between the calibration dataset and different downstream tasks). As such, the ground state is not guaranteed to have the best average score, and we do not claim it (see Fig. 3 of the paper and the related discussion). The energy is nevertheless a robust approximate proxy, since good solutions are found in the low-energy spectrum only (as far as we can check; we cannot check an exponentially large number of combianations).

We restrict ourselves to the ground state in the paper to remove any selection bias. However, we see that nearby excited states might score better on average (in this case, the 7th excited state has the highest score). We believe this is in fact a significant strength of our method, as it can identify multiple promising compression configurations compared to other methods, which tend to provide only a single option.

RankEnergyHellaSwagWinoGrandeARC-CMMLUBBHAvg
Original—59.873.753.668.351.161.3
00.00411441133.060.234.159.939.145.3
10.00513371033.758.633.429.933.337.8
20.00540391035.961.231.034.034.539.3
30.00541387635.360.432.732.735.539.3
40.00543969134.959.429.525.434.136.7
50.00555889933.859.634.365.343.347.3
60.00556599934.157.530.739.933.339.1
70.00560779234.461.434.667.046.348.8
80.00569575436.062.532.241.638.442.1
90.00584755031.357.526.232.332.936.0

Weakness 3 (also see the answer to Q4): more compression rates and seeds

In the table below, we show another compression rate for Llama-3-70B, and two new models not in the paper: Qwen3-4B and Llama-2-13B-Chat. CBO achieves the highest average score in all instances. There is also a recurrent gap for MMLU, where CBO often achieves much higher scores than the other methods.

Model# brMethodMMLUHellaSwagWinoGrandeARC-CBBHAvg
Llama-3.3-70B-Instruct (80)32CBO:076.639.068.539.851.455.1
BI59.348.074.536.553.454.3
Norm Ratio33.544.363.531.832.841.2
Llama-2-13B-Chat (40)16CBO:0 (tabu)50.541.161.531.438.844.6
BI45.737.464.232.835.343.1
Norm Ratio51.335.058.633.238.643.3
Qwen3-4B (36)6CBO:064.039.559.736.946.449.3
BI47.740.959.234.544.545.3
Norm Ratio26.235.652.125.330.934.0
Qwen3-4B (36)8CBO:044.635.656.835.034.741.3
BI31.040.155.329.941.739.6
Norm Ratio25.233.651.523.330.832.9

Q1: first-order term

We have investigated the contribution of the first-order term. First, we note that it is not negligible and, on the contrary, has a larger contribution than the second-order term.

ModelFirst order −x·df½ Σ H_ii x_i½ Σ_{i≠j} H_ij x_i x_jE_totE_cbo = xᵀHx−x·df / (½ E_cbo)
Falcon3-7B (−8)−0.01760.0251−0.0143−0.00680.0215−1.64
Llama-3.1-8B (−12)−0.02100.0278−0.0257−0.01900.0041−10.24
Llama-3.3-70B (−40)−0.00630.0722−0.0699−0.00400.0046−2.74
Qwen3-14B (−12)−0.02270.0956−0.0868−0.01390.0177−2.57

Furthermore, we show that considering only the first-order term leads to significantly worse results.

Model# brMethodMMLUHellaSwagWinoGrandeARC-CBBHMean
Falcon3-7B-Instruct8CBO (paper)45.037.861.436.640.944.3
First-order term only: −x·df42.638.657.829.933.540.5
Llama-3.1-8B-Instruct12CBO (paper)59.933.060.234.139.145.3
First-order term only: −x·df22.925.949.321.829.930.0
Llama-3.3-70B-Instruct40CBO, tabu (paper)76.934.969.037.548.853.4
First-order term only: −x·df24.025.648.920.128.929.5
Qwen3-14B12CBO (paper)71.834.759.335.847.449.8
First-order term only: −x·df25.737.351.824.529.933.8

We will update the revised version with this analysis.

Q2: first-order term + second-order term

As shown above, although the first-order term is not negligible, including it together with the second-order term also degrades performance on Qwen3-14B. The comparison below shows the results with and without the first-order term:

# Blocks removedMethodMMLUHellaSwagWinoGrandeARC-CBBHAverage
12CBO (without first-order term)71.834.759.335.847.449.8
12CBO (with first-order term)25.436.350.125.231.333.7

The non-negligible magnitude of the first-order term is consistent with the findings of Rosenberg et al. (2024), who proposed introducing an additional hyperparameter to control its contribution in weight pruning. However, our experiments show that omitting this term yields better downstream performance and avoids introducing an additional hyperparameter requiring tuning. In the above experiments, the coefficient of the first-order term was 1.

Q3: last M blocks and consecutive solutions

The table below shows the results for removing the last M blocks of each model. First, our results demonstrate that removing the last blocks is a reasonable simple baseline for some models but leads to severe performance degradation for others; thus, its success is more related to specific model families: For Falcon3-7B-Instruct and Qwen3-14B, this solution severely degrades model performance (e.g. collapsing MMLU to its random baseline), but it works well for the Llama family.

For Llama3-8B, CBO has in fact converged to that remove-last-M solution. For Llama3-70B, it almost does the same, the only difference being the removal of block 29 instead of 42. The two solutions are very close in the benchmarks, with the consecutive solution being slightly better (0.5 points on average). We would like to point out, however, that none of the other block removal algorithms that we compare with were able to get close to this Llama baseline.

This highlights that while selecting the last M blocks might be a quick heuristic to check, it is not a good approach in general. A robust algorithm is needed that is able to recover this baseline (or at least get a comparable solution) when it works, and to ignore it when it doesn't work, which is exactly what the CBO approach does.

Regarding the best consecutive solution, we take it that the reviewer means the consecutive-block solution with the lowest energy. This turns out to be just the last-M block solution for all the models shown except Falcon3-7B-Instruct (see below). We also tried restricting block influence (BI) to the best consecutive block, but find that this does not achieve particularly strong results.

Model# brMethodRemovedMMLUHellaSwagWinoGrandeARC-CBBHMean
Falcon3-7B-Instruct8CBO (paper)13, 15, 20, 21, 23–25, 2745.037.861.436.640.944.3
CBO consecutive6–1324.341.055.528.430.836.0
BI (paper)11–17, 2430.549.057.841.834.242.6
BI consecutive10–1728.548.556.739.332.441.1
Remove last 820–2725.332.863.231.131.036.7
Llama-3.1-8B-Instruct12CBO (paper)20–3159.933.060.234.139.145.3
BI (paper)18–2938.433.256.728.335.638.4
BI consecutive18–2938.433.256.728.335.638.4
Remove last 1220–3159.933.060.234.139.145.3
Llama-3.3-70B-Instruct40CBO, tabu (paper)29, 40, 41, 43–7976.934.969.037.548.853.4
BI (paper)37–7654.028.161.328.834.741.4
BI consecutive35–7426.239.263.134.934.539.6
Remove last 4040–7977.734.670.236.949.953.9
Qwen3-14B12CBO (paper)2, 26, 30–3971.834.759.335.847.449.8
BI (paper)2, 3, 22–24, 31–3761.639.858.235.043.647.6
BI consecutive26–3735.132.953.126.332.435.9
Remove last 1228–3923.632.661.232.433.336.6

Q4: seeds and sample sizes We have now prepared the calibration dataset with three different seeds and the sample sizes 1024, 2048, and 4096 and ran CBO, block influence, and norm ratio for Llama3.1-8B instruct. All methods got the same block configuration in all instances, indicating that the methods are quite robust against different random seeds and sample sizes.

Dear Reviewer AU6H,

Thank you again for your detailed review. As the discussion period is drawing to a close, we wanted to kindly check whether our response has addressed your concerns. In your review, you raised four questions that you indicated would inform your assessment.

To make it easier to navigate our response:

  • Part 1 addresses Weaknesses 2 and 3, while Weakness 1 is discussed together with Questions 1 and 2.
  • Part 2 addresses Questions 1 and 2.
  • Part 3 addresses Question 3 and 4.

We would be very happy to clarify anything that remains unclear or insufficiently addressed. We would greatly appreciate any further feedback you may have.

Reviewer_kxSrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper formulates LLM compression by Transformer block removal as a constrained binary optimization problem. It introduces block-level gating variables and constructs a quadratic loss proxy using an empirical-Fisher approximation computed from calibration gradients. Under a fixed block-removal budget, the method jointly selects blocks for deletion using exact enumeration or tabu search. Evaluation includes analyses of proxy energy versus downstream performance, comparisons between exact and heuristic solutions, and experiments on four models: Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct, Qwen3-14B, and Falcon3-7B-Instruct. Across five downstream benchmarks, the method achieves the highest average score among the compared methods in three of the four main model configurations.

Critical strengths
  1. The formulation explicitly accounts for interactions between block-removal decisions. The quadratic objective jointly selects a subset of blocks under a fixed removal budget, rather than ranking blocks independently. This is a meaningful feature because the effect of removing one block can depend on which other blocks are removed. The diagonal ablation provides evidence that including these interactions improves performance in several evaluated settings.

  2. The optimization separates model-dependent calibration from inexpensive candidate search. The surrogate is constructed from first-order calibration gradients, and the resulting matrix can be reused for different removal budgets. Once constructed, candidate configurations can be evaluated without running the language model. The solver experiments further show that tabu search finds the exact minimum in the tested tractable cases within seconds, supporting its use when exhaustive enumeration becomes impractical.

  3. The experiments demonstrate downstream benefits across several model families and scales. Evaluation covers four models, including a 70B model, and five downstream benchmarks, with comparisons against block-removal methods including EvoPress. CBO achieves the highest average score in three of the four main model configurations and the highest MMLU score in all four. These results provide evidence that the formulation produces useful compressed models beyond the proxy objective itself.

Critical weaknesses
  1. The reliability of the quadratic proxy needs stronger validation. Section 3 omits the first-order Taylor term without demonstrating approximate stationarity of the calibration loss with respect to the block gates. Removing a block also involves a unit gate change, raising questions about the local approximation. Figures 2–3 provide useful comparisons between selected low-energy, random, and high-energy configurations, but limited evidence about ranking competitive low-energy candidates. Thus, the reliability of minimizing this proxy to select high-quality removal configurations remains insufficiently established.

  2. Robustness across calibration choices and compression levels remains unclear. The main comparisons evaluate one removal budget per model. Appendix A examines calibration data only for Qwen3-14B with 12 blocks removed, where the ground-state average score varies from 49.8 with OpenHermes-2.5 to 48.3 with LMSYS-Chat-1M and 34.6 with WikiText. Although the selected main configurations demonstrate benefits, this sensitivity and the limited coverage of removal budgets leave uncertainty about how consistently the method retains its advantage across operating conditions.

  3. The computational cost of the complete compression pipeline is insufficiently quantified. Section 4.2 demonstrates that tabu search can solve the constructed binary problem within seconds, but the manuscript does not report the time required to collect calibration gradients and construct the empirical-Fisher matrix. Although this matrix can be reused across removal budgets, its initial construction is part of the compression cost. Appendix C measures inference performance after compression rather than this preparation cost. Consequently, the overall cost of obtaining compressed models, and the quality–compute trade-off relative to competing methods, remain unclear.

Questions for authors
  1. Abstract consistency. Please reconcile the compression rates and MMLU improvements in the abstract with Table 1. The table associates Llama-3.1-8B-Instruct with 37.5% block removal and a 21.5-percentage-point MMLU improvement over the next-best method, and Llama-3.3-70B-Instruct with 50% removal and a 22.9-point improvement. These associations appear reversed in the abstract. The later reference to 35.5% compression for Llama-3.1-8B should also be corrected to 37.5%.

  2. Table presentation. Table 1 appears to lack a closing horizontal rule after its final row. Adding one would improve visual consistency. It would also help to state explicitly in the caption what the boldface entries denote.

  3. Proxy reliability. Could you evaluate how well proxy energy ranks downstream performance within a larger set of low-energy configurations? This would help establish whether minimizing the proxy reliably selects better compressed models beyond distinguishing low-energy configurations from random or high-energy ones.

4 comments

Weakness 1: Please see our response to Q3.

Weakness 2: calibration choices and compression levels
To address the dependence on the dataset, we have performed the CBO calculations for Llama-3.1-8B-Instruct and Falcon3-7B-Instruct using WikiText-2 and lmsys-chat-1m. As is the case for Qwen3-14B shown in the paper, OpenHermes remains the best dataset on average, while lmsys-chat-1m remains second best. We again observe that the low-quality WikiText-2 reduces MMLU to its random baseline (or close to it).

ModelCalibration# blocks removedMMLUHellaSwagWinoGrandeARC-CBBHMean
Llama-3.1-8B-InstructOpenHermes-2.51259.933.060.234.139.145.3
Llama-3.1-8B-Instructlmsys-chat-1m1247.733.061.032.030.840.9
Llama-3.1-8B-InstructWikiText-21225.834.253.225.929.133.6
Falcon3-7B-InstructOpenHermes-2.5660.242.563.140.447.750.8
Falcon3-7B-Instructlmsys-chat-1m657.545.761.642.743.350.2
Falcon3-7B-InstructWikiText-2631.945.758.134.131.340.2

In addition, we compare CBO to Norm Ratio and Block Influence (BI) for one more compression ratio for Llama3-70B, Llama2-13B, and two instances of Qwen3-4B beyond what is shown in the paper. CBO achieves the highest average score in all instances. There is also a recurrent gap for MMLU, whereby CBO achieves much higher scores than the other methods.

Model# brMethodMMLUHellaSwagWinoGrandeARC-CBBHAvg
Llama-3.3-70B-Instruct (80)32CBO:076.639.068.539.851.455.1
BI59.348.074.536.553.454.3
Norm Ratio33.544.363.531.832.841.2
Llama-2-13b-chat (40)16CBO:0 (tabu)50.541.161.531.438.844.6
BI45.737.464.232.835.343.1
Norm Ratio51.335.058.633.238.643.3
Qwen3-4B (36)8CBO:044.635.656.835.034.741.3
BI31.040.155.329.941.739.6
Norm Ratio25.233.651.523.330.832.9
Qwen3-4B (36)6CBO:064.039.559.736.946.449.3
BI47.740.959.234.544.545.3
Norm Ratio26.235.652.125.330.934.0

Weakness 3: computational cost We note that almost all block removal methods rely on a calibration forward pass; the only difference is that CBO needs the gradient for a few variables. We pick Block Influence (BI) as a comparison method and show the time it takes to compute the forward pass in both cases. It costs indeed about 2x more for CBO, but in absolute terms the difference is only on the order of minutes, even for a model as large as 70B.

ModelGPUsMethodForward pass (2048 samples)
Llama-3.1-8B1Block Influence70 s
Llama-3.1-8B1CBO2.4 min
Qwen3-14B1Block Influence1.9 min
Qwen3-14B1CBO4.3 min
Llama-3.3-70B2Block Influence5.4 min
Llama-3.3-70B2CBO10.6 min

Q1. We appreciate the Reviewer pointing out this typo. We will correct it in the revised version.

Q2. We also appreciate the Reviewer pointing out that the bottom horizontal line is missing. We will correct it in the revised version.

Q3 and Weakness 1: proxy reliability
We have extended the analysis to the 10 lowest-lying states for the Llama-3.1-8B-Instruct model when removing 12 blocks. The low-energy combinations are generally high-quality solutions with strong benchmarks (considering that we have an aggressive compression ratio where other methods often collapse to random). We emphasize we do not claim that the ground state has to be the optimal solution across all benchmarks. We restrict ourselves to the ground state in the paper to remove any selection bias. However, we see that excited states might score better on average (in this case, the 7th excited state). We believe this is a significant strength of our method, as it can identify multiple promising compression configurations compared to other methods that tend to provide only a single option.

RankEnergyHellaSwagWinoGrandeARC-CMMLUBBHAvg
Original—59.873.753.668.351.161.3
00.00411441133.060.234.159.939.145.3
10.00513371033.758.633.429.933.337.8
20.00540391035.961.231.034.034.539.3
30.00541387635.360.432.732.735.539.3
40.00543969134.959.429.525.434.136.7
50.00555889933.859.634.365.343.347.3
60.00556599934.157.530.739.933.339.1
70.00560779234.461.434.667.046.348.8
80.00569575436.062.532.241.638.442.1
90.00584755031.357.526.232.332.936.0
Reviewer_kxSrOpenReview ↗

Response to the Authors

Thank you for the additional experiments and runtime analysis. These results address some of my concerns, and I am raising my soundness score from 2 to 3.

First, we would like to thank the Reviewer for their careful reading of our manuscript and for their detailed and constructive feedback. We are pleased that our rebuttal and additional experiments have addressed some of the Reviewer’s concerns and led them to increase their soundness score. However, we would greatly appreciate further clarification on which concerns the Reviewer feels remain insufficiently addressed, particularly those that continue to affect their overall assessment of the paper. We would be happy to provide further explanations or evidence to address these remaining concerns.

We would also like to highlight that, in response to Reviewer 9AGj’s feedback, we have further extended our experimental evaluation and added comparisons against three additional state-of-the-art methods, all of which our approach outperforms. These new results have been incorporated into the revised manuscript and provide further evidence supporting the effectiveness of our approach.

In light of these additional results and the concerns addressed in our rebuttal, we would kindly encourage the Reviewer to reconsider their overall evaluation and score, beyond the soundness assessment alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.