acceptodds
Under review as a conference paper at ICLR 2027

Compress Locally, Predict Globally: Composable Margin Statistics for Estimating Compressed LLM Accuracy

Abstract

We introduce a statistical model that predicts the task accuracy of compressed language models without constructing and evaluating each candidate configuration. For each block and compression option, we summarize changes in answer margins (the score differences between competing answers) by a scaling coefficient and a residual variance. We estimate these statistics by compressing one block at a time, then reuse and combine them across configurations. We use the combined statistics to parameterize a Gaussian score model. For each example in a labeled reference set, we use the uncompressed model's scores to estimate the probability that the compressed model selects the correct answer. We average these probabilities to predict task accuracy. Across exhaustively evaluated MMLU quantization and pruning allocation spaces, prediction mean absolute error (MAE) ranges from 0.61 to 3.83 percentage points (pp). Across five K2-Horizon-7B quantization spaces, composing margin statistics achieves a mean per-space prediction MAE of 1.10 pp, compared with 1.55 and 1.42 pp for additive and multiplicative accuracy composition. Comparisons with covariance-based and empirical-residual predictors further support the accuracy estimates from the fitted Gaussian model. GPTQ prediction MAE remains within 0.67-1.19 pp across five checkpoints from 7B to 65B and reaches 0.90 pp on 24 presampled individual-layer allocations. The same prediction rule extends to BoolQ, HellaSwag, WinoGrande, and encoder-based AG News classification. As an application, maximizing predicted accuracy selects the best measured configuration in the complete Gemma-31B quantization space and improves pruning allocation over OWL under matched budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model compression, quantization, pruning, performance prediction, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XVpNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper composes single-block margin statistics to predict compressed-model accuracy on a labeled reference set. Experiments cover several models, compression methods, and allocation tasks.

Critical strengths
  1. Margin statistics retain information that scalar block accuracies lose, making the approach well motivated and reusable.
  2. The broad evaluation, exhaustive smaller spaces, and explicit fit/selection/test separation support the estimator. I lean toward weak acceptance.
Critical weaknesses
  1. Cross-block interactions need better validation. An effect measured with all other blocks dense need not remain valid after other blocks are compressed. Appendix D’s answer-score covariance analysis does not directly address this.
  2. Low average prediction error does not guarantee good allocation choices. The 9.1-billion-configuration experiment evaluates only 24 presampled allocations, not an optimized selection. The cost comparison should also let direct-search baselines reuse compressed-option caches.
Questions for authors
  1. How large are the errors when composing two or more block effects, particularly under heavier compression?
  2. What selection regret and total cost does a prospectively optimized allocation achieve against search baselines with the same caching and evaluation budget?
1 comment

We appreciate the reviewer's insightful comments. Here, we answer each of the questions individually. We will update our manuscript accordingly.

Q1 (and W1). How large are the errors when composing two or more block effects, particularly under heavier compression?

The errors stay small. Across 48 jointly compressed configurations, prediction MAE is 0.93 pp on K2-Horizon-7B and 1.14 pp on Gemma-4-12B PT, and it remains below 3 pp at 70% block sparsity.

We compressed two, three, or all blocks jointly at 30%, 50%, and 70% block sparsity and compared measured accuracy with the prediction composed from single-block measurements. Values are prediction MAE (pp).

ModelBlock sparsityTwo blocksThree blocksAll blocks
K2-Horizon-7B30%0.730.520.35
K2-Horizon-7B50%0.300.581.31
K2-Horizon-7B70%2.500.860.58
Gemma-4-12B PT30%0.230.230.29
Gemma-4-12B PT50%1.050.962.36
Gemma-4-12B PT70%1.642.791.00

We also measured the cross-block interaction directly. On 20 two-block configurations, the residual standard deviation of the jointly compressed model is 0.88–0.95 of the value given by the composition rule (Eq. 5). Effects measured with the other blocks dense therefore remain valid after those blocks are compressed. The complete-space results in Table 1 support the same conclusion, since every allocation there composes six block effects.

We will add these results to the appendix.

Q2 (and W2). What selection regret and total cost does a prospectively optimized allocation achieve against search baselines with the same caching and evaluation budget?

In the 9.1-billion-configuration space of Section 4.7, the optimized allocation reaches 69.12% test accuracy (the dense model reaches 69.98% and an evenly spaced allocation at the same budget reaches 64.30%) with zero selection regret, 3.3–3.4 pp above greedy and evolutionary search under the same compressed-option cache and evaluation budget.

Our predictor scores 35,392 candidate allocations without compressed-model execution and selects one before any compressed model is evaluated. All methods share the same compressed-option cache and receive 72 compressed-model evaluations; we also run search with 100 evaluations to match our total time. Selection regret is relative to the best measured allocation.

MethodEvaluationsTime (s)Test accuracy (%)Selection regret (pp)
Ours72116969.120.00
Greedy search7286165.723.40
Evolutionary search7285965.843.28
Greedy search100118966.023.10
Evolutionary search100118066.113.02

Our allocation is better at both equal evaluation budget and equal time.

The parameter library also reduces cost when several budgets are requested. For three K2-Horizon-7B GPTQ budgets (3.33, 3.50, and 3.67 bits), one library of 12 single-block measurements serves all three, while search needs 12 evaluations per budget even with the shared cache, warm starts, and reuse of earlier evaluations. Total time is 382 s versus 632 s and 646 s, about 40% less, with selection regret of 0, 0, and 0.70 pp.

We will add the optimized-selection result to Section 4.7 and the cost comparison to Appendix F.

Reviewer_yeRWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper aims to predict task accuracy for layer-block compression allocations (W3/W4 under GPTQ, AWQ or rotated GPTQ, or Wanda sparsity levels) without running a fully compressed model. The basic idea is to compress each block on its own and regress the change in the dense model's Top-2 answer margin to get a label-free scale and residual variance per block. Then compose the blocks by multiplying their scales and summing their variances, with each rescaled by the scales of later blocks. Finally, this turns into an MC estimate of accuracy over dense-model scores on held-out items.

Validation covers six- and eight-block MMLU spaces on six decoders ranging from 7B to 65B. The paper measures every allocation except for K2-65B, with reported errors from 0.6 to 2 pp on K2-7B, and a jump to about 4 pp on Gemma-4-12B Wanda. The paper claims this beats simply composing single-block accuracies.

Beyond MMLU, there are resutls on three other multiple-choice tasks provided, an individual-layer allocation space, and an AG News encoder. The allocation experiments pick a Gemma-31B configuration, and separately compare a pruning selector against OWL on K2-7B and Gemma-12B.

Critical strengths
  1. The evaluation is carefully controlled. Nearly every allocation in each blockwise space is actually measured, the block statistics are fit without labels on one split, and selection uses a second, with the predictions for the later spaces saved before the compressed models were evaluated.

  2. Where a space has real spread, the method recovers the signal cheaply, off 12 single-block runs for a six-block W3/W4 space. On the 7B model the predicted and measured rankings agree well, the method correctly flags the allocations that collapse to chance, and the item-level probabilities are well calibrated (Appendix D).

Critical weaknesses
  1. The predictions are compressed and biased upward. Across most spaces the predicted gaps between allocations come out too small, as little as half the measured gaps, and the level leans optimistic. I am reading this off the scatter plots (Figs 2, 5, 6, 7), since the paper gives no signed errors, no confidence intervals on any MAE or ρ\rho, and no rank correlation for the 65B model. The bias is clearest on Gemma-12B Wanda, where every allocation is over-predicted by a few points. This matters because anyone who uses the predicted level to decide whether a budget is acceptable gets numbers that are several points optimistic there, and the ranking itself weakens at 65B. The ranking does hold at 31-32B (ρ\rho = 0.78 and 0.67), which is enough for the Gemma-31B selector to find the best allocation, as far as I understand. I would want the signed error and the predicted vs measured slope per space, confidence intervals on MAE and ρ\rho, and ρ\rho for K2-65B.

  2. The composition rule needs isolating, it is never compared against an additive composition of the same margin statistics, so its contribution cannot be separated out. Against the scalar baselines the method wins on four of five spaces (tab 2), narrowly at 3.50 bit GPTQ, and loses at 3.33, with no paired uncertainty reported, and the allocation experiment (Tab 4) compares against an OWL baseline of which the sparsity area is much narrower than the range the selector operates in.

Questions for authors
  1. Were the complete GPTQ/AWQ/rotated GPTQ candidates built sequentially, or from dense-model inputs? If they used dense-model inputs, what happens to the error once they are compressed sequentially? For pruning, the table 4 allocations are already pruned with sequential Wanda, so reporting predicted accuracy next to the measured values would answer this directly?

  2. The control I would most want is an order-free additive version of the composition, run through the same score model, with paired confidence intervals on the MAE difference. If the paper's rule does not clearly beat it, the additive rule is the fairer baseline and the paper should report against it.

  3. For table 4, it would help to add a uniform Wanda row at matched sparsity, and an OWL run actually tuned to the selector's range.

3 comments

We appreciate the reviewer's insightful comments. Here, we answer each of the questions individually. We will update our manuscript accordingly.

W1. Signed errors, predicted-vs-measured slopes, confidence intervals, and ρ\rho for K2-65B

The requested statistics for every space in Figs. 2, 5, 6, and 7 are below. Signed error is predicted minus measured accuracy; slope is from regressing predicted on measured accuracy. Brackets are 95% paired bootstrap intervals over test items.

SpaceNSigned error (pp)SlopeMAE (pp)Spearman ρ\rho
K2-Horizon-7B, GPTQ 3.50 bits20+0.28 [−0.64, 1.17]0.461.20 [1.02, 1.85]0.877 [0.682, 0.902]
K2-Horizon-7B, GPTQ 3.33 bits15+0.96 [−0.05, 1.97]0.331.80 [1.44, 2.55]0.861 [0.729, 0.907]
K2-Horizon-7B, GPTQ 3.67 bits15+0.08 [−0.87, 0.97]0.500.91 [0.81, 1.59]0.953 [0.785, 0.968]
K2-Horizon-7B, AWQ 3.50 bits20+0.57 [−0.45, 1.54]0.600.94 [0.73, 1.75]0.922 [0.691, 0.934]
K2-Horizon-7B, Rotated GPTQ 3.50 bits20−0.12 [−0.89, 0.63]0.600.63 [0.66, 1.27]0.820 [0.488, 0.848]
K2-Horizon-7B, Wanda 60%21−0.55 [−2.04, 0.99]1.021.93 [1.61, 2.92]0.865 [0.799, 0.926]
K2-Horizon-7B, individual-layer24−0.79 [−1.78, 0.19]0.450.97 [0.82, 1.85]0.686 [0.416, 0.770]
Vicuna-7B, GPTQ 3.50 bits12−0.24 [−1.20, 0.72]0.520.66 [0.66, 1.49]0.867 [0.469, 0.905]
Gemma-4-31B PT, GPTQ 3.50 bits20+0.94 [0.29, 1.57]0.390.98 [0.63, 1.61]0.778 [0.426, 0.851]
K2-Horizon-32B, GPTQ 3.50 bits20+0.60 [−0.14, 1.29]0.330.76 [0.63, 1.37]0.674 [0.320, 0.774]
K2-65B, GPTQ 3.50 bits16+0.10 [−0.63, 0.81]0.200.68 [0.58, 1.23]0.421 [0.019, 0.686]
Gemma-4-12B PT, Wanda 55%28+3.83 [2.50, 5.16]0.943.83 [2.69, 5.16]0.956 [0.904, 0.973]

Predictions here use deterministic quadrature in place of Monte Carlo sampling, so MAEs differ from the paper's by at most 0.07 pp. Bootstrap MAE is biased upward, so an interval can sit slightly above its point estimate.

Level. The predictions are not systematically optimistic: the signed error is below 1 pp in 11 of the 12 spaces, and its interval includes zero in 10. The exception is Gemma-4-12B PT with Wanda, as the reviewer observed. There the error is a constant shift, with slope 0.94 and ρ\rho = 0.956, so the gaps between allocations are predicted correctly. Estimating one offset from one to three measured allocations on the selection split reduces prediction MAE on the remaining allocations from 3.9–4.1 pp to 1.9–2.7 pp.

Gaps between allocations. The slopes below one occur in the quantization spaces, where measured accuracy spans only 3–11 pp. In the two pruning spaces, where it spans about 30 pp, the slopes are 1.02 and 0.94. Rankings are preserved in both cases.

K2-65B. ρ\rho is 0.421 on the 16 evaluated allocations (Section 4.6). Their measured accuracies span only 3.2 pp, and the prediction MAE is 0.68 pp.

We will add the signed errors and ρ\rho for K2-65B to the main results, and the full table to the appendix.

Q1. Were the complete GPTQ/AWQ/rotated GPTQ candidates built sequentially, or from dense-model inputs? What happens to the error under sequential compression?

Each complete candidate in Tables 1 and 2 was compressed as a whole model with llm-compressor, not assembled from blocks compressed with dense-model inputs. Only the single-block measurements and the individual-layer allocations in Section 4.7 use dense-model inputs.

Prediction error stays low under sequential compression. With the predictions held fixed:

  • Eight K2-Horizon-7B configurations (GPTQ at 3.50 and 3.33 bits, AWQ, and rotated GPTQ) rebuilt with sequential compression and error propagation give 0.71 pp prediction MAE.
  • Four individual-layer allocations from the 36-layer setting rebuilt with sequential GPTQ give 1.31 pp prediction MAE, and the allocation with the highest predicted accuracy is the best measured one.

For pruning, the predicted and measured accuracies of the Table 4 allocations are:

ModelPredicted (%)Measured (%)
K2-Horizon-7B, 50%63.5065.26
Gemma-4-12B PT, 60%64.2659.56

The Gemma error is the level shift discussed in W1. We will state the construction protocol of each experiment in the appendix.

Q2 (and W2). An order-free additive composition of the same margin statistics, run through the same score model

Though the additive rule is clean, straightforward, and elegant, the composition rule clearly beats it in practical.

The additive rule uses the same single-block (αb,σb)(\alpha_b, \sigma_b), the same dense-model scores, and the same Gaussian score model (Eq. 2). Only the composition changes: α=1+∑b(αb−1)\alpha = 1 + \sum_b (\alpha_b - 1) and σ2=∑bσb2\sigma^2 = \sum_b \sigma_b^2.

SpacesOurs, MAE (pp)Additive margin composition, MAE (pp)
Five K2-Horizon-7B quantization spaces of Table 21.091.38
16 complete MMLU allocation spaces (412 configurations)1.665.14

Our rule has lower prediction MAE in each of the five Table 2 spaces, including GPTQ at 3.33 bits. Over the 16 complete spaces, the paired MAE difference is −3.49 pp, with 95% interval [−3.96, −2.84]. We will add this control to Table 2.

Q3 (and W2). A uniform Wanda row at matched sparsity, and an OWL tuned to the selector's range

With both baselines added, the gains over OWL remain, including when our predictor searches only a narrow range of block sparsities around the target.

We tune OWL on the selection split over threshold multiplier M∈3,5,7M \in \\{3, 5, 7\\} and nine values of λ\lambda from 0 to 0.96, wide enough for its layerwise sparsities to cover the selector's 30–70% range. The pruning procedure and matched sparsity budgets follow Section 4.8. Values are MMLU test accuracy (%).

Model, sparsityBlock sparsity rangeUniform WandaTuned OWLOurs
K2-Horizon-7B, 50%40–60%62.1162.1167.37
K2-Horizon-7B, 50%30–70%62.1162.1166.75
Gemma-4-12B PT, 60%50–70%37.0237.0259.56
Gemma-4-12B PT, 60%30–70%37.0233.4262.46

All four gains over tuned OWL have 95% intervals above zero after correcting for the four comparisons. The 5.26 and 22.54 pp gains in the narrow ranges show that the gains do not come from a wider sparsity range. We will add uniform Wanda and tuned OWL to the comparison in Section 4.8.

I appreciate the detailed response and additional experiments. The additive composition control and the clarification regarding sequential construction adequately address my primary concerns, and I concur that the level bias is confined to the Gemma models.

Some concerns remain. Slopes ranging from 0.20 to 0.60 indicate that the predicted gaps between allocations are compressed, and ranking quality deteriorates at larger scales (ρ=0.42\rho = 0.42 on K2-65B). As the Gemma offset correction relies on measured allocations, it should be framed as an optional calibration step. I recommend that the final version explicitly report these slopes and the K2-65B rank correlation in the main text.

Overall I will keep my score and lean toward acceptance.

Reviewer_AwwoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a statistical method to predict the task accuracy of a mixed-precision/sparsity compressed LLM without building or running it. By compressing each block individually and measuring the change in the dense model's answer margins, we can obtain a scale and a noise level for each block. These per-block characteristics are combined and fed into a Gaussian score model, from which we can then predict accuracy. Empirical results are demonstrated on various model sizes between 7B and 65B, using several quantization and pruning methods, showing comparable performance on multiple-choice benchmarks with base dense model.

Critical strengths

The paper's margin-based method provides a simple and interpretable link between local compression effects and task accuracy. The evaluation is more exhaustive and convincing than typical proxy-correlation studies. The writing is easy to follow. Once the library is built, the method provides a way to cheaply evaluate compression candidates.

Critical weaknesses

There are several questions I would like the authors to address.

  1. Does the method beat a simple additive proxy? Predicted accuracy depends on a configuration only through σ/α, which is close to a sum of per-block variances. How do rankings from a plain sum of per-block σ² (or summed per-block KL) compare with your full prediction?

  2. In this paper, the predictor searches block sparsities from 30% to 70%, while OWL with λ=0.08 can only move about +/- 8 points around the target. How do uniform-sparsity Wanda and an OWL tuned on the selection split perform, or with λ wide enough to cover the same range? I believe a better tuned OWL might give better performance.

  3. How generalizable this method on auto-regressive task? The predictor needs fixed candidate answers and teacher-forced likelihoods. Can predicted multiple-choice accuracy rank allocations by generation quality (for example on GSM8K or code)?

Questions for authors

See weakness.

1 comment

We appreciate the reviewer's insightful comments. Here, we answer each of the questions individually. We will also update our manuscript with revisions.

Q1. Does the method beat a simple additive proxy, i.e., a plain sum of per-block σ2\sigma^2 or summed per-block KL?

Yes for both. The plain sum of per-block σ2\sigma^2 ranks allocations much worse than our full prediction. Summed per-block KL matches our rankings on the original models only when computed on the task's answer scores, and it falls behind on additional models in both allocation selection and accuracy prediction.

On σ/α\sigma/\alpha: as noted in Section 3.2, decision probabilities depend on this ratio for α>0\alpha > 0. Under the composition rule in Eq. 5,

[σ(a)α(a)]2=∑b=1Lσb,ab2(∏j≤bαj,aj)2\left[\frac{\sigma(a)}{\alpha(a)}\right]^2 = \sum_{b=1}^{L} \frac{\sigma_{b,a_b}^2}{\left(\prod_{j\le b}\alpha_{j,a_j}\right)^2}

so each block's residual variance is weighted by the scaling coefficients of the configuration. The ratio is close to a sum of per-block variances only when these scaling coefficients are all close to one.

We compared rankings on 16 complete MMLU allocation spaces (412 configurations), all using the same single-block measurements.

MethodSpearman ρ\rhoGap to best (pp)
Ours (full prediction)0.9260.049
Plain sum of per-block σ2\sigma^20.6226.376
Summed per-block KL, next-token (C4)0.3073.975
Summed per-block KL, answer scores0.9220.049

Here, Gap to best is the test-accuracy difference between the selected allocation and the best measured allocation in each space. Next-token KL ranks poorly. KL on the same answer scores we measure ranks similarly on these spaces, so we tested it further on three additional models (Qwen3-8B-Base, Mistral-7B-v0.3, OLMo-2-7B; 124 configurations). KL does not give predicted accuracy, so we mapped it to accuracy with a linear or isotonic fit.

OursKL(linear)KL(isotonic)
Prediction MAE (pp)3.6810.137.56
Prediction MAE (pp), after fitting to three measured accuracies2.544.674.22
Gap to best (pp)1.162.511.79

Our predictor has lower prediction MAE in both settings and selects better allocations; the paired differences in the last two rows all have 95% intervals excluding zero. We will include these comparisons in the appendix.

Q2. How do uniform-sparsity Wanda and an OWL tuned on the selection split perform? Will a better tuned OWL give better performance?

Yes, a better tuned OWL can give better performance. In three of the four comparisons below, tuning on the selection split improves OWL over the λ=0.08\lambda = 0.08 setting used in the paper (from 61.67% to 62.11% on K2-Horizon-7B and from 34.74% to 37.02% on Gemma-4-12B PT). The gains of our predictor over OWL remain after tuning, including when our predictor searches only a narrow range of block sparsities around the target.

We tune OWL on the selection split over threshold multiplier M∈3,5,7M \in \\{3, 5, 7\\} and nine values of λ\lambda from 0 to 0.96, wide enough for its layerwise sparsities to cover the same 30–70% range. In those three comparisons, the selected setting is λ=0\lambda = 0, which coincides with uniform-sparsity Wanda. The pruning procedure and matched sparsity budgets follow Section 4.8. Values are MMLU test accuracy (%).

Model, sparsityBlock sparsity rangeUniform WandaTuned OWLOurs
K2-Horizon-7B, 50%40–60%62.1162.1167.37
K2-Horizon-7B, 50%30–70%62.1162.1166.75
Gemma-4-12B PT, 60%50–70%37.0237.0259.56
Gemma-4-12B PT, 60%30–70%37.0233.4262.46

All four gains over tuned OWL have 95% intervals above zero after correcting for the four comparisons. The 5.26 and 22.54 pp gains in the narrow ranges show that the gains do not come from a wider sparsity range. We will add uniform-sparsity Wanda and tuned OWL to the comparison in Section 4.8.

Q3. How generalizable this method on auto-regressive task?

As the reviewer noted, our predictor needs fixed candidate answers (Section 3.1), so applying predicted multiple-choice accuracy to generation benchmarks is not straightforward.

Within this scope, the predictor is not limited to continuation log-likelihoods from decoder LLMs. The same composition and probability equations apply to class logits in encoder-based AG News classification (Section 4.5, Table 3), which involves no auto-regressive decoding. We leave extending margin-based accuracy prediction to generation tasks for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.