acceptodds
Withdrawn from ICLR 2027

Importance Is Not Gauge-Invariant: Measuring, Not Estimating, Unit Redundancy in LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Structured pruning ranks attention groups and FFN channels by importance proxies such as weight magnitude or activation traffic. We show these proxies do not measure the network's function: residual transformers admit per-group *gauge freedoms* — function-preserving rescalings of , , and FFN pairs — under which magnitude-based rankings reorder almost arbitrarily (Spearman –) while no output bit changes (max logit difference ). Interventional ablation rankings are exactly invariant under the bit-exact gauge on Llama-3.1-8B (rank survival ); on QK-normalized architectures the gauge is only -approximate, and the observed rank change (Qwen3: ) tracks the induced function drift. Across six backbones from five institutions, estimation-based proxies correlate with interventional importance at ; *exactly* invariant proxies (Taylor, Fisher) do no better (; a Taylor top- keep-set yields the NLL of random, vs. ) — invariance is necessary, not sufficient. A pre-registered mechanism hypothesis (QK-norm predicts proxy sign) is falsified by our own test, and gauge canonicalization does not rescue the proxies. Measurement has pitfalls of its own: on Mistral-7B's flat redundancy landscape, extreme solo-score selection is catastrophic (NLL vs. random) until one mid-budget score refresh restores safety. We therefore develop MAPS (Measured Allocation of Pruning Structure), a measurement-based pruner with a single primary hyperparameter that selects heterogeneous units (KV groups, FFN groups, whole layers) by *measured* candidate comparison. Without recovery training, at a nominal keep ratio of it retains 99.5% of zero-shot MC ( vs. ) and 99.8% of GSM8K on OLMo-3-32B. Across four backbones and three budgets, measurement-based selection tops every comparison: MAPS takes 10 of 12 backbone–budget cells (up to points over the best structured baseline), a fully-measured depth search (SLEB) takes the deepest hub cell — a dose–response in the amount of measurement — and estimation-based criteria (magnitude, Wanda-style, Taylor, FLAP-style) trail throughout, with score families backbone-dependent under a fixed allocator. An allocation-only ablation (global per-parameter greedy over the same scores, even with one mid-budget re-measurement) still trails random: *where* you remove matters as much as *what* you score, and *how much* you measure decides who wins.

Primary area:
foundation or frontier models, including LLMs
Keywords:
structured pruning, network pruning, large language models, model compression, efficient inference, importance scores, interventional ablation, gauge invariance, reparameterization symmetry, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mGzLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper study the reliability of importance scores in structured pruning for LLMs. It first argues that common proxy scores, for example: magnitude and activation traffic are unreliable because they are not gauge-invariant under function preserving reparameterizations. Then the paper demonstrates that even gauge-invariant scores (e.g., Wanda's paired aggregation, Taylor, Fisher) remain weakly correlated with causal importance, and relying purely on causal solo scores fails on flat landscapes. The authors propose MAPS, a heuristic strategy that generates a small candidate set of pruning plans and measures their Masked NLL to select the best allocation. The authors conduct extensive experiments across multiple backbones (Llama, Qwen, Mistral, OLMo, Gemma) and various budgets.

Critical strengths
  1. The paper has a lot experiments and evaluates multiple setting across various budgets.
Critical weaknesses
  1. Extremely Poor Clarity: The narrative of this paper fails to establish a logical bridge between the theoretical gauge diagnosis and the final method. This paper build initial around the necessity of "gauge invariance". Then, the paper invalidates its own point (the necessity of "gauge invariance") by proving that gauge-invariant proxies also fail. It then shifts to proposing causal solo scores, only to show they collapse on flat landscapes. This constant shifting makes the paper extremely exhausting to read.

  2. Trivial Method: The proposed MAPS algorithm is essentially a localized, brute-force Grid Search. MAPS is a straightforward heuristic that simply constructs a small set of five candidate plans and selects the best based on calibration loss. The paper fails to provide clear evidence that this specific candidate construction rule is optimal or robust compared to alternative rules under the same computational budget.

  3. High Measurement Costs and Poor Scalability: The efficiency claims are misleading. While Section 5 reports a substantial scoring phase (~5 minutes), Table 5 reports the MAPS timing (39-62s) using cached scores, omitting the dominant cost. The paper lacks a unified, end-to-end accounting of total pruning time and peak memory across 8B, 14B, and 32B models, making scalability claims unsupported.

  4. Missing Baselines and Unfair Comparison: Key baselines like SparseGPT and SliceGPT are entirely missing for 14B and 32B models. More critically, the performance comparisons are unfair. Table 5 reveals that at a nominal retention ratio of ρ=0.85\rho=0.85, MAPS removes only 2.7% of parameters, whereas the layer-removal baseline removes 10.9%. Comparing accuracy under such mismatched realized compression budgets does not establish a genuine pruning advantage.

Questions for authors

See weaknesses.

Additional question

Table 4 describes nominal retention ratios of at least 0.75 as a viable deployment range. However, Table 12 reports GSM8K accuracy of only 0.004 to 0.046 for MAPS on Llama 3.1 8B at a retention ratio of 0.75, compared with 0.514 for the original model. Given this catastrophic collapse in generation capabilities, how do the authors justify this as deployment range based solely on multiple-choice results?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mGzL

Thank you; we agree on budgets, cost accounting and the deployment claim and corrected them.

Narrative. You are right that the submission read as a chain of reversals. The revision asks three questions separately: is a score unchanged when the weights are rewritten without changing the function (a consistency requirement, not a guarantee of good pruning); does it agree with measured singleton effects; and do those effects add up. MAPS then decides by measured change of the output distribution (itself a proxy); the controls show where it helps and where a cheaper schedule of the same measurement is better.

What is new. (a) On checkpoints with bit-identical outputs, the decisions of two official pruners change substantially (2SSP and MultiPruner: 45-56% of FFN channel choices; 2SSP also on the power-of-two-rounded AWQ copy; part 1), while measured decisions (MAPS, SLEB's rule, BlockPruner) are identical. (b) Mixed granularities compete in one measured ranking: FFN channel chunks, KV groups, sub-blocks and blocks, by a coarsened forward KL per removed parameter, with proposals built from an order the gauges cannot change. SLEB measures blocks, BlockPruner and FinerCut (Jensen-Shannon to the dense output) sub-blocks; 2SSP and MultiPruner already combine measured decisions with width and depth pruning, but in fixed stages over activation- or norm-ordered width candidates. The pre-registered ablation (part 3) shows that the finer proposals matter; FinerCut's rule with the same prompts is 2.6 / 5.0 / 8.4 MC points below MAPS (part 2).

Trivial method; alternatives at equal compute. The five-candidate procedure did not hold up at matched removal (0.642 / 0.541 / 0.432 MC, below SLEB's rule). The revision keeps its principle, removals decided by measured effects, and replaces the procedure: round-wise re-measurement, KL instead of NLL, finer proposals, 64 prompts (part 2). Controls (part 3):

  • Candidate construction: the ablation keeps everything but the finer proposals. GSM8K falls from 0.467 / 0.335 / 0.217 to 0.236 / 0.024 / 0.002; MC is lower in every draw at 25-37.5% but not at 12.5% (three draws); on Qwen3-14B (one draw) both are lower at every budget. Removal is near-matched; compute is not equal (~740 vs 2,499 forward-equivalents; the ablation also re-measures every round).
  • Equal-compute search and objective comparison: at 25% (draw 0), random search with the same prompts, KL and forward budget as MAPS (1,672 allocations) reaches 0.552 MC, versus 0.634; with the same units, KL instead of calibration NLL adds 1.8 to 10.5 MC points.
  • Once-measured (KL-once) vs round-wise (MAPS; part 3): with mixed calibration on Llama-3.1-8B, KL-once has higher MC in all nine comparisons (on Llama-3.2-3B at every budget, one draw) and more mean GSM8K at 12.5-25% with 712 versus 2,499 forward-equivalents; MAPS has lower perplexity and, on 8B, more mean GSM8K at 37.5%. With C4, MAPS has lower MC at 12.5% and higher MC in every draw at 37.5%. On Qwen3-14B (two draws) KL-once collapses at 37.5% (MC 0.386 vs 0.546, GSM8K 0.248 vs 0.689) and MAPS has higher MC at every budget. We retain the frozen configuration and report both. KL-once is cheaper; its largest mean MC shortfall against MAPS is 16.0 points (Qwen3-14B, 37.5%), MAPS's against it 3.0.

Cost. Table 20 gives the timings and memory (loading boundaries: part 4). MAPS takes 45 min / 24 GiB for all budgets at 8B; at 14B / 32B it ran only on shared GPUs: about 2 h to 25% and 3 h to 37.5% (upper bounds), peak 33 / 63 GiB.

Missing baselines and unfair comparison. Realized removal is now reported for every benchmark row and near-matched rows are labelled; the submission's claims are withdrawn (part 1).

  • Added at matched removal: the official 2SSP, MultiPruner and BlockPruner, the first two also with MAPS's prompts, and FinerCut's rule (our implementation; no released code found). Their MC is at most 0.701 / 0.617 / 0.519, against 0.703 / 0.635 / 0.544 for MAPS (level at 12.5%; C4: part 2).
  • SparseGPT (50% unstructured; a different regime) at 14B and 32B; SliceGPT at 14B only (it needs RMSNorm at sublayer inputs, which OLMo-3 lacks).

GSM8K and deployment. You are right that MC alone cannot justify a deployment range. We withdraw it and report GSM8K-500 for every row. Task-specific calibration can help the targeted task (Khanal & Capone 2024; Williams & Aletras 2024; Bandari et al. 2024). With GSM8K training problems in the calibration set (as for the same-prompt baselines), MAPS keeps 0.467 / 0.335 / 0.217 GSM8K on Llama-3.1-8B (91 / 65 / 42% of dense; best external: 2SSP, 0.437 / 0.275 / 0.069; KL-once keeps more at 12.5-25%: 0.480 / 0.419) and 94% of dense at 25% on Qwen3-14B. These Llama-3.1-8B plans keep every block, so they save less KV cache than block removal (a stated limitation). With generic C4 calibration (pre-registered, part 2) the GSM8K margins did not transfer: 0.112 / 0.021 / 0.009, below the best official recipe at 12.5-25%.

To be honest, I genuinely tried to remain professional and provide further constructive feedback in response to the authors’ rebuttal response. However, the paper is still clearly below basic standards for readability and logical coherence, making the entire review process extremely painful. Hence, I decline to serve as free human feedback for this submission. The authors' shows a careless attitude towards the work itself and a lack of respect for reviewers' time. I really doubt that even the rebuttal comments were AI-generated. (I personally quite sure about that)

I am not opposed to automated research. In fact, I believe it may become a future trend, much like how the combustion engine transformed power sources. However, automated research does not inherently guarantee a higher quality paper. While LLMs can significantly facilitate research process and help implement our insights and ideas, AI is ultimately just an amplifier: it will amplifies everything, both the strengths and the sloppiness. As researchers, the minimum we can do is to minimize the sloppiness and maximize clarity: verifying that the central claim remains consistent throughout the paper, and that the experiments genuinely validate that claim. The work should clearly convey your core insights and motivation, rather than (this rather than is human generated) leave this to the reviewers.

In the AI statement, the authors assert that "theoretical claims and their proofs were checked by the authors. We take responsibility for the final content of this work." I have serious doubts about both statements.

I will maintain my original score.

Thank you for the time you spent on our submission. We are sorry that the paper and our responses were hard to read. That is our responsibility. We stand by the AI Use Statement in the paper. We checked the theoretical statements ourselves and we take responsibility for the content, including the budget mismatch you pointed out. We also used LLM tools when we prepared these responses. We will withdraw the submission and rewrite the paper so that it is easier to read.

Reviewer_TvSxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines the reliability of importance scores for structured pruning of large language models. It shows that magnitude-based rankings can change under function-preserving rescalings and that gauge invariance alone does not ensure agreement with measured removal effects. The authors propose MAPS, which combines individual unit ablation scores with direct evaluation of candidate pruning plans involving attention groups, FFN groups, and entire layers.

Critical strengths

[1] The gauge experiment clearly exposes the sensitivity of magnitude-based importance to parameterization, offering a useful diagnostic for pruning criteria.

[2] The failure analysis shows why individual ablation scores can be insufficient, providing insight into the importance of pruning-plan selection.

Critical weaknesses
  1. At rho=0.875, Table 5 shows that MAPS removes 2.7% of the parameters, while layer-drop removes 10.9%. Could you explain why the same nominal budget leads to different reductions and report the actual reductions for all methods, including on OLMo-3-32B? It would also help to see whether MAPS still performs better when the methods keep the same number of parameters.
  2. Could you clarify how Proposition 1 applies to OLMo-2, which normalizes Q and K across heads[1]? In this setting, the proposed rescaling may change the function even when epsilon is zero.
  3. Could you compare the predicted L* plan with the plan selected by measured loss, keeping everything else the same? This would help clarify what plan-level measurement adds.
  4. Please check the undefined Proposition 2 reference and the missing random entries in Table 7.

[1] https://github.com/huggingface/transformers/blob/v4.57.1/src/transformers/models/olmo2/modeling_olmo2.py

Questions for authors

See weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer TvSx

Thank you.

  1. Unequal realized reductions. You are right: the submitted MAPS rounded the kept group count up, so it removed 2.7% where layer removal removed 10.9%. Removal is now counted over attention and MLP weights (Llama-3.1-8B: 6.98B of 8.03B parameters; embeddings, LM head and norms excluded), so 25% = 1.74B parameters = 21.7% of the model, and your 2.7% of all parameters is the 3.1% of attention and MLP weights in part 1. The revision reports the removed fraction for every benchmark row (Tables 5, 6, 19, 27) and labels near-matched rows (BlockPruner 11.7 / 25.5 / 38.7%, MultiPruner 12.7 / 25.1 / 37.6-37.7%). For OLMo-3-32B, the revised MAPS plans remove 12.5 / 25.0 / 37.5% of the attention and MLP weights (realized; Table 6), as do its first revision and SLEB's rule there.

Does MAPS still perform better at the same number of parameters? The submitted procedure does not, so we replaced it. The revised one has the highest mean MC on Llama-3.1-8B against the released pruners with their prompts or ours (level at 12.5%); with generic C4 calibration, only at 25-37.5%. At matched or labelled near-matched removal:

  • the submitted MAPS (budget-exact) is above the estimated scores in the same unit scheme on Llama-3.1-8B (one exception: Wanda with 128 calibration sequences at 12.5%), but below SLEB and TP-Taylor; on OLMo-3-32B it is below layer removal and SLEB at every budget;
  • the revised MAPS (part 2), with the same prompts and three draws: 0.703 / 0.635 / 0.544 MC vs SLEB's rule 0.684 / 0.572 / 0.484 and TP-Taylor 0.670 / 0.558 / 0.436; the released 2SSP, MultiPruner and BlockPruner, with their prompts or ours, reach at most 0.701 / 0.617 / 0.519 (level at 12.5%);
  • with generic C4 calibration instead (pre-registered control, part 2): 0.691 / 0.623 / 0.567 MC against 0.701 / 0.611 / 0.519 for the best official recipe: below in every draw at 12.5%, above on average at 25% and in every draw at 37.5%; the GSM8K margins of part 2 do not transfer (0.112 / 0.021 / 0.009); in later runs (not pre-registered) with the same C4 prompts, SLEB's rule, TP-Taylor and FinerCut's rule are below MAPS in MC in every draw and budget, and the official 2SSP code in every draw at 25-37.5% but not at 12.5% (0.696 vs 0.691 mean; part 2);
  • OLMo-3-32B (one draw; 32 prompts; no KV-group units, see point 2; block-deleting rows re-evaluated after a deletion fix, part 4): MAPS has lower MC at 12.5% and higher MC at 25% and 37.5%, and retains more GSM8K and has lower perplexity at every budget. MAPS vs SLEB's rule, same prompts: MC 0.748 / 0.733 / 0.695 vs 0.754 / 0.728 / 0.681; GSM8K 0.832 / 0.794 / 0.778 vs 0.760 / 0.578 / 0.384; perplexity 6.7 / 7.9 / 9.9 vs 7.2 / 9.4 / 14.2.
  1. Proposition 1 on OLMo-2. Correct. The (Q,K) rescaling is a gauge without QK normalization, a gauge up to the normalization epsilon under per-head QK-norm, and not a gauge under cross-head QK-norm (OLMo-2/3). Our test agrees: on OLMo-2 it changes NLL by +4.28 to +4.32 nats (three seeds), while the (V,O) and FFN rescalings stay bit-exact.
  • Proposition 1 now states these three cases; the covariance table marks which rescalings are gauges where.
  • Gauge validity is separate from mask-deletion equivalence: under cross-head QK-norm, masking a KV group is not the same as deleting it. On OLMo-3, MAPS therefore proposes no KV-group units.
  1. Predicted L* plan vs. plan selected by measured loss (submitted procedure). Same five candidates (each removes the budget up to one group), three draws: the L* plan is 13.4 / 14.2 / 6.3 MC points lower at 12.5 / 25 / 37.5%. The measured choice equals the retrospective best candidate in 11 of 15 backbone-draw-budget cells (Llama-3.1-8B, three draws; Llama-3.2-3B and Qwen3-14B, one draw) and is at most 0.027 MC below it in the other four. Plan-level measurement adds what summed singleton scores miss, consistent with their non-additivity (0.48-2.40 nats).

For the revised MAPS the corresponding control measures every effect once on the dense model (part 3). Re-measuring is not a clean win: with the mixed calibration the once-measured plans have higher MC on Llama-3.1-8B in every draw-budget comparison (0.660 vs 0.635 at 25%; also on Llama-3.2-3B, one draw) at 712 versus 2,499 forward-equivalents, and with C4 at 12.5%. The round-wise procedure has lower perplexity in every mixed-calibration comparison (six of nine with C4), and higher MC with C4 at 37.5% (every draw) and on Qwen3-14B at every budget (0.546 vs 0.386 at 37.5%, two draws). We keep the frozen configuration and report both.

  1. Proposition 2 reference and Table 7. The "Prop. 2" reference was a broken cross-reference; it now reads Proposition 1. Table 7 (now Table 9) is completed under one protocol, with random as the mean [min, max] of five seeds. The measured arm is lowest (lower is better) on 2 of 6 backbones, Fisher on 3 and Taylor on 1, which we state.

If any of points 1-4 is still open, please tell us; we can answer until Nov 18.

I appreciate the rebuttal and the transparent reporting of unfavorable results. My concerns about the OLMo-2 gauge and the editorial errors are now resolved. The new L* comparison shows that plan-level measurement contributes substantially for the submitted procedure. Nonetheless my overall assessment remains that the central empirical claim does not hold. The authors confirm that the submitted MAPS loses at matched parameter counts and replace the allocation procedure in response. The revised method wins only at 25 to 37.5% removal under the authors’ own calibration, is level or below at 12.5%, and loses its GSM8K margin under the pre-registered C4 control. The rebuttal explains these limitations clearly and scopes the claims more carefully but does not overcome them. Since the gauge argument builds on established symmetries and the method evaluated here differs from the one submitted, I do not think the remaining advance is strong enough at this point to change my weak-reject recommendation.

Thank you for reading the revision and for your summary. Your summary of the revised results is accurate. We agree with it. We will withdraw the submission and use your comments in the next version of the paper.

Reviewer_7BemOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that existing pruning criteria, such as weight magnitude or activation traffic, fail to measure the actual function of a network. To address this, the authors propose an LLM pruning method that exploits gauge freedoms, i.e., rescalings that preserve the network’s function. Specifically, they introduce MAPS (Measured Allocation of Pruning Structure), which performs measurement-based pruning over KV groups, FFN groups, and whole layers, controlled by a single hyperparameter. Without any retraining, the proposed method outperforms prior approaches across four backbones and three pruning budgets.

Critical strengths

Instead of relying on weight magnitude as in conventional approaches, the proposed method directly measures the change in negative log-likelihood (ΔNLL) caused by removing each structural unit, and uses this measurement to decide which groups to prune. Based on these measurements, the method removes KV and FFN groups, drops entire layers, and then renumbers the layer indices of the remaining network so that the pruned model remains a valid, compact architecture. In this way, pruning decisions are grounded in the actual functional impact on the model's output distribution rather than in proxy statistics that may not reflect the network's behavior.

The authors evaluate the method on four backbones of different families and scales, namely Llama-8B, Llama-3B, Qwen-14B, and OLMo-32B. Across these models, the proposed approach consistently achieves higher performance than the greedy, group-wise, and layer-drop baselines. Moreover, the advantage is maintained across multiple pruning budgets, indicating that the gains are not limited to a particular compression ratio but hold over a range of sparsity levels. These results support the effectiveness of the proposed measurement-based allocation strategy.

A notable practical advantage of the method is that it requires neither gradient computation nor any retraining or fine-tuning after pruning. Since the pruning decisions rely only on forward-pass measurements of ΔNLL, the approach is computationally lightweight and memory-efficient, which makes it particularly attractive for compressing large language models where backpropagation or recovery training would be costly.

Critical weaknesses

The descriptions of the algorithm and the experimental setup are not presented clearly enough, which makes some parts of the paper difficult to follow. In particular, it would be helpful if the authors provided a more detailed, step-by-step explanation of the procedure (for example, as pseudocode) and clarified the experimental settings so that readers can fully understand and reproduce the results.

In addition, there is some concern as to whether Table 4 includes comparisons with recent state-of-the-art LLM pruning methods. The authors are encouraged to clarify how the baselines were selected and, if possible, to include comparisons with more recent approaches to better position the contribution of this work.

Finally, Table 5 reports the pruning time of the proposed method only. To properly assess the practical impact of this work, the pruning times of the competing methods should also be reported under the same hardware and experimental conditions, so that readers can directly compare the computational cost of the proposed approach with that of existing methods.

Questions for authors
  • How exactly is ΔNLL measured in practice? Please clarify the calibration data used (dataset, number of samples, sequence length), and whether the measurements are computed once or updated iteratively after each removal to account for interactions among the pruned groups and layers.
  • Regarding Table 4, why were the current baselines selected? Could the authors include or discuss comparisons with recent state-of-the-art LLM pruning methods to better position the contribution of this work?
  • Regarding Table 5, could the authors report the pruning time of the competing methods under the same hardware and experimental settings, so that the computational cost of MAPS can be directly compared with that of existing approaches?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 7Bem

Thank you for the careful questions. All three are answered with new evidence, and the algorithm is now given as pseudocode with a settings table. The revision separates three quantities: ΔNLL diagnoses unit removal (Sec. 3); KL selects pruning moves (Algorithm 1); MC, GSM8K and perplexity evaluate the resulting model.

The result your review credited. Your summary says the method "outperforms prior approaches across four backbones and three pruning budgets". The other two reviews showed that those comparisons were made at unequal removal; at matched removal the submitted procedure is below SLEB on Llama-3.1-8B, and we withdrew those claims (general response, part 1). The revised procedure is compared at matched removal. Against SLEB's rule with the same prompts it keeps more GSM8K and has lower perplexity at every budget on all four backbones, and it has higher MC except at 12.5% on OLMo-3-32B (0.748 vs 0.754; Tables 5 and 6; one draw on Llama-3.2-3B and OLMo-3-32B). It still uses no gradients and no recovery training; it takes longer than 2SSP and SLEB and less time than BlockPruner and MultiPruner, whose timings include their own loading (Q3). Its limits are listed in part 1.

Q1. How ΔNLL is measured, and is it updated after each removal?

  • As submitted: 6 GSM8K training problems with worked answers (draw 0), each truncated to 512 tokens; ΔNLL is the change in pooled token-mean NLL when a unit is masked (the inputs of o_proj for a KV group, of down_proj for FFN channels, the residual branch for a block). Each unit was measured once on the dense model, and the five candidate plans were then measured jointly.
  • Revised: at the start of every round each candidate removal is measured given all earlier rounds, on 64 prompts per draw (32 GSM8K + 32 WikiText-2, each at most 512 tokens, disjoint draws), by the KL divergence of the next-token distribution from the dense model's instead of NLL; the energy order and FFN chunks are also re-derived each round. Proposals committed within a round are not rescored; from the stored rounds, a round's measured KL change is within 0.0010-0.0015 of the sum of its members' changes on average on Llama-3.1-8B (mean change 0.016-0.020 per round; larger interactions on Qwen3-14B and OLMo-3-32B, App. J). Round-wise versus once-measured plans (general response, part 3): with calibration NLL the round-wise one scores higher (0.616 vs 0.555 MC at 25%, same units). With the KL and the mixed calibration on Llama-3.1-8B the once-measured plans have higher MC at every budget (0.660 vs 0.635 at 25%, three draws) but higher perplexity (14.8 vs 13.1), as on Llama-3.2-3B (one draw); with C4 calibration the round-wise one has lower MC at 12.5% and higher MC at 37.5% (every draw); on Qwen3-14B (two draws) it has higher MC at every budget (0.546 vs 0.386 at 37.5%).
  • Pseudocode (Algorithm 1 in Sec. 4, which states the sequence length and the budget rule; the submitted and first-revision procedures in the appendix) and one settings table (Table 26: calibration and sequence length, measured quantity, units, decision rule, rounds, cost).

Q2. Recent methods. The submission isolated scoring with score-swap controls (fixed units and materialization) alongside external pruning systems (App. C). We added three recent methods with released code that prune the same unit types: 2SSP (TMLR 2025), MultiPruner and BlockPruner, with their own calibration and, for 2SSP and MultiPruner, with ours, plus FinerCut's rule with our prompts (general response, part 2). MAPS has the highest mean MC among them at 12.5 / 25 / 37.5% (0.703 / 0.635 / 0.544 vs at most 0.701 / 0.617 / 0.519), with a margin within run-to-run variation at 12.5%; with generic C4 calibration it is above the best official recipe in every draw only at 37.5% and below it at 12.5%; in later runs (not pre-registered) with the same C4 prompts, SLEB's rule, TP-Taylor and FinerCut's rule are below MAPS at every budget, and the official 2SSP code at 25-37.5% but not at 12.5% (part 2).

Q3. Pruning time on the same hardware. On one idle GPU at 25% on Llama-3.1-8B, end-to-end with scoring: 2SSP 31 s, SLEB 48 s, submitted MAPS 68 s (measured end-to-end with scoring; this replaces the submission's approximate "~5 min" for scoring), revised MAPS 30 min (45 min for all three budgets), BlockPruner 54 min (one search for all budgets), MultiPruner 79 min (per budget). Table 20 gives time and, where recorded, peak memory at 8B (MultiPruner's peak was not recorded) and for the methods run at 14B and 32B; MAPS at 14B / 32B ran only on shared GPUs (upper bounds). Most timings exclude model loading; BlockPruner's and MultiPruner's include their own loading. The submission's Table 5 omitted the scoring phase; that is corrected.

Clarity. Sec. 4 opens with the design the findings motivate; protocols are in Tables 18 and 26 and App. J (C4 control: Table 29).

If any of Q1-Q3 is still open for you, please tell us; we can answer until Nov 18.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response (1/4): corrections, and what the paper now shows

We thank the reviewers. Two of you found that our plans removed less than the stated budget, and one found the normalization limit of Proposition 1. Both findings were correct, and they changed the paper. The revised PDF and supplementary material are uploaded; App. L lists the withdrawals, replacements, scope corrections and additions, and where the revised evidence is.

  • Withdrawn: the submission's benchmark claims ("10 of 12 cells", also in the TL;DR, which can no longer be edited) and the deployment range read from MC alone. They rested on plans that removed less than the nominal budget (Llama-3.1-8B: 3.1 / 15.3 / 29.5% at a nominal 12.5 / 25 / 37.5%).
  • Corrected: realized removal of attention and MLP weights is reported for every benchmark row (Tables 5, 6 and 27; the submission's matrix: Table 19; rows that remove whole sub-blocks are labelled near-matched), with MC (5 tasks), GSM8K-500 (8-shot) and WikiText-2 perplexity, including 19.8 / 32.6 / 168.5 for the budget-corrected submitted-MAPS row.
  • Replaced, following your reviews: the MAPS procedure. Once removal is matched (TvSx, mGzL), the submitted five-candidate rule is below SLEB on Llama-3.1-8B; the revised one re-measures after earlier removals (7Bem Q1) and is tested against alternative rules (mGzL; part 3). Its principle, removals decided by measured effects, is kept; the submitted procedure and our first revision, an earlier, unposted version of MAPS (App. J), remain reference rows. At matched removal on Llama-3.1-8B the revised MAPS has 0.703 / 0.635 / 0.544 MC at 12.5 / 25 / 37.5%, against at most 0.701 / 0.617 / 0.519 for the released 2SSP, MultiPruner and BlockPruner with their prompts or ours (level at 12.5%; part 2).
  • Limits we report: with a pre-registered generic C4 calibration the GSM8K margins did not transfer, and MC is above the best official recipe in every draw only at 37.5% and below it at 12.5%; with the same C4 documents, the official 2SSP code keeps more mean GSM8K at every budget (part 2). A cheaper control that measures the same KL once has higher MC on both Llama backbones with the mixed calibration (Llama-3.2-3B: one draw) and collapses on Qwen3-14B at 37.5% (part 3).
  • Where to verify: Table 1 (official 2SSP and MultiPruner on bit-identical copies), Table 5 (matched-removal benchmark; our control and the submitted procedure are rows), Tables 4 and 28 (component controls), Table 29 (C4 control), Table 20 (cost), App. L (changes). Findings 1-3 below do not depend on MAPS, submitted or revised.

Finding 1: importance rankings, and two official pruners built on them, depend on how the weights are written. Function-preserving rescalings (gauges) leave every logit bit-identical in the tested settings (for (Q,K) this depends on QK normalization; Proposition 1). They change magnitude and standard-aggregation Wanda rankings substantially (FFN rank survival 0.14-0.61 and 0.34-0.91 on six backbones). We ran the official codes on rescaled copies of Llama-3.1-8B and evaluated their plans on the original checkpoint:

  • 2SSP, three random power-of-two copies: 52.9-56.0% of its removed FFN channels change, and it removes a different attention module. MC falls from 0.612 to 0.592-0.606, GSM8K from 0.046 to 0.018-0.028, perplexity rises from 15.4 to 17.5-18.8 (reference values from this experiment's own run; the benchmark row in part 2 is a separate run). On an identical re-saved copy its plan does not change at all.
  • 2SSP after AWQ's own scale search (official AutoAWQ settings, scaling only, no quantization). With the unrounded scales, which preserve the function up to bf16 rounding, 12.6% of its FFN choices and its attention module change, and MC falls by 0.1 point. With the scales rounded to powers of two, a bit-exact copy, 22.5% change, and MC falls by 2.3 points (0.612 to 0.589). The rounded copy is an exact-function stress test. It does not show that ordinary AWQ preprocessing costs 2.3 points.
  • MultiPruner, two copies: 45.5-50.9% of its removed FFN channels change. Here the pruned models score higher (MC 0.609 to 0.612 / 0.625).
  • For scale: runs of the same code on two GPU types differ in 1.1-1.8% of channels and 0.1-0.2 MC points.
  • Measured decisions do not move. On a rescaled copy, MAPS takes identical decisions in all 30 rounds, with bit-identical measured values; so does SLEB's rule (8 steps), and the official BlockPruner deletes the same sub-blocks at all 40 steps of its search. On the bit-exact AWQ copy, SLEB's rule again takes identical decisions.

Finding 2: invariant statistics are a weak guide to measured effects. Statistics invariant to the gauges agree with measured removal effects only weakly to moderately (one protocol, six backbones).

Finding 3: measured singleton effects do not compose. Units chosen by their own effects cost 0.48-2.40 nats more together than the sum of their effects, so summed singleton effects understate a plan's joint cost.

AuthorsOpenReview ↗

General response (2/4): revised MAPS and the benchmark at matched removal

MAPS keeps the submission's principle (removals decided by measured effects, over units of several granularities) and replaces the procedure: at each round's start every proposal is re-measured given all earlier rounds (7Bem), by the KL divergence of the next-token distribution from the dense model's instead of calibration NLL; the proposals are finer (FFN channel chunks, KV groups, attention and MLP sub-blocks, blocks); calibration uses 64 prompts, not 6. The proposals with the smallest change per removed parameter are removed; those committed within a round are not rescored. Controls (mGzL): part 3. Within the mixed-calibration protocol of our first revision, the final configuration was selected on development items disjoint from the test sets (App. J, Selection), recorded before its test evaluation and retained. Two earlier choices were informed by test results: the bulk proposal that removes all but one KV group of a layer is not offered (motivated by part 3's analysis; decided on development items), and the mixed calibration set was chosen after seeing GSM8K-only test results.

Recent methods at matched removal (7Bem, mGzL). Llama-3.1-8B (dense 0.745 / 0.514 / 7.7), MC / GSM8K / PPL, three-draw means (3d). "Same 64" = MAPS's 64 calibration prompts.

method12.5%25%37.5%
MAPS (3d)0.703 / 0.467 / 9.70.635 / 0.335 / 13.10.544 / 0.217 / 20.5
MAPS, KL measured once (control, 3d)†0.715 / 0.480 / 10.10.660 / 0.419 / 14.80.574 / 0.186 / 27.1
MultiPruner, official recipe0.701 / 0.254 / 11.70.611 / 0.012 / 21.90.469 / 0.000 / 55.8
MultiPruner, same 64 (3d)0.692 / 0.211 / 11.20.617 / 0.049 / 17.50.518 / 0.011 / 39.9
2SSP, official (C4)0.698 / 0.150 / 10.20.611 / 0.054 / 15.40.519 / 0.010 / 25.6
2SSP, same 64 (3d)0.690 / 0.437 / 10.00.599 / 0.275 / 14.60.487 / 0.069 / 25.9
BlockPruner, official*0.682 / 0.104 / 13.40.584 / 0.002 / 23.90.483 / 0.000 / 61.3
FinerCut's rule, same 64 (3d)*0.677 / 0.162 / 14.60.586 / 0.003 / 38.50.460 / 0.000 / 321.9
SLEB rule, same 64 (3d)0.684 / 0.163 / 11.70.572 / 0.005 / 21.40.484 / 0.000 / 44.0
TP-Taylor, same 64 (3d)0.670 / 0.277 / 15.30.558 / 0.126 / 40.40.436 / 0.035 / 120.3
MAPS, first revision (3d)0.678 / 0.343 / 10.30.611 / 0.231 / 15.10.530 / 0.095 / 23.3
Submitted MAPS, budget-corrected (3d; 6 prompts)0.642 / 0.049 / 19.80.541 / 0.008 / 32.60.432 / 0.000 / 168.5

*Near-matched (whole sub-blocks): BlockPruner 11.7 / 25.5 / 38.7%; FinerCut's rule 11.7-12.3 / 25.5 / 36.8-38.7%. MultiPruner: 12.7 / 25.1 / 37.6-37.7%. †Evaluated after the selection was frozen; not a selection candidate. GSM8K and WikiText-2 training texts are in the calibration of MAPS and of every "same 64" row, so GSM8K and perplexity are partly in-domain there; the same-prompt rows are the like-for-like comparison (at 12.5%, 2SSP scores 0.150 GSM8K with its official recipe, 0.437 with the same 64).

  • With this calibration, among the external baselines MAPS has the highest mean MC and GSM8K and the lowest perplexity at every budget. Where the lead is not clear: at 12.5% the official MultiPruner recipe is level in MC (0.701, one run); at 25% MultiPruner with the same prompts is higher in one of three draws (0.645 vs 0.634); at 12.5%, draw 0, 2SSP with the same prompts keeps slightly more GSM8K (0.462 vs 0.458).

Calibration domain (pre-registered C4 control on Llama-3.1-8B; three draws; all else frozen). Calibration data matter (Williams & Aletras, 2024; Bandari et al., 2024), and task-specific calibration can help the targeted task (Khanal & Capone, 2024); ours is task-aware (table note). With 64 generic C4 documents per draw, MAPS gets 0.691 / 0.623 / 0.567 MC, 0.112 / 0.021 / 0.009 GSM8K, 10.1 / 14.5 / 22.8 PPL (best official recipe per metric 0.701 / 0.611 / 0.519, 0.254 / 0.054 / 0.010, 10.2 / 15.4 / 25.6): the GSM8K margins did not transfer; the MC lead over that recipe holds in every draw only at 37.5% (on average at 25%, below at 12.5%); perplexity stays lower at 25-37.5%. KL-once, recalibrated alike: 0.714 / 0.623 / 0.537 MC, above MAPS at 12.5% (GSM8K 0.305 vs 0.112), nearly equal at 25%, below at 37.5% (every draw). In later runs (not pre-registered), SLEB's rule, TP-Taylor and FinerCut's rule with the same C4 prompts have lower MC than MAPS in every draw at every budget and higher perplexity in all 27 comparisons; at 12.5% FinerCut's rule keeps more mean GSM8K (0.142 vs 0.112). The official 2SSP code with the same C4 documents gets 0.696 / 0.614 / 0.517 MC, 0.129 / 0.036 / 0.011 GSM8K, 10.3 / 15.3 / 26.1 PPL: MAPS has higher MC in every draw at 25-37.5% (by 1.0 / 5.0 points on average) but lower mean MC at 12.5% (by 0.6; higher in one draw), less mean GSM8K at every budget and lower perplexity in all nine comparisons. MultiPruner was not rerun.

AuthorsOpenReview ↗

General response (3/4): what each part of MAPS contributes (controls)

What each part contributes (mGzL). 25%, draw 0, same prompts; 25% realized removal except the granularity ablation, 24.88% (MC / GSM8K / PPL; differences in all comments are from unrounded values):

measured quantity, unitsmeasured oncere-measured every round
calibration NLL, first-revision units0.546 / 0.084 / 29.50.600 / 0.256 / 15.3
calibration NLL, revised units0.555 / 0.032 / 30.20.616 / 0.290 / 14.5
KL to the dense model, revised units0.660 / 0.382 / 14.40.634 / 0.312 / 13.2 (MAPS)
KL, sub-blocks and blocks only-0.597 / 0.020 / 19.5 (24.88% removed)
  • Measuring beats estimating in this control: the first revision's units ranked by energy alone reach 0.455 MC, against 0.546 measured once and 0.600 re-measured. Where the forwards go matters: the best of 1,672 random allocations scored by the same KL at MAPS's forward budget reaches 0.552 / 0.012 / 20.1, against 0.634 / 0.312 / 13.2.
  • The measured quantity matters: with the same units, KL instead of calibration NLL adds 1.8 MC points when both are re-measured and 10.5 when both are measured once. Calibration NLL also hides a failure: a layer reduced to a single KV group keeps calibration likelihood but loses downstream accuracy. On Qwen3-14B this move, used in six layers, collapsed GSM8K in our first revision (App. J; 12.5%, draw 0: 0.158 vs 0.578 for SLEB's rule with the same prompts; dense 0.924), and teacher-forced likelihood of the reference answers does not reveal it. The revised MAPS therefore measures changes in the next-token distribution, still a proxy for downstream behavior, and does not offer that move (part 2).
  • So do the finer proposals (pre-registered ablation: same KL, prompts, layers, round-quota rule and ranking, but sub-blocks and blocks only; three draws): GSM8K and perplexity are worse in all nine draw-budget pairs (GSM8K 0.236 / 0.024 / 0.002 vs MAPS 0.467 / 0.335 / 0.217), although the ablation removes slightly less (24.88% at 25%); MC is lower in every draw at 25% and 37.5% but higher in two of three draws at 12.5%. It too re-measures all its proposals every round; its whole path costs ~740 forward-equivalents vs 2,499 (a proposal-set ablation). The pre-specified Qwen3-14B replication (draw 0) agrees: GSM8K 0.732 / 0.234 / 0.026 vs 0.894 / 0.864 / 0.718, MC 1.3 / 4.8 / 2.0 points lower, perplexity higher.

Once-measured vs round-wise procedures (MAPS also re-derives its energy order and FFN chunks each round). Mean MC of the once-measured plans minus MAPS, in points, both under the KL:

setting12.5%25%37.5%
Llama-3.1-8B, mixed calibration (three draws)+1.2+2.5+3.0
Llama-3.2-3B, mixed calibration (draw 0; run later, not pre-registered)+2.7+2.3+2.1
Llama-3.1-8B, C4 calibration (three draws; part 2)+2.3+0.05-3.0
Qwen3-14B, mixed calibration (two draws; run after the Llama results)-0.9-7.2-16.0
  • The sign is the same in every draw except with C4 at 25%. With calibration NLL instead of the KL, the round-wise procedure is 5.4-6.1 MC points higher (25%, draw 0).
  • Perplexity: lower for the round-wise procedure in every comparison with the mixed calibration (six of nine with C4); on Llama-3.1-8B, draw 0, so are its held-out NLL and calibration KL (0.270 vs 0.334).
  • GSM8K: the once-measured plans keep more on average at 12.5-25% on Llama-3.1-8B (0.480 / 0.419 vs 0.467 / 0.335) and less at 37.5% (0.186 vs 0.217); more at every budget on Llama-3.2-3B (0.252 / 0.178 / 0.038 vs 0.226 / 0.116 / 0.036); more with C4 at 12.5% (0.305 vs 0.112). On Qwen3-14B they are level at 12.5% (0.906 vs 0.901) and collapse at 37.5% (two-draw means: MC 0.386 vs 0.546, GSM8K 0.248 vs 0.689, perplexity 368.4 vs 21.2).
  • Cost: 712 vs 2,499 forward-equivalents on Llama-3.1-8B (13 vs 45 minutes for one pruning run producing all budgets, an offline cost; part 4); 626 vs 2,198 on Llama-3.2-3B.
  • How we read this: both procedures decide by the same measured KL over the same unit types, so the open question is when to measure, not whether. On Llama-3.1-8B with the mixed calibration both are at or above every released pruner in mean MC (0.703 / 0.635 / 0.544 and 0.715 / 0.660 / 0.574 against at most 0.701 / 0.617 / 0.519; part 2). Re-measuring gave lower calibration KL and held-out NLL where we checked, and lower perplexity with the mixed calibration, but not reliably higher MC: a conditional trade-off, not a general accuracy gain. In these four settings the round-wise procedure's largest mean MC shortfall is 3.0 points, the once-measured one's 16.0 (Qwen3-14B, 37.5%).
  • We keep the configuration frozen before these tests instead of choosing a schedule on test results, and report both.
AuthorsOpenReview ↗

General response (4/4): other backbones, cost and fidelity

Other backbones (MC / GSM8K / PPL; same prompts within each backbone, task-aware as in part 2; C4 control: Llama-3.1-8B only; draw 0 unless stated).

method12.5%25%37.5%
Qwen3-14B: MAPS (2 draws)0.717 / 0.901 / 12.70.648 / 0.865 / 14.60.546 / 0.689 / 21.2
Qwen3-14B: first revision (2 draws)0.705 / 0.208 / 13.30.648 / 0.215 / 15.10.555 / 0.233 / 20.7
Qwen3-14B: SLEB's rule0.667 / 0.578 / 14.00.599 / 0.094 / 22.70.493 / 0.008 / 52.8
Qwen3-14B: FinerCut's rule*0.689 / 0.550 / 17.60.593 / 0.034 / 28.50.493 / 0.000 / 266.7
Llama-3.2-3B: MAPS0.607 / 0.226 / 12.50.537 / 0.116 / 18.80.469 / 0.036 / 30.7
Llama-3.2-3B: KL-once (control)0.634 / 0.252 / 13.20.560 / 0.178 / 20.00.490 / 0.038 / 33.3
Llama-3.2-3B: first revision0.585 / 0.184 / 12.90.526 / 0.096 / 19.00.463 / 0.050 / 30.9
Llama-3.2-3B: SLEB's rule†0.582 / 0.012 / 14.60.484 / 0.000 / 33.10.444 / 0.002 / 62.6
OLMo-3-32B: MAPS0.748 / 0.832 / 6.70.733 / 0.794 / 7.90.695 / 0.778 / 9.9
OLMo-3-32B: first revision0.747 / 0.788 / 6.80.727 / 0.744 / 8.20.685 / 0.646 / 10.3
OLMo-3-32B: SLEB's rule0.754 / 0.760 / 7.20.728 / 0.578 / 9.40.681 / 0.384 / 14.2

Dense: 0.749 / 0.924 / 11.1 (14B), 0.679 / 0.266 / 9.8 (3B), 0.759 / 0.830 / 5.8 (32B). *12.9 / 24.4 / 38.0% removed. †Whole blocks, next to the budgets (10.7 / 25.0 / 35.7%; 14.3 and 39.3% are as close). OLMo-3-32B: 32 prompts, no KV-group units; block-deleting rows re-evaluated after the fix below.

  • Qwen3-14B: the first revision's GSM8K collapse is gone (94% of dense GSM8K at 25%); MC is 1.1 points higher at 12.5%, equal at 25% and 0.9 points lower at 37.5%. Both the measured quantity and the units changed between the two, so the recovery is not attributable to the KL alone.
  • Llama-3.2-3B: at 37.5% MAPS keeps less GSM8K than our first revision; the once-measured control has higher MC and GSM8K and higher perplexity at every budget (part 3).
  • OLMo-3-32B: SLEB's rule has higher MC at 12.5%; MAPS keeps more GSM8K and has lower perplexity at every budget.

Cost on one idle GPU (Llama-3.1-8B): MAPS 30 min through 25% (1,672 forward-equivalents) and 45 min for all three budgets (peak 24 GiB, 2,499 forward-equivalents); the once-measured control 13 min for all budgets (712); 2SSP 31 s and SLEB 48 s at 25%; BlockPruner 54 min for all budgets; MultiPruner 79 min per budget. MAPS at 14B / 32B ran only on shared GPUs: about 2 h to 25% and 3 h to 37.5% (upper bounds), peak 33 / 63 GiB. Most timings in Table 20 exclude model loading; the official BlockPruner and MultiPruner timings include their own model and data loading. On the same GPU, MAPS on a rescaled copy repeats every decision of all 30 rounds; across GPU types decisions are not bit-reproducible (App. J).

Fidelity. Every plan is evaluated with one harness. In float32, our application of the 2SSP and MultiPruner plans reproduces the official pruned models: max |delta log p| at most 1.3e-4, argmax agreement 1.000. One correction: when a plan deletes whole blocks of OLMo-3, transformers selects each layer's attention type and rotary embedding by its index, so after a deleted block the later layers must keep their original types. Our evaluation code missed this for the new OLMo-3-32B runs (masked and deleted models differed by 0.007-0.02 nats); we fixed it and re-evaluated every affected OLMo-3 row. Selection masks units and never deletes layers, so no plan changed; only evaluations were rerun. Llama and Qwen rows are unaffected. An output check (masked selection network vs materialized evaluation network, eight held-out inputs, greedy decoding with the KV cache): the two checked deletion-only plans (SLEB's OLMo-3-32B plan with 24 deleted blocks; a FinerCut plan) give identical log-probs and decodes; the two checked Llama plans that slice channels or KV groups match in float32 (mean KL <= 5e-9, 8/8 decodes) and, in bf16, the evaluation precision, have mean KL 7e-4 / 3e-3, argmax agreement 0.991 / 0.978 and 8/8 / 4/8 matching decodes (OLMo sliced plans: 6/8). For two Llama-3.1-8B draw-1 plans at 37.5%, masked rather than materialized bf16 evaluation changes GSM8K-500 by +1.8 points for KL-once (0.212 versus 0.194; 22 masked-only and 13 materialized-only correct answers) and 0.0 points for MAPS (0.232; 12 correct only in each implementation). These checks demonstrate numerical sensitivity for the tested plans; they do not establish a general two-point error bound.

Supplementary material. The supplementary zip is replaced by the artifact of this version: the code of every experiment in the paper (the submission's runners, and runners for the experiments added in this revision that list their commands and cost) and every result file the paper uses, including the failure ledger, with a checksummed manifest; README_REPRO.md gives the run order and the pinned environments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.