- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper study the reliability of importance scores in structured pruning for LLMs. It first argues that common proxy scores, for example: magnitude and activation traffic are unreliable because they are not gauge-invariant under function preserving reparameterizations. Then the paper demonstrates that even gauge-invariant scores (e.g., Wanda's paired aggregation, Taylor, Fisher) remain weakly correlated with causal importance, and relying purely on causal solo scores fails on flat landscapes. The authors propose MAPS, a heuristic strategy that generates a small candidate set of pruning plans and measures their Masked NLL to select the best allocation. The authors conduct extensive experiments across multiple backbones (Llama, Qwen, Mistral, OLMo, Gemma) and various budgets.
Critical strengths
- The paper has a lot experiments and evaluates multiple setting across various budgets.
Critical weaknesses
-
Extremely Poor Clarity: The narrative of this paper fails to establish a logical bridge between the theoretical gauge diagnosis and the final method. This paper build initial around the necessity of "gauge invariance". Then, the paper invalidates its own point (the necessity of "gauge invariance") by proving that gauge-invariant proxies also fail. It then shifts to proposing causal solo scores, only to show they collapse on flat landscapes. This constant shifting makes the paper extremely exhausting to read.
-
Trivial Method: The proposed MAPS algorithm is essentially a localized, brute-force Grid Search. MAPS is a straightforward heuristic that simply constructs a small set of five candidate plans and selects the best based on calibration loss. The paper fails to provide clear evidence that this specific candidate construction rule is optimal or robust compared to alternative rules under the same computational budget.
-
High Measurement Costs and Poor Scalability: The efficiency claims are misleading. While Section 5 reports a substantial scoring phase (~5 minutes), Table 5 reports the MAPS timing (39-62s) using cached scores, omitting the dominant cost. The paper lacks a unified, end-to-end accounting of total pruning time and peak memory across 8B, 14B, and 32B models, making scalability claims unsupported.
-
Missing Baselines and Unfair Comparison: Key baselines like SparseGPT and SliceGPT are entirely missing for 14B and 32B models. More critically, the performance comparisons are unfair. Table 5 reveals that at a nominal retention ratio of , MAPS removes only 2.7% of parameters, whereas the layer-removal baseline removes 10.9%. Comparing accuracy under such mismatched realized compression budgets does not establish a genuine pruning advantage.
Questions for authors
See weaknesses.
Additional question
Table 4 describes nominal retention ratios of at least 0.75 as a viable deployment range. However, Table 12 reports GSM8K accuracy of only 0.004 to 0.046 for MAPS on Llama 3.1 8B at a retention ratio of 0.75, compared with 0.514 for the original model. Given this catastrophic collapse in generation capabilities, how do the authors justify this as deployment range based solely on multiple-choice results?
3 comments
To be honest, I genuinely tried to remain professional and provide further constructive feedback in response to the authors’ rebuttal response. However, the paper is still clearly below basic standards for readability and logical coherence, making the entire review process extremely painful. Hence, I decline to serve as free human feedback for this submission. The authors' shows a careless attitude towards the work itself and a lack of respect for reviewers' time. I really doubt that even the rebuttal comments were AI-generated. (I personally quite sure about that)
I am not opposed to automated research. In fact, I believe it may become a future trend, much like how the combustion engine transformed power sources. However, automated research does not inherently guarantee a higher quality paper. While LLMs can significantly facilitate research process and help implement our insights and ideas, AI is ultimately just an amplifier: it will amplifies everything, both the strengths and the sloppiness. As researchers, the minimum we can do is to minimize the sloppiness and maximize clarity: verifying that the central claim remains consistent throughout the paper, and that the experiments genuinely validate that claim. The work should clearly convey your core insights and motivation, rather than (this rather than is human generated) leave this to the reviewers.
In the AI statement, the authors assert that "theoretical claims and their proofs were checked by the authors. We take responsibility for the final content of this work." I have serious doubts about both statements.
I will maintain my original score.
Thank you for the time you spent on our submission. We are sorry that the paper and our responses were hard to read. That is our responsibility. We stand by the AI Use Statement in the paper. We checked the theoretical statements ourselves and we take responsibility for the content, including the budget mismatch you pointed out. We also used LLM tools when we prepared these responses. We will withdraw the submission and rewrite the paper so that it is easier to read.
Response to Reviewer mGzL
Thank you; we agree on budgets, cost accounting and the deployment claim and corrected them.
Narrative. You are right that the submission read as a chain of reversals. The revision asks three questions separately: is a score unchanged when the weights are rewritten without changing the function (a consistency requirement, not a guarantee of good pruning); does it agree with measured singleton effects; and do those effects add up. MAPS then decides by measured change of the output distribution (itself a proxy); the controls show where it helps and where a cheaper schedule of the same measurement is better.
What is new. (a) On checkpoints with bit-identical outputs, the decisions of two official pruners change substantially (2SSP and MultiPruner: 45-56% of FFN channel choices; 2SSP also on the power-of-two-rounded AWQ copy; part 1), while measured decisions (MAPS, SLEB's rule, BlockPruner) are identical. (b) Mixed granularities compete in one measured ranking: FFN channel chunks, KV groups, sub-blocks and blocks, by a coarsened forward KL per removed parameter, with proposals built from an order the gauges cannot change. SLEB measures blocks, BlockPruner and FinerCut (Jensen-Shannon to the dense output) sub-blocks; 2SSP and MultiPruner already combine measured decisions with width and depth pruning, but in fixed stages over activation- or norm-ordered width candidates. The pre-registered ablation (part 3) shows that the finer proposals matter; FinerCut's rule with the same prompts is 2.6 / 5.0 / 8.4 MC points below MAPS (part 2).
Trivial method; alternatives at equal compute. The five-candidate procedure did not hold up at matched removal (0.642 / 0.541 / 0.432 MC, below SLEB's rule). The revision keeps its principle, removals decided by measured effects, and replaces the procedure: round-wise re-measurement, KL instead of NLL, finer proposals, 64 prompts (part 2). Controls (part 3):
Cost. Table 20 gives the timings and memory (loading boundaries: part 4). MAPS takes 45 min / 24 GiB for all budgets at 8B; at 14B / 32B it ran only on shared GPUs: about 2 h to 25% and 3 h to 37.5% (upper bounds), peak 33 / 63 GiB.
Missing baselines and unfair comparison. Realized removal is now reported for every benchmark row and near-matched rows are labelled; the submission's claims are withdrawn (part 1).
GSM8K and deployment. You are right that MC alone cannot justify a deployment range. We withdraw it and report GSM8K-500 for every row. Task-specific calibration can help the targeted task (Khanal & Capone 2024; Williams & Aletras 2024; Bandari et al. 2024). With GSM8K training problems in the calibration set (as for the same-prompt baselines), MAPS keeps 0.467 / 0.335 / 0.217 GSM8K on Llama-3.1-8B (91 / 65 / 42% of dense; best external: 2SSP, 0.437 / 0.275 / 0.069; KL-once keeps more at 12.5-25%: 0.480 / 0.419) and 94% of dense at 25% on Qwen3-14B. These Llama-3.1-8B plans keep every block, so they save less KV cache than block removal (a stated limitation). With generic C4 calibration (pre-registered, part 2) the GSM8K margins did not transfer: 0.112 / 0.021 / 0.009, below the best official recipe at 12.5-25%.