- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses LLM pruning where the uniform sparsity rate allocation ignores the inter-layer sensitivity differences, resulting in performance degradation at high sparsity rates. To tackle, a layer-wise sparsity rate allocation method based on gradient guidance, named GLAS, is proposed. This method models the sparsity rate allocation as a minimization problem of loss under a fixed global pruning budget. It estimates the gradients and curvatures of the language model loss with respect to each layer's sparsity rate using finite difference. Among them, GLAS-1 allocates the sparsity rates of each layer based on gradient ranking at one time, while GLAS-K uses second-order approximation and constrained optimization to iteratively optimize the allocation scheme. Experiments show that GLAS can reduce perplexity, improve zero-shot task accuracy, and can be extended to structured pruning and visual Transformer.
Critical strengths
-
GLAS directly utilizes the sensitivity of the language model loss to the sparsity rate of each layer to guide the allocation of pruning budgets. Compared with heuristic methods based on weight statistics, it is more consistent with the final optimization objective.
-
The author conducted experiments on various model architectures and parameter scales (7B-32B) such as LLaMA, Qwen, and Mistral, and combined multiple pruning criteria including Wanda, SparseGPT, and RIA, comparing them with methods like OWL, DLP, and LSA. The ablation and extension experiments were quite comprehensive, analyzing the impact of different calibration sample quantities, sparsity rates, and initialization allocation schemes on performance, and verifying that GLAS-K can further optimize the sparse rate allocation produced by other methods.
Critical weaknesses
-
The method is rather limited in terms of innovation. Measuring pruning sensitivity using gradients, curvatures, and second-order approximations has been widely studied in the field of model compression. While the authors argue that conditioning the allocation on the specific pruning metric in use is a novel angle, the core technical components (finite-difference gradient estimation, diagonal Hessian approximation, and trust-region constrained optimization) are all well-established. GLAS mainly applies these ideas to optimize the sparsity rate layer by layer, and the incremental nature of this combination reduces the perceived contribution.
-
Compared with other methods, GLAS incurs high additional computational overhead. The paper does report allocation time and memory in Table 27, but this only covers a single model (LLaMA3-8B at 8B scale). The memory footprint already reaches 32.7 GB at this scale, compared to 21.7 GB for OWL and DLP. There is no discussion of how memory and time scale as model size grows beyond 8B, which is precisely where non-uniform allocation would be most valuable in practice.
Questions for authors
-
Gradients, curvatures, and second-order approximations have been widely used in model pruning and resource allocation. The authors should better clarify the core differences between GLAS and existing gradient or curvature-guided pruning methods beyond the metric-awareness argument.
-
Table 27 reports allocation cost only for LLaMA3-8B. Could the authors provide corresponding numbers for larger models (e.g., 32B or 70B scale) to demonstrate whether the memory and time overhead remain practical at the scales where compression is most needed?
-
The evaluation primarily focuses on perplexity and conventional zero-shot benchmarks. Could the authors extend the evaluation to more challenging downstream tasks like reasoning, instruction following, or long-context understanding, to better demonstrate the practical effectiveness of GLAS?
Dear Reviewer,
As the discussion period approaches its end, we would appreciate hearing whether our responses and additional experiments have addressed your concerns. Please let us know if there are any further points we can clarify. Thank you again for your time and consideration.
Authors