acceptodds
Under review as a conference paper at ICLR 2027

GLAS: Gradient-Guided Layer-wise Sparsity Allocation for LLM Pruning

Abstract

Large language models (LLMs) have achieved remarkable success across diverse tasks, yet their colossal size renders deployment expensive in both memory and inference. Pruning offers a direct remedy, but prevailing techniques apply a uniform sparsity ratio across Transformer layers, disregarding their varying importance and incurring severe performance degradation at high sparsity. Recent efforts therefore shift toward non-uniform layer-wise pruning. However, they rank layers by empirically designed proxy scores and overlook the dependence of layer importance on the pruning metric in use. To address this, we propose GLAS (Gradient-Guided Layer-wise Sparsity Allocation), which casts sparsity allocation as a constrained optimization problem. GLAS- allocates sparsity guided by metric-dependent gradients in a single pass, while GLAS- further refines the allocation iteratively. Experiments across mainstream LLM families show that GLAS surpasses state-of-the-art baselines in both perplexity and zero-shot accuracy, and transfers to semi-structured and structured pruning as well as to vision transformers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Network Pruning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_e5dyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses LLM pruning where the uniform sparsity rate allocation ignores the inter-layer sensitivity differences, resulting in performance degradation at high sparsity rates. To tackle, a layer-wise sparsity rate allocation method based on gradient guidance, named GLAS, is proposed. This method models the sparsity rate allocation as a minimization problem of loss under a fixed global pruning budget. It estimates the gradients and curvatures of the language model loss with respect to each layer's sparsity rate using finite difference. Among them, GLAS-1 allocates the sparsity rates of each layer based on gradient ranking at one time, while GLAS-K uses second-order approximation and constrained optimization to iteratively optimize the allocation scheme. Experiments show that GLAS can reduce perplexity, improve zero-shot task accuracy, and can be extended to structured pruning and visual Transformer.

Critical strengths
  1. GLAS directly utilizes the sensitivity of the language model loss to the sparsity rate of each layer to guide the allocation of pruning budgets. Compared with heuristic methods based on weight statistics, it is more consistent with the final optimization objective.

  2. The author conducted experiments on various model architectures and parameter scales (7B-32B) such as LLaMA, Qwen, and Mistral, and combined multiple pruning criteria including Wanda, SparseGPT, and RIA, comparing them with methods like OWL, DLP, and LSA. The ablation and extension experiments were quite comprehensive, analyzing the impact of different calibration sample quantities, sparsity rates, and initialization allocation schemes on performance, and verifying that GLAS-K can further optimize the sparse rate allocation produced by other methods.

Critical weaknesses
  1. The method is rather limited in terms of innovation. Measuring pruning sensitivity using gradients, curvatures, and second-order approximations has been widely studied in the field of model compression. While the authors argue that conditioning the allocation on the specific pruning metric in use is a novel angle, the core technical components (finite-difference gradient estimation, diagonal Hessian approximation, and trust-region constrained optimization) are all well-established. GLAS mainly applies these ideas to optimize the sparsity rate layer by layer, and the incremental nature of this combination reduces the perceived contribution.

  2. Compared with other methods, GLAS incurs high additional computational overhead. The paper does report allocation time and memory in Table 27, but this only covers a single model (LLaMA3-8B at 8B scale). The memory footprint already reaches 32.7 GB at this scale, compared to 21.7 GB for OWL and DLP. There is no discussion of how memory and time scale as model size grows beyond 8B, which is precisely where non-uniform allocation would be most valuable in practice.

Questions for authors
  1. Gradients, curvatures, and second-order approximations have been widely used in model pruning and resource allocation. The authors should better clarify the core differences between GLAS and existing gradient or curvature-guided pruning methods beyond the metric-awareness argument.

  2. Table 27 reports allocation cost only for LLaMA3-8B. Could the authors provide corresponding numbers for larger models (e.g., 32B or 70B scale) to demonstrate whether the memory and time overhead remain practical at the scales where compression is most needed?

  3. The evaluation primarily focuses on perplexity and conventional zero-shot benchmarks. Could the authors extend the evaluation to more challenging downstream tasks like reasoning, instruction following, or long-context understanding, to better demonstrate the practical effectiveness of GLAS?

Reviewer_cvuKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces GLAS, an LLM pruning method that dynamically allocates sparsity non-uniformly between layers while maintaining the global sparsity budget. Concretely, the method estimates the loss changes when perturbing the sparsity of individual layers. The paper introduces 2 variants: 1) GLAS-1: estimating layer-wise loss sensitivity and allocating sparsity for layers ranked by these sensitivities; and 2) GLAS-K: an iterative variant employing estimated curvature information to further enhance the sparsity allocation. Experiments show perplexity improvement compared to existing non-uniform sparsity allocation baselines.

Critical strengths
  • Pruning metric-aware dynamic sparsity allocation is well-motivated and experimental results show that this approach outperforms metric-agnostic allocation baselines.
  • The method is evaluated across multiple LLM families, model sizes, and pruning metrics, demonstrating consistent improvements in perplexity over existing baselines
  • GLAS-K provides a generic optimization procedure that further improves the allocations obtained by not only GLAS-1 but also the other baselines, including OWL, DLP, and LSA. This demonstrates its flexibility and applicability beyond the proposed GLAS-1 initialization.
Critical weaknesses
  • The loss smoothness assumption provides no quantitative error bounds for finite-difference gradients or curvature. Curvature estimation is particularly sensitive to discretization noise, thus potentially affecting the updates.
  • There is an incorrect Hessian approximation justification in Appendix D, claiming that normalized coupling κab<0.2\kappa_{ab} < 0.2 implies off-diagonal Hessian entries are at least five times smaller than the corresponding diagonal entries. This inference is mathematically invalid and weakens the justification for the diagonal approximation.
  • Despite substantial perplexity improvement, GLAS-1 achieves only marginal gains in average zero-shot accuracy over the strongest baselines. This raises concerns about whether the perplexity improvements translate into meaningful downstream capability gains, particularly given the substantial additional computational cost.
  • GLAS-1 introduces substantial computational overhead compared to the baselines, which increases even further with GLAS-K. This makes the trade-off between computational cost and downstream task performance remain questionable given the modest reported improvement.
Questions for authors

Could the authors clarify and address the concerns raised in the Weaknesses section?

Reviewer_xq8HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The article proposes the GLAS method, which directly measures the sensitivity of language modeling loss to the sparsity rate of each layer, and allocates the budget to the layers where the loss increases more slowly. Since the sparse rate itself cannot be back-propagated, this sensitivity is obtained by perturbing the sparse rates of each layer and estimating the perturbation. GLAS-1 makes an initial estimation based on uniform distribution and completes the allocation; GLAS-K iteratively adjusts on this basis, re-estimating the sensitivity in each step, and only accepting updates that actually reduce the language modeling loss.

Critical strengths

The idea is clear and the method is well laid out. Under a fixed total sparsity budget, GLAS first estimates how sensitive the language-modeling loss is to each layer’s sparsity and then allocates sparsity accordingly: GLAS-1 makes this estimate once from a uniform allocation, and GLAS-K refines it iteratively while keeping only updates that reduce the language-modeling loss.

Critical weaknesses
  1. Why were the results of the experimental part all based on a 70% sparsity level?

  2. How was the superiority of the gradient metric demonstrated (were any ablation experiments conducted)?

  3. It is necessary to report the computational cost of the allocation stage.

Questions for authors

The same as the weakness part.

1 comment

Dear Reviewer,

As the discussion period approaches its end, we would appreciate hearing whether our responses and additional experiments have addressed your concerns. Please let us know if there are any further points we can clarify. Thank you again for your time and consideration.

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.