acceptodds
Under review as a conference paper at ICLR 2027

LILA: Data-Free Structured Pruning of LLMs via NMF Spectral Geometry

Abstract

Structured pruning of large language models (LLMs) has universally required calibration data, gradient computation, or large auxiliary policy networks at pruning time. LILA (Latent-Informed Layer Analysis) demonstrates that this dependency is avoidable: each FFN weight matrix is factorized with Non-negative Matrix Factorization (NMF) to obtain a parts-based spectral representation, and neuron importance is scored via the Kolmogorov Smirnov (KS) distance between singular value distributions of the full and neuron-ablated NMF matrix, a closed-form criterion requiring no calibration corpus, forward pass, or gradient computation. Two data-free matrix variants (Abs, Split) generate pruning masks from weight spectral geometry alone; a calibration-guided variant (Act) is provided as an internal ablation. Evaluation spans 7 model families (up to 70B). Without fine-tuning, data-free LILA surpasses PruneNet (45M-parameter RL policy) by up to 2.12 pp in zero-shot accuracy on LLaMA-2-7B and outperforms WikiText-2-calibrated SliceGPT by up to 6.1 pp across all sparsity levels, confirming data-free performance is not an artifact of omitting activation statistics. After one LoRA RFT epoch (matched WikiText-2 protocol), LILA Act surpasses SliceGPT+RFT by 6.39 pp on average using 98.4% fewer calibration samples (16 vs 1,024) across LLaMA-2-7B and Phi-2. NTK analysis confirms a reduction in functional distortion versus random pruning, providing theoretical grounding for the spectral criterion.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
large language model pruning, structured pruning, calibration-free compression, non-negative matrix factorization, spectral geometry, Kolmogorov-Smirnov distance, singular value decomposition, neural tangent kernel, model compression, feed-forward network, parameter efficiency, data-free methods, LoRA fine-tuning, transformer compression, neuron importance scoring

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xhbwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LILA proposes structured FFN pruning using weight-derived importance scores, with data-free Abs/Split variants and an activation-guided Act variant. Its main contribution is a per-neuron KS distance between original and row-ablated singular-value distributions. The paper reports zero-shot accuracy, recovery fine-tuning, throughput, and NTK analyses across models up to 70B parameters.

Critical strengths
  • Calibration-free compression while retaining standard dense inference operations addresses a useful deployment problem.
  • The evaluation covers several architectures and model sizes, with detailed per-task results and an accessible implementation. Separating pruning from optional recovery fine-tuning is also useful.
Critical weaknesses

W1. The proposed KS score is degenerate under exact singular-value computation.

For a row mjm_j, neuron deletion gives

M−j⊤M−j=M⊤M−mj⊤mj.M_{-j}^{\top}M_{-j}=M^{\top}M-m_j^{\top}m_j.

Rank-one interlacing implies σi(M)≥σi(M−j)≥σi+1(M)\sigma_i(M)\geq\sigma_i(M_{-j})\geq\sigma_{i+1}(M). Consequently, the equally weighted ECDFs of their top-kk singular values have KS distance 1/k1/k whenever the retained spectra differ, and zero otherwise. With Appendix A’s k=50k=50, generic neuron ablations therefore receive the same score, 0.02, irrespective of perturbation magnitude. This undermines the central importance-ranking mechanism. The authors should explain how informative rankings arise and provide exact-SVD controls, score distributions, and sensitivity to randomized-SVD approximation accuracy.

W2. The role of NMF and the distinction from prior spectral pruning remain insufficiently established.

The principal score in Eqs. (8)–(9) compares spectra of the transformed matrix MM and its row-ablated counterpart; it does not involve the NMF factors or reconstruction. Appendix H evaluates NMF versus SVD only for the Residual and Sensitivity criteria, so it does not establish that NMF benefits the primary Spectrum criterion.

Moreover, PruneNet[1] already uses KS distance between original and compressed weight spectra as its policy penalty. The potentially distinct contribution is therefore individual-ablation scoring without policy optimization, rather than introducing KS spectral preservation. The paper should clarify this distinction and isolate the contribution of factorization to its principal method.

W3. Several comparisons do not isolate the claimed improvement.

Table 2 compares LILA on Mistral-7B against SliceGPT on LLaMA-2-7B, which cannot establish superiority on Mistral. Table 6 attributes gains to adaptive allocation while comparing different achieved sparsities: 22.84% versus 25%, and 26.61% versus 30%

Reference

[1] Sengupta et al. You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning

Questions for authors
  1. Can the authors provide exact-SVD neuron-score distributions and explain how informative rankings survive the interlacing degeneracy?
  2. What improvements remain with fixed configuration selection, equal achieved compression, matched recovery budgets, and uncertainty estimates across different pruning seeds?
1 comment
Reviewer_xhbwOpenReview ↗

Response to Author Rebuttal

Thank you for the additional experiments and clarification of NMF’s role. However, my main concerns remain unresolved. The exact-SVD control confirms the score degeneracy, while variation introduced by independent randomized SVD does not establish that the rankings reliably reflect neuron importance. Seed stability and approximation-error controls are still needed.

Moreover, under the stated k=50k=50 unweighted ECDF definition, KS scores remain multiples of 0.02 even with randomized SVD. The reported range [0.04, 0.08] therefore permits only three distinct scores, which is inconsistent with “0% tied” across 100 neurons and requires clarification.

The cross-model comparison and unequal achieved sparsities also remain confounded; removing fewer neurons does not make the adaptive gains conservative. I appreciate the expanded evaluation, but it does not resolve these central concerns. I therefore maintain my score.

Reviewer_zvJROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes LILA, an approach towards structured pruning of LLMs via Non-negative Matrix factorization. The way I understand it, the main motivation is to propose a calibration-data free criterion, claiming it "is sufficient" to avoid using calibration-data for successful pruning. The authors derive an approach based on matrix factorization, and propose several variants to build pruning criteria based on this factorization. Their approach is validated on a relatively small set of benchmark datasets and models, many of them not the state of the art.

Critical strengths

Generally, I would say that the paper is relatively easy to follow, at least most parts. Better approaches for structured pruning are certainly interesting, and although I personally find the calibration-free setting not that exciting, it is well motivated. The authors do a good job at reviewing the existing literature as far as I can tell, and the methodological section is generally well written and mathematically relatively clear.

Critical weaknesses

I have several concerns w.r.t. this paper that make me lean towards rejection. I will outline my concerns below. I will put Questions separately, although many of my concerns are related to questions where I just don't know how the authors ended up with their respective decisions. For the moment, I recommend a score of 1, but I'm looking forward to seeing the authors responses.

Soundness

  • The computational study is a) badly executed and b) does not really show the superiority of the method.

Bad execution of the computational study

  • All the models you evaluate on are old by today's standards. Pruning OPT should be not considered a valid benchmark. What about open source sota models?
  • Relatively small models: You have included a 70B model, which is not even the newest family of these models. You should include modern architectures with more parameters, if possible. Offloading and pruning should not be a problem with your method, if I am not mistaken.
  • The benchmark dataset includes only a subset of the EleutherAI set. In general, I think that you should include benchmarks that involve reasoning etc (MATH?). Furthermore, if you're including Eleuther, include all of them.
  • You included only two benchmark methods. Why not Wanda-sp which is typical (the Wanda variant adapted to structured pruning). I would definitely include something like magnitude pruning, because in my own experience, especially when allowing post-pruning FT, magnitude performs relatively good the larger these models become, see also [1].
  • If I understand correctly, you are doing some FT on some of the models. Following [1], I'd not be so sure that the superiority of your models holds in the big-model regime.

Claimed superiority of the method

  • According to Table 1, your method seems to be outperformed by a large margin (e.g. 7%) by say PruneNet. Its apparent that the claim that the calibration-data setting is basically redundant seem to be not true.

Clarity

  • Figure 1 is basically unreadable in its current form.
  • It's not clear to me why imposing non-negativity constraints yields a benefit over not doing it, see also my question.
  • In Table 1: I don't understand where this effective sparsity comes from. Are the values on the right (20, 25, 30, 25, 30) the imposed sparsities?
  • In Table 1: highlighting the "best calibration-free result per column" in bold is close to misleading.

Minor

  • The paragraph "Pruning mask construction." feel really unnecessary here.

References

  • [1]: Zimmer, Max, et al. "Perp: Rethinking the prune-retrain paradigm in the era of llms." arXiv preprint arXiv:2312.15230 (2023).
Questions for authors
  • Why are we imposing an additional constraint (non-negativity) in the first place? Whats the reason for not using SVD as people have done since forever? Whats the mathematical motivation in that regard?
Reviewer_S9eHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a data-free structured pruning framework for large language models. The method constructs non-negative matrices from FFN weights and employs NMF-based reconstruction or spectral criteria to estimate neuron importance. In particular, the proposed Spectrum criterion uses the Kolmogorov–Smirnov (KS) distance between singular value distributions before and after neuron removal. The paper also provides NTK-based theoretical analysis and evaluates the method across multiple models, demonstrating competitive accuracy and inference efficiency.

Critical strengths
  1. The paper addresses the important problem of structured LLM pruning without calibration data or gradient computation, potentially reducing the dependency on external datasets.

  2. Experiments cover multiple model architectures and scales, including 70B models. The paper also evaluates different scoring criteria, recovery fine-tuning strategies, and inference efficiency.

Critical weaknesses
  1. Neuron importance depends on both model weights and input data, as demonstrated by activation-aware pruning methods such as Wanda. Although data-free pruning is appealing, the experimental results show substantial performance degradation in several settings, raising concerns about its practical effectiveness.

  2. The paper lacks comparisons with strong data-efficient pruning methods, such as Olica (ICML 2025) and SlimLLM (ICML 2025). How does LILA compare with these methods under matched compression ratios and evaluation settings? Such comparisons are necessary to demonstrate the practical advantages of completely eliminating calibration data.

Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.