- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LILA proposes structured FFN pruning using weight-derived importance scores, with data-free Abs/Split variants and an activation-guided Act variant. Its main contribution is a per-neuron KS distance between original and row-ablated singular-value distributions. The paper reports zero-shot accuracy, recovery fine-tuning, throughput, and NTK analyses across models up to 70B parameters.
Critical strengths
- Calibration-free compression while retaining standard dense inference operations addresses a useful deployment problem.
- The evaluation covers several architectures and model sizes, with detailed per-task results and an accessible implementation. Separating pruning from optional recovery fine-tuning is also useful.
Critical weaknesses
W1. The proposed KS score is degenerate under exact singular-value computation.
For a row , neuron deletion gives
Rank-one interlacing implies . Consequently, the equally weighted ECDFs of their top- singular values have KS distance whenever the retained spectra differ, and zero otherwise. With Appendix A’s , generic neuron ablations therefore receive the same score, 0.02, irrespective of perturbation magnitude. This undermines the central importance-ranking mechanism. The authors should explain how informative rankings arise and provide exact-SVD controls, score distributions, and sensitivity to randomized-SVD approximation accuracy.
W2. The role of NMF and the distinction from prior spectral pruning remain insufficiently established.
The principal score in Eqs. (8)–(9) compares spectra of the transformed matrix and its row-ablated counterpart; it does not involve the NMF factors or reconstruction. Appendix H evaluates NMF versus SVD only for the Residual and Sensitivity criteria, so it does not establish that NMF benefits the primary Spectrum criterion.
Moreover, PruneNet[1] already uses KS distance between original and compressed weight spectra as its policy penalty. The potentially distinct contribution is therefore individual-ablation scoring without policy optimization, rather than introducing KS spectral preservation. The paper should clarify this distinction and isolate the contribution of factorization to its principal method.
W3. Several comparisons do not isolate the claimed improvement.
Table 2 compares LILA on Mistral-7B against SliceGPT on LLaMA-2-7B, which cannot establish superiority on Mistral. Table 6 attributes gains to adaptive allocation while comparing different achieved sparsities: 22.84% versus 25%, and 26.61% versus 30%
Reference
[1] Sengupta et al. You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning
Questions for authors
- Can the authors provide exact-SVD neuron-score distributions and explain how informative rankings survive the interlacing degeneracy?
- What improvements remain with fixed configuration selection, equal achieved compression, matched recovery budgets, and uncertainty estimates across different pruning seeds?
Response to Author Rebuttal
Thank you for the additional experiments and clarification of NMF’s role. However, my main concerns remain unresolved. The exact-SVD control confirms the score degeneracy, while variation introduced by independent randomized SVD does not establish that the rankings reliably reflect neuron importance. Seed stability and approximation-error controls are still needed.
Moreover, under the stated k=50 unweighted ECDF definition, KS scores remain multiples of 0.02 even with randomized SVD. The reported range [0.04, 0.08] therefore permits only three distinct scores, which is inconsistent with “0% tied” across 100 neurons and requires clarification.
The cross-model comparison and unequal achieved sparsities also remain confounded; removing fewer neurons does not make the adaptive gains conservative. I appreciate the expanded evaluation, but it does not resolve these central concerns. I therefore maintain my score.