acceptodds
Withdrawn from ICLR 2027

Small LLMs: Pruning vs. Training from Scratch

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pruning promises a shortcut to strong small models. However, Liu et al. (2019) found that in CNNs, this shortcut often loses to training from scratch. LLMs are substantially different, with larger model and data scales and different architectures, so the same conclusions may not hold. We examine this by pruning Llama-3.1-8B at pruning ratios of 0.5–0.8 with six methods spanning depth, width, and sparse granularities, under two token-matched settings. (1) With the same training token budget, pruned initialization consistently outperforms random initialization. This shows that the parent model provides a strong starting point, although the advantage narrows as the training token budget grows and, for depth pruning, as the pruning ratio rises, nearly vanishing at the highest ratio we study. (2) When training from scratch is instead given the full token budget consumed by the whole pipeline, pruning at finer granularities still retains a small advantage, while coarser structured pruning can be matched or surpassed. This suggests that the parent model may transfer knowledge that additional tokens alone cannot fully recover, but only at fine granularity. This forms a clear recommendation: with a strong parent and a limited token budget, pruning is more effective; when tokens are plentiful, training from scratch is competitive at coarse granularity, so a parent is not always necessary.

Primary area:
foundation or frontier models, including LLMs
Keywords:
network pruning, large language models, initialization, structured pruning, sparse pruning, model compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_zcESOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This study compares pruning pretrained LLMs with training smaller models from scratch. Using Llama-3.1-8B, the authors evaluate six pruning methods covering depth, width, and sparse pruning under two token-matched settings. The first compares pruned and randomly initialized models using the same training token budget, while the second allows scratch training to use the total token budget of the pruning pipeline. The results show that pruning provides a clear advantage under limited training budgets, although this advantage decreases with longer training. When the full token budget is considered, scratch training becomes competitive with structured pruning, while unstructured sparse pruning shows a slight advantage in average accuracy.

Critical strengths
  • The paper empirically investigates whether pruning pretrained LLMs is more effective than training smaller models from scratch.
  • The paper provides a broad comparison of six pruning methods across different pruning granularities, ratios, and training budgets, using carefully controlled token-matched settings to assess the benefits of pruning initialization.
Critical weaknesses
  • While I generally appreciate empirical studies, the findings of this paper do not seem substantially different from what is already known. The experiments are limited to Llama-3.1-8B and primarily use a single training corpus, making it difficult to generalize the conclusions. Moreover, although the paper includes post-pruning retraining, modern LLM development often involves more complex training pipelines with multiple mid-training and post-training stages. It remains unclear how the findings would translate to such real-world training pipelines.
  • The differences in average accuracy between P200-R50 and S250 in Table 2 are relatively small, making it unclear whether they are statistically or practically meaningful without multiple training runs. Moreover, the evaluation focuses on conventional zero-shot benchmarks, and it remains unclear whether these findings extend to more challenging downstream tasks, such as multi-step reasoning and instruction following.
  • The comparison between pruning and training from scratch may be sensitive to pruning configurations and hyperparameters. Although the authors perform some architecture search, it is unclear whether the pruning methods were sufficiently optimized. Could the authors clarify whether the conclusions, particularly the cases where training from scratch outperforms pruning, would remain consistent under different pruning configurations?
  • The comparison between pruning and training from scratch may be sensitive to pruning configurations and hyperparameters. Although the authors perform some architecture search, it is unclear whether the pruning methods were sufficiently optimized. Could the authors clarify whether the conclusions, particularly the cases where training from scratch outperforms pruning (e.g., S250 vs. P200-R50 for Minitron-D and Minitron-W in Table 2), would remain consistent under different pruning configurations?
Questions for authors
  • What are the main new insights of this study compared with prior findings on pruning versus training from scratch? Given that the experiments are limited to Llama-3.1-8B and DCLM, how broadly do the authors expect these conclusions to apply?
  • Modern LLM development often involves multiple mid-training and post-training stages. How should the findings of this study be interpreted in the context of such training pipelines?
  • Beyond standard dense Transformer architectures, models using alternative attention mechanisms, such as linear attention, or mixture-of-experts architectures are becoming increasingly common. Have the authors considered whether their findings would generalize to these architectures?
  • The differences in average accuracy between P200-R50 and S250 in Table 2 are relatively small. How robust are the reported performance differences to training variability?
  • How sensitive are the pruning results to the selected configurations and hyperparameters?
Reviewer_2SUPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This empirical study compares pruning-based and random initialization for small language models across six pruning methods. An 8B Llama model pretrained on 200B DCLM tokens is pruned and retrained for 50B tokens; scratch baselines use either the same 50B-token stream or the full 250B-token union. Pruned initialization consistently improves the 50B-token comparison at the main pruning ratio. With 250B scratch tokens, the outcome varies by method: scratch overtakes both Minitron variants, while FLAP, Sheared LLaMA, and unstructured sparse pruning retain advantages. Additional experiments examine compression ratios, longer training, and hardware efficiency.

Critical strengths

The comparison is carefully controlled: pruned and randomly initialized models share the same target architecture, and training the parent on known data allows the full pipeline to be compared with scratch training on exactly the same total data. Across six pruning methods, the substantial gains over S50 provide convincing evidence that inherited weights are useful when only a limited amount of training can be devoted to the target model.

Critical weaknesses
  1. The claimed structured-versus-sparse split is inconsistent with Table 16. FLAP retains a 1.4-point average accuracy advantage over S250, larger than either unstructured sparse method's advantage, and improves all four perplexities. Sheared LLaMA also retains a small accuracy lead. These results do not support attributing the residual advantage to granularity alone. The conclusion should distinguish method-specific outcomes and explain this counterevidence before recommending different strategies by granularity.

  2. The practical budget recommendation depends on fixing recovery at 50B tokens. In Table 13, Minitron-W P200-R30 reaches 70.0% average accuracy using 230B total tokens, exceeding S250's 67.5%; P200-R50 falls to 66.3%. Thus, a reported pruning configuration already beats scratch within the nominal 250B budget. Compare the best validation-selected configurations available within each budget and explain this deterioration before interpreting the P200-R50 endpoint as evidence favoring scratch training.

  3. The abstract's interpretation of unrecovered knowledge depends on an uncertain sparse-model gap. Wanda-U and SparseGPT-U lead S250 by only 0.9 and 1.2 points, without training-seed variance; Section 4.2 calls this suggestive. Repeat these comparisons with uncertainty estimates. A reproducible gap would establish an advantage at 250B tokens, while Appendix G's alternative of difficult fixed-mask optimization would still leave the knowledge-versus-head-start question unresolved.

Questions for authors

See Critical Weaknesses.

Reviewer_bzjGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates whether pruning a larger LLM is a better way to initialize a small model than random initialization. The paper concludes that: (a) With the same training token budget, pruned initialization consistently outperforms random initialization; (b) when tokens are plentiful, training from scratch is competitive at coarse granularity, so a parent is not always necessary.

Critical strengths
  1. This paper provides a clean, token-controlled comparison reframing pruning as an initialization and comparing against the same architecture trained from scratch, rather than against the parent.
  2. I appreciate the authors clear resource-dependent recommendation (which resource is scarce such as target tokens, total tokens, or inference hardware).
  3. The findings are validated across multiple pruning methods.
Critical weaknesses
  1. Equal tokens implies the amount of text processed, but not necessarily the computational cost. As the author mentioned the need for FLOP-matched comparisons, I think it is critical to the paper's motivation: pruning is presented as a way to avoid expensive pretraining.
  2. For equal-total-token setting, the reported average-accuracy differences between pruning and scratch training are quite marginal (0.5-1%) and this is important because the paper's conclusion is that fine-grained pruning preserves knowledge that additional tokens alone cannot recover.
  3. While I can understand the limit of compute resources, for empirical papers with ICLR standard, Llama-3.1-8B and DCLM-Baseline-1.0 only investigations is not sufficient.
  4. The paper states sparse pruning preserves weight-level information that additional training cannot fully recover BUT experiments compare final performance and do not directly measure what information transfers from the parent model or why sparse models retain an advantage.
Questions for authors
  1. For the sparse S50/S250 baselines, is the mask random or inherited from the pruned parent?
  2. Which WikiText split is used for perplexity, and does it overlap the Minitron calibration set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.