- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This study compares pruning pretrained LLMs with training smaller models from scratch. Using Llama-3.1-8B, the authors evaluate six pruning methods covering depth, width, and sparse pruning under two token-matched settings. The first compares pruned and randomly initialized models using the same training token budget, while the second allows scratch training to use the total token budget of the pruning pipeline. The results show that pruning provides a clear advantage under limited training budgets, although this advantage decreases with longer training. When the full token budget is considered, scratch training becomes competitive with structured pruning, while unstructured sparse pruning shows a slight advantage in average accuracy.
Critical strengths
- The paper empirically investigates whether pruning pretrained LLMs is more effective than training smaller models from scratch.
- The paper provides a broad comparison of six pruning methods across different pruning granularities, ratios, and training budgets, using carefully controlled token-matched settings to assess the benefits of pruning initialization.
Critical weaknesses
- While I generally appreciate empirical studies, the findings of this paper do not seem substantially different from what is already known. The experiments are limited to Llama-3.1-8B and primarily use a single training corpus, making it difficult to generalize the conclusions. Moreover, although the paper includes post-pruning retraining, modern LLM development often involves more complex training pipelines with multiple mid-training and post-training stages. It remains unclear how the findings would translate to such real-world training pipelines.
- The differences in average accuracy between P200-R50 and S250 in Table 2 are relatively small, making it unclear whether they are statistically or practically meaningful without multiple training runs. Moreover, the evaluation focuses on conventional zero-shot benchmarks, and it remains unclear whether these findings extend to more challenging downstream tasks, such as multi-step reasoning and instruction following.
- The comparison between pruning and training from scratch may be sensitive to pruning configurations and hyperparameters. Although the authors perform some architecture search, it is unclear whether the pruning methods were sufficiently optimized. Could the authors clarify whether the conclusions, particularly the cases where training from scratch outperforms pruning, would remain consistent under different pruning configurations?
- The comparison between pruning and training from scratch may be sensitive to pruning configurations and hyperparameters. Although the authors perform some architecture search, it is unclear whether the pruning methods were sufficiently optimized. Could the authors clarify whether the conclusions, particularly the cases where training from scratch outperforms pruning (e.g., S250 vs. P200-R50 for Minitron-D and Minitron-W in Table 2), would remain consistent under different pruning configurations?
Questions for authors
- What are the main new insights of this study compared with prior findings on pruning versus training from scratch? Given that the experiments are limited to Llama-3.1-8B and DCLM, how broadly do the authors expect these conclusions to apply?
- Modern LLM development often involves multiple mid-training and post-training stages. How should the findings of this study be interpreted in the context of such training pipelines?
- Beyond standard dense Transformer architectures, models using alternative attention mechanisms, such as linear attention, or mixture-of-experts architectures are becoming increasingly common. Have the authors considered whether their findings would generalize to these architectures?
- The differences in average accuracy between P200-R50 and S250 in Table 2 are relatively small. How robust are the reported performance differences to training variability?
- How sensitive are the pruning results to the selected configurations and hyperparameters?