- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SparseForge, a semi-structured LLM pruning method that jointly optimizes model weights and continuous masks. It uses Hessian-guided importance scores and progressively anneals soft masks into hard 2:4 sparse patterns, followed by recovery. Experiments on several model families compare model quality, training cost, and different components of the method. The main claim is that SparseForge achieves better accuracy than existing approaches, including CAST, with fewer training tokens.
Critical strengths
-
The method jointly considers weight recovery and mask optimization. This is a reasonable approach because the quality of a sparse model depends on both the selected mask and the adjustment of the remaining weights.
-
Progressive soft-to-hard mask optimization is technically motivated. Gradually converting continuous masks into structured sparse patterns may reduce the optimization difficulty compared with applying hard constraints from the beginning.
-
The experiments cover several model families and include ablations. The evaluation extends beyond a single LLM family, and the ablations provide some evidence about the contribution of individual design choices.
Critical weaknesses
-
The verification status of the main 2:4 results is unclear. The caption of the main comparison table states that SparseForge results are pending sparsity/export verification. This is a central concern because the paper's main claims depend on the exported models satisfying the required 2:4 pattern. The paper needs to establish that the evaluated checkpoints meet this constraint.
-
The reported accuracy results are inconsistent across tables. For the 1.25B-token setting, the main comparison table reports 55.96% average accuracy, while an ablation table reports 57.27% for the full SparseForge configuration. The paper should explain whether these numbers use different evaluation suites, checkpoints, or settings. Without clarification, it is difficult to identify which result supports the main claim.
-
The training-cost accounting and comparison with CAST need clarification. The reported cost of approximately 50 GPU-hours for processing 5B tokens with a 7B model on a 32-GPU cluster is ambiguous: it is unclear whether this means aggregate GPU-hours or elapsed wall-clock time. In addition, differences in training recipes and data configurations make it difficult to attribute the claimed advantage over CAST solely to the proposed method.
Questions for authors
-
Have the checkpoints used for the main results been verified to satisfy the exact exported 2:4 sparsity pattern? If so, can you clarify why the main table still marks this verification as pending?
-
Why do the main comparison table and ablation table report 55.96% and 57.27% average accuracy, respectively, for the 1.25B-token setting? Please clarify the evaluation suite, checkpoint, and settings used for each result.
-
Does the reported 50 GPU-hours refer to aggregate GPU-hours or wall-clock time on the 32-GPU cluster? What stages are included, and can you provide a matched-protocol comparison with CAST using clearly specified data, training budgets, and export verification?