- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an iterative pruning method for reasoning language models. The main idea is to use trajectories generated by the evolving sparse model to guide optimization, rather than relying entirely on fixed calibration data. SCOUT alternates between weight updates and mask updates. The mask is determined using Adam-based saliency scores, while a KL-based trust region controls how aggressively the mask can change. The authors evaluate SCOUT on Qwen3 models from 1.7B to 8B parameters at 50–80% unstructured sparsity. The results show substantial improvements over existing methods at high sparsity. The paper also provides a theoretical analysis of an idealized optimization procedure and reports decoding speedups using sparse kernels.
Critical strengths
Strong results at high sparsity. The performance improvements are substantial, particularly at 70–80% sparsity. For example, at 80% sparsity, SCOUT improves the four-task average from 18.6% to 33.0% on Qwen3-4B and from 30.4% to 41.2% on Qwen3-8B compared with ALPS+training. The inclusion of post-pruning trained baselines makes these results more convincing than comparisons with one-shot methods alone.
An interesting approach with useful empirical analysis. Combining on-policy distillation with adaptive mask refinement is a reasonable way to address the distribution shift caused by pruning. I also appreciate the additional analysis of the sparse model's own rollouts and the decoding experiments. These provide useful evidence beyond downstream accuracy alone.
Critical weaknesses
-
The contribution of the KL-based trust region could be better isolated: According to Appendix A, the "w/o TR" variant directly applies the target mask at the first projection event, whereas SCOUT gradually increases sparsity. This comparison therefore changes the sparsification trajectory as well as the use of the trust-region constraint. Since gradual pruning with continued training is already known to be effective, it is not entirely clear how much of the improvement comes specifically from KL-based mask selection. A comparison with a gradual pruning baseline using a predefined sparsity schedule and comparable training settings would make the contribution more convincing.
-
The connection between the trust-region formulation and its implementation could be clarified more: Equation (2) constrains the divergence between consecutive sparse models, accounting for changes in both weights and masks. However, the empirical KL constraint in Equation (13) is applied to the mask transition with the updated weights held fixed. The weight updates themselves are not explicitly checked against the same constraint. The KL screening also uses only four sequences of 512 tokens, sampled from a rollout cache that is refreshed periodically. Although Appendix C.4 provides additional analysis on longer rollouts, it remains unclear how sensitive the mask selection is to the screening data and its staleness. These issues do not invalidate the empirical results, but they limit how directly the theoretical motivation supports the implemented procedure.
Questions for authors
-
Could the authors compare SCOUT against gradual pruning with a predefined sparsity schedule, using the same training objective and a comparable computational budget? This would help isolate the benefit of KL-based mask selection.
-
How sensitive is the method to the size and length of the KL screening set, as well as the rollout refresh interval? Also, how much additional KL drift is introduced by the weight updates between mask projections?
-
In the "w/o on-policy" ablation, the authors remove the OPD objective and replace the rollout-based KL screening data with fixed-corpus data. Could these two effects be evaluated separately? It would also be interesting to understand the contribution of mask revival after the target sparsity has been reached.
3 comments
Rebuttal by Authors
Sensitivity of KL screening and KL drift from weight updates
How sensitive is the method to the size and length of the KL screening set, as well as the rollout refresh interval? Also, how much additional KL drift is introduced by the weight updates between mask projections? The connection between the trust-region formulation and its implementation could be clarified more
We examined the sensitivity of SCOUT to screening set size, sequence length, and rollout refresh interval. We also measured the KL drift induced by weight updates between mask projections.
1. Sensitivity to screening set size and sequence length
Along with extending the training rollout length to 2,048 tokens, we increased the default KL screening sequence length from 512 to 2,048 tokens. We then varied the screening set size and sequence length to evaluate their effects.
Screening set size (Qwen3-1.7B, 70% sparsity, Avg4, %)
| Screening sequences | 4 (default) | 8 | 16 |
|---|---|---|---|
| Avg4 | 38.63 | 38.51 | 38.52 |
Screening sequence length (Qwen3-1.7B, 70% sparsity, Avg4, %)
| Sequence length | 256 | 512 | 2,048 (default) |
|---|---|---|---|
| Avg4 | 38.70 | 38.86 | 38.63 |
Performance varies only slightly across the evaluated screening set sizes and sequence lengths, suggesting that SCOUT is relatively insensitive to these choices within the tested ranges.
2. Sensitivity to rollout refresh interval
Rollout refresh interval (Qwen3-1.7B, 70% sparsity, Avg4, %)
| Refresh interval (steps) | 8 | 32 (default) | 128 | 2,048 (no refresh) |
|---|---|---|---|---|
| Avg4 | 39.02 | 38.63 | 37.54 | 34.93 |
Unlike screening set size and length, rollout freshness has a noticeable effect. Accuracy decreases from 39.0% with an 8-step refresh interval to 34.9% without intermediate refreshes, highlighting the importance of periodically updating on-policy information.
3. KL drift from weight updates
SCOUT explicitly screens mask transitions, assuming that individual continuous weight updates induce relatively small distributional changes. To examine this assumption, we measured KL divergence after individual weight updates and cumulatively over the eight updates between mask projections.
KL drift from weight updates (Qwen3-1.7B, 70% sparsity)
| Measurement | Mean KL | Median KL | 95th percentile | Max KL |
|---|---|---|---|---|
| Single weight update | 0.0014 | 0.0011 | 0.0029 | 0.0053 |
| Between mask projections (8 steps) | 0.015 | 0.0076 | 0.047 | 0.086 |
Individual weight updates induce small KL divergence (mean 0.0014, maximum 0.0053), below the threshold of . Although the accumulated KL over eight updates can be larger (up to 0.086), gradients are recomputed at the current weights after each step, allowing the optimization direction to adapt as the model changes. Thus, a larger cumulative KL does not necessarily violate the locality assumption underlying individual weight updates.
This also helps clarify the relationship between Eq. (2) and Eq. (13). Eq. (2) presents an idealized formulation that jointly considers weight and mask updates, whereas Eq. (13) explicitly constrains mask transitions with the updated weights held fixed. Our implementation focuses on controlling these discrete transitions while relying on individual continuous weight updates to remain local, as supported by the measurements above. Importantly, the theoretical analysis in Appendix E is formulated specifically for mask transitions, consistent with Eq. (13).
We acknowledge that Eq. (2) may obscure this distinction, as the implemented constraint does not guarantee that the combined change in weights and masks remains within the same KL threshold. We will revise the manuscript to clarify the relationship between the idealized formulation, the implemented constraint, and the scope of the theoretical analysis.
Rebuttal by Authors
Separating the effects of OPD and KL screening data
In the "w/o on-policy" ablation, the authors remove the OPD objective and replace the rollout-based KL screening data with fixed-corpus data. Could these two effects be evaluated separately?
We agree that our original "w/o on-policy" ablation simultaneously removed OPD and replaced on-policy KL screening data with a fixed corpus, making their individual contributions difficult to distinguish.
To separate these effects, we conducted additional controlled experiments. First, we replaced on-policy KL screening data with a fixed corpus while retaining OPD. Second, we excluded OPD gradients from saliency estimation while keeping the full NTP+KD+OPD objective for weight optimization.
Contributions of OPD, saliency estimation, and KL screening data (Qwen3-1.7B, 70% sparsity, Avg4, %)
| Training objective | Saliency | KL screening data | Avg4 |
|---|---|---|---|
| NTP+KD+OPD | Full objective | On-policy rollout | 38.63 |
| NTP+KD+OPD | Full objective | Fixed corpus | 38.33 |
| NTP+KD+OPD | Excluding OPD | On-policy rollout | 37.07 |
| NTP+KD | Excluding OPD | Fixed corpus | 29.50 |
Replacing on-policy KL screening data with a fixed corpus has negligible effect on accuracy (38.63% vs. 38.33%). In contrast, excluding OPD gradients from saliency estimation reduces accuracy by 1.56 percentage points (38.63% to 37.07%), despite keeping the training objective unchanged. Removing OPD entirely leads to a larger degradation (38.63% to 29.50%).
These results suggest that on-policy information contributes more to mask selection through saliency estimation than through KL screening data, while removing OPD entirely leads to a substantial degradation in overall performance. We will include these ablations in the revised paper.
Contribution of mask revival after reaching target sparsity
It would also be interesting to understand the contribution of mask revival after the target sparsity has been reached.
SCOUT continues updating the mask after reaching the target sparsity, allowing previously pruned weights to be revived. To evaluate its contribution, we compared SCOUT with a variant that freezes the mask upon reaching 70% sparsity. Both variants follow the same procedure before reaching the target and continue weight optimization under identical conditions.
Effect of mask refinement after reaching target sparsity (Qwen3-1.7B, 70% sparsity, Avg4, %)
| Method | Seed 42 | Seed 1 | Seed 2 | Mean |
|---|---|---|---|---|
| SCOUT (revival enabled) | 38.73 | 38.66 | 38.50 | 38.63 |
| Fixed mask after reaching 70% | 37.82 | 37.13 | 37.84 | 37.60 |
Allowing continued mask refinement improves average accuracy by 1.03 percentage points. We also measured the frequency of revival events after reaching the target sparsity.
Revival events per interval
| Steps | 201–512 | 513–1,024 | 1,025–1,536 | 1,537–2,048 | Total |
|---|---|---|---|---|---|
| Revival events (%) | 0.011 | 0.046 | 0.030 | 0.005 | 0.092 |
Percentages are relative to the number of active weights at the target sparsity.
Cumulative revival events account for only 0.092% of active weights, indicating that SCOUT does not rely on extensive mask changes after reaching the target sparsity. Continued mask refinement provides a modest additional benefit rather than being the primary source of SCOUT's performance gains. We will include this analysis in the revised paper.
Rebuttal by Authors
We appreciate the reviewer’s careful assessment of our work and the positive evaluation of our high-sparsity results and empirical analyses. We understand the concerns regarding the contribution of the KL trust region, its implementation, and the role of on-policy information. To address these concerns, we conducted additional controlled experiments.
Following Reviewer Q4RB's feedback, we extended the on-policy training rollout length from 512 to 2,048 tokens to better reflect the long-generation setting of reasoning models. Unless otherwise specified, the following experiments use Qwen3-1.7B at 70% sparsity, with 2,048 optimization steps, the NTP+KD+OPD objective, and mask updates every 8 steps. We report Avg4, the average accuracy across MATH-500, GSM8K, LiveCodeBench, and IFEval, and describe the specific changes made in each ablation below. For the default SCOUT configuration, we report the mean over three training seeds (38.63%); all other variants and baselines use a single training seed unless otherwise noted.
Comparison with predefined gradual pruning schedules
To distinguish the contribution of KL-based mask updates from conventional gradual pruning, we compared SCOUT with cubic and linear sparsity schedules. Both schedules were configured to reach the target sparsity at step 200, matching SCOUT. We report results across three training seeds.
Comparison of sparsity schedules (Qwen3-1.7B, 70% sparsity, Avg4, %)SCOUT achieves the highest accuracy across all three seeds, although cubic pruning remains competitive. Although cubic pruning exceeds the KL threshold slightly more often, it induces substantially smaller maximum KL divergence than linear pruning, consistent with our motivation to limit abrupt distributional changes during pruning.
However, both predefined schedules produce mask updates that exceed the KL threshold. While such schedules implicitly control distributional changes through their pruning rates, SCOUT explicitly measures and constrains KL divergence at each mask update. Although SCOUT still requires choosing δ, this parameter directly specifies the allowable distributional change rather than relying on a predefined sparsity trajectory. We will incorporate these comparisons into the revised paper.