- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The study examines which parts of a model should have their activations sparsified during training to achieve real-world inference speedups. The study distinguishes between placing activation sparsification based on a fixed threshold versus placing it based on activation pressure. Additionally, the study develops an Adam-preconditioned, conflict-projected L1 correction (OL1) method. The study compares different intervention strategies using matched Pythia-family pretraining runs on MiniPile, primarily at 14M parameters, but also at 31M, 70M, and 410M parameters. The study uses a custom inference implementation and conditional skipping ablations to distinguish between logical sparsity and executable savings. The primary findings are that applying activation sparsification too broadly can create many zeros but will not improve latency; the FFN hidden activations h and the attention-output projection inputs z are the best locations for activation sparsification in this implementation; and increasing pressure on h will increase sparsity at unpressured z locations. The targeted strategy improves both validation loss and latency at 14M, however, larger-scale experiments demonstrate quality losses and significant kernel overhead.
Critical strengths
-
This study links training interventions to actual performance measures instead of using activation-zero counts as the sole measure of efficiency. Table 2 illustrates how increasing the overall sparsity of the model from 12.71% to 27.48% did not result in lower latency; however, Table 4 distinguishes between logical sparsity, bypassed matrix instructions, and conditional time savings. This is an important distinction when developing sparse models. Additionally, Appendix E describes a timing protocol for cudagraphs and compares them numerically to the corresponding pytorch computations.
-
Matched initialization, data order and token budgets make the intervention comparisons interpretable. Figure 3 provides an interesting observation: pressure only at h increases sparsity at z under both T4 and T7, suggesting nonlocal adaptation. The paper also reports negative scaling results and implementation limitations candidly rather than presenting only the favorable 14M operating point.
Critical weaknesses
-
The experiments provide useful observations, but the depth of the resulting insight is still limited. The general principle that sparsity must match the execution system is well motivated; the more interesting question is what the study establishes beyond that principle. The increase in z sparsity under pressure only at h is a promising finding (Figure 3), but its mechanism and conditions of occurrence remain unclear. The conditional timing analysis explains why h/z are profitable in the presented implementation, whose execution paths can avoid weight reads and exploit sparse rows (Appendix E). That implementation dependence is appropriate for a co-design study, rather than a flaw by itself. However, the paper offers limited evidence about which findings should transfer to another kernel or model configuration, making its contribution stronger as a case study than as a general guide to sparse model design.
-
The evidence for the proposed targeted recipe is not yet strong enough to compensate for this limited explanatory depth. At 14M, the simultaneous improvement in validation loss and latency is encouraging. At 31M, the targeted recipe gives speedups with higher loss: for example, kappa=0.1 yields 0.908 ms and loss 4.757 versus 1.024 ms and 4.566 for Base (Table 9). At 70M, that setting is both slower and higher-loss than Base (1.96 versus 1.61 ms; 4.18 versus 4.10), while the faster kappa=0.5 point has loss 4.84 (Table 8). The authors acknowledge these limitations, and they do not invalidate the small-model observations, but they leave the practical benefit beyond 14M unestablished at comparable quality. In addition, a matched T2/P0 control is absent, and Table 5 compares standard L1 with OL1 only in the single-site T1/P1 setting. Thus, the incremental value of OL1 within the proposed targeted recipe is not clearly isolated from the effects of threshold placement and the changed nonlinearities. My concern is the strength and significance of the demonstrated contribution, rather than a requirement for production-scale serving experiments.
Questions for authors
-
What evidence or explanation can you offer for the h-to-z sparsity spillover? In particular, which aspects of this observation do you expect to persist beyond the tested recipes, and what supports that expectation? This is potentially the most informative finding in the paper, so a clearer account would materially strengthen the contribution.
-
Do you have a matched T2/P0 control and a standard-L1 comparison for the targeted recipe? These would clarify whether OL1 provides an advantage beyond the choice of thresholding sites and nonlinearities.
-
Do the existing 31M/70M results contain an operating point with a latency improvement and validation loss comparable to Base? If not, can you clarify what the scaling experiments establish about the usefulness of the proposed recipe, separately from the overhead of the current kernel? I am asking about the interpretation of the present results, rather than requiring an expanded production-serving evaluation.