acceptodds
Under review as a conference paper at ICLR 2027

Activation Sparsity as Training–Inference Co-Design

Abstract

Activation sparsity can accelerate transformer inference when a specialized kernel can skip computation associated with zero activations. We ask how broadly and aggressively train-time sparsification should be applied to obtain such exploitable sparsity without excessive quality loss. Using matched Pythia-family pretraining runs, we add fixed-threshold nonlinearities at selected sites in the network forward pass, systematically vary activation pressure, and connect the resulting sparsity to a specialized-kernel execution. We find that extending sparsification across more activation sites can substantially increase model-wide sparsity without reducing latency. At 14M parameters, more than doubling model-wide sparsity from 12.71% to 27.48% slightly increases latency. Conditional kernel analysis identifies the FFN hidden activation \(h\) and attention-output projection input \(z\) as the most useful sparsification sites. Pressure applied only at \(h\) also increases sparsity at \(z\), despite \(z\) receiving no direct pressure. Broader pressure produces stronger optimization conflict at less responsive sites. These observations motivate a targeted intervention that achieves lower validation loss than the dense control while reducing latency from 0.652 to 0.573 ms. Scaling analysis confirms the susceptibility to sparsification of but reveals an increasing overhead from the specialized kernel. Our results suggest that sparsification is best treated as a training–inference co-design problem: rather than maximizing sparsity globally, training should induce specific exploitable structure that matches the inference system.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
activation sparsity, transformer sparsification, sparse inference, training–inference co-design, specialized kernels, inference efficiency, hardware-aware training

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_BjuxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The study examines which parts of a model should have their activations sparsified during training to achieve real-world inference speedups. The study distinguishes between placing activation sparsification based on a fixed threshold versus placing it based on activation pressure. Additionally, the study develops an Adam-preconditioned, conflict-projected L1 correction (OL1) method. The study compares different intervention strategies using matched Pythia-family pretraining runs on MiniPile, primarily at 14M parameters, but also at 31M, 70M, and 410M parameters. The study uses a custom inference implementation and conditional skipping ablations to distinguish between logical sparsity and executable savings. The primary findings are that applying activation sparsification too broadly can create many zeros but will not improve latency; the FFN hidden activations h and the attention-output projection inputs z are the best locations for activation sparsification in this implementation; and increasing pressure on h will increase sparsity at unpressured z locations. The targeted strategy improves both validation loss and latency at 14M, however, larger-scale experiments demonstrate quality losses and significant kernel overhead.

Critical strengths
  1. This study links training interventions to actual performance measures instead of using activation-zero counts as the sole measure of efficiency. Table 2 illustrates how increasing the overall sparsity of the model from 12.71% to 27.48% did not result in lower latency; however, Table 4 distinguishes between logical sparsity, bypassed matrix instructions, and conditional time savings. This is an important distinction when developing sparse models. Additionally, Appendix E describes a timing protocol for cudagraphs and compares them numerically to the corresponding pytorch computations.

  2. Matched initialization, data order and token budgets make the intervention comparisons interpretable. Figure 3 provides an interesting observation: pressure only at h increases sparsity at z under both T4 and T7, suggesting nonlocal adaptation. The paper also reports negative scaling results and implementation limitations candidly rather than presenting only the favorable 14M operating point.

Critical weaknesses
  1. The experiments provide useful observations, but the depth of the resulting insight is still limited. The general principle that sparsity must match the execution system is well motivated; the more interesting question is what the study establishes beyond that principle. The increase in z sparsity under pressure only at h is a promising finding (Figure 3), but its mechanism and conditions of occurrence remain unclear. The conditional timing analysis explains why h/z are profitable in the presented implementation, whose execution paths can avoid weight reads and exploit sparse rows (Appendix E). That implementation dependence is appropriate for a co-design study, rather than a flaw by itself. However, the paper offers limited evidence about which findings should transfer to another kernel or model configuration, making its contribution stronger as a case study than as a general guide to sparse model design.

  2. The evidence for the proposed targeted recipe is not yet strong enough to compensate for this limited explanatory depth. At 14M, the simultaneous improvement in validation loss and latency is encouraging. At 31M, the targeted recipe gives speedups with higher loss: for example, kappa=0.1 yields 0.908 ms and loss 4.757 versus 1.024 ms and 4.566 for Base (Table 9). At 70M, that setting is both slower and higher-loss than Base (1.96 versus 1.61 ms; 4.18 versus 4.10), while the faster kappa=0.5 point has loss 4.84 (Table 8). The authors acknowledge these limitations, and they do not invalidate the small-model observations, but they leave the practical benefit beyond 14M unestablished at comparable quality. In addition, a matched T2/P0 control is absent, and Table 5 compares standard L1 with OL1 only in the single-site T1/P1 setting. Thus, the incremental value of OL1 within the proposed targeted recipe is not clearly isolated from the effects of threshold placement and the changed nonlinearities. My concern is the strength and significance of the demonstrated contribution, rather than a requirement for production-scale serving experiments.

Questions for authors
  1. What evidence or explanation can you offer for the h-to-z sparsity spillover? In particular, which aspects of this observation do you expect to persist beyond the tested recipes, and what supports that expectation? This is potentially the most informative finding in the paper, so a clearer account would materially strengthen the contribution.

  2. Do you have a matched T2/P0 control and a standard-L1 comparison for the targeted recipe? These would clarify whether OL1 provides an advantage beyond the choice of thresholding sites and nonlinearities.

  3. Do the existing 31M/70M results contain an operating point with a latency improvement and validation loss comparable to Base? If not, can you clarify what the scaling experiments establish about the usefulness of the proposed recipe, separately from the overhead of the current kernel? I am asking about the interpretation of the present results, rather than requiring an expanded production-serving evaluation.

Reviewer_f53GOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies activation sparsification in language models. They propose a different version of L1 regularization, which they call orthogonal L1 (OL1). OL1 removes the component of the update aligned with the task direction. The authors apply OL1 to different components of the model and measure the effective sparsity induced and the wall-clock speed. Experiments are done on Pythia models. The authors claim that increasing sparsity in the activation does not necessarily reduce latency .

Critical strengths

This paper studies a key question about activation sparsity to speed up LLM inference. The idea is of using orthogonal L1 update is interesting but the paper has several limitations (discussed below).

Critical weaknesses
  1. The paper is difficult to follow, and many part of the manuscript needs to re-written to make it easier for the reader to understand the paper.

  2. Missing benchmark: There are tons of methods proposed in recent years for activation sparsity. The paper has no comparison with the existing methods, and currently reads more like an ablation than a new methodology.

  3. When h is targeted, the thresholding nonlinearity replaces GELU.

Why do we need to replace GELU? It is not clear if this confounds with the OL1 regularizer being proposed.

  1. Limited Inference eval: the authors should compare wall-clock timing under difference eval settings (e.g diff batch size etc)

Questions for authors
  1. Can you provide comparison with exsiting methods?

  2. Was the wall clock timing measured under same conditon and env? Are the numbers reported avg of multiple runs?

Reviewer_peQmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies where training time activation sparsification should be applied to obtain useful inference savings. Under. matched Pythia-family pretraining runs on MiniPile, the authors compare interventions across seven activation sites and evaluate validation loss, model wide logical sparsity and end-to-end latency using specialized GPU kernels. The main results are that increasing logical sparsity does not necessarily reduce latency, and that the FFN hidden activation and attention-output projection input provide the clearest execution benefits in the studied implementation. Experiments at 31M and 70M examine the resulting trade-offs and reveal substantial kernel overhead at larger scales.

Critical strengths

1. Separates sparsity from realized speedup. The paper usefully distinguishes zero-valued activations, skipped matrix instructions, and end-to-end latency. At 14M and κ=0.5\kappa = 0.5, increasing model-wide sparsity from 12.71% to 27.48% does not improve latency, which rises slightly from 0.459 to 0.473 ms (Table 2). The conditional execution ablations in Table 4 then localize the benefit: disabling skipping at hh or zz increases latency by 150 µs and 29 µs, respectively, while no other path yields a measurable saving in this implementation. This is consequential because it provides concrete evidence about which training-induced structures the inference system actually exploits.

2. Identifies a nonlocal sparsity effect. With threshold placement and threshold value (κ=0.05\kappa = 0.05) held fixed, applying pressure only at hh increases the zero fraction at zz in every layer under both T4T_4 and T7T_7 (Figure 3). Although zz is thresholded, it receives no direct activation pressure. This suggests that training can extend sparsity beyond the directly regularized representation. The observation is potentially useful for designing interventions that obtain sparsity at multiple sites without directly penalizing all of them.

Critical weaknesses

1. The controls neither establish the added value of the targeted recipe nor isolate the placement explanation. The most consequential missing comparison is the end-to-end quality–latency frontier of the pressured FFN-only recipe, T1/P1T_1/P_1. Without it, it remains unclear whether adding thresholding at zz improves the usable quality–latency frontier beyond simpler FFN-only regularization. A T2/P0T_2/P_0 control is also missing, so the paper cannot separate the effect of pressure at hh from that of thresholding alone within the recommended recipe, or confirm that the spillover shown under T4T_4 and T7T_7 also occurs under T2T_2.

2. The joint quality–latency improvement is shown only at the smallest scale. The improvement over the dense control is demonstrated at 14M with one training seed. Relative to native dense execution, the larger-model results are substantially less favorable (Tables 8–9). At 31M, T2/PhT_2/P_h with κ=0.1\kappa = 0.1 reduces latency from 1.024 to 0.908 ms but increases loss from 4.566 to 4.757. At 70M, the same setting increases both latency (1.61 → 1.96 ms) and loss (4.10 → 4.18). The only targeted 70M setting that reduces latency, κ=0.5\kappa = 0.5 (1.21 ms), raises loss to 4.84. Thus, the scaling experiments do not establish a quality-preserving efficiency benefit beyond the smallest model.

Questions for authors
  1. Could you report end-to-end latencies for the eight already-trained T1/P1T_1/P_1 checkpoints in Table 5 (L1 and OL1, λ∈{0.05,0.1,0.5,1}\lambda \in \{0.05, 0.1, 0.5, 1\}), measured with the same specialized 14M kernel and timing protocol as Table 2, and add them to the validation-loss–latency plot in Figure 1(c) alongside T2/PhT_2/P_h?

  2. Under broad pressure, hh and zz become less sparse than under PhP_h (Figure 3). How much of this, and of the poorer broad-pressure behavior more generally, can be attributed to reduced effective pressure on hh? This could arise both from the 1/J1/J averaging in Eq. (1) and, under T7/PallT_7/P_{\text{all}}, from the active norm cap (median r/b≈46r/b \approx 46, versus about 0.1 under PhP_h). Section 5.1 acknowledges the averaging confound. Do you have a controlled comparison that keeps the pressure on hh unchanged when another target is added (e.g., summing rather than averaging across sites, with the cap inactive)?

Reviewer_2ZuBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper addresses where and how strongly a language model should be pushed toward zero activations during training so that a sparse GPU kernel can turn the zeros into real speedups.
  • It proposes a controlled study that adds threshold activations and a sparsity penalty at different points of small transformers and times each trained model with a matching sparse kernel.
  • The study finds that spreading sparsity over many sites raises the number of zeros but not the speed, and that a targeted recipe on two sites is both faster and slightly better than the dense model at the smallest scale.
Critical strengths
  1. The paper shows clearly that more zeros do not mean more speed.
Critical weaknesses
  1. The main positive result does not hold beyond the smallest model. Only at 14M parameters is the targeted recipe both faster and better than the dense model.
Questions for authors
  1. At 31M or 70M parameters, is there any setting where the targeted recipe is faster than plain dense execution with only a small increase in loss?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.