acceptodds
Under review as a conference paper at ICLR 2027

The Fixed Cost Is Worth Cutting: End-to-End Speedups for Activation-Sparse LLMs

Abstract

The roughly 90% of FFN intermediate rows that sparsity-trained language models leave dead at inference time cannot be skipped for free: conditional-skipping mechanisms pay costs that do not shrink with the amount they skip, and end-to-end speedups have stayed far below this byte accounting. We capture this in a compact cost model, T(a) = F + a·B/β, which yields a decision rule—cut the fixed cost. Guided by this rule, we rebuild the sparse decode path around a lean row-skipping kernel, a sparsity-aware int8 recipe, and whole-step CUDA graphs. The rebuilt stack reaches 2.66× over the model as shipped and posts 1.38×, 1.55×, and 1.94× at matched graph caliber over the official CUDA-graph baseline, a state-of-the-art sparse kernel, and the strongest measured dense stack, while sitting 1.36× above the sustained-bandwidth floor. The stack is a sparsity-trained LLaMA-2-7B under single-batch decoding on one RTX 5090, with scale probed separately (Sec. 4.4). The same experiments expose a second, temporal axis of redundancy absent in dense controls: deep FFN inputs hold a cosine of at least 0.999 across adjacent steps on 46.4% of step pairs at 7B and 82.1% at 13B, orthogonal to spatial skipping and strengthening with scale. A training-free, zero-parameter reuse mechanism cuts 18.7% off the decode step on repetition-heavy decoding (a 1.23× speedup at greedy-token agreement 1.0) at zero measured cost on natural text, with its reused steps' divergence measured and bounded. On long-chain arithmetic reasoning the full stack stays within the pre-registered tolerance with a 2.0% paired flip rate, while aggressive accelerations exceed 10%. All claims rest on a pre-registered fidelity protocol (distribution KL, decision agreement, task flip). Code for the full stack will be released.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
activation sparsity, LLM inference, CUDA kernels, quantization, decoding efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_g7ZWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the cause of activation sparsity in LLMs does not translate into proportional end-to-end speedups. It introduces a fixed-cost term to characterize the overhead of conditional skipping, and builds an inference stack based on a lean row-skipping kernel, sparsity-aware int8 quantization, CUDA graphs, and a temporal FFN reuse mechanism with cross-step hidden-state similarity. The evaluation on ProSparse-LLaMA-2 reports up to 1.38× end-to-end speed over the official graph baseline and 1.55× over TEAL.

Critical strengths

None.

Critical weaknesses
  1. The presentation is poor. All the figures are not properly sized and difficult to read. The paper is also not organized clearly. It's hard to comprehend the key insight of the proposed method. Moreover, the technical novelty is limited, as the final design largely builds on existing techniques such as CUDA graphs, quantization.

  2. The end-to-end evaluation is not well controlled. In Sec. 4.3 / Table 1, the final L3 stack combines multiple optimizations, while the reported 1.38× over the official baseline and 1.55× over TEAL use bf16 baselines against the proposed int8 stack. It is therefore difficult to know how much of the gain comes from the newly proposed optimizations, and the accuracy impact of FFN reuse is not sufficiently evaluated under realistic free-running generation.

Questions for authors

See weaknesses.

Reviewer_FJTHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why high activation sparsity in LLMs does not translate into proportional inference speedups. The authors introduce a fixed-cost model to explain the overhead of conditional computation and propose an optimized inference stack combining a lightweight sparse kernel, INT8 quantization, and CUDA graphs. They also identify temporal redundancy in sparse models and explore reusing FFN outputs across decoding steps. The paper reports up to 2.66× end-to-end speedup over the original implementation on an RTX 5090.

Critical strengths
  • Relevant systems problem. Understanding why activation sparsity fails to translate into practical speedups is important for efficient LLM inference.
  • Detailed performance analysis. The paper provides extensive profiling, latency breakdowns, and ablations that help identify implementation bottlenecks.
  • Interesting empirical observation. The temporal redundancy observed in highly sparse models is potentially interesting and could motivate future research on cross-token computation reuse.
Critical weaknesses
  • Limited novelty. The main optimization stack largely combines existing techniques, including CUDA graphs, INT8 quantization, and sparse-kernel engineering. The fixed-cost model is also closely related to standard Amdahl's law and roofline analysis. It is unclear what constitutes a substantial new methodological contribution.
  • Overstated speedup claims. The headline 2.66× speedup compares against an eager baseline, while much of the improvement comes from switching to CUDA graphs. Even the reported 1.38× and 1.55× improvements over graph-enabled sparse baselines include INT8 quantization, making it difficult to attribute the gains to the proposed sparsity optimization. The BF16-only improvement over the official graph-enabled baseline is just 1.08×.
  • Limited practical value of temporal reuse. Although the observed temporal redundancy is interesting, the proposed reuse mechanism achieves essentially no speedup on natural text. Its larger gains are measured primarily on repetition-heavy generation, and the phenomenon is highly sensitive to input formatting and model choice. Furthermore, free-running generation shows nontrivial token disagreement, raising concerns about fidelity.
  • Limited generality and fidelity evaluation. The main efficiency results focus on a single ProSparse-LLaMA-2-7B model, one GPU, and batch-1 decoding. The model also has only 16% baseline accuracy on the primary GSM8K evaluation. The reported paired flip rate has a confidence interval extending beyond the authors' 3% tolerance, and one initially pre-registered fidelity threshold was subsequently relaxed. These limitations weaken the broader claims about accuracy preservation and practical applicability.
Questions for authors

See weakness.

Reviewer_jT6EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explains why activation sparsity in LLM decoding yields far smaller end-to-end speedups than byte accounting promises: conditional skipping pays a rate-independent fixed cost FF that does not shrink with the skipped fraction. Modeling a skippable segment as T(a)=F+aB/βT(a)=F+aB/\beta , the authors derive a decision-granularity rule—skipping pays only when the fixed decision cost falls below the proportional savings—and show that the proportional term already reaches 98.2% of sustained bandwidth, so only FF is worth cutting. Guided by this ledger, they combine a lean row-skipping kernel, a sparsity-aware int8 recipe, a training-free lagged-cosine dual-graph reuse mechanism, and whole-step CUDA graphs, reaching 1.38× over the official CUDA graph baseline, 1.55× over TEAL, and 2.66× over the official eager release form (cross-caliber) at 2.0% paired flip. They also map a temporal-redundancy regime in deep FFN layers and use negative results to validate the rule.

Critical strengths
  1. The paper provides an accurate mathematical model and detailed system-level analysis of the activation-sparse decoding pipeline. The fixed-cost model T(a)=F+aB/βT(a)=F+aB/\beta and the decision-granularity rule are clearly formulated and recomputable.

  2. Most measured latency numbers are supported by a ledger-style theoretical explanation. The authors also transparently disclose a small unresolved residual in the per-layer gate ablation rather than overclaiming a complete decomposition.

Critical weaknesses
  1. The end-to-end main experiments are restricted to ProSparse-LLaMA-2 7B/13B, a single RTX 5090, and the authors' self-built batch=1 harness, rather than production engines such as vLLM or SGLang. The 12-model analysis is mainly a phenomenon-mapping study, not an end-to-end acceleration validation.

  2. The novelty of individual algorithmic/system components is limited: none of the four components is new in isolation, and the main contributions lie in the cost model, design rule, and temporal-redundancy phenomenon.

  3. The paper focuses on batch=1 decode latency and does not evaluate prefill, continuous batching, or speculative decoding, which are central to industrial deployment. The fixed-cost ledger is diagnostically useful for decode, but its external validity for these scenarios remains unverified.

Questions for authors
  1. The paper reports 1.000x on natural text, 1.230x in the reuse-heavy regime, and 1.154x under the guarded 13B setting. Could the authors clarify which real deployment workloads would stably enter the reuse-heavy regime, and whether this phenomenon could be exploited beyond deep-FFN skipping, e.g., for KV-cache reuse, speculative decoding, or prefill?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.