acceptodds
Under review as a conference paper at ICLR 2027

SGA-Sparse: Training Native SwiGLU Gates for Activation-Sparse and Low-Bit LLMs

Abstract

Activation sparsity can reduce FFN weight traffic during single-user LLM decoding, but active channels must be identified early enough to avoid fetching unused weights. In pretrained SwiGLU models, training-free routing with the native gate performs substantially worse than routing with the up activation at high sparsity. We show that this ordering reverses after sparsity-aware continued pretraining. To the best of our knowledge, we present the first systematic study that converts pretrained SwiGLU LLMs into fixed-budget activation-sparse models by training the existing gate projection as the router. Our method, SGA-Sparse, updates all model parameters through continued pretraining while preserving the original SwiGLU activation and introducing no auxiliary router parameters. After approximately 30B training tokens, LLaMA models from 1B to 8B achieve uniform per-token 75–85% FFN channel sparsity while remaining within 0.2–2.7 pp of their original dense checkpoints in average zero-shot accuracy. We further study two complementary QAT recipes: continued-QAT largely recovers the performance at 4-bit weight quantization, while joint-QAT performs better for the more aggressive sub-4-bit target. To compare activation sparsity with dense low-bit quantization, we translate the sparsity-induced reduction in weight traffic into an equivalent bit-width. Under this common idealized metric, our 2.5-bit-equivalent models achieve average zero-shot accuracy comparable to dense 3-bit QAT models, while our 1.5-bit-equivalent models approach dense 2-bit models and outperform dense ternary models by 1.3 pp on average. These results establish native-gate training as a practical approach for combining high FFN sparsity with low-bit weights.

Primary area:
foundation or frontier models, including LLMs
Keywords:
activation sparsity, efficient LLM inference, SwiGLU, on-device LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sSpPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Studies whether the existing gate projection in a pretrained SwiGLU FFN can be trained to select a small subset of intermediate channels. SGA-Sparse computes the gate densely, selects its top-k activations, and evaluates only the corresponding channels in the up and down projections. All model parameters are updated through continued pretraining, without adding a separate router or replacing SiLU. The main finding is that gate-based routing, although worse than up-based routing in the frozen model, becomes competitive or better after training. The paper also studies how to combine this sparsity with weight quantization through continued and joint QAT.

Critical strengths
  • The comparison between gate- and up-based routing is the most interesting part of the paper.
  • The recovered accuracy at high sparsity is promising. After approximately 30B training tokens, the 3B model retains 61.2% average accuracy at 85% FFN-channel sparsity, compared with 62.6% for the original dense checkpoint.
  • The method is straightforward and preserves the pretrained FFN structure.
Critical weaknesses
  • The paper does not yet demonstrate the end-to-end inference benefit that motivates the method. Figure 3(b) measures the gate, up, and down GEMVs separately and sums their latencies, explicitly excluding top-k selection and elementwise operations. However, selecting channels before fetching their weights is the central systems challenge here. The complete execution path includes a dependency from the dense gate computation through selection to the sparse projections, along with indexing and low-bit weight access. I would like to see complete FFN latency and model-level decoding throughput before treating the reported projection speedups as evidence of practical acceleration.
  • The equivalent-bit comparison is useful, but does not establish equivalent deployment cost. The 1.5-bit and 2.5-bit labels describe idealized per-token weight traffic, not model storage. All weights still need to be stored, and the calculation excludes quantization metadata, routing information, and memory-access inefficiencies.
  • The dense low-bit comparison uses unequal training budgets.
  • The closest gate-based baseline deserves a stronger comparison. CATS is evaluated as a frozen, controlled top-k baseline, while SGA-Sparse receives substantial continued pretraining. It may also be worthwhile to compare and contrast against RECAP [1]
  • Most results concern eight zero-shot tasks and WikiText-2 perplexity. The MMLU and GSM8K ablations are useful, but there is little evidence on sustained generation, coding, long-context use, or instruction following.

[1] Lee, Mingyu, Akshat Ramachandran, and Tushar Krishna. "Recap: Training-free compensation for coarse activation channel pruning in compressed llms." Machine Learning for Computer Architecture and Systems 2 (2025).

Questions for authors

Please see above.

3 comments
AuthorsOpenReview ↗

response to Reviewer sSpP, part-1

We thank the reviewer for the valuable and detailed comments. For the discussion, we provide concise summaries of each response below. The complete tables and writeups will be included in the revised PDF, which we will upload shortly.

a) FFN-level latency and end-to-end decoding performance. (Q1)

Answer: Thank you for the detailed comment! We agree that selecting channels before fetching their weights is the central systems challenge, and we now measure the complete execution path. Our FFN runs as three kernels in one CUDA graph:

  • (1) a dense gate GEMV, with SiLU and index selection fused into its epilogue.
  • (2) a gathered up projection with the element-wise product fused in.
  • (3) a gathered down projection.

Unlike predictor-based methods, SGA-Sparse uses the model's own gate projection as the router. It adds no auxiliary predictor and no extra weight traffic, and the gate is already on the critical path of dense SwiGLU. The only new ordering constraint is that the up projection starts after selection, which costs one kernel boundary and is included in all our measurements.

On an A100, our complete FFN (including exact top-k, all metadata, and element-wise operations) takes 34.5/40.8/62.0 µs per layer on LLaMA-3-1B/3B/8B, at 75% sparsity on top of W4-g128. Compared to dense W4-g128, our sparse FFN is 1.11x/1.22x/1.42x faster. At the model level, decoding reaches 1062/493/304 tokens/s, which is 2.02x/2.27x/2.90x over dense bf16 and 1.06x/1.12x/1.25x over dense W4. Please kindly refer to our response (b) to Reviewer NRUr for details.

b) Cost of quantization metadata and routing indices. (Q2)

Answer: Thanks for the comment. We agree, and we will clarify in the revision that the 2.5- and 1.5-bit labels describe per-token weight traffic, not storage. For example, for our SGA-S(W2.5), all weights remain stored in 4-bit (~4.9 GB for LLaMA-3.1-8B vs. ~16 GB in bf16), while only ~2.4 GB are read per token. Counting quantization metadata raises the labels to 2.58 and 1.58 bits, and routing indices add only ~0.01 bit more. Please kindly refer to our response (a) to Reviewer NRUr and (d) to Reviewer HeBK for more details. Memory-access inefficiencies are captured by our end-to-end measurements: on the A100, the sparse FFN reaches 60–78% of its ideal byte-proportional speedup over dense W4, and the model-level gains above already include all of these costs. We will make sure to add these clarifications, measurements, and explanations in the appendix of the revised manuscript.

c) Training budget mismatch between QAT and SGA-Sparse. (Q3)

Answer: Thanks for the comment. We used a smaller training budget for the dense QAT baselines to save GPU resources, following the same practice as ParetoQ, which also cuts the training budget once performance saturates. For example, extending 2-bit QAT on LLaMA-3.2-1B from 20B to 30B tokens changes the average accuracy by only −0.2 pp (53.9 to 53.7). Our dense QAT runs also reproduce the degradation reported in the ParetoQ paper. Please kindly refer to our response (e) to Reviewer HeBK for details.

AuthorsOpenReview ↗

response to Reviewer sSpP, part-2

d) Additional comparison with CATS and RECAP. (Q4)

Answer: Thanks for the valuable suggestion. We agree that CATS deserves a comparison beyond the controlled top-k version. We therefore additionally implement the original threshold-based CATS: per-layer thresholds are set from a percentile of the dense |SiLU(gate)| activations in one calibration pass over C4 on the pretrained model. For a fair comparison, we also apply thresholding to SGA-Sparse, calibrating thresholds on the sparsity-trained checkpoint to the same target sparsity.

Native threshold-based CATS performs no better than its top-k counterpart (and is worse at 85% sparsity), so the controlled top-k baseline in Table 1 does not understate CATS. SGA-Sparse remains robust under thresholding, staying within 0.7 pp of its top-k accuracy, and outperforms CATS by 15–24 pp under both selection schemes. Note that applying our gate-based selector without training is exactly CATS; the difference lies in sparsity-aware training, under which the native gate signal becomes the better router (Fig. 2).

Avg. Acc.1B 75%1B 85%3B 75%3B 85%8B 75%8B 85%
CATS (top-k)39.737.543.037.551.042.0
CATS (threshold)38.837.041.136.251.040.6
SGA-S (top-k)54.754.161.561.266.065.0
SGA-S (threshold)54.553.761.060.566.064.3

Thank you also for pointing us to RECAP. We implement its main method (channel pruning with Ng = 8 compensation) and evaluate it under our training-free protocol, sparsifying the inputs of the up and down projections of the pretrained LLaMA models at 75% and 85%. RECAP achieves the highest average accuracy among the training-free baselines, while remaining well below SGA-Sparse. RECAP selects channels per input sequence and compensates the pruned channels with group-wise average magnitudes. It is designed to reduce GEMM computation, and its compensation term reads the pruned weight rows from HBM. It therefore targets computation rather than per-token weight traffic in decoding, and we view it as complementary to our method. We will add RECAP to the related work and include the results and discussion in the appendix.

Avg. Acc.1B 75%1B 85%3B 75%3B 85%8B 75%8B 85%
R-Sparse (best in Table 1)43.638.249.740.957.044.8
RECAP46.244.353.951.259.357.0
SGA-S54.754.161.561.266.065.0

e) More evaluation datasets (Q5)

Answer: Thanks for the suggestion. We additionally evaluate SGA-Sparse on instruction following (IFEval), coding (MBPP), and harder reasoning and knowledge benchmarks (MMLU-Pro, BBH, MuSR, GPQA), together with 5-shot MMLU and GSM8K, comparing against Q-Sparse on Qwen-3.5-4B and -9B at 75% and 85% sparsity. IFEval, MBPP and GSM8K require free-form multi-step generation rather than likelihood scoring, so they also probe sustained generation. SGA-Sparse achieves a higher overall average in all four settings (+2.7/+4.8 pp on 4B and +1.0/+2.6 pp on 9B at 75%/85% sparsity), with the largest gains on generation-heavy GSM8K (+7.4/+13.2 pp on 4B). Please kindly refer to our response (c) to Reviewer NRUr for the full table. Long-context evaluation requires long-context training data and post-training beyond our continued-pretraining setup, which we list as future work.

AuthorsOpenReview ↗

followup to Reviewer sSpP, updated PDF

We have uploaded the revised PDF, with all changes highlighted in magenta. The new Appendix C section on the GPU kernel reports the complete FFN latency and end-to-end decoding speedups, and clarifies that the equivalent bit widths describe per-token weight traffic rather than storage; we will release the kernel code upon acceptance. The appendix D.5 and D.6 add the comparisons with threshold-based CATS and with RECAP (RECAP is also discussed in the related work), and Table 6 extends the evaluation to additional benchmarks. Thank you again for the detailed and constructive comments.

Reviewer_HeBKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SGA-Sparse that exploits activation sparsity to reduce the FFN weight traffic that must be loaded when serving LLMs, speeding up the decoding phase. The method is a solution to the unstable activation sparsity of SwiGLU models, where the post-gating activation, which evidently provides the most accurate channel-importance signal, is only available at a late point in the FFN, and the gate signal ignores the up-projection output, so a channel with a small gate value can remain important when its activation has a large magnitude. The paper presents itself as the first systematic study that converts pretrained SwiGLU LLMs into activation-sparse models by training the original native gate as the router.

The method is evaluated across LLaMA 3 models ranging from 1B to 8B, achieving 75–85% sparsity with roughly 30B training tokens while remaining within 2.7 p.p. of the dense models. The paper investigates both QAT applied after sparsity-aware training (continued-QAT) and joint-QAT, where sparsity and quantization are trained together. It demonstrates that although the strongest routing signal in frozen models is the up activation, this ordering changes with training, and the native gate becomes the best native routing signal for the trained model. Finally, the 2.5- and 1.5-bit-equivalent SGA-Sparse models are claimed to introduce new Pareto-efficient operating points.

Critical strengths
  • A clear problem that is addressed, here through training without further architectural changes or specially trained add-ons.
  • Clear mathematical definitions, written as equations backed by textual descriptions, making them easy to follow. This also includes a clearly described gradient flow.
  • Clearly outperforms the competitors, both the training-free baselines in Table 1 and the trained baselines at a matched token budget in Table 2.
  • The experimental setup is clearly described.
  • The table comparing Joint-QAT and Continued-QAT is important as both insights on the method and perspective in the QAT landscape, where there has been reporting of different things on different bit-depths.
Critical weaknesses
  • Even though the method is tested against relevant methods, it is only tested on older LLaMA family models up to 8B, with a sole cross-validity check on a Qwen 3.5 4B model.
  • Regarding the above, it would, e.g., be interesting to see whether the native routing signal is in general the best across architectures.
  • The paper reports the latency on an A100 GPU, measured for the gate, up and down projections, and sums the latencies. We do not have any evidence on the actual decoding speedup to weigh against the invested compute of the method.
  • Regarding the above, we would also like an overview of the memory savings, e.g. at runtime: can this run on hardware with smaller memory consumption?
  • The Pareto comparison in Fig. 3a is not budget-matched: SGA-S trains 30-35B tokens while the dense ParetoQ baselines train only ~10.5B (W3) and ~21B (W2/W1.58) tokens (Table 8). The claim of new Pareto-efficient operating points may therefore reflect the extra training rather than the sparsity itself.
  • The quantized models (Tables 3, 4 and Fig. 3a) are only evaluated on Wiki2 and eight zero-shot commonsense tasks. Sub-4-bit quantization is known to affect knowledge-heavy tasks the most, yet no MMLU/GSM8K results are reported for any quantized model, and Table 4 has no per-task breakdown.
  • The claim that PTQ consistently degrades sparsity-trained checkpoints is shown with GPTQ only. The main results also use a single seed, and margins as small as 0.2 p.p. are highlighted without error bars.
Questions for authors
  • Would you expect this to generalize to other architectures using SwiGLU?
  • Do you have any idea on how this method will work on looped transformers?
  • Do we see any gains for decoding performance across architectures?
3 comments
AuthorsOpenReview ↗

response to Reviewer HeBK, part-1

We thank the reviewer for the valuable, detailed and supportive comments, and questions. For the discussion, we provide concise summaries of each response below. The complete tables and writeups will be included in the revised PDF, which we will upload shortly.

a) The generalization to other SwiGLU model architectures. (W1, W2, Q1)

Answer: Thanks for the comments and the question. As you mention, in the paper, we mainly study the GQA-based SwiGLU models (LLaMA-3), and we add one ablation for the GDN-based SwiGLU models (Qwen-3.5 4B). To answer your question, we further study four additional models to check whether the native-routing signal idea generalizes.

We run additional Table 5 experiments on Qwen-3.5-9B, Mobile-MoE-S (1.3B-A-272M), Mobile-MoE-L (5.3B-A-922M), and Ouro-1.4B (R4). Those additional studies cover different sizes of GDN-based, MoE-based, and Looped-based SwiGLU models.

The native gate signal generalizes across all of them. Gate-based routing gives lower PPL than up-based routing in all settings. At 85% sparsity, it also achieves higher average accuracy on every model (+0.9 to +1.7pp). At 75%, it is better or within 0.1 pp on all models.

Training budgets: 10B tokens for Qwen-3.5 and Mobile-MoE-L, 30B tokens for Mobile-MoE-S, and 5B tokens for Ouro.

Wiki2 / Avg. Acc.Dense75% up75% gate85% up85% gate
Qwen-3.5-4B8.0 / 66.69.1 / 64.78.8 / 65.69.5 / 63.48.9 / 64.7
Qwen-3.5-9B6.9 / 68.88.1 / 67.07.7 / 67.98.5 / 65.87.9 / 67.4
Mobile-MoE-S10.7 / 56.414.9 / 53.014.5 / 53.015.7 / 51.815.1 / 52.7
Mobile-MoE-L7.3 / 66.39.9 / 60.49.4 / 61.310.9 / 58.310.1 / 60.0
Ouro-1.4B (R4)8.6 / 63.910.1 / 60.99.9 / 60.810.9 / 58.310.6 / 59.2

b) Implication on Looped transformers. (Q2)

Answer: Thank you, this is a really interesting question. We repeat the same analysis in Fig. 2 on Ouro-1.4B, a 1.4B looped language model that runs 4 loops. The same trends of up- vs gate-based signals have been identified for both training-free and training scenarios.

Without training, up-based routing is better at every sparsity level, and the gap widens with sparsity. After 5B tokens of sparsity-aware training, the trend reverses as in Fig. 2: gate-based routing gives lower perplexity at both sparsity levels. For downstream accuracy, it ties at 75% (60.8 vs. 60.9) and is better at 85% (59.2 vs. 58.3). This suggests that the training-induced preference for the native gate signal also holds when the FFN weights are reused across loops.

Wiki2 / Avg. Acc.TF (training-free) 25%TF 50%TF 75%TF 85%SGA-S 75%SGA-S 85%
up8.7 / 63.79.7 / 62.125.8 / 52.3490.6 / 42.510.1 / 60.910.9 / 58.3
gate8.8 / 63.211.6 / 59.8100 / 45.62075 / 39.89.9 / 60.810.6 / 59.2

c) The A100 decoding speed. (W3, Q3)

Answer: Thanks for the question and the comment. We agree that the actual decoding speedup is needed to justify the training cost beyond sum-GEMV latencies. We now measure complete FFN kernels, including top-k selection and all element-wise operations, inside full-model decoding on an A100. In a 128-token prompt + 256-token generation run, SGA-S (W2.5) reaches 1062/493/304 tokens/s on LLaMA-3-1B/3B/8B. This is a 2.02x/2.27x/2.90x speedup over dense bf16, and a further 1.06x/1.12x/1.25x over dense W4 (1.12x/1.19x/1.35x with offline-calibrated thresholds, which cost at most 0.7 pp of accuracy; kindly refer to our response (d) to Reviewer sSpP). Please kindly refer to our response (b) to Reviewer NRUr for the details of the kernel design, baselines, and memory-boundedness analysis. As for the invested compute, our training on 30B tokens takes roughly 2500 A100 GPU hours for LLaMA-3-3B, a one-time cost comparable to our QAT baseline, while the decoding speedup applies to every token served after deployment. We will add both to the revised manuscript.

d) The runtime memory saving. (W4)

Answer: Thanks for the question. Yes. SGA-Sparse reduces the HBM traffic per decoded token, which is the bottleneck of single-user decoding, and weight quantization reduces the storage. Since the selected rows/cols change with every token, all weights remain stored. LLaMA-3.1-8B shrinks from ~16 GB (bf16) to ~4.9 GB (W4-g128) through quantization, and the sparsity adds negligible runtime memory. Per token, however, SGA-Sparse reads only ~2.4 GB of weights at 75% sparsity. This makes our method well suited to local deployment, consistent with our observations on an edge NPU (see our response (b) to Reviewer NRUr). It is also promising for local scenarios such as CPU offloading, which we leave to future work.

AuthorsOpenReview ↗

response to Reviewer HeBK, part-2

e) Training budget mismatch between QAT and SGA-Sparse. (W5)

Answer: We thank the reviewer for this comment. We used a smaller training budget for the dense QAT baselines merely to save GPU resources, following the same practice as ParetoQ, which also reduces the training budget once performance saturates. For reference, we additionally train 2-bit QAT on LLaMA-3.2-1B for 60000 iterations (~30B tokens) and the average accuracy is 53.7 (-0.2 pp relative to 53.9 at 20B tokens).

Another important consideration is that our QAT runs reproduce the degradation reported in ParetoQ’s paper. On LLaMA-3.2-3B, the accuracy gap from the dense model for W3 / W2 / W1.58 is −0.7 / −1.8 / −3.7 pp in our runs vs. −0.7 / −2.0 / −3.3 pp in ParetoQ’s paper. We will further clarify the training budget in the QAT settings of the appendix.

f) Quantized model’s performance on knowledge-heavy tasks. (W6)

Answer: Thanks for this suggestion. We have additionally evaluated the quantized models on knowledge- and reasoning-heavy benchmarks. We apply the same compression settings to Qwen-3.5-4B: SGA-S(W2.5), SGA-S(W1.5), and dense ParetoQ at W2 and W1.58. For each model we report the 8-task zero-shot average, 5-shot MMLU and 5-shot GSM8K, as well as the harder MMLU-Pro, BBH, MuSR and GPQA from the Open LLM Leaderboard.

Qwen-3.5-4BWiki2Avg. 8-taskMMLU (5)GSM8K (5)MMLU-ProBBHMuSRGPQAAvg.
SGA-S (75%, BF16)8.865.664.855.332.444.339.929.047.3
SGA-S (W2.5)9.065.465.552.232.244.144.327.847.4
ParetoQ (W2)9.364.561.742.527.741.141.425.743.5
ParetoQ (W1.58)10.161.956.028.021.435.038.028.338.4
SGA-S (W1.5)9.863.458.034.825.939.042.327.941.6

Firstly, the trend in Fig. 3(a) carries over to knowledge-heavy tasks. The SGA-S(W2.5) outperforms dense W2 on every task and SGA-S(W1.5) beats dense W1.58 with the same equivalent weight traffic.

Additionally, consistent with the reviewer's comment, extreme dense weight quantization hurts knowledge- and reasoning-heavy tasks the most. For example, GSM8K drops to 42.5 at W2 and 28.0 at W1.58. Combining moderate weight quantization with activation sparsity amortizes this degradation considerably. This encourages activation sparsity combined with weight quantization as a promising alternative to extreme weight quantization.

We have also added the per-task breakdown of Table 4 to the appendix. Joint-QAT outperforms continued-QAT on 7 of 8 tasks at 3B and on most tasks at 1B and 8B, consistent with the averages reported in the paper.

g) Additional PTQ methods and multi-seed runs. (W7)

Answer: We thank the reviewer for this comment and have extended the PTQ study accordingly. In addition to GPTQ, we now evaluate AWQ on the same checkpoints. In the updated Table 3, each PTQ result is reported as mean ± std over 5 calibration draws, and continued-QAT as mean ± std over 3 training seeds (2 seeds for 8B).

Both PTQ methods reduce the accuracy of the 30BT sparse BF16 checkpoints by up to 1.3 pp. At the same 35B-token budget, continued-QAT achieves higher mean accuracy than both PTQ methods at every model size. On 3B the gain (+0.84 pp over AWQ, +1.45 pp over GPTQ) is significant under a paired bootstrap test (p < 0.002).

Regarding the small margins, with error bars some QAT–PTQ gaps fall within the measured variance. This does not change our main conclusion that a small continued-QAT budget avoids the accuracy loss of PTQ and recovers BF16-level accuracy. We will include these multi-seed results and standard deviations in our revised manuscript to provide a more complete characterization.

W4, 75% sparsity: Wiki2 / Avg. Acc.1B3B8B
BF16 (30BT)12.37 / 54.79.64 / 61.57.90 / 66.0
GPTQ (30BT)13.06±0.05 / 53.6±0.310.84±0.79 / 60.2±0.18.22±0.02 / 65.0±0.5
AWQ (30BT)12.88±0.01 / 54.6±0.29.92±0.02 / 61.0±0.18.12±0.01 / 64.7±0.3
GPTQ (35BT)13.00±0.02 / 54.2±0.210.04±0.07 / 60.0±0.28.21±0.01 / 65.1±0.4
AWQ (35BT)12.88±0.02 / 54.5±0.19.84±0.01 / 60.6±0.18.12±0.01 / 65.0±0.1
Continued-QAT (30BT + 5BT)12.59±0.02 / 54.7±0.29.77±0.03 / 61.4±0.28.01±0.05 / 65.5±0.1
AuthorsOpenReview ↗

followup to Reviewer HeBK, updated PDF

We have uploaded the revised PDF, with all changes highlighted in magenta. Table 5 now includes Qwen-3.5-9B, Mobile-MoE-S/L, and Ouro-1.4B, with the looped-transformer analysis in the appendix D.9. Table 3 adds AWQ with mean±std over calibration draws and training seeds, with per-task results in the appendix (Table 10). The appendix now also includes the quantized models on knowledge-heavy tasks (D.7 and Table 17), the per-task breakdown of Table 4 (D.8 and Table 18), the QAT training-budget clarification, and the end-to-end decoding results (Figure 5). Thank you again for the supportive and detailed review.

Reviewer_NRUrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Activation sparsity can cut FFN weight traffic in single-user decoding, but the active channels have to be identified before the weights are fetched. In a SwiGLU FFN the gate projection is computed densely anyway, so its output is available early and can act as a router for the up and down projections. Prior training-free work concludes the gate is a bad router, because a channel with a small gate value can still matter when its up activation is large. This paper continues pretraining the whole model with a fixed top-k mask applied to the gate-selected channels, and reports that after training the gate becomes as good as or better than the up signal.

Critical strengths
  1. The paper fills a real gap in a small design space. Prior work splits into training-free gate routing (CATS), training-free input sparsity (TEAL), training with the activation swapped to ReLU (ReLUfication, ProSparse), and training with input sparsity (Q-Sparse).
  2. The motivation seems reasonable.
Critical weaknesses
  1. The calculation in Section 5.3 explicitly excludes quantization metadata, routing indices, and top-k overhead. Those are precisely the costs that decide whether a sparse model beats a dense low-bit one on real hardware.
  2. The deployment claim is never measured end to end. The whole motivation is reducing FFN weight traffic in memory-bound single-user decoding, but Figure 3(b) benchmarks the gate, up, and down GEMVs separately on an A100, reports their summed latency, and excludes top-k selection and elementwise operations. An A100 is also not the memory-bound setting the paper is targeting.
Questions for authors
  1. What is the measured end-to-end decode speedup including top-k selection and elementwise ops, on hardware where single-user decoding is actually memory bound? Figure 3(b) excludes both.
  2. The Eq. 7 bit-width excludes routing indices and top-k overhead. How large are those at 75% sparsity with n = 8192? Does the 2.5-bit survive once they are counted??
  3. The benchmarks in the Table 1 are mostly out-of-date, Would it be possible to try some more latest benchmarks for agentic/coding/tool using like deep_swe, terminal bench ?
3 comments
AuthorsOpenReview ↗

response to Reviewer NRUr, part-1

We thank the reviewer for the valuable comments and questions, especially those regarding the system-level clarification. For the discussion, we provide concise summaries of each response below. The complete tables and writeups will be included in the revised PDF, which we will upload shortly.

a) The cost of metadata and other overheads on hardware (W1, Q2).

Answer: Thanks for the question. We address each item below.

  • Quantization metadata: For each group of 128 weights, we use one bf16 scale and no zero point (symmetric). The extra equivalent bit exposed to the HBM traffic is 16/128 = 0.125 bit per weight. Replacing bqb_q and bsb_s in Eq. 7 with 4.125 and 2.125 gives 2.58 and 1.58 bits (2.49 and 1.49 before). We did not include this in the original calculation because 2.5- and 1.5-bit serve as a notation to anchor the comparison with dense QAT models, which share the same metadata overhead (they become 3.125, 2.125 and 1.705 bits).

  • Routing indices: Each gathered row/col of the up/down projections needs a position index. Since we fetch weights at the granularity of an entire row/col, and the up and down weights are stored row/col-major in HBM, one routing index is shared by the whole burst. At n = 8192 (LLaMA-3-1B/3B), this is one INT32 per 2048/3072 weights, i.e., 0.016/0.010 bpw (0.008 bpw for LLaMA-3-8B). The indices are generated on-chip by the gate projection at runtime. They add no HBM weight traffic and stay resident in L2 (tens of KB per layer).

  • Top-k cost: Top-k selection does not change the HBM weight traffic, but it does cost latency. We implemented the complete FFN kernel on an A100 and measured it end to end inside full-model decoding. Exact top-k adds 3–4 µs per layer at n = 8192 (up to 7 µs for LLaMA-3-8B), which is 5-7% of a decode step. We will clarify all items in the revised version. The end-to-end results in the next response include all of these overheads.

b) The end-to-end performance on memory-bound hardware (W2, Q1).

Answer: Thanks for the question and suggestions. We agree that the summed-GEMV latency in Fig. 3(b) is not sufficient, and we now provide end-to-end measurements.

We implemented the full SGA-Sparse FFN as three fused kernels captured in one CUDA graph:

  • (1) a dense gate GEMV with SiLU and index selection fused into its epilogue.
  • (2) a gathered up projection with the element-wise product by the gate fused in.
  • (3) a gathered down projection.

All overheads from the previous response are included. We plugged this FFN into full LLaMA decoding and compared against strong dense baselines: torch.compile (max-autotune) with CUDA graphs for bf16, and per-shape tuned Triton GEMVs for W4.

On the memory-boundedness of the A100: single-user (batch-1) decoding is GEMV-dominated, with an arithmetic intensity of ~1 FLOP/byte, far below the A100's ridge point (~150 FLOP/byte). Our measurements confirm this: the dense bf16 decoding step of LLaMA-3.1-8B runs at 77% of the peak HBM bandwidth. While we can only report detailed numbers on the A100, we have also implemented our method on a memory-bound edge NPU, where we observe consistent speedups and roughly 1.5x energy efficiency.

For an end-to-end generation run (128-token prompt, 256 generated tokens) at 75% FFN sparsity, SGA-S (W2.5) achieves 1062, 493, and 304 tokens/s on LLaMA-3-1B, -3B, and -8B. This is a 2.02x, 2.27x, and 2.90x speedup over dense bf16, and a further 1.06x, 1.12x, and 1.25x over dense W4.

We also implemented a second mode that precomputes the thresholds offline, as widely adopted by prior activation sparsity work (training still uses top-k). This mode has no top-k overhead at runtime, and the speedup over dense W4 increases to 1.12x, 1.19x, and 1.35x. Thresholding costs at most 0.7 pp of average accuracy relative to top-k (kindly refer to our response (d) to Reviewer sSpP). We will complement Fig. 3(b) with these measurements and explanations in the appendix of the revised manuscript.

AuthorsOpenReview ↗

response to Reviewer NRUr, part-2

c) Latest benchmarks (Q3).

Answer: Thanks for the suggestion. We additionally evaluate SGA-Sparse on recent and more challenging benchmarks: the Open LLM Leaderboard v2 tasks (MMLU-Pro, BBH, MuSR, GPQA), instruction following (IFEval) and coding (MBPP), together with 5-shot MMLU and GSM8K. We extend Table 6 by comparing against the training-based baseline Q-Sparse on Qwen-3.5-4B and Qwen-3.5-9B at 75% and 85% sparsity, under the same 10B-token training budget.

SGA-Sparse achieves a higher overall average in all four settings: +2.7 / +4.8 pp on 4B and +1.0 / +2.6 pp on 9B at 75% / 85% sparsity. The gains are largest on the reasoning- and knowledge-heavy tasks and grow with sparsity. We will add these results to the revised manuscript as an extension of Table 6.

Agentic and tool-use benchmarks such as Terminal-Bench and SWE-bench require instruction and tool-use post-training, so base-model results would not be meaningful. Evaluating our method there requires sparsity-aware post-training, which we list as future work.

ModelMethodMMLU (5)GSM8K (5)MMLU-ProBBHMuSRGPQAIFEvalMBPPAvg. 8-taskAvg.
Qwen-3.5-4BQ-Sparse (75%)61.847.929.742.140.127.518.73.064.137.2
SGA-S (75%)64.855.332.444.339.929.021.36.065.639.9
Q-Sparse (85%)57.237.526.339.337.427.818.52.462.634.3
SGA-S (85%)63.150.731.643.144.327.021.36.464.739.1
Qwen-3.5-9BQ-Sparse (75%)70.066.337.451.948.431.521.815.466.745.5
SGA-S (75%)72.269.441.254.343.432.622.715.267.946.5
Q-Sparse (85%)66.456.933.647.842.731.423.313.465.742.4
SGA-S (85%)71.067.440.252.140.930.820.914.267.445.0
AuthorsOpenReview ↗

followup to Reviewer NRUr, updated PDF

We have uploaded the revised PDF, with all changes highlighted in magenta. The new Appendix C describes our complete fused FFN kernel, quantifies the metadata, routing-index, and top-k overheads, and reports end-to-end decoding and FFN-level speedups (Figure 5). We will also release our GPU kernel code upon acceptance. The new benchmarks are added as an extension of Table 6. Thank you again for the constructive feedback.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Final Remarks from the Authors

We are grateful for the constructive feedback and suggestions from all three reviewers and the opportunity to provide these final comments.

Takeaways from rebuttal discussions:

  • Showed that the routing-signal reversal generalizes beyond the GQA-based LLaMA-3 models. We trained four additional models and showed that our method holds across GDN-based (Qwen-3.5-4B and 9B), MoE (Mobile-MoE-S and -L) and looped (Ouro-1.4B) SwiGLU architectures.

  • Implemented the complete fused GPU kernel and reported end-to-end decoding measurements on the A100, achieving up to 2.9× speedup over BF16 and further gains over dense W4 models, with all overheads raised by the reviewers included.

  • Extended the evaluation to recent and more challenging benchmarks, covering reasoning, knowledge, instruction following and coding (Open LLM Leaderboard v2, IFEval, MBPP).

  • Strengthened the quantization study with knowledge-heavy evaluations, an additional PTQ method (AWQ), and results across multiple seeds and calibration draws.

  • Clarified that the training budget of the QAT baselines follows the convention in ParetoQ, which cuts the budget once performance saturates. We additionally trained LLaMA-3.2-1B on 30B tokens to confirm the saturation, and our dense QAT models reproduce the trends reported in ParetoQ.

  • Clarified that the equivalent bit width describes per-token weight traffic, and quantified the metadata, routing-index and top-k overheads, all of which are included in the A100 kernel measurements.

  • Added comparisons with further training-free baselines suggested by the reviewers, including threshold-based CATS and RECAP.

We have incorporated these results and clarifications into the revised manuscript and will further refine them in the final version.

We believe this work offers a new perspective and a practical solution for activation sparsity training. We show that it is not necessary to replace the SiLU activation with ReLU or to route with other signals: once trained, the native gate becomes an effective router. Building on this, SGA-Sparse provides a simple yet effective training method that outperforms both training-free and training-based baselines at up to 85% sparsity under matched token budgets. Combined with low-bit QAT, it introduces new Pareto-efficient operating points and delivers measured end-to-end decoding speedups on the A100.

We will open-source our training and GPU kernel code upon acceptance, and we hope it encourages further exploration of both algorithms and system-level support for activation-sparse LLM deployment, where we believe significant algorithm–hardware co-design opportunities remain. We look forward to contributing to the efficient LLM community with our work.

Thanks very much again to all the reviewers and ACs for their efforts and support during the reviewing process.

I have read the author responses, other reviews, and looked at the revised paper. I greatly appreciate the additional effort in conducting new experiments. I was already leaning positive about the paper and I would support an acceptance decision. The high number of new experiments, however, could also warrant a view of going through another review cycle. Balancing those two sides, I'd like to keep my score at 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.