- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Studies whether the existing gate projection in a pretrained SwiGLU FFN can be trained to select a small subset of intermediate channels. SGA-Sparse computes the gate densely, selects its top-k activations, and evaluates only the corresponding channels in the up and down projections. All model parameters are updated through continued pretraining, without adding a separate router or replacing SiLU. The main finding is that gate-based routing, although worse than up-based routing in the frozen model, becomes competitive or better after training. The paper also studies how to combine this sparsity with weight quantization through continued and joint QAT.
Critical strengths
- The comparison between gate- and up-based routing is the most interesting part of the paper.
- The recovered accuracy at high sparsity is promising. After approximately 30B training tokens, the 3B model retains 61.2% average accuracy at 85% FFN-channel sparsity, compared with 62.6% for the original dense checkpoint.
- The method is straightforward and preserves the pretrained FFN structure.
Critical weaknesses
- The paper does not yet demonstrate the end-to-end inference benefit that motivates the method. Figure 3(b) measures the gate, up, and down GEMVs separately and sums their latencies, explicitly excluding top-k selection and elementwise operations. However, selecting channels before fetching their weights is the central systems challenge here. The complete execution path includes a dependency from the dense gate computation through selection to the sparse projections, along with indexing and low-bit weight access. I would like to see complete FFN latency and model-level decoding throughput before treating the reported projection speedups as evidence of practical acceleration.
- The equivalent-bit comparison is useful, but does not establish equivalent deployment cost. The 1.5-bit and 2.5-bit labels describe idealized per-token weight traffic, not model storage. All weights still need to be stored, and the calculation excludes quantization metadata, routing information, and memory-access inefficiencies.
- The dense low-bit comparison uses unequal training budgets.
- The closest gate-based baseline deserves a stronger comparison. CATS is evaluated as a frozen, controlled top-k baseline, while SGA-Sparse receives substantial continued pretraining. It may also be worthwhile to compare and contrast against RECAP [1]
- Most results concern eight zero-shot tasks and WikiText-2 perplexity. The MMLU and GSM8K ablations are useful, but there is little evidence on sustained generation, coding, long-context use, or instruction following.
[1] Lee, Mingyu, Akshat Ramachandran, and Tushar Krishna. "Recap: Training-free compensation for coarse activation channel pruning in compressed llms." Machine Learning for Computer Architecture and Systems 2 (2025).
Questions for authors
Please see above.
3 comments
response to Reviewer sSpP, part-2
d) Additional comparison with CATS and RECAP. (Q4)
Answer: Thanks for the valuable suggestion. We agree that CATS deserves a comparison beyond the controlled top-k version. We therefore additionally implement the original threshold-based CATS: per-layer thresholds are set from a percentile of the dense |SiLU(gate)| activations in one calibration pass over C4 on the pretrained model. For a fair comparison, we also apply thresholding to SGA-Sparse, calibrating thresholds on the sparsity-trained checkpoint to the same target sparsity.
Native threshold-based CATS performs no better than its top-k counterpart (and is worse at 85% sparsity), so the controlled top-k baseline in Table 1 does not understate CATS. SGA-Sparse remains robust under thresholding, staying within 0.7 pp of its top-k accuracy, and outperforms CATS by 15–24 pp under both selection schemes. Note that applying our gate-based selector without training is exactly CATS; the difference lies in sparsity-aware training, under which the native gate signal becomes the better router (Fig. 2).
| Avg. Acc. | 1B 75% | 1B 85% | 3B 75% | 3B 85% | 8B 75% | 8B 85% |
|---|---|---|---|---|---|---|
| CATS (top-k) | 39.7 | 37.5 | 43.0 | 37.5 | 51.0 | 42.0 |
| CATS (threshold) | 38.8 | 37.0 | 41.1 | 36.2 | 51.0 | 40.6 |
| SGA-S (top-k) | 54.7 | 54.1 | 61.5 | 61.2 | 66.0 | 65.0 |
| SGA-S (threshold) | 54.5 | 53.7 | 61.0 | 60.5 | 66.0 | 64.3 |
Thank you also for pointing us to RECAP. We implement its main method (channel pruning with Ng = 8 compensation) and evaluate it under our training-free protocol, sparsifying the inputs of the up and down projections of the pretrained LLaMA models at 75% and 85%. RECAP achieves the highest average accuracy among the training-free baselines, while remaining well below SGA-Sparse. RECAP selects channels per input sequence and compensates the pruned channels with group-wise average magnitudes. It is designed to reduce GEMM computation, and its compensation term reads the pruned weight rows from HBM. It therefore targets computation rather than per-token weight traffic in decoding, and we view it as complementary to our method. We will add RECAP to the related work and include the results and discussion in the appendix.
| Avg. Acc. | 1B 75% | 1B 85% | 3B 75% | 3B 85% | 8B 75% | 8B 85% |
|---|---|---|---|---|---|---|
| R-Sparse (best in Table 1) | 43.6 | 38.2 | 49.7 | 40.9 | 57.0 | 44.8 |
| RECAP | 46.2 | 44.3 | 53.9 | 51.2 | 59.3 | 57.0 |
| SGA-S | 54.7 | 54.1 | 61.5 | 61.2 | 66.0 | 65.0 |
e) More evaluation datasets (Q5)
Answer: Thanks for the suggestion. We additionally evaluate SGA-Sparse on instruction following (IFEval), coding (MBPP), and harder reasoning and knowledge benchmarks (MMLU-Pro, BBH, MuSR, GPQA), together with 5-shot MMLU and GSM8K, comparing against Q-Sparse on Qwen-3.5-4B and -9B at 75% and 85% sparsity. IFEval, MBPP and GSM8K require free-form multi-step generation rather than likelihood scoring, so they also probe sustained generation. SGA-Sparse achieves a higher overall average in all four settings (+2.7/+4.8 pp on 4B and +1.0/+2.6 pp on 9B at 75%/85% sparsity), with the largest gains on generation-heavy GSM8K (+7.4/+13.2 pp on 4B). Please kindly refer to our response (c) to Reviewer NRUr for the full table. Long-context evaluation requires long-context training data and post-training beyond our continued-pretraining setup, which we list as future work.
followup to Reviewer sSpP, updated PDF
We have uploaded the revised PDF, with all changes highlighted in magenta. The new Appendix C section on the GPU kernel reports the complete FFN latency and end-to-end decoding speedups, and clarifies that the equivalent bit widths describe per-token weight traffic rather than storage; we will release the kernel code upon acceptance. The appendix D.5 and D.6 add the comparisons with threshold-based CATS and with RECAP (RECAP is also discussed in the related work), and Table 6 extends the evaluation to additional benchmarks. Thank you again for the detailed and constructive comments.
response to Reviewer sSpP, part-1
We thank the reviewer for the valuable and detailed comments. For the discussion, we provide concise summaries of each response below. The complete tables and writeups will be included in the revised PDF, which we will upload shortly.
a) FFN-level latency and end-to-end decoding performance. (Q1)
Answer: Thank you for the detailed comment! We agree that selecting channels before fetching their weights is the central systems challenge, and we now measure the complete execution path. Our FFN runs as three kernels in one CUDA graph:
Unlike predictor-based methods, SGA-Sparse uses the model's own gate projection as the router. It adds no auxiliary predictor and no extra weight traffic, and the gate is already on the critical path of dense SwiGLU. The only new ordering constraint is that the up projection starts after selection, which costs one kernel boundary and is included in all our measurements.
On an A100, our complete FFN (including exact top-k, all metadata, and element-wise operations) takes 34.5/40.8/62.0 µs per layer on LLaMA-3-1B/3B/8B, at 75% sparsity on top of W4-g128. Compared to dense W4-g128, our sparse FFN is 1.11x/1.22x/1.42x faster. At the model level, decoding reaches 1062/493/304 tokens/s, which is 2.02x/2.27x/2.90x over dense bf16 and 1.06x/1.12x/1.25x over dense W4. Please kindly refer to our response (b) to Reviewer NRUr for details.
b) Cost of quantization metadata and routing indices. (Q2)
Answer: Thanks for the comment. We agree, and we will clarify in the revision that the 2.5- and 1.5-bit labels describe per-token weight traffic, not storage. For example, for our SGA-S(W2.5), all weights remain stored in 4-bit (~4.9 GB for LLaMA-3.1-8B vs. ~16 GB in bf16), while only ~2.4 GB are read per token. Counting quantization metadata raises the labels to 2.58 and 1.58 bits, and routing indices add only ~0.01 bit more. Please kindly refer to our response (a) to Reviewer NRUr and (d) to Reviewer HeBK for more details. Memory-access inefficiencies are captured by our end-to-end measurements: on the A100, the sparse FFN reaches 60–78% of its ideal byte-proportional speedup over dense W4, and the model-level gains above already include all of these costs. We will make sure to add these clarifications, measurements, and explanations in the appendix of the revised manuscript.
c) Training budget mismatch between QAT and SGA-Sparse. (Q3)
Answer: Thanks for the comment. We used a smaller training budget for the dense QAT baselines to save GPU resources, following the same practice as ParetoQ, which also cuts the training budget once performance saturates. For example, extending 2-bit QAT on LLaMA-3.2-1B from 20B to 30B tokens changes the average accuracy by only −0.2 pp (53.9 to 53.7). Our dense QAT runs also reproduce the degradation reported in the ParetoQ paper. Please kindly refer to our response (e) to Reviewer HeBK for details.