- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Prox, a training-free framework for sparse SwiGLU FFNs, motivated by the observation that channel selection can rely on approximate intermediate-staet magnitudes rather than exact intermediate values. Prox operates in two stages. In the first stage, it uses a sparse-input proxy to estimate channel importance and construct a selection mask. In the second stage, it exactly recomputes the retained intermediate channels using the full input and the original full-precision weights, enabling coordinated sparse execution across the gate, up, and down projections.
Critical strengths
-
Combining a low-cost proxy with exact recomputation provides a practical training-free sparsification approach. Prox uses input sparsification and INT2 weights to construct a proxy for the joint intermediate state, which is used solely for channel selection. It then recomputes the retained channels using the full current-layer input and the original weights. This enables sparse execution across the up, gate, and down projections while avoiding direct propagation of proxy value errors, without requiring an additional trained predictor.
-
The paper evaluates ten models from six model families. At matched target effective FFN sparsity, Prox achieves higher aggregate downstream scores than TEAL on nine models at 50% sparsity and on all ten models at 60% and 70% sparsity. These results indicate that the gains are not confined to a single architecture or a few isolated configurations, supporting the central claim that Prox improves quality preservation under aggresive sparsification.
-
The paper provides custom sparse kernels implemented in TileLang, achieving 1.48 - 2.07 x speedups over dense execution at 60%-70% effective sparsity in the tested batch-size-one decoding configurations on an A6000. It also reports measturements on an RTX 3090 and a stage-wise latency breakdown. The cost model accounts for proxy overhead, with the INT2 cost coefficient set based on measured latency.
Critical weaknesses
-
While I understand the authors's approach of approximating the intermediate states in advance to reduce computation across all three FFN projections, I find Figure 4 difficult to follow, particularly for readers unfamiliar with activation sparsity. For example, what does the dashed line represent? Abstracting away nonessential details and highlighting the core machanism would make the figure more accessible and help readers better understand the paper's novelty.
-
The paper omits two closely related training-free activation-sparsification methods, LaRoSA [1] and WINA [2], from both its related-work discussion and experimental comparisons. These omissions make it difficult to assess Prox’s contribution relative to a broader set of relevant approaches, rather than primarily against the selected baselines.
-
The main concern relates to the "exact sparse computation" described at line 280. At line 180, the authors state that TEAL's two successive sparsification steps allow approximation errors to compound through the FFN. At line 796, they further argue that these errors accumulates as they propagate through the up and gate branches. However, the paper provides neither a theoretical analysis nor targeted end-to-end empirical evidence that directly quantifies this error accumulation in TEAL and demonstrates that Prox's exact sparse computation yields lower accumulated approximation error. Consequently, although the proposed design avoids directly propagating proxy value errors, its claimed advantage in mitigating error accumulation remains insufficiently substantiated.
-
The ablations in Prox are more extensive than the main-text experiments A1-A3 alone suggest: The fixed-threshold design is evaluated using a single calibration corpus, without the cross-corpus controls examined in TEAL/CATS. The systems evaluation does not ioslate sensitivity to output sequence length, as CATS and LaRoSA do. Moreover, the latency breakdown does not substitute for a controlled comparison between baseline the optimized kernels, such as that presented in Appendix D of CATS.
[1] LaRoSA: Liu et al., “La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation.” ICML 2025, PMLR 267:39968–39986.
[2] WINA: Chen et al., “WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference.” ICLR 2026.
Questions for authors
-
Could the authors assess calibration-corpus sensitivity on one representative model at target sparsity levels of 60% and 70%, calibrating separately with equal numbers of tokens from the Alpaca, C4, and WikiText training corpora while using the same independent test data? Please report perplexity, downstream task scores, realized s1 and s2, the corresponding effective sparsity under the cost model, and measured latency.
-
Could the authors evaluate one representative model on a supported GPU at two sparsity levels, keeping the prompt length fixed while varying the output length across 128, 512, and 2,048 tokens? The main experiment should retain dense attention to isolate the effect of FFN sparsification. Please report both decode-only throughput and total request latency, including prefill, together with the realized sparsity during execution.
-
Could the authors design a comprehensive experiment to quantify the error accumulation they attribute to TEAL and characterize its distribution and numerical properties in as much detail as possible?
-
The results of ablation A1 appear somewhat counterintuitive: why does using FP16 for the proxy computation lead to worse performance than using INT2? Could the authors provide a detailed explanation for this observation?
-
Unless I have overlooked it, Figure 5 does not directly compare the speedups of Prox and TEAL at matched sparsity levels; TEAL’s throughput in tokens per second is instead reported in the appendix. Given that TEAL is a central baseline, could the authors provide a fair, direct comparison of end-to-end acceleration under matched hardware, workloads, and effective sparsity budgets? The observation that higher sparsity generally leads to faster execution is, by itself, only moderately informative. Readers would be more interested in how much additional acceleration Prox provides over an existing sparse-inference baseline, rather than over dense execution alone.
I would appreciate the authors’ responses to these questions and will reconsider my final score in light of their clarifications.
5 comments
Continued response to reviewer Ebcc
W4, Q1 & Q2. Missing ablations: calibration-corpus sensitivity, output-length sensitivity, and a controlled kernel comparison.
A4. We added all three in the revision.
Calibration corpus (Q1). We recalibrated Qwen3-8B on C4 and WikiText with the same number of tokens as Alpaca (20,480), using the WikiText training split, which is disjoint from the evaluation split. The realized , , and runtime effective sparsity are measured during perplexity evaluation:
Target Corpus PPL Avg. score 60% Alpaca 8.84 73.5 25.3% 69.7% 59.8% C4 8.85 73.2 25.7% 71.4% 61.5% WikiText 8.82 72.8 25.4% 71.4% 61.4% 70% Alpaca 9.61 67.2 71.2% 74.0% 70.1% C4 9.83 67.7 71.4% 75.9% 72.1% WikiText 9.72 67.1 70.7% 75.6% 71.7% Across the three corpora, the average score varies by at most 0.7 points and the runtime sparsity stays within 2.1 points of the target, so Prox is robust to the choice of calibration corpus. Since the realized sparsities are nearly identical across corpora, the latency is also nearly identical.
Output length (Q2). On an A6000, we feed Qwen3-8B 512-token prompts and generate 128 to 2048 tokens with dense attention, and measure the runtime sparsity over all generated tokens:
Target Metric 128 256 512 1024 2048 60% Runtime (%) 59.2 60.6 61.8 62.5 63.1 Decode throughput (tok/s) 69.6 70.5 71.9 72.0 69.4 Throughput incl. prefill (tok/s) 66.1 68.7 70.9 71.5 69.2 70% Runtime (%) 69.7 70.6 71.9 73.1 74.4 Decode throughput (tok/s) 77.7 78.2 80.2 79.3 76.9 Throughput incl. prefill (tok/s) 73.3 75.9 79.0 78.7 76.6 The runtime sparsity rises slightly with output length, and the decode throughput remains stable, deviating by at most 3.4% from its 128-token value.
Kernel analysis. We added the kernel pseudocode and an ablation in Appendix B.5, following CATS. Each Prox FFN runs four kernels:
- Stage 1 selector (two kernels). The first kernel tests each input against its threshold before loading any proxy weights, so a rejected input costs neither memory traffic nor MACs. Each thread block writes partial sums to a dedicated slot, avoiding atomic accumulation. The second kernel reduces the partials, applies SwiGLU, and writes out only the mask.
- Stage 2 (two kernels). A fused kernel fetches the gate and up columns only for active channels and applies SwiGLU in registers, followed by an input-sparse down-projection GEMV.
In the ablation, our Stage 1 kernel is 1.5–2.0× faster than an atomic-reduction variant and 1.2–1.6× faster than a basic partial-reduction variant. At the FFN-block level, we compare Prox with Optimal, a compact dense FFN restricted to the retained dimensions (A6000, latency in μs):
Model Method 40% 50% 60% 70% Qwen3-8B Dense 453 454 454 454 Optimal 291 233 191 152 Prox 295 232 192 159 Qwen3-14B Dense 788 788 788 789 Optimal 461 384 313 249 Prox 459 384 318 260 Prox stays within 5% of Optimal and runs 1.5–3.0× faster than the dense FFN, so the run-time thresholding and mask handling add little overhead.
Continued response to reviewer Ebcc
Q4. Why does the FP16 proxy in ablation A1 perform worse than the INT2 proxy?
A4. A1 is a budget-matched comparison. The FP16 proxy costs as much as the INT2 proxy, so to fit the same effective sparsity, A1 must run Stage 1 at a higher input sparsity and keep fewer channels in Stage 2. Higher proxy precision does not offset this reduced Stage 2 budget. Since the mask depends only on the ranking of the proxy values, the INT2 proxy is accurate enough for channel selection (see W3 (i)), and the saved budget is better spent on retaining more channels in Stage 2.
Q5. A fair, direct comparison of end-to-end acceleration between Prox and TEAL under matched hardware, workloads, and effective sparsity budgets.
A5. Effective sparsity is the fraction of FFN computation a method removes relative to the dense FFN, including its auxiliary operations, so configurations with the same have the same theoretical FFN speedup of (see Q1 of the General Response). Under matched hardware, workloads, and , the end-to-end decoding speedups over dense inference are (Prox / TEAL):
GPU Model 40% 50% 60% 70% A6000 Qwen3-4B 1.27 / 1.34 1.36 / 1.45 1.48 / 1.51 1.61 / 1.74 Qwen3-8B 1.41 / 1.40 1.53 / 1.53 1.71 / 1.72 1.92 / 1.95 Qwen3-14B 1.48 / 1.43 1.62 / 1.59 1.80 / 1.82 2.04 / 2.08 Llama3-8B 1.45 / 1.42 1.61 / 1.55 1.74 / 1.75 1.96 / 2.02 Mistral-7B 1.49 / 1.46 1.68 / 1.64 1.84 / 1.89 2.07 / 2.14 RTX 3090 Qwen3-4B 1.28 / 1.25 1.37 / 1.32 1.47 / 1.44 1.57 / 1.56 Qwen3-8B 1.35 / 1.34 1.47 / 1.41 1.58 / 1.57 1.74 / 1.75 Llama3-8B 1.39 / 1.31 1.52 / 1.41 1.64 / 1.58 1.84 / 1.81 Mistral-7B 1.42 / 1.34 1.56 / 1.46 1.72 / 1.69 1.91 / 1.99 Jetson Thor Qwen3-4B 1.28 / 1.28 1.41 / 1.37 1.54 / 1.47 1.67 / 1.64 Qwen3-8B 1.34 / 1.34 1.49 / 1.46 1.65 / 1.56 1.85 / 1.67 Qwen3.5-4B 1.24 / 1.22 1.30 / 1.33 1.38 / 1.42 1.49 / 1.47 Mistral-7B 1.42 / 1.44 1.57 / 1.62 1.79 / 1.83 2.03 / 2.06 RTX 5070 Ti Qwen3-4B 1.24 / 1.21 1.34 / 1.27 1.43 / 1.39 1.57 / 1.51 Qwen3.5-4B 1.16 / 1.18 1.23 / 1.26 1.28 / 1.33 1.41 / 1.43 Mistral-7B 1.43 / 1.35 1.58 / 1.54 1.74 / 1.77 2.02 / 2.04 At matched , Prox and TEAL achieve similar speedups, while Prox achieves higher quality. To compare them directly on quality versus speed, we added a speed–quality Pareto front in the revision (Figure 9). On Qwen3-8B with an A6000:
Sparsity TEAL speedup TEAL score Prox speedup Prox score 40% 1.40× 75.1 1.41× 75.3 50% 1.53× 73.7 1.53× 75.1 60% 1.72× 72.2 1.71× 73.5 70% 1.95× 64.4 1.92× 67.2 The Pareto front of Prox lies above TEAL's over their shared speedup range, and at an equal speedup Prox scores up to 1.9 points higher. For example, Prox at 50% matches the score of TEAL at 40% (75.1) while running at 1.53× instead of 1.40×.
Response to reviewer Ebcc
We appreciate the detailed review and the concrete experiment suggestions. Several questions overlap with the weaknesses, so we answer them together: Q3 under W3, and Q1 and Q2 under W4.
W1. Figure 4 is difficult to follow, e.g., what does the dashed line represent?
W2. LaRoSA and WINA are missing from the related-work discussion and experimental comparisons.
W3 & Q3. The paper does not quantify the error accumulation in TEAL or show that exact sparse computation in Prox yields lower error.