acceptodds
Under review as a conference paper at ICLR 2027

Prox: Training-Free FFN Activation Sparsification in LLMs via Approximate Intermediate-State Salience

Abstract

Feed-forward networks (FFNs) dominate memory traffic and computation in large language model (LLM) inference, making them a primary target for activation sparsification. However, existing training-free methods suffer substantial model-quality degradation at high FFN sparsity due to limitations in their channel-selection strategies. We observe that the SwiGLU intermediate state provides a highly effective channel-selection signal, but obtaining it requires costly dense computation. To address this, we present Prox, a training-free framework for sparse SwiGLU FFNs based on the insight that sparse execution needs only the channel mask induced by the intermediate state, which can be constructed from the magnitude ranking of its entries (i.e., channel salience) rather than their precise values. Prox works in two stages: Stage1 approximates the ranking via input sparsity and quantized proxy weights to construct the channel mask; Stage2 computes the selected channels exactly using full-precision weights, enabling sparse execution of all three projections. Extensive evaluations across ten LLMs from six model families show that Prox consistently outperforms training-free baselines at various sparsity levels, achieving up to a end-to-end decoding speedup at 70% FFN sparsity, and is compatible with quantization and sparse attention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Efficient Inference, Activation Sparsity, Feed-Forward Networks, Training-Free Sparsification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EbccOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Prox, a training-free framework for sparse SwiGLU FFNs, motivated by the observation that channel selection can rely on approximate intermediate-staet magnitudes rather than exact intermediate values. Prox operates in two stages. In the first stage, it uses a sparse-input proxy to estimate channel importance and construct a selection mask. In the second stage, it exactly recomputes the retained intermediate channels using the full input and the original full-precision weights, enabling coordinated sparse execution across the gate, up, and down projections.

Critical strengths
  • Combining a low-cost proxy with exact recomputation provides a practical training-free sparsification approach. Prox uses input sparsification and INT2 weights to construct a proxy for the joint intermediate state, which is used solely for channel selection. It then recomputes the retained channels using the full current-layer input and the original weights. This enables sparse execution across the up, gate, and down projections while avoiding direct propagation of proxy value errors, without requiring an additional trained predictor.

  • The paper evaluates ten models from six model families. At matched target effective FFN sparsity, Prox achieves higher aggregate downstream scores than TEAL on nine models at 50% sparsity and on all ten models at 60% and 70% sparsity. These results indicate that the gains are not confined to a single architecture or a few isolated configurations, supporting the central claim that Prox improves quality preservation under aggresive sparsification.

  • The paper provides custom sparse kernels implemented in TileLang, achieving 1.48 - 2.07 x speedups over dense execution at 60%-70% effective sparsity in the tested batch-size-one decoding configurations on an A6000. It also reports measturements on an RTX 3090 and a stage-wise latency breakdown. The cost model accounts for proxy overhead, with the INT2 cost coefficient set based on measured latency.

Critical weaknesses
  • While I understand the authors's approach of approximating the intermediate states in advance to reduce computation across all three FFN projections, I find Figure 4 difficult to follow, particularly for readers unfamiliar with activation sparsity. For example, what does the dashed line represent? Abstracting away nonessential details and highlighting the core machanism would make the figure more accessible and help readers better understand the paper's novelty.

  • The paper omits two closely related training-free activation-sparsification methods, LaRoSA [1] and WINA [2], from both its related-work discussion and experimental comparisons. These omissions make it difficult to assess Prox’s contribution relative to a broader set of relevant approaches, rather than primarily against the selected baselines.

  • The main concern relates to the "exact sparse computation" described at line 280. At line 180, the authors state that TEAL's two successive sparsification steps allow approximation errors to compound through the FFN. At line 796, they further argue that these errors accumulates as they propagate through the up and gate branches. However, the paper provides neither a theoretical analysis nor targeted end-to-end empirical evidence that directly quantifies this error accumulation in TEAL and demonstrates that Prox's exact sparse computation yields lower accumulated approximation error. Consequently, although the proposed design avoids directly propagating proxy value errors, its claimed advantage in mitigating error accumulation remains insufficiently substantiated.

  • The ablations in Prox are more extensive than the main-text experiments A1-A3 alone suggest: The fixed-threshold design is evaluated using a single calibration corpus, without the cross-corpus controls examined in TEAL/CATS. The systems evaluation does not ioslate sensitivity to output sequence length, as CATS and LaRoSA do. Moreover, the latency breakdown does not substitute for a controlled comparison between baseline the optimized kernels, such as that presented in Appendix D of CATS.

[1] LaRoSA: Liu et al., “La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation.” ICML 2025, PMLR 267:39968–39986.

[2] WINA: Chen et al., “WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference.” ICLR 2026.

Questions for authors
  • Could the authors assess calibration-corpus sensitivity on one representative model at target sparsity levels of 60% and 70%, calibrating separately with equal numbers of tokens from the Alpaca, C4, and WikiText training corpora while using the same independent test data? Please report perplexity, downstream task scores, realized s1 and s2, the corresponding effective sparsity under the cost model, and measured latency.

  • Could the authors evaluate one representative model on a supported GPU at two sparsity levels, keeping the prompt length fixed while varying the output length across 128, 512, and 2,048 tokens? The main experiment should retain dense attention to isolate the effect of FFN sparsification. Please report both decode-only throughput and total request latency, including prefill, together with the realized sparsity during execution.

  • Could the authors design a comprehensive experiment to quantify the error accumulation they attribute to TEAL and characterize its distribution and numerical properties in as much detail as possible?

  • The results of ablation A1 appear somewhat counterintuitive: why does using FP16 for the proxy computation lead to worse performance than using INT2? Could the authors provide a detailed explanation for this observation?

  • Unless I have overlooked it, Figure 5 does not directly compare the speedups of Prox and TEAL at matched sparsity levels; TEAL’s throughput in tokens per second is instead reported in the appendix. Given that TEAL is a central baseline, could the authors provide a fair, direct comparison of end-to-end acceleration under matched hardware, workloads, and effective sparsity budgets? The observation that higher sparsity generally leads to faster execution is, by itself, only moderately informative. Readers would be more interested in how much additional acceleration Prox provides over an existing sparse-inference baseline, rather than over dense execution alone.

I would appreciate the authors’ responses to these questions and will reconsider my final score in light of their clarifications.

5 comments
AuthorsOpenReview ↗

Response to reviewer Ebcc

We appreciate the detailed review and the concrete experiment suggestions. Several questions overlap with the weaknesses, so we answer them together: Q3 under W3, and Q1 and Q2 under W4.

W1. Figure 4 is difficult to follow, e.g., what does the dashed line represent?

A1. Thank you for the suggestion. We have redrawn Figure 4 in the revision. The new figure uses colors to distinguish the two stages: orange for Stage 1, which builds the channel mask from the INT2 proxy weights, and blue for Stage 2, which computes only the selected channels with the original weights. We also added a legend that explains every line style and removed nonessential details to highlight the core mechanism.

W2. LaRoSA and WINA are missing from the related-work discussion and experimental comparisons.

A2. We have added WINA and LaRoSA to the related-work discussion and compared Prox with both on Qwen3-8B using their official open-source implementations (see Q2 of our General Response for the full results). All methods sparsify only the FFNs, and following the same accounting as for the proxy of Prox, the online input rotation of WINA and LaRoSA is counted toward their effective sparsity. This rotation cannot be absorbed into the weights offline: consecutive layers use different rotations, so the residual stream must be re-rotated online at every layer, through a per-layer H×HH\times H residual adapter in LaRoSA and two H×HH\times H shortcut rotations per layer in the official WINA code. The average downstream scores on Qwen3-8B are:

Method40%50%60%70%
WINA72.970.261.738.5
LaRoSA73.069.761.941.1
TEAL75.173.772.264.4
Prox75.375.173.567.2

W3 & Q3. The paper does not quantify the error accumulation in TEAL or show that exact sparse computation in Prox yields lower error.

A3. We added an approximation error analysis on Qwen3-8B in Appendix D.5, from three perspectives.

(i) By error type. Let m\mathbf{m} be the mask applied to the down-projection input and s^\hat{\mathbf{s}} the intermediate state before masking. The FFN output error splits exactly into

y−y^=((1−m)⊙s)Wdown⏟selection+(m⊙(s−s^))Wdown⏟value.\mathbf{y}-\hat{\mathbf{y}}=\underbrace{\big((\mathbf{1}-\mathbf{m})\odot\mathbf{s}\big)\mathbf{W}_{\mathrm{down}}}_{\text{selection}}+\underbrace{\big(\mathbf{m}\odot(\mathbf{s}-\hat{\mathbf{s}})\big)\mathbf{W}_{\mathrm{down}}}_{\text{value}}.

SparsityMethodSelectionValueTotal
60%TEAL0.1440.1490.213
Prox0.15600.156
70%TEAL0.2080.2140.309
Prox0.21500.215

In TEAL, the value error is as large as the selection error. Prox recomputes the retained channels exactly and has no value error, while its INT2 proxy raises the selection error by only 8% and 3% over TEAL's FP16 selection. As a result, Prox lowers the total error by 26.9% and 30.4%.

(ii) By sparsification step. TEAL sparsifies its computation at two points: the input x\mathbf{x} to the up and gate projections, and the intermediate state s\mathbf{s} to the down projection. The relative FFN output error of TEAL when only x\mathbf{x}, only s\mathbf{s}, or both are sparsified is:

Sparsityx\mathbf{x} onlys\mathbf{s} onlyBoth
40%0.0740.0560.093
50%0.1140.0830.143
60%0.1670.1240.213
70%0.2440.1750.309

Applying both steps gives a larger error than either step alone, at every sparsity level and in all 36 layers. Prox avoids the value error from sparsifying x\mathbf{x} because it uses the proxy only to select channels.

(iii) At the model level. We sparsify all FFN layers and measure the per-token KL divergence from the dense next-token distribution over 65,536 WikiText tokens:

Method60%70%
TEAL0.2330.789
Prox0.1300.251

Prox reduces the KL divergence by 44.3% at 60% and 68.1% at 70%, and the gap widens with sparsity.

AuthorsOpenReview ↗

Continued response to reviewer Ebcc

W4, Q1 & Q2. Missing ablations: calibration-corpus sensitivity, output-length sensitivity, and a controlled kernel comparison.

A4. We added all three in the revision.

Calibration corpus (Q1). We recalibrated Qwen3-8B on C4 and WikiText with the same number of tokens as Alpaca (20,480), using the WikiText training split, which is disjoint from the evaluation split. The realized s1s_1, s2s_2, and runtime effective sparsity e^\hat{e} are measured during perplexity evaluation:

TargetCorpusPPLAvg. scores1s_1s2s_2e^\hat{e}
60%Alpaca8.8473.525.3%69.7%59.8%
C48.8573.225.7%71.4%61.5%
WikiText8.8272.825.4%71.4%61.4%
70%Alpaca9.6167.271.2%74.0%70.1%
C49.8367.771.4%75.9%72.1%
WikiText9.7267.170.7%75.6%71.7%

Across the three corpora, the average score varies by at most 0.7 points and the runtime sparsity stays within 2.1 points of the target, so Prox is robust to the choice of calibration corpus. Since the realized sparsities are nearly identical across corpora, the latency is also nearly identical.

Output length (Q2). On an A6000, we feed Qwen3-8B 512-token prompts and generate 128 to 2048 tokens with dense attention, and measure the runtime sparsity over all generated tokens:

TargetMetric12825651210242048
60%Runtime e^\hat{e} (%)59.260.661.862.563.1
Decode throughput (tok/s)69.670.571.972.069.4
Throughput incl. prefill (tok/s)66.168.770.971.569.2
70%Runtime e^\hat{e} (%)69.770.671.973.174.4
Decode throughput (tok/s)77.778.280.279.376.9
Throughput incl. prefill (tok/s)73.375.979.078.776.6

The runtime sparsity rises slightly with output length, and the decode throughput remains stable, deviating by at most 3.4% from its 128-token value.

Kernel analysis. We added the kernel pseudocode and an ablation in Appendix B.5, following CATS. Each Prox FFN runs four kernels:

  • Stage 1 selector (two kernels). The first kernel tests each input against its threshold before loading any proxy weights, so a rejected input costs neither memory traffic nor MACs. Each thread block writes partial sums to a dedicated slot, avoiding atomic accumulation. The second kernel reduces the partials, applies SwiGLU, and writes out only the mask.
  • Stage 2 (two kernels). A fused kernel fetches the gate and up columns only for active channels and applies SwiGLU in registers, followed by an input-sparse down-projection GEMV.

In the ablation, our Stage 1 kernel is 1.5–2.0× faster than an atomic-reduction variant and 1.2–1.6× faster than a basic partial-reduction variant. At the FFN-block level, we compare Prox with Optimal, a compact dense FFN restricted to the retained dimensions (A6000, latency in μs):

ModelMethod40%50%60%70%
Qwen3-8BDense453454454454
Optimal291233191152
Prox295232192159
Qwen3-14BDense788788788789
Optimal461384313249
Prox459384318260

Prox stays within 5% of Optimal and runs 1.5–3.0× faster than the dense FFN, so the run-time thresholding and mask handling add little overhead.

AuthorsOpenReview ↗

Continued response to reviewer Ebcc

Q4. Why does the FP16 proxy in ablation A1 perform worse than the INT2 proxy?

A4. A1 is a budget-matched comparison. The FP16 proxy costs 5×5\times as much as the INT2 proxy, so to fit the same effective sparsity, A1 must run Stage 1 at a higher input sparsity and keep fewer channels in Stage 2. Higher proxy precision does not offset this reduced Stage 2 budget. Since the mask depends only on the ranking of the proxy values, the INT2 proxy is accurate enough for channel selection (see W3 (i)), and the saved budget is better spent on retaining more channels in Stage 2.

Q5. A fair, direct comparison of end-to-end acceleration between Prox and TEAL under matched hardware, workloads, and effective sparsity budgets.

A5. Effective sparsity ee is the fraction of FFN computation a method removes relative to the dense FFN, including its auxiliary operations, so configurations with the same ee have the same theoretical FFN speedup of 1/(1−e)1/(1-e) (see Q1 of the General Response). Under matched hardware, workloads, and ee, the end-to-end decoding speedups over dense inference are (Prox / TEAL):

GPUModel40%50%60%70%
A6000Qwen3-4B1.27 / 1.341.36 / 1.451.48 / 1.511.61 / 1.74
Qwen3-8B1.41 / 1.401.53 / 1.531.71 / 1.721.92 / 1.95
Qwen3-14B1.48 / 1.431.62 / 1.591.80 / 1.822.04 / 2.08
Llama3-8B1.45 / 1.421.61 / 1.551.74 / 1.751.96 / 2.02
Mistral-7B1.49 / 1.461.68 / 1.641.84 / 1.892.07 / 2.14
RTX 3090Qwen3-4B1.28 / 1.251.37 / 1.321.47 / 1.441.57 / 1.56
Qwen3-8B1.35 / 1.341.47 / 1.411.58 / 1.571.74 / 1.75
Llama3-8B1.39 / 1.311.52 / 1.411.64 / 1.581.84 / 1.81
Mistral-7B1.42 / 1.341.56 / 1.461.72 / 1.691.91 / 1.99
Jetson ThorQwen3-4B1.28 / 1.281.41 / 1.371.54 / 1.471.67 / 1.64
Qwen3-8B1.34 / 1.341.49 / 1.461.65 / 1.561.85 / 1.67
Qwen3.5-4B1.24 / 1.221.30 / 1.331.38 / 1.421.49 / 1.47
Mistral-7B1.42 / 1.441.57 / 1.621.79 / 1.832.03 / 2.06
RTX 5070 TiQwen3-4B1.24 / 1.211.34 / 1.271.43 / 1.391.57 / 1.51
Qwen3.5-4B1.16 / 1.181.23 / 1.261.28 / 1.331.41 / 1.43
Mistral-7B1.43 / 1.351.58 / 1.541.74 / 1.772.02 / 2.04

At matched ee, Prox and TEAL achieve similar speedups, while Prox achieves higher quality. To compare them directly on quality versus speed, we added a speed–quality Pareto front in the revision (Figure 9). On Qwen3-8B with an A6000:

SparsityTEAL speedupTEAL scoreProx speedupProx score
40%1.40×75.11.41×75.3
50%1.53×73.71.53×75.1
60%1.72×72.21.71×73.5
70%1.95×64.41.92×67.2

The Pareto front of Prox lies above TEAL's over their shared speedup range, and at an equal speedup Prox scores up to 1.9 points higher. For example, Prox at 50% matches the score of TEAL at 40% (75.1) while running at 1.53× instead of 1.40×.

Reviewer_TJaNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

A training-free method for accelerating gated FFNs through input-dependent activation sparsity. Its central idea is to separate channel selection from value computation.

Critical strengths
  • The proxy only needs to identify important channels, allowing aggressive approximation without directly passing its numerical errors downstream.
  • Prox outperforms TEAL on all models at the 60% and 70% operating points, and the implemented kernels demonstrate practical decoding gains on two GPUs.
Critical weaknesses
  • R-Sparse is discussed in the related work but is absent from the experiments. RECAP [1] is also relevant because it compensates for coarse activation-channel pruning using statistics of the omitted channels.
  • Prox’s effective sparsity discounts proxy operations using a latency-derived factor, α = 0.2, whereas the baseline budgets are based on projection sparsities.
Questions for authors
  • Could you compare Prox with RECAP [1] and R-Sparse under a common FFN-only evaluation protocol, accounting for auxiliary computation and storage?
  • How accurately does the fixed α = 0.2 cost model predict measured latency across different stage allocations and sparsity levels?
  • What explains the large within-model perplexity increase for Gemma-3-12B alongside relatively stable downstream averages?
  • What prompt lengths, generation lengths, dense backend, and compilation settings underlie Figure 5 and Table 7? Are dense and sparse implementations comparably optimized, and are all proxy, thresholding, and mask-handling costs included in the reported throughput?
3 comments
AuthorsOpenReview ↗

Response to reviewer TJaN

We appreciate the constructive review and comments. Below, we provide our detailed answers in line with your questions and suggestions.

W1 & Q1. Comparison with R-Sparse and RECAP, accounting for auxiliary computation and storage.

A1. We compared Prox with R-Sparse and RECAP, as well as WINA (ICLR 2026) and LaRoSA (ICML 2025), on Qwen3-8B under a common FFN-only protocol, where, following the same accounting as for the proxy of Prox, the online auxiliary computation of each method is counted toward its effective sparsity. Please see Q2 of our General Response for the cost accounting and full results: Prox achieves the highest average score at all sparsity levels.

In terms of storage, each method keeps the original FP16 weights and stores the following auxiliary data on Qwen3-8B:

MethodAuxiliary storage% of FFN weights% of model weights
TEALPer-layer thresholds≈0≈0
RECAPPer-channel magnitude statistics<0.1%<0.1%
R-SparseRank-rr factors Ar,Br\mathbf{A}_r,\mathbf{B}_r of each projection2.2–7.2%1.5–4.8%
WINA / LaRoSAOne H×HH\times H rotation per layer11.1%7.4%
ProxINT2 copy of the up and gate weights~9%~6%
  • For R-Sparse and the rotation-based methods, the auxiliary weights are read once per decoding step, so their storage equals their online auxiliary cost in the budget table of the General Response.
  • RECAP needs almost no extra storage, but its accuracy is the lowest among all methods (53.5 at 70%).
  • Prox's storage overhead is comparable to those of R-Sparse and the rotation-based methods, and it achieves the best quality at every sparsity level. As discussed in Sec. 5.3, it could be further reduced by loading proxy weights on demand.

W2 & Q2. Accuracy of the α=0.2\alpha=0.2 cost model and its comparability with baseline budgets.

A2. We measured the INT2-to-FP16 proxy-cost ratio on A6000 and RTX 3090 across four Qwen3 scales and five input sparsities:

GPUModel0%10%25%55%70%
A6000Qwen3-4B0.1660.1660.1660.1660.161
Qwen3-8B0.1560.1600.1650.1800.193
Qwen3-14B0.1530.1570.1610.1830.206
Qwen3-32B0.1530.1570.1650.1870.214
RTX 3090Qwen3-4B0.1840.1890.1930.2050.200
Qwen3-8B0.1610.1660.1710.1850.202
Qwen3-14B0.1460.1460.1490.1660.192
Qwen3-32B0.1660.1700.1770.2020.223
  • At input sparsities up to 25%, which cover our allocations for e≤60e\le 60\\%, the measured ratios range from 0.146 to 0.193, all below 0.2.
  • The ratios exceed 0.2 only at 55% input sparsity or higher, where the proxy processes few inputs and accounts for a small share of the FFN cost. Even with the largest measured ratio (0.223), the effective sparsity changes by less than 0.5 percentage points.

To test the full cost model, we fix each target ee, vary the Stage 1/Stage 2 allocation over five settings, and measure the FFN latency of Prox relative to the dense FFN on an A6000:

Target eePredicted latency (× dense)Measured latency (× dense), min–max over 5 allocations
40%0.6000.581–0.611
50%0.5000.488–0.513
60%0.4000.393–0.413
70%0.3000.302–0.316

Across all allocations and sparsity levels, the measured latency stays within 2 percentage points of the prediction, so the cost model reliably predicts latency. Moreover, TEAL, whose budget is based purely on projection sparsity, achieves end-to-end speedups similar to those of Prox at the same ee (Table 7), so the two budgets translate into comparable latency.

AuthorsOpenReview ↗

Continued response to reviewer TJaN

Q3. What explains the large within-model perplexity increase for Gemma-3-12B alongside relatively stable downstream averages?

A3. We attribute this gap to the different quantities the two metrics measure. Perplexity is the exponential of the mean negative log-likelihood over every token of a long sequence, so it grows whenever the probability of the correct token drops. Most downstream tasks, in contrast, only compare a few candidate answers, so their accuracy is unchanged as long as the correct answer remains top-ranked.

Prior work reports consistent observations. Szatkowski et al. (ICLR 2026) find that Gemma-3-12B retains about 99% of its dense downstream accuracy at 78.77% activation sparsity, and Dutta et al. (NeurIPS 2024) show that compressed models can keep their benchmark accuracy even when their predictive distributions change substantially.

Our experiments also show that this phenomenon is not specific to Prox. TEAL shows the same pattern on Gemma-3-12B, with an even larger perplexity increase: at 40% sparsity, its perplexity reaches 806.98 while its average score remains 72.9.

MethodPPL 40%50%60%70%Avg. 40%50%60%70%
Dense7.4477.3
TEAL806.98898.751.20e32.94e372.969.067.564.5
Prox24.6096.541.10e31.28e377.877.370.570.7

To rule out numerical issues, we also re-ran the evaluation in FP32, and the perplexity remains large.

Q4. What prompt lengths, generation lengths, dense backend, and compilation settings underlie Figure 5 and Table 7? Are dense and sparse implementations comparably optimized, and are all proxy, thresholding, and mask-handling costs included?

A4. All end-to-end measurements use batch size 1 with 512 prompt tokens and 128 generated tokens. Dense inference, TEAL, and Prox all run in the same gpt-fast framework, and the decoding step of every method is compiled with torch.compile(mode="reduce-overhead", fullgraph=True), so the dense and sparse implementations are comparably optimized. We have added these settings to the revision. With the same settings, we also ran the end-to-end benchmarks on Jetson Thor and RTX 5070 Ti, and the speedups are similar to those on the A6000 and RTX 3090 (Table 7).

For TEAL, we use the kernels from its official open-source repository. For Prox, the reported throughput includes all proxy, thresholding, and mask-handling costs. To show that these overheads are small, we added an FFN-block comparison on an A6000 against Optimal, a compact dense FFN restricted to the retained dimensions (latency in μs):

ModelMethod40%50%60%70%
Qwen3-8BDense453454454454
Optimal291233191152
Prox295232192159
Qwen3-14BDense788788788789
Optimal461384313249
Prox459384318260

Prox stays within 5% of Optimal and runs 1.5–3.0× faster than the dense FFN, so the run-time selection overhead is small.

AuthorsOpenReview ↗

We are keen to discuss further with you

Dear Reviewer TJaN,

We sincerely appreciate your positive assessment and the time you have dedicated to reviewing our work. Your constructive suggestions have helped us improve the quality of our work, and we have uploaded the revised paper accordingly. Specifically, we have compared Prox with R-Sparse and RECAP, as well as WINA and LaRoSA, under a common FFN-only protocol that accounts for auxiliary computation. Prox achieves the highest average score at all sparsity levels. We have also validated the α=0.2\alpha=0.2 cost model: across different stage allocations and sparsity levels, the measured FFN latency stays within 2 percentage points of the prediction.

With the discussion period coming to a close, we would appreciate any feedback on our responses and would be happy to address any remaining questions. Thank you!

Best,

Authors

Reviewer_f4F2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Prox, a training-free method for FFN activation sparsification in LLMs. The key idea is to use a lightweight, quantized proxy to estimate intermediate-channel importance, then recompute only the selected channels using the original weights. This avoids propagating approximation errors while reducing computation across all three SwiGLU projections. Experiments across ten LLMs show improved accuracy over existing training-free sparsification methods, with up to 2.07× end-to-end decoding speedup over dense inference.

Critical strengths

Prox consistently improves upon existing training-free baselines, especially TEAL, at 60–70% sparsity. The improvements in perplexity are particularly notable.

Critical weaknesses
  • Limited novelty over TEAL. The method appears fairly incremental. Both approaches rely on magnitude-based activation sparsity, with Prox primarily introducing a lightweight proxy for channel selection followed by exact recomputation. While effective, this is a relatively straightforward extension rather than a fundamentally new sparsification approach.
  • Insufficient ablation against TEAL. The paper does not isolate how much of the improvement comes from exact recomputation versus the proposed proxy construction. In particular, a TEAL variant that recomputes selected channels exactly would be an important baseline.
  • Limited practical efficiency advantage. Prox achieves better accuracy at high sparsity but is not consistently faster than TEAL at matched sparsity levels. The performance evaluation also focuses on batch-1 decoding on relatively older GPUs, leaving its benefits for modern serving workloads unclear.
Questions for authors
  • Can the authors provide a more direct quality-versus-latency comparison with TEAL, rather than primarily comparing at matched effective sparsity levels?
  • Beyond the two-stage proxy-and-recomputation design, what do the authors consider the main algorithmic novelty relative to existing training-free activation sparsification methods?
4 comments
AuthorsOpenReview ↗

Response to reviewer f4F2

We appreciate the constructive review and comments. Below, we provide our detailed answers in line with your questions and suggestions.

W1. Limited novelty over TEAL: both rely on magnitude-based activation sparsity.

A1. Magnitude-based selection is shared with TEAL. Prox's methodological contribution is to select channels by the joint intermediate state with one shared mask and to separate approximate channel selection from exact value computation, for SwiGLU FFNs in a training-free manner. This separation allows a lightweight proxy to build the mask while confining its errors to channel-selection decisions.

Recall that a SwiGLU FFN computes

y=s,Wdown,s=u⊙h,u=xWup,h=SiLU(xWgate),\mathbf{y}=\mathbf{s}\\,\mathbf{W}_{\mathrm{down}},\quad \mathbf{s}=\mathbf{u}\odot\mathbf{h},\quad \mathbf{u}=\mathbf{x}\mathbf{W}_{\mathrm{up}},\quad \mathbf{h}=\mathrm{SiLU}(\mathbf{x}\mathbf{W}_{\mathrm{gate}}),

where x\mathbf{x} is the FFN input, u\mathbf{u} the up-branch activation, h\mathbf{h} the gate activation, and s\mathbf{s} the intermediate state.

(i) One shared mask derived from both SwiGLU branches. Prox estimates the joint intermediate state before executing the original up and gate projections. Each selected index corresponds to an output channel of up and gate and an input channel of down, so a single mask sparsifies all three projections.

MethodSelection signalUpGateDown
CATSh\mathbf{h}, gate computed denselyoutputdenseinput
COUNTDOWNu\mathbf{u}, up computed denselydenseoutputinput
TEALx\mathbf{x}, then approximate s\mathbf{s}inputinputinput
Proxproxy of the joint state s\mathbf{s}outputoutputinput

Here, input/output denote the sparsified dimension; the Prox row describes Stage 2.

(ii) A proxy used exclusively for selection. Our key finding is that generating the mask requires only the relative ranking of s\mathbf{s}, not its values. This relaxes the accuracy requirement on the approximation, so Prox can use an aggressive INT2, input-sparse proxy to build the mask, and then recompute the selected channels exactly.

W2. The paper does not isolate the contributions of exact recomputation and the proxy; a TEAL variant that recomputes selected channels exactly would be an important baseline.

A2. We added an approximation error analysis in Appendix D.5. Let m\mathbf{m} be the mask applied to the down-projection input and s^\hat{\mathbf{s}} the intermediate state before masking, so that y^=(m⊙s^)Wdown\hat{\mathbf{y}}=(\mathbf{m}\odot\hat{\mathbf{s}})\mathbf{W}_{\mathrm{down}}. The FFN output error splits exactly into

y−y^=((1−m)⊙s)Wdown⏟selection+(m⊙(s−s^))Wdown⏟value,\mathbf{y}-\hat{\mathbf{y}}=\underbrace{\big((\mathbf{1}-\mathbf{m})\odot\mathbf{s}\big)\mathbf{W}_{\mathrm{down}}}_{\text{selection}}+\underbrace{\big(\mathbf{m}\odot(\mathbf{s}-\hat{\mathbf{s}})\big)\mathbf{W}_{\mathrm{down}}}_{\text{value}},

where the selection term comes from dropped channels and the value term from errors on retained channels. The relative errors on Qwen3-8B are:

SparsityMethodSelectionValueTotal
60%TEAL0.1440.1490.213
Prox0.15600.156
70%TEAL0.2080.2140.309
Prox0.21500.215
  • Exact recomputation eliminates the value error, which is as large as the selection error in TEAL.
  • The INT2 proxy raises the selection error by only 8% and 3% over TEAL's FP16 selection, consistent with our finding that the mask depends only on the ranking.
  • Together, Prox lowers the total error by 26.9% and 30.4% relative to TEAL.

The suggested TEAL variant corresponds to our ablation A1, which selects channels with an FP16 input-sparse proxy as TEAL does and then recomputes them exactly. Under the same budget, the FP16 proxy costs 5×5\times as much as the INT2 proxy, so A1 must sparsify the input more and keep fewer channels in Stage 2. As a result, A1 underperforms Prox at all sparsity levels:

Variant40%50%60%70%
TEAL75.173.772.264.4
A1: FP16 proxy + exact recomputation72.571.667.359.6
Prox: INT2 proxy + exact recomputation75.375.173.567.2

The gain of Prox therefore requires both components: exact recomputation removes the value error, and the low-cost proxy makes it affordable.

AuthorsOpenReview ↗

Continued response to reviewer f4F2

W3. Prox is not consistently faster than TEAL at matched sparsity, and the evaluation focuses on batch-1 decoding on older GPUs.

A3. For a fair comparison, we compare all methods at the same effective sparsity, which normalizes FFN computation so that configurations with the same ee have the same theoretical speedup (see Q1 of the General Response). Under this setting, Prox and TEAL are expected to achieve similar speedups by design, and the advantage of Prox lies in higher quality at a matched speedup.

We further evaluated Prox and TEAL on two Blackwell GPUs, Jetson Thor and RTX 5070 Ti (Table 7). For the two models measured on all four GPUs, their end-to-end decoding speedups over dense inference remain close (Prox / TEAL):

GPUModel40%50%60%70%
A6000Qwen3-4B1.27 / 1.341.36 / 1.451.48 / 1.511.61 / 1.74
Mistral-7B1.49 / 1.461.68 / 1.641.84 / 1.892.07 / 2.14
RTX 3090Qwen3-4B1.28 / 1.251.37 / 1.321.47 / 1.441.57 / 1.56
Mistral-7B1.42 / 1.341.56 / 1.461.72 / 1.691.91 / 1.99
Jetson ThorQwen3-4B1.28 / 1.281.41 / 1.371.54 / 1.471.67 / 1.64
Mistral-7B1.42 / 1.441.57 / 1.621.79 / 1.832.03 / 2.06
RTX 5070 TiQwen3-4B1.24 / 1.211.34 / 1.271.43 / 1.391.57 / 1.51
Mistral-7B1.43 / 1.351.58 / 1.541.74 / 1.772.02 / 2.04

We focus on batch-1 decoding because it is the memory-bound regime targeted by training-free activation sparsity methods such as TEAL and CATS.

Q1. A more direct quality-versus-latency comparison with TEAL.

A1. We added a speed–quality Pareto front of Prox and TEAL in the revision (Figure 9). On Qwen3-8B, we sweep both methods from 40% to 70% sparsity and measure the end-to-end speedup on an A6000:

SparsityTEAL speedupTEAL scoreProx speedupProx scoreΔ score
40%1.40×75.11.41×75.3+0.1
45%1.47×74.21.47×75.0+0.9
50%1.53×73.71.53×75.1+1.4
55%1.63×72.71.62×74.2+1.5
60%1.72×72.21.71×73.5+1.3
65%1.84×68.81.82×71.2+2.5
70%1.95×64.41.92×67.2+2.8

The Pareto front of Prox lies above TEAL's over their shared speedup range, and at an equal speedup Prox scores up to 1.9 points higher. For example, Prox at 50% matches the score of TEAL at 40% (75.1) while running at 1.53× instead of 1.40×.

Q2. Beyond the two-stage proxy-and-recomputation design, what is the main algorithmic novelty?

A2. The two-stage design is how we implement Prox; it is built on two findings:

  1. Mask generation needs only an approximate ranking, not accurate values. Since the mask is decided by the relative ranking of channels, the selector can be approximated aggressively, as long as it preserves the ranking.
  2. The mask strategy should follow the SwiGLU computation flow. A mask on the intermediate state s\mathbf{s} applies to all three projections at once: it induces output sparsity in up and gate and input sparsity in down, aligned with the SwiGLU computation flow. In contrast, TEAL treats the three projections as independent operations and sparsifies each of them separately.

We believe these findings can guide future work. By the first finding, any low-cost selector that preserves the ranking can be plugged into Stage 1. By the second, other gated FFNs such as GeGLU can also be sparsified with a single mask on their intermediate state.

AuthorsOpenReview ↗

We are keen to discuss further with you

Dear Reviewer f4F2,

We sincerely appreciate the time and effort you have dedicated to reviewing our work. Your insightful comments have greatly strengthened our paper, and we have carefully addressed them in the uploaded revision. In particular, to isolate the contributions of Prox, we have added an error decomposition (Appendix D.5) that splits the FFN output error into selection error and value error. The results show that exact recomputation eliminates the value error, which is as large as the selection error in TEAL. Meanwhile, the INT2 proxy raises the selection error by only 3–8%. We have also added a speed–quality Pareto comparison with TEAL and new results on Blackwell GPUs (Jetson Thor and RTX 5070 Ti).

As the discussion period deadline approaches, we would be grateful if you could review our responses and let us know if there are any additional questions. Thanks!

Best,

Authors

Thank you for the detailed rebuttal and additional experiments. I appreciate the new error decomposition, quality–latency comparisons, and evaluations on newer GPUs, which address several of my experimental concerns.

My remaining concern is primarily about novelty. The new error analysis suggests that Prox's improvement comes mainly from avoiding the propagation of approximate activation values, rather than producing a more accurate channel-selection mask than TEAL.

Therefore, I'd keep the current rating.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We sincerely appreciate all reviewers and ACs for the constructive reviews and insightful comments. We are glad to see the broad agreement that:

  • Separating channel selection from value computation is effective: the proxy can be aggressively approximated without passing its errors downstream, and no training is required (reviewer f4F2, TJaN, Ebcc).
  • Prox consistently outperforms TEAL at high sparsity across ten models from six families (reviewer f4F2, TJaN, Ebcc).
  • The custom kernels deliver practical end-to-end speedups (reviewer TJaN, Ebcc).

Below we address two common questions shared across reviewers. Responses to the remaining questions are provided in the individual replies, and changes in the revised paper are marked in blue\color{blue}{\text{blue}}.

Q1. What is effective sparsity, and does matching it provide a fair basis for comparing methods? (reviewer f4F2, TJaN, Ebcc)

A1. Effective sparsity is defined as e=1−Cmethod/Cdensee=1-C_{\mathrm{method}}/C_{\mathrm{dense}}, where CmethodC_{\mathrm{method}} includes both the sparse FFN projections and the auxiliary computation accounted for in our cost model. Matching ee therefore gives an equal modeled FFN compute budget, corresponding to a theoretical FFN speedup of 1/(1−e)1/(1-e) under this model. Under this setting, all methods are expected to achieve similar end-to-end speedups, so the differences in task performance objectively reflect the advantage of the sparsification strategy itself. Empirically, Prox and TEAL achieve broadly comparable end-to-end decoding speedups at matched ee across the evaluated models, sparsity levels, and GPUs (Table 7), supporting the practical relevance of this comparison.

  • Prox: The INT2 proxy cost is included in the budget. At e=70e=70\\%, the three Stage 2 projections therefore use s2=74s_2=74\\%, whereas TEAL's three projection sparsities average 70%.
  • Baselines: CATS and COUNTDOWN compute one projection densely. When each of the other two projections has sparsity kk, their effective sparsity is e=2k/3e=2k/3. For TEAL, effective sparsity is the average sparsity across its three FFN projections.

Q2. Comparison with WINA (ICLR 2026), LaRoSA (ICML 2025), R-Sparse (ICLR 2025), and RECAP (ISCA 2025 Workshop). (reviewer TJaN, Ebcc)

A2. We evaluated all four methods on Qwen3-8B, using the official open-source implementations of WINA, LaRoSA, and R-Sparse, and added them to Appendix D.2. Since no official code of RECAP is available, we reimplemented it strictly following the method described in its paper. All methods sparsify only the FFNs, and following the same accounting as for the proxy of Prox, the online auxiliary computation of each method is counted toward its effective sparsity. To meet the target effective sparsity, these methods must therefore sparsify the three FFN projections (up, gate, and down) beyond the target.

For WINA and LaRoSA, the input rotation cannot be absorbed into the weights offline: consecutive layers use different rotations, so the residual stream must be re-rotated online at every layer, through a per-layer H×HH\times H residual adapter in LaRoSA and two H×HH\times H shortcut rotations per layer in the official WINA code. For RECAP, we count only its multiplications and not the sign-guided accumulation over pruned channels, which is the most lenient accounting.

MethodAuxiliary operationCostProj. sparsity 40%50%60%70%
WINAH×HH\times H SVD rotation11.1%51.1%61.1%71.1%81.1%
LaRoSAH×HH\times H PCA rotation11.1%51.1%61.1%71.1%81.1%
R-SparseLow-rank branch2.2–7.2%44.3%57.2%62.9%72.2%
RECAPGroup-wise weight scaling≈0.2%40.2%50.2%60.2%70.3%
TEALNone040%50%60%70%
ProxINT2 proxy4.0–13.3%53.3%63.3%70%74%

The results are as follows:

MethodPPL↓ 40%50%60%70%Avg.↑ 40%50%60%70%
Dense8.1976.3
WINA8.9610.1613.9767.4372.970.261.738.5
LaRoSA8.789.5711.2118.9173.069.761.941.1
R-Sparse8.509.0710.4015.6473.971.269.563.2
RECAP13.7119.9833.0961.6768.062.757.553.5
TEAL8.338.659.5616.1475.173.772.264.4
Prox8.448.598.849.6175.375.173.567.2

Prox achieves the highest average score at all sparsity levels and the lowest perplexity from 50% onward. The low scores of RECAP are consistent with its own paper, where it reduces the MMLU accuracy of Llama3-8B from 65.4 to 40.5 at only 30% sparsity.

Thanks all again. We look forward to further discussions.

Yours,

Authors

Area_Chair_BsVpOpenReview ↗

AC Comment

Dear Reviewers

As we are now halfway through the discussion period, I encourage all reviewers to engage in the responses and discuss should there be more questions.

---AC

AuthorsOpenReview ↗

Summary of Reviews and Rebuttal Updates

Dear ACs and Reviewers,

Thank you very much for your valuable time, constructive feedback, and engagement throughout the review process. To assist with evaluation, we provide a concise summary of the review and discussion so far, along with the updates we have made during the rebuttal.

Paper Summary

This work proposes Prox, a training-free activation sparsification framework for SwiGLU FFNs in LLMs. Our main contributions are:

  • One shared mask from the joint intermediate state: We identify the joint intermediate state of both SwiGLU branches as an effective channel-selection signal, so a single mask following the SwiGLU computation flow sparsifies all three FFN projections at once, enabling simple sparse execution and saving computation in every projection.
  • Approximate selection separated from exact computation: Our key finding is that the mask requires only the ranking of the intermediate state, which a low-cost proxy can preserve. Prox therefore builds it with an aggressive INT2, input-sparse proxy and then recomputes the selected channels exactly, confining proxy errors to channel-selection decisions.
  • Consistent quality gains: Prox outperforms training-free baselines across ten LLMs from six families, as well as WINA, LaRoSA, R-Sparse, and RECAP (added during rebuttal).
  • Real-world efficiency: Custom kernels deliver up to 2.07× end-to-end decoding speedup across four GPUs, with a better speed–quality trade-off than TEAL.

Review Status Before Author Response

We received three reviews (ratings: 3, 2, 2). Across reviewers, the consensus is:

  • Effective design: separating channel selection from value computation allows aggressive approximation without passing errors downstream, and requires no training (reviewers f4F2, TJaN, Ebcc).
  • Consistent gains: Prox outperforms TEAL at high sparsity across ten models from six families (reviewers f4F2, TJaN, Ebcc).
  • Practical speedups: the custom kernels deliver end-to-end decoding speedups (reviewers TJaN, Ebcc).

The main review questions centered on (1) comparison with more related work (WINA, LaRoSA, R-Sparse, and RECAP), and (2) the fairness of effective sparsity and the speed benchmarks against TEAL.

Review Status After Author Response

We have uploaded a revised version of our manuscript, with all changes marked in blue, and addressed both main questions as follows.

For (1), we compared Prox with all four methods on Qwen3-8B and analyzed their extra online computation and storage (e.g., the online rotations of WINA and LaRoSA cost 11.1% of the FFN computation). With these overheads counted, Prox achieves the highest average score at all sparsity levels.

For (2), we clarified that effective sparsity counts all auxiliary costs, so matched sparsity means an equal compute budget, and validated this cost model against measured latency. On two additional Blackwell GPUs, Prox and TEAL again achieve comparable speedups at matched sparsity, and a new speed–quality Pareto front shows that Prox dominates TEAL, achieving higher quality at an equal speedup and higher speedup at an equal quality.

Following these updates:

  • Reviewer Ebcc increased the rating from 2 → 3 (Nov 16, 2026), acknowledging our additional related-work comparisons, error analysis, and redrawn main figure.
  • Reviewer f4F2 acknowledged that our new results address several experimental concerns, and kept the rating of 2 (discussed below).
  • Reviewer TJaN (rating 3) has not yet responded; our responses provide the requested comparisons with R-Sparse and RECAP and validate the cost model against measured latency.

Regarding reviewer f4F2, whose initial concerns on effective sparsity and speed have been clarified, the remaining question is where the improvement comes from. The reviewer contrasts "avoiding the propagation of approximate activation values" with "producing a more accurate channel-selection mask." However, the former is enabled by our distinct mask design: since the shared mask on the joint intermediate state requires only its ranking, a cheap proxy suffices to generate it, which makes exact recomputation of the selected channels affordable. Without this design, simply adding exact recomputation to TEAL still underperforms Prox at every sparsity level (ablation A1).

We would be happy to provide any further clarification or results if needed. Thank you again for your time and thoughtful consideration.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.