There two major weaknesses in the current work:
- A) It is difficult to isolate the contribution of each component to the final results.
- B) The targeted intended regime (1 token decoding at batch size 1, without evaluating speculative draft tokens) is unclear, and its practical relevance is not sufficiently justified.
The paper general approach reads as “reducing FFN cost via gate activations by any means necessary”, combining activation replacement, adaptive L1 regularization, low-rank+fixed sparse gate, RMSNorm, second activation and width reallocation (for LoPA trained from scratch), custom kernels, and quantization. Each of these interventions affect various aspects of the model and has different costs. The experiments should make clear which components are necessary, how they interact, and what each contributes to the final quality and practical speedup.
For instance, pre-training a LoPA-aware model from scratch and fine-tuning/adaptation of an existing model are entirely different challenges. For an off-the-shelf model like Qwen3-1.7B-Base, extensive continued training on a different mixture risks affecting downstream performance and causing catastrophic forgetting. The matched SiLU control is useful here: it separates degradation caused by the mid-training mixture from the additional degradation caused by casting. However, both costs matter when assessing whether adapting an existing checkpoint is practically worthwhile. For already post-trained models, preserving instruction-following and alignment is another question: casting a base model and then applying SFT does not establish that casting an already post-trained checkpoint preserves its behavior.
The authors already mention that FFN encode parameteric memorization, and thus modifying them should be evaluated on knowledge tasks. Such evaluations are present for Qwen, but the observed task-specific losses need to be central to the interpretation of the results. Metrics such NLL are not sufficient for this purpose.
How the various combined techniques interact is also not sufficiently clear: at equal achieved activation sparsity, and controlled parameter and training budgets, what does LoPA add on top of R-S+ + L1? Does it improve quality, measured latency, or both compared with standard gating using the same activation and regularization? Does this benefit hold both for LoPA-aware pre-training and for Architecture Casting of an existing standard model? LoPA and R-S+ change different components, so comparing them as alternatives would not isolate this contribution.
Under the assumption that gate pre-activation are equally distributed on both sides of 0, simple ReLU-fication would give around 50% sparsit. However, this symmetry is not guaranteed in a trained model. How does adding LoPA change the pre-activation distribution and the sparsity obtained before applying L1? Table 1 already reports 69% sparsity for standard gating with R-S+ versus 51% for LoPA with R-S+, without L1. What explains this difference, and how much comes from the gate parameterization versus the accompanying normalization, second activation, and width changes?
Regarding the second point, the practical case seem to mainly address a “single-user” assistant on an edge-device without any speculative drafter. This can be a legitimate setting, but the paper needs to justify the hardware and memory constraints that make it the intended target. An agentic application may involve long accumulated contexts, repeated prefills, and parallel generations. Longer contexts increase the attention and KV-cache costs, while prefill and speculative verification process multiple tokens with potentially different activation masks. In these regimes, the benefit of per-token activation sparsification may shrink or disappear, and its overhead could even introduce a slowdown. These possibilities need to be evaluated rather than extrapolating from 1 token FFN timings.
For Hopper GPUs, the full recipe is hard to justify. Such substantial adaptation needs to earn its cost against optimized FP8 inference, at least, rather than centering the practical claims on the “nerfed” FP32 baseline. A relevant additional baseline is FP8 + hardware-supported 2:4 weight sparsity, using a method such as SparseGPT (picking it because it's easy & fast) for pruning and an appropriate sparse serving backend. This preserves the architecture and activation function and avoids extended gradient-based adaptation, although its quality loss and calibration cost still need to be measured. The relevant comparison is the actual end-to-end quality–latency trade-off, including preparation and recovery costs. These results should be central to any practical claim, instead of the approximately 8x FFN speedups reported w.r.t. an FP32 reference.
The amount of effort in this work and all the ablations performed is undeniable. Unfortunately, the current evaluation does not establish how much practical value remains once speculative drafters, parallel decoding, and optimized low-precision kernels are considered. Activation-mask construction, indexing, and sparse dispatch during verification could outweigh the saved work compared with optimized dense or fused kernels. Given alternatives that avoid changing the activation or require much less adaptation, the suggested approach needs to demonstrate what its additional complexity buys at comparable downstream quality. Without these comparisons, it holds little demonstrated legitemacy as an optimization for a practical serving pipeline.
Extra experiments with ReLU+L1 and batch size>1
Could you add results for RELU + adaptive ℓ1 @90% (with both Standard and LoPA gating) to separate the contributions of the activation function and the regularizer?
We have added the ReLU with L1 results to Table 1 for both activation and LoPA casting. We observe that the NLL is higher compared to the model casting ones that use R-S+with L1. This is expected, since the adaptation required by the model that uses ReLU+L1 is bigger during mid-training than when using R-S+ and the corresponding speedups are about the same for activation casting. For LoPA and architecture casting, the speedup is less when we use ReLU+L1. In conclusion, this ablation confirms our hypothesis that having an activation that follows SiLU on the positive side enables the model to require less adaptation than if we fully swap to ReLU:
† New rows (ReLU + ℓ1@90%), added in response to the reviewer's request.
Could you move the bf16 results into the main text and add FFN timings with batch size > 1 as well as GPU end-to-end generation timings? Are the sparsity patterns used in the kernel benchmarks taken from real activations or generated synthetically?
We will move the bf16 FFN timing into the main text.
The sparsity patterns in the kernel benchmarks are synthetic: we time a 38-layer FFN stack with level13 dimensions (D=3840; hidden size 10240 for the base model, 14336 with a rank-480 low-rank W1 for LoPA) using random weights and a random batch-size-1 input, with masks that zero a fixed fraction of hidden neurons chosen uniformly at random per layer and per step, swept from 0% to 99%.
Our kernels are matrix-vector (batch-1) kernels, so we do not report batch-k wall-clock timings. Instead, we measure on real activations the quantity that bounds them: a row-skipping kernel can only skip a W3/W2 row if the neuron is inactive for all k tokens in the batch, so this union sparsity caps the achievable FFN saving for any implementation. For LoPA casting@90%, 69–74% of neurons remain skippable at k=4 (an FFN FLOP reduction of up to 2.75–3.11×) and 53–59% at k=8 (1.96–2.19×), falling to 19–25% at k=32. Small-batch decoding retains most of the benefit, while large-batch serving does not; we will state this as a limitation in the new paper version.