acceptodds
Under review as a conference paper at ICLR 2027

Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs

Abstract

This paper introduces model casting, a mid-training recipe that drastically sparsifies the activations within the Feed-Forward Network (FFN) layer. With this strategy, at inference time, we first compute the output of the gating matrix and, thanks to its high sparsity, we avoid computations with the two other matrices, reducing the FLOP count by up to 3×. While this theoretical speedup is an upper bound, model casting translates into significant speedups both on CPU and GPU. We then introduce LoPA Gating, a new FFN design that increases the maximum theoretical speedup. It is a low-FLOPs parameterization of the gating matrix that overcomes the 3× cap by allocating fewer FLOPs and parameters to the gating matrix, compared to the two other FFN matrices that are sparsely activated. We consider two cases: (i) we cast a pre-trained model with a sparsity inducing activation; (ii) we train with LoPA from scratch. In all settings, we significantly outperform existing pruning solutions and regular RELU-fication. For instance, at matched quality, we achieve a 3.2× FLOP speedup with LoPA Casting, against 1.6× at best for competing methods top-p and TEAL. Using dedicated kernels, we achieve an actual 3.31× speed-up on GPU at 90% sparsity, past the 3× ceiling of standard gating; RELU-fication, meanwhile, plateaus below 80% sparsity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, activation functions, sparsity, feed forward network

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pX9zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes model casting: partway through pre-training, SILU is replaced by R-S+ (zero on the negative half-axis and equal to SILU on the positive half-axis). This is combined with an adaptive ℓ1 regularizer tuned toward a target sparsity, so that the FFN gate outputs become highly sparse and the corresponding rows of W3 and columns of W2 can be skipped at inference. The authors further propose LoPA Gating, W1 = G + UV, where G is a fixed matrix with a single 1 in each row. The paper reports CPU/GPU kernel timings and additional experiments on MoE, Qwen3-1.7B, quantization and post-training.

Critical strengths
  1. The controls are carefully designed, and the speedups are measured with real kernels.
  2. LoPA Gating is a meaningful design which lifts the 3× FLOP ceiling of standard gating.
  3. The method is simple to implement.
Critical weaknesses
  1. The core comparison does not separate the contribution of the activation function from that of the regularizer, and the speedup evidence is limited in scope. The RELU-fication baseline explicitly uses no ℓ1 term: "No additional ℓ1 loss is used so the sparsity cannot be controlled." [p.17, 880]. The paper contains no RELU + adaptive ℓ1 @90% control. Without ℓ1, RELU is in fact sparser: in Table 24, RELU reaches 76% versus 69% for R-S+.
  2. Quality retention and several central claims lack support, and the provenance of some results is inconsistent. The contributions claim "without degrading model quality". Yet on Qwen3-1.7B, Activation Casting is still about 1.4% worse than the matched-budget control. For MoE at the 90% target, ΔNLL is +4.19%. In Table 24, the level-8 SILU row is identical to the R-S+ row in all four columns , whereas Table 1 reports 2.145 and 2.153. The FP32 column of Table 20 also differs from the identically named configurations in Table 1.
Questions for authors
  1. Could you add results for RELU + adaptive ℓ1 @90% (with both Standard and LoPA gating) to separate the contributions of the activation function and the regularizer?
  2. Could you move the bf16 results into the main text and add FFN timings with batch size > 1 as well as GPU end-to-end generation timings? Are the sparsity patterns used in the kernel benchmarks taken from real activations or generated synthetically?
  3. Is the level-8 SILU row in Table 24 a transcription error? Why do the FP32 values in Table 20 differ from those in Table 1?
1 comment
AuthorsOpenReview ↗

Extra experiments with ReLU+L1 and batch size>1

Could you add results for RELU + adaptive ℓ1 @90% (with both Standard and LoPA gating) to separate the contributions of the activation function and the regularizer?

We have added the ReLU with L1 results to Table 1 for both activation and LoPA casting. We observe that the NLL is higher compared to the model casting ones that use R-S+with L1. This is expected, since the adaptation required by the model that uses ReLU+L1 is bigger during mid-training than when using R-S+ and the corresponding speedups are about the same for activation casting. For LoPA and architecture casting, the speedup is less when we use ReLU+L1. In conclusion, this ablation confirms our hypothesis that having an activation that follows SiLU on the positive side enables the model to require less adaptation than if we fully swap to ReLU:

Setup (pre-training → mid-training)NLLΔNLLSparsitySpeedup
Baselines: mid-training and ReLU-fication
Base[SiLU] → Base[SiLU] (dense baseline)2.145——1.00×
Base[SiLU] → Base[ReLU] (ReLU-fication)2.159+0.65%76%2.00×
(a) Activation casting
Base[SiLU] → Base[R-S+AO]2.153+0.36%69%1.84×
Base[SiLU] → Base[ReLU, ℓ1@90%] †2.167+1.04%89%2.48×
Base[SiLU] → Base[R-S+AO, ℓ1@90%]2.160+0.68%89%2.48×
(b) LoPA casting
LoPA[SiLU] → LoPA[SiLU]2.150+0.24%—1.00×
LoPA[SiLU] → LoPA[R-S+AO]2.148+0.13%51%1.89×
LoPA[SiLU] → LoPA[ReLU]2.149+0.19%57%2.11×
LoPA[SiLU] → LoPA[ReLU, ℓ1@90%] †2.170+1.16%89%5.78×
LoPA[SiLU] → LoPA[R-S+AO, ℓ1@90%]2.165+0.93%89%5.83×
(c) Architecture casting (−21% parameters and max FLOPs)
Base[SiLU] → LoPA[SiLU]2.180+1.63%—1.39×
Base[SiLU] → LoPA[R-S+AO]2.186+1.90%73%4.21×
Base[SiLU] → LoPA[ReLU]2.194+2.27%78%4.87×
Base[SiLU] → LoPA[ReLU, ℓ1@90%] †2.198+2.47%89%7.86×
Base[SiLU] → LoPA[R-S+AO, ℓ1@90%]2.193+2.20%90%7.94×

† New rows (ReLU + ℓ1@90%), added in response to the reviewer's request.

Could you move the bf16 results into the main text and add FFN timings with batch size > 1 as well as GPU end-to-end generation timings? Are the sparsity patterns used in the kernel benchmarks taken from real activations or generated synthetically?

We will move the bf16 FFN timing into the main text.

The sparsity patterns in the kernel benchmarks are synthetic: we time a 38-layer FFN stack with level13 dimensions (D=3840; hidden size 10240 for the base model, 14336 with a rank-480 low-rank W1 for LoPA) using random weights and a random batch-size-1 input, with masks that zero a fixed fraction of hidden neurons chosen uniformly at random per layer and per step, swept from 0% to 99%.

Our kernels are matrix-vector (batch-1) kernels, so we do not report batch-k wall-clock timings. Instead, we measure on real activations the quantity that bounds them: a row-skipping kernel can only skip a W3/W2 row if the neuron is inactive for all k tokens in the batch, so this union sparsity caps the achievable FFN saving for any implementation. For LoPA casting@90%, 69–74% of neurons remain skippable at k=4 (an FFN FLOP reduction of up to 2.75–3.11×) and 53–59% at k=8 (1.96–2.19×), falling to 19–25% at k=32. Small-batch decoding retains most of the benefit, while large-batch serving does not; we will state this as a limitation in the new paper version.

k12481632
LoPA casting@90%
Batched decoding, self-generated text (T=0.7, top-p 0.9)91.0% (6.34x)84.1% (4.50x)73.5% (3.11x)59.1% (2.19x)42.0% (1.63x)25.1% (1.29x)
Batched decoding, real validation text89.1% (5.70x)81.0% (3.98x)68.9% (2.75x)53.1% (1.96x)35.4% (1.48x)19.1% (1.21x)
  • Self-generated text: 32 sequences of 64 real prompt tokens followed by 400 tokens the model generated itself; masks taken on the generated tokens.
  • Real validation text: 32 real validation documents (9 domains), 400 tokens each, run through the model.
Reviewer_MM6fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes model casting, a mid-training recipe that changes part of the model architecture partway through training to improve inference efficiency. During mid-training, the SiLU activation in each FFN is replaced with R-S+ (a ReLU/SiLU hybrid), and an adaptive ℓ1 loss is added so that FFN activation sparsity can be pushed to a specified target (around 90%). The paper also introduces LoPA Gating, which reparameterizes the gate matrix W1 as G + UV, where G is a fixed matrix with one nonzero entry per row and UV is a low-rank term. The parameters saved this way are redistributed to W2 and W3, which lets the method exceed the 3× FLOP ceiling of standard gating.

Critical strengths
  1. The experimental validation is broad. It covers multiple model scales (up to 5.79B), MoE models, an off-the-shelf model (Qwen3-1.7B), and quantization-aware training.
  2. The speedups are measured with actual CPU/GPU kernels, not only as theoretical FLOP reductions (Section 5.3, Appendix R).
  3. The adaptive ℓ1 loss lets sparsity be set directly as a target value, which makes the method practical.
Critical weaknesses
  1. Unclear relation to SLTrain. LoPA Gating appears structurally similar to SLTrain (Han et al., 2024), which also combines a sparse component with a low-rank component. The paper cites SLTrain but does not explain precisely how LoPA differs from it, and it does not compare the two empirically.
  2. Missing baseline: ReLU-fication with ℓ1 loss. The current comparison changes the activation and the regularizer at the same time. Without ReLU-fication + ℓ1, it is unclear whether the gains in sparsity and NLL come from R-S+ itself or mainly from the ℓ1 loss.
  3. Weak justification for the design of G. The positions and signs of the nonzero entries in G are arbitrary; Appendix A states that any fixed ±1 per row performs equally well. Table 4 shows that Low-rank + Diagonal outperforms Low-rank + Identity even though both have the same maximum rank, but the paper does not explain why.
  4. Narrow practical scope. The reported speedups appear to hold only at batch size 1. Section 5.3 acknowledges that sparsity patterns differ across batch elements, so the rows and columns needed are the union of the masks. However, no throughput results are reported for batch sizes larger than 1, which limits how relevant the method is for real serving.
Questions for authors
  1. What exactly distinguishes LoPA Gating from SLTrain (for example, trainable vs. fixed sparse component, or applying it to the gate only vs. the whole FFN)? Could the authors provide an empirical comparison under matched parameter and FLOP budgets?
  2. Could the authors report results for ReLU-fication + adaptive ℓ1 at the same 90% sparsity target and compare it with R-S+ + ℓ1 in terms of NLL and achieved sparsity? This would isolate the contribution of R-S+.
  3. What is the rationale behind the design of G? Why does a fixed matrix with exactly one nonzero per row outperform Low-rank + Identity in Table 4, given that both have the same maximum rank?
  4. How does throughput change for batch sizes larger than 1 (e.g., 4, 8, 16, 32)? How quickly does effective sparsity (the union of masks) decrease as batch size grows, and at what batch size does the speedup disappear?
3 comments

We thank the reviewer for their feedback, here are our replies:

What exactly distinguishes LoPA Gating from SLTrain (for example, trainable vs. fixed sparse component, or applying it to the gate only vs. the whole FFN)? Could the authors provide an empirical comparison under matched parameters and FLOP budgets?

SLTrain and LoPA Gating share the same basic idea: write a weight matrix as low-rank plus sparse, W = BA + S, where BA is a low-rank factorization and S is a sparse matrix. However, we use this similar idea for different purposes: SLTrain reduces the training cost (with no changes at inference time), while LoPA Gating enables cheaper inference (without reducing the cost of pre- and mid-training).

Both corresponding low rank factorisation starts like LoRA: A uses Kaiming initialization, B starts at zero, and the product is scaled by α/r. In both cases, where the sparse entries go doesn't matter much. SLTrain shows random positions work about as well as the largest residual entries (perplexity 51.98 vs 53.75). In the case of LoPA Gating, we show in Table 4 of Appendix A that various patterns with a single +/-1 per row work equally well, and we found in preliminary experiments that designs with more than one non-zero coefficient underperformed.

Main differences:

  • During the forward pass, SLTrain still requires to build W1 and does a full matmul whereas in LoPA Gating, we bypass building W1 and just compute Sx + B·RMSNorm(Ax). Hence, SLTrain’s inference cost is the same as a dense layer or slightly more, whereas the inference cost of LoPA gating is r(D+H′) instead of HD; with r = D/8 that is 13.5× cheaper at 100% sparsity. The hidden size H′ is larger, to keep the total parameter count fixed.

  • In SLTrain, the sparse matrix S sets some trainable values at a fixed set of positions whereas in LoPA Gating the sparse matrix S has exactly one ±1 per row (no parameters), these are in a fixed structured pattern (a repeated diagonal or tiled identity).

  • SLTrain is used for training from scratch, plus a fine-tuning variant in its appendix. In contrast, LoPA Gating can be used for training from scratch or during mid-training starting from a standard dense model, but ultimately targets inference speedups. It converts an existing dense model (Architecture Casting) by fitting W1 ≈ M + UV with alternating steps: pick each row's ±1 at the largest-magnitude entry of W − UV, then take a truncated SVD of W − M.

The key difference between SLTrain and LoPA Gating is the purpose and scope of their reparameterization. Although both use a low-rank-plus-sparse form, SLTrain reparameterizes all linear layers to reduce training memory and parameter count, whereas LoPA Gating reparameterizes only the FFN gate (W1) to reduce inference FLOPs. LoPA Gating reallocates the saved parameters to W2/W3, keeping the total parameter count unchanged. We will expand the SLTrain discussion in the related work to clarify these differences.

What is the rationale behind the design of G? Why does a fixed matrix with exactly one nonzero per row outperform Low-rank + Identity in Table 4, given that both have the same maximum rank?

In the Low-rank + Identity part, while the d*D matrix is indeed the same rank as in our proposal, however note that D-d out of the D non-linearities after the matmul are associated to the low-rank parametrization only. Only the first d non-linearities benefit from the full rank identity matrix. This contributes to reducing the diversity of these activations compared to our design.

How does throughput change for batch sizes larger than 1 (e.g., 4, 8, 16, 32)? How quickly does effective sparsity (the union of masks) decrease as batch size grows, and at what batch size does the speedup disappear?

Small batch sizes>1. Small-batch GPU inference is essential for latency-sensitive assistants and coding tools, where fast individual responses directly shape the user experience. Optimizing low-batch decoding serves this need alongside larger-batch workloads optimized for throughput and cost per token.

In the table below, we see that our approach remains competitive for small batch sizes for both self-generated text and real validation text: with 4 concurrent prompts, 74% (self-generated) and 69% (real text) of FFN neurons can still be skipped, and with 8 prompts, 59% and 53%. The corresponding speedups are 2.75x-3.11x for 4 prompts and 1.96x-2.19x for 8 prompts. Beyond that, the shared sparsity drops quickly (19–25% at batch size 32). This is because unrelated prompts share few inactive neurons.

k12481632
LoPA casting@90%
Batched decoding, self-generated text (T=0.7, top-p 0.9)91.0% (6.34x)84.1% (4.50x)73.5% (3.11x)59.1% (2.19x)42.0% (1.63x)25.1% (1.29x)
Batched decoding, real validation text89.1% (5.70x)81.0% (3.98x)68.9% (2.75x)53.1% (1.96x)35.4% (1.48x)19.1% (1.21x)

[Continues]

AuthorsOpenReview ↗

Extra experiment using ReLU+L1

  • Self-generated text: 32 sequences of 64 real prompt tokens followed by 400 tokens the model generated itself; masks taken on the generated tokens.
  • Real validation text: 32 real validation documents (9 domains), 400 tokens each, run through the model.

Could the authors report results for ReLU-fication + adaptive ℓ1 at the same 90% sparsity target and compare it with R-S+ + ℓ1 in terms of NLL and achieved sparsity? This would isolate the contribution of R-S+.

We have added the ReLU with L1 results to Table 1 for both activation and LoPA casting. We observe that the NLL is higher compared to the model casting ones that use R-S+with L1. This is expected, since the adaptation required by the model that uses ReLU+L1 is bigger during mid-training than when using R-S+ and the corresponding speedups are about the same for activation casting. For LoPA and architecture casting, the speedup is less when we use ReLU+L1. In conclusion, this ablation confirms our hypothesis that having an activation that follows SiLU on the positive side enables the model to require less adaptation than if we fully swap to ReLU:

Setup (pre-training → mid-training)NLLΔNLLSparsitySpeedup
Baselines: mid-training and ReLU-fication
Base[SiLU] → Base[SiLU] (dense baseline)2.145——1.00×
Base[SiLU] → Base[ReLU] (ReLU-fication)2.159+0.65%76%2.00×
(a) Activation casting
Base[SiLU] → Base[R-S+AO]2.153+0.36%69%1.84×
Base[SiLU] → Base[ReLU, ℓ1@90%] †2.167+1.04%89%2.48×
Base[SiLU] → Base[R-S+AO, ℓ1@90%]2.160+0.68%89%2.48×
(b) LoPA casting
LoPA[SiLU] → LoPA[SiLU]2.150+0.24%—1.00×
LoPA[SiLU] → LoPA[R-S+AO]2.148+0.13%51%1.89×
LoPA[SiLU] → LoPA[ReLU]2.149+0.19%57%2.11×
LoPA[SiLU] → LoPA[ReLU, ℓ1@90%] †2.170+1.16%89%5.78×
LoPA[SiLU] → LoPA[R-S+AO, ℓ1@90%]2.165+0.93%89%5.83×
(c) Architecture casting (−21% parameters and max FLOPs)
Base[SiLU] → LoPA[SiLU]2.180+1.63%—1.39×
Base[SiLU] → LoPA[R-S+AO]2.186+1.90%73%4.21×
Base[SiLU] → LoPA[ReLU]2.194+2.27%78%4.87×
Base[SiLU] → LoPA[ReLU, ℓ1@90%] †2.198+2.47%89%7.86×
Base[SiLU] → LoPA[R-S+AO, ℓ1@90%]2.193+2.20%90%7.94×

† New rows (ReLU + ℓ1@90%), added in response to the reviewer's request.

Reviewer_MM6fOpenReview ↗

Response for Rebuttal

Thank you for your patience while waiting for my response, and for the additional experiments. They address several of my concerns. From the ReLU + ℓ1 results, R-S+ helps, but the gain looks small. Most of the sparsity and speedup seem to come from the ℓ1 loss. So I will keep my rating.

Reviewer_YTcpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a recipe to the reduce the time spent on the SwiGLU at inference for decode regime with token =1 (no speculative decoding). The main target is the MLP's gating activations and its weight matrix W1W_1: zero gating neuron imply that the corresponding up-proj output neuron and down-proj input neurons are not necessary.

To induce neuron sparsity, the paper introduces "activation casting": replace SiLU(a) by 1(a>0)*SiLU(a), and adds ℓ1\ell_1 regularization during mid-training to tip other entries into the flat zone. That still requires using the full W1W_1 matrix, which the authors address via low-rank parametrization (LoPA? casting) as W1x=Gx+U⋅RMSNorm(Vx)W_1 x=Gx+U\cdot RMSNorm(Vx) (G has one none-zero per row to "fake" a full rank) . The activation and LoPA casting can be enforced during the pre/mid-training, or integrated on a pretained model, which the author call "Architecture" casting.

The authors provide an extensive set of results that demonstrate the feasibility of the suggested approach, and its exploitation of the induced sparsity to speed-up the FFN layer via custom kernels, with pretraining-from scratch of a model scaling ladder using up to 9T tokens, as well as "Architecture casting" of Qwen3 1.7B.

Critical strengths

The problem tackled has practical applications and benefits most from the pre-training phase where the model can adapt to the added architectural and activation constraints. To decide which neurons are inactive, the MLP's gate still needs to be evaluated. The LoPA gating is thus a promising approach at reducing the gate's size before the inactive neurons are determined, and costs almost as much as a simple LoRA parameterization. The authors also exploit the theoretical guarantees to implement custom kernels and demonstrate the feasibility of their proposed approach. The alignment of a theoretical approaches with hardware constraints require a holistic mastery of LLM pre-training and inference, which should be appreciated in this era of pure empirical work.

Critical weaknesses

There two major weaknesses in the current work:

  • A) It is difficult to isolate the contribution of each component to the final results.
  • B) The targeted intended regime (1 token decoding at batch size 1, without evaluating speculative draft tokens) is unclear, and its practical relevance is not sufficiently justified.

The paper general approach reads as “reducing FFN cost via gate activations by any means necessary”, combining activation replacement, adaptive L1 regularization, low-rank+fixed sparse gate, RMSNorm, second activation and width reallocation (for LoPA trained from scratch), custom kernels, and quantization. Each of these interventions affect various aspects of the model and has different costs. The experiments should make clear which components are necessary, how they interact, and what each contributes to the final quality and practical speedup.

For instance, pre-training a LoPA-aware model from scratch and fine-tuning/adaptation of an existing model are entirely different challenges. For an off-the-shelf model like Qwen3-1.7B-Base, extensive continued training on a different mixture risks affecting downstream performance and causing catastrophic forgetting. The matched SiLU control is useful here: it separates degradation caused by the mid-training mixture from the additional degradation caused by casting. However, both costs matter when assessing whether adapting an existing checkpoint is practically worthwhile. For already post-trained models, preserving instruction-following and alignment is another question: casting a base model and then applying SFT does not establish that casting an already post-trained checkpoint preserves its behavior.

The authors already mention that FFN encode parameteric memorization, and thus modifying them should be evaluated on knowledge tasks. Such evaluations are present for Qwen, but the observed task-specific losses need to be central to the interpretation of the results. Metrics such NLL are not sufficient for this purpose.

How the various combined techniques interact is also not sufficiently clear: at equal achieved activation sparsity, and controlled parameter and training budgets, what does LoPA add on top of R-S+ + L1? Does it improve quality, measured latency, or both compared with standard gating using the same activation and regularization? Does this benefit hold both for LoPA-aware pre-training and for Architecture Casting of an existing standard model? LoPA and R-S+ change different components, so comparing them as alternatives would not isolate this contribution.

Under the assumption that gate pre-activation are equally distributed on both sides of 0, simple ReLU-fication would give around 50% sparsit. However, this symmetry is not guaranteed in a trained model. How does adding LoPA change the pre-activation distribution and the sparsity obtained before applying L1? Table 1 already reports 69% sparsity for standard gating with R-S+ versus 51% for LoPA with R-S+, without L1. What explains this difference, and how much comes from the gate parameterization versus the accompanying normalization, second activation, and width changes?

Regarding the second point, the practical case seem to mainly address a “single-user” assistant on an edge-device without any speculative drafter. This can be a legitimate setting, but the paper needs to justify the hardware and memory constraints that make it the intended target. An agentic application may involve long accumulated contexts, repeated prefills, and parallel generations. Longer contexts increase the attention and KV-cache costs, while prefill and speculative verification process multiple tokens with potentially different activation masks. In these regimes, the benefit of per-token activation sparsification may shrink or disappear, and its overhead could even introduce a slowdown. These possibilities need to be evaluated rather than extrapolating from 1 token FFN timings.

For Hopper GPUs, the full recipe is hard to justify. Such substantial adaptation needs to earn its cost against optimized FP8 inference, at least, rather than centering the practical claims on the “nerfed” FP32 baseline. A relevant additional baseline is FP8 + hardware-supported 2:4 weight sparsity, using a method such as SparseGPT (picking it because it's easy & fast) for pruning and an appropriate sparse serving backend. This preserves the architecture and activation function and avoids extended gradient-based adaptation, although its quality loss and calibration cost still need to be measured. The relevant comparison is the actual end-to-end quality–latency trade-off, including preparation and recovery costs. These results should be central to any practical claim, instead of the approximately 8x FFN speedups reported w.r.t. an FP32 reference.

The amount of effort in this work and all the ablations performed is undeniable. Unfortunately, the current evaluation does not establish how much practical value remains once speculative drafters, parallel decoding, and optimized low-precision kernels are considered. Activation-mask construction, indexing, and sparse dispatch during verification could outweigh the saved work compared with optimized dense or fused kernels. Given alternatives that avoid changing the activation or require much less adaptation, the suggested approach needs to demonstrate what its additional complexity buys at comparable downstream quality. Without these comparisons, it holds little demonstrated legitemacy as an optimization for a practical serving pipeline.

Questions for authors

The following questions are more of a summary of the concerns I raised, but are not exhaustive:

    1. Which components actually improve quality or speed, and which can be removed?
    1. At equal sparsity and matched budgets, what does LoPA add to R-S+ + L1? Does this hold both from scratch and when casting an existing model?
    1. What hardware and workload justify 1 token decoding without speculative draft tokens?
    1. What end-to-end gain remains against FP8 and SparseGPT + 2:4 FP8, including prefill, batching, and speculative verification?
    1. How much adaptation is needed, what downstream performance is lost, and does an already trained model need recovery training?
    1. What does the acronym LoPA stand for?
5 comments

We thank the reviewer for the detailed review. We first add the snippet of the relevant point we address together with our reply:

Each of these interventions affect various aspects of the model and has different costs. The experiments should make clear which components are necessary, how they interact, and what each contributes to the final quality and practical speedup.

Firstly, note that neither the custom kernels nor the quantization study are part of the paper's main contribution: the custom kernel enables to materialise the theoretical speedups as wall clock time. The quantization study is only there to showcase that model casting is complementary to quantization, and therefore, we can combine both to get further speed ups. It is not part of the main contribution, which is why it is discussed in Appendix but it is an orthogonal technique.

We are not sure why the reviewer thinks that there isn’t evidence of the contribution of each part of the model casting recipe, we provide evidence to where each part is located herein:

  1. We present an ablation for the effect of using different activation functions during mid-training in Table 24 of Appendix P, we showcase the contribution of only swapping the activation to R-S+ without L1 compared to other popular choices for activation functions.

  2. Furthermore, Table 1 showcases the effect of every ingredient in terms of quality (validation NLL). Here is a detailed description of the information provided in the Table:

  • Under the Activation casting section, the first row shows the effect of only swapping the activation to R-S+ during mid-training (no L1)
  • Under the Activation casting section, the second row shows the effect of only changing the activation plus using L1.
  • Under the LoPA casting section, in the first row, we show the effect of a model that was pre-trained with LoPA gating but mid-trained with the standard SiLU activation
  • Under the LoPA casting section, in the third row, a model that was pre-trained with LoPa gating and mid-trained with the sparse activation without L1.
  • Under the LoPA casting section, in the fourth row, a model that was pre-trained with LoPa gating and mid-trained with the sparse activation plus L1.
  • Under the Architecture casting section, in the first row, a model that was pre-trained in the standard way and mid-trained with LoPA gating only
  • Under the Architecture casting section, in the third row, a model that was pre-trained in the standard way and mid-trained with both LoPA and the sparse activation swap, without using L1.
  • Under the Architecture casting section, in the fourth row, a model that was pre-trained in the standard way and mid-trained with both LoPA and the sparse activation swap with L1.
  1. Additionally, in Table 18 in Appendix J, we present the downstream performance using standard evaluation benchmarks for activation casting with and without L1 and for LoPA casting, so we aren’t basing our analysis only on NLL.

…pre-training a LoPA-aware model from scratch and fine-tuning/adaptation of an existing model are entirely different challenges. For an off-the-shelf model like Qwen3-1.7B-Base, extensive continued training on a different mixture risks affecting downstream performance and causing catastrophic forgetting. The matched SiLU control is useful here: it separates degradation caused by the mid-training mixture from the additional degradation caused by casting. However, both costs matter when assessing whether adapting an existing checkpoint is practically worthwhile.

We agree with this statement. This is why we present both scenarios (train from scratch with LoPA plus mid-trained with activation swap plus L1 vs standard pre-training and mid-trained with LoPA and activation swap plus L1 is to demonstrate the generality of our approach. Furthermore,

i) In Table 17 in Appendix J, we report the delta against the SILU, no swap control (the same model mid-trained on the same data mixture without the activation swap) to isolate the cost of casting from the effect of the mid-training data mixture. This provides evidence that the QWEN3 mid-training degradation is due to the data mix and not the model casting component.

ii) In Table 18 of Appendix J, we showcase that once we have control over the data mix, e.g. we use a model trained from scratch in-house, then the mid-trained model does well both in terms of validation NLL and the evaluation suite. This latter experiment enables us to remove the effect of catastrophic forgetting once we have full control of the entire training pipeline. QWEN3 data mix is not public as far as we know.

[Continues in next comment]

We know answer the specific questions raised:

  • How the various combined techniques interact is also not sufficiently clear: at equal achieved activation sparsity, and controlled parameter and training budgets, what does LoPA add on top of R-S+ L1? Does it improve quality, measured latency, or both compared with standard gating using the same activation and regularization? Does this benefit hold both for LoPA-aware pre-training and for Architecture Casting of an existing standard model? LoPA and R-S+ change different components, so comparing them as alternatives would not isolate this contribution.*

See Table 1 and our reply above. Moreover, in Table 14 of Appendix G, we compare the Architecture Casting mid-trained model to a SwiGLU model trained from scratch with a reduced number of parameters. This is achieved by reducing the hidden dimension of the model to H = 3712, so that the number of parameters per FFN is equal to the parameter count of the FFNs after Architecture Casting. The gap between the dense baseline with reduced parameters and the Architecture casting model is initially large, because casting is highly disruptive: the model must simultaneously absorb a new activation and a reduced-rank gate. The NLL gap narrows as the casting time increases, and Architecture casting with SILU outperforms the baseline at iso-parameters.

What explains this difference, and how much comes from the gate parameterization versus the accompanying normalization, second activation, and width changes?

Note that neither the normalisation, second activation or width changes can induce sparsity on their own. You might start with a different distribution for the pre-activations but it is not sparse if we use Swiglu. In the paper and appendices, we already report ablations or explanations that should already provide answers on most of the points raised by the reviewer:

  • For adding the second activation ablation, in Table 4 in Appendix A, we show that a double activation improves the NLL for a trained from scratch model with LoPA Gating (in W1 and W3) versus using an activation in W1 only.

  • In LoPA Gating, we increase the FFN hidden dimension from H to H′ > H, widening W2 and W3, so that the parameters saved by the low-rank gate are reallocated to these two matrices and the FFN parameter count stays unchanged.

  • In LoPA Gating, we inserted an RMSNorm between U and V because factorizing W1 adds depth, and in our experiments, this made large-scale training unstable without normalization. Its effect at small scale may be minor, or even slightly negative, but it is what keeps large-scale training stable. A small-scale ablation would therefore not show its real purpose, which is why we did not include one.

How much adaptation is needed, what downstream performance is lost, and does an already trained model need recovery training?

In Table 19 in Appendix K, we show how much adaptation is needed in terms of the mid-training duration required to catch up with the SILU baseline model’s performance, without the decay phase using varying model sizes. Typically, the model adapts quickly, requiring from 1T-1.25T for activation casting and 0.5T-1T for LoPA casting.

We are not sure what “recovery” training means here: as explained above, we show that if we have full control over the data mix, using model casting during mid training does not degrade model performance. In Table 18 in Appendix J, we present the downstream performance using evaluation benchmarks for Activation casting with and without L1 and for LoPA casting. We show the deltas with respect to the dense SILU baseline, in general, the degradation for activation and LoPA casting is less than one point in terms of average performance.

What does the acronym LoPA stand for?

LoPA Gating refers to the new FFN design that increases the maximum theoretical speedup. The LoPA acronym stands for Low-FLOPs parameterization of the gating matrix, as mentioned in the abstract when the term is first introduced.

[Continues in next comment]

AuthorsOpenReview ↗

Speculative decoding additional experiment

The targeted intended regime (1 token decoding at batch size 1, without evaluating speculative draft tokens) is unclear, and its practical relevance is not sufficiently justified.

We conducted an experiment to address the reviewer’s comment and show that we are competitive for speculative decoding (k drafted tokens of a single sequence verified in one pass). We used a 5.79B-parameter models (level 13 in Table 12 from Appendix E) to compute the fraction of FFN neurons that can be skipped when k tokens are processed together, we also include the FFN FLOP speedup.

Regime. The single-sequence autoregressive decoding is the latency-critical setting for interactive assistants, coding tools, and on-device/CPU inference. In this regime, the FFN is memory-bandwidth bound, so skipping inactive rows of W_2/W_3 directly reduces the bytes loaded per token, which is why the sparsity translates into wall-clock gains. This complements throughput-oriented large-batch serving (e.g. server-side inference) rather than directly competing with it.

Speculative decoding. Our method is orthogonal to speculative decoding: it changes the model itself, not the decoding algorithm. Hence, model casting can be applied to the drafter, to the target model, or to both. When the target model verifies k draft tokens, the active neurons are the union over those tokens, so the effective sparsity decreases with k. However, at 90% per-token sparsity, the union over a small number of draft tokens remains substantially sparse because typically, k is between 4 and 8, which keeps the sparsity between 81.4%-74.2% with speedup between 5.12-4.05x for a LoPA casted model. We also observe that at the same k, speculative verification keeps more sparsity than batching, which is not surprising thanks to the correlation between hypotheses. In the following table, we report the fraction of FFN neurons that can be skipped when k tokens are processed together, we also include the FFN FLOP speedup in parentheses:

k12481632
LoPA casting@90%
Speculative verify (self-generated)91.0% (6.34x)87.0% (5.12x)81.4% (4.05x)74.2% (3.18x)65.8% (2.54x)56.8% (2.10x)
Batched decoding (self-generated)91.0% (6.34x)84.1% (4.50x)73.5% (3.11x)59.1% (2.19x)42.0% (1.63x)25.1% (1.29x)
Independent-tokens baseline (b0^k)91.0% (6.34x)83.2% (4.34x)70.4% (2.86x)52.3% (1.93x)31.6% (1.40x)13.9% (1.14x)
Greedy (upper bound)94.6% (8.02x)91.9% (6.70x)88.1% (5.40x)83.0% (4.30x)77.1% (3.47x)71.4% (2.94x)

Each row corresponds to:

Speculative verify: k consecutive tokens of one sequence, as in the speculative-decoding verify pass over k drafted tokens. The model generated the text itself (T=0.7, top-p 0.9).

Batched decoding: k unrelated prompts at the same position, i.e. ordinary batched decoding with batch size k.

Independent-tokens baseline: what we'd get if each token's sparsity pattern were independent: batch-1 sparsity raised to the power k.

Greedy (upper bound): speculative verify again, but on greedy-decoded text. It's inflated because most greedy generations fall into repetition loops.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.