acceptodds
Under review as a conference paper at ICLR 2027

PowLU: An Activation Function for Stable Pre-Training of LLMs

Abstract

In contemporary large language models (LLMs), the swish-gated linear unit (SwiGLU) activation function is widely adopted to regulate the information flow and introduce non-linearity. In its scalar unary representation, SwiGLU approximates the quadratic function for large positive inputs, providing strong nonlinearity and expressive capacity. However, this property also causes numerical instability as the input or model scale increases, particularly in low-precision LLM training. The main reason is the approximate quadratic amplification reflected by its unary form, which enlarges the output range and exacerbates outliers. To address this issue, we propose a stable activation function, Power Linear Unit (PowLU), for large-scale LLM pre-training. Specifically, PowLU employs a rational power function to achieve adaptive nonlinearity, thereby improving representation ability and enabling stable training in spike regions. Moreover, we provide theoretical justification for several key properties of PowLU. Scaling law experiments confirm that the performance is consistent across model sizes, and further experimental results with the Ling-3.0 architecture (7.9B and 124B total parameters) demonstrate that PowLU achieves competitive results against SwiGLU and SwiGLU-Clip in large-scale training of LLMs. In addition, the experimental results also show that PowLU effectively improves the scalability of the large-scale training of LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Pre-Training, Activation Function

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CqRhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new activation function, PowLU (Power Linear Unit), for the FFN in LLM architectures, attempting to address the shortcomings of SwiGLU, and analyzes its applicability for training from perspectives such as differentiability.

Critical strengths

1.The paper offers innovations and improvements from the perspectives of pretraining and framework design, which I consider highly contributive. 2.In addition, the insights presented in the paper are reasonable. SwiGLU does indeed have certain limitations. 3.The paper proves that the positive gating factor of PowLU is bounded.

Critical weaknesses
  1. I think the authors have made considerable efforts to conduct experiments demonstrating the effectiveness of their architecture; however, some figures are too obscure for me to understand easily.

  2. The essence of the paper is a gated activation, and I feel that there are currently quite a few works of this kind. The authors need to clarify why their architecture is superior and how it compares with other methods.

3.I have some concerns about the part of the paper that analyzes monotonicity. I feel that analyzing monotonicity is not like analyzing derivative properties—one cannot treat x as a scalar.

  1. The proof in Appendix B.1 (4)seems to have a minor issue, but it is a careless mistake and does not affect the key conclusions.
Questions for authors

To the best of your knowledge, has the architecture you proposed been applied in practice? Many people have previously said that gated methods cannot scale to very large models, and I am not sure whether the architecture you propose can scale. I understand that it is difficult for you to verify this experimentally, so could you provide a brief analysis?

During post-training and actual deployment, can the model's activation be switched back to SwiGLU?

2 comments
AuthorsOpenReview ↗

Response to Reviewer CqRh [Part 1/2]

We are immensely grateful for your effort in reviewing our work, as well as for your constructive suggestions and insightful questions. In the following, we will address each of your suggestions and concerns point-by-point.

Weaknesses

I think the authors have made considerable efforts to conduct experiments demonstrating the effectiveness of their architecture; however, some figures are too obscure for me to understand easily.

We revised the captions of Figures 2, 4, 5, 6, and 9 in the main text to make the plots easier to read. In Figure 2, the horizontal axis indexes layers, and the red, purple, and orange bands show the min–max, P1–P99, and P25–P75 ranges, respectively. Figures 5 and 6 show where large values occur: columns represent tokens, rows represent channels, and red and blue indicate positive and negative values. In Figure 4, the orange (SwiGLU), green (SwiGLU-Clip), and red (PowLU) curves show FP8 runs continued from the same checkpoint. The blue BF16 SwiGLU curve serves as a reference. After an initial transient, PowLU stays near 1.32 for the rest of its plotted segment. Figure 9 shows training loss for PowLU and two ablated variants, with an inset that zooms in on the later steps. We also revised the caption of Figure 10 in Appendix D.1 to explain that it uses the same axes and percentile bands as Figure 2 to show the corresponding distributions for shared experts.

The essence of the paper is a gated activation, and I feel that there are currently quite a few works of this kind. The authors need to clarify why their architecture is superior and how it compares with other methods.

PowLU retains the two-branch GLU structure and targets outliers in FP8 pretraining by bounding how much the positive gate can amplify the linear branch. It uses a rational power function for the positive gate. SwiGLU-Clip uses hard clipping, while SiTU-GLU applies smooth caps to both branches.

In the same-checkpoint FP8 continuation (Figure 4), SwiGLU spikes first and SwiGLU-Clip spikes later. After an initial transient, PowLU stays near a loss of 1.32 for the rest of its plotted segment. In from-scratch pretraining (Table 2), PowLU scores higher than both baselines on 13 of 17 benchmarks at 7.9B and higher than SwiGLU on 12 of 17 at 124B. These results support improved stability in the tested FP8 continuation and competitive downstream performance at both scales. Our comparisons with SiTU-GLU and PolyNorm [1] are qualitative.

I have some concerns about the part of the paper that analyzes monotonicity. I feel that analyzing monotonicity is not like analyzing derivative properties—one cannot treat x as a scalar.

For 0<m<100<m<10, the PowLU output at each coordinate increases strictly with the common scaling factor of its two positive projected inputs, including when x1≠x2x_1\ne x_2. Writing the output as F(x1,x2)=x1qm(x2)F(x_1,x_2)=x_1q_m(x_2) gives F(sx1,sx2)=(x1/x2)fm(sx2)F(sx_1,sx_2)=(x_1/x_2)f_m(sx_2) for s>0s>0, so the unary proof applies directly. With x1>0x_1>0 fixed, increasing x2x_2 alone can decrease the output at m=3m=3, since q3(4)≈3.93q_3(4)\approx3.93 and q3(x2)→1q_3(x_2)\to1 as x2→∞x_2\to\infty. We revised the Introduction and Section 3.2 to state the scope of the monotonicity result. Appendix B.1 now derives the two-input result and gives the fixed-x1x_1 example.

The proof in Appendix B.1 (4)seems to have a minor issue, but it is a careless mistake and does not affect the key conclusions.

We made the chain-rule step explicit in Eq. (4) of Appendix B.1. After substituting t=xt=\sqrt{x}, differentiating the exponent contributes the term −mtln⁡t-mt\ln t to the numerator of the first fraction. The resulting expression yields strict monotonicity of the positive unary branch for 0<m<100<m<10.

AuthorsOpenReview ↗

Response to Reviewer CqRh [Part 2/2]

Questions

To the best of your knowledge, has the architecture you proposed been applied in practice? Many people have previously said that gated methods cannot scale to very large models, and I am not sure whether the architecture you propose can scale. I understand that it is difficult for you to verify this experimentally, so could you provide a brief analysis?

Yes. We used PowLU to pretrain a forthcoming LLM with trillion-parameter-scale model, and training remained stable throughout, with no loss spikes. PowLU changes the elementwise gate while preserving the GLU projections and communication pattern. Its bounded positive gate limits amplification from large gate inputs. In our FP8 experiments, we also trained 7.9B and 124B MoE models from scratch for 600B and 800B tokens, respectively (Table 2). Our fused Triton implementation added about 1% end-to-end FP8 training overhead relative to fused SwiGLU (Section 4.1 and Appendix E).

During post-training and actual deployment, can the model's activation be switched back to SwiGLU?

Yes, a PowLU checkpoint can be switched back to SwiGLU. The switch replaces the PowLU gate with SiLU while retaining the learned projections. Before deploying the switched model, continued training on additional tokens is needed to recover training loss and downstream benchmark performance.

References

[1] Zhijian Zhuo, Ya Wang, Yutao Zeng, Xiaoqing Li, Xun Zhou, and Jinwen Ma. Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models. ICLR, 2025.

Reviewer_85ucOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PowLU, a gated activation that replaces the positive SiLU gate with a rational power transformation while retaining the conventional negative branch. Its positive gate is bounded independently of the gate input, reducing one source of amplification in SwiGLU. The paper analyzes the unary form's continuity, differentiability, and monotonicity, and proves a bound for the practical positive gate. Experiments include scaling comparisons with 26M to 368M active parameters and Ling models with 7.9B and 124B total parameters, corresponding to 1.3B and 5.1B active parameters. Reported benchmark performance is competitive with SwiGLU. A separate FP8 experiment switches activation functions during training and shows fewer visible loss spikes under PowLU, alongside reduced activation and gradient outliers.

Critical strengths

The method controls a concrete amplification mechanism and is evaluated at meaningful scale. The positive gate bound directly distinguishes PowLU from SwiGLU: with the default coefficient, amplification relative to the other projected input is bounded by approximately 5.32. The reported activation and gradient distributions are consistent with reduced outliers, while the 7.9B and 124B total parameter comparisons show competitive downstream capability. This combination of an interpretable design property and large model feasibility makes the idea worth further investigation.

Critical weaknesses
  1. The main stability experiment replaces SwiGLU during training rather than evaluating PowLU throughout pretraining under a matched FP8 setup. The short continuation results lack repeated runs and failure statistics. It is also unclear whether the larger models use PowLU from initialization and whether their training demonstrates improved stability. More controlled full-training experiments are needed to support the central claim.

  2. Limited theoretical and practical validation. The theoretical analysis focuses on a simplified unary function that does not fully reflect the implemented gated activation. The bounded positive gate alone does not guarantee overall gradient stability. Additionally, the paper lacks measurements of the computational overhead and numerical behavior of PowLU in actual FP8 training, as well as comparisons with alternative stabilization methods. These gaps leave its practical advantages insufficiently established.

If the authors can address these concerns, I would be willing to reconsider and potentially raise my overall score.

Questions for authors
  1. Were the models in Table 2 trained with their reported activation from initialization or obtained by switching from SwiGLU?

  2. Under an otherwise identical training protocol, what are the loss spike, failure, and restart statistics over a sufficiently long continuation or complete run? Do the gains persist across seeds and against tuned clipped SwiGLU or the smooth capped gate discussed in Section 2.2?

  3. How is the practical gate's derivative discontinuity at zero handled, and what are the observed numerical and throughput costs of its power operation in the actual training implementation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 85uc [Part 1/2]

We thank the reviewer for the suggestions. In response to the issues raised, we provide the following clarifications.

Weaknesses

The main stability experiment replaces SwiGLU during training rather than evaluating PowLU throughout pretraining under a matched FP8 setup. The short continuation results lack repeated runs and failure statistics. It is also unclear whether the larger models use PowLU from initialization and whether their training demonstrates improved stability. More controlled full-training experiments are needed to support the central claim.

Additionally, the paper lacks measurements of the computational overhead and numerical behavior of PowLU in actual FP8 training, as well as comparisons with alternative stabilization methods. These gaps leave its practical advantages insufficiently established.

All models in Table 2 are trained from scratch, and were trained with the FP8 recipe to the target token counts. They were not obtained by switching from SwiGLU. Figures 2 and 10 present the FP8 numerical distributions of the models trained from scratch. The results show that PowLU substantially compresses the dynamic ranges of activations and gradients, thereby reducing the risk of numerical overflow.

The activation function switching experiment is used only in Figure 4. This experiment is motivated by a real loss spike observed in another 124B SwiGLU baseline at around 5T tokens. Starting from the same checkpoint, we compare SwiGLU, SwiGLU-Clip, and PowLU while keeping all other training configurations identical; all three experiments use FP8 training. The results show that SwiGLU exhibits a pronounced loss spike and that SwiGLU-Clip only postpones the spike, whereas PowLU remains stable throughout continued training. Therefore, this experiment validates PowLU’s ability to mitigate loss spikes in real unstable training scenarios.

Limited theoretical and practical validation. The theoretical analysis focuses on a simplified unary function that does not fully reflect the implemented gated activation. The bounded positive gate alone does not guarantee overall gradient stability.

For the implemented two-input PowLU, we bound both input derivatives on the positive branch at the training setting m=3m=3. At each coordinate, the output is F(x1,x2)=x1q3(x2)F(x_1,x_2)=x_1q_3(x_2) for x2>0x_2>0, where x1x_1 is the linear input, x2x_2 is the gate input, and q3(x)=x3/(1+x)σ(x)q_3(x)=x^{3/(1+\sqrt{x})}\sigma(x).

Differentiating with respect to each input gives ∂F/∂x1=q3(x2)\partial F/\partial x_1=q_3(x_2) and ∂F/∂x2=x1q3′(x2)\partial F/\partial x_2=x_1q_3'(x_2). The first derivative is bounded by C3≈5.32C_3\approx5.32 because q3(x)≤C3q_3(x)\le C_3 for all x>0x>0. For the second derivative, differentiating the gate gives

\frac{3}{x(1+\sqrt{x})} -\frac{3\log x}{2\sqrt{x}(1+\sqrt{x})^2} +1-\sigma(x) \right].$$ This derivative is continuous for $x>0$ and tends to zero at both endpoints, so $|q_3'(x)|$ is bounded by a finite constant $K_3$. Thus $|\partial F/\partial x_2|\le K_3|x_1|$. We clarified in Appendix B.2 that these bounds control the local activation Jacobian. Gradients through the model also depend on the upstream gradients and learned projections. In the 7.9B FP8 runs trained from scratch, PowLU showed tighter activation and gradient ranges than SwiGLU at 400B tokens in both routing and shared experts (Figures 2 and 10).
AuthorsOpenReview ↗

Response to Reviewer 85uc [Part 2/2]

Questions

Were the models in Table 2 trained with their reported activation from initialization or obtained by switching from SwiGLU?

All models in Table 2 are trained from scratch.

Under an otherwise identical training protocol, what are the loss spike, failure, and restart statistics over a sufficiently long continuation or complete run? Do the gains persist across seeds and against tuned clipped SwiGLU or the smooth capped gate discussed in Section 2.2?

In pre-training, loss spikes usually do not appear in small LLMs or early in the training of larger models. However, no loss spike was observed in the 7.9B experiments. During the 124B training up to 800B tokens, only the SwiGLU baseline experienced one loss spike, from which it recovered automatically.

How is the practical gate's derivative discontinuity at zero handled, and what are the observed numerical and throughput costs of its power operation in the actual training implementation?

The current implementation of PowLU uses gate > 0, so zero values deterministically enter the SiLU branch, and the gradient value used in the backward pass is 0.5. Although the derivative is discontinuous at this point, we did not observe any numerical or training anomalies caused by this behavior in the large-scale FP8 training runs described above.

In terms of performance, PowLU introduces a small number of additional scalar operations compared to SwiGLU, but this computation accounts for only a small fraction of the overall training flops. We have implemented Triton fused kernels covering both dense and MoE forward and backward paths, fusing activation recomputation, input gradients, and routing-weight gradients into a single kernel while reusing intermediate results in registers. This reduces kernel launches, temporary tensors, and memory accesses. Compared to the original PowLU implementation, the fused kernels achieve approximately a 2–3× operator-level speedup; in end-to-end FP8 training, the optimized PowLU incurs only about a 1% performance overhead relative to fused SwiGLU.

While the clarifications and additional analyses are helpful, the evidence for improved training stability remains limited. In particular, observing one recoverable loss spike in SwiGLU and none in PowLU is insufficient to establish a robust stability advantage.

Reviewer_b4c1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce PowGLU, a new activation function that aims to improve stability in the learning process by avoiding the quasi-quadratic scaling of the unary part of the main baseline SwiGLU and SwiGLU-clip. The activation seems to provide better metrics and more stable learning.

Critical strengths

The authors deliver what they promised, an activation function that is more stable and seems to provide comparable or even better results on many different scenarios.

Critical weaknesses
  • With the large amount of activation functions proposed in the literature it is hard to establish a good baseline.
  • The computational cost of the activation is larger that other simpler potential strategies, though this weakness is compensated by other properties which establish a standard practical trade-off.
Questions for authors

The definition of new activation functions is a difficult subfield to work on. One of the main reasons is that the actual reasons for adopting one option or another are heavily influenced by fashion and “novelty” and also limited but promising performance results. At the end, usually giving rise to marginal improvements on a small set of experiments. This sets this article in difficult starting point. I applaud the authors for trying to compete in this arena. The classical trade-off of computational simplicity versus smoothness (which many times lead to increased stability) is highly contextual as models require more and more speedups that sometimes work in detriment of the good properties of the activation. For example, in scenarios where training is performed in quantized computation the continuous activation function becomes a step-wise function in practice, destroying many of the of the benefits the activation could theoretically be endowed. After these introductory words, let me as some questions regarding the proposal: 1- It looks like the authors are specifically worried on the positive side of the activation. Without trying to propose anything new and just going back to the basics, this could be easily achieved by some version of leaky RELU (and it is easy to extend to the gating strategy and also in case the negative valley is something to be preserved it could be also be done using a piecewise strategy). In this sense, I would expect not only to compare to SwiGLU but also to the most obvious implementations that solve the problem. So, my question is, wouldn’t a gated leaky Relu be endowed with similar properties to the ones proposed? In what sense the proposal effectively beats this simple baseline? 2- Setting parameter m seems critical as shown in the ablation studies. The authors report m=2.5 or 3. To what extend the authors believe that this may be the default parameter. In other words, if another group wanted to replicate the results or propose alternative activations build on this proposal, would setting m to these default parameters be satisfying for the authors compare any other proposal? 3- A minor note regarding swiGLU-clip, it also appears to spike in the experiment of figure 4. May the authors provide some intuition of why this happens despite the clipping? 4- The arguments in this article brings me to my mind Polynomial Composition of Activations (PolyNorm) technique that uses a polynomial on the unary side and also seemed to consistently beat swiglu. Though different to the proposal, it would be interesting to know the opinion of the authors with respect of their proposal and this technique. 5- I usually like to reflect on less technical issues and some community practice. Just out of curiosity, the authors to what extend they believe the commutity should switch their practice in favour of the proposal after seeing for example that PolyNorm and many other activation fucntions proposed that consistently beat swiGLU are not adopted.

3 comments
AuthorsOpenReview ↗

Response to Reviewer b4c1 [Part 1/2]

We sincerely thank you for the time and effort you have invested in reviewing our paper. Your comments and questions are very insightful and highly valuable for us to enhance the paper. Your concerns and questions are addressed point-by-point in the following.

Weaknesses

With the large amount of activation functions proposed in the literature it is hard to establish a good baseline.

We chose SwiGLU and SwiGLU-Clip as baselines because both are used in recent LLMs. SwiGLU is widely adopted, including in Llama 3 and Qwen3, and clipped variants are used in gpt-oss and DeepSeek-V4. These baselines let us compare PowLU with standard SwiGLU and with a practical way to control activation outliers.

The computational cost of the activation is larger that other simpler potential strategies, though this weakness is compensated by other properties which establish a standard practical trade-off.

With our fused Triton implementation, PowLU incurs about 1% end-to-end FP8 training overhead relative to fused SwiGLU. The implementation covers both forward and backward passes for dense and MoE layers. In the backward pass, each kernel fuses activation recomputation with input-gradient computation and, for MoE, routing-weight-gradient computation. Reusing intermediates in registers reduces kernel launches, temporary tensors, and GPU memory accesses. These changes make the PowLU operator approximately 2–3× faster than its original implementation. We added the end-to-end result to Section 4.1 (Training Settings). Appendix E now explains how the fused kernels reduce overhead and reports both performance comparisons.

AuthorsOpenReview ↗

Response to Reviewer b4c1 [Part 2/2]

Questions

It looks like the authors are specifically worried on the positive side of the activation. Without trying to propose anything new and just going back to the basics, this could be easily achieved by some version of leaky RELU (and it is easy to extend to the gating strategy and also in case the negative valley is something to be preserved it could be also be done using a piecewise strategy). In this sense, I would expect not only to compare to SwiGLU but also to the most obvious implementations that solve the problem. So, my question is, wouldn’t a gated leaky Relu be endowed with similar properties to the ones proposed? In what sense the proposal effectively beats this simple baseline?

PowLU bounds positive gate amplification, which remains unbounded with a standard gated Leaky ReLU. For our empirical comparison, we chose GeLU because it grows approximately linearly for large positive inputs, like Leaky ReLU, and also has the negative-side valley you suggested preserving. On the MoE model with 47M activated parameters, GeLU yielded a loss 0.015 higher than PowLU's, so we retained PowLU for subsequent experiments.

Setting parameter m seems critical as shown in the ablation studies. The authors report m=2.5 or 3. To what extend the authors believe that this may be the default parameter. In other words, if another group wanted to replicate the results or propose alternative activations build on this proposal, would setting m to these default parameters be satisfying for the authors compare any other proposal?

We recommend m=3m=3 for reproducing our main experiments and for using PowLU as a baseline when comparing new activation variants. In our MoE sweep, m=2.5m=2.5 gave the lowest loss at 47M and 92M activated parameters, and m=3m=3 gave the lowest loss at 199M and 368M. Across all five tested scales, the reported loss with m=3m=3 was within 0.004 of the lowest reported loss at each scale (Section 4.4.1 and Appendix D.2).

A minor note regarding swiGLU-clip, it also appears to spike in the experiment of figure 4. May the authors provide some intuition of why this happens despite the clipping?

SwiGLU-Clip can delay loss spikes by limiting activation outliers, and large gradients can still build up through channels that stay within the clipping range. Inside that range, the clamp has a derivative of one, so it passes the incoming gradient unchanged. During backpropagation through the preceding linear projection, contributions from many such channels can add up, providing one possible route to an unstable update. In Figure 4, SwiGLU-Clip spikes later than SwiGLU, with a pronounced spike around step 77,000.

The arguments in this article brings me to my mind Polynomial Composition of Activations (PolyNorm) technique that uses a polynomial on the unary side and also seemed to consistently beat swiglu. Though different to the proposal, it would be interesting to know the opinion of the authors with respect of their proposal and this technique.

We see PolyNorm’s main contribution as making higher-order features practical by normalizing each power before combining them. Its reported loss gains over SwiGLU in both dense and MoE models make it a promising alternative [1]. PowLU takes a complementary approach to controlling activation magnitudes by explicitly bounding how much a positive gate input can amplify the other branch, which directly targets outliers in FP8 pretraining.

I usually like to reflect on less technical issues and some community practice. Just out of curiosity, the authors to what extend they believe the commutity should switch their practice in favour of the proposal after seeing for example that PolyNorm and many other activation fucntions proposed that consistently beat swiGLU are not adopted.

We would try PowLU first in large-scale FP8 MoE pretraining where loss spikes are a concern. PolyNorm’s reported gains over SwiGLU [1] illustrate the value of comparing model quality. For FP8 adoption, we would weigh quality alongside training stability and throughput. In our matched FP8 continuation, PowLU remained spike-free over the window shown in Figure 4. With fused Triton kernels, it added about 1% end-to-end FP8 training overhead relative to fused SwiGLU.

References

[1] Zhijian Zhuo, Ya Wang, Yutao Zeng, Xiaoqing Li, Xun Zhou, and Jinwen Ma. Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models. ICLR, 2025.

My questions are addressed. I ratify the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.