- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new activation function, PowLU (Power Linear Unit), for the FFN in LLM architectures, attempting to address the shortcomings of SwiGLU, and analyzes its applicability for training from perspectives such as differentiability.
Critical strengths
1.The paper offers innovations and improvements from the perspectives of pretraining and framework design, which I consider highly contributive. 2.In addition, the insights presented in the paper are reasonable. SwiGLU does indeed have certain limitations. 3.The paper proves that the positive gating factor of PowLU is bounded.
Critical weaknesses
-
I think the authors have made considerable efforts to conduct experiments demonstrating the effectiveness of their architecture; however, some figures are too obscure for me to understand easily.
-
The essence of the paper is a gated activation, and I feel that there are currently quite a few works of this kind. The authors need to clarify why their architecture is superior and how it compares with other methods.
3.I have some concerns about the part of the paper that analyzes monotonicity. I feel that analyzing monotonicity is not like analyzing derivative properties—one cannot treat x as a scalar.
- The proof in Appendix B.1 (4)seems to have a minor issue, but it is a careless mistake and does not affect the key conclusions.
Questions for authors
To the best of your knowledge, has the architecture you proposed been applied in practice? Many people have previously said that gated methods cannot scale to very large models, and I am not sure whether the architecture you propose can scale. I understand that it is difficult for you to verify this experimentally, so could you provide a brief analysis?
During post-training and actual deployment, can the model's activation be switched back to SwiGLU?
2 comments
Response to Reviewer CqRh [Part 2/2]
Questions
To the best of your knowledge, has the architecture you proposed been applied in practice? Many people have previously said that gated methods cannot scale to very large models, and I am not sure whether the architecture you propose can scale. I understand that it is difficult for you to verify this experimentally, so could you provide a brief analysis?
Yes. We used PowLU to pretrain a forthcoming LLM with trillion-parameter-scale model, and training remained stable throughout, with no loss spikes. PowLU changes the elementwise gate while preserving the GLU projections and communication pattern. Its bounded positive gate limits amplification from large gate inputs. In our FP8 experiments, we also trained 7.9B and 124B MoE models from scratch for 600B and 800B tokens, respectively (Table 2). Our fused Triton implementation added about 1% end-to-end FP8 training overhead relative to fused SwiGLU (Section 4.1 and Appendix E).
During post-training and actual deployment, can the model's activation be switched back to SwiGLU?
Yes, a PowLU checkpoint can be switched back to SwiGLU. The switch replaces the PowLU gate with SiLU while retaining the learned projections. Before deploying the switched model, continued training on additional tokens is needed to recover training loss and downstream benchmark performance.
References
[1] Zhijian Zhuo, Ya Wang, Yutao Zeng, Xiaoqing Li, Xun Zhou, and Jinwen Ma. Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models. ICLR, 2025.
Response to Reviewer CqRh [Part 1/2]
We are immensely grateful for your effort in reviewing our work, as well as for your constructive suggestions and insightful questions. In the following, we will address each of your suggestions and concerns point-by-point.
Weaknesses
We revised the captions of Figures 2, 4, 5, 6, and 9 in the main text to make the plots easier to read. In Figure 2, the horizontal axis indexes layers, and the red, purple, and orange bands show the min–max, P1–P99, and P25–P75 ranges, respectively. Figures 5 and 6 show where large values occur: columns represent tokens, rows represent channels, and red and blue indicate positive and negative values. In Figure 4, the orange (SwiGLU), green (SwiGLU-Clip), and red (PowLU) curves show FP8 runs continued from the same checkpoint. The blue BF16 SwiGLU curve serves as a reference. After an initial transient, PowLU stays near 1.32 for the rest of its plotted segment. Figure 9 shows training loss for PowLU and two ablated variants, with an inset that zooms in on the later steps. We also revised the caption of Figure 10 in Appendix D.1 to explain that it uses the same axes and percentile bands as Figure 2 to show the corresponding distributions for shared experts.
PowLU retains the two-branch GLU structure and targets outliers in FP8 pretraining by bounding how much the positive gate can amplify the linear branch. It uses a rational power function for the positive gate. SwiGLU-Clip uses hard clipping, while SiTU-GLU applies smooth caps to both branches.
In the same-checkpoint FP8 continuation (Figure 4), SwiGLU spikes first and SwiGLU-Clip spikes later. After an initial transient, PowLU stays near a loss of 1.32 for the rest of its plotted segment. In from-scratch pretraining (Table 2), PowLU scores higher than both baselines on 13 of 17 benchmarks at 7.9B and higher than SwiGLU on 12 of 17 at 124B. These results support improved stability in the tested FP8 continuation and competitive downstream performance at both scales. Our comparisons with SiTU-GLU and PolyNorm [1] are qualitative.
For 0<m<10, the PowLU output at each coordinate increases strictly with the common scaling factor of its two positive projected inputs, including when x1=x2. Writing the output as F(x1,x2)=x1qm(x2) gives F(sx1,sx2)=(x1/x2)fm(sx2) for s>0, so the unary proof applies directly. With x1>0 fixed, increasing x2 alone can decrease the output at m=3, since q3(4)≈3.93 and q3(x2)→1 as x2→∞. We revised the Introduction and Section 3.2 to state the scope of the monotonicity result. Appendix B.1 now derives the two-input result and gives the fixed-x1 example.
We made the chain-rule step explicit in Eq. (4) of Appendix B.1. After substituting t=x, differentiating the exponent contributes the term −mtlnt to the numerator of the first fraction. The resulting expression yields strict monotonicity of the positive unary branch for 0<m<10.