- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes SwiGNLU, an architecture-level design to improve SwiGLU by applying non-linearity to the value branch. Experimental results show that this alternative achieves the same generation quality (measured in 'bits per byte') with a smaller training token budget.
Critical strengths
None.
Critical weaknesses
- Incoherent presentation and missing explanations. The unit 'mbpb' appears frequently in the manuscript, but the authors do not clearly define it. In addition, the paper also includes many ambiguous terms, such as 'zero-shot battery' (Line 24 in the Abstract).
- Unclear motivation for the design choice. The manuscript proposes a variant of SwiGLU but lacks an in-depth discussion and supporting theoretical or empirical evidence for this choice. As a result, SwiGNLU is not necessarily a better alternative to SwiGLU without convincing support or insight.
- Missing critical experimental setting details. The experiments use a series of LLMs scaling from 160M to 1.1 B. However, it is unclear which models these are, as the author does not identify the model family.
Questions for authors
Given these weaknesses, I hope the author can respond to the following questions to address my primary concerns.
- Could you explain the unclear terminology mentioned in Weakness 1 in more detail?
- Is there theoretical or empirical evidence supporting SwiGNLU as a strong alternative to SwiGLU?
- What models are used throughout the experiments?
We thank the reviewer for the careful reading and respond by item.
1(a) GTU precedent. Dauphin et al. attribute GTU's deficit to the downscaling factor tanh′ on the content path. SiLU does not saturate for positive inputs (σ′(z) → 1), so that argument does not transfer to SiLU directly. Our SwiGLU-dualact arm is the SiLU analog of GTU, a nonlinear full-width value branch, and it improves on SwiGLU in all three factorial conditions (0.45–2.22 mbpb, Table 5) rather than trailing it. We should have made this connection explicit in Section 1.
1(b) Placement. The distinction we claim is replacing the value projection at a fixed budget, and the 2×2 isolates exactly that placement. At matched parameters, the same bottleneck without its interior activation captures 36–52% of the gain, while the activation inside it supplies the remainder super-additively. NOBLE adds a branch at a parameter premium and requires its activation in order to exist, so it admits no matched linear null of this kind. We have exploratory runs of faithful NOBLE (+6–7% parameters) and MoA at 160M and 599M. They used AdamW rather than our Muon protocol and are mostly single-seed; NOBLE matches or exceeds SwiGNLU at 160M and trails it at 599M. Because optimizer, budget and replication all differ, we should have reported these runs in Appendix D as descriptive but we make no ranking claim.
1(c) A principled account. Separation. SwiGNLU represents f(x) = Σⱼ wⱼ σ(aⱼ·x) σ(u·x), in which h gates read one shared nonlinear feature, exactly at v_h = 1. A SwiGLU channel is a gate times a linear form, so it must approximate each two-dimensional product σ(aⱼ·x)σ(u·x) separately. Conversely, SwiGLU exactly represents gated linear values spanning more than v_h directions, which SwiGNLU can only approximate. The modification should therefore help when shared nonlinear value features matter more than value rank above v_h.
The added nonlinearity is used. A linear fit from x to the trained value branch U₂σ(U₁x) leaves on average 12% of its variance unexplained at 160M and 6% at 599M, peaking at 22–24% in layers 2–3, whereas SwiGLU's value branch is linear by construction. These probes come from earlier-campaign checkpoints at v_h = 512, and we will repeat them on the paper's cells. Interaction. The linear factorization U₂U₁ is unchanged under U₂ → U₂A, U₁ → A⁻¹U₁ for any invertible A, so v_h² of its parameters are non-functional. The interior SiLU removes that redundancy, whereas SiLU on the full-width value removes only h per-channel scalings. This predicts that the activation is worth far more inside the bottleneck than at full width, and Table 5 gives 3.65–5.37 mbpb for adding it inside the bottleneck against 0.45–2.22 at full width. The same count bounds the bottleneck-only arm's functional dimension about v_h² below SwiGLU's, so that arm's own gain cannot come from expressivity. We therefore treat that component as an optimization effect, as Section 7 suggests. The full gain also survives pure AdamW in all nine pairs (3.4–5.5 mbpb across three learning rates, about half the Muon margin), so Muon amplifies it without being necessary.
2(a) Configuration alignment. We will remeasure step time at the exact geometry, batch size and execution mode of each learning-curve cell, and we will tabulate the configuration behind every efficiency number. As a bound in the meantime, the largest training-step overheads we measured across batch sizes are +4.3%, +2.1% and +2.9% at 305M, 599M and 762M. Combining those worst cases with the token multiples still gives 10.4%, 11.2% and 13.1% less wall-clock to the control's final quality.
2(b) Conditionality of the crossings. Per seed, SwiGNLU reaches the control's final quality with 13.6–15.1% fewer tokens at 305M (three seeds), 12.8–13.5% at 599M (three seeds) and 13.3–16.7% at 762M (four seeds), and we will report these ranges. The best-learning-rate reruns at 305M and 762M recorded full curves, so we will recompute crossings on them and confirm them on the 256-window metric where saved checkpoints permit. The headline figure measures early stopping along the full-length schedule, and we will label it that way. Separately, the Longmino token ladder trains a separate run for each budget from 2B to 32B tokens, and the gain holds at every budget (10.6–12.5 mbpb), so the advantage does not depend on stopping a longer schedule early.
2(c) Alternative FFNs. We agree the comparison is narrow, as Limitation 8 states. It does cover two optimizer routings with three seeds each, and SwiGNLU leads every variant under both routings by at least 2.6 mbpb (squared ReLU under the qkv routing, where SwiGNLU wins all three pairs). GEGLU and ReGLU never beat SwiGLU in our tests.