acceptodds
Withdrawn from ICLR 2027

SwiGNLU: Nonlinear Value Bottlenecks for Parameter-Matched Gated FFNs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The feed-forward network holds most non-embedding parameters in a modern decoder, yet in SwiGLU the branch that carries its content is affine: a nonlinear gate multiplies a linear value projection. We ask whether the same feed-forward budget is better spent giving that value branch a learned nonlinear feature dictionary. SwiGNLU replaces with , computing ; choosing makes the replacement no larger than SwiGLU and 0.03–1.0% cheaper in matmul FLOPs. Under one shared Muon/AdamW-on-qkv protocol, all twenty-eight completed paired comparisons favor SwiGNLU: eighteen at matched budgets, spanning 160M to 1.1B parameters on FineWeb and on independently trained Longmino models, with mean gains of 7.1–9.6 millibits per byte per cell; and ten that vary the token budget from to the Chinchilla-optimal ratio. The gain converts to 13–16% fewer tokens to reach the control's final quality, survives training to Chinchilla undiminished, transfers to held-out corpora with a larger relative margin in seven of eight cells, and moves a seven-task zero-shot battery by +0.59 points (16 of 17 pairs). Measured end to end on identical hardware, training step time matches within 1.5%, so equal quality arrives 11.8–15.2% sooner in wall-clock; the serial value path costs 2.7–5.1% per decoded token at batch 1 and shrinks with batch. Both families share the same best measured learning rate at three of four rungs, so SwiGNLU retrofits at the baseline's own tuned LR. A parameter-matched shows a consistently positive interaction between the bottleneck and its interior nonlinearity, accounting for 34–37% of the combined improvement under the qkv routing. Muon amplifies the gain without creating it: under pure AdamW at 305M, SwiGNLU wins all nine pairs at about half the margin. At 160M it also beats squared ReLU, GEGLU, and ReGLU at matched parameters. We release modules, recipe, paired run-level evidence, and complete curves.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Neural Networks, Feed Forward Networks, Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7nyKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SwiGNLU, an architecture-level design to improve SwiGLU by applying non-linearity to the value branch. Experimental results show that this alternative achieves the same generation quality (measured in 'bits per byte') with a smaller training token budget.

Critical strengths

None.

Critical weaknesses
  1. Incoherent presentation and missing explanations. The unit 'mbpb' appears frequently in the manuscript, but the authors do not clearly define it. In addition, the paper also includes many ambiguous terms, such as 'zero-shot battery' (Line 24 in the Abstract).
  2. Unclear motivation for the design choice. The manuscript proposes a variant of SwiGLU but lacks an in-depth discussion and supporting theoretical or empirical evidence for this choice. As a result, SwiGNLU is not necessarily a better alternative to SwiGLU without convincing support or insight.
  3. Missing critical experimental setting details. The experiments use a series of LLMs scaling from 160M to 1.1 B. However, it is unclear which models these are, as the author does not identify the model family.
Questions for authors

Given these weaknesses, I hope the author can respond to the following questions to address my primary concerns.

  1. Could you explain the unclear terminology mentioned in Weakness 1 in more detail?
  2. Is there theoretical or empirical evidence supporting SwiGNLU as a strong alternative to SwiGLU?
  3. What models are used throughout the experiments?
Reviewer_KhroOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SwiGNLU, which replaces SwiGLU’s linear value projection with a nonlinear bottleneck, U2SiLU⁡(U1x)U_2\operatorname{SiLU}(U_1x), while retaining the nonlinear gate. A constraint on the bottleneck width approximately preserves the parameter and matrix-multiplication budgets. The method is evaluated through paired language-model pretraining experiments spanning 160M–1.1B parameters on FineWeb and Longmino. The reported comparisons consistently favor SwiGNLU. Additional experiments examine learning rates, optimizers, training budgets, downstream tasks, and hardware costs. A four-arm factorial study reports a positive interaction between the bottleneck and its interior nonlinearity. Learning-curve crossings suggest reduced token requirements, accompanied by nearly unchanged training-step latency and a modest decoding penalty.

Critical strengths

1. Consistent empirical improvements under multiple controlled settings. Paired comparisons across scales, corpora, and token budgets support the existence of an advantage over the evaluated SwiGLU baseline. Learning-rate checks, extended training, and a pure-AdamW experiment address several plausible alternative explanations. Reporting individual paired differences also makes the results easier to assess.

2. Explicit investigation of architectural components and practical costs. The four-arm comparison separates a full-width nonlinear value branch, a linear bottleneck, and their combination. The paper also measures training, prefill, decoding, and memory costs, appropriately acknowledging that matched matrix FLOPs do not guarantee matched latency.

Critical weaknesses

1. The architectural novelty is limited, and the paper does not sufficiently explain why this particular return to a nonlinear value branch is advantageous.

  • (a) Nonlinearity in both branches has clear historical precedent.
    The GTU architecture discussed in [1] already computes a nonlinear content branch multiplied by a nonlinear gate: tanh⁡(Wx+b)⊙sigmoid⁡(Vx+c).\tanh(Wx+b)\odot\operatorname{sigmoid}(Vx+c). Moreover, that work motivates GLU’s linear content path through improved gradient propagation and reports advantages over GTU in its experimental setting. Consequently, identifying SwiGLU’s value branch as affine does not, by itself, establish an architectural deficiency or a novel motivation for making it nonlinear. The paper should explain more precisely what changes in the present setting make the nonlinear value path beneficial.

  • (b) The additional bottleneck is also an established architectural building block.
    Compression–nonlinearity–expansion modules have long been used in bottleneck adapters [2], and NOBLE [3] more recently investigates nonlinear low-rank branches during pretraining. SwiGNLU’s specific distinction is therefore the replacement of the gated FFN’s value projection under a fixed budget, rather than either nonlinear content processing or nonlinear bottlenecks themselves. This is a relatively incremental combination. What evidence establishes the benefit of this particular placement and replacement relative to closely related nonlinear bottleneck designs under comparable budgets and training protocols?

  • (c) The formal argument establishes budget feasibility, but not an advantage of the modified function or optimization dynamics.
    The parameter inequality establishes that the replacement can fit within SwiGLU’s budget. Equation (1) characterizes the resulting feature interactions, but neither establishes superior expressivity, optimization, or generalization. The positive factorial interaction is useful empirical evidence, yet its interpretation remains intertwined with initialization, optimizer behavior, and width reallocation. In particular, the paper reports a substantially smaller initial value-branch RMS and acknowledges that Muon updates the factor matrices separately. Given the limited architectural novelty, a formal analysis under clearly stated assumptions would materially strengthen the contribution—for example, identifying functions represented more efficiently by the proposed structure, or conditions under which useful gradient-propagation properties are retained. Can the authors provide such an analysis and connect its predictions to the experiments? I am not asking for universal dominance over SwiGLU, but for a principled account of when and why the modification should help.

2. The practical efficiency claims need a more consistent comparison against tuned baselines.

  • (a) Learning-curve and timing configurations are not clearly aligned.
    Section 6 combines token-to-target ratios with separately measured training-step overheads. However, Table 4 uses production configurations, while the main comparisons include different geometries at 305M and 599M. Timing also uses fused, compiled execution, whereas Appendix E describes the training module as unfused. Please identify the exact geometry, learning rate, execution mode, and batch configuration underlying each efficiency estimate. Does the reported 12–15% wall-clock reduction hold when learning curves and timing measurements use the same configuration?

  • (b) First-crossing savings remain conditional on the chosen training protocol.
    The 13–16% token reduction uses the first crossing of the control population’s mean final loss on a 16-window validation metric. The paper also shows that learning-rate tuning changes some margins. Do comparable savings hold when both families use their best measured settings? Please provide uncertainty estimates and, where checkpoints permit, confirm crossings using the wider validation set. The comparison should also distinguish early stopping along the existing schedule from training with a shortened decay horizon.

  • (c) Comparisons with alternative FFNs have limited tuning coverage.
    The squared-ReLU, GEGLU, and ReGLU comparisons cover one scale and one learning rate. The approximately 2.6 mbpb advantage over squared ReLU is smaller than some optimizer-routing effects reported elsewhere in the paper. Does this ordering survive an equal learning-rate search budget? Stronger comparisons would help determine whether SwiGNLU offers a consequential improvement over established alternatives.

References

[1] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language Modeling with Gated Convolutional Networks. ICML, 2017.

[2] Neil Houlsby et al. Parameter-Efficient Transfer Learning for NLP. ICML, 2019.

[3] Ethan Smith. NOBLE: Accelerating Transformers with Nonlinear Low-Rank Branches. arXiv:2603.06492, 2026.

Questions for authors

see the weaknesses.

1 comment

We thank the reviewer for the careful reading and respond by item.

1(a) GTU precedent. Dauphin et al. attribute GTU's deficit to the downscaling factor tanh′ on the content path. SiLU does not saturate for positive inputs (σ′(z) → 1), so that argument does not transfer to SiLU directly. Our SwiGLU-dualact arm is the SiLU analog of GTU, a nonlinear full-width value branch, and it improves on SwiGLU in all three factorial conditions (0.45–2.22 mbpb, Table 5) rather than trailing it. We should have made this connection explicit in Section 1.

1(b) Placement. The distinction we claim is replacing the value projection at a fixed budget, and the 2×2 isolates exactly that placement. At matched parameters, the same bottleneck without its interior activation captures 36–52% of the gain, while the activation inside it supplies the remainder super-additively. NOBLE adds a branch at a parameter premium and requires its activation in order to exist, so it admits no matched linear null of this kind. We have exploratory runs of faithful NOBLE (+6–7% parameters) and MoA at 160M and 599M. They used AdamW rather than our Muon protocol and are mostly single-seed; NOBLE matches or exceeds SwiGNLU at 160M and trails it at 599M. Because optimizer, budget and replication all differ, we should have reported these runs in Appendix D as descriptive but we make no ranking claim.

1(c) A principled account. Separation. SwiGNLU represents f(x) = Σⱼ wⱼ σ(aⱼ·x) σ(u·x), in which h gates read one shared nonlinear feature, exactly at v_h = 1. A SwiGLU channel is a gate times a linear form, so it must approximate each two-dimensional product σ(aⱼ·x)σ(u·x) separately. Conversely, SwiGLU exactly represents gated linear values spanning more than v_h directions, which SwiGNLU can only approximate. The modification should therefore help when shared nonlinear value features matter more than value rank above v_h.

The added nonlinearity is used. A linear fit from x to the trained value branch U₂σ(U₁x) leaves on average 12% of its variance unexplained at 160M and 6% at 599M, peaking at 22–24% in layers 2–3, whereas SwiGLU's value branch is linear by construction. These probes come from earlier-campaign checkpoints at v_h = 512, and we will repeat them on the paper's cells. Interaction. The linear factorization U₂U₁ is unchanged under U₂ → U₂A, U₁ → A⁻¹U₁ for any invertible A, so v_h² of its parameters are non-functional. The interior SiLU removes that redundancy, whereas SiLU on the full-width value removes only h per-channel scalings. This predicts that the activation is worth far more inside the bottleneck than at full width, and Table 5 gives 3.65–5.37 mbpb for adding it inside the bottleneck against 0.45–2.22 at full width. The same count bounds the bottleneck-only arm's functional dimension about v_h² below SwiGLU's, so that arm's own gain cannot come from expressivity. We therefore treat that component as an optimization effect, as Section 7 suggests. The full gain also survives pure AdamW in all nine pairs (3.4–5.5 mbpb across three learning rates, about half the Muon margin), so Muon amplifies it without being necessary.

2(a) Configuration alignment. We will remeasure step time at the exact geometry, batch size and execution mode of each learning-curve cell, and we will tabulate the configuration behind every efficiency number. As a bound in the meantime, the largest training-step overheads we measured across batch sizes are +4.3%, +2.1% and +2.9% at 305M, 599M and 762M. Combining those worst cases with the token multiples still gives 10.4%, 11.2% and 13.1% less wall-clock to the control's final quality.

2(b) Conditionality of the crossings. Per seed, SwiGNLU reaches the control's final quality with 13.6–15.1% fewer tokens at 305M (three seeds), 12.8–13.5% at 599M (three seeds) and 13.3–16.7% at 762M (four seeds), and we will report these ranges. The best-learning-rate reruns at 305M and 762M recorded full curves, so we will recompute crossings on them and confirm them on the 256-window metric where saved checkpoints permit. The headline figure measures early stopping along the full-length schedule, and we will label it that way. Separately, the Longmino token ladder trains a separate run for each budget from 2B to 32B tokens, and the gain holds at every budget (10.6–12.5 mbpb), so the advantage does not depend on stopping a longer schedule early.

2(c) Alternative FFNs. We agree the comparison is narrow, as Limitation 8 states. It does cover two optimizer routings with three seeds each, and SwiGNLU leads every variant under both routings by at least 2.6 mbpb (squared ReLU under the qkv routing, where SwiGNLU wins all three pairs). GEGLU and ReGLU never beat SwiGLU in our tests.

Reviewer_iibiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SwiGNLU, a drop-in change to the SwiGLU feed-forward block. I had difficulty reading the paper from the abstract where notations appear with no prior definition. I recommend the authors include the main message of the paper directly in the abstract in plain text so that the reader can decide whether or not if the paper is useful to him/her. The paper is not fllowing the expected fromat of a top AI venue such as ICLR.

Critical strengths

Their operator is simple, and the closest prior work (nonlinear low-rank side branches, activation mixing) is different enough. The gain is modest but highly consistent. The cost accounting and the factorial are more thorough than is usual for FFN-variant papers, which makes the result useful to practitioners. Its generality beyond ≤1.1B models and this optimizer protocol is still open.

Critical weaknesses

I strongly recommend the authors to work on their scientific presentation in plain words and pass the message early in the paper with minimal use of notation and numbers.

Questions for authors

N/A.

1 comment

We thank the reviewer for their time and effort.

We understand that the formatting and flow of the paper made it difficult to digest. In the future we will seek to optimize our delivery style to aid in understanding and present the value proposition up front in clear language.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.