acceptodds
Withdrawn from ICLR 2027

Dynamic FFNs Improve Representation Learning in Transformer Pretraining

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modular architectures allow different inputs to invoke different computations, but existing approaches often couple this flexibility with increased model capacity. We ask how far input-dependent modularity can take us when parameter and compute budgets are held fixed. We introduce ynamic eedorward etworks (DFFNs), which reorganize a standard Transformer feedforward block into a shared static FFN and a set of routed low-rank Submodules (SMs). A learned router selects an SM for each token, providing input-dependent computation without increasing the parameter or compute budget of the dense FFN. Across multiple model scales, DFFN consistently improves language modeling and downstream performance over matched dense Transformers. Controlled experiments further show that these gains depend on learned input-dependent routing rather than simply the presence of the submodules. Together, our results show that modularity can improve Transformer pretraining without relying on increased model capacity.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
llm, pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_urdkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether input-dependent modularity can improve Transformer representation learning during pretraining when both the parameter count and compute budget are held strictly fixed. To decouple modular computation from the increased total capacity typically found in MoE architectures, the authors propose Dynamic Feedforward Networks (DFFNs). A DFFN reorganizes a standard Transformer FFN layer by slightly reducing the intermediate dimension of the shared static FFN and reallocating the freed parameter budget to a linear router and low-rank Submodules. For each token, the router selects the top-1 submodule to apply a low-rank residual update to the static up-projection prior to the nonlinear activation function. The authors evaluate DFFNs across three decoder-only Transformer scales—0.88B, 1.58B, and 2.99B—pretrained on the FineWeb-Edu dataset. Across all three scales, DFFN achieves lower validation perplexity than parameter-matched dense baselines both at the final checkpoints and at approximate Chinchilla compute-optimal checkpoints.

Critical strengths
  1. This is a well-formulated research question with clean parameter control.
  2. The empirical improvement is clear and consistent. Within the tested experimental setup, DFFN outperforms the matched dense baseline in validation perplexity across all three model scales at both compute-optimal and final checkpoints.
Critical weaknesses
  1. The baseline is weak. While Section 2 notes gated FFN variants such as GLU, GEGLU, and SwiGLU, all experiments are conducted on a standard non-gated FFN. However, in DFFN, the routed update multiplies the linear projection by the input-dependent router score, which introduces a multiplicative gating interaction. Therefore, it is not clear whether the improvement comes from modularity or non-linearity.
  2. Incremental architectural novelty over routed LoRA frameworks: Structurally, DFFN closely mirrors routed multi-LoRA architectures previously developed for parameter-efficient fine-tuning, such as Mixture-of-LoRA Experts and LoRAMoE. The primary distinction is applying this formulation to the FFN up-projection during from-scratch pretraining while shrinking the static FFN dimension.
Questions for authors

Comparing empirical training efficiency in Section 5.3 (Table 5) seems unnecessary in this paper, as this work is primarily a mechanistic study on representation learning rather than a systems-efficiency paper. Moreover, the current efficiency comparison is not entirely fair: for example, the dense baseline uses a hardware-unfriendly FFN intermediate dimension (6664 vs. 6144 in DFFN), and practical throughput is heavily sensitive to model scale, GPU architecture, and numerical precision. How do the authors justify the fairness of this comparison, and what would the runtime overhead be if the dense baseline used a hardware-aligned FFN dimension?

Reviewer_VRp9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose augmenting the MLP up-projection of transformers with an additional low-rank path, where the low-rank matrix is chosen from a set of low rank matrices based on the input activation (dynamic, input dependent routing over various low rank matrices). They parameter-match this to a default MLP by reducing MLP inner dimension and find that this change improves language modeling.

Critical strengths

The paper is written clearly and easy to follow in its approach.

Critical weaknesses

I have concerns that make me question the validity of the results:

  • Table 2 right shows that the 1.58B model trained on 20 tokens/parameter performs worse than the 0.88B model trained on 20 tokens/parameters. There must be something wrong here.
  • the baseline is a two-layer MLP with expansion factor 3.25 and unspecified activation function. This choice seems very odd and is unjustified. Where is the 3.25x coming from? What activation function is used? Why not just compare against a Swiglu mlp with 8/3 expansion factor as is standard? The authors should show that this method generally works on (modern) MLPs, independent of whether it is self-gated or not, or the specific activation function. The baseline they chose seems constructed.
  • the general experimental setup is odd. Why train a 0.88B model on 345B tokens, 1.58B model on 400B tokens, 2.99B model on 95B tokens, what is the rationale here? Why is the model size swept through the number of blocks only, without changing model dimension?
Questions for authors

See questions in critical weaknesses. Additionally, it might help if the authors provided code to reproduce experiments.

Reviewer_MyEpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates DFFNs, a modification to the up-projection of MLPs. Under this modification, the up-projection contains both a standard, full-rank component and a routed set of low-rank components. Models of 3 scales are trained with the old MLP structure and this new MLP structure, where their performances are compared both on raw performance and on downstream performance on an array of tasks. The work also goes beyond performance and looks into the structure learned in the routed components. The input subspaces for each component learn to be roughly orthogonal to each other, indicating some functional specialization. Additionally, by only selecting a single component per prompt rather than allowing the router to create a smooth average of multiple components, performance is shown to be roughly unchanged, further supporting that these different components learn functionally different behaviors.

Critical strengths

The results show fair comparisons between the prior MLP structure and the new DFFN structure. These carefully hold the parameter count and compute constant. Both model groups are trained on identical datasets and presumably identical training scripts (code has yet to be released), removing possibly confounding factors from using prior results on possibly different codebases. These experiments are conducted on 3 model scales, reasonable under compute constraints. Model and data scale are carefully chosen from scaling laws.

Critical weaknesses

DFFNs are not closely related to MoE, but have an equal parameter count. The addition of a full-rank component on top of the routed mixture components does separate the two methods; however, ablations that only use the routed component, which would identify that this mixture is important rather than just the router, are not conducted.

Questions for authors
  1. Are there experiments that compare the allocated parameters to the routed component and the full-rank component? Even on the smallest model with less compute as a comparative setting, it would be interesting to see the behavior of changing other parameters in this method.
  2. Building on this previous point, are there experiments that perform other ablations on learning rather than post-training ablations changing the number of used components in the router? The method is promising, but the combination of components responsible is not quite clear.
  3. In Figure 3, each of the heads is connected by its index in each layer. Is there some reason we should expect head k of layer l to be similar to head k of layer l+1? This isn't critical, but there may be a cleaner way to represent these results.

Small things: Line 294: The citation of Hoffmann et al. appears to need a different structure, \citep? Or some other sentence structure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.