- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether input-dependent modularity can improve Transformer representation learning during pretraining when both the parameter count and compute budget are held strictly fixed. To decouple modular computation from the increased total capacity typically found in MoE architectures, the authors propose Dynamic Feedforward Networks (DFFNs). A DFFN reorganizes a standard Transformer FFN layer by slightly reducing the intermediate dimension of the shared static FFN and reallocating the freed parameter budget to a linear router and low-rank Submodules. For each token, the router selects the top-1 submodule to apply a low-rank residual update to the static up-projection prior to the nonlinear activation function. The authors evaluate DFFNs across three decoder-only Transformer scales—0.88B, 1.58B, and 2.99B—pretrained on the FineWeb-Edu dataset. Across all three scales, DFFN achieves lower validation perplexity than parameter-matched dense baselines both at the final checkpoints and at approximate Chinchilla compute-optimal checkpoints.
Critical strengths
- This is a well-formulated research question with clean parameter control.
- The empirical improvement is clear and consistent. Within the tested experimental setup, DFFN outperforms the matched dense baseline in validation perplexity across all three model scales at both compute-optimal and final checkpoints.
Critical weaknesses
- The baseline is weak. While Section 2 notes gated FFN variants such as GLU, GEGLU, and SwiGLU, all experiments are conducted on a standard non-gated FFN. However, in DFFN, the routed update multiplies the linear projection by the input-dependent router score, which introduces a multiplicative gating interaction. Therefore, it is not clear whether the improvement comes from modularity or non-linearity.
- Incremental architectural novelty over routed LoRA frameworks: Structurally, DFFN closely mirrors routed multi-LoRA architectures previously developed for parameter-efficient fine-tuning, such as Mixture-of-LoRA Experts and LoRAMoE. The primary distinction is applying this formulation to the FFN up-projection during from-scratch pretraining while shrinking the static FFN dimension.
Questions for authors
Comparing empirical training efficiency in Section 5.3 (Table 5) seems unnecessary in this paper, as this work is primarily a mechanistic study on representation learning rather than a systems-efficiency paper. Moreover, the current efficiency comparison is not entirely fair: for example, the dense baseline uses a hardware-unfriendly FFN intermediate dimension (6664 vs. 6144 in DFFN), and practical throughput is heavily sensitive to model scale, GPU architecture, and numerical precision. How do the authors justify the fairness of this comparison, and what would the runtime overhead be if the dense baseline used a hardware-aligned FFN dimension?