acceptodds
Under review as a conference paper at ICLR 2027

Investigating FFN Parameter Allocation via Cross-Layer Sharing in Transformer LMs

Abstract

This paper studies feed-forward network (FFN) parameter sharing in Transformer-based language models. We share FFN parameters across a contiguous middle block and reallocate the saved budget to widen FFNs, retaining FFN computation in every layer. A broad search using models with 570M non-embedding parameters identifies many configurations with lower average downstream task loss than a standard Transformer baseline. Improvements concentrate in late-heavy configurations that retain more layer-specific FFNs in the late layers. Mirrored comparisons with identical parameter counts and FLOPs within each pair also favor late-heavy allocation on average task loss in most cases. The selected configuration improves average downstream accuracy at the 2B scale without retuning the sharing configuration. These results highlight the importance of where independently parameterized FFNs are retained and suggest that sharing and reallocation can improve downstream performance at a nearly matched parameter budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformer-based Language Models, Feed-Forward Networks, Cross-Layer Parameter Sharing, Layerwise Parameter Allocation, Parameter Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8mb1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how feed-forward network (FFN) parameters should be allocated across Transformer layers. The authors propose a Sandwich-style architecture in which middle layers share FFN parameters, while early and late layers retain independent FFNs. The saved parameters are used to widen the FFNs, keep the total parameter count approximately fixed. By training 147 configurations from scratch at the 570M scale, the authors find that most configurations, especially those retaining more independent FFNs in later layers, improve downstream task loss over a standard Transformer. The authors also show the best configuration improves average downstream accuracy at the 2B scale, although it requires substantially more computation.

Critical strengths
  • The paper studies a simple and clear research question: which layers of LLMs need independent FFNs? The proposed method and experiments are clearly presented and easy to reproduce.
Critical weaknesses
  • The authors motivate their method using prior findings about layer-wise representations, but I find this motivation rather high-level and insufficient to justify the specific design. There is also limited analysis of why later FFNs may need more independent parameters. The authors propose two hypotheses in Section 6 but do not provide experiments to verify them, which I consider a major weakness.
  • The paper does not show that the proposed method outperforms the baseline under matched FLOPs. I understand that pretraining models from scratch is computationally expensive, but given the relatively small performance improvements and the use of only a single training seed, it is difficult to assess how robust the gains are. The authors could consider approaching this research question from a slightly different angle, with more emphasis on understanding the observed behaviour.
Questions for authors
  • The motivation in Section 3.2 relies mainly on prior observations about layer-wise representations and redundancy. Can the authors provide more direct evidence or additional experiments supporting the proposed sharing strategy, particularly why sharing FFNs across middle layers should be effective?
  • Can the authors provide additional analysis to test either of the two hypotheses proposed in Section 6 for why late-heavy configurations perform better?
Reviewer_cTPPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This work proposes a “sandwich-style” parameter sharing technique for feed-forward networks (FFN) in transformer-based LLMs, in which the first and last few layers have independent FFNs, while the middle layers share a single FFN. The saved parameter budget is used to widen the FFNs instead. The authors try many configurations for this approach and find a scheme in which more parameters are shared in early FFNs to work best. The authors report this and other configurations to lead to improved downstream loss, and find that the configuration identified on a smaller scale also transfers well to one examined larger scale.

Critical strengths
  1. The proposed FFN-sharing strategy is well-motivated and conceptually simple.
  2. The configuration search is systematically designed and executed, effectively covering the search space, and yielding a compelling comparison of the parameter-and-compute-matched early-vs-late pairs.
  3. The authors are largely transparent about the limitations of this study and negative results, though I still find that the limitations around compute matching are underclaimed, especially in the abstract and intro (see below).
Critical weaknesses
  1. While the paired configurations with different early/late-layer counts are naturally compute-matched, making the comparison somewhat unfair, or at least limited. The compute-matched comparison in Sec. 5.2 reveals the scheme to not meaningfully improve on the baseline on an even FLOP budget, but this is understated in the abstract and intro. I also don't find Fig. 4 in Sec. 6 alone convincing, as many configurations are clearly sub-optimal, and this is only looking at validation loss (e.g., according to Tab. 9, (0, 0) almost matches (1, 5) in avg. loss, leading me to suspect that the added FLOPs play a major role here). One additional experiment that could shed more light on this would be to also include a FLOP-matched, non-sharing baseline (at least for a few configurations). I.e., in addition to reporting a variation of the FFN-sharing model that matches the baseline FLOP as in Sec. 5.2, authors should compare some of their best-performing setting to a non-FFN-sharing baseline model that matches the FLOPs of the FFN-sharing model.
  2. The scaling results are limited, considering only a single additional datapoint, and scaling only to a limited 2B scale. Switching the evaluation from loss to accuracy also makes it difficult here to ascertain how the reported benefit of the FFN sharing scheme is affected by overall scale. In addition, for the 2B model, the last 5 checkpoints are averaged, which is not the case for the smaller models (App. D). 
  3. I've mentioned this in 1. and 2. but also want to highlight it as a separate point here: The evaluation metrics are applied very inconsistently across the paper, the authors jump from looking at validation loss, to downstream loss, to downstream accuracy, and rarely are all 3 compared, making it very difficult to assess the validity of the claims.
  4. All evaluations use a single seed, making it difficult to ascertain what differences should be considered significant.
  5. It remains unclear how sensitive the reported benefits are to the specific overall model and FFN setup, or training recipe. I understand that replicating the whole grid search with different settings is expensive, but without at least any spot-checks it is hard to trust this method.
Questions for authors

Minor: The connection to Ikeda et al. (L40 “a previous study has shown that reallocating FFN capacity to a consecutive block of middle layers can outperform standard uniform configurations under a comparable parameter budget”) is unclear to me: wouldn't that imply the opposite scheme, i.e., middle layers should remain independent and early/late layers should share FFNs, since sharing effectively reduces the overall capacity of middle layers, or at best keeps it unchanged if the FFN is widened? Could you elaborate on this point?

Reviewer_QABEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a Sandwich-style parameter-sharing method for FFNs in Transformer language models. The proposed methodology shares a single FFN across the middle layers while keeping unique, layer-specific FFNs in the early and late layers, then uses the saved parameter budget to widen all the remaining FFNs. The authors demonstrate that keeping more unique FFNs near the end of the model (late-heavy setup) exhibits better performance.

Critical strengths
  • Instead of removing FFN computation entirely from certain layers, the paper introduces an approach where middle layers share a single FFN while maintaining computation everywhere, reinvesting the saved parameters to widen all remaining FFNs

  • The paper also does an interesting experiment by testing "mirrored configurations" that swap early- and late-layer counts while keeping parameter counts and FLOPs identical. The study provides compelling evidence isolating the exact impact of where unique FFN parameters are retained.

Critical weaknesses
  • The authors already note several constraints of their work, including a limited model architecture, specific experimental design choices, and a restricted model scale.

  • The main concern is the requirement to pretrain the model from scratch using the proposed configuration, which yields only a marginal improvement (around 0.6) over the baseline. Although the baseline model has twice the parameter size of the model used during pretraining, the proposed methodology might actually benefit from an unfair advantage by being explicitly pretrained in this configuration. Considering the high time and financial costs of pretraining weighed against such limited accuracy gains, the practical applicability of the approach is questionable.

  • Existing literature (such as ReplaceMe [1] and Jump to Conclusion [2]) demonstrates efficient, training-free methods for Transformer architectures that preserve base accuracy with far less overhead, whereas this proposed methodology relies on computationally expensive pretraining strategies.

References:

[1] ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization

[2] Jump to Conclusions: Short-Cutting Transformers with Linear Transformations

Questions for authors
  • What is the performance of the proposed method on downstream tasks if applied without a full pretraining step (e.g., via post-training weight sharing, pruning, or fine-tuning existing pretrained models)? Since full pretraining is computationally expensive, understanding whether the sandwich-style sharing can be adopted post-hoc would clarify its practical utility.

  • What if the proposed approach were pretrained on a larger model, such as a 4B-parameter model? Would it still outperform a larger baseline, such as an 8B-parameter model? The baseline used in the experiments, a 2B-parameter model, may have inherent limitations due to its smaller size.

Comments on the paper

1 comment
Area_Chair_z6AmOpenReview ↗

Reminder: Author–Reviewer Discussion Deadline Approaching

Dear Reviewers and Authors,

Thank you to the reviewers for your time, effort, and dedication in reviewing this paper. This AC also appreciates the authors' detailed rebuttal, which addresses the reviewers' comments and includes additional clarifications and experimental results.

To the reviewers: As the author–reviewer discussion deadline approaches, please carefully review the authors' responses and assess whether they adequately address your concerns, if you haven't already. If any concerns have not been fully addressed, please let the authors know what further clarification or evidence would be helpful.

Thank you again for your valuable contributions and constructive discussion.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.