acceptodds
Under review as a conference paper at ICLR 2027

Distributed Bias Terms in Bias-Free Transformer Feed-Forward Networks

Abstract

Modern language models omit bias parameters from their feed-forward networks (FFNs). We show that implicit input and output biases emerge in these networks regardless, and that training organizes them by a structure that is consistent across more than twenty public dense and mixture-of-experts (MoE) models from 70M to 30B parameters. Every token's FFN input contains a large, nearly constant component along the mean input, which gives each neuron a fixed offset, and the average FFN output is added to every position; these are the input and output biases. Training arranges the neurons around the mean input: the gate vectors of most neurons point against it, while those of the few neurons that write the most are strongly aligned or anti-aligned with it. For an anti-aligned gate, the mean component drives the gate preactivation negative but does not shut the neuron off, because SiLU remains nonzero there. The output bias is therefore not the work of a few always-on neurons but a small surplus left over in almost every neuron, the largest one percent contributing between a fifth and a half. Nor are the biases constants. The channels that hold the mean input carry information about the token, including the norm of the residual stream that normalization discards, and in the middle layers context reduces how much most tokens write along the output bias. Models that do have bias parameters form the same implicit biases and keep the explicit ones at one to twenty percent of their size, and training with the mean input removed removes the structure without a measurable change in loss. In MoE models each expert has its own biases, and the output biases of different experts point in a common direction. They add up coherently in the block output, and they account for much of the similarity between experts that is measured on averaged outputs and commonly read as redundancy. Generally, our results show that mean activations are not merely nuisance statistics for mechanistic interpretability, but part of a recurring computational structure across LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
SwiGLU, feed-forward networks, implicit bias, large language models, mixture of experts, mechanistic interpretability, activation geometry, cross-expert alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cgS8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies implicit input and output biases in unbiased Transformer FFNs. We find that in both Dense and MoE models, gate weights tend to align with the input mean, and that many neurons contribute to the output mean rather than a few neurons producing a constant output. We test these patterns using randomized controlled experiments and intervention methods, and show that a shared output bias explains most of the similarity between experts as measured by their average output.

Critical strengths
  • The paper studies dense and MoE models and checks the findings from several angles. The randomized controls and intervention experiments help separate patterns in the learned weights from effects of the mean input alone. This gives more support to the observations than an analysis of trained models alone.
Critical weaknesses
  • The presentation makes the findings difficult to understand. For example, because the analysis moves back and forth between average output and output at average input, it is hard to understand which quantity each experiment measures and which claim it supports.
  • The functional significance remains unclear. While randomized control groups and demeaned-replacement experiments provide some evidence, they do not establish what computational role gate alignment and variance contributions play.
  • The implications for expert redundancy are limited. While the paper shows that similarities between averaged outputs differ from those at common inputs, it does not show whether accounting for this difference changes existing conclusions about expert redundancy.
Questions for authors

See weakness.

Reviewer_YRoMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a detailed study of distributed bias terms in bias-free Transformer feed-forward networks. It demonstrates the existence of implicit input and output biases and examines their properties. The evaluation covers different types of models, including mixture-of-experts (MoE) models. The paper shows that mean activations are part of a recurring computational structure across LLMs.

Critical strengths

S1. The paper provides a thorough analysis of the phenomenon, supported by extensive experiments across different model scales and architectures.

S2. The methodology is sound

Critical weaknesses

W1. Despite its extensive experiments, the paper reads largely as a collection of observations, with limited insight into the underlying causes of the implicit biases or their implications.

W2. The readability could be improved. For example, the motivation behind the chosen metrics should be briefly explained (Lines 108–114, 160–162). Providing clearer motivation at the beginning of each experimental section would also help readers follow the analysis. Additionally, the notation βh(t)\beta_h(t) and ctc_t in Line 140 should be clarified, as they have not been explicitly defined in the preceding text, making Section 2.1 difficult to follow.

Questions for authors

Q1. What are the implications of these findings?

Q2. Could the authors provide further analysis of what drives the organization of gate and read vectors around the mean input during training?

Q3. Figure 4 suggests that all neurons matter, although some contribute much more than others. Could the authors perform neuron ablation experiments to investigate what happens when the most influential neurons are removed, compared with removing the majority of the less influential neurons? Could the authors perform neuron ablation experiments and measure the resulting changes in model loss? Relatedly what happens to the loss if the output bias b is substracted from the layer output at inference?

Q4. The experiments mainly report statistical results at the model scale. Could the authors provide a case study at the layer or neuron level and attempt to interpret the biases, for example, using a logit lens to investigate whether they carry any human-interpretable semantic information?

Q5. Does the output bias play a functional role in model predictions? Or it is just a statistiacl property of the FFN outputs?

Q6. Could the authors provide a proof or derivation showing that b−f(μ)b-f(\mu) is the average of the second- and higher-order terms in the expansion of ff around μ\mu?

Q7. Line 161: I do not understand why the factor sign⁡(aˉh)\operatorname{sign}(\bar a_h) is necessary if the goal is simply to measure the alignment between the neuron vector uhu_h and the mean input direction μ^\hat\mu. The current explanation is not clear to me. Am I missing something?

Reviewer_q9TuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Paper attempts to draw the readers attention to the interesting insight that even though FFN blocks of modern transformers have no explicit bias, there is none the less an implicit bias present. They then perform some analysis on why this matters for MoEs in particular that these different experts have very similar biases. They use the reader/writer decomposition to analyze how this bias emerges. They explore several pre-trained LLOMs, including both dense and MoE. Their big finding seems to be that MoE experts exhibit strongly aligned output bias directions. This analysis, however, is predominantly empirical.

Critical strengths

The main contribution of the paper is to analyze that FFNs implicitly learn a bias, even though there are no explicit bias terms in the FFN. Through this insight they make an interesting empirical discovery that for MoEs the experts exhibit strongly aligned output bias directions. This indicates a potential redundancy that could be leveraged. The authors do a good job with the sheer breadth of the empirical analysis.

Critical weaknesses

The writing of this paper is poor. It actually does a disservice to the idea which I think is neat and intriguing. It also does not make a punch, so if there is a common bias being learned across MoEs how could this be leveraged to improve efficiency or performance? I am sure there is some angle on this, but the authors do a good job of obfuscating this in the main text. It is heavy going with the writing, the authors spend time with their mathematical definitions, but do not give the reader much of a heads up for how they are going to be used or why they matter. The idea itself seems pretty simple, modern FFN blocks no longer use biases, but they occur implicitly anyway through the statistical bias that occurs within the activations themselves. The authors do not need three equations (2-4) to show this, as it seems pretty trivial. They launch into empirical analysis of their statistics too early before setting up why we should care, and what the whole point is. The paper is not at ICLR level at the moment.

Questions for authors
  1. Figure 1 shows all the statistics but there is nothing in the caption or text that connects this to the central narrative of the paper. Would it be possible to make the caption more self containing, saying why these statistics matter?
  2. Paper goes too early into analyzing the statistics of the FFN without contexualizing over what the big picture is. Could you speculate somewhere in the paper how your insights over MoEs could be useful for future transformer network design?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Withdrawal and Thanks to the Reviewers

I would like to thank the reviewers for taking the time to read the manuscript and provide thoughtful and constructive feedback. After consideration, I have decided to withdraw the submission and take additional time to revise the paper and improve the clarity and presentation of this work. I sincerely appreciate the reviewers' time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.