acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Activation Functions: Reducing Bias Can Improve Learning

Abstract

Many widely used activation functions, including ReLU, GELU, Swish and Sigmoid, produce predominantly positive outputs, resulting in a nonzero post-activation mean that we term activation bias. Despite its ubiquity, the effect of activation bias on learning dynamics remains poorly understood. Using controlled models, we show that activation bias accelerates initial loss reduction but is accompanied by overshooting, non-smooth trajectories, and a steeper loss landscape. This early advantage also coincides with the model's use of task-relevant information encoded in the bias component. In contrast, reducing activation bias slows initial learning but ultimately yields a lower final loss. Experiments on larger vision and language models exhibit a similar pattern: smaller activation bias is associated with better late-stage performance. Thus, this seemingly minor property can substantially redirect learning and warrants explicit consideration in future activation function design.

Primary area:
optimization
Keywords:
activation function, activation bias, training dynamics, loss landscape, scientific understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_ByRSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the impact of "activation bias" (the distribution of values after non-linearities) on training dynamics.

The methods are mostly empirical, using first controlled deep residual teacher-student models, then larger language and vision models. The authors compare pairs of model without and with an additive constant applied to the output of non-linearities.

The main finding is that reducing the activation bias (with an additive constant < 0) makes the loss landscape smoother, meaning a slower initial training but better final loss.

They also find that the bias carries some information, especially early during training.

Critical strengths

(S1) Studies a fundamental aspect of the design of neural networks (choice of non-linearities), which should be interesting to a broad audience.

(S2) Although the interaction of this choice with training dynamics has been extensively studied, the particular question addressed (impact of the bias in post-activation distributions) seems to have been overlooked. The finding that this bias has a such an impact on training dynamics is novel and surprising to me.

(S3) The experiments use a mix of minmal settings and larger models, which cover both controlled experiments and validation in practical settings.

Critical weaknesses

Mix of weaknesses and questions below.

(W1) General question: isn't it surprising that the observed effect is so clear yet hasn't been investigated previously? Do you have a guess why/how this could have been overlooked until now? Especially since a central claim is the consistently better (L419) late-stage performance.

(W2) I'm not sure what is the significance of the "overshoot" in the norm of the hidden states (Sect. 2.2). The term "overshoot" seems to imply that the optimization was heading somewhere but got past it because of excess momentum. So this seems to indicate a poorly-tuned optimizer, rather than an issue with the architecture? Figure 3a and 3b also do not seem clearly different to me. Both have a sort of reversal at the last step.

(W3) I'm not sure what is the utility of this paper in practice. The conclusions are vague: "These results motivate a more complete causal account of activation bias and its explicit consideration in future activation-function design". Does it mean we need to design other activation functions? Should we just use the additive constant like the experiments of this paper? How about the choice of optimizer (see Q2 below)? How does this related to recent work on learned activation functions (eg [1,2,3])? (no need to add these specific citations if not relevant, but there is not even an acknowledgement of the existence this area of the literature, so I'm not sure the authors know about it)

[1] Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

[3] Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions

[2] Do we always need the simplicity bias? looking for optimal inductive biases in the wild


Detailed or minor comments

  • In Figure 1, the meaning of "Representation" on the Y axis is unclear until much later in the paper.

  • Figure 3: include a legend/colorbar and subtitles to the panels (shifted/unshifted/...) to make reading easier (rather than stating everything in the caption).

  • Section 5 title: use singular noun "Discussion".

  • The end of the "Discussion" and "Conclusion" sections are near-identical. Consider merging them or removing some of the redundancy.

  • Isn't the proposed "adaptive shifting" identical to a layer normalization applied after the non-linearity???

Questions for authors

I mixed in the questions with the statement of weaknesses above. No need to reply to every each of them. Feel free to focus on the parts that I may have misunderstood, and those that are most critical. ⬇️

The current recommendation for "weak rejection" is because of the unanswered questions state above, in particular the fact that the story doesn't totally add up (why has this been overlooked until now?) and that the paper doesn't quite close the loop (what to do with these findings?). It's a valuable exploration and certainly a great workshop paper, but it seems to be missing a few things to reach the bar of the main conference.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer ByRS (Part 1)

We thank the reviewer for the careful reading and professional technical questions. We reply to the questions and concerns point by point below.

Weakness 1

We thank the reviewer for this valuable question.

We were indeed surprised by the clear effect and by the fact that this had not been investigated previously. We suspect this is because the research approach adopted here has been overlooked and underestimated. Large language models contain many components and complicated mechanisms, and are trained on complex tasks. These factors can interfere with one another, mask the effect of activation bias, and even produce inconsistent observations. In this work, we instead propose a simple but well-controlled teacher–student model that removes irrelevant elements, isolates the effect of activation bias, and thereby reveals consistent improvements. We believe this "do subtraction" research approach is powerful but has been underused by the community, which may explain why the effect has been overlooked.

We've also discussed this point in Section 7 of the updated paper.

Weakness 2

We thank the reviewer for this question. We apologize for any confusion caused by our use of "overshoot" and the related figures.

Here, "overshoot" does not imply excess momentum from a poorly tuned optimizer. It describes the non-monotonic learning trajectory of the model. In Section 2.2, we track the mean of the residual-stream hidden state as a scalar summary of its evolution. Combined with Section 3, the unshifted model initially exploits the activation bias component as a shortcut: this component carries task-relevant information early in training and allows the model to reduce the loss quickly. As training proceeds, the model finds this shortcut limited and gradually discards the information in the bias component. Because the output in our controlled model is directly determined by the hidden state, we expect the same non-monotonic behavior to appear in the hidden-state trajectory. This rise-and-fall pattern is what we term overshoot. Indeed, the overshoot in Section 2.2 coincides with both the early-advantage stage of the unshifted model and the reconstruction results in Section 3.

Regarding Figures 3a and 3b, the point is that the unshifted model first dashes along a steep direction and then makes a sharp turn (Figure 3a). Figure 3b visualizes the shifted model's trajectory on the same plane, where its loss varies little—showing that its early updates develop largely outside this direction. This motivated us to construct a complementary plane (Figure 3c), on which the shifted model exhibits no such "dash-then-turn" behavior and instead follows a smooth trajectory toward lower loss. The comparison in Figure 3 therefore shows that activation bias induces a sharp direction: the model rapidly dashes along it to gain an early advantage, after which it makes a sharp turn and optimization slows. Reducing activation bias instead yields a smooth trajectory pointing toward the basin. The small changes at the last step of these figures are minor and not our emphasis.

Overall, the emphasis of "overshoot," Figure 3, and Section 3 is that activation bias makes the learning trajectory non-smooth and non-monotonic. We model this from three complementary angles: the hidden-state trajectory, the weight trajectory, and the trajectory of encoded information. Since the two models use the same Adam optimizer, this issue does not come from the optimizer but from activation bias itself.

AuthorsOpenReview ↗

Reply to Reviewer ByRS (Part 2)

Weakness 3

We thank the reviewer for this valuable question about practical utility, and we address it in three points.

First, compared with an additive constant, which may be hard to choose directly, the adaptive shifting used in our scale-up experiments is a plug-and-play intervention that can improve performance and potentially reduce compute. In Appendix F.4, we estimate that for GPT-2 it could save up to one-third of the original compute. Due to our limited compute budget, we could not test adaptive shifting on larger models and tasks. This leaves a valuable opportunity for practitioners, as the operation is extremely simple and could be broadly helpful.

Second, reducing activation bias may also benefit optimizers. Our work shows that a simple change in activation bias produces substantial differences in learning dynamics: it reduces overshoot, smooths learning, and is associated with a less sharp loss landscape. Since some optimizers struggle on poorly conditioned landscapes, reducing activation bias may benefit them and widen the range of usable optimizers. These optimizers should therefore be re-tested when activation bias is reduced.

Third, we do not claim that new activation functions should be designed immediately. Rather, we argue for a more complete understanding of activation functions, so that these properties can be fully taken into account in future activation design. Bias is only one property of an activation function. The same approach—isolating a property and tracing its effect on training dynamics—could be applied to post-activation variance, derivatives, or saturation. Studying these properties separately and then jointly may support a more complete understanding and guide the design of stronger activation functions in the future.

Additionally, we thank the reviewer for pointing us to Teney et al. (2026), Ducotterd et al. (2024), and Teney et al. (2025). We agree that learnable activation functions are a highly relevant line of work within the field of activation functions that we should acknowledge. We have therefore added these references and discussed them in the related-work section. We believe learnable activation functions hold strong potential, particularly once we have accumulated a sufficient understanding of activation properties, so that these inductive biases can be incorporated into learnable activation designs.

We apologize again for not clarifying the significance of this paper in the discussion section of the original manuscript. We have therefore revised this section thoroughly to clearly state the utility and limitations of our work. Please refer to the updated PDF.

Detailed or minor comments

We thank the reviewer for these detailed and constructive suggestions on clarity and presentation. We agree that they improve the paper, and we have revised all of these details accordingly. Please refer to the updated PDF.

Concerning the question of whether the proposed adaptive shifting is identical to layer normalization applied after the nonlinearity, the two operations are similar but not identical: adaptive shifting only removes the mean and does not rescale the variance, nor does it include the affine parameters that LayerNorm contains.

We agree that comparing these similar operations is necessary, especially since some prior work has proposed post-activation LayerNorm. We have therefore added Section 5 to compare them. The results show that adaptive shifting, together with the fixed offset, outperforms the post-activation LayerNorm counterpart. This further indicates that the variance of activation functions warrants further investigation, rather than being simply set to 1 or controlled by the learnable γ in LayerNorm.

Reviewer_ByRSOpenReview ↗

Still unconvinced

Overall I'm still incredulous that this basic fact has both been overlooked until now and that a trivial fix provides consistently better performance.

Concerning the question of whether the proposed adaptive shifting is identical to layer normalization applied after the nonlinearity, the two operations are similar but not identical: adaptive shifting only removes the mean and does not rescale the variance, nor does it include the affine parameters that LayerNorm contains.

This means that the proposed adaptive shifting is strictly a subset of what LayerNorm already does. So what is proposed is essentially a LayerNorm with frozen bias/scale (?!). I see that another review (nLsZ) also pointed this out above, and it is also confirmed in the authors' reponse as I understand.

I also noted other relevant prior work that was originally overlooked by the authors. I appreciate the efforts in the response and the the patching up of paper with additional experiments and discussion, but it now seems that the story should have built by better putting up first the existing knowledge and highly-related methods.

In summary I'm not updating my negative assessment because:

  • I don't think the paper starts with a good representation fo the current state of knowledge
  • I don't think the implications and utility of the paper are clear enough (the response to my question has a lot of text but little substance ("may also benefit optimizers", "we argue for a more complete understanding of activation functions", ...). I still don't know what to do with this paper. If the authors think this is important for the choice of optimizer or for the design of activation functions, then we need some concrete hint that this is indeed the case. In its current state, this is fine as a workshop paper but does not pass the bar for the main conference,
Reviewer_yYWXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how activation bias impacts learning trajectory across different model architectures. Here, activation bias is the non-zero post-activation mean produced by commonly used activations like ReLU, sigmoid, GELU etc. First, the authors test the impact of the activation bias in a controlled setting, where they use a teacher-student setup with two student models (one with the original unshifted activation and the other with a shift term subtracted to correct for the bias) learning from the same unshifted teacher model (all models studied here have only pre-norm residual MLP blocks with attention and embeddings removed). In this controlled setup, the paper demonstrates that the unshifted model starts out with a lower loss initially, but the model with activation bias corrected reaches a lower final loss. The authors link this to the following observations: the unshifted model first "overshoots" the residual hidden state mean then settles at a lower value as training progresses while the shifted model's mean moves smoothly, the shifted model has nearly orthogonal initial update directions and lower Hessian eigenvalues, and a smoother loss landscape. Next, the authors demonstrate, using an autoencoder, that the unshifted model initially relies on information carried in the activation bias but gradually it relies on it less. Finally, the authors apply an adaptive per-token mean subtraction to ViT-B/L, DiT-B/2, GPT-2 Small, OLMo-2-1B and HyenaDNA and show that this leads to small late stage improvements.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The authors study the question through various well thought out experiments.
  3. The findings are more or less consistent across architectures/tasks.
Critical weaknesses
  1. The overall significance is unclear and the gains at scale are small. Since the main gains are at later stages of training, seeing that it doesn't scale very well on larger models makes me question the significance.
  2. Gated activations, like SwiGLU, which are more commonly used now, are not studied. It is not clear if the bias issue will be present there.
  3. The adaptive shift is equivalent to mean-only normalization of the MLP hidden layer, which is close to adding a LayerNorm after the FFN activation like the NormFormer paper. This work is neither cited nor compared against.
Questions for authors

What is shifted in OLMo-2-1B? How do results change for SwiGLU based models?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer yYWX

We thank the reviewer for the careful reading and professional technical questions. We reply to the questions and concerns point by point below.

Weakness 1

We thank the reviewer for this comment. We apologize that we did not clearly explain the significance of the scale-up results.

The models and tasks we chose for the scale-up experiments are well established and have already been extensively optimized, so further gains from adaptive shifting can be expected to be modest. Even so, reducing activation bias consistently improves performance across multiple runs and the models and tasks tested.

We have clearly discussed the overall significance in Section 7. First, adaptive shifting is a simple and plug-and-play intervention that can improve performance and potentially save a large amount of compute as shown in Appendix F.4. Second, reducing activation bias changes the learning dynamics, reduces overshoot, and is associated with a less sharp loss landscape. This may also help optimizers that struggle with poorly conditioned landscapes. Third, the way to isolate activation bias and study it provides an approach for studying other activation properties, including post-activation variance, derivatives, and saturation, which may inform future activation-function design. Please refer to the updated PDF.

Weakness 2

We thank the reviewer for this comment. We apologize that the original submission did not sufficiently examine gated activations.

In Appendix D, we add SwiGLU experiments in our controlled model. We shift only the Swish output before multiplication by the linear gate branch. As shown in Figure 16(a), the shifted model still has higher loss early in training but lower loss later, consistent with our earlier findings. Figure 16(b) shows a similar reversal in OLMo-2-1B with adaptive shifting.

We also find lower Hessian sharpness in the shifted OLMo-2-1B model (Table 4). Figure 18 shows that its post-activation values have a smoother, more stable absolute-value distribution, whereas the unshifted model develops a stronger concentration near zero, suggesting more dead neurons and a lower activation rate. Thus, activation bias in the Swish branch still affects learning in SwiGLU and other gated activation functions.

Weakness 3

We thank the reviewer for pointing out these works. We apologize for overlooking NormFormer (Shleifer et al., 2021) and Sub-LN (Wang et al., 2023). In Section 5, we now discuss both papers and compare their post-activation LayerNorm operation with our shifting methods.

Post-activation LayerNorm differs from adaptive shifting in two ways: it normalizes variance and can introduce learnable scale and bias, whereas adaptive shifting only subtracts the mean. Moreover, NormFormer combines its FFN LayerNorm with other architectural changes, and Sub-LN is evaluated together with depth-dependent initialization. NormFormer's ablations remove one operation at a time from the full architecture. These experiments therefore do not isolate the benefit of subtracting the post-activation mean alone.

We compare adaptive shifting with post-activation LayerNorm, with and without learnable scale and bias, in the controlled model in Section 5. Figure 7(b) shows that adaptive shifting performs better than both LayerNorm variants. These results suggest that additionally normalizing variance does not guarantee better performance in this setting, and the role of post-activation variance deserves further study.

Question 1

We thank the reviewer for this question. We shift the Swish output before multiplication by the linear gate branch, rather than shifting the gated product, as clarified in Appendix D. This follows our definition of activation bias as the nonzero mean produced by the nonlinear activation.

For the results of SwiGLU-based models, please refer to our answer to Weakness 2.

Thank you for the rebuttal. I have read the rebuttal and looked at the feedback from other reviewers. The SwiGLU results should be added to the main text instead of the appendix. While some of my concerns are addressed, I am still not fully convinced about the overall significance, and I maintain my score.

Reviewer_4gEhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes the effect of the nonzero mean of activation outputs on learning dynamics. The topic is highly relevant, since it can help advance the theory of activation functions and improve the design of neural network architectures.

Critical strengths

The paper provides a theoretical analysis of the effect of activation shifts on optimization geometry, alongside experiments in a controlled teacher-student architecture.

It proposes interesting experiments to assess this behavior, with a strong focus on the initial slowdown followed by an advantage at the end of training in the controlled setting.

Critical weaknesses

The experiments on real-world architectures are not fully aligned with the theoretical analysis. While the theory considers a constant shift (b), the large-scale experiments use input-dependent centering across neurons. Although the authors explicitly acknowledge this, its implications for connecting the theoretical analysis to the large-scale results are not well explored.

The reported improvements are small and their robustness is difficult to assess without information about variability across runs and statistical significance in the real-world architecture experiments.

The positioning relative to the following previous works (already cited by the authors) should be clarified:

  • Raiko et al. (2012), "Deep Learning Made Easier by Linear Transformations in Perceptrons". This work already studies activation centering and its effects on conditioning, gradient directions, and generalization. The authors should clarify what new insights emerge from their analysis of how training dynamics evolve over time.

  • Clevert et al. (2016), "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)". This work already links bringing mean activations closer to zero to reduced bias shift and improved learning. The authors should clarify to what extent their contribution lies in isolating the effect of the activation mean by shifting the same activation function and identifying the trade-off between early learning and final performance.

Moreover, the authors can also discuss the following paper: Schraudolph (1998), "Centering Neural Network Gradient Factors".

Questions for authors

Could the authors report variability across multiple runs (e.g. mean and standard deviation over seeds), at least for ViT-Base and GPT-2 Small, with also the absolute final test accuracy and perplexity, respectively, alongside the residual curves?

Beyond the time-resolved analysis of training dynamics, which findings of this work would not follow from the analyses in Raiko et al. (2012) and Clevert et al. (2016)?

Does adaptive shifting introduce measurable computational overhead in the large-scale experiments?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer 4gEh (Part 1)

We thank the reviewer for the careful reading and professional technical questions. We reply to the questions and concerns point by point below.

Weakness 1

We thank the reviewer for this comment. We apologize that we did not clearly explain the connection between fixed and adaptive shifting.

Our scale-up experiments test whether reducing activation bias also improves performance in practical models. Because the post-activation mean differs across layers and changes during training, a single fixed shift may no longer keep it near zero. We therefore use adaptive shifting. To assess the connection, we compare fixed and adaptive shifting in the same controlled model in Section 5. Their loss curves nearly overlap in Figure 7(a), supporting adaptive shifting as a practical extension of fixed shifting.

A full multilayer analysis of gradient propagation and Hessian sharpness under adaptive shifting is beyond the simplified theory in our appendix. We use that theory to interpret the fixed-shift intervention, while the controlled comparison in Section 5 provides the empirical link to the used adaptive shifting in our scale-up experiments.

Weakness 2

We thank the reviewer for this suggestion. We apologize that the original submission did not report variability across runs.

We agree that the small size of the improvements makes their consistency important. We have now repeated the ViT-B and GPT-2 Small experiments with three runs. Table 2 reports the absolute final top-1 accuracy and perplexity for each run, and Figure 24 in Appendix F.3 shows the corresponding performance differences during training. The shifted model performs better in all three reported runs for both models. We therefore describe the gains as small but consistent across runs.

These scale-up experiments use established models and tasks that have already been extensively optimized, so a large improvement from changing only the activation mean would be unexpected. Their purpose is to test whether the effect identified in the controlled setting remains observable in practical architectures. The results show consistent improvements, and Appendix F.4 further illustrates that the shifted GPT-2 run reaches the unshifted run's final evaluation loss in roughly one-third fewer training steps. We have not tested much larger models or tasks because of our compute budget, and we see this as an important direction for future work.

Weakness 3

We thank the reviewer for highlighting these connections. We apologize that we did not clearly distinguish our contribution from earlier work.

Schraudolph (1998) and Raiko et al. (2012) center quantities by averaging across examples in a batch. Our adaptive shift instead subtracts the mean across neurons within each input's post-activation vector. Note that averaging across examples does not ensure zero post-activation bias for each individual input. Thus, their centering operates along a different dimension and does not perform the per-input bias removal studied in our work.

We also agree with Clevert et al. (2016) that bringing mean activations closer to zero can reduce bias shift and improve learning. ELU, however, changes the negative branch of the activation by introducing an exponential function, so its effects cannot be attributed to the activation bias alone. Our fixed-shift experiments compare the same activation function with and without a shift, isolating the effect of the activation bias.

In short, these related works either study operations that differ substantially from ours or incorporate other factors without isolating activation bias. These distinctions clarify what is new in our findings and how our paper relates to prior work.

We additionally discuss other related post-activation normalization methods, NormFormer (Shleifer et al., 2021) and Sub-LN (Wang et al., 2023), in Section 5. Their post-activation LayerNorm operations resemble adaptive shifting in centering activations on the neuron dimension, but LayerNorm also rescales variance and can include learnable scale and bias. In our controlled comparison, Figure 7(b) shows that adaptive shifting outperforms post-activation LayerNorm both with and without learnable scale and bias.

AuthorsOpenReview ↗

Reply to Reviewer 4gEh (Part 2)

Question 1

We thank the reviewer for this question. The three-run variability results, absolute final ViT-B accuracy and GPT-2 perplexity, and residual curves are now reported in Table 2 and Figure 24 of Appendix F.3.

Question 2

We thank the reviewer for this question. Please refer to our answer to (Weakness 3).

Question 3

We thank the reviewer for this question. Adaptive shifting only computes and subtracts the post-activation mean. It has no learnable parameters and does not rescale variance which means that it is a simplified version of a Layernorm and therefore does not introduce computational overhead. Furthermore, Appendix F.4 instead reports that, in the illustrated GPT-2 run, reducing activation bias can save large amount of compute.

Reviewer_4gEhOpenReview ↗

Open Points

The results of the three runs and the absolute metrics are very good for addressing my concern about variability. The comparison between the fixed and the adaptive shifting is an empirical connection between the two methods of shifting.

What results from your experiments go beyond what Raiko et al. and Clevert et al. have shown? Also, the connection between fixed and adaptive shifting remains empirical. Could you clarify the extent to which the theoretical results help explain the large-scale experiments?

While the absence of learnable parameters and variance rescaling is a plus for the proposed method, it would be good to quantify the computational overhead of the adaptive shifting relative to the unshifted baseline. Reporting the time per training step or throughput would be useful. While Appendix F.4 shows the reduction in training steps for the different methods, this is not the same as quantifying the computational overhead.

Reviewer_nLsZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies activation bias, the nonzero post-activation mean of common activations such as ReLU and GELU. In a controlled residual teacher–student model, subtracting the initial mean from every activation slows early training but gives a lower final loss, a smoother hidden-state trajectory and smaller Hessian eigenvalues. A linear down-projection analysis shows that removing the mean minimizes sharpness. An autoencoder experiment suggests that the unshifted model transiently uses information in the mean component. At scale, subtracting the per-token mean of FFN hidden activations improves late-stage results for ViT, DiT, GPT-2 Small and OLMo-2-1B.

Critical strengths
  1. The early-fast, late-slow reversal is the most interesting finding. Classical centering arguments predict that centered activations converge faster, so the early advantage of the uncentered model is counterintuitive. I could not find a prior report of this reversal.
  2. The scale-up covers three modalities and five architectures up to 1B parameters. The intervention is a single line, and the baselines use recipes tuned for the unshifted model, which makes the reported gains conservative rather than inflated.
  3. The discussion states its limits plainly (no unified theory, no claim that zero bias is universally optimal).
Critical weaknesses
  1. The sharpness result is the classical centering argument. Efficient BackProp already shows that nonzero-mean inputs to a layer create one large Hessian eigenvalue and recommends centering the inputs of every layer, and factor-centering decomposition and mean-normalized SGD turned this into training methods. The paper cites Efficient BackProp only for sigmoid and tanh, so the "first study" claim does not hold.
  2. Adaptive shifting (Eq. 9) is a mean-only LayerNorm between the FFN activation and the down-projection. Pre-LN leaves activation bias in place because its LayerNorm sits before the up-projection, but NormFormer and Sub-LN already add a LayerNorm at exactly this position and report gains in LM pretraining. The paper neither cites nor compares against them, so it does not show what adaptive shifting adds over existing practice.
  3. The implication for current LLMs is untested. Most recent LLMs use SwiGLU, whose output is close to zero-mean because its linear branch is symmetric, so the bias the paper targets may be largely absent there. The paper never measures activation bias in a gated FFN.
  4. Section 3 is qualitative and partly tautological. After shifting, the mean component is near zero by construction, so decoding it yields little content whether or not the model uses it. The comparison needs a measure that does not depend on the magnitude of this component.
Questions for authors
  1. In Eq. 9, does the gradient flow through the subtracted mean, or is the mean detached?
  2. In OLMo-2-1B, which uses SwiGLU, was the shift applied to the gated product or to the Swish gate alone?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer nLsZ

We thank the reviewer for the careful reading and professional technical questions. We reply to the questions and concerns point by point below.

Weakness 1

We thank the reviewer for this comment. We apologize that we did not clearly explain the relation to prior work.

We agree that relevant studies, such as Efficient BackProp (LeCun et al., 1998), have discussed how centering input distributions can improve Hessian conditioning. Therefore, we have removed the "first study" claim to avoid overstating our contribution.

However, it's clear that centering layer inputs or even the pre-activation distribution does not guarantee zero-mean outputs after the activation. Additionally, in our experiments, activation bias increases several leading Hessian eigenvalues, not just the largest one. Therefore, our work goes one step further by directly studying post-activation bias and its effects on learning dynamics.

Weakness 2

We thank the reviewer for pointing out these works. We apologize for overlooking highly related works, including NormFormer (Shleifer et al., 2021) and Sub-LN (Wang et al., 2023).

However, the adaptive shifting proposed in our work is qualitatively different from post-activation LayerNorm: LayerNorm normalizes the variance and introduces learnable scale and bias, whereas adaptive shifting only subtracts the mean. Although NormFormer and Sub-LN report gains, they combine this operation with other changes: NormFormer also adds post-attention LayerNorm and head-wise attention scaling, while Sub-LN is evaluated together with depth-dependent initialization. NormFormer's ablation studies remove one operation at a time but keep the others from the full architecture. These works therefore do not isolate the benefit of reducing activation bias.

In Section 5, we now discuss both papers and compare their post-activation LayerNorm operation with our shifting methods using our controlled model. Figure 7(b) shows that whether LayerNorm is equipped with learnable scale and bias or not, adaptive shifting performs better than both LayerNorm variants. These results suggest that additionally normalizing the variance via a post-activation LayerNorm does not guarantee improvements in learning, which further implies that the variance control and studying other properties of activation functions are valuable future directions of our work.

Weakness 3

We thank the reviewer for this comment. We apologize that the original submission did not sufficiently examine gated activations.

In Appendix D, we add SwiGLU experiments in our controlled model. To clarify, for SwiGLU we only shift the Swish output before multiplication by the gate. As shown in Figure 16(a), the shifted model still exhibits a clear early-late reversal, consistent with our earlier findings. We also find that this phenomenon is consistent in the representative LLM OLMo-2-1B (Figure 16(b)).

We further analyzed the distributions of absolute values in the unshifted and shifted models. Figure 18 shows that, in the shifted model, the post-activation distributions are smoother and more stable, whereas its counterpart is very sharp and has substantial clustering of values near zero, implying that many neurons are inactive, even before the gate control. This difference further provides us with a perspective on activation bias: reducing bias may lead to a larger activation rate and fewer dead neurons which may benefit learning.

Weakness 4

We thank the reviewer for this comment. We apologize that we may have caused some confusion and misunderstanding.

We agree that shifting makes the bias component close to zero by construction. However, this is precisely the purpose of the intervention: to restrict the model's use of this component as an early shortcut. In this experiment, the shifted model has higher early loss but lower final loss (Figure 20) and the early advantage of the unshifted model coincides with the informative reconstruction from the bias component. Thus, our interpretation is that shifting restricts access to this component as an early shortcut and leads to an early lag but a later comeback, rather than indicating that the model chooses not to use it. This offers an information-angle explanation for the unshifted model's early advantage and possible late-stage disadvantage, complementing our analysis of learning dynamics.

Question 1

We thank the reviewer for this question. Gradients flow through the subtracted mean and it is not detached. Reducing activation bias therefore affects gradient propagation through the FFN.

Question 2

We thank the reviewer for this question. We shift the Swish output before multiplication by the gate as clarified in Appendix D. This follows our definition of activation bias as the nonzero mean produced by the nonlinear activation.

AuthorsOpenReview ↗

Reviewer-Author Discussion Period Closing

Dear Reviewer nLsZ,

Thank you very much for your time and effort in reviewing.

With the discussion closing in less than 24 hours (Nov. 18 AOE), we want to follow up on our rebuttal. Please let us know whether any of your concerns or questions have been addressed and whether you have follow-up questions.

We’d be grateful if you would revisit your assessment in light of the rebuttal. Deeply appreciated!

Best,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.