- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies the impact of "activation bias" (the distribution of values after non-linearities) on training dynamics.
The methods are mostly empirical, using first controlled deep residual teacher-student models, then larger language and vision models. The authors compare pairs of model without and with an additive constant applied to the output of non-linearities.
The main finding is that reducing the activation bias (with an additive constant < 0) makes the loss landscape smoother, meaning a slower initial training but better final loss.
They also find that the bias carries some information, especially early during training.
Critical strengths
(S1) Studies a fundamental aspect of the design of neural networks (choice of non-linearities), which should be interesting to a broad audience.
(S2) Although the interaction of this choice with training dynamics has been extensively studied, the particular question addressed (impact of the bias in post-activation distributions) seems to have been overlooked. The finding that this bias has a such an impact on training dynamics is novel and surprising to me.
(S3) The experiments use a mix of minmal settings and larger models, which cover both controlled experiments and validation in practical settings.
Critical weaknesses
Mix of weaknesses and questions below.
(W1) General question: isn't it surprising that the observed effect is so clear yet hasn't been investigated previously? Do you have a guess why/how this could have been overlooked until now? Especially since a central claim is the consistently better (L419) late-stage performance.
(W2) I'm not sure what is the significance of the "overshoot" in the norm of the hidden states (Sect. 2.2). The term "overshoot" seems to imply that the optimization was heading somewhere but got past it because of excess momentum. So this seems to indicate a poorly-tuned optimizer, rather than an issue with the architecture? Figure 3a and 3b also do not seem clearly different to me. Both have a sort of reversal at the last step.
(W3) I'm not sure what is the utility of this paper in practice. The conclusions are vague: "These results motivate a more complete causal account of activation bias and its explicit consideration in future activation-function design". Does it mean we need to design other activation functions? Should we just use the additive constant like the experiments of this paper? How about the choice of optimizer (see Q2 below)? How does this related to recent work on learned activation functions (eg [1,2,3])? (no need to add these specific citations if not relevant, but there is not even an acknowledgement of the existence this area of the literature, so I'm not sure the authors know about it)
[1] Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
[3] Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions
[2] Do we always need the simplicity bias? looking for optimal inductive biases in the wild
Detailed or minor comments
-
In Figure 1, the meaning of "Representation" on the Y axis is unclear until much later in the paper.
-
Figure 3: include a legend/colorbar and subtitles to the panels (shifted/unshifted/...) to make reading easier (rather than stating everything in the caption).
-
Section 5 title: use singular noun "Discussion".
-
The end of the "Discussion" and "Conclusion" sections are near-identical. Consider merging them or removing some of the redundancy.
-
Isn't the proposed "adaptive shifting" identical to a layer normalization applied after the non-linearity???
Questions for authors
I mixed in the questions with the statement of weaknesses above. No need to reply to every each of them. Feel free to focus on the parts that I may have misunderstood, and those that are most critical. ⬇️
The current recommendation for "weak rejection" is because of the unanswered questions state above, in particular the fact that the story doesn't totally add up (why has this been overlooked until now?) and that the paper doesn't quite close the loop (what to do with these findings?). It's a valuable exploration and certainly a great workshop paper, but it seems to be missing a few things to reach the bar of the main conference.
3 comments
Reply to Reviewer ByRS (Part 2)
Weakness 3
We thank the reviewer for this valuable question about practical utility, and we address it in three points.
First, compared with an additive constant, which may be hard to choose directly, the adaptive shifting used in our scale-up experiments is a plug-and-play intervention that can improve performance and potentially reduce compute. In Appendix F.4, we estimate that for GPT-2 it could save up to one-third of the original compute. Due to our limited compute budget, we could not test adaptive shifting on larger models and tasks. This leaves a valuable opportunity for practitioners, as the operation is extremely simple and could be broadly helpful.
Second, reducing activation bias may also benefit optimizers. Our work shows that a simple change in activation bias produces substantial differences in learning dynamics: it reduces overshoot, smooths learning, and is associated with a less sharp loss landscape. Since some optimizers struggle on poorly conditioned landscapes, reducing activation bias may benefit them and widen the range of usable optimizers. These optimizers should therefore be re-tested when activation bias is reduced.
Third, we do not claim that new activation functions should be designed immediately. Rather, we argue for a more complete understanding of activation functions, so that these properties can be fully taken into account in future activation design. Bias is only one property of an activation function. The same approach—isolating a property and tracing its effect on training dynamics—could be applied to post-activation variance, derivatives, or saturation. Studying these properties separately and then jointly may support a more complete understanding and guide the design of stronger activation functions in the future.
Additionally, we thank the reviewer for pointing us to Teney et al. (2026), Ducotterd et al. (2024), and Teney et al. (2025). We agree that learnable activation functions are a highly relevant line of work within the field of activation functions that we should acknowledge. We have therefore added these references and discussed them in the related-work section. We believe learnable activation functions hold strong potential, particularly once we have accumulated a sufficient understanding of activation properties, so that these inductive biases can be incorporated into learnable activation designs.
We apologize again for not clarifying the significance of this paper in the discussion section of the original manuscript. We have therefore revised this section thoroughly to clearly state the utility and limitations of our work. Please refer to the updated PDF.
Detailed or minor comments
We thank the reviewer for these detailed and constructive suggestions on clarity and presentation. We agree that they improve the paper, and we have revised all of these details accordingly. Please refer to the updated PDF.
Concerning the question of whether the proposed adaptive shifting is identical to layer normalization applied after the nonlinearity, the two operations are similar but not identical: adaptive shifting only removes the mean and does not rescale the variance, nor does it include the affine parameters that LayerNorm contains.
We agree that comparing these similar operations is necessary, especially since some prior work has proposed post-activation LayerNorm. We have therefore added Section 5 to compare them. The results show that adaptive shifting, together with the fixed offset, outperforms the post-activation LayerNorm counterpart. This further indicates that the variance of activation functions warrants further investigation, rather than being simply set to 1 or controlled by the learnable γ in LayerNorm.
Still unconvinced
Overall I'm still incredulous that this basic fact has both been overlooked until now and that a trivial fix provides consistently better performance.
Concerning the question of whether the proposed adaptive shifting is identical to layer normalization applied after the nonlinearity, the two operations are similar but not identical: adaptive shifting only removes the mean and does not rescale the variance, nor does it include the affine parameters that LayerNorm contains.
This means that the proposed adaptive shifting is strictly a subset of what LayerNorm already does. So what is proposed is essentially a LayerNorm with frozen bias/scale (?!). I see that another review (nLsZ) also pointed this out above, and it is also confirmed in the authors' reponse as I understand.
I also noted other relevant prior work that was originally overlooked by the authors. I appreciate the efforts in the response and the the patching up of paper with additional experiments and discussion, but it now seems that the story should have built by better putting up first the existing knowledge and highly-related methods.
In summary I'm not updating my negative assessment because:
- I don't think the paper starts with a good representation fo the current state of knowledge
- I don't think the implications and utility of the paper are clear enough (the response to my question has a lot of text but little substance ("may also benefit optimizers", "we argue for a more complete understanding of activation functions", ...). I still don't know what to do with this paper. If the authors think this is important for the choice of optimizer or for the design of activation functions, then we need some concrete hint that this is indeed the case. In its current state, this is fine as a workshop paper but does not pass the bar for the main conference,
Reply to Reviewer ByRS (Part 1)
We thank the reviewer for the careful reading and professional technical questions. We reply to the questions and concerns point by point below.
Weakness 1
We thank the reviewer for this valuable question.
We were indeed surprised by the clear effect and by the fact that this had not been investigated previously. We suspect this is because the research approach adopted here has been overlooked and underestimated. Large language models contain many components and complicated mechanisms, and are trained on complex tasks. These factors can interfere with one another, mask the effect of activation bias, and even produce inconsistent observations. In this work, we instead propose a simple but well-controlled teacher–student model that removes irrelevant elements, isolates the effect of activation bias, and thereby reveals consistent improvements. We believe this "do subtraction" research approach is powerful but has been underused by the community, which may explain why the effect has been overlooked.
We've also discussed this point in Section 7 of the updated paper.
Weakness 2
We thank the reviewer for this question. We apologize for any confusion caused by our use of "overshoot" and the related figures.
Here, "overshoot" does not imply excess momentum from a poorly tuned optimizer. It describes the non-monotonic learning trajectory of the model. In Section 2.2, we track the mean of the residual-stream hidden state as a scalar summary of its evolution. Combined with Section 3, the unshifted model initially exploits the activation bias component as a shortcut: this component carries task-relevant information early in training and allows the model to reduce the loss quickly. As training proceeds, the model finds this shortcut limited and gradually discards the information in the bias component. Because the output in our controlled model is directly determined by the hidden state, we expect the same non-monotonic behavior to appear in the hidden-state trajectory. This rise-and-fall pattern is what we term overshoot. Indeed, the overshoot in Section 2.2 coincides with both the early-advantage stage of the unshifted model and the reconstruction results in Section 3.
Regarding Figures 3a and 3b, the point is that the unshifted model first dashes along a steep direction and then makes a sharp turn (Figure 3a). Figure 3b visualizes the shifted model's trajectory on the same plane, where its loss varies little—showing that its early updates develop largely outside this direction. This motivated us to construct a complementary plane (Figure 3c), on which the shifted model exhibits no such "dash-then-turn" behavior and instead follows a smooth trajectory toward lower loss. The comparison in Figure 3 therefore shows that activation bias induces a sharp direction: the model rapidly dashes along it to gain an early advantage, after which it makes a sharp turn and optimization slows. Reducing activation bias instead yields a smooth trajectory pointing toward the basin. The small changes at the last step of these figures are minor and not our emphasis.
Overall, the emphasis of "overshoot," Figure 3, and Section 3 is that activation bias makes the learning trajectory non-smooth and non-monotonic. We model this from three complementary angles: the hidden-state trajectory, the weight trajectory, and the trajectory of encoded information. Since the two models use the same Adam optimizer, this issue does not come from the optimizer but from activation bias itself.