acceptodds
Under review as a conference paper at ICLR 2027

Activation tails select suboptimal profiles in expressive networks

Abstract

ReLU homogenization removes Gaussian activation tails, but those tails can still determine the leading parameter profile of a positive-risk training trajectory. For any width m ≥ 4 of the hidden layer and smoothing parameter 0 < τ ≤ 1/5, we assume a bias-free one-hidden layer neural network with Gaussian-smoothed ReLU activations. All the weights and readouts are trained via population logistic gradient flow on the eight-point binary classification task. A nonempty open set of sufficiently large finite initializations gives rise to trajectories that misclassify two points forever. Both the smooth network and its width-m homogeneous ReLU version are able to classify all eight points. However, the loss for the trajectories of the smooth network tends to a positive, non-optimal population loss. The parameter vector, normalized by √log(t + t₀), converges to one point on an explicitly constructed sphere of local constrained norm minimizers. Four constraints correspond to the margins obtained from the points that were correctly classified, while four constraints are related to the activation tails of the two points that were misclassified. The absence of the latter four constraints yields a different norm-minimizing parameter profile. Under the same Euclidean parameter metric and common layer learning rate, these Gaussian parameter profiles are incompatible with the neuronwise balance invariant of homogeneous ReLU gradient flow. Different smoothing parameters yield the same ReLU homogenization but different leading log(t + t₀)-normalized logits at a specified off-support probe, and different limiting ratios to a reference training logit. The bad basin and profile convergence property are preserved under sufficiently small perturbations of each input coordinate independently and arbitrary positive probability weights.

Primary area:
learning theory
Keywords:
Activation tails, neural network homogenization, implicit bias, gradient flow, ReLU networks, logistic loss, asymptotic training dynamics, training failure

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PARkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper constructs a counterexample showing that Gaussian activation tails can affect the asymptotic training dynamics of neural networks, even though they vanish under ReLU homogenization. They further extend the result to gradient descent and SGD, and show that adding a trainable output bias can eliminate the classification failure in their construction.

Critical strengths

Not applicable.

Critical weaknesses

The motivation of this paper is unclear. Why should we expect the homogenization of an activation function to determine the asymptotic parameter profile of gradient flow, particularly when training fails to fit the data? I am not convinced that the paper addresses an important or well-motivated question.

The presentation is unnecessarily complicated and overly defensive. Even the opening sentence of the abstract introduces several technical concepts without explaining the underlying problem. The extensive mathematical machinery makes it difficult to identify the central insight and why it matters.

Questions for authors

What is the main motivation for studying this particular failure case? What broader insight about neural network training can we learn from such a specialized construction?

Reviewer_RcA7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies population logistic gradient flow for a two-layer neural network with Gaussian-smoothed ReLU activation on an eight-point binary classification problem. Theorem 1 establishes the existence of a full-dimensional open set of initializations for which gradient flow permanently misclassifies two samples and converges to a strictly positive loss. The parameters diverge, while their normalized direction converges to an explicitly characterized direction defined by (eq-4). The authors compare this profile with homogeneous ReLU gradient flow (Corollary 2), show that different smoothing parameters yield different leading logits (Corollary 3), and extend the construction to GD and SGD. They also demonstrate that adding an output bias can repair the classification failure.

Critical strengths

The paper appears to contain substantial technical work, particularly in establishing a full-dimensional attracting region, characterizing the limiting parameter directions, and extending the analysis to discrete GD and SGD.

Critical weaknesses
  1. The paper claims that the gaussian activation tail (whose definition is not explicitly given in the paper) is what makes GF with the Gaussian-smoothed ReLU networks converges to a direction with suboptimal loss. However, Corollary 2, which provides comparison with homogeneous ReLU networks, does not rule out the possibility that GF with a homogeneous ReLU network also converges to the direction which achieves positive loss. What Corollary 2 claims is much narrower. Therefore, their claim that the gaussian activation tail is the key does not appear to be convincing. Moreover, unsuccessful training is obviously not unique to Gaussian-smoothed ReLU networks. Ordinary ReLU can also exhibit optimization failure, which further question the validity of their claim that the activation tail is the key mechanism behind convergence to a positive loss.
  2. Corollary 3 seems to claim that although all \phi_\tau have the same ReLU homogenization, GF converges in different direction for different values of \tau. However, I don't see the reason why we should expect to converge to the same direction just because they have the same ReLU homogenization. It is not surprising to have different directions since \phi_\tau obviously depends on the value of \tau. It might be an interpretation issue, but neither preceding and subsequent sentences seems to provide clear interpretation of the result.
  3. The paper is extremely hard to follow. The paper does not provide sufficient explanation for the technical notations they introduce and for their main results. For example, Section 2 introduces a coordinate transformation and quite a bit of technical notations at the middle of the section without explaining the purpose of them. The fact that the network can perfectly classify all 8 samples, which is important to appreciate the significance of Theorem 1, is only established briefly and vaguely at the end of Section 3. The term "activation tail", which appears to be their central object of study, is neither explicitly defined nor explained. Equation 7 introduces "activation-tail rate" which is even more difficult to see what it exactly means. Lastly, Corollary 3 is especially difficult to interpret.
Questions for authors
  1. Can you clarify my concern with Theorem 1 and Corollary 2 raised above?
  2. Can you clarify the significance of Corollary 3?
Reviewer_G4esOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the implicit bias and optimization dynamics of two-layer neural networks with Gaussian-smoothed ReLU activations. The central contribution is a theoretical counterexample demonstrating that activation tails, despite vanishing under homogenization, can fundamentally influence the asymptotic parameter profile selected by gradient-based optimization.

Critical strengths

The paper concerns a special angle of non-homogeneity and activation changes. It moves forward the conceptually interesting comparison and gives a technically substantial characterization of optimization trajectories with suboptimal loss. It also extends the discussion to finite-step GD and SGD.

Critical weaknesses
  1. The paper does not prove that ReLU avoids a suboptimal basin for the optimization setting. It only proves that ReLU gradient flow cannot converge to the same asymptotic parameter profile as the smoothed-ReLU network. ReLU could still exhibit suboptimal training behavior. This weakens the discussions and comparisons w.r.t. the effect of homogeneity.

  2. The paper is built on a specific construction, and require large enough initialization, which is also different from practice. The claim of the paper is limited to a specific setting and lacks linkage to general implicit bias phenomena or training practices.

Questions for authors
  1. What happens when a ReLU network is trained from the same large initializations? Does it successfully classify all samples, or can it converge to another positive-risk trajectory?
  2. Can the results be extended to other smooth activations with exponentially decaying tails, such as softplus? Which properties of the Gaussian tail are essential to the proof, and which are merely technical conveniences?
  3. Can a similar failure be constructed for more general networks and data settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.