acceptodds
Under review as a conference paper at ICLR 2027

Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks

Abstract

Weight decay is widely used as a regularization technique in neural network training, yet its role in neuron condensation (parameter direction alignment) remains unclear. Starting from a parameter initialization in the neural tangent kernel regime, we characterize training dynamics under weight decay through three stages: rapid fitting, amplitude compression, and neuron condensation. Using a two-layer ReLU network, we analyze a residual correlation field that governs both neuron amplitudes and directions. During rapid fitting, the residual approaches a quasi-static equilibrium maintained by weight decay while the tangent kernel remains nearly unchanged. In the early stage of amplitude compression, kernel decay amplifies the residual correlation field, whose isolated attracting extrema provide candidates for condensation directions. As neuron amplitudes stabilize, we bound the drift of attracting extrema and demonstrate contraction of neuron directions around them, leading to neuron condensation. This staged analysis provides a dynamical understanding of how weight decay promotes a condensed representation, beyond reducing parameter norms.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Two Layer Network, Weight Decay, Neuron Condensation, Training Dynamics

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_NMJUOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors explore the condensation phenomenon in two-layers ReLU networks when weight decay is employed. They separate training into three stages: fitting, compression, and condensation. Through an evaluation of the evolution of the weights’ amplitude and angle, they show mostly theoretically how weight decay with high initialisations is usually linked with higher levels of condensation.

Critical strengths
  1. S1 - Theoretical derivation: Most of the results are strongly backed by theoretical derivations and proofs. I have not read them in detail, in particular the parts in the Appendix, but everything I have read made sense. The main claims in the paper are appropriately backed by the theoretical results.
  2. S2 - Clarity and presentation: Although the paper mostly consists of theoretical derivations, the explanation and discussion of the research performed is very clear. The goal of the paper is easy to understand.
Critical weaknesses
  1. W1 - Significance: This paper focuses on the simplest example of two-layers ReLU neural networks representing very simple piecewise linear functions. In the Appendix, the authors explore a few more settings experimentally. The Transformer experiment somewhat addresses this issue and demonstrates that the authors’ results are empirically verified, but to do so they have to use a very large weight decay factor of $0.8$. All things considered, these results are interesting, but it is unclear how strong the condensation effect of weight decay would be in more realistic settings.
Questions for authors
  1. Why have the authors used such a high $L_2$ weight decay factor in their Transformer experiment? What would their results look like with more common values that are closer in order of magnitude to the learning rate?
  2. I have not clearly understood where the different stages, and the boundaries between them, are precisely defined. In Section 4, how exactly is it defined when to go from one stage to the next one? In particular, between stages II and III.
3 comments

We sincerely thank you for your careful review and constructive feedback. We have made the corresponding revisions in the manuscript. Below, we address your concerns point by point.

1. Choice of the weight decay coefficient (Weakness 1, Question 1)

We would like to clarify that the weight decay coefficients adopted in large language model training do not need to share the same magnitude as the learning rate. For example, the DeepSeek-V3 (https://arxiv.org/pdf/2412.19437) uses a weight decay coefficient of 0.10.1. If the weight decay coefficient is set to the same order of magnitude as the learning rate, substantially longer training may be needed before condensation becomes observable.

We have added an experiment with a weight decay coefficient of 0.10.1 in Appendix H.6, which is closer to values used in practical training settings. With the training duration extended to 1,200 epochs, we observe condensation in this setting as well.

2. Boundaries between the three stages (Question 2)

We do not specify sharp boundaries between the three stages in the paper. Rather, the stages describe changes in the dominant behavior of the training dynamics.Theorem 5 shows that, once neuron amplitudes stabilize, the extrema of qe(θ)q_e(\theta) also stabilize. The subsequent angular dynamics then drive neuron directions to condense around these extrema. Accordingly, we use when neuron amplitudes reach a plateau as an empirical indication of the transition into Stage III.

Reviewer_NMJUOpenReview ↗

Response to the authors' rebuttal

I would like to thank the authors for their rebuttal. They have cleared up the questions I raised, and after reading through the other reviews and rebuttals, I believe that the critical weaknesses have been addressed. I will raise my score.

Thank you for taking the time to reconsider our work. We are glad that our rebuttal has addressed your concerns. We sincerely appreciate your careful evaluation and constructive feedback.

Reviewer_J53NOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the dynamical mechanism of neuron condensation in two-layer ReLU networks trained with weight decay under large initialization. By introducing a residual correlation field, the authors divide the training dynamics into three stages: rapid fitting, amplitude compression, and neuron condensation. Both theory and experiments demonstrate that weight decay not only reduces parameter norms but also profoundly drives neuron directions to align around attracting extrema, offering a compelling explanation for generalization phenomena such as grokking. Furthermore, this mechanism exhibits strong generalization to complex architectures like Transformers.

Critical strengths
  • The paper explains a novel mechanism. Weight decay appears to control only parameter amplitudes, but the paper shows that it makes the stationary data MSE nonzero, keeps the residual correlation field nonzero and amplifies it, and thereby reduces the attracting stationary directions of the angular dynamics to a few directions. This reminds us that weight decay affects generalization not only through norm shrinkage but also through forming condensed representations.

  • The separation into rapid fitting, amplitude compression, and angular condensation provides a coherent account of the training dynamics. In particular, the nonzero quasi-static residual under weight decay plays an important role in driving subsequent directional dynamics. This framework helps clarify the connection between changes in parameter magnitudes and the emergence of directional alignment.

  • The evolution of neuron amplitudes and directions in synthetic tasks is broadly consistent with the proposed three-stage framework. The Transformer experiments further suggest that weight decay and directional alignment may be relevant beyond the simplified theoretical setting. Although these experiments do not establish a complete causal relationship, they motivate further investigation into the generality of the mechanism.

Critical weaknesses
  • The main theoretical results rely on one-dimensional inputs, a two-layer ReLU network, and gradient flow. The one-dimensional geometry allows the residual correlation field to take a simple sinusoidal form within each activation cell, making the analysis of attracting directions and angular contraction tractable. However, the alignment observed in the synthetic and Transformer experiments suggests that the underlying mechanism may extend beyond one-dimensional models.
Questions for authors
  • The paper exploits the trigonometric structure of the residual correlation field within each activation cell to characterize isolated attracting directions in the one-dimensional setting. How sensitive are the number and stability of these attracting directions to the training inputs and labels? Under small perturbations of the training data, do the attracting directions vary continuously, or can extrema disappear, split, or change stability?

  • The Transformer experiment compares training with and without weight decay and observes alignment among rows of WQW_Q and WKW_K, together with improved test accuracy on held-out anchor pairs. Have the authors examined the temporal relationship between directional alignment and improvements in test accuracy? For example, does alignment emerge before the test accuracy begins to improve, or do the two evolve together? This could help clarify whether condensation may act as an intermediate mechanism for improved generalization rather than merely co-occurring with it.

3 comments

We sincerely thank you for your careful reading and thoughtful suggestions. We have revised the manuscript accordingly. Below are our detailed responses to each question.

1. Extension to high dimensional inputs (Weakness 1)

For scalar inputs, we use the angle θ\theta to parameterize a unit vector on S1S^1. The trigonometric representation itself is not essential to the mechanism. For higher-dimensional inputs, analogous results can be obtained by representing the direction directly as a unit vector u∈Snu\in S^n, rather than using the scalar angle θ\theta. The residual correlation field then takes the form q(u)q(u), and its extrema provide candidate condensation directions.

We have added Section 5 in the main text, experiments with higher-dimensional inputs in Appendix G.3, and a detailed theoretical analysis in Appendix I.

2. Sensitivity of attracting directions to the training data (Question 1)

We conducted experiments in which we perturbed the training data or added training samples. Perturbations affect the NTK, which in turn affects the quasi-static residual eqse_{\mathrm{qs}} and the residual correlation field q=⟨eqs,ϕ⟩q=\langle e_{\mathrm{qs}},\phi\rangle. We can establish the following bound on the displacement of an extremum:

∣δθ∗∣≤CX∥δX∥n+Cy∥δy∥n.|\delta\theta_*| \le C_X\|\delta X\|_n+C_y\|\delta y\|_n.

Adding training samples or introducing larger perturbations can change the shape of qq and lead to additional extrema. The experiments can be found in Appendix G.4

3. Condensation and test accuracy (Question 2)

We have added plots showing the evolution of accuracy on held-out anchor pairs alongside the degree of condensation. The corresponding revisions can be found in Appendix H.5. In the experiment, the increase in accuracy occurs concurrently with condensation. We quantify the degree of condensation using

C(W)=1m(m−1)∑i≠j∣wi⊤wj∣∥wi∥2∥wj∥2.C(W)=\frac{1}{m(m-1)}\sum_{i\ne j}\frac{|w_i^\top w_j|}{\|w_i\|_2\|w_j\|_2}.

where wiw_i are row vectors of WQW_Q and WKW_K.

Thank you for the detailed rebuttal. The responses have adequately addressed my concerns. I find the paper insightful and valuable, particularly in its analysis of how weight decay induces directional neuron condensation. Overall, I hold a more positive view of the work after the rebuttal and will increase my score.

Thank you for your positive feedback. We are glad that you find our analysis insightful and valuable. We sincerely appreciate your time and consideration, as well as your constructive comments, which have helped us improve the paper.

Reviewer_hYnGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how weight decay can promote neuron condensation in a two-layer ReLU network with scalar inputs. It develops a three-stage account: rapid fitting while the initial kernel changes little, amplitude compression that amplifies a residual correlation field, and angular contraction around attracting directions under the late-stage assumptions. The analysis is illustrated with synthetic regression examples and supplemented by observations of query/key weight alignment in a small Transformer on an anchor-function task.

Critical strengths

The angular analysis is interesting and provides a coherent account of how weight decay can affect neuron directions in addition to shrinking parameter magnitudes. The connection between the residual field, amplitude evolution, and directional alignment substantially deepened my understanding of the problem. I regard the analysis as broadly complete within its stated setting, subject to the clarification questions below. This understanding is the main reason for my positive recommendation.

My overall recommendation is weak acceptance. The theoretical account is valuable to me, while the toy examples limit the demonstrated scope and leave me less convinced about a stronger recommendation.

Critical weaknesses
  1. The empirical examples are predominantly toy settings. They help illustrate the proposed dynamics, but provide limited evidence about how broadly the conclusions apply. In particular, the Transformer observations suggest a related trend, while the theory concerns a two-layer ReLU MLP. The paper explicitly acknowledges that a corresponding Transformer theory remains open. I would appreciate further explanation or empirical evidence connecting the mechanism to self-attention and establishing how representative the observed trend is. This limitation is the main reason I lean toward weak rather than clear acceptance.

  2. The role of fixed activation patterns needs clarification when interpreting the analysis as an account of training. The early-time argument in Appendix B.2 controls activation changes, but Theorem 4 assumes fixed masks and Theorems 5–6 work within persistent open activation cells or bands. How activation changes affect the later mechanism remains a question for me. Under fixed masks, the predictions are bilinear in the two parameter factors, with sample- and neuron-dependent gates; this is not generally identical to an ordinary linear network. Nevertheless, I would like a clearer explanation of the specifically ReLU contribution beyond the corresponding gated linear dynamics. These are questions for rebuttal, rather than claims that the conditional theorems are false.

The presentation is generally understandable, but I repeatedly needed to move backward and forward to locate notation definitions. More local reminders of the important notation would make the analysis easier to follow.

Post-rebuttal update: Thank you for explaining the fixed-mask assumptions and how ReLU gating differs from the corresponding linear dynamics. The boundary-crossing experiment and the higher-dimensional extension strengthen the case for the mechanism in the settings studied. The Transformer results remain suggestive, and the proposed connection to softmax is an open question. I maintain my rating of weak acceptance (3).

Questions for authors
  1. Activation changes: Beyond the initial fitting interval, how should the fixed-mask and fixed-cell assumptions be understood along the actual training trajectory? Can further elaboration or empirical measurements of activation changes explain when the later analysis is applicable and how boundary crossings affect its interpretation?

  2. ReLU versus linear dynamics: Under the fixed-mask assumptions, what aspects of the mechanism and proof are specifically due to ReLU gating, beyond known two-layer linear-network dynamics? Please clarify this distinction while accounting for the sample-dependent masks; I am not asserting that fixed masks make the entire model an ordinary linear network.

  3. Transformer connection: Which parts of the proposed mechanism plausibly account for alignment of query/key weights in self-attention? Since a corresponding theory is explicitly left open, further explanation or empirical evidence across additional configurations would help assess whether the observations reflect the same mechanism. I am not requesting a general Transformer theorem during rebuttal.

  4. Relation to dropout condensation: Could you explain the substantive distinction from the earlier work on dropout-induced condensation, including which mechanistic insights or analytical steps are new beyond studying a different regularizer? I have not independently checked whether there is substantial overlap and do not assert that the works are duplicative.

I would like these points addressed during rebuttal. For the activation-change and Transformer questions, further elaboration or empirical evidence could be helpful.

1 comment

We sincerely thank you for your careful review and constructive suggestions. We address your concerns point by point below.

1. Activation changes (Weakness 2, Question 1)

If boundary crossings occur, the ReLU derivative changes discontinuously. Consequently, when a neuron crosses an activation boundary, the neural tangent kernel (NTK) and the angular energy P(t)P(t) can exhibit jumps. At initialization, the distribution of neurons is known, and neurons close to activation boundaries can cross them during early training. We therefore control these activation changes. Regarding the assumption of fixed activation patterns:

(a) Empirically, we conducted an experiment with 20 neurons and observed no activation boundary crossings after the neurons entered Stage II as illustrated in the accompanying figure.

Training stageEpoch rangeActivation boundary crossings
Stage I0–1,44712
Stage II1,447–≈31,9000
Stage III≈31,900–80,0000

(b) The assumption that neurons do not cross activation boundaries is particularly relevant to Stages II and III. Theorem 4 can be understood as a characterization of the instantaneous velocities at the onset of Stage II. This characterization is local in time and does not involve tracking boundary crossings over an extended interval. In Stage III, neurons condense around an approximately stationary attracting point within an activation cell. Once a group of neurons has entered such a cell, assuming that their activation patterns remain unchanged is reasonable in this setting.

2. ReLU versus linear dynamics (Weakness 2, Question 2)

Condensation can also occur in a two-layer linear network under suitable weight decay, in the sense that wk/bkw_k/b_k approaches a constant. The distinction lies in the structure of the residual correlation field q(θ)q(\theta).

For a linear network, this field is a continuous trigonometric function. With ReLU activations, it has a piecewise trigonometric form, with a continuous expression within each activation cell. This piecewise structure allows more extrema and, consequently, more candidate condensation directions.

The role of ReLU gating is to create activation cells, with neurons in different cells activating on different subsets of the training data. Within a given cell, the activation function can be removed on the active samples, so the neurons in this cell is like a two-layer linear subnetwork, and these subnetworks interact with one another.

3. Connection to Transformers (Weakness 1, Question 3)

We believe that softmax may play an important role in condensation in Transformers, consistent with the earlier work on condensation of attention (https://arxiv.org/abs/2605.01199). Our preliminary hypothesis is that softmax saturation may help explain how softmax interacts with weight decay to produce eventual condensation. A detailed analysis of this interaction remains an important direction for future work.

4. Relation to dropout condensation (Question 4)

Both that work and ours seek to explain neuron condensation. A central contribution of the dropout work is to show that dropout training dynamics resemble gradient flow on a loss with an additional regularization term involving the L2L_2 norm of the outputs evaluated at the training samples. For two-layer ReLU networks, that work explains condensation by showing that condensed solutions can attain a lower loss. Our work focuses on the training dynamics that lead to condensation: how the parameters progressively align during training.

5. Notation and presentation

We have added a table of notation in Appendix A to make the symbols and definitions easier to understand.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.