acceptodds
Under review as a conference paper at ICLR 2027

A Thresholded ReLU Unit with Fixed Output Weight Goes Dormant under Target Drift: Phase Diagram and Entry Time

Abstract

Networks tracking moving targets lose ReLU units to dormancy: units active on almost no input get almost no gradient. Lacking a target-speed parameter, existing dormancy theory explains why such units stay dormant, not when they become dormant. We study one ReLU unit with a learned threshold and fixed output weight, trained online on a teacher whose weight vector random-walks on the sphere at fixed speed. In the large-dimension limit, its weight length, teacher overlap and threshold obey three differential equations with closed-form resting states. We map the boundary between tracking and dormancy over drift speed and learning rate: at slow drift the tracking state collides with an unstable state and both vanish; at fast drift it survives but is dormant. With nothing fitted, the equations predict where and when dormancy sets in, and simulations match once finite-dimension fluctuations are kept. These equations give the mechanism: drift keeps an error alive, at rates causing dormancy each update's squared length inflates the weight vector, so the threshold must outgrow the weight length. Since this inflation vanishes with the learning rate, we prove an isolated unit in the large-dimension equations needs a rate bounded away from zero for dormancy. Yet one of two units sharing one output error goes dormant where the isolated unit never does. In both settings a trained output weight settles at a small scale and prevents dormancy everywhere tested.

Primary area:
learning theory
Keywords:
loss of plasticity, dormant neurons, drifting teacher, online learning, statistical mechanics of learning, saddle-node bifurcation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eq5zOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper studies a single ReLU unit with a learned threshold and a fixed output weight, trained by online SGD to track a teacher unit that drifts at a fixed drift speed. It analyzes when such a unit goes "dormant", i.e., when its threshold becomes so large that the unit is active on almost no inputs and therefore receives almost no gradient. In the limit of large input dimension, the dynamics reduce to three coupled ODEs describing the evolution of the squared weight length, the overlap of weight length with the teacher, and the threshold. Through bifurcation analysis, the authors chart the boundary between tracking and dormancy as a function of the learning rate. At slow drift, they show that it is rules by a saddle-node bifurcation. They also predict the time to dormancy above this boundary.

The equations permit to extract a mechanism for dormancy. 1. drift keeps the unit misaligned with the teacher, so the error never vanishes. 2. at sufficiently large learning rates, each stochastic step then inflates the weight vector. 3. a longer weight vector makes the unit output overshoot the teacher on the inputs where it is active. 4. the threshold update counters by rising until the unit is dormant. Because the inflating term vanishes with the learning rate, the authors prove that the isolated unit needs a learning rate bounded away from zero to go dormant.

Extensions beyond the isolated unit show that a trained output weight settles at a small scale and prevents dormancy at every cell tested in the one-unit settings, that in a 128-unit layer (input dimension 32) dormant units shrink their weight norms rather than inflating them, so the single-unit diagram mispredicts which settings go dormant, and that at input dimension 512 no unit goes dormant at all.

Critical strengths
  • S1. The paper asks the open question of how dormancy depends on target drift speed and learning rate and addresses it in a minimal solvable setting.
  • S2. The control conditions are appropriately chosen to separate accumulated drift from per-step target noise and from residual magnitude.
Critical weaknesses

W1. Clarity and self-containment. The main text is very difficult to read and is not self-contained, and does not meet the standard of a conference paper. Overall, it made the paper very difficult to read. For instance:

  • The last paragraph of Introduction as well as many paragraphs of Sections 4–5 report numerical results whose meaning depends on definitions or protocols given only later in the text or solely in the appendix.
  • Some sentences cannot be interpreted from the main text, e.g. "Two readings fixed in advance failed: the measured crossing has no interior minimum through the end point" (Section 5.2), where it is never explained why such a minimum was expected.
  • Several terms are used without definition in the main text ("cell", "signature", "readings", "converted rate", the "leading-order scheme" as opposed to the diffusion approximation).
  • Section 5.1 shows criteria counts (93 criteria, 46 met, 26 not, 21 qualified) without stating the criteria themselves.
  • The appendices are largely a chronological record of individual runs and reruns (seeds, run counts, horizons, intermediate readings) rather than an organized argument.

W2. Relevance of the one-unit mechanism to networks. The title, abstract and motivating question center on the isolated-unit mechanism (norm inflation through the η² term, outrun by the threshold), but the paper own results indicate that this mechanism does not carry over to more practical settings:

  • The paper shows that a trained output weight prevents entry at every tested cell.
  • In the 128-unit layer, dormant units keep chance-level alignment and shrinking norms, the opposite of the one-unit mechanism.
  • At input dimension 512, no unit goes dormant. The authors acknowledge that the layer "enters another way" and point to the two-unit competition mechanism. But that mechanism is only analyzed for two units with fixed output weights, and a trained output weight also breaks this mechanism. The paper framing should therefore be revised: either the contribution is presented as the analysis of an one-unit mechanism that does not explain dormancy in networks, or the competition mechanism should be developed as the main result.

W3. Incomplete description of the bifurcation structure (Figure 1). At δ = 0.3 the right panel of Figure 1 shows the resting-state branch turning a second time at a second saddle-node bifurcation (the small open circle) after which it is stable again. However, neither the caption nor the text identifies this second bifurcation or discusses the stable dormant branch beyond it. The caption ("all stable but the arm above the ringed fold") is ambiguous, since the top arm also lies above the "ringed fold" and is stable, and the marking of the unstable arm is barely visible. The plotted dormant arm is also truncated near m ≈ 2.8, although it continues beyond the bifurcation. As a consequence, the bistability between the two bifurcations is never discussed: a stable dormant state exists below the reported boundary, which from the trained state is not reached deterministically but can be reached from other initial conditions or by noise-driven escape across the saddle. Appendix D.1 also states that no fixed point with ρ ∈ (0, 1) exists just above the fold at δ ≤ 0.3, which this branch contradicts.

Questions for authors
  • Q1. Is there any realistic regime (trained readout, minibatch training) in which the one-unit mechanism operates? If not, what is the intended takeaway of the paper?
  • Q2. A trained output weight breaks the two-unit channel mechanism too. What then produces dormancy in the layer, which trains its readout?
  • Q3. Can the authors provide a complete description of the bifurcation structure shown in Figure 1, and discuss its potential impact on the interpretation of the results?
Reviewer_fsqpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates dormant neuron problems under nonstationary learning targets. The authors use an online teacher–student setting in which a thresholded ReLU student tracks a teacher whose weight vector undergoes random drift. Using this setting, the paper characterizes the boundary between tracking and dormancy as a function of drift speed and learning rate and predicts dormancy-entry times. Further experiments investigate trainable weights with 128 neurons and reveal both additional mechanisms and important limitations of transferring the isolated-unit predictions to larger networks.

Critical strengths

With the rapid development of AI, I greatly appreciate efforts to develop theoretical frameworks for understanding AI from a statistical physics perspective, and this paper represents one such effort. The paper develops a clear theoretical framework for neuron dormant phenomena in continual learning, investigating how target drift speed and learning rate jointly determine the conditions for dormancy and its onset time. The authors identify an analytically tractable model and provide a relatively complete theoretical analysis with quantitative predictions. The identification of two different dormancy mechanisms is also particularly interesting.

Critical weaknesses

My main concerns are as follows:

  1. I understand that simplifications are necessary to obtain an analytically tractable model. However, the theory here relies heavily on fixed output weights, a restrictive assumption that differs from how neural networks are typically trained. More importantly, the explaination of dormancy appears to depend strongly on this assumption, raising concerns about their relevance to realistic networks.
  2. Although the single-neuron theory is well developed, the paper's own network model exhibits substantially different behavior and dormancy mechanisms. This discrepancy raises concerns about the generalizability of the single-neuron theory results.
  3. Although the paper aims to explain neuronal dormancy in continual learning, its theoretical predictions are not validated on any standard continual learning benchmarks.
Questions for authors

see weakness

Reviewer_3xKXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies when a thresholded ReLU unit becomes inactive while tracking a changing target. The authors derive equations for its learning process, map the boundary between active and inactive states, and predict the time to become inactive. Tests with two units and a wider network show the limits of the single-unit model.

Critical strengths
  1. A clear and useful question: how target changes and learning rate affect unit activity. An explicit mechanism linking tracking error, weight growth, and a rising activation threshold.
  2. Careful tests within the proposed model, with accurate timing predictions after accounting for random fluctuations. Clear reporting of failed predictions and limits in larger networks.
Critical weaknesses
  1. Strong reliance on a fixed output weight. No dormancy in any tested one-unit or two-unit case with learned output weights. Unclear relevance of the main mechanism to networks where both layers are trained.
  2. Poor transfer to the wider network. Shrinking weights in inactive network units, opposite to the growth in the single-unit model. Large errors in predicted entry times and learning-rate thresholds. The two-unit analysis offers a possible explanation, but no comparable predictions for the wider network.
  3. An unclear link between low activity and loss of learning ability. Dormancy defined by crossing an activity cutoff, despite later crossings back and full recovery in all tested single-unit cases after drift stops. More direct tests of reduced learning ability needed.
  4. Heavy reliance on numerical calculations for the phase boundary. Explicit formulas for fixed points, but no full proof that training reaches them. The small-learning-rate theorem applies at each fixed drift speed; no proof of the reported minimum threshold across all speeds.
Questions for authors

N/A

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the author response and of the changes in the revised PDF

We thank the three reviewers. This comment summarizes our replies and lists the main changes in the revised PDF. New text is in blue (8 and 10 November) or red (14 November).

(a) Points the reviewers state

  • Reviewer 3xKX: "An explicit mechanism linking tracking error, weight growth, and a rising activation threshold."
  • Reviewer fsqp: "The identification of two different dormancy mechanisms is also particularly interesting."
  • Reviewer eq5z: "The control conditions are appropriately chosen to separate accumulated drift from per-step target noise and from residual magnitude."

(b) Concerns raised by more than one reviewer

  1. The fixed output weight (3xKX, fsqp, eq5z). Section 4.1 states that a unit with fixed output weight v>0v>0 behaves exactly like one at learning rate v2ηv^2\eta. An output weight trained by plain gradient steps at the weight vector's rate settles at 0.14 to 0.16, and no run enters at four cells where every run with v=1v=1 enters; held fixed at 0.15, the unit is the original one at rates 0.018 to 0.027, below the boundary. The unit enters only if that weight learns 100 to 1000 times more slowly, at three of the four cells (Table 6). The contribution is now stated for units at a fixed output scale. Replies: 3xKX, comment 1; fsqp, comment 1; eq5z, comment 1.
  2. The 128-unit layer (3xKX, fsqp, eq5z). The layer lacks the isolated unit's norm inflation, and its units enter about 4.5 times later than predicted; the revision claims no quantitative single-unit prediction for it. It shares features of competition through a shared output error: 96% of its 708 dormant units at (δ,ηraw)=(0.6,0.03)(\delta,\eta_{\rm raw})=(0.6,0.03) shrink their norms, and with output weights frozen it still loses 36 and 38% of its units at the large raw step. New runs: with output weights fixed at 1, units enter and shrink at K=4K=4 and 8; trained from a small random start, the output weights keep every unit active at K=2K=2, 8 and 32 at the rates tested, so this channel does not account for the layer's dormancy (Appendix D.8). Replies: 3xKX, comment 3; fsqp, comments 2 and 3; eq5z, comment 3.
  3. Proofs and the bifurcation structure (3xKX, eq5z). Proposition 3 gives the resting states with alignment in (0,1)(0,1) in closed form; the fold is found numerically; Table 1 marks each statement as proven or computed. Proposition A.16 proves that below a positive rate the trajectory from the trained state converges without entry, at each drift speed, not uniformly in the speed; 0.554 is computed for speeds up to δ=5\delta=5. New Proposition A.10 proves by interval arithmetic that the cutoff curve stays above 0.55397 at every positive drift speed. At δ=0.3\delta=0.3 the branch folds twice, and between the folds a stable dormant state coexists with the tracking state. Sections 4.2 and 4.4 state the boundary as the fate of the trained state, so that dormant state does not move it. Replies: 3xKX, comment 2; eq5z, comment 2.

(c) Changes in the revised PDF

  1. Abstract, p. 1, and Section 1, p. 3: the trained output weight is narrowed to plain gradient steps at the weight vector's rate and a start at one (3xKX, fsqp, eq5z).
  2. Section 1, p. 2: Figure 1 (right) dashes the unstable arm, rings both folds and labels the second; new caption (eq5z).
  3. Sections 1 and 4.2, pp. 2 and 5: Question 3 asks what survives a second unit, a trained output weight and a layer (3xKX, fsqp, eq5z).
  4. Section 4.1, p. 5: fixed output weights are the soft committee machine setting (Straat et al., 2022) (3xKX, fsqp, eq5z).
  5. Sections 4.4 to 5.4, pp. 6 to 10: terms defined and protocols stated where used; Section 5.1 points to Table 12; a Section 5.2 sentence rewritten (eq5z).
  6. Sections 5.4 and 6, pp. 9 and 10: Section 5.4 retitled; Section 6 limits 0.554 to the trained state and δ≤5\delta\le5 (3xKX, fsqp, eq5z).
  7. Appendix A.5, pp. 20 and 21: new Proposition A.10 and Remark A.11 (3xKX).
  8. New Appendix A.9, pp. 35 to 38: equations of KK units sharing one output error (3xKX, fsqp, eq5z).
  9. Appendices B.7, p. 42, and D.1, pp. 49 and 50: the fixed-point search, its protocol and what it returned; a basin-sweep clause removed (eq5z).
  10. Appendix D.2, pp. 54 and 55: recovery and relearning tests (Tables 3 and 4) (3xKX).
  11. Appendix D.3, pp. 55 to 57 and 59 to 61: a slower output weight (Table 6) and intermediate models of the layer (new Table 8) (3xKX, fsqp, eq5z).
  12. Appendix D.4, p. 65: Adam (Table 11) (fsqp, eq5z).
  13. New Appendix D.6, pp. 68 to 70: MNIST with rotating inputs (Table 13) (fsqp).
  14. New Appendix D.7, pp. 69 to 71: runs started at the dormant resting state (Tables 14 and 15) (eq5z).
  15. New Appendix D.8, pp. 72 to 74: simulations of KK units sharing one output error (Table 16) (3xKX, fsqp, eq5z).
  16. Section 1, pp. 1 to 3: symbols, terms and the three extension models defined at first use, in red (eq5z).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.