- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper studies a single ReLU unit with a learned threshold and a fixed output weight, trained by online SGD to track a teacher unit that drifts at a fixed drift speed. It analyzes when such a unit goes "dormant", i.e., when its threshold becomes so large that the unit is active on almost no inputs and therefore receives almost no gradient. In the limit of large input dimension, the dynamics reduce to three coupled ODEs describing the evolution of the squared weight length, the overlap of weight length with the teacher, and the threshold. Through bifurcation analysis, the authors chart the boundary between tracking and dormancy as a function of the learning rate. At slow drift, they show that it is rules by a saddle-node bifurcation. They also predict the time to dormancy above this boundary.
The equations permit to extract a mechanism for dormancy. 1. drift keeps the unit misaligned with the teacher, so the error never vanishes. 2. at sufficiently large learning rates, each stochastic step then inflates the weight vector. 3. a longer weight vector makes the unit output overshoot the teacher on the inputs where it is active. 4. the threshold update counters by rising until the unit is dormant. Because the inflating term vanishes with the learning rate, the authors prove that the isolated unit needs a learning rate bounded away from zero to go dormant.
Extensions beyond the isolated unit show that a trained output weight settles at a small scale and prevents dormancy at every cell tested in the one-unit settings, that in a 128-unit layer (input dimension 32) dormant units shrink their weight norms rather than inflating them, so the single-unit diagram mispredicts which settings go dormant, and that at input dimension 512 no unit goes dormant at all.
Critical strengths
- S1. The paper asks the open question of how dormancy depends on target drift speed and learning rate and addresses it in a minimal solvable setting.
- S2. The control conditions are appropriately chosen to separate accumulated drift from per-step target noise and from residual magnitude.
Critical weaknesses
W1. Clarity and self-containment. The main text is very difficult to read and is not self-contained, and does not meet the standard of a conference paper. Overall, it made the paper very difficult to read. For instance:
- The last paragraph of Introduction as well as many paragraphs of Sections 4–5 report numerical results whose meaning depends on definitions or protocols given only later in the text or solely in the appendix.
- Some sentences cannot be interpreted from the main text, e.g. "Two readings fixed in advance failed: the measured crossing has no interior minimum through the end point" (Section 5.2), where it is never explained why such a minimum was expected.
- Several terms are used without definition in the main text ("cell", "signature", "readings", "converted rate", the "leading-order scheme" as opposed to the diffusion approximation).
- Section 5.1 shows criteria counts (93 criteria, 46 met, 26 not, 21 qualified) without stating the criteria themselves.
- The appendices are largely a chronological record of individual runs and reruns (seeds, run counts, horizons, intermediate readings) rather than an organized argument.
W2. Relevance of the one-unit mechanism to networks. The title, abstract and motivating question center on the isolated-unit mechanism (norm inflation through the η² term, outrun by the threshold), but the paper own results indicate that this mechanism does not carry over to more practical settings:
- The paper shows that a trained output weight prevents entry at every tested cell.
- In the 128-unit layer, dormant units keep chance-level alignment and shrinking norms, the opposite of the one-unit mechanism.
- At input dimension 512, no unit goes dormant. The authors acknowledge that the layer "enters another way" and point to the two-unit competition mechanism. But that mechanism is only analyzed for two units with fixed output weights, and a trained output weight also breaks this mechanism. The paper framing should therefore be revised: either the contribution is presented as the analysis of an one-unit mechanism that does not explain dormancy in networks, or the competition mechanism should be developed as the main result.
W3. Incomplete description of the bifurcation structure (Figure 1). At δ = 0.3 the right panel of Figure 1 shows the resting-state branch turning a second time at a second saddle-node bifurcation (the small open circle) after which it is stable again. However, neither the caption nor the text identifies this second bifurcation or discusses the stable dormant branch beyond it. The caption ("all stable but the arm above the ringed fold") is ambiguous, since the top arm also lies above the "ringed fold" and is stable, and the marking of the unstable arm is barely visible. The plotted dormant arm is also truncated near m ≈ 2.8, although it continues beyond the bifurcation. As a consequence, the bistability between the two bifurcations is never discussed: a stable dormant state exists below the reported boundary, which from the trained state is not reached deterministically but can be reached from other initial conditions or by noise-driven escape across the saddle. Appendix D.1 also states that no fixed point with ρ ∈ (0, 1) exists just above the fold at δ ≤ 0.3, which this branch contradicts.
Questions for authors
- Q1. Is there any realistic regime (trained readout, minibatch training) in which the one-unit mechanism operates? If not, what is the intended takeaway of the paper?
- Q2. A trained output weight breaks the two-unit channel mechanism too. What then produces dormancy in the layer, which trains its readout?
- Q3. Can the authors provide a complete description of the bifurcation structure shown in Figure 1, and discuss its potential impact on the interpretation of the results?
Summary of the author response and of the changes in the revised PDF
We thank the three reviewers. This comment summarizes our replies and lists the main changes in the revised PDF. New text is in blue (8 and 10 November) or red (14 November).
(a) Points the reviewers state
(b) Concerns raised by more than one reviewer
(c) Changes in the revised PDF