acceptodds
Under review as a conference paper at ICLR 2027

The Accuracy Well: Nonmonotone Accuracy Dynamics at Vanishing Initialization

Abstract

We study classification dynamics under small initialization and identify a pronounced non-monotone behavior of accuracy that we call the *accuracy well*: accuracy rises rapidly early in training, subsequently drops, and only much later recovers. We observe this phenomenon in deep linear networks, fully connected ReLU networks on MNIST, deep convolutional networks on CIFAR-10, and controlled synthetic tasks, across different losses and optimizers and in both training and test accuracy. We propose a three-stage explanation in terms of target-driven organization, a degeneracy bottleneck, and subsequent recruitment and recovery. We develop a rigorous theory of these dynamics for shallow and deep linear networks, showing in particular that high accuracy can emerge while the predictor and the change in loss remain asymptotically vanishing, before continued optimization causes accuracy to deteriorate. For a simple synthetic classification task, we prove an extreme instance in which accuracy asymptotically rises to , collapses to , and later recovers permanently to . Our results show that classification accuracy can reveal aspects of learning dynamics that are not captured by the training loss alone.

Primary area:
learning theory
Keywords:
accuracy well, nonmonotone accuracy, vanishing initialization, training dynamics, deep linear networks, predictive bottleneck, stagewise learning, gradient flow

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SaNqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper proposes a new phenomenon called accuracy well. In this phenomenon, prediction accuracy rises quickly under small initialization, only to drop later on together with training loss, followed by a final increasing in accuracy with consistently low loss. The paper shows this pattern on a set of small model experiments and aims at providing the underlying theory to explain it as well.

Critical strengths
  • The accuracy well phenomenon is shown across different empirical setups and with a clear causal reason.
Critical weaknesses
  • The motivation on why investigating the proposed phenomenon is valuable to the broader research community is unclear. The obvious relevant related work is grokking and double descent. Both show similar patterns in different ways but the paper only briefly mentions their connection without further comparisons to relevant work such as [1, 2].
  • Generally, it is not favorable that related work is moved to the appendix. The corresponding pages are lengthy and only provide limited aid for positioning the proposed work.
  • The difference between when early memorization (grokking) vs. early generalization happens would be insightful but is not fully clear. This is made more challenging as it is not consistently clear which accuracy, train or validation, is shown and discussed in the paper (e.g. Figure 2 center, Figure 3).
  • The theory in the main paper does not aid in understanding the accuracy well phenomenon itself. It is hard to follow and lacks the structure necessary to communicate the underlying ideas. The extensive proofs in the appendix complicate this further.
  1. Liu et al. "Omnigrok: Grokking Beyond Algorithmic Data". ICLR 2023.
  2. Kumar et al. "Grokking as the Transition from Lazy to Rich Training Dynamics". ICLR 2024.
Questions for authors
  • What is the connection to grokking with L2 norm phase transition? In which sense are both effects the same?
  • What key message are you trying to communicate with each of the theorems? E.g., for Theorem 4.1, what is the connection between the bound for L=1 and the accuracy well effect?
6 comments

We thank the reviewer for the careful reading of our manuscript and for the constructive comments and questions.

What is the connection to grokking with L2 norm phase transition? In which sense are both effects the same?

We believe that grokking and the accuracy well are distinct phenomena. Grokking concerns delayed generalization: training accuracy improves much earlier than test accuracy. In contrast, the accuracy well concerns nonmonotone accuracy during training and can occur on both the training and test datasets, without requiring any delay between them.

For example, Fig. 2 (left) of our paper shows similar accuracy wells on both training and test data, without grokking. Conversely, for example, Fig. 2(a) of Kumar et al. (2024) and Fig. 2(b) of Liu et al. (2023) illustrate grokking without a pronounced accuracy well: training and test accuracies are monotonically non-decreasing over time (up to minor fluctuations), but test accuracy improves substantially later. Thus, an accuracy well can occur without grokking, and grokking can occur without an accuracy well.

The two phenomena are not mutually exclusive. In our exploratory experiments on modular addition, we have observed both effects in the same training trajectory. Whether their underlying mechanisms are related remains an open question.

This is made more challenging as it is not consistently clear which accuracy, train or validation, is shown and discussed in the paper (e.g. Figure 2 center, Figure 3).

We thank the reviewer for pointing out the ambiguity regarding training versus test accuracy. We have revised the captions of Figures 2 and 3 to explicitly specify which accuracy is displayed in each panel.

What key message are you trying to communicate with each of the theorems?

Our theorems aim to explain the main experimental observations and establish when an accuracy well can, or cannot, occur.

  1. Why an accuracy well can occur. Consistent with the heuristic picture developed in Section 3, Theorems 4.2 and 4.5 rigorously establish the first two stages: an accurate classifier forms early, but subsequent concentration onto dominant singular modes can destroy its accuracy. For L=2L=2, we also establish eventual recovery, yielding a complete accuracy well when AT>Ak⋆A_{\mathbf T}>A_{k_\star}.

  2. Why accuracy can change dramatically while loss barely changes. Theorems 4.2, 4.3, and 4.5 establish that both the initial accuracy rise and subsequent collapse can occur while the predictor remains microscopic and the loss change is asymptotically vanishing, matching our experimental observations.

  3. Why network depth matters. Theorem 4.1 shows that for L=1L=1, any accuracy drop is bounded by ε=1−AT\varepsilon=1-A_{\mathbf T}, while Theorem B.5 shows that the duration of nontrivial accuracy dynamics vanishes with initialization. This explains why pronounced wells are strongly restricted for L=1L=1, consistent with Fig. 2 (bottom right), unlike factorized networks with L≥2L\ge2.

  4. Why the dataset matters. Theorem 4.3 establishes an extreme accuracy well for random unary operations, where the accuracy trajectory asymptotically follows 0→1→0→10\to1\to0\to1. Finite-width fluctuations cause the collapse even though the mean predictor remains perfectly accurate. In contrast, Theorem 4.4 proves monotone accuracy for permutation targets, where the flat singular-value spectrum eliminates the bottleneck, consistent with Fig. 5 (left column).

  5. Why initialization scale and width matter. Theorem B.7 establishes that sufficiently large width suppresses fluctuations and allows the initial classifier to reach the target accuracy, supporting the stronger early accuracy hills observed at larger HH. Theorem B.15 describes progressively deeper accuracy deterioration at later microscopic times. Combined with our timescale estimates (see our response to WhDc), this explains why smaller initialization can produce deeper accuracy wells by allowing more deterioration before nonlinear recovery begins.

We are currently revising the manuscript to address these comments. In particular, we plan to add a discussion distinguishing the accuracy well from grokking in the Introduction and further clarify the connection between our theoretical results, their underlying mechanisms, and the experimental observations.

We thank the reviewer again for their constructive feedback. We have revised the manuscript to clarify two points:

  1. Relation to grokking. We expanded the discussion of related phenomena in the Introduction, explicitly distinguishing the accuracy well from grokking, which concerns delayed generalization rather than nonmonotone classification accuracy itself.

  2. Interpretation of the theoretical results. We improved the exposition in Section 4 and added a new Summary paragraph that explicitly connects our main theorems to their scientific implications and experimental observations. In particular, it clarifies the mechanism behind accuracy wells, the separation between accuracy and loss dynamics, and the roles of network depth and the learning target.

We hope these revisions address the reviewer's concerns and would be happy to clarify any remaining questions.

I do not see any revisions and thus no revised manuscript.

My main concern about the broader applicability to the research community remains unanswered. What does the accuracy well let anyone predict or do? And while I agree that the presented accuracy well is different from grokking, it would still be helpful to elaborate on their connection, especially as both have a different yet distinct loss-accuracy mismatch during the initial training phases.

Reviewer_AFBFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper reports a phenomenon it calls the accuracy well: under small initialization, accuracy rises quickly early in training, then drops, and recovers much later. It first shows this empirically in deep linear networks, ReLU networks on MNIST, CNNs on CIFAR-10, and synthetic "unary operation" tasks. It then proposes a three-stage explanation (target-driven organization, a degeneracy bottleneck, and recruitment/recovery) and proves it for linear networks trained with squared loss. For a random unary task, the paper proves an extreme case where accuracy goes asymptotically from 100% to 0% and back to 100%.

Critical strengths
  • The phenomenon is interesting. It highlights the gap between the loss dynamics we usually monitor and the discrete training accuracy.
  • The analysis covers different depths L carefully (L = 1, L = 2, L ≥ 3), and the experiments include clean ablations, e.g., the presence and location of biases.
  • The phase 2 analysis is interesting. The times tau_org and tau_bot separate the regimes where different approximations hold. This could be useful for later analyses of linear networks.
  • The controls are informative. The L = 1 baseline (Thm. 4.1) and the permutation target (Thm. 4.4) show that the well does not occur in every setting, and the experiments confirm both (Figs. 3, 8).
Critical weaknesses
  1. The novelty of the analysis is unclear. The authors acknowledge that passing through low-rank predictors in phase 2 is "well-established" in small-initialization dynamics (L.781). Phase 1 looks like the standard linearized (fixed-kernel) result, and phases 3 rely on Saxe et al. (2014). The accuracy drop also seems nearly automatic: a rank-one linear classifier can predict only two classes, so its accuracy is at most about 2/C, which matches Table 2. Could you clarify the technical novelty of this work?

  2. There is a gap between the theory and the experiments.

    • The theory assumes whitened inputs (isotropic covariance, Sec. 4), squared loss, gradient flow, and no biases. The MNIST experiments use raw pixels without centering (App. C.2), and most experiments use cross-entropy and minibatch SGD.
    • Adding an output bias removes the well on MNIST (Figs. 6, 9), and the theory does not explain this.
Questions for authors
  • Your theorems define tau_org and tau_bot in App. B.2, Eqs. 249–250 and take σ→0\sigma \to 0 before H→∞H \to \infty. At a fixed finite sigma and H, is anything guaranteed about these times being well separated? When can phases 1 and 2 merge, or phase 2 not appear?
  • What practical takeaways do you have in mind? For example, do we need early stopping for the shallow network? The well appears only at initialization scales far below standard ones (e.g., σ=10−4\sigma=10^{-4} times the fan-in scaling on MNIST) and disappears with an output bias, which standard networks include. If the initialization scale is this small, do practitioners need to worry about this behavior at all?
  • How does the number of classes affect the theoretical results in the paper?
  • Is there an intuition for why an output bias suppresses the well while a hidden bias does not?
4 comments

We thank the reviewer for the careful reading of our manuscript and for the constructive comments and questions.

  1. The novelty of the analysis is unclear. The authors acknowledge that passing through low-rank predictors in phase 2 is "well-established" in small-initialization dynamics (L.781). Phase 1 looks like the standard linearized (fixed-kernel) result, and phases 3 rely on Saxe et al. (2014). The accuracy drop also seems nearly automatic: a rank-one linear classifier can predict only two classes, so its accuracy is at most about 2/C, which matches Table 2. Could you clarify the technical novelty of this work?

The novelty of our work lies in two complementary aspects: the discovery and systematic phenomenological analysis of the accuracy well, and the development of new rigorous results explaining the associated accuracy dynamics.

Novelty of the phenomenological analysis:

  1. Identification of the phenomenon. We identify and emphasize the accuracy well as a distinct phenomenon: training accuracy can exhibit a pronounced nonmonotone dynamics on the entire training dataset. We demonstrate that this behavior occurs across a broad range of models and learning settings, rather than being specific to a particular setup.
  2. Systematic experimental characterization. We investigate how initialization scale, network width, depth, target structure, loss, and optimization method affect the phenomenon. In particular, we demonstrate that smaller initialization generally makes the well more pronounced and separates its stages in time, while increasing width can strengthen the early classifier and deepen the well.
  3. Relationship between accuracy and loss. We identify a striking regularity: the initial accuracy rise and subsequent collapse can occur while the loss barely changes, whereas the final recovery accompanies substantial loss decay. The bottom of the well typically occurs near the onset of appreciable loss reduction.

Novelty of the theoretical analysis:

  1. Rigorous explanation of the early accuracy rise. Although early predictor dynamics can be described by linearized (fixed-kernel) approximations, this does not automatically determine classification accuracy. The predictor remains microscopic, and fluctuations that are negligible in norm can still change its predictions. Our analysis provides the additional probabilistic control needed to establish the emergence of an accurate classifier.
  2. Rigorous explanation of the accuracy collapse. Known results on low-rank dynamics do not establish the accuracy drop by themselves. First, one must establish that an accurate classifier actually forms; otherwise, there is no accuracy hill from which to collapse. Second, during the accuracy drop, the predictor is still microscopic, so small residual components and fluctuations can affect classification. Our results establish the deterioration of the actual classifier, rather than merely the emergence of low-rank structure.
  3. Recovery and the scope of previous results. The classical analysis of Saxe et al. (2014) provides useful spectral dynamics for understanding the later recovery stage, but does not directly analyze classification accuracy. More importantly, recovery is only one stage of the accuracy well, and these classical results alone do not explain the preceding emergence and collapse of an accurate classifier (stages 1 and 2).
  1. There is a gap between the theory and the experiments.

The theory assumes whitened inputs (isotropic covariance, Sec. 4), squared loss, gradient flow, and no biases. The MNIST experiments use raw pixels without centering (App. C.2), and most experiments use cross-entropy and minibatch SGD.

To address this concern, we have performed an additional experiment on centered and whitened MNIST, closely matching the assumptions of our theory.

Specifically, we trained a bias-free one-hidden-layer linear network using full-batch gradient descent and squared loss on 10,00010{,}000 MNIST examples, with width H=1024H=1024 and initialization scales σ∈{10−20,10−30,10−40}\sigma\in\{10^{-20},10^{-30},10^{-40}\}.

The agreement with our theoretical predictions is remarkably close: the observed accuracy initially rises to approximately 0.840.84, approaching the full-target accuracy A(T)≈0.85\mathcal A(\mathbf T)\approx0.85, then drops toward the leading rank-one classifier accuracy A(T1)≈0.2\mathcal A(\mathbf T_1)\approx0.2, before recovering. Decreasing the initialization scale makes the agreement increasingly closer.

We have added the experiment to Appendix C.2, paragraph “Theoretical predictions on whitened MNIST,” and referenced it immediately after the main L=2L=2 theorem in Section 4.

  • Adding an output bias removes the well on MNIST (Figs. 6, 9), and the theory does not explain this.
  • Is there an intuition for why an output bias suppresses the well while a hidden bias does not?

Yes, there is a natural explanation for the different effects of hidden and output biases in the one-hidden-layer ReLU networks considered in our MNIST experiments, which we have now included in Appendix C.2, paragraph “Explanation of the bias dependence,” and discussed in Section 2, paragraph “Architectural biases.”

Hidden bias. A hidden bias can be absorbed into an additional constant input coordinate, x~=(x⊤,1)⊤\widetilde{\mathbf x}=(\mathbf x^\top,1)^\top. This reduces the architecture to a bias-free network on augmented inputs, preserving the multiplicative parameterization responsible for the accuracy well. Thus, we expect the underlying mechanism to persist, consistent with our experiments.

Output bias. The situation is different because an output bias provides a directly trainable additive contribution. Writing fθ(x)=gθ(x)+b2\mathbf f_\theta(\mathbf x)=\mathbf g_\theta(\mathbf x)+\mathbf b_2, gradient flow under squared loss gives

b˙2=2(yˉ−gˉ−b2),\dot{\mathbf b}_2=2(\bar{\mathbf y}-\bar{\mathbf g}-\mathbf b_2),

where bars denote training-set averages. Since gθ=O(σ2)\mathbf g_\theta=O(\sigma^2) at early fixed times, while b2(0)=0\mathbf b_2(0)=0, we obtain

b2(t)=(1−e−2t)yˉ+O(σ2).\mathbf b_2(t)=(1-e^{-2t})\bar{\mathbf y}+O(\sigma^2).

Thus, unlike the feature-dependent predictions gθ(x)=O(σ2)\mathbf g_\theta(\mathbf x)=O(\sigma^2) at fixed early times, the output bias reaches macroscopic magnitude on an O(1)O(1) timescale, exponentially approaching yˉ\bar{\mathbf y}. It therefore dominates the initially microscopic contribution of the hidden neurons.

Importantly, the output bias is a vector, not a common shift of all class scores. If yˉ\bar{\mathbf y} has a unique largest component, then for sufficiently small σ\sigma, the network predicts the same class argmax⁡cyˉc\operatorname{argmax}_c\bar{\mathbf y}_c for every input at any fixed early time t>0t>0. Consequently, the output bias suppresses the early accuracy hill, even if it develops in the underlying feature-dependent predictions gθ(x)\mathbf g_\theta(\mathbf x).

Our new diagnostic experiments (added to App. C.2) support this explanation: removing the learned output bias only at evaluation reveals that the underlying accuracy well persists, even though it is suppressed in the full network. We agree that this architectural distinction deserves greater emphasis and have revised the manuscript accordingly.

We believe that these additional experiments, together with the theoretical explanation of the role of output biases, substantially narrow the gap between our theoretical analysis and experimental observations.

  • Your theorems define tau_org and tau_bot in App. B.2, Eqs. 249–250 and take σ→0\sigma \to 0 before H→∞H \to \infty. At a fixed finite sigma and H, is anything guaranteed about these times being well separated? When can phases 1 and 2 merge, or phase 2 not appear?

Generally, two conditions are important for a pronounced accuracy well: sufficiently small initialization and sufficiently large width.

1. Initialization scale σ\sigma. Small initialization separates the early microscopic dynamics from the later nonlinear dynamics responsible for recovery. If σ\sigma is too large, nonlinear effects may intervene before the degeneracy bottleneck fully develops, making the accuracy drop less pronounced. Our analysis provides a leading-order estimate for when interference occurs before nonlinear saturation:

σ≲2s1 H−s1ζ,\sigma \lesssim \sqrt{2s_1}\,H^{-s_1\zeta},

where s1s_1 is the largest singular value of the target, and ζ\zeta characterizes the interference time τ=ζlog⁡H\tau=\zeta\log H. Larger ζ\zeta corresponds to a deeper accuracy bottleneck, as shown for unary targets in Theorem B.15. Here σ\sigma follows the initialization convention of Fig. 3 (right), where initialization scale is σ/H\sigma/\sqrt{H}. We discuss this condition more in our response to WhDc. At still larger initialization, nonlinear effects can also overlap with the initial organization stage, making the stages less distinguishable.

2. Network width HH. Sufficiently large width allows the weak predictive signals to accumulate while random fluctuations average out, producing the initial accuracy rise. Theorem B.7 gives the sufficient condition H≳κ3log⁡(2NCH),H\gtrsim\kappa^3\log(2NCH), where NN is the number of examples, CC is the number of classes, and κ\kappa is an explicitly defined dataset-dependent constant (see Appendix B.2). At insufficient width, the early accuracy peak may be much lower or disappear, making the well less pronounced. This does not necessarily mean that the first two stages merge; rather, the initial organization may fail to produce an accurate classifier.

Thus, the stages are asymptotically separated under our assumptions, but their visibility at finite σ\sigma and HH depends on these two conditions.

What practical takeaways do you have in mind? For example, do we need early stopping for the shallow network? The well appears only at initialization scales far below standard ones (e.g., σ=10−4\sigma=10^{-4} times the fan-in scaling on MNIST) and disappears with an output bias, which standard networks include. If the initialization scale is this small, do practitioners need to worry about this behavior at all?

We believe our results have broader relevance for several reasons.

1. Small initialization is a well-established regime in theoretical deep learning, studied extensively in the literature. In our Related Work appendix, we discuss more than 10 papers investigating small-initialization and related early-training dynamics. Understanding this regime is therefore an important problem in its own right.

2. Learning dynamics can naturally separate into different stages, as demonstrated in previous studies [1–3]. In particular, small initialization can make successive learning stages more pronounced [2, 3], providing a useful setting for their mathematical analysis. Importantly, staged behavior does not necessarily disappear at larger initialization. For example, in Fig. 2 (top middle), even at σ=10−1\sigma=10^{-1}, we observe an initial accuracy rise, followed by slower improvement and later acceleration. Thus, vanishing initialization serves as a useful mathematical lens for studying mechanisms that may also operate at larger initialization scales.

3. Our experiments also suggest potential implications for early stopping. For example, in Fig. 3 (middle), the shallow network with H=64H=64 achieves substantially higher training accuracy at an early stage than at the end of training. Early stopping could therefore be beneficial in such settings, although its effect on generalization requires further investigation.

[1] Saxe et al., “Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks,” ICLR, 2014.

[2] Gidel et al., “Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks,” NeurIPS, 2019.

[3] Gissin et al., “The Implicit Bias of Depth: How Incremental Learning Drives Generalization,” ICLR, 2020.

How does the number of classes affect the theoretical results in the paper?

The number of classes CC does not qualitatively affect our results, but it can affect them quantitatively. For example, a rank-one classifier can predict at most two classes, so its accuracy is bounded by 2/C2/C on balanced datasets. Thus, increasing CC lowers the bottleneck accuracy, potentially making the well deeper.

Thank you for the detailed responses and the additional experiments. I find the accuracy well an interesting phenomenon. The new experiments on whitened MNIST help close the gap between the theory and the experiments, and the new remark on the output bias is useful for understanding what happens during training. Your reply on the initialization scale and the network width also clearly explains why these conditions are needed for the accuracy well. Based on these clarifications, I raise my score from 2 to 3 (weak acceptance). Thank you again for your detailed responses.

Reviewer_WhDcOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies an “accuracy well” under small initialization: classification accuracy initially improves, subsequently deteriorates, and later recovers. The authors propose a three-stage explanation involving target-driven organization, a degeneracy bottleneck, and recruitment of additional predictive directions. They establish rigorous results for linear networks, including a random-unary classification setting in which accuracy asymptotically approaches one, collapses toward zero, and eventually recovers under sufficient width. Experiments demonstrate related behavior in ReLU networks on MNIST and convolutional networks on CIFAR-10.

The central contribution is the analysis of useful class ordering that emerges before appreciable loss reduction and is subsequently destroyed by further optimization. The complete theoretical account is strongest for two-layer linear networks; generic deeper-network recovery and the mechanism underlying the nonlinear experiments remain open.

Critical strengths
  1. The two-layer theory provides a substantive explanation of classifier dynamics beyond the observation that accuracy and loss need not move together. The random-unary result is particularly interesting: the mean microscopic predictor remains perfectly accurate at finite positive times, while amplified finite-width fluctuations can destroy classification. This distinction makes the collapse mechanism nontrivial.

  2. The theoretical controls are informative. The direct-linear accuracy bound and the two-layer permutation result clarify when an accuracy well is absent. Perfect classification in the undercomplete permutation setting also illustrates the distinction between fitting target values and recovering their class ordering.

  3. The experiments convincingly establish the existence of the phenomenon beyond the synthetic setting. The MNIST initialization sweeps and five-seed control show clear nonmonotone trajectories, while CIFAR-10 exhibits corresponding training and held-out behavior.

  4. The paper carefully states several important limitations, including the order of limits, the assumptions on spectral structure and decision margins, and the distinction between generic and spectrally aligned initialization.

Overall, I lean toward acceptance because the two-layer theoretical contribution and the empirical phenomenon provide useful insight into learning dynamics. My reservations concern the quantitative connection to finite networks and the evidence for the proposed nonlinear mechanism.

Critical weaknesses

Post-rebuttal update, 15 November 2026: I now recommend 4: Clear acceptance. The timescale estimate, whitened-MNIST trajectory comparison, and output-bias intervention with centered-score spectra substantially address my principal concerns. The estimate remains heuristic and multi-seed CIFAR-10 uncertainty remains unaddressed, but these limitations do not outweigh the paper's contribution. See my follow-up comment for the detailed assessment. The original concerns below are retained for context.

  1. The sequential asymptotics leave the finite-size regime insufficiently characterized. The principal results take the initialization scale to zero before the large-system limit. For the random-unary mechanism, finite-width interference must become strong enough to destroy classification before nonlinear saturation invalidates the microscopic approximation. The paper does not provide a readily usable condition relating initialization scale, width, and spectral gaps. This limits the quantitative connection between the theorem and the experiments.

  2. The nonlinear mechanism remains suggestive rather than established. Figure 4 reports neuron participation and output diversity along one MNIST trajectory. Concentration of contribution magnitudes does not by itself establish a low-dimensional predictive representation, and output diversity is structurally related to accuracy on balanced multiclass data. Spectral measurements or a controlled intervention would provide more independent evidence. The finding that an output bias suppresses a comparably pronounced well in the MNIST MSE experiments is an important boundary of applicability that deserves greater emphasis.

  3. Table 2 provides limited dynamical evidence. A bias-free rank-one classifier Mx = u(vᵀx), with unique decisions, predicts at most two classes: the maximizing component of u when vᵀx is positive and the minimizing component when it is negative. Its accuracy is therefore bounded by the combined frequency of two classes. The low rank-one accuracies in Table 2 are partly a consequence of this restriction. The stronger test would be whether actual trajectories approach the predicted bottleneck classifier and accuracy.

  4. Empirical uncertainty is incompletely characterized. The five-seed MNIST experiment is useful, but the CIFAR-10 sweeps use single runs and moving averages of changing minibatches. Multi-seed estimates of hill height, well depth, and recovery time, evaluated on fixed sets at checkpoints, would strengthen the broader empirical claims.

Questions for authors
  1. Can the authors give a finite-size condition, or a joint initialization-scale/width experiment, showing when fluctuation-driven collapse occurs before nonlinear saturation?

  2. On centered and whitened image data, do trained linear-network trajectories approach the predicted full-target and top-singular-block classifiers? How closely do their observed accuracies match the theoretical levels?

  3. Can the nonlinear bottleneck interpretation be supported by feature or centered-score spectra, or by an intervention that helps explain why output bias suppresses the well?

  4. How reproducible are the CIFAR-10 well depths and transition times across seeds when measured on fixed evaluation sets?

The first and third questions are most important to my assessment. A convincing quantitative connection between the asymptotic mechanism and finite-network behavior would strengthen my recommendation.

3 comments

We thank the reviewer for the careful reading of our work and for these constructive questions. They have motivated additional experiments and clarifications that strengthen the connection between our theoretical analysis and empirical observations.

  1. The sequential asymptotics leave the finite-size regime insufficiently characterized...

Can the authors give a finite-size condition, or a joint initialization-scale/width experiment, showing when fluctuation-driven collapse occurs before nonlinear saturation?

Our existing analysis provides the ingredients for a quantitative finite-size estimate, which helps explain the initialization dependence observed in our experiments.

Timescale comparison. The key requirement is that fluctuation-driven interference must occur before nonlinear saturation. Lemma B.17 predicts interference at characteristic times τ=ζlog⁡H\tau=\zeta\log H, where HH is the network width and ζ\zeta is a threshold determined by the target's spectral structure, independently of the initialization scale σ\sigma or width HH. Importantly, different output occupancy levels have different thresholds ζ\zeta: examples associated with more highly occupied outputs require longer times for interference to disrupt their classification. For this interference to develop, the microscopic regime must persist sufficiently long. Comparing the interference time with the onset of nonlinear saturation gives the approximate condition

\sigma \lesssim \sqrt{2s_1} H^{-s_1\zeta}, }$$ where $s_1$ is the largest singular value of the target matrix and $\sigma$ follows the initialization convention of Fig. 3 (right), where initialization scale is $\sigma/\sqrt{H}$. **Thus, for a given width and target, sufficiently small initialization allows the predicted interference to occur before nonlinear saturation.** **Comparison with experiments.** For the actual random-unary target in Fig. 3 (right), $p=H=512$ and $s_1=2$. The interference thresholds for output occupancies $1,2,3$ are $\zeta\approx0.50,0.85,1.87$, corresponding to approximate initialization bounds $\sigma \lesssim 3.9\times10^{-3}, 4.7\times10^{-5}, 1.5\times10^{-10}$, respectively. Hence, the experimental scales $\sigma=10^{-1},10^{-4},10^{-7},10^{-10}$ successively allow interference in more occupancy groups, **consistent with the progressively deeper accuracy wells observed in Fig. 3 (right)**. We emphasize that this timescale comparison is a leading-order estimate rather than a rigorous finite-size guarantee. A more precise and rigorous quantitative characterization of the finite-size behavior is a natural direction for future work, beyond the scope of the present paper.
  1. On centered and whitened image data, do trained linear-network trajectories approach the predicted full-target and top-singular-block classifiers? How closely do their observed accuracies match the theoretical levels?

We have performed an additional experiment on centered and whitened MNIST that shows close agreement with our theoretical predictions.

We trained a bias-free two-layer linear network with width H=1024H=1024 using full-batch gradient descent and squared loss on 10,00010{,}000 MNIST examples, with initialization scales σ∈{10−20,10−30,10−40}\sigma\in\{10^{-20},10^{-30},10^{-40}\}. We directly computed the theoretical reference accuracies of the full-target classifier A(T)≈0.85\mathcal A(\mathbf T)\approx0.85 and its leading rank-one component A(T1)≈0.2\mathcal A(\mathbf T_1)\approx0.2.

The observed accuracy rises to approximately 0.840.84, closely matching A(T)\mathcal A(\mathbf T), then drops toward 0.20.2, closely matching A(T1)\mathcal A(\mathbf T_1), before recovering. Smaller initialization makes the agreement increasingly closer.

We have added this experiment to Appendix C.2, paragraph “Theoretical predictions on whitened MNIST,” together with a reference immediately following our main L=2L=2 theorem in Section 4.

The finding that an output bias suppresses a comparably pronounced well in the MNIST MSE experiments is an important boundary of applicability that deserves greater emphasis.

  1. Can the nonlinear bottleneck interpretation be supported by feature or centered-score spectra, or by an intervention that helps explain why output bias suppresses the well?

We have performed additional experiments that provide direct evidence for the proposed nonlinear bottleneck and clarify the role of output biases.

First, we trained MNIST ReLU networks with and without an output bias. Remarkably, removing the learned output bias only at evaluation reveals that the characteristic accuracy well persists in the feature-dependent predictions, even though it is suppressed in the full network. Second, we analyzed the score matrices centered across both examples and output classes. In both architectures, their spectra exhibit pronounced near-rank-one concentration followed by recovery, supporting our bottleneck interpretation.

These findings suggest that the output bias primarily masks the underlying accuracy dynamics rather than eliminating them. We agree that this architectural dependence deserves greater attention. We have therefore added the intervention, spectral diagnostics, and a theoretical explanation of why the directly trainable output bias can suppress the accuracy well to Appendix C.2, paragraphs "Output-bias intervention and discriminative-score concentration" and "Explanation of the bias dependence", together with a new discussion of the findings in Section 2, paragraph "Architectural biases."

Reviewer_WhDcOpenReview ↗

Post-rebuttal assessment: recommendation upgraded

Thank you for the detailed responses and the additional experiments. After reading both replies to my review and the revised Appendix C.2, I find that my principal concerns have been sufficiently addressed. I have raised my recommendation from 3: Weak acceptance to 4: Clear acceptance.

  1. Finite-size interpretation. The comparison between the interference time and nonlinear saturation gives a useful leading-order condition connecting initialization scale, width, and the target spectrum. The occupancy-dependent numerical estimates make the initialization dependence more concrete. I appreciate the explicit distinction between this estimate and a rigorous finite-size guarantee.

  2. Theory-to-experiment comparison. The new centered and whitened MNIST experiment directly addresses my concern that Table 2 alone was largely a static rank-one restriction. The observed rise to about 0.84, drop toward 0.20, and subsequent recovery agree closely with the computed full-target and leading-component reference accuracies. This is evidence about actual trajectories under assumptions close to the theory.

  3. Nonlinear bottleneck and output bias. The evaluation-time removal of the learned output bias, together with the doubly centered score spectra, provides the independent evidence I requested. The reappearance of the well in the feature-dependent predictions supports the interpretation that the output bias can mask the underlying dynamics in this setting.

The finite-size condition remains heuristic, and my request for multi-seed CIFAR-10 estimates on fixed evaluation sets has not been resolved. I regard these as remaining limitations and useful future work, rather than reasons to withhold acceptance given the theoretical contribution and the new targeted evidence. The manuscript should retain its distinction between the rigorous linear results and the more limited nonlinear evidence. Thank you for the constructive engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.