- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
A fully-connected two-layer neural network trained with Adam on modular arithmetic data can exhibit grokking. Assuming that the parameter space consists of a generalizing core and an overfitting shell outside of the core, the authors derived expressions for the core and shell radii, and presented numerical results supporting their theory.
Critical strengths
None.
Critical weaknesses
-
The presentation of the mental model is not clear. The introduction appear to tell me, the parameter space consists of a generalizing core and an overfitting shell, but what is the justification? Also, if that is the case, the core and shell radii should be the property of the loss function and risk function themselves, so why would they depend on the learning rate and batch size? The authors indeed say the shell-core structure is "optimizer-induced", but I think the meaning of that deserves elaboration.
-
The authors claim Eq. (7) is one of their main result, but I don't see what can be learned from the equation. Eq. (7) tells me that the shell radius depend on the ratio of the learning rate and the batch size, but this is not surprising because the minibatch noise depends on the same ratio. How does the shell radius depend on this ratio? Eq. (7) says their relation is a Taylor expansion, which is true for any smooth function. If, for example, the authors can show that the 3rd-order term must be orders or magnitudes smaller than the constant, linear, and quadratic terms, then that is indeed something new for me to learn. Unfortunatelly I don't see the authors proving that.
-
The numerical results are presented with "theoretical fits." However, they did not clearly write down the fitting functions. Therefore, how falsifiable are their predictions is not clear to me.
Questions for authors
What is the justification of the shell-core model of the parameter space?
Eqs. (7) and (15) appers to be just Taylor expansions. What can I learn from them, since all smooth functions can be written as Taylor series?
7 comments
Responses to Reviewer iaDy (part 2)
Regarding
Also, if that is the case, the core and shell radii should be the property of the loss function and risk function themselves, so why would they depend on the learning rate and batch size?
We appreciate your concerns. Let us put it simply.
- The radii are a property of neither the loss function nor the risk function.
- Their dependence on hyperparameters such as the learning rate and the batch size comes directly from Adam's joint SDE (Lemma 1):
Intuition. Why learning rate and batch size enter into Adam's SDE.
In mini-batch sampling, the optimiser uses the gradient computed from a mini-batch to estimate the true full-batch gradient. The error (variance) of this estimate appears as the diffusion term. The batch size controls the noise magnitude: a larger batch gives a smaller estimation error.
SGD as example. Let me use SGD's SDE as an example, step by step. (You can also find in appendix A5)
For example, suppose a batch contains samples
with per-sample gradients
whose mean and covariance are
The mini-batch gradient is their average
By the central limit theorem (CLT), if the batch size is sufficiently large, converges in distribution to
where is the per-sample covariance. The covariance of the batch average is scaled by because the samples are independent:
Wiener process. Therefore, for any mini-batch, the estimated gradient can be written as two terms:
where is the sampling noise. We use the negative-sign convention because gradient descent uses the negative gradient; this simplifies the later derivation.
SDE. Adam is too complicated to serve as an example, so let us use SGD. Consider the update at iteration (a stochastic process)
where is the learning rate. Substituting gives
Each iteration corresponds to a time step , i.e. . A Wiener process (Brownian motion) has increments , so and
Taking gives the SDE of SGD:
Where and enter. The batch size enters through the covariance
Responses to Reviewer iaDy (part 3)
Regarding
The authors indeed say the shell-core structure is "optimizer-induced", but I think the meaning of that deserves elaboration.
We appreciate your concerns. The shell-core structure is indeed induced by the optimiser: it follows from Adam's joint-state SDE, which we developed and validated.
Adam's joint-state SDE is
where is the parameter at time , and and are the first- and second-moment estimates at time .
The SDE has two terms. In SDE theory, is called the drift, which is deterministic, and is called the diffusion, which is stochastic.
Training starts from an initial state , where is the initial parameter given by the initialiser (e.g. PyTorch's Kaiming initialiser), and the first- and second-moment estimates are initialised to zero.
- Initialisation shell. As we derived in Section 3, concentrates on a thin hyperspherical shell: for a given initialiser, the initial radius is nearly the same whatever the random seed.
- Memorisation. Starting from , as optimisation progresses, the trajectory first reaches the memorisation boundary at the stopping time . Because the initial states concentrate, the trajectories towards memorisation behave alike. This is why, in Fig. 1, we observe very symmetric trajectories using manifold learning (MDS). It also suggests that, for any starting parameter (with the initialiser kept the same), the radius at which the trajectory hits the memorisation boundary, , is expected to be nearly identical.
- Generalisation. Once the trajectory has reached the memorisation manifold, as optimisation continues, it finally settles in the generalisation manifold, where drift and diffusion equilibrate.
This is the optimisation-induced topology. The memorisation and generalisation manifolds and are defined as the solutions reachable under Adam's dynamics.
Empirically, the three radii concentrate across seeds (Appendix A.2; , 10 seeds, mean std): initialisation , memorisation , generalisation .
Regarding
- The authors claim Eq. (7) is one of their main result, but I don't see what can be learned from the equation. Eq. (7) tells me that the shell radius depend on the ratio of the learning rate and the batch size,
We appreciate your concerns.
We would like to clarify the scope of this work: we study the scaling laws of grokking, i.e. how grokking scales with the learning rate, the batch size and the regularisation coefficient. Eq. (7) is the result that theoretically derives the relationship between these hyperparameters and the memorisation radius. Several things can be learned from it.
- The learning rate and the batch size act through the single ratio . They modulate the stochastic diffusion of Adam's SDE through the factor . To the best of our knowledge, the scaling of the memorisation radius with has not been derived in prior work.
- The regularisation coefficient acts differently. modulates the deterministic drift, and enters only through the leading constant, .
- Eq. (7) predicts a U-shape. It is consistent with Fig. 2.
Regarding
but this is not surprising because the minibatch noise depends on the same ratio. How does the shell radius depend on this ratio?
We appreciate your concerns. We welcome criticism and are open to academic disagreement.
However, knowing that the mini-batch noise depends on the ratio does not tell us how the scaling laws are quantitatively governed by and .
Responses to Reviewer iaDy (part 1)
Dear Reviewer iaDy,
We very much appreciate that you have read and assessed our work. We will organise our responses point by point to address your concerns.
Regarding
We appreciate your careful reading and the summary.
We would like to clarify that this paper does not focus specifically on two-layer neural networks; the two-layer architecture is used only in the experimental setting. Our work studies training dynamics.
To validate our equations, lemmas, theorems, and theoretical predictions, we need to compute the gradient, with O(N) entries, and the Hessian, with O(N2) entries, at every training step, where is the number of parameters. For larger networks, this computation becomes impractical.
Regarding
We appreciate your careful reading and summary.
The topological configuration comes from empirical observations under weight-shrinkage regularisers. Please refer to the experiments conducted in Figure 1 and Appendix (A.2 ADDITIONAL RESULTS: ADAM–INDUCED SHELL–CORE RADIUS & STOPPING-TIME CONCENTRATION).
Regarding
We appreciate that you highlight our contribution.
Regarding
We appreciate your concerns, welcome criticism and open to academic disagreements.