- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies deep residual networks as depth and width scale together to infinity. Existing theory and actual scaling recipes in practice mostly fall into the ODE regime, which ensures feature learning (hidden states can be updated by O(1) during training) but loses feature diversity (random features at random initialization vanish/average out as the network size approaches infinity). This paper analyzes the SDE regime, where feature learning is ensured and feature diversity does not vanish, and proposes scaling/initialization recipes for the SDE regime. A wide range of experiments, from MLPs on CIFAR to GPT-2, provide empirical evidence partly supporting the theory.
Critical strengths
- Systematic theory and experiments. This paper systematically studies the scaling limit and DMFT, including various optimizers. Experiments from different angles and different model classes are conducted. The paper has the basic "shape" of an influential work.
Critical weaknesses
- Unclear motivation/justification for the SDE regime. The SDE regime indeed preserves feature learning and feature diversity. It is easy to understand and widely accepted that feature learning is good. But it is unclear why we need feature diversity or what the benefits are. The theory seems to be unable to prove an advantage in loss of SDE against ODE. And partly the reason is that DMFT cannot be solved analytically. As far as I can tell, the paper seems to use experiments to show that the SDE regime can lead to lower loss (Figure 6b) and that real LLMs are more likely to be in the SDE regime (Figure 7). But the evidence for advantage is not that strong. If the ODE regime turns out to lead to better performance at late training, discussions in this paper will become useless.
- Tension between theory and reality. The theory says that , which seems to be how people are scaling models these days, leads to the ODE regime. But later, the paper claims that actual LLMs are in the SDE regime. I am confused; why in reality leads to SDE, then does the new recipes proposed necessary if models are already in SDE? The theory suggests , which will lead to a smaller MLP hidden dimension than the embedding dimension, being a little counterintuitive.
Questions for authors
- This is a clarification question on interpretation. The theory studies the limit of infinitely large models with a finite number of training steps. Should I interpret this theory for finite-size models as the
early stage of training, where the number of training steps is much smaller than the model size or some function of the model size? It seems that the experiments in this paper also focus on a very early stage of training. If it is not due to limited compute, I guess it is intended to match the theory? - If my interpretation is correct in 1., the question is: what will the late training be like? Will SDE be beneficial for the late stage of training? For a finite-size model, even in the SDE regime, does the randomness due to initialization (or random features) go away at very late or long training?
- If my interpretation is correct in 1., what determines the validity of the theory for a finite-size model? Or is there any quantitative threshold for the number of training steps and the model size such that the DMFT and scaling argument are accurate?
- What is the difference between Eq. (6) and Eq. (8)? My understanding is Eq. (6) from SDE and (8) from ODE? But they are the same scaling, no? Based on the error, the optimal scaling is , which leads to SDE? I am confused.
- Based on your Eq. (6), let's say the total number of parameters is , the current scaling method, will lead to a squared error as , which is actually close to or consistent with Chinchilla scaling laws. With your suggested scaling method, , the model size scaling of loss will be . If you are correct, does this mean we can change neural scaling laws by better shapes of scaling up models?
- Some recent practitioners (see
modded nanogptrepo) suggest initializing down-projections to be zero matrices. Does this effectively introduce correlation between layers, or to make the layers as ODE (zero drift, zero randomness, initially)?
10 comments
Response to Reviewer 7Ukv (2/3)
Response to Reviewer 7Ukv (2/3)
Q1–Q3 · Training duration and late-time dynamics
Your interpretation in terms of early training is broadly right. We first fix the number of training steps , then take the widths and depth to infinity. For a finite network, the longer we train, the more important it is to check whether the mean-field approximation remains accurate. The error estimate in Result 3 depends on , and we do not yet have a threshold for the number of steps as a function of network size. The left and middle panels of Figure 5 give a concrete point of reference, with the fitted slopes in Appendix G.1. On CIFAR-10, comparing the seven larger network sizes with the largest-model ensemble, the fitted slopes of output and residual-kernel error against size are and at the first epoch, close to the theoretical prediction of . By the tenth epoch, both errors still decrease with size, but the slopes have changed to and . At the sizes we could test, the theory does give a better quantitative description of the early errors.
There is also a direct reason for holding the number of training steps fixed in the recipe experiments. The learning rate is one of the hyperparameters we transfer. Once we have selected it on a small model, how far can the larger model drive down training loss with the same number of updates? Compute limited how long we could train, but intermediate measurements let us check whether a recipe's advantage happened to appear only at the endpoint. For example, at all five Tiny Shakespeare sizes , both the fixed-ratio recipe and Half Copy have lower three-seed mean training loss than ODE at both 2,500 and 5,000 steps. Half Copy's advantage is already visible at step 1,000 at all five sizes. Whether ODE would catch up with much longer training remains open.
We also do not have a definitive answer about how much randomness from initialization remains very late in training. After 15,000 steps in the 12- and 24-layer GPT-2 models, the cumulative squared energy of the layerwise residual increments remains large, and we also see positive cross-layer correlations that developed during training. This measurement suggests that variation through depth is still worth retaining in the description even after the network has learned cross-layer structure. To establish how much of this variation can still be attributed to initialization, we would have to track how the trained features depend on the initial weights. The energy at the end of training alone does not answer that question.
Response to Reviewer 7Ukv (3/3)
Response to Reviewer 7Ukv (3/3)
Q4–Q5 · Equations (6) and (8) and loss scaling
Your conversion of the exponents is correct for the approximation error in Eq. (6). We obtain Eq. (8) from that same bound. At a given residual-block parameter budget , we balance the three approximation-error terms for SGD/Adam. When the two widths and depth grow proportionally, Eq. (6) gives an upper bound on the mean squared error. With as in Eq. (8), the bound is .
In both cases Eq. (6) compares a finite network at a fixed training step with the mean-field prediction at its current ratio, (Appendix D). Under proportional growth, . Comparing directly with the final ODE prediction would also require controlling .
This is why we discuss the choice of network shape explicitly in §3.3. Even if we knew the approximation errors along both routes exactly, which model assigns a higher probability to the correct next token would still be a separate question. To predict how loss changes with scale beyond an empirical fit such as Chinchilla, we would need to know both how well each limiting model learns the task and in which direction, and by how much, task loss changes as a finite network departs from its own limit.
Q6 · Zero initialization of the down-projections
Your intuition about initialization is correct. Setting the down-projection to zero makes that branch's contribution to the initial residual variance exactly zero. Doing so in every layer does not by itself introduce random correlations between layers.
We emphasize residual variance because it is substantial in many finite networks, yet vanishes as we scale up along the ODE recipe. We do not regard it as the only quantity that matters. There is an interesting detail in modded-nanoGPT itself: some later versions added U-Net-style skip connections, so even when each branch's down-projection still starts at zero, states from earlier layers are added back into the residual stream. This example shows that zero initialization can be part of an effective recipe, but it gives insufficient evidence for the stronger claim that the best choice is to make the initial forward evolution of the residual stream through depth approach the trivial evolution suggested by the ODE limit. modded-nanoGPT record history
Once training starts and there is a nonzero backpropagated signal, the down-projection update uses the hidden features produced by the random up-projection. We have yet to test how much the ratio affects the subsequent dynamics under this initialization.
Response to Reviewer 7Ukv (1/3)
Response to Reviewer 7Ukv (1/3)
Critical weaknesses 1–2 · Finite GPT-2 and the ODE and SDE limits
The ODE limit has been a theoretically fruitful route, and what concerns us is precisely why it hasn’t been as useful to practitioners as one might have hoped. We see one possible reason. Our GPT-2 measurements suggest that, at initialization, the cumulative energy of the random residual increments through depth is about two orders of magnitude larger than the energy of the input embedding. Yet under the HP transfer recipe associated with the ODE limit, this contribution vanishes as we scale up. If a contribution that dominates at finite size vanishes in the limit, is that limit a useful point of comparison for the finite networks we actually train?
In our experiments, the ODE HP transfer recipe gives substantially higher training loss than two alternatives we came up with simply by trying to retain that random variation through depth. We therefore strongly believe that what we call the SDE limit gives a more useful description of training dynamics in residual networks, in part because it captures this contribution.
As for which asymptotic regime actual LLMs belong to overall, we’d leave that open. Outside of HP transfer experiments, people rarely follow a fixed recipe all the way to its limit. Any actual scale-up covers only a finite stretch of that path, and people may change the hyperparameters along the way if the gains fall short. Our point is that the ODE regime, which has dominated the theoretical discussion in recent years, loses an important feature of the dynamics that the SDE retains. The same standard applies to us. If the SDE turns out to miss another contribution that dominates in finite networks, then it’s our turn to revise the limit.
That leaves us more optimistic about HP transfer itself. The fixed-ratio and Half Copy recipes give us two concrete ways to retain the random residual contribution as we scale up. If they hold up at larger scales, HP transfer may be more broadly useful than it looks today. The difficulty may have been in the choice of limit.
Critical weakness 2 · Single-layer MLP width and sliding
We agree that N♯≍N♭≍L, which makes a single MLP layer narrower than the embedding, looks counterintuitive at first. We kept N♭/(N♯L) fixed to retain random residual variation and the scale of local feature learning. CIFAR-10 uses the balanced scaling in Eq. (8). On Tiny Shakespeare we instead grow (N♭,N♯,L) by (R3/2,R1/2,R) to match ODE in depth and block parameters. We had no reason to expect either shape to be optimal, yet both gave substantially lower training loss than ODE at larger sizes. The allocation of width and depth itself changes the scale of residual randomness, so this comparison cannot isolate the respective contributions of shape and random variation. Half Copy, however, retains the conventional shape and uses correlations at initialization to preserve random residual variation. It too attains lower training loss than ODE. This makes us believe that the variation ignored by the ODE limit deserves as much attention in finite networks as the choice of shape.
The sliding diagram is on p. 3 of
00_rebuttal_figures.pdfat the top level of the updated supplementary ZIP.The figure shows one way to understand this shape. First keep depth fixed and widen the embedding and the MLP in the same proportion. This preserves their width ratio, so it is easy to see why such a network might train effectively. Now split the same hidden channels from one wide block into two consecutive residual blocks. Channels 1–4 in the figure compute exactly the same functions before and after the split. Channels 5–8 keep their original weights but read the residual state after its update by the first four channels. If that update is small, the contributions from the last four channels will also be close. More generally, widening both dimensions by a factor of q2 and then splitting each wide block into q layers gives (q2N♭,qN♯,qL). Each MLP layer is indeed narrower than in the proportionally widened network, but all the hidden channels are still there. Sliding explains why, under suitable conditions, moving those channels from parallel to sequential computation can have only a limited effect on the macroscopic dynamics.