acceptodds
Under review as a conference paper at ICLR 2027

Sliding Networks: A Family of Residual Structures Preserving Feature Learning and Feature Diversity

Abstract

Deep learning models tend to perform best at large widths and depths. The growing interest in scaling laws, especially in the language modeling domain, has motivated procedures for scaling up network widths and depths in order to preserve stability and computational efficiency of training dynamics. Existing depth-scaling approaches force a choice between retaining locally maximal *feature learning* and preserving a form of *feature diversity* in the residual stream arising from random differences between channels. We examine a *joint scaling recipe* that preserves both. With residual width , hidden width , and depth , the key quantity is, by default, the limiting ratio , while correlated initialization (Half Copy) can partially relax the resulting shape constraints. Theory and residual-MLP experiments support convergence at fixed positive to an SDE limit with locally maximal feature learning. Hidden computations can then be redistributed between width and depth while preserving this limit, like turning melody into harmony and back. Language-model experiments nevertheless show performance differences under regrouping at fixed parameter count. These results distinguish approximating a limit (ODE or SDE) from scaling for better performance.

Primary area:
learning theory
Keywords:
residual networks, neural network parameterization, joint width-depth scaling, feature learning, signal propagation, stochastic differential equations, hyperparameter transfer, transformers

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_7UkvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3(+1)
Read review
Paper summary

The paper studies deep residual networks as depth and width scale together to infinity. Existing theory and actual scaling recipes in practice mostly fall into the ODE regime, which ensures feature learning (hidden states can be updated by O(1) during training) but loses feature diversity (random features at random initialization vanish/average out as the network size approaches infinity). This paper analyzes the SDE regime, where feature learning is ensured and feature diversity does not vanish, and proposes scaling/initialization recipes for the SDE regime. A wide range of experiments, from MLPs on CIFAR to GPT-2, provide empirical evidence partly supporting the theory.

Critical strengths
  1. Systematic theory and experiments. This paper systematically studies the scaling limit and DMFT, including various optimizers. Experiments from different angles and different model classes are conducted. The paper has the basic "shape" of an influential work.
Critical weaknesses
  1. Unclear motivation/justification for the SDE regime. The SDE regime indeed preserves feature learning and feature diversity. It is easy to understand and widely accepted that feature learning is good. But it is unclear why we need feature diversity or what the benefits are. The theory seems to be unable to prove an advantage in loss of SDE against ODE. And partly the reason is that DMFT cannot be solved analytically. As far as I can tell, the paper seems to use experiments to show that the SDE regime can lead to lower loss (Figure 6b) and that real LLMs are more likely to be in the SDE regime (Figure 7). But the evidence for advantage is not that strong. If the ODE regime turns out to lead to better performance at late training, discussions in this paper will become useless.
  2. Tension between theory and reality. The theory says that N♯≍N♭≍LN_{\sharp} \asymp N_{\flat} \asymp L, which seems to be how people are scaling models these days, leads to the ODE regime. But later, the paper claims that actual LLMs are in the SDE regime. I am confused; why N♯≍N♭≍LN_{\sharp} \asymp N_{\flat} \asymp L in reality leads to SDE, then does the new recipes proposed necessary if models are already in SDE? The theory suggests N♯≍N♭≍LN_{\sharp} \asymp \sqrt{N_{\flat}} \asymp L, which will lead to a smaller MLP hidden dimension than the embedding dimension, being a little counterintuitive.
Questions for authors
  1. This is a clarification question on interpretation. The theory studies the limit of infinitely large models with a finite number of training steps. Should I interpret this theory for finite-size models as the early stage of training, where the number of training steps is much smaller than the model size or some function of the model size? It seems that the experiments in this paper also focus on a very early stage of training. If it is not due to limited compute, I guess it is intended to match the theory?
  2. If my interpretation is correct in 1., the question is: what will the late training be like? Will SDE be beneficial for the late stage of training? For a finite-size model, even in the SDE regime, does the randomness due to initialization (or random features) go away at very late or long training?
  3. If my interpretation is correct in 1., what determines the validity of the theory for a finite-size model? Or is there any quantitative threshold for the number of training steps and the model size such that the DMFT and scaling argument are accurate?
  4. What is the difference between Eq. (6) and Eq. (8)? My understanding is Eq. (6) from SDE and (8) from ODE? But they are the same scaling, no? Based on the error, the optimal scaling is N♯≍N♭≍LN_{\sharp} \asymp \sqrt{N_{\flat}} \asymp L, which leads to SDE? I am confused.
  5. Based on your Eq. (6), let's say the total number of parameters is Ntot∝LN♭N♯N_{\rm tot} \propto LN_{\flat}N_{\sharp}, the current scaling method, N♯≍N♭≍LN_{\sharp} \asymp N_{\flat} \asymp L will lead to a squared error as O(1/Ntot1/3)O(1/N_{\rm tot}^{1/3}), which is actually close to or consistent with Chinchilla scaling laws. With your suggested scaling method, N♯≍N♭≍LN_{\sharp} \asymp \sqrt{N_{\flat}} \asymp L, the model size scaling of loss will be O(1/Ntot1/2)O(1/N_{\rm tot}^{1/2}). If you are correct, does this mean we can change neural scaling laws by better shapes of scaling up models?
  6. Some recent practitioners (see modded nanogpt repo) suggest initializing down-projections to be zero matrices. Does this effectively introduce correlation between layers, or to make the layers as ODE (zero drift, zero randomness, initially)?
10 comments
AuthorsOpenReview ↗

Response to Reviewer 7Ukv (1/3)

Response to Reviewer 7Ukv (1/3)

Critical weaknesses 1–2 · Finite GPT-2 and the ODE and SDE limits

The ODE limit has been a theoretically fruitful route, and what concerns us is precisely why it hasn’t been as useful to practitioners as one might have hoped. We see one possible reason. Our GPT-2 measurements suggest that, at initialization, the cumulative energy of the random residual increments through depth is about two orders of magnitude larger than the energy of the input embedding. Yet under the HP transfer recipe associated with the ODE limit, this contribution vanishes as we scale up. If a contribution that dominates at finite size vanishes in the limit, is that limit a useful point of comparison for the finite networks we actually train?

In our experiments, the ODE HP transfer recipe gives substantially higher training loss than two alternatives we came up with simply by trying to retain that random variation through depth. We therefore strongly believe that what we call the SDE limit gives a more useful description of training dynamics in residual networks, in part because it captures this contribution.

As for which asymptotic regime actual LLMs belong to overall, we’d leave that open. Outside of HP transfer experiments, people rarely follow a fixed recipe all the way to its limit. Any actual scale-up covers only a finite stretch of that path, and people may change the hyperparameters along the way if the gains fall short. Our point is that the ODE regime, which has dominated the theoretical discussion in recent years, loses an important feature of the dynamics that the SDE retains. The same standard applies to us. If the SDE turns out to miss another contribution that dominates in finite networks, then it’s our turn to revise the limit.

That leaves us more optimistic about HP transfer itself. The fixed-ratio and Half Copy recipes give us two concrete ways to retain the random residual contribution as we scale up. If they hold up at larger scales, HP transfer may be more broadly useful than it looks today. The difficulty may have been in the choice of limit.

Critical weakness 2 · Single-layer MLP width and sliding

We agree that N♯≍N♭≍LN_\sharp\asymp\sqrt{N_\flat}\asymp L, which makes a single MLP layer narrower than the embedding, looks counterintuitive at first. We kept N♭/(N♯L)N_\flat/(N_\sharp L) fixed to retain random residual variation and the scale of local feature learning. CIFAR-10 uses the balanced scaling in Eq. (8). On Tiny Shakespeare we instead grow (N♭,N♯,L)(N_\flat,N_\sharp,L) by (R3/2,R1/2,R)(R^{3/2},R^{1/2},R) to match ODE in depth and block parameters. We had no reason to expect either shape to be optimal, yet both gave substantially lower training loss than ODE at larger sizes. The allocation of width and depth itself changes the scale of residual randomness, so this comparison cannot isolate the respective contributions of shape and random variation. Half Copy, however, retains the conventional shape and uses correlations at initialization to preserve random residual variation. It too attains lower training loss than ODE. This makes us believe that the variation ignored by the ODE limit deserves as much attention in finite networks as the choice of shape.

The sliding diagram is on p. 3 of 00_rebuttal_figures.pdf at the top level of the updated supplementary ZIP.

The figure shows one way to understand this shape. First keep depth fixed and widen the embedding and the MLP in the same proportion. This preserves their width ratio, so it is easy to see why such a network might train effectively. Now split the same hidden channels from one wide block into two consecutive residual blocks. Channels 1–4 in the figure compute exactly the same functions before and after the split. Channels 5–8 keep their original weights but read the residual state after its update by the first four channels. If that update is small, the contributions from the last four channels will also be close. More generally, widening both dimensions by a factor of q2q^2 and then splitting each wide block into qq layers gives (q2N♭,qN♯,qL)(q^2N_\flat,qN_\sharp,qL). Each MLP layer is indeed narrower than in the proportionally widened network, but all the hidden channels are still there. Sliding explains why, under suitable conditions, moving those channels from parallel to sequential computation can have only a limited effect on the macroscopic dynamics.

AuthorsOpenReview ↗

Response to Reviewer 7Ukv (2/3)

Response to Reviewer 7Ukv (2/3)

Q1–Q3 · Training duration and late-time dynamics

Your interpretation in terms of early training is broadly right. We first fix the number of training steps TT, then take the widths and depth to infinity. For a finite network, the longer we train, the more important it is to check whether the mean-field approximation remains accurate. The error estimate in Result 3 depends on TT, and we do not yet have a threshold for the number of steps as a function of network size. The left and middle panels of Figure 5 give a concrete point of reference, with the fitted slopes in Appendix G.1. On CIFAR-10, comparing the seven larger network sizes with the largest-model ensemble, the fitted slopes of output and residual-kernel error against size qq are −0.95-0.95 and −0.90-0.90 at the first epoch, close to the theoretical prediction of −1-1. By the tenth epoch, both errors still decrease with size, but the slopes have changed to −0.23-0.23 and −0.26-0.26. At the sizes we could test, the theory does give a better quantitative description of the early errors.

There is also a direct reason for holding the number of training steps fixed in the recipe experiments. The learning rate is one of the hyperparameters we transfer. Once we have selected it on a small model, how far can the larger model drive down training loss with the same number of updates? Compute limited how long we could train, but intermediate measurements let us check whether a recipe's advantage happened to appear only at the endpoint. For example, at all five Tiny Shakespeare sizes R=4,5,6.25,8,10R=4,5,6.25,8,10, both the fixed-ratio recipe and Half Copy have lower three-seed mean training loss than ODE at both 2,500 and 5,000 steps. Half Copy's advantage is already visible at step 1,000 at all five sizes. Whether ODE would catch up with much longer training remains open.

We also do not have a definitive answer about how much randomness from initialization remains very late in training. After 15,000 steps in the 12- and 24-layer GPT-2 models, the cumulative squared energy of the layerwise residual increments remains large, and we also see positive cross-layer correlations that developed during training. This measurement suggests that variation through depth is still worth retaining in the description even after the network has learned cross-layer structure. To establish how much of this variation can still be attributed to initialization, we would have to track how the trained features depend on the initial weights. The energy at the end of training alone does not answer that question.

AuthorsOpenReview ↗

Response to Reviewer 7Ukv (3/3)

Response to Reviewer 7Ukv (3/3)

Q4–Q5 · Equations (6) and (8) and loss scaling

Your conversion of the exponents is correct for the approximation error in Eq. (6). We obtain Eq. (8) from that same bound. At a given residual-block parameter budget NtotN_{\rm tot}, we balance the three approximation-error terms for SGD/Adam. When the two widths and depth grow proportionally, Eq. (6) gives an O(Ntot−1/3)O(N_{\rm tot}^{-1/3}) upper bound on the mean squared error. With N♯≍L≍N♭N_\sharp\asymp L\asymp\sqrt{N_\flat} as in Eq. (8), the bound is O(Ntot−1/2)O(N_{\rm tot}^{-1/2}).

In both cases Eq. (6) compares a finite network at a fixed training step with the mean-field prediction at its current ratio, αN=N♭/(N♯L)\alpha_N=N_\flat/(N_\sharp L) (Appendix D). Under proportional growth, αN→0\alpha_N\to0. Comparing directly with the final ODE prediction would also require controlling ∣o∞(αN)−o∞(0)∣|o_\infty(\alpha_N)-o_\infty(0)|.

This is why we discuss the choice of network shape explicitly in §3.3. Even if we knew the approximation errors along both routes exactly, which model assigns a higher probability to the correct next token would still be a separate question. To predict how loss changes with scale beyond an empirical fit such as Chinchilla, we would need to know both how well each limiting model learns the task and in which direction, and by how much, task loss changes as a finite network departs from its own limit.

Q6 · Zero initialization of the down-projections

Your intuition about initialization is correct. Setting the down-projection to zero makes that branch's contribution to the initial residual variance exactly zero. Doing so in every layer does not by itself introduce random correlations between layers.

We emphasize residual variance because it is substantial in many finite networks, yet vanishes as we scale up along the ODE recipe. We do not regard it as the only quantity that matters. There is an interesting detail in modded-nanoGPT itself: some later versions added U-Net-style skip connections, so even when each branch's down-projection still starts at zero, states from earlier layers are added back into the residual stream. This example shows that zero initialization can be part of an effective recipe, but it gives insufficient evidence for the stronger claim that the best choice is to make the initial forward evolution of the residual stream through depth approach the trivial evolution suggested by the ODE limit. modded-nanoGPT record history

Once training starts and there is a nonzero backpropagated signal, the down-projection update uses the hidden features produced by the random up-projection. We have yet to test how much the ratio α=N♭/(N♯L)\alpha=N_\flat/(N_\sharp L) affects the subsequent dynamics under this initialization.

Reviewer_HiSROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a technical analysis of scaling parameters in dense MLP residual blocks. Specifically, the authors consider a joint width-depth scaling pattern that is motivated by the fact that it should preserve hidden feature learning and feature diversity. This is an alternative to scaling patterns usually studied like muP, depth-muP etc.

The authors explore the theory of their scaling limit, focusing on the limit ratio N♭/(N♯L) → α where residual width is N♭, hidden width is N♯, and depth is L. They show three results: how the training dynamics converge to something that is optimizer dependent, how the initialization converges to a continuous Gaussian random walk through depth, and how the finite-size error relative to the limit behaves.

The experiments show that the residual MLPs on CIFAR-10  do converge as predicted by the theory in that limit.

As a consequence of the theory, the authors study sliding, i.e., trading width for depth and vice versa while keeping the limit unchanged (the better architecture depends on the task), and propose two new methods: the fixed-alpha SDE recipe, as a new scaling recipe that keeps α constant but changes the network’s shape, and half copy as a way to better initialize models to save on parameters.

Critical strengths

The work is well motivated, innovative, and the theory is strong.

Critical weaknesses

The experiments are somewhat small: Tiny Shakespeare, CIFAR, have three seeds and Half Copy experiments only one seed.

The notations are heavy and can make the text hard to follow at times.

Questions for authors

Could the authors clarify what are (if any) the practical takeaways, since the theory doesn’t predict which width–depth ratio will perform best?

Why focusing on training losses? what happens to validation losses?

4 comments
AuthorsOpenReview ↗

Response to Reviewer HiSR

Critical weaknesses · Experimental scale, seed counts, and notation

We agree that the scale of both tasks is limited. Whether the recipes retain the same advantages in larger-scale language-model training remains to be tested. We would like to clarify the seed counts for two distinct Half Copy experiments. Figure 6 compares scaling recipes on CIFAR-10 and Tiny Shakespeare. Every Half Copy data point uses three seeds. In the specific pair of models used to estimate parameter savings, the smaller Half Copy model attained lower training loss in all three paired seeds on both tasks. The left panel of Figure 8 asks a different question: how does changing depth affect performance at a fixed recipe and parameter budget? That Tiny Shakespeare experiment did use just one seed, so we report only that the one-layer model attained the lowest training loss in that scan.

We also agree that the notation can make the argument hard to follow in places. In revision we will bring together, where the model is first introduced, the relationships used throughout the paper. The residual stream has width N♭N_\flat, each layer has N♯N_\sharp hidden channels, and across LL layers there are K=N♯LK=N_\sharp L hidden channels in total. The ratio N♭/KN_\flat/K determines the scale of the initial random residual variance, and we denote its limit by α\alpha. This will give readers a single place to refer back to when comparing width and depth.

Q1 · Practical implications of the fixed-ratio recipe

We kept N♭/(N♯L)N_\flat/(N_\sharp L) fixed to retain random residual variation and the scale of local feature learning. CIFAR-10 uses the balanced scaling in Eq. (8). On Tiny Shakespeare we instead grow (N♭,N♯,L)(N_\flat,N_\sharp,L) by (R3/2,R1/2,R)(R^{3/2},R^{1/2},R) to match ODE in depth and block parameters. We had no reason to expect either shape to be optimal, yet both gave substantially lower training loss than ODE at larger sizes. The allocation of width and depth itself changes the scale of residual randomness, so this comparison cannot isolate the respective contributions of shape and random variation. Half Copy, however, retains the conventional shape and uses correlations at initialization to preserve random residual variation. It too attains lower training loss than ODE. This makes us believe that the variation ignored by the ODE limit deserves as much attention in finite networks as the choice of shape.

The fixed-ratio and Half Copy recipes give us two concrete ways to retain the random residual contribution as we scale up. If they hold up at larger scales, HP transfer may be more broadly useful than it looks today. The difficulty may have been in the choice of limit.

Q2 · Training and held-out loss

We use training loss as our primary yardstick because we want to disentangle how readily a model fits the data from how well that fit generalizes. The former tells us more directly what the model can do under a given training recipe. The latter also depends on the quality and structure of the data.

The training and held-out loss plots for Tiny Shakespeare and CIFAR-10 are on p. 1 of 00_rebuttal_figures.pdf at the top level of the updated supplementary ZIP.

Tiny Shakespeare has only about a million training characters. Our Transformers can readily fit that corpus, and the effect of scaling them up is striking when we plot training and held-out loss side by side. For every recipe, each increase in size lowers training loss and raises held-out loss. Even the ordering of the three recipes shown in the figure reverses at every size: Half Copy fits the training data best, followed by fixed-ratio SDE and then ODE, while ODE has the lowest held-out loss and Half Copy the highest. At comparable training loss, we see no clear sign that any one recipe consistently generalizes better.

On CIFAR-10, our residual MLPs also show signs of overfitting, but the effect is much less pronounced. Training and test loss often improve together. At larger sizes they can diverge: when we double the fixed-ratio model from 33.5M to 67.1M residual-block parameters, training loss falls while test loss rises in all three seeds. Some of the gains in fitting do carry over, though. In the Half Copy–ODE comparison used to estimate parameter savings, Half Copy reaches essentially the same mean test loss with about 46% fewer total parameters.

We chose these experiments with our compute budget in mind. CIFAR gave us a model we could control closely enough to check the phenomena our theory predicts. To see whether those ideas might also be useful in practice, we wanted to test them on Transformers. That meant training a whole family of models across sizes, recipes, and seeds, so we ran that comparison on a small corpus. With more compute, we’d repeat it at a data scale where we could also tell how much of the improvement in fitting carries over to held-out performance.

Reviewer_HiSROpenReview ↗

Thank you for your rebuttal

Dear authors, thank you very much for the time you took to answer my questions. I will keep my score.

Thank you for your careful review and for taking the time to consider our responses!

AuthorsOpenReview ↗

Thank you and revision update

Thank you again for taking the time to read our paper and for your questions and discussion. We have uploaded a revised manuscript. In particular, Lines 109–113 now bring together the definitions of the two widths, depth, total hidden-channel count, and their ratio, in response to your comment on the notation. Lines 516–539 also expand the discussion of what the theory can offer hyperparameter transfer and where its usefulness may be limited. Your comments helped us clarify both the presentation and the practical message of the paper.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Reviewer mtZP (1/3)

Response to Reviewer mtZP (1/3)

Q1 · Concentration and stability

We derived the limiting dynamics (Result 1) using mean-field methods from statistical physics. For general networks undergoing training, we have yet to prove from the microscopic dynamics that the concentration and stability required for mean-field closure hold. When we go on to estimate the finite-network errors (Result 3), we therefore explicitly assume that the correlations induced by training are weak enough to give quantitative control of fluctuations in empirical averages.

For Eq. (156) specifically, we impose an additional sufficient condition on the sampling errors from the hidden channels in each layer. Once these errors have propagated to the output, their cross-layer covariances must have a sufficiently small cumulative contribution that averaging over depth gives the usual O((N♯L)−1/2)O((N_\sharp L)^{-1/2}) root-mean-square error. Similarly, when averaging over the residual coordinates shared across layers, we assume that their sampling fluctuations have root-mean-square scale O(N♭−1/2)O(N_\flat^{-1/2}). Equation (156) is a condition used to derive the error rate in Appendix D. We have not shown that independent initialization implies that a trained network satisfies it.

Result 3 estimates the error in macroscopic observables at a fixed number of training steps. For example, the qqth output, fq=(γ0N♭)−1∑iWqiLhiL♭f_q=(\gamma_0N_\flat)^{-1}\sum_i W^L_{qi}h_i^{L\flat}, is itself a coordinate average of the products of readout weights and residual features. In the mean-field description the histories of the residual coordinates have the law of a common effective process. There is a direct analogy with the strong and weak errors of Euler–Maruyama discretization. Under the usual regularity conditions, comparing a discrete and a continuous trajectory driven by the same noise typically gives a strong error of O(L−1/2)O(L^{-1/2}). Comparing the expectations of a smooth observable under the two processes can give a weak error of O(L−1)O(L^{-1}). The depth term in Result 3 falls in the latter category: it compares the population values of a macroscopic statistic at finite and continuous depth. In Appendix D, the local weak errors of O(L−2)O(L^{-2}) per layer accumulate under stable propagation to give O(L−1)O(L^{-1}).

The Gaussianity claim in the second part of Result 1 concerns the limiting distribution, so it does not require a quantitative rate such as Eq. (156). The concentration needed here can be stated more precisely. Fix finitely many inputs, depths, and training steps. If the empirical Gram matrix of the corresponding residual histories converges to a deterministic kernel, the joint rotation invariance preserved by SGD and spectral Muon suffices to show that the values of any fixed residual coordinate at these locations converge jointly to a Gaussian distribution. Any fixed number of distinct coordinates also become independent in the limit. Both the Gaussian shape and the asymptotic independence of coordinates are conclusions of this step. What remains to be proved rigorously is the concentration of the empirical Gram matrix for general trained networks. The prediction about distributional shape is also easy to test. For Adam, we give an explicit counterexample in which the macroscopic statistics still concentrate after one update but an individual coordinate is already non-Gaussian. We also see this departure in the trained nonlinear network on the logic task. On the other hand, in a GPT-2-style model with about 124 million parameters trained throughout with NS5 Muon, we observe nearly Gaussian residual activations and output Jacobians, in line with the prediction.

AuthorsOpenReview ↗

Response to Reviewer mtZP (2/3)

Response to Reviewer mtZP (2/3)

Q2 · The Muon spectral clipping threshold

Our derivation of the spectral bias for Muon relies on local smoothness of the spectral map, so its treatment of singular values near the clipping threshold is incomplete. This is precisely why we worked out the example in Eq. (170), placing a population singular value at the kink of the clipping function. Upward fluctuations of the sampled singular value are clipped, while downward fluctuations remain in the linear region, so the first-order fluctuations no longer cancel in expectation. When the two widths are of the same order, the mean bias in a single layer can then change from O(N♯−1)O(N_\sharp^{-1}) to O(N♯−1/2)O(N_\sharp^{-1/2}).

We have not established how close the population singular values may approach the threshold as width grows while retaining the inverse-width bias bound.

How much this extra bias affects the whole network also depends on how many layers have a gradient singular value close to the threshold at the same update. Suppose the relevant singular values cross the threshold smoothly as a function of depth, with finitely many crossings and a nonzero slope at each. For a sufficiently fine depth grid, the fraction of layers falling within an O(N♯−1/2)O(N_\sharp^{-1/2}) neighborhood of the threshold should then also be about O(N♯−1/2)O(N_\sharp^{-1/2}). Multiplying the O(N♯−1/2)O(N_\sharp^{-1/2}) extra bias per layer by the fraction of affected layers brings its depth-averaged contribution back to O(N♯−1)O(N_\sharp^{-1}). We believe this order-of-magnitude argument, but we do not yet have a rigorous proof.

Practical implementations of Muon often use a polynomial spectral map. The polynomial has no hard-clipping kink at the corresponding point, so in principle a local smooth expansion is possible at sufficiently large widths. If finite-sampling fluctuations are still large enough to move a singular value across a region where the slope of the polynomial changes sharply, the asymptotic bias rate may be difficult to observe at attainable widths.

Q3 · Half Copy

Neither Result 1 nor Result 3 covers Half Copy. In designing Half Copy, we only aimed to retain one basic macroscopic statistic as the network scales up: the random residual variance accumulated through depth at initialization. Matching that statistic does not guarantee that other macroscopic statistics agree with those under independent Gaussian initialization. We conjecture that Half Copy converges to its own limiting process, with training dynamics that differ from those under independent initialization. We plan to derive this limit in future work, but doing so may require more advanced mathematical or physical techniques. Even with an MSRJD path integral, one would have to work out carefully how correlations between channels in the same group enter the dynamical equations.

We also tested several other correlated random-matrix initializations with the same sole design goal of retaining the initial residual variance. In our limited comparisons, each of these methods achieved lower training loss than the ODE recipe in some tests, but we did not observe a consistent winner. We chose to discuss Half Copy in the paper because its construction is simple and follows directly from the idea of copying hidden channels.

AuthorsOpenReview ↗

Response to Reviewer mtZP (3/3)

Response to Reviewer mtZP (3/3)

Q4 · Held-out performance

We use training loss as our primary yardstick because we want to disentangle how readily a model fits the data from how well that fit generalizes. The former tells us more directly what the model can do under a given training recipe. The latter also depends on the quality and structure of the data.

The training and held-out loss plots for Tiny Shakespeare and CIFAR-10 are on p. 1 of 00_rebuttal_figures.pdf at the top level of the updated supplementary ZIP.

Tiny Shakespeare has only about a million training characters. Our Transformers can readily fit that corpus, and the effect of scaling them up is striking when we plot training and held-out loss side by side. For every recipe, each increase in size lowers training loss and raises held-out loss. Even the ordering of the three recipes shown in the figure reverses at every size: Half Copy fits the training data best, followed by fixed-ratio SDE and then ODE, while ODE has the lowest held-out loss and Half Copy the highest. At comparable training loss, we see no clear sign that any one recipe consistently generalizes better.

On CIFAR-10, our residual MLPs also show signs of overfitting, but the effect is much less pronounced. Training and test loss often improve together. At larger sizes they can diverge: when we double the fixed-ratio model from 33.5M to 67.1M residual-block parameters, training loss falls while test loss rises in all three seeds. At the five residual-block budgets shared by all three plotted recipes, after the same five epochs, both fixed-ratio SDE and Half Copy have lower mean test loss than ODE at four budgets and slightly higher mean test loss at 16.8M block parameters. Some of the gains in fitting do carry over, though. In the Half Copy–ODE comparison used to estimate parameter savings, Half Copy reaches essentially the same mean test loss with about 46% fewer total parameters.

We chose these experiments with our compute budget in mind. CIFAR gave us a model we could control closely enough to check the phenomena our theory predicts. To see whether those ideas might also be useful in practice, we wanted to test them on Transformers. That meant training a whole family of models across sizes, recipes, and seeds, so we ran that comparison on a small corpus. With more compute, we’d repeat it at a data scale where we could also tell how much of the improvement in fitting carries over to held-out performance.

Q5 · Finite references and multi-step self-consistent predictions

The comparison with alternative reference ensembles for Figure 5 is on p. 2 of 00_rebuttal_figures.pdf at the top level of the updated supplementary ZIP.

Figure 5 uses the mean over 32 independently initialized models at q=96q=96 as its reference, as defined in Appendix G.1. In the supplementary figure we use ensembles of 32 models at q=80q=80 and q=40q=40 as alternative references. At the first epoch the two curves nearly coincide when we switch the reference from 96 to 80. Over the common range of larger sizes, 22≤q≤6422\le q\le64, the fitted output-error slope is −0.94-0.94 for both references, and the residual-kernel slopes are about −0.91-0.91 and −0.90-0.90, respectively. At the tenth epoch, errors still generally decrease with size for all three references, but the output-error values are more sensitive to the reference, especially at q=40q=40.

We have not obtained an independent multi-step numerical solution of the continuous-depth self-consistent equations for a nonlinear residual network. A fixed-depth solution can be a starting point, but one would still have to solve for the covariances and responses across depths and training steps as the depth grid is refined, and check that the predictions stabilize. A calculation at a single fixed depth cannot answer that question. Earlier work on the depth limit of residual networks encountered the same difficulty and likewise used ensemble predictions from large finite networks as the reference in its CIFAR experiments. Bordelon et al., ICLR 2024, Figure 5

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.