acceptodds
Under review as a conference paper at ICLR 2027

Beyond Linear and Overcomplete Regimes: A Mean-Field Analysis of Bottleneck Autoencoders

Abstract

Autoencoders (AEs) learn low-dimensional representations by mapping data into a latent space while minimizing reconstruction error. Despite their empirical success, theoretical understanding remains limited and largely restricted to one of three setting: linear models, shallow non-linear models or settings without a bottleneck. In this work, we go beyond those settings and study nonlinear AEs with a fixed finite-dimensional bottleneck in the mean-field (MF) regime. We derive explicit MF learning dynamics for both encoder and decoder, providing a tractable characterization of training in the nonlinear setting. We show that, over finite time horizons, the empirical risk of finite-width networks trained with stochastic gradient descent closely tracks the MF risk trajectory with high probability. At optimality, we further establish that the finite-width risk converges to the MF optimum, demonstrating that finite networks are sufficiently expressive to approximate the infinite-width solution. As a step towards characterizing learned representations, our MF analysis shows connections between AEs, early learning of approximately linear representations, and principal curves.

Primary area:
learning theory
Keywords:
Autoencoder, Principal curve, Mean field analysis, Representational learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZM51OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops a particle mean-field framework for nonlinear autoencoders with a fixed finite-dimensional bottleneck. Using sixth-order regularization, it establishes confinement of the parameter dynamics, quantitative finite-width SGD tracking guarantees, and approximation bounds for the optimal reconstruction risk. Separately, in the unregularized setting, it studies early-time alignment under suitable initialization and connects blockwise-optimal encoder–decoder maps to self-consistent principal manifolds under additional assumptions.

Critical strengths

1. Detailed mean-field analysis of bottleneck autoencoders.

The paper develops a rigorous framework for nonlinear autoencoders with a fixed finite-dimensional bottleneck. The quantitative finite-width tracking guarantees and the approximation bounds for the optimal risk provide a substantive theoretical foundation for this architecture.

2. Clear exposition and careful technical analysis.

The proof sketches and supplementary proofs are well organized and make the main arguments accessible. The appendices provide a thorough verification of the regularity conditions and estimates required by the analysis. In particular, the sixth-order regularizer allows the authors to establish confinement of the dynamics uniformly in width and time without imposing a compact parameter domain. This is a useful technical component, and its detailed justification strengthens the overall presentation.

Critical weaknesses

1. The representation claims need a clearer presentation and more targeted numerical support.

Although early learning of approximately linear representations is highlighted in the abstract and introduction, the main text provides only an informal discussion, with the precise statement and assumptions of Theorem 10 deferred to Appendix M. A concise statement and a representative figure would make this contribution easier to assess.

The supplementary experiments also only partially test the theoretical predictions. Distance from the PCA subspace does not directly measure alignment with the kernel-weighted direction predicted by Theorem 10, and the self-consistency diagnostic does not verify the nearest-point projection condition underlying Theorem 5. More targeted validation, with selected results included in the main text, would clarify what learning behavior the theory explains.

2. The connection between the regularized mean-field framework and the representation results remains incomplete.

The main mean-field guarantees require λ>0\lambda>0, whereas the principal-curve characterization in Theorem 5 and the early-alignment result in Theorem 10 concern λ=0\lambda=0. These representation results therefore do not directly describe the regularized models covered by the finite-width tracking theorem. Moreover, Theorem 5 assumes block-wise optimality, which the finite-time dynamics are not shown to approach. Although these differences do not invalidate the individual results, they leave their connection unclear. The paper should explain which representation-level conclusions are supported by the regularized framework and which should be understood as separate results.

Minor comments

  • On p. 4 and in Appendix F, several occurrences of "SDG" should be "SGD".
  • On p. 7, line 346, “infrimum” -> “infimum".
  • On p. 7, line 346, “within within” ->“within.”
  • On p. 9, line 451, the reference to the numerical experiments appears to point to Appendix L; it should point to Appendix K.
Questions for authors
  1. Could the authors include a concise statement of Theorem 10, its key assumptions, and a representative numerical illustration in the main text, possibly by shortening some proof sketches and making use of any additional page allowance during the rebuttal period?

  2. How should the λ=0\lambda=0 representation results in Theorems 5 and 10 be interpreted in relation to the λ>0\lambda>0 mean-field guarantees? Are these intended as separate results, or can a connection be established through an analysis of small positive regularization or a controlled vanishing-regularization limit?

  3. Could the experiments more directly test the predicted kernel-PCA alignment and both conditions underlying the principal-curve interpretation—self-consistency and nearest-point projection?

2 comments

We thank the reviewer for the careful reading and such a concise and accurate summary of our contributions and the positive review. We address the posed questions below.

Connection between the λ>0\lambda>0 framework and the λ=0\lambda=0 representation results.

We thank the reviewer for this question. In the submitted version, Theorems 5 and 10 were indeed separate results for λ=0\lambda=0.

Heuristically the correct way to read Theorem 10 is indeed to consider the situation where the regularization parameter λ\lambda is very small. In this case there shouldnt be a real difference between this and λ=0\lambda =0. In an earlier draft we did have a version for small λ\lambda. However we thought that the argument and the discussion was becoming too technical so reverted to the λ=0\lambda=0 case (which provided a proof of concept). Following your review, we have now reinstated the early-alignment result. This establishes the connection through small positive regularization, as the reviewer suggests. The results are essentially similar to the λ=0\lambda=0 case and are basically a technical extension

In the revision this is added to a new Appendix M.3, which extends Theorem 10 to the regularized mean-field dynamics. For λ>0\lambda>0, the moments CsC _ s and BsB _ s again align with a single data direction during the small-output phase. This direction is the same top eigenvector vˉ1\bar v _ 1 of the kernel covariance C\mathcal{C} as in Theorem 10, and the overlap κ1\kappa _ 1 with the initialization is also unchanged. The reason is that all neurons start with the same norm R1R _ 1, so at initialization the regularizer damps every neuron at the same rate dλ=λR14d _ \lambda=\lambda R _ 1^4. In the leading linear system this damping multiplies all modes by the same factor e−dλse^{-d_\lambda s}, which slows the growth from rate ρ1\rho _ 1 to rλ=ρ1−dλr _ \lambda=\rho _ 1-d _ \lambda but does not change its direction, so no eigenvector perturbation argument is needed. The error bound is the same as Theorem 10, with the exponent θλ=(ρ1−ρ2)/rλ\theta _ \lambda=(\rho _ 1-\rho _ 2)/r _ \lambda in place of θ\theta, and one extra term λ(sδ+sδ2)\lambda(s _ \delta+s _ \delta^2), where sδ=1rλlog⁡(δ/∥Z0∥2)s _ \delta=\frac{1}{r _ \lambda}\log(\delta/\|Z _ 0\| _ 2) is the exit time; this term comes from the motion of the features under the regularizer. At λ=0\lambda=0 we have rλ=ρ1r _ \lambda=\rho _ 1, θλ=θ\theta _ \lambda=\theta, and the extra term vanishes, so we recover Theorem 10 exactly. There are two limitations that we state openly. First, λ\lambda has to be small compared with the time window, roughly λsδ2≲1\lambda s _ \delta^2\lesssim 1, that is, λlog⁡2(δ/∥Z0∥2)≲rλ2\lambda\log^2(\delta/\|Z _ 0\| _ 2)\lesssim r _ \lambda^2, so we cannot send the initialization to zero with λ\lambda fixed. Second, we additionally assume that every encoder and decoder neuron has the same norm R1R _1 at initialization, and that λR14≤ρ1/2\lambda R _ 1^4\le\rho _ 1/2. Theorem 5, on the other hand, remains a statement about the unregularized risk and about blockwise-optimal maps, which we do not show the dynamics reach, and we will say clearly in the revision which conclusions are supported by the regularized framework and which are separate results.

Additional Experiments

We added the additional experiments in Section 6 of the revised version. Specifically we show that the discrepancy between finite-width and mean-field risk trajectories scales roughly as N−1/2N^{-1/2}, and the decoder’s self-consistency and encoder’s projection residuals decrease during training. Reconstructions initially approach the weighted kernel PCA line, and the encoder moment and decoder tangent first align with the leading direction of the initial kernel. Those observations are consistent over small regularizations. We note that in the original experiments we were considering PCA of covariance matrix as we discuss in section M.2.3 that under some extra conditions the PCA of the default covariance and the kernel weighted covariance are close.

Adding early time PCA Theorems to the main

We have now extended the main to both include some experiments and a more detailed informal version of the regularized early time PCA theorem. This is now stated as Theorem 6. However the precise assumptions are still contained in the appendix as it needs substantial notation to be defined which takes up a lot of space. However the principal ideas we want to convey we hope is relatively clear.

Thank you for the revisions and additional experiments. The regularized early-alignment result, its inclusion in the main text, and the targeted kernel-PCA and projection diagnostics address my concerns.

The informal statement of Theorem 6 would benefit from a brief explanation of the key conditions in Appendix M.3, particularly equal-norm initialization and the coupling between regularization strength and alignment time.

I therefore maintain my score.

Reviewer_FvHTOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

Autoencoders can be analyzed by taking some sort of an infinite width/depth limit i.e. the mean-field approximation (actually, as far as I understand, this can be done for other neural networks as well, and surveying the literature turns up several other cases). The authors use this to study this specific case in depth and perform various empirical studies to show that for a wide variety of autoencoder-like families, the convergence to the infinite case is tracked by the mean-field setup they use which resolves the entire process into a diff-eq-like situation.

Critical strengths

The paper seems to have made a thorough effort in analyzing the infinite-case limit and done a fairly extensive set of derivations.

Critical weaknesses
  • I found the paper extremely long and difficult to follow, with a feeling that there was an attempt to blind the reader with a lot of math.

  • It was not clear to me what the takeaway was. Specifically, the set of empirical observations converges to the mean-field prediction - so what ? How can the mean field theory be used to actually train real, finite autoencoder-like structures ?

  • There didn't seem to be any actual in-depth use of the infinite case, just using a 1024-width case.

Questions for authors
  • Shouldn't the mean field theory lead to some sort of "solving in continuous domain, and then discretizing back" instead of a finding that basically shows that the points on the curve mostly lie where they should ?
3 comments

We thank the reviewer for the feedback. Before going into the individual points, we would like to give a short and informal description of the mean field viewpoint to give context to the results we present.

A neural network of width NN can be seen as a collection of NN neurons that interact with one another during training. As the network becomes wider, it becomes increasingly difficult to follow each neuron individually. The mean field viewpoint offers a different way of looking at the same training process: rather than following individual neurons, one follows how the neurons are distributed as a whole. In the large width limit, this distribution evolves smoothly in time, and its evolution is described by a PDE. This continuous description allows one to leverage the rich literature of PDEs to answer several questions. For instance it lets us ask broad questions about training, such as whether the dynamics are well behaved, where they are heading, and what kind of representation the network learns. In this work we build the basis for such an analysis in the future.

This viewpoint has a rich history. It was first developed for two-layer networks (Mei et al., 2018; Chizat & Bach, 2018; Sirignano & Spiliopoulos, 2020; Mei et al., 2019) and was subsequently extended to multilayer networks in which every layer is wide (Nguyen, 2019; Sirignano & Spiliopoulos, 2022; Nguyen & Pham, 2023). For autoencoders, the closest work is Nguyen (2021), which studies the weight-tied setting without a fixed bottleneck. Our work builds on this line of research and, to the best of our knowledge, is the first to derive the governing PDE along with an error-rate for general non-linear autoencoder with a bottleneck. We then (e.g. in Theorem 10) give some preliminary results analyzing the concerned PDE.

Takeaways

The lens of mean field theory frees us from one of the constraints of the network, its size, and gives a bird's-eye view of how the network trains. It is not a numerical algorithm (although it can inform one) to replace SGD; rather, finite-width SGD is the discretization of the PDE. So, we can think of the PDE as the equations governing and determining a phenomenon whose discretized version is the SGD autoencoder we implement in practice.

We first show (in Theorem 1) that the coupled PDEs that govern the dynamics are well-posed and have a unique solution under mild assumptions. Then we show (in Theorem 2) that they do indeed control the dynamics of the finite-width network, i.e. the finite-width SGD trajectories actually track the PDE with derived error rates depending on the width of the network (over any fixed training horizon the error is of order (N−1∨ϵ) (D+log⁡(N/ϵ))\sqrt{(N^{-1}\vee\epsilon)\,(D+\log(N/\epsilon))}, where ϵ\epsilon is the step size and D=d+bD=d+b is the input dimension plus the bottleneck dimension; in short, it decays like 1/N1/\sqrt{N} up to a logarithmic factor once ϵ≤1/N\epsilon\le 1/N). This theorem is what controls the discretization error. This informs the size of the network we choose, because the larger the width the better our autoencoder will track the real phenomenon that is described by the PDE.

In comparison to theorem 2 which tracks the real time training error, Theorem 3 compares the differences (of order 1/N1/N) in-between the best possible (i.e. smallest) risks achievable by the evolution of the PDE and the finite autoencoder. We note that since the risk is non-convex we cannot a priori guarantee that we will in fact converge to the global minimum. In such un-optimistic situations, Theorem 2 is a better tracking mechanism.

With the PDE in hand, we can start asking what the autoencoder actually learns. It is a classical result that a \emph{linear} autoencoder ends up learning PCA (Baldi & Hornik, 1989; Esser et al., 2023). In Theorem 5 we attempt to generalize this observation to non-linear models. In particular we generalize linear principal components to principal curves. We show that, under mild assumptions, a nonlinear autoencoder learns principal curves (Hastie and Stuetzle, 1989) at its optimum.

Finally in Theorem 10, we show that the PDE, in its initial phase, does behave like linear PCA.

The experiments illustrate these results as summarized in Appendix A in the revised version. The discrepancy between finite-width and mean-field risk trajectories scales roughly as N−1/2N^{-1/2}, and the decoder’s self-consistency and encoder’s projection residuals decrease during training. Reconstructions initially approach the PCA line, and the encoder moment and decoder tangent first align with the leading direction of the initial kernel. Those observations are constant over small regularizations.

... continued in the second part including references is in the following response due to character limit.

Paper length and difficulty

We understand that the density of the proofs and the length of the paper made the paper feel overwhelming, and we are sorry that it left this impression. Our goal was to communicate our findings as rigorously as possible and not hand wave arguments. For this model the risk is non-convex even in the mean field limit, so the standard arguments do not go through directly, and this made a lot of our proofs much longer.

Our aim in the first few sections of the appendix was to take the reader along the path of the mean field construction: how the heuristics work and how the PDEs are built from them. We then proceed in the later sections to rigorously derive said heuristics. We believe trying to write very concise proofs without much detail would obstruct readability.

Width 1024

For the experiments we would like to illustrate that the with increasing network width the Risk of the finite width AE converges to the MF limit. However as the PDE has no closed-form solution, we consider two approximations. Firstly we can consider a very wide network as ‘reference’ network in place of the infinite case (this is the standard approach for modeling the MF setting numerically). In the revised version in Figure 1(b) we additionally consider a numerically integrated version of the PDE.

We observe that firstly the convergence toward the wide reference network and the numerically integrated PDE behave very similarly, which validates that the wide network setting is a good proxy for the MF limit. Secondly we observe, similar to the experiments in the initial version of the paper, that as predicted by Theorem 10 the convergence roughly follows N−1/2N^{-1/2}.

References

  • Baldi & Hornik (1989). Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks.
  • Chizat & Bach (2018). On the global convergence of gradient descent for over-parameterized models using optimal transport. NeurIPS.
  • Esser, Mukherjee, Sabanayagam & Ghoshdastidar (2023). Improved representation learning through tensorized autoencoders. AISTATS.
  • Hastie & Stuetzle (1989). Principal curves. Journal of the American Statistical Association.
  • Mei, Montanari & Nguyen (2018). A mean field view of the landscape of two-layer neural networks. PNAS.
  • Mei, Misiakiewicz & Montanari (2019). Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. COLT.
  • Nguyen (2019). Mean field limit of the learning dynamics of multilayer neural networks. arXiv:1902.02880.
  • Nguyen (2021). Analysis of feature learning in weight-tied autoencoders via the mean field lens. arXiv:2102.08373.
  • Nguyen & Pham (2023). A rigorous framework for the mean field limit of multilayer neural networks. arXiv:2001.11443.
  • Sirignano & Spiliopoulos (2020). Mean field analysis of neural networks: A law of large numbers. SIAM Journal on Applied Mathematics.
  • Sirignano & Spiliopoulos (2022). Mean field analysis of deep neural networks. Mathematics of Operations Research.

As no further questions or concerns were raised, we hope that our rebuttal has satisfactorily addressed the reviewer's concerns. We therefore hope that the reviewer will reconsider their assessment, reflecting the clarifications provided.

Reviewer_fvc1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies nonlinear auto-encoders with a bottleneck. With an input and output in Rd\mathbb R^d, it uses first an encoder which maps the input to a bb dimensional bottleneck, using a hidden layer with NN variables. The decoder has the same structure, it maps the bottleneck representation to the output using a hidden layer with NN variables. The auto encoder has thus three hidden layers, with dimension N,b,NN,b,N. The paper studies what happens in the $N\to \infty $ limit, keeping dd and b<db<d fixed. In this mean-field limit, the paper first derives the learning dynamics. Then it proves that, under assumptions of smoothness and with a proper regularization, the finite NN networks trained using stochastic gradient descent have a learning trajectory close to the mean-field dynamics when NN is large. It establishes a gap of O(1/N)O(1/N) between the minimal risk at finite NN and the mean-field one. It also provides some self-consistency property of optimal encoder-decoder pairs.

Critical strengths

The paper studies a geometry of nonlinear auto encoder which combines a central bottleneck of size bb with two intermediate hidden layers of size NN. It manages to control the large NN limit at fixed bb using the mean-field limit. This combination is interesting.

It gives quantitative statements on the convergence of finite-time training in the large NN limit, and convergence of the optimal risk.

Critical weaknesses

The paper proves that the SGD training at finite NN and the mean-field dynamics get close to each other in the large NN limit, but it does not seem to show that these two dynamics reach an optimum

It is not clear if the theoretical results have any practical relevance: it seems that the bounds are not really able to provide practical guidance for how large NN should be.

The result characterizing the principle manifold is interesting but it does not prove that the result applies to the specific architecture with three hidden layers that is used in the present paper (one needs to assume that mf∗m_{f*} can be represented by the hidden layer function).

Questions for authors
  • It would be useful for the reader to clarify the notations for ww and uu in lines 194-196. There is a w=(w1,w2)w=(w_1,w_2) and then a set of wiw_i for i=1,...,Ni=1,...,N, these wiw_i are neither w1w_1 or w2w_2, but they are NN parameters of the ww type, in RD\mathbb R^D. Please find a better notation.
  • A clearer presentation of the hypotheses of Theorem 5 would be welcome
4 comments

We thank the reviewer for the careful reading and such a concise and accurate summary of our contributions. We would also like to draw some attention to our discussion on early alignment with kernel PCA which, owing to the space constraint, we moved to Section M of the Appendix. Following the third reviwer's comments we have now moved the stated an informal version of the theorem therin in the main paper as Theorem 6.

Notation for particles and their components

We shall clarify the notation to w=(w⋅,1,w⋅,2)w=(w_{\cdot,1}, w_{\cdot,2}) for a generic encoder particle (and similarly for decoder). We use the notation (wj,1k,wj,2k)(w^k_{j,1}, w^k_{j,2}) for a k-th sample of the j-th particle in the proofs and this change would be consistent with it.

Clearer Presentation of Theorem 5

We will make a revised clearer presentation of theorem 5. If the following looks good then in the next revision we will have the theorem read as follows:

Theorem 5 Let K⊂Rb\mathcal{K}\subset \mathbb{R}^b be a compact latent space and let (f∗,g∗)∈FW(K)×GU(K)(f^{\ast}, g^{\ast})\in \mathfrak{F}_W(\mathcal{K})\times \mathfrak{G}_U(\mathcal{K}) be blockwise optimal encoder-decoder pair for the unregularized risk L(f,g)=12E∥X−g(f(X))∥22\mathcal{L}(f,g)=\frac{1}{2}\mathbb{E}\|X-g(f(X))\|_2^2. Assume that the encoder and decoder classes are sufficiently expressive in the following sense:

  1. GU(K)\mathfrak{G} _ U(\mathcal{K}) contains a continuous version of conditional expectation, E[X∣f∗(X)]\mathbb{E}[X \mid f^{\ast}(X)].

  2. FW(K)\mathfrak{F} _ W(\mathcal{K}) contains an admissible nearest point encoder πg∗(x)∈arg⁡min⁡z∈K∥x−g∗(z)∥22\pi _ {g^{\ast}}(x) \in \arg\min _ {z\in \mathcal{K}} \|x-g^{\ast}(z) \| _ 2^2, then f∗(x)∈arg⁡min⁡z∈K∥x−g∗(z)∥22, and g∗(z)=E[X∣f∗(X)=z]f^{\ast}(x) \in \arg\min _ {z\in \mathcal{K}} \|x-g^{\ast}(z) \| _ 2^2, \text{ and } g^{\ast}(z) = \mathbb{E}[X \mid f^{\ast}(X) = z]

Remark (connection to principal curves and manifolds) If X∼μX\sim \mu and when the optimal decoder g∗g^{\ast} is injective and the nearest point in g∗(K)g^{\ast}(\mathcal{K}) is μ\mu-a.s. unique, then f⋆=πg⋆f^\star=\pi_{g^\star} and g⋆(z)=E[X∣πg∗(X)=z]g^\star(z)=\mathbb{E}[X \mid \pi_{g^\ast}(X) = z]. If furthermore K\mathcal{K} is a compact bb-dimensional C1C^1 manifold with boundary and g⋆g^\star is a C1C^1 embedding, then g⋆(K)g^\star(\mathcal{K}) is a self-consistent principal bb-manifold in the sense of Definition 3 in section L of Appendix. For b=1b=1, this recovers the classical principal-curve self-consistency condition of Hastie and Stuetzle.

Is the Principal Manifold result specific of the three layer AE?

We agree, that in the current formulation the statements of Theorem 5 only require the encoder and decoder class to be sufficiently expressive and are therefore not specific to the considered three layer AE setting of the paper. However using results from standard universal function approximation by neural networks offer a direct link between the setting of Theorem 5 and our AE setting. A precise understanding of which encoder and decoder neural networks exactly fulfill the requirements of Theorem 5 is an interesting future direction.

Do the dynamics reach an optimum?

The reviewer is correct that we do not prove convergence of either the finite-width SGD dynamics or the limiting mean-field dynamics to an optimum. In a standard two-layer mean-field model, the network output is linear in one measure, and as such the convexity of the risk over measures can sometimes be exploited. Here the encoder measure first produces the bottleneck and is then composed with a nonlinear decoder. Consequently, the risk need not be jointly convex in the encoder and decoder measures, and the standard single-measure convergence argument does not work. We establish global well-posedness and confinement of the regularized dynamics, but whether, and at what rate, these dynamics approach an optimum remains an important open problem.

Theorem 2 is a finite-horizon propagation-of-chaos result in the vein of [Theorem 3, Mei et al., 2018]. For every fixed T<∞T<\infty, it controls the discrepancy between the finite-width SGD trajectory and the coupled mean-field trajectory uniformly over t∈[0,T]t\in[0,T]. For sufficiently large width and sufficiently small step size, the mean-field PDE accurately describes finite-width training over that prescribed time interval. Theorem 2 provides some guidance on the choices of the width NN of the network depending on the choices of step size, training time, error and failure probability which we discuss further in the next section. Theorem 3 shows that the finite-width model class can approximate the optimal mean-field risk value. We show that the mean field risk realizes its minimum, and this minimum is within O(1/N)O(1/N) of the infimum of a finite NN-width network.

We discuss practical relvances is in the next part due to character limit.

References

S. Mei, A. Montanari, and P.M. Nguyen. A mean field view of the landscape of two-layer neural networks. PNAS, 2018.

Practical Relevances

Guidance on N

We agree that the constants in the finite-time theorem are worst-case constants and do not provide a useful numerical prescription but rather theoretical guarantees. Theorem 2 nevertheless gives an explicit sufficient condition and makes the dependence on width, step size, dimension, confidence, and training horizon transparent. Let us set the step size ϵ=1/N\epsilon=1/N, then for a target error α>0\alpha>0, a failure probability β∈(0,1)\beta\in(0,1), and a finite training horizon T>0T>0, we can show that for width Nα,β,T=O~(exp⁡T2α2)N _ {\alpha,\beta,T} = \tilde{O}\left(\frac{\exp{T^2}}{\alpha^2}\right) such that with probability at least 1−β1-\beta,

sup⁡k∈[0,T/ϵ]∩N∣RλN(Uk,Wk)−Rλ(ρU,kϵ,ρW,kϵ)∣≤α.\sup _ {k\in[0,T/\epsilon]\cap\mathbb N}\left| \mathcal{R} _ \lambda^N(U^k,W^k) - \mathcal{R} _ \lambda \left(\rho _ {U,k\epsilon},\rho _ {W,k\epsilon}\right) \right| \le \alpha. The term exp⁡(T2)\exp(T^2) seems to be unavoidable and is due to the worst case Gronwall estimates.

Early time linear behaviour.

Beyond finite-width approximation, Appendix M shows that, from small initialization, the nonlinear mean-field dynamics exhibit an early kernel PCA-like behavior: the learned representation first aligns with the leading direction of a kernel-weighted covariance, which approaches the ordinary top principal component in the linear-feature regime. This provides a theoretical foundation for our empirical finding that the learned representation passes through a transient linear phase before diverging to capture nonlinearity. We will highlight this result more clearly in the main text.

Thanks for addressing the questions in my report. I think that the new presentation of Theorem 5 is better. I'll maintain my score.

We are glad that all questions could be satisfactorily addressed. We have also now revised the manuscript to ensure that the statement of Theorem 5 reflects the wording provided in our response above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.