- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper develops a particle mean-field framework for nonlinear autoencoders with a fixed finite-dimensional bottleneck. Using sixth-order regularization, it establishes confinement of the parameter dynamics, quantitative finite-width SGD tracking guarantees, and approximation bounds for the optimal reconstruction risk. Separately, in the unregularized setting, it studies early-time alignment under suitable initialization and connects blockwise-optimal encoder–decoder maps to self-consistent principal manifolds under additional assumptions.
Critical strengths
1. Detailed mean-field analysis of bottleneck autoencoders.
The paper develops a rigorous framework for nonlinear autoencoders with a fixed finite-dimensional bottleneck. The quantitative finite-width tracking guarantees and the approximation bounds for the optimal risk provide a substantive theoretical foundation for this architecture.
2. Clear exposition and careful technical analysis.
The proof sketches and supplementary proofs are well organized and make the main arguments accessible. The appendices provide a thorough verification of the regularity conditions and estimates required by the analysis. In particular, the sixth-order regularizer allows the authors to establish confinement of the dynamics uniformly in width and time without imposing a compact parameter domain. This is a useful technical component, and its detailed justification strengthens the overall presentation.
Critical weaknesses
1. The representation claims need a clearer presentation and more targeted numerical support.
Although early learning of approximately linear representations is highlighted in the abstract and introduction, the main text provides only an informal discussion, with the precise statement and assumptions of Theorem 10 deferred to Appendix M. A concise statement and a representative figure would make this contribution easier to assess.
The supplementary experiments also only partially test the theoretical predictions. Distance from the PCA subspace does not directly measure alignment with the kernel-weighted direction predicted by Theorem 10, and the self-consistency diagnostic does not verify the nearest-point projection condition underlying Theorem 5. More targeted validation, with selected results included in the main text, would clarify what learning behavior the theory explains.
2. The connection between the regularized mean-field framework and the representation results remains incomplete.
The main mean-field guarantees require , whereas the principal-curve characterization in Theorem 5 and the early-alignment result in Theorem 10 concern . These representation results therefore do not directly describe the regularized models covered by the finite-width tracking theorem. Moreover, Theorem 5 assumes block-wise optimality, which the finite-time dynamics are not shown to approach. Although these differences do not invalidate the individual results, they leave their connection unclear. The paper should explain which representation-level conclusions are supported by the regularized framework and which should be understood as separate results.
Minor comments
- On p. 4 and in Appendix F, several occurrences of "SDG" should be "SGD".
- On p. 7, line 346, “infrimum” -> “infimum".
- On p. 7, line 346, “within within” ->“within.”
- On p. 9, line 451, the reference to the numerical experiments appears to point to Appendix L; it should point to Appendix K.
Questions for authors
-
Could the authors include a concise statement of Theorem 10, its key assumptions, and a representative numerical illustration in the main text, possibly by shortening some proof sketches and making use of any additional page allowance during the rebuttal period?
-
How should the representation results in Theorems 5 and 10 be interpreted in relation to the mean-field guarantees? Are these intended as separate results, or can a connection be established through an analysis of small positive regularization or a controlled vanishing-regularization limit?
-
Could the experiments more directly test the predicted kernel-PCA alignment and both conditions underlying the principal-curve interpretation—self-consistency and nearest-point projection?
2 comments
Thank you for the revisions and additional experiments. The regularized early-alignment result, its inclusion in the main text, and the targeted kernel-PCA and projection diagnostics address my concerns.
The informal statement of Theorem 6 would benefit from a brief explanation of the key conditions in Appendix M.3, particularly equal-norm initialization and the coupling between regularization strength and alignment time.
I therefore maintain my score.
We thank the reviewer for the careful reading and such a concise and accurate summary of our contributions and the positive review. We address the posed questions below.
Connection between the λ>0 framework and the λ=0 representation results.
We thank the reviewer for this question. In the submitted version, Theorems 5 and 10 were indeed separate results for λ=0.
Heuristically the correct way to read Theorem 10 is indeed to consider the situation where the regularization parameter λ is very small. In this case there shouldnt be a real difference between this and λ=0. In an earlier draft we did have a version for small λ. However we thought that the argument and the discussion was becoming too technical so reverted to the λ=0 case (which provided a proof of concept). Following your review, we have now reinstated the early-alignment result. This establishes the connection through small positive regularization, as the reviewer suggests. The results are essentially similar to the λ=0 case and are basically a technical extension
In the revision this is added to a new Appendix M.3, which extends Theorem 10 to the regularized mean-field dynamics. For λ>0, the moments Cs and Bs again align with a single data direction during the small-output phase. This direction is the same top eigenvector vˉ1 of the kernel covariance C as in Theorem 10, and the overlap κ1 with the initialization is also unchanged. The reason is that all neurons start with the same norm R1, so at initialization the regularizer damps every neuron at the same rate dλ=λR14. In the leading linear system this damping multiplies all modes by the same factor e−dλs, which slows the growth from rate ρ1 to rλ=ρ1−dλ but does not change its direction, so no eigenvector perturbation argument is needed. The error bound is the same as Theorem 10, with the exponent θλ=(ρ1−ρ2)/rλ in place of θ, and one extra term λ(sδ+sδ2), where sδ=rλ1log(δ/∥Z0∥2) is the exit time; this term comes from the motion of the features under the regularizer. At λ=0 we have rλ=ρ1, θλ=θ, and the extra term vanishes, so we recover Theorem 10 exactly. There are two limitations that we state openly. First, λ has to be small compared with the time window, roughly λsδ2≲1, that is, λlog2(δ/∥Z0∥2)≲rλ2, so we cannot send the initialization to zero with λ fixed. Second, we additionally assume that every encoder and decoder neuron has the same norm R1 at initialization, and that λR14≤ρ1/2. Theorem 5, on the other hand, remains a statement about the unregularized risk and about blockwise-optimal maps, which we do not show the dynamics reach, and we will say clearly in the revision which conclusions are supported by the regularized framework and which are separate results.
Additional Experiments
We added the additional experiments in Section 6 of the revised version. Specifically we show that the discrepancy between finite-width and mean-field risk trajectories scales roughly as N−1/2, and the decoder’s self-consistency and encoder’s projection residuals decrease during training. Reconstructions initially approach the weighted kernel PCA line, and the encoder moment and decoder tangent first align with the leading direction of the initial kernel. Those observations are consistent over small regularizations. We note that in the original experiments we were considering PCA of covariance matrix as we discuss in section M.2.3 that under some extra conditions the PCA of the default covariance and the kernel weighted covariance are close.
Adding early time PCA Theorems to the main
We have now extended the main to both include some experiments and a more detailed informal version of the regularized early time PCA theorem. This is now stated as Theorem 6. However the precise assumptions are still contained in the appendix as it needs substantial notation to be defined which takes up a lot of space. However the principal ideas we want to convey we hope is relatively clear.