- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the intrinsic dimension of neural network training using Neural Tangent Kernel (NTK) theory and the Johnson–Lindenstrauss lemma. The authors establish sufficient conditions under which training in a randomly projected low-dimensional parameter subspace preserves the NTK dynamics of the original network in the infinite-width limit.
The main theoretical contribution is an upper bound of order O(L^3\varepsilon^{-2}\log(NL/\delta)) on the required subspace dimension, independent of the ambient parameter dimension. The authors also establish lower bounds for random Gaussian projections and worst-case linear projections, showing the optimality of the \varepsilon^{-2}\log N dependence up to depth-dependent factors. Furthermore, they analyze the rank deficiency of the projected NTK and its implications for residual training errors.
Numerical experiments on MNIST classification and synthetic regression and classification tasks support the theoretical findings, demonstrating that low-dimensional subspace training can approximate full-parameter training while preserving the NTK structure.
Critical strengths
- Rigorous theoretical characterization of intrinsic dimension.
The paper provides a theoretical explanation of intrinsic dimension through the connection between Neural Tangent Kernel (NTK) theory and the Johnson–Lindenstrauss lemma. In particular, the authors establish an upper bound on the required subspace dimension that is independent of the ambient parameter dimension, together with nearly matching lower bounds. These results provide valuable insights into why highly overparameterized neural networks can be trained in substantially lower-dimensional parameter spaces.
- Connection between theoretical results and empirical observations.
The paper complements its theoretical analysis with numerical experiments on MNIST and synthetic tasks. The experiments support the predicted relationship between subspace dimension and NTK approximation accuracy, as well as the stability of the projected kernel during training. This combination of theoretical guarantees and empirical validation strengthens the significance of the paper.
Critical weaknesses
- Limited applicability beyond the NTK regime.
The theoretical analysis is restricted to the infinite-width NTK regime, where neural network training is approximately governed by fixed kernel dynamics. However, practical neural networks often operate outside this regime, where feature learning plays an important role. It remains unclear whether the proposed theoretical framework can explain intrinsic dimension in more realistic training settings. This restriction limits the broader applicability of the theoretical results.
- Gap between theoretical bounds and empirical intrinsic dimension.
Although the paper establishes nearly matching upper and lower bounds on the subspace dimension required to preserve the NTK, the lower bounds are based on worst-case gradient configurations that may not arise in actual neural networks. Moreover, the experiments show that successful learning can require substantially fewer dimensions than predicted by the kernel-preservation criterion. Consequently, it remains unclear to what extent the theoretical bounds explain the intrinsic dimension observed in practical machine learning tasks.
Questions for authors
- Justification of the random subspace parameterization
The paper restricts the network parameters to the form \theta(\phi)=\theta_0+P\phi, where P is a fixed random Gaussian projection. Although this is a standard formulation of random subspace training, could the authors clarify to what extent this restriction captures the essential behavior of neural network optimization? In particular, how does the choice of a data-independent random subspace affect the applicability of the theoretical results to practical training?
- Relationship between NTK preservation and intrinsic dimension
The main theoretical results characterize the subspace dimension required to preserve the NTK. However, this dimension does not necessarily coincide with the minimum dimension required for successful learning. Could the authors clarify whether NTK preservation is necessary for successful low-dimensional training, or whether substantially smaller subspaces can achieve comparable performance without preserving the original NTK?
5 comments
Thank you for the clarification. Can a random subspace that is too small to preserve the original NTK still achieve low training error within the NTK regime?
Thank you for your reply.
By "preserving the NTK" we mean entrywise, which is what our dimension condition guarantees. Below it, the projected network still trains linearly in the NTK regime, but with its own kernel : a random sketch of with and rank at most . The training error then converges to the part of the initial residual in the null space of . Two consequences follow. For , arbitrary labels cannot be fit. For a specific target, low training error requires only that the residual be captured by the top eigenspace of , which we expect holds for near the effective dimension of , since kernels typically have fast spectral decay. Our theorems do not cover this regime, so we will state it as an open direction in the revision.
We would like to thank the reviewer for their comments.
Reviewer has raised two weaknesses:
Our response: The reviewer is right. Our analysis rigorously applies only to asymptotically wide networks. However, it is not specific to our work, though. Rigorous statements about neural networks are rare, and the NTK regime is one of the few settings where they are possible. Our contribution is to combine this regime with a different mathematical tool, the JL lemma, to obtain what is, to the best of our knowledge, the first rigorous bound for the "intrinsic dimension" phenomenon. Since NTK results are also informative for sufficiently wide finite networks, we expect the picture to be indicative beyond the strict limit, though we do not claim this formally.
Our response: Similar to the previous point. The reviewer is correct that our bound is for worst-case, and the empirically observed intrinsic dimension might be smaller. This reflects a broader open problem: most current guarantees are either worst-case or limited to narrow architectures, and sharp, distribution-dependent bounds are almost non-existent, both for generalization and, in our case, for intrinsic dimension. Closing this gap would require a theory that is not yet available, and we see our work as a first rigorous step towards it. We will discuss this gap explicitly in the revised paper.
Reviewer has also asked two questions:
Our response: Our main focus has been to develope a theory for the "intrinsic dimension" phenomenom discovered by Li et al., which uses random (data-independent) projections. Changing this setting would completely change the nature of our work.
Also, Data-independent random projections require no prior data processing, pre-training, or gradient sampling. Proving that d≪D suffices under a random P demonstrates that low-dimensional trainability is an intrinsic geometric property of overparameterized loss landscapes, rather than an artifact of tailored subspace selection.
Data-dependent methods (e.g., LoRA, PCA on gradients, or Hessian eigenspace projections) select subspaces aligned with data features and can achieve even smaller dimensions. However, our work proves that even without data awareness, random projections already reduce the required dimension from D to O(ε−2logN).
Our response: This is related to a previous point. The reviewer is essentially asking whether training survives when the NTK is no longer preserved, i.e., when d falls below our critical dimension.
We do not answer this, and we will state it explicitly in the revision. Our result is one-directional: if the NTK is preserved, training is essentially unaltered. We do show one necessary condition: below d=N the projected kernel is rank-deficient, so part of the training error is permanently frozen. Beyond this, characterizing training and generalization once the NTK regime breaks requires a theory of deep networks beyond NTK, which remains a major open problem in learning theory.