acceptodds
Under review as a conference paper at ICLR 2027

How Much Can You Project? A Neural Tangent Kernel Theory of Intrinsic Dimension

Abstract

Neural networks with millions of parameters are routinely trained within a surprisingly small effective parameter space, a phenomenon formalized by Li et al. as "intrinsic dimension", which has remained without theoretical explanation since its discovery. We provide the first such explanation for asymptotically wide networks, showing that intrinsic dimension emerges from the interplay of the Neural Tangent Kernel (NTK) and the Johnson–Lindenstrauss lemma. Specifically, we ask whether the NTK regime survives when training is restricted to a randomly chosen low-dimensional subspace of the full parameter space. We prove that it does when the subspace dimension exceeds a threshold of order : a block-structured random Gaussian projection preserves the NTK entry-wise throughout training with high probability as network width grows. Crucially, this threshold is independent of the ambient parameter dimension , which can be arbitrarily large. We complement this result with matching lower bounds: the dependence is unavoidable for random Gaussian projections, while no linear map can beat the threshold for the hardest gradient configurations. Thus, we identify a critical dimension above which random subspace training preserves the NTK regime and below which the same mechanism fails. We also identify a distinct threshold at : below it, the projected kernel is rank-deficient and a component of the training error is permanently frozen, with the effect determined by its alignment with the surviving eigenspace. Experiments on standard benchmarks support our theoretical predictions.

Primary area:
learning theory
Keywords:
Neural Tangent Kernel (NTK), Intrinsic Dimension, Random Projections, Johnson–Lindenstrauss Lemma, Neural Network Theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JyzSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the intrinsic dimension of neural network training using Neural Tangent Kernel (NTK) theory and the Johnson–Lindenstrauss lemma. The authors establish sufficient conditions under which training in a randomly projected low-dimensional parameter subspace preserves the NTK dynamics of the original network in the infinite-width limit.

The main theoretical contribution is an upper bound of order O(L^3\varepsilon^{-2}\log(NL/\delta)) on the required subspace dimension, independent of the ambient parameter dimension. The authors also establish lower bounds for random Gaussian projections and worst-case linear projections, showing the optimality of the \varepsilon^{-2}\log N dependence up to depth-dependent factors. Furthermore, they analyze the rank deficiency of the projected NTK and its implications for residual training errors.

Numerical experiments on MNIST classification and synthetic regression and classification tasks support the theoretical findings, demonstrating that low-dimensional subspace training can approximate full-parameter training while preserving the NTK structure.

Critical strengths
  1. Rigorous theoretical characterization of intrinsic dimension.

The paper provides a theoretical explanation of intrinsic dimension through the connection between Neural Tangent Kernel (NTK) theory and the Johnson–Lindenstrauss lemma. In particular, the authors establish an upper bound on the required subspace dimension that is independent of the ambient parameter dimension, together with nearly matching lower bounds. These results provide valuable insights into why highly overparameterized neural networks can be trained in substantially lower-dimensional parameter spaces.

  1. Connection between theoretical results and empirical observations.

The paper complements its theoretical analysis with numerical experiments on MNIST and synthetic tasks. The experiments support the predicted relationship between subspace dimension and NTK approximation accuracy, as well as the stability of the projected kernel during training. This combination of theoretical guarantees and empirical validation strengthens the significance of the paper.

Critical weaknesses
  1. Limited applicability beyond the NTK regime.

The theoretical analysis is restricted to the infinite-width NTK regime, where neural network training is approximately governed by fixed kernel dynamics. However, practical neural networks often operate outside this regime, where feature learning plays an important role. It remains unclear whether the proposed theoretical framework can explain intrinsic dimension in more realistic training settings. This restriction limits the broader applicability of the theoretical results.

  1. Gap between theoretical bounds and empirical intrinsic dimension.

Although the paper establishes nearly matching upper and lower bounds on the subspace dimension required to preserve the NTK, the lower bounds are based on worst-case gradient configurations that may not arise in actual neural networks. Moreover, the experiments show that successful learning can require substantially fewer dimensions than predicted by the kernel-preservation criterion. Consequently, it remains unclear to what extent the theoretical bounds explain the intrinsic dimension observed in practical machine learning tasks.

Questions for authors
  1. Justification of the random subspace parameterization

The paper restricts the network parameters to the form \theta(\phi)=\theta_0+P\phi, where P is a fixed random Gaussian projection. Although this is a standard formulation of random subspace training, could the authors clarify to what extent this restriction captures the essential behavior of neural network optimization? In particular, how does the choice of a data-independent random subspace affect the applicability of the theoretical results to practical training?

  1. Relationship between NTK preservation and intrinsic dimension

The main theoretical results characterize the subspace dimension required to preserve the NTK. However, this dimension does not necessarily coincide with the minimum dimension required for successful learning. Could the authors clarify whether NTK preservation is necessary for successful low-dimensional training, or whether substantially smaller subspaces can achieve comparable performance without preserving the original NTK?

5 comments

We would like to thank the reviewer for their comments.

Reviewer has raised two weaknesses:

  • Limited applicability beyond the NTK regime...

Our response: The reviewer is right. Our analysis rigorously applies only to asymptotically wide networks. However, it is not specific to our work, though. Rigorous statements about neural networks are rare, and the NTK regime is one of the few settings where they are possible. Our contribution is to combine this regime with a different mathematical tool, the JL lemma, to obtain what is, to the best of our knowledge, the first rigorous bound for the "intrinsic dimension" phenomenon. Since NTK results are also informative for sufficiently wide finite networks, we expect the picture to be indicative beyond the strict limit, though we do not claim this formally.

  • Gap between theoretical bounds and empirical intrinsic dimension...

Our response: Similar to the previous point. The reviewer is correct that our bound is for worst-case, and the empirically observed intrinsic dimension might be smaller. This reflects a broader open problem: most current guarantees are either worst-case or limited to narrow architectures, and sharp, distribution-dependent bounds are almost non-existent, both for generalization and, in our case, for intrinsic dimension. Closing this gap would require a theory that is not yet available, and we see our work as a first rigorous step towards it. We will discuss this gap explicitly in the revised paper.


Reviewer has also asked two questions:

  • The paper restricts the network parameters to the form θ(ϕ)=θ0+Pϕ\theta(\phi)=\theta_0+P\phi, where PP is a fixed random Gaussian projection. Although this is a standard formulation of random subspace training, could the authors clarify to what extent this restriction captures the essential behavior of neural network optimization? In particular, how does the choice of a data-independent random subspace affect the applicability of the theoretical results to practical training?

Our response: Our main focus has been to develope a theory for the "intrinsic dimension" phenomenom discovered by Li et al., which uses random (data-independent) projections. Changing this setting would completely change the nature of our work.

Also, Data-independent random projections require no prior data processing, pre-training, or gradient sampling. Proving that d≪Dd \ll D suffices under a random PP demonstrates that low-dimensional trainability is an intrinsic geometric property of overparameterized loss landscapes, rather than an artifact of tailored subspace selection.

Data-dependent methods (e.g., LoRA, PCA on gradients, or Hessian eigenspace projections) select subspaces aligned with data features and can achieve even smaller dimensions. However, our work proves that even without data awareness, random projections already reduce the required dimension from DD to O(ε−2log⁡N)O(\varepsilon^{-2} \log N).

  • The main theoretical results characterize the subspace dimension required to preserve the NTK. However, this dimension does not necessarily coincide with the minimum dimension required for successful learning. Could the authors clarify whether NTK preservation is necessary for successful low-dimensional training, or whether substantially smaller subspaces can achieve comparable performance without preserving the original NTK?

Our response: This is related to a previous point. The reviewer is essentially asking whether training survives when the NTK is no longer preserved, i.e., when dd falls below our critical dimension.

We do not answer this, and we will state it explicitly in the revision. Our result is one-directional: if the NTK is preserved, training is essentially unaltered. We do show one necessary condition: below d=Nd=N the projected kernel is rank-deficient, so part of the training error is permanently frozen. Beyond this, characterizing training and generalization once the NTK regime breaks requires a theory of deep networks beyond NTK, which remains a major open problem in learning theory.

Thank you for the clarification. Can a random subspace that is too small to preserve the original NTK still achieve low training error within the NTK regime?

Thank you for your reply.

By "preserving the NTK" we mean ΘP≈K∞\Theta^P\approx K_\infty entrywise, which is what our dimension condition guarantees. Below it, the projected network still trains linearly in the NTK regime, but with its own kernel ΘP\Theta^P: a random sketch of K∞K_\infty with EΘP=K∞\mathbb E\Theta^P=K_\infty and rank at most dd. The training error then converges to the part of the initial residual in the null space of ΘP\Theta^P. Two consequences follow. For d<Nd<N, arbitrary labels cannot be fit. For a specific target, low training error requires only that the residual be captured by the top eigenspace of ΘP\Theta^P, which we expect holds for dd near the effective dimension of K∞K_\infty, since kernels typically have fast spectral decay. Our theorems do not cover this regime, so we will state it as an open direction in the revision.

Reviewer_1xK5OpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1(−2)
Read review
Paper summary

The paper investigates the NTK regime in MLPs with training restricted to a random low-dimensional subspace. The original, full-space parameter vector corresponding to each layer θt(ℓ):=vec(Wℓ,b(ℓ))∈RNℓ\theta_t^{(\ell)}:=vec(W^{\ell}, b^{(\ell)})\in\mathbb{R}^{N_{\ell}} is represented as

θt(ℓ)=θ0(ℓ)+P(ℓ)ϕt(ℓ),\theta_t^{(\ell)} = \theta_0^{(\ell)} + P^{(\ell)}\phi_t^{(\ell)},

where ϕt(ℓ)∈Rdℓ\phi_t^{(\ell)}\in\mathbb{R}^{d_\ell} and P(ℓ)∈RNℓ×dℓP^{(\ell)}\in\mathbb{R}^{N_\ell\times d_\ell} is a Gaussian matrix, sampled once at the start of training. The paper then analyzes gradient flow (GF) w.r.t. the parameters ϕt(ℓ)\phi_t^{(\ell)}, which takes the form

Δ˙(t)=−1NΘP(t)Δ(t).\dot{\Delta}(t) = -\frac{1}{N}\Theta^P(t)\Delta(t).

with the subspace NTK given by

=\sum_{\ell=1}^{L} \langle (P^{(\ell)})^\top \nabla_{\theta^{(\ell)}} f(x_i), (P^{(\ell)})^\top \nabla_{\theta^{(\ell)}} f(x_j).$$ The main result shows that, for large enough subpsace dimension, the subspace NTK $\Theta^P$ remains close to the fullspace infinite-width NTK on the training set points, both at initialization and during GF training. Given dataset size $N$, the required subspace dimension grows as $\log N$, analogous to the Johnson-Lindenstrauss lemma.
Critical strengths

Good writing: The paper is very well written and clearly structured, it was easy to follow.

Soundness: The theoretical claims appear well-supported and I could not find technical issues. (See post-rebuttal update below.)

Critical weaknesses

Only controls projected NTK on a finite set: In my view, the main issue of this work is that the proposed analysis inspired by the Johnson-Lindenstrauss lemma fundamentally applies only to a finite set of training points. The results guarantee that the projected NTK ΘP\Theta^P is close to the fullspace NTK on such NN points, but it does not tell us anything about the NTK behavior outside of this set. Given this, the results cannot be applied to analyze generalization, which makes their utility unclear.

Since this appears to be a major weakness of the considered setting, I am currently not recommending acceptance.

Update: During the discussion, I identified a major soundness issue in the main proof (as well as in the rebuttal). Namely:

Definition A.6 introduces a stopping time T∗T_* such that λmin⁡(ΘP(t))≥λ0/2>0\lambda_{\min}(\Theta^P(t))\geq\lambda_0/2>0 for all t<T∗t<T_*. Lemma A.10 then claims that T∗=∞T_*=\infty with high probability. However, rank(ΘP(t))≤drank (\Theta^P(t))\leq d, so the projected NTK is singular whenever d<Nd<N. Therefore, the stopping time must be T∗=0T_*=0.

The problem in the proof is that the authors use a bound λmin⁡(ΘP(0))≥3λ0/4\lambda_{\min}(\Theta^P(0))\geq3\lambda_0/4 without acknowledging that it requires ε≤λ0/(NM)\varepsilon \leq \lambda_0/(NM) (as the authors themselves stated in the rebuttal), which in turn is impossible when d=O(log⁡N)d= O(\log N), since dd grows with ε−2\varepsilon^{-2}.

In light of this, I have decreased my score.

Questions for authors
  1. What is the main insight/utility of the NTK regime under suspace training? (given the wekness explained above)

  2. In the proofs, are KPK_P and ΘP\Theta^P the same object? Is there a reason to denote them differently?

10 comments

We would like to thank the reviewer for their comments.

Reviewer has one major concern:

  • Only controls projected NTK on a finite set: In my view, the main issue of this work is that the proposed analysis inspired by the Johnson-Lindenstrauss lemma fundamentally applies only to a finite set of training points. The results guarantee that the projected NTK is close to the fullspace NTK on such points, but it does not tell us anything about the NTK behavior outside of this set. Given this, the results cannot be applied to analyze generalization, which makes their utility unclear.

Our response: We agree that the guarantee is stated for a finite set of points. This does not restrict it to training points, and it can count for test predictions as well. The projections P(ℓ)P^{(\ell)} are drawn independently of the data, so the JL argument applies to any fixed finite set Xtot=X∪XtestX_{tot} = X \cup X_{test} chosen independently of PP. With Ntot=N+NtestN_{tot}=N+N_{test}, the dimension condition becomes

d≥Ω(L3ε−2log⁡(2NtotLδ)),d\ge \Omega(L^3\varepsilon^{-2}\log(\frac{2N_{tot}L}{\delta})),

so a large test set costs only a logarithmic increase in dd, and the requirement remains independent of the ambient dimension DD. Our uniform-in-time stability result then controls the train-train and test-train blocks of ΘP(t)\Theta^P(t) simultaneously on [0,T][0,T]. The training residual dynamics still involve only the NN training points.

In the NTK regime, with the squared loss 12∑i(f(xi)−yi)2\tfrac12\sum_i(f(x_i)-y_i)^2, the prediction at an unseen xx is

ft(x)=f0(x)+K(x,X)K(X,X)−1(I−e−tK(X,X))(y−f0(X))f_t(x)=f_0(x)+K(x,X)K(X,X)^{-1}(I-e^{-tK(X,X)})(y-f_0(X)),

and the projected network follows the same formula with ΘP\Theta^P in place of KK. Both networks share the same f0f_0, since they share θ0\theta_0. The map A↦A−1(I−e−tA)=∫0te−sA dsA\mapsto A^{-1}(I-e^{-tA})=\int_0^t e^{-sA}\,ds is Lipschitz, uniformly in tt, with constant 1/λ021/\lambda_0^2 on {λmin⁡(A)≥λ0}\{\lambda_{\min}(A)\ge\lambda_0\} (by Duhamel's formula). Combined with the kernel closeness above, this gives

sup⁡t∈[0,T]max⁡x∈Xtest∣ftP(x∗)−ft(x)∣≤CεM∞,\sup_{t\in[0,T]} \max_{x\in X_{\mathrm{test}}}|f^P_t(x^*)-f_t(x)|\leq C\varepsilon M_\infty,

where CC depends only on λ0\lambda_0, NN, and ∥y−f0(X)∥\|y-f_0(X)\|. We do not claim to analyze the generalization of the full-space NTK predictor, which is a separate question. We show that random subspace training does not change that predictor, so it inherits whatever generalization the full-width network has.

For a guarantee over a whole compact domain D⊂Rn0D\subset\mathbb R^{n_0}, the same argument applies to an ε\varepsilon-net of DD, given Lipschitz continuity of x↦g(x)x\mapsto g(x). This adds roughly n0log⁡(1/ε)n_0\log(1/\varepsilon) to the log⁡Ntot\log N_{tot} term, which is still independent of DD.

Also, on MNIST (N=512N=512, D=804,865D=804{,}865), subspace training reaches negligible test error at d≈40d\approx 40, and test performance tracks the fidelity of the projected kernel to the full-space kernel (Figures 1 and 11), consistent with the above.

Suggested revision: We will state this scope explicitly in the revised paper, separating training (our main focus) from generalization. We will add a short corollary extending the guarantee to any finite test set fixed independently of the projections, at only a logarithmic cost in dd. We will also discuss how to obtain uniform guarantees over the input domain: assuming a data-generating mechanism for train and test samples, and replacing the finite-set JL lemma with its existing high-probability uniform versions. We believe a full treatment of this setting also requires analyzing the generalization of the full-width NTK predictor itself, so we defer it to future work.


Reviewer has also asked two question:

  • What is the main insight/utility of the NTK regime under suspace training? (given the wekness explained above)

Our response: Already answered in our response to your concern above. Preservance of the N×NN\times N NTK matrix (both at initialization and also during the training - and the latter was much harder to prove) is a sufficient condition for "un-altered training" compared to ordinary training, i.e., training in the ambient parameter space with DD parameters.

  • In the proofs, are KPK_P and ΘP\Theta^{P} the same object? Is there a reason to denote them differently?

Our response: Thank you for mentioning this issue. Yes, they are the same. We will polish the proofs so all instances become ΘP\Theta^P.

Thank you for your response. Could you make these arguments more precise, particularly the derivation of the kernel closeness guarantee on test points (including the explicit form and dependencies of the constant C in the bound), and the proposed ϵ\epsilon-net argument?

Yes, we have made it precise and it is a standard covering argument. We can make all the quantities including the constants precise.

Since this whole procedure concerns "generalization", we first need a stochastic mechanism for the generation of training samples. Below, μ\mu denotes the data distribution and we assume samples are i.i.d. . The ℓ\ell-th feature of an input xx is its gradient block g(ℓ)(x)=∇W(ℓ),b(ℓ)fθ0(x)∈RNℓg^{(\ell)}(x)=\nabla_{W^{(\ell)},b^{(\ell)}}f_{\theta_0}(x)\in\mathbb{R}^{N_\ell}, so the relevant feature distribution is the pushforward of μ\mu under g(ℓ)g^{(\ell)}, conditional on θ0\theta_0. We condition on θ0\theta_0 and on the training points. The only property of the projections used is that they are independent of the data and of θ0\theta_0.

(A) High probability with respect to μ\mu. Let BP=x: max⁡i≤N∣KP(x,xi)−K(x,xi)∣>εM(x),M(x)=max⁡(max⁡i∥g(xi)∥2,∥g(x)∥2).B_P=\\{x:\ \max_{i\le N}\big|K_P(x,x_i)-K(x,x_i)\big|>\varepsilon M(x)\\},\quad M(x)=\max(\max_i\|g(x_i)\|^2,\|g(x)\|^2).

If, for all ℓ\ell, dℓ ≥ CJLL2ε−2log⁡(N+1)Lγδ,d_\ell\ \ge\ C_{\rm JL} L^2\varepsilon^{-2}\log\frac{(N+1)L}{\gamma\delta}, then with probability at least 1−δ1-\delta over the projections, μ(BP)≤γ\mu(B_P)\le\gamma.

Proof. For a fixed xx, apply our theorem from Stage I to the N+1N+1 points x1,…,xN,x\\{x_1,\dots,x_N,x\\} with failure probability γδ\gamma\delta, so P(x∈BP)≤γδ\mathbb{P}(x\in B_P)\le\gamma\delta. By Fubini, Eμ(BP)≤γδ\mathbb{E}\mu(B_P)\le\gamma\delta, and Markov's inequality gives P(μ(BP)>γ)≤δ\mathbb{P}(\mu(B_P)>\gamma)\le\delta. □\square

Hence a fresh test point x∗∼μx^*\sim\mu has its inner products with the training set preserved with joint failure probability at most γ+δ\gamma+\delta. The cost is only log⁡(1/γδ)\log(1/\gamma\delta). No assumption on gg and no i.i.d. assumption is needed for this step; the i.i.d. structure enters only when relating training error to risk.

(B) Uniform over the input domain. Assume

  • (H1) X⊂B(0,R)⊂Rn0\mathcal X\subset B(0,R)\subset\mathbb{R}^{n_0} where n0n_0 is the input dimension, and

  • (H2) ∥g(ℓ)(x)−g(ℓ)(x′)∥2≤Λℓ∥x−x′∥2\Vert g^{(\ell)}(x)-g^{(\ell)}(x')\Vert_2\leq\Lambda_{\ell}\Vert x-x'\Vert_2 on X\mathcal{X} for some bounded Lipshitz constant.

Let ρℓ=sup⁡x∥g(ℓ)(x)∥2\rho_\ell=\sup_x\Vert g^{(\ell)}(x)\Vert_2 and κℓ=ΛℓR/ρℓ\kappa_\ell=\Lambda_\ell R/\rho_\ell. If dℓ ≥ Cεℓ−2(κℓ2 n0+log⁡(2/δℓ)),d_\ell\ \ge\ C\varepsilon_\ell^{-2}\big(\kappa_\ell^2\,n_0+\log(2/\delta_\ell)\big), then with probability at least 1−δℓ1-\delta_\ell, simultaneously for all x,x′∈Xx,x'\in\mathcal X, ∣⟨Q(ℓ)⊤g(ℓ)(x),Q(ℓ)⊤g(ℓ)(x′)⟩−⟨g(ℓ)(x),g(ℓ)(x′)⟩∣≤εℓρℓ2,\big|\langle Q^{(\ell)\top}g^{(\ell)}(x),Q^{(\ell)\top}g^{(\ell)}(x')\rangle-\langle g^{(\ell)}(x),g^{(\ell)}(x')\rangle\big|\le\varepsilon_\ell\rho_\ell^2 , where QℓQ^{\ell} denotes the projection matrix of the ℓ\ell-th layer. A union bound over ℓ\ell with ∑ℓεℓ=ε\sum_\ell\varepsilon_\ell=\varepsilon and ∑ℓδℓ=δ\sum_\ell\delta_\ell=\delta gives the kernel bound εmax⁡ℓρℓ2\varepsilon\max_\ell\rho_\ell^2 on the whole domain.

Proof sketch. Let T=g(ℓ)(X)T=g^{(\ell)}(\mathcal X) and T′=(T+T)∪(T−T)T'=(T+T)\cup(T-T). (i) The matrix deviation inequality (Liaw–Mehrabian–Plan–Vershynin, 2017) gives, with probability at least 1−2e−u21-2e^{-u^2}, sup⁡z∈T′∣∥Q⊤z∥−∥z∥∣ ≤ Cγ(T)+uρℓdℓ,\sup_{z\in T'}\big|\|Q^\top z\|-\|z\|\big|\ \le\ C\frac{\gamma(T)+u\rho_\ell}{\sqrt{d_\ell}}, where γ(⋅)\gamma(\cdot) is the Gaussian width. (ii) By (H1)--(H2), N(T,r)≤(1+2ΛℓR/r)n0\mathcal N(T,r)\le(1+2\Lambda_\ell R/r)^{n_0}, so Dudley's integral gives γ(T)≲ΛℓRn0\gamma(T)\lesssim\Lambda_\ell R\sqrt{n_0}. This is where the covering argument enters, and it is independent of DD and NℓN_\ell. (iii) The polarization identity ⟨u,v⟩=14(∥u+v∥2−∥u−v∥2)\langle u,v\rangle=\tfrac14(\|u+v\|^2-\|u-v\|^2) converts the norm deviation into the inner-product bound. □\square

The required dℓd_\ell depends on neither the ambient dimension DD, the sample size NN, nor μ\mu. The factor n0n_0 can be replaced by the intrinsic (covering) dimension of the input domain.

Summary of result: The NTK test predictor depends on the test-train and train-train kernel blocks. Together with the uniform-in-time stability result, (A) and (B) therefore control the difference between the projected and full-space predictors at new points, up to constants depending on λ0\lambda_0, NN and ∥y−f0(X)∥\|y-f_0(X)\|.


We will add all the details into the camera-ready version once the extra page limit is allowed and refine other sections. We hope this obviates the reviewer's concern.

Reviewer_6aWDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the sufficient parameter dimension that allows to preserve the full parameter space under random transformations in Neural Networks. In particular, it focuses on answering this question in the NTK/Lazy regime.

Critical strengths

The papers objective and research question are clear from the beginning. The discussion of what needs to be answered in the NTK regime in order to understand the intrinsic dimension of the parameter space is clear and intuitive.

Critical weaknesses
  1. There seems to be a discrepancy between the paper main objective (when is dimension reduction of the parameter space successful) and what is actually being show ( how to approximate NTK training with fewer parameters).
  2. See questions section.
Questions for authors
  1. Line 073 in the main text mentions multiplicative tolerance. However, Theorem 4.2 seems to establish additive tolerance. Could the authors clarify this?
  2. Theorem 4.2 states that, as the NNs width increases, the difference between projection goes to 0. However, Figure 2 in the manuscript fixes the parameter size and varies the dimension. Is this correct?
3 comments

We would like to thank the reviewer for their comments.

Reviewer has mentioned one weakness:

  • There seems to be a discrepancy between the paper main objective (when is dimension reduction of the parameter space successful) and what is actually being show ( how to approximate NTK training with fewer parameters).

Our response: In the NTK regime, the two questions are closely linked. If "successful" means that restricting training to a random subspace does not meaningfully alter the training dynamics, then our result is a sufficient condition for success: preserving the NTK dynamics (up to a controlled error ε\varepsilon) implies that the projected network trains like the full one.

Concretely, we show that in the infinite-width regime, random linear dimensionality reduction (as in Li et al.) with subspace dimension dd above an explicit threshold, depending on the sample size NN, the accuracy ε\varepsilon, and the depth LL but not on the ambient dimension DD, keeps the training trajectory ε\varepsilon-close (in some sense) to that of full-space training. Our analysis is asymptotic in width, but it should also be informative for finite networks that are sufficiently wide. This gives a theoretical counterpart to the empirical finding of Li et al. that training in a low-dimensional random subspace can match ambient-space performance across a range of architectures.

Our methodology and findings may thus shed light on the theoretical origin of "intrinsic dimension".


Reviewer has also asked two questions:

  • Line 073 in the main text mentions multiplicative tolerance. However, Theorem 4.2 seems to establish additive tolerance.

Our response: You are correct to point out this discrepancy in phrasing. In Section 1.1 (Line 073), the text informally introduced the projected kernel as ε\varepsilon-faithful if it agrees with the infinite-width kernel K∞K_\infty "to within a multiplicative tolerance ε\varepsilon". However, as stated in Theorem 4.2 and formally established in Appendix A (Theorem A.14), our theoretical guarantee proves an additive entrywise error bound scaled by the maximum diagonal entry (kernel scale) M∞=max⁡1≤i≤NK∞(i,i)=max⁡i∣gi∣2M_\infty = \max_{1 \le i \le N} K_\infty(i,i) = \max_i |g_i|^2:

sup⁡0≤t≤Tmax⁡1≤i,j≤N∣ΘijP(t)−K∞(i,j)∣≤εM∞\sup_{0 \le t \le T} \max_{1 \le i,j \le N} |\Theta_{ij} ^P(t) - K_\infty(i,j)| \le \varepsilon M_\infty

This additive scaling arises directly from the innerproduct version of the Johnson–Lindenstrauss (JL) Lemma (Lemma A.2), which guarantees ∣⟨Qu,Qv⟩−⟨u,v⟩∣≤εmax⁡(∣u∣2,∣v∣2)|\langle Qu, Qv \rangle - \langle u, v \rangle| \le \varepsilon \max(|u|^2, |v|^2). When accumulated across network layers, the distortion remains additive relative to the maximum per-sample gradient norm M∞M_\infty.

The introductory phrase "multiplicative tolerance" in Section 1.1 was an informal shorthand. In the revised manuscript, we will update Section 1.1 to explicitly define an ε\varepsilon-faithful projection as one that matches K∞K_\infty to within an additive entrywise tolerance scaled by M∞M_\infty, aligning the text strictly with Theorem 4.2 and Theorem A.14.

  • ... Figure 2 in the manuscript fixes the parameter size and varies the dimension. Is this correct?

Our response: Yes, the experimental design of Figure 2 is correct and intentionally structured to isolate the theoretical predictions of Theorem 4.2. As width grows, the network gradually enters the NTK regime. To empirically test the projected kernel constancy across time predicted by Theorem 4.2, we fix width at a large value (m=1024m = 1024). This ensures the model operates comfortably inside the NTK regime so that finite-width kernel fluctuations are negligible.

By sweeping d∈8,32,128,512d \in \\{8, 32, 128, 512\\} while keeping mm fixed, Figure 2 cleanly isolates the impact of subspace dimension dd. Figure 2A demonstrates that the relative kernel drift decreases monotonically with larger dd, while Figure 2B shows individual kernel entries remaining nearly constant at d=512d = 512.

We will update the caption of Figure 2 to clarify this issue.

I thank the authors for their answers. I have raised my score.

We sincerely thank the reviewer for reconsidering their score.

Comments on the paper

1 comment
Area_Chair_mtLJOpenReview ↗

Discussion phase coming to an end

Dear reviewer 6aWD,

As we are approaching the deadline of the discussion phase, please read the authors’ responses and share your thoughts in the discussion as soon as possible. In particular, please clarify whether the responses have addressed your main concerns. Please also update your review and scores if appropriate.

Your timely input will help us reach a well-informed decision. Thank you for your time and effort!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.