- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
In this paper, the authors derive the canonical regulariser for the nonlinear models, which recovers the anchored ridge term in the linear models as a wide NN in the neural tangent regime. They prove that this canonical regulariser is characterised by metrics on two manifolds, which the parameter map defines by its output-equivalence. The authors show that this geodesic-ridge regulariser is compatible with the gradient flow on the models whose parameter-output Jacobian changes, by showing several properties. Since geodesic ridge is intractable, as it contains two geodesic metrics with its underlying manifold unknown, the authors propose the arc ridge, which uses the length of the training path as a proxy term that upper bounds the geodesic ridge term. Finally, the authors empirically show this compatibility by measuring how strong regularisation hurts the performance of the network, and find that only arc ridge maintains the performance while standard ridge diverges.
Critical strengths
The theoretical results the authors derive are definitely interesting results. As the authors describe, the canonical regulariser for the NTK regime is well studied, and this has been used to actually define the regulariser or to understand the NTK regime in the Bayesian regime. On the other hand, recent feature learning makes these interpretations impossible, and the authors resolve this by characterising what a canonical regulariser is and suggesting a tractable proxy.
Critical weaknesses
The largest weakness I found is the writing. The text is dense and heavy, and some informal descriptions as Theorem 4.1 is unclear what it implies. Specifically, most of the claims coming after Thm 3.1 are hard to understand, and it is unclear what they imply. Similarly, after Thm 4.2, Remark 4.1 gives lots of related objects to this geodesic ridge. Rather, it would be better to give descriptions of the four properties proven in Thm 4.2 and the proof idea. Also, most of the mathematical expressions are presented in inline math, which I presume is because of the page limit. It would be better to make a few main claims in display style so that the text is sparser and it is easier to get the main claim.
Also, I don’t see anything specific to the feature learning regime in this paper. As in Appendix A (Assumptions), the main assumption is that the network is smooth when viewed as a parameter-output map, which is why the theory applies to an ideal model like f(x; theta) = theta_1 theta_2 x as in Figure 2. Therefore, it is more accurate to describe this as canonical regularisation in the nonlinear training regime (as opposed to the linear model as NTK), not the feature learning regime. This can also be seen in the experiments: Both models the authors used in the experiments are pre-muP paper, meaning that their initialisation and regime would belong to the NTK regime. It would be helpful if the authors further described how the geodesic ridge or arc ridge behaves in those feature-learning regimes in simple models, such as a two-layer network.
One of my worry is that the results in this paper are mostly basic in the theory. The reason of this worry is the proof of Thm D.11 (which is formal version of Thm 4.2), in page 38, is very short, and only few previous lemmas are used. So this paper might be understood as interpretation of existing or elementary computations in manifold theories in terms of nonlinear models.
My score is with weak confidence. I am very familiar with NTK and feature learning literature, however, I'm not familiar with the Riemannian manifold and its theory. Therefore I gave weak confidence.
Questions for authors
- Do you have experiments performed on a muP-initialised network? I believe most of the recent open models will be trained with muP, so you may use small models there. What I would expect is that the gap between the (anchored) ridge and the arc ridge increases.
- Can you use geodesic ridge or arc ridge to show some actual inductive bias of neural networks outside the NT regime? It would be okay to recover existing literature like (https://arxiv.org/abs/2002.04486).
2 comments
Thank you for the corrections
Thank you for the corrections; now I agree that the authors' experiment is done in the feature learning regime, and the authors carefully designed it. I’m not sure if there is a ‘feature learning pretrained checkpoint’ for the models for these datasets, and whether it is feasible in the rebuttal period. Therefore, the experiment can’t be viewed as a weakness.
While it is appreciated that the authors point to the result regarding the two-layer neural network, this is not what I intended. Proposition G.4 shows that the arguments on finite particles (or each neuron) can be viewed as the arguments on the distribution or empirical law, not characterising or interpreting the geodesic ridge or arc ridge. Still, the computed terms depend on the integral over the gradient flow path, which is intractable. Results I intend would be: The geodesic/arc ridge distance equals some function norm (like Sobolev norm, Barron norm, Besov norm, etc) or is equivalent to it, i.e., there is some such that $ c \| f - f_0 \|_\mathcal{L} < d^2(\theta, \theta_0) < C \| f - f_0 \|_{\mathcal{L}}.$ These allow interpretation of the geodesic ridge or arc ridge immediately, as we can think of this function space norm instead.
My current evaluation of this paper aligns with reviewer ACgY. The paper’s main contribution can be interesting and useful; the writing and presentation require substantial improvement. Promising such a revision to camera-ready, for me, is not possible, as ICLR has no conditional accept. Therefore, I maintain my score.
Rebuttal to Reviewer eBVw
We thank the reviewer for their time, for finding our results "definitely interesting", and for noting that "the authors resolve this by characterising what a canonical regulariser is and suggesting a tractable proxy", with "no substantive concerns about the paper's soundness". New results are in Appendix I of the revised PDF.
"Nothing specific to feature learning"
Our setting is wide networks. There, Tensor Programs IV establishes a dynamical dichotomy: a stable abc-parametrisation either admits feature learning or has a kernel limit, but not both. Definition 2.1 adopts exactly this split: kernel regime means a constant Jacobian, and everything else is feature learning. That the theory holds for general smooth models is a strength. It does not rely on an infinite-width limit that does not yet exist for the objects we study. What width adds is collected in Appendix G. For the two-layer networks the reviewer asks about, Proposition G.4 shows that in µP (the mean-field parametrisation) the optimiser's metric is the Wasserstein metric on neuron measures. Arc ridge is then the squared W2-length of the neuron-measure path at every width, and the flow manifold is isometric to its image in measure space.
"The experiments are in the NTK regime"
They are not, and we can now show it directly.
On the depth of the theory
A correction: Theorem D.11 is the formal version of Theorem 3.1, the ridge-bias result, not of Theorem 4.2. Its proof is short by design. As stated in Section 3.2 and Appendix B, its selection step is the classical vanishing-Tikhonov principle. What is new there is the evaluation at the gradient-flow limit, the rotation bound (Lemma D.5) and the counterexamples (Remark D.7). Theorem 4.2 (proved in §D.9) is short because it assembles earlier results. The substance lies in those results:
Presentation
We thank the reviewer for concrete suggestions. Appendix I.1 now states the main results in plain language, with one display equation each. With the camera-ready page we will also:
Questions
µP experiments. Our fine-tuning is in µP; the backbones are the standard public checkpoints, and pretraining a µP backbone was not possible within the discussion period. The theory predicts what the reviewer expects: the anchored–arc gap grows with the rotation of the horizontal space (Lemma D.5). Figures I.1 and I.2 show this rotation is already large here.
Recovering known inductive biases. arXiv:2002.04486 (Chizat & Bach, 2020) treats the logistic loss, where no interpolating minimiser exists and parameters diverge in a max-margin direction. As discussed in Appendix B, this lies outside our square-loss setting. The in-scope two-layer counterpart is Proposition G.4 above.
We hope this addresses the reviewer's concerns, and would be grateful if they would reconsider their score.