acceptodds
Under review as a conference paper at ICLR 2027

Canonical Regularisation of Wide Feature-Learning Neural Networks

Abstract

Wide neural networks in the feature-learning regime drive modern deep learning, and yet they remain far less studied than their kernel-regime counterparts. We consider a critical yet under-explored difference between these two regimes: the regulariser and prior implied by gradient flow training. This *canonical regularisation* property is well-studied in kernel regime networks – of all the infinite global minima, gradient flow selects exactly the vanishing ridge solution – and underpins the celebrated NN-GP correspondence, precisely allowing the modelling of noise during training. However, we prove ridge regularisation biases gradient flow in feature-learning regime networks, even in the infinitesimal limit of vanishing regularisation. Over training, ridge distorts the inductive bias of the network, with a particular damage done to pretrained networks where the implicit prior is informative. We resolve this by deriving the canonical regulariser from the training dynamics alone: under a single conditioning assumption, the initialisations that train to the same network form a curved surface, the basin manifold, and the optimiser's own kinetic energy, measured over the motions that preserve the training outcome, uniquely identifies ridge in the kernel regime and generalises it to *geodesic ridge* in the feature-learning regime. As a practical contribution, we propose *arc ridge* as a minimax-robust, scalable surrogate to geodesic ridge, revealing a deep relationship between early stopping and canonical regularisation across learning regimes. Finally, we demonstrate the consequences of our theory empirically on both image processing and NLP transfer-learning problems.

Primary area:
learning theory
Keywords:
regularization, feature learning, geodesic ridge

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eBVwOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

In this paper, the authors derive the canonical regulariser for the nonlinear models, which recovers the anchored ridge term in the linear models as a wide NN in the neural tangent regime. They prove that this canonical regulariser is characterised by metrics on two manifolds, which the parameter map defines by its output-equivalence. The authors show that this geodesic-ridge regulariser is compatible with the gradient flow on the models whose parameter-output Jacobian changes, by showing several properties. Since geodesic ridge is intractable, as it contains two geodesic metrics with its underlying manifold unknown, the authors propose the arc ridge, which uses the length of the training path as a proxy term that upper bounds the geodesic ridge term. Finally, the authors empirically show this compatibility by measuring how strong regularisation hurts the performance of the network, and find that only arc ridge maintains the performance while standard ridge diverges.

Critical strengths

The theoretical results the authors derive are definitely interesting results. As the authors describe, the canonical regulariser for the NTK regime is well studied, and this has been used to actually define the regulariser or to understand the NTK regime in the Bayesian regime. On the other hand, recent feature learning makes these interpretations impossible, and the authors resolve this by characterising what a canonical regulariser is and suggesting a tractable proxy.

Critical weaknesses

The largest weakness I found is the writing. The text is dense and heavy, and some informal descriptions as Theorem 4.1 is unclear what it implies. Specifically, most of the claims coming after Thm 3.1 are hard to understand, and it is unclear what they imply. Similarly, after Thm 4.2, Remark 4.1 gives lots of related objects to this geodesic ridge. Rather, it would be better to give descriptions of the four properties proven in Thm 4.2 and the proof idea. Also, most of the mathematical expressions are presented in inline math, which I presume is because of the page limit. It would be better to make a few main claims in display style so that the text is sparser and it is easier to get the main claim.

Also, I don’t see anything specific to the feature learning regime in this paper. As in Appendix A (Assumptions), the main assumption is that the network is smooth when viewed as a parameter-output map, which is why the theory applies to an ideal model like f(x; theta) = theta_1 theta_2 x as in Figure 2. Therefore, it is more accurate to describe this as canonical regularisation in the nonlinear training regime (as opposed to the linear model as NTK), not the feature learning regime. This can also be seen in the experiments: Both models the authors used in the experiments are pre-muP paper, meaning that their initialisation and regime would belong to the NTK regime. It would be helpful if the authors further described how the geodesic ridge or arc ridge behaves in those feature-learning regimes in simple models, such as a two-layer network.

One of my worry is that the results in this paper are mostly basic in the theory. The reason of this worry is the proof of Thm D.11 (which is formal version of Thm 4.2), in page 38, is very short, and only few previous lemmas are used. So this paper might be understood as interpretation of existing or elementary computations in manifold theories in terms of nonlinear models.

My score is with weak confidence. I am very familiar with NTK and feature learning literature, however, I'm not familiar with the Riemannian manifold and its theory. Therefore I gave weak confidence.

Questions for authors
  1. Do you have experiments performed on a muP-initialised network? I believe most of the recent open models will be trained with muP, so you may use small models there. What I would expect is that the gap between the (anchored) ridge and the arc ridge increases.
  2. Can you use geodesic ridge or arc ridge to show some actual inductive bias of neural networks outside the NT regime? It would be okay to recover existing literature like (https://arxiv.org/abs/2002.04486).
2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer eBVw

We thank the reviewer for their time, for finding our results "definitely interesting", and for noting that "the authors resolve this by characterising what a canonical regulariser is and suggesting a tractable proxy", with "no substantive concerns about the paper's soundness". New results are in Appendix I of the revised PDF.

"Nothing specific to feature learning"

Our setting is wide networks. There, Tensor Programs IV establishes a dynamical dichotomy: a stable abc-parametrisation either admits feature learning or has a kernel limit, but not both. Definition 2.1 adopts exactly this split: kernel regime means a constant Jacobian, and everything else is feature learning. That the theory holds for general smooth models is a strength. It does not rely on an infinite-width limit that does not yet exist for the objects we study. What width adds is collected in Appendix G. For the two-layer networks the reviewer asks about, Proposition G.4 shows that in µP (the mean-field parametrisation) the optimiser's metric is the Wasserstein metric on neuron measures. Arc ridge is then the squared W2W_2-length of the neuron-measure path at every width, and the flow manifold is isometric to its image in measure space.

"The experiments are in the NTK regime"

They are not, and we can now show it directly.

  1. Fine-tuning uses a freshly initialised µP readout and µP per-layer learning rates, and every regulariser acts on effective µP parameters (Appendix H).
  2. The kernel regime is a property of the Jacobian during fine-tuning, not of how the backbone was pretrained. New measurements (Figure I.1) show the smallest NTK eigenvalue growing by four to five orders of magnitude during fine-tuning, so the kernel is far from frozen. The anchored-ridge displacement also leaves the horizontal space, with over 97% of its gradient along the fibres within 300 steps (Figure I.2). By Corollary D.6 this is impossible in the kernel regime.
  3. Tensor Programs IV identifies feature learning as what makes pretraining and transfer work at all.

On the depth of the theory

A correction: Theorem D.11 is the formal version of Theorem 3.1, the ridge-bias result, not of Theorem 4.2. Its proof is short by design. As stated in Section 3.2 and Appendix B, its selection step is the classical vanishing-Tikhonov principle. What is new there is the evaluation at the gradient-flow limit, the rotation bound (Lemma D.5) and the counterexamples (Remark D.7). Theorem 4.2 (proved in §D.9) is short because it assembles earlier results. The substance lies in those results:

  • the basin manifold as a smooth, invariant, unique carrier (Theorems D.2–D.3);
  • forced label dependence (Theorem D.8);
  • drift dynamics with explicit rates for arbitrary regularisers (Theorem D.12);
  • uniqueness of the canonical regulariser (Lemma D.6, Theorems D.16 and D.19);
  • minimax optimality of arc ridge (Theorem E.7);
  • the NTK-Gaussian expansion of the training-cost function (Theorem F.3). We do not claim new differential geometry; we use it to obtain new results about wide networks.

Presentation

We thank the reviewer for concrete suggestions. Appendix I.1 now states the main results in plain language, with one display equation each. With the camera-ready page we will also:

  • state the main claims in display form;
  • follow Theorem 4.2 with plain-language descriptions of properties (a)–(d) and the proof idea, moving Remark 4.1 to Appendix F;
  • give each informal theorem a one-sentence reading. For example, Theorem 4.1 says: if parameter motion is priced the way gradient descent prices it, and only motions that leave the trained network unchanged are allowed, then the cheapest way to reach θ\theta costs exactly dflow(θ0,θ)2d_{\mathrm{flow}}(\theta_0,\theta)^2. That cost is geodesic ridge.

Questions

µP experiments. Our fine-tuning is in µP; the backbones are the standard public checkpoints, and pretraining a µP backbone was not possible within the discussion period. The theory predicts what the reviewer expects: the anchored–arc gap grows with the rotation of the horizontal space (Lemma D.5). Figures I.1 and I.2 show this rotation is already large here.

Recovering known inductive biases. arXiv:2002.04486 (Chizat & Bach, 2020) treats the logistic loss, where no interpolating minimiser exists and parameters diverge in a max-margin direction. As discussed in Appendix B, this lies outside our square-loss setting. The in-scope two-layer counterpart is Proposition G.4 above.

We hope this addresses the reviewer's concerns, and would be grateful if they would reconsider their score.

Reviewer_eBVwOpenReview ↗

Thank you for the corrections

Thank you for the corrections; now I agree that the authors' experiment is done in the feature learning regime, and the authors carefully designed it. I’m not sure if there is a ‘feature learning pretrained checkpoint’ for the models for these datasets, and whether it is feasible in the rebuttal period. Therefore, the experiment can’t be viewed as a weakness.

While it is appreciated that the authors point to the result regarding the two-layer neural network, this is not what I intended. Proposition G.4 shows that the arguments on finite particles (or each neuron) can be viewed as the arguments on the distribution or empirical law, not characterising or interpreting the geodesic ridge or arc ridge. Still, the computed terms depend on the integral over the gradient flow path, which is intractable. Results I intend would be: The geodesic/arc ridge distance equals some function norm ∣f0−f∣L\\| f_0 - f \\|_{\mathcal{L}} (like Sobolev norm, Barron norm, Besov norm, etc) or is equivalent to it, i.e., there is some 0<c<C0 < c < C such that $ c \| f - f_0 \|_\mathcal{L} < d^2(\theta, \theta_0) < C \| f - f_0 \|_{\mathcal{L}}.$ These allow interpretation of the geodesic ridge or arc ridge immediately, as we can think of this function space norm instead.

My current evaluation of this paper aligns with reviewer ACgY. The paper’s main contribution can be interesting and useful; the writing and presentation require substantial improvement. Promising such a revision to camera-ready, for me, is not possible, as ICLR has no conditional accept. Therefore, I maintain my score.

Reviewer_ACgYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies which regulariser is consistent with gradient-flow training of overparametrised networks outside the kernel regime. Under a uniform Jacobian-conditioning assumption on a ball around the initialisation, it shows that the initialisations training to the same interpolator form a smooth manifold (the basin or flow manifold). Anchored and standard ridge, in the vanishing-regularisation limit, select an interpolator different from the gradient-flow limit unless an alignment condition holds. The authors then propose "geodesic ridge", the squared geodesic distance from initialization along the flow manifold, as the canonical regularizer, which reduces to anchored ridge in the kernel regime. As a tractable surrogate they propose "arc ridge", the squared length of the training path, which coincides with early stopping under exact gradient flow. Experiments on fine-tuning ResNet18 (UTKFace) and DistilBERT (Yelp) show that arc ridge degrades more gracefully than standard and anchored ridge at large regularisation strengths.

Critical strengths

The question is a good one, and the geometric picture is natural and appealing: the role of the basin manifold, and the observation that ridge drifts along the interpolating set when the Jacobian rotates. The mathematical development appears careful and is well connected to the relevant dynamical-systems and inverse-problems literature. Reading arc ridge as early stopping, and early stopping as a robust form of the canonical regulariser, is an interesting perspective.

Critical weaknesses

The main problem is the presentation, which made it very hard to appreciate the contribution. The main text is extremely dense and reads largely as an index to roughly sixty pages of appendices. Nearly every sentence carries several technical qualifications and cross-references, and the key ideas are never stated simply. As a result it is difficult to separate what is genuinely new from what is a reformulation of known results, and to judge what the claims mean in practice. A conference paper should let the reader grasp the main contribution from the main text; this one does not, and I could not evaluate it with the confidence it deserves.

From what I could extract, several claims also seem overstated. Geodesic ridge is anchored on the outcome of training, so it is unbiased by construction and cannot be evaluated before training. It depends on the labels, which weakens the analogy with the NTK Gaussian-process prior that motivates the paper. In practice the proposed surrogate is early stopping. The conditioning assumption is, as the authors acknowledge, established only under near-lazy conditions, so it is unclear how much the results say about the feature-learning regime emphasised in the title. The experiments show that strong ridge hurts fine-tuning, which is expected for elementary reasons, but they do not isolate the proposed mechanism, and they show no gain at the best regularisation strength.

Questions for authors

Can you state, in a few sentences and without appendix references, what is new relative to the classical vanishing-Tikhonov selection principle and to existing results on the limit map of gradient flow?

In what sense is geodesic ridge usable, given that it requires the trained network?

Can you give evidence that Assumption 2.1 holds in a regime with substantial feature learning?

Can you measure directly, in the experiments, the component of the ridge gradient along the output fibres, to show that it is responsible for the degradation?

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer ACgY

We thank the reviewer for their time, and for noting that "the question is a good one", that the geometric picture is "natural and appealing", and that the development "appears careful and is well connected to the relevant dynamical-systems and inverse-problems literature". New results are in Appendix I of the revised PDF.

Q1. What is new

The central novelty is the notion and construction of a canonical regulariser beyond the kernel regime. In the kernel regime it comes for free: parameter norm, RKHS norm and GP prior coincide. Outside it that coincidence breaks, and the relevant prior results are negative: no data-independent function of the parameters captures the implicit bias of gradient descent in several non-linear models (Razin & Cohen, 2020; Vardi & Shamir, 2021). No canonical regulariser had been identified, nor a geometry in which to seek one. We supply both:

  1. The geometry. Two new parameter-space objects organise training: the basin manifold (the initialisations that train to the same network) and the output fibres (the motions invisible to the training loss). Prior work uses the limit map of gradient flow to describe where training goes; using its level set as the carrier of a regulariser is new. We prove this carrier must depend on the labels outside an integrable class.
  2. The canonical regulariser. We define canonicity from the training dynamics alone: the least energy, priced in the optimiser's own metric, over motions that preserve the training outcome. This singles out a unique regulariser, geodesic ridge, which reduces to anchored ridge in the kernel regime. Standard, anchored and geodesic ridge thus form a sequence of geometric corrections.
  3. Why ridge fails. Vanishing-Tikhonov theory says which interpolator a penalty selects, not whether gradient flow reaches it. Anchored ridge does so only if the training displacement lies in the final Jacobian row space, which fails once that space rotates. Any small regulariser then drifts along the zero-loss set at speed λ\lambda.
  4. Early stopping is MAP estimation under the least-favourable geometry consistent with the observed path, extending the early-stopping/ridge correspondence beyond linear and kernel models.

Q2. Usability, and "unbiased by construction"

We say the latter ourselves after Theorem 4.2. It is not the content of Section 4: unbiasedness alone admits an infinite family of regularisers (Proposition D.18), and the least-energy principle selects exactly one. Geodesic ridge is a function of the training problem (architecture, data, optimiser, θ0\theta_0), which also determines θ⋆\theta^\star; its obstacle is purely computational (geodesics on an implicit manifold). Like the NTK-GP in the kernel regime, it says what the right regulariser is. Weight decay, L2-SP and every fixed-anchor quadratic penalty are biased whenever alignment fails, and arc ridge is the minimax-robust computable surrogate, needing no validation split (Remark E.2).

Label dependence is forced, not chosen. By Theorem D.8, any regulariser unbiased for the training problem must depend on the labels unless the Jacobian row spaces integrate, which generic networks violate (Remark A.3). The GP analogy lives in the label-free training-cost function (Remark 4.1): the NTK-GP energy in the kernel regime, and NTK-Gaussian to second order in every regime.

Q3–Q4. Assumption 2.1 and the mechanism

Details are in our reply to Reviewer RUrq. In brief:

  • Assumption 2.1 is not laziness: it constrains singular values only and lets Im J⊤\mathrm{Im}\,J^\top rotate. λmin⁡(K)\lambda_{\min}(K) stays bounded away from zero on every UTKFace run while the NTK changes by four to five orders of magnitude (Figure I.1). A chained certificate removes the rotation limit of Proposition A.2 (Proposition I.1), and the negative result needs only full rank at the endpoint (Proposition I.2).
  • The mechanism. Anchored ridge's gradient is over 97% fibre-aligned within 300 steps (Figure I.2), and at matched training loss strong standard and anchored ridge generalise worse than arc ridge (Figure I.3). That strong standard ridge hurts is elementary. That anchored ridge, canonical in the kernel regime, does worse than early stopping at equal training error is what the kernel picture cannot explain and our theory predicts. On "no gain at the best strength": Section 5 predicts exactly this. Early stopping, applied to every regulariser, dominates at small λ\lambda, so all three coincide, as expected if arc ridge is optimally tuned early stopping.

Presentation

We thank the reviewer for the suggestion. Appendix I.1 now states the main results in plain language with one display equation each. The camera-ready page will bring display-form statements and a "classical vs. new" paragraph into the main text.

We hope this clarifies the contribution, and would be grateful if the reviewer would reconsider their assessment.

I thank the authors for their responses. Unfortunately, they do little to clarify the presentation or to make clear what the main contribution of the paper is. In my view, addressing this would require a substantial rewriting of the main text; the new Appendix I.1 is a useful addition, but placed at the very end of a long series of appendices, it does not resolve the issue. I believe the paper would benefit from a thorough revision before publication, and I will therefore keep my score.

Reviewer_RUrqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the canonical regularizer for wide neural networks in the feature learning regime (in the kernel learning regime, the anchored ridge regularizer is such canonical regularizer). Under a uniform Jacobian-conditioning assumption (Assumption 2.1) and squared-error training, the author characterizes the flow manifold (Definition 3.1): the initialization that converge to the same trained network, and the output fibres (Definition 3.2). The paper shows that in the feature learning regime, in which the flow manifold is curved, anchored ridge, which violates the alignment condition θ∗−θ0∈ImJ(θ∗)\theta^{*} - \theta_{0} \in Im J (\theta^{*}) can cause bias and steer the training dynamics off the flow manifold even as the regularization strength vanishes, results in the trained network to acquire a different interpolator than θ∗\theta^{*} (with illustration in Fig. 2). In Section 4, the authors constructs the geodesic ridge (Definition 4.1) using distances along the flow manifold and the output fibres, and show that this regularizer is bias-free and select the same trained output as θ∗\theta^* (Theorem 4.2). As a practical surrogate, the paper proposes arc ridge (Definition 4.2), the squared training-path length, whose force is equivalent to early stopping under exact gradient flow. Experiments on UTKFace with ResNet18 and Yelp Review with DistilBERT (Fig. 3) show less degradation at strong regularization than standard and anchored ridge.

Critical strengths

A clear geometric explanation of when ridge changes the gradient-flow solution

The paper identify the gap of a canonical regularizer in the feature-learning regime and construct well-defined geometric objects to address this gap. Its characterization of the flow manifold, which groups initialization by their trained network, and output fibres, which group parameters by their training predictions, makes the failure mechanism of anchored ridge regularization in the feature learning regime, as well as the proposed fix, the geodesic regularizer, understandable.

Theorem 3.1 gives a concrete result that anchored ridge cannot recover the gradient-flow endpoint in the feature learning regime in which the flow manifold is curved. The two-timescale analysis in Theorem D.12 further explains how a small regularizer first follows training and subsequently drives motion along the interpolating set. Fig. 2 illustrates this distinction in minimal nonlinear and linear models. These results provide a useful account of why a penalty that matches gradient flow in the kernel regime can fail outside it.

A precise connection between training geometry and early stopping

The paper connects its geometric construction to a practical quantity that can be accumulated during training. Theorem 4.3 bounds geodesic ridge by the squared training-path length, and Proposition E.3 shows that the corresponding force follows the unregularized gradient-flow path and approaches a stopping point where the regularizing and data forces balance. The distinction between a path-dependent force and the gradient of a parameter-space penalty is stated explicitly. This gives a theoretical interpretation of early stopping in the feature-learning setting, even though it does not establish a new advantage over early stopping itself.

Critical weaknesses

The relevance of the theoretical assumptions (Assumption 2.1) to substantial feature learning remains insufficiently established

Assumption 2.1 requires uniform lower and upper singular-value bounds throughout an entire ball around initialization. This is weaker than the lazy training assumption, but stronger than checking that the Jacobian remains well conditioned along the observed training trajectory. Appendix A.1 and Fig. 4 provide the latter check on a 16-point regression problem with a two-hidden-layer MLP, but do not certify the required ball or establish the hypothesis for the pretrained models in Section 5. The paper acknowledges this limitation. Nevertheless, because the basin geometry and canonical construction depend on this assumption, the gap limits how confidently these results explain realistic feature-learning networks.

The experiments do not isolate output fibre drift as the cause of the performance differences

Fig. 3 shows that standard and anchored ridge degrade more sharply than arc ridge at large regularization strengths on two transfer-learning tasks. However, test MSE alone does not distinguish the proposed fibre-drift mechanism from under-fitting caused by strong parameter penalties. The experiments do not directly measure the regularizer component along output fibres (projection into kerJ(θ))ker J(\theta)) or compare generalization at matched training error. This matters because selecting a different interpolator is not, by itself, a guarantee of worse generalization, as mentioned in Section 3.2.

Questions for authors

Applicability of Assumption 2.1. Can the certificate in Proposition A.2 give usable bounds for a neural-network experiment with substantial Jacobian rotation, beyond the analytically tractable examples? In particular, how does the certified radius compare with the radius required by the assumption? The trajectory checks in Fig. 4 do not establish this condition.

Evidence for the proposed mechanism. Can the authors measure the component of the ridge force in the Jacobian null space and relate it to changes in test performance, ideally at matched training error? This would help distinguish fibre drift from ordinary under-fitting at strong regularization.

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer RUrq

We thank the reviewer for a careful and accurate reading, and for highlighting the paper's "clear geometric explanation of when ridge changes the gradient-flow solution" and its "precise connection between training geometry and early stopping". Both questions are excellent, and the measurements they prompted strengthen the paper. They are in the new Appendix I of the revised PDF.

Q1. Applicability of Assumption 2.1 and Proposition A.2

Why the single-centre certificate is conservative. Proposition A.2 compares every Jacobian in the ball with J(θ0)J(\theta_0). With a,ba,b the extreme singular values of J(θ0)J(\theta_0) and r0=g(θ0)−Yr_0=g(\theta_0)-Y, its conditions reduce to one number: Q=H b ∥r0∥a3<q∗∈(0.11, 0.15].Q=\frac{H\,b\,\|r_0\|}{a^3}<q^*\in(0.11,\,0.15]. The certified radius (≤a/H\le a/H) therefore exceeds the required one (≥b∥r0∥/a2\ge b\|r_0\|/a^2) only when the latter is 7 to 9 times smaller, and any trajectory it certifies rotates Im J⊤\mathrm{Im}\,J^\top by less than about 19.5∘19.5^\circ (Remark I.1). This limitation belongs to the certificate, not to Assumption 2.1, which constrains singular values only.

Chained certificate (new Proposition I.1). We remove it by certifying a chain of local balls Bi=B(zi,ρi)B_i=B(z_i,\rho_i) centred at checkpoints of a numerical trajectory. Each ball has its own local constant HiH_i, with ci=ai−Hiρi>0c_i=a_i-H_i\rho_i>0 and Ci=bi+HiρiC_i=b_i+H_i\rho_i. On BiB_i the gradient field is one-sided Lipschitz with constant μi=2Hi(∥g(zi)−Y∥+Ciρi)\mu_i=2H_i(\|g(z_i)-Y\|+C_i\rho_i), which gives a Grönwall bound EiE_i between the exact flow and the numerical path. A first-exit argument keeps the exact flow in BiB_i whenever sup⁡0≤s≤hi[∥θ^(ti+s)−zi∥+Ei(s)]<ρi,\sup_{0\le s\le h_i}\big[\|\widehat\theta(t_i+s)-z_i\|+E_i(s)\big]<\rho_i, and one terminal condition, ϵN+CNcN2(rN+CNϵN)<ρN\epsilon_N+\tfrac{C_N}{c_N^2}(r_N+C_N\epsilon_N)<\rho_N, closes the infinite tail. Each ball is compared only with its own centre, so rotation is bounded per ball but unbounded cumulatively. Starting from ϵ0>0\epsilon_0>0 certifies every trajectory from B(θ0,ϵ0)B(\theta_0,\epsilon_0) at once. Full statement and proof are in Appendix I.2.

At ResNet-18 scale the remaining bottleneck is computing verified local constants HiH_i, which we do not claim to have done. What we can measure is the trajectory-level counterpart (Figure I.1). Along every UTKFace fine-tuning run, at the smallest and largest strengths, λmin⁡(K(θ))\lambda_{\min}(K(\theta)) stays bounded away from zero while growing by four to five orders of magnitude.

The negative result does not need the ball at all (new Proposition I.2). The bias conclusion of Theorem 3.1 holds whenever gradient flow converges to an interpolator θ⋆\theta^\star with J(θ⋆)J(\theta^\star) of full rank and θ⋆−θ0∉Im J(θ⋆)⊤\theta^\star-\theta_0\notin\mathrm{Im}\,J(\theta^\star)^\top. Both conditions are pointwise at the endpoint.

Q2. Evidence for the mechanism

We ran both diagnostics the reviewer proposed.

Fibre component of the force (Figure I.2). In the kernel regime, anchored ridge's gradient is exactly horizontal (Corollary D.6). Measured along fine-tuning, with the projector computed over the full training split, it is over 97% along the output fibres within 300 steps, at both strengths. The alignment condition of Theorem 3.1 therefore fails in a real fine-tuned network. Standard ridge lies along the fibres from the start (Remark D.11). The arc-ridge force, measured the same way, vanishes to solver precision throughout, as Remark E.1 predicts; this also validates the projector. The plotted fraction is a direction. The fibre force itself scales with λ\lambda (Theorem D.12), which is why it is harmless at small strength and damaging at large strength.

Matched training error (Figure I.3). Fibre motion is invisible to the training loss, so it shows up as vertical displacement in the train–test plane. At the smallest strength all three regularisers trace one curve, as Section 5 predicts. At the largest, arc ridge retraces that curve, while standard and anchored ridge leave it upwards. At equal training loss their test loss is about 1.7× and 1.27× that of arc ridge. The degradation is a generalisation effect at matched fit, not under-fitting alone.

We hope these additions resolve both questions, and would be grateful if the reviewer would take them into account in their assessment.

Thank you for the reply and running the experiments with the measurements that I proposed! I've checked the experiment on Figure I.2 and I.3 on Fibre component of the force and Matched training error and confirmed that my concerns for this problem has been resolved! I therefore keep my score on positive assessment of the paper!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.