acceptodds
Under review as a conference paper at ICLR 2027

Grokking through the Lens of Minimum-Norm Interpolation

Abstract

Grokking reveals that fitting the training data and learning the underlying signal can occur at very different stages. However, existing theories offer limited quantitative insight into how this separation depends on inductive bias and signal structure. Our work addresses this gap by developing a statistical theory that quantifies how regularization geometry and signal sparsity govern generalization near interpolation. In particular, we focus on the prototypical setting of high-dimensional regression and identify regimes in which sparsity-promoting regularization makes exact interpolation much more accurate than approximate fitting. In strongly overparameterized noiseless problems, we prove a zero-one generalization law and construct a convex family of norms whose interpolators transition from the trivial risk of the all-zero predictor to exact recovery, while keeping the training error always equal to . In the regime where feature dimension and sample size scale proportionally, we provide a precise characterization of the training and generalization errors along -regularization paths. This in turn allows us to quantify the generalization gain that remains near interpolation: we show that this gain increases as the norm becomes more sparsity-promoting and as the target becomes sparser, with a sharp drop in generalization reached for noiseless data and regularization. Experiments on diagonal linear networks and transformers trained on modular arithmetic demonstrate that the prediction of our theory extend well beyond the linear setting of the analysis.

Primary area:
learning theory
Keywords:
Grokking, minimum norm interpolation, regularized regression, square-root loss, overparameterization, benign overfitting, convex Gaussian min-max theorem (CGMT), LASSO, basis pursuit

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oSceOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the problem how regularization geometry and signal sparsity affect generalization in the regime where training data is almost perfectly fitted. The paper proposes a zero-one loss under the overparameterized (high-dimensional) noiseless regression setting. In the regime where feature dimension is proportional to the sample size, the authors characterize the estimation error of the optimal solutions of their regularized loss objective as well as the MNI solutions. They further apply their convergence results to understand, when a solution is nearly perfect for training data fitting, how much benefits to generalization does it have to make it fit the training data exactly, and how does it depend on the regularization norm geometry and weights sparsity.

Critical strengths
  1. I like this paper a lot due to its substantial theoretical contributions to the understanding of regularization-driven generalization. I checked most part of the proofs and haven't found any critical/unsolvable bug. Hence, I truly value that the results and proofs are rigorous and well-written. The proof seems novel and nontrivial, making the technical part of this paper strong.

  2. They provide a new perspective of grokking. Rather than understanding how a finite-step optimization algorithm induces grokking, they focus on understanding that, when a solution is nearly perfect for training data fitting, how much benefits to generalization does it have to make it fit the training data exactly.

Critical weaknesses
  1. I think the main theoretical results of the paper is not directly related to the topic "grokking". Basically, the paper proves a sharp transition in generalization across regularization geometries, but does not establish delayed generalization along an optimization trajectory in terms of the training steps/time. Later in section 4.2 and 4.3, the paper focuses on the properties of the optimal solutions to the regularized loss objective rather than optimization solutions with fixed training time. I noticed that the authors also propose in their conclusion section that "A natural next step is to quantify the correspondence observed between early-stopped diagonal networks and regression with their initialization-dependent implicit regularizer." Therefore, I think maybe it's better to not put the main focus on grokking but instead mention grokking as an application? That said, I still believe the current work provides a new perspective of understanding grokking and could potentially shed light on establishing end-to-end provable grokking in non-linear learning scenarios.

  2. The main results are almost asymptotic. Is it possible to derive non-asymptotic analysis with finite sample size and feature dimension to bound the gap to its asymptotic limit?

Questions for authors

My main concerns are in the above weakness part. The following are some minor things.

  1. In Equation (4), since the training loss is evaluated on the square root of MSE, I would expect the generalization/test loss to be evaluated on the square-rooted population MSE E[(y−xTβ)2]\sqrt{\mathbb{E}[(y-x^{T}\beta)^{2}]} where (x,y)(x, y) is an i.i.d. test instance. However, if I calculate correctly, its evaluation MSE(β)\text{MSE}(\beta) instead satisfies E[(y−xTβ)2]=(p/n)MSE(β)+σ2\mathbb{E}[(y-x^{T}\beta)^{2}] = (p/n)\text{MSE}(\beta) + \sigma^{2}. Therefore, I expect the generalization measure to be (p/n)MSE(β)+σ2\sqrt{(p/n)\text{MSE}(\beta) + \sigma^{2}}, though MSE(β)\text{MSE}(\beta) preserves comparisons. Moreover, I believe your results could survive under the evaluation of the square-rooted population MSE. Hence, there won't be any technical issue but just to make the results more natural and understandable.

  2. Regarding Lemma 1, since the authors study the ordinary ℓ1\ell_{1}-regularized MSE on the high-dimensional regression with sparsity (LASSO), I believe that abundant works have established convergence results for the estimator (convergence in probability or even almost-surely convergence). I don't mean that Lemma 1 is lacking novelty compared to existing works, but I think its incremental novelty needs to be clarified carefully.

  3. Regarding Theorem 1, while the result is clean and elegant, the regularization objective seems, however, artificial and impractical. It would be better if such a separation can be identified and proved with some natural choice of learning objectives/algorithms. Do you know any of such candidates?

  4. I found the writing of Section 4.2 a bit messy and not very understandable. For example, it is hard for me to follow what is the purpose to write Equations (12) and (13). I feel that add more explanations such as "why the scalar estimator studied here is related and useful to the previous high-dimensional regression problem" before the mathematical interpretation would make the idea more clear. Also, Lemma 2 is also hard to parse due to many notations are defined without explanation.

4 comments

Part 1

We thank the reviewer for the careful reading and positive assessment of our work. We respond to all comments below, and we have uploaded a revision marking the new text in red.

Weakness 1. Connection to grokking and scope of our contribution.

We agree that grokking is a temporal phenomenon. Our contribution addresses a complementary statistical question: why can further fitting yield substantial generalization improvements when training error is already nearly zero, and what determines those improvements? We quantify generalization against training residual, separating this question from the speed at which an optimizer traverses its trajectory. In the stated regimes, we prove that more sparsity-promoting regularization and sparser signals yield larger generalization gains near interpolation, with the gain becoming unbounded in the noiseless ℓ1\ell_1 case.

Our experiments on diagonal networks support the algorithmic relevance of this perspective by exhibiting the predicted dependence on initialization-induced bias and signal sparsity. Furthermore, in the paragraph “Regularization paths and training dynamics” in Section 6 of the revision, we develop a theoretical connection between gradient training and regularized regression by showing that, in the dual space, they both operate on the same strongly convex landscape. See also our response to Reviewer 3zYZ for details.

Having said that, a sharp characterization of the training curve remains an outstanding open direction. In that regard, the dual formulation put forward in Appendix F of the revision suggests exploring dynamical mean-field methods, such as those of Celentano et al. (2021), although applying them here is likely to require novel technical ideas.

Weakness 2. Non-asymptotic analysis.

Following the reviewer’s suggestion, the revised version provides non-asymptotic versions of the results in Lemma 1, Theorem 1, Lemma 2, and Lemma 3. We briefly discuss them below, and we have revised the manuscript accordingly.

Writing L=2log⁡(p/n)L=2\log(p/n) and retaining the stated calibration of k\*k_\* and the sparsity margin η\eta, for each fixed 0<κ<10 < \kappa < 1, Lemma 1 gives unique exact recovery with probability at least 1−e−cnη21-e^{-cn\eta^2}, for all sufficiently large dimensions along the stated sequence. For each fixed κ>1\kappa > 1, every minimizer satisfies

\le C_\kappa\frac{k_\*}{n}, \qquad 0\le1-\mathcal E_s(\widehat\beta_{\kappa,1}) \le C_\kappa\frac{k_\*}{n},$$ with failure probability at most $e^{-c_\kappa k_\*}$. Since $k_\*/n\asymp L^{-1}$, these are $O_\kappa(1/\log(p/n))$ deviations with exponentially small failure probability. Theorem 1 similarly gives exact interpolation with exponentially high probability and, above the transition, $$\left|\mathsf{MSE}_s(\widetilde\beta_\kappa)-1\right| \le C_\kappa\left(\frac{k_\*}{n}+\sqrt{\frac np}\right) =O_\kappa(L^{-1}).$$ Below the transition, it gives exact recovery, with an additional exponentially small failure term from the interpolation argument. Finally, for strictly positive noise and i.i.d. signal coefficients, both Lemma 2 and Lemma 3 can be made non-asymptotic, and the resulting rate is of order n-1/4. This is proved in the new Proposition 3 added to Appendix C.1 of the revision. **Question 1. Squared versus square-root generalization error.** We agree that Equation (4) should explicitly distinguish coefficient estimation error from population prediction RMSE. For an independent test covariate $x\sim N(0,I_p/n)$, conditionally on the training data, $$\mathbb E_x\langle x,b-\beta_\*\rangle^2 =\frac1n\\|b-\beta_\*\\|_2^2 =\frac pn\\,\mathsf{MSE}(b).$$ Thus, the prediction RMSE against the regression function is $\sqrt{(p/n)\mathsf{MSE}(b)}$, as the reviewer suggests. The prediction of a noisy test response additionally includes $\sigma^2$ in the squared error. Importantly, **the zero–one conclusions and quantitative rates survive the suggested square-root evaluation**. Normalization by the zero predictor gives $$\mathsf{RMSE}_s(b)=\sqrt{\mathsf{MSE}_s(b)},$$ and $|\sqrt{x}-1|\le|x-1|$ preserves the upper-side deviation bounds. Exact recovery still gives zero. We keep the MSE in Section 4.3, noting that taking square roots preserves which estimator generalizes better, but rescales derivatives and does not automatically preserve their ordering across different problem parameters.

Part II

Question 2. Novelty of Lemma 1 relative to existing ℓ1\ell_1 results.

We agree that sparse recovery and Lasso risk have an extensive literature. Lemma 1 concerns square-root Lasso, but we do not claim novelty for this estimator, the leading Donoho–Tanner recovery threshold, or Gaussian comparison methods. The novelty is in a quantitative all-or-nothing law along the regularization path: every fixed 0<κ<10 < \kappa < 1 yields exact recovery, whereas every fixed κ>1\kappa > 1 yields asymptotically trivial relative risk.

The substantive step is to control all competing predictors, including mixtures of signal and off-support coordinates. Comparing the objectives at β\*\beta_\* and zero suggests the threshold but does not exclude intermediate solutions. Above the transition, our proof establishes

\quad\Longrightarrow\quad \\|b\\|_2^2+\\|b\\|_1\le C_\kappa\frac{k_\*^2}{n}.$$ This controls the entire sublevel set, rather than only a selected optimizer. The $\ell_1$ bound controls the signal–estimator cross term and yields the $O_\kappa(L^{-1})$ relative-risk rate for every minimizer. In the revision (see the last paragraph of Section 4.1), we have expanded the comparison with [Bayati and Montanari (2012)](https://arxiv.org/abs/1008.2581) and [Miolane and Montanari (2021)](https://arxiv.org/abs/1811.01212). Their usual formulations at fixed positive aspect ratio do not directly establish our joint limit $n/p\to0$, $k_\*\sim n/(2\log(p/n))$, and $\log p=o(n)$ without additional uniform estimates. We also discuss the related all-or-nothing results of Reeves et al. ([2019a](https://proceedings.mlr.press/v99/reeves19a.html), [2019b](https://arxiv.org/abs/1912.13027)). Our trivial-risk conclusion is specific to the estimator, as regularization varies with the statistical model unchanged, rather than an information-theoretic impossibility as sampling parameters vary. **Question 3. Theorem 1’s construction and natural alternatives.** We agree that Theorem 1 is deliberately calibrated. However, the construction has a simple interpretation: with $m_\kappa=(p/\kappa)\sqrt{2/(\pi L)}$, it is the sum of the largest $m_\kappa$ absolute coordinates, with fractional interpolation when $m_\kappa$ is noninteger. It therefore preserves the $\ell_1$ cost of sufficiently sparse representations while discounting dense residual completions. We have clarified this connection to the classical top-k norm right before the statement of Theorem 1. **Question 4. Clarity of Section 4.2.** Thank you for pointing this out. We agree that the connection between the scalar estimator and the high-dimensional regression problem could use a clarification. To address this, we have revised the opening of Section 4.2 and added the new paragraph **“Scalar denoising problem.”** There, we first explain the purpose of the scalar characterization: it reduces the computation of the asymptotic training error and MSE to solving two equations for an effective noise level and a shrinkage parameter. We introduce the meaning of the scalar variables and parameters, and explicitly connect the scalar estimator to regression: under the assumptions of Lemma 2, the joint empirical distribution of the true and estimated coefficients converges to the law of $(B,f_{\tau_\kappa,\alpha_\kappa})$, as established in Remark 3. In particular, the scalar estimation error gives the limiting regression MSE. To clarify the notation used in Lemma 2, we now explain the roles of H_r,D_r,U_r before defining them in Equation (16). These quantities describe the normalized squared error, average sensitivity, and normalized coefficient moment, and enter the characterization of the MSE, training residual, and regularization calibration, respectively. Together with the explanation following Lemma 2—that the first equation in (17) determines the effective noise level from the observation noise and estimation error, while the second relates scalar shrinkage to the regression penalty—this makes explicit why these equations are introduced and how they characterize the original regression estimator.

References

I thank the authors for your careful response and the substantial amount of additional work that you have done in the rebuttal. I will maintain my score!

Reviewer_3zYZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies generalization performance of interpolators or solutions in linear regression with random Gaussian designs under different norm biases. The authors constructed a family of norms whose MNIs exhibit a zero–one generalization transition as the norm geometry varies, while maintaining interpolation.

Critical strengths
  • Clearly written, easy to follow.
  • Listed a few highly relevant work and explained the relationship with those previous work
  • The unified picture of minimum l_r norm interpolation and l_r norm regularization is interesting and useful
  • The designed ℓ1−ℓ∞\ell_1-\ell_\infty norm shows rapid transition along the κ\kappa axis is interesting
Critical weaknesses
  • The generalization of minimum L_p norm interpolators in overparameterized linear regression is well studied, as correctly cited in the related work section. It's well known that interpolators can have drastically different generalization, depending on the inductive bias and data distribution.
  • The connection to grokking seems far-fetched and is not very clear to me. In practice grokking is mostly known as a temporal phenomenom; in this paper the authors have shown that we can cleverly design a norm which does a rapid transition when tuning a parameter in the designing of that norm. If there's a way to map this norm transition onto the actual training dynamic, this would greatly improve the scope of the current paper.
Questions for authors

Is there any way to map this norm transition onto the actual training dynamic of neural networks?

2 comments

Part 1

We thank the reviewer for the insightful comments. The question concerns mapping the norm transition onto the actual training dynamics of neural networks. To address this, we discuss below a connection between regularization paths and the training dynamics of the diagonal linear networks of Section 5. We start from the experimental evidence of Figures 2–3 and then move to a theoretical connection we added to Section 6 of the revision (changes are reported in red).

First of all, let us note that Appendix E.1 empirically connects our analysis of generalization along a fixed regularization path to training dynamics. For each initialization scale kk, we have compared successive checkpoints of a two-layer diagonal network with QkQ_k-regularized regression solutions, holding QkQ_k fixed while varying regularization strength. At matched training error, Figure 3 shows that the two methods have closely matching parameter MSE. Here, training error provides a common coordinate for comparing these paths independently of optimization speed. Changing initialization between runs then changes QkQ_k, and Figure 2 shows a qualitative dependence of late-stage generalization gains on initialization that is consistent with our predictions concerning norm geometry. Therefore, we empirically show that the training dynamics of the two-layer diagonal neural network closely match the path of solutions of the corresponding QkQ_k-regularized regression problem, as both exhibit similar parameter MSE at the same training error.

In terms of theoretical results, the minimum-QkQ_k characterization at convergence follows from the implicit-bias analysis of Azulay et al. (2021). In the revision, we have added a paragraph concerning a dynamical connection at the end of the new Discussion section (Section 6 of the revision), with more details and a formal result (Proposition 6) deferred to the new Appendix F.

The key observation is that, in the dual space, gradient training and regularized regression operate on the same strongly convex landscape: both the gradient-flow iterate β^(t)\hat\beta(t) and the QkQ_k-regularized estimator β^κ\hat\beta_{\kappa} have the form ∇Qk∗(X⊤θ)\nabla Q_k^{\ast}(X^{\top}\theta) for the dual variable θ∈Rn\theta\in\mathbb{R}^{n}. In this dual variable, gradient flow on the square-root loss becomes the normalized gradient flow, started at 00, of the strongly convex function

=\sum_{j=1}^{p}q_{k}^{\ast}(\langle x_{j},\theta\rangle) -\langle y,\theta\rangle,$$ where $x_j$ are the columns of $X$ and $q_k^{\ast}(u)=\frac{\sqrt{k}}{4}(\cosh(2u)-1)$ is the convex conjugate of $q_k$. Furthermore, the regularized estimator with regularization $\kappa/\sqrt{n}$ corresponds to the minimizer of $\Phi_k$ over the ball $\\{\\|\theta\\|_{2}\le\sqrt{n}/\kappa\\}$; the minimum-$Q_k$ interpolator corresponds to the global minimizer $\theta^\star$ of $\Phi_k$; and the interpolation threshold corresponds to $\kappa_n^{\star}=\sqrt{n}/\\|\theta^{\star}\\|_{2}$. Training time can therefore be interpreted as inverse regularization in terms of the dual radius: the dual variable of $\hat\beta(t)$ has norm at most $t/\sqrt{n}$, while the dual problem defining $\hat\beta_{\kappa}$ constrains its variable to a ball of radius $\sqrt{n}/\kappa$. Thus, gradient flow at time $t$ is at least as regularized as $\kappa=n/t$ in terms of the dual radius, with equality when the residual direction does not rotate. Appendix F also provides the following connections: - Remark 5 shows that the distance between gradient flow and the regularization path is controlled by the angle between the current residual and the time average of the past normalized residuals, and it vanishes when the residual direction does not rotate. Before interpolation, $\hat\beta(t)$ is exactly the $Q_k$-regularized estimator at $\kappa_{\mathrm{eff}}(t)=\sqrt{n}/\\|\theta(t)\\|_{2}$ for the modified labels $\tilde y_t=X\hat\beta(t)+c\theta(t)$, for any $c > 0$. - Remark 6 describes the limits $k\to\infty$ and $k\to0$, which can be connected to prior work. As $k\to\infty$, after rescaling time and regularization strength, the coefficient dynamics approach ordinary linear gradient flow and the $Q_k$-regularization path approaches ridge regression, connecting to the setting studied by [Ali et al. (2019)](https://proceedings.mlr.press/v89/ali19a.html). As $k\to0$, the leading term of $Q_{k}$ is $\frac{1}{4}\log(1/k)\\,\\|\cdot\\|_{1}$, connecting to the analysis by [Berthier (2025)](https://arxiv.org/abs/2509.18766).

Part 2

Overall, these results provide a structural connection and a quantitative comparison between gradient flow and regularized regression. A sharp characterization of the training dynamics is a natural next step. The dual formulation put forward in Appendix F suggests exploring dynamical mean-field methods, such as those of Celentano et al. (2021), although applying them here is likely to require novel technical ideas.

References

Reviewer_ezDwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies generalization along a regularization path. Using the empirically observed correspondence that training with gradient descent closely follows a regularization path, such an analysis can be used to gain insight into grokking. With this setup, the paper studies the asymptotic (high-dimensional) limits of the training and generalization errors along the regularization path, with a focus on sharp changes in generalization near interpolation.

Critical strengths

The paper is technically strong and combines many results nicely.

  • Section 4.1: Lemma 1, which shows a sharp jump, is an interesting contribution.
  • Section 4.3: In the proportional regime, the generalization error and the training error change at different speeds near interpolation.
  • The machinery developed (Section 4.2) is good.
Critical weaknesses

Some of the results are a bit orchestrated through the choice of norms and the choice of regularization. For example, the jump in Lemma 1 as a function of κ comes from using the square-root loss: the standard Lasso traces the same solution path, but continuously in its regularization parameter. In Theorem 1, the ℓ∞ component of the hybrid norm only serves to fit the residual exactly. With such deliberate choices, the link to grokking is tangential at best.

Overall, it is nice to see the asymptotic limits of the training and generalization errors, and to see that for certain choices of norm and sparsity they behave very differently near interpolation.

Questions for authors

Why the square-root Lasso instead of the standard Lasso? Does the sharp transition in the MSE and the training error persist for the standard Lasso, for example in Lemma 1 and Section 4.3?

Similarly, what happens empirically when gradient descent is run on the squared loss 𝓔(β)²? Since gradient flow on the squared loss is a time reparametrization of gradient flow on the square-root loss, the ordering as a function of the training error should be unchanged. However, the squared loss slows down near interpolation. Does this create a delay in generalization, and how does it depend on k and ρ?

For the experiment with modular arithmetic, it is unclear why changing the output scale controls the implicit bias. This is a classification problem, and for exponentially-tailed losses the asymptotic implicit bias is max-margin and does not depend on the initialization scale [Chizat & Bach, 2020] (although it does at finite time [Moroshko et al., 2020]). Which norm or notion of sparsity does α control here, and how do weight decay and AdamW affect this?

[Chizat & Bach, 2020] Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss. COLT 2020. [Moroshko et al., 2020] Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy. NeurIPS 2020.

3 comments

Part 1

We thank the reviewer for the positive assessment of our technical contributions. We respond to the questions below, and we have uploaded a revision marking the new text in red.

Square-root versus squared loss.

This is a very good point, and we now clarify the rationale behind choosing square-root loss rather than squared loss in the paragraph “Square-root loss vs. squared loss” in Section 6 of the revision.

The square-root loss preserves the homogeneous geometry of minimum-norm interpolation. For a norm N with unit ball B_N, a full-row-rank design X, and τ>0\tau > 0, the optimized value Vτ(y):=inf⁡b∣y−Xb∣2+τN(b)V_\tau(y):=\inf_b\\{\\|y-Xb\\|_2+\tau N(b)\\} is itself a norm, with unit ball conv⁡(B2n∪τ−1XBN)\operatorname{conv}(B_2^n\cup\tau^{-1}XB_N), where B_2n is the Euclidean unit ball. This infimal convolution combines residual and interpolation costs through a simple convex-hull construction. Squared loss instead yields a Moreau envelope, losing this degree-one homogeneous structure.

Crucially, square-root loss permits exact interpolation at positive regularization, while squared loss does not. In fact, squared loss with a positive norm penalty cannot interpolate nonzero observations: if Xb=y≠0Xb=y\neq0, shrinking b to (1−ε)b(1-\varepsilon)b saves λεN(b)\lambda\varepsilon N(b) in penalty but costs only ε2∣y∣22/2\varepsilon^2\\|y\\|_2^2/2 in squared residual, where λ>0\lambda > 0 is the penalty coefficient. For sufficiently small ε>0\varepsilon > 0, this improves the objective. In contrast, with square-root loss, both changes are first order, allowing exact interpolation to remain optimal. This motivates our choice of square-root loss in the paper.

Nevertheless, as pointed out by the reviewer, solutions with positive training error correspond after a reparameterization. We also discuss this reparameterization in the context of training diagonal linear networks in Remark 4 in Appendix F of the revision. As a result, the findings of Section 4.3 concerning the comparison between MSE and training residual still apply. We finally note that the correspondence between squared loss and square-root loss breaks down at exact interpolation, and we do not claim that Lemma 1’s simultaneous zero–one transition persists unchanged along the standard-Lasso parameter.

What happens empirically when gradient descent is run on the squared loss?

We agree with the reviewer that gradient flow on the squared loss is a time reparameterization of gradient flow on the square-root loss and, as such, it does not change the ordering as a function of training error. In fact, if we were to replot Figure 2 (left, center) with squared loss on the x-axis instead of square-root loss, the ordering of the curves would be the same. Furthermore, with squared loss on the x-axis, all those curves would be concave, whereas they are convex with square-root loss on the x-axis. This is consistent with the delay in generalization expected by the reviewer as a consequence of having a squared loss, and it results in an amplification of the generalization gain near interpolation (as the generalization gain is now concave in the training error). As for the role of k and ρ\rho, since the training error is smaller with squared loss, k and ρ\rho will have less impact on the generalization gain (this can be seen by comparing vertical cuts in the left and center plots of Figure 2).

Part 2

Why changing the output scale controls the implicit bias.

Our choice to vary α\alpha is motivated by Kumar et al. (2024), who show that output scaling can modify feature-learning dynamics and delayed generalization, including in transformers trained with cross-entropy and AdamW. Our focus is on how much generalization improves as training error approaches zero, rather than on the duration of the generalization delay. We therefore use α\alpha to probe this behavior under different finite-time training dynamics, without claiming that it selects a particular asymptotic max-margin solution. As the reviewer notes, scale-dependent behavior at finite training loss need not imply different asymptotic implicit biases (Moroshko et al., 2020). Moreover, the assumptions underlying Chizat and Bach (2020) do not directly cover our transformer trained with AdamW and weight decay. Studying which norm or notion of sparsity is controlled by α\alpha, as well as the interaction of weight decay and AdamW, remains an interesting direction for future research. To avoid confusion, we have opted to remove the reference to implicit bias in the last paragraph of Section 5 of the revision.

References

Thank you for the detailed response and the revision. I appreciate the explanation of the difference between the square-root and squared Lasso, and the clarification on the role of the output scale and the implicit bias in the modular arithmetic experiment. However, I still find the choice of norms somewhat curated, so I maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.