acceptodds
Under review as a conference paper at ICLR 2027

ON THE TWO FACES OF ADAM IN SEPARABLE LINEAR CLASSIFICATION

Abstract

We consider the behavior of deterministic, full-batch, bias-corrected Adam in separable linear classification with softmax parametrization under log-loss. In this setting, under a wide range of conditions Adam is known to approach max-norm-margin optimality when its stability constant \(\epsilon\) is zero, while with a positive \(\epsilon\), it is known to approach Euclidean-margin optimality. Our main contribution is the quantitative description of Adam's behavior for small fixed positive . We give sufficient conditions under which an Adam-trained classifier nearly maximizes the max-norm margin before the updates become gradient-like. We also show that the classifier reaches a fixed target Euclidean margin only much later. Specifically, we show that for polynomially decreasing stepsizes with exponent \(a\), where , the updates become approximately proportional to the negative gradient after iterations. At that time, the classifier still nearly maximizes the max-norm margin. Reaching a fixed target Euclidean margin above that of every max-norm-optimal classifier, but below the optimum, is shown to require iterations. Under inverse-linear stepsize decay (), the update transition takes polynomially many iterations, whereas reaching the target margin takes exponentially many. Experiments support these predictions. The later change in the classifier can improve or worsen generalization after training error reaches zero, connecting the analysis to grokking and its reverse.

Primary area:
learning theory
Keywords:
Adam's Implicit Bias Transition, Adam's Local Update Transition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tZHZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper theoretically investigates the behavior of Adam with stability parameter ϵ\epsilon in separable linear classification. This paper characterize the transition of Adam's updates from sign-like to gradient-like behavior and analyze how the classifier evolves from near ℓ∞\ell_\infty-margin optimality toward ℓ2\ell_2-margin optimality.

Critical strengths
  • This paper studies the behavior of Adam with a small positive stability constant ϵ\epsilon in separable linear classification.
  • The paper provides a theoretical characterization of how Adam's implicit bias transitions from ℓ∞\ell_\infty-margin optimality to ℓ2\ell_2-margin optimality.
Critical weaknesses
  • The main weakness of this paper is its presentation, which makes the theoretical contributions unnecessarily difficult to understand. The paper introduces numerous auxiliary quantities and presents many results in terms of these quantities, rather than clearly explaining what the results imply. For example, Proposition 3.2 involves several layers of notation, such as rϵr_\epsilon, ℓϵ\ell_\epsilon, Htδ(ϵ)H_t^\delta(\epsilon), and Λϵ\Lambda_\epsilon. As a result, readers must repeatedly trace back through definitions and substitute multiple expressions to understand the actual dependence on ϵ\epsilon and tt. More importantly, the paper does not sufficiently explain the implications of its theoretical bounds or how the various results fit together to support the main conclusions. I believe the paper needs a substantial reorganization that emphasizes the key theoretical insights and their interpretations, rather than simply presenting a sequence of technically detailed results.
  • The claim in the Introduction that Adam's updates are nearly aligned with the sign gradient appears overstated. Theorem 2 only shows that the sign residual converges to zero, but since this residual is weighted by the magnitude of each gradient coordinate, i.e., ∥gt⊙(Dt−\sign(gt))∥\| g_t \odot (D_t - \sign(g_t))\|, Theorem 2 does not immediatelly imply that the update direction of Adam is equivalent to sign gradient descent.
  • Λϵ\Lambda_\epsilon is used in line 247, while it is defined later, i.e., in line 265.

Overall, the theoretical results appear interesting. However, the large amount of notation and the difficulty of understanding the relationships among the main results substantially reduce the readability of the paper. I strongly encourage the authors to improve the presentation.

Questions for authors

See the weakness section.

3 comments

Thank you for your feedback on our paper. We address your concerns below.

1. Presentation (Weakness 1)

We have added Figure 1 to the revised file to help illustrate our main results. However, within the time available for this rebuttal, we have not yet fully addressed all presentation concerns. We will work on further improving the exposition.

2. Sign residual (Weakness 2)

Below, we distinguish between the motivating coordinatewise intuition and the guarantee established by Theorem 2.

The intuition can be seen directly in the memoryless case, where β1=β2=0\beta_1=\beta_2=0. In this case, Adam’s update direction is exactly Dt[j]=gt[j]∣gt[j]∣+ϵD_t[j]=\frac{g_t[j]}{|g_t[j]|+\epsilon}. Thus, on coordinates where ∣gt[j]∣≫ϵ|g_t[j]|\gg\epsilon, the update is close to sign⁡(gt[j])\operatorname{sign}(g_t[j]), whereas on coordinates where ∣gt[j]∣≪ϵ|g_t[j]|\ll\epsilon, it is close to gt[j]/ϵg_t[j]/\epsilon. In particular, if every nonzero gradient coordinate dominates ϵ\epsilon, the update is coordinatewise close to the sign-gradient vector. This is the mechanism underlying the introductory intuition.

With momentum, the update also depends on past gradients, so the exact coordinatewise identity above no longer holds in general. Moreover, Theorem 2 does not assume that every nonzero gradient coordinate dominates ϵ\epsilon. We therefore motivate and define the sign residual through its connection to first-order loss decrease. A small sign residual means that Adam’s first-order loss decrease is close to that of sign descent at the same iterate and with the same step size. As the reviewer correctly notes, this does not, by itself, establish unweighted closeness of the update to the sign-gradient vector.

This is the sense in which our analysis characterizes the update as “sign-like.” The motivation for this residual and the detailed justification of its interpretation are provided in Section 2, immediately after the residual definitions.

We have revised the introductory wording to make this interpretation explicit and to distinguish it from the motivating coordinatewise intuition.

3. Placement of Λϵ\Lambda_{\epsilon} (Weakness 3)

Thank you for pointing this out. We have revised Section 3 to introduce Λϵ\Lambda_\epsilon before its first use in the definition of Tˉϵ\bar T_\epsilon. When introducing it, we state the required conditions Λϵ→∞\Lambda_\epsilon\to\infty and ϵΛϵ→0\epsilon\Lambda_\epsilon\to0 as ϵ↓0\epsilon\downarrow0 and indicate that its specific choice will be given below.

We retain the explicit formula immediately after the preliminary error bound to explain why this choice is useful. Specifically, we choose Λϵ\Lambda_\epsilon so that the two terms ℓϵ/Λϵ\ell_\epsilon/\Lambda_\epsilon and ∑t<Tˉϵηt2\sum_{t<\bar T_\epsilon}\eta_t^2 are both controlled by the common bound O(ωa(ℓϵ))O(\omega_a(\ell_\epsilon)). This preserves the motivation for the formula while ensuring that the parameter is introduced before it is used.

We thank the authors for the response. I agree that there is insufficient time during the rebuttal period to fix the presentation issues. However, the current manuscript is extremely difficult to read due to notation issues. I cannot support accepting the paper in its current state, and therefore I would like to maintain my negative score.

We believe that our presentation can be made clearer. However, we respectfully disagree that our paper has significant notation problems or that the remaining presentation concerns outweigh its theoretical contribution. Could you clarify which notation issues remain unresolved after our revisions and which specific definitions, statements, or passages are still difficult to follow?

Reviewer_kkNqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

For separable linear classification, deterministic Adam(with stability constant ϵ>0\epsilon>0)’s updates become gradient-like long before its classifier shifts from L∞L_\infty- to L2L_2-max-margin behavior, with a provably large delay for small stability constant ϵ\epsilon.

Critical strengths

A major strength is the paper’s rigorous study of Adam’s path during training for separable linear classification. Earlier work mainly studies Adam’s limiting behavior: convergence toward an L∞L_\infty-max-margin classifier when ϵ=0\epsilon=0, or an L2L_2-max-margin classifier when ϵ>0\epsilon>0, under the respective assumptions. This paper shifts the focus to how the classifier evolves when ϵ\epsilon is small and positive. It shows that the classifier can stay close to an L∞L_\infty-max-margin solution long after the updates become gradient-like. It also gives rigorous quantitative bounds on the transition times, establishing a large separation between the change in the updates and the classifier’s approach to L2L_2-max-margin behavior.

Critical weaknesses

The transition-time bound Tclassifier=ϵ−Θ(1)/(1−a)T_{\mathrm{classifier}} = \epsilon^{-\Theta(1)/(1-a)} leave the polynomial exponent for the later classifier transition unresolved. The bounds establish a clear separation between the two transitions, but give limited guidance on when the classifier transition will occur for a given dataset and training budget.

Questions for authors

Do the authors expect a sharper result of the form Tclassifier=ϵ−κ/(1−a)+o(1)T_{\mathrm{classifier}}=\epsilon^{-\kappa/(1-a)+o(1)}, where κ\kappa is independent of aa for a fixed dataset, target margin accuracy, and other optimizer settings? Or could κ\kappa itself depend on aa?

As an empirical check, could the authors vary ϵ\epsilon for several values of aa and compare the slopes of (1−a)log⁡Tclassifier(1-a)\log T_{\mathrm{classifier}} against log⁡(1/ϵ)\log(1/\epsilon), similar to Figure 1(d)? Do these slopes agree across aa, and do they change across datasets?

5 comments

Thank you for your positive feedback on our paper. We address your questions below.

1. Transition-time bound (Question 1)

We expect the leading exponent κ\kappa to be independent of aa for a fixed dataset, target margin accuracy, and other optimizer settings. Intuitively, aa mainly controls the speed of training, while the dataset geometry and target accuracy determine how much the classifier’s direction must change. Once Adam’s updates become gradient-like, we expect the learning-rate schedule to primarily affect the speed of this directional evolution.

This remains a conjecture: our current bounds do not identify the sharp exponent or rule out additional dependence on aa. We expect κ\kappa to vary across datasets and target accuracies.

2. Additional experiments for different values of aa (Question 2)

We conducted additional experiments using the paper’s original covariance setting and the learning-rate schedule ηt=10−3/(t+1)a\eta_t=10^{-3}/(t+1)^a. Figure 5 in Appendix A.1 of the revised file shows approximately linear trends between (1−a)log⁡Tclassifier(1-a)\log T_{\rm classifier} and log⁡(1/ϵ)\log(1/\epsilon). The fitted slopes of the mean curves are:

aaFitted slope
0.400.0637
0.450.0682
0.500.0710
0.550.0726

The slopes are similar in magnitude and increase with aa, so these finite-ϵ\epsilon results do not establish an aa-independent asymptotic exponent. At a=0.4a=0.4, the fitted slopes for individual datasets range from 0.0311 to 0.1004, showing variation across dataset draws.

We also present additional results for a=1a=1 in Figure 4 in Appendix A.1 of the revised file, which illustrates an approximately linear relationship between log⁡log⁡Tclassifier\log\log T_{\rm classifier} and log⁡(1/ϵ)\log(1/\epsilon).

Thank you for the response and additional experiments. In Figure 5(b), the trend is not always linear, but rather closer to convex (U-shaped curve). Do the authors have intuition on this mismatch between experiments and theoretical predictions?

Thank you for pointing this out. Our intuition is that finite-ϵ\epsilon effects can remain noticeable over the plotted range. Corollary 6 establishes that (1−a)log⁡Tclassifier(1-a)\log T_{\rm classifier} grows on the order of log⁡(1/ϵ)\log(1/\epsilon) for sufficiently small ϵ\epsilon. It does not identify a unique limiting slope or rule out curvature over the tested range. How small ϵ\epsilon needs to be also depends on the dataset and optimizer parameters.

Figure 5(b) uses ϵ∈{10−4,10−5,…,10−11}\epsilon\in\lbrace10^{-4},10^{-5},\ldots,10^{-11}\rbrace. To examine the smaller-ϵ\epsilon portion of this range, we compared linear fits using all eight values with those using the six smallest values, {10−6,…,10−11}\lbrace 10^{-6},\ldots,10^{-11} \rbrace, excluding 10−410^{-4} and 10−510^{-5}.

Dataset seedR2R^2: all eight values, 10−410^{-4} through 10−1110^{-11}R2R^2: six smallest values, 10−610^{-6} through 10−1110^{-11}
00.98360.9936
10.92490.9954
20.80880.9545
30.97290.9937
40.82850.9636
50.95880.9821
60.96360.9931
70.99260.9915

The R2R^2 values improve for seven of the eight datasets, with the largest improvements for seeds 2 and 4. This suggests that the trend can become more linear over the smaller-ϵ\epsilon range for most datasets.

Reviewer_AhAnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the implicit bias of deterministic full-batch adam in linearly separable classification, focusing on how its stability parameter ϵ\epsilon influences optimization dynamics. Previous work established that Adam with ϵ=0\epsilon=0 converges toward the ℓ∞\ell_\infty max-margin geometry, whereas fixed positive ϵ\epsilon ultimately induces ℓ2\ell_2 max-margin behavior. The paper provides a quantitative characterization of the transition between these regimes.

The central contribution is a separation between the transition of Adam's update directions and the subsequent transition of the learned classifier. For a range of polynomially decaying learning rates the authors show that Adam's updates become gradient-like after Θ(log⁡1/(1−a)(1/ϵ))\Theta(\log^{1/(1-a)}(1/\epsilon)) iterations. However, the classifier remains approximately ℓ∞\ell_\infty max-margin optimal at this point, and reaching a prescribed ℓ2\ell_2-margin accuracy requires ϵ−Θ(1)/(1−a)\epsilon^{-\Theta(1)/(1-a)} iterations, provided the two optimal margin geometries are separated. Analysis on inverse-linear learning-rate decay is provided too.

The theoretical analysis develops bounds on Adam's accumulated displacement, update residuals, and margin evolution, with extensions to multiclass classification. Experiments on synthetic linearly separable datasets support the predicted separation of timescales, while experiments using two-layer ReLU networks on MNIST provide qualitative evidence of similar behavior.

Critical strengths
  • The paper advances beyond existing asymptotic characterizations of Adam by explicitly separating the timescale at which updates become gradient-like from the timescale at which the classifier approaches Euclidean max-margin optimality. The polynomial-versus-polylog separation, and its stronger counterpart under inverse-linear stepsize decay, provide a meaningful theoretical explanation of how different implicit biases can dominate at different phases of the training

  • The paper develops a thorough analysis of bias-corrected Adam, including control of accumulated displacement, moment approximation errors, and margin evolution. The extension to multi class classification and the experiments on synthetic data and nonlinear neural nets strengthen the results. In particular, the numerical evidence illustrates that the classifier can remain close to its earlier margin geometry even after the optimizer's updates have become strongly gradient-like.

Critical weaknesses
  • The main guarantees concern deterministic full-batch Adam on linearly separable data with particular polynomial decaying learning rate schedules and constraints on the momentum parameters. Furthermore, the separation in classifier transition times requires a positive geometric gap between the ℓ∞\ell_\infty and ℓ2\ell_2 margin solutions. These assumptions are appropriate for theory, but it remains unclear how broadly the predicted behavior extends to stochastic mini-batch training, constant learning rates, or more general nonlinear objectives - which would be important for practitioners.

  • The synthetic experiments primarily examine one lr exponent and a restricted set of datasets, while the nonlinear experiments use a constant lr and therefore do not directly test the iteration-complexity predictions. The exponential classifier-transition timescale predicted for, for example a=1a=1 is not experimentally investigated. Moreover, the classifier transition is not observed within the training horizon for the smaller stability parameters in the nonlinear experiments. Currently the experiments provide limited evidence for the broader quantitative predictions and their practical relevance.

Questions for authors
  • Could the authors provide additional experiments with different decay exponents, particularly a = 1, to investigate whether the predicted polynomial and exponential timescales are reflected in observed trajectories?
  • To what extent does the separation between update and classifier transitions persist under stochastic mini-batch Adam or constant learning rates?
  • How frequently is Δgeom>0\Delta_{\mathrm{geom}}>0 satisfied, and how does its magnitude influence the observable separation between the two transitions?
3 comments

Thank you for your feedback on our paper. We address your concerns/questions below.

1. Scope of the guarantees (Weakness 1)

Our paper addresses a challenging question even in the deterministic, linearly separable setting: how long can the classifier retain its earlier geometry after Adam’s updates have become gradient-like? As the reviewer acknowledges, our analysis provides new information beyond existing endpoint implicit-bias results by distinguishing and quantifying these two transition timescales. Establishing this separation already requires a detailed analysis of Adam’s moment estimates, accumulated displacement, and margin evolution.

We agree that stochastic mini-batches, constant learning rates, and more general nonlinear objectives are important settings to investigate. Extending our quantitative guarantees to these settings would require substantial additional analysis of the optimization dynamics and the geometry selected during training, and is beyond the scope of the present theoretical study. We nevertheless believe that the qualitative separation could persist in some of these settings, potentially in a weaker or less uniform form, as discussed in our response to Question 2. Our nonlinear experiments are intended as qualitative illustrations, rather than tests of the iteration-time exponents established for the linear setting.

The positive-gap assumption has a specific role: it excludes a common optimal direction under the two geometries and is needed for a nontrivial lower bound on the subsequent classifier change. Without it, a distinct classifier transition need not occur. The update-crossover results do not require this assumption.

2. Stochastic Mini-batches and constant learning rates (Weakness 1 and Question 2)

A similar separation may occur in these settings. The key intuition is that a change in update dynamics does not immediately change the classifier direction accumulated during earlier training. If the earlier phase has built a large parameter vector, substantial subsequent movement may be needed to change its direction.

Stochastic mini-batches. If the mini-batch gradients and their fluctuations become sufficiently small for ϵ\epsilon to dominate Adam’s denominator, the updates become approximately those of rescaled SGD with momentum. However, mini-batching can alter the geometry selected during training. For example, Baek et al. (ICLR 2026) show that cyclic per-sample Adam with ϵ=0\epsilon=0 can have a different implicit bias from full-batch Adam. Thus, the earlier classifier need not be close to the ℓ∞\ell_\infty-max-margin solution. Whether a distinct classifier transition occurs depends on the sampling scheme, gradient fluctuations, and the difference between the directions favored by the earlier and later dynamics. If these directions coincide, a distinct transition need not occur.

Constant learning rates. A fixed learning rate does not imply a fixed parameter displacement. Once ϵ\epsilon dominates Adam’s denominator, the actual updates can shrink as the gradients decrease. The subsequent accumulated movement may therefore remain small relative to the inherited classifier norm for an extended period, making a qualitative separation possible under stable training dynamics.

3. Additional experiments for a=1a=1 (Weakness 2 and Question 1)

We examined the step-size schedule ηt=10−3/(t+1)\eta_t=10^{-3}/(t+1) following the same experiment setup as the paper. We fit a separate line for each seed. The table below reports the mean and sample standard deviation of the fitted slopes, together with the mean seedwise R2R^2. Figure 4 in Appendix A.1 of the revised file instead annotates the fit to the across-seed mean curve. Its R2R^2 value is also presented in the table.

Theorem 4 predicts γ∞(ST1−ST⋆)=ℓϵ+o(ℓϵ)\gamma_\infty(S_{T_1} - S_{T_\star})=\ell_\epsilon+o(\ell_\epsilon), including for a=1a=1. Corollary 6 predicts log⁡log⁡τζ=Θ(ℓϵ)\log\log\tau_\zeta=\Theta(\ell_\epsilon) for a=1a=1. The approximately linear trends observed are consistent with these scaling statements. For other values of aa, the linear trends are also preserved (see Figure 5 in Appendix A.1 of the revised file).

Mean fitted slope ± SDMean seedwise R2R^2R2R^2 of the mean-curve fit
γ∞(ST1−ST⋆)\gamma_\infty(S_{T_1}-S_{T_\star}) versus log⁡(1/ϵ)\log(1/\epsilon)0.9962 ± 0.00250.99999880.9999997
log⁡log⁡τζ\log\log\tau_\zeta versus log⁡(1/ϵ)\log(1/\epsilon)0.0930 ± 0.03400.96560.9754

4. Frequency and size of the geometric gap (Question 3)

We tested 300 datasets generated using the paper’s Gaussian-mixture construction, with 100 datasets for each of three covariance settings. The gap was positive in every dataset. The table reports the gap as a percentage of the optimal Euclidean margin; values are mean ± standard deviation.

Coordinate standard deviationsDatasets with a positive gapRelative geometric gap
All equal to 1100/10019.4% ± 1.9%
Ranging from 1 to 3 (paper setting)100/10019.3% ± 2.0%
Ranging from 1 to 10100/10019.1% ± 2.2%

These results suggest that the assumption is common in the tested data families. We do not claim that it holds for every separable dataset.

A positive gap means that every max-norm-optimal classifier falls short of the optimal Euclidean margin. Equation (175) Hτζ≥Δgeom−ζ4R2∥wT1∥2−E0H_{\tau_\zeta} \ge \frac{\Delta_{\rm geom}-\zeta}{4R_2} \lVert w_{T_1} \rVert_2-E_0 shows that, for sufficiently small ϵ\epsilon, a larger gap gives a stronger lower bound on the effective movement required after the update transition (i.e. Hτζ=∑s=T1τζ−1ηs∥gs∥2ϵH_{\tau_{\zeta}} = \sum_{s=T_1}^{\tau_{\zeta} - 1}\frac{\eta_s \lVert g_s \rVert_2}{\epsilon}), when the other quantities are held fixed. It does not, by itself, imply a later transition in iteration count: the inherited classifier norm, the optimization dynamics, and the target accuracy also matter. A smaller gap weakens the guaranteed separation and may make it harder to resolve at finite ϵ\epsilon. In our experiments, the target is ζ=0.75Δgeom\zeta=0.75\Delta_{\mathrm{geom}}, so varying the gap also varies the absolute accuracy target.

Several of my points have been addressed however I still have some concerns. The inverse-linear-decay experiments are added. I looked at it and Figure 4(b) plots log⁡log⁡τζ\log\log\tau_\zeta at values roughly 6–9. Even a value of 6 corresponds to approximately 1017510^{175} which doesn't seem consistent to iteration-by-iteration Adam runs, so I am confused at what was actually run. How are iteration counts reconstructed and how is approximation error was assessed against discrete, bias-corrected Adam. I also agree the presentation is hard to digest particularly the many stopping times and auxiliary quantities. I retain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.