acceptodds
Under review as a conference paper at ICLR 2027

Continuous Approximation for Gradient-based Optimization: Arbitrary-Order Discretization Error and Implicit Bias

Abstract

Gradient-based optimization methods are the backbone of modern machine learning and deep learning, and thus the theoretical understanding of their dynamics and implicit biases has been crucial recently. A promising line of works in this direction leverages continuous-time approximations to capture the behavior of discrete and iterative updates, shedding light on their convergence, stability, and regularization properties. However, these approximations typically suffer from discretization errors that could become significant with large learning rates. To this end, we propose a unified framework for constructing continuous-time approximations with discretization error to arbitrary order of learning rate. This framework can be applied to a wide range of gradient-based methods, including momentum methods and adaptive variants like Adam. This is achieved by segmented Taylor expansions and parameter-splitting techniques that generalize prior works of standard gradient descent, hence enabling a precise analysis of more delicate optimizer dynamics. As a key case study, we derive the first third-order continuous approximation of Adam, which exhibits novel implicit regularization terms that cannot be obtained from lower-order ones. This allows us to identify a new phase transition occurring in Adam's training, explaining its acceleration and generalization behavior, respectively.

Primary area:
learning theory
Keywords:
continuous optimization, gradient-based methods, adam, discretization error

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gJ3WOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a unified framework for constructing arbitrary-order continuous-time approximations of gradient-based optimization algorithms. By introducing segmented continuous dynamics and recursively determining higher-order correction terms, the framework aims to control the discretization error to O(ηρ)O(\eta^\rho) for an arbitrary positive integer ρ\rho. The authors demonstrate the framework on gradient descent and heavy-ball momentum and extend it to adaptive optimizers through a parameter-splitting technique.

As the main application, the paper derives a third-order continuous approximation of Adam and identifies two proposed implicit regularization terms, RaR_a and RgR_g. The authors interpret these terms as describing a gradual transition from an early optimization regime associated with acceleration to a later regime associated with improved generalization. Numerical experiments on a two-dimensional model compare approximation accuracy, while experiments using LeNet on MNIST, including a label-noise setting, illustrate the proposed regularization transition.

Critical strengths
  1. A potentially useful unified framework for higher-order continuous approximations.

The paper attempts to systematically extend continuous-time analysis from standard gradient descent to more complicated optimizers involving momentum and adaptive learning rates. The recursive construction is interesting because it could provide a reusable approach for investigating optimizer dynamics beyond leading-order approximations.

  1. An interesting application to Adam's implicit regularization.

The explicit third-order approximation and the proposed transition between different regularization regimes provide an interesting perspective on Adam's optimization behavior. The accompanying numerical experiments offer some empirical motivation for studying higher-order effects.

Critical weaknesses
  1. The technical presentation is excessively complicated and substantially obscures the main contribution.

Despite careful reading, I found the main text extremely difficult to follow. The central mathematical ideas are buried under lengthy recursive expressions, complicated notation, and extensive algebraic formulas. In particular, Theorem 1 and the third-order Adam approximation are presented without sufficiently accessible explanations of their underlying mechanisms and significance. Much of the reasoning needed to understand the main claims is deferred to the appendices.

For a theoretical paper, mathematical complexity is sometimes unavoidable. Nevertheless, the main text should allow readers to understand what the framework accomplishes, why the construction works, and what insights it provides before requiring them to navigate the full derivations. In its current form, the paper does not communicate these points effectively, making its main contributions unusually difficult to assess.

  1. The significance of the higher-order approximation and its implicit-regularization interpretation is not sufficiently established.

Continuous-time approximations and implicit regularization of Adam have already been studied, notably by Ma et al. (2022) and Cattaneo et al. (2024), while arbitrary-order approximations for momentum methods have also been investigated. Although the paper extends the approximation order for Adam, it does not sufficiently clarify what fundamentally new understanding is obtained beyond these existing results.

More importantly, the interpretation of the proposed regularization quantities Ra=−sgn⁡(g)⊤Hsgn⁡(g)R_a=-\operatorname{sgn}(g)^\top H\operatorname{sgn}(g) and Rg=g⊤HgR_g=g^\top Hg appears stronger than what these quantities directly establish. They measure Hessian quadratic forms along particular directions, rather than overall sharpness or flatness. In particular, RgR_g may decrease simply because the gradient norm decreases, even if the Hessian remains unchanged. Thus, its decrease does not necessarily indicate a preference for flatter solutions.

The experiments illustrate correlations between these quantities, the proposed regime transition, and generalization behavior, but they do not convincingly establish that the identified regularization mechanisms are responsible for Adam's acceleration or improved generalization. Since this interpretation is a central motivation for the third-order analysis, the limited evidence substantially weakens the paper's scientific significance.

Questions for authors
  1. Novelty relative to existing continuous approximations. Could the authors clarify which theoretical insights or predictions are uniquely enabled by the third-order Adam approximation, compared with the second-order analysis of Cattaneo et al. (2024)? Beyond a reduction in trajectory approximation error, what new behavior can be established that cannot be explained using existing lower-order models?

  2. Interpretation of the regularization quantities. How do the authors distinguish the decrease of Rg=g⊤HgR_g=g^\top Hg caused by a reduction in gradient magnitude from a genuine preference for flatter regions? Would normalized directional curvature or direct Hessian-based measurements provide stronger evidence for the proposed interpretation of RaR_a and RgR_g?

  3. Evidence for the relationship with acceleration and generalization. The experiments suggest that changes in the regularization quantities coincide with changes in training dynamics and test accuracy. Can the authors provide stronger evidence that the proposed regularization transition explains these changes, rather than merely correlating with the reduction of gradient magnitudes during optimization?

  4. Computational feasibility and utility. The third-order approximation involves higher-order derivatives of the loss. How computationally expensive would it be to evaluate or numerically integrate the proposed continuous dynamics for a modern high-dimensional neural network? Is the intended advantage primarily theoretical, or can the approximation provide practical predictions or computational benefits beyond the low-dimensional experiments?

Reviewer_4paAOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper provides a general method to derive continuous approximations for first-order algorithm to an arbitrary high order. The framework includes momentum based method, and adaptive methods (ADAM). High order continuous approximations can be useful to obtain insights on the associated algorithm, such as implicit bias. Here, the authors use their framework to derive the first third order continuous approximation of ADAM, and to highlight a new implicit bias, that is not visible for lower order approximations.

Critical strengths

The paper is clear and well written. If Corollary 1.2 indeed holds, it is a very interesting result about the implicit bias of ADAM.

Critical weaknesses
  • I have some concerns about how the paper presents its contributions of Section 2. The presentation suggests it is a new approach, while it is significantly inspired from Lyu et al., 2025. Specifically, Theorem 1 is very similar in its form to Theorem 2.1 in Lyu et al., 2025, precisely replacing the gradient of the loss with a function fkf_k and replacing constant momentum with variable μk\mu_k. The figure on page 3 is almost the same as the one of Lyu et al. 2025 (p 14). The related proofs are thus also similar. Lyu et al., 2025 is cited in this work, but it is not acknowledged that Section 2 is heavily inspired from this work.

  • The other contribution (Section 3) seems interesting, but I am not convinced by its derivation. It uses lots of approximation ≈\approx that are not quantified, and arguments of the form "generally/usually it holds that...". It seems specifically critical to me as the quantities here are relatively small. For instance, between (94) and (95), it is stated that gi2≈0g_i^2 \approx 0, and after (95) it is argued that "gi2≫ε{g_i}^{2} \gg \varepsilon holds most of the time." I note that the latter statement is justified by "During the training of large-scale neural networks, we generally do not seek to converge to a strict minima, so for the optimizers, the gradient is usually not zero". Even if the gradient is not zero, why couldnt we have many small coordinates ? (and I think the standard definition of "non-strict minima" also implies that the gradient is zero). Also, the justification to get rid of P4P_4 on page 30 "the sign of gig_i can be regarded as completely random and cannot form a stable implicit bias", seems very light to me. I also note that according to (96), a factor "sgn(u)" is missing in the sum P4P_4. Overall I believe it is a major concern, as this "implicit bias of ADAM" result would certainly be the most original and significant of the paper.

Questions for authors

Please see my remarks in the weakness section.

Reviewer_oomuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper develops a framework for approximating discrete optimization algorithms with continuous-time dynamics to arbitrary order in the learning rate. It uses segmented expansions and a decomposition of optimizer updates to handle momentum and adaptive methods. Adam is the main application, the authors derive a proposed third-order approximation and interpret its additional terms as implicit regularization. They argue that Adam changes behavior as squared gradient components become small relative to its stabilization constant, moving from a preference for sharper regions toward a flatter-region preference resembling momentum. Toy examples and MNIST experiments illustrate this proposed transition.

Critical strengths
  1. A systematic framework covering gradient descent, momentum, and Adam would be valuable. The recursive construction gives the paper a coherent methodological direction.
  2. Going beyond the second-order analysis of Cattaneo et al. (2024) is a meaningful objective, especially if higher-order terms reveal behavior that lower-order approximations miss.
  3. The proposed regime transition is concrete enough to test. Tracking gradient size, curvature-related quantities, and generalization together is a useful experimental starting point.
Critical weaknesses
  1. The Adam derivation replaces moving averages with mk≈gm_k\approx g and vk≈g2v_k\approx g^2, but does not adequately bound the resulting errors. These approximations need quantitative control before they can support an O(η3)O(\eta^3) claim.
  2. The implicit-bias argument makes additional unsupported simplifications. In particular, Appendix B.1 drops P4P_4 because gradient signs are considered random, without establishing cancellation or showing that this same-order term is negligible.
  3. The numerical validation uses first-order Euler integration with step η/10\eta/10. Its integration error can obscure the claimed approximation order, a more accurate solver and explicit error-scaling tests are needed.
Questions for authors

How are the errors introduced by replacing Adam’s moment estimates with instantaneous gradients and squared gradients controlled tightly enough to preserve the claimed third-order approximation accuracy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.