acceptodds
Under review as a conference paper at ICLR 2027

The Precision Complexity of Dithered Adam

Abstract

Low-precision Adam can reduce the memory used by optimizer states. We study the cumulative precision needed to reach a target accuracy when write widths can change over time and respond to observed numerical errors. We first consider bias-corrected Adam on the scalar quadratic objective, with fresh uniform additive dither in the parameter, gradient, and two moment writes. We also extend the complexity bounds to scalar strongly convex smooth objectives and higher-dimensional separable objectives. A causal policy chooses each width using the revealed history, and we measure cost by the sum of all widths used during the run. Under an explicit step-size condition, we establish matching lower and upper bounds of order on the expected cumulative precision required to achieve . These bounds hold for every sufficiently small and every deterministic horizon above a threshold of order . The lower bound applies to all causal policies using the full error history, while a deterministic schedule attains the upper bound by keeping second-moment precision fixed and raising the other three widths near the end of the run. For the same target accuracy and horizon, keeping all four write widths fixed throughout the run gives a minimum cost of . Thus, for , temporal precision allocation saves a factor of order . If the deterministic horizon can also be chosen, a stronger step-size condition gives an optimal cost of . Without this additional condition, suitable parameters and initialization can permit finite-time cancellation, reducing the cost to .

Primary area:
optimization
Keywords:
Adam, adaptive optimization, quantization, numerical precision, precision complexity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_F7CoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the cumulative write precision required by bias-corrected Adam under an additive uniform-dither model. For a scalar quadratic and sufficiently long prescribed horizons, it establishes matching bounds of Θ(T+log⁡2(1/ϵ))\Theta(T+\log^2(1/\epsilon)) over causal precision policies. A deterministic schedule attains this order by increasing parameter, gradient, and first-moment precision near the end while keeping second-moment precision fixed.

Critical strengths
  • The constructive schedule illustrates how temporal precision allocation can reduce cumulative write cost under the stated assumptions.
Critical weaknesses

Insufficient justification of the central problem formulation

  • The paper motivates its study through low-precision training, but cumulative write precision is not demonstrated to predict peak memory, actual memory traffic, or training time. Computation costs are excluded, and shared dither, formats, and width information are treated as free. Moreover, the experiments use NumPy float64 with injected noise rather than actual changing numerical formats.
  • They illustrate the assumed model’s behavior but do not validate its relevance to practical low-precision computation. Acknowledging these omissions does not resolve this central concern.

Restrictive theoretical scope and unclear implications for neural-network training.

  • The analysis assume quadratics functions (extend to strongly convex smooth functions with some extra requirements);
  • These restrictions leave unclear whether the temporal schedule remains useful for coupled neural-network parameters.

In summary, the paper presents theoretical results under highly simplified settings, without sufficient justification of the underlying computational cost model. The empirical evaluation merely reproduces the assumed numerical model rather than testing its practical implications. Overall, I find the limited theoretical scope and insufficient empirical validation to be major weaknesses, particularly given the paper's motivation from efficient low-precision training.

Questions for authors
  • Do the claimed savings persist when actual numerical formats and precision-change costs are used?
  • Can the proposed schedule be evaluated on a neural network with fixed forward/backward computational precision and varying precision for the stored quantities modeled in the paper
Reviewer_NzSbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that dynamically allocating numerical precision during Adam optimization, by increasing the precision of selected variables near the end of the run, can achieve the target accuracy while reducing cumulative precision cost by a logarithmic factor compared with fixed-precision methods.

Critical strengths

A precise formulation of a novel problem: the paper formulates and studies the optimization of the cumulative precision cost across the four types of writes

Critical weaknesses

Simplified Optimization Setting. The main analysis focuses on a scalar quadratic objective without stochastic gradient noise. Although this setting facilitates the theoretical analysis, it remains unclear whether the results extend to more general objectives or settings with stochastic gradients, which limits the direct applicability of the findings to practical Adam implementations

Simplified model of numerical errors. The analysis models write errors as additive random variables with uniform distributions, with their magnitudes controlled by the chosen precision widths. This model differs from the rounding errors induced by standard floating-point formats, which are generally value-dependent and constrained by the representable numbers. It would be helpful if the authors clarified how representative this error model is of practical low-precision Adam implementations and discussed whether the main theoretical conclusions are expected to extend to standard floating-point arithmetic.

Limitations of the Bit-Cost Model. The bit-cost interpretation assumes that the dither, quantization formats, and selected write widths are shared between the encoder and decoder at no cost. While this assumption may be reasonable for the proposed predetermined schedule, adaptive policies may require additional information to communicate their precision choices. Could the authors clarify how this free-information assumption affects the interpretation of the lower bound and whether the main conclusions remain meaningful when the overhead of communicating adaptive precision choices is accounted for?

Practical Significance and Scope of the Experimental Evaluation. The theoretical results establish an asymptotic reduction in cumulative precision cost through temporal precision allocation. However, the experiments evaluate this benefit only in the scalar dithered oracle and compare empirically selected schedules with a finite set of fixed-width baselines. As the authors acknowledge, these experiments do not measure actual memory savings, runtime improvements, or training performance on general optimization problems. Moreover, the empirically selected ramp parameters do not necessarily coincide with those in the theoretically guaranteed schedule. A clearer discussion of how the proposed precision-cost reduction translates into practical benefits in low-precision Adam implementations, and of the extent to which the scalar experiments support this interpretation, would strengthen the paper.

Questions for authors

Theorem 1 establishes the precision-complexity bound for a prescribed horizon under Condition (7), whereas Corollary 2 requires the stronger Condition (9) to establish the optimal precision complexity over all deterministic horizons. Could the authors clarify why the stronger step-size restriction is needed when optimizing over the horizon, and whether it arises from the proof technique or reflects a fundamental difference between the two settings? In particular, outside Condition (9), is the Θ(log⁡2(1/ε))\Theta(\log^2(1/\varepsilon)) complexity guarantee merely unestablished, or can the optimal complexity be lower? Appendix A.4 exhibits a one-step cancellation example with optimal complexity Θ(log⁡(1/ε))\Theta(\log(1/\varepsilon)); it would be helpful to clarify whether this behavior extends to a broader range of parameter settings or depends on specific choices of the initialization and step size.

Reviewer_NaxKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work derives the minimal cumulative numerical precision for Adam to reach a target accuracy, considering time-varying, history-dependent precision for parameter, gradient, first-moment, and second-moment iterates. For scalar quadratic objectives with independent uniform additive dither, the authors give matching bounds under certain conditions, and they are compared favorably to those under fixed uniform precision over a long horizon. I found the main contribution to be a tight lower bound for all precision policies that are adaptive to history. I like the result for its theoretical contributions, but its practical implications are nevertheless narrower than the introduction of the paper claims.

Critical strengths
  • The primary strength is a novel lower bound, which can be applied to any causal precision policy given the full error history. As I mentioned above, it is stronger than optimality over fixed schedules. The parabolic maximal theorem itself is not new, but applying it to prove lower bounds for adaptive precision is new.

  • The authors proved matching bounds. It is also insightful to observe that the second moment's precision can stay fixed unless the other iterates, whose precisions require going higher later.

Critical weaknesses
  • Well, the refinement from Θ(TL)\Theta(TL) to Θ(T+L2)\Theta(T+L^2) with L=log⁡(1/ε)L=\log(1/\varepsilon) only matters when T≫LT \gg L. If T=Θ(L)T = \Theta(L), both become Θ(L2)\Theta(L^2), so fixed precision can also achieve the same as the adaptive one when the horizon TT is on the order of L=log⁡(1/ε)L=\log(1/\varepsilon).

  • Some conditions may limit practical applicability. They include independent uniform additive dither, exact arithmetic between noisy writes, and no stochastic gradient noise. I wonder if the authors could test whether the core phenomena match realistic quantization and standard optimizer hyperparameters.

  • I wonder if the authors could relax the independent (i.e. scalar) per-coordinate restriction.

Overall, the main issue is that the claimed gain exists only under long horizons and a simplified noise model. Its contribution to real AI training appears to be limited.

Questions for authors

Please address the weaknesses I mentioned above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.