- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the cumulative write precision required by bias-corrected Adam under an additive uniform-dither model. For a scalar quadratic and sufficiently long prescribed horizons, it establishes matching bounds of over causal precision policies. A deterministic schedule attains this order by increasing parameter, gradient, and first-moment precision near the end while keeping second-moment precision fixed.
Critical strengths
- The constructive schedule illustrates how temporal precision allocation can reduce cumulative write cost under the stated assumptions.
Critical weaknesses
Insufficient justification of the central problem formulation
- The paper motivates its study through low-precision training, but cumulative write precision is not demonstrated to predict peak memory, actual memory traffic, or training time. Computation costs are excluded, and shared dither, formats, and width information are treated as free. Moreover, the experiments use NumPy float64 with injected noise rather than actual changing numerical formats.
- They illustrate the assumed model’s behavior but do not validate its relevance to practical low-precision computation. Acknowledging these omissions does not resolve this central concern.
Restrictive theoretical scope and unclear implications for neural-network training.
- The analysis assume quadratics functions (extend to strongly convex smooth functions with some extra requirements);
- These restrictions leave unclear whether the temporal schedule remains useful for coupled neural-network parameters.
In summary, the paper presents theoretical results under highly simplified settings, without sufficient justification of the underlying computational cost model. The empirical evaluation merely reproduces the assumed numerical model rather than testing its practical implications. Overall, I find the limited theoretical scope and insufficient empirical validation to be major weaknesses, particularly given the paper's motivation from efficient low-precision training.
Questions for authors
- Do the claimed savings persist when actual numerical formats and precision-change costs are used?
- Can the proposed schedule be evaluated on a neural network with fixed forward/backward computational precision and varying precision for the stored quantities modeled in the paper