Stability and Generalization of Noisy signSGD: From Sign Discontinuity to Gaussian Smoothing
Withdrawn from ICLR 2027, so left out of the paper lists and the map.
Abstract
Efficient and scalable stochastic optimization algorithms have made the training of large foundation models possible. More recently, sign-based optimizers such as signSGD have attracted growing attention for their low storage and communication costs, as they update models using only 1-bit gradient signs. Although some studies have begun to investigate the convergence properties of signSGD, its generalization properties remain poorly understood. We establish a lower bound on the generalization error of signSGD. For Lipschitz-continuous and smooth objectives, this lower bound is in convex, nonconvex, and strongly convex settings. Establishing a meaningful upper bound is more challenging: for gradient coordinates near zero, even arbitrarily small perturbations can flip their signs and cause non-negligible changes in the updates. To address this difficulty, we interpret Gaussian signSGD as a smoothed surrogate for signSGD. Adding Gaussian noise to the gradients before applying the sign operator smooths the resulting update distribution and enables us to derive a noise-dependent upper bound on the generalization error. Through this Gaussian-smoothed counterpart, the resulting bound provides an indirect upper-bound characterization of signSGD. Its explicit dependence on the noise scale further allows us to quantify how Gaussian perturbations affect generalization. We also conduct experiments on synthetic data, full-parameter fine-tuning of large language models, and computer vision tasks. The results across different models and tasks support our theoretical findings.
- Primary area:
- learning theory
- Keywords:
- Learning Theory, SignSGD, Generalization Error, Algorithm Stability