Stability and Generalization of Noisy signSGD: From Sign Discontinuity to Gaussian Smoothing

Efficient and scalable stochastic optimization algorithms have made the training of large foundation models possible. More recently, sign-based optimizers such as signSGD have attracted growing attention for their low storage and communication costs, as they update models using only 1-bit gradient signs. Although some studies have begun to investigate the convergence properties of signSGD, its generalization properties remain poorly understood. We establish a lower bound on the generalization error of signSGD. For Lipschitz-continuous and smooth objectives, this lower bound is $\Omega (L\eta \sqrt{d}/n)$ in convex, nonconvex, and strongly convex settings. Establishing a meaningful upper bound is more challenging: for gradient coordinates near zero, even arbitrarily small perturbations can flip their signs and cause non-negligible changes in the updates. To address this difficulty, we interpret Gaussian signSGD as a smoothed surrogate for signSGD. Adding Gaussian noise to the gradients before applying the sign operator smooths the resulting update distribution and enables us to derive a noise-dependent upper bound on the generalization error. Through this Gaussian-smoothed counterpart, the resulting bound provides an indirect upper-bound characterization of signSGD. Its explicit dependence on the noise scale further allows us to quantify how Gaussian perturbations affect generalization. We also conduct experiments on synthetic data, full-parameter fine-tuning of large language models, and computer vision tasks. The results across different models and tasks support our theoretical findings.