SUN: Stochastic Unbiased Momentum via Filter Design

Momentum is a central component of deep learning optimizers, yet its gradient-tracking behavior in practical training remains incompletely understood. Naturally, averaging past stochastic gradients suppresses noise, but the temporal center of mass of this weighted average lies in past iterations, causing the estimate to lag behind the current iteration. We interpret this source of gradient-tracking bias through the group delay of a discrete-time filter and analyze the tradeoff between delay and noise attenuation using the filter's frequency response. Building on this interpretation, we propose a filter-design approach that uses amplitude response and group delay to guide the construction and tuning of momentum estimators. We instantiate this approach with SUN, a family of momentum filters that apply filtered gradient lookahead to reduce group delay in selected frequency bands while retaining noise attenuation. This design enables targeted phase compensation beyond the uniform lookahead used in existing methods. Experiments on language model pre-training indicate that the resulting momentum estimators improve training performance, supporting filter design as a practical approach to momentum-based optimization.