- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies momentum through a signal processing perspective, motivated by the trade-off between noise suppression and the lag introduced by averaging past stochastic gradients. It formulates momentum as a temporal filter and uses amplitude response and group delay to characterize noise attenuation and tracking delay. Based on this analysis, it proposes SUN, which applies filtered gradient difference lookahead to selectively reduce delay across frequency bands, with SUN-Bandpass instantiated through frequency response constrained filter design.
Critical strengths
Clear intuition. The filter view of momentum is easy to follow. Using amplitude response and group delay gives a simple way to understand the tradeoff between noise reduction and delay.
Experiments. The paper tests the method on language model pretraining and compares it with several momentum baselines. The results show small improvements over the baselines in the tested setting.
Critical weaknesses
- Limited novelty. The problem studied in this paper is the trade off between noise suppression and gradient tracking delay: averaging historical gradients suppresses stochastic noise, but shifts the estimate toward past iterations and makes it lag behind the current gradient. [2] essentially studies the same underlying trade off from a statistical estimation perspective, directly characterizing gradient estimation bias and variance (with explicit asymptotic results for both) and deriving an online filtering method from this analysis. Compared with [2], this paper mainly characterizes the two sides through group delay and frequency response, but it is unclear what substantially new understanding this reformulation provides. [1] is also closely related and already analyzes momentum in the frequency domain and designs momentum filters based on frequency response.
- Method overlap. Most computations in SUN already appear in prior optimizers. SUN computes gradient differences, applies EMA filtering, and combines the filtered correction with standard momentum. A similar computation appears in PID optimizer [3]; gradient difference corrections are central to MARS [4]; multi timescale EMA states are used in AdEMAMix [5]. SUN-Bandpass combines two gradient difference branches with different decay rates and selects coefficients using the proposed frequency response objective. So the remaining novelty is mostly in the parameterization and coefficient selection, rather than a fundamentally new momentum update.
- Limited experiments. The empirical evaluation is quite narrow, with essentially one 124M NanoGPT setup on FineWeb. I also did not find an ablation that separates the benefit of the proposed group delay based design from the additional filter states and parameterization used by SUN. This makes it difficult to tell whether the gain actually comes from the claimed mechanism.
- Missing literature. The related work discussion is incomplete. Several methods are already very close to SUN in both motivation and update rule, especially [2] and [3], but they are not discussed. This makes it hard to assess the claimed contribution against prior work.
References
[1] Xianliang Li, Jun Luo, Zhiwei Zheng, Hanxiao Wang, Li Luo, Lingkun Wen, Linlong Wu, and Sheng Xu. On the Performance Analysis of Momentum Method: A Frequency Domain Perspective. ICLR, 2025.
[2] Zhipeng Yao, Rui Yu, Guisong Chang, Ying Li, Yu Zhang, and Dazhou Li. Dynamic Momentum Recalibration in Online Gradient Learning. CVPR, 2026.
[3] Wangpeng An, Haoqian Wang, Qingyun Sun, Jun Xu, Qionghai Dai, and Lei Zhang. A PID Controller Approach for Stochastic Optimization of Deep Networks. CVPR, 2018.
[4] Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. MARS: Unleashing the Power of Variance Reduction for Training Large Models. ICML, 2025.
[5] Matteo Pagliardini, Pierre Ablin, and David Grangier. The AdEMAMix Optimizer: Better, Faster, Older. ICLR, 2025.
Questions for authors
- How are the frequency ranges in Eqs. (12)–(14) chosen? Is there any evidence from actual language model training that useful gradient changes and stochastic noise are separated in frequency in the way assumed in Sec. 4.2?
- For the results in Fig. 6 and Table 1, were the momentum hyperparameters of the baselines tuned, or only the learning rate? SUN itself involves tuning the filter constraints and coefficients, so I am concerned about whether the comparison uses a similar tuning budget.
- Li et al. (2025) already uses frequency domain analysis to study and design momentum filters. What is new in SUN beyond adding group delay to the design objective and filtering the gradient difference?
6 comments
- Limited experiments. The empirical evaluation is quite narrow, with essentially one 124M NanoGPT setup on FineWeb. I also did not find an ablation that separates the benefit of the proposed group delay based design from the additional filter states and parameterization used by SUN. This makes it difficult to tell whether the gain actually comes from the claimed mechanism.
To provide more evidence on the correlation of group delay and training performance, in Appendix C.2 of our revised version, we performed an additional ablation study on comparing filters of the same gain response under the same number of state variables and number of coefficients, but with different group delay response by maximizing the optimization formulation in Eq. (12) - (14). The experiment result shows that filter with higher delay performed worse than filter with lower delay. We have further proposed a new metric of normalized delay in Section 4.5, Eq. (21) which shows clear correlation with the training performance, as illustrated in Table 1 and Figure 8.
- Missing literature. The related work discussion is incomplete. Several methods are already very close to SUN in both motivation and update rule, especially [2] and [3], but they are not discussed. This makes it hard to assess the claimed contribution against prior work.
We thank the reviewer for providing the additional reference that we missed in the current draft. We remark that the PID optimizer [3] can be reduced to MARS as discussed above, and for which we have analyzed MARS in detail by Section 3 and 4. On the other hand, please refer to the above Comparison with [2] to understand our contribution upon [2]. We remark that the motivation of existing works is very different from ours, as they do not consider the group delay analysis of filters.
- How are the frequency ranges in Eqs. (12)–(14) chosen? Is there any evidence from actual language model training that useful gradient changes and stochastic noise are separated in frequency in the way assumed in Sec. 4.2?
The frequency ranges are chosen to attain similar gain response covered by the baseline filters (e.g., small high-frequency gain for ), e.g., see Figure 3 where our SUN-Bandpass filter has an amplitude gain that sits in-between the baseline filters in this frequency band.
In Appendix C.1 of our revised version, we have added an additional experiment on a smaller scale setting, where we compute the full gradient during the middle phase of training and compute the gradient frequency spectrum. The spectrum in Fig. 7 shows that the full gradient has very weak high-frequency signal, indicating that the high frequency spectrum is dominated by gradient error.
- For the results in Fig. 6 and Table 1, were the momentum hyperparameters of the baselines tuned, or only the learning rate? SUN itself involves tuning the filter constraints and coefficients, so I am concerned about whether the comparison uses a similar tuning budget.
Besides for EMAMIX which we tuned the additional momentum coefficient and momentum scaling as the default suggested value performs poorly (see the additional hyperparameter tuning for EMAMIX in Appendix C.4), all other baseline algorithms use the default momentum coefficients. This also reveals the fundamental difference between our approach and existing baselines: the coefficients of prior work algorithms might require expensive experiment tuning to further optimize their performance, while the coefficients of SUN momentum algorithms are optimized using the filter delay minimization formulation and does not depend on expensive training experiments for tuning. Therefore, the coefficient selection process of SUN is much more efficient and practical.
- Li et al. (2025) already uses frequency domain analysis to study and design momentum filters. What is new in SUN beyond adding group delay to the design objective and filtering the gradient difference?
Please refer to the above Comparison with [1] for our separation with Li et al. (2025), and in particular that Li et al. (2025) only analyzed the classical momentum algorithm. To summarize, in addition to group delay optimization formulation and the SUN filter families construction, our work gives a clear insight about momentum algorithm design ---- that group delay is a useful design criterion that is highly correlated with training performance (e.g., see the metric of normalized delay in Table 1). This insight provides practical guideline for future works to use/design better momentum algorithms, as group delay appears in all filtering systems. The name of our paper (Stochastic Unbiased Momentum) highlights the goal of achieving unbiased gradient estimate through filtering stochastic gradient with a low delay filter.
Response to the Authors
I appreciate the detailed rebuttal from the authors. I summarize my assessment below.
1. Limited Novelty.
I discuss this in two parts.
(a) Problem Analysis
Prior work and SUN reach similar conclusions:
| Paper | Main Conclusions |
|---|---|
| Li et al. [1] | Momentum acts as a frequency-domain filter. Suppressing high frequencies and progressively amplifying low frequencies can improve training. |
| Yao et al. [2] | Gradient averaging reduces variance but introduces estimation bias. |
| Sahu et al. [3] | Momentum causes tracking lag and can degrade performance under nonstationarity. |
| SUN | Gradient averaging suppresses noise but introduces group delay. |
Li et al. [1], Appendix A, Eq. (11) derive the EMA phase response:
SUN, Sections 3–4.1, defines . Differentiating Li et al.'s result directly yields SUN's formula:
Yao et al. [2], Theorem 2.3, Eq. (3) give the EMA estimation-bias bound:
Its dominant dependence on matches SUN's zero-frequency delay:
Sahu et al. [3], Theorems 3.3 and 3.7 establish momentum-dependent tracking-error bounds and an unavoidable inertia window of order .
Conclusion: SUN's EMA group-delay formula follows directly from Li et al. [1], while Yao et al. [2] and Sahu et al. [3] already establish the underlying noise–lag trade-off. SUN mainly reformulates known results using different terminology.
(b) Method Design
SUN's design criterion has some novelty, but its implementation adds hyperparameters and memory overhead for limited gains.
| Method | Design | Hyperparameters | Persistent States |
|---|---|---|---|
| Momentum | EMA | 1 | 1 |
| Adam | First/second moments | 2 | 2 |
| FSGDM [1] | Momentum scheduling | 3 | 1 |
| SGDF [2] | Adaptive filtering | 2 | 2 |
| SUN-Bandpass | EMA + gradient differences | 5 | 3 |
| SUN-HighOrder | Higher-order EMA | Up to 37 | 18 |
Counts exclude learning rate and weight decay.
Then what is the point of proposing SUN?
2. Limited Experiments
- The experiments are largely limited to NanoGPT2-124M on FineWeb. More comparisons with other optimizers across different architectures and datasets are needed to validate SUN's effectiveness.
- I noticed the additional experiments in Appendix C, but the results in Fig. 11 are not promising. Could the authors do a quick experiment combining SUN with Adam, SignSGD, and Muon using ResNet-34 on CIFAR-100, and compare the results with SGDF [2] and FSGDM [1]?
3. The Meaning of "Unbiased".
The authors state:
The name of our paper (Stochastic Unbiased Momentum) highlights the goal of achieving unbiased gradient estimate through filtering stochastic gradient with a low delay filter.
I am concerned about what the authors mean by an "unbiased gradient estimate."
- GD: Training on the entire dataset gives the exact full gradient, without sampling bias.
- SGD: Randomly sampled gradients are unbiased but have variance. Importantly, unbiasedness is defined at the same model parameters.
- Gradient accumulation: Gradients are accumulated at the same model parameters. For networks with LayerNorm, this is equivalent to computing the full-batch gradient when all samples are included, assuming no other batch-dependent operations.
However, SUN averages and filters gradients evaluated at different model parameters across iterations. Reducing group delay does not imply that the resulting gradient estimator is statistically unbiased with respect to the current true gradient. What is the precise definition of "unbiased" in this paper?
References
[1] Xianliang Li, Jun Luo, Zhiwei Zheng, Hanxiao Wang, Li Luo, Lingkun Wen, Linlong Wu, and Sheng Xu. On the Performance Analysis of Momentum Method: A Frequency Domain Perspective. ICLR, 2025.
[2] Zhipeng Yao, Rui Yu, Guisong Chang, Ying Li, Yu Zhang, and Dazhou Li. Dynamic Momentum Recalibration in Online Gradient Learning. CVPR, 2026.
[3] Sharan Sahu, Cameron J. Hogan, and Martin T. Wells. On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization. ICML, 2026.
Thank you for the thoughtful review. We will address each point of your concern by the following response.
We highlight the contribution of our work by the following direct comparison with the concerned existing works.
Comparison with [2]. The bias and variance decomposition in [2] is an upper bound analysis to explain the factors that control the gradient error of the classical momentum algorithm. Meanwhile, our formulation provides a new understanding that frequency analysis of momentum filters decomposes gradient tracking error along the frequency spectrum. The existing analysis in [2] cannot capture the fact that gradient bias differs for gradient signal in different frequency bands, while our group delay analysis in Figure 3 captures the different amount of gradient bias due to signal delay over the frequency spectrum. On the other hand, our analysis holds broadly for the family of linear time-invariant momentum algorithm, whereas the analysis in [2] is limited to the classical momentum algorithms and their proposed algorithm. In our revised version, we have further quantified the amount of delay across the frequency spectrum using a metric of normalized delay as defined in Eq. (21), where it predicts the training performance as shown in Table 1. This metric is possible only by our frequency analysis of group delay and gain response, and cannot be seen from the results of [2].
Comparison with [1]. [1] focuses on analysing the classical momentum algorithm with time-varying coefficients and tackles the problem of choosing the optimal coefficient schedule, without considering the effect of group delay. Meanwhile, our work focuses on analyzing general time-invariant momentum algorithms and minimizes the absolute group delay. Therefore, our paper and [1] are addressing different problems, despite using the same language of frequency response which is a common tool for a lot of research.
Discussion on Related Works. We apologize for missing the PID optimizer reference in our submission. Meanwhile, through a few steps of derivation to identify the corresponding gradient filter of PID, we notice that the PID optimizer can be reduced to MARS with a change in choosing γ<0 in Eq. (2). We have provided the proof in Appendix F of our revised version to clarify accordingly.
Discussion on Novelty. We emphasize that our novelty lies in identifying the role of signal delay in optimization algorithms, which none of the above works have analyzed. Our interpretation provides new explanation on the purpose of these existing algorithmic components, which makes it unfair to say that we have limited novelty when we have studied these existing components for drawing connection to the existing literature. The central philosophy of our paper is a principled formulation for finding filters that minimize its absolute group delay through an optimization problem in Eq. (12) - (14), which replaces the heuristic coefficient selection in existing works (e.g., γ in MARS and α in AdEMAMIX).