acceptodds
Under review as a conference paper at ICLR 2027

SUN: Stochastic Unbiased Momentum via Filter Design

Abstract

Momentum is a central component of deep learning optimizers, yet its gradient-tracking behavior in practical training remains incompletely understood. Naturally, averaging past stochastic gradients suppresses noise, but the temporal center of mass of this weighted average lies in past iterations, causing the estimate to lag behind the current iteration. We interpret this source of gradient-tracking bias through the group delay of a discrete-time filter and analyze the tradeoff between delay and noise attenuation using the filter's frequency response. Building on this interpretation, we propose a filter-design approach that uses amplitude response and group delay to guide the construction and tuning of momentum estimators. We instantiate this approach with SUN, a family of momentum filters that apply filtered gradient lookahead to reduce group delay in selected frequency bands while retaining noise attenuation. This design enables targeted phase compensation beyond the uniform lookahead used in existing methods. Experiments on language model pre-training indicate that the resulting momentum estimators improve training performance, supporting filter design as a practical approach to momentum-based optimization.

Primary area:
optimization
Keywords:
Momentum optimization, deep learning optimization, filter design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wbKGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies momentum through a signal processing perspective, motivated by the trade-off between noise suppression and the lag introduced by averaging past stochastic gradients. It formulates momentum as a temporal filter and uses amplitude response and group delay to characterize noise attenuation and tracking delay. Based on this analysis, it proposes SUN, which applies filtered gradient difference lookahead to selectively reduce delay across frequency bands, with SUN-Bandpass instantiated through frequency response constrained filter design.

Critical strengths

Clear intuition. The filter view of momentum is easy to follow. Using amplitude response and group delay gives a simple way to understand the tradeoff between noise reduction and delay.

Experiments. The paper tests the method on language model pretraining and compares it with several momentum baselines. The results show small improvements over the baselines in the tested setting.

Critical weaknesses
  1. Limited novelty. The problem studied in this paper is the trade off between noise suppression and gradient tracking delay: averaging historical gradients suppresses stochastic noise, but shifts the estimate toward past iterations and makes it lag behind the current gradient. [2] essentially studies the same underlying trade off from a statistical estimation perspective, directly characterizing gradient estimation bias and variance (with explicit asymptotic results for both) and deriving an online filtering method from this analysis. Compared with [2], this paper mainly characterizes the two sides through group delay and frequency response, but it is unclear what substantially new understanding this reformulation provides. [1] is also closely related and already analyzes momentum in the frequency domain and designs momentum filters based on frequency response.
  2. Method overlap. Most computations in SUN already appear in prior optimizers. SUN computes gradient differences, applies EMA filtering, and combines the filtered correction with standard momentum. A similar computation appears in PID optimizer [3]; gradient difference corrections are central to MARS [4]; multi timescale EMA states are used in AdEMAMix [5]. SUN-Bandpass combines two gradient difference branches with different decay rates and selects coefficients using the proposed frequency response objective. So the remaining novelty is mostly in the parameterization and coefficient selection, rather than a fundamentally new momentum update.
  3. Limited experiments. The empirical evaluation is quite narrow, with essentially one 124M NanoGPT setup on FineWeb. I also did not find an ablation that separates the benefit of the proposed group delay based design from the additional filter states and parameterization used by SUN. This makes it difficult to tell whether the gain actually comes from the claimed mechanism.
  4. Missing literature. The related work discussion is incomplete. Several methods are already very close to SUN in both motivation and update rule, especially [2] and [3], but they are not discussed. This makes it hard to assess the claimed contribution against prior work.

References

[1] Xianliang Li, Jun Luo, Zhiwei Zheng, Hanxiao Wang, Li Luo, Lingkun Wen, Linlong Wu, and Sheng Xu. On the Performance Analysis of Momentum Method: A Frequency Domain Perspective. ICLR, 2025.
[2] Zhipeng Yao, Rui Yu, Guisong Chang, Ying Li, Yu Zhang, and Dazhou Li. Dynamic Momentum Recalibration in Online Gradient Learning. CVPR, 2026.
[3] Wangpeng An, Haoqian Wang, Qingyun Sun, Jun Xu, Qionghai Dai, and Lei Zhang. A PID Controller Approach for Stochastic Optimization of Deep Networks. CVPR, 2018.
[4] Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. MARS: Unleashing the Power of Variance Reduction for Training Large Models. ICML, 2025.
[5] Matteo Pagliardini, Pierre Ablin, and David Grangier. The AdEMAMix Optimizer: Better, Faster, Older. ICLR, 2025.

Questions for authors
  1. How are the frequency ranges in Eqs. (12)–(14) chosen? Is there any evidence from actual language model training that useful gradient changes and stochastic noise are separated in frequency in the way assumed in Sec. 4.2?
  2. For the results in Fig. 6 and Table 1, were the momentum hyperparameters of the baselines tuned, or only the learning rate? SUN itself involves tuning the filter constraints and coefficients, so I am concerned about whether the comparison uses a similar tuning budget.
  3. Li et al. (2025) already uses frequency domain analysis to study and design momentum filters. What is new in SUN beyond adding group delay to the design objective and filtering the gradient difference?
6 comments

Thank you for the thoughtful review. We will address each point of your concern by the following response.

  1. Limited novelty. The problem studied in this paper is the trade off between noise suppression and gradient tracking delay: averaging historical gradients suppresses stochastic noise, but shifts the estimate toward past iterations and makes it lag behind the current gradient. [2] essentially studies the same underlying trade off from a statistical estimation perspective, directly characterizing gradient estimation bias and variance (with explicit asymptotic results for both) and deriving an online filtering method from this analysis. Compared with [2], this paper mainly characterizes the two sides through group delay and frequency response, but it is unclear what substantially new understanding this reformulation provides. [1] is also closely related and already analyzes momentum in the frequency domain and designs momentum filters based on frequency response.

We highlight the contribution of our work by the following direct comparison with the concerned existing works.

  • Comparison with [2]. The bias and variance decomposition in [2] is an upper bound analysis to explain the factors that control the gradient error of the classical momentum algorithm. Meanwhile, our formulation provides a new understanding that frequency analysis of momentum filters decomposes gradient tracking error along the frequency spectrum. The existing analysis in [2] cannot capture the fact that gradient bias differs for gradient signal in different frequency bands, while our group delay analysis in Figure 3 captures the different amount of gradient bias due to signal delay over the frequency spectrum. On the other hand, our analysis holds broadly for the family of linear time-invariant momentum algorithm, whereas the analysis in [2] is limited to the classical momentum algorithms and their proposed algorithm. In our revised version, we have further quantified the amount of delay across the frequency spectrum using a metric of normalized delay as defined in Eq. (21), where it predicts the training performance as shown in Table 1. This metric is possible only by our frequency analysis of group delay and gain response, and cannot be seen from the results of [2].

  • Comparison with [1]. [1] focuses on analysing the classical momentum algorithm with time-varying coefficients and tackles the problem of choosing the optimal coefficient schedule, without considering the effect of group delay. Meanwhile, our work focuses on analyzing general time-invariant momentum algorithms and minimizes the absolute group delay. Therefore, our paper and [1] are addressing different problems, despite using the same language of frequency response which is a common tool for a lot of research.

  1. Method overlap. Most computations in SUN already appear in prior optimizers. SUN computes gradient differences, applies EMA filtering, and combines the filtered correction with standard momentum. A similar computation appears in PID optimizer [3]; gradient difference corrections are central to MARS [4]; multi timescale EMA states are used in AdEMAMix [5]. SUN-Bandpass combines two gradient difference branches with different decay rates and selects coefficients using the proposed frequency response objective. So the remaining novelty is mostly in the parameterization and coefficient selection, rather than a fundamentally new momentum update.
  • Discussion on Related Works. We apologize for missing the PID optimizer reference in our submission. Meanwhile, through a few steps of derivation to identify the corresponding gradient filter of PID, we notice that the PID optimizer can be reduced to MARS with a change in choosing γ<0\gamma < 0 in Eq. (2). We have provided the proof in Appendix F of our revised version to clarify accordingly.

  • Discussion on Novelty. We emphasize that our novelty lies in identifying the role of signal delay in optimization algorithms, which none of the above works have analyzed. Our interpretation provides new explanation on the purpose of these existing algorithmic components, which makes it unfair to say that we have limited novelty when we have studied these existing components for drawing connection to the existing literature. The central philosophy of our paper is a principled formulation for finding filters that minimize its absolute group delay through an optimization problem in Eq. (12) - (14), which replaces the heuristic coefficient selection in existing works (e.g., γ\gamma in MARS and α\alpha in AdEMAMIX).

  1. Limited experiments. The empirical evaluation is quite narrow, with essentially one 124M NanoGPT setup on FineWeb. I also did not find an ablation that separates the benefit of the proposed group delay based design from the additional filter states and parameterization used by SUN. This makes it difficult to tell whether the gain actually comes from the claimed mechanism.

To provide more evidence on the correlation of group delay and training performance, in Appendix C.2 of our revised version, we performed an additional ablation study on comparing filters of the same gain response under the same number of state variables and number of coefficients, but with different group delay response by maximizing the optimization formulation in Eq. (12) - (14). The experiment result shows that filter with higher delay performed worse than filter with lower delay. We have further proposed a new metric of normalized delay in Section 4.5, Eq. (21) which shows clear correlation with the training performance, as illustrated in Table 1 and Figure 8.

  1. Missing literature. The related work discussion is incomplete. Several methods are already very close to SUN in both motivation and update rule, especially [2] and [3], but they are not discussed. This makes it hard to assess the claimed contribution against prior work.

We thank the reviewer for providing the additional reference that we missed in the current draft. We remark that the PID optimizer [3] can be reduced to MARS as discussed above, and for which we have analyzed MARS in detail by Section 3 and 4. On the other hand, please refer to the above Comparison with [2] to understand our contribution upon [2]. We remark that the motivation of existing works is very different from ours, as they do not consider the group delay analysis of filters.

  1. How are the frequency ranges in Eqs. (12)–(14) chosen? Is there any evidence from actual language model training that useful gradient changes and stochastic noise are separated in frequency in the way assumed in Sec. 4.2?

The frequency ranges are chosen to attain similar gain response covered by the baseline filters (e.g., small high-frequency gain for k∈[0.01,0.5]k \in [0.01, 0.5]), e.g., see Figure 3 where our SUN-Bandpass filter has an amplitude gain that sits in-between the baseline filters in this frequency band.

In Appendix C.1 of our revised version, we have added an additional experiment on a smaller scale setting, where we compute the full gradient during the middle phase of training and compute the gradient frequency spectrum. The spectrum in Fig. 7 shows that the full gradient has very weak high-frequency signal, indicating that the high frequency spectrum is dominated by gradient error.

  1. For the results in Fig. 6 and Table 1, were the momentum hyperparameters of the baselines tuned, or only the learning rate? SUN itself involves tuning the filter constraints and coefficients, so I am concerned about whether the comparison uses a similar tuning budget.

Besides for EMAMIX which we tuned the additional momentum coefficient and momentum scaling as the default suggested value performs poorly (see the additional hyperparameter tuning for EMAMIX in Appendix C.4), all other baseline algorithms use the default momentum coefficients. This also reveals the fundamental difference between our approach and existing baselines: the coefficients of prior work algorithms might require expensive experiment tuning to further optimize their performance, while the coefficients of SUN momentum algorithms are optimized using the filter delay minimization formulation and does not depend on expensive training experiments for tuning. Therefore, the coefficient selection process of SUN is much more efficient and practical.

  1. Li et al. (2025) already uses frequency domain analysis to study and design momentum filters. What is new in SUN beyond adding group delay to the design objective and filtering the gradient difference?

Please refer to the above Comparison with [1] for our separation with Li et al. (2025), and in particular that Li et al. (2025) only analyzed the classical momentum algorithm. To summarize, in addition to group delay optimization formulation and the SUN filter families construction, our work gives a clear insight about momentum algorithm design ---- that group delay is a useful design criterion that is highly correlated with training performance (e.g., see the metric of normalized delay in Table 1). This insight provides practical guideline for future works to use/design better momentum algorithms, as group delay appears in all filtering systems. The name of our paper (Stochastic Unbiased Momentum) highlights the goal of achieving unbiased gradient estimate through filtering stochastic gradient with a low delay filter.

Reviewer_wbKGOpenReview ↗

Response to the Authors

I appreciate the detailed rebuttal from the authors. I summarize my assessment below.

1. Limited Novelty.

I discuss this in two parts.

(a) Problem Analysis

Prior work and SUN reach similar conclusions:

PaperMain Conclusions
Li et al. [1]Momentum acts as a frequency-domain filter. Suppressing high frequencies and progressively amplifying low frequencies can improve training.
Yao et al. [2]Gradient averaging reduces variance but introduces estimation bias.
Sahu et al. [3]Momentum causes tracking lag and can degrade performance under nonstationarity.
SUNGradient averaging suppresses noise but introduces group delay.

Li et al. [1], Appendix A, Eq. (11) derive the EMA phase response:

ϕLi(ω)=−tan⁡−1(βsin⁡ω1−βcos⁡ω).\phi_{\mathrm{Li}}(\omega)=-\tan^{-1}\left(\frac{\beta\sin\omega}{1-\beta\cos\omega}\right).

SUN, Sections 3–4.1, defines τ(ω)=−ϕ′(ω)\tau(\omega)=-\phi'(\omega). Differentiating Li et al.'s result directly yields SUN's formula:

−ϕLi′(ω)=β(cos⁡ω−β)1−2βcos⁡ω+β2,−ϕLi′(0)=β1−β=τSUN(0).-\phi'_{\mathrm{Li}}(\omega)=\frac{\beta(\cos\omega-\beta)}{1-2\beta\cos\omega+\beta^2},\qquad -\phi'_{\mathrm{Li}}(0)=\frac{\beta}{1-\beta}=\tau_{\mathrm{SUN}}(0).

Yao et al. [2], Theorem 2.3, Eq. (3) give the EMA estimation-bias bound:

∥Bias⁡(m)∥2≤(αLG1−β+ασL2(1−β))2.\|\operatorname{Bias}(m)\|^2\leq\left(\frac{\alpha LG}{1-\beta}+\frac{\alpha\sigma L}{\sqrt{2(1-\beta)}}\right)^2.

Its dominant dependence on β\beta matches SUN's zero-frequency delay:

∥Bias⁡(m)∥=O(11−β),τSUN(0)=Θ(11−β).\|\operatorname{Bias}(m)\|=O\left(\frac{1}{1-\beta}\right),\qquad \tau_{\mathrm{SUN}}(0)=\Theta\left(\frac{1}{1-\beta}\right).

Sahu et al. [3], Theorems 3.3 and 3.7 establish momentum-dependent tracking-error bounds and an unavoidable inertia window of order Ω(κ/(1−β))\Omega(\kappa/(1-\beta)).

E∥θt−θt⋆∥2≲e−γμt/(1−β)(1−β)2∥θ0−θ0⋆∥2+(2+β)2Δ2γ2μ2+σ2γμ(1−β).\mathbb{E}\|\theta_t-\theta_t^\star\|^2\lesssim\frac{e^{-\gamma\mu t/(1-\beta)}}{(1-\beta)^2}\|\theta_0-\theta_0^\star\|^2+\frac{(2+\beta)^2\Delta^2}{\gamma^2\mu^2}+\frac{\sigma^2\gamma}{\mu(1-\beta)}.

τβ=Ω(κ1−β),τSUN(0)=Θ(11−β).\tau_\beta=\Omega\left(\frac{\kappa}{1-\beta}\right),\qquad \tau_{\mathrm{SUN}}(0)=\Theta\left(\frac{1}{1-\beta}\right).

Conclusion: SUN's EMA group-delay formula follows directly from Li et al. [1], while Yao et al. [2] and Sahu et al. [3] already establish the underlying noise–lag trade-off. SUN mainly reformulates known results using different terminology.

(b) Method Design

SUN's design criterion has some novelty, but its implementation adds hyperparameters and memory overhead for limited gains.

MethodDesignHyperparametersPersistent States
MomentumEMA11
AdamFirst/second moments22
FSGDM [1]Momentum scheduling31
SGDF [2]Adaptive filtering22
SUN-BandpassEMA + gradient differences53
SUN-HighOrderHigher-order EMAUp to 3718

Counts exclude learning rate and weight decay.

Then what is the point of proposing SUN?


2. Limited Experiments

  1. The experiments are largely limited to NanoGPT2-124M on FineWeb. More comparisons with other optimizers across different architectures and datasets are needed to validate SUN's effectiveness.
  2. I noticed the additional experiments in Appendix C, but the results in Fig. 11 are not promising. Could the authors do a quick experiment combining SUN with Adam, SignSGD, and Muon using ResNet-34 on CIFAR-100, and compare the results with SGDF [2] and FSGDM [1]?

3. The Meaning of "Unbiased".

The authors state:

The name of our paper (Stochastic Unbiased Momentum) highlights the goal of achieving unbiased gradient estimate through filtering stochastic gradient with a low delay filter.

I am concerned about what the authors mean by an "unbiased gradient estimate."

  1. GD: Training on the entire dataset gives the exact full gradient, without sampling bias.
  2. SGD: Randomly sampled gradients are unbiased but have variance. Importantly, unbiasedness is defined at the same model parameters.
  3. Gradient accumulation: Gradients are accumulated at the same model parameters. For networks with LayerNorm, this is equivalent to computing the full-batch gradient when all samples are included, assuming no other batch-dependent operations.

However, SUN averages and filters gradients evaluated at different model parameters across iterations. Reducing group delay does not imply that the resulting gradient estimator is statistically unbiased with respect to the current true gradient. What is the precise definition of "unbiased" in this paper?

References

[1] Xianliang Li, Jun Luo, Zhiwei Zheng, Hanxiao Wang, Li Luo, Lingkun Wen, Linlong Wu, and Sheng Xu. On the Performance Analysis of Momentum Method: A Frequency Domain Perspective. ICLR, 2025.

[2] Zhipeng Yao, Rui Yu, Guisong Chang, Ying Li, Yu Zhang, and Dazhou Li. Dynamic Momentum Recalibration in Online Gradient Learning. CVPR, 2026.

[3] Sharan Sahu, Cameron J. Hogan, and Martin T. Wells. On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization. ICML, 2026.

Reviewer_PVj5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SUN, a filter-design approach to momentum optimization. It selectively compensates for group delay in chosen frequency bands while preserving noise attenuation. The language-model experiment shows a modest improvement in validation perplexity.

Critical strengths

Introducing group delay as an explicit design criterion gives a clear framework for balancing gradient tracking and noise suppression.

Frequency-selective phase compensation provides an interpretable and extensible way to design momentum filters.

Critical weaknesses

Evaluation is limited to one model and dataset

Questions for authors

Does SUN retain its performance advantage in controlled comparisons between filters with similar magnitude responses but different group delays?

1 comment

Thank you for the positive review. We will address your remaining concern as follows.

Evaluation is limited to one model and dataset.

We will try our best to gather computational resources and provide additional evaluation as soon as possible.

Does SUN retain its performance advantage in controlled comparisons between filters with similar magnitude responses but different group delays?

This is a very good question. We have added Appendix C.2 in our revised version to compare between two SUN-Bandpass filters, where both filters have the same gain response while the additional SUN-Bandpass filter has maximized its group delay. The experiment result in Fig. 8 shows that the filter with higher delay performed worse than the filter with lower delay. We have further designed a new filter SUN-HighOrder, which has similar gain response as the optimized SUN-Bandpass, e.g., see the updated Fig. 3, and achieved better performance as illustrated in the updated Table 1.

Meanwhile, we are willing to answer any further clarifying questions to improve the reviewer's confidence on our work.

Reviewer_tvnJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper interprets momentum estimators as discrete-time filters and uses amplitude response and group delay to characterize the tradeoff between noise attenuation and gradient-tracking lag. It proposes SUN, which filters gradient-difference lookahead signals to compensate for delay in selected frequency bands. A SUN-Bandpass configuration is constructed through constrained numerical optimization and evaluated by replacing Muon’s momentum in 124M-parameter language-model pre-training. The approach is interpretable and promising, but its connection to actual gradient-estimation quality and the robustness of its empirical benefits require stronger validation.

Critical strengths
  1. An interpretable perspective on momentum. The joint analysis of amplitude response and group delay provides a useful framework for comparing EMA, MARS, Muon-style momentum, and EMA mixtures. It highlights temporal behavior that is not apparent from the update equations alone.

  2. A constructive and modular method. SUN goes beyond descriptive analysis by introducing filtered lookahead and formulating a constrained filter-design problem. The analytical transfer function in Appendix A and explicit coefficients in Eq. (15) make the proposed momentum estimator concrete and implementable.

  3. Evaluation on language-model pre-training with learning-rate sweeps. The experiments test the method beyond synthetic filtering examples and compare several momentum variants within the same Muon framework. SUN achieves the best tuned validation perplexity among the evaluated variants, providing encouraging initial evidence of practical usefulness.

Critical weaknesses
  1. The connection between group delay and optimization quality is insufficiently established. Reduced group delay does not by itself guarantee smaller gradient-tracking error, statistical unbiasedness, or improved closed-loop stability. The paper does not measure the useful-gradient and noise spectra during training or directly verify that SUN improves gradient-estimation accuracy.

  2. The filter-design procedure is underspecified. Although the final coefficients are provided, the frequency bands, numerical constraint boundaries, solver initialization, and iterative tuning budget used to obtain them are not fully specified. This makes it difficult to reproduce the design process or determine whether the improvement reflects a transferable principle rather than configuration-specific tuning.

  3. The empirical evidence is narrow and lacks uncertainty estimates. Results cover one model size, one dataset, and one optimizer framework, without reported multi-seed variability. The improvement over the strongest baseline, EMAMIX, is only 0.056 perplexity, and its significance, additional optimizer-state cost, and robustness to equally extensive baseline momentum tuning remain unclear.

Questions for authors
  1. Can you directly measure the tracking error between the momentum estimate and a large-batch approximation to the current true gradient? Does lower group delay consistently predict lower tracking error after accounting for amplitude distortion and noise attenuation?

  2. Could you provide the exact bands and constraint functions in Eqs. (12)–(14), frequency discretization, SLSQP initialization, stopping criteria, and total number of configurations explored? Were these choices informed by language-model validation results?

  3. Does the 0.056-perplexity improvement over EMAMIX persist across independent seeds and comparable momentum-hyperparameter searches? A comparison with the closely related FSGDM method would also help clarify the benefit of explicitly optimizing group delay.

  4. Do the same SUN coefficients improve training at another model scale or when combined with AdamW rather than Muon? Please also report additional optimizer-state memory, per-step overhead, and performance at matched wall-clock budgets.

3 comments

Thank you for the thoughtful review. We will address each point of your concern by the following response.

  1. The connection between group delay and optimization quality is insufficiently established. Reduced group delay does not by itself guarantee smaller gradient-tracking error, statistical unbiasedness, or improved closed-loop stability. The paper does not measure the useful-gradient and noise spectra during training or directly verify that SUN improves gradient-estimation accuracy.

Thank you for pointing out this gap of evidence. Accordingly, we measure the gradient spectrum through DTFT in a smaller scale experiment in Appendix C.1 of our revised version, verifying that the full gradient has very weak high frequency components, and SUN-Bandpass has lower tracking error than EMA. We leave the development of theoretical convergence guarantee on low delay momentum algorithms as future work, because it involves in-depth re-working on the existing framework of optimization analysis through frequency analysis. We position the current paper as the first practical approach on designing low delay momentum algorithms by constrained group delay optimization, which we believe it will be in the interest of the fast-paced community on practical deep learning optimization.

  1. The filter-design procedure is underspecified. Although the final coefficients are provided, the frequency bands, numerical constraint boundaries, solver initialization, and iterative tuning budget used to obtain them are not fully specified. This makes it difficult to reproduce the design process or determine whether the improvement reflects a transferable principle rather than configuration-specific tuning.

We have added the details of our optimization process in Appendix D of our revised version accordingly. Please also see our explanation to your Q2 regarding the same concern.

  1. The empirical evidence is narrow and lacks uncertainty estimates. Results cover one model size, one dataset, and one optimizer framework, without reported multi-seed variability. The improvement over the strongest baseline, EMAMIX, is only 0.056 perplexity, and its significance, additional optimizer-state cost, and robustness to equally extensive baseline momentum tuning remain unclear.

We have extended Table 1 to show the mean-variance of 3 random runs, the peak GPU memory usage, and the per iteration runtime cost for each algorithm. Due to limited time and computational resource, we are yet unable to deliver experiment results on larger model sizes and other datasets. In the remaining time we will try our best to arrange experiments on a larger model before the end of discussion phase. Meanwhile, we have additionally proposed a new filter family SUN-HighOrder which enlarged the improvement over EMAMIX to 0.1 perplexity, as updated in Table 1 of our revised version.

  1. Can you directly measure the tracking error between the momentum estimate and a large-batch approximation to the current true gradient? Does lower group delay consistently predict lower tracking error after accounting for amplitude distortion and noise attenuation?

Thank you for this great suggestion. In Appendix C.1 of our revised version, we add an additional experiment on a smaller scale setting, where we compute the full gradient during the middle phase of training and directly measure the gradient tracking error. Our SUN-Bandpass filter consistently shows lower tracking error than the EMA baseline during this middle phase of 1000 iterations.

In order to account for the difference in amplitude gain, we further propose a metric of effective normalized delay D{\cal D} in Section 4.5, Eq. (21) of our revised version, which measures the total amount of delay weighted by its frequency output-power at the filtered signal. This new metric is a good predictive measurement even without a prior knowledge on the input signal's spectrum, e.g., in Table 1 where low normalized delay D(A=1){\cal D}(A=1) with a uniform prior input spectrum of A=1A=1 is correlated with low perplexity, and in Table 2 where we apply actual input signal spectrum $A=| {\rm DTFT}({\sf Full \ Grad.})|$ and obtain the same relative ordering of delay on different filters.

  1. Could you provide the exact bands and constraint functions in Eqs. (12)–(14), frequency discretization, SLSQP initialization, stopping criteria, and total number of configurations explored? Were these choices informed by language-model validation results?

The SLSQP solver is used for multiple rounds of refinement on the filter coefficients. In Appendix D, we have added the exact problem parameter choice and solver configuration for the last round of optimization that gives us the SUN-Bandpass coefficients. For the SUN-Bandpass filter, we have explored around 10-20 configurations to arrive at the coefficients that achieve a desired low delay behavior, i.e., 10-20 rounds of the optimization process described in Appendix D. Note that this process of optimizing the filter coefficients does not involve any training experiments, and each round only takes around 1 minute of computation. The choices of gain bounds are determined according to the gain response covered by the baseline filters (high frequency band k∈[0.01,0.5]k \in [0.01, 0.5]), e.g., see Figure 3 where our SUN-Bandpass filter has an amplitude gain that sits in-between the baseline filters in this frequency band. All the other choices of solver parameters are decided according to the solver's behavior, e.g., increasing the number of iterations when the returned solution does not improve.

  1. Does the 0.056-perplexity improvement over EMAMIX persist across independent seeds and comparable momentum-hyperparameter searches? A comparison with the closely related FSGDM method would also help clarify the benefit of explicitly optimizing group delay.

We have extended the current results with 3 random runs and report the mean-variance in Table 1 and provided more hyperparameter search for EMAMIX in Fig. 10 of our revised version, where further tuning of the EMAMIX parameters has improved its optimal performance over SUN-Bandpass. Meanwhile, we have constructed SUN-HighOrder to further push the optimal performance boundary with minimal delay, such that SUN-HighOrder has 0.1 lower perplexity than the fine-tuned EMAMIX. (We notice the missing mean-variance of the new EMAMIX result but we do not have the computational resource at the moment. We will fill this in soon.)

When comparing the expense of hyperparameter search, notice that both SUN-Bandpass and SUN-HighOrder do not require running actual training experiments to search for their coefficients. The runtime cost of executing the SLSQP solver is less than 1 minute on common consumer CPU. Therefore, the hyperparameter searching cost of our framework is incomparable to existing momentum algorithms that requires expensive training experiments for tuning.

Also, we have provided an additional comparison with FSGDM in Appendix C.3, Fig. 9. With their proposed default parameters, time-varying filter FSGDM does not out-perform the time-invariant EMA baseline.

  1. Do the same SUN coefficients improve training at another model scale or when combined with AdamW rather than Muon? Please also report additional optimizer-state memory, per-step overhead, and performance at matched wall-clock budgets.

Unfortunately we are not able to deliver additional results on another model scale due to the limited time and computation. Meanwhile, we provide an early attempt on replacing the first-order momentum of Adam with the SUN-Bandpass filter in Appendix C.5 of our revised version. Our current results show that it is insufficient to just replace the first-order momentum of Adam with low delay filter. We will continue to investigate the possible ways to adopt low delay filters in Adam, e.g., further replacing the second-order momentum with low delay filters.

For the additional optimizer-state memory and computation overhead, we have added the number of state variables, the exact peak GPU memory usage and per-step wall-clock runtime for different momentum algorithms in Table 1 of our revised version. Comparing performance at matched wall-clock budget requires training different algorithms for different number of iterations, for which we unfortunately do not have sufficient time to prepare. Meanwhile, we have provided the learning curves against wall-clock runtime in the updated Figure 6, which can serve as an approximation to what you have suggested.

AuthorsOpenReview ↗

Reminder about the end of discussion period

Dear Reviewer tvnJ,

We hope our message finds you well. Since the discussion period is ending soon, we would like to see if you have any other concerns regarding our response, and let us know if you are satisfied with our response.

Best regards,

Authors of 50578

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.