- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies stochastic momentum methods that choose update directions from Schatten norm balls, including an idealized Muon direction as a special case. It proves an bound on the average expected gradient norm for a momentum variance reduction (MVR) method under generalized smoothness, gradient dependent noise, and a centered gradient difference condition. EMA and heavy ball momentum obtain an bound without the gradient difference condition. Experiments compare MVR and EMA on DeiT-Tiny and GPT-2. MVR performs better in some runs using theorem-scaled parameters, whereas EMA generally reaches a lower final loss with the empirical settings.
Overall, I find the extension to gradient dependent noise technically interesting, but I am not yet convinced that the paper makes a strong enough case for acceptance. Related MVR methods already obtain the rate in arbitrary norm settings, so the value of this paper rests largely on the scope of its new assumptions. I would like to see a concrete problem where those assumptions hold together, and a comparison that accounts for MVR’s two gradient evaluations per step. I am currently on the borderline ( leaning reject), but could revise my assessment if the authors address these points.
Critical strengths
- The paper addresses a useful limitation of existing analyses: gradient noise need not have uniformly bounded variance. The centered gradient difference condition also targets the quantity used by the MVR proof.
- The average-iterate guarantee is a meaningful part of the result. Together with the treatment of different Schatten norms, it makes the theoretical contribution more specific than the headline rate alone suggests.
- The experiments report cases where EMA outperforms MVR. This helps readers distinguish the convergence guarantee from the observed training performance.
Critical weaknesses
My main question is how often the assumptions behind the MVR result hold in the intended applications. Assumption 4 places a particular It is hard to assess the reach of the main theorem without a worked learning problem satisfying the smoothness, affine noise, and centered gradient difference assumptions together. In particular, an example where bounded variance fails would make the gain over earlier analyses much more concrete.
The empirical comparisons are plotted against updates, although MVR evaluates a fresh sample’s gradient at two parameter values per update. A comparison by gradient evaluations or wall-clock time could change the practical interpretation.
The GPT-2 empirical runs use approximate matrix directions and AdamW for the remaining parameters, while the theorem analyzes exact Schatten directions. The paper should be clearer about what these runs illustrate. The single-seed DeiT-Tiny and empirical GPT-2 results also make small differences difficult to judge.
Questions for authors
- Can the authors give a concrete stochastic learning model satisfying Assumptions 3–5 in which gradient-noise variance is unbounded?
- How do MVR and EMA compare at equal gradient-evaluation or wall-clock budgets, with variability across seeds?
- How sensitive are the GPT-2 results to the accuracy of the power-iteration and Newton–Schulz directions?
We thank the reviewer for taking the time to review our work and for their helpful feedback. We address the weaknesses and questions:
W1/Q1. An example where bounded variance fails. Appendix B of the revision gives an example with a quadratic population objective that satisfies Assumptions 3-5. Its single-point gradient-noise variance is unbounded, and its centered same-sample gradient-difference second moment grows without bound even when the two query points stay a fixed distance apart. Thus, the example violates both the commonly used uniform variance bound and the constant-coefficient centered gradient-difference bound while satisfying the general conditions introduced and used in this work.
W2/Q2. Additional experimental comparison.
We have added three-seed comparisons on two related tasks, ResNet-32 on CIFAR-10 and GPT-2 small on Tiny Shakespeare. The curves report the mean and min-max range across seeds (Figure 1-2 in Section 4). We now also report learning curves for MVR and EMA under the same budget of minibatch-gradient evaluations (Figure 11 in Appendix E.3 and Figure 12 in Appendix E.4).
Our available compute limits the number of seeds for the original DeiT-Tiny and empirical GPT-2 runs.
On CIFAR-10, EMA achieves lower mean test loss for p=1 and p=∞, while the two methods have nearly identical test loss for p=2 (0.448 for MVR and 0.446 for EMA).
On Tiny Shakespeare, EMA achieves lower final training and validation losses for all three values of p at batch size 16.
These results are qualitatively consistent with the original experiments under empirical settings, where EMA generally achieved comparable or lower held-out loss. Our theoretical rate concerns first-order stationarity of the population objective, whereas these experiments probe an aspect outside the ambit of our theory i.e comparing held-out loss within a finite training budget.
W3/Q3. Approximate Schatten directions and sensitivity in GPT-2. Muon-style language model training commonly uses Newton--Schulz iterations to compute updates for hidden weight matrices and AdamW for the remaining parameters, including embeddings and normalization parameters. Power iteration is also used to approximate leading singular directions of gradient matrices in neural network training. Computing a full SVD for every matrix block at every update would substantially increase the cost of our GPT-2 runs, so we use iterative approximations for p=1 and p=∞.
The GPT-2 experiments is to compare MVR and EMA as two ways of forming the momentum estimate used in the Schatten update. For each p, we apply the same direction solver to the two momentum estimates and use a single iteration count shared by both methods when the solver is approximate. Both methods use identical AdamW settings for the remaining parameters.