- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces the concept of gradient cluster imbalance as a potential mechanism by which the structure of per-sample gradients can influence optimization and give rise to the difference between standard gradient descent (GD) and normalized optimizers such as Adam and Muon. The authors hypothesize that per-sample gradients form clusters with similar directions, that different clusters are approximately orthogonal, and that the cluster sizes are imbalanced. Under this structure, frequent clusters dominate the averaged gradient and slow down progress on directions corresponding to rare clusters.
The theoretical analysis assumes a mixture model with power-law cluster frequencies and strong orthogonality and smoothness assumptions. Under these assumptions, the authors analyze full-batch GD and batch-size-one SGD. They also show, with additional axis-alignment and spectral-alignment assumptions, how idealized sign-based and spectral optimization methods reduce the effect of cluster-frequency imbalance. The authors conduct empirical analyses on gradient-similarity matrices of a two-layer Transformer, Pythia-1B, and ResNet-20 trained on CIFAR-100, using clustering and visualization techniques. They report unequal gradient clusters across multiple layers and find patterns associated with output tokens, token sequences, and visually similar examples.
In short, this is an interesting paper with an intriguing idea and promising preliminary results. However, the results do not clearly support the central hypothesis that gradient cluster imbalance explains why normalized optimizers benefit from larger batch sizes.
Critical strengths
-
Interesting diagnostic beyond the output layer: the paper extends the investigation of gradient-frequency imbalance to layers beyond the output layer of a model. Its per-sample gradient-similarity matrices provide a means of investigating hidden-layer structure beyond the need of predefined class or token labels.
-
Experiments across different model types: the experiments include language models and a vision model, offering evidence that unequal gradient clusters are not restricted to a single architecture or data modality.
-
Interesting observations about cluster structure: the reported associations between gradient clusters and output tokens, shared token patterns, and visually similar images are informative; they could motivate further work on the relationship between data frequency, gradient geometry, and optimization.
-
A potentially useful theoretical framework: the paper ties cluster frequencies to effective update sizes under GD and explains why coordinate-wise or spectral normalization might be beneficial under particular structural assumptions. Although the assumptions are restrictive, this framework offers a starting point for studying the interaction between gradient structure and optimizer behavior.
Critical weaknesses
-
The main theory does not properly establish the relationship between batch size and normalized optimizers. The theory primarily focuses on the cases of full batch GD and batch size one SGD, without guarantees for intermediate batch sizes, and without a rigorous demonstration that Adam and Muon improve with batch size. The idealized sign- and spectral-descent analyses also omit stochastic rates, cutting off the connection to the main claim of the paper.
-
The theoretical assumptions substantially restrict the conclusions drawn in the paper. Global orthogonality makes the objective separable, which makes it easy to analyze, but doesn t reflect realistic interactions between components. Frequency imbalance is not cleanly separated from other effects such as curvature differences, making the contribution of cluster imbalance unclear.
-
The quantitative evidence for the assumed cluster-size distribution is limited. While the theory assumes a power-law of cluster frequencies, the empirical analysis does not report the exponent or its uncertainty, and the cropped matrices and clustering parameters weaken claims of robustness. Absolute cosine similarity treats anti-aligned gradients as similar to aligned gradients, which may cancel.
-
The evidence for spectral alignment in hidden layers is not clearly presented. Spectral alignment is used to motivate Muon-like approaches, but the appendix reports substantial row overlap, and only tentative evidence in favor of the main assumption, weakening claims that Muon is better supported by the evidence.
-
Some theoretical conclusions seem to go beyond the stated bounds. While the upper bounds do not by themselves demonstrate that GD is worst at , GD and SGD rates contain different constants. The logarithmic relationship requires clarification or additional conditions before the stated conclusion can be reached.
Questions for authors
- Can the authors generalize their analysis to intermediate batch sizes, and give convergence guarantees for sign-based or spectral optimization in such a setting? In particular, can they give conditions under which the dependence on the cluster frequencies is reduced, and the advantage of their method improves with batch size?
- Can the authors estimate the distribution of cluster sizes and a power-law exponent with uncertainty, on the full analyzed sample set, not cropped visualizations? Do the findings hold when using one sample per sequence, to reduce correlations between examples from the same sequence or document?
- Can the authors provide a causal validation experiment, varying some aspect of cluster frequency imbalance or the batch size, and measuring the change in performance between SGD, Adam, and Muon? For instance, could rebalancing frequent clusters reduce the predicted optimizer advantage?
- Can the authors reconcile the description in the main-text of small violations of the spectral alignment, with Appendix E.2, which reports large row overlaps in several layers, with only partial empirical support? Are the findings not sensitive to using signed rather than absolute cosine similarity?
- Can the authors explain why the results establish that GD is worst at , given that the stated results are upper bounds? Can they also specify the conditions under which the comparison between GD and batch-size-one SGD yields only a logarithmic difference, given that the exponents involve different constants?
We thank the reviewer for their evaluation and address their concerns below.
As all reviewers have asked similar questions, we have added additional ablation experiments to show this effect. Please see our global response.
We agree that our theoretical results only cover the two extreme regimes of fully stochastic or fully deterministic. Intermediate batch size results will only interpolate between those two -- we have reworded the discussion of Theorems 1 and 2 to be more explicit.
Our results regarding sign descent show independence to imbalance. As the methods perform the same regardless of imbalance, they will not suffer from the same slowdown as GD in the imbalanced case. We clarified this point in the discussion.
We agree that those results are not a complete description of the behavior of sign descent or Muon. Our goal is to explain why increasing batch size brings limited benefits to SGD, but work for sign/muon. Precise rates would require additional method-specific assumptions, shifting the focus from imbalance to small technical differences.
We agree that global orthogonality is strong and unlikely to hold in practice, as noted in the discussion of Assumption 1. Our goal is to give intuition as to how imbalance impacts training. Explicitly assuming no interaction between components allows us to isolate the effect of frequency imbalance and its impact on curvature in the deterministic and stochastic setting (changing the smoothness constant from L=1 for SGD in Theorem 2 to L=1/π1 for GD in Theorem 1).
Good suggestion! We added an uncertainty analysis in Appendix C.6, fitting a power law to the cluster sizes. On layers with a clear cluster structure, especially those close to the output, the results are consistent with data drawn from Zipfian distributions.
We have added this experiment to Appendix C.7, repeating the analysis using one sample per sequence. Overall, the results are qualitatively similar.
We agree that the current wording was unclear. We updated §3 to better reflect what we meant to convey: that while spectral alignment is not a perfect fit, it is a better description than axis-alignment (which clearly doesn't hold), which might explain why spectral methods outperform sign methods.
We agree that frequent directions might still have small overall signal if the gradients cancel out. We focus on absolute cosine similarity to identify distinct subspaces where gradients live, regardless of direction, as this is what we need to control to establish the theoretical results.
On layers that are well clustered, taking into account the sign would give twice the number of clusters, a positive and negative one for each, but otherwise similar looking matrices. This approach could provide another way to find clusters with opposite signal, as in the work of Rosenfeld and Risteski 2023.
Indeed, our results compare available guarantees rather than exact dynamics. Those upper bounds are however tight in terms of the dependence on the number of clusters, based on simpler models that fit our assumptions (Kunstner and Bach, 2025)
We updated the discussion after Theorem 2 to avoid the inconsistency of the difference in constants.
We kindly ask the reviewers to consider increasing their scores if they find our responses satisfactory. We remain available to address any further questions during the discussion period.