acceptodds
Under review as a conference paper at ICLR 2027

Gradient Cluster Imbalance: Why Large Batches Favor Normalized Optimizers

12 views

Abstract

Adam and Muon often outperform plain stochastic gradient descent in neural network training, but their advantages vary across layers and batch sizes. Previous work has found that Adam is robust to class imbalance in the first and last layers, while Muon is effective in hidden layers. However, a unifying mechanism remains unclear, particularly how these advantages depend on batch size. We identify a property for per-sample gradients that we refer to as gradient cluster imbalance: per-sample gradients form clusters with similar directions that are nearly orthogonal to other clusters, and those clusters have imbalanced sizes. Under this property, we show that GD makes slow progress in directions corresponding to rare clusters for smooth, convex objectives, and that large batches offer limited gains over stochastic gradient descent as averaging leads to ill-conditioning. Under additional structural assumptions, SignGD, a simplified model of Adam, and idealized Muon mitigate this imbalance by normalizing gradients from different clusters separately. Measurements of individual layers provide partial support for these structural assumptions. Experiments reveal unequal gradient clusters across hidden layers of language models and vision models, extending existing observations of frequency imbalance beyond class labels. We show that clusters vary across layers and over training and reflect shared token patterns and visual features.

Primary area:
optimization
Keywords:
Neural Network Optimization, Batch Size Scaling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eHXvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces the concept of gradient cluster imbalance as a potential mechanism by which the structure of per-sample gradients can influence optimization and give rise to the difference between standard gradient descent (GD) and normalized optimizers such as Adam and Muon. The authors hypothesize that per-sample gradients form clusters with similar directions, that different clusters are approximately orthogonal, and that the cluster sizes are imbalanced. Under this structure, frequent clusters dominate the averaged gradient and slow down progress on directions corresponding to rare clusters.

The theoretical analysis assumes a mixture model with power-law cluster frequencies and strong orthogonality and smoothness assumptions. Under these assumptions, the authors analyze full-batch GD and batch-size-one SGD. They also show, with additional axis-alignment and spectral-alignment assumptions, how idealized sign-based and spectral optimization methods reduce the effect of cluster-frequency imbalance. The authors conduct empirical analyses on gradient-similarity matrices of a two-layer Transformer, Pythia-1B, and ResNet-20 trained on CIFAR-100, using clustering and visualization techniques. They report unequal gradient clusters across multiple layers and find patterns associated with output tokens, token sequences, and visually similar examples.

In short, this is an interesting paper with an intriguing idea and promising preliminary results. However, the results do not clearly support the central hypothesis that gradient cluster imbalance explains why normalized optimizers benefit from larger batch sizes.

Critical strengths
  1. Interesting diagnostic beyond the output layer: the paper extends the investigation of gradient-frequency imbalance to layers beyond the output layer of a model. Its per-sample gradient-similarity matrices provide a means of investigating hidden-layer structure beyond the need of predefined class or token labels.

  2. Experiments across different model types: the experiments include language models and a vision model, offering evidence that unequal gradient clusters are not restricted to a single architecture or data modality.

  3. Interesting observations about cluster structure: the reported associations between gradient clusters and output tokens, shared token patterns, and visually similar images are informative; they could motivate further work on the relationship between data frequency, gradient geometry, and optimization.

  4. A potentially useful theoretical framework: the paper ties cluster frequencies to effective update sizes under GD and explains why coordinate-wise or spectral normalization might be beneficial under particular structural assumptions. Although the assumptions are restrictive, this framework offers a starting point for studying the interaction between gradient structure and optimizer behavior.

Critical weaknesses
  1. The main theory does not properly establish the relationship between batch size and normalized optimizers. The theory primarily focuses on the cases of full batch GD and batch size one SGD, without guarantees for intermediate batch sizes, and without a rigorous demonstration that Adam and Muon improve with batch size. The idealized sign- and spectral-descent analyses also omit stochastic rates, cutting off the connection to the main claim of the paper.

  2. The theoretical assumptions substantially restrict the conclusions drawn in the paper. Global orthogonality makes the objective separable, which makes it easy to analyze, but doesn t reflect realistic interactions between components. Frequency imbalance is not cleanly separated from other effects such as curvature differences, making the contribution of cluster imbalance unclear.

  3. The quantitative evidence for the assumed cluster-size distribution is limited. While the theory assumes a power-law of cluster frequencies, the empirical analysis does not report the exponent or its uncertainty, and the cropped matrices and clustering parameters weaken claims of robustness. Absolute cosine similarity treats anti-aligned gradients as similar to aligned gradients, which may cancel.

  4. The evidence for spectral alignment in hidden layers is not clearly presented. Spectral alignment is used to motivate Muon-like approaches, but the appendix reports substantial row overlap, and only tentative evidence in favor of the main assumption, weakening claims that Muon is better supported by the evidence.

  5. Some theoretical conclusions seem to go beyond the stated bounds. While the upper bounds do not by themselves demonstrate that GD is worst at α=1\alpha=1, GD and SGD rates contain different constants. The logarithmic relationship requires clarification or additional conditions before the stated conclusion can be reached.

Questions for authors
  1. Can the authors generalize their analysis to intermediate batch sizes, and give convergence guarantees for sign-based or spectral optimization in such a setting? In particular, can they give conditions under which the dependence on the cluster frequencies πk\pi_k is reduced, and the advantage of their method improves with batch size?
  2. Can the authors estimate the distribution of cluster sizes and a power-law exponent with uncertainty, on the full analyzed sample set, not cropped visualizations? Do the findings hold when using one sample per sequence, to reduce correlations between examples from the same sequence or document?
  3. Can the authors provide a causal validation experiment, varying some aspect of cluster frequency imbalance or the batch size, and measuring the change in performance between SGD, Adam, and Muon? For instance, could rebalancing frequent clusters reduce the predicted optimizer advantage?
  4. Can the authors reconcile the description in the main-text of small violations of the spectral alignment, with Appendix E.2, which reports large row overlaps in several layers, with only partial empirical support? Are the findings not sensitive to using signed rather than absolute cosine similarity?
  5. Can the authors explain why the results establish that GD is worst at α=1\alpha=1, given that the stated results are upper bounds? Can they also specify the conditions under which the comparison between GD and batch-size-one SGD yields only a logarithmic difference, given that the exponents involve different constants?
1 comment

We thank the reviewer for their evaluation and address their concerns below.


Q3: Can the authors provide a [experimental validation for the effect of frequency imbalance or the batch size on the] performance between SGD, Adam, and Muon?

As all reviewers have asked similar questions, we have added additional ablation experiments to show this effect. Please see our global response.

Q1: Generalize the analysis to intermediate batch sizes, and give convergence guarantees for sign-based or spectral optimization in such a setting

We agree that our theoretical results only cover the two extreme regimes of fully stochastic or fully deterministic. Intermediate batch size results will only interpolate between those two -- we have reworded the discussion of Theorems 1 and 2 to be more explicit.

Our results regarding sign descent show independence to imbalance. As the methods perform the same regardless of imbalance, they will not suffer from the same slowdown as GD in the imbalanced case. We clarified this point in the discussion.

We agree that those results are not a complete description of the behavior of sign descent or Muon. Our goal is to explain why increasing batch size brings limited benefits to SGD, but work for sign/muon. Precise rates would require additional method-specific assumptions, shifting the focus from imbalance to small technical differences.

W2: Global orthogonality [...] doesn't reflect realistic interactions between components. Frequency imbalance is not cleanly separated from other effects such as curvature.

We agree that global orthogonality is strong and unlikely to hold in practice, as noted in the discussion of Assumption 1. Our goal is to give intuition as to how imbalance impacts training. Explicitly assuming no interaction between components allows us to isolate the effect of frequency imbalance and its impact on curvature in the deterministic and stochastic setting (changing the smoothness constant from L=1L=1 for SGD in Theorem 2 to L=1/π1L=1/\pi_1 for GD in Theorem 1).

W3/Q2: Can the authors estimate the distribution of cluster sizes and a power-law exponent with uncertainty?

Good suggestion! We added an uncertainty analysis in Appendix C.6, fitting a power law to the cluster sizes. On layers with a clear cluster structure, especially those close to the output, the results are consistent with data drawn from Zipfian distributions.

Q2: Do the findings hold when using one sample per sequence, to reduce correlations between examples from the same sequence?

We have added this experiment to Appendix C.7, repeating the analysis using one sample per sequence. Overall, the results are qualitatively similar.

W4: Evidence for spectral alignment in hidden layers is not clearly presented Q4: Can the authors reconcile the description [...] of small violations [...] with Appendix E.2, which reports large row overlaps?

We agree that the current wording was unclear. We updated §3 to better reflect what we meant to convey: that while spectral alignment is not a perfect fit, it is a better description than axis-alignment (which clearly doesn't hold), which might explain why spectral methods outperform sign methods.

Q4. Are the findings not sensitive to using signed rather than absolute cosine similarity?

We agree that frequent directions might still have small overall signal if the gradients cancel out. We focus on absolute cosine similarity to identify distinct subspaces where gradients live, regardless of direction, as this is what we need to control to establish the theoretical results.

On layers that are well clustered, taking into account the sign would give twice the number of clusters, a positive and negative one for each, but otherwise similar looking matrices. This approach could provide another way to find clusters with opposite signal, as in the work of Rosenfeld and Risteski 2023.

Q5: Can the authors explain why the results establish that GD is worst at (\alpha=1), given that the stated results are upper bounds?

Indeed, our results compare available guarantees rather than exact dynamics. Those upper bounds are however tight in terms of the dependence on the number of clusters, based on simpler models that fit our assumptions (Kunstner and Bach, 2025)

Can they also specify the conditions under which the comparison between GD and batch-size-one SGD yields only a logarithmic difference, given that the exponents involve different constants?

We updated the discussion after Theorem 2 to avoid the inconsistency of the difference in constants.


We kindly ask the reviewers to consider increasing their scores if they find our responses satisfactory. We remain available to address any further questions during the discussion period.

Reviewer_4q9ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces gradient cluster imbalance as an explanation for layer- and batch-size-dependent discrepancies between ordinary and normalized optimizers. It establishes GD and interpolating-SGD upper bounds for smooth convex objectives with globally orthogonal mixture components, and demonstrates that sign and ideal spectral updates remove frequency weights under additional coordinate or spectral separation. Synthetic experiments demonstrate these mechanisms, while transformer and ResNet analyses show unequal gradient clusters, including groups that reach beyond target labels.

Critical strengths
  1. The paper offers a valuable conceptual link between frequency imbalance and optimization geometry. The component-wise interpretation of GD shows how rare directions receive smaller effective steps, while the distinction between coordinate and spectral alignment clarifies when various normalization rules can help. The rotation experiment in Figure 3 makes this distinction particularly concrete.
  2. The theoretical analysis and findings make this a potentially valuable direction for understanding optimization.
Critical weaknesses

The neural-network experiments do not explicitly establish the practical consequences of the observed gradient clusters. The synthetic experiments compare optimization behavior, yet the real-model experiments mainly describe gradient structure without checking whether cluster frequencies predict learning progress under various optimizers and batch sizes. This makes the central account inadequately supported in the scenarios motivating the paper. More focused tests are needed. I would also appreciate clearer statements in the abstract and introduction about the scope of the conclusions: whether they concern optimization alone, or whether any implications for performance on unseen data are intended. The current evidence does not establish the latter.

I am not an expert in the theory of this field, and I am unsure how limiting the assumptions are relative to similar theoretical work. If they are suitable for isolating the proposed mechanism, I consider the paper’s message promising. At the moment, I incline toward weak rejection primarily because of the gap in experimental validation. I will consider the other reviews and the authors’ response, and I am open to revising my assessment.

Questions for authors
  1. Can you provide a targeted neural-network experiment checking whether measured cluster frequencies predict per-cluster learning progress across optimizers and batch sizes, with comparable tuning?
  2. Are the conclusions limited to optimization and loss convergence, or do they additionally address generalization to unseen data? If generalization is intended, which setting is meant (standard held-out performance, distribution shift, or another setting), and what evidence supports it? Please make this scope explicit in the abstract and introduction.
  3. How restrictive are the structural assumptions compared with those in closely related theoretical work? Which empirical observations support their relevance to the neural-network settings studied here, and which assumptions remain mainly idealizations?
2 comments

We thank the reviewer for recognizing our contribution and address their concerns below.


I am unsure how limiting the assumptions are relative to similar theoretical work. If they are suitable for isolating the proposed mechanism, I consider the paper’s message promising.

We agree with your assessment; as described in the paper, the assumptions are idealized and too strong for practical problems, but we do believe that they capture the core idea of what is happening, and we choose to use the simplest version to convey this message.

Q1: Can you provide a targeted neural-network experiment checking whether measured cluster frequencies predict per-cluster learning progress across optimizers and batch sizes, with comparable tuning?

As all reviewers have asked similar questions, we have added additional ablation experiments to show this effect. Please see our global response.

Q2. Are the conclusions limited to optimization and loss convergence?

Yes, our focus is on the optimization and training performance. However, training performance might translate to generalization in settings where learning rare instances is necessary (Feldman, 2019). We have made small edits to the abstract, introduction and conclusion to make it clearer that we are considering optimization/training performance.

Q.3 How restrictive are the structural assumptions compared with those in closely related theoretical work? Which empirical observations support their relevance to the neural-network settings studied here, and which assumptions remain mainly idealizations?

Gradients forming orthogonal clusters of different sizes is a novel assumption and our main contribution. Existing assumptions on the training of neural networks typically work at a "higher" level, such as smoothness measured in different norms. We believe assumptions that better reflect the behavior of real training dynamics are needed to explain the advantages of modern optimizers. We see the proposed assumption as a promising first step in this direction.

We evaluate the imbalanced gradient cluster assumption on different tasks/models, layers and training steps, and while the strong orthogonality required by the assumption used for our theoretical results (Assumption 1) does not hold literally, the results do support the idea that imbalanced gradient clustering (Definition 1) appears.

The other structural assumptions of axis-alignment and spectral-alignment (Assumptions 3 and 4) are similarly too strong as defined, but relaxed variants should be reasonable descriptions of real problems.


We kindly ask the reviewers to consider increasing their scores if they find our responses and updated experiments satisfactory. We remain available and happy to address any further questions during the discussion period.

Thank you for the additional experiments and for clarifying that the paper’s focus. I will increase my score.

Reviewer_nCavOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces gradient cluster imbalance, where per-sample gradients form approximately orthogonal clusters with unequal frequencies, as a mechanism explaining the advantages of normalized optimizers at large batch sizes. Under strong separation assumptions, the authors analyze how this imbalance slows gradient descent and limits the benefits of batching, while sign and spectral updates can remove frequency dependence under additional alignment conditions. Experiments on language and vision models reveal imbalanced gradient clusters whose structure and semantics vary across layers and training stages.

Critical strengths

The paper studies an important problem in machine learning: why normalized optimizers outperform SGD, particularly at large batch sizes. It introduces gradient cluster imbalance as a unifying explanation, supported by solid theoretical analysis and empirical evidence from language and vision models. The paper is well written, with clear explanations of its assumptions and main findings.

Critical weaknesses
  1. Some theoretical assumptions are overly strong and idealized, particularly global exact orthogonality and the additional alignment conditions, which may not hold in realistic models and data.
  2. The real-model experiments provide evidence for the existence of gradient clusters, but do not establish that cluster imbalance causally explains the performance differences between optimizers.
  3. Although the title emphasizes batch size, the experimental support largely comes from comparing single-sample SGD with full-gradient GD on a synthetic problem. Direct validation across intermediate batch sizes and in realistic neural networks is missing.
Questions for authors

The paper demonstrates gradient clustering across several model–dataset combinations. For a fixed dataset, which aspects of the cluster structure remain stable across models, and which are primarily determined by the architecture or training process?

1 comment

We thank the reviewer for their consideration and feedback. We address their concerns below.


W1: Some theoretical assumptions are overly strong and idealized, particularly global exact orthogonality and the additional alignment conditions

We agree that the assumptions are idealized and too strong for a practical problem. We do however believe that those assumptions capture the core idea of what is happening, and we choose to use the simplest version to convey this message. We have modified the contribution section to mention that those assumptions are idealized.

W2: The real-model experiments provide evidence for the existence of gradient clusters, but [not that] cluster imbalance causally explains the performance differences between optimizers.

As all reviewers have asked similar questions, we have added additional ablation experiments to show this effect. Please see our global response.

W3: Although the title emphasizes batch size, the experimental support largely comes from comparing single-sample SGD with full-gradient GD.

The experiments mentioned in the global response also compare different batch sizes, showing that the problem is continuous in batch size. Our theoretical results only cover the two extreme regimes, as the intermediate batch sizes will interpolate between those two cases. We have edited the discussion of Theorems 1 and 2 to be more explicit about this point.

Q1: For a fixed dataset, which aspects of the cluster structure remain stable across models, and which are primarily determined by the architecture or training process?

This is an interesting question. Some aspects will depend primarily on the dataset, such as the distribution of classes. However, architecture is likely to make a difference in what patterns get learned and what samples are considered "similar". Our initial probing in this regard cannot answer the question definitively; it would deserve its own independent study.


We hope our responses and updated experiments address your concerns. We kindly ask the reviewers to consider increasing their scores if they find our responses satisfactory. We are happy to address any further questions during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response

We thank all reviewers for their feedback. In this response, we address common concerns raised by all reviewers.

Additional experiments showcasing the impact of gradient cluster imbalance


All reviewers suggested additional experiments to test whether imbalance has a substantial impact in realistic training problems, the relevant imbalance is the one identified by our theory and clustering, and the effect depends on batch size.

Some of these points are already supported by existing experimental results in the literature. On the impact of imbalance, Kunstner et al. 2024 shows that imbalance in the distribution of the classes makes SGD slow compared to Adam. Wang et al. 2026(a) shows that a similar behavior appears when tweaking the distributions of "facts" or associations that an associative memory needs to learn. When this imbalance is not axis-aligned with specific parameters, Adam is slower than Muon at learning those imbalanced associations, as predicted by Zhang et al. 2025. On the impact of batch size, Kunstner et al. 2023 show that the performance gap between SGD and Adam grows with batch size, while Marek et al. 2024 and Srećković et al. 2025 show that the performance gap is minimal at small batch size. We have updated the discussion section to make those connections more explicit.

However, we do agree with the reviewers that the submission would be strengthened by experiments that explicitly tie those points together. We have updated the submission with new experiments in Appendix H.

We consider a problem with a controlled level of imbalance, following Wang et al. 2026(a). We generate data containing pairs of "facts", where each "fact" appears with some frequency in the data, following a power law πi∝1/iα\pi_i \propto 1/i^\alpha, and tweak the value of α\alpha. We consider a small transformer model working on Q/A sentences about those facts (e.g. " $Firstname $Lastname live in which city? Answer: $CityName"), and a linear model mapping one-hot encoding of these input and output facts. The model has to memorize the relationship.

With a linear model, the results support the theoretical predictions: the Muon>Adam>SGD-M gaps emerge under frequency imbalance (α≈1\alpha \approx 1), the gaps grow at large batch sizes, and the gap is primarily due to rare facts being learned slowly, which is reflected in the gradient clustering.

Using sentence data with a transformer, we see a similar behavior. The gap is larger at larger batch sizes. The slow training in the imbalanced setting (α=1\alpha=1) corresponds to the lower-frequency facts. And per-sample gradient clustering at training checkpoints shows that Muon reaches fact-level gradient clusters earlier than Adam and SGD, especially in the attention O/V layers. We do however observe that training is slow in the balanced case too, but for a qualitatively different reason; the loss plateaus, as if the optimizer is stuck in a saddle point, where Muon and Adam also seem to help. We suspect the plateau may reflect representation-learning difficulty with uniform data, which has recently been documented by Wang et al. 2026(b), indicating that the training dynamics on deep networks require more nuance.

Together with our earlier findings that the gradient Gram matrices show imbalance across layers, training stages, and models, these results further support gradient cluster imbalance as an important contributing factor to optimizer performance differences in the imbalanced setting at large batch sizes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.