acceptodds
Under review as a conference paper at ICLR 2027

On the Comparison of Optimizers for Imbalanced Learning

Abstract

Data imbalance is pervasive in machine learning, from rare words and anomalies to underrepresented patterns in heterogeneous or cross-tabulated data. We study idealized optimizers geometries in continuous time to model small-step training in deep learning. We assume that the source of imbalance is unobserved: the optimizer has only access to the aggregate training loss ignoring the exact contributions of the majority and minority groups. In this setting, we characterize a region where majority losses are optimized regardless of the admissible minority structure. We derive explicit equations of this zone and bounds on the time needed to leave it. These bounds exhibit a milder dependence on minority amplitude for sign, spectral, and Newton descent than for Euclidean gradient descent. Experiments with AdamW and Muon suggest similar advantages over SGD across language, tabular, and image tasks.

Primary area:
optimization
Keywords:
optimisation, imbalanced learning, Fairness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1J45OpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how optimizer geometry affects learning under hidden imbalance. The loss is split into an unknown majority L₁ and minority L₀, with only an amplitude ψ bounding the minority’s gradient. Optimizers are idealized as continuous-time flows (gradient descent for SGD, sign descent for Adam, spectral descent for Muon, Newton). The authors define a minority-aware zone, where at least one admissible minority is favored over its majority, and derive upper bounds on the time to enter it: O(ψ⁻²) for gradient descent, O(ψ⁻¹) for sign and spectral descent, and O(log 1/ψ) for Newton. They conclude that normalized optimizers suffer less under severe imbalance. The theory is validated by entry-time ratios on linear regression and by minority/majority accuracy or loss for AdamW, SGD and Muon on Covertype, CIFAR-10 and SmallGPT-2.

I am not an expert in the theory of continuous-time optimization dynamics and was not able to verify all proofs. My assessment therefore focuses on the presentation, the link between theory and experiments, and the experimental setup, and should be read with low confidence.

Critical strengths
  1. A coherent theoretical framework: explicit zone equations for five optimizers, with lower bounds for gradient, sign and spectral descent. This allows a precise comparison of how optimizers react to the minority amplitude.
  2. A relevant question: when minority groups are hidden, reweighting or resampling cannot be applied, which makes optimizer choice a well-motivated alternative.
Critical weaknesses
  1. Weak link between theory and practice.
    1. In my undersatnding for gradient descent, entering the zone simply means the gradient has become small (‖∇L‖ < ψ(‖θ‖ + D)), so the entry time is largely a convergence-speed measure.
    2. The clock comparison rests on κ ≈ 20, but Appendix D reports κ ≈ 700 for language models and ≈ 500/1000 for tabular data.
    3. Only the linear-regression experiment measures entry times. The other experiments use one untuned learning rate per optimizer, 3 seeds, no standard deviations and inconsistent momentum settings.
    4. The practical conclusion (prefer AdamW/Muon over SGD) is already known (Kunstner et al., 2024; Vasudeva et al., 2025).
  2. Incomplete structure. There is no related work and no conclusion; the introduction jumps straight to the method and experiments. The last paragraph of Section 4 reads like an early conclusion, drawn before the experiments. Limitations and future work appear only in Appendix A.1.
Questions for authors
  1. How do the conclusions hold with κ ≈ 700 (Appendix D), and is κ ≈ 500 or ≈ 1000 for tabular data?
  2. Does the entry time capture anything beyond reaching a small gradient norm?
1 comment

We thank the reviewer for these constructive comments and address both questions below.

Q1- How do the conclusions hold with κ≈700\kappa \approx 700 (Appendix D), and is κ≈500\kappa \approx 500 or ≈1000\approx 1000 for tabular data?

As explained in the general comments, we have now refined the definition of κ\kappa. Under this revised definition, the reported values rather range from 11 to 100100. Large ratios typically arise when comparing SGD with AdamW or Muon on deep models. In these settings, geometric factors may partially offset the effect of a larger κ\kappa by reducing the entry-time bounds. Please, observe that our additional experiments in the general answer (Table 1) examine the m∥⋅∥(θ)m_{\Vert \cdot \Vert}(\theta) factor and suggest that they can contribute substantially to the predicted entry-time advantage of sign and spectral descent methods. More experiments on these factors will be provided in our revision.

Q2- Does the entry time capture anything beyond reaching a small gradient norm?

Let us emphasize two reasons why the entry zones and their associated entry times are meaningful:

  • Calibration to the minority scale. The zone thresholds depend explicitly on imbalance, making the required optimization accuracy precise. The entry-time bounds quantify the training needed to reach these zones, while the thresholds may also provide a basis for future stopping criteria.

  • Dependence on geometry. For sign and spectral descent, the criteria involve, respectively,

    \quad\text{and}\quad \frac{\Vert\nabla L\Vert_{\mathrm{nuc}}}{\sqrt{\operatorname{rank}(\nabla L)}}.$$

These criteria can yield larger zones, independently of the choice of clock, and smaller entry-time bounds through the geometry-dependent factor m∥⋅∥(θ)m_{\Vert \cdot \Vert}(\theta). The tables 1 and 3 provide new empirical insights into this effect, as discussed in the general comments and discussion with referee Ge2A.

The factors governing our zones and entry-time bounds thus connect imbalance, optimizer geometry, and training duration, offering, we believe, quantitative insights for future theory and algorithms for imbalanced data.

Reviewer_Ge2AOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper models an imbalanced learning setting in which the identity of the minority group is unknown and only an upper bound on its size is known. In this setting, the paper defines the region in which the optimizer begins to sacrifice the majority for the sake of some admissible minority, termed the minority-aware zone, and derives explicit equations for this region as well as upper and lower bounds on its entry time for several idealized optimizers. Based on these results, the authors argue that normalized geometry exhibits a milder dependence than GD, and support this claim with the observation that AdamW and Muon learn the minority faster than SGD in linear regression, tabular, image, and language model experiments.

Critical strengths
  • This paper presents a general framework for handling imbalance without group labels. It defines minorities as all groups meeting a gradient-magnitude condition, covering ambiguous or multiple minorities such as hidden subclasses and rare tokens. A minimax argument reduces the worst-case condition to a closed-form expression, which the authors apply across optimizers using a unified steepest-descent perspective. Framing optimizer choice as a label-free way to address imbalance also connects to recent work on Adam and Muon’s ability to learn the tail.

  • The theoretical results are rigorous while covering a broad scope, and the authors provides lower bounds matching the upper bounds. Under five dynamics, it establishes upper bounds and lower-bound examples whose order matches the upper bounds, thereby showing that the gap between ψ−2\psi^{-2} and ψ−1\psi^{-1} is not merely an artifact of the proof technique.

Critical weaknesses

W1. It remains unclear whether the key gap between ψ−2\psi^{-2} and ψ−1\psi^{-1} reflects the effect of the optimizer's "geometry" or merely a difference in time scale (clock), as these two factors are not disentangled. Applying Theorem 4.1 directly to the ℓ2\ell_2 norm, normalized GD (∇L/∣∇L∣2\nabla L / |\nabla L|_2) also attains an upper bound of O(ΔL/(ψD))O(\Delta L/(\psi D)) with m=1m=1. However, normalized GD traverses exactly the same trajectory as GD, differing only by a reparameterization of time, and since its minority-aware zone is given by the same equation as that of GD, it enters the zone at the same point along the trajectory. In other words, the advantage in the order of ψ\psi stems not from the point of entry but from "how many seconds it is counted to take to reach that point," and the genuine geometric effect appears to be limited to the enlargement of the zone by factors of supp\sqrt{\text{supp}} and rank\sqrt{\text{rank}}. The authors address this issue by arguing that κ=ηmax⁡/ηmin⁡\kappa = \eta_{\max}/\eta_{\min} is small; however, the time reparameterization factor between GD and normalized methods is not a constant but a state-dependent quantity proportional to ∣∇L∣|\nabla L|, which shrinks to approximately ψD\psi D near the zone, and therefore cannot be controlled by a constant κ\kappa. Moreover, Appendix D reports κ\kappa values of approximately 500 to 1000 for the tabular and language modelling settings, meaning that the justification of κ≈20\kappa \approx 20 in the main text does not hold for precisely these two experimental domains (the main text states κ≈500\kappa \approx 500, whereas the caption of Table 13 states κ≈1000\kappa \approx 1000, so the two are not even consistent with each other).

W2. The implications of the theoretical results for practical optimizers and deep learning losses are limited, whereas the practical recommendations in the conclusion are comparatively strong. The analysis targets idealized dynamics in which momentum, second-moment averaging, weight decay, and minibatch noise are all removed, while the language modelling experiments are conducted with practical optimizers using momentum of 0.9 to 0.95 and β=(0.9,0.999)\beta = (0.9, 0.999), so the theory and the experiments address different objects. Nevertheless, the conclusion offers the practical recommendation that "AdamW or Muon should be used instead of SGD when imbalance is suspected"; since the theory is confined to a comparison of upper bounds and the experiments are subject to the limitations described above, this recommendation is not sufficiently supported.

Questions For Authors

Questions for authors

Q1. Disentangling the effect of geometry from the effect of time reparameterization (clock). Could the authors explain which part of the ψ−1\psi^{-1} advantage of sign descent and spectral descent stems from the norm geometry and which part stems merely from velocity normalization? Could the authors add normalized GD to Table 2 and present a comparison that is independent of the time scale (e.g., a comparison of the loss values at the time of entry or of the entry points themselves)?

Q2. Validity of the κ\kappa argument. (a) The time reparameterization factor between GD and normalized methods is a state-dependent quantity proportional to ∣∇L∣|\nabla L|, which shrinks to approximately ψD\psi D near the zone. Could the authors explain how a constant κ\kappa controls this state-dependent factor? (b) Appendix D reports κ\kappa values of approximately 500 and 700 for the tabular and language modelling settings, respectively (approximately 1000 according to the caption of Table 13), which are far larger than the κ≈20\kappa \approx 20 used as justification in the main text. Is there any basis for regarding the continuous-time comparison as meaningful in these two cases, which constitute the main domains of the experiments?

Q3. Comparison of minority performance at matched majority performance. On CIFAR-10, Muon and AdamW also improve majority accuracy faster than SGD, so the advantage on the minority cannot be distinguished from a difference in overall convergence speed. Could the authors present a comparison of the minority loss and accuracy of each optimizer at the point where they reach the same majority loss (or the same overall training loss)?

Q4. Per-optimizer hyperparameter tuning and statistical reliability. Both the classification and language modelling experiments use only a single learning rate per optimizer. Do the conclusions still hold when the learning rate of each optimizer is swept? In particular, can the authors confirm that the continued increase in the minority loss of SGD in the language modelling experiments is not caused by insufficient tuning of SGD? In addition, could the authors report the variance across seeds, add Muon results on the Covertype dataset, and clarify whether the accuracy reported in Table 9 refers to train or test accuracy?

6 comments

We thank the reviewer for these thoughtful and insightful questions, which have helped us sharpen our analysis and improve the presentation of our work.

Q1. Disentangling the effect of geometry from the effect of time reparameterization (clock). Could the authors explain which part of the ψ−1\psi^{-1} advantage of sign descent and spectral descent stems from the norm geometry and which part stems merely from velocity normalization? Could the authors add normalized GD to Table 2 and present a comparison that is independent of the time scale (e.g., a comparison of the loss values at the time of entry or of the entry points themselves)?

We thank the referee for this important question that pushed us to provide new elements and more explanations as one can see in the general comments.

  • On normalization through continuous times dynamics. Zones are invariant under positive rescaling of the velocity field, provided the zero-velocity set is preserved. Thus, for any given norm, gradient descent and normalized descent have the same zones. Consider now continuous time dynamics and their normalized versions which give the same curves with different time parametrizations: as recalled in the general comments, normalized forms are faster whenever 0<∥dL(θ)∥∗<10<\Vert dL(\theta)\Vert _*<1, slower when ∥dL(θ)∥∗>1\Vert dL(\theta)\Vert _*>1 and yield better entry-time bounds. The question for us is whether the curves they produce can be followed with small step sizes and remain effective. In Deep Learning the answer appears broadly positive, as illustrated by the widespread use of RMSProp, Adam, Muon, and related methods. This empirical evidence motivates our view that normalized continuous-time dynamics are useful models whose trajectories are practically followable through small-step discretizations. This connection to training is what makes the 1/ψ1/\psi dependence of the entry-time bounds a possible indicator of improved performance, despite the mitigation that comes from κ\kappa (see Table 2).

  • On geometry. Geometry is also a decisive factor: the shape of the unit ball determines the gradient structure: it is the phenomenon behind Adam and Muon. In our framework, these optimizers are represented by sign descent and spectral descent. Their entry-time bounds are divided by the pre-entry averages of the square root of gradient support size and gradient rank, respectively (Theorem 4.1 and Table 2). Our ResNet10 experiments (Table 1 of our general response) show a large, stable full-network diagnostic along the AdamW trajectory and a substantial, though varying, matrix-group diagnostic for Muon. These observations motivate further empirical investigation of the role of these geometric factors in imbalanced training. We will provide additional evidences in our revision.

  • Time-independant measures on entry points. For each dataset, we track the entry points (θSDentry,θGDentry)(\theta_{\mathrm{SD}}^{\mathrm{entry}},\theta_{\mathrm{GD}}^{\mathrm{entry}}) and we measure the difference of the following functions (L,L0,∥∇L∥2,∥∇L0∥2)\Big( L, L_0, \Vert \nabla L \Vert _2, \Vert \nabla L_0 \Vert _2 \Big) between the two entry points. Our results are reported in Table 3: at their respective entry points, sign descent generally yields a lower minority-related measure but a higher overall-loss measure than gradient descent. This suggests a more favorable balance for the minority.

Table 3. Normalized paired differences between SD and GD, evaluated at each optimizer's respective entry point. We compute the relative differences and count how often they are positive.

QuantityYearPredictionMSDSuperconductivity
Average of 2(L(θSDentry)−L(θGDentry))L(θSDentry)+L(θGDentry)\frac{2(L(\theta_{\mathrm{SD}}^{\mathrm{entry}}) - L(\theta_{\mathrm{GD}}^{\mathrm{entry}}))}{L(\theta_{\mathrm{SD}}^{\mathrm{entry}}) + L(\theta_{\mathrm{GD}}^{\mathrm{entry}})}0.70.7\\%22\\%
Frequency L(θSDentry)>L(θGDentry)L(\theta_{\mathrm{SD}}^{\mathrm{entry}}) > L(\theta_{\mathrm{GD}}^{\mathrm{entry}})100100\\%63.5863.58\\%
Average of 2(L0(θSDentry)−L0(θGDentry))L0(θSDentry)+L0(θGDentry)\frac{2(L_0(\theta_{\mathrm{SD}}^{\mathrm{entry}}) - L_0(\theta_{\mathrm{GD}}^{\mathrm{entry}}))}{L_0(\theta_{\mathrm{SD}}^{\mathrm{entry}}) + L_0(\theta_{\mathrm{GD}}^{\mathrm{entry}})}−5-5\\%−63-63\\%
Frequency L0(θSDentry)>L0(θGDentry)L_0(\theta_{\mathrm{SD}}^{\mathrm{entry}}) > L_0(\theta_{\mathrm{GD}}^{\mathrm{entry}})36.1136.11\\%10.9810.98\\%
Average of 2(∥∇L(θSDentry)∥2−∥∇L(θGDentry)∥2)∥∇L(θSDentry)∥2+∥∇L(θGDentry)∥2\frac{2(\Vert \nabla L(\theta_{\mathrm{SD}}^{\mathrm{entry}})\Vert_2 - \Vert \nabla L(\theta_{\mathrm{GD}}^{\mathrm{entry}})\Vert_2)}{\Vert \nabla L(\theta_{\mathrm{SD}}^{\mathrm{entry}})\Vert_2 + \Vert \nabla L(\theta_{\mathrm{GD}}^{\mathrm{entry}})\Vert_2}6363\\%135135\\%
Frequency ∥∇L(θSDentry)∥2>∥∇L(θGDentry)∥2\Vert \nabla L(\theta_{\mathrm{SD}}^{\mathrm{entry}})\Vert_2 > \Vert \nabla L(\theta_{\mathrm{GD}}^{\mathrm{entry}})\Vert_2100100\\%100100\\%
Average of 2(∥∇L0(θSDentry)∥2−∥∇L0(θGDentry)∥2)∥∇L0(θSDentry)∥2+∥∇L0(θGDentry)∥2\frac{2(\Vert \nabla L_0(\theta_{\mathrm{SD}}^{\mathrm{entry}})\Vert_2 - \Vert \nabla L_0(\theta_{\mathrm{GD}}^{\mathrm{entry}})\Vert_2)}{\Vert \nabla L_0(\theta_{\mathrm{SD}}^{\mathrm{entry}})\Vert_2 + \Vert \nabla L_0(\theta_{\mathrm{GD}}^{\mathrm{entry}})\Vert_2}−2-2\\%−65-65\\%
Frequency ∥∇L0(θSDentry)∥2>∥∇L0(θGDentry)∥2\Vert \nabla L_0(\theta_{\mathrm{SD}}^{\mathrm{entry}})\Vert_2 > \Vert \nabla L_0(\theta_{\mathrm{GD}}^{\mathrm{entry}})\Vert_242.7842.78\\%10.9810.98\\%

Q2. Validity of the κ\kappa argument. (a) The time reparameterization factor between GD and normalized methods is a state-dependent quantity proportional to $\vert \nabla L\vert $, which shrinks to approximately ψD\psi D near the zone. Could the authors explain how a constant κ\kappa controls this state-dependent factor? (b) Appendix D reports κ\kappa values of approximately 500500 and 700700 for the tabular and language modelling settings, respectively (approximately 10001000 according to the caption of Table 13), which are far larger than the κ≈20\kappa \approx 20 used as justification in the main text. Is there any basis for regarding the continuous-time comparison as meaningful in these two cases, which constitute the main domains of the experiments?

(a) We have now provided a more precise definition of κ\kappa in the general comments. Its role nevertheless remains empirical which is meant to support our idealized view of this complex problem, to make this more concrete; we have added a table to document its values. In our discussions, we use κ\kappa as a proxy to bound the relative scaling of fixed-step-size clocks through t≃ηkt\simeq\eta k, not the state-dependent time change between GD and normalized GD.

(b) With the more controlled definition of κ\kappa introduced in the general comments, these large values do not arise in our experimental protocols: the learning-rate ratios instead range from 11 to 100100. As expected, higher ratios tend to occur when comparing SGD with normalized methods. However, larger SGD step sizes tend to cause training to collapse before comparable accuracy is reached, so these larger ratios are also accompanied by poorer convergence of SGD.

Q3. Comparison of minority performance at matched majority performance. On CIFAR-10, Muon and AdamW also improve majority accuracy faster than SGD, so the advantage on the minority cannot be distinguished from a difference in overall convergence speed. Could the authors present a comparison of the minority loss and accuracy of each optimizer at the point where they reach the same majority loss (or the same overall training loss)?

We thank the reviewer for highlighting the importance of distinguishing minority-class learning speed from overall training speed. To address this point, we compare optimizers at similar levels of majority-class performance rather than at the same iteration. Specifically, we report cumulative average training accuracies for the majority (L1L_1) and minority (L0L_0) groups when cumulative average L1L_1 accuracy is closest to each target threshold.

Table 4. ResNet-10 experiments on CIFAR. Cumulative average L1/L0L_1/L_0 training accuracy at the iterations where cumulative average L1L_1 accuracy is closest to 7070\\%, 8080\\%, 9090\\%, and 9595\\%.

ThresholdAdamWSGDMuon
Threshold = 70%67.83% / 55.37%69.37% / 39.27%× / ×
Threshold = 80%80.17% / 62.44%80.56% / 61.29%79.53% / 63.07%
Threshold = 90%89.86% / 81.85%90.06% / 79.64%89.51% / 77.20%
Threshold = 95%94.98% / 91.09%95.02% / 89.81%94.88% / 88.90%

Table 4 suggests that the minority-class learning advantage of normalized methods is most evident early in training. AdamW achieves substantially higher minority-class accuracy than SGD near the 7070\\% threshold, while both AdamW and Muon show modest advantages near 8080\\%. At the higher thresholds, AdamW retains a small advantage over SGD, whereas Muon falls slightly behind. These results suggest that AdamW's gains include a benefit to minority-class learning beyond overall convergence, while Muon's gains appear more closely tied to faster overall convergence.

Reviewer_YKcSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper provides a method of analyzing the behaviour of (a proxy of) common deep learning optimizers under a general form of imbalance in the dataset. There has been a decent amount of previous work on more explicit imbalance, such as class imbalance in next token prediction LLMs, but these authors create a more general form where imbalance could be defined in the labels or features or a subset of the input data. For their analysis, they assume that because learning rates are usually small, they can use a continuous time approximation and consider each optimizer as the flow defined by an ODE. Their main goal is define the conditions in which an optimizer reaches a “minority aware zone” and how long that takes. Intuitively this zone is where the optimizer will move the parameters in a direction which may increase the loss of the majority component of the loss. From there, the authors characterize the continuous time version of a few different modern optimizers such as GD, Adam, and Muon. Their first result regards the characterization of the minority aware zones, that is a condition on some norm of the gradient of the full loss with respect to the parameters that implies the optimizer has reached a minority aware zone. The second main result concerns how long reaching those zones takes, where the entry time as defined as the lower bounding time value in which the parameters lead to satisfying the zone condition. The authors conclude with a few experiments trying to verify their results. These include attempting to verify the entry time results on a simple model, verifying a result that normalized optimizers perform better in a class imbalanced setting, and showing that gradient descent struggles in heavily imbalanced settings.

Critical strengths

This paper studies a very relevant issue, much of modern optimizer development hinges on managing the differing characteristics of directions of the loss surface of machine learning problems, many of which are due to imbalances in the data. The optimization methods they study are relevant to practitioners. The results appear to be sound based on my reading, although at times unclear.

Critical weaknesses

The main weakness of this paper in my view is the organization and presentation of it. It seems to me the authors have focused very much on presenting their results, but give little room for interpreting those results and making them a useful lesson for either the theoretical or practical community. The majority of the paper involves (somewhat terse) definitions which occasionally have an intuitive explanation but often not, followed by results without proof or intuition of proof. The authors refer the reader to the appendix, but it is customary that at the very least an intuition of the proof if not a full proof sketch or simplified case be shown. This is particularly important in a work like this where the theory is the primary contribution and the referenced proofs are long and complex. I would additionally note that the appendix itself could be better organized, for example A.1 is simply listing bullet points rather than focusing on the actual content, a summary of the contributions and future work. Some standard criticism also applies, for example I find the claim that a small learning rate justifies the use of continuous time to be quite strong, it is well known that even an ϵ\epsilon too large learning rate can lead to divergence of a method, indicating a strong dependence on the learning rate even at small magnitude.

At this point I will vote for a weak acceptance given I think the issues with this paper are largely in the presentation rather than the results, but I strongly suggest the authors review their manuscript for the education of the community.

Questions for authors
  • In Figure 1, is the optimizer run as in practice, or are the authors integrating the ODEs they are describing along this trajectory?
  • Can the authors provide a brief proof sketch or intuition for at least one of their main theorems?
  • In the experiments, Appendix C.2 states the learning rates used in practice are actually higher than 10−210^{-2} as described in the main text, do the authors feel that this could contradict their experimental results? Do the results still hold for a vanishing learning rate?
  • Do the authors have any recommendations on how these insights could be used to explain the effectiveness of non-GD optimizers in settings without imbalance or do they predict that all settings have some hidden imbalance?
3 comments

We thank the reviewer for the constructive feedback and for recognizing the relevance and soundness of the results. We address the questions below and will make the theoretical mechanisms more explicit in the main text.

Q1- In Figure 1, is the optimizer run as in practice, or are the authors integrating the ODEs they are describing along this trajectory?

Figure 1 illustrates the zones defined in our paper alongside discrete optimization trajectories computed using a step size of 5e-3 along the gradient-descent and sign-descent.

Q2- Can the authors provide a brief proof sketch or intuition for at least one of their main theorems?

Yes, we illustrate this using the minority awareness zone equations, which are central to our results. The formula follows in two steps. Lemma 3.3 shows that admissible minority gradients fill the ball of radius ψ(∥θ∥2+D)\psi(\Vert \theta\Vert_2+D). Thus,

Q(θ)=min⁡∥ξ∥2≤ψ(∥θ∥2+D)max⁡v∈VL(θ)⟨v,∇L(θ)−ξ⟩.\mathcal Q(\theta)=\min_{\Vert \xi\Vert_2\leq\psi(\Vert \theta\Vert_2+D)}\max_{v\in V_L(\theta)}\langle v,\nabla L(\theta)-\xi\rangle.

By a classical convex-concave minimax argument, this equals

\mathcal Q(\theta)=\max_{v\in V_L(\theta)}\bigl\\{\langle v,\nabla L(\theta)\rangle-\psi(\Vert \theta\Vert_2+D)\Vert v\Vert_2\bigr\\}.

This is Theorem 3.4. For GD, VL(θ)=∇L(θ)V_L(\theta)=\\{\nabla L(\theta)\\}, so Q(θ)<0\mathcal Q(\theta)<0 becomes

0<∥∇L(θ)∥2<ψ(∥θ∥2+D).0<\Vert \nabla L(\theta)\Vert_2<\psi(\Vert \theta\Vert_2+D).

Within the zone, one admissible minority decreases at its majority's expense under every permitted update.

Q3- In the experiments, Appendix C.2 states the learning rates used in practice are actually higher than as described in the main text, do the authors feel that this could contradict their experimental results? Do the results still hold for a vanishing learning rate?

The introduction was unclear and was using vision tasks to illustrate learning-rate scales and referred to Appendix D for other settings. As explained in the general comments we will clarify this distinction with a clearer definition of κ\kappa, please see general comments for thorough answers.

The rates used in our experiments do not contradict the reported results; they show that a single numerical threshold is not an adequate justification for the continuous-time model.

Our theorems concern continuous dynamics, not convergence of discrete entry times as the step size vanishes. Similarly our experiments are empirical validations in the small range step size or with SOTA stepsizes so there is no contradiction as well.

We believe that our results may extend to vanishing step sizes. However, establishing such results is not necessarily easier than for nonvanishing step sizes, since the zones may be reached well before the steps become genuinely small. We are currently investigating this direction.

Q4- Do the authors have any recommendations on how these insights could be used to explain the effectiveness of non-GD optimizers in settings without imbalance or do they predict that all settings have some hidden imbalance?

This is a very interesting and deep question that we cannot yet answer. Under moderate imbalance, methods may favor majority groups, as studied for gradient descent in Bachoc et al (2025). It is also possible that normalized methods based on the ℓ∞\ell_\infty or spectral norm, such as sign descent and spectral descent, promote more balanced learning throughout training. At this stage, however, these possibilities remain speculative.

I thank the authors for their clarifications and discussion of how they will revise the paper. Given my main concerns of the paper were initially structural and I had supported acceptance I will keep that rating. I do hope to see the changes in structure the authors have discussed in a future revision but I do think the state of the paper as is does not warrant more than a weak acceptance.

We thank the reviewer for her/his thorough assessment, we will do our best to improve our presentation along the lines we presented in this rebuttal.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers - Part 1/2

We thank the reviewers for their thoughtful and constructive reviews.

1. Presentation We agree with the referees that the presentation needs to be clearer. We will give greater prominence to the underlying intuitions and develop the discussion of our conclusions. These revisions will be reflected throughout the paper, including an augmented conclusion. We shall center our presentation on the following contributions of our work:

  • Provide a mathematical formulation of an idealized imbalanced-training problem with explicit equations for the minority-awareness region

  • Bounds on the time needed to reach such a zone

  • Identification of factors that may facilitate more balanced training, in particular, more insights into the geometric factor associated with gradient support and matrix rank.

Our conclusion will be largely developped in these directions. Let us discuss more carefully the roles of the factors: κ\kappa, ψ\psi, and m ∥⋅∥(θ)m_{\:\Vert \cdot\Vert }(\theta).

2. Clock comparisons and the factor κ\kappa: Our previous empirical discussion of κ\kappa was too limited and insufficiently controlled. As noted by some of the reviewers, a meaningful comparison requires a fixed objective, dataset, and architecture, together with comparable convergence quality and training budget. Table 2 reports comparisons on real-world examples of various origins.

3. GD, normalized GD, and the factor ψ\psi

For a smooth loss LL, gradient flow and normalized gradient flow have the same trajectories, GD moves faster where ∥∇L∥>1\Vert \nabla L\Vert >1 and slower where ∥∇L∥<1\Vert \nabla L\Vert <1. The powers ψ−2\psi^{-2} and ψ−1\psi^{-1} appearing in the respective entry-time bounds reflect this difference in speed along trajectories.

Our hypothesis is that when the step sizes are sufficiently small their discrete iterations can be expected to resemble the corresponding continuous trajectories with `comparable' discrete-to-continuous clocks. Under this approximation, the different entry-time factors provide an indication of relative performance in reaching the minority-awareness region. See Table 2.

4. The geometric factor mθm_\theta

Recall (Theorem 4.1, Equation (16)) the `M factors'

\leq \frac{L(\theta_{\mathrm{init}})-L_\star}{\psi D \overline M_\theta}, \qquad \overline M_\theta =\frac{1}{t_{\mathrm{entry}}(\theta)} \int_0^{t_{\mathrm{entry}}(\theta)} m_{\Vert \cdot\Vert }(\theta(s))ds.$$ For sign and spectral descent, it is the average square root of gradient support size and gradient rank, respectively. As shown in Table 1 these have considerable impact on entry times (further experiments will be provided in a revision). **Table 1.** $M$ factors for ResNet-10 on CIFAR-10 across three random seeds. For simplicity, the Muon diagnostic excludes parameters updated by the auxiliary AdamW optimizer. | | AdamW | Muon | | --- | --- | --- | | Epoch | $\bar{M}_{\mathrm{GD}}$ | $\bar{R}$ | | 5 | $1468.98 \pm 0.00$ | $47.39 \pm 0.21$ | | 10 | $1468.98 \pm 0.00$ | $48.21 \pm 0.08$ | | 15 | $1468.98 \pm 0.00$ | $47.44 \pm 0.22$ | | 20 | $1468.98 \pm 0.00$ | $47.35 \pm 0.39$ | | 25 | $1468.98 \pm 0.00$ | $45.85 \pm 0.15$ | | 30 | $1468.98 \pm 0.00$ | $41.96 \pm 0.62$ | | 35 | $1468.98 \pm 0.00$ | $34.53 \pm 0.35$ | | 40 | $1468.98 \pm 0.00$ | $36.87 \pm 0.97$ | | 45 | $1468.98 \pm 0.00$ | $40.79 \pm 0.53$ | | 50 | $1468.98 \pm 0.00$ | $42.81 \pm 1.48$ |
AuthorsOpenReview ↗

General response to all reviewers - Part 2/2

Table 2. Selected learning rates and reported outcomes on shared datasets, architectures, and training budgets.

Dataset/model; budgetOptimizers A/BA/BRates ηA/ηB\eta_A/\eta_BκLR\kappa_{\mathrm{LR}}Outcome A/BA/B
Image classification: top-1 accuracy (%)
CIFAR-10/ResNet-18/300 ep.; BN affine off [1]SGD / Adam0.10{.}1 / 0.0010{.}001100.00100{.}0091.80/93.4091{.}80/93{.}40
CIFAR-100/ResNet-18/300 ep.; BN affine off [1]SGD / Adam0.10{.}1 / 0.0010{.}001100.00100{.}0070.90/70.2070{.}90/70{.}20
ImageNet/ResNet-50/90 ep.; BN affine off [1]SGD / Adam0.10{.}1 / 0.010{.}0110.0010{.}0075.90/73.3075{.}90/73{.}30
ImageNet/ResNet-50/100 epochs [2]SGD / Adam0.10{.}1 / 0.070{.}071.431{.}4374.90/73.9574{.}90/73{.}95
ImageNet/ResNet-50/100 epochs [3]Muon/SGD0.10{.}1 / 0.050{.}052.002{.}00SGD: Fig. 7;/Muon: 76.9776{.}97
GPT-2 and larger language models: validation loss
OpenWebText/GPT-2-125M/ 100 000100\,000 steps [4]AdamW / Lion6 10−46\,10^{-4} / 1.5 10−41{.}5\,10^{-4}4,004{,}002,921/2,9242{,}921/2{,}924
OpenWebText/GPT-2-355M/100 000100\,000 steps [4]Sophia-G /Lion4 10−44\,10^{-4} / 6 10−56 \,10^{-5}6.666{.}662,627/2,6912{,}627/2{,}691
The Pile/GPT-NeoX 6.6B/100 000100\,000 steps [4]AdamW /Sophia-G1.2 10−41{.}2\,10^{-4} / 6 10−56\,10^{-5}2.002{.}001,992/1,9691{,}992/1{,}969
Llama pretraining: C4-EN loss
DCLM/code/math/Llama 1.2B/48B tokens [5]Muon/ AdamW0.0040{.}004 / 0.0020{.}0022.002{.}002,827/2,8362{,}827/2{,}836
DCLM/code/math/Llama 1.2B/193B tokens [5]Muon/AdamW0.0040{.}004 / 0.0020{.}0022.002{.}002,748/2,7522{,}748/2{,}752
DCLM/code/math/ Llama 1.2B/193B tokens [5]SOAP/AdamW0.0040{.}004 / 0.0020{.}0022.002{.}002,749/2,7522{,}749/2{,}752

  • C4-EN: English Colossal Clean Crawled Corpus.

  • DCLM/code/math DataComp for Language Models, StarCoder Data, and ProofPile 2.

Bibliography

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.