- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how optimizer geometry affects learning under hidden imbalance. The loss is split into an unknown majority L₁ and minority L₀, with only an amplitude ψ bounding the minority’s gradient. Optimizers are idealized as continuous-time flows (gradient descent for SGD, sign descent for Adam, spectral descent for Muon, Newton). The authors define a minority-aware zone, where at least one admissible minority is favored over its majority, and derive upper bounds on the time to enter it: O(ψ⁻²) for gradient descent, O(ψ⁻¹) for sign and spectral descent, and O(log 1/ψ) for Newton. They conclude that normalized optimizers suffer less under severe imbalance. The theory is validated by entry-time ratios on linear regression and by minority/majority accuracy or loss for AdamW, SGD and Muon on Covertype, CIFAR-10 and SmallGPT-2.
I am not an expert in the theory of continuous-time optimization dynamics and was not able to verify all proofs. My assessment therefore focuses on the presentation, the link between theory and experiments, and the experimental setup, and should be read with low confidence.
Critical strengths
- A coherent theoretical framework: explicit zone equations for five optimizers, with lower bounds for gradient, sign and spectral descent. This allows a precise comparison of how optimizers react to the minority amplitude.
- A relevant question: when minority groups are hidden, reweighting or resampling cannot be applied, which makes optimizer choice a well-motivated alternative.
Critical weaknesses
- Weak link between theory and practice.
- In my undersatnding for gradient descent, entering the zone simply means the gradient has become small (‖∇L‖ < ψ(‖θ‖ + D)), so the entry time is largely a convergence-speed measure.
- The clock comparison rests on κ ≈ 20, but Appendix D reports κ ≈ 700 for language models and ≈ 500/1000 for tabular data.
- Only the linear-regression experiment measures entry times. The other experiments use one untuned learning rate per optimizer, 3 seeds, no standard deviations and inconsistent momentum settings.
- The practical conclusion (prefer AdamW/Muon over SGD) is already known (Kunstner et al., 2024; Vasudeva et al., 2025).
- Incomplete structure. There is no related work and no conclusion; the introduction jumps straight to the method and experiments. The last paragraph of Section 4 reads like an early conclusion, drawn before the experiments. Limitations and future work appear only in Appendix A.1.
Questions for authors
- How do the conclusions hold with κ ≈ 700 (Appendix D), and is κ ≈ 500 or ≈ 1000 for tabular data?
- Does the entry time capture anything beyond reaching a small gradient norm?
We thank the reviewer for these constructive comments and address both questions below.
Q1- How do the conclusions hold with κ≈700 (Appendix D), and is κ≈500 or ≈1000 for tabular data?
As explained in the general comments, we have now refined the definition of κ. Under this revised definition, the reported values rather range from 1 to 100. Large ratios typically arise when comparing SGD with AdamW or Muon on deep models. In these settings, geometric factors may partially offset the effect of a larger κ by reducing the entry-time bounds. Please, observe that our additional experiments in the general answer (Table 1) examine the m∥⋅∥(θ) factor and suggest that they can contribute substantially to the predicted entry-time advantage of sign and spectral descent methods. More experiments on these factors will be provided in our revision.
Q2- Does the entry time capture anything beyond reaching a small gradient norm?
Let us emphasize two reasons why the entry zones and their associated entry times are meaningful:
Calibration to the minority scale. The zone thresholds depend explicitly on imbalance, making the required optimization accuracy precise. The entry-time bounds quantify the training needed to reach these zones, while the thresholds may also provide a basis for future stopping criteria.
Dependence on geometry. For sign and spectral descent, the criteria involve, respectively,
\quad\text{and}\quad \frac{\Vert\nabla L\Vert_{\mathrm{nuc}}}{\sqrt{\operatorname{rank}(\nabla L)}}.$$These criteria can yield larger zones, independently of the choice of clock, and smaller entry-time bounds through the geometry-dependent factor m∥⋅∥(θ). The tables 1 and 3 provide new empirical insights into this effect, as discussed in the general comments and discussion with referee Ge2A.
The factors governing our zones and entry-time bounds thus connect imbalance, optimizer geometry, and training duration, offering, we believe, quantitative insights for future theory and algorithms for imbalanced data.