- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies heavy-ball's effective angular learning rate and suggests that there is no free lunch. It also proposes a two phase leanring rate schedule, where the first phase holds lr constant and second phase linearly decays leanring rate. The switching trigger is based on stability of gradient norm.
Critical strengths
- The insights about the angular effective learning rate is quite interesting.
Critical weaknesses
- I believe there is a gap in motivation of ELR-SD. it's either not stated clearly or it's not been established why we need to keep angular effective lr the consistent with outside scheduling. Especially,
"Thus, the scale effective learning rate can guide learning-rate scheduling for loss reduction nearly as effectively as the angular effective learning rate. Together, these results suggest that Hyperball alone may not be a free lunch: improving performance requires a more effective learning-rate schedule."
Frankly, I do not see how these experiments lead to the said conclusion. It also has not established why angular changes of weights are the more important aspect in loss reduction.
-
The empirical evidence feels inconclusive, for example figure 5, why was it not run longer, if the claims sustained it should be a clear win?
-
The gradient instability trigger thresholds seem arbitrary and there is neither ablation nor explanation in the paper nor in the appendix. Why is it defined based on gradient norm, but not the angular changes?
-
The theorem 7.1 provides no justification for any of the optimizers: The example in Appendix C constructs multiple projected Riemannian gradient learners, each with a fixed step size, and combines them through an exponential weighting rule. It does not establish this guarantee for MuonH, AdamH, or ELR-SD.
Questions for authors
I am a bit confused by what it means by angular effective learning rate at j-th stage in equation 12? what is j and where was it defined?