- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the author show a bound for the convergence of under two weaker assumptions that
1/. the activation is continuously differentiable and
2/. the loss is samplewise, non-negative and lipschitz smooth.
Importantly this work does not make particular assumption about the initialization, width, or depth of the network. They evaluate their framework by training dense neural net on the Cifar10 vision task.
Critical strengths
I appreciated the effort done to make this paper accessible to the general deep learning community. Importantly, this works show a decent focus on applicability: this paper derive a bound that works under pragmatic assumptions, which can be applied to realistic training settings.
Also the experiments illustrate the flexibility of the framework which setting involving dynamic learning rates, or weight sharing for instance. Among others, experiment in fig1 shows that this framework can be used to derive practical applications such as a dynamic lr scheduler.
Critical weaknesses
I couldn't review this paper's proof with enough attention, so my observations will mainly be about the experimental section part. Although I see why these experiments can be interesting, this part of the text is very descriptive and miss its important objective: to validate that the bound holds in practice. An easy way to show this could be to plot and do a fit test with a function in In general the experiments are framed as "what works best?" instead of "what are the nominal and limit cases of this framework?". Experiment in fig 1 is a good example of this, as it tells that "adaptive learning rate works better than constant learning rate" where it could illustrate a tightness of condition 9.
Similarly, the experiment in fig2 uses weight repetition, while this setting is correctly motivated, it can also induce confusing factor. Again, tackling the question "would the framework predict this correctly?" would be more valuable than an insight such as "deeper networks train slower if we repeat the weights".
Questions for authors
How does the batch size affect the results ? Given that the proof is for full batch setting, evaluating how the batch size empirically degrades the hypotheses would strengthen the empirical results. Gradient accumulation allows to artificially increase the batch size when the hardware becomes insufficient.
How were the learning rate tuned in the experiments of fig 2 ? An extensive literature shows that optimal lr is width and depth dependent, choosing the same lr for all models can bias results. The same phenomenon can occur with the tuning of alpha in the adaptive setting.
4 comments
Thank you again for your review. As the discussion period is coming to an end, we would like to check if our rebuttal sufficiently addressed your concerns. If so, we would appreciate it if you kindly considered updating your score.
Thank you for the detailed response and additional experiments, which provide valuable insights into the tightness of your bound in a worst-case setting, and also the influence of batch size.
We are glad to hear that our response has addressed your concerns. We kindly ask if you would consider updating your score or review to reflect your current opinion of the work.
Thank you for your review. In the following, we respond to your comments.
Although I see why these experiments can be interesting, this part of the text is very descriptive and miss its important objective: to validate that the bound holds in practice.
We respectfully disagree that the purpose of the experiments is to validate that the bound holds, because the mathematical proof (which is the main contribution of this work) is sufficient to establish the result. Moreover, since the bound is a worst-case analysis over all possible functions, the actual convergence rate in any particular instance may not be precisely 1/T1/L. To illustrate, if in practice the model was initialized at a stationary point, then the algorithm would converge in zero iterations. Instead, our experiments serve to provide additional context for how scaling the learning rate with the parameter norm, as suggested by theory, affects the convergence of gradient descent for a real-world feedforward neural network and dataset. We note that we perform experiments on both standard neural networks and the shared-weights architecture, the latter being a good test case for isolating the impact of depth in this real-world setting.
In addition, we added experiments to the paper illustrating the stated 1/T1/L convergence rate on the hard example used to establish the lower bound. This shows the "limit case" of our theoretical setting where the rate cannot be improved among all deterministic learning rate rules. The plots for these experiments are located in Appendix D.2.
How does the batch size affect the results ? Given that the proof is for full batch setting, evaluating how the batch size empirically degrades the hypotheses would strengthen the empirical results.
Proving theoretical guarantees for the stochastic setting would require significant development of new analytical techniques, so we leave this for future work. However, we ran additional experiments varying the batch size to see how they affect our hypotheses. The plots for these experiments have been added to Appendix D.3. We observe that smaller batch size/increased noise do slow down the convergence, which aligns with theoretical expectations from analyses of SGD/GD on Lipschitz smooth functions. Moreover, for small networks (L=2) and small batch size (32), the advantage of the adaptive learning rate is diminished for earlier iterates. This suggests that the relationship between L and convergence rate may be more complicated in the stochastic regime, and may depend also all the scale of the noise. However, after extended training, we observe that even when the noise causes the constant learning rate iterates to diverge over time, the adaptive learning rates do eventually converge. This suggests that the learning rate should still be scaled with the parameter norm in order to adapt to the non-uniform smoothness of the loss landscape.
**How were the learning rate tuned in the experiments of fig 2 ? **
The appropriate range of learning rates was chosen using a cyclical learning rate finder algorithm on the two-layer neural network, as detailed in Appendix D. Since deeper neural networks tend to require smaller learning rates, this establishes a general upper bound on the constant learning rate, and we tested a range of values designed to converge stably yet quickly for the range of models considered. As expected, larger constant learning rates lead to faster convergence, but as the learning rate is increased, training becomes more unstable. In contrast, the adaptive learning rates show faster and/or more stable convergence than all these constant learning rates, so it is likely that learning rates outside of this range will also underperform in one way or the other.