acceptodds
Under review as a conference paper at ICLR 2027

Online Learning-guided Learning Rate Adaptation via Gradient Alignment

Abstract

The performance of an optimizer on large-scale deep learning models depends critically on *fine-tuning* the learning rate, often requiring an extensive grid search over base learning rates, schedules, and other hyperparameters. We propose GALA (*Gradient Alignment-based Learning rate Adaptation*), a theory-inspired rule that dynamically adjusts the learning rate by tracking the alignment between consecutive gradients and using a local curvature estimate. Guided by the convergence analysis, we formulate the problem of selecting the learning rate as a one-dimensional online learning problem. When paired with an online learning algorithm such as Follow-the-Regularized-Leader, our method produces a flexible, adaptive learning rate schedule that tends to increase when consecutive gradients are aligned and decrease otherwise. We establish a data-adaptive convergence rate for normalized SGD equipped with GALA in the smooth, nonconvex setting. Empirically, common optimizers such as SGD and Adam, when augmented with GALA, demonstrate robust performance across a wide range of initial learning rates and perform competitively without the need for tuning.

Primary area:
optimization
Keywords:
non-convex optimization, learning rate adaptation, online learning, adaptive methods

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_JmvdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GALA, a framework for adapting learning rates using gradient alignment and an estimate of local smoothness. It derives a quadratic surrogate loss for learning-rate selection and applies online learning, principally FTRL, to obtain an explicit update. For a normalized SGD variant with momentum, independent sampling, and optimistic FTRL, the paper establishes a regret-dependent convergence bound with trajectory-dependent smoothness terms and a rate of O~(T−1/4)\widetilde O(T^{-1/4}) in the stochastic setting. Practical SGD and Adam implementations introduce additional heuristics. Experiments cover a series of computer vision tasks. The results demonstrate reduced sensitivity to initialization and competitive accuracy, with noticeable additional runtime.

Critical strengths
  1. A substantive connection between optimization progress and online learning-rate selection. Lemma 3.1 translates function-value progress into a bound involving regret on a scalar quadratic loss. The normalized-momentum analysis further establishes a convergence guarantee for a specific instantiation. This provides a mathematical rationale for combining alignment with curvature regularization, beyond simply proposing an empirical scheduling rule. The dependence on observed smoothness along the trajectory is particularly appealing.

  2. Meaningful empirical evidence of reduced initialization sensitivity. The experiments sweep initial values from 10−810^{-8} to 1, include several relevant adaptive baselines, and report repeated-seed results in the appendix (but error bar is missing). SGD-GALA is especially consistent across this range. The hypergradient-descent comparison and explicit runtime measurements strengthen the evidence that the method addresses a practical tuning problem. These results support the usefulness of the approach, even though they do not fully establish its claimed computational advantage.

Critical weaknesses
  1. The theoretical guarantee does not explain the mechanism essential to the practical results. Section 4 analyzes normalized SGD with momentum, independent samples, a random interpolation point, clipping, and a modified surrogate optimized by optimistic FTRL. The experiments instead use unnormalized updates, different alignment signals, endpoint evaluation, and no clipping. Most consequentially, Section 5.1 states that replacing independent-batch alignment with same-batch alignment is crucial for performance. This replacement changes the statistical meaning of the signal. As the step displacement tends to zero, same-batch alignment approaches E∣g(x;ξ)∣2=∣∇F(x)∣2+E∣g(x;ξ)−∇F(x)∣2\mathbb E\\|g(x;\xi)\\|^2=\\|\nabla F(x)\\|^2+\mathbb E\\|g(x;\xi)-\nabla F(x)\\|^2, so it can remain positive at a stationary point because of gradient noise. The independent-sample construction avoids this variance contribution. Moreover, the practical Adam rule uses alignment between the current gradient and Adam direction, rather than consecutive gradients. Figure 12 investigates endpoint versus midpoint evaluation, but does not isolate these more fundamental changes. Consequently, the theory supports a related algorithm while the strongest empirical claims concern heuristics whose behavior remains insufficiently explained.

  2. The evaluation does not establish the claimed tuning-cost advantage under controlled computational budgets. GALA requires two gradient evaluations per iteration, yet the principal performance comparisons use equal epoch budgets. Table 1 documents the resulting overhead but does not show whether GALA remains competitive at equal wall-clock time or equal gradient-evaluation budgets. The ViT comparison also changes weight decay and scheduling simultaneously: baselines use weight decay 0.05 and cosine schedules, while GALA uses zero weight decay and no external schedule. This prevents clean attribution of accuracy differences to learning-rate adaptation. Additionally, Appendix E.1 describes the initial distance certificate d0d_0 of Prodigy and D-Adaptation as equivalent to an initial learning rate, without sufficiently documenting how these distinct controls were configured. Since reduced tuning effort is the central practical motivation, an initialization sweep alone does not establish a net advantage over a modest baseline search. Together with the theory–implementation gap, this leaves me leaning toward rejection. And this paper does not compare with variance reduction methods, such as STORM[1] and MARS[2], which also requires 2 forward-backward pass but achieves better performance than optimizers like Adam(W). Moreover, the experiments are limited to computer vision tasks, where most of the baseline and proposed optimizers can achieve very high performance and comparison on loss (especially for training acc. and losses) becomes meaningless. It is better to implement experiments on other fields such as LLMs.

[1] Cutkosky A, Orabona F. Momentum-based variance reduction in non-convex sgd[J]. Advances in neural information processing systems, 2019, 32.

[2] Yuan H, Liu Y, Wu S, et al. Mars: Unleashing the power of variance reduction for training large models[J]. arXiv preprint arXiv:2411.10438, 2024.

Questions for authors
  1. How does the exact Section 4 algorithm perform on the reported tasks? Can you isolate the effects of normalization, independent versus same-batch alignment, and optimistic versus ordinary FTRL? Figure 12 addresses only the interpolation choice.

  2. What controls the variance contribution in same-batch alignment near stationary points? Is there an argument that this signal selects an appropriate learning rate for the population objective, rather than primarily responding to minibatch noise?

  3. For Prodigy and D-Adaptation, was the swept parameter d0d_0, the learning-rate multiplier, or both? What exact configurations were used, and were their recommended default configurations included?

  4. Additional evidence needed for assessment: Does GALA retain its advantage with matched weight decay and under an equal total wall-clock budget that includes a small baseline learning-rate search? This directly tests the claimed practical benefit; it is not merely a suggestion for future work.

3 comments

We thank the reviewer for their detailed feedback.


Q1: How does the exact Section 4 algorithm perform on the reported tasks? Can you isolate the effects of normalization, independent versus same-batch alignment, and optimistic versus ordinary FTRL? Figure 12 addresses only the interpolation choice.

A1: We thank the reviewer for their suggestion. We ran the exact implementations of both algorithms to understand the gap between the theory and our heuristic.

To begin with, the difference between FTL and FTRL is negligible; the regularization parameter δ\delta, which plays the same role as ϵ\epsilon in Adam, appears in the denominator of the learning rate and makes sure that its initial value is not undefined due to division by zero. We tested both in our experiments and essentially saw no noticeable difference in practice. Similarly, the optimistic version only adds the hint ⟨gt+1(xt+1),mt+1/∥mt+1∥⟩\langle g_{t+1}(x_{t+1}), m_{t+1}/\|m_{t+1}\| \rangle to the numerator, so it cannot change the behavior we observe below, which is driven by the growth of the denominator.

As for the ablations on the exact implementation of the algorithm, we have two sets of new figures: (i) exact GALA-SGD in Algorithm 1 and (ii) GALA on normalized SGD with momentum (Eq (7-8)) as analyzed in Theorem 4.2. We implemented both algorithms and re-ran the Resnet18/CIFAR-10 experiments exactly as in Section 5, with two different batch sizes: a small batch size of 128 (same as the setup in the manuscript) and a large batch size of 2048. Overall, neither version matches the performance of our heuristic. Algorithm 1 achieves at most 30% (batch size 128) and 47% (batch size 2048) test accuracy, while normalized SGD with momentum experiences a similar gain with the larger batch, but achieves lower accuracy than Algorithm 1. In both cases, the learning rate shrinks significantly in the first epochs. The main reason is the local Lipschitz estimate: since the distance between xtx_t and wtw_t is small due to small learning rate, LtL_t becomes very large, which shrinks the learning rate even further. For normalized SGD with momentum, the denominator additionally grows by 8(1−α)3αL~t\frac{8(1-\alpha)}{3\alpha}\tilde L_t at every step. Since both versions fail at this step, we cannot isolate the effect of normalization in a meaningful way; our heuristic avoids this issue by measuring LtL_t over the full step using the same mini-batch.


Q2: What controls the variance contribution in same-batch alignment near stationary points? Is there an argument that this signal selects an appropriate learning rate for the population objective, rather than primarily responding to minibatch noise?

A2: We thank the reviewer for raising this question. Our use of the same-batch alignment is motivated by empirical observations; while it is intuitive that this modification may control the variance and reduce the impact of minibatch noise, we currently do not have a formal theoretical justification. As discussed in our response above, the additional experiments in Section E.3.2 show that the independent-batch variants suffer from rapid learning-rate shrinkage and substantially underperform the practical implementation. Increasing the batch size from 128 to 2048 improves performance but does not close this gap. Our evidence for the practical variant therefore rests on its observed optimization and test performance, and we will clarify this limitation in the revised manuscript.


Q3: For Prodigy and D-Adaptation, was the swept parameter d0d_0, the learning-rate multiplier, or both? What exact configurations were used, and were their recommended default configurations included?

A3: Both methods, as described in the original papers, use d0d_0 as the initial estimate of the distance to the solution, and keep the learning-rate multiplier fixed at its default value of 1 without tuning, so that d0d_0 acts as the initial learning rate. We followed the exact recipe and ran both methods with d0∈{1,10−1,10−2,10−3,10−4,10−5,10−8}d_0 \in \{1, 10^{-1}, 10^{-2}, 10^{-3}, 10^{-4}, 10^{-5}, 10^{-8} \}, same as every other baseline, with all other parameters at their default values. The recommended default d0=10−6d_0 = 10^{-6} lies within our tested range, and in our sweeps the best performance is not attained at the default: on ResNet-18 with CIFAR-10, both methods perform best at d0=10−3d_0 = 10^{-3}, and for ViT, where we also ran d0=10−6d_0 = 10^{-6} for Prodigy, it performs on par with 10−510^{-5} and 10−810^{-8}.

Q4: Additional evidence needed for assessment: Does GALA retain its advantage with matched weight decay and under an equal total wall-clock budget that includes a small baseline learning-rate search? This directly tests the claimed practical benefit; it is not merely a suggestion for future work.

A4: To answer your question in the correct context, we ran ablations over weight decay and scheduler choice for GALA-Adam. Please refer to Appendix E.3.3, Figure 16 in the updated manuscript for the ablation results. On top of the existing setup for GALA-Adam for ViT training (no weight decay and constant schedule), we ran the following three configurations: (i) weight decay 0.05 + cosine schedule, (ii) weight decay 0.05 + constant schedule, (iii) no weight decay + cosine schedule. As Figure 16 demonstrates, weight decay does not help GALA-Adam improve its performance, but cosine schedule does improve its performance by an important margin.

One could explain the negative effect of weight decay through how it interacts with the online loss. In our implementation, the alignment and LtL_t are computed with the gradients before weight decay is added, while the iterates still move with the decay term.

Indeed, to compare methods on equal terms, we also ran AdamW without weight decay and with a constant schedule. Its best test accuracy remains similar (76.8% vs. 77.6%), but it becomes much more sensitive to the initial learning rate; for instance, at 10−210^{-2} it drops from 48.4% to 18.0%. Therefore, for each baseline and ours, we reported the results based on the setup that yields the best possible performance for each, which we believe was reasonable. We will clarify this better in the final version and include the new experiments on the ablation with discussions to explain the impact of weight decay and cosine schedule in empirical performance.

Regarding your question about GALA’s advantage on equal terms, which we find quite important to highlight, we would like to make an empirically driven assessment. For the ViT experiment, GALA-SGD matches and even surpasses the best test performance of AdamW, while for Resnet-18 runs, GALA-SGD with η0∈{0.1,1}\eta_0 \in \{0.1, 1\} is close to the best performance across all baselines on CIFAR-10, CIFAR-100 and Flower102. Clearly, GALA introduces an additional ~50% computational overhead (Table 1), so a sweep over η0∈{0.1,1}\eta_0 \in \{0.1, 1\} with GALA-SGD costs about the same as 3 runs of SGD. For a single run, the difference is clearer: e.g., on ViT, AdamW yields about 77.5% at 10−410^{-4} but 69% at 10−310^{-3} and 48.7% at 10−210^{-2}, whereas GALA-SGD reaches 77–79% from every η0∈[10−8,1]\eta_0 \in [10^{-8}, 1]. Unfortunately, it is rather difficult to make an exact comparison on this front; however, we believe our experiments demonstrate that GALA’s robustness to initialization and consistent performance from a narrow initialization range make it a strong candidate when the training budget is limited.

Thank you for the additional experiments and for clarifying the baseline configurations. Two concerns remain central to my assessment. First, the new CIFAR-10 results show that the analyzed algorithm’s learning rate shrinks rapidly and its accuracy falls well short of the practical heuristic’s. It remains unclear why same-batch alignment should guide learning-rate selection near stationary points, where minibatch noise can keep the signal positive. Second, the weight-decay and schedule ablations address part of my concern, but the claimed tuning-cost advantage remains untested under a fixed total wall-clock budget that includes a modest baseline learning-rate search. Given these gaps, I maintain my score.

Reviewer_yrR1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces GALA (Gradient Alignment‑based Learning Rate Adaptation), an online learning rate adaptation framework leveraging stochastic gradient alignment and local gradient curvature estimates. Casting learning‑rate tuning as a one‑dimensional online learning task, the authors derive closed‑form updates via FTRL and present regret‑based convergence guarantees for normalized SGD with momentum under smooth stochastic nonconvex settings. Evaluated on SGD and Adam across vision benchmarks (ResNet‑18 on CIFAR‑10/CIFAR‑100/Flowers102, pretrained ViT‑Tiny on Tiny‑ImageNet), GALA yields stronger robustness to initial learning‑rate choices, though practical implementations deviate from the theoretical algorithm with additional heuristics.

Critical strengths
  1. A coherent and potentially useful formulation of learning-rate adaptation.
    The paper derives an online surrogate that combines gradient alignment with a local curvature estimate, rather than introducing another purely heuristic learning-rate schedule. This construction provides an intuitive mechanism for bidirectional adaptation: aligned updates encourage larger learning rates, while misalignment suppresses them. The connection from objective decrease to the online learning formulation is technically interesting and is, in my view, the strongest conceptual contribution of the paper.

  2. The experiments directly evaluate robustness to learning-rate initialization.
    Instead of reporting only the best tuned configuration, the paper evaluates performance across several orders of magnitude of initial learning rates. Across the ResNet experiments, GALA variants are substantially less sensitive to initialization than their corresponding base optimizers. The appendix further includes multi-seed experiments, learning-rate evolution, and runtime measurements, which provide useful empirical support for the robustness motivation.

Critical weaknesses
  1. The empirically successful practical algorithm differs materially from the algorithm supported by the theory.

    The convergence analysis is developed for a normalized-SGD-with-momentum construction with an independent auxiliary stochastic gradient and a particular strongly convex surrogate. In contrast, the practical SGD implementation uses unnormalized updates, endpoint evaluation, same-mini-batch gradient alignment, and removes learning-rate clipping. Importantly, the paper states that using the same mini-batch for the two gradients is crucial for empirical performance, while its theoretical justification is left for future work.

    This difference is substantive because same-batch alignment is not simply an unbiased substitute for the independent-batch alignment used in the analysis. Writing g(x;ξ)=∇F(x)+ε(x;ξ)g(x;\xi)=\nabla F(x)+\varepsilon(x;\xi), the expected same-batch alignment becomes

    E⟨g(y;ξ),g(x;ξ)⟩=⟨∇F(y),∇F(x)⟩+E⟨ε(y;ξ),ε(x;ξ)⟩.\mathbb{E}\langle g(y;\xi),g(x;\xi)\rangle = \langle\nabla F(y),\nabla F(x)\rangle + \mathbb{E}\langle\varepsilon(y;\xi),\varepsilon(x;\xi)\rangle.

    Thus, the practical signal contains an additional noise-covariance term that is absent from the independent-sample argument.

  2. The theoretical guarantee and empirical attribution need additional clarification.
    Algorithm 1 accepts an arbitrary initial learning rate, while the regret analysis later invokes optimistic FTRL and Remark 4.4 interprets the resulting bound as being robust to initialization. It is not clear to me how an arbitrary first-round action is accounted for in the optimistic-FTRL regret argument. This may be fixable, but the current statement should be clarified or corrected.

    In addition, the ViT comparison does not fully isolate the effect of the proposed learning-rate adaptation. The baseline optimizers use cosine scheduling and weight decay 0.050.05, while the GALA variants use a constant schedule and zero weight decay. As a result, the observed performance differences cannot be attributed solely to GALA.

Questions for authors
  1. Clarification on the initialization in the optimistic-FTRL analysis.
    Algorithm 1 allows an arbitrary initial learning rate, while Lemma 4.3 invokes optimistic FTRL and Remark 4.4 concludes that the convergence/regret bounds do not explicitly depend on the initial learning rate. Could the authors clarify how the first-round action is handled in the regret analysis? In particular, is η0\eta_0 intended to be the action prescribed by optimistic FTRL, or may it be chosen arbitrarily? If it may be arbitrary, where is the corresponding first-round regret contribution accounted for?

  2. Clarification on the theory-to-practice gap for same-batch alignment.
    The practical implementation states that using the same mini-batch for the two aligned gradients is crucial for empirical performance, whereas the theoretical construction relies on an independent auxiliary sample. Could the authors provide an ablation comparing independent-batch alignment and same-batch alignment under otherwise matched settings? This would help clarify whether the practical gain comes from the theoretically motivated alignment mechanism or from the additional same-batch gradient correlation.

  3. Matched comparison for the ViT experiment.
    For the ViT experiment, the baseline optimizers use cosine scheduling with weight decay 0.050.05, whereas the GALA variants use a constant schedule and zero weight decay. Could the authors provide a matched comparison in which the base optimizer, weight decay, and other training choices are held fixed, changing only the learning-rate adaptation mechanism? This would make it easier to attribute the observed robustness and performance differences specifically to GALA.

1 comment

We thank the reviewer for their detailed feedback.


Q1: Clarification on the initialization in the optimistic-FTRL analysis.

A1: We thank the reviewer for identifying this omission. Indeed, Lemma 4.3 requires the initial learning rate η0\eta_0 to be prescribed by optimistic FTRL, instead of being arbitrary. Algorithm 1 permits a user-selected initial learning rate, but Lemma 4.3 does not establish the same guarantee for an arbitrary initial action. We have clarified this distinction and revised Remark 4.4 accordingly: the absence of an explicit initialization parameter in the theoretical bound should not be interpreted as a guarantee of robustness to arbitrary initialization. Our evidence for robustness across user-selected initial learning rates is empirical.


Q2: Clarification on the theory-to-practice gap for same-batch alignment.

A2: This is indeed a reasonable way to explain our thinking process while coming up with the heuristics. The main difference between Algorithm 1 and the heuristics introduced in Section 5 is the computation of inner product. In Algorithm 1, the inner product computes ⟨∇f(wt,ξt′),∇f(xt,ξt)⟩\langle \nabla f(w_t, \xi_t'), \nabla f(x_t, \xi_t) \rangle, where the different samples ξt\xi_t and ξt’\xi_t’ are chosen for unbiased estimation but this term is prone to high variance when the batches corresponding to ξt\xi_t and ξt’\xi_t’ are small in size. The heuristic choice ⟨∇f(xt+1,ξt+1),∇f(xt,ξt+1)⟩\langle\nabla f(x_{t+1}, \xi_{t+1}), \nabla f(x_t, \xi_{t+1})\rangle eliminates the noise due to different batches but introduces bias. We essentially trade off bias and the variance of the estimation and observe that low variance heuristic estimate is a more useful signal.

We follow your suggestion and aim to depict the negative effect of variance in the inner product by re-running ResNet-18 with CIFAR-10 experiment with the method as in Algorithm 1 (i.e., without the same-batch heuristic and using the random middle point wsw_s). We ran the same configuration with two different batch sizes 128 and 2048. We expected to see that runs with small batch size will suffer from noise and larger batches will help perform better by detecting the alignment signal better. Figure 14 reports the results of this ablation. We observe that large batch setting yields better performance for the theoretical algorithm, although it is still not as strong as the heuristic we propose and implement. Looking closer, the learning rate collapses early in training, mainly due to the local Lipschitz estimate LtL_t. It is computed over the short distance between xtx_t and wtw_t (due to the small learning rate), and as the stochastic gradient difference might not shrink proportionally to the distance, LtL_t becomes very large. When we cap LtL_t in a short diagnostic run, we see noticeable improvement, but the performance still stays below our heuristic. We deduce that the noise in both LtL_t and the alignment term plays an important role, and the heuristic we propose trades low variance for higher bias, which succeeds in practice.


Q3: Matched comparison for the ViT experiment.

A3: We thank the reviewer for their comment. Indeed, following the suggestion, we ran ablations for GALA-Adam with weight decay and cosine schedule. Please refer to Appendix E.3.3, Figure 16 in the updated manuscript. On top of the existing setup for GALA-Adam on ViT training (no weight decay and constant schedule), we ran the following three configurations: (i) weight decay 0.05 + cosine schedule, (ii) weight decay 0.05 + constant schedule, (iii) no weight decay + cosine schedule. As Figure 16 demonstrates, weight decay does not help GALA-Adam improve its performance, but cosine schedule does.

One could explain the negative effect of weight decay through how it interacts with the online loss. In our implementation, the alignment and LtL_t are computed with the gradients before weight decay is added, while the iterates still move with the decay term.

Indeed, we also ran AdamW without weight decay and with a constant schedule. Its best test accuracy remains similar (76.8% vs. 77.6%), but it becomes much more sensitive to the initial learning rate; for instance, at 10−210^{-2} it drops from 48.4% to 18.0%. Therefore, for each baseline and ours, we reported the results based on the setup that yields the best possible performance for each. We will clarify this better in the final version and include the new experiments on the ablation with discussions to explain the impact of weight decay and cosine schedule in empirical performance.

Reviewer_no56OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an adaptive learning rate schedule Gradient Alignment-based Learning rate Adaptation (GALA) which provides more robust performance of the optimizer across a wide range of initial learning rates. The core observation is: one can write the objective change as a negative expected gradient alignment and bound it by the proposed surrogate loss defined in equation (5). The learning rate update is then obtained by applying online learning algorithm such as Follow-the-Regularized-Leader. The convergence analysis is established with the normalized SGD while the practical implementation includes additional heuristics. Numerical experiments apply GALA with base optimizers like SGD and Adam to train a traditional ML classification models. GALA achieves comparable test accuracy to the best accuracy across wide range of initial learning rate, justifying GALA can save the efforts of hyperparameter tuning for these tasks.

Critical strengths
  1. Presentation is clear and the method is well-motivated. The derivation from the observation of gradient-alignment to the surrogate loss is well-explained and can be followed easily. The authors are also clear on the additional heuristics needed to boost the practical performance.

  2. The numerical results support the claim of robust performance across wide range of initial learning rate. GALA consistently achieves similar accuracy for the learning rate ranging from 10−810^{-8} to 10−110^{-1} or 10−310^{-3} across the presented classification tasks. The evolution of the learning rate is illustrative on how the algorithm adaptively adjust the learning rate for the given problem.

Critical weaknesses
  1. Numerical experiments present only classification task. The ML task considered in the experiment is restricted to 4 classification problems. The classification accuracy is typically less sensitive to the optimality of model parameters, and hence robust test accuracy might not necessarily translate to comparable optimality in terms of optimality gap or gradient norm. It is also not clear whether the same robustness conclusion can be generalized to other ML tasks like regression or other generative models.

  2. GALA does not seem to dominate other adaptive learning rate schedule. The authors compares GALA with other adaptive learning rate schedules in Appendix E. However, it is not clear what the empirical advantages over other existing adaptive methods. Method like D-AdaptAdam seems to be robust across a wide range of learning rate as well. It is not clear in what circumstances, GALA can present its advantages and the practitioner should use GALA instead of other methods.

Questions for authors
  1. Is the robustness to initial learning rate generalizable to other ML tasks other than classification problems?

  2. Does GALA achieve same level of gradient norm as other vanilla optimizers and across different initial learning rate?

  3. Figure 4 shows that GALA does not "converge" to a single learning rate but only stabilizes to a learning rate that is initial learning rate dependent. When will GALA stop increasing/decreasing the learning rate? Are we able to infer which learning rate GALA will eventually arrive at from your analysis?

  4. Can you clarify the advantages of GALA over other adaptive methods? What would be the scenarios that you would recommend practitioners to use GALA rather than other adaptive methods?

3 comments

We thank the reviewer for their detailed feedback.


Q1: Does robustness to initial learning rate extend beyond classification?

A1: We agree that evaluating other tasks, such as regression and generative modeling, would strengthen the empirical evidence. Due to resource constraints, we haven't conducted these additional experiments, so we cannot yet claim that the observed robustness extends beyond classification. That said, we believe that reducing learning-rate tuning across the classification settings studied is a useful contribution to practitioners.


Q2: Does GALA achieve comparable gradient norms across optimizers and initial learning rate?

A2: In general, GALA versions of SGD and Adam achieve good performance in minimizing the gradient norm compared to standalone SGD and Adam, as well as other baselines. We re-ran the Resnet-18 training experiments on CIFAR-10 to record additional statistics.

In Figure 13, we compared the gradient minimization performance of GALA-SGD and GALA-Adam w.r.t. all other baselines. We picked, for each algorithm, the configuration with the best final training accuracy (when several configurations are within 0.5% of the best training accuracy, we pick the one with the highest test accuracy). Then, we re-ran these experiments for all 3 seeds and recorded the full gradient periodically once every 10 epochs. GALA-SGD and GALA-Adam perform consistently well in both training loss and gradient norm: over the last 50 epochs, their gradient norm stays in the same range as Adam, Prodigy and D-Adaptation, and below SGD and Mechanic. On the other hand, AdGD and Mechanic perform well in one metric but not as much in the other.


Q3: When will GALA stop increasing/decreasing the learning rate? Are we able to infer which learning rate GALA will eventually arrive at from your analysis?

A3: This is an interesting observation. Our analysis, which is currently based on FTRL, does not provide an immediate bound on the learning rate evolution. From an empirical standpoint, we do not expect to see learning rates from different initializations converge to the same value. In the initial stages, the learning rate fluctuates to find the correct local scale, during which the trajectories for different initializations likely end up in different basins. Therefore, learning rate evolution in the later stages will differ slightly by following different local landscapes. However, we do have a principled explanation for the increasing behavior in the later stages.

Define at=⟨∇f(xt+1,ξt+1),∇f(xt,ξt+1)⟩a_t = \langle\nabla f(x_{t+1}, \xi_{t+1}), \nabla f(x_{t}, \xi_{t+1})\rangle where At=∑k=0takA_t = \sum_{k=0}^{t} a_k, and bt=Lt∥∇f(xt,ξt)∥2b_t = L_t \| \nabla f(x_t, \xi_t) \|^2 with Bt=∑k=0tbkB_t = \sum_{k=0}^{t} b_k. Then, we could write the step size recursion as ηt+1=(1−btBt)ηt+atBt\eta_{t+1} = \left( 1 - \frac{b_t}{B_t} \right) \eta_t + \frac{a_t}{B_t}. Essentially, the first term scales the learning rate down, while the second term increases or decreases the learning rate depending on the alignment. In the beginning, when BtB_t is small, the learning rate is allowed to fluctuate with respect to how aligned the stochastic gradients are. As we progress, BtB_t gets large such that the alignment magnitude is scaled down by the large BtB_t and the learning rate is more stable. Moreover, since the gradients are larger in the early iterations, they dominate both sums, so the value at which the learning rate stabilizes is impacted by the early phase of training, which differs between initializations. Towards convergence to a local basin, we expect to see better alignment between ∇f(xt+1,ξt+1)\nabla f(x_{t+1}, \xi_{t+1}) and ∇f(xt,ξt+1)\nabla f(x_{t}, \xi_{t+1}). Combined with the fact that (1−btBt)→1\left( 1 - \frac{b_t}{B_t} \right) \to 1, this could explain the increasing behavior towards the end of execution.

We also have one empirical observation. The learning rate of the GALA version of the algorithms, from a wide range of initializations, tends to converge towards the range of values that perform well for the standalone algorithms. For instance, for GALA-Adam on ResNet with CIFAR-100, the learning rates stabilize roughly between 10−410^{-4} and 10−210^{-2}, which is the regime where standalone Adam performs well. In terms of theoretical step size behavior, if we consider the case where gradient norms evolve smoothly, the learning rate for GALA stays roughly between 1Lmax\frac{1}{ L_{\text{max}} } and 1Lmin\frac{1}{L_{\text{min}} }.

Q4: What are GALA’s advantages, and when should practitioners use it?

A4: The core idea behind our work is designing methods that follow a provable principle to increase or decrease the learning rate while being tuning-efficient and robust to initializations. Compared to other adaptive schemes we discuss, i.e., D-Adapt, AdGD, Mechanic and Prodigy, GALA is the only one that has convergence guarantees for non-convex problems; the guarantees of the other methods are for convex problems. Moreover, D-Adaptation and Prodigy rely on an estimate of the distance to the solution that can only increase by construction (dk+1=max⁡(dk,d^k+1)d_{k+1} = \max(d_k, \hat d_{k+1})). Therefore, they can recover from a small initial value but cannot correct a large one, which is consistent with the sharp drop in their performance for large initializations in Figure 2. GALA, on the other hand, can increase or decrease the learning rate depending on the alignment. In addition, one has to introduce a new hyperparameter to make AdGD work for training neural networks, and training with Prodigy may suffer unexpected spikes.

Importantly, GALA is not a standalone algorithm but an external scaler; one might apply it to algorithms like SGD and Adam without interfering with their internal mechanics to compute update directions. For this reason, we recommend using GALA to make exploration efficient. For instance, training a new architecture on a new or custom dataset requires exploring different algorithms and their initializations to achieve the best possible performance, whereas GALA applied to the optimizer of choice would give competitive performance from most initializations, reducing the cost of tuning.

The reviewer thanks the authors for their detailed response. While it resolves most of my concerns about the theoretical insights, my concerns about how GALA is evaluated against other off-the-shelf adaptive step-size methods remain. I believe the paper would benefit from experiments on a wider variety of ML tasks, including at least one regression task, to better support its claims. I therefore maintain my original assessment.

Reviewer_PTbYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provided an adaptive learning rate that is based on an online-learning problem using a suitable algorithm. They provided theoretical analysis on normalized SGD with momentum using independent samples. The empirical results show stable and good convergence performance.

Critical strengths

Online problem derivation with theoretical guarantee: The paper provides an interesting connection between gradient alignment, local curvature estimation, and online learning. The resulting quadratic surrogate objective admits a simple closed-form FTRL update, making the framework mathematically interpretable.

Robust empirical results: Experiments demonstrate that GALA maintains relatively stable performance across various initial learning rates.

Critical weaknesses

Limited experimental scale and generalizability: The experiments focus primarily on ResNet-18 and ViT-Tiny with relatively small image-classification datasets. There are no language-model pretraining experiments or evaluations on larger modern architectures. Given the paper's motivation of reducing learning-rate tuning costs for large-scale deep learning, the empirical evidence is insufficient to establish scalability and practical relevance in such settings.

Questions for authors

NA

1 comment

We thank the reviewer for this comment. We agree that evaluations on larger architectures and tasks such as language-model pretraining would strengthen the empirical evidence. Resource constraints have prevented us from conducting these experiments yet, so our current results do not include large-scale deep learning. Nevertheless, they demonstrate the potential of GALA to reduce sensitivity to learning-rate initialization across the image-classification settings studied, which we believe offers practical value. We will clarify the scope of our empirical conclusions and identify larger-scale evaluation as an important direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.