- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes GALA, a framework for adapting learning rates using gradient alignment and an estimate of local smoothness. It derives a quadratic surrogate loss for learning-rate selection and applies online learning, principally FTRL, to obtain an explicit update. For a normalized SGD variant with momentum, independent sampling, and optimistic FTRL, the paper establishes a regret-dependent convergence bound with trajectory-dependent smoothness terms and a rate of in the stochastic setting. Practical SGD and Adam implementations introduce additional heuristics. Experiments cover a series of computer vision tasks. The results demonstrate reduced sensitivity to initialization and competitive accuracy, with noticeable additional runtime.
Critical strengths
-
A substantive connection between optimization progress and online learning-rate selection. Lemma 3.1 translates function-value progress into a bound involving regret on a scalar quadratic loss. The normalized-momentum analysis further establishes a convergence guarantee for a specific instantiation. This provides a mathematical rationale for combining alignment with curvature regularization, beyond simply proposing an empirical scheduling rule. The dependence on observed smoothness along the trajectory is particularly appealing.
-
Meaningful empirical evidence of reduced initialization sensitivity. The experiments sweep initial values from to 1, include several relevant adaptive baselines, and report repeated-seed results in the appendix (but error bar is missing). SGD-GALA is especially consistent across this range. The hypergradient-descent comparison and explicit runtime measurements strengthen the evidence that the method addresses a practical tuning problem. These results support the usefulness of the approach, even though they do not fully establish its claimed computational advantage.
Critical weaknesses
-
The theoretical guarantee does not explain the mechanism essential to the practical results. Section 4 analyzes normalized SGD with momentum, independent samples, a random interpolation point, clipping, and a modified surrogate optimized by optimistic FTRL. The experiments instead use unnormalized updates, different alignment signals, endpoint evaluation, and no clipping. Most consequentially, Section 5.1 states that replacing independent-batch alignment with same-batch alignment is crucial for performance. This replacement changes the statistical meaning of the signal. As the step displacement tends to zero, same-batch alignment approaches , so it can remain positive at a stationary point because of gradient noise. The independent-sample construction avoids this variance contribution. Moreover, the practical Adam rule uses alignment between the current gradient and Adam direction, rather than consecutive gradients. Figure 12 investigates endpoint versus midpoint evaluation, but does not isolate these more fundamental changes. Consequently, the theory supports a related algorithm while the strongest empirical claims concern heuristics whose behavior remains insufficiently explained.
-
The evaluation does not establish the claimed tuning-cost advantage under controlled computational budgets. GALA requires two gradient evaluations per iteration, yet the principal performance comparisons use equal epoch budgets. Table 1 documents the resulting overhead but does not show whether GALA remains competitive at equal wall-clock time or equal gradient-evaluation budgets. The ViT comparison also changes weight decay and scheduling simultaneously: baselines use weight decay 0.05 and cosine schedules, while GALA uses zero weight decay and no external schedule. This prevents clean attribution of accuracy differences to learning-rate adaptation. Additionally, Appendix E.1 describes the initial distance certificate of Prodigy and D-Adaptation as equivalent to an initial learning rate, without sufficiently documenting how these distinct controls were configured. Since reduced tuning effort is the central practical motivation, an initialization sweep alone does not establish a net advantage over a modest baseline search. Together with the theory–implementation gap, this leaves me leaning toward rejection. And this paper does not compare with variance reduction methods, such as STORM[1] and MARS[2], which also requires 2 forward-backward pass but achieves better performance than optimizers like Adam(W). Moreover, the experiments are limited to computer vision tasks, where most of the baseline and proposed optimizers can achieve very high performance and comparison on loss (especially for training acc. and losses) becomes meaningless. It is better to implement experiments on other fields such as LLMs.
[1] Cutkosky A, Orabona F. Momentum-based variance reduction in non-convex sgd[J]. Advances in neural information processing systems, 2019, 32.
[2] Yuan H, Liu Y, Wu S, et al. Mars: Unleashing the power of variance reduction for training large models[J]. arXiv preprint arXiv:2411.10438, 2024.
Questions for authors
-
How does the exact Section 4 algorithm perform on the reported tasks? Can you isolate the effects of normalization, independent versus same-batch alignment, and optimistic versus ordinary FTRL? Figure 12 addresses only the interpolation choice.
-
What controls the variance contribution in same-batch alignment near stationary points? Is there an argument that this signal selects an appropriate learning rate for the population objective, rather than primarily responding to minibatch noise?
-
For Prodigy and D-Adaptation, was the swept parameter , the learning-rate multiplier, or both? What exact configurations were used, and were their recommended default configurations included?
-
Additional evidence needed for assessment: Does GALA retain its advantage with matched weight decay and under an equal total wall-clock budget that includes a small baseline learning-rate search? This directly tests the claimed practical benefit; it is not merely a suggestion for future work.
3 comments
Q4: Additional evidence needed for assessment: Does GALA retain its advantage with matched weight decay and under an equal total wall-clock budget that includes a small baseline learning-rate search? This directly tests the claimed practical benefit; it is not merely a suggestion for future work.
A4: To answer your question in the correct context, we ran ablations over weight decay and scheduler choice for GALA-Adam. Please refer to Appendix E.3.3, Figure 16 in the updated manuscript for the ablation results. On top of the existing setup for GALA-Adam for ViT training (no weight decay and constant schedule), we ran the following three configurations: (i) weight decay 0.05 + cosine schedule, (ii) weight decay 0.05 + constant schedule, (iii) no weight decay + cosine schedule. As Figure 16 demonstrates, weight decay does not help GALA-Adam improve its performance, but cosine schedule does improve its performance by an important margin.
One could explain the negative effect of weight decay through how it interacts with the online loss. In our implementation, the alignment and are computed with the gradients before weight decay is added, while the iterates still move with the decay term.
Indeed, to compare methods on equal terms, we also ran AdamW without weight decay and with a constant schedule. Its best test accuracy remains similar (76.8% vs. 77.6%), but it becomes much more sensitive to the initial learning rate; for instance, at it drops from 48.4% to 18.0%. Therefore, for each baseline and ours, we reported the results based on the setup that yields the best possible performance for each, which we believe was reasonable. We will clarify this better in the final version and include the new experiments on the ablation with discussions to explain the impact of weight decay and cosine schedule in empirical performance.
Regarding your question about GALA’s advantage on equal terms, which we find quite important to highlight, we would like to make an empirically driven assessment. For the ViT experiment, GALA-SGD matches and even surpasses the best test performance of AdamW, while for Resnet-18 runs, GALA-SGD with is close to the best performance across all baselines on CIFAR-10, CIFAR-100 and Flower102. Clearly, GALA introduces an additional ~50% computational overhead (Table 1), so a sweep over with GALA-SGD costs about the same as 3 runs of SGD. For a single run, the difference is clearer: e.g., on ViT, AdamW yields about 77.5% at but 69% at and 48.7% at , whereas GALA-SGD reaches 77–79% from every . Unfortunately, it is rather difficult to make an exact comparison on this front; however, we believe our experiments demonstrate that GALA’s robustness to initialization and consistent performance from a narrow initialization range make it a strong candidate when the training budget is limited.
Thank you for the additional experiments and for clarifying the baseline configurations. Two concerns remain central to my assessment. First, the new CIFAR-10 results show that the analyzed algorithm’s learning rate shrinks rapidly and its accuracy falls well short of the practical heuristic’s. It remains unclear why same-batch alignment should guide learning-rate selection near stationary points, where minibatch noise can keep the signal positive. Second, the weight-decay and schedule ablations address part of my concern, but the claimed tuning-cost advantage remains untested under a fixed total wall-clock budget that includes a modest baseline learning-rate search. Given these gaps, I maintain my score.
We thank the reviewer for their detailed feedback.
Q1: How does the exact Section 4 algorithm perform on the reported tasks? Can you isolate the effects of normalization, independent versus same-batch alignment, and optimistic versus ordinary FTRL? Figure 12 addresses only the interpolation choice.
A1: We thank the reviewer for their suggestion. We ran the exact implementations of both algorithms to understand the gap between the theory and our heuristic.
To begin with, the difference between FTL and FTRL is negligible; the regularization parameter δ, which plays the same role as ϵ in Adam, appears in the denominator of the learning rate and makes sure that its initial value is not undefined due to division by zero. We tested both in our experiments and essentially saw no noticeable difference in practice. Similarly, the optimistic version only adds the hint ⟨gt+1(xt+1),mt+1/∥mt+1∥⟩ to the numerator, so it cannot change the behavior we observe below, which is driven by the growth of the denominator.
As for the ablations on the exact implementation of the algorithm, we have two sets of new figures: (i) exact GALA-SGD in Algorithm 1 and (ii) GALA on normalized SGD with momentum (Eq (7-8)) as analyzed in Theorem 4.2. We implemented both algorithms and re-ran the Resnet18/CIFAR-10 experiments exactly as in Section 5, with two different batch sizes: a small batch size of 128 (same as the setup in the manuscript) and a large batch size of 2048. Overall, neither version matches the performance of our heuristic. Algorithm 1 achieves at most 30% (batch size 128) and 47% (batch size 2048) test accuracy, while normalized SGD with momentum experiences a similar gain with the larger batch, but achieves lower accuracy than Algorithm 1. In both cases, the learning rate shrinks significantly in the first epochs. The main reason is the local Lipschitz estimate: since the distance between xt and wt is small due to small learning rate, Lt becomes very large, which shrinks the learning rate even further. For normalized SGD with momentum, the denominator additionally grows by 3α8(1−α)L~t at every step. Since both versions fail at this step, we cannot isolate the effect of normalization in a meaningful way; our heuristic avoids this issue by measuring Lt over the full step using the same mini-batch.
Q2: What controls the variance contribution in same-batch alignment near stationary points? Is there an argument that this signal selects an appropriate learning rate for the population objective, rather than primarily responding to minibatch noise?
A2: We thank the reviewer for raising this question. Our use of the same-batch alignment is motivated by empirical observations; while it is intuitive that this modification may control the variance and reduce the impact of minibatch noise, we currently do not have a formal theoretical justification. As discussed in our response above, the additional experiments in Section E.3.2 show that the independent-batch variants suffer from rapid learning-rate shrinkage and substantially underperform the practical implementation. Increasing the batch size from 128 to 2048 improves performance but does not close this gap. Our evidence for the practical variant therefore rests on its observed optimization and test performance, and we will clarify this limitation in the revised manuscript.
Q3: For Prodigy and D-Adaptation, was the swept parameter d0, the learning-rate multiplier, or both? What exact configurations were used, and were their recommended default configurations included?
A3: Both methods, as described in the original papers, use d0 as the initial estimate of the distance to the solution, and keep the learning-rate multiplier fixed at its default value of 1 without tuning, so that d0 acts as the initial learning rate. We followed the exact recipe and ran both methods with d0∈{1,10−1,10−2,10−3,10−4,10−5,10−8}, same as every other baseline, with all other parameters at their default values. The recommended default d0=10−6 lies within our tested range, and in our sweeps the best performance is not attained at the default: on ResNet-18 with CIFAR-10, both methods perform best at d0=10−3, and for ViT, where we also ran d0=10−6 for Prodigy, it performs on par with 10−5 and 10−8.