- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper theoretically studies how Lookahead affects SAM when they are tied together. Detailedly, it analyzes the generalization and optimization bounds of SALA and investigates the different roles of the Lookahead hyperparameters. The theorerical findings are also validated with empirical results.
Critical strengths
This paper is well-motivated and easy to follow. It theoretically reveals that the optimization bound negatively correlates with the lookahead parameter, while the generalization bound positively correlates with . This finding suggests a tradeoff, which may be useful in tuning hyperparameters. Moreover, different from previous studies, the authors present the first generalization guarantee for SALA in the non-smooth setting.
Critical weaknesses
None
Questions for authors
-
I noticed most experiments are run with SGD, but I am not sure if the mometum has ever been used. If not, please run such a comparision and report whether the findings are universal.
-
In Section 4.1, the perturbation radiu is set to 0.0005, which is relatively small to the popular choice of 0.05. Please justify.
-
Given the theoretical implications, is there any practical advice on the choice of and to balance the generalization and generalization?
3 comments
Many thanks for the detailed response. I would like to raise my score.
Thank you for taking the time to carefully consider our response. We sincerely appreciate your positive reassessment and support.
Q1: I noticed most experiments are run with SGD, but I am not sure if the momentum has ever been used. If not, please run such a comparison and report whether the findings are universal.
A1: Thank you for this suggestion. We have added a comparison between SGD and SGD with momentum in Appendix A.3. Figure 3 shows consistent trends across both optimizer settings: increasing α accelerates optimization convergence, whereas decreasing α reduces the empirical generalization gap. Moreover, the observed dependence on α is consistent with the exponential dependence predicted by our theoretical bounds. These results support the optimization and generalization trade-off and indicate that our findings are universal across the SGD and momentum-SGD settings evaluated here.
Q2: In Section 4.1, the perturbation radii ρ is set to 0.0005, which is relatively small to the popular choice of 0.05. Please justify.
A2: Thank you for this comment. We have added experiments with ρ=0.05 using SALA and ResNet-18 on CIFAR-10 in Appendix A.4: “Additional Results with a Larger Perturbation Radius”. The observed trends remain consistent: increasing α accelerates optimization convergence, whereas decreasing α reduces the empirical generalization gap. The results are also consistent with the exponential dependence on α predicted by our theoretical bounds, supporting our findings at a larger perturbation radius.
Our original choice of ρ=0.0005 was motivated by the small-perturbation regime considered in our excess-risk analysis, where we set ρ=1/n. Although this fixed value does not equal 1/n for every dataset, it is closer to the theoretical scale than ρ=0.05 for the sample sizes considered. Moreover, comparable perturbation radii have also been used in prior work. For instance, Wang et al. [1] used ρ=0.0001 for their LookSAM baseline when training molecular graph transformer models, providing precedent for considering small perturbation radii in SAM-based methods.
[1] Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models.
Q3: Given the theoretical implications, is there any practical advice on the choice of α and k to balance the generalization and generalization?
A3: Our theoretical results suggest that the choice of α depends on several factors. Theorem 4.1 gives the excess-risk bound
ϵexc=O(T−4Lαμ+nkT4L3ακ+nkE[FS(w(S))]T8L3ακ).
If E[FS(w(S))]=0, balancing the optimization and generalization terms yields the heuristic choice
α~=min1,(μ+3κ)logT4Llog(3κμnk).
If E[FS(w(S))]>0 is fixed, the square-root generalization term nkT8L3ακ dominates the other generalization term. Balancing it with the optimization term yields
α~=min1,(2μ+3κ)logT4Llog(9κ2E[FS(w(S))]4μ2nk).
These estimates offer additional insight into the trade-off of α established above. Within the corresponding interior regime and with n and k fixed, shorter training horizons favor a larger α to accelerate optimization, whereas longer training horizons favor a smaller α to control the growing generalization contribution. See Theorem 4.1 and Remark 4.2 for details.
Our current analysis does not provide a principled rule for selecting the number of inner updates k. We acknowledge this limitation and leave a more detailed investigation of its selection to future work.