- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Curvature-Scaled Sharpness-Aware Minimization (CS-SAM), which scales both the gradient-aligned perturbation and the subsequent descent step using a gradient-dependent or loss-dependent curvature bound. It motivates this construction through complementary failure modes: fixed-radius SAM can cycle near stationarity, whereas USAM can diverge at large gradients. Validation consists of analytic one-dimensional examples and ResNet-20/32 experiments on CIFAR-10/100. The loss-dependent variant achieves the highest mean best-test accuracy among the evaluated SAM/USAM baselines in all four settings.
Critical strengths
- This paper derives a necessary condition for uniform distortion control among gradient-aligned, gradient-norm-dependent perturbations and provides a construction with a sufficient guarantee. This gives a clear mathematical rationale for the update.
Critical weaknesses
- Appendix E.2 selects hyperparameters using the maximum test accuracy over training, and Table 3 also reports maximum test accuracy over epochs. This introduces test-set selection bias. Each CS-SAM variant receives 36 configurations per setting, compared with 16 for each baseline, and results use only three seeds.
- The comparisons are restricted to SAM/USAM on two ResNet architectures and CIFAR-10/100, without adaptive SAM baselines or a curvature-scaled gradient descent ablation. Thus, the modest gains cannot clearly be attributed to the proposed sharpness-aware mechanism rather than tuning budget or adaptive descent steps.
- The analysis assumes exact gradients and valid admissible curvature scales, whereas experiments use minibatch quantities and tuned coefficients without establishing their curvature-bound properties.
Questions for authors
- Can you provide results with hyperparameters and checkpoints selected on a separate validation set, equal search budgets across methods, and uncertainty estimates for the resulting performance differences? Do the reported improvements persist?
- How does CS-SAM compare with its curvature-scaled gradient descent counterpart under the same selection and tuning protocol?
- How should the selected configurations outside the convergence conditions be interpreted? Do configurations satisfying those conditions remain competitive, and is there empirical evidence that the tuned scales control curvature or relative gradient distortion during network training?
- Are comparisons available against relevant adaptive methods, particularly LightSAM and LE-SAM, or ASAM and Fisher SAM?
2 comments
On the models and datasets (W2): Restricting the deep learning experiments to ResNet-family models on CIFAR-10/100 is standard in the SAM literature when the main contribution is theoretical. See, for example, [Andriushchenko, 2022], [Khanh, 2024], [Oikonomou, 2025], [Kim, 2026]. In the same spirit, comparing only with the methods that the theory studies is also common: [Khanh, 2024] compares only SAM with constant and diminishing perturbation radii, and [Oikonomou, 2025] compares only members of the Unified SAM family with SGD.
On adaptive SAM baselines (W2, Q4): Thank you for the suggestion. We ran LE-SAM and ASAM, one from each of the two pairs mentioned by the reviewer, in all four settings (see the new DNN experiment in the general response), with the hyperparameters recommended in the corresponding papers and the learning rate selected for SAM with cosine annealing. CS-SAM with has the highest mean in three of the four settings: ASAM is ahead by points on ResNet-20/CIFAR-100 and behind by to points in the other three, with all four differences comparable to one standard deviation, while LE-SAM is behind in all four settings, by to points. Let us also note that the four methods mentioned by the reviewer are complementary to our approach (Appendix A): ASAM and Fisher SAM change the geometry of the perturbation, LightSAM replaces the base optimizer of both steps by AdaGrad/Adam-type updates, and LE-SAM replaces the radius by a loss-space budget. In contrast, CS-SAM rescales the two steps by one scalar curvature bound, which can in principle be combined with any of the above. To the best of our knowledge, none of these methods has convergence guarantees under generalized smoothness (for example, the analysis of LightSAM assumes -smoothness), which is the focus of our work.
Reading your comments, we believe that the pointed weaknesses concern the experimental protocol and can be handled in the updated version of our work. None of them is related to the correctness or the significance of our theoretical results, which are the main contribution of the paper. If you agree that we managed to address all issues, please consider raising your mark to support our work. If you believe this is not the case, please let us know so that we have a chance to respond.
References (not appearing in our paper):
- [Li, 2023] Bingcong Li and Georgios Giannakis. Enhancing sharpness-aware optimization through variance suppression. In NeurIPS, 2023.
- [Yue, 2023] Yun Yue et al. Sharpness-aware minimization revisited: Weighted sharpness as a regularization term. In KDD, 2023.
- [Xie, 2024] Wanyun Xie, Thomas Pethick, and Volkan Cevher. SAMPa: Sharpness-aware minimization parallelized. In NeurIPS, 2024.
- [Khirirat, 2024] Sarit Khirirat, Abdurakhmon Sadiev, Artem Riabinin, Eduard Gorbunov, and Peter Richtárik. Communication-efficient algorithms under generalized smoothness assumptions. In OPT 2024 (NeurIPS Workshop on Optimization for Machine Learning), 2024.
- [Kim, 2026] Hoki Kim, Jinseong Park, Yujin Choi, and Jaewook Lee. Stability analysis of sharpness-aware minimization. In ICML, 2026.
We thank the reviewer for their detailed review and for recognizing that our necessary condition, together with the construction that provides the sufficient guarantee, gives a clear mathematical rationale for the update.
Below, we address the concerns and the questions raised by the reviewer.
On the selection protocol (W1, Q1): We agree that selecting the configuration and the checkpoint by the test accuracy could make the reported numbers optimistic. However, all six rows of Table 3 are produced by exactly the same protocol (Appendix E.2): a grid search on one seed, the configuration with the best test accuracy is rerun on two more seeds, and we report the mean and the standard deviation of the best test accuracy. Since the protocol is identical for the baselines and for CS-SAM, the comparison between the methods, which is what Table 3 is used for, does not favor our method. Let us also note that reporting the maximum test accuracy is standard in the SAM literature: see, for example, [Kwon, 2021] and [Oikonomou, 2025].
On the tuning budget (W1): We kindly note that the comparison is not 36 versus 16 configurations. SAM is tuned on a 4×4 grid of (η,r) for each of the two schedules, that is, 32 configurations for SAM and 32 for USAM, versus 36 for each variant of CS-SAM. Moreover, in Table 3 each variant of CS-SAM is compared against the best of the four baseline rows, that is, against the best of 64 tuned baseline configurations in every setting.
On the number of seeds and the size of the margins (W1): Reporting the mean and the standard deviation over three runs is common in this literature (see, for example, [Oikonomou, 2025]). We agree that in some settings the margin is comparable to the standard deviation: the claim of the paper is that the two scales are competitive with tuned SAM and USAM (Section 4.2: "The experiments are designed to test whether the two scales are competitive in practice"). Let us also note that small margins over the previous baselines are standard in the SAM literature, even in papers whose main contribution is empirical. See, for example: [Kwon, 2021], [Kim, 2022], [Li, 2023], [Yue, 2023].
On the comparison with curvature-scaled gradient descent (W2, Q2): This is a fair request, and we ran this control: see the new DNN experiment in the general response. For both scales and in all four settings, we set ρ=0 in the selected configuration of CS-SAM. Removing the perturbation reduces the best test accuracy by 1.40, 0.90, 3.83 and 4.00 points for MH, and by 1.46, 1.83, 5.00 and 5.07 points for ML. Therefore, the accuracy of CS-SAM is not obtained by the adaptive step size alone: with exactly the same step rule, the perturbation is what makes the difference. In the camera-ready version, we can include these experiments and discussion. Let us also note that CS-GD is covered by our theory: for ρ=0 our theorems recover its known rates (Section 3), and the deterministic experiment in the general response shows that the steps prescribed by our analysis for ρ=0 are competitive with, or better than, the existing ones.
On the deterministic theory and the minibatch experiments (W3): We agree, and we state this explicitly in the paper: in Section 4.2 we write that "we go beyond the deterministic setting of Section 4.1 and of our guarantees", and in Appendix A that "the present analysis is deterministic, assumes exact gradients and function values". Let us note that combining a deterministic theory with minibatch DNN experiments is standard in the literature. See, for example: [Khanh, 2024], [Xie, 2024], [Khirirat, 2024].
On the selected configurations and the convergence conditions (Q3): Let us first clarify one point: the values of ρ reported in Table 4 cannot be compared directly with the threshold ρ<log2 of our theorems. As we note in Appendix E.2, multiplying (ρ,γ,L0,L1), or (ρ,γ,H0,H1), by a positive constant leaves CS-SAM unchanged, and Table 4 reports the coefficients in the normalization L1=1 (respectively H0=1). For example, the configuration (ρ,γ,H0,H1)=(3,3,1,3) is exactly the same algorithm as (1/8,1/8,1/24,1/8), and (ρ,γ,L0,L1)=(0.8,0.4,0.02,1) is the same algorithm as (0.2,0.1,0.005,0.25); both rescaled pairs (ρ,γ) satisfy χ(ρ,γ)>0 and c0≥0. Whether a tuned configuration is covered by our theorems therefore depends on whether the corresponding constants are valid curvature bounds for the network, which is unknown, exactly as the smoothness constant L is unknown in every analysis of SAM under classical smoothness. The DNN runs should thus be interpreted as an empirical test of the update rule beyond the regime where the assumptions can be certified, and this is how we present them in the paper.