acceptodds
Under review as a conference paper at ICLR 2027

Curvature-Scaled Sharpness-Aware Minimization: Convergence Beyond Smoothness

Abstract

Sharpness-Aware Minimization (SAM) has established itself as a widely used method for training deep neural networks. Despite its popularity, existing convergence guarantees for SAM-type methods require smoothness, a restrictive assumption. Generalized smoothness conditions have recently been proposed in the analysis of gradient methods but remain underexplored for SAM. To analyze the effect of SAM's perturbation under generalized smoothness, we use relative gradient distortion to quantify how the sharpness-aware direction departs from the original gradient. Under -smoothness, we show that this distortion can become unbounded near stationarity for SAM and at large gradients for its unnormalized variant (USAM), with examples where SAM cycles and USAM diverges. We then establish a necessary condition for uniform distortion control among perturbations along the gradient whose nonnegative multipliers depend only on its norm. Motivated by this condition, we propose Curvature-Scaled SAM (CS-SAM), which scales both the perturbation and the descent step by suitable curvature upper bounds, and prove that it uniformly bounds the distortion. Under both - and -smoothness, we establish deterministic rates for the best squared gradient norm on nonconvex objectives, last-iterate rates for the function gap on convex objectives with a minimizer, and linear convergence under the Polyak-Lojasiewicz condition. With zero perturbation, our guarantees recover the rates for gradient descent with - or -based step-sizes. Minibatch experiments on computer vision tasks demonstrate competitive performance for both variants, with the variant achieving the highest mean best-test accuracy in all four settings against tuned SAM and USAM with constant and cosine annealing schedules.

Primary area:
optimization
Keywords:
Sharpness-aware minimization, generalized smoothness, convergence analysis, nonconvex optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_owb6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Curvature-Scaled Sharpness-Aware Minimization (CS-SAM), which scales both the gradient-aligned perturbation and the subsequent descent step using a gradient-dependent or loss-dependent curvature bound. It motivates this construction through complementary failure modes: fixed-radius SAM can cycle near stationarity, whereas USAM can diverge at large gradients. Validation consists of analytic one-dimensional examples and ResNet-20/32 experiments on CIFAR-10/100. The loss-dependent variant achieves the highest mean best-test accuracy among the evaluated SAM/USAM baselines in all four settings.

Critical strengths
  1. This paper derives a necessary condition for uniform distortion control among gradient-aligned, gradient-norm-dependent perturbations and provides a construction with a sufficient guarantee. This gives a clear mathematical rationale for the update.
Critical weaknesses
  1. Appendix E.2 selects hyperparameters using the maximum test accuracy over training, and Table 3 also reports maximum test accuracy over epochs. This introduces test-set selection bias. Each CS-SAM variant receives 36 configurations per setting, compared with 16 for each baseline, and results use only three seeds.
  2. The comparisons are restricted to SAM/USAM on two ResNet architectures and CIFAR-10/100, without adaptive SAM baselines or a curvature-scaled gradient descent ablation. Thus, the modest gains cannot clearly be attributed to the proposed sharpness-aware mechanism rather than tuning budget or adaptive descent steps.
  3. The analysis assumes exact gradients and valid admissible curvature scales, whereas experiments use minibatch quantities and tuned coefficients without establishing their curvature-bound properties.
Questions for authors
  1. Can you provide results with hyperparameters and checkpoints selected on a separate validation set, equal search budgets across methods, and uncertainty estimates for the resulting performance differences? Do the reported improvements persist?
  2. How does CS-SAM compare with its curvature-scaled gradient descent counterpart under the same selection and tuning protocol?
  3. How should the selected configurations outside the convergence conditions be interpreted? Do configurations satisfying those conditions remain competitive, and is there empirical evidence that the tuned scales control curvature or relative gradient distortion during network training?
  4. Are comparisons available against relevant adaptive methods, particularly LightSAM and LE-SAM, or ASAM and Fisher SAM?
2 comments

We thank the reviewer for their detailed review and for recognizing that our necessary condition, together with the construction that provides the sufficient guarantee, gives a clear mathematical rationale for the update.

Below, we address the concerns and the questions raised by the reviewer.

On the selection protocol (W1, Q1): We agree that selecting the configuration and the checkpoint by the test accuracy could make the reported numbers optimistic. However, all six rows of Table 3 are produced by exactly the same protocol (Appendix E.2): a grid search on one seed, the configuration with the best test accuracy is rerun on two more seeds, and we report the mean and the standard deviation of the best test accuracy. Since the protocol is identical for the baselines and for CS-SAM, the comparison between the methods, which is what Table 3 is used for, does not favor our method. Let us also note that reporting the maximum test accuracy is standard in the SAM literature: see, for example, [Kwon, 2021] and [Oikonomou, 2025].

On the tuning budget (W1): We kindly note that the comparison is not 36 versus 16 configurations. SAM is tuned on a 4×44\times4 grid of (η,r)(\eta,r) for each of the two schedules, that is, 32 configurations for SAM and 32 for USAM, versus 36 for each variant of CS-SAM. Moreover, in Table 3 each variant of CS-SAM is compared against the best of the four baseline rows, that is, against the best of 64 tuned baseline configurations in every setting.

On the number of seeds and the size of the margins (W1): Reporting the mean and the standard deviation over three runs is common in this literature (see, for example, [Oikonomou, 2025]). We agree that in some settings the margin is comparable to the standard deviation: the claim of the paper is that the two scales are competitive with tuned SAM and USAM (Section 4.2: "The experiments are designed to test whether the two scales are competitive in practice"). Let us also note that small margins over the previous baselines are standard in the SAM literature, even in papers whose main contribution is empirical. See, for example: [Kwon, 2021], [Kim, 2022], [Li, 2023], [Yue, 2023].

On the comparison with curvature-scaled gradient descent (W2, Q2): This is a fair request, and we ran this control: see the new DNN experiment in the general response. For both scales and in all four settings, we set ρ=0\rho=0 in the selected configuration of CS-SAM. Removing the perturbation reduces the best test accuracy by 1.401.40, 0.900.90, 3.833.83 and 4.004.00 points for MHM_H, and by 1.461.46, 1.831.83, 5.005.00 and 5.075.07 points for MLM_L. Therefore, the accuracy of CS-SAM is not obtained by the adaptive step size alone: with exactly the same step rule, the perturbation is what makes the difference. In the camera-ready version, we can include these experiments and discussion. Let us also note that CS-GD is covered by our theory: for ρ=0\rho=0 our theorems recover its known rates (Section 3), and the deterministic experiment in the general response shows that the steps prescribed by our analysis for ρ=0\rho=0 are competitive with, or better than, the existing ones.

On the deterministic theory and the minibatch experiments (W3): We agree, and we state this explicitly in the paper: in Section 4.2 we write that "we go beyond the deterministic setting of Section 4.1 and of our guarantees", and in Appendix A that "the present analysis is deterministic, assumes exact gradients and function values". Let us note that combining a deterministic theory with minibatch DNN experiments is standard in the literature. See, for example: [Khanh, 2024], [Xie, 2024], [Khirirat, 2024].

On the selected configurations and the convergence conditions (Q3): Let us first clarify one point: the values of ρ\rho reported in Table 4 cannot be compared directly with the threshold ρ<log⁡2\rho<\log2 of our theorems. As we note in Appendix E.2, multiplying (ρ,γ,L0,L1)(\rho,\gamma,L_0,L_1), or (ρ,γ,H0,H1)(\rho,\gamma,H_0,H_1), by a positive constant leaves CS-SAM unchanged, and Table 4 reports the coefficients in the normalization L1=1L_1=1 (respectively H0=1H_0=1). For example, the configuration (ρ,γ,H0,H1)=(3,3,1,3)(\rho,\gamma,H_0,H_1)=(3,3,1,3) is exactly the same algorithm as (1/8,1/8,1/24,1/8)(1/8,1/8,1/24,1/8), and (ρ,γ,L0,L1)=(0.8,0.4,0.02,1)(\rho,\gamma,L_0,L_1)=(0.8,0.4,0.02,1) is the same algorithm as (0.2,0.1,0.005,0.25)(0.2,0.1,0.005,0.25); both rescaled pairs (ρ,γ)(\rho,\gamma) satisfy χ(ρ,γ)>0\chi(\rho,\gamma)>0 and c0≥0c_0\ge0. Whether a tuned configuration is covered by our theorems therefore depends on whether the corresponding constants are valid curvature bounds for the network, which is unknown, exactly as the smoothness constant LL is unknown in every analysis of SAM under classical smoothness. The DNN runs should thus be interpreted as an empirical test of the update rule beyond the regime where the assumptions can be certified, and this is how we present them in the paper.

On the models and datasets (W2): Restricting the deep learning experiments to ResNet-family models on CIFAR-10/100 is standard in the SAM literature when the main contribution is theoretical. See, for example, [Andriushchenko, 2022], [Khanh, 2024], [Oikonomou, 2025], [Kim, 2026]. In the same spirit, comparing only with the methods that the theory studies is also common: [Khanh, 2024] compares only SAM with constant and diminishing perturbation radii, and [Oikonomou, 2025] compares only members of the Unified SAM family with SGD.

On adaptive SAM baselines (W2, Q4): Thank you for the suggestion. We ran LE-SAM and ASAM, one from each of the two pairs mentioned by the reviewer, in all four settings (see the new DNN experiment in the general response), with the hyperparameters recommended in the corresponding papers and the learning rate selected for SAM with cosine annealing. CS-SAM with MHM_H has the highest mean in three of the four settings: ASAM is ahead by 0.240.24 points on ResNet-20/CIFAR-100 and behind by 0.140.14 to 0.330.33 points in the other three, with all four differences comparable to one standard deviation, while LE-SAM is behind in all four settings, by 0.450.45 to 1.171.17 points. Let us also note that the four methods mentioned by the reviewer are complementary to our approach (Appendix A): ASAM and Fisher SAM change the geometry of the perturbation, LightSAM replaces the base optimizer of both steps by AdaGrad/Adam-type updates, and LE-SAM replaces the radius by a loss-space budget. In contrast, CS-SAM rescales the two steps by one scalar curvature bound, which can in principle be combined with any of the above. To the best of our knowledge, none of these methods has convergence guarantees under generalized smoothness (for example, the analysis of LightSAM assumes LL-smoothness), which is the focus of our work.

Reading your comments, we believe that the pointed weaknesses concern the experimental protocol and can be handled in the updated version of our work. None of them is related to the correctness or the significance of our theoretical results, which are the main contribution of the paper. If you agree that we managed to address all issues, please consider raising your mark to support our work. If you believe this is not the case, please let us know so that we have a chance to respond.

References (not appearing in our paper):

  • [Li, 2023] Bingcong Li and Georgios Giannakis. Enhancing sharpness-aware optimization through variance suppression. In NeurIPS, 2023.
  • [Yue, 2023] Yun Yue et al. Sharpness-aware minimization revisited: Weighted sharpness as a regularization term. In KDD, 2023.
  • [Xie, 2024] Wanyun Xie, Thomas Pethick, and Volkan Cevher. SAMPa: Sharpness-aware minimization parallelized. In NeurIPS, 2024.
  • [Khirirat, 2024] Sarit Khirirat, Abdurakhmon Sadiev, Artem Riabinin, Eduard Gorbunov, and Peter Richtárik. Communication-efficient algorithms under generalized smoothness assumptions. In OPT 2024 (NeurIPS Workshop on Optimization for Machine Learning), 2024.
  • [Kim, 2026] Hoki Kim, Jinseong Park, Yujin Choi, and Jaewook Lee. Stability analysis of sharpness-aware minimization. In ICML, 2026.
Reviewer_M54JOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies Sharpness-Aware Minimization (SAM) and its variants. The paper first provides examples of cases where SAM/USAM fail to converge, identifying a failure mode of existing methods. Then the paper introduces a curvature adjusted version of SAM, CS-SAM which aims to address those failure modes. The authors provide theoretical convergence guarantees, and numerical experiments on some DNN architectures.

Critical strengths
  • Finding failure examples of SAM and USAM is interesting, and provides insights into algorithm performance. This motivates the paper's proposal of CS-SAM.
Critical weaknesses
  • The theoretical guarantees are standard convergence-to-local-solution type, that do not really explain why CS-SAM might reach a better solution compared to SAM/USAM. Recall that SAM's main selling point is that it can achieve solutions with better sharpness. Faster or more stable convergence is definitely welcome, but is not a primary focus of SAM.

  • Experiments are on a limited set of models (ResNet) and benchmarks (CIFAR), which fall significantly below the expected level of experimentation richness. No vision transformers, NLP or large-scale benchmarking with ImageNet has been done. Moreover, the authors do not discuss why CS-SAM generalizes better (lower sharpness for example?).

Questions for authors

I would suggest authors consider expanding the experimental results. In particular:

  • Discuss why CS-SAM generalizes better
  • How CS-SAM works in vision transformer or NLP setups
4 comments

We thank the reviewer for their comments and for recognizing that the failure examples of SAM and USAM are interesting and provide insights into the performance of these algorithms.

Below, we address the concerns raised by the reviewer.

On the significance of the theory: The reviewer mentioned: "The theoretical guarantees are standard convergence-to-local-solution type [...] Faster or more stable convergence is definitely welcome, but is not a primary focus of SAM."

We agree that generalization was the original motivation of SAM. However, in our opinion, for any training algorithm the optimization and the generalization aspects are of the same importance: generalization theory aims to understand which solutions the method prefers, while optimization theory explains whether the method reaches a solution at all, and how fast. Studying only the convergence guarantees of SAM-type methods, without studying the sharpness of the solutions, is standard in the literature. See, for example: [Si, 2023], [Khanh, 2024], [Oikonomou, 2025], [Xie, 2024]. None of these works studies the sharpness of the solutions found by the method. Our work belongs to this category of papers, and we believe that both directions are valuable and should be explored by the community.

Having said that, we politely disagree that our guarantees are standard. Let us explain why:

  1. Convergence of SAM-type methods cannot be taken for granted. Proposition 2.1 shows that fixed-radius SAM cycles on a convex quadratic and that USAM diverges on a convex quartic. A method that cycles or diverges does not reach a good solution of any kind, so understanding which perturbations allow convergence comes before any discussion on the quality of the solution.
  2. First guarantees beyond smoothness. All existing convergence results for SAM-type methods require classical LL-smoothness (Table 1), an assumption that can fail even for simple neural network objectives. To the best of our knowledge, Theorems 3.1 and 3.2 are the first convergence rates for a SAM-type method under (L0,L1)(L_0,L_1)- and (H0,H1)(H_0,H_1)-smoothness.
  3. Tightness. For ρ=0\rho=0 our theorems recover the known rates of curvature-scaled gradient descent, and for the loss scale they allow a step that is 77 to 1414 times larger than the one of [Alimisis, 2026]. The new deterministic experiment in the general response shows that this is not only a theoretical improvement: it gives 77 to 1313 times fewer gradient evaluations.

On why CS-SAM generalizes well: We do not claim a generalization theory for CS-SAM: as we mention in Appendix A, CS-SAM "should be viewed as an optimization method", and proving that it prefers flatter solutions is beyond the scope of this work. Having said that, the new DNN experiment in the general response shows which part of the method is responsible for the test accuracy: when we remove the perturbation from CS-SAM and keep exactly the same curvature-scaled step, the test accuracy drops by 0.90.9 to 1.81.8 points on CIFAR-10 and by 3.83.8 to 5.15.1 points on CIFAR-100. That is, the improvement in test accuracy comes from the sharpness-aware perturbation and not from the adaptive step size alone.

On the scope of the experiments: We respectfully disagree that the experiments fall "significantly below the expected level" for a paper whose main contribution is theoretical. Let us highlight what the experimental section contains:

  • Section 4.1 verifies our theory exactly: the predicted two-cycle of SAM, the divergence threshold of USAM, and the distortion bound of CS-SAM for both scales. Reviewer 3eF6 notes that "the theory is verified with unusual care".
  • Section 4.2 evaluates six methods (SAM and USAM with constant and cosine annealing schedules, and the two variants of CS-SAM) on four model–dataset settings, with every method tuned separately in every setting.
  • In the general response, we add two new experiments. For DNNs, we add two adaptive SAM baselines (ASAM and LE-SAM) and the zero-perturbation control of CS-SAM for both scales, in all four settings. In the deterministic setting, we test the step sizes that our analysis gives for ρ=0\rho=0. In total, the DNN experiments now correspond to more than 600 full training runs of 200 epochs.

Let us also note that restricting the deep learning experiments to ResNet-family models on CIFAR-10/100 is standard in the SAM literature when the main contribution is theoretical. See, for example: [Andriushchenko, 2022], [Khanh, 2024], [Oikonomou, 2025], [Kim, 2026].

On vision transformers and NLP: Large-scale experiments such as ImageNet are not feasible in the rebuttal period, but we can aim to include a vision transformer experiment in the camera-ready version. Having said that, in our opinion, the lack of such experiments should not be seen as a limitation of a theory-oriented paper.

If you agree that we managed to address all issues, please consider raising your mark to support our work. From our viewpoint, the concerns raised are related to the scope of the paper (the optimization theory of SAM-type methods under generalized smoothness) and not to flaws of our results, and they are not reasons justifying a rejection score. If you believe this is not the case, please let us know so that we have a chance to respond.

References (not appearing in our paper):

  • [Xie, 2024] Wanyun Xie, Thomas Pethick, and Volkan Cevher. SAMPa: Sharpness-aware minimization parallelized. In NeurIPS, 2024.
  • [Kim, 2026] Hoki Kim, Jinseong Park, Yujin Choi, and Jaewook Lee. Stability analysis of sharpness-aware minimization. In ICML, 2026.

Thank you for your response. It appears that the authors frame the paper as a theoretical paper, but my concerns with theory still stand. Therefore, I maintain my score.

We thank the reviewer for engaging in the discussion. However, we believe that there is a misunderstanding about the claims and the nature of our work, and we would like to clarify both points.

On the claims of our paper: The theoretical concern of the reviewer is that our guarantees "do not really explain why CS-SAM might reach a better solution compared to SAM/USAM". Let us clarify that we never claim that CS-SAM generalizes better than SAM, nor that our theory explains generalization. Our claims are:

  1. Under generalized smoothness, SAM and USAM can fail to converge at all: SAM cycles on a convex quadratic and USAM diverges on a convex quartic (Proposition 2.1).
  2. We prove a necessary condition for uniformly bounded distortion (Theorem 2.2), and CS-SAM satisfies it for any admissible curvature scale.
  3. CS-SAM has the first convergence rates of a SAM-type method under (L0,L1)(L_0,L_1)- and (H0,H1)(H_0,H_1)-smoothness, and they are tight: for ρ=0\rho=0 they recover the known rates of gradient descent.
  4. Empirically, CS-SAM is competitive with tuned SAM, USAM, ASAM and LE-SAM.

None of these claims is about sharpness, and our theorems prove exactly what we claim.

This is not only a theoretical paper: We used the term "theory-oriented" only to explain our choice of benchmarks, and we are sorry if this gave a different impression. The paper and the rebuttal contain the following experiments:

  • Verification of the theory (Section 4.1): the two-cycle of SAM, the divergence of USAM, and the distortion bound of CS-SAM for both scales.
  • DNN training (Section 4.2): SAM and USAM with constant and cosine schedules and the two variants of CS-SAM, on ResNet-20/32 and CIFAR-10/100, with every method tuned separately in every setting.
  • New, adaptive baselines: ASAM and LE-SAM in all four settings. CS-SAM with MHM_H has the highest mean in three of the four settings, and it is ahead of LE-SAM in all four.
  • New, zero-perturbation control: with exactly the same step rule but no perturbation, the test accuracy drops by 0.90.9 to 1.81.8 points on CIFAR-10 and by 3.83.8 to 5.15.1 points on CIFAR-100. That is, the accuracy of CS-SAM comes from the sharpness-aware perturbation and not from the adaptive step alone.
  • New, deterministic step sizes: our steps need 77 to 1313 times fewer gradient evaluations than those of [Alimisis, 2026], and are within a factor of 1.21.2 to 1.41.4 of the steps of [Vankov, 2025], which are designed specifically for gradient descent.

In total, the DNN experiments correspond to more than 600 training runs of 200 epochs.

We respectfully believe that a rating of 1 is not justified: no correctness issue has been raised, and the remaining concern is about a claim (better generalization) that our paper does not make. If you agree, please consider raising your mark to support our work. Otherwise, please let us know and we will be most happy to respond.

Reviewer_3eF6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops convergence theory for sharpness-aware minimization (SAM) under generalized smoothness conditions. All existing convergence guarantees for SAM-type methods rely on global Lipschitz smoothness, whereas the recently proposed gradient-norm-dependent and loss-value-dependent smoothness models fit neural-network training far better, yet had never been analyzed for SAM. The paper's concentration is relative gradient distortion: the relative deviation between the gradient at the perturbed point and the original gradient. The authors first show that the two base rules fail in complementary ways: SAM's distortion is unbounded as the gradient vanishes, while USAM's distortion is unbounded as the gradient grows. Building on this, Theorem 2.2 establishes a necessary condition for uniformly bounded distortion within the class of perturbations that are collinear with the gradient with a multiplier depending only on its norm. Motivated by this condition, the authors define an admissible curvature scale and propose Curvature-Scaled SAM (CS-SAM), which divides both the perturbation step and the descent step by a curvature upper bound evaluated at the point where the corresponding gradient is taken. Validation consists of two parts: one numerically reproduces Proposition 2.1 and Theorem 2.5 on the one-dimensional examples (two-cycle, divergence, and distortion below the predicted bound on both scales); The other trains ResNet-20/32 on CIFAR-10/100 against tuned constant-step and cosine-annealed SAM and USAM, where the loss-scale variant achieves the highest mean best test accuracy in all four settings and the gradient-scale variant in three of four.

Critical strengths

First, the theoretical contribution is well positioned and the construction is derivational rather than decorative. The paper starts from the right quantity (relative gradient distortion), establishes complementary failure modes, derives a necessary condition on the perturbation form, abstracts admissible scales and proves sufficiency, and finally delivers complete rates on both concrete scales. The chain is self-consistent, and the recovery of the known curvature-scaled gradient-descent results at zero perturbation shows the analysis is neither loose nor degenerate, embedding SAM theory genuinely into the generalized-smoothness landscape.

Second, the theory is verified with unusual care, and the empirical part mirrors it precisely. It verifies both the failure modes (the predicted two-cycle amplitude and the divergence threshold) and the new method's distortion bound (below the theoretical envelope across the entire gradient range, on both scales), so theory and experiment interlock. The baselines are SAM and USAM each tuned over learning rate and radius (multiplier), under both constant and cosine schedules, with gradient-evaluation counts matched, and the loss-scale variant winning all four settings consistently strengthens the claim.

Critical weaknesses

The experimental evaluation protocol is thin, and several conclusions lack statistical support. Hyperparameters are selected by best test accuracy over 200 epochs (test set used for model selection), tuning uses a single seed, and final numbers average only three seeds with no significance testing; in some settings the margin (about 0.3 percentage points on ResNet-32/CIFAR-100) is comparable to the standard deviation, with visibly overlapping error bars. The baseline set contains only vanilla SAM and USAM. In particular, normalized gradient descent is missing even though it is the closest relative of the gradient-scale variant and the minimal control for attributing gains to the sharpness-aware perturbation rather than to the adaptive step size; m-sharpness, ASAM, and the concurrent loss-dependent-radius work the paper itself cites are also absent.

Questions for authors

Did the authors run the zero-perturbation variant, that is, pure curvature-scaled gradient descent, as a control in the deep-network experiments? The gradient-scale variant as configured is very close to normalized gradient descent, so without this control the gains cannot be attributed to the sharpness-aware perturbation itself rather than to the adaptive step size.

The protocol selects configurations by best test accuracy over 200 epochs on a single tuning seed and reports means over three seeds. Would the ranking survive a validation-based selection protocol and/or additional seeds, and does the margin on ResNet-32/CIFAR-100 remain significant under it?

2 comments

We thank the reviewer for the careful reading and the very accurate summary of our work, and for recognizing that the theoretical contribution is "well positioned", that the construction is "derivational rather than decorative", and that the theory is "verified with unusual care".

Below, we address the concerns and the two questions raised by the reviewer.

On the zero-perturbation experiment (Q1): This is a fair request, and we ran this control: see the new DNN experiment in the general response. For both scales and in all four settings, we set ρ=0\rho=0 in the selected configuration of CS-SAM. Removing the perturbation reduces the best test accuracy by 1.401.40, 0.900.90, 3.833.83 and 4.004.00 points for MHM_H, and by 1.461.46, 1.831.83, 5.005.00 and 5.075.07 points for MLM_L. Therefore, the accuracy of CS-SAM is not obtained by the adaptive step size alone: with exactly the same step rule, the perturbation is what makes the difference. In the camera-ready version, we can include these experiments and discussion.

On normalized gradient descent: The reviewer is right that, with the selected L0=0.02L_0=0.02, the gradient-scale variant is close to a normalized method whenever the minibatch gradient norm is much larger than L0L_0. This is by design: under (L0,L1)(L_0,L_1)-smoothness, the curvature-scaled step γ/(L0+L1∥∇f(x)∥)\gamma/(L_0+L_1\lVert\nabla f(x)\rVert) is the step of clipped and normalized gradient methods [Zhang, 2020], [Gorbunov, 2025], [Vankov, 2025]: it behaves like gradient descent for small gradients and like normalized gradient descent for large ones. For ρ=0\rho=0, CS-SAM with MLM_L is exactly this method. Hence normalized gradient descent is not an external baseline but, essentially, the ρ=0\rho=0 member of our family, and the row "CS-GD with MLM_L" of the new DNN experiment is precisely the comparison with it: it is 1.461.46 to 5.075.07 points below CS-SAM with MLM_L.

On the selection protocol and the tuning seed (Q2): We agree that selecting the configuration and the checkpoint by the test accuracy could make the reported numbers optimistic. However, all six rows of Table 3 are produced by exactly the same protocol (Appendix E.2), so the comparison between the methods does not favor ours. Reporting the maximum test accuracy is also standard in the SAM literature: see, for example: [Kwon, 2021], [Oikonomou, 2025]. Regarding the single tuning seed, the tuning campaign alone comprises 544 training runs of 200 epochs (256 for the baselines and 288 for CS-SAM), and tuning on one seed and rerunning the selected configuration on additional seeds is the usual compromise. Let us also note that the tuning budgets are comparable: SAM and USAM receive 32 configurations each (a 4×44\times4 grid for each of the two schedules), versus 36 for each variant of CS-SAM, and each variant of CS-SAM is compared against the best of 64 tuned baseline configurations in every setting.

On the margin on ResNet-32/CIFAR-100 (Q2): We agree that in this setting the margin over the best baseline is within one standard deviation, and we do not claim a statistically significant improvement there. The claim of the paper is that the two scales are competitive with tuned SAM and USAM. Let us add two remarks. First, relative to tuned SAM, which is the standard method, the improvement in this setting is 0.990.99 points for MHM_H and 0.730.73 points for MLM_L. Second, small margins over the previous baselines are standard in the SAM literature, even in papers whose main contribution is empirical. See, for example: [Kwon, 2021], [Kim, 2022], [Li, 2023], [Yue, 2023].

On additional baselines: Our main empirical goal was to compare CS-SAM with the two rules that our theory studies, each with a constant and with a cosine annealing schedule. This is common in theory-oriented SAM papers: for example: [Khanh, 2024], [Oikonomou, 2025]. Having said that, we ran ASAM and the concurrent LE-SAM in all four settings, with the hyperparameters recommended in the corresponding papers (see the new DNN experiment in the general response). CS-SAM with MHM_H has the highest mean in three of the four settings: ASAM is ahead by 0.240.24 points on ResNet-20/CIFAR-100, and LE-SAM is behind in all four settings. Let us also note that ASAM changes the geometry of the perturbation, which is complementary to the scalar curvature scaling of CS-SAM, and the two can in principle be combined.

Thanks again for the detailed review and the positive evaluation of our theoretical contribution. Reading your comments, we believe that the pointed weaknesses concern the experimental protocol and can be handled in the updated version of our work. If you agree that we managed to address all issues, please consider raising your mark to support our work. If you believe this is not the case, please let us know so that we have a chance to respond.

References (not appearing in our paper):

  • [Li, 2023] Bingcong Li and Georgios Giannakis. Enhancing sharpness-aware optimization through variance suppression. In NeurIPS, 2023.
  • [Yue, 2023] Yun Yue et al. Sharpness-aware minimization revisited: Weighted sharpness as a regularization term. In KDD, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.