acceptodds
Under review as a conference paper at ICLR 2027

Preconditioned Sharpness-Aware Minimization Can Exploit Large Perturbation Radii

Abstract

When sharpness-aware minimization (SAM) is combined with Adam-type optimizers, its perturbation and update operate in different geometries. Preconditioned SAM (PSAM) (Singh et al., 2025) addresses this mismatch. We analyze PSAM in its own right and examine its practical use. We show that each PSAM step is standard SAM in scaled geometry when the perturbation and update share a preconditioner. This yields a stationarity bound and a fixed-metric PAC-Bayes bound. Our descent analysis under quasiconvexity introduces an admissible radius and suggests how PSAM's perturbation direction can allow larger radii. In practice, early preconditioner instability motivates a warm start for PSAM. Across vision and language tasks, warm-started PSAM performs best among the evaluated methods. In language modeling, the best-performing radius for warm-started PSAM is up to 25 times that for warm-started SAM, while other methods deteriorate at these radii. We observe larger radius gaps when the SAM and PSAM perturbation directions are less aligned, consistent with our geometric analysis.

Primary area:
optimization
Keywords:
Sharpness-aware minimization, preconditioning, adaptive optimization, loss geometry, generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FaBwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the theoretical properties and practical behavior of preconditioned sharpness-aware minimization (PSAM). The authors show that, when the perturbation and update share a preconditioner, each PSAM step is equivalent to a standard SAM step in scaled coordinates. They derive a stationarity bound and a fixed-metric PAC-Bayes bound, and use a quasiconvex descent argument to motivate an explanation for larger admissible perturbation radii.

Critical strengths

S1. The scaled-coordinate interpretation provides a clear perspective on the interaction between preconditioned perturbations and parameter updates.

S2. The finding that warm-started PSAM remains effective at radii where other methods deteriorate is practically useful, particularly in the NLP settings.

S3. The paper goes beyond final performance by examining preconditioner drift, perturbation alignment, and a loss-plane visualization.

Critical weaknesses
  • I am not fully convinced that the shared-preconditioner analysis directly explains the behavior of the practical Adam(W) implementation. In the theory, the perturbation and parameter update use the same preconditioner. In practice, however, the perturbation is constructed using Dt−1D_{t-1}, while the parameter update uses DtD_t together with momentum. The authors do acknowledge this mismatch, but Figure 3 only reports the relative Frobenius drift of the preconditioner, which does not directly tell us how much the actual update direction changes. It would be more convincing to compare Dt−1−1gt+D_{t-1}^{-1}g_t^+ with Dt−1gt+D_t^{-1}g_t^+, or to include an ablation where the preconditioner is kept fixed within each step. This would make the connection between the theory and the actual implementation much clearer.

  • The evidence linking perturbation direction to the larger useful radius remains limited. Table 2 measures the cosine similarity between SAM and PSAM perturbations only at the end of training and compares five different model/task settings. Since the architecture, dataset, and training setup also differ, the table does not isolate whether the directional difference explains the gap in useful radii. Figure 7 covers only one model, one seed, and one mini-batch. The authors could strengthen this explanation by measuring the similarity at several training stages and across mini-batches within the same model and task, and checking whether lower similarity consistently corresponds to a larger useful-radius gap.

  • The paper shows that warm starting improves PSAM, but the explanation that this benefit comes from preconditioner stabilization needs more direct evidence. The diagnostics in Figures 3 and 5 use vision runs without warm starting, so they do not show how the preconditioner behaves at the actual WS-PSAM switch or in the NLP experiments. Warm starting also changes the model parameters, loss level, and optimizer state, so the current results do not separate these possible explanations. The authors could report preconditioner drift and update-direction changes around the switch, especially in the NLP settings. Comparing runs that retain or reset the second-moment estimate at the same switch checkpoint, with initialization and bias correction handled consistently, would also help assess the contribution of the accumulated preconditioner.

Questions for authors
  1. The analysis uses a shared preconditioner for perturbation and update, whereas the practical implementation uses different preconditioner estimates and includes momentum. Please measure how closely the actual update direction matches the direction analyzed in the theory, especially before and after the warm-start switch. Reporting cosine similarity between these directions would be more informative than preconditioner drift alone. A controlled ablation using the same preconditioner for both operations would also help assess the practical effect of this difference.
  2. Table 2 measures alignment only at the end of training across different model/task settings, and Figure 7 covers only one model, one seed, and one mini-batch. Please track alignment at several training stages and across mini-batches and radii within the same model and task, and check whether better alignment consistently corresponds to a larger useful radius. For the warm-start explanation, please also report preconditioner behavior around the switch in the NLP experiments.
  3. How are the perturbation radius and warm-start ratio selected? Please clarify whether independent validation data are used for the vision and translation experiments and whether all methods receive comparable tuning budgets. The current description leaves this protocol unclear.
  4. The reported 25-fold radius ratio compares optima on a relatively coarse grid. Please qualify it as the best ratio on the tested grid, or provide a finer sweep around the best-performing radii. It is important to provide a guideline to choose ρ\rho in practice.
  5. Please specify how the preconditioner is initialized for the first step of non-warm-started PSAM and how normalized perturbations are computed when the gradient norm is zero. Including the initialization and any numerical safeguards in the pseudocode would improve reproducibility.
  6. Warm starting uses only one gradient evaluation per step during its initial phase, so equal numbers of training steps do not necessarily imply equal computational cost. Please report total gradient-evaluation counts or runtime for the compared methods, together with the warm-start ratios.
  7. The experiments only compare argainst PSAM and SAM, lacking other popular baselines such as ASAM and FisherSAM. More importantly, the results on ImageNet-1K is also missing. It is worth noting that the proposed optimizer may perform dramatically different on small-scale and large-scale datasets.
  8. Some recent works in the literature review are missing. For example, (Baek et al, 2024) studies why SAM is robust to the label noise and (Tan et al, 2026) also investigates the benefits of SAM on model calibration.
  • Baek et al, WHY IS SAM ROBUST TO LABEL NOISE (ICLR, 2024)
  • Tan et al, Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization (ICLR, 2026)
  1. Frankly speaking, SAM is not a good choice for Transformer-based NLP tasks. This is because on one hand, it requires two backpropagations each step. On the other hand, SAM performs even much worse than the base optimizer in the large-batch training. This is a crtical issue. Just as Figure 1 depicts, the improvement is neglible and increasing ρ\rho may even explode the training process. It would be more convincing if the authors could clarify why it is essential to use large ρ\rho for PSAM.
5 comments
AuthorsOpenReview ↗

Response to Reviewer FaBw (1/3): W1–W3, Q1–Q2

We thank the reviewer for the time and the constructive comments. We have revised the paper accordingly, and the responses below refer to the revised version.

W1 / Q1. Shared preconditioner in theory and practice.

With Pt=DtP_t = D_t, our analysis uses the same update preconditioner as practical PSAM, so the gap lies in the perturbation, which uses Dt−1D_{t-1}. At the best radius (ρ=5\rho = 5) on GPT-2 (30M), we therefore measured cos⁡(Dt−1−1gt,Dt−1gt)\cos(D_{t-1}^{-1}g_t, D_t^{-1}g_t) along with the suggested cos⁡(Dt−1−1gt+,Dt−1gt+)\cos(D_{t-1}^{-1}g_t^+, D_t^{-1}g_t^+) [Appendix E.4]:

before switch (steps 401–500)after switch (501–600)last epoch
cos⁡(Dt−1−1gt,Dt−1gt)\cos(D_{t-1}^{-1}g_t, D_t^{-1}g_t)0.0890.1100.136
cos⁡(Dt−1−1gt+,Dt−1gt+)\cos(D_{t-1}^{-1}g_t^+, D_t^{-1}g_t^+)–0.0740.085

The former is higher, so the perturbation deviates less than the suggested comparison indicates; still, in both cases the directions differ, and we acknowledge this gap. In the suggested ablation that shares Dt−1D_{t-1} between the perturbation and the update, training failed at every radius, likely because the update is prone to diverge without the update gradient in its preconditioner.

W2 / Q2. Reliability of the alignment in Table 2.

Our alignment analysis assumes quasiconvexity, which is more plausible near a minimizer (Section 5.3), so Table 2 reports the alignment at the final training step. We agree that a single step is a limited sample. We therefore computed cos⁡(gt,Dt−1−1gt)\cos(g_t, D_{t-1}^{-1}g_t) at every step of the last epoch, that is, over many mini-batches, for WS-PSAM on GPT-2 (30M) with 11 radii and 5 seeds [Appendix E.3]:

ρ\rho0.0050.112510
cos0.049 ± 0.0050.049 ± 0.0050.047 ± 0.0060.046 ± 0.0060.043 ± 0.0060.038 ± 0.005

The other five radii give 0.048–0.049. At every radius, the alignment stays at the level in Table 2 (0.040) with small variation across steps and seeds, so the reported value is reliable. We address the preconditioner behavior around the switch in W3.

W3. Warm start and the preconditioner.

Our warm start aims to skip the unstable early phase of training rather than to carry over the accumulated preconditioner (Sections 5.3 and 6). To separate the two, we ran the suggested ablation on GPT-2 (30M), resetting mm, vv, and both bias-correction counters at the switch [Appendix E.2]. Resetting did not hurt at any radius (validation perplexity, 5 seeds):

ρ\rho12510
keep157.24155.92155.61161.29
reset156.93155.73154.85160.82

The benefit therefore comes from skipping the early phase, not from the accumulated preconditioner.

As also suggested, we measured the preconditioner drift in language modeling (GPT-2 (30M)) [Appendix E.2]. At the best radius (ρ=5\rho = 5), it is large only in the first 10 steps, which the warm start skips, and stays small afterward, including around the switch at step 501:

steps2–10401–500501–600last epoch
drift0.1300.0530.0570.058

Update directions around the switch are reported in W1.

AuthorsOpenReview ↗

Response to Reviewer FaBw (2/3): Q3–Q6

Q3. Tuning protocol.

For each task, we sweep ρ\rho over the same grid for all SAM-type methods and take each method's best radius; the results at every radius are reported in Appendix D. WS-SAM and WS-PSAM share the warm-start ratio: τ=0.4\tau = 0.4 in vision, chosen by the sweep in Section 7.1, and τ=0.2\tau = 0.2 in the language tasks, fixed without tuning.

No separate validation data are used for this selection. Our experiments compare methods under a common protocol: every method is evaluated at every radius on the same data, which is the test set for vision and translation. With the shared ρ\rho grid, base-optimizer settings, and τ\tau, all methods receive comparable tuning budgets.

Q4. Radius ratio and choosing ρ\rho.

The 25×25\times ratio comes from GPT-2 (30M), so we ran a finer sweep around the best radii there [Appendix E.3]. The best radius moved from 0.2 to 0.16 for WS-SAM and from 5 to 4 for WS-PSAM, so the ratio remains 25×25\times (validation perplexity, 5 seeds):

WS-SAM ρ\rho0.10.130.160.20.3
ppl158.54158.36158.24158.37159.05
WS-PSAM ρ\rho234568
ppl155.92155.13154.88155.61156.74159.38

For choosing ρ\rho in practice, we suggest increasing it on a logarithmic grid until performance clearly degrades, and then refining the grid around the best radius, as above.

Q5. First step and numerical safeguards.

At the first step of PSAM without a warm start, the second-moment estimate is not yet available, so we use P1=IP_1 = I, that is, the SAM perturbation; from the second step on, Pt=Dt−1P_t = D_{t-1}. For the normalization, we compute ϵt=ρPt−1gt/(∥Pt−1gt∥2+10−12)\epsilon_t = \rho P_t^{-1}g_t / (\lVert P_t^{-1}g_t \rVert_2 + 10^{-12}), so ϵt=0\epsilon_t = 0 when gt=0g_t = 0. In addition, Pt−1P_t^{-1} is always well defined, since Dt−1D_{t-1} includes Adam's stability constant c=10−8c = 10^{-8}. We have added pseudocode including these details to Appendix A.2 [Algorithm 1].

Q6. Computational cost.

With TT training steps, AdamW uses TT gradient evaluations, SAM and PSAM use 2T2T, and their warm-started variants use (2−τ)T(2-\tau)T, that is, 1.6T1.6T in vision (τ=0.4\tau = 0.4) and 1.8T1.8T in the language tasks (τ=0.2\tau = 0.2). The warm start thus uses 20% (vision) and 10% (language) fewer gradient evaluations than SAM and PSAM, so its gains do not come from additional computation. On GPT-2 (30M), the counts and the runtime per run on a V100 GPU are [Appendix E.2]:

AdamWSAMPSAMWS-SAMWS-PSAM
gradient evaluations2,5005,0005,0004,5004,500
runtime (min)3.87.58.17.07.3

PSAM is slightly slower than SAM because it divides the gradient elementwise by Dt−1D_{t-1} to form the perturbation.

AuthorsOpenReview ↗

Response to Reviewer FaBw (3/3): Q7–Q9

Q7. Other SAM variants and ImageNet-1K.

Our main contribution is to analyze the mechanism of PSAM and how to use it effectively, rather than to benchmark SAM variants. Nevertheless, we agree that the comparison is useful, and we added ASAM (Kwon et al., 2021), Functional SAM (Func-SAM), and preconditioned Functional SAM (PFSAM), each with and without our warm start (WS) [Table 3]. We used GPT-2 (30M) on WikiText-2 with the same ρ\rho grid as in the paper; AdamW gives 160.73 ± 0.80. Note that the actual perturbation size of ASAM differs from ρ\rho.

Methodbest ρ\rhoval. pplWS best ρ\rhoWS val. ppl
SAM0.01160.23 ± 0.680.2158.37 ± 0.69
ASAM0.1159.99 ± 0.480.005160.49 ± 0.89
Func-SAM0.1158.40 ± 0.280.2156.01 ± 0.49
PSAM1157.30 ± 0.315155.61 ± 0.74
PFSAM1156.69 ± 0.6410151.62 ± 0.36

PSAM can also be combined with other SAM variants: PFSAM uses the PSAM perturbation with the Func-SAM gradient. PSAM and PFSAM outperform the other SAM variants both with and without WS. With WS, they also have much larger best radii (5 and 10), whereas Func-SAM, which uses the SAM perturbation, has the same best radius as SAM (0.2). This suggests that the large best radius of PFSAM, also reported by Singh et al. (2025), stems from the PSAM perturbation, consistent with our analysis.

Unfortunately, ImageNet-1K experiments are not feasible with our computational resources within the rebuttal period.

Q8. Missing references.

Thank you for the pointers. We have added both works to the related work (Section 2).

Q9. SAM on NLP tasks and large radii.

We agree that standard SAM often brings little gain on Transformer-based NLP tasks. However, Singh et al. (2025) showed that this can be resolved: PFSAM, which uses the PSAM perturbation, improves on AdamW in language modeling up to billion-parameter models. Building on this, our work explains why PSAM works with Adam-type optimizers and how to use it effectively.

The cost of two gradient evaluations per step and the behavior in large-batch training are shared by SAM-type methods in general rather than specific to PSAM; we report the cost in Q6.

With a warm start, PSAM improves on both AdamW and SAM by more than the standard deviation in both settings of Figure 1 (validation perplexity):

AdamWWS-SAMWS-PSAM
GPT-2 (30M)160.73 ± 0.80158.37 ± 0.69155.61 ± 0.74
GPT-2 (124M)18.28 ± 0.0518.20 ± 0.0718.09 ± 0.03

As with any SAM-type method, a larger radius strengthens the regularization but eventually harms convergence, so a smaller radius should be used if training diverges (see Q4). PSAM's perturbation can keep the descent guarantee up to larger radii than SAM's (Section 5.3), which explains why its best radius is larger.

We hope these responses address the reviewer's concerns, and we would be glad to answer any further questions.

Reviewer_NyfPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the theoretical properties and practical behaviour of Preconditioned Sharpness-Aware Minimization (PSAM). Specifically, why it could allow a larger perturbation radius than standard SAM. The authors show that PSAM is equivalent to SAM in scaled geometry when the perturbation and update share a preconditioner, and derive stationarity and fixed-metric PAC-Bayes bounds. They further establish a descent property under quasiconvexity, find that geometry mismatch may invalidate this guarantee, and give hypothesis to explain large radius of PSAM. Moreover, the authors find that a warm start is quite important for PSAM and helps improve performance across vision and NLP tasks.

Critical strengths
  1. The descent analysis under quasiconvexity offers a useful perspective on the importance of matching optimization geometries for inner and outer steps in SAM, which is an interesting theoretical finding.
  2. The admissible-radius analysis and the relationship between perturbation-direction alignment and optimal radius provide an interesting hypothesis for PSAM.
  3. The experiments show that PSAM with warm start shows improvement across vision and language tasks.
  4. Empirical results of sweeping radius for SAM and PSAM align with theoretical analysis.
Critical weaknesses
  1. The paper's central claim is that PSAM can use a larger perturbation radius than SAM, but whether the larger radius can lead to better performance/generalization is unclear and hasn't been discussed from a theoretical perspective.
  2. The explanation for PSAM's larger admissible radius is heuristic. While the paper proves a descent property under quasiconvexity, it doesn't establish that PSAM admits a larger radius than SAM.
  3. The paper discusses existing works on preconditioned perturbations, but the main difference between them could be clearer. Riemannian SAM (Yun & Yang, 2023) has explored gradient alignment in preconditioned geometry and established descent inequalities. Although this paper proposes quasiconvexity-based descent guarantee and geometry mismatch counterexample, it doesn't sufficiently clarify theoretical significance over these existing analyses.
  4. Although the paper has discussed existing warm-start SAM methods in Related Works, the proposed method is closely related to prior work. Specifically, Singh et al. (2025) have explored perturbation-radius warm-up to stabilize PSAM. Thus, warm-start PSAM introduced in this paper wouldn't be a main contribution.
  5. Experiments are limited to SAM and PSAM variants. Since there is no broad agreement that PSAM is the best geometry-aware SAM variant, comparing with other SAM variants is necessary, especially the most relevant ones like Riemannian SAM (Yun & Yang, 2023), Func-SAM (Singh et al., 2025), and InfoSAM (Zhang et al., 2025).
Questions for authors
  1. PreSAM in (Zhang et al., 2025) may be more aligned with the preconditioned SAM used in this paper. The perturbation in (Singh et al., 2025) is actually different from Eq.6, which is ρP−1g∥g∥\rho P^{-1} \frac{g}{\|g\|}. The authors should double-check this detail.
  2. (Singh et al., 2025) explores why SAM works in vision tasks while not in language tasks. This paper finds that cosine similarity between gg and P−1gP^{-1}g in language tasks is lower than in vision tasks. A deeper discussion relating this mismatch to performance rather than just larger radius would be interesting.
4 comments
AuthorsOpenReview ↗

Response to Reviewer NyfP (1/2): W1–W5

We thank the reviewer for the time and the constructive comments. We have revised the paper accordingly, and the responses below refer to the revised version.

W1. Larger radius and performance.

We do not claim that a larger radius makes PSAM generalize better than SAM. In both methods, the radius is the Euclidean length of the perturbation, but the neighborhood of PSAM is an ellipsoid rather than a ball, so the radii are hard to compare in terms of generalization. Our theory instead suggests that PSAM may work better with Adam: it keeps the descent guarantee [Corollary 5.10], which a SAM perturbation combined with a preconditioned update can lose [Appendix B.7]. In practice, PSAM outperforms SAM across vision, translation, and language modeling when both use a warm start.

W2. Heuristic explanation of the larger radius.

We agree, and we note this in Section 5.3. The larger radius itself is observed empirically in translation and language modeling; the admissible radius is our geometric explanation for it. Although a rigorous proof is difficult for deep networks, the observations agree with what the explanation predicts: the radius gap between PSAM and SAM is larger when their perturbations are less aligned (Table 2, Section 7.4).

W3. Theoretical significance over Riemannian SAM.

We agree and have clarified this in Section 2. Yun & Yang (2023) obtain the alignment between the gradient and the SAM gradient in a preconditioned geometry from smoothness, so the radius at which it is guaranteed is limited only by the smoothness constant. We show that quasiconvexity guarantees the alignment at any radius, as long as the perturbed point stays in the quasiconvex region. The admissible radius then depends on the local geometry and the perturbation direction, which underlies our explanation of why PSAM can use larger radii (Section 5.3).

In addition, their analysis assumes that the perturbation and the update share the same metric, whereas we show why this matching is needed: descent can fail without it [Appendix B.7].

W4. Relation to the radius warm-up of Singh et al. (2025).

Singh et al. (2025) mention a radius warm-up for PSAM and SAM only briefly, as an occasional remedy for numerical instability. We added a comparison between our warm start and a radius warm-up [Table 13], detailed in our response to Reviewer NuqS (W1). At their best radii, our warm start performs better for both SAM and PSAM, with 10% fewer gradient evaluations.

We agree that the warm start itself is not new, as it follows Park et al. (2026). Our contribution is to show how to use it effectively with PSAM, and to analyze why.

W5. Comparison with other SAM variants.

We do not claim that PSAM is the best geometry-aware SAM variant. Our main contribution is to analyze the mechanism of PSAM and how to use it effectively. Nevertheless, we agree that the comparison is useful, and we added ASAM (Kwon et al., 2021), Functional SAM (Func-SAM), and preconditioned Functional SAM (PFSAM), each with and without our warm start (WS) [Table 3]. We used GPT-2 (30M) on WikiText-2; AdamW gives 160.73 ± 0.80. Note that the actual perturbation size of ASAM differs from ρ\rho.

Methodbest ρ\rhoval. pplWS best ρ\rhoWS val. ppl
SAM0.01160.23 ± 0.680.2158.37 ± 0.69
ASAM0.1159.99 ± 0.480.005160.49 ± 0.89
Func-SAM0.1158.40 ± 0.280.2156.01 ± 0.49
PSAM1157.30 ± 0.315155.61 ± 0.74
PFSAM1156.69 ± 0.6410151.62 ± 0.36

PSAM can also be combined with other SAM variants: PFSAM uses the PSAM perturbation with the Func-SAM gradient. PSAM and PFSAM outperform the other SAM variants both with and without WS. With WS, they also have much larger best radii (5 and 10), whereas Func-SAM, which uses the SAM perturbation, has the same best radius as SAM (0.2). This suggests that the large best radius of PFSAM, also reported by Singh et al. (2025), stems from the PSAM perturbation, consistent with our analysis.

AuthorsOpenReview ↗

Response to Reviewer NyfP (2/2): Q1–Q2

Q1. PreSAM and Eq. 6.

Our Eq. 6, ϵt=ρPt−1gt/∥Pt−1gt∥2\epsilon_t = \rho P_t^{-1}g_t/\lVert P_t^{-1}g_t\rVert_2, follows the published version of Singh et al. (ICML 2025, Eq. 12), which normalizes the preconditioned gradient; the form ρP−1g/∥g∥\rho P^{-1}g/\lVert g\rVert appears in their earlier arXiv version. In the preSAM framework of Zhang et al. (2025), Eq. 6 is the OP instance with Ct=Pt−1C_t = P_t^{-1}, as noted in Section 2. PreSAM, however, pairs this perturbation with an SGD update, whereas PSAM pairs it with Adam, whose preconditioner defines PtP_t; this matched setting is what we study.

Q2. Relating the alignment to performance.

Thank you for the suggestion. In our experiments, the performance gain of PSAM does not directly follow the alignment: WS-PSAM outperforms WS-SAM both on the vision models, where the cosine is 0.40–0.50, and on the NLP tasks, where it is 0.04–0.06. We instead attribute the gain to the matched geometry: with Adam, a SAM perturbation is mismatched with the update and can lose the descent guarantee, whereas PSAM keeps it (Section 5.3). The alignment thus relates to the radius gap (Section 7.4) rather than directly to performance.

We hope these responses address the reviewer's concerns, and we would be glad to answer any further questions.

Thanks to the authors for the clarifications and additional experiments. My main concerns remain:

  1. I understand that the authors don't claim that a larger radius in PSAM would lead to better performance or generalization. Then the core question is: what does this independent finding, "PSAM allows a larger perturbation radius", provide practical or theoretical value? Why is it a key question to explore?

  2. The descent analysis about geometry matching is interesting, which serves as an additional explanation for PSAM's motivation but not the key contribution in this paper. However, Appendix E.4 shows substantial mismatch, and enforcing exact matching causes training to fail. This observation is interesting but lacks analysis. It remains unclear how the theoretical results explain the observed performance improvements.

AuthorsOpenReview ↗

Response to Reviewer NyfP's follow-up

Thank you for the follow-up.

1. Value of the larger radius. We first note two points.

  • Why PSAM. PSAM is worth studying in its own right: it was proposed for Adam-trained Transformers in NLP, where SAM brings little gain (Singh et al., 2025), and when both use a warm start, PSAM on its own outperforms SAM on all our tasks.
  • Radius and performance. To clarify our W1 response, within a method a larger radius regularizes more strongly and improves performance until it starts to harm convergence; we only meant that radii are hard to compare across methods (ball vs. ellipsoid).

We now explain the practical and theoretical value of the finding that PSAM allows a larger radius.

Practical value.

  • PSAM's gain requires a larger radius. On the NLP tasks, WS-PSAM outperforms WS-SAM only at radii beyond WS-SAM's best. At WS-SAM's best radius (0.2), WS-PSAM is no better than WS-SAM (GPT-2 30M: 159.52 vs. 158.37 ppl), whereas at its own best radius (5) it outperforms WS-SAM (155.61); the same holds on GPT-2 (124M) and IWSLT14 (Tables 7–9). The radius thus decides whether PSAM improves on SAM.
  • Tuning guideline. WS-PSAM's radius search must therefore extend beyond WS-SAM's best radius (our reply to FaBw, Q4).

Theoretical value.

  • A new view of the radius. The radius is usually seen only as a hyperparameter that sets the strength of regularization. The finding shows that it also has an upper limit explained by the descent property: a direction less aligned with the gradient, such as PSAM's, travels farther before leaving the region where descent is guaranteed, so it can use a larger radius (Section 5.3).
  • Theory and experiment agree. The radius gap appears only on the NLP tasks, where the alignment is low (Table 2). Our analysis also explains why preconditioned Functional SAM works well at large radii, as Singh et al. (2025) reported: it uses the PSAM perturbation (Table 3). The finding is thus both explained by our theory and evidence for it.

2. What the theory explains, and Appendix E.4. We acknowledge that our descent analysis does not fully apply to practical PSAM. We therefore measured the descent directly, re-running the WS-SAM (ρ=0.2\rho = 0.2) and WS-PSAM (ρ=5\rho = 5) runs of Table 8 (GPT-2 30M, 5 seeds), which reproduce Table 8 exactly. Over the last epoch, we tested several radii and recorded the descent term of Corollary 5.10, gt⊤Dt−1gt+g_t^\top D_t^{-1} g_t^+, and the actual decrease in the minibatch loss after the AdamW step:

perturbation radiusnone0.21510
WS-SAM: descent term16251698125812221
WS-SAM: loss decrease (10−310^{-3})2.042.131.560.140.02
WS-PSAM: descent term1826182618291840826
WS-PSAM: loss decrease (10−310^{-3})2.312.312.312.321.26

As the table shows, the descent term tracks the actual loss decrease. At each method's best radius (bold), WS-PSAM decreases the loss more than WS-SAM. This is broadly consistent with our analysis and suggests that PSAM converges better.

The final training and validation losses of WS-PSAM and WS-SAM then make the source of the gain clear (training loss averaged over the last epoch; 5 seeds):

training lossvalidation lossgap
AdamW3.823 ± 0.0125.080 ± 0.0051.257 ± 0.012
WS-SAM (ρ=0.2\rho = 0.2)3.952 ± 0.0125.065 ± 0.0041.113 ± 0.014
WS-PSAM (ρ=5\rho = 5)3.893 ± 0.0145.047 ± 0.0051.154 ± 0.014

WS-SAM has the smaller gap, but WS-PSAM reaches a lower training loss and, in the end, a lower validation loss. This indicates that better convergence plays a major role in PSAM's gain.

Finally, we note that the ablation that shares Dt−1D_{t-1} (Appendix E.4) fails even at ρ=0.005\rho = 0.005, where the perturbation is negligible, so the failure comes from changing Adam's update, not from matching itself.

We hope this addresses your concerns.

Reviewer_NuqSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies an existing preconditioned SAM (PSAM) method and its combination with a warmup Adam(W) phase. It expresses a matched-preconditioner PSAM step as ordinary SAM in scaled coordinates, derives a stationarity bound and a fixed-metric PAC-Bayes bound, and proves a minibatch descent property under local quasiconvexity. Experiments on CIFAR-100, IWSLT14, and WikiText show that warm-started PSAM achieves the best mean performance among the evaluated methods.

Critical strengths
  1. The theory arguments are rigorous, and the proofs are clearly presented.
  2. Experiments across vision and language tasks, together with ablations of the warm-start ratio and perturbation radius, provide consistent evidence for the practical benefits of warm-started PSAM.
Critical weaknesses
  1. The empirical advance over prior works is not adequately established. Singh et al. [1] already discuss perturbation warm-up for PSAM and SAM, and evaluate preconditioned Functional SAM. The comparisons against a smooth radius warm-up are important for judging its practical contribution.
  2. Limited novelty. PSAM itself was introduced in prior work, and Park et al. [2] already explored a similar warm start using the base optimizer. Moreover, interpreting a PSAM step as SAM in scaled coordinates follows naturally from applying matched preconditioners to the perturbation and update. The contribution therefore appears primarily to lie in the analysis and empirical investigation of these existing components.

[1] Singh et al. (2025). Avoiding spurious sharpness minimization broadens applicability of SAM.

[2] Park et al. (2025). Sharpness-Aware Minimization Can Hallucinate Minimizers.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer NuqS

We thank the reviewer for the time and the constructive comments. We have revised the paper accordingly, and the responses below refer to the revised version.

W1. Prior work and comparison with a radius warm-up.

Singh et al. (2025) mention a radius warm-up for PSAM and SAM only briefly, as an occasional remedy for numerical instability. They also report a larger best radius for preconditioned Functional SAM, but do not study the radius of PSAM. We show that, with a warm start, PSAM itself has a much larger best radius than SAM on the NLP tasks, and explain why.

As suggested, we added a comparison between our warm start (WS) and a radius warm-up [Table 13]. Since the warm-up details are not given, we use a linear warm-up over the same first 20% of training, ρt=ρmin⁡(1,t/(0.2T))\rho_t = \rho \min(1, t/(0.2T)) (GPT-2 (30M) on WikiText-2, 5 seeds):

Methodbest ρ\rhoval. pplgrad. evals
SAM0.01160.23 ± 0.685,000
SAM + radius warm-up0.05159.85 ± 0.715,000
WS-SAM0.2158.37 ± 0.694,500
PSAM1157.30 ± 0.315,000
PSAM + radius warm-up2156.13 ± 0.815,000
WS-PSAM5155.61 ± 0.744,500
  • Our warm start is better than the radius warm-up for both SAM and PSAM, with 10% fewer gradient evaluations.
  • In every setting, the best radius of PSAM is much larger than that of SAM.

W2. Novelty.

Existing components. We agree that our methodological novelty is limited: PSAM is from Singh et al. (2025), and our warm start follows Park et al. (2026). Our contribution is to identify how to use them effectively together, and to analyze why, both theoretically and empirically. For example, in language modeling, warm-started PSAM performs best at radii up to 25 times those of warm-started SAM, where SAM itself fails. To our knowledge, this has not been studied before.

Scaled-coordinate equivalence. We agree that the equivalence is intuitive. Turning it into guarantees, however, is not immediate: global smoothness is hard to justify in the scaled space, so we identify the assumptions under which the guarantees of SAM transfer to PSAM. In our ViT experiments, we also track the quantities these assumptions bound and find that they stay stable after the early phase of training [Appendix C]. The equivalence also underlies our descent and admissible-radius analyses.

We hope these responses address the reviewer's concerns, and we would be glad to answer any further questions.

Thank you for your clarification. My main concern has been addressed. I've increased my score to 3.

AuthorsOpenReview ↗

Thank you for the follow-up

Thank you for taking the time to read our responses and for raising your score. We are glad the clarification was useful, and your comments genuinely helped us improve the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank all reviewers for their constructive comments. We have uploaded a revised version of the paper. The main changes are listed below, with the reviewers whose comments prompted them; all new experiments use GPT-2 (30M) on WikiText-2 with five seeds.

Main text

  • Section 7.5 (new). We compare PSAM with ASAM, Functional SAM, and preconditioned Functional SAM, each with and without a warm start (Table 3; NyfP, FaBw). We also analyze the warm start further: the preconditioner drift in language modeling, an optimizer-state reset at the switch, and a comparison with a radius warm-up (FaBw, NuqS, NyfP). Finally, we check the robustness of the alignment in Table 2 and of the 25×25\times radius gap (FaBw).
  • Section 2. We added Baek et al. (2024) and Tan et al. (2026) (FaBw) and clarified how our analysis differs from that of Yun & Yang (2023) (NyfP).

Appendix

  • Algorithm 1 (Appendix A.2). The full WS-PSAM procedure, including the first step and the numerical safeguards (FaBw).
  • Appendix E (new). Details and full results of the additional experiments:
    • E.1: the comparison with other SAM variants (NyfP, FaBw).
    • E.2: the warm-start analysis and the computational cost (FaBw, NuqS, NyfP).
    • E.3: robustness checks, namely the alignment over the last epoch and a finer radius sweep around the best radii (FaBw).
    • E.4: the perturbation and update directions with Dt−1D_{t-1} and DtD_t (FaBw).

Our individual responses discuss each change in detail.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.