- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the theoretical properties and practical behavior of preconditioned sharpness-aware minimization (PSAM). The authors show that, when the perturbation and update share a preconditioner, each PSAM step is equivalent to a standard SAM step in scaled coordinates. They derive a stationarity bound and a fixed-metric PAC-Bayes bound, and use a quasiconvex descent argument to motivate an explanation for larger admissible perturbation radii.
Critical strengths
S1. The scaled-coordinate interpretation provides a clear perspective on the interaction between preconditioned perturbations and parameter updates.
S2. The finding that warm-started PSAM remains effective at radii where other methods deteriorate is practically useful, particularly in the NLP settings.
S3. The paper goes beyond final performance by examining preconditioner drift, perturbation alignment, and a loss-plane visualization.
Critical weaknesses
-
I am not fully convinced that the shared-preconditioner analysis directly explains the behavior of the practical Adam(W) implementation. In the theory, the perturbation and parameter update use the same preconditioner. In practice, however, the perturbation is constructed using , while the parameter update uses together with momentum. The authors do acknowledge this mismatch, but Figure 3 only reports the relative Frobenius drift of the preconditioner, which does not directly tell us how much the actual update direction changes. It would be more convincing to compare with , or to include an ablation where the preconditioner is kept fixed within each step. This would make the connection between the theory and the actual implementation much clearer.
-
The evidence linking perturbation direction to the larger useful radius remains limited. Table 2 measures the cosine similarity between SAM and PSAM perturbations only at the end of training and compares five different model/task settings. Since the architecture, dataset, and training setup also differ, the table does not isolate whether the directional difference explains the gap in useful radii. Figure 7 covers only one model, one seed, and one mini-batch. The authors could strengthen this explanation by measuring the similarity at several training stages and across mini-batches within the same model and task, and checking whether lower similarity consistently corresponds to a larger useful-radius gap.
-
The paper shows that warm starting improves PSAM, but the explanation that this benefit comes from preconditioner stabilization needs more direct evidence. The diagnostics in Figures 3 and 5 use vision runs without warm starting, so they do not show how the preconditioner behaves at the actual WS-PSAM switch or in the NLP experiments. Warm starting also changes the model parameters, loss level, and optimizer state, so the current results do not separate these possible explanations. The authors could report preconditioner drift and update-direction changes around the switch, especially in the NLP settings. Comparing runs that retain or reset the second-moment estimate at the same switch checkpoint, with initialization and bias correction handled consistently, would also help assess the contribution of the accumulated preconditioner.
Questions for authors
- The analysis uses a shared preconditioner for perturbation and update, whereas the practical implementation uses different preconditioner estimates and includes momentum. Please measure how closely the actual update direction matches the direction analyzed in the theory, especially before and after the warm-start switch. Reporting cosine similarity between these directions would be more informative than preconditioner drift alone. A controlled ablation using the same preconditioner for both operations would also help assess the practical effect of this difference.
- Table 2 measures alignment only at the end of training across different model/task settings, and Figure 7 covers only one model, one seed, and one mini-batch. Please track alignment at several training stages and across mini-batches and radii within the same model and task, and check whether better alignment consistently corresponds to a larger useful radius. For the warm-start explanation, please also report preconditioner behavior around the switch in the NLP experiments.
- How are the perturbation radius and warm-start ratio selected? Please clarify whether independent validation data are used for the vision and translation experiments and whether all methods receive comparable tuning budgets. The current description leaves this protocol unclear.
- The reported 25-fold radius ratio compares optima on a relatively coarse grid. Please qualify it as the best ratio on the tested grid, or provide a finer sweep around the best-performing radii. It is important to provide a guideline to choose in practice.
- Please specify how the preconditioner is initialized for the first step of non-warm-started PSAM and how normalized perturbations are computed when the gradient norm is zero. Including the initialization and any numerical safeguards in the pseudocode would improve reproducibility.
- Warm starting uses only one gradient evaluation per step during its initial phase, so equal numbers of training steps do not necessarily imply equal computational cost. Please report total gradient-evaluation counts or runtime for the compared methods, together with the warm-start ratios.
- The experiments only compare argainst PSAM and SAM, lacking other popular baselines such as ASAM and FisherSAM. More importantly, the results on ImageNet-1K is also missing. It is worth noting that the proposed optimizer may perform dramatically different on small-scale and large-scale datasets.
- Some recent works in the literature review are missing. For example, (Baek et al, 2024) studies why SAM is robust to the label noise and (Tan et al, 2026) also investigates the benefits of SAM on model calibration.
- Baek et al, WHY IS SAM ROBUST TO LABEL NOISE (ICLR, 2024)
- Tan et al, Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization (ICLR, 2026)
- Frankly speaking, SAM is not a good choice for Transformer-based NLP tasks. This is because on one hand, it requires two backpropagations each step. On the other hand, SAM performs even much worse than the base optimizer in the large-batch training. This is a crtical issue. Just as Figure 1 depicts, the improvement is neglible and increasing may even explode the training process. It would be more convincing if the authors could clarify why it is essential to use large for PSAM.
5 comments
Response to Reviewer FaBw (2/3): Q3–Q6
Q3. Tuning protocol.
For each task, we sweep over the same grid for all SAM-type methods and take each method's best radius; the results at every radius are reported in Appendix D. WS-SAM and WS-PSAM share the warm-start ratio: in vision, chosen by the sweep in Section 7.1, and in the language tasks, fixed without tuning.
No separate validation data are used for this selection. Our experiments compare methods under a common protocol: every method is evaluated at every radius on the same data, which is the test set for vision and translation. With the shared grid, base-optimizer settings, and , all methods receive comparable tuning budgets.
Q4. Radius ratio and choosing .
The ratio comes from GPT-2 (30M), so we ran a finer sweep around the best radii there [Appendix E.3]. The best radius moved from 0.2 to 0.16 for WS-SAM and from 5 to 4 for WS-PSAM, so the ratio remains (validation perplexity, 5 seeds):
| WS-SAM | 0.1 | 0.13 | 0.16 | 0.2 | 0.3 |
|---|---|---|---|---|---|
| ppl | 158.54 | 158.36 | 158.24 | 158.37 | 159.05 |
| WS-PSAM | 2 | 3 | 4 | 5 | 6 | 8 |
|---|---|---|---|---|---|---|
| ppl | 155.92 | 155.13 | 154.88 | 155.61 | 156.74 | 159.38 |
For choosing in practice, we suggest increasing it on a logarithmic grid until performance clearly degrades, and then refining the grid around the best radius, as above.
Q5. First step and numerical safeguards.
At the first step of PSAM without a warm start, the second-moment estimate is not yet available, so we use , that is, the SAM perturbation; from the second step on, . For the normalization, we compute , so when . In addition, is always well defined, since includes Adam's stability constant . We have added pseudocode including these details to Appendix A.2 [Algorithm 1].
Q6. Computational cost.
With training steps, AdamW uses gradient evaluations, SAM and PSAM use , and their warm-started variants use , that is, in vision () and in the language tasks (). The warm start thus uses 20% (vision) and 10% (language) fewer gradient evaluations than SAM and PSAM, so its gains do not come from additional computation. On GPT-2 (30M), the counts and the runtime per run on a V100 GPU are [Appendix E.2]:
| AdamW | SAM | PSAM | WS-SAM | WS-PSAM | |
|---|---|---|---|---|---|
| gradient evaluations | 2,500 | 5,000 | 5,000 | 4,500 | 4,500 |
| runtime (min) | 3.8 | 7.5 | 8.1 | 7.0 | 7.3 |
PSAM is slightly slower than SAM because it divides the gradient elementwise by to form the perturbation.
Response to Reviewer FaBw (3/3): Q7–Q9
Q7. Other SAM variants and ImageNet-1K.
Our main contribution is to analyze the mechanism of PSAM and how to use it effectively, rather than to benchmark SAM variants. Nevertheless, we agree that the comparison is useful, and we added ASAM (Kwon et al., 2021), Functional SAM (Func-SAM), and preconditioned Functional SAM (PFSAM), each with and without our warm start (WS) [Table 3]. We used GPT-2 (30M) on WikiText-2 with the same grid as in the paper; AdamW gives 160.73 ± 0.80. Note that the actual perturbation size of ASAM differs from .
| Method | best | val. ppl | WS best | WS val. ppl |
|---|---|---|---|---|
| SAM | 0.01 | 160.23 ± 0.68 | 0.2 | 158.37 ± 0.69 |
| ASAM | 0.1 | 159.99 ± 0.48 | 0.005 | 160.49 ± 0.89 |
| Func-SAM | 0.1 | 158.40 ± 0.28 | 0.2 | 156.01 ± 0.49 |
| PSAM | 1 | 157.30 ± 0.31 | 5 | 155.61 ± 0.74 |
| PFSAM | 1 | 156.69 ± 0.64 | 10 | 151.62 ± 0.36 |
PSAM can also be combined with other SAM variants: PFSAM uses the PSAM perturbation with the Func-SAM gradient. PSAM and PFSAM outperform the other SAM variants both with and without WS. With WS, they also have much larger best radii (5 and 10), whereas Func-SAM, which uses the SAM perturbation, has the same best radius as SAM (0.2). This suggests that the large best radius of PFSAM, also reported by Singh et al. (2025), stems from the PSAM perturbation, consistent with our analysis.
Unfortunately, ImageNet-1K experiments are not feasible with our computational resources within the rebuttal period.
Q8. Missing references.
Thank you for the pointers. We have added both works to the related work (Section 2).
Q9. SAM on NLP tasks and large radii.
We agree that standard SAM often brings little gain on Transformer-based NLP tasks. However, Singh et al. (2025) showed that this can be resolved: PFSAM, which uses the PSAM perturbation, improves on AdamW in language modeling up to billion-parameter models. Building on this, our work explains why PSAM works with Adam-type optimizers and how to use it effectively.
The cost of two gradient evaluations per step and the behavior in large-batch training are shared by SAM-type methods in general rather than specific to PSAM; we report the cost in Q6.
With a warm start, PSAM improves on both AdamW and SAM by more than the standard deviation in both settings of Figure 1 (validation perplexity):
| AdamW | WS-SAM | WS-PSAM | |
|---|---|---|---|
| GPT-2 (30M) | 160.73 ± 0.80 | 158.37 ± 0.69 | 155.61 ± 0.74 |
| GPT-2 (124M) | 18.28 ± 0.05 | 18.20 ± 0.07 | 18.09 ± 0.03 |
As with any SAM-type method, a larger radius strengthens the regularization but eventually harms convergence, so a smaller radius should be used if training diverges (see Q4). PSAM's perturbation can keep the descent guarantee up to larger radii than SAM's (Section 5.3), which explains why its best radius is larger.
We hope these responses address the reviewer's concerns, and we would be glad to answer any further questions.
Response to Reviewer FaBw (1/3): W1–W3, Q1–Q2
We thank the reviewer for the time and the constructive comments. We have revised the paper accordingly, and the responses below refer to the revised version.
W1 / Q1. Shared preconditioner in theory and practice.
With Pt=Dt, our analysis uses the same update preconditioner as practical PSAM, so the gap lies in the perturbation, which uses Dt−1. At the best radius (ρ=5) on GPT-2 (30M), we therefore measured cos(Dt−1−1gt,Dt−1gt) along with the suggested cos(Dt−1−1gt+,Dt−1gt+) [Appendix E.4]:
The former is higher, so the perturbation deviates less than the suggested comparison indicates; still, in both cases the directions differ, and we acknowledge this gap. In the suggested ablation that shares Dt−1 between the perturbation and the update, training failed at every radius, likely because the update is prone to diverge without the update gradient in its preconditioner.
W2 / Q2. Reliability of the alignment in Table 2.
Our alignment analysis assumes quasiconvexity, which is more plausible near a minimizer (Section 5.3), so Table 2 reports the alignment at the final training step. We agree that a single step is a limited sample. We therefore computed cos(gt,Dt−1−1gt) at every step of the last epoch, that is, over many mini-batches, for WS-PSAM on GPT-2 (30M) with 11 radii and 5 seeds [Appendix E.3]:
The other five radii give 0.048–0.049. At every radius, the alignment stays at the level in Table 2 (0.040) with small variation across steps and seeds, so the reported value is reliable. We address the preconditioner behavior around the switch in W3.
W3. Warm start and the preconditioner.
Our warm start aims to skip the unstable early phase of training rather than to carry over the accumulated preconditioner (Sections 5.3 and 6). To separate the two, we ran the suggested ablation on GPT-2 (30M), resetting m, v, and both bias-correction counters at the switch [Appendix E.2]. Resetting did not hurt at any radius (validation perplexity, 5 seeds):
The benefit therefore comes from skipping the early phase, not from the accumulated preconditioner.
As also suggested, we measured the preconditioner drift in language modeling (GPT-2 (30M)) [Appendix E.2]. At the best radius (ρ=5), it is large only in the first 10 steps, which the warm start skips, and stays small afterward, including around the switch at step 501:
Update directions around the switch are reported in W1.