acceptodds
Under review as a conference paper at ICLR 2027

SHARD-SAM: Signed Hessian-Amplified Radius and Direction for Sharpness-Aware Minimizatio

18 views

Abstract

Reconciling optimization and generalization remains a fundamental challenge in deep learning. Sharpness-Aware Minimization (SAM) advances this goal, yet its fixed perturbation radius and first-order geometric approximation limit its adaptability to heterogeneous loss landscapes. We propose SHARD-SAM (Signed Hessian-Amplified Radius and Direction), a sharpness-aware framework that jointly adapts perturbation magnitude and orientation through a shared curvature signal. SHARD-SAM constructs a signed Hessian-amplified spectral surrogate using zeroth-order curvature probes. A symmetric asymptotic envelope adjusts the perturbation radius relative to a running curvature baseline, while magnitude-adaptive gating combines the gradient with a spectral correction. A shared curvature-temperature parameter controls both modules, coupling radius and direction adaptation within a unified perturbation rule. Our theoretical analysis characterizes the surrogate's spectral properties, estimation error, and conditions for local directional benefits. Experiments on CIFAR-10, CIFAR-100, and ImageNet across CNN and ViT architectures demonstrate consistent generalization improvements over representative first-order and second-order geometry-adaptive SAM optimizers, with a favorable accuracy–efficiency trade-off among the evaluated curvature-aware methods. Ablations and direction-source controls support the contribution of the shared curvature signal and its two adaptive modules.

Primary area:
optimization
Keywords:
sharpness-aware minimization, deep learning optimization, curvature-aware optimization, zeroth-order curvature estimation, generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zH99OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SHARD-SAM, which adaptively adjusts the perturbation radius and direction through shared curvature signals. The core of the method is to use zeroth-order Gaussian probes to construct a signed Hessian-amplified spectral surrogate, extract curvature responses from forward-only loss queries, and then obtain the principal direction and curvature score through power iteration. The radius is dynamically adjusted by a symmetric asymptotic envelope based on the ratio of the score magnitude to its EMA baseline, while the effective direction combines the gradient with a spectral correction term through magnitude-adaptive gating.

Critical strengths

1.This paper uses an interesting method to simultaneously adjust the radius and direction in SAM, solving a known problem. 2. Numerous experiments show that the proposed optimizer has improved performance.

Critical weaknesses

1.The problem described in the paper is well-known, but the solution is not intuitive. Specifically, curvature awareness comes from CR-SAM, constructing spectral proxies similar to TSAM, and adaptive radii similar to momentum construction in MSAM, but it is not yet known why these modules must be combined for an intuitive reason. 2.Theoretical analysis is only based on specific Hessian assumptions and training objectives and is not related to specific neural networks, which weakens persuasiveness.

Questions for authors

1.Can the proposed optimizer theoretically derive its advantages in simple neural networks, such as two-layer ReLU and other widely used simple networks? 2.Can the proposed optimizer demonstrate advantages in other fields, such as language modeling, time series prediction, etc.?

2 comments
AuthorsOpenReview ↗

Why SHARD couples radius and direction, with ReLU theory and forecasting results

Response to Reviewer zH99

Thank you for suggesting the two-layer ReLU example.

Rationale for coordinating radius and direction (W1)

A rotation toward higher curvature can raise the inner loss at one radius and lower it at another. This is why SHARD coordinates radius and direction. In the local loss model, rotating away from the gradient gives up some alignment. That loss in the gradient term scales linearly with the radius, while the gain in the curvature term scales quadratically (B.1, Eq. B.11).

CR-SAM regularizes curvature, Tilted SAM reweights neighborhood losses, and Momentum-SAM uses momentum to construct the perturbation direction. SHARD takes both a direction and a score from a single signed spectral operator. The gate compares the score magnitude with the gradient norm to set the strength of the rotation. The radius controller compares the same magnitude with its running average. If the score stays high, the direction can remain tilted even as the radius returns near its base value (Secs. 4.1–4.3).

The two ablations on CIFAR-100/WRN test different parts of the design. Adapting radius and direction together gives 84.93% accuracy, compared with 83.92% for radius alone and 84.18% for direction alone (Fig. 2(a), Table S1). To test sharing, we kept both controls active. Sharing ten probes gives 84.912%, while splitting them into two independent sets of five gives 84.510%. Even with ten independent probes per control, accuracy is 84.750% (new F.1, Fig. S13).

Network-specific curvature assumptions (W2)

In a two-layer ReLU network with squared loss and fixed output weights, the loss is exactly quadratic over any region where the activation signs stay fixed. The Hessian in that region is H = ΦᵀΦ/n, where the features in Φ depend on the inputs, output weights, and active units (new G.1, Proposition 1). This network thus satisfies the quadratic assumption in A.1. When both layers are trainable, the loss also contains cubic and quartic terms. Propositions 4–5 keep those terms and give a sufficient condition for the rotation to increase the inner loss (G.4).

Local advantage over equal-radius SAM (Q1)

At the same radius, SHARD can attain a higher inner loss than SAM and a lower training loss after one outer update. For the fixed-output ReLU model, let u be the unit gradient, v the sampled direction, and c = uᵀv. If Γ(v) = (v − cu)ᵀHu > 0, a small rotation toward v improves the curvature term to first order in the gate and costs gradient alignment only to second order. One Hessian-vector product suffices to check this condition. Proposition 2 gives the gate interval for which the inner loss increases (G.2, Eq. G.10).

For sufficiently small positive values of both the gate and the learning rate, the same condition gives a lower training loss after one update if the perturbed points and update endpoints stay in the same activation region (Proposition 3, Eqs. G.16–G.17). If SHARD also shrinks the radius, the gain from rotation must offset the effect of that reduction for SHARD to have lower loss to first order in the learning rate (Eq. G.18). Appendix G.5 derives corresponding conditions for linear regression, two-layer linear networks, and fixed-feature logistic regression.

Beyond image classification (Q2)

In time-series forecasting, SHARD has the lowest five-seed mean test MSE in all 16 settings. We ran PatchTST and DLinear on ETTh1 and Weather at four forecast horizons, comparing SHARD with AdamW, SAM, and SASSHA. SHARD reduces MSE by 1.40%–2.96% relative to SASSHA, the strongest baseline in every setting, and by 3.33%–6.63% relative to AdamW (new F.6, data underlying Fig. S34). We split the data chronologically, fitted normalization on training data, and evaluated the final checkpoint at 50 epochs. Figs. S35–S36 report test MAE.

Dear Reviewer,

We’re nearing the end of the discussion period. This is a friendly reminder to leave a comment for the authors.

Thank you

~ AC

Reviewer_q1DZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose a SAM variants called SHARD-SAM that construct a signed Hessian-amplified spectral surrogate using zero order finite difference probes and extracts a dominate spectral direction through power iteration. This method use the shared curvature signal to adjust perturbation radius and direction. This paper conduct experiment on CIFAR-10, CIFAR-100, ImageNet using CNN and VIT architectures shows that SHARED-SAM can overperform various SAM variants.

Critical strengths
  1. using a shared signal to control both component is an interesting idea.

  2. the empirical evaluation results shows that the proposed method can consistently outperform several representative SAM variants.

Critical weaknesses
  1. the key idea of this method is to use a shared curvature signal to direct perturbation radius and direction, the core components of this method have been explored in prior work. Curvature (or hessian) aware perturbation has been discussed in paper like EigenSAM(nips2024), cr-sam(aaai2024), adaptive perturbation is also been discussed in methods such as TRAM(iclr2024) and lesam(icml2026). Thus, this paper can be seen as a combinitation of current existing ideas, although using a shared signal is interesting. The paper does not sufficiently explain what fundamentally new insight emerges from this combination.

  2. SHARD-SAM introduces multiple additional components like curvature probes, spectural estimations, power iterations, gating, making the method less elegant compared to original SAM and some variants(eg. FriendlySAM, ASAM,LESAM) which do not need many additional components. For SHARD-SAM each optimization step needs extra 20 forward steps which needs more training time, while the performance gain is relatively small.

  3. Limited evaluation of generalization and robustness. The empirical evaluation primarily focuses on standard image classification settings while many other SAM paper investigate the proposed method performance under more challenging settings such as domain generalization, noisy label and advesarial robustness.

Questions for authors

See Weakness.

2 comments
AuthorsOpenReview ↗

Shared curvature control, accuracy gains at equal training time, and robustness tests

Response to Reviewer q1DZ

Thank you for asking whether the accuracy gain justifies the extra computation.

What the shared curvature signal adds (W1)

SHARD couples radius and direction because the same rotation can improve inner ascent at one radius and impair it at another. In the local loss model, the gradient term scales linearly with the radius and the directional curvature term scales quadratically (B.1, Eq. B.11). We use one curvature signal to set the radius and decide how far to rotate.

Eigen-SAM steers toward the leading Hessian eigenvector, CR-SAM regularizes curvature, TRAM uses a trust region based on predictive distributions, and LE-SAM uses a first-order loss budget. In SHARD, the same set of probes supplies both the direction and the score. The gate compares the score magnitude with the gradient norm, while the radius depends on that magnitude relative to its running average (Secs. 4.1–4.3).

For a positive semidefinite quadratic Hessian with a leading eigengap, weighting improves population power-iteration contraction over the unweighted moment (Theorem 1, A.1). For the two-layer ReLU model, we compare the training losses after one SHARD update and one SAM update (Eq. G.16). To first order in the learning rate, the difference has two terms (Eq. G.18). The radius scales the gain from rotation. Changing the radius also changes the perturbation along the gradient, so shrinking it can offset the gain from rotation.

On CIFAR-100/WRN, adapting both controls gives 84.93% accuracy, versus 83.92% for radius alone and 84.18% for direction alone (Fig. 2(a)). Sharing ten probes gives 84.912%, versus 84.510% for two independent sets of five at equal query cost. Even when each control gets ten independent probes, accuracy is 84.750% (new F.1, Fig. S13).

Training time and accuracy (W2)

On CIFAR-100/WRN, the 20 extra forward queries per step increase training time by 40% at matched epochs (14 GPU-hours for SHARD, 10 for SAM). At the same 10-GPU-hour budget, SHARD reaches 84.54% accuracy, ahead of SASSHA (84.30%, the strongest baseline), Eigen-SAM (84.22%), and SAM (83.31%). SHARD also reaches 84% accuracy sooner, in 8.6 GPU-hours versus 9.3 for SASSHA and 9.1 for Eigen-SAM (E.5, Table S6).

Most of the overhead comes from evaluating the probes, which both controls and all three power iterations reuse. On one A6000, grouped probes take 325.6 ms per WRN step, compared with 13.2 ms for the spectral computation and 5.3 ms for gating. Grouping brings step time down from 1.716 to 1.232 s at a peak of 36.63 GiB. SAM takes 0.880 s and uses 13.03 GiB. On ImageNet, SHARD takes 24% longer than SAM on ResNet-50 and 32% longer on ViT-B/32 (new F.4, Figs. S25–S26, S29).

Generalization and robustness (W3)

SHARD improves on SAM under label noise, domain shift, and common corruptions. With 40% symmetric label noise, SHARD reaches 78.22% accuracy on clean test labels versus 75.42% for SAM. It fits fewer of the corrupted training labels (90.3% vs. 95.7%). At 20% symmetric and asymmetric noise, the clean-test gains over SAM are 1.81 and 1.73 points, respectively (new F.5, Fig. S31).

On PACS and OfficeHome, we fine-tuned an ImageNet-pretrained ResNet-50 on three source domains, selected configurations using validation data from those domains, and tested on the fourth. Across five seeds, SHARD beats SAM on all eight held-out domains and has the highest mean accuracy on seven (Fig. S32). On CIFAR-100-C at severity 5, SHARD gains 5.92 points over SAM, versus 2.43 for SASSHA and 1.96 for CR-SAM (Fig. S30).

At a perturbation budget of 8/255, all methods have near-zero robust accuracy under both attacks. SHARD reaches 0.222% under PGD-20 and 0.078% under AutoAttack, with no meaningful advantage in either test (Fig. S33).

Dear Reviewer,

We’re nearing the end of the discussion period. This is a friendly reminder to leave a comment for the authors.

Thank you

~ AC

Reviewer_U8aVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SHARD-SAM, a curvature-aware variant of SAM that uses a single shared curvature signal to jointly adapt both the perturbation radius and the perturbation direction. Validation covers CIFAR-10, CIFAR-100, and ImageNet across a wide range of CNNs and Vision Transformers. Component ablations separate the contributions of radius adaptation, direction adaptation, and their combination, and direction-source replacement controls swap in random, unweighted, and true-Hessian directions while holding the training recipe fixed, attributing the gains specifically to the proposed curvature signal. Theoretical analyses of the surrogate's spectral structure, estimation error, local directional-benefit conditions, convergence, and a PAC-Bayesian motivation are also conducted.

Critical strengths
  1. The direction-source replacement experiment, where swapping random unweighted true-HVP (Lanczos) directions into the same adaptive training loop together with the frozen-state, fixed-deflection-angle diagnostics and the probe-budget sweep is a level of experimental care well above the typical SAM-variant paper. The negative-branch measurement is the admirable corner-case check.

  2. A favorable and honestly characterized cost–accuracy trade-off. The curvature signal is built from forward-only queries plus O(bd) implicit matvecs (no backward passes or HVPs), reaching ~1.24–1.4× SAM runtime where Eigen-SAM/CR-SAM/SASSHA require substantially more machinery. This candor, plus a measured efficiency claim that survives a fixed-budget comparison, makes the practical relevance claim trustworthy.

Critical weaknesses
  1. The central "spectral" claim is contradicted by the paper's own controls, and the theory that could resolve this is unverifiable. The surrogate has rank at most ten in parameter spaces of tens of millions of dimensions, and its direction beats a random rotation by only about nineteen percent in worst-case loss increase. More decisively, in the direction-source replacement test, the true Hessian eigendirection gains less than one point while the surrogate gains about 1.8 points under the identical training loop. The proposed signal outperforms the spectral object it is named after, indicating the benefit comes from the probe weighting itself rather than spectral fidelity.

  2. All experiments use deliberately weak augmentation, the regime where sharpness-aware methods look best, and nothing shows the half-point margin over the strongest baseline survives stronger augmentation or other tasks. It is also unclear whether the Table 1 baselines received tuning comparable to SHARD-SAM's held-out selection of its curvature temperature, which matters because the margins are a few tenths of a point in a hyperparameter-sensitive family.

Questions for authors
  1. In the direction-source replacement test, the true Hessian eigendirection gains less than one point while the surrogate gains about 1.8 points under identical training and control rules. What property of the magnitude-weighted, sign-preserving surrogate explains this gap, if not spectral fidelity?

  2. At a fixed checkpoint and batch, how variable are the spectral score and the extracted direction across independent probe resamples? The controlled test shows only a modest loss-increment advantage over random at the default probe budget; at what budget does the direction stabilize, and does the radius envelope respond to resampling noise rather than genuine landscape change?

  3. For Table 1, on what protocol were the baseline hyperparameters chosen per setting? Please make the per-method selection budgets explicit.

  4. Does the advantage over the strongest baselines persist under strong augmentation (MixUp, CutMix, RandAugment) or on at least one non-classification task? If not, the consistency claim should be scoped to the weak-augmentation regime.

2 comments
AuthorsOpenReview ↗

SHARD versus HVP, probe stability, and results under stronger augmentation and matched tuning

Response to Reviewer U8aV

Thank you for the close reading of the HVP comparison and the tuning protocol.

Spectral interpretation and the HVP control (W1, Q1)

We agree that probe weighting helps explain SHARD's training advantage over HVP. SHARD preserves the sign of each curvature response and gives more weight to responses with larger absolute values. In adaptive training, SHARD gains most when that weighting supplies both the direction and the score. At a fixed radius and gate, HVP gives the larger loss increase.

For a quadratic loss, the weighted Gaussian population operator shares an eigenbasis with the Hessian (Theorem 1). For a positive semidefinite Hessian with a leading eigengap, weighting improves power-iteration contraction over the unweighted population moment (A.1). With ten probes, the surrogate has rank at most ten and its leading direction can differ from the Hessian's (A.5).

On ResNet-50, weighting raises the fixed-control loss increase from 0.00868 to 0.00970, about 19% above random (0.00816). HVP reaches 0.01014 (Fig. 7, Table S7) and leads SHARD in fixed-control WRN training (84.23% vs. 84.07%, Fig. S10(b)). With adaptive control, SHARD gains 1.78 points over SAM and HVP gains 0.83 (Fig. 2(b)). Both use the same rules to turn the score into a radius and gate (E.3). With the SHARD direction fixed, its own score gives higher mean accuracy than the calibrated HVP score (84.912% vs. 84.790%). The HVP direction favors the HVP score (84.660% vs. 84.580%, Fig. S11). In a fixed-output ReLU network with squared loss and unchanged activation signs, one HVP can check a sufficient condition for a small rotation toward the sampled direction to raise the inner loss at fixed radius (new G.2, Proposition 2, Eq. G.10).

Probe resampling and radius stability (Q2)

More probes make the score more reproducible. The raw directions from independent draws remain nearly orthogonal at every budget we tested (5–100 probes). The gated perturbation is already stable at ten probes, where training accuracy plateaus (Fig. 6). At a fixed ResNet-50 checkpoint and batch, increasing the budget from 10 to 100 lowers the score's coefficient of variation from 6.38% to 2.01%. The median absolute cosine between raw directions changes from 0.0012 to 0.0040. For the gated perturbation, mean cosine at ten probes ranges from 0.975 to 0.987 across three models (F.2, Figs. S16–S17).

Resampling alone changes the radius. At fixed ResNet-50 checkpoints, its standard deviation falls from 0.00827 to 0.00269 as the budget rises from 10 to 100. During training the landscape also changes, so these measurements cannot give the fraction of radius variation due to noise. With the same gate rule, current feedback gives 84.91% accuracy, versus 84.63% for a prescribed schedule and 84.54% for feedback delayed by 200 steps (Table S1).

Augmentation and other tasks (W2, Q4)

SHARD retains its lead under strong augmentation. We originally used random cropping and horizontal flipping to isolate optimizer effects (Sec. 5.1, Table S2). In the new CIFAR-100 comparison, all methods used RandAugment with MixUp/CutMix. The strongest competitors were LE-SAM on WRN-28-10 and SASSHA on pretrained ViT-B/16. SHARD led them by 0.31 and 0.30 points, respectively (new F.3, Figs. S21, S24).

In forecasting, SHARD reduces five-seed mean test MSE by 1.40%–2.96% relative to SASSHA, the strongest baseline in all 16 settings. We tested PatchTST and DLinear on ETTh1 and Weather at four horizons (F.6, data underlying Fig. S34).

Table 1 selection budgets (W2, Q3)

For the original Table 1, we used the optimizer-specific defaults recommended in each baseline's paper or official implementation. We ran no additional tuning trials for any baseline in any setting. Within each setting, all methods shared the training schedule, batch size, and augmentation (Sec. 5.1, Table S2). For SHARD-SAM, we selected the curvature temperature from 0.3, 0.5, and 0.7.

In the new experiment in F.3, all nine methods had the same tuning budget in each of the eight Table 1 settings. We varied learning rate and weight decay, base radius for SAM variants, and temperature for SHARD-SAM. We ranked configurations by final-checkpoint top-1 accuracy on held-out training data, reran the leaders, and chose the configuration with the highest mean accuracy across its initial run and reruns. On CIFAR-10 and CIFAR-100, we evaluated 18 configurations with seed 11 on a stratified 5,000-image holdout and reran the top three with seeds 22 and 33 (18 + 3 × 2 = 24 full runs per method per setting). On ImageNet, we held out 50 training images per class, evaluated eight configurations with seed 11, and reran the top two with seed 22 (8 + 2 × 1 = 10 full runs per method per setting). SHARD-SAM had the highest mean development accuracy in all eight settings (Fig. S22).

Dear Reviewer,

We’re nearing the end of the discussion period. This is a friendly reminder to leave a comment for the authors.

Thank you

~ AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.