- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SHARD-SAM, which adaptively adjusts the perturbation radius and direction through shared curvature signals. The core of the method is to use zeroth-order Gaussian probes to construct a signed Hessian-amplified spectral surrogate, extract curvature responses from forward-only loss queries, and then obtain the principal direction and curvature score through power iteration. The radius is dynamically adjusted by a symmetric asymptotic envelope based on the ratio of the score magnitude to its EMA baseline, while the effective direction combines the gradient with a spectral correction term through magnitude-adaptive gating.
Critical strengths
1.This paper uses an interesting method to simultaneously adjust the radius and direction in SAM, solving a known problem. 2. Numerous experiments show that the proposed optimizer has improved performance.
Critical weaknesses
1.The problem described in the paper is well-known, but the solution is not intuitive. Specifically, curvature awareness comes from CR-SAM, constructing spectral proxies similar to TSAM, and adaptive radii similar to momentum construction in MSAM, but it is not yet known why these modules must be combined for an intuitive reason. 2.Theoretical analysis is only based on specific Hessian assumptions and training objectives and is not related to specific neural networks, which weakens persuasiveness.
Questions for authors
1.Can the proposed optimizer theoretically derive its advantages in simple neural networks, such as two-layer ReLU and other widely used simple networks? 2.Can the proposed optimizer demonstrate advantages in other fields, such as language modeling, time series prediction, etc.?
2 comments
Dear Reviewer,
We’re nearing the end of the discussion period. This is a friendly reminder to leave a comment for the authors.
Thank you
~ AC
Why SHARD couples radius and direction, with ReLU theory and forecasting results
Response to Reviewer zH99
Thank you for suggesting the two-layer ReLU example.
Rationale for coordinating radius and direction (W1)
A rotation toward higher curvature can raise the inner loss at one radius and lower it at another. This is why SHARD coordinates radius and direction. In the local loss model, rotating away from the gradient gives up some alignment. That loss in the gradient term scales linearly with the radius, while the gain in the curvature term scales quadratically (B.1, Eq. B.11).
CR-SAM regularizes curvature, Tilted SAM reweights neighborhood losses, and Momentum-SAM uses momentum to construct the perturbation direction. SHARD takes both a direction and a score from a single signed spectral operator. The gate compares the score magnitude with the gradient norm to set the strength of the rotation. The radius controller compares the same magnitude with its running average. If the score stays high, the direction can remain tilted even as the radius returns near its base value (Secs. 4.1–4.3).
The two ablations on CIFAR-100/WRN test different parts of the design. Adapting radius and direction together gives 84.93% accuracy, compared with 83.92% for radius alone and 84.18% for direction alone (Fig. 2(a), Table S1). To test sharing, we kept both controls active. Sharing ten probes gives 84.912%, while splitting them into two independent sets of five gives 84.510%. Even with ten independent probes per control, accuracy is 84.750% (new F.1, Fig. S13).
Network-specific curvature assumptions (W2)
In a two-layer ReLU network with squared loss and fixed output weights, the loss is exactly quadratic over any region where the activation signs stay fixed. The Hessian in that region is H = ΦᵀΦ/n, where the features in Φ depend on the inputs, output weights, and active units (new G.1, Proposition 1). This network thus satisfies the quadratic assumption in A.1. When both layers are trainable, the loss also contains cubic and quartic terms. Propositions 4–5 keep those terms and give a sufficient condition for the rotation to increase the inner loss (G.4).
Local advantage over equal-radius SAM (Q1)
At the same radius, SHARD can attain a higher inner loss than SAM and a lower training loss after one outer update. For the fixed-output ReLU model, let u be the unit gradient, v the sampled direction, and c = uᵀv. If Γ(v) = (v − cu)ᵀHu > 0, a small rotation toward v improves the curvature term to first order in the gate and costs gradient alignment only to second order. One Hessian-vector product suffices to check this condition. Proposition 2 gives the gate interval for which the inner loss increases (G.2, Eq. G.10).
For sufficiently small positive values of both the gate and the learning rate, the same condition gives a lower training loss after one update if the perturbed points and update endpoints stay in the same activation region (Proposition 3, Eqs. G.16–G.17). If SHARD also shrinks the radius, the gain from rotation must offset the effect of that reduction for SHARD to have lower loss to first order in the learning rate (Eq. G.18). Appendix G.5 derives corresponding conditions for linear regression, two-layer linear networks, and fixed-feature logistic regression.
Beyond image classification (Q2)
In time-series forecasting, SHARD has the lowest five-seed mean test MSE in all 16 settings. We ran PatchTST and DLinear on ETTh1 and Weather at four forecast horizons, comparing SHARD with AdamW, SAM, and SASSHA. SHARD reduces MSE by 1.40%–2.96% relative to SASSHA, the strongest baseline in every setting, and by 3.33%–6.63% relative to AdamW (new F.6, data underlying Fig. S34). We split the data chronologically, fitted normalization on training data, and evaluated the final checkpoint at 50 epochs. Figs. S35–S36 report test MAE.