acceptodds
Under review as a conference paper at ICLR 2027

Revisiting sharpness and Model Generalization through Local Rademacher Complexity

Abstract

Sharpness-aware minimization (SAM) and its variants can improve test accuracy by minimizing the maximum increase in empirical loss under local model-parameter perturbations. This improvement is commonly attributed to the view that higher loss sharpness is strongly associated with poorer generalization. However, several studies challenge this view and report weak or even opposite correlations. Regardless of the correlation, one more fundamental question is whether the sharpness alone can explain model generalization. Using local Rademacher complexity, we develop a new theorem on excess risk, which shows that, in comparison with reducing the sharpness alone, jointly reducing sharpness and complexity will lead to a compacter excess-risk bound. Focusing on Rademacher complexity within the theorem, we derive a bound which increases as the sample-wise classification margins decrease. Furthermore, we build a Rademacher-guided plug-in mechanism, termed Rad, that adjusts the direction of parameter perturbations according to sample-wise classification margins. Extensive experiments on multiple datasets and model architectures demonstrate that Rad combined with SAM or its variants improves the test accuracy and reduces the magnitude of the train–test accuracy gap. Grounded in local Rademacher complexity, our work offers a new interpretation of the relationship between loss sharpness and model generalization.

Primary area:
learning theory
Keywords:
Sharpness-aware minimization, excess risk bound, flatness, local Rademacher complexity

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_inCeOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits the relationship between sharpness and generalization through local Rademacher complexity. The authors derive an excess-risk bound involving both empirical sharpness and the local Rademacher complexity of a local loss-difference class, then derive a margin-dependent upper bound on that complexity. Based on this analysis, they propose Rad, a plug-in mechanism that gives larger influence to low-margin samples when constructing the SAM perturbation. Experiments across CIFAR, ImageNet, and fine-grained classification show consistent, although generally modest, improvements when Rad is combined with SAM or ASAM.

Critical strengths

This paper provides a principled perspective on why sharpness alone may not fully capture generalization behavior.

The excess-risk analysis connects empirical sharpness with local Rademacher complexity, and the subsequent margin-dependent bound gives a concrete bridge from an abstract complexity measure to sample-wise quantities.

Critical weaknesses

Theorem 4.1 gives a specific margin-dependent quantity in the upper bound on local Rademacher complexity, but Rad does not directly optimize this bound. Instead, it introduces a heuristic coefficient based on deviation from the mini-batch mean margin. Theorem 3.5 provides an excess-risk upper bound containing both sharpness and local Rademacher complexity. This shows that both quantities appear in the proposed bound, but does not by itself prove that sharpness is fundamentally insufficient as an explanatory quantity.

Since Rad mainly increases the influence of low-margin samples, it would be useful to compare against simpler loss-, confidence-, or margin-based reweighting strategies within SAM. Such comparisons would help establish whether the gains are specifically tied to the proposed Rademacher-complexity motivation.

Questions for authors

See Weaknesses

4 comments

Thank you for your careful evaluation of our work. We will try our best to address your questions. All page and line numbers below refer to the revised manuscript.

Q1: Theorem 4.1 gives a specific margin-dependent quantity in the upper bound on local Rademacher complexity, but Rad does not directly optimize this bound. Instead, it introduces a heuristic coefficient based on deviation from the mini-batch mean margin. Theorem 3.5 provides an excess-risk upper bound containing both sharpness and local Rademacher complexity. This shows that both quantities appear in the proposed bound, but does not by itself prove that sharpness is fundamentally insufficient as an explanatory quantity.

W1: Direct qiq_i weighting may allow a few samples to dominate the perturbation direction, which may limit the increase in the average loss over the entire batch. Thus, to get a bigger increase of loss, we adopt Rad as a theory-inspired implementation rather than directly optimize the bound (Lines 285-287 on Page 6). Rad exhibits a similar weight-distribution trend to qiq_i and limits weight differences to reduce the risk of a few samples dominating the shared perturbation. Theorem 3.5 is one of many possible ways to bound excess risk and does not imply that other bounds must depend on both quantities (Lines 243-244 on Page 5).

Specifically, Theorem 4.1 introduces qi(w)q_i(w) to characterize each sample’s contribution to the Rademacher complexity upper bound we derive: qi(w)=(K−1)e−mi(w)1+(K−1)e−mi(w).q_i(w)=\frac{(K-1)e^{-m_i(w)}}{1+(K-1)e^{-m_i(w)}}. where KK is the number of classes and mi(w)m_i(w) is the classification margin of sample ii. When mi(w)≫log⁡(K−1)m_i(w) \gg\log(K-1), qi(w)≈(K−1)e−mi(w).q_i(w)\approx(K-1)e^{-m_i(w)}.

If qi(w)q_i(w) is used directly as a sample weight, a few samples may receive excessively large relative weights and dominate the perturbation direction, which may limit the increase in the average loss over the entire batch. Therefore, motivated by qi(w)q_i(w), we adopt a more moderate implementation, Rad, which preserves the emphasis on small-margin samples while limiting weight differences through normalization and clipping.

In the revised manuscript, we have added a weight-distribution comparison using the same trained SAM model and dataset (Lines 1036-1059 on Page 20), as shown in Figure 4 of Appendix D. We compute both Rad and qi(w)q_i(w) weights and divide them by their respective batch means for comparison. The results show that: (1) both distributions concentrate in the lower relative-weight region; (2) direct qi(w)q_i(w) weighting assigns large relative weights to a small fraction of samples, whereas Rad limits this concentration of weights.

In addition, Theorem 3.5 imply that our excess-risk upper bound contains both empirical sharpness and local Rademacher complexity. We have revised the statement to read: “Our excess-risk upper bound contains both empirical sharpness and the Rademacher complexity of the local loss-difference function class. (Lines 243-244 on Page 5)” We have also adjusted the introduction (Lines 85--86 on Page 2) and contribution statements (Lines 95--97 on Page 2) to clarify that our discussion concerns the generalization guarantee provided by our bound, rather than excess-risk bounds in general.

Q2:Since Rad mainly increases the influence of low-margin samples, it would be useful to compare against simpler loss-, confidence-, or margin-based reweighting strategies within SAM. Such comparisons would help establish whether the gains are specifically tied to the proposed Rademacher-complexity motivation.

W2: We compare Rad with six baselines, including direct qiq_i weighting and confidence-based weighting (Lines 482-495 on Pages 9-10). The results show that general hard-example reweighting does not necessarily improve performance and suggest that Rad’s specific weight design contributes to the improvements.

Specifically, we compare seven methods: (1) SAM, without sample weighting; (2) Loss, using sample losses as sample weights; (3) Confidence, using 1−pw(yi∣xi)1-p_w(y_i\mid x_i) as sample weights; (4) qiq_i, using the quantity derived in Theorem 4.1; (5) Normalized qiq_i, applying the same centering, MAD normalization, and clipping as Rad to qiq_i; (6) Shuffled Rad, randomly permuting Rad weights within each batch; and (7) Rad, our proposed weighting scheme. All methods use the same CIFAR-10/ResNet-50 training setup: 200 epochs, a single-GPU batch size of 128, and seed 42. We report the best test accuracy during training. The results are shown below.

MethodSAMLossConfidenceqiq_iNormalized qiq_iShuffled RadRad (ours)
Best accuracy (%)96.6196.3696.3496.7396.7896.5496.89

Compared with SAM, direct qiq_i weighting improves accuracy, while Rad achieves the highest accuracy among all compared methods.

Dear Reviewer inCe,

Thank you again for your thoughtful and constructive feedback on our submission. We sincerely appreciate your time and effort throughout the review process.

As the November 18 discussion deadline approaches, we would like to kindly follow up on our previous responses and additional experiments. We hope they have adequately addressed your concerns.

If you have had a chance to review our responses, we would greatly appreciate knowing whether any concerns remain unresolved. We would be more than happy to provide further clarification or additional information before the discussion period ends.

Thank you once again for your time and consideration.

Best regards,

The Authors of Paper 3889

I thank the authors for their detailed response. I think the addtional analysis and experiments have solve my main concerns.

First, the auther claify the sope of the theoretical results, and point out that Rad is motivated by the theorem, not directly optimizing the boud.

Then, the addtional baseline is valuable and support the conclusion that Rad is effective, however, the advantage is somehow small compared with other reweighting methods, and it would be better to validate the robustness.

Overall, I think those modification made the paper more strong, though there is still some gap between the theorem and the algo., considering that these response has solve my concern, I am willing to raise my rating by 1.

Thank you for your positive assessment and for recognizing the improvements made in the revised manuscript. We appreciate your suggestion to further validate the robustness of Rad.

To address this concern, we have conducted three independent runs with seed 42 and report the mean and standard deviation of test accuracy and train-test accuracy gap. The results are summarized below.

MethodSAMLossConfidenceqiq_iNormalized qiq_iShuffled RadSAM+Rad
Best accuracy (%) ↑96.63 ± 0.0696.37 ± 0.0596.35 ± 0.0396.71 ± 0.0496.75 ± 0.0696.54 ± 0.0396.87 ± 0.06
Gap (pp) ↓2.52 ± 0.072.92 ± 0.042.99 ± 0.022.39 ± 0.052.32 ± 0.052.68 ± 0.072.17 ± 0.08

SAM+Rad achieves the highest average test accuracy and the lowest average accuracy gap among all weighting strategies. Direct qiq_i weighting also outperforms SAM, while Rad provides further improvements through its weight design. Compared with shuffled Rad, SAM+Rad achieves better performance, which highlights the importance of the correspondence between weights and samples.

Reviewer_1ZCcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits the relationship between loss sharpness and model generalization from the perspective of local Rademacher complexity. The authors derive an excess-risk bound involving both empirical sharpness and local Rademacher complexity, and further establish a margin-dependent complexity bound. Based on this analysis, they propose Rad, a plug-in mechanism that adjusts SAM perturbation directions using sample-wise classification margins. Experiments on several image classification benchmarks show improvements in test accuracy when Rad is combined with SAM and ASAM.

Critical strengths

The paper investigates the relationship between sharpness and generalization from the perspective of local Rademacher complexity, providing an additional theoretical interpretation.

Rad is simple to implement and can be integrated into existing SAM variants without additional forward or backward passes, according to the proposed algorithm.

The experiments cover multiple datasets and architectures, including CIFAR-10/100, ImageNet-1K, and fine-grained classification tasks. Rad generally improves test accuracy over the corresponding SAM and ASAM baselines.

Critical weaknesses
  1. The sample-aware perturbation approach in this paper overlaps with some SAM studies. δ\delta-SAM (Findings of EMNLP 2022) [a] has already discussed the differences between per-batch and per-instance weight perturbations, and approximates per-instance SAM through dynamic reweighting. CA-SAM (2025) [b] also studies the influence of different sample gradients on shared SAM perturbations. Rad modifies the shared perturbation direction through sample reweighting. The authors need to clearly distinguish the differences between these methods in perturbation construction, weighting strategy, and approximation quality, and explain Rad's unique advantages.

  2. There is a lack of perturbation-based baselines directly related to the core issue. This paper mainly compares SGD, SAM, ASAM, and XSAM, but lacks per-instance SAM, δ\delta-SAM, and other sample-aware SAM methods. Additionally, AT (Adversarial Training)+SAM [c] first generates adversarial examples in the input space and then performs SAM perturbation in the parameter space, which is also a related multi-layer perturbation optimization strategy. Although its optimization goal is not exactly the same as Rad, it can still be used as a supplementary comparison. The authors should report performance, perturbation quality, and computational cost under comparable training settings.

  3. Rad does not directly eliminate gradient cancellation. Ordinary SAM constructs shared perturbations using ∑igi\sum_i g_i, while Rad uses ∑i(1+λγi)gi\sum_i(1+\lambda\gamma_i)g_i. This adjustment only changes the relative contribution of sample gradients. When different samples have opposite gradients and similar weights, cancellation still exists. The authors should preferably verify to what extent Rad reduces gradient cancellation or compare the perturbations with per-instance SAM. It is recommended to supplement analysis of gradient alignment, perturbation direction, and sample-wise loss variation.

  4. Theorem 3.5 requires the learned parameter w^\hat w to be within the ρ\rho-neighborhood of the population-risk minimizer w⋆w^\star, but it is difficult to actually guarantee this condition during training. Additionally, sharpness and local Rademacher complexity in the theory are defined around w⋆w^\star, while Rad constructs perturbations at the current wtw_t. The author did not prove that Rad can reduce the corresponding complexity term. Therefore, this theorem is more like an excess-risk bound that includes sharpness and complexity, rather than a direct generalization guarantee of Rad.

  5. The weighting design in Eq. (9) lacks some basis and necessary ablation experiments. Theorem 4.1 constructs the complexity upper bound using margin-dependent qi(w)q_i(w), but Eq. (9) actually obtains γi\gamma_i through batch-centered margin deviation, mean absolute deviation normalization, and clipping. There is no direct optimization relationship between the two. The author should explain why this signed weighting is used and compare it with direct qiq_i weighting, loss-based weighting, other margin-based weighting, and ordinary sample reweighting to prove the superiority of the specific design of Eq. (9).

  6. There is no direct verification of local Rademacher complexity reduction. This paper mainly reports accuracy, train-test accuracy gap, Hessian eigenvalues, and margin changes. These results do not prove that Rad actually reduces the actual local Rademacher complexity, and it is difficult to distinguish whether the improvement comes from complexity reduction or ordinary hard-example reweighting. It would be better to directly evaluate the computable complexity surrogate and estimate the empirical local Rademacher complexity on small-scale tasks to verify the connection between the theoretical motivation and the algorithmic effect.

  7. The experimental improvement is relatively small, and some evaluation methods and experimental settings also need to be further improved. The Top-1 accuracy of SAM+Rad and ASAM+Rad on ImageNet-1K only increased by 0.13 and 0.23 percentage points respectively, and some train-test accuracy gaps increased. The author should also report final-epoch results and follow some baselines by adding label noise or distribution shift experiments to test the applicability range of emphasizing low-margin samples.

[a] Sharpness-Aware Minimization with Dynamic Reweighting

[b] Learning with noisy labels via clean aware sharpness aware minimization

[c] On the Duality Between Sharpness-Aware Minimization and Adversarial Training

Questions for authors

Please see the weakness

9 comments

(1/5)

Thank you for your careful evaluation of our work. We will try our best to address your questions. All page and line numbers below refer to the revised manuscript.

Q1: The sample-aware perturbation approach in this paper overlaps with some SAM studies. δ\delta-SAM (Findings of EMNLP 2022) [a] has already discussed the differences between per-batch and per-instance weight perturbations, and approximates per-instance SAM through dynamic reweighting. CA-SAM (2025) [b] also studies the influence of different sample gradients on shared SAM perturbations. Rad modifies the shared perturbation direction through sample reweighting. The authors need to clearly distinguish the differences between these methods in perturbation construction, weighting strategy, and approximation quality, and explain Rad's unique advantages.

W1: SAM, δ\delta-SAM[1], CA-SAM[2], RW-SAM[3], and Rad have the same perturbation construction framework but different weighting strategies. The advantages of Rad are the highest test accuracy among the compared methods and negligible computational overhead over SAM. Rad also achieves better perturbation and approximation quality than SAM, δ\delta-SAM, and RW-SAM in the evaluated setting (Lines 1177-1227 on Pages 22-23).

Specifically, these methods construct a batch-shared parameter perturbation from normalized, weighted sample gradients: ϵ=ρ∑iai∇wℓi(w)∥∑iai∇wℓi(w)∥,\epsilon=\rho\frac{\sum_i a_i\nabla_w\ell_i(w)} {\left\|\sum_i a_i\nabla_w\ell_i(w)\right\|}, where ρ\rho is the perturbation radius and aia_i is the sample weight. For sample weighting, SAM uses the same weights to all samples; δ\delta-SAM computes weights from per-sample losses at the original parameters and after positive and negative random parameter perturbations; CA-SAM assigns binary weights based on historical predictions, with weight 1 for likely clean samples and 0 for the others. RW-SAM computes weights from per-sample loss changes before and after a random parameter perturbation; and Rad computes weights from sample classification margins.

We have added comparisons under the same CIFAR-10/ResNet-50 training setup with three independent runs for each method. The results are shown below (Lines 420-425 on Page 8).

MethodSAMδ\delta-SAMCA-SAMRW-SAMSAM+Rad
Accuracy (%) ↑96.65±0.0696.42±0.0996.40±0.0696.55±0.0796.82±0.06
Gap (pp) ↓2.51±0.072.83±0.052.49±0.032.67±0.082.26±0.07

The experiments are conducted using available official code or implementations based on the published formulations. δ\delta-SAM mainly targets text tasks, while CA-SAM focuses on learning with noisy labels; their relative performance may change when transferred to clean image classification. SAM+Rad achieves the highest accuracy among the compared methods in this task. It is noted that a full per-instance SAM training run on CIFAR-10/ResNet-50 is estimated to take approximately 25 days on our current hardware, so we have not completed this comparison.

At each step, δ\delta-SAM requires 5 forward and 2 backward passes; RW-SAM requires 3 forward and 2 backward passes; CA-SAM requires 3 subset forward and 3 subset backward passes when both the clean and noisy subsets are nonempty. Rad, like SAM, requires only 2 forward and 2 backward passes. Thus, Rad offers a computationally efficient weighting strategy. We compare the computational cost under the same CIFAR-10/ResNet-50 training setup. Compared with the other weighting methods, Rad introduces negligible additional computational cost over SAM.

MethodForward passesBackward passesTime per epoch (s)Peak GPU memory (GiB)
SAM2292.783.48
δ-SAM52143.133.51
CA-SAM≤3≤3107.5518.32
RW-SAM32114.103.59
SAM+Rad2292.813.48

We compare perturbation quality among SAM, δ-SAM, RW-SAM, and SAM+Rad; CA-SAM is excluded because it requires historical perturbations. Using the same frozen SAM checkpoint, we evaluate all perturbations on the CIFAR-10 training samples with a perturbation radius of 0.05. We compute each sample’s loss increase after perturbation and report its mean, together with the mean absolute difference from the loss increase under the corresponding per-instance perturbation. Rad achieves the largest mean loss increase and the smallest mean absolute gap in loss increase to the per-instance perturbation reference.

MetricSAMδ\delta-SAMRW-SAMSAM+Rad
Mean loss increase ↑0.03260.04210.03560.0479
Mean absolute gap to per-instance reference ↓0.55510.54600.55210.5397

[1] Sharpness-Aware Minimization with Dynamic Reweighting

[2] Learning with noisy labels via clean aware sharpness aware minimization

[3] Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise

(2/5)

Q2: There is a lack of perturbation-based baselines directly related to the core issue. This paper mainly compares SGD, SAM, ASAM, and XSAM, but lacks per-instance SAM, δ\delta-SAM, and other sample-aware SAM methods. Additionally, AT (Adversarial Training)+SAM [c] first generates adversarial examples in the input space and then performs SAM perturbation in the parameter space, which is also a related multi-layer perturbation optimization strategy. Although its optimization goal is not exactly the same as Rad, it can still be used as a supplementary comparison. The authors should report performance, perturbation quality, and computational cost under comparable training settings.

W2: The added perturbation-based baselines are δ\delta-SAM, CA-SAM, and RW-SAM. In our response to Q1, we report the performance and computational cost of SAM, δ\delta-SAM, CA-SAM, RW-SAM, and Rad, as well as the perturbation and approximation quality of SAM, δ\delta-SAM, RW-SAM, and Rad. For adversarial training, we compare AT+SAM, AT+SAM+Rad, AWP, and AWP+Rad. Rad provides improvements in robust accuracy over both baselines, with AWP+Rad achieving the highest robust accuracy among the compared methods (Lines 1240-1269 on Pages 23-24).

Specifically, we train for 200 epochs with batch size 128 and seed 42. The learning rate starts at 0.1 and drops by a factor of 10 at epochs 80, 120, and 160. Training uses PGD-10 with ∥δ∥∞≤8/255\Vert\delta\Vert_\infty\leq8/255 and step size 2/2552/255. We report clean, PGD-20, and AutoAttack accuracy for the best PGD-10 and final-epoch checkpoints.

MethodBest clean (%)Best PGD-20 (%)Best AA (%)Final clean (%)Final PGD-20 (%)Final AA (%)
AT+SAM84.3552.2248.7685.2550.3747.13
AT+SAM+Rad82.8352.6048.9584.8650.8847.72
AWP81.0853.8448.9779.9353.2848.96
AWP+Rad80.9853.9549.2680.0953.5749.06

Compared with the baselines, Rad-enhanced methods achieve higher PGD-20 and AutoAttack accuracy at the best and final checkpoints. These results suggest that Rad’s benefits extend beyond standard classification to adversarial training.

[1] Adversarial Weight Perturbation Helps Robust Generalization

Q3: Rad does not directly eliminate gradient cancellation. Ordinary SAM constructs shared perturbations using ∑igi\sum_i g_i, while Rad uses ∑i(1+λγi)gi\sum_i (1+\lambda\gamma_i)g_i. This adjustment only changes the relative contribution of sample gradients. When different samples have opposite gradients and similar weights, cancellation still exists. The authors should preferably verify to what extent Rad reduces gradient cancellation or compare the perturbations with per-instance SAM. It is recommended to supplement analysis of gradient alignment, perturbation direction, and sample-wise loss variation.

W3: In our methods, we believe that perturbation quality should be assessed by the loss increase rather than changes in perturbation direction. We compare the mean sample-wise loss increase and its mean absolute gap to the per-instance SAM reference (in our response to Q1). Rad achieves a larger mean loss increase and a smaller reference gap. In addition, we consider that Rad can indirectly mitigate gradient cancellation. We evaluate gradient cancellation and find that Rad increases the mean gradient aggregation ratio, which indicates less relative gradient cancellation under the evaluated settings (Lines 1294-1315 on Pages 24-25).

First, Local Rademacher complexity considers different combinations of sample-wise loss changes under random signs and can capture variations masked by cancellation in the average loss. However, this complexity is difficult to compute and optimize directly. We derive a margin-dependent upper bound to guide the design of sample weights. We compare the gradient aggregation ratios with uniform and Rad weights to assess whether Rad mitigates gradient cancellation.

Specifically, we define the gradient aggregation ratio as :

Cb=∥∑i∈Bbaigi∥2∑i∈Bbai∥gi∥2C_b=\frac{\Vert\sum_{i\in B_b}a_i g_i\Vert_2}{\sum_{i\in B_b}a_i\Vert g_i\Vert_2},

where gig_i is the sample gradient and aia_i is the sample weight. A higher ratio indicates less relative gradient cancellation. We evaluate uniform and Rad weights at the same frozen checkpoint on CIFAR-10. We use the trained SAM and ASAM models and report the mean ratio across all batches. The results show that Rad increases the mean gradient aggregation ratio across these settings, which suggests that Rad can mitigate gradient cancellation.

SAMASAM
BackboneVGG16ResNet-18ResNet-50VGG16ResNet-18ResNet-50
Uniform weights0.50940.22460.23040.51700.17520.1863
Rad weights ↑0.54770.26170.26410.55960.18620.2216

(3/5)

Q4: Theorem 3.5 requires the learned parameter w^\widehat w to be within the ρ\rho-neighborhood of the population-risk minimizer, but it is difficult to actually guarantee this condition during training. Additionally, sharpness and local Rademacher complexity in the theory are defined around w⋆w^\star, while Rad constructs perturbations at the current wtw_t. The author did not prove that Rad can reduce the corresponding complexity term. Therefore, this theorem is more like an excess-risk bound that includes sharpness and complexity, rather than a direct generalization guarantee of Rad.

W4: We acknowledge a gap between the theoretical assumptions and practical training, which remains difficult to fully resolve. So far, all of references (such as [e.g., 1,2]) we found deal with this issue by using this approach. The assumption w^∈Wρ(w∗)\widehat w\in\mathcal W_\rho(w^*) ensures that the learned model belongs to the local parameter set considered in our analysis, so that the sharpness, complexity, and associated assumptions apply. In addition, we numerically estimate empirical local Rademacher complexity and find that Rad-enhanced methods yield lower estimates than their corresponding baselines.

Specifically, the assumption w^∈Wρ(w∗)\widehat w\in\mathcal W_\rho(w^*) specifies the scope of the theorem and does not guarantee that SAM satisfies the condition (Lines 245-249 on Page 5). In practice, the population-risk minimizer is unknown, so we use the current parameters as a practical substitute. Rad is motivated by our theoretical results rather than directly guaranteed by Theorem 3.5 (Lines 311-313 on Page 6). We have clarified this distinction in the revised manuscript.

We approximate the local Rademacher complexity from its definition (Lines 1317-1341 on Page 25):

RS(Gρ(w))=Eσ[sup⁡ϵ:∑jϵj2≤ρ21n∑i=1nσi(ℓ(w+ϵ;zi)−ℓ(w;zi))]R_S(G_\rho(w))=E_\sigma[\sup_{\epsilon:\sum_j\epsilon_j^2\leq\rho^2}\frac{1}{n}\sum_{i=1}^{n}\sigma_i(\ell(w+\epsilon;z_i)-\ell(w;z_i))]

We use trained models and the same 128 samples from the CIFAR-10 training set with seed 42. We use 64 sets of independent Rademacher signs. For each set, we approximate the supremum through projected gradient ascent with 100 steps, two step sizes, and two initializations. We evaluate neighborhood radii of 0.025, 0.05, and 0.1 and report the mean estimate over the 64 sets. These values are numerical estimates rather than certified complexity bounds. The results show that, the Rad-enhanced methods yield lower complexity estimates than their corresponding baselines, which provides empirical support for our theoretical motivation.

RadiusRadSAMASAM
VGG16 ↓ResNet-18 ↓ResNet-50 ↓VGG16 ↓ResNet-18 ↓ResNet-50 ↓
0.025w/o0.025400.019550.038580.043290.025410.02581
w/0.022740.018440.034300.040620.024490.02216
0.05w/o0.112240.081210.167130.154880.143560.13893
w/0.097310.064950.135310.140350.135810.11515
0.1w/o0.520220.813820.993640.660841.328061.46272
w/0.518620.775510.879980.621731.306111.01058

[1] SAM:Sharpness-Aware Minimization for Efficiently Improving Generalization

[2] ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks

Reviewer_UXjzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits the relationship between loss sharpness and model generalization from the perspective of local Rademacher complexity. The authors argue that sharpness alone is insufficient to characterize generalization, as it measures changes in average empirical loss while potentially overlooking individual sample-level variations. To address this limitation, the paper derives an excess-risk bound involving both empirical sharpness and local Rademacher complexity, followed by a margin-dependent upper bound on the latter. Motivated by these theoretical results, the authors propose Rad, a plug-and-play mechanism that adjusts the perturbation direction of SAM-based optimizers by emphasizing low-margin samples.

Critical strengths

(1) Meaningful theoretical motivation and an interesting perspective on generalization. The introduction of local Rademacher complexity provides a potentially valuable perspective for understanding generalization beyond conventional sharpness measures.

(2) Consistent empirical improvements with a lightweight optimization mechanism. Rad improves test accuracy over its corresponding SAM and ASAM baselines across all reported datasets and architectures, including CNNs trained from scratch and pretrained ViTs.

Critical weaknesses
  1. Theorem 3.5 does not explicitly require the learned parameter to satisfy an empirical risk minimization condition, which raises concerns about the validity of the excess-risk bound. For example, consider the bounded loss (\ell(w;z)=w^2/(1+w^2)), with (w^*=0), (\rho=1), and (\hat w=1). The excess risk remains (1/2), while the proposed upper bound approaches zero as the sample size increases.

2.Although Theorem 4.1 derives a margin-dependent Rademacher complexity bound, Rad uses a heuristic margin-based weighting scheme without demonstrating that it reduces the complexity or its upper bound. Moreover, the proposed mechanism resembles conventional sample reweighting during SAM perturbation construction.

Questions for authors

(1) Regarding the validity of Theorem 3.5: Does the theorem implicitly assume that the learned parameter is an empirical risk minimizer or satisfies an approximate optimization condition? If so, could the authors explicitly state this assumption and explain how it is satisfied by SAM-based optimization?

(2) Theorem 4.1 derives a complexity upper bound involving (q_i(w)), whereas Rad uses a clipped, batch-normalized margin deviation.

(3) Have the authors compared Rad against simpler perturbation-weighting strategies, particularly direct (q_i)-based weighting and confidence-based weighting? These comparisons would be important for determining whether the improvements can be attributed to the proposed mechanism rather than general hard-example reweighting.

5 comments

Thank you for your careful evaluation of our work. We will try our best to address your questions. All page and line numbers below refer to the revised manuscript.

Q1: Regarding the validity of Theorem 3.5: Does the theorem implicitly assume that the learned parameter is an empirical risk minimizer or satisfies an approximate optimization condition? If so, could the authors explicitly state this assumption and explain how it is satisfied by SAM-based optimization?

W1: Theorem 2.1 and Theorem 3.5 require the learned parameter to satisfy an approximate ERM condition. We have added an explicit empirical optimization error term in the revised manuscript. Since SAM optimizes the worst-case empirical loss within a neighborhood rather than the empirical loss itself, we cannot guarantee that SAM attains ERM. We have retained this error in the bound without assuming it is zero. We also consider that reducing this optimization error in SAM-based optimization is a meaningful direction for future research.

Specifically, Theorem 2.1 in our manuscript, cited from Xu et al. (2016), is based on an empirical risk minimizer (ERM) setting. Since SAM optimizes a neighborhood-wise worst-case empirical loss rather than the empirical loss itself, it is not guaranteed to minimize the empirical risk. We transform the exact ERM requirement in Theorem 2.1 into an explicit optimization error term in the bound (Lines 170-183 on Page 4). The optimization error is defined as: ϵopt=LS(w^)−inf⁡w∈W′LS(w).\epsilon_{\mathrm{opt}}=L_S(\widehat w)-\inf_{w\in\mathcal W'}L_S(w).

In Theorem 3.5, we set W′=Wρ(w∗)\mathcal W'=\mathcal W_\rho(w^*) and denote the corresponding optimization error by Δopt\Delta_{\mathrm{opt}} (Lines 229-239 on Page 5). We have updated the derivation in Appendix B accordingly (Lines 832-881 on Pages 16-17). In your counterexample, Δopt=1/2\Delta_{\mathrm{opt}}=1/2, so the revised upper bound no longer approaches zero as the sample size increases, consistent with the excess risk.

Theorem 4.1 derives a complexity upper bound involving qi(w)q_i(w), whereas Rad uses a clipped, batch-normalized margin deviation.

W2: Direct qiq_i weighting may allow a few samples to dominate the perturbation direction, which may limit the increase in the average loss over the entire batch. Thus, to get a bigger increase of loss, we adopt Rad as a theory-inspired implementation rather than directly optimize the bound (Lines 285-287 on Page 6). We have explicitly clarified in the revised manuscript that Rad is a heuristic mechanism motivated by the theoretical upper bound. Rad exhibits a similar weight-distribution trend to qiq_i and limits weight differences to reduce the risk of a few samples dominating the shared perturbation.

Specifically, Theorem 4.1 introduces qi(w)q_i(w) to characterize each sample’s contribution to the Rademacher complexity upper bound we derive: qi(w)=(K−1)e−mi(w)1+(K−1)e−mi(w).q_i(w)=\frac{(K-1)e^{-m_i(w)}}{1+(K-1)e^{-m_i(w)}}. where KK is the number of classes and mi(w)m_i(w) is the classification margin of sample ii. When mi(w)≫log⁡(K−1)m_i(w) \gg\log(K-1), qi(w)≈(K−1)e−mi(w).q_i(w)\approx(K-1)e^{-m_i(w)}. If qi(w)q_i(w) is used directly as a sample weight, a few samples may receive excessively large relative weights and dominate the perturbation direction, which may limit the increase in the average loss over the entire batch. Therefore, motivated by qi(w)q_i(w), we adopt a more moderate implementation, Rad, which preserves the emphasis on small-margin samples while limiting weight differences through normalization and clipping.

In the revised manuscript, we have added a weight-distribution comparison using the same trained SAM model and dataset (Lines 1036-1059 on Page 20), as shown in Figure 4 of Appendix D. We compute both Rad and qi(w)q_i(w) weights and divide them by their respective batch means for comparison. The results show that: (1) both distributions concentrate in the lower relative-weight region; (2) direct qi(w)q_i(w) weighting assigns large relative weights to a small fraction of samples, whereas Rad limits this concentration of weights.

Have the authors compared Rad against simpler perturbation-weighting strategies, particularly direct qiq_i-based weighting and confidence-based weighting? These comparisons would be important for determining whether the improvements can be attributed to the proposed mechanism rather than general hard-example reweighting.

W3: We compare Rad with six baselines, including direct qiq_i weighting and confidence-based weighting (Lines 482-495 on Pages 9-10). The results show that general hard-example reweighting does not necessarily improve performance and suggest that Rad’s specific weight design contributes to the improvements.

Specifically, we compare seven methods: (1) SAM, without sample weighting; (2) Loss, using sample losses as sample weights; (3) Confidence, using 1−pw(yi∣xi)1-p_w(y_i\mid x_i) as sample weights; (4) qiq_i, using the quantity derived in Theorem 4.1; (5) Normalized qiq_i, applying the same centering, MAD normalization, and clipping as Rad to qiq_i; (6) Shuffled Rad, randomly permuting Rad weights within each batch; and (7) Rad, our proposed weighting scheme. All methods use the same CIFAR-10/ResNet-50 training setup: 200 epochs, a single-GPU batch size of 128, and three independent runs with seed 42. We report the best test accuracy during training. The results are shown below.

MethodSAMLossConfidenceqiq_iNormalized qiq_iShuffled RadSAM+Rad
Best accuracy (%) ↑96.63 ± 0.0696.37 ± 0.0596.35 ± 0.0396.71 ± 0.0496.75 ± 0.0696.54 ± 0.0396.87 ± 0.06
Gap (pp) ↓2.52 ± 0.072.92 ± 0.042.99 ± 0.022.39 ± 0.052.32 ± 0.052.68 ± 0.072.17 ± 0.08

SAM+Rad achieves the highest average test accuracy and the lowest average accuracy gap among all weighting strategies. Direct qiq_i weighting also outperforms SAM, while Rad provides further improvements through its weight design. Compared with shuffled Rad, SAM+Rad achieves better performance, which highlights the importance of the correspondence between weights and samples.

Dear Reviewer UXjz,

Thank you again for your thoughtful and constructive feedback on our submission. We sincerely appreciate your time and effort throughout the review process.

As the November 18 discussion deadline approaches, we would like to kindly follow up on our previous responses and additional experiments. We hope they have adequately addressed your concerns.

If you have had a chance to review our responses, we would greatly appreciate knowing whether any concerns remain unresolved. We would be more than happy to provide further clarification or additional information before the discussion period ends.

Thank you once again for your time and consideration.

Best regards,

The Authors of Paper 3889

Reviewer_1nPBOpenReview ↗
Rating
3
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper extends the analysis of the local sharpness of loss functions and its correlation with increasing or decreasing excess risk. The authors establish a new upper bound of population risk that (i.e., the bound) depends on both local sharpness as well as the local Rademacher complexity. This leads the authors to target minimising both the sharpness and the Rademacher complexity in order to minimise the excess risk.

The work shows that in the case of K-class classification, the Rademacher complexity can be bounded above by a term that depends on the classification margin. Hence, minimising that margin would lead to minimising the Rademacher complexity. This is the basis for their proposed plug-and-play optimisation procedure termed Rad.

The authors demonstrate in their evaluations over multiple vision benchmarks and architectures that plugging Rad into baselines optimisation procedures improves the accuracy and generalisation (measured via the train-test accuracy gap).

Critical strengths

S1. The exposition is quite precise and clear. It is easy to follow the arguments made in the paper and to understand how a conclusion follows from a presented result. The contribution, significance, and positioning of this work are all clear and easy to see.

Critical weaknesses

None.

Questions for authors

Q1. Peng et al. (2025) seems to be dismissed quickly without a clear engagement (cited in line 117). Can you please highlight how your bound improves over theirs?

Q2. In lines 240-241, you make the following claim

Our new bound shows that the excess risk depends not only on empirical sharpness but also on the Rademacher complexity of the local loss-difference function class.

Equation (7) gives simply an upper bound. It does not really "show" that excess risk depends on both sharpness and Rademacher complexity. Someone may come up with another tighter bound that only depends on one of those quantities or none of them at all. Hence, the dependence is only in your upper-bound, not in excess risk itself. Please tighten your claim or justify it.

3 comments

Thank you for your careful evaluation and positive assessment of our work. We will try our best to address your questions. All page and line numbers below refer to the revised manuscript.

Q1: Peng et al. (2025) seems to be dismissed quickly without a clear engagement (cited in line 117). Can you please highlight how your bound improves over theirs?

W1: Compared with Peng et al. [1], our bound does not require cross-model parameter-distribution statistics. Specifically, Peng et al. require parameter-distribution statistics, including the parameter mean estimated from six independently trained models in their experiments, and Hessian-related quantities (Lines 116--118 on Page 3). Our bound involves empirical sharpness and local Rademacher complexity. The former can be approximated using Hessian-related quantities, while we derive a margin-dependent upper bound for the latter. Thus, our bound avoids the need to estimate parameter-distribution statistics across multiple trained models.

In addition, we derive an excess-risk bound from a local Rademacher complexity perspective. Peng et al. adopt an information-theoretic perspective, exploiting the relationship between parameter distributions and curvature to tighten generalization bounds. The two approaches provide complementary perspectives.

[1] Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGD. ICLR 2025.

Q2: Equation (7) gives simply an upper bound. It does not really "show" that excess risk depends on both sharpness and Rademacher complexity. Someone may come up with another tighter bound that only depends on one of those quantities or none of them at all. Hence, the dependence is only in your upper-bound, not in excess risk itself. Please tighten your claim or justify it.

W2: Our method is one of many possible ways to bound excess risk. It does not mean that other bounds must depend on both quantities, i.e., the empirical sharpness and local Rademacher complexity. We have revised the statement to read: “Our excess-risk upper bound contains both empirical sharpness and the Rademacher complexity of the local loss-difference function class. (Lines 243-244 on Page 5)” We have also adjusted the introduction (Lines 85--86 on Page 2) and contribution statements (Lines 95--97 on Page 2) to clarify that our discussion concerns the generalization guarantee provided by our bound, rather than excess-risk bounds in general.

I thank the authors for their answers.

This addresses my questions, and I will maintain my positive score. Please upload the updated version and highlight the made changes in a different colour (not done yet) to make it easier for the reviewers to see what has been updated since the first version.

Good luck.

Thank you for your encouragement and support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.