- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper revisits the relationship between sharpness and generalization through local Rademacher complexity. The authors derive an excess-risk bound involving both empirical sharpness and the local Rademacher complexity of a local loss-difference class, then derive a margin-dependent upper bound on that complexity. Based on this analysis, they propose Rad, a plug-in mechanism that gives larger influence to low-margin samples when constructing the SAM perturbation. Experiments across CIFAR, ImageNet, and fine-grained classification show consistent, although generally modest, improvements when Rad is combined with SAM or ASAM.
Critical strengths
This paper provides a principled perspective on why sharpness alone may not fully capture generalization behavior.
The excess-risk analysis connects empirical sharpness with local Rademacher complexity, and the subsequent margin-dependent bound gives a concrete bridge from an abstract complexity measure to sample-wise quantities.
Critical weaknesses
Theorem 4.1 gives a specific margin-dependent quantity in the upper bound on local Rademacher complexity, but Rad does not directly optimize this bound. Instead, it introduces a heuristic coefficient based on deviation from the mini-batch mean margin. Theorem 3.5 provides an excess-risk upper bound containing both sharpness and local Rademacher complexity. This shows that both quantities appear in the proposed bound, but does not by itself prove that sharpness is fundamentally insufficient as an explanatory quantity.
Since Rad mainly increases the influence of low-margin samples, it would be useful to compare against simpler loss-, confidence-, or margin-based reweighting strategies within SAM. Such comparisons would help establish whether the gains are specifically tied to the proposed Rademacher-complexity motivation.
Questions for authors
See Weaknesses
4 comments
Dear Reviewer inCe,
Thank you again for your thoughtful and constructive feedback on our submission. We sincerely appreciate your time and effort throughout the review process.
As the November 18 discussion deadline approaches, we would like to kindly follow up on our previous responses and additional experiments. We hope they have adequately addressed your concerns.
If you have had a chance to review our responses, we would greatly appreciate knowing whether any concerns remain unresolved. We would be more than happy to provide further clarification or additional information before the discussion period ends.
Thank you once again for your time and consideration.
Best regards,
The Authors of Paper 3889
I thank the authors for their detailed response. I think the addtional analysis and experiments have solve my main concerns.
First, the auther claify the sope of the theoretical results, and point out that Rad is motivated by the theorem, not directly optimizing the boud.
Then, the addtional baseline is valuable and support the conclusion that Rad is effective, however, the advantage is somehow small compared with other reweighting methods, and it would be better to validate the robustness.
Overall, I think those modification made the paper more strong, though there is still some gap between the theorem and the algo., considering that these response has solve my concern, I am willing to raise my rating by 1.
Thank you for your positive assessment and for recognizing the improvements made in the revised manuscript. We appreciate your suggestion to further validate the robustness of Rad.
To address this concern, we have conducted three independent runs with seed 42 and report the mean and standard deviation of test accuracy and train-test accuracy gap. The results are summarized below.
| Method | SAM | Loss | Confidence | Normalized | Shuffled Rad | SAM+Rad | |
|---|---|---|---|---|---|---|---|
| Best accuracy (%) ↑ | 96.63 ± 0.06 | 96.37 ± 0.05 | 96.35 ± 0.03 | 96.71 ± 0.04 | 96.75 ± 0.06 | 96.54 ± 0.03 | 96.87 ± 0.06 |
| Gap (pp) ↓ | 2.52 ± 0.07 | 2.92 ± 0.04 | 2.99 ± 0.02 | 2.39 ± 0.05 | 2.32 ± 0.05 | 2.68 ± 0.07 | 2.17 ± 0.08 |
SAM+Rad achieves the highest average test accuracy and the lowest average accuracy gap among all weighting strategies. Direct weighting also outperforms SAM, while Rad provides further improvements through its weight design. Compared with shuffled Rad, SAM+Rad achieves better performance, which highlights the importance of the correspondence between weights and samples.
Thank you for your careful evaluation of our work. We will try our best to address your questions. All page and line numbers below refer to the revised manuscript.
W1: Direct qi weighting may allow a few samples to dominate the perturbation direction, which may limit the increase in the average loss over the entire batch. Thus, to get a bigger increase of loss, we adopt Rad as a theory-inspired implementation rather than directly optimize the bound (Lines 285-287 on Page 6). Rad exhibits a similar weight-distribution trend to qi and limits weight differences to reduce the risk of a few samples dominating the shared perturbation. Theorem 3.5 is one of many possible ways to bound excess risk and does not imply that other bounds must depend on both quantities (Lines 243-244 on Page 5).
Specifically, Theorem 4.1 introduces qi(w) to characterize each sample’s contribution to the Rademacher complexity upper bound we derive: qi(w)=1+(K−1)e−mi(w)(K−1)e−mi(w). where K is the number of classes and mi(w) is the classification margin of sample i. When mi(w)≫log(K−1), qi(w)≈(K−1)e−mi(w).
If qi(w) is used directly as a sample weight, a few samples may receive excessively large relative weights and dominate the perturbation direction, which may limit the increase in the average loss over the entire batch. Therefore, motivated by qi(w), we adopt a more moderate implementation, Rad, which preserves the emphasis on small-margin samples while limiting weight differences through normalization and clipping.
In the revised manuscript, we have added a weight-distribution comparison using the same trained SAM model and dataset (Lines 1036-1059 on Page 20), as shown in Figure 4 of Appendix D. We compute both Rad and qi(w) weights and divide them by their respective batch means for comparison. The results show that: (1) both distributions concentrate in the lower relative-weight region; (2) direct qi(w) weighting assigns large relative weights to a small fraction of samples, whereas Rad limits this concentration of weights.
In addition, Theorem 3.5 imply that our excess-risk upper bound contains both empirical sharpness and local Rademacher complexity. We have revised the statement to read: “Our excess-risk upper bound contains both empirical sharpness and the Rademacher complexity of the local loss-difference function class. (Lines 243-244 on Page 5)” We have also adjusted the introduction (Lines 85--86 on Page 2) and contribution statements (Lines 95--97 on Page 2) to clarify that our discussion concerns the generalization guarantee provided by our bound, rather than excess-risk bounds in general.
W2: We compare Rad with six baselines, including direct qi weighting and confidence-based weighting (Lines 482-495 on Pages 9-10). The results show that general hard-example reweighting does not necessarily improve performance and suggest that Rad’s specific weight design contributes to the improvements.
Specifically, we compare seven methods: (1) SAM, without sample weighting; (2) Loss, using sample losses as sample weights; (3) Confidence, using 1−pw(yi∣xi) as sample weights; (4) qi, using the quantity derived in Theorem 4.1; (5) Normalized qi, applying the same centering, MAD normalization, and clipping as Rad to qi; (6) Shuffled Rad, randomly permuting Rad weights within each batch; and (7) Rad, our proposed weighting scheme. All methods use the same CIFAR-10/ResNet-50 training setup: 200 epochs, a single-GPU batch size of 128, and seed 42. We report the best test accuracy during training. The results are shown below.
Compared with SAM, direct qi weighting improves accuracy, while Rad achieves the highest accuracy among all compared methods.