- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Chameleon, a sharpness-aware optimizer that improves generalization by learning anisotropic perturbation geometry. Unlike standard SAM, which employs a fixed spherical perturbation neighborhood, Chameleon introduces a learnable ellipsoidal neighborhood with coordinate-wise adaptive perturbation scales. These scales are optimized through a normalized sharpness surrogate that implicitly captures curvature information without explicit second-order computations. The authors provide theoretical analyses of the learned perturbation geometry, convergence properties, and PAC-Bayes generalization bounds, along with a matched-budget curvature analysis. Extensive experiments validate the effectiveness of the proposed method.
Critical strengths
- The paper clearly identifies a key limitation of standard SAM: its uniform perturbation radius across parameter directions fails to account for the anisotropic curvature of the loss landscape. Motivated by this observation, the authors introduce a learnable anisotropic perturbation geometry, providing a natural and well-justified extension of SAM. The overall motivation is clear, and the proposed design aligns well with the identified problem.
- The experimental evaluation is comprehensive, covering a wide range of tasks, including image classification, domain generalization, natural language understanding, and click-through rate prediction, across diverse datasets and model architectures. The consistent improvements over existing baselines demonstrate the effectiveness and generalizability of the proposed method across different learning scenarios.
Critical weaknesses
-
The derivation of step (i) in Appendix C.5 is insufficiently justified. The approximation equates a vector-valued quantity with a scalar, without explaining the necessary assumptions. Furthermore, even accepting this approximation, substituting it into (h) appears to yield a denominator of , rather than as written in (i). These inconsistencies undermine the justification of the proposed scale update.
-
The experimental comparisons are insufficient. While several SAM variants are evaluated on vision tasks, experiments on language understanding and CTR prediction only compare against vanilla SAM and ERM. Moreover, most baselines are relatively outdated. Including more recent SAM variants, such as XSAM [1], RSAM [2], and TSAM [3], would help better establish the competitiveness of the proposed method.
[1] Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation, ICLR 2026.
[2] Rényi Sharpness: A Novel Sharpness That Strongly Correlates with Generalization, ICLR 2026.
[3] Tilted Sharpness-Aware Minimization, ICML 2025.
Questions for authors
- Could the authors provide a more rigorous derivation of Equation (17), particularly the transitions from (h) to (i) and from (i) to (j)? What assumptions justify these approximations, and how is the discrepancy in the gradient norm factor explained?
- Could the authors provide comparisons with more recent SAM variants, such as XSAM, RSAM, and TSAM, particularly on language understanding, CTR prediction, and domain generalization benchmarks? These comparisons would help establish the competitiveness of Chameleon beyond standard SAM.
3 comments
I appreciate the authors' thorough response and efforts to address my concerns. Their clarifications have resolved most of my questions, and I will therefore retain my positive assessment of the paper.
We sincerely thank you for the positive assessment and valuable suggestions, which have helped us improve our work. We greatly appreciate your continued support.
Rigorous Scale-Gradient Derivation and Comparisons with Recent SAM Variants
We thank the reviewer for the constructive feedback and address both concerns.
W1&Q1: Derivation of Eq. (17) and scale updates.
We agree that steps (i)–(j) in Appendix C.5 are not rigorously justified. Let v=g0/∥g0∥2, w1=w+ρ⊙v, g1=∇L(w1), H1=∇2L(w1), s=∥g1∥2>0, and ΔL=L1−L0. Holding w,g0 fixed, the exact gradient of J=L1/s is
∇ρJ=v⊙[sg1−s3L1H1g1].
Define the displacement and direction:
δ=ρ⊙v,r=∥δ∥2,u=δ/r.
For a Lipschitz Hessian and small r,
rg1−g0=H1u+O(r).
Replacing H1(g1/s) also requires g1/s≈u, beyond smoothness. Taylor expansion gives
rΔL=g0⊤u+O(r).
Thus, 1/r≈s/ΔL additionally requires g0⊤u≈s, i.e., alignment between the displacement and gradient directions. Coordinate-wise scaling does not generally preserve this alignment. Consequently, the transition from (h) to (i) is conditional, not a general identity. We also agree that the subsequent transition to (j) introduces an inconsistent gradient-norm factor that cannot be absorbed into the learning rate.
For completeness, centering the numerator defines a different objective JΔ=ΔL/s, whose exact gradient is
∇ρJΔ=v⊙[sg1−s3ΔLH1g1].
This alternative does not algebraically recover Eq. (6b). We will correct Appendix C.5, remove the unsupported equivalence, and explicitly characterize Eq. (6b) as a first-order-computable heuristic rather than the exact gradient of J.
We further compare the update with exact HVP-based gradients. The cosine similarity reaches 0.93. On CIFAR-100/CutMix and RTE, the HVP variant achieves 83.04%/81.68%, versus 82.98%/81.59% for Chameleon and 82.49%/80.51% for the damped uncentered variant. The corresponding relative FLOPs are 1.84× for HVP and 1.09× for Chameleon. We will report relative gradient errors and objective changes alongside cosine similarity to characterize the approximation more completely.
W2&Q2: Comparisons with recent SAM variants.
We agree that comparisons beyond vanilla SAM are important. We therefore evaluate three recent methods using matched data splits, training steps, model selection, and hyperparameter search-trial counts, following their official implementations. Specifically, XSAM [1] estimates the direction toward the neighborhood maximum; RSAM [2] regularizes Rényi sharpness; TSAM [3] uses exponential tilting to reweight local losses. In contrast, Chameleon learns parameter-wise, axis-aligned perturbation scales.
Table 1. GLUE (RoBERTa-base), CTR (DeepFM), and relative runtime (SAM=1).
CoLA: MCC; MRPC/RTE: accuracy/F1; STS-B: Pearson/Spearman; CTR: LogLoss/AUC (×100). Lower LogLoss and higher AUC are better.
Table 2. PACS and DomainNet accuracy (%).
PACS: Painting/Cartoon/Photo/Sketch. DomainNet: Clipart/Infograph/Painting/Quickdraw/Real/Sketch. ERM and SAM results are also provided in the manuscript's Tables 2–4.
Chameleon achieves the best reported results across all listed benchmarks. Relative to the strongest recent baseline, gains range from 0.11–0.49 points on GLUE and 0.06–0.61 points on PACS/DomainNet. CTR improvements are smaller but consistent in both LogLoss and AUC. Chameleon also incurs the lowest reported cost (1.08×), compared with 1.26× for XSAM, 1.38× for RSAM, and 2.45× for TSAM. We will clarify the evaluation protocols, computational-cost measurement, and seed-level variability. These results establish competitiveness under the tested settings without implying universal dominance across all architectures.