Chameleon: Learning Anisotropic Perturbation Geometry in Sharpness-Aware Optimization

Sharpness-Aware Minimization (SAM) improves generalization by optimizing the worst-case loss within a neighborhood of model parameters, thereby biasing training toward flatter minima. Despite its effectiveness, standard SAM uses a fixed isotropic perturbation region, which can be mismatched with the highly anisotropic curvature of modern loss landscapes. This geometric mismatch becomes particularly limiting under distribution shifts, where sensitive directions may drift over time and a spherical neighborhood can either over-regularize flat directions or under-explore sharp ones. We propose Chameleon, a lightweight sharpness-aware optimizer that learns an anisotropic perturbation geometry. Chameleon replaces SAM’s fixed-radius ball with a learnable ellipsoidal neighborhood by tensorizing perturbation scales into adaptive parameters. These scales are optimized through a normalized sharpness surrogate that preserves first-order efficiency while encouraging perturbations to align with curvature-sensitive directions. Theoretically, we characterize the learned geometry as a curvature-sensitive axis-aligned metric that coordinate-wise reweights dominant spectral components, establish sublinear convergence under clipped adaptive scales, and derive containment-based PAC-Bayes generalization bounds complemented by a matched-budget curvature analysis. Extensive experiments evaluating multi-modal capabilities across vision, language, and industrial click-through rate (CTR) prediction tasks demonstrate consistent improvements in cross-domain adaptation. Specifically, Chameleon achieves up to 3.91% cross-domain and 2.09% in-domain accuracy gains on vision tasks, 4.19% F1 improvement on language tasks, and 0.71% AUC gains on CTR tasks.