OrthoSAM: Orthogonal Perturbations for Sharpness-Aware Minimization

Sharpness-Aware Minimization (SAM) improves generalization by minimizing the worst-case loss within a local neighborhood, encouraging the optimizer to find flatter minima. However, standard SAM treats neural network weights as flattened vectors and applies unconstrained perturbations in the Euclidean space, ignoring the intrinsic matrix structure of modern neural networks. As a result, the adversarial perturbation can become dominated by a few large singular directions, while other important directions receive much less attention. In this work, we propose **OrthoSAM**, a matrix-aware optimization method based on the idea that flatness should be explored more evenly across different directions. Instead of using unrestricted perturbations, OrthoSAM constrains perturbations to a structured manifold with orthogonality properties, reducing the imbalance caused by dominant singular values. This encourages a more uniform exploration of the local loss landscape across matrix directions. To make the method efficient in practice, we use the Newton–Schulz iteration to obtain approximately orthogonal perturbations without requiring expensive singular value decomposition (SVD). Extensive experiments on multiple architectures and datasets show that OrthoSAM consistently improves generalization and provides stronger robustness to label noise.