Randomized Manifold Smoothing for Formal Robustness to Semantically Meaningful Perturbations

Model predictions can be sensitive to small, unrelated variations in the input, which questions the reliability of the model. Randomized smoothing provides robustness guarantees against small, adversarial perturbations, giving a certifiably robust predictor for such perturbations. However, these perturbations are applied across all directions of the ambient space, irrespective of the underlying data manifold. Consequently, its guarantees largely cover perturbations of the input that are not seen in reality rather than semantically meaningful variations in the data. To address this, we introduce Randomized Manifold Smoothing, which adapts the smoothing distribution to local data geometry by shaping Gaussian perturbations along principal directions of a locally estimated subspace. This yields anisotropic certificates over locally supported variations while retaining the statistical certification machinery of randomized smoothing. We evaluate manifold smoothing across image classification, semantic segmentation, interpretable concept representations, and vision-language retrieval tasks, including semantic distribution shifts. Across these settings, we show that manifold smoothing consistently preserves task performance while providing stronger robustness guarantees along semantically meaningful perturbations compared to standard randomized smoothing in high-dimensional data and representation spaces.