Beyond Likelihood: Robust Multiclass Classification with Transformers

We propose a robust multiclass classification approach based on the rho-estimation framework. An oracle inequality for the resulting estimator controls estimation error when the model is well specified and provides robustness guarantees under model misspecification and data contamination. For i.i.d. data, we further derive an upper bound on the excess misclassification error of the corresponding plug-in classifier. We then explore its implementation with Transformers and establish an approximation result for Hölder functions defined on neighborhoods of low-dimensional manifolds. The construction uses standard self-attention to implement a stable projection and separates network approximation error from the effect of neighborhood thickness. Building on this result, we derive explicit convergence rates for softmax Transformer logit models, with exponents governed by the projection dimension rather than the ambient dimension. Experiments on synthetic and real-world datasets support the theoretical findings.